2024-04-08

Title: SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection

• Authors: Bradley P. Allen, Fina Polat, Paul Groth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03732
• Pdf URL: https://arxiv.org/pdf/2404.03732
• Copy Paste: [[2404.03732]] SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection(https://arxiv.org/abs/2404.03732)
• Keywords: language model, llm, hallucination, prompt
• Abstract: We describe the University of Amsterdam Intelligent Data Engineering Lab team's entry for the SemEval-2024 Task 6 competition. The SHROOM-INDElab system builds on previous work on using prompt programming and in-context learning with large language models (LLMs) to build classifiers for hallucination detection, and extends that work through the incorporation of context-specific definition of task, role, and target concept, and automated generation of examples for use in a few-shot prompting approach. The resulting system achieved fourth-best and sixth-best performance in the model-agnostic track and model-aware tracks for Task 6, respectively, and evaluation using the validation sets showed that the system's classification decisions were consistent with those of the crowd-sourced human labellers. We further found that a zero-shot approach provided better accuracy than a few-shot approach using automatically generated examples. Code for the system described in this paper is available on Github.
• 摘要：我们描述了阿姆斯特丹大学智能数据工程实验室团队参加 SemEval-2024 Task 6 竞赛的情况。 SHROOM-INDElab 系统建立在之前的工作基础上，即使用即时编程和上下文学习与大型语言模型 (LLM) 来构建用于幻觉检测的分类器，并通过结合任务、角色和任务的上下文特定定义来扩展该工作。目标概念，以及自动生成示例以用于几次提示方法。由此产生的系统在任务 6 的模型不可知轨道和模型感知轨道中分别取得了第四好和第六好的性能，并且使用验证集的评估表明系统的分类决策与众包的分类决策一致人类贴标签员。我们进一步发现，使用自动生成的示例，零样本方法比少样本方法提供了更好的准确性。本文所述系统的代码可在 Github 上找到。

Title: PRobELM: Plausibility Ranking Evaluation for Language Models

• Authors: Zhangdie Yuan, Chenxi Whitehouse, Eric Chamoun, Rami Aly, Andreas Vlachos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03818
• Pdf URL: https://arxiv.org/pdf/2404.03818
• Copy Paste: [[2404.03818]] PRobELM: Plausibility Ranking Evaluation for Language Models(https://arxiv.org/abs/2404.03818)
• Keywords: language model, prompt
• Abstract: This paper introduces PRobELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' ability to discern more plausible from less plausible scenarios through their parametric knowledge. While benchmarks such as TruthfulQA emphasise factual accuracy or truthfulness, and others such as COPA explore plausible scenarios without explicitly incorporating world knowledge, PRobELM seeks to bridge this gap by evaluating models' capabilities to prioritise plausible scenarios that leverage world knowledge over less plausible alternatives. This design allows us to assess the potential of language models for downstream use cases such as literature-based discovery where the focus is on identifying information that is likely but not yet known. Our benchmark is constructed from a dataset curated from Wikidata edit histories, tailored to align the temporal bounds of the training data for the evaluated models. PRobELM facilitates the evaluation of language models across multiple prompting types, including statement, text completion, and question-answering. Experiments with 10 models of various sizes and architectures on the relationship between model scales, training recency, and plausibility performance, reveal that factual accuracy does not directly correlate with plausibility performance and that up-to-date training data enhances plausibility assessment across different model architectures.
• 摘要：本文介绍了 PRObELM（语言模型的合理性排名评估），这是一个基准，旨在评估语言模型通过其参数知识辨别更合理场景和不太合理场景的能力。虽然 TruthfulQA 等基准强调事实的准确性或真实性，而 COPA 等其他基准则在没有明确纳入世界知识的情况下探索合理的场景，而 PRObELM 试图通过评估模型的能力来弥合这一差距，以优先考虑利用世界知识的合理场景，而不是不太合理的替代方案。这种设计使我们能够评估语言模型在下游用例中的潜力，例如基于文献的发现，其重点是识别可能但尚未知道的信息。我们的基准是根据维基数据编辑历史整理的数据集构建的，旨在调整评估模型的训练数据的时间范围。 PRObELM 有助于跨多种提示类型评估语言模型，包括陈述、文本完成和问答。对 10 个不同规模和架构的模型进行了关于模型规模、训练新近度和合理性性能之间关系的实验，结果表明事实准确性并不与合理性性能直接相关，而且最新的训练数据增强了不同模型架构之间的合理性评估。

Title: CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialogues

• Authors: Makesh Narsimhan Sreedhar, Traian Rebedea, Shaona Ghosh, Christopher Parisien
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03820
• Pdf URL: https://arxiv.org/pdf/2404.03820
• Copy Paste: [[2404.03820]] CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialogues(https://arxiv.org/abs/2404.03820)
• Keywords: language model, gpt, llm, chat
• Abstract: Recent advancements in instruction-tuning datasets have predominantly focused on specific tasks like mathematical or logical reasoning. There has been a notable gap in data designed for aligning language models to maintain topic relevance in conversations - a critical aspect for deploying chatbots to production. We introduce the CantTalkAboutThis dataset to help language models remain focused on the subject at hand during task-oriented interactions. It consists of synthetic dialogues on a wide range of conversation topics from different domains. These dialogues are interspersed with distractor turns that intentionally divert the chatbot from the predefined topic. Fine-tuning language models on this dataset helps make them resilient to deviating from the role assigned and improves their ability to maintain topical coherence compared to general-purpose instruction-tuned LLMs like GPT-4-turbo and Mixtral-Instruct. Additionally, preliminary observations suggest that training models on this dataset also enhance their performance on fine-grained instruction following tasks.
• 摘要：指令调整数据集的最新进展主要集中在数学或逻辑推理等特定任务上。旨在调整语言模型以保持对话中主题相关性的数据存在显着差距——这是将聊天机器人部署到生产中的一个关键方面。我们引入 CantTalkAboutThis 数据集来帮助语言模型在面向任务的交互过程中保持关注当前的主题。它由来自不同领域的广泛对话主题的综合对话组成。这些对话中穿插着一些干扰性的对话，有意将聊天机器人从预定义的主题上转移开。与 GPT-4-turbo 和 Mixtral-Instruct 等通用指令调整的 LLM 相比，在此数据集上微调语言模型有助于使它们能够适应偏离分配的角色，并提高其保持主题连贯性的能力。此外，初步观察表明，该数据集上的训练模型还可以提高其在细粒度指令跟踪任务中的性能。

Title: Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data

• Authors: Jingyu Zhang, Marc Marone, Tianjian Li, Benjamin Van Durme, Daniel Khashabi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03862
• Pdf URL: https://arxiv.org/pdf/2404.03862
• Copy Paste: [[2404.03862]] Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data(https://arxiv.org/abs/2404.03862)
• Keywords: language model, llm
• Abstract: For humans to trust the fluent generations of large language models (LLMs), they must be able to verify their correctness against trusted, external sources. Recent efforts aim to increase verifiability through citations of retrieved documents or post-hoc provenance. However, such citations are prone to mistakes that further complicate their verifiability. To address these limitations, we tackle the verifiability goal with a different philosophy: we trivialize the verification process by developing models that quote verbatim statements from trusted sources in pre-training data. We propose Quote-Tuning, which demonstrates the feasibility of aligning LLMs to leverage memorized information and quote from pre-training data. Quote-Tuning quantifies quoting against large corpora with efficient membership inference tools, and uses the amount of quotes as an implicit reward signal to construct a synthetic preference dataset for quoting, without any human annotation. Next, the target model is aligned to quote using preference optimization algorithms. Experimental results show that Quote-Tuning significantly increases the percentage of LLM generation quoted verbatim from high-quality pre-training documents by 55% to 130% relative to untuned models while maintaining response quality. Further experiments demonstrate that Quote-Tuning generalizes quoting to out-of-domain data, is applicable in different tasks, and provides additional benefits to truthfulness. Quote-Tuning not only serves as a hassle-free method to increase quoting but also opens up avenues for improving LLM trustworthiness through better verifiability.
• 摘要：为了让人类信任大型语言模型 (LLM) 的流畅生成，他们必须能够根据可信的外部来源验证其正确性。最近的努力旨在通过引用检索到的文件或事后出处来提高可验证性。然而，此类引文很容易出现错误，从而使其可验证性进一步复杂化。为了解决这些限制，我们用不同的理念来解决可验证性目标：我们通过开发在预训练数据中引用来自可信来源的逐字陈述的模型来简化验证过程。我们提出了 Quote-Tuning，它展示了调整法学硕士以利用记忆信息和来自预训练数据的引用的可行性。 Quote-Tuning 使用高效的成员推理工具对大型语料库的引用进行量化，并使用引用数量作为隐式奖励信号来构建用于引用的综合偏好数据集，而无需任何人工注释。接下来，使用偏好优化算法将目标模型对齐以进行报价。实验结果表明，与未调整的模型相比，Quote-Tuning 显着提高了 LLM 生成逐字引用高质量预训练文档的百分比，提高了 55% 至 130%，同时保持了响应质量。进一步的实验表明，引用调优将引用推广到域外数据，适用于不同的任务，并为真实性提供了额外的好处。报价调整不仅可以作为一种增加报价的无忧方法，而且还为通过更好的可验证性来提高 LLM 可信度开辟了途径。

Title: FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping

• Authors: Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Shiwei Liu, Tianlong Chen, Aditya Akella
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03865
• Pdf URL: https://arxiv.org/pdf/2404.03865
• Copy Paste: [[2404.03865]] FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping(https://arxiv.org/abs/2404.03865)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges for autoregressive token-by-token generation. To mitigate computation overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. Despite some promising success due to the redundancy across LLMs layers on metrics like Rough-L/BLUE, our careful knowledge-intensive evaluation unveils issues such as generation collapse, hallucination of wrong facts, and noticeable performance drop even at the trivial exit ratio of 10-15% of layers. We attribute these errors primarily to ineffective handling of the KV cache through state copying during early-exit. In this work, we observed the saturation of computationally expensive feed-forward blocks of LLM layers and proposed FFN-SkipLLM, which is a novel fine-grained skip strategy of autoregressive LLMs. More specifically, FFN-SkipLLM is an input-adaptive feed-forward skipping strategy that can skip 25-30% of FFN blocks of LLMs with marginal change in performance on knowledge-intensive generation tasks without any requirement to handle KV cache. Our extensive experiments and ablation across benchmarks like MT-Bench, Factoid-QA, and variable-length text summarization illustrate how our simple and ease-at-use method can facilitate faster autoregressive decoding.
• 摘要：自回归大型语言模型（例如 LLaMa、GPT）无处不在，在语言理解和生成方面取得了显着的成功。然而，如此令人印象深刻的功能通常伴随着巨大的模型大小，这对逐个令牌的自回归生成提出了重大挑战。为了减轻生成过程中发生的计算过载，已经提出了几种提前退出和分层策略。尽管由于法学硕士层在 Rough-L/BLUE 等指标上的冗余而取得了一些有希望的成功，但我们仔细的知识密集型评估揭示了诸如世代崩溃、错误事实的幻觉以及即使在 10 的微不足道的退出率下性能也明显下降等问题。 -15% 的层数。我们将这些错误主要归因于提前退出期间通过状态复制对 KV 缓存的无效处理。在这项工作中，我们观察了 LLM 层计算成本高昂的前馈块的饱和，并提出了 FFN-SkipLLM，这是一种新型的自回归 LLM 细粒度跳跃策略。更具体地说，FFN-SkipLLM 是一种输入自适应前馈跳跃策略，可以跳过 LLM 的 25-30% FFN 块，在知识密集型生成任务上的性能略有变化，而无需处理 KV 缓存。我们在 MT-Bench、Factoid-QA 和可变长度文本摘要等基准测试中进行了大量实验和消融，说明了我们简单易用的方法如何促进更快的自回归解码。

Title: Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction

• Authors: Bowen Zhang, Harold Soh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03868
• Pdf URL: https://arxiv.org/pdf/2404.03868
• Copy Paste: [[2404.03868]] Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction(https://arxiv.org/abs/2404.03868)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: In this work, we are interested in automated methods for knowledge graph creation (KGC) from input text. Progress on large language models (LLMs) has prompted a series of recent works applying them to KGC, e.g., via zero/few-shot prompting. Despite successes on small domain-specific datasets, these models face difficulties scaling up to text common in many real-world applications. A principal issue is that in prior methods, the KG schema has to be included in the LLM prompt to generate valid triplets; larger and more complex schema easily exceed the LLMs' context window length. To address this problem, we propose a three-phase framework named Extract-Define-Canonicalize (EDC): open information extraction followed by schema definition and post-hoc canonicalization. EDC is flexible in that it can be applied to settings where a pre-defined target schema is available and when it is not; in the latter case, it constructs a schema automatically and applies self-canonicalization. To further improve performance, we introduce a trained component that retrieves schema elements relevant to the input text; this improves the LLMs' extraction performance in a retrieval-augmented generation-like manner. We demonstrate on three KGC benchmarks that EDC is able to extract high-quality triplets without any parameter tuning and with significantly larger schemas compared to prior works.
• 摘要：在这项工作中，我们对从输入文本创建知识图（KGC）的自动化方法感兴趣。大语言模型 (LLM) 的进展促使最近一系列将其应用于 KGC 的工作，例如通过零/少样本提示。尽管在小型特定领域数据集上取得了成功，但这些模型在扩展到许多实际应用程序中常见的文本时面临着困难。一个主要问题是，在以前的方法中，KG 模式必须包含在 LLM 提示中才能生成有效的三元组；更大、更复杂的模式很容易超出法学硕士的上下文窗口长度。为了解决这个问题，我们提出了一个名为提取-定义-规范化（EDC）的三阶段框架：开放信息提取，然后是模式定义和事后规范化。 EDC 非常灵活，因为它可以应用于预定义目标模式可用和不可用的设置；在后一种情况下，它会自动构建模式并应用自我规范化。为了进一步提高性能，我们引入了一个经过训练的组件，用于检索与输入文本相关的模式元素；这以类似检索增强生成的方式提高了法学硕士的提取性能。我们在三个 KGC 基准测试中证明，EDC 能够在不进行任何参数调整的情况下提取高质量的三元组，并且与之前的工作相比，其模式要大得多。

Title: SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models

• Authors: Hyeonwoo Kim, Gyoungjin Gim, Yungi Kim, Jihoo Kim, Byungju Kim, Wonseok Lee, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03887
• Pdf URL: https://arxiv.org/pdf/2404.03887
• Copy Paste: [[2404.03887]] SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models(https://arxiv.org/abs/2404.03887)
• Keywords: language model, llm, chain-of-thought
• Abstract: This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.
• 摘要：这项研究提出了一种新颖的学习方法，旨在增强大型语言模型（LLM）的数学推理和解决问题的能力。我们专注于整合思维链（CoT）和思维程序（PoT）学习，假设优先学习数学推理能力有助于放大解决问题的能力。因此，CoT 的初始学习对于解决具有挑战性的数学问题至关重要。为此，我们提出了一种顺序学习方法，称为SAAS（解决能力放大策略），策略性地从CoT学习过渡到PoT学习。我们的实证研究涉及使用多个基准进行广泛的性能比较，表明我们的 SAAS 实现了最先进的 (SOTA) 性能。结果强调了我们顺序学习方法的有效性，标志着法学硕士数学推理领域的重大进步。

Title: Forget NLI, Use a Dictionary: Zero-Shot Topic Classification for Low-Resource Languages with Application to Luxembourgish

• Authors: Fred Philippy, Shohreh Haddadan, Siwen Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03912
• Pdf URL: https://arxiv.org/pdf/2404.03912
• Copy Paste: [[2404.03912]] Forget NLI, Use a Dictionary: Zero-Shot Topic Classification for Low-Resource Languages with Application to Luxembourgish(https://arxiv.org/abs/2404.03912)
• Keywords: language model
• Abstract: In NLP, zero-shot classification (ZSC) is the task of assigning labels to textual data without any labeled examples for the target classes. A common method for ZSC is to fine-tune a language model on a Natural Language Inference (NLI) dataset and then use it to infer the entailment between the input document and the target labels. However, this approach faces certain challenges, particularly for languages with limited resources. In this paper, we propose an alternative solution that leverages dictionaries as a source of data for ZSC. We focus on Luxembourgish, a low-resource language spoken in Luxembourg, and construct two new topic relevance classification datasets based on a dictionary that provides various synonyms, word translations and example sentences. We evaluate the usability of our dataset and compare it with the NLI-based approach on two topic classification tasks in a zero-shot manner. Our results show that by using the dictionary-based dataset, the trained models outperform the ones following the NLI-based approach for ZSC. While we focus on a single low-resource language in this study, we believe that the efficacy of our approach can also transfer to other languages where such a dictionary is available.
• 摘要：在 NLP 中，零样本分类 (ZSC) 是为文本数据分配标签的任务，而无需目标类的任何标记示例。 ZSC 的常见方法是在自然语言推理 (NLI) 数据集上微调语言模型，然后使用它来推断输入文档和目标标签之间的蕴涵。然而，这种方法面临着一定的挑战，特别是对于资源有限的语言。在本文中，我们提出了一种替代解决方案，利用字典作为 ZSC 的数据源。我们关注卢森堡语（卢森堡语）这种资源匮乏的语言，并基于提供各种同义词、单词翻译和例句的词典构建了两个新的主题相关性分类数据集。我们评估数据集的可用性，并以零样本方式将其与基于 NLI 的方法在两个主题分类任务上进行比较。我们的结果表明，通过使用基于字典的数据集，训练后的模型优于采用基于 NLI 的 ZSC 方法的模型。虽然我们在这项研究中专注于单一的低资源语言，但我们相信我们的方法的功效也可以转移到有此类词典的其他语言。

Title: Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

• Authors: Bowen Zhang, Kehua Chang, Chunping Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03921
• Pdf URL: https://arxiv.org/pdf/2404.03921
• Copy Paste: [[2404.03921]] Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models(https://arxiv.org/abs/2404.03921)
• Keywords: language model, prompt
• Abstract: Sentence Embedding stands as a fundamental task within the realm of Natural Language Processing, finding extensive application in search engines, expert systems, and question-and-answer platforms. With the continuous evolution of large language models such as LLaMA and Mistral, research on sentence embedding has recently achieved notable breakthroughs. However, these advancements mainly pertain to fine-tuning scenarios, leaving explorations into computationally efficient direct inference methods for sentence representation in a nascent stage. This paper endeavors to bridge this research gap. Through comprehensive experimentation, we challenge the widely held belief in the necessity of an Explicit One-word Limitation for deriving sentence embeddings from Pre-trained Language Models (PLMs). We demonstrate that this approach, while beneficial for generative models under direct inference scenario, is not imperative for discriminative models or the fine-tuning of generative PLMs. This discovery sheds new light on the design of manual templates in future studies. Building upon this insight, we propose two innovative prompt engineering techniques capable of further enhancing the expressive power of PLMs' raw embeddings: Pretended Chain of Thought and Knowledge Enhancement. We confirm their effectiveness across various PLM types and provide a detailed exploration of the underlying factors contributing to their success.
• 摘要：句子嵌入是自然语言处理领域的一项基本任务，在搜索引擎、专家系统和问答平台中得到广泛应用。随着LLaMA、Mistral等大型语言模型的不断演进，句子嵌入的研究最近取得了显着的突破。然而，这些进步主要与微调场景有关，使得对句子表示的计算有效的直接推理方法的探索仍处于初级阶段。本文致力于弥合这一研究空白。通过全面的实验，我们挑战了人们普遍认为从预训练语言模型 (PLM) 中导出句子嵌入时需要使用显式单字限制的信念。我们证明，这种方法虽然有利于直接推理场景下的生成模型，但对于判别模型或生成 PLM 的微调来说并不是必需的。这一发现为未来研究中手动模板的设计提供了新的思路。基于这一见解，我们提出了两种创新的即时工程技术，能够进一步增强 PLM 原始嵌入的表达能力：假想的思维链和知识增强。我们确认了它们在各种 PLM 类型中的有效性，并详细探讨了促成其成功的潜在因素。

Title: Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving

• Authors: Gulsum Yigit, Mehmet Fatih Amasyali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03938
• Pdf URL: https://arxiv.org/pdf/2404.03938
• Copy Paste: [[2404.03938]] Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving(https://arxiv.org/abs/2404.03938)
• Keywords: language model, prompt
• Abstract: Math Word Problem (MWP) solving presents a challenging task in Natural Language Processing (NLP). This study aims to provide MWP solvers with a more diverse training set, ultimately improving their ability to solve various math problems. We propose several methods for data augmentation by modifying the problem texts and equations, such as synonym replacement, rule-based: question replacement, and rule based: reversing question methodologies over two English MWP datasets. This study extends by introducing a new in-context learning augmentation method, employing the Llama-7b language model. This approach involves instruction-based prompting for rephrasing the math problem texts. Performance evaluations are conducted on 9 baseline models, revealing that augmentation methods outperform baseline models. Moreover, concatenating examples generated by various augmentation methods further improves performance.
• 摘要：数学应用题 (MWP) 解决在自然语言处理 (NLP) 中提出了一项具有挑战性的任务。这项研究旨在为 MWP 求解者提供更加多样化的训练集，最终提高他们解决各种数学问题的能力。我们通过修改问题文本和方程提出了几种数据增强方法，例如同义词替换、基于规则的：问题替换和基于规则的：在两个英语 MWP 数据集上反转问题方法。这项研究通过引入一种新的情境学习增强方法（采用 Llama-7b 语言模型）进行了扩展。这种方法涉及基于指令的提示来重新表述数学问题文本。对 9 个基线模型进行了性能评估，结果表明增强方法优于基线模型。此外，连接由各种增强方法生成的示例进一步提高了性能。

Title: SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials

• Authors: Mathilde Aguiar, Pierre Zweigenbaum, Nona Naderi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03977
• Pdf URL: https://arxiv.org/pdf/2404.03977
• Copy Paste: [[2404.03977]] SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.03977)
• Keywords: language model, prompt, chain-of-thought
• Abstract: This paper describes our submission to Task 2 of SemEval-2024: Safe Biomedical Natural Language Inference for Clinical Trials. The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) consists of a Textual Entailment (TE) task focused on the evaluation of the consistency and faithfulness of Natural Language Inference (NLI) models applied to Clinical Trial Reports (CTR). We test 2 distinct approaches, one based on finetuning and ensembling Masked Language Models and the other based on prompting Large Language Models using templates, in particular, using Chain-Of-Thought and Contrastive Chain-Of-Thought. Prompting Flan-T5-large in a 2-shot setting leads to our best system that achieves 0.57 F1 score, 0.64 Faithfulness, and 0.56 Consistency.
• 摘要：本文介绍了我们向 SemEval-2024 任务 2 提交的内容：临床试验的安全生物医学自然语言推理。临床试验数据的多证据自然语言推理 (NLI4CT) 由文本蕴涵 (TE) 任务组成，重点评估应用于临床试验报告 (CTR) 的自然语言推理 (NLI) 模型的一致性和可信度。我们测试了两种不同的方法，一种基于微调和集成屏蔽语言模型，另一种基于使用模板提示大型语言模型，特别是使用思想链和对比思想链。在 2 次设置中提示 Flan-T5-large 会导致我们最好的系统达到 0.57 F1 分数、0.64 忠实度和 0.56 一致性。

Title: Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study

• Authors: Myrthe Reuver, Suzan Verberne, Antske Fokkens
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03987
• Pdf URL: https://arxiv.org/pdf/2404.03987
• Copy Paste: [[2404.03987]] Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study(https://arxiv.org/abs/2404.03987)
• Keywords: llm
• Abstract: For a viewpoint-diverse news recommender, identifying whether two news articles express the same viewpoint is essential. One way to determine "same or different" viewpoint is stance detection. In this paper, we investigate the robustness of operationalization choices for few-shot stance detection, with special attention to modelling stance across different topics. Our experiments test pre-registered hypotheses on stance detection. Specifically, we compare two stance task definitions (Pro/Con versus Same Side Stance), two LLM architectures (bi-encoding versus cross-encoding), and adding Natural Language Inference knowledge, with pre-trained RoBERTa models trained with shots of 100 examples from 7 different stance detection datasets. Some of our hypotheses and claims from earlier work can be confirmed, while others give more inconsistent results. The effect of the Same Side Stance definition on performance differs per dataset and is influenced by other modelling choices. We found no relationship between the number of training topics in the training shots and performance. In general, cross-encoding out-performs bi-encoding, and adding NLI training to our models gives considerable improvement, but these results are not consistent across all datasets. Our results indicate that it is essential to include multiple datasets and systematic modelling experiments when aiming to find robust modelling choices for the concept `stance'.
• 摘要：对于观点多样化的新闻推荐者来说，识别两篇新闻文章是否表达相同的观点至关重要。确定“相同或不同”观点的一种方法是姿态检测。在本文中，我们研究了少镜头姿态检测的操作化选择的稳健性，特别关注跨不同主题的姿态建模。我们的实验测试了预先注册的姿态检测假设。具体来说，我们比较了两种立场任务定义（赞成/反对与同方立场）、两种 LLM 架构（双编码与交叉编码），并添加自然语言推理知识，以及使用 100 个示例镜头训练的预训练 RoBERTa 模型来自 7 个不同姿态检测数据集。我们早期工作中的一些假设和主张可以得到证实，而另一些则给出了更加不一致的结果。同侧姿态定义对性能的影响因数据集而异，并且受到其他建模选择的影响。我们发现训练镜头中训练主题的数量与表现之间没有关系。一般来说，交叉编码的性能优于双向编码，并且在我们的模型中添加 NLI 训练可以带来相当大的改进，但这些结果在所有数据集上并不一致。我们的结果表明，当旨在为概念“立场”找到稳健的建模选择时，必须包含多个数据集和系统建模实验。

Title: BuDDIE: A Business Document Dataset for Multi-task Information Extraction

• Authors: Ran Zmigrod, Dongsheng Wang, Mathieu Sibue, Yulong Pei, Petr Babkin, Ivan Brugere, Xiaomo Liu, Nacho Navarro, Antony Papadimitriou, William Watson, Zhiqiang Ma, Armineh Nourbakhsh, Sameena Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04003
• Pdf URL: https://arxiv.org/pdf/2404.04003
• Copy Paste: [[2404.04003]] BuDDIE: A Business Document Dataset for Multi-task Information Extraction(https://arxiv.org/abs/2404.04003)
• Keywords: language model
• Abstract: The field of visually rich document understanding (VRDU) aims to solve a multitude of well-researched NLP tasks in a multi-modal domain. Several datasets exist for research on specific tasks of VRDU such as document classification (DC), key entity extraction (KEE), entity linking, visual question answering (VQA), inter alia. These datasets cover documents like invoices and receipts with sparse annotations such that they support one or two co-related tasks (e.g., entity extraction and entity linking). Unfortunately, only focusing on a single specific of documents or task is not representative of how documents often need to be processed in the wild - where variety in style and requirements is expected. In this paper, we introduce BuDDIE (Business Document Dataset for Information Extraction), the first multi-task dataset of 1,665 real-world business documents that contains rich and dense annotations for DC, KEE, and VQA. Our dataset consists of publicly available business entity documents from US state government websites. The documents are structured and vary in their style and layout across states and types (e.g., forms, certificates, reports, etc.). We provide data variety and quality metrics for BuDDIE as well as a series of baselines for each task. Our baselines cover traditional textual, multi-modal, and large language model approaches to VRDU.
• 摘要：视觉丰富的文档理解（VRDU）领域旨在解决多模态领域中大量经过深入研究的 NLP 任务。有几个数据集用于研究 VRDU 的特定任务，例如文档分类 (DC)、关键实体提取 (KEE)、实体链接、视觉问答 (VQA) 等。这些数据集涵盖带有稀疏注释的发票和收据等文档，以便它们支持一两个相关任务（例如，实体提取和实体链接）。不幸的是，仅关注单个特定的文档或任务并不能代表文档通常需要如何在野外处理 - 在这种情况下，期望有多种风格和要求。在本文中，我们介绍了 BuDDIE（用于信息提取的业务文档数据集），这是第一个包含 1,665 个真实业务文档的多任务数据集，其中包含 DC、KEE 和 VQA 的丰富而密集的注释。我们的数据集包含来自美国州政府网站的公开商业实体文件。这些文件的结构、风格和布局因州和类型而异（例如表格、证书、报告等）。我们为 BuDDIE 提供数据多样性和质量指标，以及每项任务的一系列基线。我们的基线涵盖 VRDU 的传统文本、多模式和大型语言模型方法。

Title: Willkommens-Merkel, Chaos-Johnson, and Tore-Klose: Modeling the Evaluative Meaning of German Personal Name Compounds

• Authors: Annerose Eichel, Tana Deeg, André Blessing, Milena Belosevic, Sabine Arndt-Lappe, Sabine Schulte im Walde
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.04031
• Pdf URL: https://arxiv.org/pdf/2404.04031
• Copy Paste: [[2404.04031]] Willkommens-Merkel, Chaos-Johnson, and Tore-Klose: Modeling the Evaluative Meaning of German Personal Name Compounds(https://arxiv.org/abs/2404.04031)
• Keywords: language model
• Abstract: We present a comprehensive computational study of the under-investigated phenomenon of personal name compounds (PNCs) in German such as Willkommens-Merkel ('Welcome-Merkel'). Prevalent in news, social media, and political discourse, PNCs are hypothesized to exhibit an evaluative function that is reflected in a more positive or negative perception as compared to the respective personal full name (such as Angela Merkel). We model 321 PNCs and their corresponding full names at discourse level, and show that PNCs bear an evaluative nature that can be captured through a variety of computational methods. Specifically, we assess through valence information whether a PNC is more positively or negatively evaluative than the person's name, by applying and comparing two approaches using (i) valence norms and (ii) pretrained language models (PLMs). We further enrich our data with personal, domain-specific, and extra-linguistic information and perform a range of regression analyses revealing that factors including compound and modifier valence, domain, and political party membership influence how a PNC is evaluated.
• 摘要：我们对德国人名复合词 (PNC) 中未得到充分调查的现象进行了全面的计算研究，例如 Willkommens-Merkel（“欢迎-默克尔”）。 PNC 在新闻、社交媒体和政治话语中普遍存在，据推测，PNC 表现出一种评价功能，与各自的个人全名（例如安吉拉·默克尔）相比，这种功能反映在更积极或消极的看法中。我们在话语层面对 321 个 PNC 及其相应的全名进行建模，并表明 PNC 具有可以通过各种计算方法捕获的评价性质。具体来说，我们通过应用和比较两种方法（使用（i）效价规范和（ii）预训练语言模型（PLM）），通过效价信息评估 PNC 是否比人名更具积极性或消极性评价。我们进一步利用个人、特定领域和语言外信息丰富我们的数据，并进行一系列回归分析，揭示复合和修饰价、领域和政党成员资格等因素影响 PNC 的评估方式。

Title: Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer

• Authors: Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, Mark Fishel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04042
• Pdf URL: https://arxiv.org/pdf/2404.04042
• Copy Paste: [[2404.04042]] Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer(https://arxiv.org/abs/2404.04042)
• Keywords: language model, llm
• Abstract: This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named \textsc{Llammas}, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.
• 摘要：本文探讨了使预训练大型语言模型 (LLM) 适应新的低资源语言的经济高效方法，特别关注爱沙尼亚语。利用 Llama 2 模型，我们研究了跨语言指令调整与额外的单语预训练相结合的影响。我们的结果表明，即使是相对少量的额外单语预训练，然后进行跨语言教学调整，也能显着提高爱沙尼亚语的结果。此外，我们展示了从高质量英语指令到爱沙尼亚语的跨语言知识转移，从而提高了常识推理和多轮对话能力。我们最好的模型名为 \textsc{Llammas}，代表了爱沙尼亚语的第一个开源指令遵循法学硕士。此外，我们还发布了 Alpaca-est，这是爱沙尼亚的第一个通用任务指令数据集。这些贡献标志着爱沙尼亚语开源法学硕士方向的初步进展。

Title: CLUE: A Clinical Language Understanding Evaluation for LLMs

• Authors: Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Koraş, Constantin Marc Seibold, Kaleb E Smith, Jens Kleesiek
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04067
• Pdf URL: https://arxiv.org/pdf/2404.04067
• Copy Paste: [[2404.04067]] CLUE: A Clinical Language Understanding Evaluation for LLMs(https://arxiv.org/abs/2404.04067)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown the potential to significantly contribute to patient care, diagnostics, and administrative processes. Emerging biomedical LLMs address healthcare-specific challenges, including privacy demands and computational constraints. However, evaluation of these models has primarily been limited to non-clinical tasks, which do not reflect the complexity of practical clinical applications. Additionally, there has been no thorough comparison between biomedical and general-domain LLMs for clinical tasks. To fill this gap, we present the Clinical Language Understanding Evaluation (CLUE), a benchmark tailored to evaluate LLMs on real-world clinical tasks. CLUE includes two novel datasets derived from MIMIC IV discharge letters and four existing tasks designed to test the practical applicability of LLMs in healthcare settings. Our evaluation covers several biomedical and general domain LLMs, providing insights into their clinical performance and applicability. CLUE represents a step towards a standardized approach to evaluating and developing LLMs in healthcare to align future model development with the real-world needs of clinical application. We publish our evaluation and data generation scripts: https://github.com/dadaamin/CLUE
• 摘要：大型语言模型 (LLM) 已显示出对患者护理、诊断和管理流程做出重大贡献的潜力。新兴的生物医学法学硕士解决了特定于医疗保健的挑战，包括隐私要求和计算限制。然而，这些模型的评估主要局限于非临床任务，不能反映实际临床应用的复杂性。此外，生物医学法学硕士和通用领域法学硕士在临床任务方面还没有进行彻底的比较。为了填补这一空白，我们提出了临床语言理解评估（CLUE），这是一个专门用于评估法学硕士在现实世界临床任务中的基准。 CLUE 包括两个源自 MIMIC IV 出院信的新颖数据集和四个旨在测试法学硕士在医疗保健环境中的实际适用性的现有任务。我们的评估涵盖多个生物医学和一般领域法学硕士，提供对其临床表现和适用性的见解。 CLUE 代表了朝着评估和开发医疗保健领域法学硕士的标准化方法迈出的一步，以使未来的模型开发与临床应用的实际需求保持一致。我们发布评估和数据生成脚本：https://github.com/dadaamin/CLUE

Title: Assessing the quality of information extraction

• Authors: Filip Seitl, Tomáš Kovářík, Soheyla Mirshahi, Jan Kryštůfek, Rastislav Dujava, Matúš Ondreička, Herbert Ullrich, Petr Gronat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04068
• Pdf URL: https://arxiv.org/pdf/2404.04068
• Copy Paste: [[2404.04068]] Assessing the quality of information extraction(https://arxiv.org/abs/2404.04068)
• Keywords: language model
• Abstract: Advances in large language models have notably enhanced the efficiency of information extraction from unstructured and semi-structured data sources. As these technologies become integral to various applications, establishing an objective measure for the quality of information extraction becomes imperative. However, the scarcity of labeled data presents significant challenges to this endeavor. In this paper, we introduce an automatic framework to assess the quality of the information extraction and its completeness. The framework focuses on information extraction in the form of entity and its properties. We discuss how to handle the input/output size limitations of the large language models and analyze their performance when iteratively extracting the information. Finally, we introduce metrics to evaluate the quality of the extraction and provide an extensive discussion on how to interpret the metrics.
• 摘要：大型语言模型的进步显着提高了从非结构化和半结构化数据源中提取信息的效率。随着这些技术成为各种应用程序不可或缺的一部分，建立信息提取质量的客观衡量标准变得势在必行。然而，标记数据的稀缺给这一努力带来了重大挑战。在本文中，我们介绍了一个自动框架来评估信息提取的质量及其完整性。该框架侧重于实体及其属性形式的信息提取。我们讨论如何处理大型语言模型的输入/输出大小限制，并在迭代提取信息时分析其性能。最后，我们引入了评估提取质量的指标，并就如何解释指标进行了广泛的讨论。

Title: BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models

• Authors: Jacek Wiland, Max Ploner, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04113
• Pdf URL: https://arxiv.org/pdf/2404.04113
• Copy Paste: [[2404.04113]] BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models(https://arxiv.org/abs/2404.04113)
• Keywords: language model
• Abstract: Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurations. However, previous approaches rely on the objective function used in pre-training LMs and are thus applicable only to masked or causal LMs. As a result, comparing different types of LMs becomes impossible. To address this, we propose an approach that uses an LM's inherent ability to estimate the log-likelihood of any given textual statement. We carefully design an evaluation dataset of 7,731 instances (40,916 in a larger variant) from which we produce alternative statements for each relational fact, one of which is correct. We then evaluate whether an LM correctly assigns the highest log-likelihood to the correct statement. Our experimental evaluation of 22 common LMs shows that our proposed framework, BEAR, can effectively probe for knowledge across different LM types. We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs.
• 摘要：知识探索评估语言模型（LM）在预训练期间成功学习关系知识的程度。探测是比较不同规模和训练配置的 LM 的一种廉价方法。然而，以前的方法依赖于预训练 LM 中使用的目标函数，因此仅适用于屏蔽或因果 LM。因此，比较不同类型的语言模型变得不可能。为了解决这个问题，我们提出了一种方法，利用 LM 的固有能力来估计任何给定文本语句的对数似然。我们精心设计了一个包含 7,731 个实例（较大变体中为 40,916 个）的评估数据集，从中我们为每个关系事实生成替代陈述，其中一个是正确的。然后，我们评估 LM 是否正确地将最高对数似然分配给正确的语句。我们对 22 种常见 LM 的实验评估表明，我们提出的框架 BEAR 可以有效地探索不同 LM 类型的知识。我们向研究社区发布了 BEAR 数据集和一个开源框架，该框架实施探测方法，以促进 LM 的评估和开发。

Title: Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

• Authors: Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04167
• Pdf URL: https://arxiv.org/pdf/2404.04167
• Copy Paste: [[2404.04167]] Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model(https://arxiv.org/abs/2404.04167)
• Keywords: language model, llm
• Abstract: In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.
• 摘要：在本研究中，我们介绍了 CT-LLM，这是一种 2B 大语言模型 (LLM)，它说明了在开发 LLM 时向优先考虑中文的关键转变。 CT-LLM独特地从零开始，与传统方法不同，主要纳入中文文本数据，利用12000亿个令牌的广泛语料库，其中8000亿个中文令牌、3000亿个英文令牌和1000亿个代码令牌。这种战略组合促进了模型在理解和处理中文方面的卓越能力，通过对齐技术进一步增强了这种能力。 CT-LLM在CHC-Bench上表现出色，在中文任务中表现出色，并通过SFT展示了其对英语的熟练程度。这项研究挑战了主要以英语语料库培训法学硕士，然后将其适应其他语言的主流模式，拓宽了法学硕士培训方法的视野。通过开源中文法学硕士培养的全流程，包括使用获得的海量合适预训练中文语料库（MAP-CC）、精心挑选的多学科中文硬案例基准（CHC-Bench）和2B规模的中文Tiny LLM（CT-LLM），我们的目标是促进学术界和工业界的进一步探索和创新，为更具包容性和多功能的语言模型铺平道路。

Title: Do Sentence Transformers Learn Quasi-Geospatial Concepts from General Text?

• Authors: Ilya Ilyankou, Aldo Lipani, Stefano Cavazzi, Xiaowei Gao, James Haworth
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04169
• Pdf URL: https://arxiv.org/pdf/2404.04169
• Copy Paste: [[2404.04169]] Do Sentence Transformers Learn Quasi-Geospatial Concepts from General Text?(https://arxiv.org/abs/2404.04169)
• Keywords: language model
• Abstract: Sentence transformers are language models designed to perform semantic search. This study investigates the capacity of sentence transformers, fine-tuned on general question-answering datasets for asymmetric semantic search, to associate descriptions of human-generated routes across Great Britain with queries often used to describe hiking experiences. We find that sentence transformers have some zero-shot capabilities to understand quasi-geospatial concepts, such as route types and difficulty, suggesting their potential utility for routing recommendation systems.
• 摘要：句子转换器是设计用于执行语义搜索的语言模型。这项研究调查了句子转换器的能力，在用于非对称语义搜索的一般问答数据集上进行了微调，将人类生成的穿越英国路线的描述与经常用于描述徒步旅行体验的查询相关联。我们发现句子转换器具有一些零样本能力来理解准地理空间概念，例如路线类型和难度，这表明它们对于路线推荐系统的潜在效用。

Title: Social Skill Training with Large Language Models

• Authors: Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S. Bernstein, John Mitchell
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.04204
• Pdf URL: https://arxiv.org/pdf/2404.04204
• Copy Paste: [[2404.04204]] Social Skill Training with Large Language Models(https://arxiv.org/abs/2404.04204)
• Keywords: language model
• Abstract: People rely on social skills like conflict resolution to communicate effectively and to thrive in both work and personal life. However, practice environments for social skills are typically out of reach for most people. How can we make social skill training more available, accessible, and inviting? Drawing upon interdisciplinary research from communication and psychology, this perspective paper identifies social skill barriers to enter specialized fields. Then we present a solution that leverages large language models for social skill training via a generic framework. Our AI Partner, AI Mentor framework merges experiential learning with realistic practice and tailored feedback. This work ultimately calls for cross-disciplinary innovation to address the broader implications for workforce development and social equality.
• 摘要：人们依靠解决冲突等社交技能来有效沟通并在工作和个人生活中蓬勃发展。然而，社交技能的练习环境对于大多数人来说通常是遥不可及的。我们如何才能使社交技能培训变得更容易获得、更容易获得、更有吸引力？这篇观点论文借鉴了传播学和心理学的跨学科研究，确定了进入专业领域的社交技能障碍。然后，我们提出了一个解决方案，通过通用框架利用大型语言模型进行社交技能培训。我们的 AI 合作伙伴 AI Mentor 框架将体验式学习与现实实践和定制反馈相结合。这项工作最终需要跨学科创新，以解决对劳动力发展和社会平等的更广泛影响。

Title: Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation

• Authors: Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, En-Shiun Annie Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04212
• Pdf URL: https://arxiv.org/pdf/2404.04212
• Copy Paste: [[2404.04212]] Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation(https://arxiv.org/abs/2404.04212)
• Keywords: language model
• Abstract: Parameter-efficient fine-tuning (PEFT) methods are increasingly vital in adapting large-scale pre-trained language models for diverse tasks, offering a balance between adaptability and computational efficiency. They are important in Low-Resource Language (LRL) Neural Machine Translation (NMT) to enhance translation accuracy with minimal resources. However, their practical effectiveness varies significantly across different languages. We conducted comprehensive empirical experiments with varying LRL domains and sizes to evaluate the performance of 8 PEFT methods with in total of 15 architectures using the SacreBLEU score. We showed that 6 PEFT architectures outperform the baseline for both in-domain and out-domain tests and the Houlsby+Inversion adapter has the best performance overall, proving the effectiveness of PEFT methods.
• 摘要：参数高效微调（PEFT）方法对于使大规模预训练语言模型适应不同任务变得越来越重要，从而在适应性和计算效率之间提供平衡。它们在低资源语言 (LRL) 神经机器翻译 (NMT) 中非常重要，可以用最少的资源提高翻译准确性。然而，它们的实际效果在不同语言之间存在很大差异。我们对不同的 LRL 域和大小进行了全面的实证实验，以使用 SacreBLEU 评分评估 8 种 PEFT 方法（总共 15 种架构）的性能。我们表明，6 个 PEFT 架构在域内和域外测试中均优于基线，并且 Houlsby+Inversion 适配器具有最佳的整体性能，证明了 PEFT 方法的有效性。

Title: Cleared for Takeoff? Compositional & Conditional Reasoning may be the Achilles Heel to (Flight-Booking) Language Agents

• Authors: Harsh Kohli, Huan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04237
• Pdf URL: https://arxiv.org/pdf/2404.04237
• Copy Paste: [[2404.04237]] Cleared for Takeoff? Compositional & Conditional Reasoning may be the Achilles Heel to (Flight-Booking) Language Agents(https://arxiv.org/abs/2404.04237)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: The rapid progress of large language models (LLMs) has seen them excel and frequently surpass human performance on standard benchmarks. This has enabled many downstream applications, such as LLM agents, to rely on their sophisticated reasoning to navigate complex task requirements. However, LLMs are known to unexpectedly falter in simple tasks and under seemingly straightforward circumstances - underscoring the need for better and more diverse evaluation setups to measure their true capabilities. To this end, we choose to study compositional and conditional reasoning, two cornerstones of human cognition, and introduce GroundCocoa - a lexically diverse benchmark connecting these reasoning skills to the real-world problem of flight booking. Our task involves aligning detailed user preferences with available flight options presented in a multiple-choice format. Results indicate a significant disparity in performance among current state-of-the-art LLMs with even the best performing model, GPT-4 Turbo, not exceeding 67% accuracy despite advanced prompting techniques.
• 摘要：大型语言模型 (LLM) 的快速进步让它们在标准基准测试中表现出色，并经常超越人类的表现。这使得许多下游应用程序（例如 LLM 代理）能够依靠其复杂的推理来满足复杂的任务要求。然而，众所周知，法学硕士在简单的任务和看似简单的情况下会出乎意料地表现不佳——这凸显出需要更好、更多样化的评估设置来衡量他们的真实能力。为此，我们选择研究构成推理和条件推理（人类认知的两个基石），并引入 GroundCocoa - 一个词汇多样化的基准，将这些推理技能与现实世界的航班预订问题联系起来。我们的任务包括将详细的用户偏好与以多项选择格式呈现的可用航班选项结合起来。结果表明，当前最先进的法学硕士之间的性能存在显着差异，即使是性能最好的模型 GPT-4 Turbo，尽管采用了先进的提示技术，准确率也不会超过 67%。