2024-09-20

Title: ARTICLE: Annotator Reliability Through In-Context Learning

• Authors: Sujan Dutta, Deepak Pandita, Tharindu Cyril Weerasooriya, Marcos Zampieri, Christopher M. Homan, Ashiqur R. KhudaBukhsh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12218
• Pdf URL: https://arxiv.org/pdf/2409.12218
• Copy Paste: [[2409.12218]] ARTICLE: Annotator Reliability Through In-Context Learning(https://arxiv.org/abs/2409.12218)
• Keywords: llm
• Abstract: Ensuring annotator quality in training and evaluation data is a key piece of machine learning in NLP. Tasks such as sentiment analysis and offensive speech detection are intrinsically subjective, creating a challenging scenario for traditional quality assessment approaches because it is hard to distinguish disagreement due to poor work from that due to differences of opinions between sincere annotators. With the goal of increasing diverse perspectives in annotation while ensuring consistency, we propose \texttt{ARTICLE}, an in-context learning (ICL) framework to estimate annotation quality through self-consistency. We evaluate this framework on two offensive speech datasets using multiple LLMs and compare its performance with traditional methods. Our findings indicate that \texttt{ARTICLE} can be used as a robust method for identifying reliable annotators, hence improving data quality.
• 摘要：确保训练和评估数据中的注释者质量是 NLP 机器学习的关键部分。情绪分析和攻击性语音检测等任务本质上是主观的，这为传统的质量评估方法带来了挑战，因为很难区分由于工作不力而导致的分歧和由于真诚的注释者之间的意见分歧而导致的分歧。为了在确保一致性的同时增加注释中的多样化视角，我们提出了 \texttt{ARTICLE}，这是一个上下文学习 (ICL) 框架，可通过自洽性来估计注释质量。我们使用多个 LLM 在两个攻击性语音数据集上评估该框架，并将其性能与传统方法进行比较。我们的研究结果表明，\texttt{ARTICLE} 可用作识别可靠注释者的稳健方法，从而提高数据质量。

Title: MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language

• Authors: Muhammad Asif Ali, Nawal Daftardar, Mutayyaba Waheed, Jianbin Qin, Di Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12257
• Pdf URL: https://arxiv.org/pdf/2409.12257
• Copy Paste: [[2409.12257]] MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language(https://arxiv.org/abs/2409.12257)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated significant capabilities across numerous application domains. A key challenge is to keep these models updated with latest available information, which limits the true potential of these models for the end-applications. Although, there have been numerous attempts for LLMs Knowledge Editing (KE), i.e., to edit the LLMs prior knowledge and in turn test it via Multi-hop Question Answering (MQA), yet so far these studies are primarily focused on English language. To bridge this gap, in this paper we propose: Multi-hop Questioning Answering under Knowledge Editing for Arabic Language (MQA-KEAL). MQA-KEAL stores knowledge edits as structured knowledge units in the external memory. In order to solve multi-hop question, it first uses task-decomposition to decompose the question into smaller sub-problems. Later for each sub-problem, it iteratively queries the external memory and/or target LLM in order to generate the final response. In addition, we also contribute MQUAKE-AR (Arabic translation of English benchmark MQUAKE), as well as a new benchmark MQA-AEVAL for rigorous performance evaluation of MQA under KE for Arabic language. Experimentation evaluation reveals MQA-KEAL outperforms the baseline models by a significant margin.
• 摘要：大型语言模型 (LLM) 已在众多应用领域展现出强大的能力。一个关键挑战是让这些模型保持最新可用信息，这限制了这些模型在最终应用中的真正潜力。尽管已经多次尝试进行 LLM 知识编辑 (KE)，即编辑 LLM 的先验知识并通过多跳问答 (MQA) 对其进行测试，但到目前为止，这些研究主要集中在英语上。为了弥补这一差距，我们在本文中提出：知识编辑下的阿拉伯语多跳问答 (MQA-KEAL)。MQA-KEAL 将知识编辑作为结构化知识单元存储在外部存储器中。为了解决多跳问题，它首先使用任务分解将问题分解为更小的子问题。然后对于每个子问题，它会迭代查询外部存储器和/或目标 LLM 以生成最终响应。此外，我们还贡献了 MQUAKE-AR（英语基准 MQUAKE 的阿拉伯语翻译），以及新的基准 MQA-AEVAL，用于对阿拉伯语 KE 下的 MQA 进行严格的性能评估。实验评估表明，MQA-KEAL 的表现明显优于基线模型。

Title: Making Large Language Models into World Models with Precondition and Effect Knowledge

• Authors: Kaige Xie, Ian Yang, John Gunerli, Mark Riedl
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12278
• Pdf URL: https://arxiv.org/pdf/2409.12278
• Copy Paste: [[2409.12278]] Making Large Language Models into World Models with Precondition and Effect Knowledge(https://arxiv.org/abs/2409.12278)
• Keywords: language model, llm, agent
• Abstract: World models, which encapsulate the dynamics of how actions affect environments, are foundational to the functioning of intelligent agents. In this work, we explore the potential of Large Language Models (LLMs) to operate as world models. Although LLMs are not inherently designed to model real-world dynamics, we show that they can be induced to perform two critical world model functions: determining the applicability of an action based on a given world state, and predicting the resulting world state upon action execution. This is achieved by fine-tuning two separate LLMs-one for precondition prediction and another for effect prediction-while leveraging synthetic data generation techniques. Through human-participant studies, we validate that the precondition and effect knowledge generated by our models aligns with human understanding of world dynamics. We also analyze the extent to which the world model trained on our synthetic data results in an inferred state space that supports the creation of action chains, a necessary property for planning.
• 摘要：世界模型是智能代理运作的基础，它囊括了动作如何影响环境的动态。在这项工作中，我们探索了大型语言模型 (LLM) 作为世界模型运行的潜力。尽管 LLM 并非天生设计用于模拟现实世界动态，但我们表明它们可以执行两个关键的世界模型功能：根据给定的世界状态确定动作的适用性，并预测动作执行后产生的世界状态。这是通过微调两个独立的 LLM（一个用于先决条件预测，另一个用于效果预测）同时利用合成数据生成技术来实现的。通过人类参与者研究，我们验证了我们的模型生成的先决条件和效果知识与人类对世界动态的理解相一致。我们还分析了在我们的合成数据上训练的世界模型在多大程度上产生了支持创建动作链（规划的必要属性）的推断状态空间。

Title: Small Language Models are Equation Reasoners

• Authors: Bumjun Kim, Kunha Lee, Juyeon Kim, Sangam Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12393
• Pdf URL: https://arxiv.org/pdf/2409.12393
• Copy Paste: [[2409.12393]] Small Language Models are Equation Reasoners(https://arxiv.org/abs/2409.12393)
• Keywords: language model, llm, chain-of-thought
• Abstract: Chain-of-Thought (CoT) reasoning has enabled Large Language Model (LLM) to achieve remarkable performance in various NLP tasks, including arithmetic problem-solving. However, this success does not generalize to small language model (sLM) like T5, due to their limited capacity and absence of emergent abilities associated with larger models. Recent works to enhance sLM through knowledge distillation have yielded some improvements but still face significant limitations, particularly high ambiguity from the variability in natural language expressions and substantial computational costs. In this paper, we investigate why sLM perform poorly on arithmetic reasoning tasks and hypothesize that natural language format variability introduces high ambiguity for these smaller models. Based on this hypothesis, we conduct experiments with equation-only format, which is a reasoning format that unifies arithmetic reasoning previously expressed in natural language formats into mathematical equations. Experiment results demonstrate that equation-only format effectively boosts the arithmetic reasoning abilities of sLM, especially in very small models like T5-Tiny.
• 摘要：思维链 (CoT) 推理使大型语言模型 (LLM) 在各种 NLP 任务（包括算术问题解决）中取得了显著的表现。然而，这种成功并不适用于 T5 等小型语言模型 (sLM)，因为它们的容量有限，并且缺乏与大型模型相关的新兴能力。最近通过知识提炼增强 sLM 的研究取得了一些进展，但仍然面临重大限制，特别是自然语言表达的多变性和大量的计算成本导致的高歧义性。在本文中，我们研究了 sLM 在算术推理任务上表现不佳的原因，并假设自然语言格式的多变性会给这些较小的模型带来高歧义性。基于这一假设，我们对仅方程式格式进行了实验，这是一种将先前以自然语言格式表达的算术推理统一为数学方程式的推理格式。实验结果表明，仅方程式格式有效地提高了 sLM 的算术推理能力，尤其是在像 T5-Tiny 这样的非常小的模型中。

Title: Preference Alignment Improves Language Model-Based TTS

• Authors: Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12403
• Pdf URL: https://arxiv.org/pdf/2409.12403
• Copy Paste: [[2409.12403]] Preference Alignment Improves Language Model-Based TTS(https://arxiv.org/abs/2409.12403)
• Keywords: language model
• Abstract: Recent advancements in text-to-speech (TTS) have shown that language model (LM)-based systems offer competitive performance to their counterparts. Further optimization can be achieved through preference alignment algorithms, which adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. This study presents a thorough empirical evaluation of how preference alignment algorithms, particularly Direct Preference Optimization (DPO), enhance LM-based TTS. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores, with the latter two metrics surpassing even human speech in certain evaluations. We also show preference alignment is applicable to low-resource scenarios and effectively generalized to out-of-domain applications.
• 摘要：文本转语音 (TTS) 的最新进展表明，基于语言模型 (LM) 的系统比同类系统具有更出色的性能。可以通过偏好对齐算法实现进一步优化，该算法会调整 LM 以与奖励模型的偏好保持一致，从而增强生成内容的吸引力。本研究对偏好对齐算法（尤其是直接偏好优化 (DPO)）如何增强基于 LM 的 TTS 进行了全面的实证评估。借助 1.15B 参数的基于 LM 的 TTS 模型，我们证明偏好对齐可以持续提高可理解性、说话者相似性和代理主观评价分数，后两个指标在某些评估中甚至超过了人类语音。我们还表明偏好对齐适用于资源匮乏的场景，并可有效推广到域外应用。

Title: Textualized Agent-Style Reasoning for Complex Tasks by Multiple Round LLM Generation

• Authors: Chen Liang, Zhifan Feng, Zihe Liu, Wenbin Jiang, Jinan Xu, Yufeng Chen, Yong Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12411
• Pdf URL: https://arxiv.org/pdf/2409.12411
• Copy Paste: [[2409.12411]] Textualized Agent-Style Reasoning for Complex Tasks by Multiple Round LLM Generation(https://arxiv.org/abs/2409.12411)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought, agent
• Abstract: Chain-of-thought prompting significantly boosts the reasoning ability of large language models but still faces three issues: hallucination problem, restricted interpretability, and uncontrollable generation. To address these challenges, we present AgentCOT, a llm-based autonomous agent framework, which can solve complex problems in an agent-style manner by multiple round LLM generation. At each step, AgentCOT selects an action and executes it to yield an intermediate result with supporting evidence. In addition, we integrate the step's index into the reasoning process to form a graph structure for complex inference logic. We introduce two new strategies to enhance the performance of AgentCOT.We conduct extensive experiments to verify the effectiveness of our method on six common benchmarks. Results exhibit that our method brings in substantial improvements over current competitive approaches.
• 摘要：思路链提示显著提升了大型语言模型的推理能力，但仍面临三个问题：幻觉问题、可解释性受限和生成不可控。为了解决这些挑战，我们提出了基于 LLM 的自主代理框架 AgentCOT，它可以通过多轮 LLM 生成以代理式方式解决复杂问题。在每个步骤中，AgentCOT 选择一个动作并执行它以产生具有支持证据的中间结果。此外，我们将步骤的索引集成到推理过程中，以形成复杂推理逻辑的图形结构。我们引入了两种新策略来增强 AgentCOT 的性能。我们在六个常见基准上进行了广泛的实验来验证我们方法的有效性。结果表明，我们的方法比当前的竞争方法带来了显着的改进。

Title: Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels

• Authors: Chaoqun Liu, Qin Chao, Wenxuan Zhang, Xiaobao Wu, Boyang Li, Anh Tuan Luu, Lidong Bing
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12425
• Pdf URL: https://arxiv.org/pdf/2409.12425
• Copy Paste: [[2409.12425]] Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels(https://arxiv.org/abs/2409.12425)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance through supervised fine-tuning or in-context learning using gold labels. However, this paradigm is limited by the availability of gold labels, while in certain scenarios, LLMs may need to perform tasks that are too complex for humans to provide such labels. To tackle this challenge, this study explores whether solely utilizing unlabeled data can elicit strong model capabilities. We propose a new paradigm termed zero-to-strong generalization. We iteratively prompt LLMs to annotate unlabeled data and retain high-quality labels by filtering. Surprisingly, we obverse that this iterative process gradually unlocks LLMs' potential on downstream tasks. Our experiments on extensive classification and reasoning tasks confirm the effectiveness of our proposed framework. Our analysis indicates that this paradigm is effective for both in-context learning and fine-tuning, and for various model sizes.
• 摘要：大型语言模型 (LLM) 通过使用黄金标签进行监督微调或上下文学习，表现出了卓越的性能。然而，这种范式受到黄金标签可用性的限制，而在某些情况下，LLM 可能需要执行过于复杂的任务，而人类无法提供此类标签。为了应对这一挑战，本研究探讨了仅使用未标记数据是否可以引发强大的模型能力。我们提出了一种称为从零到强泛化的新范式。我们迭代地提示 LLM 注释未标记数据并通过过滤保留高质量标签。令人惊讶的是，我们观察到这个迭代过程逐渐释放了 LLM 在下游任务上的潜力。我们在广泛的分类和推理任务上的实验证实了我们提出的框架的有效性。我们的分析表明，这种范式对于上下文学习和微调以及各种模型大小都有效。

Title: Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models

• Authors: Xinyu Zhou, Delong Chen, Samuel Cahyawijaya, Xufeng Duan, Zhenguang G. Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12435
• Pdf URL: https://arxiv.org/pdf/2409.12435
• Copy Paste: [[2409.12435]] Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models(https://arxiv.org/abs/2409.12435)
• Keywords: language model, llm
• Abstract: We introduce a novel analysis that leverages linguistic minimal pairs to probe the internal linguistic representations of Large Language Models (LLMs). By measuring the similarity between LLM activation differences across minimal pairs, we quantify the and gain insight into the linguistic knowledge captured by LLMs. Our large-scale experiments, spanning 100+ LLMs and 150k minimal pairs in three languages, reveal properties of linguistic similarity from four key aspects: consistency across LLMs, relation to theoretical categorizations, dependency to semantic context, and cross-lingual alignment of relevant phenomena. Our findings suggest that 1) linguistic similarity is significantly influenced by training data exposure, leading to higher cross-LLM agreement in higher-resource languages. 2) Linguistic similarity strongly aligns with fine-grained theoretical linguistic categories but weakly with broader ones. 3) Linguistic similarity shows a weak correlation with semantic similarity, showing its context-dependent nature. 4) LLMs exhibit limited cross-lingual alignment in their understanding of relevant linguistic phenomena. This work demonstrates the potential of minimal pairs as a window into the neural representations of language in LLMs, shedding light on the relationship between LLMs and linguistic theory.
• 摘要：我们引入了一种新颖的分析方法，利用语言最小对来探究大型语言模型 (LLM) 的内部语言表征。通过测量最小对之间 LLM 激活差异的相似性，我们量化并深入了解 LLM 捕获的语言知识。我们的大规模实验涵盖了三种语言的 100 多个 LLM 和 150k 个最小对，从四个关键方面揭示了语言相似性的属性：LLM 之间的一致性、与理论分类的关系、对语义上下文的依赖性以及相关现象的跨语言一致性。我们的研究结果表明：1) 语言相似性受训练数据暴露的显著影响，导致资源较多的语言的跨 LLM 一致性更高。2) 语言相似性与细粒度的理论语言类别高度一致，但与更广泛的语言类别的一致性较弱。3) 语言相似性与语义相似性显示出弱相关性，显示出其依赖于上下文的性质。 4) LLM 在理解相关语言现象时表现出有限的跨语言一致性。这项研究展示了最小对作为 LLM 中语言神经表征的窗口的潜力，揭示了 LLM 与语言理论之间的关系。

Title: Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data

• Authors: Jiaming Zhou, Abbas Ghaddar, Ge Zhang, Liheng Ma, Yaochen Hu, Soumyasundar Pal, Mark Coates, Bin Wang, Yingxue Zhang, Jianye Hao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12437
• Pdf URL: https://arxiv.org/pdf/2409.12437
• Copy Paste: [[2409.12437]] Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data(https://arxiv.org/abs/2409.12437)
• Keywords: language model, llm, prompt
• Abstract: Despite recent advances in training and prompting strategies for Large Language Models (LLMs), these models continue to face challenges with complex logical reasoning tasks that involve long reasoning chains. In this work, we explore the potential and limitations of using graph-based synthetic reasoning data as training signals to enhance LLMs' reasoning capabilities. Our extensive experiments, conducted on two established natural language reasoning tasks -- inductive reasoning and spatial reasoning -- demonstrate that supervised fine-tuning (SFT) with synthetic graph-based reasoning data effectively enhances LLMs' reasoning performance without compromising their effectiveness on other standard evaluation benchmarks.
• 摘要：尽管大型语言模型 (LLM) 的训练和提示策略最近取得了进展，但这些模型在涉及长推理链的复杂逻辑推理任务中仍然面临挑战。在这项工作中，我们探索了使用基于图形的合成推理数据作为训练信号来增强 LLM 推理能力的潜力和局限性。我们对两个已建立的自然语言推理任务（归纳推理和空间推理）进行了广泛的实验，结果表明，使用基于图形的合成推理数据进行监督微调 (SFT) 可有效增强 LLM 的推理性能，同时不会影响其在其他标准评估基准上的有效性。

Title: Incremental and Data-Efficient Concept Formation to Support Masked Word Prediction

• Authors: Xin Lian, Nishant Baglodi, Christopher J. MacLellan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12440
• Pdf URL: https://arxiv.org/pdf/2409.12440
• Copy Paste: [[2409.12440]] Incremental and Data-Efficient Concept Formation to Support Masked Word Prediction(https://arxiv.org/abs/2409.12440)
• Keywords: language model
• Abstract: This paper introduces Cobweb4L, a novel approach for efficient language model learning that supports masked word prediction. The approach builds on Cobweb, an incremental system that learns a hierarchy of probabilistic concepts. Each concept stores the frequencies of words that appear in instances tagged with that concept label. The system utilizes an attribute value representation to encode words and their surrounding context into instances. Cobweb4L uses the information theoretic variant of category utility and a new performance mechanism that leverages multiple concepts to generate predictions. We demonstrate that with these extensions it significantly outperforms prior Cobweb performance mechanisms that use only a single node to generate predictions. Further, we demonstrate that Cobweb4L learns rapidly and achieves performance comparable to and even superior to Word2Vec. Next, we show that Cobweb4L and Word2Vec outperform BERT in the same task with less training data. Finally, we discuss future work to make our conclusions more robust and inclusive.
• 摘要：本文介绍了一种支持掩码词预测的高效语言模型学习新方法 Cobweb4L。该方法以 Cobweb 为基础，Cobweb 是一种增量系统，可学习概率概念的层次结构。每个概念都存储了在标有该概念标签的实例中出现的单词的频率。该系统利用属性值表示将单词及其周围的上下文编码为实例。Cobweb4L 使用类别效用的信息论变体和一种利用多个概念生成预测的新性能机制。我们证明，通过这些扩展，它的性能明显优于以前的仅使用单个节点生成预测的 Cobweb 性能机制。此外，我们证明 Cobweb4L 学习速度快，性能可与 Word2Vec 媲美甚至优于 Word2Vec。接下来，我们表明 Cobweb4L 和 Word2Vec 在训练数据较少的相同任务中胜过 BERT。最后，我们讨论了未来的工作，以使我们的结论更加稳健和包容。

Title: CodePlan: Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planning

• Authors: Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, Minlie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12452
• Pdf URL: https://arxiv.org/pdf/2409.12452
• Copy Paste: [[2409.12452]] CodePlan: Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planning(https://arxiv.org/abs/2409.12452)
• Keywords: language model, llm, prompt
• Abstract: Despite the remarkable success of large language models (LLMs) on traditional natural language processing tasks, their planning ability remains a critical bottleneck in tackling complex multi-step reasoning tasks. Existing approaches mainly rely on prompting or task-specific fine-tuning, often suffering from weak robustness and cross-task generalization. To address the limitation, we introduce CODEPLAN, a scalable paradigm that empowers LLMs to generate and follow code-form plans pseudocode that outlines high-level, structured reasoning processes. By leveraging the structured and versatile nature of code, CODEPLAN effectively captures the rich semantics and control flows inherent to sophisticated reasoning. Importantly, CODEPLAN allows the automatic extraction of code-form plans from massive, wide-ranging text corpora without the need for curated, task-specific datasets. This enables it to scale up efficiently and improve reasoning capabilities across diverse scenarios. To train CODEPLAN, we construct a large-scale dataset of 2M examples that integrate code-form plans with standard prompt-response pairs from existing corpora. With minimal computation overhead during both training and inference, CODEPLAN achieves a 25.1% relative improvement compared with directly generating responses, averaged across 13 challenging multi-step reasoning benchmarks, spanning mathematical reasoning, symbolic reasoning, instruction-following, multi-hop QA, and decision-making tasks. Further analysis reveals CODEPLAN's increasing performance gains on more complex reasoning tasks, as well as significant data efficiency thanks to its generalization ability.
• 摘要：尽管大型语言模型 (LLM) 在传统自然语言处理任务上取得了显著成功，但它们的规划能力仍然是解决复杂的多步骤推理任务的关键瓶颈。现有方法主要依赖于提示或特定于任务的微调，通常存在较弱的鲁棒性和跨任务泛化能力。为了解决这一限制，我们引入了 CODEPLAN，这是一种可扩展的范例，它使 LLM 能够生成和遵循代码形式计划伪代码，这些伪代码概述了高级、结构化的推理过程。通过利用代码的结构化和多功能性，CODEPLAN 有效地捕获了复杂推理所固有的丰富语义和控制流。重要的是，CODEPLAN 允许从大量、广泛的文本语料库中自动提取代码形式计划，而无需精心策划的特定于任务的数据集。这使它能够有效地扩展并提高跨不同场景的推理能力。为了训练 CODEPLAN，我们构建了一个包含 200 万个示例的大规模数据集，将代码形式计划与现有语料库中的标准提示-响应对集成在一起。在训练和推理过程中，CODEPLAN 的计算开销都极小，与直接生成响应相比，其相对改进幅度达到 25.1%，这是在 13 个具有挑战性的多步骤推理基准测试中的平均水平，涵盖数学推理、符号推理、指令遵循、多跳问答和决策任务。进一步的分析表明，CODEPLAN 在更复杂的推理任务上的性能提升越来越大，并且得益于其泛化能力，数据效率显著提高。

Title: Familiarity-aware Evidence Compression for Retrieval Augmented Generation

• Authors: Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, Muhao Chen
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12468
• Pdf URL: https://arxiv.org/pdf/2409.12468
• Copy Paste: [[2409.12468]] Familiarity-aware Evidence Compression for Retrieval Augmented Generation(https://arxiv.org/abs/2409.12468)
• Keywords: language model, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) improves large language models (LMs) by incorporating non-parametric knowledge through evidence retrieval from external sources. However, it often struggles to filter out inconsistent and irrelevant information that can distract the LM from its tasks. While compressing the retrieved evidence with a compression model aims to address this issue, the compressed evidence may still be unfamiliar to the target model used for downstream task, potentially failing to utilize the evidence effectively. We propose FaviComp (Familiarity-aware Evidence Compression), a novel training-free evidence compression technique that makes retrieved evidence more familiar to the target model, while seamlessly integrating parametric knowledge from the model. Specifically, FaviComp proactively lowers the perplexity of the compressed evidence with regard to the target model by combining token probabilities from both the compression model and the target model to generate context that is more familiar to the target model. This approach balances the integration of parametric and non-parametric knowledge, which is especially helpful in complex tasks where the retrieved evidence set may not contain all the necessary information. Experimental results demonstrate that FaviComp consistently outperforms existing baselines in multiple open-domain QA datasets, achieving high compression rates and showcasing the effective integration of both parametric and non-parametric knowledge.
• 摘要：检索增强生成 (RAG) 通过从外部来源检索证据来整合非参数知识，从而改进大型语言模型 (LM)。然而，它通常很难过滤掉不一致和不相关的信息，这些信息可能会分散 LM 的任务注意力。虽然使用压缩模型压缩检索到的证据旨在解决这个问题，但压缩证据可能仍然不为用于下游任务的目标模型所熟悉，可能无法有效利用证据。我们提出了 FaviComp（熟悉度感知证据压缩），这是一种新颖的无训练证据压缩技术，它使检索到的证据对目标模型更熟悉，同时无缝集成来自模型的参数知识。具体而言，FaviComp 通过结合来自压缩模型和目标模型的标记概率来生成对目标模型更熟悉的上下文，主动降低压缩证据相对于目标模型的困惑度。这种方法平衡了参数和非参数知识的集成，这在复杂任务中尤其有用，因为检索到的证据集可能不包含所有必要的信息。实验结果表明，FaviComp 在多个开放域 QA 数据集中始终优于现有基线，实现了高压缩率，并展示了参数和非参数知识的有效集成。

Title: CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs

• Authors: Junlin Lv, Yuan Feng, Xike Xie, Xin Jia, Qirong Peng, Guiming Xie
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12490
• Pdf URL: https://arxiv.org/pdf/2409.12490
• Copy Paste: [[2409.12490]] CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs(https://arxiv.org/abs/2409.12490)
• Keywords: language model, llm
• Abstract: Large language models have achieved notable success across various domains, yet efficient inference is still limited by the quadratic computation complexity of the attention mechanism. The inference consists of prefilling and decoding phases. Although several attempts have been made to accelerate decoding, the inefficiency of the prefilling phase, especially for long-context tasks, remains a challenge. In this paper, we observe a locality in query criticality during the prefilling phase of long-context processing: adjacent query tokens tend to focus on similar subsets of the past Key-Value (KV) cache. Based on this observation, we propose CritiPrefill, a criticality-based segment-wise prefilling method. This method partitions the input sequence's queries and KV cache into segments and blocks, utilizing a segment-wise algorithm to estimate the query criticality. By pruning non-critical computations between query segments and cache blocks in the self-attention mechanism, the prefilling process can be significantly accelerated. Extensive evaluations on multiple long-context datasets show up to 2.7x speedup on Llama3-8B and 3.0x speedup on Yi-9B for 128K context length on a single A100 GPU, with minimal quality degradation.
• 摘要：大型语言模型在各个领域都取得了显著的成功，然而注意力机制的二次计算复杂度仍然限制了推理的效率。推理包括预填充和解码阶段。尽管已经进行了多次尝试来加速解码，但预填充阶段的低效率（尤其是对于长上下文任务而言）仍然是一个挑战。在本文中，我们观察到长上下文处理的预填充阶段查询关键性存在局部性：相邻的查询标记倾向于关注过去键值 (KV) 缓存的相似子集。基于这一观察，我们提出了基于关键性的分段预填充方法 CritiPrefill。该方法将输入序列的查询和 KV 缓存划分为段和块，利用分段算法来估计查询关键性。通过在自注意力机制中修剪查询段和缓存块之间的非关键计算，可以显著加速预填充过程。对多个长上下文数据集进行的广泛评估表明，在单个 A100 GPU 上，对于 128K 上下文长度，Llama3-8B 的速度提高了 2.7 倍，Yi-9B 的速度提高了 3.0 倍，而质量下降最小。

Title: LLMR: Knowledge Distillation with a Large Language Model-Induced Reward

• Authors: Dongheng Li, Yongchang Hao, Lili Mou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12500
• Pdf URL: https://arxiv.org/pdf/2409.12500
• Copy Paste: [[2409.12500]] LLMR: Knowledge Distillation with a Large Language Model-Induced Reward(https://arxiv.org/abs/2409.12500)
• Keywords: language model, llm
• Abstract: Large language models have become increasingly popular and demonstrated remarkable performance in various natural language processing (NLP) tasks. However, these models are typically computationally expensive and difficult to be deployed in resource-constrained environments. In this paper, we propose LLMR, a novel knowledge distillation (KD) method based on a reward function induced from large language models. We conducted experiments on multiple datasets in the dialogue generation and summarization tasks. Empirical results demonstrate that our LLMR approach consistently outperforms traditional KD methods in different tasks and datasets.
• 摘要：大型语言模型越来越受欢迎，并在各种自然语言处理 (NLP) 任务中表现出色。然而，这些模型通常计算成本高昂，难以在资源受限的环境中部署。在本文中，我们提出了 LLMR，这是一种基于大型语言模型诱导的奖励函数的新型知识蒸馏 (KD) 方法。我们在对话生成和摘要任务中对多个数据集进行了实验。实证结果表明，我们的 LLMR 方法在不同任务和数据集中始终优于传统的 KD 方法。

Title: Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models

• Authors: Jun Rao, Xuebo Liu, Zepeng Lin, Liang Ding, Jing Li, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12512
• Pdf URL: https://arxiv.org/pdf/2409.12512
• Copy Paste: [[2409.12512]] Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models(https://arxiv.org/abs/2409.12512)
• Keywords: language model
• Abstract: Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. The success of KD in auto-regressive language models mainly relies on Reverse KL for mode-seeking and student-generated output (SGO) to combat exposure bias. Our theoretical analyses and experimental validation reveal that while Reverse KL effectively mimics certain features of the teacher distribution, it fails to capture most of its behaviors. Conversely, SGO incurs higher computational costs and presents challenges in optimization, particularly when the student model is significantly smaller than the teacher model. These constraints are primarily due to the immutable distribution of the teacher model, which fails to adjust adaptively to models of varying sizes. We introduce Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. This strategy abolishes the necessity for on-policy sampling and merely requires minimal updates to the parameters of the teacher's online module during training, thereby allowing dynamic adaptation to the student's distribution to make distillation better. Extensive results across multiple generation datasets show that OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
• 摘要：知识蒸馏 (KD) 是一种通过训练较小的学生模型来模仿大型教师模型来压缩大型教师模型的技术。KD 在自回归语言模型中的成功主要依赖于反向 KL 进行模式搜索和学生生成输出 (SGO) 来对抗曝光偏差。我们的理论分析和实验验证表明，虽然反向 KL 有效地模仿了教师分布的某些特征，但它无法捕捉到其大部分行为。相反，SGO 会产生更高的计算成本，并在优化方面带来挑战，尤其是当学生模型明显小于教师模型时。这些限制主要是由于教师模型的不可变分布，无法自适应地调整以适应不同大小的模型。我们引入了在线知识蒸馏 (OKD)，其中教师网络集成了小型在线模块以与学生模型同时训练。这种策略消除了在线策略采样的必要性，只需要在训练期间对教师在线模块的参数进行最少的更新，从而允许动态适应学生的分布以使蒸馏效果更好。跨多代数据集的大量结果表明，OKD 达到或超过了各种模型架构和规模中领先方法的性能，将训练时间缩短了多达四倍。

Title: Should RAG Chatbots Forget Unimportant Conversations? Exploring Importance and Forgetting with Psychological Insights

• Authors: Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12524
• Pdf URL: https://arxiv.org/pdf/2409.12524
• Copy Paste: [[2409.12524]] Should RAG Chatbots Forget Unimportant Conversations? Exploring Importance and Forgetting with Psychological Insights(https://arxiv.org/abs/2409.12524)
• Keywords: chat, retrieval-augmented generation
• Abstract: While Retrieval-Augmented Generation (RAG) has shown promise in enhancing long-term conversations, the increasing memory load as conversations progress degrades retrieval accuracy. Drawing on psychological insights, we propose LUFY, a simple yet effective method that focuses on emotionally arousing memories and retains less than 10% of the conversation. In the user experiment, participants interacted with three types of RAG chatbots, each for 2 hours over 4 sessions, marking the most extensive assessment of a chatbot's long-term capabilities to date -- more than four times longer than any existing benchmark. The results demonstrate that prioritizing arousing memories while forgetting the majority of the conversation significantly enhances user experience. This study pushes the frontier of long-term conversations and highlights the importance of forgetting unimportant parts of conversations. Code and Dataset: this https URL
• 摘要：虽然检索增强生成 (RAG) 在增强长期对话方面表现出了良好的前景，但随着对话的进行，记忆负荷的增加会降低检索的准确性。借鉴心理学见解，我们提出了 LUFY，这是一种简单而有效的方法，它专注于唤起情感的记忆，并保留不到 10% 的对话。在用户实验中，参与者与三种类型的 RAG 聊天机器人进行了互动，每种互动持续 2 小时，共 4 个会话，这是迄今为止对聊天机器人长期能力最广泛的评估——比任何现有基准都要长四倍多。结果表明，优先唤起记忆而忘记大部分对话可显著提升用户体验。这项研究推动了长期对话的发展，并强调了忘记对话中不重要部分的重要性。代码和数据集：此 https URL

Title: Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection

• Authors: Chin-Po Chen, Jeng-Lin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12541
• Pdf URL: https://arxiv.org/pdf/2409.12541
• Copy Paste: [[2409.12541]] Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection(https://arxiv.org/abs/2409.12541)
• Keywords: language model, llm
• Abstract: Alzheimer's disease (AD) stands as the predominant cause of dementia, characterized by a gradual decline in speech and language capabilities. Recent deep-learning advancements have facilitated automated AD detection through spontaneous speech. However, common transcript-based detection methods directly model text patterns in each utterance without a global view of the patient's linguistic characteristics, resulting in limited discriminability and interpretability. Despite the enhanced reasoning abilities of large language models (LLMs), there remains a gap in fully harnessing the reasoning ability to facilitate AD detection and model interpretation. Therefore, we propose a patient-level transcript profiling framework leveraging LLM-based reasoning augmentation to systematically elicit linguistic deficit attributes. The summarized embeddings of the attributes are integrated into an Albert model for AD detection. The framework achieves 8.51\% ACC and 8.34\% F1 improvements on the ADReSS dataset compared to the baseline without reasoning augmentation. Our further analysis shows the effectiveness of our identified linguistic deficit attributes and the potential to use LLM for AD detection interpretation.
• 摘要：阿尔茨海默病 (AD) 是导致痴呆症的主要原因，其特征是说话和语言能力逐渐下降。最近深度学习的进步促进了通过自发语音自动检测 AD。然而，常见的基于转录的检测方法直接对每句话中的文本模式进行建模，而没有对患者语言特征的全局了解，导致可辨别性和可解释性有限。尽管大型语言模型 (LLM) 的推理能力增强，但在充分利用推理能力促进 AD 检测和模型解释方面仍然存在差距。因此，我们提出了一个患者级转录分析框架，利用基于 LLM 的推理增强来系统地引出语言缺陷属性。属性的汇总嵌入被集成到用于 AD 检测的 Albert 模型中。与没有推理增强的基线相比，该框架在 ADReSS 数据集上实现了 8.51\% ACC 和 8.34\% F1 改进。我们进一步的分析表明了我们所确定的语言缺陷属性的有效性以及使用 LLM 进行 AD 检测解释的潜力。

Title: Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment

• Authors: Tianyu Peng, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12545
• Pdf URL: https://arxiv.org/pdf/2409.12545
• Copy Paste: [[2409.12545]] Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment(https://arxiv.org/abs/2409.12545)
• Keywords: language model, llm
• Abstract: Knowledge distillation (KD) is an effective model compression method that can transfer the internal capabilities of large language models (LLMs) to smaller ones. However, the multi-modal probability distribution predicted by teacher LLMs causes difficulties for student models to learn. In this paper, we first demonstrate the importance of multi-modal distribution alignment with experiments and then highlight the inefficiency of existing KD approaches in learning multi-modal distributions. To address this problem, we propose Ranking Loss based Knowledge Distillation (RLKD), which encourages the consistency of the ranking of peak predictions between the teacher and student models. By incorporating word-level ranking loss, we ensure excellent compatibility with existing distillation objectives while fully leveraging the fine-grained information between different categories in peaks of two predicted distribution. Experimental results demonstrate that our method enables the student model to better learn the multi-modal distributions of the teacher model, leading to a significant performance improvement in various downstream tasks.
• 摘要：知识蒸馏 (KD) 是一种有效的模型压缩方法，可以将大型语言模型 (LLM) 的内部能力转移到较小的语言模型中。然而，教师 LLM 预测的多模态概率分布给学生模型的学习带来了困难。在本文中，我们首先通过实验证明了多模态分布对齐的重要性，然后强调了现有 KD 方法在学习多模态分布方面的低效性。为了解决这个问题，我们提出了基于排名损失的知识蒸馏 (RLKD)，它鼓励教师和学生模型之间峰值预测的排名保持一致。通过结合词级排名损失，我们确保与现有蒸馏目标的出色兼容性，同时充分利用两个预测分布峰值中不同类别之间的细粒度信息。实验结果表明，我们的方法使学生模型能够更好地学习教师模型的多模态分布，从而显着提高各种下游任务的性能。

Title: RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues

• Authors: Tzu-Lin Kuo, Feng-Ting Liao, Mu-Wei Hsieh, Fu-Chieh Chang, Po-Chun Hsu, Da-Shan Shiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12558
• Pdf URL: https://arxiv.org/pdf/2409.12558
• Copy Paste: [[2409.12558]] RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues(https://arxiv.org/abs/2409.12558)
• Keywords: language model, llm, chat, retrieval-augmented generation
• Abstract: In real-world applications with Large Language Models (LLMs), external retrieval mechanisms - such as Search-Augmented Generation (SAG), tool utilization, and Retrieval-Augmented Generation (RAG) - are often employed to enhance the quality of augmented generations in dialogues. These approaches often come with multi-turn dialogue, where each interaction is enriched by relevant information retrieved from external sources. Existing benchmarks either assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in single-turn settings. However, there is a gap in evaluating LLMs' ability to leverage retrieval for more precise responses across multiple turns. To address this limitation, we introduce RAD-Bench (Retrieval Augmented Dialogue), a benchmark designed to evaluate LLMs' capabilities in multi-turn dialogues following retrievals, essential for their deployment in context-rich applications. RAD-Bench evaluates two key abilities of LLMs: Retrieval Synthesis and Retrieval Reasoning. These are measured using discriminative questions and retrieved contexts, and corresponding reference answers, assessing how effectively LLMs integrate and reason with context to maintain and enhance conversation quality over multiple turns. Our evaluation results on commonly used LLMs reveal that model performance deteriorates as additional layers of conditions or constraints are applied across conversation turns, even when accurate retrieved contexts are provided.
• 摘要：在大型语言模型 (LLM) 的实际应用中，外部检索机制（例如搜索增强生成 (SAG)、工具利用和检索增强生成 (RAG)）通常用于提高对话中增强生成的质量。这些方法通常伴随着多轮对话，其中每次交互都通过从外部来源检索到的相关信息来丰富。现有的基准要么评估 LLM 在多轮对话中的聊天能力，要么评估它们在单轮设置中使用检索进行增强响应的能力。然而，在评估 LLM 利用检索在多轮对话中获得更精确响应的能力方面存在差距。为了解决这一限制，我们引入了 RAD-Bench（检索增强对话），这是一个基准，旨在评估 LLM 在检索后进行多轮对话的能力，这对于将其部署到上下文丰富的应用程序中至关重要。RAD-Bench 评估 LLM 的两项关键能力：检索综合和检索推理。这些是通过判别性问题和检索到的上下文以及相应的参考答案来衡量的，评估 LLM 如何有效地整合和推理上下文以保持和提高多轮对话的质量。我们对常用 LLM 的评估结果表明，即使提供了准确的检索到的上下文，随着在对话轮次中应用额外的条件或约束层，模型性能也会下降。

Title: Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights

• Authors: Mohamad Ballout, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12586
• Pdf URL: https://arxiv.org/pdf/2409.12586
• Copy Paste: [[2409.12586]] Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights(https://arxiv.org/abs/2409.12586)
• Keywords: language model
• Abstract: Enhancing small language models for real-life application deployment is a significant challenge facing the research community. Due to the difficulties and costs of using large language models, researchers are seeking ways to effectively deploy task-specific small models. In this work, we introduce a simple yet effective knowledge distillation method to improve the performance of small language models. Our approach utilizes a teacher model with approximately 3 billion parameters to identify the most influential tokens in its decision-making process. These tokens are extracted from the input based on their attribution scores relative to the output, using methods like saliency maps. These important tokens are then provided as rationales to a student model, aiming to distill the knowledge of the teacher model. This method has proven to be effective, as demonstrated by testing it on four diverse datasets, where it shows improvement over both standard fine-tuning methods and state-of-the-art knowledge distillation models. Furthermore, we explore explanations of the success of the model by analyzing the important tokens extracted from the teacher model. Our findings reveal that in 68\% of cases, specifically in datasets where labels are part of the answer, such as multiple-choice questions, the extracted tokens are part of the ground truth.
• 摘要：增强小型语言模型以用于实际应用是研究界面临的重大挑战。由于使用大型语言模型的困难和成本，研究人员正在寻找有效部署特定任务小型模型的方法。在这项工作中，我们引入了一种简单但有效的知识提炼方法来提高小型语言模型的性能。我们的方法利用一个具有大约 30 亿个参数的教师模型来识别其决策过程中最有影响力的标记。这些标记是根据它们相对于输出的归因分数从输入中提取出来的，使用显着性图等方法。然后将这些重要的标记作为原理提供给学生模型，旨在提炼教师模型的知识。这种方法已被证明是有效的，通过在四个不同的数据集上进行测试，它显示出比标准微调方法和最先进的知识提炼模型都有所改进。此外，我们通过分析从教师模型中提取的重要标记来探索模型成功的原因。我们的研究结果表明，在 68% 的情况下，特别是在标签是答案一部分的数据集中，例如多项选择题，提取的标记是基本事实的一部分。

Title: Enhancing SLM via ChatGPT and Dataset Augmentation

• Authors: Tom Pieper, Mohamad Ballout, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12599
• Pdf URL: https://arxiv.org/pdf/2409.12599
• Copy Paste: [[2409.12599]] Enhancing SLM via ChatGPT and Dataset Augmentation(https://arxiv.org/abs/2409.12599)
• Keywords: language model, gpt, llm, chat
• Abstract: This paper explores the enhancement of small language models through strategic dataset augmentation via ChatGPT-3.5-Turbo, in the domain of Natural Language Inference (NLI). By employing knowledge distillation-based techniques and synthetic dataset augmentation, we aim to bridge the performance gap between large language models (LLMs) and small language models (SLMs) without the immense cost of human annotation. Our methods involve two forms of rationale generation--information extraction and informed reasoning--to enrich the ANLI dataset. We then fine-tune T5-Small on these augmented datasets, evaluating its performance against an established benchmark. Our findings reveal that the incorporation of synthetic rationales significantly improves the model's ability to comprehend natural language, leading to 1.3\% and 2.3\% higher classification accuracy, respectively, on the ANLI dataset, demonstrating the potential of leveraging LLMs for dataset augmentation. This approach not only enhances the performance of smaller models on complex tasks but also introduces a cost-effective method for fine-tuning smaller language models. By advancing our understanding of knowledge distillation and fine-tuning strategies, this work contributes to the ongoing effort to create more capable and efficient NLP systems.
• 摘要：本文探讨了在自然语言推理 (NLI) 领域，通过 ChatGPT-3.5-Turbo 进行战略性数据集增强来增强小型语言模型。通过采用基于知识蒸馏的技术和合成数据集增强，我们旨在弥合大型语言模型 (LLM) 和小型语言模型 (SLM) 之间的性能差距，而无需付出巨大的人工注释成本。我们的方法涉及两种形式的原理生成——信息提取和知情推理——以丰富 ANLI 数据集。然后，我们在这些增强的数据集上对 T5-Small 进行微调，根据既定基准评估其性能。我们的研究结果表明，加入合成原理显著提高了模型理解自然语言的能力，在 ANLI 数据集上分别将分类准确率提高了 1.3% 和 2.3%，展示了利用 LLM 进行数据集增强的潜力。这种方法不仅提高了小型模型在复杂任务上的性能，而且还引入了一种经济高效的方法来微调小型语言模型。通过加深我们对知识提炼和微调策略的理解，这项工作为创建更强大、更高效的 NLP 系统的持续努力做出了贡献。

Title: Iteration of Thought: Leveraging Inner Dialogue for Autonomous Large Language Model Reasoning

• Authors: Santosh Kumar Radha, Yasamin Nouri Jelyani, Ara Ghukasyan, Oktay Goktas
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2409.12618
• Pdf URL: https://arxiv.org/pdf/2409.12618
• Copy Paste: [[2409.12618]] Iteration of Thought: Leveraging Inner Dialogue for Autonomous Large Language Model Reasoning(https://arxiv.org/abs/2409.12618)
• Keywords: language model, llm, prompt, agent
• Abstract: Iterative human engagement is a common and effective means of leveraging the advanced language processing power of large language models (LLMs). Using well-structured prompts in a conversational manner, human users can effectively influence an LLM to develop more thoughtful and accurate responses. Motivated by this insight, we propose the Iteration of Thought (IoT) framework for enhancing LLM responses by generating "thought"-provoking prompts vis a vis an input query and the current iteration of an LLM's response. Unlike static or semi-static approaches, e.g. Chain of Thought (CoT) or Tree of Thoughts (ToT), IoT adapts its reasoning path dynamically, based on evolving context, and without generating alternate explorative thoughts which are ultimately discarded. The three components of the IoT framework are (1) an Inner Dialogue Agent (IDA) responsible for generating instructive, context-specific prompts; (2) an LLM Agent (LLMA) that processes these prompts to refine its responses; and (3) an iterative prompting loop that implements a conversation between the former two components. We introduce two variants of our framework: Autonomous Iteration of Thought (AIoT), where an LLM decides when to stop iterating, and Guided Iteration of Thought (GIoT), which always forces a fixed number iterations. We investigate the performance of IoT across various datasets, spanning complex reasoning tasks from the GPQA dataset, explorative problem-solving in Game of 24, puzzle solving in Mini Crosswords, and multi-hop question answering from the HotpotQA dataset. Our results show that IoT represents a viable paradigm for autonomous response refinement in LLMs, showcasing significant improvements over CoT and thereby enabling more adaptive and efficient reasoning systems that minimize human intervention.
• 摘要：迭代式人类参与是利用大型语言模型 (LLM) 的高级语言处理能力的一种常见且有效的方法。通过以对话方式使用结构良好的提示，人类用户可以有效地影响 LLM 以开发更周到和准确的响应。受此启发，我们提出了思维迭代 (IoT) 框架，通过针对输入查询和 LLM 响应的当前迭代生成“引发思考”的提示来增强 LLM 响应。与静态或半静态方法（例如思维链 (CoT) 或思维树 (ToT)）不同，IoT 会根据不断变化的上下文动态调整其推理路径，而不会生成最终被丢弃的替代探索性想法。IoT 框架的三个组成部分是 (1) 内部对话代理 (IDA)，负责生成具有指导意义的、特定于上下文的提示；(2) LLM 代理 (LLMA)，处理这些提示以改进其响应；以及 (3) 实现前两个组件之间对话的迭代提示循环。我们引入了我们框架的两个变体：自主思维迭代 (AIoT)，其中 LLM 决定何时停止迭代，以及引导思维迭代 (GIoT)，它总是强制进行固定次数的迭代。我们研究了 IoT 在各种数据集上的性能，涵盖来自 GPQA 数据集的复杂推理任务、24 游戏中的探索性问题解决、Mini Crosswords 中的谜题解决以及来自 HotpotQA 数据集的多跳问答。我们的结果表明，IoT 代表了 LLM 中自主响应细化的可行范例，展示了对 CoT 的显着改进，从而实现了更具适应性、更高效的推理系统，从而最大限度地减少了人为干预。

Title: CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks

• Authors: Zhaozhi Qian, Faroq Altam, Muhammad Saleh Saeed Alqurishi, Riad Souissi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12623
• Pdf URL: https://arxiv.org/pdf/2409.12623
• Copy Paste: [[2409.12623]] CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks(https://arxiv.org/abs/2409.12623)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are the cornerstones of modern artificial intelligence systems. This paper introduces Juhaina, a Arabic-English bilingual LLM specifically designed to align with the values and preferences of Arabic speakers. Juhaina inherently supports advanced functionalities such as instruction following, open-ended question answering, information provisioning, and text processing. Our model contains 9.24 billion parameters and is trained on a context window of up to 8,192 tokens. This paper details the creation process of Juhaina and provides an extensive empirical evaluation. Furthermore, we identify the limitations of widely-adopted Open Arabic LLM Leaderboard (OALL) and propose a new evaluation benchmark, CamelEval. Our findings demonstrate that Juhaina surpasses existing LLMs of comparable sizes, such as the Llama and Gemma families, in generating helpful responses in Arabic, providing factually accurate information about the region, and understanding nuanced cultural aspects. We aspire for Juhaina to democratize cutting-edge AI technologies, serving over 400 million Arabic speakers by offering LLMs that not only communicate in their language but also comprehend their culture. We publicly release all models on Huggingface \url{this https URL}.
• 摘要：大型语言模型 (LLM) 是现代人工智能系统的基石。本文介绍了 Juhaina，这是一种阿拉伯语-英语双语 LLM，专门设计用于符合阿拉伯语使用者的价值观和偏好。Juhaina 本身支持高级功能，例如指令遵循、开放式问答、信息提供和文本处理。我们的模型包含 92.4 亿个参数，并在最多 8,192 个标记的上下文窗口上进行训练。本文详细介绍了 Juhaina 的创建过程，并提供了广泛的实证评估。此外，我们确定了广泛采用的开放阿拉伯语 LLM 排行榜 (OALL) 的局限性，并提出了一个新的评估基准 CamelEval。我们的研究结果表明，Juhaina 在生成有用的阿拉伯语回复、提供有关该地区的事实准确信息以及理解细微的文化方面优于现有的同等规模的 LLM，例如 Llama 和 Gemma 家族。我们希望 Juhaina 能够普及尖端的人工智能技术，为超过 4 亿阿拉伯语使用者提供服务，为他们提供不仅能用阿拉伯语交流，还能理解阿拉伯语文化的法学硕士学位。我们在 Huggingface \url{此 https URL} 上公开发布所有模型。

Title: Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

• Authors: Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, Jean-Baptiste Lespiau, Nithya Attaluri, Kate Olszewska
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12640
• Pdf URL: https://arxiv.org/pdf/2409.12640
• Copy Paste: [[2409.12640]] Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries(https://arxiv.org/abs/2409.12640)
• Keywords: language model, long context
• Abstract: We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via a novel, unifying framework for evaluations over arbitrarily long contexts which measure the model's ability to do more than retrieve a single piece of information from its context. The central idea of the \frameworkname framework (\frameworkshort) is to construct tasks which require a model to ``chisel away'' the irrelevant information in the context, revealing a latent structure in the context. To verify a model's understanding of this latent structure, we query the model for details of the structure. Using \frameworkshort, we produce three diagnostic long-context evaluations across code and natural-language domains intended to provide a stronger signal of long-context language model capabilities. We perform evaluations on several state-of-the-art models and demonstrate both that a) the proposed evaluations are high-signal and b) that there is significant room for improvement in synthesizing long-context information.
• 摘要：我们引入了 Michelangelo：一种用于大型语言模型的最小、合成且未泄露的长上下文推理评估，并且易于自动评分。此评估通过一个新颖的统一框架得出，该框架用于对任意长的上下文进行评估，该框架衡量模型执行更多操作的能力，而不仅仅是从其上下文中检索单个信息。 \frameworkname 框架（\frameworkshort）的核心思想是构建需要模型“剔除”上下文中不相关信息的任务，从而揭示上下文中的潜在结构。为了验证模型对这种潜在结构的理解，我们查询模型以获取结构的详细信息。使用 \frameworkshort，我们在代码和自然语言领域生成三个诊断性长上下文评估，旨在提供长上下文语言模型能力的更强信号。我们对几种最先进的模型进行了评估，并证明：a）所提出的评估具有高信号性；b）在合成长上下文信息方面还有很大的改进空间。

Title: Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards

• Authors: Furkan Şahinuç, Thy Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12656
• Pdf URL: https://arxiv.org/pdf/2409.12656
• Copy Paste: [[2409.12656]] Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards(https://arxiv.org/abs/2409.12656)
• Keywords: language model, llm
• Abstract: Scientific leaderboards are standardized ranking systems that facilitate evaluating and comparing competitive methods. Typically, a leaderboard is defined by a task, dataset, and evaluation metric (TDM) triple, allowing objective performance assessment and fostering innovation through benchmarking. However, the exponential increase in publications has made it infeasible to construct and maintain these leaderboards manually. Automatic leaderboard construction has emerged as a solution to reduce manual labor. Existing datasets for this task are based on the community-contributed leaderboards without additional curation. Our analysis shows that a large portion of these leaderboards are incomplete, and some of them contain incorrect information. In this work, we present SciLead, a manually-curated Scientific Leaderboard dataset that overcomes the aforementioned problems. Building on this dataset, we propose three experimental settings that simulate real-world scenarios where TDM triples are fully defined, partially defined, or undefined during leaderboard construction. While previous research has only explored the first setting, the latter two are more representative of real-world applications. To address these diverse settings, we develop a comprehensive LLM-based framework for constructing leaderboards. Our experiments and analysis reveal that various LLMs often correctly identify TDM triples while struggling to extract result values from publications. We make our code and data publicly available.
• 摘要：科学排行榜是标准化的排名系统，有助于评估和比较竞争方法。通常，排行榜由任务、数据集和评估指标 (TDM) 三元组定义，从而允许客观地评估绩效并通过基准测试促进创新。然而，出版物的指数级增长使得手动构建和维护这些排行榜变得不可行。自动排行榜构建已成为减少人工劳动的解决方案。此任务的现有数据集基于社区贡献的排行榜，没有额外的管理。我们的分析表明，这些排行榜中很大一部分是不完整的，其中一些包含不正确的信息。在这项工作中，我们提出了 SciLead，这是一个手动管理的科学排行榜数据集，它克服了上述问题。基于这个数据集，我们提出了三种实验设置，模拟在排行榜构建过程中 TDM 三元组完全定义、部分定义或未定义的真实场景。虽然以前的研究只探索了第一种设置，但后两种设置更能代表现实世界的应用。为了解决这些不同的情况，我们开发了一个基于 LLM 的综合框架来构建排行榜。我们的实验和分析表明，各种 LLM 通常能够正确识别 TDM 三元组，但很难从出版物中提取结果值。我们公开提供我们的代码和数据。

Title: Exploring the topics, sentiments and hate speech in the Spanish information environment

• Authors: ALEJANDRO BUITRAGO LOPEZ, Javier Pastor-Galindo, José Antonio Ruipérez-Valiente
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12658
• Pdf URL: https://arxiv.org/pdf/2409.12658
• Copy Paste: [[2409.12658]] Exploring the topics, sentiments and hate speech in the Spanish information environment(https://arxiv.org/abs/2409.12658)
• Keywords: language model, llm
• Abstract: In the digital era, the internet and social media have transformed communication but have also facilitated the spread of hate speech and disinformation, leading to radicalization, polarization, and toxicity. This is especially concerning for media outlets due to their significant role in shaping public discourse. This study examines the topics, sentiments, and hate prevalence in 337,807 response messages (website comments and tweets) to news from five Spanish media outlets (La Vanguardia, ABC, El País, El Mundo, and 20 Minutos) in January 2021. These public reactions were originally labeled as distinct types of hate by experts following an original procedure, and they are now classified into three sentiment values (negative, neutral, or positive) and main topics. The BERTopic unsupervised framework was used to extract 81 topics, manually named with the help of Large Language Models (LLMs) and grouped into nine primary categories. Results show social issues (22.22%), expressions and slang (20.35%), and political issues (11.80%) as the most discussed. Content is mainly negative (62.7%) and neutral (28.57%), with low positivity (8.73%). Toxic narratives relate to conversation expressions, gender, feminism, and COVID-19. Despite low levels of hate speech (3.98%), the study confirms high toxicity in online responses to social and political topics.
• 摘要：在数字时代，互联网和社交媒体改变了交流方式，但也助长了仇恨言论和虚假信息的传播，导致激进化、两极分化和毒性。这对媒体来说尤其令人担忧，因为它们在塑造公共话语方面发挥着重要作用。这项研究调查了 2021 年 1 月来自五家西班牙媒体（La Vanguardia、ABC、El País、El Mundo 和 20 Minutos）的 337,807 条新闻回复信息（网站评论和推文）中的主题、情绪和仇恨流行程度。这些公众反应最初由专家按照原始程序标记为不同类型的仇恨，现在它们被分为三个情绪值（负面、中性或正面）和主要主题。BERTopic 无监督框架用于提取 81 个主题，在大型语言模型 (LLM) 的帮助下手动命名并分为九个主要类别。结果显示，社会问题（22.22%）、表达和俚语（20.35%）和政治问题（11.80%）是讨论最多的问题。内容主要是负面的（62.7%）和中性的（28.57%），积极性较低（8.73%）。有毒的叙述与对话表达、性别、女权主义和 COVID-19 有关。尽管仇恨言论的水平较低（3.98%），但研究证实，在线对社会和政治话题的回应具有很高的毒性。

Title: Text2Traj2Text: Learning-by-Synthesis Framework for Contextual Captioning of Human Movement Trajectories

• Authors: Hikaru Asano, Ryo Yonetani, Taiki Sekii, Hiroki Ouchi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12670
• Pdf URL: https://arxiv.org/pdf/2409.12670
• Copy Paste: [[2409.12670]] Text2Traj2Text: Learning-by-Synthesis Framework for Contextual Captioning of Human Movement Trajectories(https://arxiv.org/abs/2409.12670)
• Keywords: language model
• Abstract: This paper presents Text2Traj2Text, a novel learning-by-synthesis framework for captioning possible contexts behind shopper's trajectory data in retail stores. Our work will impact various retail applications that need better customer understanding, such as targeted advertising and inventory management. The key idea is leveraging large language models to synthesize a diverse and realistic collection of contextual captions as well as the corresponding movement trajectories on a store map. Despite learned from fully synthesized data, the captioning model can generalize well to trajectories/captions created by real human subjects. Our systematic evaluation confirmed the effectiveness of the proposed framework over competitive approaches in terms of ROUGE and BERT Score metrics.
• 摘要：本文介绍了 Text2Traj2Text，这是一种新颖的综合学习框架，用于为零售店购物者轨迹数据背后的可能背景添加字幕。我们的工作将影响需要更好地了解客户的各种零售应用，例如定向广告和库存管理。关键思想是利用大型语言模型来合成多样化且逼真的上下文字幕集合以及商店地图上的相应移动轨迹。尽管是从完全合成的数据中学习的，但字幕模型可以很好地推广到真实人类受试者创建的轨迹/字幕。我们的系统评估证实了所提出的框架在 ROUGE 和 BERT 分数指标方面优于竞争方法。

Title: Connecting Ideas in 'Lower-Resource' Scenarios: NLP for National Varieties, Creoles and Other Low-resource Scenarios

• Authors: Aditya Joshi, Diptesh Kanojia, Heather Lent, Hour Kaing, Haiyue Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12683
• Pdf URL: https://arxiv.org/pdf/2409.12683
• Copy Paste: [[2409.12683]] Connecting Ideas in 'Lower-Resource' Scenarios: NLP for National Varieties, Creoles and Other Low-resource Scenarios(https://arxiv.org/abs/2409.12683)
• Keywords: language model
• Abstract: Despite excellent results on benchmarks over a small subset of languages, large language models struggle to process text from languages situated in `lower-resource' scenarios such as dialects/sociolects (national or social varieties of a language), Creoles (languages arising from linguistic contact between multiple languages) and other low-resource languages. This introductory tutorial will identify common challenges, approaches, and themes in natural language processing (NLP) research for confronting and overcoming the obstacles inherent to data-poor contexts. By connecting past ideas to the present field, this tutorial aims to ignite collaboration and cross-pollination between researchers working in these scenarios. Our notion of `lower-resource' broadly denotes the outstanding lack of data required for model training - and may be applied to scenarios apart from the three covered in the tutorial.
• 摘要：尽管大型语言模型在一小部分语言的基准测试中取得了优异的成绩，但它们在处理“资源匮乏”场景中的语言文本时仍举步维艰，例如方言/社会方言（一种语言的国家或社会变体）、克里奥尔语（由多种语言之间的语言接触而产生的语言）和其他资源匮乏的语言。本入门教程将确定自然语言处理 (NLP) 研究中常见的挑战、方法和主题，以应对和克服数据匮乏环境中固有的障碍。通过将过去的想法与当前的领域联系起来，本教程旨在激发在这些场景中工作的研究人员之间的合作和相互交流。我们的“资源匮乏”概念大致表示模型训练所需的数据严重不足 - 并且可以应用于本教程中介绍的三种场景之外的场景。

Title: Exploring Large Language Models for Product Attribute Value Identification

• Authors: Kassem Sabeh, Mouna Kacimi, Johann Gamper, Robert Litschko, Barbara Plank
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.12695
• Pdf URL: https://arxiv.org/pdf/2409.12695
• Copy Paste: [[2409.12695]] Exploring Large Language Models for Product Attribute Value Identification(https://arxiv.org/abs/2409.12695)
• Keywords: language model, llm, prompt
• Abstract: Product attribute value identification (PAVI) involves automatically identifying attributes and their values from product information, enabling features like product search, recommendation, and comparison. Existing methods primarily rely on fine-tuning pre-trained language models, such as BART and T5, which require extensive task-specific training data and struggle to generalize to new attributes. This paper explores large language models (LLMs), such as LLaMA and Mistral, as data-efficient and robust alternatives for PAVI. We propose various strategies: comparing one-step and two-step prompt-based approaches in zero-shot settings and utilizing parametric and non-parametric knowledge through in-context learning examples. We also introduce a dense demonstration retriever based on a pre-trained T5 model and perform instruction fine-tuning to explicitly train LLMs on task-specific instructions. Extensive experiments on two product benchmarks show that our two-step approach significantly improves performance in zero-shot settings, and instruction fine-tuning further boosts performance when using training data, demonstrating the practical benefits of using LLMs for PAVI.
• 摘要：产品属性值识别 (PAVI) 涉及从产品信息中自动识别属性及其值，从而实现产品搜索、推荐和比较等功能。现有方法主要依赖于微调预训练语言模型，例如 BART 和 T5，这些模型需要大量特定于任务的训练数据，并且难以推广到新属性。本文探讨了大型语言模型 (LLM)，例如 LLaMA 和 Mistral，作为 PAVI 的数据高效且稳健的替代方案。我们提出了各种策略：在零样本设置中比较一步和两步基于提示的方法，并通过上下文学习示例利用参数和非参数知识。我们还引入了一个基于预训练 T5 模型的密集演示检索器，并执行指令微调以明确训练 LLM 以执行特定于任务的指令。在两个产品基准上进行的大量实验表明，我们的两步方法显着提高了零样本设置下的性能，并且指令微调在使用训练数据时进一步提高了性能，证明了使用 LLM 进行 PAVI 的实际好处。

Title: LLM-Measure: Generating Valid, Consistent, and Reproducible Text-Based Measures for Social Science Research

• Authors: Yi Yang, Hanyu Duan, Jiaxin Liu, Kar Yan Tam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12722
• Pdf URL: https://arxiv.org/pdf/2409.12722
• Copy Paste: [[2409.12722]] LLM-Measure: Generating Valid, Consistent, and Reproducible Text-Based Measures for Social Science Research(https://arxiv.org/abs/2409.12722)
• Keywords: language model, llm
• Abstract: The increasing use of text as data in social science research necessitates the development of valid, consistent, reproducible, and efficient methods for generating text-based concept measures. This paper presents a novel method that leverages the internal hidden states of large language models (LLMs) to generate these concept measures. Specifically, the proposed method learns a concept vector that captures how the LLM internally represents the target concept, then estimates the concept value for text data by projecting the text's LLM hidden states onto the concept vector. Three replication studies demonstrate the method's effectiveness in producing highly valid, consistent, and reproducible text-based measures across various social science research contexts, highlighting its potential as a valuable tool for the research community.
• 摘要：在社会科学研究中，文本作为数据的使用越来越多，这要求开发有效、一致、可重复且高效的方法来生成基于文本的概念度量。本文介绍了一种新方法，该方法利用大型语言模型 (LLM) 的内部隐藏状态来生成这些概念度量。具体来说，所提出的方法学习一个概念向量，该向量捕捉 LLM 如何在内部表示目标概念，然后通过将文本的 LLM 隐藏状态投影到概念向量上来估计文本数据的概念值。三项重复研究表明，该方法在各种社会科学研究环境中生成高度有效、一致且可重复的基于文本的度量方面非常有效，凸显了其作为研究界宝贵工具的潜力。

Title: Edu-Values: Towards Evaluating the Chinese Education Values of Large Language Models

• Authors: Peiyi Zhang, Yazhou Zhang, Bo Wang, Lu Rong, Jing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12739
• Pdf URL: https://arxiv.org/pdf/2409.12739
• Copy Paste: [[2409.12739]] Edu-Values: Towards Evaluating the Chinese Education Values of Large Language Models(https://arxiv.org/abs/2409.12739)
• Keywords: language model, llm, prompt
• Abstract: With the recent evolution of large language models (LLMs), concerns about aligning such models with human values have grown. Previous research has primarily focused on assessing LLMs' performance in terms of the Helpful, Honest, Harmless (3H) basic principles, while often overlooking their alignment with educational values in the Chinese context. To fill this gap, we present Edu-Values, the first Chinese education values evaluation benchmark designed to measure LLMs' alignment ability across seven dimensions: professional ideology, cultural literacy, educational knowledge and skills, education laws and regulations, teachers' professional ethics, basic competencies, and subject knowledge. We meticulously design and compile 1,418 questions, including multiple-choice, multi-modal question answering, subjective analysis, adversarial prompts, and questions on traditional Chinese culture. We conduct both human evaluation and automatic evaluation over 11 state-of-the-art (SoTA) LLMs, and highlight three main findings: (1) due to differences in educational culture, Chinese LLMs significantly outperform English LLMs, with Qwen 2 ranking the first with a score of 81.37; (2) LLMs perform well in subject knowledge and teaching skills but struggle with teachers' professional ethics and basic competencies; (3) LLMs excel at multiple-choice questions but perform poorly on subjective analysis and multi-modal tasks. This demonstrates the effectiveness and potential of the proposed benchmark. Our dataset is available at this https URL.
• 摘要：随着大型语言模型（LLM）的不断发展，人们越来越关注如何将这些模型与人类价值观相一致。以往的研究主要侧重于评估LLM在“有益、诚实、无害”（3H）基本原则方面的表现，而往往忽略了它们与中国背景下的教育价值观的契合程度。为了填补这一空白，我们提出了第一个中国教育价值观评估基准Edu-Values，旨在衡量LLM在七个维度上的契合能力：职业意识形态、文化素养、教育知识和技能、教育法律法规、教师职业道德、基本能力和学科知识。我们精心设计和编写了1,418道题目，包括多项选择题、多模态问答题、主观分析题、对抗性提示题和中国传统文化题。我们对 11 门最先进的 (SoTA) LLM 课程进行了人工评估和自动评估，并得出了三个主要发现：（1）由于教育文化的差异，中国 LLM 课程的表现明显优于英国 LLM 课程，其中 Qwen 2 以 81.37 分排名第一；（2）LLM 课程在学科知识和教学技能方面表现良好，但在教师职业道德和基本能力方面存在不足；（3）LLM 课程擅长多项选择题，但在主观分析和多模态任务方面表现不佳。这证明了所提出的基准的有效性和潜力。我们的数据集可在此 https URL 上找到。

Title: Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Parameter Optimization

• Authors: Thomas Savage, Stephen Ma, Abdessalem Boukil, Vishwesh Patel, Ekanath Rangan, Ivan Rodriguez, Jonathan H Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12741
• Pdf URL: https://arxiv.org/pdf/2409.12741
• Copy Paste: [[2409.12741]] Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Parameter Optimization(https://arxiv.org/abs/2409.12741)
• Keywords: language model, llm
• Abstract: Large Language Model (LLM) fine tuning is underutilized in the field of medicine. Two of the most common methods of fine tuning are Supervised Fine Tuning (SFT) and Direct Parameter Optimization (DPO), but there is little guidance informing users when to use either technique. In this investigation, we compare the performance of SFT and DPO for five common natural language tasks in medicine: Classification with text data, Classification with numeric data, Clinical Reasoning, Summarization, and Clinical Triage. We find that SFT alone is sufficient for Classification with text data, whereas DPO improves performance for the more complex tasks of Clinical Reasoning, Summarization and Clinical Triage. Our results establish the role and importance of DPO fine tuning within medicine, and consequently call attention to current software gaps that prevent widespread deployment of this technique.
• 摘要：大型语言模型 (LLM) 微调在医学领域尚未得到充分利用。两种最常见的微调方法是监督微调 (SFT) 和直接参数优化 (DPO)，但很少有指导告知用户何时使用这两种技术。在本研究中，我们比较了 SFT 和 DPO 在医学中五种常见自然语言任务的性能：文本数据分类、数字数据分类、临床推理、总结和临床分诊。我们发现，仅使用 SFT 就足以进行文本数据分类，而 DPO 可以提高临床推理、总结和临床分诊等更复杂任务的性能。我们的研究结果确立了 DPO 微调在医学中的作用和重要性，并因此提醒人们注意阻碍该技术广泛部署的当前软件缺陷。

Title: Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination

• Authors: Eva Sánchez Salido, Roser Morante, Julio Gonzalo, Guillermo Marco, Jorge Carrillo-de-Albornoz, Laura Plaza, Enrique Amigó, Andrés Fernández, Alejandro Benito-Santos, Adrián Ghajari Espinosa, Victor Fresno
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12746
• Pdf URL: https://arxiv.org/pdf/2409.12746
• Copy Paste: [[2409.12746]] Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination(https://arxiv.org/abs/2409.12746)
• Keywords: language model
• Abstract: In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline.
• 摘要：在本文中，我们介绍了 UNED-ACCESS 2024，这是一个双语数据集，包含 1003 道西班牙语和英语大学入学考试多项选择题。问题最初以西班牙语编写，并手动翻译成英语，从未公开发布过。在 UNED-ACCESS 2024 数据集和 MMLU 问题等效子集上，在统一的零样本实验环境中评估了当前开源和专有模型的选择。结果表明：(i) 推理问题对模型具有挑战性，(ii) 较小的模型比较大的模型表现更差，西班牙语的下降速度比英语更快，(iii) 最佳模型之间的性能差距可以忽略不计，而较小模型的性能差距则增长到 37%。UNED-ACCESS 2024 上的模型排名在英语和西班牙语中几乎相同，并且与 MMLU 上的排名也具有很高的相关性（0.98 Pearson），这表明小型数据集足够多样化和具有代表性，可以衡量学科的表现。

Title: Language Models Learn to Mislead Humans via RLHF

• Authors: Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Boman, He He, Shi Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12822
• Pdf URL: https://arxiv.org/pdf/2409.12822
• Copy Paste: [[2409.12822]] Language Models Learn to Mislead Humans via RLHF(https://arxiv.org/abs/2409.12822)
• Keywords: language model
• Abstract: Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY" since it is Unintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and calculate humans' accuracy against gold labels. On a question-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convincing our subjects but not at completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show that probing, a state-of-the-art approach for detecting Intended Sophistry (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results highlight an important failure mode of RLHF and call for more research in assisting humans to align them.
• 摘要：语言模型 (LM) 会产生人类难以察觉的错误，尤其是在任务复杂的情况下。RLHF 是最流行的后训练方法，它可能会加剧这一问题：为了获得更高的回报，LM 可能会更好地说服人类，即使他们错了，他们也是对的。我们在标准 RLHF 流程下研究这种现象，称之为“U-SOPHISTRY”，因为它不是模型开发人员想要的。具体来说，我们要求时间受限（例如 3-10 分钟）的人类受试者评估模型输出的正确性，并计算人类对黄金标签的准确率。在问答任务（QuALITY）和编程任务（APPS）中，RLHF 使 LM 更善于说服我们的受试者，但不能正确完成任务。RLHF 还使模型更难评估：我们的受试者在 QuALITY 上的假阳性率增加了 24.1%，在 APPS 上的假阳性率增加了 18.3%。最后，我们表明，探测是一种用于检测故意诡辩（例如后门 LM）的最先进的方法，它并不适用于 U-诡辩。我们的结果突出了 RLHF 的一个重要失败模式，并呼吁开展更多研究来帮助人类对齐它们。

Title: FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists

• Authors: Tenghao Huang, Donghee Lee, John Sweeney, Jiatong Shi, Emily Steliotes, Matthew Lange, Jonathan May, Muhao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12832
• Pdf URL: https://arxiv.org/pdf/2409.12832
• Copy Paste: [[2409.12832]] FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists(https://arxiv.org/abs/2409.12832)
• Keywords: language model, agent
• Abstract: Flavor development in the food industry is increasingly challenged by the need for rapid innovation and precise flavor profile creation. Traditional flavor research methods typically rely on iterative, subjective testing, which lacks the efficiency and scalability required for modern demands. This paper presents three contributions to address the challenges. Firstly, we define a new problem domain for scientific agents in flavor science, conceptualized as the generation of hypotheses for flavor profile sourcing and understanding. To facilitate research in this area, we introduce the FoodPuzzle, a challenging benchmark consisting of 978 food items and 1,766 flavor molecules profiles. We propose a novel Scientific Agent approach, integrating in-context learning and retrieval augmented techniques to generate grounded hypotheses in the domain of food science. Experimental results indicate that our model significantly surpasses traditional methods in flavor profile prediction tasks, demonstrating its potential to transform flavor development practices.
• 摘要：食品行业的风味开发越来越受到快速创新和精确风味特征创建需求的挑战。传统的风味研究方法通常依赖于迭代的主观测试，这种测试缺乏现代需求所需的效率和可扩展性。本文提出了三点来应对这些挑战。首先，我们为风味科学中的科学代理定义了一个新的问题领域，概念化为风味特征来源和理解的假设的生成。为了促进这一领域的研究，我们引入了 FoodPuzzle，这是一个具有挑战性的基准，由 978 种食品和 1,766 种风味分子特征组成。我们提出了一种新颖的科学代理方法，整合了情境学习和检索增强技术，以在食品科学领域生成扎实的假设。实验结果表明，我们的模型在风味特征预测任务中明显超越了传统方法，展示了其改变风味开发实践的潜力。

Title: Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Models

• Authors: Bryan Zhang, Taichi Nakatani, Stephan Walter
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12880
• Pdf URL: https://arxiv.org/pdf/2409.12880
• Copy Paste: [[2409.12880]] Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Models(https://arxiv.org/abs/2409.12880)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: E-commerce stores enable multilingual product discovery which require accurate product title translation. Multilingual large language models (LLMs) have shown promising capacity to perform machine translation tasks, and it can also enhance and translate product titles cross-lingually in one step. However, product title translation often requires more than just language conversion because titles are short, lack context, and contain specialized terminology. This study proposes a retrieval-augmented generation (RAG) approach that leverages existing bilingual product information in e-commerce by retrieving similar bilingual examples and incorporating them as few-shot prompts to enhance LLM-based product title translation. Experiment results show that our proposed RAG approach improve product title translation quality with chrF score gains of up to 15.3% for language pairs where the LLM has limited proficiency.
• 摘要：电子商务商店支持多语言产品发现，这需要准确的产品标题翻译。多语言大型语言模型 (LLM) 已显示出执行机器翻译任务的良好能力，它还可以一步完成跨语言增强和翻译产品标题。然而，产品标题翻译通常需要的不仅仅是语言转换，因为标题很短、缺乏上下文并且包含专业术语。本研究提出了一种检索增强生成 (RAG) 方法，该方法利用电子商务中现有的双语产品信息，通过检索类似的双语示例并将其作为少量提示合并在一起来增强基于 LLM 的产品标题翻译。实验结果表明，我们提出的 RAG 方法提高了产品标题翻译质量，对于 LLM 熟练程度有限的语言对，chrF 分数提高了 15.3%。

Title: Knowledge-Based Domain-Oriented Data Augmentation for Enhancing Unsupervised Sentence Embedding

• Authors: Peichao Lai, Zhengfeng Zhang, Bin Cui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12887
• Pdf URL: https://arxiv.org/pdf/2409.12887
• Copy Paste: [[2409.12887]] Knowledge-Based Domain-Oriented Data Augmentation for Enhancing Unsupervised Sentence Embedding(https://arxiv.org/abs/2409.12887)
• Keywords: language model, llm
• Abstract: Recently, unsupervised sentence embedding models have received significant attention in downstream natural language processing tasks. Using large language models (LLMs) for data augmentation has led to considerable improvements in previous studies. Nevertheless, these strategies emphasize data augmentation with extensive generic corpora, neglecting the consideration of few-shot domain data. The synthesized data lacks fine-grained information and may introduce negative sample noise. This study introduces a novel pipeline-based data augmentation method that leverages LLM to synthesize the domain-specific dataset. It produces both positive and negative samples through entity- and quantity-aware augmentation, utilizing an entity knowledge graph to synthesize samples with fine-grained semantic distinctions, increasing training sample diversity and relevance. We then present a Gaussian-decayed gradient-assisted Contrastive Sentence Embedding (GCSE) model to reduce synthetic data noise and improve model discrimination to reduce negative sample noise. Experimental results demonstrate that our approach achieves state-of-the-art semantic textual similarity performance with fewer synthetic data samples and lesser LLM parameters, demonstrating its efficiency and robustness in varied backbones.
• 摘要：近年来，无监督句向量模型在下游自然语言处理任务中受到了广泛关注。使用大型语言模型（LLM）进行数据增强已在先前的研究中取得了显着的进步。然而，这些策略强调使用大量通用语料库进行数据增强，而忽略了对小样本领域数据的考虑。合成数据缺乏细粒度信息，可能会引入负样本噪声。本研究介绍了一种新颖的基于流水线的数据增强方法，该方法利用LLM合成特定领域的数据集。它通过实体和数量感知的增强来生成正样本和负样本，利用实体知识图谱合成具有细粒度语义区分的样本，增加训练样本的多样性和相关性。然后，我们提出了一个高斯衰减梯度辅助的对比句向量（GCSE）模型来减少合成数据噪声并提高模型区分度以减少负样本噪声。实验结果表明，我们的方法以更少的合成数据样本和更少的 LLM 参数实现了最先进的语义文本相似度性能，证明了其在不同主干网中的效率和稳健性。

Title: Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

• Authors: Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, Mehrdad Farajtabar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12903
• Pdf URL: https://arxiv.org/pdf/2409.12903
• Copy Paste: [[2409.12903]] Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization(https://arxiv.org/abs/2409.12903)
• Keywords: language model
• Abstract: The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters can be extremely slow and costly. In contrast, small language models are less expensive to train, but they often cannot achieve the accuracy of large models. In this paper, we explore an intriguing idea to connect these two different regimes: Can we develop a method to initialize large language models using smaller pre-trained models? Will such initialization bring any benefits in terms of training time and final accuracy? In this paper, we introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. Our method ensures that the larger model retains the functionality of the smaller model. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts. We demonstrate that training such an initialized model results in significant savings in terms of GPU hours required for pre-training large language models.
• 摘要：语言模型的预训练阶段通常从随机初始化的参数开始。在当前模型扩展的趋势下，训练大量参数可能非常缓慢且成本高昂。相比之下，小型语言模型的训练成本较低，但它们通常无法达到大型模型的准确性。在本文中，我们探索了一个有趣的想法来连接这两种不同的机制：我们能否开发一种方法来使用较小的预训练模型来初始化大型语言模型？这种初始化是否会在训练时间和最终准确性方面带来任何好处？在本文中，我们引入了 HyperCloning，这是一种可以将预训练语言模型的参数扩展为具有增加隐藏维度的较大模型的参数的方法。我们的方法确保较大的模型保留了较小模型的功能。因此，较大的模型在训练开始之前就已经继承了较小模型的预测能力和准确性。我们证明，训练这种初始化模型可以大大节省预训练大型语言模型所需的 GPU 小时数。

Title: LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning

• Authors: Jin Jiang, Yuchen Yan, Yang Liu, Yonggang Jin, Shuai Peng, Mengdi Zhang, Xunliang Cai, Yixin Cao, Liangcai Gao, Zhi Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12929
• Pdf URL: https://arxiv.org/pdf/2409.12929
• Copy Paste: [[2409.12929]] LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning(https://arxiv.org/abs/2409.12929)
• Keywords: language model, llm
• Abstract: In this paper, we present a novel approach, called LogicPro, to enhance Large Language Models (LLMs) complex Logical reasoning through Program Examples. We do this effectively by simply utilizing widely available algorithmic problems and their code solutions. First, we constructed diverse test samples input based on algorithmic questions and code solutions. Then, we designed different complex reasoning questions based on algorithmic problems and test samples. Finally, combining the intermediate variable outputs of the code solutions and the complex reasoning questions, we derived the reasoning process and the final answer. With this approach, we can construct a dataset that is sufficiently difficult (all models are ineffective), diverse (synthesized from 2,360 different algorithmic questions), and scalable (building different test samples and collecting more algorithmic questions). In addition, we obtain a high-quality reasoning process guided by the values of intermediate variables. As a result, our approach achieves significant improvements in multiple models for the BBH$^{27}$, GSM8K, HellSwag, Logicqa, Reclor, and RTE datasets, outperforming a wide range of existing reasoning datasets.
• 摘要：在本文中，我们提出了一种新方法，称为 LogicPro，通过程序示例增强大型语言模型 (LLM) 的复杂逻辑推理。我们通过简单地利用广泛可用的算法问题及其代码解决方案来有效地做到这一点。首先，我们根据算法问题和代码解决方案构建了多样化的测试样本输入。然后，我们根据算法问题和测试样本设计了不同的复杂推理问题。最后，结合代码解决方案和复杂推理问题的中间变量输出，我们推导出推理过程和最终答案。通过这种方法，我们可以构建一个足够困难（所有模型均无效）、多样化（由 2,360 个不同的算法问题合成）和可扩展（构建不同的测试样本并收集更多算法问题）的数据集。此外，我们获得了由中间变量值引导的高质量推理过程。因此，我们的方法在 BBH$^{27}$、GSM8K、HellSwag、Logicqa、Reclor 和 RTE 数据集的多个模型中取得了显著的改进，表现优于现有的各种推理数据集。

Title: Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation

• Authors: Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12941
• Pdf URL: https://arxiv.org/pdf/2409.12941
• Copy Paste: [[2409.12941]] Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation(https://arxiv.org/abs/2409.12941)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.
• 摘要：大型语言模型 (LLM) 已证明在各种认知任务中具有显著的性能改进。一种新兴应用是使用 LLM 来增强检索增强生成 (RAG) 功能。这些系统需要 LLM 理解用户查询、检索相关信息并合成连贯而准确的响应。鉴于此类系统在现实世界中的部署日益增多，全面评估变得至关重要。为此，我们提出了 FRAMES（事实性、检索和推理测量集），这是一个高质量的评估数据集，旨在测试 LLM 提供事实响应、评估检索能力和评估生成最终答案所需推理的能力。虽然以前的工作提供了数据集和基准来单独评估这些能力，但 FRAMES 提供了一个统一的框架，可以更清楚地了解端到端 RAG 场景中的 LLM 性能。我们的数据集包含具有挑战性的多跳问题，需要整合来自多个来源的信息。我们展示了基准结果，表明即使是最先进的 LLM 也难以完成这项任务，在没有检索的情况下准确率仅为 0.40。通过我们提出的多步检索流程，准确率显著提高，准确率达到 0.66（提高了 50% 以上）。我们希望我们的工作将有助于弥补评估差距，并有助于开发更强大、更强大的 RAG 系统。

Title: MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions

• Authors: Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, Hinrich Schütze
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12958
• Pdf URL: https://arxiv.org/pdf/2409.12958
• Copy Paste: [[2409.12958]] MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions(https://arxiv.org/abs/2409.12958)
• Keywords: language model, llm
• Abstract: Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets face serious challenges for low-resource languages due to their dependence on data annotation. This work introduces a novel method, Multilingual Reverse Instructions (MURI), which generates high-quality instruction tuning datasets for low-resource languages without requiring human annotators or pre-existing multilingual models. Utilizing reverse instructions and a translation pipeline, MURI produces instruction-output pairs from existing human-written texts in low-resource languages. This method ensures cultural relevance and diversity by sourcing texts from different native domains and applying filters to eliminate inappropriate content. Our dataset, MURI-IT, includes more than 2 million instruction-output pairs across 200 languages. Evaluation by native speakers and fine-tuning experiments with mT5 models demonstrate the approach's effectiveness for both NLU and open-ended generation. We publicly release datasets and models at this https URL.
• 摘要：指令调优通过使大型语言模型 (LLM) 与人类在不同任务中的偏好保持一致来增强大型语言模型 (LLM)。由于依赖数据注释，创建指令调优数据集的传统方法在资源匮乏的语言中面临严峻挑战。这项工作引入了一种新方法，即多语言反向指令 (MURI)，它无需人工注释者或预先存在的多语言模型即可为资源匮乏的语言生成高质量的指令调优数据集。利用反向指令和翻译管道，MURI 从资源匮乏的语言中现有的人工编写文本生成指令输出对。该方法通过从不同的本地域获取文本并应用过滤器消除不适当的内容来确保文化相关性和多样性。我们的数据集 MURI-IT 包含 200 种语言的 200 多万个指令输出对。母语人士的评估和使用 mT5 模型的微调实验证明了该方法对 NLU 和开放式生成的有效性。我们在此 https URL 上公开发布数据集和模型。

Title: CLAIR-A: Leveraging Large Language Models to Judge Audio Captions

• Authors: Tsung-Han Wu, Joseph E. Gonzalez, Trevor Darrell, David M. Chan
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.12962
• Pdf URL: https://arxiv.org/pdf/2409.12962
• Copy Paste: [[2409.12962]] CLAIR-A: Leveraging Large Language Models to Judge Audio Captions(https://arxiv.org/abs/2409.12962)
• Keywords: language model, llm
• Abstract: The Automated Audio Captioning (AAC) task asks models to generate natural language descriptions of an audio input. Evaluating these machine-generated audio captions is a complex task that requires considering diverse factors, among them, auditory scene understanding, sound-object inference, temporal coherence, and the environmental context of the scene. While current methods focus on specific aspects, they often fail to provide an overall score that aligns well with human judgment. In this work, we propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models (LLMs) to evaluate candidate audio captions by directly asking LLMs for a semantic distance score. In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics, with a 5.8% relative accuracy improvement compared to the domain-specific FENSE metric and up to 11% over the best general-purpose measure on the Clotho-Eval dataset. Moreover, CLAIR-A offers more transparency by allowing the language model to explain the reasoning behind its scores, with these explanations rated up to 30% better by human evaluators than those provided by baseline methods. CLAIR-A is made publicly available at this https URL.
• 摘要：自动音频字幕 (AAC) 任务要求模型生成音频输入的自然语言描述。评估这些机器生成的音频字幕是一项复杂的任务，需要考虑多种因素，其中包括听觉场景理解、声音对象推理、时间连贯性和场景的环境背景。虽然当前的方法侧重于特定方面，但它们往往无法提供与人类判断相符的总体分数。在这项工作中，我们提出了 CLAIR-A，这是一种简单而灵活的方法，它利用大型语言模型 (LLM) 的零样本功能通过直接要求 LLM 提供语义距离分数来评估候选音频字幕。在我们的评估中，与传统指标相比，CLAIR-A 可以更好地预测人类对质量的判断，与特定领域的 FENSE 指标相比，相对准确度提高了 5.8%，比 Clotho-Eval 数据集上的最佳通用指标高出 11%。此外，CLAIR-A 允许语言模型解释其得分背后的原因，从而提供更高的透明度，而这些解释在人类评估者看来比基线方法提供的解释高出 30%。CLAIR-A 可通过此 https URL 公开获取。