2024-09-10

Title: Leveraging Large Language Models for Solving Rare MIP Challenges

• Authors: Teng Wang, Wing-Yin Yu, Ruifeng She, Wenhan Yang, Taijie Chen, Jianping Zhang
• Subjects: cs.CL, cs.AI, cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.04464
• Pdf URL: https://arxiv.org/pdf/2409.04464
• Copy Paste: [[2409.04464]] Leveraging Large Language Models for Solving Rare MIP Challenges(https://arxiv.org/abs/2409.04464)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: Mixed Integer Programming (MIP) has been extensively applied in areas requiring mathematical solvers to address complex instances within tight time constraints. However, as the problem scale increases, the complexity of model formulation and finding feasible solutions escalates significantly. In contrast, the model-building cost for end-to-end models, such as large language models (LLMs), remains largely unaffected by problem scale due to their pattern recognition capabilities. While LLMs, like GPT-4, without fine-tuning, can handle some traditional medium-scale MIP problems, they struggle with uncommon or highly specialized MIP scenarios. Fine-tuning LLMs can yield some feasible solutions for medium-scale MIP instances, but these models typically fail to explore diverse solutions when constrained by a low and constant temperature, limiting their performance. In this paper, we propose and evaluate a recursively dynamic temperature method integrated with a chain-of-thought approach. Our findings show that starting with a high temperature and gradually lowering it leads to better feasible solutions compared to other dynamic temperature strategies. Additionally, by comparing results generated by the LLM with those from Gurobi, we demonstrate that the LLM can produce solutions that complement traditional solvers by accelerating the pruning process and improving overall efficiency.
• 摘要：混合整数规划 (MIP) 已广泛应用于需要数学求解器在严格时间限制内解决复杂实例的领域。然而，随着问题规模的增加，模型制定和寻找可行解决方案的复杂性显著增加。相比之下，端到端模型（如大型语言模型 (LLM)）的模型构建成本由于其模式识别能力而基本不受问题规模的影响。虽然 LLM（如 GPT-4）无需微调即可处理一些传统的中等规模 MIP 问题，但它们在处理不常见或高度专业化的 MIP 场景时会遇到困难。微调 LLM 可以为中等规模的 MIP 实例提供一些可行解决方案，但这些模型在受到低温和恒定温度的限制时通常无法探索多种解决方案，从而限制了它们的性能。在本文中，我们提出并评估了一种结合思路链方法的递归动态温度方法。我们的研究结果表明，与其他动态温度策略相比，从高温开始并逐渐降低温度可以得到更好的可行解决方案。此外，通过将 LLM 生成的结果与 Gurobi 的结果进行比较，我们证明 LLM 可以通过加速修剪过程和提高整体效率来产生补充传统求解器的解决方案。

Title: Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languages

• Authors: Tejas Deshpande, Nidhi Kowtal, Raviraj Joshi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04512
• Pdf URL: https://arxiv.org/pdf/2409.04512
• Copy Paste: [[2409.04512]] Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languages(https://arxiv.org/abs/2409.04512)
• Keywords: language model, llm, prompt
• Abstract: This paper introduces Chain of Translation Prompting (CoTR), a novel strategy designed to enhance the performance of language models in low-resource languages. CoTR restructures prompts to first translate the input context from a low-resource language into a higher-resource language, such as English. The specified task like generation, classification, or any other NLP function is then performed on the translated text, with the option to translate the output back to the original language if needed. All these steps are specified in a single prompt. We demonstrate the effectiveness of this method through a case study on the low-resource Indic language Marathi. The CoTR strategy is applied to various tasks, including sentiment analysis, hate speech classification, subject classification and text generation, and its efficacy is showcased by comparing it with regular prompting methods. Our results underscore the potential of translation-based prompting strategies to significantly improve multilingual LLM performance in low-resource languages, offering valuable insights for future research and applications. We specifically see the highest accuracy improvements with the hate speech detection task. The technique also has the potential to enhance the quality of synthetic data generation for underrepresented languages using LLMs.
• 摘要：本文介绍了翻译提示链 (CoTR)，这是一种旨在提高低资源语言中语言模型性能的新策略。CoTR 重新构建提示，首先将输入上下文从低资源语言翻译成高资源语言，例如英语。然后对翻译后的文本执行指定的任务，如生成、分类或任何其他 NLP 功能，并可选择在需要时将输出翻译回原始语言。所有这些步骤都在一个提示中指定。我们通过对低资源印度语马拉地语的案例研究证明了该方法的有效性。CoTR 策略应用于各种任务，包括情绪分析、仇恨言论分类、主题分类和文本生成，并通过将其与常规提示方法进行比较来展示其有效性。我们的结果强调了基于翻译的提示策略在低资源语言中显着提高多语言 LLM 性能的潜力，为未来的研究和应用提供了宝贵的见解。我们特别看到了仇恨言论检测任务的最高准确率提升。该技术还有可能提高使用 LLM 的代表性不足的语言的合成数据生成的质量。

Title: How Does Code Pretraining Affect Language Model Task Performance?

• Authors: Jackson Petty, Sjoerd van Steenkiste, Tal Linzen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04556
• Pdf URL: https://arxiv.org/pdf/2409.04556
• Copy Paste: [[2409.04556]] How Does Code Pretraining Affect Language Model Task Performance?(https://arxiv.org/abs/2409.04556)
• Keywords: language model
• Abstract: Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data. Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: additive, in which the total volume of data seen during pretraining is held constant; and competitive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) a diverse collection of tasks included in the BigBench benchmark, and (b) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformations. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.
• 摘要：大型语言模型越来越多地在包含自然语言和非语言数据（如源代码）的语料库上进行训练。除了帮助完成与编程相关的任务外，有传闻证据表明，在预训练语料库中包含代码可能会提高其他不相关任务的性能，但迄今为止，还没有任何工作能够通过控制语言和代码数据来建立因果关系。我们在这里就是这么做的。我们在两种不同设置中交错自然语言和代码的数据集上对语言模型进行预训练：加法，其中预训练期间看到的总数据量保持不变；竞争，其中语言数据量保持不变。我们研究预训练混合物如何影响（a）BigBench 基准中包含的多种任务集合的性能，以及（b）组合性，通过语义解析和句法转换的泛化准确性来衡量。我们发现，对更高比例的代码进行预训练可以提高涉及结构化输出（如语义解析）和数学的组合任务的性能。相反，增加代码混合可能会损害其他任务的性能，包括需要对语法或形态等语言结构敏感性的任务，以及衡量现实世界知识的任务。

Title: Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning

• Authors: Xinyue Liu, Harshita Diddee, Daphne Ippolito
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04574
• Pdf URL: https://arxiv.org/pdf/2409.04574
• Copy Paste: [[2409.04574]] Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning(https://arxiv.org/abs/2409.04574)
• Keywords: language model, llm
• Abstract: One-size-fits-all large language models (LLMs) are increasingly being used to help people with their writing. However, the style these models are trained to write in may not suit all users or use cases. LLMs would be more useful as writing assistants if their idiolect could be customized to match each user. In this paper, we explore whether parameter-efficient finetuning (PEFT) with Low-Rank Adaptation can effectively guide the style of LLM generations. We use this method to customize LLaMA-2 to ten different authors and show that the generated text has lexical, syntactic, and surface alignment with the target author but struggles with content memorization. Our findings highlight the potential of PEFT to support efficient, user-level customization of LLMs.
• 摘要：一刀切的大型语言模型 (LLM) 越来越多地被用于帮助人们写作。然而，这些模型训练的写作风格可能并不适合所有用户或用例。如果 LLM 的个人语言可以根据每个用户进行定制，那么它们作为写作助手会更有用。在本文中，我们探讨了具有低秩自适应的参数高效微调 (PEFT) 是否可以有效指导 LLM 生成的风格。我们使用此方法将 LLaMA-2 定制为十位不同的作者，并表明生成的文本在词汇、句法和表面方面与目标作者一致，但在内容记忆方面存在困难。我们的研究结果凸显了 PEFT 支持高效、用户级 LLM 定制的潜力。

Title: Paper Copilot: A Self-Evolving and Efficient LLM System for Personalized Academic Assistance

• Authors: Guanyu Lin, Tao Feng, Pengrui Han, Ge Liu, Jiaxuan You
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04593
• Pdf URL: https://arxiv.org/pdf/2409.04593
• Copy Paste: [[2409.04593]] Paper Copilot: A Self-Evolving and Efficient LLM System for Personalized Academic Assistance(https://arxiv.org/abs/2409.04593)
• Keywords: llm
• Abstract: As scientific research proliferates, researchers face the daunting task of navigating and reading vast amounts of literature. Existing solutions, such as document QA, fail to provide personalized and up-to-date information efficiently. We present Paper Copilot, a self-evolving, efficient LLM system designed to assist researchers, based on thought-retrieval, user profile and high performance optimization. Specifically, Paper Copilot can offer personalized research services, maintaining a real-time updated database. Quantitative evaluation demonstrates that Paper Copilot saves 69.92\% of time after efficient deployment. This paper details the design and implementation of Paper Copilot, highlighting its contributions to personalized academic support and its potential to streamline the research process.
• 摘要：随着科学研究的蓬勃发展，研究人员面临着浏览和阅读大量文献的艰巨任务。现有的解决方案（例如文档问答）无法有效地提供个性化和最新的信息。我们推出了 Paper Copilot，这是一个自我进化的高效 LLM 系统，旨在基于思维检索、用户配置文件和高性能优化来协助研究人员。具体来说，Paper Copilot 可以提供个性化的研究服务，维护实时更新的数据库。定量评估表明，Paper Copilot 在高效部署后可节省 69.92% 的时间。本文详细介绍了 Paper Copilot 的设计和实施，强调了它对个性化学术支持的贡献以及简化研究流程的潜力。

Title: BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

• Authors: Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, Ivan P. Yamshchikov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04599
• Pdf URL: https://arxiv.org/pdf/2409.04599
• Copy Paste: [[2409.04599]] BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training(https://arxiv.org/abs/2409.04599)
• Keywords: language model
• Abstract: Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that our method does not reduce the downstream performance, and in several cases improves it.
• 摘要：语言模型可以从高效的标记化中受益匪浅。然而，它们仍然主要使用经典的 BPE 算法，这是一种简单可靠的方法。事实证明，这会导致诸如训练不足的标记和次优压缩等问题，从而影响下游性能。我们引入了 Picky BPE，这是一种经过修改的 BPE 算法，可在标记器训练期间执行词汇细化。我们的方法提高了词汇效率，消除了训练不足的标记，并且不会影响文本压缩。我们的实验表明，我们的方法不会降低下游性能，并且在某些情况下会提高性能。

Title: Sparse Rewards Can Self-Train Dialogue Agents

• Authors: Barrett Martin Lattimer, Varun Gangal, Ryan McDonald, Yi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04617
• Pdf URL: https://arxiv.org/pdf/2409.04617
• Copy Paste: [[2409.04617]] Sparse Rewards Can Self-Train Dialogue Agents(https://arxiv.org/abs/2409.04617)
• Keywords: language model, llm, agent
• Abstract: Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub.
• 摘要：最新 (SOTA) 大型语言模型 (LLM) 代理的最新进展，尤其是在多轮对话任务中，主要由监督微调和高质量人工反馈推动。然而，随着基础 LLM 模型的不断改进，获取有意义的人工反馈变得越来越具有挑战性和成本高昂。在某些领域，基础 LLM 代理最终可能会超越人类的能力，从而使传统的反馈驱动方法变得不切实际。在本文中，我们介绍了一种新颖的自我改进范式，使 LLM 代理能够在没有外部人工反馈的情况下自主提高其性能。我们的方法，并列模拟收获结果 (JOSH)，是一种自对齐算法，它利用稀疏奖励模拟环境来提取理想行为并进一步在其自身输出上训练 LLM。我们提出了 ToolWOZ，这是一个源自 MultiWOZ 的稀疏奖励工具调用模拟环境。我们证明使用 JOSH 训练的模型（无论是小型模型还是前沿模型）都可以显着改善基于工具的交互，同时保留跨不同基准的一般模型能力。我们的代码和数据在 GitHub 上公开。

Title: Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

• Authors: Michael Günther, Isabelle Mohr, Bo Wang, Han Xiao
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.04701
• Pdf URL: https://arxiv.org/pdf/2409.04701
• Copy Paste: [[2409.04701]] Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models(https://arxiv.org/abs/2409.04701)
• Keywords: long context
• Abstract: Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be "over-compressed" in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in suboptimal representations. In this paper, we introduce a novel method called "late chunking," which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks without the need for additional training. Moreover, our method is generic enough to be applied to any long-context embedding model.
• 摘要：许多用例需要检索较小的文本部分，而基于密集向量的检索系统通常在较短的文本段中表现更好，因为语义在嵌入中“过度压缩”的可能性较小。因此，从业者经常将文本文档拆分成较小的块并分别对其进行编码。但是，以这种方式创建的块嵌入可能会丢失周围块的上下文信息，从而导致表示不理想。在本文中，我们介绍了一种称为“后期分块”的新方法，该方法利用长上下文嵌入模型首先嵌入长文本的所有标记，在转换器模型之后和均值池之前应用分块。生成的块嵌入捕获了完整的上下文信息，无需额外训练即可在各种检索任务中获得出色的结果。此外，我们的方法足够通用，可以应用于任何长上下文嵌入模型。

Title: Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

• Authors: Junfeng Tian, Da Zheng, Yang Cheng, Rui Wang, Colin Zhang, Debing Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04774
• Pdf URL: https://arxiv.org/pdf/2409.04774
• Copy Paste: [[2409.04774]] Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models(https://arxiv.org/abs/2409.04774)
• Keywords: language model, llm, long context
• Abstract: Large language models (LLM) have prioritized expanding the context window from which models can incorporate more information. However, training models to handle long contexts presents significant challenges. These include the scarcity of high-quality natural long-context data, the potential for performance degradation on short-context tasks, and the reduced training efficiency associated with attention mechanisms. In this paper, we introduce Untie the Knots (\textbf{UtK}), a novel data augmentation strategy employed during the continue pre-training phase, designed to efficiently enable LLMs to gain long-context capabilities without the need to modify the existing data mixture. In particular, we chunk the documents, shuffle the chunks, and create a complex and knotted structure of long texts; LLMs are then trained to untie these knots and identify relevant segments within seemingly chaotic token sequences. This approach greatly improves the model's performance by accurately attending to relevant information in long context and the training efficiency is also largely increased. We conduct extensive experiments on models with 7B and 72B parameters, trained on 20 billion tokens, demonstrating that UtK achieves 75\% and 84.5\% accurracy on RULER at 128K context length, significantly outperforming other long context strategies. The trained models will open-source for further research.
• 摘要：大型语言模型 (LLM) 优先扩展上下文窗口，以便模型可以整合更多信息。然而，训练模型来处理长上下文提出了重大挑战。这些挑战包括高质量自然长上下文数据的稀缺、短上下文任务的性能下降的可能性以及与注意力机制相关的训练效率降低。在本文中，我们引入了 Untie the Knots (\textbf{UtK})，这是一种在持续预训练阶段采用的新型数据增强策略，旨在有效地使 LLM 获得长上下文能力，而无需修改现有的数据混合。具体来说，我们对文档进行分块、打乱块顺序，并创建一个复杂而打结的长文本结构；然后训练 LLM 解开这些结并在看似混乱的标记序列中识别相关段。这种方法通过准确地关注长上下文中的相关信息，大大提高了模型的性能，训练效率也大大提高。我们对具有 7B 和 72B 参数的模型进行了广泛的实验，并在 200 亿个标记上进行了训练，结果表明 UtK 在 128K 上下文长度的 RULER 上实现了 75\% 和 84.5\% 的准确率，明显优于其他长上下文策略。经过训练的模型将开源以供进一步研究。

Title: Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models

• Authors: Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Mayank Mishra, Gaurav Pandey, Dinesh Raghu, Sachindra Joshi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04787
• Pdf URL: https://arxiv.org/pdf/2409.04787
• Copy Paste: [[2409.04787]] Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models(https://arxiv.org/abs/2409.04787)
• Keywords: language model, llm
• Abstract: Fine-tuning Large Language Models (LLMs) on specific datasets is a common practice to improve performance on target tasks. However, this performance gain often leads to overfitting, where the model becomes too specialized in either the task or the characteristics of the training data, resulting in a loss of generalization. This paper introduces Selective Self-Rehearsal (SSR), a fine-tuning approach that achieves performance comparable to the standard supervised fine-tuning (SFT) while improving generalization. SSR leverages the fact that there can be multiple valid responses to a query. By utilizing the model's correct responses, SSR reduces model specialization during the fine-tuning stage. SSR first identifies the correct model responses from the training set by deploying an appropriate LLM as a judge. Then, it fine-tunes the model using the correct model responses and the gold response for the remaining samples. The effectiveness of SSR is demonstrated through experiments on the task of identifying unanswerable queries across various datasets. The results show that standard SFT can lead to an average performance drop of up to $16.7\%$ on multiple benchmarks, such as MMLU and TruthfulQA. In contrast, SSR results in close to $2\%$ drop on average, indicating better generalization capabilities compared to standard SFT.
• 摘要：在特定数据集上微调大型语言模型 (LLM) 是提高目标任务性能的常见做法。然而，这种性能提升往往会导致过度拟合，即模型在任务或训练数据的特征上变得过于专业化，从而导致泛化能力下降。本文介绍了选择性自我排练 (SSR)，这是一种微调方法，可实现与标准监督微调 (SFT) 相当的性能，同时提高泛化能力。SSR 利用查询可以有多个有效响应这一事实。通过利用模型的正确响应，SSR 在微调阶段减少了模型专业化。SSR 首先通过部署适当的 LLM 作为判断者从训练集中识别正确的模型响应。然后，它使用正确的模型响应和剩余样本的黄金响应对模型进行微调。通过在各种数据集上识别无法回答的查询的任务上进行的实验证明了 SSR 的有效性。结果表明，标准 SFT 在 MMLU 和 TruthfulQA 等多个基准测试中平均性能下降高达 $16.7\%$。相比之下，SSR 平均性能下降近 $2\%$，表明与标准 SFT 相比，其泛化能力更佳。

Title: Exploring Straightforward Conversational Red-Teaming

• Authors: George Kour, Naama Zwerdling, Marcel Zalmanovici, Ateret Anaby-Tavor, Ora Nova Fandina, Eitan Farchi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04822
• Pdf URL: https://arxiv.org/pdf/2409.04822
• Copy Paste: [[2409.04822]] Exploring Straightforward Conversational Red-Teaming(https://arxiv.org/abs/2409.04822)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly used in business dialogue systems but they pose security and ethical risks. Multi-turn conversations, where context influences the model's behavior, can be exploited to produce undesired responses. In this paper, we examine the effectiveness of utilizing off-the-shelf LLMs in straightforward red-teaming approaches, where an attacker LLM aims to elicit undesired output from a target LLM, comparing both single-turn and conversational red-teaming tactics. Our experiments offer insights into various usage strategies that significantly affect their performance as red teamers. They suggest that off-the-shelf models can act as effective red teamers and even adjust their attack strategy based on past attempts, although their effectiveness decreases with greater alignment.
• 摘要：大型语言模型 (LLM) 在商业对话系统中的应用越来越广泛，但它们也带来了安全和道德风险。多轮对话（其中上下文会影响模型的行为）可能会被利用来产生不受欢迎的响应。在本文中，我们研究了在直接的红队攻击方法中使用现成的 LLM 的有效性，其中攻击者 LLM 旨在从目标 LLM 中引出不受欢迎的输出，并比较了单轮和对话式红队攻击策略。我们的实验提供了对各种使用策略的见解，这些策略会显著影响他们作为红队成员的表现。他们表明，现成的模型可以充当有效的红队成员，甚至可以根据过去的尝试调整他们的攻击策略，尽管它们的有效性会随着一致性的增加而降低。

Title: Achieving Peak Performance for Large Language Models: A Systematic Review

• Authors: Zhyar Rzgar K Rostam, Sándor Szénási, Gábor Kertész
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04833
• Pdf URL: https://arxiv.org/pdf/2409.04833
• Copy Paste: [[2409.04833]] Achieving Peak Performance for Large Language Models: A Systematic Review(https://arxiv.org/abs/2409.04833)
• Keywords: language model, llm
• Abstract: In recent years, large language models (LLMs) have achieved remarkable success in natural language processing (NLP). LLMs require an extreme amount of parameters to attain high performance. As models grow into the trillion-parameter range, computational and memory costs increase significantly. This makes it difficult for many researchers to access the resources needed to train or apply these models. Optimizing LLM performance involves two main approaches: fine-tuning pre-trained models for specific tasks to achieve state-of-the-art performance, and reducing costs or improving training time while maintaining similar performance. This paper presents a systematic literature review (SLR) following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We reviewed 65 publications out of 983 from 2017 to December 2023, retrieved from 5 databases. The study presents methods to optimize and accelerate LLMs while achieving cutting-edge results without sacrificing accuracy. We begin with an overview of the development of language modeling, followed by a detailed explanation of commonly used frameworks and libraries, and a taxonomy for improving and speeding up LLMs based on three classes: LLM training, LLM inference, and system serving. We then delve into recent optimization and acceleration strategies such as training optimization, hardware optimization, scalability and reliability, accompanied by the taxonomy and categorization of these strategies. Finally, we provide an in-depth comparison of each class and strategy, with two case studies on optimizing model training and enhancing inference efficiency. These case studies showcase practical approaches to address LLM resource limitations while maintaining performance.
• 摘要：近年来，大型语言模型 (LLM) 在自然语言处理 (NLP) 中取得了显著的成功。LLM 需要大量参数才能实现高性能。随着模型参数数量增长到万亿级，计算和内存成本显著增加。这使得许多研究人员难以获得训练或应用这些模型所需的资源。优化 LLM 性能涉及两种主要方法：针对特定任务微调预训练模型以实现最先进的性能，以及在保持类似性能的同时降低成本或缩短训练时间。本文根据系统评价和荟萃分析的首选报告项目 (PRISMA) 声明进行了系统文献综述 (SLR)。我们审查了 2017 年至 2023 年 12 月从 5 个数据库中检索到的 983 篇出版物中的 65 篇。该研究提出了优化和加速 LLM 的方法，同时在不牺牲准确性的情况下实现尖端结果。我们首先概述语言建模的发展，然后详细解释常用的框架和库，并根据三个类别（LLM 训练、LLM 推理和系统服务）对改进和加速 LLM 进行分类。然后，我们深入研究最近的优化和加速策略，例如训练优化、硬件优化、可扩展性和可靠性，并对这些策略进行分类和分类。最后，我们对每个类别和策略进行了深入比较，并提供了两个关于优化模型训练和提高推理效率的案例研究。这些案例研究展示了在保持性能的同时解决 LLM 资源限制的实用方法。

Title: Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue

• Authors: Junkai Wu, Xulin Fan, Bo-Ru Lu, Xilin Jiang, Nima Mesgarani, Mark Hasegawa-Johnson, Mari Ostendorf
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.04927
• Pdf URL: https://arxiv.org/pdf/2409.04927
• Copy Paste: [[2409.04927]] Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue(https://arxiv.org/abs/2409.04927)
• Keywords: language model, llm
• Abstract: In recent years, we have observed a rapid advancement in speech language models (SpeechLLMs), catching up with humans' listening and reasoning abilities. Remarkably, SpeechLLMs have demonstrated impressive spoken dialogue question-answering (SQA) performance in benchmarks like Gaokao, the English listening test of the college entrance exam in China, which seemingly requires understanding both the spoken content and voice characteristics of speakers in a conversation. However, after carefully examining Gaokao's questions, we find the correct answers to many questions can be inferred from the conversation context alone without identifying the speaker asked in the question. Our evaluation of state-of-the-art models Qwen-Audio and WavLLM in both Gaokao and our proposed "What Do You Like?" dataset shows a significantly higher accuracy in these context-based questions than in identity-critical questions, which can only be answered correctly with correct speaker identification. Our results and analysis suggest that when solving SQA, the current SpeechLLMs exhibit limited speaker awareness from the audio and behave similarly to an LLM reasoning from the conversation transcription without sound. We propose that our definitions and automated classification of context-based and identity-critical questions could offer a more accurate evaluation framework of SpeechLLMs in SQA tasks.
• 摘要：近年来，我们观察到语音语言模型 (SpeechLLM) 的快速发展，赶上了人类的听力和推理能力。值得注意的是，SpeechLLM 在高考等基准测试中表现出令人印象深刻的口语对话问答 (SQA) 性能，高考是中国大学入学考试的英语听力测试，这似乎需要理解对话中说话者的口语内容和语音特征。然而，在仔细研究了高考的问题后，我们发现许多问题的正确答案可以仅从对话上下文中推断出来，而无需识别问题中提到的说话者。我们对高考和我们提出的“你喜欢什么？”数据集中最先进的模型 Qwen-Audio 和 WavLLM 的评估表明，这些基于上下文的问题的准确率明显高于身份关键问题，身份关键问题只有通过正确的说话者识别才能正确回答。我们的结果和分析表明，在解决 SQA 时，当前的 SpeechLLM 对音频中的说话者意识有限，其行为类似于 LLM 从没有声音的对话转录中进行推理。我们认为，我们对基于上下文和身份关键问题的定义和自动分类可以为 SQA 任务中的 SpeechLLM 提供更准确的评估框架。

Title: Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities

• Authors: Anushka Swarup, Avanti Bhandarkar, Olivia P. Dizon-Paradis, Ronald Wilson, Damon L. Woodard
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04934
• Pdf URL: https://arxiv.org/pdf/2409.04934
• Copy Paste: [[2409.04934]] Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities(https://arxiv.org/abs/2409.04934)
• Keywords: language model, chat
• Abstract: Relation extraction is a Natural Language Processing task aiming to extract relationships from textual data. It is a critical step for information extraction. Due to its wide-scale applicability, research in relation extraction has rapidly scaled to using highly advanced neural networks. Despite their computational superiority, modern relation extractors fail to handle complicated extraction scenarios. However, a comprehensive performance analysis of the state-of-the-art relation extractors that compile these challenges has been missing from the literature, and this paper aims to bridge this gap. The goal has been to investigate the possible data-centric characteristics that impede neural relation extraction. Based on extensive experiments conducted using 15 state-of-the-art relation extraction algorithms ranging from recurrent architectures to large language models and seven large-scale datasets, this research suggests that modern relation extractors are not robust to complex data and relation characteristics. It emphasizes pivotal issues, such as contextual ambiguity, correlating relations, long-tail data, and fine-grained relation distributions. In addition, it sets a marker for future directions to alleviate these issues, thereby proving to be a critical resource for novice and advanced researchers. Efficient handling of the challenges described can have significant implications for the field of information extraction, which is a critical part of popular systems such as search engines and chatbots. Data and relevant code can be found at this https URL.
• 摘要：关系提取是一项自然语言处理任务，旨在从文本数据中提取关系。这是信息提取的关键步骤。由于其广泛的适用性，关系提取研究已迅速扩展到使用高度先进的神经网络。尽管现代关系提取器具有计算优势，但它们无法处理复杂的提取场景。然而，文献中缺少对这些挑战的最先进的关系提取器的全面性能分析，本文旨在弥补这一空白。目标是研究可能阻碍神经关系提取的数据中心特性。基于使用 15 种最先进的关系提取算法（从递归架构到大型语言模型）和 7 个大型数据集进行的大量实验，这项研究表明现代关系提取器对复杂数据和关系特征不具有鲁棒性。它强调了关键问题，例如上下文歧义、关联关系、长尾数据和细粒度关系分布。此外，它为缓解这些问题的未来方向设定了标杆，从而成为新手和高级研究人员的重要资源。有效处理所描述的挑战对信息提取领域具有重要意义，而信息提取是搜索引擎和聊天机器人等流行系统的关键部分。数据和相关代码可在此 https URL 中找到。

Title: Evaluation of Google Translate for Mandarin Chinese translation using sentiment and semantic analysis

• Authors: Xuechun Wang, Rodney Beard, Rohitash Chandra
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04964
• Pdf URL: https://arxiv.org/pdf/2409.04964
• Copy Paste: [[2409.04964]] Evaluation of Google Translate for Mandarin Chinese translation using sentiment and semantic analysis(https://arxiv.org/abs/2409.04964)
• Keywords: language model, llm
• Abstract: Machine translation using large language models (LLMs) is having a significant global impact, making communication easier. Mandarin Chinese is the official language used for communication by the government, education institutes, and media in China. In this study, we provide an automated assessment of machine translation models with human experts using sentiment and semantic analysis. In order to demonstrate our framework, we select classic early twentieth-century novel 'The True Story of Ah Q' with selected Mandarin Chinese to English translations. We also us Google Translate to generate the given text into English and then conduct a chapter-wise sentiment analysis and semantic analysis to compare the extracted sentiments across the different translations. We utilise LLMs for semantic and sentiment analysis. Our results indicate that the precision of Google Translate differs both in terms of semantic and sentiment analysis when compared to human expert translations. We find that Google Translate is unable to translate some of the specific words or phrases in Chinese, such as Chinese traditional allusions. The mistranslations have to its lack of contextual significance and historical knowledge of China. Thus, this framework brought us some new insights about machine translation for Chinese Mandarin. The future work can explore other languages or types of texts with this framework.
• 摘要：使用大型语言模型 (LLM) 的机器翻译正在对全球产生重大影响，使交流更加容易。普通话是中国政府、教育机构和媒体用于交流的官方语言。在本研究中，我们使用情感和语义分析对机器翻译模型进行了自动评估，并由人类专家进行评估。为了展示我们的框架，我们选择了经典的二十世纪早期小说《阿 Q 正传》，并选择了普通话到英文的翻译。我们还使用谷歌翻译将给定的文本生成英文，然后进行章节情感分析和语义分析，以比较不同翻译中提取的情感。我们利用 LLM 进行语义和情感分析。我们的结果表明，与人类专家翻译相比，谷歌翻译在语义和情感分析方面的准确性都不同。我们发现谷歌翻译无法翻译一些特定的中文单词或短语，例如中国传统典故。误译是由于它缺乏语境意义和对中国的历史知识。因此，这个框架为我们带来了一些关于中文普通话机器翻译的新见解。未来的工作可以用这个框架探索其他语言或类型的文本。

Title: Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation

• Authors: Yanni Xue, Haojie Hao, Jiakai Wang, Qiang Sheng, Renshuai Tao, Yu Liang, Pu Feng, Xianglong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05021
• Pdf URL: https://arxiv.org/pdf/2409.05021
• Copy Paste: [[2409.05021]] Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation(https://arxiv.org/abs/2409.05021)
• Keywords: language model, gpt, llm
• Abstract: While neural machine translation (NMT) models achieve success in our daily lives, they show vulnerability to adversarial attacks. Despite being harmful, these attacks also offer benefits for interpreting and enhancing NMT models, thus drawing increased research attention. However, existing studies on adversarial attacks are insufficient in both attacking ability and human imperceptibility due to their sole focus on the scope of language. This paper proposes a novel vision-fused attack (VFA) framework to acquire powerful adversarial text, i.e., more aggressive and stealthy. Regarding the attacking ability, we design the vision-merged solution space enhancement strategy to enlarge the limited semantic solution space, which enables us to search for adversarial candidates with higher attacking ability. For human imperceptibility, we propose the perception-retained adversarial text selection strategy to align the human text-reading mechanism. Thus, the finally selected adversarial text could be more deceptive. Extensive experiments on various models, including large language models (LLMs) like LLaMA and GPT-3.5, strongly support that VFA outperforms the comparisons by large margins (up to 81%/14% improvements on ASR/SSIM).
• 摘要：虽然神经机器翻译 (NMT) 模型在我们的日常生活中取得了成功，但它们也容易受到对抗性攻击。尽管这些攻击有害，但也为解释和增强 NMT 模型提供了好处，因此引起了越来越多的研究关注。然而，现有的对抗性攻击研究仅关注语言范围，在攻击能力和人类不可感知性方面都存在不足。本文提出了一种新颖的视觉融合攻击 (VFA) 框架来获取强大的对抗性文本，即更具攻击性和隐蔽性。关于攻击能力，我们设计了视觉融合解决方案空间增强策略来扩大有限的语义解决方案空间，这使我们能够搜索具有更高攻击能力的对抗性候选者。对于人类不可感知性，我们提出了感知保留的对抗性文本选择策略来与人类文本阅读机制保持一致。因此，最终选择的对抗性文本可能更具欺骗性。对各种模型（包括 LLaMA 和 GPT-3.5 等大型语言模型 (LLM)）进行的大量实验有力地证明了 VFA 的表现远胜于比较结果（ASR/SSIM 上分别提高了 81%/14%）。

Title: WaterSeeker: Efficient Detection of Watermarked Segments in Large Documents

• Authors: Leyi Pan, Aiwei Liu, Yijian Lu, Zitian Gao, Yichen Di, Lijie Wen, Irwin King, Philip S. Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05112
• Pdf URL: https://arxiv.org/pdf/2409.05112
• Copy Paste: [[2409.05112]] WaterSeeker: Efficient Detection of Watermarked Segments in Large Documents(https://arxiv.org/abs/2409.05112)
• Keywords: language model, llm
• Abstract: Watermarking algorithms for large language models (LLMs) have attained high accuracy in detecting LLM-generated text. However, existing methods primarily focus on distinguishing fully watermarked text from non-watermarked text, overlooking real-world scenarios where LLMs generate only small sections within large documents. In this scenario, balancing time complexity and detection performance poses significant challenges. This paper presents WaterSeeker, a novel approach to efficiently detect and locate watermarked segments amid extensive natural text. It first applies an efficient anomaly extraction method to preliminarily locate suspicious watermarked regions. Following this, it conducts a local traversal and performs full-text detection for more precise verification. Theoretical analysis and experimental results demonstrate that WaterSeeker achieves a superior balance between detection accuracy and computational efficiency. Moreover, WaterSeeker's localization ability supports the development of interpretable AI detection systems. This work pioneers a new direction in watermarked segment detection, facilitating more reliable AI-generated content identification.
• 摘要：大型语言模型 (LLM) 的水印算法在检测 LLM 生成的文本方面已达到很高的准确率。然而，现有方法主要侧重于区分完全带水印的文本和无水印的文本，而忽略了 LLM 仅在大型文档中生成小部分的实际场景。在这种情况下，平衡时间复杂度和检测性能带来了重大挑战。本文提出了 WaterSeeker，一种在大量自然文本中有效检测和定位带水印片段的新方法。它首先应用一种有效的异常提取方法来初步定位可疑的水印区域。然后，它进行局部遍历并执行全文检测以进行更精确的验证。理论分析和实验结果表明，WaterSeeker 在检测精度和计算效率之间实现了出色的平衡。此外，WaterSeeker 的定位能力支持可解释的 AI 检测系统的开发。这项工作开辟了带水印片段检测的新方向，促进了更可靠的 AI 生成内容识别。

Title: OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs

• Authors: Jintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05152
• Pdf URL: https://arxiv.org/pdf/2409.05152
• Copy Paste: [[2409.05152]] OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs(https://arxiv.org/abs/2409.05152)
• Keywords: language model, llm
• Abstract: Despite the recent advancements in Large Language Models (LLMs), which have significantly enhanced the generative capabilities for various NLP tasks, LLMs still face limitations in directly handling retrieval tasks. However, many practical applications demand the seamless integration of both retrieval and generation. This paper introduces a novel and efficient One-pass Generation and retrieval framework (OneGen), designed to improve LLMs' performance on tasks that require both generation and retrieval. The proposed framework bridges the traditionally separate training approaches for generation and retrieval by incorporating retrieval tokens generated autoregressively. This enables a single LLM to handle both tasks simultaneously in a unified forward pass. We conduct experiments on two distinct types of composite tasks, RAG and Entity Linking, to validate the pluggability, effectiveness, and efficiency of OneGen in training and inference. Furthermore, our results show that integrating generation and retrieval within the same context preserves the generative capabilities of LLMs while improving retrieval performance. To the best of our knowledge, OneGen is the first to enable LLMs to conduct vector retrieval during the generation.
• 摘要：尽管大型语言模型 (LLM) 近期取得了进展，大大增强了各种 NLP 任务的生成能力，但 LLM 在直接处理检索任务方面仍然面临限制。然而，许多实际应用需要无缝集成检索和生成。本文介绍了一种新颖而高效的一次性生成和检索框架 (OneGen)，旨在提高 LLM 在需要生成和检索的任务上的性能。所提出的框架通过合并自回归生成的检索标记，弥合了传统上分开的生成和检索训练方法。这使得单个 LLM 能够在统一的前向传递中同时处理这两个任务。我们对两种不同类型的复合任务 RAG 和实体链接进行了实验，以验证 OneGen 在训练和推理中的可插入性、有效性和效率。此外，我们的结果表明，在同一上下文中集成生成和检索可以保留 LLM 的生成能力，同时提高检索性能。据我们所知，OneGen 是第一个允许 LLM 在生成过程中进行向量检索的系统。

Title: Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?

• Authors: Neeladri Bhuiya, Viktor Schlegel, Stefan Winkler
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05197
• Pdf URL: https://arxiv.org/pdf/2409.05197
• Copy Paste: [[2409.05197]] Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?(https://arxiv.org/abs/2409.05197)
• Keywords: language model, llm
• Abstract: State-of-the-art Large Language Models (LLMs) are accredited with an increasing number of different capabilities, ranging from reading comprehension, over advanced mathematical and reasoning skills to possessing scientific knowledge. In this paper we focus on their multi-hop reasoning capability: the ability to identify and integrate information from multiple textual sources. Given the concerns with the presence of simplifying cues in existing multi-hop reasoning benchmarks, which allow models to circumvent the reasoning requirement, we set out to investigate, whether LLMs are prone to exploiting such simplifying cues. We find evidence that they indeed circumvent the requirement to perform multi-hop reasoning, but they do so in more subtle ways than what was reported about their fine-tuned pre-trained language model (PLM) predecessors. Motivated by this finding, we propose a challenging multi-hop reasoning benchmark, by generating seemingly plausible multi-hop reasoning chains, which ultimately lead to incorrect answers. We evaluate multiple open and proprietary state-of-the-art LLMs, and find that their performance to perform multi-hop reasoning is affected, as indicated by up to 45% relative decrease in F1 score when presented with such seemingly plausible alternatives. We conduct a deeper analysis and find evidence that while LLMs tend to ignore misleading lexical cues, misleading reasoning paths indeed present a significant challenge.
• 摘要：最先进的大型语言模型 (LLM) 被认可拥有越来越多的不同能力，从阅读理解、高级数学和推理技能到拥有科学知识。在本文中，我们重点介绍它们的多跳推理能力：识别和整合来自多个文本源的信息的能力。考虑到对现有多跳推理基准中存在的简化线索的担忧，这些线索允许模型规避推理要求，我们着手调查 LLM 是否容易利用此类简化线索。我们发现证据表明它们确实规避了执行多跳推理的要求，但它们这样做的方式比其经过微调的预训练语言模型 (PLM) 前身报道的方式更微妙。受这一发现的启发，我们提出了一个具有挑战性的多跳推理基准，通过生成看似合理的多跳推理链，最终导致错误答案。我们评估了多个开放和专有的最先进的 LLM，发现它们执行多跳推理的性能受到影响，当呈现这些看似合理的替代方案时，F1 分数相对下降高达 45%。我们进行了更深入的分析，发现证据表明，虽然 LLM 倾向于忽略误导性的词汇线索，但误导性的推理路径确实带来了重大挑战。

Title: Interactive Machine Teaching by Labeling Rules and Instances

• Authors: Giannis Karamanolakis, Daniel Hsu, Luis Gravano
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05199
• Pdf URL: https://arxiv.org/pdf/2409.05199
• Copy Paste: [[2409.05199]] Interactive Machine Teaching by Labeling Rules and Instances(https://arxiv.org/abs/2409.05199)
• Keywords: language model, prompt
• Abstract: Weakly supervised learning aims to reduce the cost of labeling data by using expert-designed labeling rules. However, existing methods require experts to design effective rules in a single shot, which is difficult in the absence of proper guidance and tooling. Therefore, it is still an open question whether experts should spend their limited time writing rules or instead providing instance labels via active learning. In this paper, we investigate how to exploit an expert's limited time to create effective supervision. First, to develop practical guidelines for rule creation, we conduct an exploratory analysis of diverse collections of existing expert-designed rules and find that rule precision is more important than coverage across datasets. Second, we compare rule creation to individual instance labeling via active learning and demonstrate the importance of both across 6 datasets. Third, we propose an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns (e.g., by prompting a language model), and effectiveness by soliciting expert feedback on both candidate rules and individual instances. Across 6 datasets, INTERVAL outperforms state-of-the-art weakly supervised approaches by 7% in F1. Furthermore, it requires as few as 10 queries for expert feedback to reach F1 values that existing active learning methods cannot match even with 100 queries.
• 摘要：弱监督学习旨在通过使用专家设计的标记规则来降低标记数据的成本。然而，现有的方法要求专家一次性设计有效的规则，这在缺乏适当指导和工具的情况下很难实现。因此，专家是否应该将有限的时间花在编写规则上，还是通过主动学习提供实例标签，这仍然是一个悬而未决的问题。在本文中，我们研究如何利用专家有限的时间来创建有效的监督。首先，为了制定规则创建的实用指南，我们对现有的专家设计规则的各种集合进行了探索性分析，发现规则的精确度比数据集的覆盖率更重要。其次，我们将规则创建与通过主动学习进行的单个实例标记进行比较，并在 6 个数据集中展示了两者的重要性。第三，我们提出了一个交互式学习框架 INTERVAL，该框架通过基于丰富的模式自动提取候选规则（例如，通过提示语言模型）来实现效率，并通过征求专家对候选规则和单个实例的反馈来实现有效性。在 6 个数据集中，INTERVAL 的 F1 值比最先进的弱监督方法高出 7%。此外，它仅需要 10 次查询即可获得专家反馈，从而达到现有的主动学习方法即使进行 100 次查询也无法达到的 F1 值。

Title: Socially Responsible Data for Large Multilingual Language Models

• Authors: Andrew Smart, Ben Hutchinson, Lameck Mbangula Amugongo, Suzanne Dikker, Alex Zito, Amber Ebinama, Zara Wudiri, Ding Wang, Erin van Liemt, João Sedoc, Seyi Olojo, Stanley Uwakwe, Edem Wornyo, Sonja Schmer-Galunder, Jamila Smith-Loud
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05247
• Pdf URL: https://arxiv.org/pdf/2409.05247
• Copy Paste: [[2409.05247]] Socially Responsible Data for Large Multilingual Language Models(https://arxiv.org/abs/2409.05247)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have rapidly increased in size and apparent capabilities in the last three years, but their training data is largely English text. There is growing interest in multilingual LLMs, and various efforts are striving for models to accommodate languages of communities outside of the Global North, which include many languages that have been historically underrepresented in digital realms. These languages have been coined as "low resource languages" or "long-tail languages", and LLMs performance on these languages is generally poor. While expanding the use of LLMs to more languages may bring many potential benefits, such as assisting cross-community communication and language preservation, great care must be taken to ensure that data collection on these languages is not extractive and that it does not reproduce exploitative practices of the past. Collecting data from languages spoken by previously colonized people, indigenous people, and non-Western languages raises many complex sociopolitical and ethical questions, e.g., around consent, cultural safety, and data sovereignty. Furthermore, linguistic complexity and cultural nuances are often lost in LLMs. This position paper builds on recent scholarship, and our own work, and outlines several relevant social, cultural, and ethical considerations and potential ways to mitigate them through qualitative research, community partnerships, and participatory design approaches. We provide twelve recommendations for consideration when collecting language data on underrepresented language communities outside of the Global North.
• 摘要：大型语言模型 (LLM) 在过去三年中规模和表观能力迅速增长，但其训练数据主要是英文文本。人们对多语言 LLM 的兴趣日益浓厚，各种努力都在努力建立能够适应全球北方以外社区语言的模型，其中包括许多历史上在数字领域代表性不足的语言。这些语言被称为“低资源语言”或“长尾语言”，LLM 在这些语言上的表现通常很差。虽然将 LLM 的使用扩展到更多语言可能会带来许多潜在好处，例如协助跨社区交流和语言保护，但必须非常小心，确保这些语言的数据收集不是提取性的，也不会重现过去的剥削性做法。从以前被殖民的人、土著人和非西方语言使用的语言中收集数据会引发许多复杂的社会政治和道德问题，例如围绕同意、文化安全和数据主权。此外，语言复杂性和文化细微差别在 LLM 中往往会丢失。本立场文件以近期研究成果和我们自己的工作为基础，概述了若干相关的社会、文化和道德考量因素，以及通过定性研究、社区伙伴关系和参与式设计方法缓解这些因素的潜在方法。我们提供了十二条建议，供在收集全球北方以外代表性不足的语言社区的语言数据时参考。

Title: On the Relationship between Truth and Political Bias in Language Models

• Authors: Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, Jad Kabbara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05283
• Pdf URL: https://arxiv.org/pdf/2409.05283
• Copy Paste: [[2409.05283]] On the Relationship between Truth and Political Bias in Language Models(https://arxiv.org/abs/2409.05283)
• Keywords: language model
• Abstract: Language model alignment research often attempts to ensure that models are not only helpful and harmless, but also truthful and unbiased. However, optimizing these objectives simultaneously can obscure how improving one aspect might impact the others. In this work, we focus on analyzing the relationship between two concepts essential in both language model alignment and political science: \textit{truthfulness} and \textit{political bias}. We train reward models on various popular truthfulness datasets and subsequently evaluate their political bias. Our findings reveal that optimizing reward models for truthfulness on these datasets tends to result in a left-leaning political bias. We also find that existing open-source reward models (i.e. those trained on standard human preference datasets) already show a similar bias and that the bias is larger for larger models. These results raise important questions about both the datasets used to represent truthfulness and what language models capture about the relationship between truth and politics.
• 摘要：语言模型对齐研究通常试图确保模型不仅有用且无害，而且真实且无偏见。然而，同时优化这些目标可能会掩盖改进一个方面对其他方面的影响。在这项工作中，我们专注于分析语言模型对齐和政治科学中必不可少的两个概念之间的关系：\textit{真实性} 和 \textit{政治偏见}。我们在各种流行的真实性数据集上训练奖励模型，然后评估它们的政治偏见。我们的研究结果表明，在这些数据集上优化真实性的奖励模型往往会导致左倾的政治偏见。我们还发现，现有的开源奖励模型（即在标准人类偏好数据集上训练的模型）已经显示出类似的偏见，并且模型越大，偏见越大。这些结果提出了关于用于表示真实性的数据集以及语言模型捕捉到的真相与政治之间关系的重要问题。

Title: Seek and Solve Reasoning for Table Question Answering

• Authors: Ruya Jiang, Chun Wang, Weihong Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05286
• Pdf URL: https://arxiv.org/pdf/2409.05286
• Copy Paste: [[2409.05286]] Seek and Solve Reasoning for Table Question Answering(https://arxiv.org/abs/2409.05286)
• Keywords: language model, llm, prompt
• Abstract: Table-based Question Answering (TQA) involves answering questions based on tabular data. The complexity of table structures and question logic makes this task difficult even for Large Language Models (LLMs). This paper improves TQA performance by leveraging LLMs' reasoning capabilities. Inspired by how humans solve TQA tasks, we propose a Seek-and-Solve pipeline that instructs the LLM to first seek relevant information and then answer questions. The two stages are integrated at the reasoning level, and their Chain of Thought (CoT) paths are integrated into a coherent Seek-and-Solve CoT (SS-CoT). Furthermore, we present a compact single-stage TQA-solving prompt distilled from the pipeline. Experiments demonstrate that under In-Context Learning settings, using samples with SS-CoT paths as demonstrations, the TQA-solving prompt can effectively guide the LLM to solve complex TQA tasks, resulting in improved performance and reliability. Our results highlight the importance of properly eliciting LLMs' reasoning capabilities in solving complex TQA tasks.
• 摘要：基于表格的问答系统 (TQA) 涉及根据表格数据回答问题。表格结构和问题逻辑的复杂性使得这项任务即使对于大型语言模型 (LLM) 来说也十分困难。本文通过利用 LLM 的推理能力来提高 TQA 性能。受人类解决 TQA 任务的方式的启发，我们提出了一种“寻找并解决”流程，指示 LLM 首先寻找相关信息，然后回答问题。这两个阶段在推理层面上整合在一起，它们的思路链 (CoT) 路径被整合成一个连贯的“寻找并解决”CoT (SS-CoT)。此外，我们提出了一个从流程中提炼出来的紧凑的单阶段 TQA 解决提示。实验表明，在情境学习设置下，使用具有 SS-CoT 路径的样本作为演示，TQA 解决提示可以有效地指导 LLM 解决复杂的 TQA 任务，从而提高性能和可靠性。我们的研究结果强调了在解决复杂的 TQA 任务中正确引出 LLM 的推理能力的重要性。

Title: Diagnostic Reasoning in Natural Language: Computational Model and Application

• Authors: Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05367
• Pdf URL: https://arxiv.org/pdf/2409.05367
• Copy Paste: [[2409.05367]] Diagnostic Reasoning in Natural Language: Computational Model and Application(https://arxiv.org/abs/2409.05367)
• Keywords: llm
• Abstract: Diagnostic reasoning is a key component of expert work in many domains. It is a hard, time-consuming activity that requires expertise, and AI research has investigated the ways automated systems can support this process. Yet, due to the complexity of natural language, the applications of AI for diagnostic reasoning to language-related tasks are lacking. To close this gap, we investigate diagnostic abductive reasoning (DAR) in the context of language-grounded tasks (NL-DAR). We propose a novel modeling framework for NL-DAR based on Pearl's structural causal models and instantiate it in a comprehensive study of scientific paper assessment in the biomedical domain. We use the resulting dataset to investigate the human decision-making process in NL-DAR and determine the potential of LLMs to support structured decision-making over text. Our framework, open resources and tools lay the groundwork for the empirical study of collaborative diagnostic reasoning in the age of LLMs, in the scholarly domain and beyond.
• 摘要：诊断推理是许多领域专家工作的关键组成部分。这是一项艰巨、耗时且需要专业知识的活动，而人工智能研究已经研究了自动化系统如何支持这一过程。然而，由于自然语言的复杂性，人工智能在语言相关任务中的诊断推理应用还很缺乏。为了弥补这一差距，我们在语言基础任务 (NL-DAR) 的背景下研究了诊断溯因推理 (DAR)。我们基于 Pearl 的结构因果模型提出了一种新颖的 NL-DAR 建模框架，并在一项关于生物医学领域科学论文评估的综合研究中将其实例化。我们使用生成的数据集来研究 NL-DAR 中的人类决策过程，并确定 LLM 支持文本结构化决策的潜力。我们的框架、开放资源和工具为 LLM 时代、学术领域及其他领域的协作诊断推理的实证研究奠定了基础。

Title: Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models

• Authors: Hong Xingyun Hong, Shao Yan Shao, Wang Zhilin Wang, Duan Manni Duan, Jin Xiongnan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05385
• Pdf URL: https://arxiv.org/pdf/2409.05385
• Copy Paste: [[2409.05385]] Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models(https://arxiv.org/abs/2409.05385)
• Keywords: language model, gpt, llm
• Abstract: The development of LLMs has greatly enhanced the intelligence and fluency of question answering, while the emergence of retrieval enhancement has enabled models to better utilize external information. However, the presence of noise and errors in retrieved information poses challenges to the robustness of LLMs. In this work, to evaluate the model's performance under multiple interferences, we first construct a dataset based on machine reading comprehension datasets simulating various scenarios, including critical information absence, noise, and conflicts. To address the issue of model accuracy decline caused by noisy external information, we propose a data augmentation-based fine-tuning method to enhance LLM's robustness against noise. Additionally, contrastive learning approach is utilized to preserve the model's discrimination capability of external information. We have conducted experiments on both existing LLMs and our approach, the results are evaluated by GPT-4, which indicates that our proposed methods improve model robustness while strengthening the model's discrimination capability.
• 摘要：LLM 的发展大大提升了问答的智能化和流畅性，而检索增强的出现使得模型能够更好地利用外部信息。然而，检索到的信息中存在噪声和错误，对 LLM 的鲁棒性提出了挑战。在本文中，为了评估模型在多种干扰下的性能，我们首先基于机器阅读理解数据集构建了一个数据集，模拟了各种场景，包括关键信息缺失、噪声和冲突。针对外部信息噪声导致模型准确率下降的问题，我们提出了一种基于数据增强的微调方法来增强 LLM 对噪声的鲁棒性。此外，利用对比学习方法来保持模型对外部信息的辨别能力。我们对现有的 LLM 和我们的方法都进行了实验，并通过 GPT-4 对结果进行了评估，结果表明我们提出的方法在增强模型辨别能力的同时提高了模型的鲁棒性。

Title: STLM Engineering Report: Dropout

• Authors: Dylan Hillier, Leon Guertler, Bobby Cheng, Cheston Tan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05423
• Pdf URL: https://arxiv.org/pdf/2409.05423
• Copy Paste: [[2409.05423]] STLM Engineering Report: Dropout(https://arxiv.org/abs/2409.05423)
• Keywords: language model
• Abstract: In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance firstly in the regime of improving the sample efficiency of models given small, high quality datasets, and secondly in the regime of improving the quality of its fit on larger datasets where models may underfit. We find that concordant with conventional wisdom, dropout remains effective in the overfitting scenario, and that furthermore it may have some relevance for improving the fit of models even in the case of excess data, as suggested by previous research. In the process we find that the existing explanation for the mechanism behind this performance gain is not applicable in the case of language modelling.
• 摘要：在这项研究中，我们探索了 dropout 对现代语言模型的相关性，特别是在参数规模小于 100M 的模型中。我们首先探索了 dropout 在给定小型高质量数据集的情况下提高模型采样效率的相关性，其次探索了 dropout 在模型可能欠拟合的较大数据集上提高拟合质量的相关性。我们发现，与传统观点一致，dropout 在过拟合场景中仍然有效，而且正如先前的研究表明的那样，即使在数据过剩的情况下，dropout 也可能对提高模型拟合度有一定相关性。在此过程中，我们发现，对这种性能提升背后机制的现有解释并不适用于语言建模的情况。

Title: Representational Analysis of Binding in Large Language Models

• Authors: Qin Dai, Benjamin Heinzerling, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05448
• Pdf URL: https://arxiv.org/pdf/2409.05448
• Copy Paste: [[2409.05448]] Representational Analysis of Binding in Large Language Models(https://arxiv.org/abs/2409.05448)
• Keywords: language model
• Abstract: Entity tracking is essential for complex reasoning. To perform in-context entity tracking, language models (LMs) must bind an entity to its attribute (e.g., bind a container to its content) to recall attribute for a given entity. For example, given a context mentioning ``The coffee is in Box Z, the stone is in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later, LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs, Feng and Steinhardt (2023) introduce a Binding ID mechanism and state that LMs use a abstract concept called Binding ID (BI) to internally mark entity-attribute pairs. However, they have not directly captured the BI determinant information from entity activations. In this work, we provide a novel view of the Binding ID mechanism by localizing the prototype of BI information. Specifically, we discover that there exists a low-rank subspace in the hidden state (or activation) of LMs, that primarily encodes the order of entity and attribute and which is used as the prototype of BI to causally determine the binding. To identify this subspace, we choose principle component analysis as our first attempt and it is empirically proven to be effective. Moreover, we also discover that when editing representations along directions in the subspace, LMs tend to bind a given entity to other attributes accordingly. For example, by patching activations along the BI encoding direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z contains the map''.
• 摘要：实体跟踪对于复杂推理至关重要。要执行上下文实体跟踪，语言模型 (LM) 必须将实体绑定到其属性（例如，将容器绑定到其内容）以回忆给定实体的属性。例如，给定一个提到“咖啡在盒子 Z 中，石头在盒子 M 中，地图在盒子 H 中”的上下文，为了推断“盒子 Z 包含咖啡”，LM 必须将“盒子 Z”绑定到“咖啡”。为了解释 LM 的绑定行为，Feng 和 Steinhardt (2023) 引入了一种绑定 ID 机制，并指出 LM 使用一个称为绑定 ID (BI) 的抽象概念在内部标记实体-属性对。但是，他们没有直接从实体激活中捕获 BI 决定因素信息。在这项工作中，我们通过定位 BI 信息的原型提供了绑定 ID 机制的新视图。具体来说，我们发现在 LM 的隐藏状态（或激活）中存在一个低秩子空间，它主要编码实体和属性的顺序，并用作 BI 的原型来因果地确定绑定。为了识别这个子空间，我们选择主成分分析作为我们的第一次尝试，并且经验证明它是有效的。此外，我们还发现，当沿子空间中的方向编辑表示时，LM 倾向于将给定实体相应地绑定到其他属性。例如，通过沿 BI 编码方向修补激活，我们可以让 LM 推断“盒子 Z 包含石头”和“盒子 Z 包含地图”。

Title: Elsevier Arena: Human Evaluation of Chemistry/Biology/Health Foundational Large Language Models

• Authors: Camilo Thorne, Christian Druckenbrodt, Kinga Szarkowska, Deepika Goyal, Pranita Marajan, Vijay Somanath, Corey Harper, Mao Yan, Tony Scerri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05486
• Pdf URL: https://arxiv.org/pdf/2409.05486
• Copy Paste: [[2409.05486]] Elsevier Arena: Human Evaluation of Chemistry/Biology/Health Foundational Large Language Models(https://arxiv.org/abs/2409.05486)
• Keywords: language model, gpt
• Abstract: The quality and capabilities of large language models cannot be currently fully assessed with automated, benchmark evaluations. Instead, human evaluations that expand on traditional qualitative techniques from natural language generation literature are required. One recent best-practice consists in using A/B-testing frameworks, which capture preferences of human evaluators for specific models. In this paper we describe a human evaluation experiment focused on the biomedical domain (health, biology, chemistry/pharmacology) carried out at Elsevier. In it a large but not massive (8.8B parameter) decoder-only foundational transformer trained on a relatively small (135B tokens) but highly curated collection of Elsevier datasets is compared to OpenAI's GPT-3.5-turbo and Meta's foundational 7B parameter Llama 2 model against multiple criteria. Results indicate -- even if IRR scores were generally low -- a preference towards GPT-3.5-turbo, and hence towards models that possess conversational abilities, are very large and were trained on very large datasets. But at the same time, indicate that for less massive models training on smaller but well-curated training sets can potentially give rise to viable alternatives in the biomedical domain.
• 摘要：目前，大型语言模型的质量和能力无法通过自动化的基准评估得到全面评估。相反，需要对自然语言生成文献中传统的定性技术进行扩展的人工评估。最近的一个最佳实践是使用 A/B 测试框架，该框架可以捕获人类评估者对特定模型的偏好。在本文中，我们描述了一项在爱思唯尔进行的针对生物医学领域（健康、生物学、化学/药理学）的人工评估实验。在其中，一个大型但不庞大（8.8B 参数）的解码器专用基础转换器在相对较小（135B 个标记）但经过高度策划的爱思唯尔数据集上进行训练，并与 OpenAI 的 GPT-3.5-turbo 和 Meta 的基础 7B 参数 Llama 2 模型根据多个标准进行了比较。结果表明——即使 IRR 分数通常较低——人们更喜欢 GPT-3.5-turbo，因此更喜欢具有对话能力、非常大并在非常大的数据集上进行训练的模型。但同时表明，对于规模较小的模型，在规模较小但精心策划的训练集上进行训练可能会在生物医学领域产生可行的替代方案。

Title: Harmonic Reasoning in Large Language Models

• Authors: Anna Kruspe
• Subjects: cs.CL, cs.AI, cs.SD
• Abstract URL: https://arxiv.org/abs/2409.05521
• Pdf URL: https://arxiv.org/pdf/2409.05521
• Copy Paste: [[2409.05521]] Harmonic Reasoning in Large Language Models(https://arxiv.org/abs/2409.05521)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) are becoming very popular and are used for many different purposes, including creative tasks in the arts. However, these models sometimes have trouble with specific reasoning tasks, especially those that involve logical thinking and counting. This paper looks at how well LLMs understand and reason when dealing with musical tasks like figuring out notes from intervals and identifying chords and scales. We tested GPT-3.5 and GPT-4o to see how they handle these tasks. Our results show that while LLMs do well with note intervals, they struggle with more complicated tasks like recognizing chords and scales. This points out clear limits in current LLM abilities and shows where we need to make them better, which could help improve how they think and work in both artistic and other complex areas. We also provide an automatically generated benchmark data set for the described tasks.
• 摘要：大型语言模型 (LLM) 正变得非常流行，并用于许多不同的目的，包括艺术中的创造性任务。然而，这些模型有时在特定的推理任务中会遇到困难，尤其是那些涉及逻辑思维和计数的任务。本文研究了 LLM 在处理音乐任务（例如从音程中找出音符以及识别和弦和音阶）时​​的理解和推理能力。我们测试了 GPT-3.5 和 GPT-4o，以了解它们如何处理这些任务。我们的结果表明，虽然 LLM 在音符间隔方面表现良好，但它们在识别和弦和音阶等更复杂的任务上却举步维艰。这指出了当前 LLM 能力的明显局限性，并表明了我们需要改进的地方，这有助于改善它们在艺术和其他复杂领域的思维和工作方式。我们还为所述任务提供了自动生成的基准数据集。

Title: MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery

• Authors: Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, Zhicheng Dou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05591
• Pdf URL: https://arxiv.org/pdf/2409.05591
• Copy Paste: [[2409.05591]] MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery(https://arxiv.org/abs/2409.05591)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) leverages retrieval tools to access external databases, thereby enhancing the generation quality of large language models (LLMs) through optimized context. However, the existing retrieval methods are constrained inherently, as they can only perform relevance matching between explicitly stated queries and well-formed knowledge, but unable to handle tasks involving ambiguous information needs or unstructured knowledge. Consequently, existing RAG systems are primarily effective for straightforward question-answering tasks. In this work, we propose \textbf{MemoRAG}, a novel retrieval-augmented generation paradigm empowered by long-term memory. MemoRAG adopts a dual-system architecture. On the one hand, it employs a \textit{light but long-range} LLM to form the global memory of database. Once a task is presented, it generates draft answers, cluing the retrieval tools to locate useful information within the database. On the other hand, it leverages an \textit{expensive but expressive} LLM, which generates the ultimate answer based on the retrieved information. Building on this general framework, we further optimize MemoRAG's performance by enhancing its cluing mechanism and memorization capacity. In our experiment, MemoRAG achieves superior performance across a variety of evaluation tasks, including both complex ones where conventional RAG fails and straightforward ones where RAG is commonly applied.
• 摘要：检索增强生成 (RAG) 利用检索工具访问外部数据库，从而通过优化上下文来提高大型语言模型 (LLM) 的生成质量。然而，现有的检索方法存在固有限制，因为它们只能在明确陈述的查询和格式良好的知识之间进行相关性匹配，而无法处理涉及模糊信息需求或非结构化知识的任务。因此，现有的 RAG 系统主要对简单的问答任务有效。在本文中，我们提出了 \textbf{MemoRAG}，一种由长期记忆赋能的新型检索增强生成范式。MemoRAG 采用双系统架构。一方面，它采用 \textit{轻量但长距离} LLM 来形成数据库的全局记忆。一旦提出任务，它就会生成草稿答案，为检索工具在数据库中找到有用的信息提供线索。另一方面，它利用了 \textit{昂贵但富有表现力} 的 LLM，根据检索到的信息生成最终答案。在此通用框架的基础上，我们通过增强其线索机制和记忆能力进一步优化了 MemoRAG 的性能。在我们的实验中，MemoRAG 在各种评估任务中都取得了优异的表现，包括传统 RAG 无法胜任的复杂任务和通常应用 RAG 的简单任务。

Title: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case

• Authors: Vagrant Gautam, Julius Steuer, Eileen Bingert, Ray Johns, Anne Lauscher, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05653
• Pdf URL: https://arxiv.org/pdf/2409.05653
• Copy Paste: [[2409.05653]] Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case(https://arxiv.org/abs/2409.05653)
• Keywords: language model
• Abstract: While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them with. Winogender schemas (Rudinger et al., 2018) are an influential dataset proposed to evaluate gender bias in coreference resolution, but a closer look at the data reveals issues with the instances that compromise their use for reliable evaluation, including treating different grammatical cases of pronouns in the same way, violations of template constraints, and typographical errors. We identify these issues and fix them, contributing a new dataset: Winogender 2.0. Our changes affect performance with state-of-the-art supervised coreference resolution systems as well as all model sizes of the language model FLAN-T5, with F1 dropping on average 0.1 points. We also propose a new method to evaluate pronominal bias in coreference resolution that goes beyond the binary. With this method and our new dataset which is balanced for grammatical case, we empirically demonstrate that bias characteristics vary not just across pronoun sets, but also across surface forms of those sets.
• 摘要：虽然测量共指解析中的偏见和稳健性是重要目标，但这些测量的好坏取决于我们用来测量它们的工具。Winogender 模式（Rudinger 等人，2018 年）是一个有影响力的数据集，旨在评估共指解析中的性别偏见，但仔细观察数据会发现实例存在问题，这些问题会影响其用于可靠的评估，包括以相同的方式处理代词的不同语法情况、违反模板约束和印刷错误。我们发现了这些问题并进行了修复，并贡献了一个新的数据集：Winogender 2.0。我们的更改会影响最先进的监督共指解析系统以及语言模型 FLAN-T5 的所有模型大小的性能，F1 平均下降 0.1 分。我们还提出了一种超越二元的评估共指解析中代词偏见的新方法。通过这种方法和我们新的针对语法格进行平衡的数据集，我们通过实证证明偏见特征不仅在代词集之间有所不同，而且在这些集合的表面形式之间也有所不同。

Title: Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach

• Authors: Meng Zhou, Surajsinh Parmar, Anubhav Bhatti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05732
• Pdf URL: https://arxiv.org/pdf/2409.05732
• Copy Paste: [[2409.05732]] Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach(https://arxiv.org/abs/2409.05732)
• Keywords: language model, llm
• Abstract: Open-source, multilingual medical large language models (LLMs) have the potential to serve linguistically diverse populations across different regions. Adapting generic LLMs for healthcare often requires continual pretraining, but this approach is computationally expensive and sometimes impractical. Instruction fine-tuning on a specific task may not always guarantee optimal performance due to the lack of broader domain knowledge that the model needs to understand and reason effectively in diverse scenarios. To address these challenges, we introduce two multilingual instruction fine-tuning datasets, MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in six languages. We propose a two-stage training paradigm: the first stage injects general medical knowledge using MMed-IFT, while the second stage fine-tunes task-specific multiple-choice questions with MMed-IFT-MC. Our method achieves competitive results on both English and multilingual benchmarks, striking a balance between computational efficiency and performance. We plan to make our dataset and model weights public at \url{this https URL} in the future.
• 摘要：开源、多语言医学大型语言模型 (LLM) 有可能服务于不同地区的语言多样化人群。将通用 LLM 调整为医疗保健通常需要持续的预训练，但这种方法在计算上成本高昂，有时也不切实际。由于缺乏模型理解和有效推理不同场景所需的更广泛领域知识，特定任务的指令微调可能并不总能保证最佳性能。为了应对这些挑战，我们引入了两个多语言指令微调数据集 MMed-IFT 和 MMed-IFT-MC，其中包含六种语言的 20 多万个高质量医学样本。我们提出了一个两阶段训练范式：第一阶段使用 MMed-IFT 注入一般医学知识，而第二阶段使用 MMed-IFT-MC 微调特定于任务的多项选择题。我们的方法在英语和多语言基准上都取得了有竞争力的结果，在计算效率和性能之间取得了平衡。我们计划将来在 \url{此 https URL} 上公开我们的数据集和模型权重。

Title: Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Models

• Authors: Emily Cheng, Richard J. Antonello
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05771
• Pdf URL: https://arxiv.org/pdf/2409.05771
• Copy Paste: [[2409.05771]] Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Models(https://arxiv.org/abs/2409.05771)
• Keywords: language model, llm
• Abstract: Research has repeatedly demonstrated that intermediate hidden states extracted from large language models are able to predict measured brain response to natural language stimuli. Yet, very little is known about the representation properties that enable this high prediction performance. Why is it the intermediate layers, and not the output layers, that are most capable for this unique and highly general transfer task? In this work, we show that evidence from language encoding models in fMRI supports the existence of a two-phase abstraction process within LLMs. We use manifold learning methods to show that this abstraction process naturally arises over the course of training a language model and that the first "composition" phase of this abstraction process is compressed into fewer layers as training continues. Finally, we demonstrate a strong correspondence between layerwise encoding performance and the intrinsic dimensionality of representations from LLMs. We give initial evidence that this correspondence primarily derives from the inherent compositionality of LLMs and not their next-word prediction properties.
• 摘要：研究一再证明，从大型语言模型中提取的中间隐藏状态能够预测大脑对自然语言刺激的测量反应。然而，人们对实现这种高预测性能的表示属性知之甚少。为什么中间层而不是输出层最适合这种独特且高度通用的传输任务？在这项工作中，我们表明，来自 fMRI 语言编码模型的证据支持 LLM 中存在两阶段抽象过程。我们使用流形学习方法来表明，这种抽象过程在训练语言模型的过程中自然产生，并且随着训练的继续，这种抽象过程的第一个“组合”阶段被压缩为更少的层。最后，我们证明了分层编码性能与 LLM 表示的固有维度之间存在很强的对应关系。我们给出初步证据，表明这种对应关系主要源于 LLM 固有的组合性，而不是它们的下一个词预测属性。

Title: Benchmarking Chinese Knowledge Rectification in Large Language Models

• Authors: Tianhe Lu, Jizhan Fang, Yunzhi Yao, Xin Xu, Ningyu Zhang, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05806
• Pdf URL: https://arxiv.org/pdf/2409.05806
• Copy Paste: [[2409.05806]] Benchmarking Chinese Knowledge Rectification in Large Language Models(https://arxiv.org/abs/2409.05806)
• Keywords: language model, llm, hallucination
• Abstract: While Large Language Models (LLMs) exhibit remarkable generative capabilities, they are not without flaws, particularly in the form of hallucinations. This issue is even more pronounced when LLMs are applied to specific languages and domains. For example, LLMs may generate nonsense information when handling Chinese ancient poetry, proverbs, or idioms, owing to the lack of specific knowledge. To this end, this paper introduces a benchmark for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically, we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of knowledge from various sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony, antithesis, and logical constructs inherent in the Chinese language. Through the analysis of this dataset, we uncover the challenges faced by current LLMs in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge editing techniques on this dataset unveil the substantial scope for advancement in the rectification of Chinese knowledge. Code and dataset are available at this https URL.
• 摘要：虽然大型语言模型 (LLM) 表现出卓越的生成能力，但它们并非没有缺陷，尤其是幻觉。当 LLM 应用于特定语言和领域时，这个问题更加明显。例如，由于缺乏具体的知识，LLM 在处理中国古诗、谚语或成语时可能会生成无意义的信息。为此，本文介绍了一个通过知识编辑纠正 LLM 中中文知识的基准。具体来说，我们引入了一个新的中文数据集 CKnowEdit，通过从各种来源收集七种类型的知识，包括古典文本、成语和百度贴吧若知吧的内容，从而解释了中文固有的独特复音、对偶和逻辑结构。通过对该数据集的分析，我们发现了当前 LLM 在掌握中文方面面临的挑战。此外，我们对该数据集上最先进的知识编辑技术的评估揭示了中文知识纠正的巨大进步空间。代码和数据集可在此 https URL 上获取。

Title: Improving Pretraining Data Using Perplexity Correlations

• Authors: Tristan Thrush, Christopher Potts, Tatsunori Hashimoto
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.05816
• Pdf URL: https://arxiv.org/pdf/2409.05816
• Copy Paste: [[2409.05816]] Improving Pretraining Data Using Perplexity Correlations(https://arxiv.org/abs/2409.05816)
• Keywords: language model, llm
• Abstract: Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own. Our work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark performance, and selecting high-correlation documents is an effective pretraining data selection method. We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations and perform data selection using a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 benchmarks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram classifier.
• 摘要：高质量的预训练数据通常被视为高性能语言模型的关键。然而，由于数据选择实验需要昂贵的预训练运行，理解预训练数据的进展一直很缓慢。我们提出了一个框架，可以避免这些成本，并在不进行任何 LLM 训练的情况下选择高质量的预训练数据。我们的工作基于一个简单的观察：许多预训练文本的 LLM 损失与下游基准性能相关，选择高相关性文档是一种有效的预训练数据选择方法。我们构建了一个以困惑度-基准相关性估计为中心的数据选择新统计框架，并使用从 Open LLM 排行榜上获取的 90 个 LLM 样本对来自数万个网络域的文本进行数据选择。在 8 个基准上 160M 参数规模的受控预训练实验中，我们的方法在每个基准上都优于 DSIR，同时匹配在手工设计的二元分类器 DataComp-LM 中发现的最佳数据选择器。

Title: MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct

• Authors: Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, Yongbin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05840
• Pdf URL: https://arxiv.org/pdf/2409.05840
• Copy Paste: [[2409.05840]] MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct(https://arxiv.org/abs/2409.05840)
• Keywords: language model, gpt, llm
• Abstract: The development of Multimodal Large Language Models (MLLMs) has seen significant advancements. However, the quantity and quality of multimodal instruction data have emerged as significant bottlenecks in their progress. Manually creating multimodal instruction data is both time-consuming and inefficient, posing challenges in producing instructions of high complexity. Moreover, distilling instruction data from black-box commercial models (e.g., GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains performance to that of these models. The challenge of curating diverse and complex instruction data remains substantial. We propose MMEvol, a novel multimodal instruction data evolution framework that combines fine-grained perception evolution, cognitive reasoning evolution, and interaction evolution. This iterative approach breaks through data quality bottlenecks to generate a complex and diverse image-text instruction dataset, thereby empowering MLLMs with enhanced capabilities. Beginning with an initial set of instructions, SEED-163K, we utilize MMEvol to systematically broadens the diversity of instruction types, integrates reasoning steps to enhance cognitive capabilities, and extracts detailed information from images to improve visual understanding and robustness. To comprehensively evaluate the effectiveness of our data, we train LLaVA-NeXT using the evolved data and conduct experiments across 13 vision-language tasks. Compared to the baseline trained with seed data, our approach achieves an average accuracy improvement of 3.1 points and reaches state-of-the-art (SOTA) performance on 9 of these tasks.
• 摘要：多模态大型语言模型 (MLLM) 的发展取得了重大进展。然而，多模态教学数据的数量和质量已成为其发展的重要瓶颈。手动创建多模态教学数据既耗时又低效，对生成高复杂度的指令提出了挑战。此外，从黑盒商业模型（例如 GPT-4o、GPT-4V）中提取教学数据通常会产生过于简单的教学数据，从而限制了这些模型的性能。整理多样化和复杂的教学数据的挑战仍然很大。我们提出了 MMEvol，这是一种新颖的多模态教学数据演进框架，它结合了细粒度感知演进、认知推理演进和交互演进。这种迭代方法突破了数据质量瓶颈，生成了复杂多样的图像文本教学数据集，从而增强了 MLLM 的功能。从初始指令集 SEED-163K 开始，我们利用 MMEvol 系统地拓宽指令类型的多样性，整合推理步骤以增强认知能力，并从图像中提取详细信息以提高视觉理解和稳健性。为了全面评估我们数据的有效性，我们使用演化后的数据训练 LLaVA-NeXT，并在 13 个视觉语言任务中进行实验。与使用种子数据训练的基线相比，我们的方法实现了平均 3.1 分的准确率提升，并在其中 9 个任务中达到了最佳 (SOTA) 性能。