2024-08-29

Title: Awes, Laws, and Flaws From Today's LLM Research

• Authors: Adrian de Wynter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15409
• Pdf URL: https://arxiv.org/pdf/2408.15409
• Copy Paste: [[2408.15409]] Awes, Laws, and Flaws From Today's LLM Research(https://arxiv.org/abs/2408.15409)
• Keywords: language model, llm
• Abstract: We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works based on criteria typical of what is considered good research (e.g. presence of statistical tests and reproducibility) and cross-validate it with arguments that are at the centre of controversy (e.g., claims of emergent behaviour, the use of LLMs as evaluators). We find multiple trends, such as declines in claims of emergent behaviour and the presence of ethics disclaimers; and the rise of LLMs as evaluators. This paper underscores the need for more scrutiny and rigour by and from this field. Critical reading and familiarity with the literature are crucial to live up to the fundamentals of a responsible scientific method that is ethical, reproducible, systematic, and open to criticism.
• 摘要：我们对当代大型语言模型 (LLM) 研究背后的科学方法进行了严格的审查。为此，我们根据被视为良好研究的典型标准（例如，统计测试和可重复性）评估了 2,000 多项研究，并将其与处于争议中心的论点（例如，关于突发行为的主张、使用 LLM 作为评估者）进行交叉验证。我们发现了多种趋势，例如关于突发行为的主张和道德免责声明的减少；以及 LLM 作为评估者的兴起。本文强调了该领域需要进行更多的审查和严谨。批判性阅读和熟悉文献对于实现负责任的科学方法的基本原则至关重要，这种方法是合乎道德的、可重复的、系统的，并且接受批评。

Title: Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations

• Authors: Yize Zhao, Tina Behnia, Vala Vakilian, Christos Thrampoulidis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15417
• Pdf URL: https://arxiv.org/pdf/2408.15417
• Copy Paste: [[2408.15417]] Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations(https://arxiv.org/abs/2408.15417)
• Keywords: language model
• Abstract: Next-token prediction (NTP) over large text corpora has become the go-to paradigm to train large language models. Yet, it remains unclear how NTP influences the mapping of linguistic patterns to geometric properties of the resulting model representations. We frame training of large language models as soft-label classification over sparse probabilistic label vectors, coupled with an analytical approximation that allows unrestricted generation of context embeddings. This approach links NTP training to rank-constrained, nuclear-norm regularized optimization in the logit domain, offering a framework for analyzing the geometry of word and context embeddings. In large embedding spaces, we find that NTP implicitly favors learning logits with a sparse plus low-rank structure. While the sparse component captures the co-occurrence frequency of context-word pairs, the orthogonal low-rank component, which becomes dominant as training progresses, depends solely on the sparsity pattern of the co-occurrence matrix. Consequently, when projected onto an appropriate subspace, representations of contexts that are followed by the same set of next-tokens collapse, a phenomenon we term subspace-collapse. We validate our findings on synthetic and small-scale real language datasets. Finally, we outline potential research directions aimed at deepening the understanding of NTP's influence on the learning of linguistic patterns and regularities.
• 摘要：大型文本语料库上的下一标记预测 (NTP) 已成为训练大型语言模型的首选范例。然而，NTP 如何影响语言模式到结果模型表示的几何属性的映射仍不清楚。我们将大型语言模型的训练定义为稀疏概率标签向量上的软标签分类，并结合允许无限制生成上下文嵌入的分析近似。这种方法将 NTP 训练与 logit 域中的秩约束、核范数正则化优化联系起来，为分析单词和上下文嵌入的几何形状提供了一个框架。在大型嵌入空间中，我们发现 NTP 隐含地倾向于学习具有稀疏加低秩结构的 logit。虽然稀疏分量捕获了上下文-单词对的共现频率，但随着训练的进行而占主导地位的正交低秩分量仅取决于共现矩阵的稀疏模式。因此，当投影到适当的子空间时，同一组下一个标记后面的上下文表示会崩溃，我们将这种现象称为子空间崩溃。我们在合成和小规模真实语言数据集上验证了我们的发现。最后，我们概述了潜在的研究方向，旨在加深对 NTP 对语言模式和规律学习的影响的理解。

Title: Legilimens: Practical and Unified Content Moderation for Large Language Model Services

• Authors: Jialin Wu, Jiangyi Deng, Shengyuan Pang, Yanjiao Chen, Jiayang Xu, Xinfeng Li, Wenyuan Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15488
• Pdf URL: https://arxiv.org/pdf/2408.15488
• Copy Paste: [[2408.15488]] Legilimens: Practical and Unified Content Moderation for Large Language Model Services(https://arxiv.org/abs/2408.15488)
• Keywords: language model, llm, chat
• Abstract: Given the societal impact of unsafe content generated by large language models (LLMs), ensuring that LLM services comply with safety standards is a crucial concern for LLM service providers. Common content moderation methods are limited by an effectiveness-and-efficiency dilemma, where simple models are fragile while sophisticated models consume excessive computational resources. In this paper, we reveal for the first time that effective and efficient content moderation can be achieved by extracting conceptual features from chat-oriented LLMs, despite their initial fine-tuning for conversation rather than content moderation. We propose a practical and unified content moderation framework for LLM services, named Legilimens, which features both effectiveness and efficiency. Our red-team model-based data augmentation enhances the robustness of Legilimens against state-of-the-art jailbreaking. Additionally, we develop a framework to theoretically analyze the cost-effectiveness of Legilimens compared to other methods. We have conducted extensive experiments on five host LLMs, seventeen datasets, and nine jailbreaking methods to verify the effectiveness, efficiency, and robustness of Legilimens against normal and adaptive adversaries. A comparison of Legilimens with both commercial and academic baselines demonstrates the superior performance of Legilimens. Furthermore, we confirm that Legilimens can be applied to few-shot scenarios and extended to multi-label classification tasks.
• 摘要：鉴于大型语言模型 (LLM) 生成的不安全内容对社会的影响，确保 LLM 服务符合安全标准是 LLM 服务提供商的关键关注点。常见的内容审核方法受到有效性和效率困境的限制，其中简单模型很脆弱，而复杂模型则消耗过多的计算资源。在本文中，我们首次揭示了通过从面向聊天的 LLM 中提取概念特征可以实现有效和高效的内容审核，尽管它们最初是针对对话而不是内容审核进行微调的。我们为 LLM 服务提出了一个实用且统一的内容审核框架，名为 Legilimens，它兼具有效性和效率。我们基于红队模型的数据增强增强了 Legilimens 对最先进越狱的鲁棒性。此外，我们还开发了一个框架来从理论上分析 Legilimens 与其他方法相比的成本效益。我们对五个主机 LLM、十七个数据集和九种越狱方法进行了广泛的实验，以验证 Legilimens 对抗普通和自适应对手的有效性、效率和稳健性。将 Legilimens 与商业和学术基线进行比较，证明了 Legilimens 的卓越性能。此外，我们确认 Legilimens 可以应用于少样本场景并扩展到多标签分类任务。

Title: Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression

• Authors: Haowen Hou, Fei Ma, Binwen Bai, Xinxin Zhu, Fei Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15491
• Pdf URL: https://arxiv.org/pdf/2408.15491
• Copy Paste: [[2408.15491]] Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression(https://arxiv.org/abs/2408.15491)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have garnered widespread attention due to their remarkable performance across various tasks. However, to mitigate the issue of hallucinations, LLMs often incorporate retrieval-augmented pipeline to provide them with rich external knowledge and context. Nevertheless, challenges stem from inaccurate and coarse-grained context retrieved from the retriever. Supplying irrelevant context to the LLMs can result in poorer responses, increased inference latency, and higher costs. This paper introduces a method called Instruction-Aware Contextual Compression, which filters out less informative content, thereby accelerating and enhancing the use of LLMs. The experimental results demonstrate that Instruction-Aware Contextual Compression notably reduces memory consumption and minimizes generation latency while maintaining performance levels comparable to those achieved with the use of the full context. Specifically, we achieved a 50% reduction in context-related costs, resulting in a 5% reduction in inference memory usage and a 2.2-fold increase in inference speed, with only a minor drop of 0.047 in Rouge-1. These findings suggest that our method strikes an effective balance between efficiency and performance.
• 摘要：大型语言模型 (LLM) 因其在各种任务中的出色表现而受到广泛关注。然而，为了缓解幻觉问题，LLM 通常采用检索增强管道，为其提供丰富的外部知识和上下文。然而，挑战源于从检索器检索到的不准确和粗粒度的上下文。向 LLM 提供不相关的上下文会导致响应较差、推理延迟增加和成本增加。本文介绍了一种称为指令感知上下文压缩的方法，它可以过滤掉信息量较少的内容，从而加速和增强 LLM 的使用。实验结果表明，指令感知上下文压缩显著降低了内存消耗并最大限度地减少了生成延迟，同时保持了与使用完整上下文相当的性能水平。具体来说，我们实现了上下文相关成本的降低 50%，推理内存使用量减少了 5%，推理速度提高了 2.2 倍，而 Rouge-1 中仅下降了 0.047。这些结果表明我们的方法在效率和性能之间取得了有效的平衡。

Title: ReMamba: Equip Mamba with Effective Long-Sequence Modeling

• Authors: Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang, Xunliang Cai, Dongyan Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15496
• Pdf URL: https://arxiv.org/pdf/2408.15496
• Copy Paste: [[2408.15496]] ReMamba: Equip Mamba with Effective Long-Sequence Modeling(https://arxiv.org/abs/2408.15496)
• Keywords: long context
• Abstract: While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models. In this study, we investigate the long-context efficiency issues of the Mamba models and propose ReMamba, which enhances Mamba's ability to comprehend long contexts. ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. Experimental results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy, improving over the baselines by 3.2 and 1.6 points, respectively, and attaining performance almost on par with same-size transformer models.
• 摘要：虽然 Mamba 架构在短上下文自然语言处理 (NLP) 任务上表现出卓越的推理效率和竞争性能，但经验证据表明，与基于 Transformer 的模型相比，其理解长上下文的能力有限。在本研究中，我们研究了 Mamba 模型的长上下文效率问题，并提出了 ReMamba，它增强了 Mamba 理解长上下文的能力。ReMamba 在两阶段重新转发过程中结合了选择性压缩和自适应技术，将额外的推理成本开销降至最低。在 LongBench 和 L-Eval 基准测试中的实验结果证明了 ReMamba 的有效性，分别比基线提高了 3.2 和 1.6 个百分点，并且性能几乎与相同大小的 Transformer 模型相当。

Title: Dolphin: Long Context as a New Modality for Energy-Efficient On-Device Language Models

• Authors: Wei Chen, Zhiyuan Li, Shuo Xin, Yihao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15518
• Pdf URL: https://arxiv.org/pdf/2408.15518
• Copy Paste: [[2408.15518]] Dolphin: Long Context as a New Modality for Energy-Efficient On-Device Language Models(https://arxiv.org/abs/2408.15518)
• Keywords: language model, long context
• Abstract: This paper presents Dolphin, a novel decoder-decoder architecture for energy-efficient processing of long contexts in language models. Our approach addresses the significant energy consumption and latency challenges inherent in on-device models. Dolphin employs a compact 0.5B parameter decoder to distill extensive contextual information into a memory embedding, substantially reducing the input length for the primary 7B parameter decoder model. Inspired by vision-language models, we repurpose the image embedding projector to encode long textual contexts, effectively treating extended context as a distinct modality. This innovative method enables processing of substantially longer contexts without the typical computational overhead associated with extended input sequences. Empirical evaluations demonstrate a 10-fold improvement in energy efficiency and a 5-fold reduction in latency compared to conventional full-length context processing methods without losing quality of the response. Our work contributes to the development of more sustainable and scalable language models for on-device applications, addressing the critical need for energy-efficient and responsive AI technologies in resource-constrained environments while maintaining the accuracy to understand long contexts. This research has implications for the broader field of natural language processing, particularly in the domain of efficient model design for resource-limited settings. By enabling more sophisticated AI capabilities on edge devices, Dolphin paves the way for advanced language processing in a wide range of applications where computational resources are at a premium. The Dolphin model is publicly available at this https URL.
• 摘要：本文介绍了 Dolphin，这是一种新型解码器-解码器架构，用于语言模型中长上下文的节能处理。我们的方法解决了设备端模型固有的大量能耗和延迟挑战。Dolphin 采用紧凑的 0.5B 参数解码器将大量上下文信息提炼到内存嵌入中，大大减少了主要 7B 参数解码器模型的输入长度。受视觉语言模型的启发，我们重新利用图像嵌入投影仪来编码长文本上下文，有效地将扩展上下文视为一种独特的模态。这种创新方法能够处理更长的上下文，而无需与扩展输入序列相关的典型计算开销。实证评估表明，与传统的全长上下文处理方法相比，能源效率提高了 10 倍，延迟减少了 5 倍，且响应质量没有降低。我们的工作有助于为设备端应用程序开发更可持续、更可扩展的语言模型，满足资源受限环境中对节能和响应迅速的 AI 技术的关键需求，同时保持理解长上下文的准确性。这项研究对更广泛的自然语言处理领域具有重要意义，特别是在资源有限环境下的高效模型设计领域。通过在边缘设备上启用更复杂的 AI 功能，Dolphin 为计算资源极为宝贵的广泛应用中的高级语言处理铺平了道路。Dolphin 模型可在此 https URL 上公开获取。

Title: LRP4RAG: Detecting Hallucinations in Retrieval-Augmented Generation via Layer-wise Relevance Propagation

• Authors: Haichuan Hu, Yuhan Sun, Qunjun Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15533
• Pdf URL: https://arxiv.org/pdf/2408.15533
• Copy Paste: [[2408.15533]] LRP4RAG: Detecting Hallucinations in Retrieval-Augmented Generation via Layer-wise Relevance Propagation(https://arxiv.org/abs/2408.15533)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has become a primary technique for mitigating hallucinations in large language models (LLMs). However, incomplete knowledge extraction and insufficient understanding can still mislead LLMs to produce irrelevant or even contradictory responses, which means hallucinations persist in RAG. In this paper, we propose LRP4RAG, a method based on the Layer-wise Relevance Propagation (LRP) algorithm for detecting hallucinations in RAG. Specifically, we first utilize LRP to compute the relevance between the input and output of the RAG generator. We then apply further extraction and resampling to the relevance matrix. The processed relevance data are input into multiple classifiers to determine whether the output contains hallucinations. To the best of our knowledge, this is the first time that LRP has been used for detecting RAG hallucinations, and extensive experiments demonstrate that LRP4RAG outperforms existing baselines.
• 摘要：检索增强生成 (RAG) 已成为缓解大型语言模型 (LLM) 中幻觉的主要技术。然而，不完整的知识提取和不充分的理解仍然会误导 LLM 产生不相关甚至矛盾的反应，这意味着幻觉在 RAG 中持续存在。在本文中，我们提出了 LRP4RAG，一种基于分层相关性传播 (LRP) 算法的用于检测 RAG 中幻觉的方法。具体而言，我们首先利用 LRP 计算 RAG 生成器的输入和输出之间的相关性。然后我们对相关性矩阵进行进一步的提取和重采样。处理后的相关性数据被输入到多个分类器中以确定输出是否包含幻觉。据我们所知，这是 LRP 首次用于检测 RAG 幻觉，大量实验表明 LRP4RAG 的表现优于现有基线。

Title: An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication

• Authors: Yunmeng Li, Jun Suzuki, Makoto Morishita, Kaori Abe, Kentaro Inui
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.15543
• Pdf URL: https://arxiv.org/pdf/2408.15543
• Copy Paste: [[2408.15543]] An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication(https://arxiv.org/abs/2408.15543)
• Keywords: chat
• Abstract: The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat Translation (MQM-Chat). Through the experiments of five models using MQM-Chat, we observed that all models generated certain fundamental errors, while each of them has different shortcomings, such as omission, overly correcting ambiguous source content, and buzzword issues, resulting in the loss of stylized information. Our findings underscore the effectiveness of MQM-Chat in evaluating chat translation, emphasizing the importance of stylized content and dialogue consistency for future studies.
• 摘要：聊天的复杂性对机器翻译模型提出了重大挑战。认识到需要一个精确的评估指标来解决聊天翻译问题，本研究引入了聊天翻译多维质量指标 (MQM-Chat)。通过使用 MQM-Chat 对五个模型进行的实验，我们观察到所有模型都产生了某些基本错误，而每个模型都有不同的缺点，例如遗漏、过度纠正模棱两可的源内容和流行语问题，导致风格化信息的丢失。我们的研究结果强调了 MQM-Chat 在评估聊天翻译方面的有效性，强调了风格化内容和对话一致性对未来研究的重要性。

Title: WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

• Authors: Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Xiaofeng Xu, Xia Song, Jennifer Neville
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15549
• Pdf URL: https://arxiv.org/pdf/2408.15549
• Copy Paste: [[2408.15549]] WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback(https://arxiv.org/abs/2408.15549)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages real-time, in-situ user interactions to create preference datasets that more accurately reflect authentic human values. WildFeedback operates through a three-step process: feedback signal identification, preference data construction, and user-guided evaluation. We applied this framework to a large corpus of user-LLM conversations, resulting in a rich preference dataset that reflects genuine user preferences. This dataset captures the nuances of user preferences by identifying and classifying feedback signals within natural conversations, thereby enabling the construction of more representative and context-sensitive alignment data. Our extensive experiments demonstrate that LLMs fine-tuned on WildFeedback exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed user-guided evaluation. By incorporating real-time feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users. In summary, WildFeedback offers a robust, scalable solution for aligning LLMs with true human values, setting a new standard for the development and evaluation of user-centric language models.
• 摘要：随着大型语言模型 (LLM) 的不断发展，将这些模型与人类偏好进行对齐已成为一项关键挑战。传统的对齐方法依赖于人类或 LLM 注释的数据集，但其资源密集型性质、固有的主观性以及放大模型偏差的反馈循环风险限制了它们。为了克服这些限制，我们引入了 WildFeedback，这是一个新颖的框架，它利用实时、现场的用户交互来创建更准确反映真实人类价值观的偏好数据集。WildFeedback 通过三个步骤运行：反馈信号识别、偏好数据构建和用户指导评估。我们将此框架应用于大量用户-LLM 对话语料库，从而生成了一个反映真实用户偏好的丰富偏好数据集。该数据集通过识别和分类自然对话中的反馈信号来捕捉用户偏好的细微差别，从而能够构建更具代表性和上下文敏感的对齐数据。我们进行了大量的实验，结果表明，在 WildFeedback 上进行微调的 LLM 能够显著提高与用户偏好的一致性，传统基准和我们提出的用户指导评估都证明了这一点。通过整合来自实际用户的实时反馈，WildFeedback 解决了困扰现有方法的可扩展性、主观性和偏见挑战，标志着朝着开发更能响应用户多样化和不断变化的需求的 LLM 迈出了重要一步。总之，WildFeedback 提供了一种强大、可扩展的解决方案，使 LLM 与真正的人类价值观保持一致，为以用户为中心的语言模型的开发和评估树立了新标准。

Title: Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation

• Authors: Lujun Gui, Bin Xiao, Lei Su, Weipeng Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15562
• Pdf URL: https://arxiv.org/pdf/2408.15562
• Copy Paste: [[2408.15562]] Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation(https://arxiv.org/abs/2408.15562)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Lossless speculative decoding accelerates target large language model (LLM) inference by employing a lightweight draft model for generating tree-structured candidates, which are subsequently verified in parallel by the target LLM. Currently, effective approaches leverage feature-level rather than token-level autoregression within the draft model to facilitate more straightforward predictions and enhanced knowledge distillation. In this paper, we reassess these approaches and propose FSPAD (Feature Sampling and Partial Alignment Distillation for Lossless Speculative Decoding), which introduces two straightforward and effective components within the existing framework to boost lossless speculative decoding. Firstly, FSPAD utilizes token embeddings to sample features of the target LLM in high-dimensional space before feeding them into the draft model, due to the inherent uncertainty of the features preventing the draft model from obtaining the specific token output by the target LLM. Secondly, FSPAD introduces partial alignment distillation to weaken the draft model's connection between features and logits, aiming to reduce the conflict between feature alignment and logit confidence during training. Our experiments include both greedy and non-greedy decoding on the largest and smallest models from the Vicuna and LLaMA3-Instruct series, as well as tasks in multi-turn conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation. The results show that FSPAD outperforms the state-of-the-art method across all the aforementioned tasks and target LLMs.
• 摘要：无损推测解码通过使用轻量级草稿模型来生成树形候选，然后由目标 LLM 并行验证这些候选，从而加速目标大型语言模型 (LLM) 推理。目前，有效的方法利用草稿模型中的特征级而非 token 级自回归来促进更直接的预测和增强的知识提炼。在本文中，我们重新评估了这些方法并提出了 FSPAD（无损推测解码的特征采样和部分对齐提炼），它在现有框架中引入了两个简单有效的组件来促进无损推测解码。首先，FSPAD 利用 token 嵌入在高维空间中对目标 LLM 的特征进行采样，然后再将它们输入到草稿模型中，因为特征固有的不确定性会阻止草稿模型获得目标 LLM 输出的特定 token。其次，FSPAD 引入了部分对齐提炼来削弱草稿模型中特征与 logit 之间的联系，旨在减少训练过程中特征对齐和 logit 置信度之间的冲突。我们的实验包括对 Vicuna 和 LLaMA3-Instruct 系列中最大和最小模型进行贪婪和非贪婪解码，以及多轮对话、翻译、总结、问答、数学推理和检索增强生成等任务。结果表明，FSPAD 在上述所有任务和目标 LLM 中的表现均优于最先进的方法。

Title: SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models

• Authors: Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15565
• Pdf URL: https://arxiv.org/pdf/2408.15565
• Copy Paste: [[2408.15565]] SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models(https://arxiv.org/abs/2408.15565)
• Keywords: language model, llm, prompt
• Abstract: There is a growing trend of teaching large language models (LLMs) to solve mathematical problems through coding. Existing studies primarily focus on prompting powerful, closed-source models to generate seed training data followed by in-domain data augmentation, equipping LLMs with considerable capabilities for code-aided mathematical reasoning. However, continually training these models on augmented data derived from a few datasets such as GSM8K may impair their generalization abilities and restrict their effectiveness to a narrow range of question types. Conversely, the potential of improving such LLMs by leveraging large-scale, expert-written, diverse math question-answer pairs remains unexplored. To utilize these resources and tackle unique challenges such as code response assessment, we propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. We also explore different alignment algorithms with self-generated instruction/preference data to foster continuous improvement. Experiments across both in-domain (up to +5.7%) and out-of-domain (+4.4%) benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
• 摘要：教授大型语言模型 (LLM) 通过编码解决数学问题的趋势日益增长。现有研究主要侧重于促使强大的闭源模型生成种子训练数据，然后进行域内数据增强，从而使 LLM 具备强大的代码辅助数学推理能力。然而，持续使用来自少数数据集（如 GSM8K）的增强数据训练这些模型可能会损害其泛化能力，并将其有效性限制在较窄的问题类型范围内。相反，通过利用大规模、专家编写的多样化数学问答对来改进此类 LLM 的潜力仍未得到探索。为了利用这些资源并应对代码响应评估等独特挑战，我们提出了一种新颖的范式，该范式使用基于代码的批评模型来指导问题代码数据构建、质量控制和补充评估等步骤。我们还探索了使用自生成指令/偏好数据的不同对齐算法，以促进持续改进。在英语和中文的领域内（高达+5.7%）和领域外（+4.4%）基准上进行的实验证明了所提出范式的有效性。

Title: Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

• Authors: Lingyu Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15650
• Pdf URL: https://arxiv.org/pdf/2408.15650
• Copy Paste: [[2408.15650]] Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings(https://arxiv.org/abs/2408.15650)
• Keywords: language model, prompt
• Abstract: Text classification is crucial for applications such as sentiment analysis and toxic text filtering, but it still faces challenges due to the complexity and ambiguity of natural language. Recent advancements in deep learning, particularly transformer architectures and large-scale pretraining, have achieved inspiring success in NLP fields. Building on these advancements, this thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models (PLMs). Firstly, to address the challenge of selecting misleading yet incorrect distractors for cloze questions, we develop models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy. Secondly, to enhance model generalization to unseen labels, we create small finetuning datasets with domain-independent task label descriptions, improving model performance and robustness. Lastly, we tackle the sensitivity of large language models to in-context learning prompts by selecting effective demonstrations, focusing on misclassified examples and resolving model ambiguity regarding test example labels.
• 摘要：文本分类对于情绪分析和有害文本过滤等应用至关重要，但由于自然语言的复杂性和模糊性，它仍然面临挑战。深度学习的最新进展，特别是转换器架构和大规模预训练，在 NLP 领域取得了令人鼓舞的成功。基于这些进步，本论文通过利用预训练语言模型 (PLM) 的内在知识，探索了文本分类中的三个具有挑战性的设置。首先，为了解决为完形填空问题选择误导性但不正确的干扰项的挑战，我们开发了利用基于 PLM 的上下文词表示的特征的模型，实现了与人类准确性相媲美或超越人类准确性的性能。其次，为了增强模型对看不见的标签的泛化，我们创建了具有与领域无关的任务标签描述的小型微调数据集，从而提高了模型性能和鲁棒性。最后，我们通过选择有效的演示、关注错误分类的示例和解决与测试示例标签有关的模型歧义来解决大型语言模型对上下文学习提示的敏感性。

Title: StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements

• Authors: Jillian Fisher, Skyler Hallinan, Ximing Lu, Mitchell Gordon, Zaid Harchaoui, Yejin Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15666
• Pdf URL: https://arxiv.org/pdf/2408.15666
• Copy Paste: [[2408.15666]] StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements(https://arxiv.org/abs/2408.15666)
• Keywords: language model, llm
• Abstract: Authorship obfuscation, rewriting a text to intentionally obscure the identity of the author, is an important but challenging task. Current methods using large language models (LLMs) lack interpretability and controllability, often ignoring author-specific stylistic features, resulting in less robust performance overall. To address this, we develop StyleRemix, an adaptive and interpretable obfuscation method that perturbs specific, fine-grained style elements of the original input text. StyleRemix uses pre-trained Low Rank Adaptation (LoRA) modules to rewrite an input specifically along various stylistic axes (e.g., formality and length) while maintaining low computational cost. StyleRemix outperforms state-of-the-art baselines and much larger LLMs in a variety of domains as assessed by both automatic and human evaluation. Additionally, we release AuthorMix, a large set of 30K high-quality, long-form texts from a diverse set of 14 authors and 4 domains, and DiSC, a parallel corpus of 1,500 texts spanning seven style axes in 16 unique directions
• 摘要：作者身份混淆，即重写文本以故意掩盖作者身份，是一项重要但具有挑战性的任务。当前使用大型语言模型 (LLM) 的方法缺乏可解释性和可控制性，通常会忽略作者特定的风格特征，导致整体性能较差。为了解决这个问题，我们开发了 StyleRemix，这是一种自适应且可解释的混淆方法，可扰乱原始输入文本的特定细粒度样式元素。StyleRemix 使用预先训练的低秩自适应 (LoRA) 模块，专门沿着各种风格轴（例如形式和长度）重写输入，同时保持较低的计算成本。根据自动和人工评估的评估，StyleRemix 在各种领域的表现均优于最先进的基线和更大的 LLM。此外，我们还发布了 AuthorMix，这是来自 14 位作者和 4 个领域的 30K 篇高质量长篇文本的大型数据集，以及 DiSC，这是一个包含 1,500 篇文本的平行语料库，涵盖 16 个独特方向的 7 个风格轴

Title: Conan-embedding: General Text Embedding with More and Better Negative Samples

• Authors: Shiyu Li, Yang Tang, Shizhe Chen, Xi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15710
• Pdf URL: https://arxiv.org/pdf/2408.15710
• Copy Paste: [[2408.15710]] Conan-embedding: General Text Embedding with More and Better Negative Samples(https://arxiv.org/abs/2408.15710)
• Keywords: llm, prompt
• Abstract: With the growing popularity of RAG, the capabilities of embedding models are gaining increasing attention. Embedding models are primarily trained through contrastive loss learning, with negative examples being a key component. Previous work has proposed various hard negative mining strategies, but these strategies are typically employed as preprocessing steps. In this paper, we propose the conan-embedding model, which maximizes the utilization of more and higher-quality negative examples. Specifically, since the model's ability to handle preprocessed negative examples evolves during training, we propose dynamic hard negative mining method to expose the model to more challenging negative examples throughout the training process. Secondly, contrastive learning requires as many negative examples as possible but is limited by GPU memory constraints. Therefore, we use a Cross-GPU balancing Loss to provide more negative examples for embedding training and balance the batch size across multiple tasks. Moreover, we also discovered that the prompt-response pairs from LLMs can be used for embedding training. Our approach effectively enhances the capabilities of embedding models, currently ranking first on the Chinese leaderboard of Massive text embedding benchmark
• 摘要：随着 RAG 的日益普及，嵌入模型的能力也越来越受到关注。嵌入模型主要通过对比损失学习进行训练，其中负样本是关键组成部分。先前的研究提出了各种硬负样本挖掘​​策略，但这些策略通常用作预处理步骤。在本文中，我们提出了柯南嵌入模型，该模型可以最大限度地利用更多更高质量的负样本。具体而言，由于模型处理预处理负样本的能力在训练过程中不断发展，我们提出了动态硬负样本挖掘​​方法，在整个训练过程中将模型暴露于更具挑战性的负样本。其次，对比学习需要尽可能多的负样本，但受到 GPU 内存限制的限制。因此，我们使用跨 GPU 平衡损失来为嵌入训练提供更多负样本，并在多个任务之间平衡批大小。此外，我们还发现 LLM 中的提示-响应对可用于嵌入训练。我们的方法有效地增强了嵌入模型的能力，目前在 Massive 文本嵌入基准的中文排行榜上排名第一

Title: LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models

• Authors: Max Ploner, Jacek Wiland, Sebastian Pohl, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15729
• Pdf URL: https://arxiv.org/pdf/2408.15729
• Copy Paste: [[2408.15729]] LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models(https://arxiv.org/abs/2408.15729)
• Keywords: language model
• Abstract: Knowledge probing evaluates the extent to which a language model (LM) has acquired relational knowledge during its pre-training phase. It provides a cost-effective means of comparing LMs of different sizes and training setups and is useful for monitoring knowledge gained or lost during continual learning (CL). In prior work, we presented an improved knowledge probe called BEAR (Wiland et al., 2024), which enables the comparison of LMs trained with different pre-training objectives (causal and masked LMs) and addresses issues of skewed distributions in previous probes to deliver a more unbiased reading of LM knowledge. With this paper, we present LM-PUB- QUIZ, a Python framework and leaderboard built around the BEAR probing mechanism that enables researchers and practitioners to apply it in their work. It provides options for standalone evaluation and direct integration into the widely-used training pipeline of the Hugging Face TRANSFORMERS library. Further, it provides a fine-grained analysis of different knowledge types to assist users in better understanding the knowledge in each evaluated LM. We publicly release LM-PUB-QUIZ as an open-source project.
• 摘要：知识探测评估语言模型 (LM) 在预训练阶段获得关系知识的程度。它提供了一种经济有效的方法来比较不同大小和训练设置的 LM，并且对于监控持续学习 (CL) 期间获得或丢失的知识非常有用。在之前的工作中，我们提出了一种改进的知识探测器，称为 BEAR (Wiland et al., 2024)，它可以比较使用不同预训练目标 (因果和掩蔽 LM) 训练的 LM，并解决先前探测中的偏斜分布问题，以提供对 LM 知识的更公正的解读。在本文中，我们介绍了 LM-PUB-QUIZ，这是一个围绕 BEAR 探测机制构建的 Python 框架和排行榜，使研究人员和从业者能够将其应用于他们的工作中。它提供了独立评估和直接集成到 Hugging Face TRANSFORMERS 库的广泛使用的训练管道中的选项。此外，它还提供了不同知识类型的细粒度分析，以帮助用户更好地理解每个评估的 LM 中的知识。我们公开发布了 LM-PUB-QUIZ 作为开源项目。

Title: Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions

• Authors: Huachuan Qiu, Zhenzhong Lan
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.15787
• Pdf URL: https://arxiv.org/pdf/2408.15787
• Copy Paste: [[2408.15787]] Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions(https://arxiv.org/abs/2408.15787)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Virtual counselors powered by large language models (LLMs) aim to create interactive support systems that effectively assist clients struggling with mental health challenges. To replicate counselor-client conversations, researchers have built an online mental health platform that allows professional counselors to provide clients with text-based counseling services for about an hour per session. Notwithstanding its effectiveness, challenges exist as human annotation is time-consuming, cost-intensive, privacy-protected, and not scalable. To address this issue and investigate the applicability of LLMs in psychological counseling conversation simulation, we propose a framework that employs two LLMs via role-playing for simulating counselor-client interactions. Our framework involves two LLMs, one acting as a client equipped with a specific and real-life user profile and the other playing the role of an experienced counselor, generating professional responses using integrative therapy techniques. We implement both the counselor and the client by zero-shot prompting the GPT-4 model. In order to assess the effectiveness of LLMs in simulating counselor-client interactions and understand the disparities between LLM- and human-generated conversations, we evaluate the synthetic data from various perspectives. We begin by assessing the client's performance through automatic evaluations. Next, we analyze and compare the disparities between dialogues generated by the LLM and those generated by professional counselors. Furthermore, we conduct extensive experiments to thoroughly examine the performance of our LLM-based counselor trained with synthetic interactive dialogues by benchmarking against state-of-the-art models for mental health.
• 摘要：由大型语言模型 (LLM) 驱动的虚拟咨询师旨在创建交互式支持系统，有效地帮助那些面临心理健康挑战的客户。为了复制咨询师与客户的对话，研究人员建立了一个在线心理健康平台，允许专业咨询师为客户提供每次约一小时的基于文本的咨询服务。尽管它很有效，但也存在挑战，因为人工注释既费时又费钱，既需要保护隐私又不可扩展。为了解决这个问题并研究 LLM 在心理咨询对话模拟中的适用性，我们提出了一个框架，该框架通过角色扮演使用两个 LLM 来模拟咨询师与客户的互动。我们的框架涉及两个 LLM，一个扮演具有特定真实用户资料的客户，另一个扮演经验丰富的咨询师的角色，使用综合疗法技术生成专业回应。我们通过零样本提示 GPT-4 模型来实现咨询师和客户。为了评估 LLM 在模拟咨询师与客户互动方面的有效性，并了解 LLM 和人工对话之间的差异，我们从各个角度评估了合成数据。我们首先通过自动评估来评估客户的表现。接下来，我们分析和比较 LLM 生成的对话与专业咨询师生成的对话之间的差异。此外，我们进行了广泛的实验，以通过与最先进的心理健康模型进行对比，彻底检查我们基于 LLM 的咨询师的表现，这些咨询师接受了合成交互式对话的训练。

Title: Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough

• Authors: Konstantin Dobler, Gerard de Melo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15793
• Pdf URL: https://arxiv.org/pdf/2408.15793
• Copy Paste: [[2408.15793]] Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough(https://arxiv.org/abs/2408.15793)
• Keywords: llm
• Abstract: We investigate continued pretraining of LLMs for language adaptation on a tight academic budget: a setting in which only a few GPUs can be used in parallel, for a heavily constrained duration. We focus on adapting Mistral-7B to German or Arabic and evaluate several techniques to improve efficiency and effectiveness in this setting. Our German models adapted on this tight compute budget underperform compared to the base Mistral-7B, while our Arabic models outperform several baselines, showing that for sufficiently well-represented languages, continued pretraining for specialization is not always helpful. Our main findings focus on training precision and tokenizer swapping. Our results show that pure bfloat16 training is a viable alternative to mixed-precision training, while being much faster when only using a few GPUs. Swapping the tokenizer for a specialized one yields more efficient tokenization and is competitive with the original tokenizer, which already contains some German tokens, but did not significantly increase performance for German. Code and model weights are available at on GitHub.
• 摘要：我们研究了在紧张的学术预算下对 LLM 进行持续预训练以适应语言：在这种情况下，只能并行使用少量 GPU，并且持续时间受到严重限制。我们专注于将 Mistral-7B 改编为德语或阿拉伯语，并评估了几种在这种环境下提高效率和有效性的技术。我们在这种紧张的计算预算下改编的德语模型与基本 Mistral-7B 相比表现不佳，而我们的阿拉伯语模型则优于几个基线，这表明对于具有足够代表性的语言，持续进行专业化预训练并不总是有帮助。我们的主要发现集中在训练精度和标记器交换上。我们的结果表明，纯 bfloat16 训练是混合精度训练的可行替代方案，同时在仅使用少量 GPU 时速度要快得多。将标记器换成专门的标记器可以产生更高效的标记，并且与原始标记器相媲美，原始标记器已经包含一些德语标记，但并没有显着提高德语的性能。代码和模型权重可在 GitHub 上找到。

Title: Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization

• Authors: Léo Hemamou, Mehdi Debiane
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15801
• Pdf URL: https://arxiv.org/pdf/2408.15801
• Copy Paste: [[2408.15801]] Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization(https://arxiv.org/abs/2408.15801)
• Keywords: language model, llm, hallucination, chat
• Abstract: In an era where digital text is proliferating at an unprecedented rate, efficient summarization tools are becoming indispensable. While Large Language Models (LLMs) have been successfully applied in various NLP tasks, their role in extractive text summarization remains underexplored. This paper introduces EYEGLAXS (Easy Yet Efficient larGe LAnguage model for eXtractive Summarization), a framework that leverages LLMs, specifically LLAMA2-7B and ChatGLM2-6B, for extractive summarization of lengthy text documents. Instead of abstractive methods, which often suffer from issues like factual inaccuracies and hallucinations, EYEGLAXS focuses on extractive summarization to ensure factual and grammatical integrity. Utilizing state-of-the-art techniques such as Flash Attention and Parameter-Efficient Fine-Tuning (PEFT), EYEGLAXS addresses the computational and resource challenges typically associated with LLMs. The system sets new performance benchmarks on well-known datasets like PubMed and ArXiv. Furthermore, we extend our research through additional analyses that explore the adaptability of LLMs in handling different sequence lengths and their efficiency in training on smaller datasets. These contributions not only set a new standard in the field but also open up promising avenues for future research in extractive text summarization.
• 摘要：在数字文本以前所未有的速度激增的时代，高效的摘要工具变得不可或缺。虽然大型语言模型 (LLM) 已成功应用于各种 NLP 任务，但它们在提取文本摘要中的作用仍未得到充分探索。本文介绍了 EYEGLAXS（用于提取摘要的简单而高效的大型语言模型），这是一个利用 LLM（特别是 LLAMA2-7B 和 ChatGLM2-6B）对长文本文档进行提取摘要的框架。EYEGLAXS 专注于提取摘要，以确保事实和语法的完整性，而不是抽象方法（抽象方法经常存在事实不准确和幻觉等问题）。利用 Flash Attention 和参数高效微调 (PEFT) 等最先进的技术，EYEGLAXS 解决了通常与 LLM 相关的计算和资源挑战。该系统在 PubMed 和 ArXiv 等知名数据集上设定了新的性能基准。此外，我们通过额外的分析扩展了我们的研究，探索了 LLM 在处理不同序列长度方面的适应性及其在较小数据集上的训练效率。这些贡献不仅为该领域树立了新标准，而且为未来的文本摘要提取研究开辟了有希望的途径。

Title: Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models

• Authors: Sebastian Vallejo Vera, Hunter Driggers
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15895
• Pdf URL: https://arxiv.org/pdf/2408.15895
• Copy Paste: [[2408.15895]] Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models(https://arxiv.org/abs/2408.15895)
• Keywords: language model, llm, prompt
• Abstract: Human coders are biased. We test similar biases in Large Language Models (LLMs) as annotators. By replicating an experiment run by Ennser-Jedenastik and Meyer (2018), we find evidence that LLMs use political information, and specifically party cues, to judge political statements. Not only do LLMs use relevant information to contextualize whether a statement is positive, negative, or neutral based on the party cue, they also reflect the biases of the human-generated data upon which they have been trained. We also find that unlike humans, who are only biased when faced with statements from extreme parties, LLMs exhibit significant bias even when prompted with statements from center-left and center-right parties. The implications of our findings are discussed in the conclusion.
• 摘要：人类编码员有偏见。我们在大型语言模型 (LLM) 中测试了类似的偏见，将其作为注释器。通过复制 Ennser-Jedenastik 和 Meyer (2018) 进行的实验，我们发现证据表明 LLM 使用政治信息，特别是政党线索来判断政治言论。LLM 不仅使用相关信息根据政党线索来判断言论是正面、负面还是中立，而且还反映了它们所训练的人类生成的数据的偏见。我们还发现，与人类不同，人类只有在面对极端政党的言论时才会有偏见，而 LLM 即使在面对中左翼和中右翼政党的言论时也会表现出明显的偏见。我们的研究结果的含义在结论中进行了讨论。

Title: Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts

• Authors: Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker, Ahmet Üstün
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15901
• Pdf URL: https://arxiv.org/pdf/2408.15901
• Copy Paste: [[2408.15901]] Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts(https://arxiv.org/abs/2408.15901)
• Keywords: language model
• Abstract: Efficiency, specialization, and adaptability to new data distributions are qualities that are hard to combine in current Large Language Models. The Mixture of Experts (MoE) architecture has been the focus of significant research because its inherent conditional computation enables such desirable properties. In this work, we focus on "upcycling" dense expert models into an MoE, aiming to improve specialization while also adding the ability to adapt to new tasks easily. We introduce Nexus, an enhanced MoE architecture with adaptive routing where the model learns to project expert embeddings from domain representations. This approach allows Nexus to flexibly add new experts after the initial upcycling through separately trained dense models, without requiring large-scale MoE training for unseen data domains. Our experiments show that Nexus achieves a relative gain of up to 2.1% over the baseline for initial upcycling, and a 18.8% relative gain for extending the MoE with a new expert by using limited finetuning data. This flexibility of Nexus is crucial to enable an open-source ecosystem where every user continuously assembles their own MoE-mix according to their needs.
• 摘要：效率、专业化和对新数据分布的适应性是当前大型语言模型中难以兼顾的品质。混合专家 (MoE) 架构一直是重要研究的焦点，因为其固有的条件计算可以实现这些理想的特性。在这项工作中，我们专注于将密集专家模型“升级”为 MoE，旨在提高专业化水平，同时增加轻松适应新任务的能力。我们引入了 Nexus，这是一种具有自适应路由的增强型 MoE 架构，其中模型学习从域表示中投射专家嵌入。这种方法允许 Nexus 在通过单独训练的密集模型进行初始升级后灵活地添加新专家，而无需对看不见的数据域进行大规模 MoE 训练。我们的实验表明，Nexus 在初始升级时实现了比基线高达 2.1% 的相对增益，而通过使用有限的微调数据用新专家扩展 MoE 时实现了 18.8% 的相对增益。 Nexus 的这种灵活性对于实现开源生态系统至关重要，在这个生态系统中，每个用户都可以根据自己的需要不断组装自己的 MoE 组合。

Title: LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments

• Authors: Ruirui Chen, Weifeng Jiang, Chengwei Qin, Ishaan Singh Rawal, Cheston Tan, Dongkyu Choi, Bo Xiong, Bo Ai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15903
• Pdf URL: https://arxiv.org/pdf/2408.15903
• Copy Paste: [[2408.15903]] LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments(https://arxiv.org/abs/2408.15903)
• Keywords: language model, llm
• Abstract: The rapid obsolescence of information in Large Language Models (LLMs) has driven the development of various techniques to incorporate new facts. However, existing methods for knowledge editing still face difficulties with multi-hop questions that require accurate fact identification and sequential logical reasoning, particularly among numerous fact updates. To tackle these challenges, this paper introduces Graph Memory-based Editing for Large Language Models (GMeLLo), a straitforward and effective method that merges the explicit knowledge representation of Knowledge Graphs (KGs) with the linguistic flexibility of LLMs. Beyond merely leveraging LLMs for question answering, GMeLLo employs these models to convert free-form language into structured queries and fact triples, facilitating seamless interaction with KGs for rapid updates and precise multi-hop reasoning. Our results show that GMeLLo significantly surpasses current state-of-the-art knowledge editing methods in the multi-hop question answering benchmark, MQuAKE, especially in scenarios with extensive knowledge edits.
• 摘要：大型语言模型 (LLM) 中信息的快速过时推动了各种技术的发展以纳入新的事实。然而，现有的知识编辑方法在处理需要准确事实识别和顺序逻辑推理的多跳问题时仍然面临困难，尤其是在大量事实更新中。为了应对这些挑战，本文介绍了基于图内存的大型语言模型编辑 (GMeLLo)，这是一种直接有效的方法，它将知识图谱 (KG) 的显性知识表示与 LLM 的语言灵活性相结合。除了仅仅利用 LLM 进行问​​答之外，GMeLLo 还使用这些模型将自由形式的语言转换为结构化查询和事实三元组，促进与 KG 的无缝交互以实现快速更新和精确的多跳推理。我们的结果表明，GMeLLo 在多跳问答基准 MQuAKE 中显著超越了目前最先进的知识编辑方法，尤其是在具有大量知识编辑的场景中。

Title: BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems

• Authors: Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15971
• Pdf URL: https://arxiv.org/pdf/2408.15971
• Copy Paste: [[2408.15971]] BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems(https://arxiv.org/abs/2408.15971)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) are becoming increasingly powerful and capable of handling complex tasks, e.g., building single agents and multi-agent systems. Compared to single agents, multi-agent systems have higher requirements for the collaboration capabilities of language models. Many benchmarks are proposed to evaluate their collaborative abilities. However, these benchmarks lack fine-grained evaluations of LLM collaborative capabilities. Additionally, multi-agent collaborative and competitive scenarios are ignored in existing works. To address these two problems, we propose a benchmark, called BattleAgentBench, which defines seven sub-stages of three varying difficulty levels and conducts a fine-grained evaluation of language models in terms of single-agent scenario navigation capabilities, paired-agent task execution abilities, and multi-agent collaboration and competition capabilities. We conducted extensive evaluations on leading four closed-source and seven open-source models. Experimental results indicate that API-based models perform excellently on simple tasks but open-source small models struggle with simple tasks. Regarding difficult tasks that require collaborative and competitive abilities, although API-based models have demonstrated some collaborative capabilities, there is still enormous room for improvement.
• 摘要：大型语言模型 (LLM) 的功能越来越强大，能够处理复杂的任务，例如构建单智能体和多智能体系统。与单智能体相比，多智能体系统对语言模型的协作能力有更高的要求。许多基准测试被提出来评估它们的协作能力。然而，这些基准测试缺乏对 LLM 协作能力的细粒度评估。此外，现有工作忽略了多智能体协作和竞争场景。为了解决这两个问题，我们提出了一个名为 BattleAgentBench 的基准测试，它定义了三个不同难度级别的七个子阶段，并从单智能体场景导航能力、配对智能体任务执行能力以及多智能体协作和竞争能力方面对语言模型进行了细粒度评估。我们对领先的四个闭源模型和七个开源模型进行了广泛的评估。实验结果表明，基于 API 的模型在简单任务上表现出色，但开源小模型在简单任务上表现不佳。对于需要协作和竞争能力的困难任务，虽然基于 API 的模型已经展示出一些协作能力，但仍有巨大的改进空间。