2024-06-12

Title: Enhancing Presentation Slide Generation by LLMs with a Multi-Staged End-to-End Approach

• Authors: Sambaran Bandyopadhyay, Himanshu Maheshwari, Anandhavelu Natarajan, Apoorv Saxena
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06556
• Pdf URL: https://arxiv.org/pdf/2406.06556
• Copy Paste: [[2406.06556]] Enhancing Presentation Slide Generation by LLMs with a Multi-Staged End-to-End Approach(https://arxiv.org/abs/2406.06556)
• Keywords: llm, prompt
• Abstract: Generating presentation slides from a long document with multimodal elements such as text and images is an important task. This is time consuming and needs domain expertise if done manually. Existing approaches for generating a rich presentation from a document are often semi-automatic or only put a flat summary into the slides ignoring the importance of a good narrative. In this paper, we address this research gap by proposing a multi-staged end-to-end model which uses a combination of LLM and VLM. We have experimentally shown that compared to applying LLMs directly with state-of-the-art prompting, our proposed multi-staged solution is better in terms of automated metrics and human evaluation.
• 摘要：从包含文本和图像等多模态元素的长文档生成演示文稿幻灯片是一项重要任务。如果手动完成，这非常耗时，并且需要领域专业知识。现有的从文档生成丰富演示文稿的方法通常是半自动化的，或者仅在幻灯片中放入平淡的摘要，而忽略了良好叙述的重要性。在本文中，我们通过提出一种使用 LLM 和 VLM 组合的多阶段端到端模型来解决这一研究空白。我们通过实验表明，与直接将 LLM 与最先进的提示相结合相比，我们提出的多阶段解决方案在自动化指标和人工评估方面更胜一筹。

Title: Enhancing Text Authenticity: A Novel Hybrid Approach for AI-Generated Text Detection

• Authors: Ye Zhang, Qian Leng, Mengran Zhu, Rui Ding, Yue Wu, Jintong Song, Yulu Gong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06558
• Pdf URL: https://arxiv.org/pdf/2406.06558
• Copy Paste: [[2406.06558]] Enhancing Text Authenticity: A Novel Hybrid Approach for AI-Generated Text Detection(https://arxiv.org/abs/2406.06558)
• Keywords: language model, llm
• Abstract: The rapid advancement of Large Language Models (LLMs) has ushered in an era where AI-generated text is increasingly indistinguishable from human-generated content. Detecting AI-generated text has become imperative to combat misinformation, ensure content authenticity, and safeguard against malicious uses of AI. In this paper, we propose a novel hybrid approach that combines traditional TF-IDF techniques with advanced machine learning models, including Bayesian classifiers, Stochastic Gradient Descent (SGD), Categorical Gradient Boosting (CatBoost), and 12 instances of Deberta-v3-large models. Our approach aims to address the challenges associated with detecting AI-generated text by leveraging the strengths of both traditional feature extraction methods and state-of-the-art deep learning models. Through extensive experiments on a comprehensive dataset, we demonstrate the effectiveness of our proposed method in accurately distinguishing between human and AI-generated text. Our approach achieves superior performance compared to existing methods. This research contributes to the advancement of AI-generated text detection techniques and lays the foundation for developing robust solutions to mitigate the challenges posed by AI-generated content.
• 摘要：大型语言模型 (LLM) 的快速发展开启了一个 AI 生成的文本与人类生成的内容越来越难以区分的时代。检测 AI 生成的文本已成为打击错误信息、确保内容真实性和防止恶意使用 AI 的必要条件。在本文中，我们提出了一种新颖的混合方法，将传统的 TF-IDF 技术与先进的机器学习模型相结合，包括贝叶斯分类器、随机梯度下降 (SGD)、分类梯度提升 (CatBoost) 和 12 个 Deberta-v3-large 模型实例。我们的方法旨在通过利用传统特征提取方法和最先进的深度学习模型的优势来解决与检测 AI 生成的文本相关的挑战。通过对综合数据集进行大量实验，我们证明了我们提出的方法在准确区分人类和 AI 生成的文本方面的有效性。与现有方法相比，我们的方法实现了卓越的性能。这项研究促进了人工智能生成的文本检测技术的进步，并为开发强大的解决方案以减轻人工智能生成内容带来的挑战奠定了基础。

Title: Harnessing Business and Media Insights with Large Language Models

• Authors: Yujia Bao, Ankit Parag Shah, Neeru Narang, Jonathan Rivers, Rajeev Maksey, Lan Guan, Louise N. Barrere, Shelley Evenson, Rahul Basole, Connie Miao, Ankit Mehta, Fabien Boulay, Su Min Park, Natalie E. Pearson, Eldhose Joy, Tiger He, Sumiran Thakur, Koustav Ghosal, Josh On, Phoebe Morrison, Tim Major, Eva Siqi Wang, Gina Escobar, Jiaheng Wei, Tharindu Cyril Weerasooriya, Queena Song, Daria Lashkevich, Clare Chen, Gyuhak Kim, Dengpan Yin, Don Hejna, Mo Nomeli, Wei Wei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06559
• Pdf URL: https://arxiv.org/pdf/2406.06559
• Copy Paste: [[2406.06559]] Harnessing Business and Media Insights with Large Language Models(https://arxiv.org/abs/2406.06559)
• Keywords: language model, llm
• Abstract: This paper introduces Fortune Analytics Language Model (FALM). FALM empowers users with direct access to comprehensive business analysis, including market trends, company performance metrics, and expert insights. Unlike generic LLMs, FALM leverages a curated knowledge base built from professional journalism, enabling it to deliver precise and in-depth answers to intricate business questions. Users can further leverage natural language queries to directly visualize financial data, generating insightful charts and graphs to understand trends across diverse business sectors clearly. FALM fosters user trust and ensures output accuracy through three novel methods: 1) Time-aware reasoning guarantees accurate event registration and prioritizes recent updates. 2) Thematic trend analysis explicitly examines topic evolution over time, providing insights into emerging business landscapes. 3) Content referencing and task decomposition enhance answer fidelity and data visualization accuracy. We conduct both automated and human evaluations, demonstrating FALM's significant performance improvements over baseline methods while prioritizing responsible AI practices. These benchmarks establish FALM as a cutting-edge LLM in the business and media domains, with exceptional accuracy and trustworthiness.
• 摘要：本文介绍了财富分析语言模型 (FALM)。FALM 使用户能够直接访问全面的业务分析，包括市场趋势、公司绩效指标和专家见解。与通用的 LLM 不同，FALM 利用从专业新闻报道中构建的精选知识库，使其能够为复杂的业务问题提供精确而深入的答案。用户还可以利用自然语言查询直接可视化财务数据，生成富有洞察力的图表和图形，以清楚地了解不同业务部门的趋势。FALM 通过三种新颖的方法培养用户信任并确保输出准确性：1) 时间感知推理保证准确的事件注册并优先考虑最近的更新。2) 主题趋势分析明确检查主题随时间的变化，提供对新兴商业格局的洞察。3) 内容引用和任务分解提高了答案保真度和数据可视化准确性。我们进行了自动和人工评估，证明了 FALM 相对于基线方法的显着性能改进，同时优先考虑负责任的 AI 实践。这些基准确立了 FALM 作为商业和媒体领域的前沿法学硕士的地位，具有卓越的准确性和可信度。

Title: Inverse Constitutional AI: Compressing Preferences into Principles

• Authors: Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Samuel Albanie, Robert Mullins
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06560
• Pdf URL: https://arxiv.org/pdf/2406.06560
• Copy Paste: [[2406.06560]] Inverse Constitutional AI: Compressing Preferences into Principles(https://arxiv.org/abs/2406.06560)
• Keywords: language model, llm, chat
• Abstract: Feedback data plays an important role in fine-tuning and evaluating state-of-the-art AI models. Often pairwise text preferences are used: given two texts, human (or AI) annotators select the "better" one. Such feedback data is widely used to align models to human preferences (e.g., reinforcement learning from human feedback), or to rank models according to human preferences (e.g., Chatbot Arena). Despite its wide-spread use, prior work has demonstrated that human-annotated pairwise text preference data often exhibits unintended biases. For example, human annotators have been shown to prefer assertive over truthful texts in certain contexts. Models trained or evaluated on this data may implicitly encode these biases in a manner hard to identify. In this paper, we formulate the interpretation of existing pairwise text preference data as a compression task: the Inverse Constitutional AI (ICAI) problem. In constitutional AI, a set of principles (or constitution) is used to provide feedback and fine-tune AI models. The ICAI problem inverts this process: given a dataset of feedback, we aim to extract a constitution that best enables a large language model (LLM) to reconstruct the original annotations. We propose a corresponding initial ICAI algorithm and validate its generated constitutions quantitatively based on reconstructed annotations. Generated constitutions have many potential use-cases -- they may help identify undesirable biases, scale feedback to unseen data or assist with adapting LLMs to individual user preferences. We demonstrate our approach on a variety of datasets: (a) synthetic feedback datasets with known underlying principles; (b) the AlpacaEval dataset of cross-annotated human feedback; and (c) the crowdsourced Chatbot Arena data set. We release the code for our algorithm and experiments at this https URL .
• 摘要：反馈数据在微调和评估最先进的 AI 模型中起着重要作用。通常使用成对文本偏好：给定两个文本，人类（或 AI）注释者选择“更好”的文本。此类反馈数据被广泛用于将模型与人类偏好对齐（例如，从人类反馈中进行强化学习），或根据人类偏好对模型进行排名（例如，Chatbot Arena）。尽管被广泛使用，但先前的研究表明，人类注释的成对文本偏好数据经常表现出意想不到的偏见。例如，在某些情况下，人类注释者被证明更喜欢自信的文本而不是真实的文本。基于这些数据训练或评估的模型可能会以难以识别的方式隐式编码这些偏见。在本文中，我们将现有成对文本偏好数据的解释表述为压缩任务：逆宪法 AI (ICAI) 问题。在宪法 AI 中，一组原则（或宪法）用于提供反馈和微调 AI 模型。 ICAI 问题颠倒了这个过程：给定一个反馈数据集，我们的目标是提取一个最能使大型语言模型 (LLM) 重建原始注释的章程。我们提出了相应的初始 ICAI 算法，并根据重建的注释定量验证了其生成的章程。生成的章程有许多潜在的用例——它们可能有助于识别不良偏差、将反馈扩展到看不见的数据或帮助根据个人用户偏好调整 LLM。我们在各种数据集上展示了我们的方法：(a) 具有已知基本原理的合成反馈数据集；(b) 交叉注释的人工反馈的 AlpacaEval 数据集；(c) 众包的 Chatbot Arena 数据集。我们在此 https URL 上发布了我们的算法和实验的代码。

Title: Brainstorming Brings Power to Large Language Models of Knowledge Reasoning

• Authors: Zining Qin, Chenhao Wang, Huiling Qin, Weijia Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06561
• Pdf URL: https://arxiv.org/pdf/2406.06561
• Copy Paste: [[2406.06561]] Brainstorming Brings Power to Large Language Models of Knowledge Reasoning(https://arxiv.org/abs/2406.06561)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated amazing capabilities in language generation, text comprehension, and knowledge reasoning. While a single powerful model can already handle multiple tasks, relying on a single perspective can lead to biased and unstable results. Recent studies have further improved the model's reasoning ability on a wide range of tasks by introducing multi-model collaboration. However, models with different capabilities may produce conflicting answers on the same problem, and how to reasonably obtain the correct answer from multiple candidate models has become a challenging problem. In this paper, we propose the multi-model brainstorming based on prompt. It incorporates different models into a group for brainstorming, and after multiple rounds of reasoning elaboration and re-inference, a consensus answer is reached within the group. We conducted experiments on three different types of datasets, and demonstrate that the brainstorming can significantly improve the effectiveness in logical reasoning and fact extraction. Furthermore, we find that two small-parameter models can achieve accuracy approximating that of larger-parameter models through brainstorming, which provides a new solution for distributed deployment of LLMs.
• 摘要：大型语言模型（LLM）在语言生成、文本理解和知识推理等方面展现出了惊人的能力。虽然单个强大的模型已经可以处理多个任务，但依赖单一视角可能会导致结果出现偏差和不稳定。近期研究通过引入多模型协作，进一步提升了模型在广泛任务上的推理能力。然而，不同能力的模型对同一问题可能产生相互冲突的答案，如何从多个候选模型中合理地获取正确答案成为一个具有挑战性的问题。本文提出了基于提示的多模型头脑风暴，将不同的模型纳入小组进行头脑风暴，经过多轮推理细化和重新推理，在小组内达成共识答案。我们在三种不同类型的数据集上进行了实验，结果表明头脑风暴可以显著提高逻辑推理和事实提取的有效性。此外，我们发现两个小参数模型通过头脑风暴可以达到与大参数模型相近的准确率，为LLM的分布式部署提供了一种新的解决方案。

Title: Achieving Sparse Activation in Small Language Models

• Authors: Jifeng Song, Kai Huang, Xiangyu Yin, Boyuan Yang, Wei Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06562
• Pdf URL: https://arxiv.org/pdf/2406.06562
• Copy Paste: [[2406.06562]] Achieving Sparse Activation in Small Language Models(https://arxiv.org/abs/2406.06562)
• Keywords: language model, llm
• Abstract: Sparse activation, which selectively activates only an input-dependent set of neurons in inference, is a useful technique to reduce the computing cost of Large Language Models (LLMs) without retraining or adaptation efforts. However, whether it can be applied to the recently emerging Small Language Models (SLMs) remains questionable, because SLMs are generally less over-parameterized than LLMs. In this paper, we aim to achieve sparse activation in SLMs. We first show that the existing sparse activation schemes in LLMs that build on neurons' output magnitudes cannot be applied to SLMs, and activating neurons based on their attribution scores is a better alternative. Further, we demonstrated and quantified the large errors of existing attribution metrics when being used for sparse activation, due to the interdependency among attribution scores of neurons across different layers. Based on these observations, we proposed a new attribution metric that can provably correct such errors and achieve precise sparse activation. Experiments over multiple popular SLMs and datasets show that our approach can achieve 80% sparsification ratio with <5% model accuracy loss, comparable to the sparse activation achieved in LLMs. The source code is available at: this https URL.
• 摘要：稀疏激活在推理过程中选择性地仅激活一组输入相关的神经元，是一种有用的技术，可以降低大型语言模型 (LLM) 的计算成本，而无需重新训练或调整工作。然而，它是否可以应用于最近出现的小型语言模型 (SLM) 仍有疑问，因为 SLM 的过度参数化程度通常低于 LLM。在本文中，我们的目标是在 SLM 中实现稀疏激活。我们首先表明，LLM 中现有的基于神经元输出幅度的稀疏激活方案不能应用于 SLM，而基于其归因分数激活神经元是一种更好的选择。此外，我们展示并量化了现有归因指标在用于稀疏激活时的巨大误差，这是由于不同层之间神经元的归因分数相互依赖。基于这些观察，我们提出了一种新的归因指标，可以证明可以纠正此类错误并实现精确的稀疏激活。在多个流行的 SLM 和数据集上进行的实验表明，我们的方法可以实现 80% 的稀疏化率，模型精度损失小于 5%，与 LLM 中实现的稀疏激活相当。源代码可在以下网址获取：此 https URL。

Title: Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

• Authors: Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang, Yahui Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06563
• Pdf URL: https://arxiv.org/pdf/2406.06563
• Copy Paste: [[2406.06563]] Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models(https://arxiv.org/abs/2406.06563)
• Keywords: language model, llm
• Abstract: In this technical report, we introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is initialized from the pre-existing dense checkpoints of our Skywork-13B model. We explore the comparative effectiveness of upcycling versus training from scratch initializations. Our findings suggest that the choice between these two approaches should consider both the performance of the existing dense checkpoints and the MoE training budget. We highlight two innovative techniques: gating logit normalization, which improves expert diversification, and adaptive auxiliary loss coefficients, allowing for layer-specific adjustment of auxiliary loss coefficients. Our experimental results validate the effectiveness of these methods. Leveraging these techniques and insights, we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile corpus. The evaluation results demonstrate that our model delivers strong performance across a wide range of benchmarks.
• 摘要：在本技术报告中，我们介绍了在 Skywork-MoE 开发过程中实施的训练方法，Skywork-MoE 是一种高性能混合专家 (MoE) 大型语言模型 (LLM)，拥有 1460 亿个参数和 16 位专家。它从 Skywork-13B 模型中预先存在的密集检查点进行初始化。我们探索了升级与从头开始训练初始化的比较效果。我们的研究结果表明，在这两种方法之间进行选择时，应同时考虑现有密集检查点的性能和 MoE 训练预算。我们重点介绍了两种创新技术：门控逻辑归一化（可改善专家多样化）和自适应辅助损失系数（允许对辅助损失系数进行层特定调整）。我们的实验结果验证了这些方法的有效性。利用这些技术和见解，我们在 SkyPile 语料库的压缩子集上训练了升级的 Skywork-MoE。评估结果表明，我们的模型在广泛的基准测试中都表现出色。

Title: MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

• Authors: Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, Yang You
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06565
• Pdf URL: https://arxiv.org/pdf/2406.06565
• Copy Paste: [[2406.06565]] MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures(https://arxiv.org/abs/2406.06565)
• Keywords: language model, llm, chat
• Abstract: Evaluating large language models (LLMs) is challenging. Traditional ground-truth-based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, we propose MixEval, a new paradigm for establishing efficient, gold-standard LLM evaluation by strategically mixing off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on MixEval, we further build MixEval-Hard, which offers more room for model improvement. Our benchmarks' advantages lie in (1) a 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive meta-evaluation and analysis for our and existing LLM benchmarks to deepen the community's understanding of LLM evaluation and guide future research directions.
• 摘要：评估大型语言模型 (LLM) 具有挑战性。传统的基于真实值的基准测试无法捕捉真实世界查询的全面性和细微差别，而 LLM 作为评判基准测试则存在评分偏差和查询数量有限等问题。随着时间的推移，这两种方法都可能受到污染。面向用户的评估（例如 Chatbot Arena）提供了可靠的信号，但成本高昂且速度慢。在这项工作中，我们提出了 MixEval，这是一种通过策略性地混合现成的基准测试来建立高效、黄金标准的 LLM 评估的新范式。它通过将从网络挖掘出的查询与现有基准测试中的类似查询进行匹配，连接了 (1) 全面且分布良好的真实世界用户查询和 (2) 高效且公平评分的基于真实值的基准测试。基于 MixEval，我们进一步构建了 MixEval-Hard，为模型改进提供了更多空间。我们的基准测试的优势在于：(1) 由于高度公正的查询分布和评分机制，与 Chatbot Arena 的模型排名相关性达到 0.96；(2) 执行速度快、成本低且可重复（MMLU 时间和成本的 6%）；(3) 通过快速稳定的数据更新管道实现动态评估。我们为我们和现有的 LLM 基准测试提供了广泛的元评估和分析，以加深社区对 LLM 评估的理解并指导未来的研究方向。

Title: RAG Enabled Conversations about Household Electricity Monitoring

• Authors: Carolina Fortuna, Vid Hanžel, Blaž Bertalanič
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] RAG Enabled Conversations about Household Electricity Monitoring(https://arxiv.org/abs/)
• Keywords: language model, gpt, llm, chat, retrieval augmented generation
• Abstract: In this paper, we investigate the integration of Retrieval Augmented Generation (RAG) with large language models (LLMs) such as ChatGPT, Gemini, and Llama to enhance the accuracy and specificity of responses to complex questions about electricity datasets. Recognizing the limitations of LLMs in generating precise and contextually relevant answers due to their dependency on the patterns in training data rather than factual understanding, we propose a solution that leverages a specialized electricity knowledge graph. This approach facilitates the retrieval of accurate, real-time data which is then synthesized with the generative capabilities of LLMs. Our findings illustrate that the RAG approach not only reduces the incidence of incorrect information typically generated by LLMs but also significantly improves the quality of the output by grounding responses in verifiable data. This paper details our methodology, presents a comparative analysis of responses with and without RAG, and discusses the implications of our findings for future applications of AI in specialized sectors like energy data analysis.
• 摘要：在本文中，我们研究了检索增强生成 (RAG) 与大型语言模型 (LLM)（如 ChatGPT、Gemini 和 Llama）的集成，以提高对电力数据集复杂问题的回答的准确性和特异性。认识到 LLM 在生成精确且上下文相关的答案方面的局限性（因为它们依赖于训练数据中的模式而不是事实理解），我们提出了一种利用专门的电力知识图谱的解决方案。这种方法有助于检索准确的实时数据，然后将其与 LLM 的生成功能进行合成。我们的研究结果表明，RAG 方法不仅可以减少通常由 LLM 生成的不正确信息的发生率，而且还可以通过将响应建立在可验证数据中来显著提高输出的质量。本文详细介绍了我们的方法，对有无 RAG 的响应进行了比较分析，并讨论了我们的研究结果对未来 AI 在能源数据分析等专业领域的应用的影响。

Title: SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

• Authors: Quandong Wang, Yuxuan Yuan, Xiaoyu Yang, Ruike Zhang, Kang Zhao, Wei Liu, Jian Luan, Daniel Povey, Bin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06571
• Pdf URL: https://arxiv.org/pdf/2406.06571
• Copy Paste: [[2406.06571]] SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM(https://arxiv.org/abs/2406.06571)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass Large Language Model, an innovative architecture that extends the core decoder-only framework by incorporating subsampling, upsampling, and bypass modules. The subsampling modules are responsible for shortening the sequence, while the upsampling modules restore the sequence length, and the bypass modules enhance convergence. In comparison to LLaMA, the proposed SUBLLM exhibits significant enhancements in both training and inference speeds as well as memory usage, while maintaining competitive few-shot performance. During training, SUBLLM increases speeds by 26% and cuts memory by 10GB per GPU. In inference, it boosts speeds by up to 37% and reduces memory by 1GB per GPU. The training and inference speeds can be enhanced by 34% and 52% respectively when the context window is expanded to 8192. We shall release the source code of the proposed architecture in the published version.
• 摘要：虽然大型语言模型 (LLM) 在各个领域取得了显著成功，但训练和推理的效率仍然是一个重大挑战。为了解决这个问题，我们提出了 SUBLLM，即子采样-上采样-旁路大型语言模型 (Subsampling-Upsampling-Bypass Large Language Model)，这是一种创新架构，通过结合子采样、上采样和旁路模块扩展了核心解码器专用框架。子采样模块负责缩短序列，上采样模块恢复序列长度，旁路模块增强收敛。与 LLaMA 相比，所提出的 SUBLLM 在训练和推理速度以及内存使用方面均表现出显着的增强，同时保持了具有竞争力的少样本性能。在训练期间，SUBLLM 将速度提高了 26%，并且每个 GPU 的内存减少了 10GB。在推理方面，它将速度提高了 37%，并且每个 GPU 的内存减少了 1GB。当上下文窗口扩展到8192时，训练和推理速度可分别提高34％和52％。我们将在发布的版本中发布所提架构的源代码。

Title: Graph Neural Network Enhanced Retrieval for Question Answering of LLMs

• Authors: Zijian Li, Qingyan Guo, Jiawei Shao, Lei Song, Jiang Bian, Jun Zhang, Rui Wang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2406.06572
• Pdf URL: https://arxiv.org/pdf/2406.06572
• Copy Paste: [[2406.06572]] Graph Neural Network Enhanced Retrieval for Question Answering of LLMs(https://arxiv.org/abs/2406.06572)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Retrieval augmented generation has revolutionized large language model (LLM) outputs by providing factual supports. Nevertheless, it struggles to capture all the necessary knowledge for complex reasoning questions. Existing retrieval methods typically divide reference documents into passages, treating them in isolation. These passages, however, are often interrelated, such as passages that are contiguous or share the same keywords. Therefore, recognizing the relatedness is crucial for enhancing the retrieval process. In this paper, we propose a novel retrieval method, called GNN-Ret, which leverages graph neural networks (GNNs) to enhance retrieval by considering the relatedness between passages. Specifically, we first construct a graph of passages by connecting passages that are structure-related and keyword-related. A graph neural network (GNN) is then leveraged to exploit the relationships between passages and improve the retrieval of supporting passages. Furthermore, we extend our method to handle multi-hop reasoning questions using a recurrent graph neural network (RGNN), named RGNN-Ret. At each step, RGNN-Ret integrates the graphs of passages from previous steps, thereby enhancing the retrieval of supporting passages. Extensive experiments on benchmark datasets demonstrate that GNN-Ret achieves higher accuracy for question answering with a single query of LLMs than strong baselines that require multiple queries, and RGNN-Ret further improves accuracy and achieves state-of-the-art performance, with up to 10.4% accuracy improvement on the 2WikiMQA dataset.
• 摘要：检索增强生成通过提供事实支持彻底改变了大型语言模型 (LLM) 的输出。然而，它很难捕捉到复杂推理问题所需的所有知识。现有的检索方法通常将参考文档分成段落，单独处理它们。然而，这些段落往往是相互关联的，例如连续的段落或共享相同关键词的段落。因此，识别相关性对于增强检索过程至关重要。在本文中，我们提出了一种名为 GNN-Ret 的新型检索方法，该方法利用图神经网络 (GNN) 通过考虑段落之间的相关性来增强检索。具体来说，我们首先通过连接与结构相关和与关键词相关的段落来构建段落图。然后利用图神经网络 (GNN) 来利用段落之间的关系并改进支持段落的检索。此外，我们扩展了我们的方法，使用名为 RGNN-Ret 的循环图神经网络 (RGNN) 来处理多跳推理问题。在每个步骤中，RGNN-Ret 都会整合来自之前步骤的段落图，从而增强对支持段落的检索。在基准数据集上进行的大量实验表明，GNN-Ret 在单次查询 LLM 时实现的问答准确率高于需要多次查询的强基线，并且 RGNN-Ret 进一步提高了准确率并实现了最佳性能，在 2WikiMQA 数据集上的准确率提高了 10.4%。

Title: MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering

• Authors: Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, Eric Horvitz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06573
• Pdf URL: https://arxiv.org/pdf/2406.06573
• Copy Paste: [[2406.06573]] MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering(https://arxiv.org/abs/2406.06573)
• Keywords: language model, llm
• Abstract: Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance generalizes to real-world clinical settings. Medical question-answering benchmarks rely on assumptions consistent with quantifying LLM performance but that may not hold in the open world of the clinic. Yet LLMs learn broad knowledge that can help the LLM generalize to practical conditions regardless of unrealistic assumptions in celebrated benchmarks. We seek to quantify how well LLM medical question-answering benchmark performance generalizes when benchmark assumptions are violated. Specifically, we present an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz attempts to modify benchmark questions in ways aimed at confounding the LLM. We demonstrate the approach by targeting strong assumptions about patient characteristics presented in the MedQA benchmark. Successful "attacks" modify a benchmark item in ways that would be unlikely to fool a medical expert but nonetheless "trick" the LLM into changing from a correct to an incorrect answer. Further, we present a permutation test technique that can ensure a successful attack is statistically significant. We show how to use performance on a "MedFuzzed" benchmark, as well as individual successful attacks. The methods show promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.
• 摘要：大型语言模型 (LLM) 在医学问答基准上取得了令人印象深刻的表现。然而，高基准准确度并不意味着性能可以推广到现实世界的临床环境中。医学问答基准依赖于与量化 LLM 性能一致的假设，但在开放的临床世界中可能不成立。然而，LLM 学习广泛的知识，可以帮助 LLM 推广到实际情况，而不管著名基准中的不切实际的假设如何。我们试图量化当基准假设被违反时 LLM 医学问答基准性能的推广程度。具体来说，我们提出了一种对抗性方法，我们称之为 MedFuzz（用于医学模糊测试）。MedFuzz 试图以旨在混淆 LLM 的方式修改基准问题。我们通过针对 MedQA 基准中提出的关于患者特征的强假设来展示这种方法。成功的“攻击”会以不太可能欺骗医学专家的方式修改基准项目，但仍然会“诱使”LLM 从正确答案变为错误答案。此外，我们提出了一种置换测试技术，可以确保成功的攻击具有统计意义。我们展示了如何使用“MedFuzzed”基准测试中的表现以及个别成功的攻击。这些方法有望深入了解 LLM 在更现实的环境中稳健运行的能力。

Title: Towards Transparency: Exploring LLM Trainings Datasets through Visual Topic Modeling and Semantic Frame

• Authors: Charles de Dampierre, Andrei Mogoutov, Nicolas Baumard
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06574
• Pdf URL: https://arxiv.org/pdf/2406.06574
• Copy Paste: [[2406.06574]] Towards Transparency: Exploring LLM Trainings Datasets through Visual Topic Modeling and Semantic Frame(https://arxiv.org/abs/2406.06574)
• Keywords: llm
• Abstract: LLMs are now responsible for making many decisions on behalf of humans: from answering questions to classifying things, they have become an important part of everyday life. While computation and model architecture have been rapidly expanding in recent years, the efforts towards curating training datasets are still in their beginnings. This underappreciation of training datasets has led LLMs to create biased and low-quality content. In order to solve that issue, we present Bunka, a software that leverages AI and Cognitive Science to improve the refinement of textual datasets. We show how Topic Modeling coupled with 2-dimensional Cartography can increase the transparency of datasets. We then show how the same Topic Modeling techniques can be applied to Preferences datasets to accelerate the fine-tuning process and increase the capacities of the model on different benchmarks. Lastly, we show how using Frame Analysis can give insights into existing biases in the training corpus. Overall, we argue that we need better tools to explore and increase the quality and transparency of LLMs training datasets.
• 摘要：LLM 现在负责代表人类做出许多决定：从回答问题到对事物进行分类，它们已成为日常生活的重要组成部分。虽然近年来计算和模型架构迅速扩展，但对训练数据集的策划工作仍处于起步阶段。对训练数据集的低估导致 LLM 创建了有偏见和低质量的内容。为了解决这个问题，我们介绍了 Bunka，这是一款利用人工智能和认知科学来改进文本数据集细化的软件。我们展示了主题建模与二维制图相结合如何提高数据集的透明度。然后，我们展示了如何将相同的主题建模技术应用于偏好数据集，以加速微调过程并提高模型在不同基准上的容量。最后，我们展示了如何使用框架分析来深入了解训练语料库中现有的偏见。总的来说，我们认为我们需要更好的工具来探索和提高 LLM 训练数据集的质量和透明度。

Title: Ask-EDA: A Design Assistant Empowered by LLM, Hybrid RAG and Abbreviation De-hallucination

• Authors: Luyao Shi, Michael Kazda, Bradley Sears, Nick Shropshire, Ruchir Puri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06575
• Pdf URL: https://arxiv.org/pdf/2406.06575
• Copy Paste: [[2406.06575]] Ask-EDA: A Design Assistant Empowered by LLM, Hybrid RAG and Abbreviation De-hallucination(https://arxiv.org/abs/2406.06575)
• Keywords: language model, llm, hallucination, chat, retrieval augmented generation, agent
• Abstract: Electronic design engineers are challenged to find relevant information efficiently for a myriad of tasks within design construction, verification and technology development. Large language models (LLM) have the potential to help improve productivity by serving as conversational agents that effectively function as subject-matter experts. In this paper we demonstrate Ask-EDA, a chat agent designed to serve as a 24x7 expert available to provide guidance to design engineers. Ask-EDA leverages LLM, hybrid retrieval augmented generation (RAG) and abbreviation de-hallucination (ADH) techniques to deliver more relevant and accurate responses. We curated three evaluation datasets, namely q2a-100, cmds-100 and abbr-100. Each dataset is tailored to assess a distinct aspect: general design question answering, design command handling and abbreviation resolution. We demonstrated that hybrid RAG offers over a 40% improvement in Recall on the q2a-100 dataset and over a 60% improvement on the cmds-100 dataset compared to not using RAG, while ADH yields over a 70% enhancement in Recall on the abbr-100 dataset. The evaluation results show that Ask-EDA can effectively respond to design-related inquiries.
• 摘要：电子设计工程师面临的挑战是高效地找到相关信息，以完成设计构建、验证和技术开发中的大量任务。大型语言模型 (LLM) 可以充当对话代理，有效地充当主题专家，从而帮助提高生产率。在本文中，我们展示了 Ask-EDA，这是一个聊天代理，旨在充当 24x7 专家，为设计工程师提供指导。Ask-EDA 利用 LLM、混合检索增强生成 (RAG) 和缩写去幻觉 (ADH) 技术来提供更相关和准确的响应。我们整理了三个评估数据集，分别是 q2a-100、cmds-100 和 abbr-100。每个数据集都经过量身定制，以评估一个不同的方面：一般设计问题回答、设计命令处理和缩写解析。我们证明，与不使用 RAG 相比，混合 RAG 在 q2a-100 数据集上的召回率提高了 40% 以上，在 cmds-100 数据集上的召回率提高了 60% 以上，而 ADH 在 abbr-100 数据集上的召回率提高了 70% 以上。评估结果表明 Ask-EDA 可以有效响应与设计相关的查询。

Title: OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step

• Authors: Owen Dugan, Donato Manuel Jimenez Beneto, Charlotte Loh, Zhuo Chen, Rumen Dangovski, Marin Soljačić
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06576
• Pdf URL: https://arxiv.org/pdf/2406.06576
• Copy Paste: [[2406.06576]] OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step(https://arxiv.org/abs/2406.06576)
• Keywords: language model, gpt, llm
• Abstract: Despite significant advancements in text generation and reasoning, Large Language Models (LLMs) still face challenges in accurately performing complex arithmetic operations. To achieve accurate calculations, language model systems often enable LLMs to generate code for arithmetic operations. However, this approach compromises speed and security and, if finetuning is involved, risks the language model losing prior capabilities. We propose a framework that enables exact arithmetic in \textit{a single autoregressive step}, providing faster, more secure, and more interpretable LLM systems with arithmetic capabilities. We use the hidden states of an LLM to control a symbolic architecture which performs arithmetic. Our implementation using Llama 3 8B Instruct with OccamNet as a symbolic model (OccamLlama) achieves 100\% accuracy on single arithmetic operations ($+,-,\times,÷,\sin{},\cos{},\log{},\exp{},\sqrt{}$), outperforming GPT 4o and on par with GPT 4o using a code interpreter. OccamLlama also outperforms both Llama 3 8B Instruct and GPT 3.5 Turbo on multistep reasoning problems involving challenging arithmetic, thus enabling small LLMs to match the arithmetic performance of even much larger models. We will make our code public shortly.
• 摘要：尽管在文本生成和推理方面取得了重大进展，大型语言模型 (LLM) 在准确执行复杂算术运算方面仍然面临挑战。为了实现精确计算，语言模型系统通常允许 LLM 生成算术运算代码。但是，这种方法会损害速度和安全性，如果涉及微调，语言模型可能会失去先前的功能。我们提出了一个框架，可以在 \textit{单个自回归步骤} 中实现精确算术，从而提供更快、更安全、更易于解释的具有算术功能的 LLM 系统。我们使用 LLM 的隐藏状态来控制执行算术的符号架构。我们使用 Llama 3 8B Instruct 和 OccamNet 作为符号模型 (OccamLlama) 实现的算法在单个算术运算 ($+、-、\times、÷、\sin{}、\cos{}、\log{}、\exp{}、\sqrt{}$) 上实现了 100\% 的准确率，优于 GPT 4o，与使用代码解释器的 GPT 4o 相当。OccamLlama 在涉及具有挑战性的算术的多步推理问题上也优于 Llama 3 8B Instruct 和 GPT 3.5 Turbo，从而使小型 LLM 能够与甚至更大模型的算术性能相匹配。我们将很快公开我们的代码。

Title: RAG-based Crowdsourcing Task Decomposition via Masked Contrastive Learning with Prompts

• Authors: Jing Yang, Xiao Wang, Yu Zhao, Yuhang Liu, Fei-Yue Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06577
• Pdf URL: https://arxiv.org/pdf/2406.06577
• Copy Paste: [[2406.06577]] RAG-based Crowdsourcing Task Decomposition via Masked Contrastive Learning with Prompts(https://arxiv.org/abs/2406.06577)
• Keywords: language model, hallucination, prompt, retrieval-augmented generation
• Abstract: Crowdsourcing is a critical technology in social manufacturing, which leverages an extensive and boundless reservoir of human resources to handle a wide array of complex tasks. The successful execution of these complex tasks relies on task decomposition (TD) and allocation, with the former being a prerequisite for the latter. Recently, pre-trained language models (PLMs)-based methods have garnered significant attention. However, they are constrained to handling straightforward common-sense tasks due to their inherent restrictions involving limited and difficult-to-update knowledge as well as the presence of hallucinations. To address these issues, we propose a retrieval-augmented generation-based crowdsourcing framework that reimagines TD as event detection from the perspective of natural language understanding. However, the existing detection methods fail to distinguish differences between event types and always depend on heuristic rules and external semantic analyzing tools. Therefore, we present a Prompt-Based Contrastive learning framework for TD (PBCT), which incorporates a prompt-based trigger detector to overcome dependence. Additionally, trigger-attentive sentinel and masked contrastive learning are introduced to provide varying attention to trigger and contextual features according to different event types. Experiment results demonstrate the competitiveness of our method in both supervised and zero-shot detection. A case study on printed circuit board manufacturing is showcased to validate its adaptability to unknown professional domains.
• 摘要：众包是社会制造中的一项关键技术，它利用广泛而无限的人力资源来处理各种复杂任务。这些复杂任务的成功执行依赖于任务分解 (TD) 和分配，前者是后者的先决条件。最近，基于预训练语言模型 (PLM) 的方法引起了广泛关注。然而，由于它们固有的限制，包括知识有限且难以更新以及存在幻觉，它们只能处理简单的常识性任务。为了解决这些问题，我们提出了一个基于检索增强生成的众包框架，从自然语言理解的角度将 TD 重新想象为事件检测。然而，现有的检测方法无法区分事件类型之间的差异，并且总是依赖于启发式规则和外部语义分析工具。因此，我们提出了一种基于提示的 TD 对比学习框架 (PBCT)，它结合了基于提示的触发检测器来克服依赖性。此外，还引入了触发注意哨兵和掩蔽对比学习，以根据不同的事件类型对触发和上下文特征提供不同的关注。实验结果证明了我们的方法在监督和零样本检测中的竞争力。展示了印刷电路板制造案例研究，以验证其对未知专业领域的适应性。

Title: From Redundancy to Relevance: Enhancing Explainability in Multimodal Large Language Models

• Authors: Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, Jieping Ye
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2406.06579
• Pdf URL: https://arxiv.org/pdf/2406.06579
• Copy Paste: [[2406.06579]] From Redundancy to Relevance: Enhancing Explainability in Multimodal Large Language Models(https://arxiv.org/abs/2406.06579)
• Keywords: language model, llm, prompt
• Abstract: Recently, multimodal large language models have exploded with an endless variety, most of the popular Large Vision Language Models (LVLMs) depend on sequential visual representation, where images are converted into hundreds or thousands of tokens before being input into the Large Language Model (LLM) along with language prompts. The black-box design hinders the interpretability of visual-language models, especially regarding more complex reasoning tasks. To explore the interaction process between image and text in complex reasoning tasks, we introduce the information flow method to visualize the interaction mechanism. By analyzing the dynamic flow of the information flow, we find that the information flow appears to converge in the shallow layer. Further investigation revealed a redundancy of the image token in the shallow layer. Consequently, a truncation strategy was introduced to aggregate image tokens within these shallow layers. This approach has been validated through experiments across multiple models, yielding consistent improvements.
• 摘要：近年来，多模态大型语言模型如雨后春笋般涌现，种类繁多，目前流行的大型视觉语言模型（LVLM）大多依赖于顺序视觉表示，即图像被转换成数百或数千个 token，然后与语言提示一起输入到大型语言模型（LLM）中。黑箱设计阻碍了视觉语言模型的可解释性，特别是在更复杂的推理任务中。为了探索复杂推理任务中图像与文本的交互过程，我们引入信息流方法来可视化交互机制。通过分析信息流的动态流动，我们发现信息流似乎在浅层汇聚。进一步的研究表明，浅层中图像 token 存在冗余。因此，我们引入了一种截断策略来聚合这些浅层中的图像 token。该方法已经通过多个模型的实验得到验证，并取得了持续的改进。

Title: Break the Chain: Large Language Models Can be Shortcut Reasoners

• Authors: Mengru Ding, Hanmeng Liu, Zhizhang Fu, Jian Song, Wenbo Xie, Yue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06580
• Pdf URL: https://arxiv.org/pdf/2406.06580
• Copy Paste: [[2406.06580]] Break the Chain: Large Language Models Can be Shortcut Reasoners(https://arxiv.org/abs/2406.06580)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Recent advancements in Chain-of-Thought (CoT) reasoning utilize complex modules but are hampered by high token consumption, limited applicability, and challenges in reproducibility. This paper conducts a critical evaluation of CoT prompting, extending beyond arithmetic to include complex logical and commonsense reasoning tasks, areas where standard CoT methods fall short. We propose the integration of human-like heuristics and shortcuts into language models (LMs) through "break the chain" strategies. These strategies disrupt traditional CoT processes using controlled variables to assess their efficacy. Additionally, we develop innovative zero-shot prompting strategies that encourage the use of shortcuts, enabling LMs to quickly exploit reasoning clues and bypass detailed procedural steps. Our comprehensive experiments across various LMs, both commercial and open-source, reveal that LMs maintain effective performance with "break the chain" strategies. We also introduce ShortcutQA, a dataset specifically designed to evaluate reasoning through shortcuts, compiled from competitive tests optimized for heuristic reasoning tasks such as forward/backward reasoning and simplification. Our analysis confirms that ShortcutQA not only poses a robust challenge to LMs but also serves as an essential benchmark for enhancing reasoning efficiency in AI.
• 摘要：思维链 (CoT) 推理的最新进展利用了复杂的模块，但受到高 token 消耗、有限的适用性和可重复性挑战的阻碍。本文对 CoT 提示进行了严格的评估，范围超出了算术，包括复杂的逻辑和常识推理任务，这是标准 CoT 方法不足的领域。我们建议通过“打破链条”策略将类似人类的启发式方法和捷径集成到语言模型 (LM) 中。这些策略使用受控变量来评估其有效性，从而破坏了传统的 CoT 流程。此外，我们开发了创新的零样本提示策略，鼓励使用捷径，使 LM 能够快速利用推理线索并绕过详细的程序步骤。我们对各种 LM（包括商业和开源）进行的全面实验表明，LM 使用“打破链条”策略保持了有效的性能。我们还介绍了 ShortcutQA，这是一个专门用于评估通过捷径进行推理的数据集，由针对启发式推理任务（例如正向/反向推理和简化）优化的竞争性测试编译而成。我们的分析证实，ShortcutQA 不仅对 LM 提出了严峻的挑战，而且也是提高 AI 推理效率的重要基准。

Title: Set-Based Prompting: Provably Solving the Language Model Order Dependency Problem

• Authors: Reid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang, Cynthia Dwork
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06581
• Pdf URL: https://arxiv.org/pdf/2406.06581
• Copy Paste: [[2406.06581]] Set-Based Prompting: Provably Solving the Language Model Order Dependency Problem(https://arxiv.org/abs/2406.06581)
• Keywords: language model, llm, prompt
• Abstract: The development of generative language models that can create long and coherent textual outputs via autoregression has lead to a proliferation of uses and a corresponding sweep of analyses as researches work to determine the limitations of this new paradigm. Unlike humans, these 'Large Language Models' (LLMs) are highly sensitive to small changes in their inputs, leading to unwanted inconsistency in their behavior. One problematic inconsistency when LLMs are used to answer multiple-choice questions or analyze multiple inputs is order dependency: the output of an LLM can (and often does) change significantly when sub-sequences are swapped, despite both orderings being semantically identical. In this paper we present , a technique that guarantees the output of an LLM will not have order dependence on a specified set of sub-sequences. We show that this method provably eliminates order dependency, and that it can be applied to any transformer-based LLM to enable text generation that is unaffected by re-orderings. Delving into the implications of our method, we show that, despite our inputs being out of distribution, the impact on expected accuracy is small, where the expectation is over the order of uniformly chosen shuffling of the candidate responses, and usually significantly less in practice. Thus, can be used as a 'dropped-in' method on fully trained models. Finally, we discuss how our method's success suggests that other strong guarantees can be obtained on LLM performance via modifying the input representations.
• 摘要：随着研究人员努力确定这一新范式的局限性，能够通过自回归创建长而连贯的文本输出的生成语言模型的发展导致了用途的激增和相应的分析范围的扩大。与人类不同，这些“大型语言模型”（LLM）对输入的微小变化高度敏感，导致其行为出现不必要的不​​一致。当使用 LLM 回答多项选择题或分析多个输入时，一个有问题的不一致是顺序依赖性：当交换子序列时，LLM 的输出可能会（并且通常会）发生显着变化，尽管两种顺序在语义上是相同的。在本文中，我们提出了一种技术，该技术可保证 LLM 的输出不会对指定的一组子序列具有顺序依赖性。我们表明，这种方法可以证明消除顺序依赖性，并且可以应用于任何基于转换器的 LLM，以实现不受重新排序影响的文本生成。深入研究我们的方法的含义，我们表明，尽管我们的输入不属于分布，但对预期准确度的影响很小，其中期望值超过候选响应的均匀选择改组的阶数，并且在实践中通常要小得多。因此，可以用作完全训练模型的“插入式”方法。最后，我们讨论了我们的方法的成功如何表明可以通过修改输入表示来获得 LLM 性能的其他强有力保证。

Title: Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing

• Authors: Viet Anh Trinh, Rosy Southwell, Yiwen Guan, Xinlu He, Zhiyong Wang, Jacob Whitehill
• Subjects: cs.CL, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2406.06582
• Pdf URL: https://arxiv.org/pdf/2406.06582
• Copy Paste: [[2406.06582]] Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing(https://arxiv.org/abs/2406.06582)
• Keywords: language model, llm
• Abstract: Recent work on discrete speech tokenization has paved the way for models that can seamlessly perform multiple tasks across modalities, e.g., speech recognition, text to speech, speech to speech translation. Moreover, large language models (LLMs) pretrained from vast text corpora contain rich linguistic information that can improve accuracy in a variety of tasks. In this paper, we present a decoder-only Discrete Multimodal Language Model (DMLM), which can be flexibly applied to multiple tasks (ASR, T2S, S2TT, etc.) and modalities (text, speech, vision). We explore several critical aspects of discrete multi-modal models, including the loss function, weight initialization, mixed training supervision, and codebook. Our results show that DMLM benefits significantly, across multiple tasks and datasets, from a combination of supervised and unsupervised training. Moreover, for ASR, it benefits from initializing DMLM from a pretrained LLM, and from a codebook derived from Whisper activations.
• 摘要：离散语音标记化的最新研究为能够无缝执行跨模态的多项任务的模型铺平了道路，例如语音识别、文本到语音、语音到语音翻译。此外，从大量文本语料库中预训练的大型语言模型 (LLM) 包含丰富的语言信息，可以提高各种任务的准确性。在本文中，我们提出了一种仅用于解码器的离散多模态语言模型 (DMLM)，它可以灵活地应用于多项任务（ASR、T2S、S2TT 等）和模态（文本、语音、视觉）。我们探讨了离散多模态模型的几个关键方面，包括损失函数、权重初始化、混合训练监督和码本。我们的结果表明，DMLM 在多个任务和数据集中从监督和无监督训练的组合中受益匪浅。此外，对于 ASR，它受益于从预训练的 LLM 初始化 DMLM 以及从 Whisper 激活派生的码本。

Title: Evaluating the Efficacy of Large Language Models in Detecting Fake News: A Comparative Analysis

• Authors: Sahas Koka, Anthony Vuong, Anish Kataria
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06584
• Pdf URL: https://arxiv.org/pdf/2406.06584
• Copy Paste: [[2406.06584]] Evaluating the Efficacy of Large Language Models in Detecting Fake News: A Comparative Analysis(https://arxiv.org/abs/2406.06584)
• Keywords: language model, gpt, llm
• Abstract: In an era increasingly influenced by artificial intelligence, the detection of fake news is crucial, especially in contexts like election seasons where misinformation can have significant societal impacts. This study evaluates the effectiveness of various LLMs in identifying and filtering fake news content. Utilizing a comparative analysis approach, we tested four large LLMs -- GPT-4, Claude 3 Sonnet, Gemini Pro 1.0, and Mistral Large -- and two smaller LLMs -- Gemma 7B and Mistral 7B. By using fake news dataset samples from Kaggle, this research not only sheds light on the current capabilities and limitations of LLMs in fake news detection but also discusses the implications for developers and policymakers in enhancing AI-driven informational integrity.
• 摘要：在这个人工智能影响日益深远的时代，检测虚假新闻至关重要，尤其是在选举季等错误信息可能对社会产生重大影响的背景下。本研究评估了各种 LLM 在识别和过滤虚假新闻内容方面的有效性。利用比较分析方法，我们测试了四个大型 LLM——GPT-4、Claude 3 Sonnet、Gemini Pro 1.0 和 Mistral Large——以及两个较小的 LLM——Gemma 7B 和 Mistral 7B。通过使用来自 Kaggle 的虚假新闻数据集样本，本研究不仅揭示了 LLM 在虚假新闻检测方面的当前能力和局限性，还讨论了对开发人员和政策制定者在增强人工智能驱动的信息完整性方面的影响。

Title: Bi-Chainer: Automated Large Language Models Reasoning with Bidirectional Chaining

• Authors: Shuqi Liu, Bowei He, Linqi Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06586
• Pdf URL: https://arxiv.org/pdf/2406.06586
• Copy Paste: [[2406.06586]] Bi-Chainer: Automated Large Language Models Reasoning with Bidirectional Chaining(https://arxiv.org/abs/2406.06586)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown human-like reasoning abilities but still face challenges in solving complex logical problems. Existing unidirectional chaining methods, such as forward chaining and backward chaining, suffer from issues like low prediction accuracy and efficiency. To address these, we propose a bidirectional chaining method, Bi-Chainer, which dynamically switches to depth-first reasoning in the opposite reasoning direction when it encounters multiple branching options within the current direction. Thus, the intermediate reasoning results can be utilized as guidance to facilitate the reasoning process. We show that Bi-Chainer achieves sizable accuracy boots over unidirectional chaining frameworks on four challenging logical reasoning datasets. Moreover, Bi-Chainer enhances the accuracy of intermediate proof steps and reduces the average number of inference calls, resulting in more efficient and accurate reasoning.
• 摘要：大型语言模型 (LLM) 已展现出类似人类的推理能力，但在解决复杂逻辑问题方面仍面临挑战。现有的单向链接方法，例如前向链接和后向链接，存在预测精度低和效率低等问题。为了解决这些问题，我们提出了一种双向链接方法 Bi-Chainer，当它在当前方向遇到多个分支选项时，它会动态切换到相反推理方向上的深度优先推理。因此，中间推理结果可用作指导以促进推理过程。我们表明，Bi-Chainer 在四个具有挑战性的逻辑推理数据集上实现了比单向链接框架更高的准确率。此外，Bi-Chainer 提高了中间证明步骤的准确性并减少了平均推理调用次数，从而实现更高效、更准确的推理。

Title: Exploring Human-AI Perception Alignment in Sensory Experiences: Do LLMs Understand Textile Hand?

• Authors: Shu Zhong, Elia Gatti, Youngjun Cho, Marianna Obrist
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2406.06587
• Pdf URL: https://arxiv.org/pdf/2406.06587
• Copy Paste: [[2406.06587]] Exploring Human-AI Perception Alignment in Sensory Experiences: Do LLMs Understand Textile Hand?(https://arxiv.org/abs/2406.06587)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This work investigates how well LLMs align with human touch experiences using the "textile hand" task. We created a "Guess What Textile" interaction in which participants were given two textile samples -- a target and a reference -- to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space. Our results suggest that a degree of perceptual alignment exists, however varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants didn't perceive their textile experiences closely matched by the LLM predictions. This is only the first exploration into perceptual alignment around touch, exemplified through textile hand. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks.
• 摘要：将大型语言模型 (LLM) 的行为与人类意图相一致对于未来的人工智能至关重要。这种一致性的一个重要但经常被忽视的方面是感知一致性。与视觉等其他感官模式相比，触觉等感知模式更加多面和微妙。这项研究使用“纺织品手”任务研究了 LLM 与人类触觉体验的一致性。我们创建了一个“猜猜什么纺织品”互动，参与者被给予两个纺织品样本——一个目标和一个参考——来处理。在没有看到它们的情况下，参与者向 LLM 描述了它们之间的差异。使用这些描述，LLM 试图通过评估其高维嵌入空间内的相似性来识别目标纺织品。我们的结果表明存在一定程度的感知一致性，但在不同的纺织品样本之间差异很大。例如，LLM 预测对于丝绸缎子是很好的一致性，但对于棉质牛仔布则不是。此外，参与者没有感觉到他们的纺织品体验与 LLM 预测紧密匹配。这只是对触觉感知一致性的首次探索，以纺织品手为例。我们讨论了这种对齐差异的可能来源，以及更好的人机感知对齐如何有益于未来的日常任务。

Title: Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models

• Authors: Flavio Petruzzellis, Alberto Testolin, Alessandro Sperduti
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2406.06588
• Pdf URL: https://arxiv.org/pdf/2406.06588
• Copy Paste: [[2406.06588]] Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models(https://arxiv.org/abs/2406.06588)
• Keywords: language model, llm, prompt, chat
• Abstract: Large Language Models (LLMs) achieve impressive performance in a wide range of tasks, even if they are often trained with the only objective of chatting fluently with users. Among other skills, LLMs show emergent abilities in mathematical reasoning benchmarks, which can be elicited with appropriate prompting methods. In this work, we systematically investigate the capabilities and limitations of popular open-source LLMs on different symbolic reasoning tasks. We evaluate three models of the Llama 2 family on two datasets that require solving mathematical formulas of varying degrees of difficulty. We test a generalist LLM (Llama 2 Chat) as well as two fine-tuned versions of Llama 2 (MAmmoTH and MetaMath) specifically designed to tackle mathematical problems. We observe that both increasing the scale of the model and fine-tuning it on relevant tasks lead to significant performance gains. Furthermore, using fine-grained evaluation measures, we find that such performance gains are mostly observed with mathematical formulas of low complexity, which nevertheless often remain challenging even for the largest fine-tuned models.
• 摘要：大型语言模型 (LLM) 在各种任务中都取得了令人印象深刻的表现，即使它们通常以与用户流畅地聊天为唯一目标进行训练。除其他技能外，LLM 在数学推理基准中表现出新兴能力，这些能力可以通过适当的提示方法引出。在这项工作中，我们系统地研究了流行的开源 LLM 在不同符号推理任务上的能力和局限性。我们在两个需要解决不同难度的数学公式的数据集上评估了 Llama 2 系列的三个模型。我们测试了一个通用 LLM（Llama 2 Chat）以及两个专门为解决数学问题而设计的 Llama 2 微调版本（MAmmoTH 和 MetaMath）。我们观察到，增加模型的规模和在相关任务上对其进行微调都会带来显着的性能提升。此外，使用细粒度评估指标，我们发现这种性能提升主要体现在低复杂度的数学公式中，尽管如此，即使对于最大的微调模型来说，这通常仍然具有挑战性。

Title: PatentEval: Understanding Errors in Patent Generation

• Authors: You Zuo (ALMAnaCH), Kim Gerdes (LISN), Eric Villemonte de La Clergerie (ALMAnaCH), Benoît Sagot (ALMAnaCH)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06589
• Pdf URL: https://arxiv.org/pdf/2406.06589
• Copy Paste: [[2406.06589]] PatentEval: Understanding Errors in Patent Generation(https://arxiv.org/abs/2406.06589)
• Keywords: language model, llm
• Abstract: In this work, we introduce a comprehensive error typology specifically designed for evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones. We have also developed a benchmark, PatentEval, for systematically assessing language models in this context. Our study includes a comparative analysis, annotated by humans, of various models. These range from those specifically adapted during training for tasks within the patent domain to the latest general-purpose large language models (LLMs). Furthermore, we explored and evaluated some metrics to approximate human judgments in patent text evaluation, analyzing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.
• 摘要：在本研究中，我们引入了一种全面的错误类型学，专门用于评估机器生成的专利文本中的两个不同任务：权利要求到摘要的生成，以及根据先前的权利要求生成下一个权利要求。我们还开发了一个基准 PatentEval，用于系统地评估此背景下的语言模型。我们的研究包括对各种模型的比较分析（由人类注释）。这些模型的范围从在专利领域任务训练期间专门调整的模型到最新的通用大型语言模型 (LLM)。此外，我们探索并评估了一些指标，以近似专利文本评估中的人类判断，分析这些指标与专家评估的一致程度。这些方法为了解专利文本生成专业领域中当前语言模型的能力和局限性提供了宝贵的见解。

Title: Are LLMs classical or nonmonotonic reasoners? Lessons from generics

• Authors: Alina Leidinger, Robert van Rooij, Ekaterina Shutova
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06590
• Pdf URL: https://arxiv.org/pdf/2406.06590
• Copy Paste: [[2406.06590]] Are LLMs classical or nonmonotonic reasoners? Lessons from generics(https://arxiv.org/abs/2406.06590)
• Keywords: llm
• Abstract: Recent scholarship on reasoning in LLMs has supplied evidence of impressive performance and flexible adaptation to machine generated or human feedback. Nonmonotonic reasoning, crucial to human cognition for navigating the real world, remains a challenging, yet understudied task. In this work, we study nonmonotonic reasoning capabilities of seven state-of-the-art LLMs in one abstract and one commonsense reasoning task featuring generics, such as 'Birds fly', and exceptions, 'Penguins don't fly' (see Fig. 1). While LLMs exhibit reasoning patterns in accordance with human nonmonotonic reasoning abilities, they fail to maintain stable beliefs on truth conditions of generics at the addition of supporting examples ('Owls fly') or unrelated information ('Lions have manes'). Our findings highlight pitfalls in attributing human reasoning behaviours to LLMs, as well as assessing general capabilities, while consistent reasoning remains elusive.
• 摘要：最近关于法学硕士 (LLM) 推理的研究提供了令人印象深刻的性能和灵活适应机器生成或人类反馈的证据。非单调推理对于人类认知驾驭现实世界至关重要，仍然是一项具有挑战性但研究不足的任务。在这项工作中，我们在一个抽象推理任务和一个常识推理任务中研究了七种最先进的法学硕士 (LLM) 的非单调推理能力，这些任务以“鸟会飞”等泛型和“企鹅不会飞”等例外为特色（见图 1）。虽然法学硕士 (LLM) 表现出与人类非单调推理能力一致的推理模式，但在添加支持示例（“猫头鹰会飞”）或不相关信息（“狮子有鬃毛”）后，它们无法对泛型的真值条件保持稳定的信念。我们的研究结果强调了将人类推理行为归因于法学硕士 (LLM) 以及评估一般能力的陷阱，而一致的推理仍然难以捉摸。

Title: Exploring Multilingual Large Language Models for Enhanced TNM classification of Radiology Report in lung cancer staging

• Authors: Hidetoshi Matsuo, Mizuho Nishio, Takaaki Matsunaga, Koji Fujimoto, Takamichi Murakami
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06591
• Pdf URL: https://arxiv.org/pdf/2406.06591
• Copy Paste: [[2406.06591]] Exploring Multilingual Large Language Models for Enhanced TNM classification of Radiology Report in lung cancer staging(https://arxiv.org/abs/2406.06591)
• Keywords: language model, gpt, llm
• Abstract: Background: Structured radiology reports remains underdeveloped due to labor-intensive structuring and narrative-style reporting. Deep learning, particularly large language models (LLMs) like GPT-3.5, offers promise in automating the structuring of radiology reports in natural languages. However, although it has been reported that LLMs are less effective in languages other than English, their radiological performance has not been extensively studied. Purpose: This study aimed to investigate the accuracy of TNM classification based on radiology reports using GPT3.5-turbo (GPT3.5) and the utility of multilingual LLMs in both Japanese and English. Material and Methods: Utilizing GPT3.5, we developed a system to automatically generate TNM classifications from chest CT reports for lung cancer and evaluate its performance. We statistically analyzed the impact of providing full or partial TNM definitions in both languages using a Generalized Linear Mixed Model. Results: Highest accuracy was attained with full TNM definitions and radiology reports in English (M = 94%, N = 80%, T = 47%, and ALL = 36%). Providing definitions for each of the T, N, and M factors statistically improved their respective accuracies (T: odds ratio (OR) = 2.35, p < 0.001; N: OR = 1.94, p < 0.01; M: OR = 2.50, p < 0.001). Japanese reports exhibited decreased N and M accuracies (N accuracy: OR = 0.74 and M accuracy: OR = 0.21). Conclusion: This study underscores the potential of multilingual LLMs for automatic TNM classification in radiology reports. Even without additional model training, performance improvements were evident with the provided TNM definitions, indicating LLMs' relevance in radiology contexts.
• 摘要：背景：由于结构化和叙述式报告需要大量劳动力，因此结构化放射学报告仍未得到充分开发。深度学习，尤其是像 GPT-3.5 这样的大型语言模型 (LLM)，有望实现以自然语言自动构建放射学报告。然而，尽管有报道称 LLM 在英语以外的语言中效果较差，但它们的放射学性能尚未得到广泛研究。目的：本研究旨在调查使用 GPT3.5-turbo (GPT3.5) 根据放射学报告进行 TNM 分类的准确性以及日语和英语多语言 LLM 的实用性。材料和方法：利用 GPT3.5，我们开发了一个系统，可自动从肺癌胸部 CT 报告中生成 TNM 分类并评估其性能。我们使用广义线性混合模型对提供两种语言的完整或部分 TNM 定义的影响进行了统计分析。结果：英文的完整 TNM 定义和放射学报告的准确率最高（M = 94%、N = 80%、T = 47% 和 ALL = 36%）。为 T、N 和 M 因素中的每一个提供定义在统计上提高了各自的准确率（T：比值比 (OR) = 2.35，p < 0.001；N：OR = 1.94，p < 0.01；M：OR = 2.50，p < 0.001）。日语报告的 N 和 M 准确率下降（N 准确率：OR = 0.74 和 M 准确率：OR = 0.21）。结论：本研究强调了多语言 LLM 在放射学报告中自动 TNM 分类的潜力。即使没有额外的模型训练，提供的 TNM 定义也能明显提高性能，表明 LLM 与放射学环境相关。

Title: Improve Mathematical Reasoning in Language Models by Automated Process Supervision

• Authors: Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, Abhinav Rastogi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06592
• Pdf URL: https://arxiv.org/pdf/2406.06592
• Copy Paste: [[2406.06592]] Improve Mathematical Reasoning in Language Models by Automated Process Supervision(https://arxiv.org/abs/2406.06592)
• Keywords: language model, llm
• Abstract: Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named \textit{OmegaPRM} for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train a Process Reward Model (PRM). Utilizing this fully automated process supervision alongside the weighted self-consistency algorithm, we have enhanced the instruction tuned Gemini Pro model's math reasoning performance, achieving a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from the 51\% base model performance. Additionally, the entire process operates without any human intervention, making our method both financially and computationally cost-effective compared to existing methods.
• 摘要：复杂的多步骤推理任务（例如解决数学问题或生成代码）对于最先进的大型语言模型 (LLM) 来说仍然是一个重大障碍。使用结果奖励模型 (ORM) 验证 LLM 输出是一种标准的推理时间技术，旨在提高 LLM 的推理性能。然而，对于具有较长或多跳推理链的推理任务，这仍然不够，因为中间结果既没有得到适当的奖励也没有受到惩罚。过程监督通过在推理过程中分配中间奖励来解决这一限制。到目前为止，用于收集过程监督数据的方法依赖于人工注释或每步蒙特卡洛估计，这两种方法的扩展成本都过高，从而阻碍了这种技术的广泛应用。为了应对这一挑战，我们提出了一种新颖的分而治之式蒙特卡洛树搜索 (MCTS) 算法，名为 \textit{OmegaPRM}，用于高效收集高质量的过程监督数据。该算法通过二分搜索快速识别思路链 (CoT) 中的第一个错误，并平衡正反例，从而确保效率和质量。因此，我们能够收集超过 150 万个流程监督注释来训练流程奖励模型 (PRM)。利用这种全自动流程监督以及加权自洽算法，我们增强了指令调整的 Gemini Pro 模型的数学推理性能，在 MATH 基准上实现了 69.4% 的成功率，与 51% 的基础模型性能相比提高了 36%。此外，整个过程无需任何人工干预，与现有方法相比，我们的方法在财务和计算上都更具成本效益。

Title: Are Large Language Models the New Interface for Data Pipelines?

• Authors: Sylvio Barbon Junior, Paolo Ceravolo, Sven Groppe, Mustafa Jarrar, Samira Maghool, Florence Sèdes, Soror Sahri, Maurice Van Keulen
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2406.06596
• Pdf URL: https://arxiv.org/pdf/2406.06596
• Copy Paste: [[2406.06596]] Are Large Language Models the New Interface for Data Pipelines?(https://arxiv.org/abs/2406.06596)
• Keywords: language model, llm
• Abstract: A Language Model is a term that encompasses various types of models designed to understand and generate human communication. Large Language Models (LLMs) have gained significant attention due to their ability to process text with human-like fluency and coherence, making them valuable for a wide range of data-related tasks fashioned as pipelines. The capabilities of LLMs in natural language understanding and generation, combined with their scalability, versatility, and state-of-the-art performance, enable innovative applications across various AI-related fields, including eXplainable Artificial Intelligence (XAI), Automated Machine Learning (AutoML), and Knowledge Graphs (KG). Furthermore, we believe these models can extract valuable insights and make data-driven decisions at scale, a practice commonly referred to as Big Data Analytics (BDA). In this position paper, we provide some discussions in the direction of unlocking synergies among these technologies, which can lead to more powerful and intelligent AI solutions, driving improvements in data pipelines across a wide range of applications and domains integrating humans, computers, and knowledge.
• 摘要：语言模型是一个术语，涵盖旨在理解和生成人类交流的各种模型。大型语言模型 (LLM) 因其能够以类似人类的流畅性和连贯性处理文本的能力而备受关注，这使得它们对于以管道形式出现的各种数据相关任务很有价值。LLM 在自然语言理解和生成方面的能力，加上其可扩展性、多功能性和最先进的性能，使各种与人工智能相关的领域的创新应用成为可能，包括可解释人工智能 (XAI)、自动机器学习 (AutoML) 和知识图谱 (KG)。此外，我们相信这些模型可以提取有价值的见解并大规模做出数据驱动的决策，这种做法通常称为大数据分析 (BDA)。在这份立场文件中，我们讨论了一些如何释放这些技术之间的协同作用，从而带来更强大、更智能的人工智能解决方案，推动各种应用和领域中数据管道的改进，将人类、计算机和知识融为一体。

Title: Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners

• Authors: Abigail Gurin Schleifer, Beata Beigman Klebanov, Moriah Ariely, Giora Alexandron
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06599
• Pdf URL: https://arxiv.org/pdf/2406.06599
• Copy Paste: [[2406.06599]] Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners(https://arxiv.org/abs/2406.06599)
• Keywords: llm
• Abstract: Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by experts into theory-driven Knowledge Profiles (KPs). Comparing these KPs to ones discovered by purely data-driven clustering techniques, we report poor discoverability of most KPs, except for the ones including the correct answers. We trace this "discoverability bias" to the representations of KPs in the pre-trained LLM embeddings space.
• 摘要：使用预训练的 LLM 嵌入将学生对开放式问题的回答无监督地聚类为行为和认知档案是一种新兴技术，但人们对其如何很好地捕捉具有教学意义的信息知之甚少。我们在学生对生物学开放式问题的回答的背景下对此进行了调查，这些问题之前已由专家分析并聚类为理论驱动的知识档案 (KP)。将这些 KP 与纯数据驱动的聚类技术发现的 KP 进行比较，我们发现大多数 KP 的可发现性较差，除了包含正确答案的 KP。我们将这种“可发现性偏差”追溯到预训练的 LLM 嵌入空间中 KP 的表示。

Title: Prototypical Reward Network for Data-Efficient RLHF

• Authors: Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen, Xinhao Zhang, Kunpeng Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06606
• Pdf URL: https://arxiv.org/pdf/2406.06606
• Copy Paste: [[2406.06606]] Prototypical Reward Network for Data-Efficient RLHF(https://arxiv.org/abs/2406.06606)
• Keywords: language model, llm
• Abstract: The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to scalability issues for LLMs and complex tasks. Our proposed framework Proto-RM leverages prototypical networks to enhance reward models under limited human feedback. By enabling stable and reliable structural learning from fewer samples, Proto-RM significantly enhances LLMs' adaptability and accuracy in interpreting human preferences. Extensive experiments on various datasets demonstrate that Proto-RM significantly improves the performance of reward models and LLMs in human feedback tasks, achieving comparable and usually better results than traditional methods, while requiring significantly less data. in data-limited scenarios. This research offers a promising direction for enhancing the efficiency of reward models and optimizing the fine-tuning of language models under restricted feedback conditions.
• 摘要：人类反馈强化学习 (RLHF) 的奖励模型已被证明可有效微调大型语言模型 (LLM)。值得注意的是，为 RLHF 收集人类反馈可能耗费大量资源，并导致 LLM 和复杂任务的可扩展性问题。我们提出的框架 Proto-RM 利用原型网络在有限的人类反馈下增强奖励模型。通过从较少的样本中进行稳定可靠的结构学习，Proto-RM 显著提高了 LLM 在解释人类偏好方面的适应性和准确性。在各种数据集上进行的大量实验表明，Proto-RM 显著提高了奖励模型和 LLM 在人类反馈任务中的表现，在数据有限的场景中实现了与传统方法相当且通常更好的结果，同时需要的数据明显减少。这项研究为在受限反馈条件下提高奖励模型的效率和优化语言模型的微调提供了一个有希望的方向。

Title: The Prompt Report: A Systematic Survey of Prompting Techniques

• Authors: Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, Philip Resnik
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06608
• Pdf URL: https://arxiv.org/pdf/2406.06608
• Copy Paste: [[2406.06608]] The Prompt Report: A Systematic Survey of Prompting Techniques(https://arxiv.org/abs/2406.06608)
• Keywords: prompt
• Abstract: Generative Artificial Intelligence (GenAI) systems are being increasingly deployed across all parts of industry and research settings. Developers and end users interact with these systems through the use of prompting or prompt engineering. While prompting is a widespread and highly researched concept, there exists conflicting terminology and a poor ontological understanding of what constitutes a prompt due to the area's nascency. This paper establishes a structured understanding of prompts, by assembling a taxonomy of prompting techniques and analyzing their use. We present a comprehensive vocabulary of 33 vocabulary terms, a taxonomy of 58 text-only prompting techniques, and 40 techniques for other modalities. We further present a meta-analysis of the entire literature on natural language prefix-prompting.
• 摘要：生成人工智能 (GenAI) 系统正越来越多地应用于工业和研究环境的各个领域。开发人员和最终用户通过使用提示或提示工程与这些系统进行交互。虽然提示是一个广泛且研究程度很高的概念，但由于该领域的新兴性，存在着术语上的矛盾以及对提示构成的本体论理解不足。本文通过汇编提示技术的分类法并分析其使用情况，建立了对提示的结构化理解。我们提供了包含 33 个词汇术语的综合词汇表、58 种纯文本提示技术的分类法和 40 种其他模态技术。我们还对自然语言前缀提示的全部文献进行了元分析。

Title: Reinterpreting 'the Company a Word Keeps': Towards Explainable and Ontologically Grounded Language Models

• Authors: Walid S. Saba
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06610
• Pdf URL: https://arxiv.org/pdf/2406.06610
• Copy Paste: [[2406.06610]] Reinterpreting 'the Company a Word Keeps': Towards Explainable and Ontologically Grounded Language Models(https://arxiv.org/abs/2406.06610)
• Keywords: language model, llm
• Abstract: We argue that the relative success of large language models (LLMs) is not a reflection on the symbolic vs. subsymbolic debate but a reflection on employing a successful bottom-up strategy of a reverse engineering of language at scale. However, and due to their subsymbolic nature whatever knowledge these systems acquire about language will always be buried in millions of weights none of which is meaningful on its own, rendering such systems utterly unexplainable. Furthermore, and due to their stochastic nature, LLMs will often fail in making the correct inferences in various linguistic contexts that require reasoning in intensional, temporal, or modal contexts. To remedy these shortcomings we suggest employing the same successful bottom-up strategy employed in LLMs but in a symbolic setting, resulting in explainable, language-agnostic, and ontologically grounded language models.
• 摘要：我们认为，大型语言模型 (LLM) 的相对成功并不是对符号与亚符号之争的反映，而是对采用成功的自下而上策略进行大规模语言逆向工程的反映。然而，由于它们的亚符号性质，这些系统获得的任何有关语言的知识都将始终埋藏在数百万个权重中，而这些权重本身都没有意义，这使得此类系统完全无法解释。此外，由于其随机性，LLM 通常无法在需要在内涵、时间或模态语境中进行推理的各种语言语境中做出正确的推理。为了弥补这些缺点，我们建议在符号环境中采用与 LLM 相同的成功自下而上策略，从而产生可解释、与语言无关且基于本体论的语言模型。

Title: GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents

• Authors: Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Arjun Yadav
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06613
• Pdf URL: https://arxiv.org/pdf/2406.06613
• Copy Paste: [[2406.06613]] GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents(https://arxiv.org/abs/2406.06613)
• Keywords: language model, gpt, llm, prompt, chain-of-thought, agent
• Abstract: Large language models have demonstrated remarkable few-shot performance on many natural language understanding tasks. Despite several demonstrations of using large language models in complex, strategic scenarios, there lacks a comprehensive framework for evaluating agents' performance across various types of reasoning found in games. To address this gap, we introduce GameBench, a cross-domain benchmark for evaluating strategic reasoning abilities of LLM agents. We focus on 9 different game environments, where each covers at least one axis of key reasoning skill identified in strategy games, and select games for which strategy explanations are unlikely to form a significant portion of models' pretraining corpuses. Our evaluations use GPT-3 and GPT-4 in their base form along with two scaffolding frameworks designed to enhance strategic reasoning ability: Chain-of-Thought (CoT) prompting and Reasoning Via Planning (RAP). Our results show that none of the tested models match human performance, and at worse GPT-4 performs worse than random action. CoT and RAP both improve scores but not comparable to human levels.
• 摘要：大型语言模型在许多自然语言理解任务中都表现出了出色的小样本性能。尽管有几次演示在复杂的战略场景中使用大型语言模型，但缺乏一个全面的框架来评估代理在游戏中各种推理类型中的表现。为了解决这一差距，我们引入了 GameBench，这是一个跨领域基准，用于评估 LLM 代理的战略推理能力。我们专注于 9 种不同的游戏环境，其中每种环境都涵盖了战略游戏中确定的至少一个关键推理技能轴，并选择了策略解释不太可能构成模型预训练语料库重要部分的游戏。我们的评估使用了 GPT-3 和 GPT-4 的基本形式以及两个旨在增强战略推理能力的支架框架：思维链 (CoT) 提示和通过规划进行推理 (RAP)。我们的结果表明，没有一个测试模型可以与人类的表现相匹配，最差的情况下 GPT-4 的表现还不如随机动作。CoT 和 RAP 都提高了分数，但无法与人类水平相提并论。

Title: Language Guided Skill Discovery

• Authors: Seungeun Rho, Laura Smith, Tianyu Li, Sergey Levine, Xue Bin Peng, Sehoon Ha
• Subjects: cs.CL, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2406.06615
• Pdf URL: https://arxiv.org/pdf/2406.06615
• Copy Paste: [[2406.06615]] Language Guided Skill Discovery(https://arxiv.org/abs/2406.06615)
• Keywords: language model, llm, prompt, agent
• Abstract: Skill discovery methods enable agents to learn diverse emergent behaviors without explicit rewards. To make learned skills useful for unknown downstream tasks, obtaining a semantically diverse repertoire of skills is essential. While some approaches introduce a discriminator to distinguish skills and others aim to increase state coverage, no existing work directly addresses the "semantic diversity" of skills. We hypothesize that leveraging the semantic knowledge of large language models (LLMs) can lead us to improve semantic diversity of resulting behaviors. In this sense, we introduce Language Guided Skill Discovery (LGSD), a skill discovery framework that aims to directly maximize the semantic diversity between skills. LGSD takes user prompts as input and outputs a set of semantically distinctive skills. The prompts serve as a means to constrain the search space into a semantically desired subspace, and the generated LLM outputs guide the agent to visit semantically diverse states within the subspace. We demonstrate that LGSD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt. Furthermore, we show that language guidance aids in discovering more diverse skills compared to five existing skill discovery methods in robot-arm manipulation environments. Lastly, LGSD provides a simple way of utilizing learned skills via natural language.
• 摘要：技能发现方法使代理能够在没有明确奖励的情况下学习各种新兴行为。要使学习到的技能对未知的下游任务有用，获得语义多样化的技能库至关重要。虽然一些方法引入了鉴别器来区分技能，而另一些方法旨在增加状态覆盖率，但现有的研究并没有直接解决技能的“语义多样性”。我们假设利用大型语言模型 (LLM) 的语义知识可以让我们提高结果行为的语义多样性。从这个意义上讲，我们引入了语言引导技能发现 (LGSD)，这是一个旨在直接最大化技能之间语义多样性的技能发现框架。LGSD 将用户提示作为输入并输出一组语义上不同的技能。提示是一种将搜索空间限制在语义上所需的子空间中的手段，生成的 LLM 输出引导代理访问子空间内语义上不同的状态。我们证明，LGSD 使有腿机器人能够通过简单地更改提示来访问平面上不同的用户预期区域。此外，我们表明，与机械臂操作环境中现有的五种技能发现方法相比，语言指导有助于发现更多样化的技能。最后，LGSD 提供了一种通过自然语言利用所学技能的简单方法。

Title: Transforming Dental Diagnostics with Artificial Intelligence: Advanced Integration of ChatGPT and Large Language Models for Patient Care

• Authors: Masoumeh Farhadi Nia, Mohsen Ahmadi, Elyas Irankhah
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06616
• Pdf URL: https://arxiv.org/pdf/2406.06616
• Copy Paste: [[2406.06616]] Transforming Dental Diagnostics with Artificial Intelligence: Advanced Integration of ChatGPT and Large Language Models for Patient Care(https://arxiv.org/abs/2406.06616)
• Keywords: language model, gpt, llm, chat
• Abstract: Artificial intelligence has dramatically reshaped our interaction with digital technologies, ushering in an era where advancements in AI algorithms and Large Language Models (LLMs) have natural language processing (NLP) systems like ChatGPT. This study delves into the impact of cutting-edge LLMs, notably OpenAI's ChatGPT, on medical diagnostics, with a keen focus on the dental sector. Leveraging publicly accessible datasets, these models augment the diagnostic capabilities of medical professionals, streamline communication between patients and healthcare providers, and enhance the efficiency of clinical procedures. The advent of ChatGPT-4 is poised to make substantial inroads into dental practices, especially in the realm of oral surgery. This paper sheds light on the current landscape and explores potential future research directions in the burgeoning field of LLMs, offering valuable insights for both practitioners and developers. Furthermore, it critically assesses the broad implications and challenges within various sectors, including academia and healthcare, thus mapping out an overview of AI's role in transforming dental diagnostics for enhanced patient care.
• 摘要：人工智能极大地重塑了我们与数字技术的互动，开启了一个人工智能算法和大型语言模型 (LLM) 的进步拥有 ChatGPT 等自然语言处理 (NLP) 系统的时代。本研究深入探讨了尖端 LLM（尤其是 OpenAI 的 ChatGPT）对医疗诊断的影响，重点关注牙科领域。利用可公开访问的数据集，这些模型增强了医疗专业人员的诊断能力，简化了患者与医疗保健提供者之间的沟通，并提高了临床程序的效率。ChatGPT-4 的出现有望在牙科实践中取得重大进展，尤其是在口腔外科领域。本文阐明了当前的形势，并探讨了 LLM 这一新兴领域未来的潜在研究方向，为从业者和开发者提供了宝贵的见解。此外，它批判性地评估了包括学术界和医疗保健在内的各个领域的广泛影响和挑战，从而概述了人工智能在转变牙科诊断以增强患者护理方面的作用。

Title: LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering

• Authors: Harry Li, Gabriel Appleby, Ashley Suh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06621
• Pdf URL: https://arxiv.org/pdf/2406.06621
• Copy Paste: [[2406.06621]] LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering(https://arxiv.org/abs/2406.06621)
• Keywords: language model, llm
• Abstract: We present LinkQ, a system that leverages a large language model (LLM) to facilitate knowledge graph (KG) query construction through natural language question-answering. Traditional approaches often require detailed knowledge of complex graph querying languages, limiting the ability for users -- even experts -- to acquire valuable insights from KG data. LinkQ simplifies this process by first interpreting a user's question, then converting it into a well-formed KG query. By using the LLM to construct a query instead of directly answering the user's question, LinkQ guards against the LLM hallucinating or generating false, erroneous information. By integrating an LLM into LinkQ, users are able to conduct both exploratory and confirmatory data analysis, with the LLM helping to iteratively refine open-ended questions into precise ones. To demonstrate the efficacy of LinkQ, we conducted a qualitative study with five KG practitioners and distill their feedback. Our results indicate that practitioners find LinkQ effective for KG question-answering, and desire future LLM-assisted systems for the exploratory analysis of graph databases.
• 摘要：我们推出了 LinkQ，这是一个利用大型语言模型 (LLM) 通过自然语言问答来促进知识图谱 (KG) 查询构建的系统。传统方法通常需要详细了解复杂的图形查询语言，这限制了用户（甚至是专家）从 KG 数据中获取有价值见解的能力。LinkQ 首先解释用户的问题，然后将其转换为格式良好的 KG 查询，从而简化了此过程。通过使用 LLM 构建查询而不是直接回答用户的问题，LinkQ 可以防止 LLM 产生幻觉或生成虚假、错误的信息。通过将 LLM 集成到 LinkQ 中，用户可以进行探索性和确认性数据分析，LLM 有助于迭代地将开放式问题细化为精确的问题。为了证明 LinkQ 的有效性，我们对五位 KG 从业者进行了一项定性研究并提炼了他们的反馈。我们的结果表明，从业者认为 LinkQ 对 KG 问答很有效，并希望未来有 LLM 辅助系统来进行图形数据库的探索性分析。

Title: Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs

• Authors: Fan Liu, Zhao Xu, Hao Liu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2406.06622
• Pdf URL: https://arxiv.org/pdf/2406.06622
• Copy Paste: [[2406.06622]] Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs(https://arxiv.org/abs/2406.06622)
• Keywords: language model, llm, prompt
• Abstract: Although safely enhanced Large Language Models (LLMs) have achieved remarkable success in tackling various complex tasks in a zero-shot manner, they remain susceptible to jailbreak attacks, particularly the unknown jailbreak attack. To enhance LLMs' generalized defense capabilities, we propose a two-stage adversarial tuning framework, which generates adversarial prompts to explore worst-case scenarios by optimizing datasets containing pairs of adversarial prompts and their safe responses. In the first stage, we introduce the hierarchical meta-universal adversarial prompt learning to efficiently and effectively generate token-level adversarial prompts. In the second stage, we propose the automatic adversarial prompt learning to iteratively refine semantic-level adversarial prompts, further enhancing LLM's defense capabilities. We conducted comprehensive experiments on three widely used jailbreak datasets, comparing our framework with six defense baselines under five representative attack scenarios. The results underscore the superiority of our proposed methods. Furthermore, our adversarial tuning framework exhibits empirical generalizability across various attack strategies and target LLMs, highlighting its potential as a transferable defense mechanism.
• 摘要：尽管安全增强的大型语言模型 (LLM) 在以零样本方式处理各种复杂任务方面取得了显著成功，但它们仍然容易受到越狱攻击，尤其是未知越狱攻击。为了增强 LLM 的广义防御能力，我们提出了一个两阶段对抗性调整框架，该框架通过优化包含对抗性提示及其安全响应对的数据集来生成对抗性提示以探索最坏情况。在第一阶段，我们引入了分层元通用对抗性提示学习，以高效、有效地生成 token 级对抗性提示。在第二阶段，我们提出了自动对抗性提示学习来迭代细化语义级对抗性提示，进一步增强 LLM 的防御能力。我们在三个广泛使用的越狱数据集上进行了全面的实验，将我们的框架与五种代表性攻击场景下的六条防御基线进行了比较。结果强调了我们提出的方法的优越性。此外，我们的对抗性调整框架在各种攻击策略和目标 LLM 中表现出经验的普遍性，凸显了其作为可转移防御机制的潜力。

Title: LLM Questionnaire Completion for Automatic Psychiatric Assessment

• Authors: Gony Rosenman, Lior Wolf, Talma Hendler
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06636
• Pdf URL: https://arxiv.org/pdf/2406.06636
• Copy Paste: [[2406.06636]] LLM Questionnaire Completion for Automatic Psychiatric Assessment(https://arxiv.org/abs/2406.06636)
• Keywords: language model, llm, prompt
• Abstract: We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The LLM is prompted to answer these questionnaires by impersonating the interviewee. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C), using a Random Forest regressor. Our approach is shown to enhance diagnostic accuracy compared to multiple baselines. It thus establishes a novel framework for interpreting unstructured psychological interviews, bridging the gap between narrative-driven and data-driven approaches for mental health assessment.
• 摘要：我们采用大型语言模型 (LLM) 将非结构化心理访谈转换为涵盖各种精神病学和人格领域的结构化问卷。LLM 被要求通过模仿受访者来回答这些问卷。获得的答案被编码为特征，用于使用随机森林回归器预测抑郁症 (PHQ-8) 和 PTSD (PCL-C) 的标准化精神病学测量。与多个基线相比，我们的方法被证明可以提高诊断准确性。因此，它建立了一个解释非结构化心理访谈的新框架，弥合了心理健康评估的叙述驱动和数据驱动方法之间的差距。

Title: Harnessing AI for efficient analysis of complex policy documents: a case study of Executive Order 14110

• Authors: Mark A. Kramer, Allen Leavens, Alexander Scarlat
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2406.06657
• Pdf URL: https://arxiv.org/pdf/2406.06657
• Copy Paste: [[2406.06657]] Harnessing AI for efficient analysis of complex policy documents: a case study of Executive Order 14110(https://arxiv.org/abs/2406.06657)
• Keywords: language model, llm
• Abstract: Policy documents, such as legislation, regulations, and executive orders, are crucial in shaping society. However, their length and complexity make interpretation and application challenging and time-consuming. Artificial intelligence (AI), particularly large language models (LLMs), has the potential to automate the process of analyzing these documents, improving accuracy and efficiency. This study aims to evaluate the potential of AI in streamlining policy analysis and to identify the strengths and limitations of current AI approaches. The research focuses on question answering and tasks involving content extraction from policy documents. A case study was conducted using Executive Order 14110 on "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence" as a test case. Four commercial AI systems were used to analyze the document and answer a set of representative policy questions. The performance of the AI systems was compared to manual analysis conducted by human experts. The study found that two AI systems, Gemini 1.5 Pro and Claude 3 Opus, demonstrated significant potential for supporting policy analysis, providing accurate and reliable information extraction from complex documents. They performed comparably to human analysts but with significantly higher efficiency. However, achieving reproducibility remains a challenge, necessitating further research and development.
• 摘要：政策文件（例如立法、法规和行政命令）对于塑造社会至关重要。然而，这些文件的篇幅和复杂性使得解释和应用具有挑战性且耗时。人工智能（AI），尤其是大型语言模型（LLM），有可能使分析这些文件的过程自动化，从而提高准确性和效率。本研究旨在评估人工智能在简化政策分析方面的潜力，并确定当前人工智能方法的优势和局限性。该研究侧重于问答和涉及从政策文件中提取内容的任务。案例研究以关于“安全、可靠和值得信赖地开发和使用人工智能”的行政命令 14110 为测试案例。四个商业人工智能系统被用于分析文档并回答一组代表性政策问题。将人工智能系统的性能与人类专家进行的手动分析进行了比较。研究发现，两个人工智能系统 Gemini 1.5 Pro 和 Claude 3 Opus 表现出支持政策分析的巨大潜力，可以从复杂文档中提取准确可靠的信息。它们的表现与人类分析师相当，但效率明显更高。然而，实现可重复性仍然是一个挑战，需要进一步的研究和开发。

Title: In-Context Learning and Fine-Tuning GPT for Argument Mining

• Authors: Jérémie Cabessa, Hugo Hernault, Umer Mushtaq
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06699
• Pdf URL: https://arxiv.org/pdf/2406.06699
• Copy Paste: [[2406.06699]] In-Context Learning and Fine-Tuning GPT for Argument Mining(https://arxiv.org/abs/2406.06699)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have become ubiquitous in NLP and deep learning. In-Context Learning (ICL) has been suggested as a bridging paradigm between the training-free and fine-tuning LLMs settings. In ICL, an LLM is conditioned to solve tasks by means of a few solved demonstration examples included as prompt. Argument Mining (AM) aims to extract the complex argumentative structure of a text, and Argument Type Classification (ATC) is an essential sub-task of AM. We introduce an ICL strategy for ATC combining kNN-based examples selection and majority vote ensembling. In the training-free ICL setting, we show that GPT-4 is able to leverage relevant information from only a few demonstration examples and achieve very competitive classification accuracy on ATC. We further set up a fine-tuning strategy incorporating well-crafted structural features given directly in textual form. In this setting, GPT-3.5 achieves state-of-the-art performance on ATC. Overall, these results emphasize the emergent ability of LLMs to grasp global discursive flow in raw text in both off-the-shelf and fine-tuned setups.
• 摘要：大型语言模型 (LLM) 在 NLP 和深度学习中已经无处不在。上下文学习 (ICL) 被认为是无训练和微调 LLM 设置之间的桥梁范例。在 ICL 中，LLM 被训练为通过提示中包含的几个已解决的演示示例来解决任务。论证挖掘 (AM) 旨在提取文本的复杂论证结构，论证类型分类 (ATC) 是 AM 的重要子任务。我们引入了一种结合基于 kNN 的示例选择和多数投票集成的 ATC ICL 策略。在无训练 ICL 设置中，我们表明 GPT-4 能够利用仅来自少数演示示例的相关信息，并在 ATC 上实现非常有竞争力的分类准确率。我们进一步建立了一种微调策略，结合了以文本形式直接给出的精心设计的结构特征。在这种设置下，GPT-3.5 在 ATC 上实现了最先进的性能。总体而言，这些结果强调了 LLM 在现成和微调设置中掌握原始文本中整体话语流的新兴能力。

Title: Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing

• Authors: Enshuo Hsu, Kirk Roberts
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2406.06723
• Pdf URL: https://arxiv.org/pdf/2406.06723
• Copy Paste: [[2406.06723]] Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing(https://arxiv.org/abs/2406.06723)
• Keywords: language model, llm, prompt
• Abstract: The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7% to 47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.
• 摘要：基于深度学习的自然语言处理系统的性能取决于大量标记训练数据，而在临床领域，这些数据并不容易获得或负担得起。弱监督和上下文学习为这一问题提供了部分解决方案，尤其是使用大型语言模型 (LLM) 时，但它们的性能仍然落后于具有中等量黄金标准数据的传统监督方法。特别是，使用 LLM 进行推理需要大量计算。我们提出了一种利用微调 LLM 和弱监督的方法，几乎​​不需要任何领域知识，但仍能实现始终如一的主导性能。使用基于提示的方法，LLM 用于生成弱标记数据以训练下游 BERT 模型。然后在少量黄金标准数据上进一步微调弱监督模型。我们使用 Llama2 在三个不同的 n2c2 数据集上评估了这种方法。在不超过 10 条金标准注释的情况下，我们最终的 BERT 模型在经过微调的 Llama2-13B 弱监督下，F1 分数始终比开箱即用的 PubMedBERT 高出 4.7% 至 47.9%。在只有 50 条金标准注释的情况下，我们的模型实现了与完全微调系统接近的性能。

Title: Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrieval

• Authors: Ravisri Valluri, Akash Kumar Mohankumar, Kushal Dave, Amit Singh, Jian Jiao, Manik Varma, Gaurav Sinha
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06739
• Pdf URL: https://arxiv.org/pdf/2406.06739
• Copy Paste: [[2406.06739]] Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrieval(https://arxiv.org/abs/2406.06739)
• Keywords: language model
• Abstract: Generative Retrieval introduces a new approach to Information Retrieval by reframing it as a constrained generation task, leveraging recent advancements in Autoregressive (AR) language models. However, AR-based Generative Retrieval methods suffer from high inference latency and cost compared to traditional dense retrieval techniques, limiting their practical applicability. This paper investigates fully Non-autoregressive (NAR) language models as a more efficient alternative for generative retrieval. While standard NAR models alleviate latency and cost concerns, they exhibit a significant drop in retrieval performance (compared to AR models) due to their inability to capture dependencies between target tokens. To address this, we question the conventional choice of limiting the target token space to solely words or sub-words. We propose PIXAR, a novel approach that expands the target vocabulary of NAR models to include multi-word entities and common phrases (up to 5 million tokens), thereby reducing token dependencies. PIXAR employs inference optimization strategies to maintain low inference latency despite the significantly larger vocabulary. Our results demonstrate that PIXAR achieves a relative improvement of 31.0% in MRR@10 on MS MARCO and 23.2% in Hits@5 on Natural Questions compared to standard NAR models with similar latency and cost. Furthermore, online A/B experiments on a large commercial search engine show that PIXAR increases ad clicks by 5.08% and revenue by 4.02%.
• 摘要：生成检索通过将其重新定义为受约束的生成任务，利用自回归 (AR) 语言模型的最新进展，为信息检索引入了一种新方法。然而，与传统的密集检索技术相比，基于 AR 的生成检索方法存在推理延迟和成本高的问题，限制了它们的实际适用性。本文研究了完全非自回归 (NAR) 语言模型作为生成检索的更有效替代方案。虽然标准 NAR 模型可以缓解延迟和成本问题，但由于无法捕获目标标记之间的依赖关系，它们的检索性能显著下降（与 AR 模型相比）。为了解决这个问题，我们质疑将目标标记空间限制为单词或子单词的传统选择。我们提出了 PIXAR，这是一种新方法，它扩展了 NAR 模型的目标词汇表以包括多词实体和常用短语（最多 500 万个标记），从而减少了标记依赖关系。尽管词汇量大得多，但 PIXAR 采用推理优化策略来保持较低的推理延迟。我们的结果表明，与具有相似延迟和成本的标准 NAR 模型相比，PIXAR 在 MS MARCO 上的 MRR@10 相对提高了 31.0%，在 Natural Questions 上的 Hits@5 相对提高了 23.2%。此外，在大型商业搜索引擎上进行的在线 A/B 实验表明，PIXAR 的广告点击量提高了 5.08%，收入提高了 4.02%。

Title: Evaluating Zero-Shot Long-Context LLM Compression

• Authors: Chenyu Wang, Yihan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06773
• Pdf URL: https://arxiv.org/pdf/2406.06773
• Copy Paste: [[2406.06773]] Evaluating Zero-Shot Long-Context LLM Compression(https://arxiv.org/abs/2406.06773)
• Keywords: language model, llm
• Abstract: This study evaluates the effectiveness of zero-shot compression techniques on large language models (LLMs) under long-context. We identify the tendency for computational errors to increase under long-context when employing certain compression methods. We propose a hypothesis to explain the varied behavior of different LLM compression techniques and explore remedies to mitigate the performance decline observed in some techniques under long-context. This is a course report for COS 598D Machine Learning and Systems by Prof. Kai Li at Princeton University. Due to limited computational resources, our experiments were conducted only on LLaMA-2-7B-32K.
• 摘要：本研究评估了零样本压缩技术在长上下文中对大型语言模型 (LLM) 的有效性。我们发现，在使用某些压缩方法时，计算错误在长上下文中趋于增加。我们提出了一个假设来解释不同 LLM 压缩技术的不同行为，并探索了缓解某些技术在长上下文中观察到的性能下降的补救措施。这是普林斯顿大学 Kai Li 教授的 COS 598D 机器学习与系统课程报告。由于计算资源有限，我们的实验仅在 LLaMA-2-7B-32K 上进行。

Title: AGB-DE: A Corpus for the Automated Legal Assessment of Clauses in German Consumer Contracts

• Authors: Daniel Braun, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.06809
• Pdf URL: https://arxiv.org/pdf/2406.06809
• Copy Paste: [[2406.06809]] AGB-DE: A Corpus for the Automated Legal Assessment of Clauses in German Consumer Contracts(https://arxiv.org/abs/2406.06809)
• Keywords: language model, gpt
• Abstract: Legal tasks and datasets are often used as benchmarks for the capabilities of language models. However, openly available annotated datasets are rare. In this paper, we introduce AGB-DE, a corpus of 3,764 clauses from German consumer contracts that have been annotated and legally assessed by legal experts. Together with the data, we present a first baseline for the task of detecting potentially void clauses, comparing the performance of an SVM baseline with three fine-tuned open language models and the performance of GPT-3.5. Our results show the challenging nature of the task, with no approach exceeding an F1-score of 0.54. While the fine-tuned models often performed better with regard to precision, GPT-3.5 outperformed the other approaches with regard to recall. An analysis of the errors indicates that one of the main challenges could be the correct interpretation of complex clauses, rather than the decision boundaries of what is permissible and what is not.
• 摘要：法律任务和数据集通常被用作语言模型能力的基准。然而，公开可用的带注释数据集很少。在本文中，我们介绍了 AGB-DE，这是一个包含 3,764 条德国消费者合同条款的语料库，这些条款已由法律专家注释并进行了法律评估。结合数据，我们为检测潜在无效条款的任务提供了第一个基线，比较了 SVM 基线与三个微调开放语言模型的性能以及 GPT-3.5 的性能。我们的结果表明这项任务的挑战性，没有一种方法的 F1 分数超过 0.54。虽然微调模型在准确率方面通常表现更好，但 GPT-3.5 在召回率方面优于其他方法。对错误的分析表明，主要挑战之一可能是对复杂条款的正确解释，而不是什么是允许的和什么是不允许的决策边界。

Title: Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles

• Authors: Julia Kruk, Michela Marchini, Rijul Ragu, Caleb Ziems, David Muchlinski, Diyi Yang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.06840
• Pdf URL: https://arxiv.org/pdf/2406.06840
• Copy Paste: [[2406.06840]] Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles(https://arxiv.org/abs/2406.06840)
• Keywords: language model, llm
• Abstract: A dog whistle is a form of coded communication that carries a secondary meaning to specific audiences and is often weaponized for racial and socioeconomic discrimination. Dog whistling historically originated from United States politics, but in recent years has taken root in social media as a means of evading hate speech detection systems and maintaining plausible deniability. In this paper, we present an approach for word-sense disambiguation of dog whistles from standard speech using Large Language Models (LLMs), and leverage this technique to create a dataset of 16,550 high-confidence coded examples of dog whistles used in formal and informal communication. Silent Signals is the largest dataset of disambiguated dog whistle usage, created for applications in hate speech detection, neology, and political science. The dataset can be found at this https URL.
• 摘要：狗哨是一种编码交流形式，对特定受众具有第二含义，经常被用来进行种族和社会经济歧视。狗哨起源于美国政治，但近年来已在社交媒体中扎根，成为逃避仇恨言论检测系统和保持合理否认的一种手段。在本文中，我们提出了一种使用大型语言模型 (LLM) 从标准语音中消除狗哨词义歧义的方法，并利用该技术创建了一个包含 16,550 个高置信度编码狗哨示例的数据集，这些示例用于正式和非正式交流。Silent Signals 是消除歧义狗哨使用情况的最大数据集，专为仇恨言论检测、新词和政治学应用而创建。该数据集可在此 https URL 中找到。

Title: Modeling language contact with the Iterated Learning Model

• Authors: Seth Bullock, Conor Houghton
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.06878
• Pdf URL: https://arxiv.org/pdf/2406.06878
• Copy Paste: [[2406.06878]] Modeling language contact with the Iterated Learning Model(https://arxiv.org/abs/2406.06878)
• Keywords: agent
• Abstract: Contact between languages has the potential to transmit vocabulary and other language features; however, this does not always happen. Here, an iterated learning model is used to examine, in a simple way, the resistance of languages to change during language contact. Iterated learning models are agent-based models of language change, they demonstrate that languages that are expressive and compositional arise spontaneously as a consequence of a language transmission bottleneck. A recently introduced type of iterated learning model, the Semi-Supervised ILM is used to simulate language contact. These simulations do not include many of the complex factors involved in language contact and do not model a population of speakers; nonetheless the model demonstrates that the dynamics which lead languages in the model to spontaneously become expressive and compositional, also cause a language to maintain its core traits even after mixing with another language.
• 摘要：语言之间的接触有可能传递词汇和其他语言特征；然而，这种情况并不总是发生。这里，迭代学习模型被用来简单地检查语言在语言接触过程中对变化的抵抗力。迭代学习模型是基于代理的语言变化模型，它们表明，富有表现力和组合性的语言是语言传输瓶颈的结果。最近引入的一种迭代学习模型，半监督 ILM 用于模拟语言接触。这些模拟不包括语言接触中涉及的许多复杂因素，也不模拟说话者群体；尽管如此，该模型表明，导致模型中的语言自发变得富有表现力和组合性的动态，也导致一种语言在与另一种语言混合后仍保持其核心特征。

Title: PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models

• Authors: Dylan Zhang, Shizhe Diao, Xueyan Zou, Hao Peng
• Subjects: cs.CL, cs.AI, cs.LG, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2406.06887
• Pdf URL: https://arxiv.org/pdf/2406.06887
• Copy Paste: [[2406.06887]] PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models(https://arxiv.org/abs/2406.06887)
• Keywords: language model, chat
• Abstract: Instruction-finetuned code language models (LMs) have shown promise in various programming tasks. They are trained, using a language modeling objective, on natural language instructions and gold code snippet pairs. Recent evidence suggests that these models, never exposed to incorrect solutions during training, often struggle to distinguish between correct and incorrect solutions. This observation raises our inquiry: Can preference learning, which trains models to prefer correct solutions over incorrect ones, help push the boundaries of code LMs even further? We propose PLUM, a novel \textbf{p}reference \textbf{l}earning framework a\textbf{u}gmented with test cases tailored for code L\textbf{M}s.PLUM aims to investigate the key success factors and potential benefits of preference learning in code LMs, which remain elusive despite its success in aligning LMs with human values. PLUM consists of three stages: (1) Generating test cases for natural language instructions, (2) sampling candidate solutions from the policy and evaluating them against the test cases to create a preference dataset, which is then used to (3) train the policy with a preference learning algorithm. Experiments demonstrate that PLUM substantially improves the performance of existing code LMs on established code generation benchmarks such as HumanEval (+) and MBPP (+), even for the state-of-the-art open-source language model CodeQwen-1.5-7B-Chat. PLUM complements the supervised fine-tuning (SFT) stage, demonstrating synergistic effects.
• 摘要：指令微调的代码语言模型 (LM) 在各种编程任务中都表现出良好的前景。它们使用语言建模目标，在自然语言指令和黄金代码片段对上进行训练。最近的证据表明，这些模型在训练期间从未接触过错误的解决方案，因此经常难以区分正确和错误的解决方案。这一观察引发了我们的疑问：偏好学习（训练模型偏好正确解决方案而不是错误解决方案）是否有助于进一步突破代码语言模型的界限？我们提出了 PLUM，这是一种新颖的 \textbf{p} 参考 \textbf{l} 学习框架，并添加了针对代码语言模型量身定制的测试用例。PLUM 旨在研究偏好学习在代码语言模型中的关键成功因素和潜在好处，尽管它成功地将语言模型与人类价值观相结合，但这些因素仍然难以捉摸。 PLUM 包含三个阶段：(1) 生成自然语言指令的测试用例，(2) 从策略中抽取候选解决方案并根据测试用例对其进行评估以创建偏好数据集，然后使用该数据集 (3) 使用偏好学习算法训练策略。实验表明，PLUM 大大提高了现有代码语言模型在已建立的代码生成基准（例如 HumanEval (+) 和 MBPP (+)）上的性能，甚至对于最先进的开源语言模型 CodeQwen-1.5-7B-Chat 也是如此。PLUM 补充了监督微调 (SFT) 阶段，表现出协同效应。

Title: Agent-SiMT: Agent-assisted Simultaneous Machine Translation with Large Language Models

• Authors: Shoutao Guo, Shaolei Zhang, Zhengrui Ma, Min Zhang, Yang Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.06910
• Pdf URL: https://arxiv.org/pdf/2406.06910
• Copy Paste: [[2406.06910]] Agent-SiMT: Agent-assisted Simultaneous Machine Translation with Large Language Models(https://arxiv.org/abs/2406.06910)
• Keywords: language model, llm, agent
• Abstract: Simultaneous Machine Translation (SiMT) generates target translations while reading the source sentence. It relies on a policy to determine the optimal timing for reading sentences and generating translations. Existing SiMT methods generally adopt the traditional Transformer architecture, which concurrently determines the policy and generates translations. While they excel at determining policies, their translation performance is suboptimal. Conversely, Large Language Models (LLMs), trained on extensive corpora, possess superior generation capabilities, but it is difficult for them to acquire translation policy through the training methods of SiMT. Therefore, we introduce Agent-SiMT, a framework combining the strengths of LLMs and traditional SiMT methods. Agent-SiMT contains the policy-decision agent and the translation agent. The policy-decision agent is managed by a SiMT model, which determines the translation policy using partial source sentence and translation. The translation agent, leveraging an LLM, generates translation based on the partial source sentence. The two agents collaborate to accomplish SiMT. Experiments demonstrate that Agent-SiMT attains state-of-the-art performance.
• 摘要：同步机器翻译 (SiMT) 在阅读源句子的同时生成目标翻译。它依靠策略来确定阅读句子和生成翻译的最佳时机。现有的 SiMT 方法通常采用传统的 Transformer 架构，该架构同时确定策略和生成翻译。虽然它们在确定策略方面表现出色，但翻译性能并不理想。相反，在大量语料上训练的大型语言模型 (LLM) 具有出色的生成能力，但它们很难通过 SiMT 的训练方法获得翻译策略。因此，我们引入了 Agent-SiMT，这是一个结合了 LLM 和传统 SiMT 方法优势的框架。Agent-SiMT 包含策略决策代理和翻译代理。策略决策代理由 SiMT 模型管理，该模型使用部分源句子和翻译来确定翻译策略。翻译代理利用 LLM 根据部分源句子生成翻译。这两个代理协作完成 SiMT。实验表明，Agent-SiMT 达到了最佳性能。

Title: A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagation

• Authors: Bairu Hou, Yang Zhang, Jacob Andreas, Shiyu Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.06950
• Pdf URL: https://arxiv.org/pdf/2406.06950
• Copy Paste: [[2406.06950]] A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagation(https://arxiv.org/abs/2406.06950)
• Keywords: llm, hallucination
• Abstract: This paper focuses on the task of hallucination detection, which aims to determine the truthfulness of LLM-generated statements. To address this problem, a popular class of methods utilize the LLM's self-consistencies in its beliefs in a set of logically related augmented statements generated by the LLM, which does not require external knowledge databases and can work with both white-box and black-box LLMs. However, in many existing approaches, the augmented statements tend to be very monotone and unstructured, which makes it difficult to integrate meaningful information from the LLM beliefs in these statements. Also, many methods work with the binarized version of the LLM's belief, instead of the continuous version, which significantly loses information. To overcome these limitations, in this paper, we propose Belief Tree Propagation (BTProp), a probabilistic framework for LLM hallucination detection. BTProp introduces a belief tree of logically related statements by recursively decomposing a parent statement into child statements with three decomposition strategies, and builds a hidden Markov tree model to integrate the LLM's belief scores in these statements in a principled way. Experiment results show that our method improves baselines by 3%-9% (evaluated by AUROC and AUC-PR) on multiple hallucination detection benchmarks. Code is available at this https URL.
• 摘要：本文重点关注幻觉检测任务，旨在确定 LLM 生成的语句的真实性。为了解决这个问题，一类流行的方法利用 LLM 信念的自洽性，将其应用于由 LLM 生成的一组逻辑相关的增强语句中，这种方法不需要外部知识数据库，并且可以与白盒和黑盒 LLM 配合使用。然而，在许多现有方法中，增强语句往往非常单调和非结构化，这使得很难将来自 LLM 信念的有意义的信息整合到这些语句中。此外，许多方法使用的是 LLM 信念的二值化版本，而不是连续版本，这会严重丢失信息。为了克服这些限制，在本文中，我们提出了信念树传播 (BTProp)，这是一种用于 LLM 幻觉检测的概率框架。 BTProp 通过使用三种分解策略将父语句递归分解为子语句，引入了逻辑相关语句的信念树，并构建了隐马尔可夫树模型，以原则性的方式整合了 LLM 对这些语句的信念得分。实验结果表明，我们的方法在多个幻觉检测基准上将基线提高了 3%-9%（通过 AUROC 和 AUC-PR 评估）。代码可从此 https URL 获取。

Title: Evolving Subnetwork Training for Large Language Models

• Authors: Hanqi Li, Lu Chen, Da Ma, Zijian Wu, Su Zhu, Kai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.06962
• Pdf URL: https://arxiv.org/pdf/2406.06962
• Copy Paste: [[2406.06962]] Evolving Subnetwork Training for Large Language Models(https://arxiv.org/abs/2406.06962)
• Keywords: language model, gpt
• Abstract: Large language models have ushered in a new era of artificial intelligence research. However, their substantial training costs hinder further development and widespread adoption. In this paper, inspired by the redundancy in the parameters of large language models, we propose a novel training paradigm: Evolving Subnetwork Training (EST). EST samples subnetworks from the layers of the large language model and from commonly used modules within each layer, Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP). By gradually increasing the size of the subnetworks during the training process, EST can save the cost of training. We apply EST to train GPT2 model and TinyLlama model, resulting in 26.7\% FLOPs saving for GPT2 and 25.0\% for TinyLlama without an increase in loss on the pre-training dataset. Moreover, EST leads to performance improvements in downstream tasks, indicating that it benefits generalization. Additionally, we provide intuitive theoretical studies based on training dynamics and Dropout theory to ensure the feasibility of EST. Our code is available at this https URL.
• 摘要：大型语言模型开启了人工智能研究的新纪元。然而，它们高昂的训练成本阻碍了进一步发展和广泛采用。在本文中，受大型语言模型参数冗余的启发，我们提出了一种新颖的训练范式：演进子网训练 (EST)。EST 从大型语言模型的各层以及每层内常用的模块多头注意力 (MHA) 和多层感知器 (MLP) 中抽取子网络样本。通过在训练过程中逐渐增加子网络的大小，EST 可以节省训练成本。我们应用 EST 训练 GPT2 模型和 TinyLlama 模型，结果为 GPT2 节省了 26.7% 的 FLOP，为 TinyLlama 节省了 25.0% 的 FLOP，而预训练数据集上的损失没有增加。此外，EST 可以提高下游任务的性能，表明它有利于泛化。此外，我们提供了基于训练动态和 Dropout 理论的直观理论研究，以确保 EST 的可行性。我们的代码可在此 https URL 上获取。

Title: Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models

• Authors: Zhenyi Lu, Jie Tian, Wei Wei, Xiaoye Qu, Yu Cheng, Wenfeng xie, Dangyang Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07001
• Pdf URL: https://arxiv.org/pdf/2406.07001
• Copy Paste: [[2406.07001]] Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models(https://arxiv.org/abs/2406.07001)
• Keywords: language model, llm, chain-of-thought
• Abstract: Text classification is a crucial task encountered frequently in practical scenarios, yet it is still under-explored in the era of large language models (LLMs). This study shows that LLMs are vulnerable to changes in the number and arrangement of options in text classification. Our extensive empirical analyses reveal that the key bottleneck arises from ambiguous decision boundaries and inherent biases towards specific tokens and positions. To mitigate these issues, we make the first attempt and propose a novel two-stage classification framework for LLMs. Our approach is grounded in the empirical observation that pairwise comparisons can effectively alleviate boundary ambiguity and inherent bias. Specifically, we begin with a self-reduction technique to efficiently narrow down numerous options, which contributes to reduced decision space and a faster comparison process. Subsequently, pairwise contrastive comparisons are employed in a chain-of-thought manner to draw out nuances and distinguish confusable options, thus refining the ambiguous decision boundary. Extensive experiments on four datasets (Banking77, HWU64, LIU54, and Clinic150) verify the effectiveness of our framework. Furthermore, benefitting from our framework, various LLMs can achieve consistent improvements. Our code and data are available in \url{this https URL}.
• 摘要：文本分类是实际场景中经常遇到的一项重要任务，但在大型语言模型 (LLM) 时代，它仍未得到充分探索。本研究表明，LLM 容易受到文本分类中选项数量和排列变化的影响。我们广泛的实证分析表明，关键瓶颈来自模糊的决策边界和对特定标记和位置的固有偏见。为了缓解这些问题，我们进行了首次尝试，并提出了一种新颖的 LLM 两阶段分类框架。我们的方法基于实证观察，即成对比较可以有效缓解边界模糊性和固有偏见。具体而言，我们首先采用一种自我缩减技术来有效缩小众多选项，这有助于减少决策空间并加快比较过程。随后，以思路链的方式采用成对对比比较来提取细微差别并区分易混淆的选项，从而细化模糊的决策边界。在四个数据集（Banking77、HWU64、LIU54 和 Clinic150）上进行的大量实验验证了我们框架的有效性。此外，得益于我们的框架，各种 LLM 可以实现持续改进。我们的代码和数据可在 \url{此 https URL} 中找到。

Title: Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference

• Authors: Jihwan Bang, Juntae Lee, Kyuhong Shim, Seunghan Yang, Simyung Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07007
• Pdf URL: https://arxiv.org/pdf/2406.07007
• Copy Paste: [[2406.07007]] Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference(https://arxiv.org/abs/2406.07007)
• Keywords: language model, llm
• Abstract: The customization of large language models (LLMs) for user-specified tasks gets important. However, maintaining all the customized LLMs on cloud servers incurs substantial memory and computational overheads, and uploading user data can also lead to privacy concerns. On-device LLMs can offer a promising solution by mitigating these issues. Yet, the performance of on-device LLMs is inherently constrained by the limitations of small-scaled models. To overcome these restrictions, we first propose Crayon, a novel approach for on-device LLM customization. Crayon begins by constructing a pool of diverse base adapters, and then we instantly blend them into a customized adapter without extra training. In addition, we develop a device-server hybrid inference strategy, which deftly allocates more demanding queries or non-customized tasks to a larger, more capable LLM on a server. This ensures optimal performance without sacrificing the benefits of on-device customization. We carefully craft a novel benchmark from multiple question-answer datasets, and show the efficacy of our method in the LLM customization.
• 摘要：为用户指定的任务定制大型语言模型 (LLM) 变得非常重要。然而，在云服务器上维护所有定制的 LLM 会产生大量的内存和计算开销，而上传用户数据也会导致隐私问题。设备上的 LLM 可以通过缓解这些问题来提供有希望的解决方案。然而，设备上的 LLM 的性能本质上受到小型模型的限制。为了克服这些限制，我们首先提出了 Crayon，这是一种用于设备上 LLM 定制的新方法。Crayon 首先构建一个多样化的基础适配器池，然后我们立即将它们混合成一个定制的适配器，而无需额外的训练。此外，我们开发了一种设备-服务器混合推理策略，该策略巧妙地将要求更高的查询或非定制任务分配给服务器上更大、更强大的 LLM。这确保了最佳性能，而不会牺牲设备上定制的好处。我们从多个问答数据集中精心制作了一个新颖的基准，并展示了我们的方法在 LLM 定制中的有效性。

Title: Delving into ChatGPT usage in academic writing through excess vocabulary

• Authors: Dmitry Kobak, Rita González Márquez, Emőke-Ágnes Horvát, Jan Lause
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, cs.SI
• Abstract URL: https://arxiv.org/abs/2406.07016
• Pdf URL: https://arxiv.org/pdf/2406.07016
• Copy Paste: [[2406.07016]] Delving into ChatGPT usage in academic writing through excess vocabulary(https://arxiv.org/abs/2406.07016)
• Keywords: language model, gpt, llm, chat
• Abstract: Recent large language models (LLMs) can generate and revise text with human-level performance, and have been widely commercialized in systems like ChatGPT. These models come with clear limitations: they can produce inaccurate information, reinforce existing biases, and be easily misused. Yet, many scientists have been using them to assist their scholarly writing. How wide-spread is LLM usage in the academic literature currently? To answer this question, we use an unbiased, large-scale approach, free from any assumptions on academic LLM usage. We study vocabulary changes in 14 million PubMed abstracts from 2010-2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. Our analysis based on excess words usage suggests that at least 10% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, and was as high as 30% for some PubMed sub-corpora. We show that the appearance of LLM-based writing assistants has had an unprecedented impact in the scientific literature, surpassing the effect of major world events such as the Covid pandemic.
• 摘要：最近的大型语言模型 (LLM) 可以以人类水平的性能生成和修改文本，并且已在 ChatGPT 等系统中广泛商业化。这些模型有明显的局限性：它们可能会产生不准确的信息、强化现有的偏见并且容易被滥用。然而，许多科学家一直在使用它们来协助他们的学术写作。目前 LLM 在学术文献中的使用范围有多广？为了回答这个问题，我们使用了一种无偏见的大规模方法，不做任何关于学术 LLM 使用的假设。我们研究了 2010 年至 2024 年 1400 万篇 PubMed 摘要中的词汇变化，并展示了 LLM 的出现如何导致某些风格词的频率急剧增加。我们基于多余词汇使用的分析表明，2024 篇摘要中至少有 10% 是用 LLM 处理的。这个下限因学科、国家和期刊而异，对于某些 PubMed 子语料库，这个下限高达 30%。我们表明，基于法学硕士的写作助手的出现对科学文献产生了前所未有的影响，超过了新冠疫情等重大世界事件的影响。

Title: Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model

• Authors: Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07036
• Pdf URL: https://arxiv.org/pdf/2406.07036
• Copy Paste: [[2406.07036]] Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model(https://arxiv.org/abs/2406.07036)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have showcased impressive multilingual machine translation ability. However, unlike encoder-decoder style models, decoder-only LLMs lack an explicit alignment between source and target contexts. Analyzing contribution scores during generation processes revealed that LLMs can be biased towards previously generated tokens over corresponding source tokens, leading to unfaithful translations. To address this issue, we propose to encourage LLMs to pay more attention to the source context from both source and target perspectives in zeroshot prompting: 1) adjust source context attention weights; 2) suppress irrelevant target prefix influence; Additionally, we propose 3) avoiding over-reliance on the target prefix in instruction tuning. Experimental results from both human-collected unfaithfulness test sets focusing on LLM-generated unfaithful translations and general test sets, verify our methods' effectiveness across multiple language pairs. Further human evaluation shows our method's efficacy in reducing hallucinatory translations and facilitating faithful translation generation.
• 摘要：大型语言模型 (LLM) 展示了令人印象深刻的多语言机器翻译能力。然而，与编码器-解码器样式的模型不同，仅解码器的 LLM 缺乏源上下文和目标上下文之间的明确对齐。分析生成过程中的贡献分数表明，LLM 可能偏向于先前生成的标记而不是相应的源标记，从而导致不忠实的翻译。为了解决这个问题，我们建议鼓励 LLM 在零样本提示中从源和目标角度更多地关注源上下文：1）调整源上下文注意权重；2）抑制不相关的目标前缀影响；此外，我们建议 3）避免在指令调整中过度依赖目标前缀。来自人工收集的不忠实测试集（重点关注 LLM 生成的不忠实翻译）和一般测试集的实验结果验证了我们的方法在多种语言对中的有效性。进一步的人工评估表明我们的方法在减少幻觉翻译和促进忠实翻译生成方面的有效性。

Title: CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation

• Authors: Renhao Li, Minghuan Tan, Derek F. Wong, Min Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07054
• Pdf URL: https://arxiv.org/pdf/2406.07054
• Copy Paste: [[2406.07054]] CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation(https://arxiv.org/abs/2406.07054)
• Keywords: language model, llm, agent
• Abstract: In recent years, instruction fine-tuning (IFT) on large language models (LLMs) has garnered considerable attention to enhance model performance on unseen tasks. Attempts have been made on automatic construction and effective selection for IFT data. However, we posit that previous methods have not fully harnessed the potential of LLMs for enhancing data quality. The responses within IFT data could be further enhanced by leveraging the capabilities of LLMs themselves. In this paper, we propose CoEvol, an LLM-based multi-agent cooperation framework for the improvement of responses to instructions. To effectively refine the responses, we develop an iterative framework following a debate-advise-edit-judge paradigm. A two-stage multi-agent debate strategy is further devised to ensure the diversity and reliability of editing suggestions within the framework. Empirically, models equipped with CoEvol outperform competitive baselines evaluated by MT-Bench and AlpacaEval, demonstrating its effectiveness in enhancing instruction-following capabilities for LLMs.
• 摘要：近年来，大型语言模型 (LLM) 上的指令微调 (IFT) 引起了广泛关注，以提高模型在未见过的任务上的性能。人们已经尝试自动构建和有效选择 IFT 数据。然而，我们认为以前的方法尚未充分利用 LLM 提高数据质量的潜力。通过利用 LLM 本身的功能，可以进一步增强 IFT 数据中的响应。在本文中，我们提出了 CoEvol，这是一个基于 LLM 的多智能体合作框架，用于改进对指令的响应。为了有效地改进响应，我们开发了一个遵循辩论-建议-编辑-评判范式的迭代框架。进一步设计了一种两阶段多智能体辩论策略，以确保框架内编辑建议的多样性和可靠性。从经验上看，配备 CoEvol 的模型优于 MT-Bench 和 AlpacaEval 评估的竞争基线，证明了其在增强 LLM 的指令遵循能力方面的有效性。

Title: Effectively Compress KV Heads for LLM

• Authors: Hao Yu, Zelan Yang, Shen Li, Yong Li, Jianxin Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07056
• Pdf URL: https://arxiv.org/pdf/2406.07056
• Copy Paste: [[2406.07056]] Effectively Compress KV Heads for LLM(https://arxiv.org/abs/2406.07056)
• Keywords: language model, llm
• Abstract: The advent of pre-trained large language models (LLMs) has revolutionized various natural language processing tasks. These models predominantly employ an auto-regressive decoding mechanism that utilizes Key-Value (KV) caches to eliminate redundant calculations for previous tokens. Nevertheless, as context lengths and batch sizes increase, the linear expansion in memory footprint of KV caches becomes a key bottleneck of LLM deployment, which decreases generation speeds significantly. To mitigate this issue, previous techniques like multi-query attention (MQA) and grouped-query attention (GQA) have been developed, in order to reduce KV heads to accelerate inference with comparable accuracy to multi-head attention (MHA). Despite their effectiveness, existing strategies for compressing MHA often overlook the intrinsic properties of the KV caches. In this work, we explore the low-rank characteristics of the KV caches and propose a novel approach for compressing KV heads. In particular, we carefully optimize the MHA-to-GQA transformation to minimize compression error, and to remain compatible with rotary position embeddings (RoPE), we also introduce specialized strategies for key caches with RoPE. We demonstrate that our method can compress half or even three-quarters of KV heads while maintaining performance comparable to the original LLMs, which presents a promising direction for more efficient LLM deployment in resource-constrained environments.
• 摘要：预训练大型语言模型 (LLM) 的出现彻底改变了各种自然语言处理任务。这些模型主要采用自回归解码机制，利用键值 (KV) 缓存来消除先前标记的冗余计算。然而，随着上下文长度和批次大小的增加，KV 缓存内存占用的线性扩展成为 LLM 部署的主要瓶颈，这显著降低了生成速度。为了缓解这个问题，已经开发了多查询注意 (MQA) 和分组查询注意 (GQA) 等先前的技术，以减少 KV 头以加速推理，同时达到与多头注意 (MHA) 相当的准确度。尽管现有的压缩 MHA 策略很有效，但它们往往忽略了 KV 缓存的内在属性。在这项工作中，我们探索了 KV 缓存的低秩特性，并提出了一种压缩 KV 头的新方法。具体来说，我们精心优化了 MHA 到 GQA 的转换，以最大限度地减少压缩误差，并且为了与旋转位置嵌入 (RoPE) 保持兼容，我们还引入了 RoPE 的密钥缓存专用策略。我们证明我们的方法可以压缩一半甚至四分之三的 KV 头，同时保持与原始 LLM 相当的性能，这为在资源受限的环境中更有效地部署 LLM 提供了一个有希望的方向。

Title: Benchmarking Trustworthiness of Multimodal Large Language Models: A Comprehensive Study

• Authors: Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, Hang Su, Yinpeng Dong, Jun Zhu
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07057
• Pdf URL: https://arxiv.org/pdf/2406.07057
• Copy Paste: [[2406.07057]] Benchmarking Trustworthiness of Multimodal Large Language Models: A Comprehensive Study(https://arxiv.org/abs/2406.07057)
• Keywords: language model, llm
• Abstract: Despite the superior capabilities of Multimodal Large Language Models (MLLMs) across diverse tasks, they still face significant trustworthiness challenges. Yet, current literature on the assessment of trustworthy MLLMs remains limited, lacking a holistic evaluation to offer thorough insights into future improvements. In this work, we establish MultiTrust, the first comprehensive and unified benchmark on the trustworthiness of MLLMs across five primary aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark employs a rigorous evaluation strategy that addresses both multimodal risks and cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets. Extensive experiments with 21 modern MLLMs reveal some previously unexplored trustworthiness issues and risks, highlighting the complexities introduced by the multimodality and underscoring the necessity for advanced methodologies to enhance their reliability. For instance, typical proprietary models still struggle with the perception of visually confusing images and are vulnerable to multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to disclose privacy in text and reveal ideological and cultural biases even when paired with irrelevant images in inference, indicating that the multimodality amplifies the internal risks from base LLMs. Additionally, we release a scalable toolbox for standardized trustworthiness research, aiming to facilitate future advancements in this important field. Code and resources are publicly available at: this https URL.
• 摘要：尽管多模态大型语言模型 (MLLM) 在不同任务上都具有出色的能力，但它们仍然面临着重大的可信度挑战。然而，目前关于可信 MLLM 评估的文献仍然有限，缺乏整体评估以提供对未来改进的透彻见解。在这项工作中，我们建立了 MultiTrust，这是第一个全面统一的基准，涵盖 MLLM 五个主要方面的可信度：真实性、安全性、稳健性、公平性和隐私性。我们的基准采用严格的评估策略，涵盖 32 个不同任务和自我管理的数据集。对 21 种现代 MLLM 进行的大量实验揭示了一些以前未曾探索过的可信度问题和风险，突出了多模态引入的复杂性，并强调了需要先进的方法来提高其可靠性。例如，典型的专有模型仍然难以感知视觉上令人困惑的图像，并且容易受到多模态越狱和对抗攻击； MLLM 更倾向于在文本中泄露隐私，即使在推理中与不相关的图像配对，也会揭示意识形态和文化偏见，这表明多模态放大了基础 LLM 的内部风险。此外，我们发布了一个可扩展的标准化可信度研究工具箱，旨在促进这一重要领域的未来发展。代码和资源可从以下网址公开获取：此 https URL。

Title: HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation

• Authors: Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, Xi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07070
• Pdf URL: https://arxiv.org/pdf/2406.07070
• Copy Paste: [[2406.07070]] HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation(https://arxiv.org/abs/2406.07070)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), achieving remarkable performance across diverse tasks and enabling widespread real-world applications. However, LLMs are prone to hallucination, generating content that either conflicts with established knowledge or is unfaithful to the original sources. Existing hallucination benchmarks primarily focus on sentence- or passage-level hallucination detection, neglecting dialogue-level evaluation, hallucination localization, and rationale provision. They also predominantly target factuality hallucinations while underestimating faithfulness hallucinations, often relying on labor-intensive or non-specialized evaluators. To address these limitations, we propose HalluDial, the first comprehensive large-scale benchmark for automatic dialogue-level hallucination evaluation. HalluDial encompasses both spontaneous and induced hallucination scenarios, covering factuality and faithfulness hallucinations. The benchmark includes 4,094 dialogues with a total of 146,856 samples. Leveraging HalluDial, we conduct a comprehensive meta-evaluation of LLMs' hallucination evaluation capabilities in information-seeking dialogues and introduce a specialized judge language model, HalluJudge. The high data quality of HalluDial enables HalluJudge to achieve superior or competitive performance in hallucination evaluation, facilitating the automatic assessment of dialogue-level hallucinations in LLMs and providing valuable insights into this phenomenon. The dataset and the code are available at this https URL.
• 摘要：大型语言模型 (LLM) 极大地推动了自然语言处理 (NLP) 领域的发展，在各种任务中取得了卓越的表现，并实现了广泛的实际应用。然而，LLM 容易产生幻觉，生成的内容要么与既定知识相冲突，要么与原始来源不符。现有的幻觉基准主要侧重于句子或段落级别的幻觉检测，而忽略了对话级别的评估、幻觉定位和理由提供。它们还主要针对事实性幻觉，而低估了真实性幻觉，通常依赖于劳动密集型或非专业的评估者。为了解决这些限制，我们提出了 HalluDial，这是第一个全面的大型自动对话级幻觉评估基准。HalluDial 涵盖自发和诱发幻觉场景，涵盖事实性和真实性幻觉。该基准包括 4,094 个对话，共计 146,856 个样本。利用 HalluDial，我们对 LLM 在信息寻求型对话中的幻觉评估能力进行了全面的元评估，并引入了专门的判断语言模型 HalluJudge。HalluDial 的高数据质量使 HalluJudge 在幻觉评估中取得了优异或有竞争力的表现，促进了 LLM 中对话级幻觉的自动评估，并为这一现象提供了宝贵的见解。数据集和代码可在此 https URL 上找到。

Title: DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs

• Authors: Haishuo Fang, Xiaodan Zhu, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07080
• Pdf URL: https://arxiv.org/pdf/2406.07080
• Copy Paste: [[2406.07080]] DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs(https://arxiv.org/abs/2406.07080)
• Keywords: language model, gpt, llm, agent
• Abstract: Answering Questions over Knowledge Graphs (KGQA) is key to well-functioning autonomous language agents in various real-life applications. To improve the neural-symbolic reasoning capabilities of language agents powered by Large Language Models (LLMs) in KGQA, we propose the DecompositionAlignment-Reasoning Agent (DARA) framework. DARA effectively parses questions into formal queries through a dual mechanism: high-level iterative task decomposition and low-level task grounding. Importantly, DARA can be efficiently trained with a small number of high-quality reasoning trajectories. Our experimental results demonstrate that DARA fine-tuned on LLMs (e.g. Llama-2-7B, Mistral) outperforms both in-context learning-based agents with GPT-4 and alternative fine-tuned agents, across different benchmarks in zero-shot evaluation, making such models more accessible for real-life applications. We also show that DARA attains performance comparable to state-of-the-art enumerating-and-ranking-based methods for KGQA.
• 摘要：基于知识图谱的问答 (KGQA) 是各种实际应用中自主语言代理正常运行的关键。为了提高 KGQA 中由大型语言模型 (LLM) 驱动的语言代理的神经符号推理能力，我们提出了分解对齐推理代理 (DARA) 框架。DARA 通过双重机制有效地将问题解析为形式查询：高级迭代任务分解和低级任务基础。重要的是，DARA 可以通过少量高质量推理轨迹进行有效训练。我们的实验结果表明，在零样本评估的不同基准上，在 LLM（例如 Llama-2-7B、Mistral）上微调的 DARA 优于使用 GPT-4 的基于上下文学习的代理和其他微调的代理，使此类模型更适用于实际应用。我们还表明，DARA 的性能与最先进的基于枚举和排名的 KGQA 方法相当。

Title: Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning

• Authors: Menglong Cui, Jiangcun Du, Shaolin Zhu, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07081
• Pdf URL: https://arxiv.org/pdf/2406.07081
• Copy Paste: [[2406.07081]] Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning(https://arxiv.org/abs/2406.07081)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) exhibit outstanding performance in machine translation via in-context learning. In contrast to sentence-level translation, document-level translation (DOCMT) by LLMs based on in-context learning faces two major challenges: firstly, document translations generated by LLMs are often incoherent; secondly, the length of demonstration for in-context learning is usually limited. To address these issues, we propose a Context-Aware Prompting method (CAP), which enables LLMs to generate more accurate, cohesive, and coherent translations via in-context learning. CAP takes into account multi-level attention, selects the most relevant sentences to the current one as context, and then generates a summary from these collected sentences. Subsequently, sentences most similar to the summary are retrieved from the datastore as demonstrations, which effectively guide LLMs in generating cohesive and coherent translations. We conduct extensive experiments across various DOCMT tasks, and the results demonstrate the effectiveness of our approach, particularly in zero pronoun translation (ZPT) and literary translation tasks.
• 摘要：大型语言模型 (LLM) 通过上下文学习在机器翻译中表现出色。与句子级翻译相比，基于上下文学习的 LLM 的文档级翻译 (DOCMT) 面临两大挑战：首先，LLM 生成的文档翻译通常不连贯；其次，上下文学习的演示长度通常有限。为了解决这些问题，我们提出了一种上下文感知提示方法 (CAP)，使 LLM 能够通过上下文学习生成更准确、更有凝聚力和更连贯的翻译。CAP 考虑了多级注意力，选择与当前句子最相关的句子作为上下文，然后从这些收集到的句子中生成摘要。随后，从数据存储中检索与摘要最相似的句子作为演示，从而有效地指导 LLM 生成有凝聚力和连贯性的翻译。我们在各种 DOCMT 任务中进行了广泛的实验，结果证明了我们的方法的有效性，特别是在零代词翻译 (ZPT) 和文学翻译任务中。

Title: Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees

• Authors: Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07115
• Pdf URL: https://arxiv.org/pdf/2406.07115
• Copy Paste: [[2406.07115]] Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees(https://arxiv.org/abs/2406.07115)
• Keywords: language model, llm, agent
• Abstract: Tool-augmented large language models (LLMs) leverage tools, often in the form of APIs, to enhance their reasoning capabilities on complex tasks, thus taking on the role of intelligent agents interacting with the real world. The recently introduced ToolLLaMA model by Qin et al. [2024] utilizes the depth-first search-based decision tree (DFSDT) method for reasoning with $16000+$ real-world APIs, which effectively improves the planning and inferencing performance of tool-augmented LLMs compared to traditional chain reasoning approaches. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT) during training, which does not fully exploit the advantages of the tree of thought. In this study, we propose an inference trajectory optimization framework based on the preference data extracted from decision trees to address this limitation. We first introduce a novel method for constructing preference data from the tree of thought, capitalizing on the failed explorations previously overlooked in the trees. Specifically, we generate an effective step-wise preference dataset, named ToolPreference, for tool use based on the ToolBench dataset. In the subsequent training phase, we first fine-tune the LLM with tool-usage expert trajectories and then use these step-wise preference pairs for direct preference optimization (DPO) to update the policy of the LLM, resulting in our ToolPrefer-LLaMA (TP-LLaMA) model. Our experiments demonstrate that by obtaining insights from errors in inference trees, TP-LLaMA significantly outperforms the baselines across almost all test scenarios by a large margin and exhibits better generalization capabilities with unseen APIs. At the same time, TP-LLaMA has also demonstrated superior reasoning efficiency compared to the baselines, making it more suitable for complex tool-usage reasoning tasks.
• 摘要：工具增强型大型语言模型 (LLM) 利用工具（通常以 API 的形式）来增强其在复杂任务上的推理能力，从而承担与现实世界交互的智能代理的角色。Qin 等人 [2024] 最近引入的 ToolLLaMA 模型利用深度优先搜索的决策树 (DFSDT) 方法进行推理，使用 $16000+$ 个现实世界 API，与传统的链式推理方法相比，有效地提高了工具增强型 LLM 的规划和推理性能。然而，他们的方法在训练期间仅使用决策树（也称为推理树）中的成功路径进行监督微调 (SFT)，这并没有充分利用思维树的优势。在本研究中，我们提出了一种基于从决策树中提取的偏好数据的推理轨迹优化框架来解决这一限制。我们首先介绍了一种从思维树中构建偏好数据的新方法，利用了树中以前被忽略的失败探索。具体来说，我们基于 ToolBench 数据集生成了一个针对工具使用的有效分步偏好数据集 ToolPreference。在后续的训练阶段，我们首先使用工具使用专家轨迹对 LLM 进行微调，然后使用这些分步偏好对进行直接偏好优化（DPO）来更新 LLM 的策略，从而得到我们的 ToolPrefer-LLaMA（TP-LLaMA）模型。我们的实验表明，通过从推理树中的错误中获得洞察力，TP-LLaMA 在几乎所有测试场景中都显著优于基线，并且在未知 API 上表现出更好的泛化能力。同时，TP-LLaMA 与基线相比也表现出了卓越的推理效率，使其更适合复杂的工具使用推理任务。

Title: Never Miss A Beat: An Efficient Recipe for Context Window Extension of Large Language Models with Consistent "Middle" Enhancement

• Authors: Tong Wu, Yanpeng Zhao, Zilong Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07138
• Pdf URL: https://arxiv.org/pdf/2406.07138
• Copy Paste: [[2406.07138]] Never Miss A Beat: An Efficient Recipe for Context Window Extension of Large Language Models with Consistent "Middle" Enhancement(https://arxiv.org/abs/2406.07138)
• Keywords: language model, llm, chat
• Abstract: Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ($\gg4K$) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle (CREAM), which interpolates positional encodings by manipulating position indices. Apart from being simple, CREAM is training-efficient: it only requires fine-tuning at the pre-trained context window (eg, Llama 2-4K) and can extend LLMs to a much longer target context length (eg, 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the ``Lost-in-the-Middle'' problem faced by long-context LLMs. Experimental results show that CREAM successfully extends LLMs to the target length for both Base and Chat versions of $\texttt{Llama2-7B}$ with ``Never Miss A Beat''. Our code will be publicly available soon.
• 摘要：最近，已经开发出许多方法来扩展预训练大型语言模型 (LLM) 的上下文长度，但它们通常需要在目标长度 ($\gg4K$) 上进行微调，并且难以有效利用上下文中间部分的信息。为了解决这些问题，我们提出了 $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle (CREAM)，它通过操纵位置索引来插入位置编码。除了简单之外，CREAM 还具有训练效率：它只需要在预训练的上下文窗口（例如 Llama 2-4K）上进行微调，并且可以将 LLM 扩展到更长的目标上下文长度（例如 256K）。为了确保模型更关注中间的信息，我们引入了截断高斯，以鼓励在微调期间从上下文的中间部分进行采样，从而缓解长上下文 LLM 面临的“中间丢失”问题。实验结果表明，CREAM 成功地将 LLM 扩展到 $\texttt{Llama2-7B}$ 的基本版和聊天版的目标长度，并且“永不错过任何节拍”。我们的代码将很快公开发布。

Title: Teaching Language Models to Self-Improve by Learning from Language Feedback

• Authors: Chi Hu, Yimin Hu, Hang Cao, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07168
• Pdf URL: https://arxiv.org/pdf/2406.07168
• Copy Paste: [[2406.07168]] Teaching Language Models to Self-Improve by Learning from Language Feedback(https://arxiv.org/abs/2406.07168)
• Keywords: language model, gpt, llm
• Abstract: Aligning Large Language Models (LLMs) with human intentions and values is crucial yet challenging. Current methods primarily rely on human preferences, which are costly and insufficient in capturing nuanced feedback expressed in natural language. In this paper, we present Self-Refinement Tuning (SRT), a method that leverages model feedback for alignment, thereby reducing reliance on human annotations. SRT uses a base language model (e.g., Tulu2) to generate initial responses, which are critiqued and refined by a more advanced model (e.g., GPT-4-Turbo). This process enables the base model to self-evaluate and improve its outputs, facilitating continuous learning. SRT further optimizes the model by learning from its self-generated feedback and refinements, creating a feedback loop that promotes model improvement. Our empirical evaluations demonstrate that SRT significantly outperforms strong baselines across diverse tasks and model sizes. When applied to a 70B parameter model, SRT increases the win rate from 9.6\% to 25.8\% on the AlpacaEval 2.0 benchmark, surpassing well-established systems such as GPT-4-0314, Claude 2, and Gemini. Our analysis highlights the crucial role of language feedback in the success of SRT, suggesting potential for further exploration in this direction.
• 摘要：将大型语言模型 (LLM) 与人类的意图和价值观相一致至关重要，但又极具挑战性。当前的方法主要依赖于人类的偏好，而这些偏好成本高昂，并且不足以捕捉到自然语言中表达的细微反馈。在本文中，我们提出了自优化调整 (SRT)，这是一种利用模型反馈进行对齐的方法，从而减少对人工注释的依赖。SRT 使用基础语言模型（例如 Tulu2）来生成初始响应，然后由更高级的模型（例如 GPT-4-Turbo）对其进行批评和改进。此过程使基础模型能够自我评估并改进其输出，从而促进持续学习。SRT 通过从其自生成的反馈和改进中学习来进一步优化模型，从而创建促进模型改进的反馈循环。我们的实证评估表明，SRT 在不同任务和模型大小上的表现明显优于强大的基线。当应用于 70B 参数模型时，SRT 在 AlpacaEval 2.0 基准上将胜率从 9.6% 提高到 25.8%，超越了 GPT-4-0314、Claude 2 和 Gemini 等成熟系统。我们的分析强调了语言反馈在 SRT 成功中的关键作用，表明在这方面有进一步探索的潜力。

Title: Merging Improves Self-Critique Against Jailbreak Attacks

• Authors: Victor Gallego
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07188
• Pdf URL: https://arxiv.org/pdf/2406.07188
• Copy Paste: [[2406.07188]] Merging Improves Self-Critique Against Jailbreak Attacks(https://arxiv.org/abs/2406.07188)
• Keywords: language model, llm, prompt
• Abstract: The robustness of large language models (LLMs) against adversarial manipulations, such as jailbreak attacks, remains a significant challenge. In this work, we propose an approach that enhances the self-critique capability of the LLM and further fine-tunes it over sanitized synthetic data. This is done with the addition of an external critic model that can be merged with the original, thus bolstering self-critique capabilities and improving the robustness of the LLMs response to adversarial prompts. Our results demonstrate that the combination of merging and self-critique can reduce the attack success rate of adversaries significantly, thus offering a promising defense mechanism against jailbreak attacks. Code, data and models released at this https URL .
• 摘要：大型语言模型 (LLM) 抵御越狱攻击等对抗性操纵的鲁棒性仍然是一项重大挑战。在这项工作中，我们提出了一种方法，可以增强 LLM 的自我批评能力，并进一步针对经过清理的合成数据对其进行微调。这是通过添加一个可以与原始模型合并的外部批评模型来实现的，从而增强了自我批评能力并提高了 LLM 对对抗性提示的响应的鲁棒性。我们的结果表明，合并和自我批评的结合可以显著降低对手的攻击成功率，从而为越狱攻击提供了一种有希望的防御机制。代码、数据和模型在此 https URL 上发布。

Title: Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models

• Authors: Joshua Strong, Qianhui Men, Alison Noble
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2406.07212
• Pdf URL: https://arxiv.org/pdf/2406.07212
• Copy Paste: [[2406.07212]] Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models(https://arxiv.org/abs/2406.07212)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) present a valuable technology for various applications in healthcare, but their tendency to hallucinate introduces unacceptable uncertainty in critical decision-making situations. Human-AI collaboration (HAIC) can mitigate this uncertainty by combining human and AI strengths for better outcomes. This paper presents a novel guided deferral system that provides intelligent guidance when AI defers cases to human decision-makers. We leverage LLMs' verbalisation capabilities and internal states to create this system, demonstrating that fine-tuning smaller LLMs with data from larger models enhances performance while maintaining computational efficiency. A pilot study showcases the effectiveness of our deferral system.
• 摘要：大型语言模型 (LLM) 为医疗保健领域的各种应用提供了宝贵的技术，但它们容易产生幻觉，在关键的决策情况下会带来不可接受的不确定性。人机协作 (HAIC) 可以通过结合人类和人工智能的优势来减轻这​​种不确定性，以获得更好的结果。本文介绍了一种新颖的引导式延期系统，当人工智能将案例推迟给人类决策者时，该系统可提供智能指导。我们利用 LLM 的语言化能力和内部状态来创建此系统，并证明使用来自较大模型的数据对较小的 LLM 进行微调可以提高性能，同时保持计算效率。一项初步研究展示了我们的延期系统的有效性。

Title: Improving Autoformalization using Type Checking

• Authors: Auguste Poiroux, Gail Weiss, Viktor Kunčak, Antoine Bosselut
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07222
• Pdf URL: https://arxiv.org/pdf/2406.07222
• Copy Paste: [[2406.07222]] Improving Autoformalization using Type Checking(https://arxiv.org/abs/2406.07222)
• Keywords: language model, gpt
• Abstract: Large language models show promise for autoformalization, the task of automatically translating natural language into formal languages. However, current autoformalization methods remain limited. The last reported state-of-the-art performance on the ProofNet formalization benchmark for the Lean proof assistant, achieved using Codex for Lean 3, only showed successful formalization of 16.1% of informal statements. Similarly, our evaluation of GPT-4o for Lean 4 only produces successful translations 34.9% of the time. Our analysis shows that the performance of these models is largely limited by their inability to generate formal statements that successfully type-check (i.e., are syntactically correct and consistent with types) - with a whopping 86.6% of GPT-4o errors starting from a type-check failure. In this work, we propose a method to fix this issue through decoding with type-check filtering, where we initially sample a diverse set of candidate formalizations for an informal statement, then use the Lean proof assistant to filter out candidates that do not type-check. Using GPT-4o as a base model, and combining our method with self-consistency, we obtain a +18.3% absolute increase in formalization accuracy, and achieve a new state-of-the-art of 53.2% on ProofNet with Lean 4.
• 摘要：大型语言模型有望实现自动形式化，即将自然语言自动翻译成形式语言。然而，目前的自动形式化方法仍然有限。上一次报告的 Lean 证明助手在 ProofNet 形式化基准上的最佳性能是使用 Codex for Lean 3 实现的，仅显示 16.1% 的非正式语句成功形式化。同样，我们对 GPT-4o for Lean 4 的评估只有 34.9% 的时间成功翻译。我们的分析表明，这些模型的性能在很大程度上受到它们无法生成成功类型检查（即语法正确且与类型一致）的形式语句的限制——高达 86.6% 的 GPT-4o 错误源于类型检查失败。在这项工作中，我们提出了一种通过类型检查过滤解码来解决这个问题的方法，我们首先对一个非正式语句抽样一组不同的候选形式化，然后使用 Lean 证明助手过滤掉没有类型检查的候选。使用 GPT-4o 作为基础模型，并将我们的方法与自洽性相结合，我们获得了 +18.3% 的形式化准确率绝对提升，并在使用 Lean 4 的 ProofNet 上实现了 53.2% 的新最佳水平。

Title: Improving Commonsense Bias Classification by Mitigating the Influence of Demographic Terms

• Authors: JinKyu Lee, Jihie Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07229
• Pdf URL: https://arxiv.org/pdf/2406.07229
• Copy Paste: [[2406.07229]] Improving Commonsense Bias Classification by Mitigating the Influence of Demographic Terms(https://arxiv.org/abs/2406.07229)
• Keywords: gpt, chat
• Abstract: Understanding commonsense knowledge is crucial in the field of Natural Language Processing (NLP). However, the presence of demographic terms in commonsense knowledge poses a potential risk of compromising the performance of NLP models. This study aims to investigate and propose methods for enhancing the performance and effectiveness of a commonsense polarization classifier by mitigating the influence of demographic terms. Three methods are introduced in this paper: (1) hierarchical generalization of demographic terms (2) threshold-based augmentation and (3) integration of hierarchical generalization and threshold-based augmentation methods (IHTA). The first method involves replacing demographic terms with more general ones based on a term hierarchy ontology, aiming to mitigate the influence of specific terms. To address the limited bias-related information, the second method measures the polarization of demographic terms by comparing the changes in the model's predictions when these terms are masked versus unmasked. This method augments commonsense sentences containing terms with high polarization values by replacing their predicates with synonyms generated by ChatGPT. The third method combines the two approaches, starting with threshold-based augmentation followed by hierarchical generalization. The experiments show that the first method increases the accuracy over the baseline by 2.33%, and the second one by 0.96% over standard augmentation methods. The IHTA techniques yielded an 8.82% and 9.96% higher accuracy than threshold-based and standard augmentation methods, respectively.
• 摘要：理解常识知识在自然语言处理 (NLP) 领域至关重要。然而，常识知识中人口统计术语的存在可能会损害 NLP 模型的性能。本研究旨在研究并提出通过减轻人口统计术语的影响来提高常识极化分类器的性能和有效性的方法。本文介绍了三种方法：(1) 人口统计术语的分层泛化 (2) 基于阈值的增强和 (3) 分层泛化和基于阈值的增强方法 (IHTA) 的集成。第一种方法涉及用基于术语层次本体的更通用的术语替换人口统计术语，旨在减轻特定术语的影响。为了解决有限的偏见相关信息，第二种方法通过比较这些术语被屏蔽和未被屏蔽时模型预测的变化来衡量人口统计术语的极化。该方法通过用 ChatGPT 生成的同义词替换其谓词来增强包含高极化值术语的常识句子。第三种方法结合了这两种方法，首先基于阈值进行增强，然后进行分层泛化。实验表明，第一种方法比基线提高了 2.33% 的准确率，第二种方法比标准增强方法提高了 0.96%。IHTA 技术的准确率分别比基于阈值和标准增强方法高出 8.82% 和 9.96%。

Title: Decipherment-Aware Multilingual Learning in Jointly Trained Language Models

• Authors: Grandee Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07231
• Pdf URL: https://arxiv.org/pdf/2406.07231
• Copy Paste: [[2406.07231]] Decipherment-Aware Multilingual Learning in Jointly Trained Language Models(https://arxiv.org/abs/2406.07231)
• Keywords: language model
• Abstract: The principle that governs unsupervised multilingual learning (UCL) in jointly trained language models (mBERT as a popular example) is still being debated. Many find it surprising that one can achieve UCL with multiple monolingual corpora. In this work, we anchor UCL in the context of language decipherment and show that the joint training methodology is a decipherment process pivotal for UCL. In a controlled setting, we investigate the effect of different decipherment settings on the multilingual learning performance and consolidate the existing opinions on the contributing factors to multilinguality. From an information-theoretic perspective we draw a limit to the UCL performance and demonstrate the importance of token alignment in challenging decipherment settings caused by differences in the data domain, language order and tokenization granularity. Lastly, we apply lexical alignment to mBERT and investigate the contribution of aligning different lexicon groups to downstream performance.
• 摘要：联合训练语言模型（mBERT 就是一个流行的例子）中无监督多语言学习 (UCL) 的支配原理仍在争论中。许多人惊讶于可以使用多个单语料库实现 UCL。在这项工作中，我们将 UCL 锚定在语言解读的背景下，并表明联合训练方法是 UCL 的关键解读过程。在受控环境中，我们研究了不同解读设置对多语言学习性能的影响，并巩固了对多语言性影响因素的现有观点。从信息论的角度来看，我们为 UCL 性能划定了一个界限，并证明了在由数据域、语言顺序和标记化粒度的差异导致的具有挑战性的解读设置中标记对齐的重要性。最后，我们将词汇对齐应用于 mBERT，并研究对齐不同词汇组对下游性能的贡献。

Title: DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanisms

• Authors: Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07232
• Pdf URL: https://arxiv.org/pdf/2406.07232
• Copy Paste: [[2406.07232]] DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanisms(https://arxiv.org/abs/2406.07232)
• Keywords: language model, llm
• Abstract: Recently, large language models (LLMs) enhanced by self-reflection have achieved promising performance on machine translation. The key idea is guiding LLMs to generate translation with human-like feedback. However, existing self-reflection methods lack effective feedback information, limiting the translation performance. To address this, we introduce a DUAL-REFLECT framework, leveraging the dual learning of translation tasks to provide effective feedback, thereby enhancing the models' self-reflective abilities and improving translation performance. The application of this method across various translation tasks has proven its effectiveness in improving translation accuracy and eliminating ambiguities, especially in translation tasks with low-resource language pairs.
• 摘要：最近，通过自我反思增强的大型语言模型 (LLM) 在机器翻译上取得了令人瞩目的表现。其核心思想是引导 LLM 生成具有类似人类反馈的翻译。然而，现有的自我反思方法缺乏有效的反馈信息，限制了翻译性能。为了解决这个问题，我们引入了一个 DUAL-REFLECT 框架，利用翻译任务的对偶学习提供有效反馈，从而增强模型的自我反思能力并提高翻译性能。该方法在各种翻译任务中的应用已证明其在提高翻译准确性和消除歧义方面的有效性，尤其是在资源较少的语言对的翻译任务中。

Title: On the Hallucination in Simultaneous Machine Translation

• Authors: Meizhi Zhong, Kehai Chen, Zhengshan Xue, Lemao Liu, Mingming Yang, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07239
• Pdf URL: https://arxiv.org/pdf/2406.07239
• Copy Paste: [[2406.07239]] On the Hallucination in Simultaneous Machine Translation(https://arxiv.org/abs/2406.07239)
• Keywords: hallucination
• Abstract: It is widely known that hallucination is a critical issue in Simultaneous Machine Translation (SiMT) due to the absence of source-side information. While many efforts have been made to enhance performance for SiMT, few of them attempt to understand and analyze hallucination in SiMT. Therefore, we conduct a comprehensive analysis of hallucination in SiMT from two perspectives: understanding the distribution of hallucination words and the target-side context usage of them. Intensive experiments demonstrate some valuable findings and particularly show that it is possible to alleviate hallucination by decreasing the over usage of target-side information for SiMT.
• 摘要：众所周知，由于缺乏源端信息，幻觉是同步机器翻译 (SiMT) 中的一个关键问题。虽然人们为提高 SiMT 的性能做出了许多努力，但很少有人尝试理解和分析 SiMT 中的幻觉。因此，我们从两个角度对 SiMT 中的幻觉进行了全面分析：了解幻觉词的分布和它们的目标端上下文使用情况。大量的实验证明了一些有价值的发现，并特别表明可以通过减少 SiMT 对目标端信息的过度使用来缓解幻觉。

Title: MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

• Authors: Vera Neplenbroek, Arianna Bisazza, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07243
• Pdf URL: https://arxiv.org/pdf/2406.07243
• Copy Paste: [[2406.07243]] MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs(https://arxiv.org/abs/2406.07243)
• Keywords: language model, llm, prompt
• Abstract: Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at this https URL.
• 摘要：已证明生成式大型语言模型 (LLM) 存在有害的偏见和刻板印象。虽然安全微调通常以英语进行，但这些模型被许多不同语言的使用者使用。现有证据表明，这些模型在不同语言中的表现不一致，并且会根据用户的人口统计因素进行区分。受此启发，我们研究 LLM 所表现出的社会刻板印象是否因提示它们的语言而不同，同时控制文化差异和任务准确性。为此，我们提出了 MBBQ（问答多语言偏见基准），这是英语 BBQ 数据集的精心策划版本，扩展到荷兰语、西班牙语和土耳其语，用于衡量这些语言中普遍存在的刻板印象。我们进一步用并行控制数据集补充 MBBQ，以独立于偏见测量问答任务的任务表现。我们基于几个开源和专有 LLM 的结果证实，即使控制文化变化，某些非英语语言也比英语更容易受到偏见的影响。此外，我们观察到，除了最准确的模型外，所有模型的偏见行为都存在显著的跨语言差异。随着 MBBQ 的发布，我们希望鼓励对多语言环境中的偏见进行进一步研究。数据集和代码可在此 https URL 上找到。

Title: Scholarly Question Answering using Large Language Models in the NFDI4DataScience Gateway

• Authors: Hamed Babaei Giglou, Tilahun Abedissa Taffa, Rana Abdullah, Aida Usmanova, Ricardo Usbeck, Jennifer D'Souza, Sören Auer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07257
• Pdf URL: https://arxiv.org/pdf/2406.07257
• Copy Paste: [[2406.07257]] Scholarly Question Answering using Large Language Models in the NFDI4DataScience Gateway(https://arxiv.org/abs/2406.07257)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: This paper introduces a scholarly Question Answering (QA) system on top of the NFDI4DataScience Gateway, employing a Retrieval Augmented Generation-based (RAG) approach. The NFDI4DS Gateway, as a foundational framework, offers a unified and intuitive interface for querying various scientific databases using federated search. The RAG-based scholarly QA, powered by a Large Language Model (LLM), facilitates dynamic interaction with search results, enhancing filtering capabilities and fostering a conversational engagement with the Gateway search. The effectiveness of both the Gateway and the scholarly QA system is demonstrated through experimental analysis.
• 摘要：本文介绍了一种基于 NFDI4DataScience 网关的学术问答 (QA) 系统，该系统采用基于检索增强生成 (RAG) 的方法。NFDI4DS 网关作为基础框架，提供统一且直观的界面，可使用联合搜索查询各种科学数据库。基于 RAG 的学术问答系统由大型语言模型 (LLM) 提供支持，可促进与搜索结果的动态交互，增强过滤功能并促进与网关搜索的对话互动。通过实验分析证明了网关和学术问答系统的有效性。

Title: Scientific Computing with Large Language Models

• Authors: Christopher Culver, Peter Hicks, Mihailo Milenkovic, Sanjif Shanmugavelu, Tobias Becker
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07259
• Pdf URL: https://arxiv.org/pdf/2406.07259
• Copy Paste: [[2406.07259]] Scientific Computing with Large Language Models(https://arxiv.org/abs/2406.07259)
• Keywords: language model, chat
• Abstract: We provide an overview of the emergence of large language models for scientific computing applications. We highlight use cases that involve natural language processing of scientific documents and specialized languages designed to describe physical systems. For the former, chatbot style applications appear in medicine, mathematics and physics and can be used iteratively with domain experts for problem solving. We also review specialized languages within molecular biology, the languages of molecules, proteins, and DNA where language models are being used to predict properties and even create novel physical systems at much faster rates than traditional computing methods.
• 摘要：我们概述了用于科学计算应用的大型语言模型的出现。我们重点介绍了涉及科学文档的自然语言处理和用于描述物理系统的专用语言的用例。对于前者，聊天机器人风格的应用程序出现在医学、数学和物理学中，可以与领域专家一起反复使用以解决问题。我们还回顾了分子生物学中的专用语言，即分子、蛋白质和 DNA 的语言，其中语言模型用于预测属性，甚至以比传统计算方法快得多的速度创建新的物理系统。

Title: Speaking Your Language: Spatial Relationships in Interpretable Emergent Communication

• Authors: Olaf Lipinski, Adam J. Sobey, Federico Cerutti, Timothy J. Norman
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2406.07277
• Pdf URL: https://arxiv.org/pdf/2406.07277
• Copy Paste: [[2406.07277]] Speaking Your Language: Spatial Relationships in Interpretable Emergent Communication(https://arxiv.org/abs/2406.07277)
• Keywords: agent
• Abstract: Effective communication requires the ability to refer to specific parts of an observation in relation to others. While emergent communication literature shows success in developing various language properties, no research has shown the emergence of such positional references. This paper demonstrates how agents can communicate about spatial relationships within their observations. The results indicate that agents can develop a language capable of expressing the relationships between parts of their observation, achieving over 90% accuracy when trained in a referential game which requires such communication. Using a collocation measure, we demonstrate how the agents create such references. This analysis suggests that agents use a mixture of non-compositional and compositional messages to convey spatial relationships. We also show that the emergent language is interpretable by humans. The translation accuracy is tested by communicating with the receiver agent, where the receiver achieves over 78% accuracy using parts of this lexicon, confirming that the interpretation of the emergent language was successful.
• 摘要：有效的沟通需要能够将观察结果的特定部分与其他部分联系起来。虽然新兴沟通文献表明在开发各种语言属性方面取得了成功，但没有研究表明出现了这种位置参考。本文展示了代理如何就其观察结果中的空间关系进行交流。结果表明，代理可以开发一种能够表达其观察结果各部分之间关系的语言，在需要此类交流的参考游戏中进行训练时，准确率超过 90%。使用搭配测量，我们展示了代理如何创建此类参考。该分析表明，代理使用非组合和组合消息的混合来传达空间关系。我们还表明，新兴语言可以被人类解释。通过与接收方代理进行通信来测试翻译准确性，接收方使用该词典的部分内容实现了超过 78% 的准确率，证实了对新兴语言的解释是成功的。

Title: Bilingual Sexism Classification: Fine-Tuned XLM-RoBERTa and GPT-3.5 Few-Shot Learning

• Authors: AmirMohammad Azadi, Baktash Ansari, Sina Zamani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07287
• Pdf URL: https://arxiv.org/pdf/2406.07287
• Copy Paste: [[2406.07287]] Bilingual Sexism Classification: Fine-Tuned XLM-RoBERTa and GPT-3.5 Few-Shot Learning(https://arxiv.org/abs/2406.07287)
• Keywords: gpt, prompt
• Abstract: Sexism in online content is a pervasive issue that necessitates effective classification techniques to mitigate its harmful impact. Online platforms often have sexist comments and posts that create a hostile environment, especially for women and minority groups. This content not only spreads harmful stereotypes but also causes emotional harm. Reliable methods are essential to find and remove sexist content, making online spaces safer and more welcoming. Therefore, the sEXism Identification in Social neTworks (EXIST) challenge addresses this issue at CLEF 2024. This study aims to improve sexism identification in bilingual contexts (English and Spanish) by leveraging natural language processing models. The tasks are to determine whether a text is sexist and what the source intention behind it is. We fine-tuned the XLM-RoBERTa model and separately used GPT-3.5 with few-shot learning prompts to classify sexist content. The XLM-RoBERTa model exhibited robust performance in handling complex linguistic structures, while GPT-3.5's few-shot learning capability allowed for rapid adaptation to new data with minimal labeled examples. Our approach using XLM-RoBERTa achieved 4th place in the soft-soft evaluation of Task 1 (sexism identification). For Task 2 (source intention), we achieved 2nd place in the soft-soft evaluation.
• 摘要：网络内容中的性别歧视是一个普遍存在的问题，需要有效的分类技术来减轻其有害影响。网络平台经常会有性别歧视的评论和帖子，营造出一种敌对的环境，尤其是对女性和少数群体而言。这些内容不仅传播了有害的刻板印象，还造成了情感伤害。可靠的方法对于查找和删除性别歧视内容至关重要，从而使网络空间更安全、更受欢迎。因此，CLEF 2024 上的“社交网络中的性别歧视识别”（EXIST）挑战赛解决了这个问题。这项研究旨在通过利用自然语言处理模型来提高双语环境（英语和西班牙语）中的性别歧视识别能力。任务是确定文本是否具有性别歧视以及其背后的源意图是什么。我们对 XLM-RoBERTa 模型进行了微调，并分别使用 GPT-3.5 和小样本学习提示对性别歧视内容进行分类。 XLM-RoBERTa 模型在处理复杂语言结构方面表现出了强大的性能，而 GPT-3.5 的少样本学习能力则允许使用最少的标记示例快速适应新数据。我们使用 XLM-RoBERTa 的方法在任务 1（性别歧视识别）的软-软评估中获得了第 4 名。对于任务 2（源意图），我们在软-软评估中获得了第 2 名。

Title: Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models

• Authors: Daniela Occhipinti, Michele Marchi, Irene Mondella, Huiyuan Lai, Felice Dell'Orletta, Malvina Nissim, Marco Guerini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07288
• Pdf URL: https://arxiv.org/pdf/2406.07288
• Copy Paste: [[2406.07288]] Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models(https://arxiv.org/abs/2406.07288)
• Keywords: language model
• Abstract: Automatic methods for generating and gathering linguistic data have proven effective for fine-tuning Language Models (LMs) in languages less resourced than English. Still, while there has been emphasis on data quantity, less attention has been given to its quality. In this work, we investigate the impact of human intervention on machine-generated data when fine-tuning dialogical models. In particular, we study (1) whether post-edited dialogues exhibit higher perceived quality compared to the originals that were automatically generated; (2) whether fine-tuning with post-edited dialogues results in noticeable differences in the generated outputs; and (3) whether post-edited dialogues influence the outcomes when considering the parameter size of the LMs. To this end we created HED-IT, a large-scale dataset where machine-generated dialogues are paired with the version post-edited by humans. Using both the edited and unedited portions of HED-IT, we fine-tuned three different sizes of an LM. Results from both human and automatic evaluation show that the different quality of training data is clearly perceived and it has an impact also on the models trained on such data. Additionally, our findings indicate that larger models are less sensitive to data quality, whereas this has a crucial impact on smaller models. These results enhance our comprehension of the impact of human intervention on training data in the development of high-quality LMs.
• 摘要：事实证明，自动生成和收集语言数据的方法对于资源比英语少的语言的语言模型 (LM) 的微调非常有效。尽管人们一直强调数据量，但对其质量的关注较少。在这项工作中，我们研究了在微调对话模型时人为干预对机器生成的数据的影响。具体来说，我们研究 (1) 与自动生成的原始对话相比，后期编辑的对话是否表现出更高的感知质量；(2) 使用后期编辑的对话进行微调是否会导致生成的输出出现明显差异；(3) 考虑到 LM 的参数大小，后期编辑的对话是否会影响结果。为此，我们创建了 HED-IT，这是一个大规模数据集，其中机器生成的对话与人工后期编辑的版本配对。使用 HED-IT 的编辑部分和未编辑部分，我们对三种不同大小的 LM 进行了微调。人工和自动评估的结果表明，训练数据的质量差异显而易见，而且这也会对基于此类数据训练的模型产生影响。此外，我们的研究结果表明，较大的模型对数据质量的敏感度较低，而这对较小的模型则有至关重要的影响。这些结果增强了我们对人工干预在开发高质量 LM 过程中对训练数据的影响的理解。

Title: BertaQA: How Much Do Language Models Know About Local Culture?

• Authors: Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, Mikel Artetxe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07302
• Pdf URL: https://arxiv.org/pdf/2406.07302
• Copy Paste: [[2406.07302]] BertaQA: How Much Do Language Models Know About Local Culture?(https://arxiv.org/abs/2406.07302)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To address this gap, we introduce BertaQA, a multiple-choice trivia dataset that is parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We find that state-of-the-art LLMs struggle with local cultural knowledge, even as they excel on global topics. However, we show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English. To our knowledge, this is the first solid evidence of knowledge transfer from a low-resource to a high-resource language. Our analysis sheds light on the complex interplay between language and knowledge, and reveals that some prior findings do not fully hold when reassessed on local topics. Our dataset and evaluation code are available under open licenses at this https URL.
• 摘要：大型语言模型 (LLM) 展现了有关世界的广泛知识，但大多数评估仅限于全球或以英语为中心的主题。这引发了一个问题：这些模型在与其他文化相关的主题上表现如何，而这些文化在网络上的存在并不那么突出。为了解决这一差距，我们引入了 BertaQA，这是一个英语和巴斯克语并行的多项选择琐事数据集。该数据集由一个与巴斯克文化相关的问题的本地子集和一个包含更广泛兴趣问题的全球子集组成。我们发现，最先进的 LLM 在处理本地文化知识时会遇到困难，即使在全球主题上表现出色。然而，我们表明，继续用巴斯克语进行预训练可以显著提高模型在巴斯克文化上的表现，即使在用英语查询时也是如此。据我们所知，这是知识从低资源语言转移到高资源语言的第一个确凿证据。我们的分析揭示了语言和知识之间复杂的相互作用，并揭示了一些先前的发现在重新评估本地主题时并不完全成立。我们的数据集和评估代码可通过此 https URL 以开放许可方式获得。

Title: Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities

• Authors: Delfina Sol Martinez Pandiani, Erik Tjong Kim Sang, Davide Ceolin
• Subjects: cs.CL, cs.AI, cs.CV, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2406.07353
• Pdf URL: https://arxiv.org/pdf/2406.07353
• Copy Paste: [[2406.07353]] Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities(https://arxiv.org/abs/2406.07353)
• Keywords: language model, llm
• Abstract: Internet memes, channels for humor, social commentary, and cultural expression, are increasingly used to spread toxic messages. Studies on the computational analyses of toxic memes have significantly grown over the past five years, and the only three surveys on computational toxic meme analysis cover only work published until 2022, leading to inconsistent terminology and unexplored trends. Our work fills this gap by surveying content-based computational perspectives on toxic memes, and reviewing key developments until early 2024. Employing the PRISMA methodology, we systematically extend the previously considered papers, achieving a threefold result. First, we survey 119 new papers, analyzing 158 computational works focused on content-based toxic meme analysis. We identify over 30 datasets used in toxic meme analysis and examine their labeling systems. Second, after observing the existence of unclear definitions of meme toxicity in computational works, we introduce a new taxonomy for categorizing meme toxicity types. We also note an expansion in computational tasks beyond the simple binary classification of memes as toxic or non-toxic, indicating a shift towards achieving a nuanced comprehension of toxicity. Third, we identify three content-based dimensions of meme toxicity under automatic study: target, intent, and conveyance tactics. We develop a framework illustrating the relationships between these dimensions and meme toxicities. The survey analyzes key challenges and recent trends, such as enhanced cross-modal reasoning, integrating expert and cultural knowledge, the demand for automatic toxicity explanations, and handling meme toxicity in low-resource languages. Also, it notes the rising use of Large Language Models (LLMs) and generative AI for detecting and generating toxic memes. Finally, it proposes pathways for advancing toxic meme detection and interpretation.
• 摘要：网络模因是幽默、社会评论和文化表达的渠道，越来越多地用于传播有害信息。过去五年来，对有毒模因的计算分析的研究显着增长，仅有的三项关于计算有毒模因分析的调查仅涵盖了截至 2022 年的著作，导致术语不一致和趋势未被探索。我们的工作通过调查基于内容的有毒模因计算观点并回顾 2024 年初的关键发展来填补这一空白。采用 PRISMA 方法，我们系统地扩展了之前考虑的论文，取得了三重成果。首先，我们调查了 119 篇新论文，分析了 158 篇专注于基于内容的有毒模因分析的计算著作。我们确定了 30 多个用于有毒模因分析的数据集并检查了它们的标记系统。其次，在观察到计算著作中模因毒性定义不明确后，我们引入了一种新的分类法来对模因毒性类型进行分类。我们还注意到，计算任务的范围已经超出了将模因简单地二元分类为有毒或无毒的范围，这表明人们正在转向对毒性进行细致入微的理解。第三，我们在自动研究中确定了模因毒性的三个基于内容的维度：目标、意图和传达策略。我们开发了一个框架来说明这些维度与模因毒性之间的关系。该调查分析了关键挑战和最新趋势，例如增强跨模态推理、整合专家和文化知识、对自动毒性解释的需求以及处理资源匮乏的语言中的模因毒性。此外，它还指出，大型语言模型 (LLM) 和生成式 AI 在检测和生成有毒模因方面的使用越来越多。最后，它提出了推进有毒模因检测和解释的途径。

Title: BvSP: Broad-view Soft Prompting for Few-Shot Aspect Sentiment Quad Prediction

• Authors: Yinhao Bai, Yalan Xie, Xiaoyi Liu, Yuhua Zhao, Zhixin Han, Mengting Hu, Hang Gao, Renhong Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07365
• Pdf URL: https://arxiv.org/pdf/2406.07365
• Copy Paste: [[2406.07365]] BvSP: Broad-view Soft Prompting for Few-Shot Aspect Sentiment Quad Prediction(https://arxiv.org/abs/2406.07365)
• Keywords: language model, prompt
• Abstract: Aspect sentiment quad prediction (ASQP) aims to predict four aspect-based elements, including aspect term, opinion term, aspect category, and sentiment polarity. In practice, unseen aspects, due to distinct data distribution, impose many challenges for a trained neural model. Motivated by this, this work formulates ASQP into the few-shot scenario, which aims for fast adaptation in real applications. Therefore, we first construct a few-shot ASQP dataset (FSQP) that contains richer categories and is more balanced for the few-shot study. Moreover, recent methods extract quads through a generation paradigm, which involves converting the input sentence into a templated target sequence. However, they primarily focus on the utilization of a single template or the consideration of different template orders, thereby overlooking the correlations among various templates. To tackle this issue, we further propose a Broadview Soft Prompting (BvSP) method that aggregates multiple templates with a broader view by taking into account the correlation between the different templates. Specifically, BvSP uses the pre-trained language model to select the most relevant k templates with Jensen-Shannon divergence. BvSP further introduces soft prompts to guide the pre-trained language model using the selected templates. Then, we aggregate the results of multi-templates by voting mechanism. Empirical results demonstrate that BvSP significantly outperforms the stateof-the-art methods under four few-shot settings and other public datasets. Our code and dataset are available at this https URL.
• 摘要：方面情绪四元组预测 (ASQP) 旨在预测四个基于方面的元素，包括方面术语、观点术语、方面类别和情绪极性。在实践中，由于数据分布不同，看不见的方面对训练有素的神经模型提出了许多挑战。受此启发，这项工作将 ASQP 公式化为少样本场景，旨在快速适应实际应用。因此，我们首先构建一个少样本 ASQP 数据集 (FSQP)，其中包含更丰富的类别，并且对于少样本研究更加平衡。此外，最近的方法通过生成范式提取四元组，这涉及将输入句子转换为模板化的目标序列。然而，它们主要关注单个模板的使用或不同模板顺序的考虑，从而忽略了各种模板之间的相关性。为了解决这个问题，我们进一步提出了一种 Broadview 软提示 (BvSP) 方法，该方法通过考虑不同模板之间的相关性以更广阔的视角聚合多个模板。具体来说，BvSP 使用预训练语言模型选择具有 Jensen-Shannon 散度的最相关 k 个模板。BvSP 进一步引入软提示来指导预训练语言模型使用所选模板。然后，我们通过投票机制汇总多模板的结果。实证结果表明，BvSP 在四个少样本设置和其他公共数据集下明显优于最先进的方法。我们的代码和数据集可在此 https URL 上找到。

Title: When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models

• Authors: Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, Yingyan (Celine)Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07368
• Pdf URL: https://arxiv.org/pdf/2406.07368
• Copy Paste: [[2406.07368]] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models(https://arxiv.org/abs/2406.07368)
• Keywords: language model, llm
• Abstract: Autoregressive Large Language Models (LLMs) have achieved impressive performance in language tasks but face two significant bottlenecks: (1) quadratic complexity in the attention module as the number of tokens increases, and (2) limited efficiency due to the sequential processing nature of autoregressive LLMs during generation. While linear attention and speculative decoding offer potential solutions, their applicability and synergistic potential for enhancing autoregressive LLMs remain uncertain. We conduct the first comprehensive study on the efficacy of existing linear attention methods for autoregressive LLMs, integrating them with speculative decoding. We introduce an augmentation technique for linear attention that ensures compatibility with speculative decoding, enabling more efficient training and serving of LLMs. Extensive experiments and ablation studies involving seven existing linear attention models and five encoder/decoder-based LLMs consistently validate the effectiveness of our augmented linearized LLMs. Notably, our approach achieves up to a 6.67 reduction in perplexity on the LLaMA model and up to a 2$\times$ speedup during generation compared to prior linear attention methods. Codes and models are available at this https URL.
• 摘要：自回归大型语言模型 (LLM) 在语言任务中取得了令人瞩目的表现，但也面临两个重大瓶颈：(1) 随着 token 数量的增加，注意力模块的复杂度呈二次方增长，(2) 由于自回归 LLM 在生成过程中的顺序处理特性，效率有限。虽然线性注意力和推测解码提供了潜在的解决方案，但它们的适用性和增强自回归 LLM 的协同潜力仍不确定。我们对现有线性注意力方法对自回归 LLM 的有效性进行了首次全面研究，并将其与推测解码相结合。我们引入了一种线性注意力增强技术，可确保与推测解码兼容，从而更有效地训练和服务 LLM。涉及七种现有线性注意力模型和五种基于编码器/解码器的 LLM 的大量实验和消融研究一致验证了我们增强线性化 LLM 的有效性。值得注意的是，与之前的线性注意方法相比，我们的方法在 LLaMA 模型上实现了高达 6.67 的困惑度降低，并且在生成过程中实现了高达 2$\times$ 的加速。代码和模型可在此 https URL 上找到。

Title: Limited Out-of-Context Knowledge Reasoning in Large Language Models

• Authors: Peng Hu, Changjiang Gao, Ruiqi Gao, Jiajun Chen, Shujian Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07393
• Pdf URL: https://arxiv.org/pdf/2406.07393
• Copy Paste: [[2406.07393]] Limited Out-of-Context Knowledge Reasoning in Large Language Models(https://arxiv.org/abs/2406.07393)
• Keywords: language model, llm, prompt, chat
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities as knowledge bases and significant in-context reasoning capabilities. However, previous work challenges their out-of-context reasoning ability, i.e., the ability to infer information from their training data, instead of from the context or prompt. This paper focuses on a significant facet of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge. We designed a synthetic dataset with seven representative OCKR tasks to systematically assess the OCKR capabilities of LLMs. Using this dataset, we evaluated the LLaMA2-13B-chat model and discovered that its proficiency in this aspect is limited, regardless of whether the knowledge is trained in a separate or adjacent training settings. Moreover, training the model to reason with complete reasoning data did not result in significant improvement. Training the model to perform explicit knowledge retrieval helps in only one of the tasks, indicating that the model's limited OCKR capabilities are due to difficulties in retrieving relevant knowledge. Furthermore, we treat cross-lingual knowledge transfer as a distinct form of OCKR, and evaluate this ability. Our results show that the evaluated model also exhibits limited ability in transferring knowledge across languages. The dataset used in this study is available at this https URL.
• 摘要：大型语言模型 (LLM) 已展现出强大的知识库能力和显著的上下文推理能力。然而，先前的研究挑战了它们的非上下文推理能力，即从训练数据而不是上下文或提示中推断信息的能力。本文重点关注非上下文推理的一个重要方面：非上下文知识推理 (OCKR)，即结合多种知识来推断新知识。我们设计了一个包含七个代表性 OCKR 任务的合成数据集，以系统地评估 LLM 的 OCKR 能力。使用该数据集，我们评估了 LLaMA2-13B-chat 模型，发现无论知识是在单独还是相邻的训练环境中训练的，其在这方面的熟练程度都是有限的。此外，使用完整的推理数据训练模型进行推理并没有带来显著的改进。训练模型执行显性知识检索仅在其中一项任务中有帮助，这表明模型有限的 OCKR 能力是由于难以检索相关知识造成的。此外，我们将跨语言知识转移视为 OCKR 的一种独特形式，并评估其能力。我们的结果表明，所评估的模型在跨语言转移知识方面也表现出有限的能力。本研究中使用的数据集可在此 https URL 上找到。

Title: MINERS: Multilingual Language Models as Semantic Retrievers

• Authors: Genta Indra Winata, Ruochen Zhang, David Ifeoluwa Adelani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07424
• Pdf URL: https://arxiv.org/pdf/2406.07424
• Copy Paste: [[2406.07424]] MINERS: Multilingual Language Models as Semantic Retrievers(https://arxiv.org/abs/2406.07424)
• Keywords: language model
• Abstract: Words have been represented in a high-dimensional vector space that encodes their semantic similarities, enabling downstream applications such as retrieving synonyms, antonyms, and relevant contexts. However, despite recent advances in multilingual language models (LMs), the effectiveness of these models' representations in semantic retrieval contexts has not been comprehensively explored. To fill this gap, this paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual LMs in semantic retrieval tasks, including bitext mining and classification via retrieval-augmented contexts. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches, without requiring any fine-tuning.
• 摘要：单词已被表示在一个高维向量空间中，该空间编码了它们的语义相似性，从而支持下游应用，例如检索同义词、反义词和相关上下文。然而，尽管多语言语言模型 (LM) 取得了最新进展，但这些模型在语义检索上下文中的表示效果尚未得到全面探索。为了填补这一空白，本文介绍了 MINERS，这是一个基准，旨在评估多语言 LM 在语义检索任务中的能力，包括通过检索增强上下文进行双语文本挖掘和分类。我们创建了一个全面的框架来评估 LM 在具有挑战性的跨语言和代码切换环境中检索 200 多种不同语言（包括资源极其匮乏的语言）样本的稳健性。我们的结果表明，仅通过检索语义相似的嵌入就可以获得与最先进方法相媲美的性能，而无需任何微调。

Title: On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations

• Authors: Shiao Meng, Xuming Hu, Aiwei Liu, Fukun Ma, Yawen Yang, Shuang Li, Lijie Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07444
• Pdf URL: https://arxiv.org/pdf/2406.07444
• Copy Paste: [[2406.07444]] On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations(https://arxiv.org/abs/2406.07444)
• Keywords: language model
• Abstract: Driven by the demand for cross-sentence and large-scale relation extraction, document-level relation extraction (DocRE) has attracted increasing research interest. Despite the continuous improvement in performance, we find that existing DocRE models which initially perform well may make more mistakes when merely changing the entity names in the document, hindering the generalization to novel entity names. To this end, we systematically investigate the robustness of DocRE models to entity name variations in this work. We first propose a principled pipeline to generate entity-renamed documents by replacing the original entity names with names from Wikidata. By applying the pipeline to DocRED and Re-DocRED datasets, we construct two novel benchmarks named Env-DocRED and Env-Re-DocRED for robustness evaluation. Experimental results show that both three representative DocRE models and two in-context learned large language models consistently lack sufficient robustness to entity name variations, particularly on cross-sentence relation instances and documents with more entities. Finally, we propose an entity variation robust training method which not only improves the robustness of DocRE models but also enhances their understanding and reasoning capabilities. We further verify that the basic idea of this method can be effectively transferred to in-context learning for DocRE as well.
• 摘要：在跨句和大规模关系抽取需求的驱动下，文档级关系抽取 (DocRE) 引起了越来越多的研究兴趣。尽管性能不断提高，但我们发现现有的 DocRE 模型最初表现良好，但在仅仅更改文档中的实体名称时可能会犯更多错误，从而阻碍了对新实体名称的推广。为此，我们在这项工作中系统地研究了 DocRE 模型对实体名称变化的鲁棒性。我们首先提出了一个原则性的流程，通过将原始实体名称替换为来自 Wikidata 的名称来生成实体重命名的文档。通过将流程应用于 DocRED 和 Re-DocRED 数据集，我们构建了两个名为 Env-DocRED 和 Env-Re-DocRED 的新基准进行鲁棒性评估。实验结果表明，三个代表性 DocRE 模型和两个上下文学习的大型语言模型始终缺乏对实体名称变化的足够鲁棒性，特别是在跨句关系实例和包含更多实体的文档上。最后，我们提出了一种实体变化鲁棒训练方法，不仅可以提高 DocRE 模型的鲁棒性，还可以增强其理解和推理能力。我们进一步验证了该方法的基本思想也可以有效地转移到 DocRE 的上下文学习中。

Title: Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing

• Authors: Mao Li, Frederick Conrad
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07483
• Pdf URL: https://arxiv.org/pdf/2406.07483
• Copy Paste: [[2406.07483]] Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing(https://arxiv.org/abs/2406.07483)
• Keywords: language model, gpt, llm, chat
• Abstract: In the rapidly evolving landscape of Natural Language Processing (NLP), the use of Large Language Models (LLMs) for automated text annotation in social media posts has garnered significant interest. Despite the impressive innovations in developing LLMs like ChatGPT, their efficacy, and accuracy as annotation tools are not well understood. In this paper, we analyze the performance of eight open-source and proprietary LLMs for annotating the stance expressed in social media posts, benchmarking their performance against human annotators' (i.e., crowd-sourced) judgments. Additionally, we investigate the conditions under which LLMs are likely to disagree with human judgment. A significant finding of our study is that the explicitness of text expressing a stance plays a critical role in how faithfully LLMs' stance judgments match humans'. We argue that LLMs perform well when human annotators do, and when LLMs fail, it often corresponds to situations in which human annotators struggle to reach an agreement. We conclude with recommendations for a comprehensive approach that combines the precision of human expertise with the scalability of LLM predictions. This study highlights the importance of improving the accuracy and comprehensiveness of automated stance detection, aiming to advance these technologies for more efficient and unbiased analysis of social media.
• 摘要：在快速发展的自然语言处理 (NLP) 领域，使用大型语言模型 (LLM) 对社交媒体帖子进行自动文本注释引起了人们的极大兴趣。尽管在开发 ChatGPT 等 LLM 方面取得了令人印象深刻的创新，但它们作为注释工具的有效性和准确性尚不清楚。在本文中，我们分析了八个开源和专有 LLM 在注释社交媒体帖子中表达的立场时的性能，并将其性能与人类注释者（即众包）的判断进行对比。此外，我们调查了 LLM 可能与人类判断不一致的情况。我们研究的一个重要发现是，表达立场的文本的明确性在 LLM 的立场判断与人类的立场判断的忠实程度方面起着关键作用。我们认为，当人类注释者表现良好时，LLM 表现良好，而当 LLM 失败时，通常对应于人类注释者难以达成一致的情况。最后，我们提出了一种综合方法的建议，将人类专业知识的精确性与 LLM 预测的可扩展性相结合。这项研究强调了提高自动立场检测的准确性和全面性的重要性，旨在推进这些技术，以便更高效、公正地分析社交媒体。

Title: Paraphrasing in Affirmative Terms Improves Negation Understanding

• Authors: MohammadHossein Rezaei, Eduardo Blanco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.07492
• Pdf URL: https://arxiv.org/pdf/2406.07492
• Copy Paste: [[2406.07492]] Paraphrasing in Affirmative Terms Improves Negation Understanding(https://arxiv.org/abs/2406.07492)
• Keywords: language model
• Abstract: Negation is a common linguistic phenomenon. Yet language models face challenges with negation in many natural language understanding tasks such as question answering and natural language inference. In this paper, we experiment with seamless strategies that incorporate affirmative interpretations (i.e., paraphrases without negation) to make models more robust against negation. Crucially, our affirmative interpretations are obtained automatically. We show improvements with CondaQA, a large corpus requiring reasoning with negation, and five natural language understanding tasks.
• 摘要：否定是一种常见的语言现象。然而，语言模型在许多自然语言理解任务（例如问答和自然语言推理）中都面临着否定的挑战。在本文中，我们尝试了无缝策略，这些策略结合了肯定解释（即没有否定的释义），使模型对否定更具鲁棒性。至关重要的是，我们的肯定解释是自动获得的。我们在 CondaQA（一个需要否定推理的大型语料库）和五个自然语言理解任务中展示了改进。

Title: CADS: A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization

• Authors: Frederic Kirstein, Jan Philip Wahle, Bela Gipp, Terry Ruas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07494
• Pdf URL: https://arxiv.org/pdf/2406.07494
• Copy Paste: [[2406.07494]] CADS: A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization(https://arxiv.org/abs/2406.07494)
• Keywords: language model
• Abstract: Abstractive dialogue summarization is the task of distilling conversations into informative and concise summaries. Although reviews have been conducted on this topic, there is a lack of comprehensive work detailing the challenges of dialogue summarization, unifying the differing understanding of the task, and aligning proposed techniques, datasets, and evaluation metrics with the challenges. This article summarizes the research on Transformer-based abstractive summarization for English dialogues by systematically reviewing 1262 unique research papers published between 2019 and 2024, relying on the Semantic Scholar and DBLP databases. We cover the main challenges present in dialog summarization (i.e., language, structure, comprehension, speaker, salience, and factuality) and link them to corresponding techniques such as graph-based approaches, additional training tasks, and planning strategies, which typically overly rely on BART-based encoder-decoder models. We find that while some challenges, like language, have seen considerable progress, mainly due to training methods, others, such as comprehension, factuality, and salience, remain difficult and hold significant research opportunities. We investigate how these approaches are typically assessed, covering the datasets for the subdomains of dialogue (e.g., meeting, medical), the established automatic metrics and human evaluation approaches for assessing scores and annotator agreement. We observe that only a few datasets span across all subdomains. The ROUGE metric is the most used, while human evaluation is frequently reported without sufficient detail on inner-annotator agreement and annotation guidelines. Additionally, we discuss the possible implications of the recently explored large language models and conclude that despite a potential shift in relevance and difficulty, our described challenge taxonomy remains relevant.
• 摘要：抽象对话摘要是将对话提炼为信息丰富且简洁的摘要的任务。尽管已经对这一主题进行了评论，但缺乏全面的工作来详细说明对话摘要的挑战，统一对该任务的不同理解，并将提出的技术、数据集和评估指标与挑战相结合。本文通过系统地回顾 2019 年至 2024 年期间发表的 1262 篇独特研究论文，总结了基于 Transformer 的英语对话抽象摘要的研究，这些研究依赖于 Semantic Sc​​holar 和 DBLP 数据库。我们涵盖了对话摘要中存在的主要挑战（即语言、结构、理解、说话者、显着性和事实性），并将它们与相应的技术（例如基于图的方法、额外的训练任务和规划策略）联系起来，这些技术通常过度依赖基于 BART 的编码器-解码器模型。我们发现，虽然某些挑战（如语言）取得了长足进步，这主要归功于训练方法，但其他挑战（如理解、事实性和显著性）仍然困难重重，并具有重大的研究机会。我们研究了这些方法的典型评估方式，涵盖对话子域（例如会议、医疗）的数据集、已建立的自动指标和用于评估分数和注释者一致性的人工评估方法。我们观察到只有少数数据集涵盖所有子域。ROUGE 指标是最常用的，而人工评估经常报告，但没有足够详细的内部注释者一致性和注释指南。此外，我们讨论了最近探索的大型语言模型的可能影响，并得出结论，尽管相关性和难度可能会发生变化，但我们描述的挑战分类法仍然具有相关性。

Title: TextGrad: Automatic "Differentiation" via Text

• Authors: Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, James Zou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07496
• Pdf URL: https://arxiv.org/pdf/2406.07496
• Copy Paste: [[2406.07496]] TextGrad: Automatic "Differentiation" via Text(https://arxiv.org/abs/2406.07496)
• Keywords: language model, gpt, llm, prompt
• Abstract: AI is undergoing a paradigm shift, with breakthroughs achieved by systems orchestrating multiple large language models (LLMs) and other complex components. As a result, developing principled and automated optimization methods for compound AI systems is one of the most important new challenges. Neural networks faced a similar challenge in its early days until backpropagation and automatic differentiation transformed the field by making optimization turn-key. Inspired by this, we introduce TextGrad, a powerful framework performing automatic ``differentiation'' via text. TextGrad backpropagates textual feedback provided by LLMs to improve individual components of a compound AI system. In our framework, LLMs provide rich, general, natural language suggestions to optimize variables in computation graphs, ranging from code snippets to molecular structures. TextGrad follows PyTorch's syntax and abstraction and is flexible and easy-to-use. It works out-of-the-box for a variety of tasks, where the users only provide the objective function without tuning components or prompts of the framework. We showcase TextGrad's effectiveness and generality across a diverse range of applications, from question answering and molecule optimization to radiotherapy treatment planning. Without modifying the framework, TextGrad improves the zero-shot accuracy of GPT-4o in Google-Proof Question Answering from $51\%$ to $55\%$, yields $20\%$ relative performance gain in optimizing LeetCode-Hard coding problem solutions, improves prompts for reasoning, designs new druglike small molecules with desirable in silico binding, and designs radiation oncology treatment plans with high specificity. TextGrad lays a foundation to accelerate the development of the next-generation of AI systems.
• 摘要：人工智能正在经历范式转变，通过协调多个大型语言模型 (LLM) 和其他复杂组件的系统取得了突破。因此，为复合人工智能系统开发原则性和自动化的优化方法是最重要的新挑战之一。神经网络在早期也面临着类似的挑战，直到反向传播和自动微分通过使优化成为一站式解决方案而改变了这个领域。受此启发，我们推出了 TextGrad，这是一个通过文本执行自动“微分”的强大框架。TextGrad 反向传播 LLM 提供的文本反馈，以改进复合人工智能系统的各个组件。在我们的框架中，LLM 提供丰富、通用的自然语言建议来优化计算图中的变量，从代码片段到分子结构。TextGrad 遵循 PyTorch 的语法和抽象，灵活且易于使用。它可以开箱即用地用于各种任务，用户只需提供目标函数，而无需调整框架的组件或提示。我们展示了 TextGrad 在各种应用中的有效性和通用性，从问答和分子优化到放射治疗计划。在不修改框架的情况下，TextGrad 将 GPT-4o 在 Google-Proof Question Answering 中的零样本准确率从 $51\%$ 提高到 $55\%$，在优化 LeetCode-Hard 编码问题解决方案时获得了 $20\%$ 的相对性能提升，改进了推理提示，设计了具有理想计算机结合的新型类药小分子，并设计了具有高特异性的放射肿瘤治疗计划。TextGrad 为加速下一代 AI 系统的开发奠定了基础。

Title: THaLLE: Text Hyperlocally Augmented Large Language Extension -- Technical Report

• Authors: KBTG Labs, Danupat Khamnuansin, Atthakorn Petchsod, Anuruth Lertpiya, Pornchanan Balee, Thanawat Lodkaew, Tawunrat Chalothorn, Thadpong Pongthawornkamol, Monchai Lertsutthiwong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] THaLLE: Text Hyperlocally Augmented Large Language Extension -- Technical Report(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have revealed new capabilities and opportunities across the technological landscape. However, the practicality of very large LLMs is challenged by their high compute cost, which does not justify the benefits given their limited capability compared to humans. While smaller, more practical LLMs have shown potential in financial analysis, though they are not yet fully proficient, as evidenced by their near-passing performance on the Chartered Financial Analyst (CFA) exam. In this work, we present Financial Analyst Extension to our Text Hyperlocally Augmented Large Language Extension (THaLLE), a series of 8B LLMs consistently achieving highest performance on mock CFA exams against models of comparable size. We thoroughly document the fine-tuning techniques used to facilitate future research. Additionally, we introduce the use of Flare CFA, a publicly available dataset for evaluating LLMs as a financial advisor.
• 摘要：大型语言模型 (LLM) 的最新进展揭示了技术领域的新功能和机遇。然而，非常大的 LLM 的实用性受到其高计算成本的挑战，鉴于它们与人类相比能力有限，这并不能证明其好处。虽然规模较小，但更实用的 LLM 已显示出在金融分析方面的潜力，尽管它们尚未完全熟练，这一点从它们在特许金融分析师 (CFA) 考试中接近通过的表现就可以看出。在这项工作中，我们将金融分析师扩展引入我们的文本超本地增强大型语言扩展 (THaLLE)，这是一系列 8B LLM，在模拟 CFA 考试中与同等规模的模型相比始终取得最高成绩。我们详细记录了用于促进未来研究的微调技术。此外，我们还介绍了 Flare CFA 的使用，这是一个用于评估 LLM 作为财务顾问的公开数据集。

Title: Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

• Authors: Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, Weizhu Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07522
• Pdf URL: https://arxiv.org/pdf/2406.07522
• Copy Paste: [[2406.07522]] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling(https://arxiv.org/abs/2406.07522)
• Keywords: language model, prompt
• Abstract: Efficiently modeling sequences with infinite context length has been a long-standing problem. Past works suffer from either the quadratic computation complexity or the limited extrapolation ability on length generalization. In this work, we present Samba, a simple hybrid architecture that layer-wise combines Mamba, a selective State Space Model (SSM), with Sliding Window Attention (SWA). Samba selectively compresses a given sequence into recurrent hidden states while still maintaining the ability to precisely recall memories with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T training tokens and show that Samba substantially outperforms the state-of-the-art models based on pure attention or SSMs on a wide range of benchmarks. When trained on 4K length sequences, Samba can be efficiently extrapolated to 256K context length with perfect memory recall and show improved token predictions up to 1M context length. As a linear-time sequence model, Samba enjoys a 3.73x higher throughput compared to Transformers with grouped-query attention when processing user prompts of 128K length, and 3.64x speedup when generating 64K tokens with unlimited streaming. A sample implementation of Samba is publicly available in this https URL.
• 摘要：有效地对具有无限上下文长度的序列进行建模一直是一个长期存在的问题。过去的工作要么受到二次计算复杂性的影响，要么在长度泛化方面外推能力有限。在这项工作中，我们提出了 Samba，这是一种简单的混合架构，它逐层结合了 Mamba（一种选择性状态空间模型 (SSM)）和滑动窗口注意 (SWA)。Samba 选择性地将给定序列压缩为循环隐藏状态，同时仍保持使用注意机制精确回忆记忆的能力。我们将 Samba 扩展到 3.8B 参数和 3.2T 训练标记，并表明 Samba 在各种基准测试中的表现大大优于基于纯注意或 SSM 的最先进的模型。在 4K 长度序列上进行训练时，Samba 可以有效地外推到 256K 上下文长度，具有完美的记忆回忆，并且显示出高达 1M 上下文长度的改进的标记预测。作为线性时间序列模型，Samba 在处理长度为 128K 的用户提示时，与具有分组查询注意机制的 Transformers 相比，吞吐量提高了 3.73 倍；在生成 64K 令牌且不限流量的情况下，速度提高了 3.64 倍。Samba 的示例实现在此 https URL 中公开提供。

Title: Simple and Effective Masked Diffusion Language Models

• Authors: Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, Volodymyr Kuleshov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.07524
• Pdf URL: https://arxiv.org/pdf/2406.07524
• Copy Paste: [[2406.07524]] Simple and Effective Masked Diffusion Language Models(https://arxiv.org/abs/2406.07524)
• Keywords: language model
• Abstract: While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We release our code at: this https URL
• 摘要：虽然扩散模型擅长生成高质量图像，但先前的研究报告称，在语言建模中，扩散和自回归 (AR) 方法之间存在显著的性能差距。在这项工作中，我们表明简单的掩蔽离散扩散比以前认为的更有效。我们应用了一种有效的训练方法，可以提高掩蔽扩散模型的性能，并得出一个简化的 Rao-Blackwellized 目标，从而带来额外的改进。我们的目标形式简单——它是经典掩蔽语言建模损失的混合——可用于训练仅编码器的语言模型，这些模型可以接受高效的采样器，包括可以像传统语言模型一样半自回归生成任意长度文本的采样器。在语言建模基准上，一系列使用现代工程实践训练的掩蔽扩散模型在扩散模型中达到了新的最先进水平，并接近 AR 困惑度。我们在以下网址发布我们的代码：此 https URL

Title: Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena

• Authors: Aidar Myrzakhan, Sondos Mahmoud Bsharat, Zhiqiang Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.07545
• Pdf URL: https://arxiv.org/pdf/2406.07545
• Copy Paste: [[2406.07545]] Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena(https://arxiv.org/abs/2406.07545)
• Keywords: language model, gpt, llm
• Abstract: Multiple-choice questions (MCQ) are frequently used to assess large language models (LLMs). Typically, an LLM is given a question and selects the answer deemed most probable after adjustments for factors like length. Unfortunately, LLMs may inherently favor certain answer choice IDs, such as A/B/C/D, due to inherent biases of priori unbalanced probabilities, influencing the prediction of answers based on these IDs. Previous research has introduced methods to reduce this ''selection bias'' by simply permutating options on a few test samples and applying to new ones. Another problem of MCQ is the lottery ticket choice by ''random guessing''. The LLM does not learn particular knowledge, but the option is guessed correctly. This situation is especially serious for those small-scale LLMs. To address them, a more thorough approach involves shifting from MCQ to open-style questions, which can fundamentally eliminate selection bias and random guessing issues. However, transitioning causes its own set of challenges in (1) identifying suitable open-style questions and (2) validating the correctness of LLM open-style responses against human-annotated ground-truths. This work aims to tackle these significant difficulties, and establish a new LLM evaluation benchmark through entirely open-style questions. Consequently, we introduce the Open-LLM-Leaderboard to track various LLMs' performance and reflect true capability of them, such as GPT-4o/4/3.5, Claude 3, Gemini, etc. Our code and dataset are available at this https URL.
• 摘要：多项选择题 (MCQ) 经常用于评估大型语言模型 (LLM)。通常，LLM 会收到一个问题，并在调整长度等因素后选择最有可能的答案。不幸的是，由于先验不平衡概率的固有偏差，LLM 可能天生就偏向某些答案选项 ID，例如 A/B/C/D，从而影响基于这些 ID 的答案预测。先前的研究引入了减少这种“选择偏差”的方法，只需在几个测试样本上排列选项并应用于新样本即可。MCQ 的另一个问题是通过“随机猜测”进行彩票选择。LLM 不会学习特定的知识，但选项被猜对了。这种情况对于那些小规模的 LLM 来说尤其严重。为了解决这些问题，更彻底的方法是将 MCQ 转变为开放式问题，这可以从根本上消除选择偏差和随机猜测问题。然而，转型本身也带来了一系列挑战，包括 (1) 确定合适的开放式问题和 (2) 根据人工注释的事实验证 LLM 开放式回答的正确性。这项工作旨在解决这些重大困难，并通过完全开放式的问题建立新的 LLM 评估基准。因此，我们引入了 Open-LLM-Leaderboard 来跟踪各种 LLM 的表现并反映它们的真实能力，例如 GPT-4o/4/3.5、Claude 3、Gemini 等。我们的代码和数据集可在此 https URL 上找到。