2024-07-18

Title: Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasing

• Authors: Shafiuddin Rehan Ahmed, Zhiyong Eric Wang, George Arthur Baker, Kevin Stowe, James H. Martin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.11988
• Pdf URL: https://arxiv.org/pdf/2407.11988
• Copy Paste: [[2407.11988]] Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasing(https://arxiv.org/abs/2407.11988)
• Keywords: gpt, chat
• Abstract: The most popular Cross-Document Event Coreference Resolution (CDEC) datasets fail to convey the true difficulty of the task, due to the lack of lexical diversity between coreferring event triggers (words or phrases that refer to an event). Furthermore, there is a dearth of event datasets for figurative language, limiting a crucial avenue of research in event comprehension. We address these two issues by introducing ECB+META, a lexically rich variant of Event Coref Bank Plus (ECB+) for CDEC on symbolic and metaphoric language. We use ChatGPT as a tool for the metaphoric transformation of sentences in the documents of ECB+, then tag the original event triggers in the transformed sentences in a semi-automated manner. In this way, we avoid the re-annotation of expensive coreference links. We present results that show existing methods that work well on ECB+ struggle with ECB+META, thereby paving the way for CDEC research on a much more challenging dataset. Code/data: this https URL
• 摘要：最流行的跨文档事件共指解析 (CDEC) 数据集无法传达任务的真正难度，因为共指事件触发器（指代事件的单词或短语）之间缺乏词汇多样性。此外，比喻性语言的事件数据集匮乏，限制了事件理解研究的重要途径。我们通过引入 ECB+META 来解决这两个问题，ECB+META 是 Event Coref Bank Plus (ECB+) 的词汇丰富变体，用于符号和隐喻语言的 CDEC。我们使用 ChatGPT 作为 ECB+ 文档中句子隐喻转换的工具，然后以半自动化方式标记转换后的句子中的原始事件触发器。通过这种方式，我们避免了重新注释昂贵的共指链接。我们展示的结果显示，现有的在 ECB+ 上运行良好的方法在 ECB+META 上遇到了困难，从而为在更具挑战性的数据集上进行 CDEC 研究铺平了道路。代码/数据：此 https URL

Title: LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems

• Authors: Jisoo Mok, Mohammad Kachuee, Shuyang Dai, Shayan Ray, Tara Taghavi, Sungroh Yoon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12016
• Pdf URL: https://arxiv.org/pdf/2407.12016
• Copy Paste: [[2407.12016]] LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems(https://arxiv.org/abs/2407.12016)
• Keywords: language model, llm, prompt, agent
• Abstract: Task-orientated conversational agents interact with users and assist them via leveraging external APIs. A typical task-oriented conversational system can be broken down into three phases: external API selection, argument filling, and response generation. The focus of our work is the task of argument filling, which is in charge of accurately providing arguments required by the selected API. Upon comprehending the dialogue history and the pre-defined API schema, the argument filling task is expected to provide the external API with the necessary information to generate a desirable agent action. In this paper, we study the application of Large Language Models (LLMs) for the problem of API argument filling task. Our initial investigation reveals that LLMs require an additional grounding process to successfully perform argument filling, inspiring us to design training and prompting frameworks to ground their responses. Our experimental results demonstrate that when paired with proposed techniques, the argument filling performance of LLMs noticeably improves, paving a new way toward building an automated argument filling framework.
• 摘要：面向任务的对话代理通过利用外部 API 与用户交互并为他们提供帮助。典型的面向任务的对话系统可以分为三个阶段：外部 API 选择、参数填充和响应生成。我们工作的重点是参数填充任务，该任务负责准确提供所选 API 所需的参数。在理解对话历史和预定义的 API 模式后，参数填充任务有望为外部 API 提供生成所需代理操作所需的信息。在本文中，我们研究了大型语言模型 (LLM) 在 API 参数填充任务问题中的应用。我们的初步调查显示，LLM 需要额外的基础过程才能成功执行参数填充，这启发我们设计训练和提示框架来为其响应奠定基础。我们的实验结果表明，当与所提出的技术结合使用时，LLM 的参数填充性能显着提高，为构建自动参数填充框架铺平了新道路。

Title: Follow-Up Questions Improve Documents Generated by Large Language Models

• Authors: Bernadette J Tix
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12017
• Pdf URL: https://arxiv.org/pdf/2407.12017
• Copy Paste: [[2407.12017]] Follow-Up Questions Improve Documents Generated by Large Language Models(https://arxiv.org/abs/2407.12017)
• Keywords: language model, prompt
• Abstract: This study investigates the impact of Large Language Models generating follow up questions in response to user requests for short text documents. Users provided prompts requesting documents they would like the AI to produce. The AI then generated questions to clarify the user needs before generating the requested documents. Users answered the questions and then indicated their preference between a document generated using both the initial prompt and the questions and answers, and a document generated using only the initial prompt, and gave feedback about their experience with the question-answering process. The findings of this study show clear benefits to question-asking both in document preference and in the qualitative user experience.
• 摘要：本研究调查了大型语言模型生成后续问题以响应用户对短文本文档的请求的影响。用户提供提示，请求他们希望人工智能生成的文档。然后，人工智能在生成所请求的文档之前生成问题以澄清用户的需求。用户回答问题，然后表明他们更喜欢使用初始提示和问题和答案生成的文档，还是仅使用初始提示生成的文档，并就他们在问答过程中的体验提供反馈。本研究的结果显示，提问在文档偏好和定性用户体验方面都有明显的好处。

Title: DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model

• Authors: Shezheng Song, Shasha Li, Jie Yu, Shan Zhao, Xiaopeng Li, Jun Ma, Xiaodong Liu, Zhuo Li, Xiaoguang Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12019
• Pdf URL: https://arxiv.org/pdf/2407.12019
• Copy Paste: [[2407.12019]] DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model(https://arxiv.org/abs/2407.12019)
• Keywords: language model, gpt, llm, chat
• Abstract: Our study delves into Multimodal Entity Linking, aligning the mention in multimodal information with entities in knowledge base. Existing methods are still facing challenges like ambiguous entity representations and limited image information utilization. Thus, we propose dynamic entity extraction using ChatGPT, which dynamically extracts entities and enhances datasets. We also propose a method: Dynamically Integrate Multimodal information with knowledge base (DIM), employing the capability of the Large Language Model (LLM) for visual understanding. The LLM, such as BLIP-2, extracts information relevant to entities in the image, which can facilitate improved extraction of entity features and linking them with the dynamic entity representations provided by ChatGPT. The experiments demonstrate that our proposed DIM method outperforms the majority of existing methods on the three original datasets, and achieves state-of-the-art (SOTA) on the dynamically enhanced datasets (Wiki+, Rich+, Diverse+). For reproducibility, our code and collected datasets are released on \url{this https URL}.
• 摘要：我们的研究深入研究了多模态实体链接，将多模态信息中的提及与知识库中的实体对齐。现有方法仍然面临着诸如实体表示不明确和图像信息利用有限等挑战。因此，我们提出使用 ChatGPT 进行动态实体提取，它可以动态提取实体并增强数据集。我们还提出了一种方法：动态集成多模态信息与知识库 (DIM)，利用大型语言模型 (LLM) 的功能进行视觉理解。LLM（例如 BLIP-2）提取与图像中实体相关的信息，这可以促进实体特征的改进提取并将它们与 ChatGPT 提供的动态实体表示链接起来。实验表明，我们提出的 DIM 方法在三个原始数据集上的表现优于大多数现有方法，并在动态增强数据集（Wiki+、Rich+、Diverse+）上达到了最新水平 (SOTA)。为了可重复性，我们的代码和收集的数据集在 \url{此 https URL} 上发布。

Title: Adaptive Draft-Verification for Efficient Large Language Model Decoding

• Authors: Xukun Liu, Bowen Lei, Ruqi Zhang, Dongkuan Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12021
• Pdf URL: https://arxiv.org/pdf/2407.12021
• Copy Paste: [[2407.12021]] Adaptive Draft-Verification for Efficient Large Language Model Decoding(https://arxiv.org/abs/2407.12021)
• Keywords: language model, llm
• Abstract: Large language model (LLM) decoding involves generating a sequence of tokens based on a given context, where each token is predicted one at a time using the model's learned probabilities. The typical autoregressive decoding method requires a separate forward pass through the model for each token generated, which is computationally inefficient and poses challenges for deploying LLMs in latency-sensitive scenarios. The main limitations of current decoding methods stem from their inefficiencies and resource demands. Existing approaches either necessitate fine-tuning smaller models, which is resource-intensive, or rely on fixed retrieval schemes to construct drafts for the next tokens, which lack adaptability and fail to generalize across different models and contexts. To address these issues, we introduce a novel methodology called ADED, which accelerates LLM decoding without requiring fine-tuning. Our approach involves an adaptive draft-verification process that evolves over time to improve efficiency. We utilize a tri-gram matrix-based LLM representation to dynamically approximate the output distribution of the LLM, allowing the model to adjust to changing token probabilities during the decoding process. Additionally, we implement a draft construction mechanism that effectively balances exploration and exploitation, ensuring that the drafts generated are both diverse and close to the true output distribution of the LLM. The importance of this design lies in its ability to optimize the draft distribution adaptively, leading to faster and more accurate decoding. Through extensive experiments on various benchmark datasets and LLM architectures, we demonstrate that ADED significantly accelerates the decoding process while maintaining high accuracy, making it suitable for deployment in a wide range of practical applications.
• 摘要：大型语言模型 (LLM) 解码涉及根据给定上下文生成一系列标记，其中使用模型的学习概率一次预测一个标记。典型的自回归解码方法需要对生成的每个标记进行单独的前向传递，这在计算上效率低下，并且对在延迟敏感场景中部署 LLM 构成挑战。当前解码方法的主要限制源于其效率低下和资源需求。现有方法要么需要对较小的模型进行微调，这需要大量资源，要么依靠固定的检索方案来构建下一个标记的草稿，这缺乏适应性并且无法在不同的模型和上下文中推广。为了解决这些问题，我们引入了一种名为 ADED 的新方法，它可以加速 LLM 解码而无需微调。我们的方法涉及一个自适应草稿验证过程，该过程会随着时间的推移而发展以提高效率。我们利用基于三元矩阵的 LLM 表示来动态近似 LLM 的输出分布，从而使模型能够在解码过程中适应不断变化的标记概率。此外，我们实施了一种草稿构建机制，可以有效地平衡探索和开发，确保生成的草稿既多样化又接近 LLM 的真实输出分布。这种设计的重要性在于它能够自适应地优化草稿分布，从而实现更快、更准确的解码。通过在各种基准数据集和 LLM 架构上进行大量实验，我们证明 ADED 在保持高精度的同时显著加快了解码过程，使其适合部署在广泛的实际应用中。

Title: ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation

• Authors: Peiyang Wu, Nan Guo, Xiao Xiao, Wenming Li, Xiaochun Ye, Dongrui Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12022
• Pdf URL: https://arxiv.org/pdf/2407.12022
• Copy Paste: [[2407.12022]] ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation(https://arxiv.org/abs/2407.12022)
• Keywords: language model, llm
• Abstract: Recently, large language models (LLMs) have demonstrated excellent performance in understanding human instructions and generating code, which has inspired researchers to explore the feasibility of generating RTL code with LLMs. However, the existing approaches to fine-tune LLMs on RTL codes typically are conducted on fixed datasets, which do not fully stimulate the capability of LLMs and require large amounts of reference data. To mitigate these issues , we introduce a simple yet effective iterative training paradigm named ITERTL. During each iteration, samples are drawn from the model trained in the previous cycle. Then these new samples are employed for training in this loop. Through this iterative approach, the distribution mismatch between the model and the training samples is reduced. Additionally, the model is thus enabled to explore a broader generative space and receive more comprehensive feedback. Theoretical analyses are conducted to investigate the mechanism of the effectiveness. Experimental results show the model trained through our proposed approach can compete with and even outperform the state-of-the-art (SOTA) open-source model with nearly 37\% reference samples, achieving remarkable 42.9\% and 62.2\% pass@1 rate on two VerilogEval evaluation datasets respectively. While using the same amount of reference samples, our method can achieved a relative improvement of 16.9\% and 12.5\% in pass@1 compared to the non-iterative method. This study facilitates the application of LLMs for generating RTL code in practical scenarios with limited data.
• 摘要：近年来，大型语言模型（LLM）在理解人类指令和生成代码方面表现出色，这启发了研究人员探索利用LLM生成RTL代码的可行性。然而，现有的在RTL代码上微调LLM的方法通常是在固定的数据集上进行的，这并不能充分激发LLM的能力，并且需要大量的参考数据。为了缓解这些问题，我们引入了一个简单而有效的迭代训练范式——ITERTL。在每次迭代中，从上一次训练的模型中抽取样本。然后，这些新样本被用于本次循环的训练。通过这种迭代方法，模型和训练样本之间的分布不匹配减少了。此外，这使模型能够探索更广阔的生成空间并获得更全面的反馈。并通过理论分析来探究该算法有效性的机制。实验结果表明，通过我们提出的方法训练的模型可以与最先进的 (SOTA) 开源模型相媲美，甚至超越后者，参考样本接近 37%，在两个 VerilogEval 评估数据集上分别实现了 42.9% 和 62.2% 的 pass@1 率。在使用相同数量的参考样本时，与非迭代方法相比，我们的方法在 pass@1 方面可以实现 16.9% 和 12.5% 的相对改进。这项研究有助于在数据有限的实际场景中应用 LLM 来生成 RTL 代码。

Title: CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models

• Authors: Zhong-Zhi Li, Ming-Liang Zhang, Fei Yin, Zhi-Long Ji, Jin-Feng Bai, Zhen-Ru Pan, Fan-Hu Zeng, Jian Xu, Jia-Xin Zhang, Cheng-Lin Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12023
• Pdf URL: https://arxiv.org/pdf/2407.12023
• Copy Paste: [[2407.12023]] CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models(https://arxiv.org/abs/2407.12023)
• Keywords: language model, gpt
• Abstract: Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available.
• 摘要：随着多模态大型语言模型的快速发展，对其多模态数学能力的评估持续受到广泛关注。尽管MathVista等数据集提出了多模态场景下数学能力评估的基准，但在中文K12教育背景下，仍然缺乏相应的评估工具和细粒度评估数据集。为了系统地评估多模态大型模型解决中文多模态数学问题的能力，我们提出了中文多模态数学技能评估基准CMMaTH，训练了23k多模态K12数学相关问题，形成了迄今为止最大的中文多模态数学问题基准。CMMaTH问题从小学到高中水平，在问题类型、解决方案目标、视觉元素、详细知识点和标准解决方案注释方面提供了更多的多样性。我们已经构建了一个与CMMaTH数据集集成的开源工具GradeGPT，以便于稳定、快速和免费的模型评估。我们的数据和代码都是可用的。

Title: The Pitfalls of Publishing in the Age of LLMs: Strange and Surprising Adventures with a High-Impact NLP Journal

• Authors: Rakesh M. Verma, Nachum Dershowitz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12026
• Pdf URL: https://arxiv.org/pdf/2407.12026
• Copy Paste: [[2407.12026]] The Pitfalls of Publishing in the Age of LLMs: Strange and Surprising Adventures with a High-Impact NLP Journal(https://arxiv.org/abs/2407.12026)
• Keywords: llm
• Abstract: We show the fraught side of the academic publishing realm and illustrate it through a recent case study with an NLP journal.
• 摘要：我们展示了学术出版领域的紧张一面，并通过最近对 NLP 期刊的案例研究进行了说明。

Title: TreeSeg: Hierarchical Topic Segmentation of Large Transcripts

• Authors: Dimitrios C. Gklezakos, Timothy Misiak, Diamond Bishop
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12028
• Pdf URL: https://arxiv.org/pdf/2407.12028
• Copy Paste: [[2407.12028]] TreeSeg: Hierarchical Topic Segmentation of Large Transcripts(https://arxiv.org/abs/2407.12028)
• Keywords: language model, llm
• Abstract: From organizing recorded videos and meetings into chapters, to breaking down large inputs in order to fit them into the context window of commoditized Large Language Models (LLMs), topic segmentation of large transcripts emerges as a task of increasing significance. Still, accurate segmentation presents many challenges, including (a) the noisy nature of the Automatic Speech Recognition (ASR) software typically used to obtain the transcripts, (b) the lack of diverse labeled data and (c) the difficulty in pin-pointing the ground-truth number of segments. In this work we present TreeSeg, an approach that combines off-the-shelf embedding models with divisive clustering, to generate hierarchical, structured segmentations of transcripts in the form of binary trees. Our approach is robust to noise and can handle large transcripts efficiently. We evaluate TreeSeg on the ICSI and AMI corpora, demonstrating that it outperforms all baselines. Finally, we introduce TinyRec, a small-scale corpus of manually annotated transcripts, obtained from self-recorded video sessions.
• 摘要：从将录制的视频和会议组织成章节，到将大量输入分解以使其适合商品化的大型语言模型 (LLM) 的上下文窗口，大型转录本的主题分割成为一项越来越重要的任务。然而，准确的分割仍面临许多挑战，包括 (a) 通常用于获取转录本的自动语音识别 (ASR) 软件的噪声性质，(b) 缺乏多样化的标记数据，以及 (c) 难以确定片段的真实数量。在这项工作中，我们提出了 TreeSeg，这是一种将现成的嵌入模型与分裂聚类相结合的方法，以二叉树的形式生成转录本的分层结构化分割。我们的方法对噪声具有鲁棒性，可以有效处理大型转录本。我们在 ICSI 和 AMI 语料库上评估了 TreeSeg，表明它优于所有基线。最后，我们介绍 TinyRec，这是一个从自录视频会话中获得的小型手动注释记录语料库。

Title: Understanding Transformers via N-gram Statistics

• Authors: Timothy Nguyen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12034
• Pdf URL: https://arxiv.org/pdf/2407.12034
• Copy Paste: [[2407.12034]] Understanding Transformers via N-gram Statistics(https://arxiv.org/abs/2407.12034)
• Keywords: language model, llm
• Abstract: Transformer based large-language models (LLMs) display extreme proficiency with language yet a precise understanding of how they work remains elusive. One way of demystifying transformer predictions would be to describe how they depend on their context in terms of simple template functions. This paper takes a first step in this direction by considering families of functions (i.e. rules) formed out of simple N-gram based statistics of the training data. By studying how well these rulesets approximate transformer predictions, we obtain a variety of novel discoveries: a simple method to detect overfitting during training without using a holdout set, a quantitative measure of how transformers progress from learning simple to more complex statistical rules over the course of training, a model-variance criterion governing when transformer predictions tend to be described by N-gram rules, and insights into how well transformers can be approximated by N-gram rulesets in the limit where these rulesets become increasingly complex. In this latter direction, we find that for 78% of LLM next-token distributions on TinyStories, their top-1 predictions agree with those provided by our N-gram rulesets.
• 摘要：基于 Transformer 的大型语言模型 (LLM) 表现出极其熟练的语言能力，但对其工作原理的准确理解仍然难以捉摸。揭开 Transformer 预测神秘面纱的一种方法是用简单的模板函数来描述它们如何依赖于它们的上下文。本文朝这个方向迈出了第一步，考虑了由简单的基于 N-gram 的训练数据统计数据形成的函数族 (即规则)。通过研究这些规则集对 Transformer 预测的近似程度，我们获得了各种新发现：一种无需使用保留集即可在训练期间检测过度拟合的简单方法、一种定量测量 Transformer 在训练过程中如何从学习简单统计规则进展到更复杂的统计规则、一种控制 Transformer 预测何时倾向于用 N-gram 规则描述的模型方差标准，以及在这些规则集变得越来越复杂的极限下，Transformer 能被 N-gram 规则集近似的程度的见解。在后一个方向，我们发现对于 TinyStories 上 78% 的 LLM 下一个标记分布，它们的 top-1 预测与我们的 N-gram 规则集提供的预测一致。

Title: Exploring Advanced Large Language Models with LLMsuite

• Authors: Giorgio Roffo
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12036
• Pdf URL: https://arxiv.org/pdf/2407.12036
• Copy Paste: [[2407.12036]] Exploring Advanced Large Language Models with LLMsuite(https://arxiv.org/abs/2407.12036)
• Keywords: language model, gpt, llm, chat, retrieval augmented generation
• Abstract: This tutorial explores the advancements and challenges in the development of Large Language Models (LLMs) such as ChatGPT and Gemini. It addresses inherent limitations like temporal knowledge cutoffs, mathematical inaccuracies, and the generation of incorrect information, proposing solutions like Retrieval Augmented Generation (RAG), Program-Aided Language Models (PAL), and frameworks such as ReAct and LangChain. The integration of these techniques enhances LLM performance and reliability, especially in multi-step reasoning and complex task execution. The paper also covers fine-tuning strategies, including instruction fine-tuning, parameter-efficient methods like LoRA, and Reinforcement Learning from Human Feedback (RLHF) as well as Reinforced Self-Training (ReST). Additionally, it provides a comprehensive survey of transformer architectures and training techniques for LLMs. The toolbox for implementing these techniques is publicly available at this https URL
• 摘要：本教程探讨了 ChatGPT 和 Gemini 等大型语言模型 (LLM) 开发中的进步和挑战。它解决了固有的局限性，例如时间知识截断、数学不准确性和错误信息的生成，并提出了检索增强生成 (RAG)、程序辅助语言模型 (PAL) 和 ReAct 和 LangChain 等框架等解决方案。这些技术的集成提高了 LLM 的性能和可靠性，尤其是在多步推理和复杂任务执行中。本文还介绍了微调策略，包括指令微调、LoRA 等参数高效方法、从人类反馈中强化学习 (RLHF) 以及强化自我训练 (ReST)。此外，它还全面介绍了 LLM 的转换器架构和训练技术。实现这些技术的工具箱可在此 https URL 上公开获取

Title: The Art of Saying No: Contextual Noncompliance in Language Models

• Authors: Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.12043
• Pdf URL: https://arxiv.org/pdf/2407.12043
• Copy Paste: [[2407.12043]] The Art of Saying No: Contextual Noncompliance in Language Models(https://arxiv.org/abs/2407.12043)
• Keywords: language model, gpt, prompt, chat
• Abstract: Chat-based language models are designed to be helpful, yet they should not comply with every user request. While most existing work primarily focuses on refusal of "unsafe" queries, we posit that the scope of noncompliance should be broadened. We introduce a comprehensive taxonomy of contextual noncompliance describing when and how models should not comply with user requests. Our taxonomy spans a wide range of categories including incomplete, unsupported, indeterminate, and humanizing requests (in addition to unsafe requests). To test noncompliance capabilities of language models, we use this taxonomy to develop a new evaluation suite of 1000 noncompliance prompts. We find that most existing models show significantly high compliance rates in certain previously understudied categories with models like GPT-4 incorrectly complying with as many as 30% of requests. To address these gaps, we explore different training strategies using a synthetically-generated training set of requests and expected noncompliant responses. Our experiments demonstrate that while direct finetuning of instruction-tuned models can lead to both over-refusal and a decline in general capabilities, using parameter efficient methods like low rank adapters helps to strike a good balance between appropriate noncompliance and other capabilities.
• 摘要：基于聊天的语言模型旨在提供帮助，但它们不应遵守每个用户请求。虽然大多数现有工作主要侧重于拒绝“不安全”查询，但我们认为不合规的范围应该扩大。我们引入了一个全面的上下文不合规分类法，描述了模型何时以及如何不遵守用户请求。我们的分类法涵盖了广泛的类别，包括不完整、不受支持、不确定和人性化请求（除了不安全的请求）。为了测试语言模型的不合规能力，我们使用此分类法开发了一个包含 1000 个不合规提示的新评估套件。我们发现大多数现有模型在某些以前研究不足的类别中表现出非常高的合规率，而 GPT-4 等模型错误地遵守了多达 30% 的请求。为了解决这些差距，我们使用合成生成的请求训练集和预期的不合规响应探索了不同的训练策略。我们的实验表明，虽然对指令调整模型进行直接微调可能会导致过度拒绝和一般能力的下降，但使用低秩适配器等参数有效方法有助于在适当的不合规和其他能力之间取得良好的平衡。

Title: NinjaLLM: Fast, Scalable and Cost-effective RAG using Amazon SageMaker and AWS Trainium and Inferentia2

• Authors: Tengfei Xue, Xuefeng Li, Roman Smirnov, Tahir Azim, Arash Sadrieh, Babak Pahlavan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12057
• Pdf URL: https://arxiv.org/pdf/2407.12057
• Copy Paste: [[2407.12057]] NinjaLLM: Fast, Scalable and Cost-effective RAG using Amazon SageMaker and AWS Trainium and Inferentia2(https://arxiv.org/abs/2407.12057)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) techniques are widely used today to retrieve and present information in a conversational format. This paper presents a set of enhancements to traditional RAG techniques, focusing on large language models (LLMs) fine-tuned and hosted on AWS Trainium and Inferentia2 AI chips via SageMaker. These chips are characterized by their elasticity, affordability, and efficient performance for AI compute tasks. Besides enabling deployment on these chips, this work aims to improve tool usage, add citation capabilities, and mitigate the risks of hallucinations and unsafe responses due to context bias. We benchmark our RAG system's performance on the Natural Questions and HotPotQA datasets, achieving an accuracy of 62% and 59% respectively, exceeding other models such as DBRX and Mixtral Instruct.
• 摘要：如今，检索增强生成 (RAG) 技术被广泛用于以对话形式检索和呈现信息。本文介绍了一系列对传统 RAG 技术的增强，重点关注通过 SageMaker 在 AWS Trainium 和 Inferentia2 AI 芯片上微调和托管的大型语言模型 (LLM)。这些芯片的特点是其弹性、可负担性和高效的 AI 计算任务性能。除了在这些芯片上实现部署外，这项工作还旨在提高工具使用率、增加引用功能并降低由于上下文偏差导致幻觉和不安全反应的风险。我们在 Natural Questions 和 HotPotQA 数据集上对我们的 RAG 系统的性能进行了基准测试，分别实现了 62% 和 59% 的准确率，超过了 DBRX 和 Mixtral Instruct 等其他模型。

Title: Identifying Speakers in Dialogue Transcripts: A Text-based Approach Using Pretrained Language Models

• Authors: Minh Nguyen, Franck Dernoncourt, Seunghyun Yoon, Hanieh Deilamsalehy, Hao Tan, Ryan Rossi, Quan Hung Tran, Trung Bui, Thien Huu Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12094
• Pdf URL: https://arxiv.org/pdf/2407.12094
• Copy Paste: [[2407.12094]] Identifying Speakers in Dialogue Transcripts: A Text-based Approach Using Pretrained Language Models(https://arxiv.org/abs/2407.12094)
• Keywords: language model
• Abstract: We introduce an approach to identifying speaker names in dialogue transcripts, a crucial task for enhancing content accessibility and searchability in digital media archives. Despite the advancements in speech recognition, the task of text-based speaker identification (SpeakerID) has received limited attention, lacking large-scale, diverse datasets for effective model training. Addressing these gaps, we present a novel, large-scale dataset derived from the MediaSum corpus, encompassing transcripts from a wide range of media sources. We propose novel transformer-based models tailored for SpeakerID, leveraging contextual cues within dialogues to accurately attribute speaker names. Through extensive experiments, our best model achieves a great precision of 80.3\%, setting a new benchmark for SpeakerID. The data and code are publicly available here: \url{this https URL}
• 摘要：我们介绍了一种识别对话记录中说话者姓名的方法，这是一项关键任务，可提高数字媒体档案中内容的可访问性和可搜索性。尽管语音识别取得了进展，但基于文本的说话者识别 (SpeakerID) 任务受到的关注有限，缺乏大规模、多样化的数据集来进行有效的模型训练。为了解决这些差距，我们提出了一个源自 MediaSum 语料库的新型大规模数据集，其中包含来自各种媒体来源的记录。我们提出了针对 SpeakerID 量身定制的新型基于 Transformer 的模型，利用对话中的上下文线索来准确归因说话者姓名。通过大量实验，我们的最佳模型实现了 80.3% 的高精度，为 SpeakerID 树立了新的标杆。数据和代码可在此处公开获取：\url{此 https URL}

Title: Better RAG using Relevant Information Gain

• Authors: Marc Pickett, Jeremy Hartman, Ayan Kumar Bhowmick, Raquib-ul Alam, Aditya Vempaty
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12101
• Pdf URL: https://arxiv.org/pdf/2407.12101
• Copy Paste: [[2407.12101]] Better RAG using Relevant Information Gain(https://arxiv.org/abs/2407.12101)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: A common way to extend the memory of large language models (LLMs) is by retrieval augmented generation (RAG), which inserts text retrieved from a larger memory into an LLM's context window. However, the context window is typically limited to several thousand tokens, which limits the number of retrieved passages that can inform a model's response. For this reason, it's important to avoid occupying context window space with redundant information by ensuring a degree of diversity among retrieved passages. At the same time, the information should also be relevant to the current task. Most prior methods that encourage diversity among retrieved results, such as Maximal Marginal Relevance (MMR), do so by incorporating an objective that explicitly trades off diversity and relevance. We propose a novel simple optimization metric based on relevant information gain, a probabilistic measure of the total information relevant to a query for a set of retrieved results. By optimizing this metric, diversity organically emerges from our system. When used as a drop-in replacement for the retrieval component of a RAG system, this method yields state-of-the-art performance on question answering tasks from the Retrieval Augmented Generation Benchmark (RGB), outperforming existing metrics that directly optimize for relevance and diversity.
• 摘要：扩展大型语言模型 (LLM) 内存的常用方法是通过检索增强生成 (RAG)，它将从较大内存中检索到的文本插入 LLM 的上下文窗口。但是，上下文窗口通常仅限于几千个标记，这限制了可以告知模型响应的检索段落的数量。因此，重要的是避免用冗余信息占用上下文窗口空间，确保检索到的段落之间具有一定程度的多样性。同时，信息也应该与当前任务相关。大多数鼓励检索结果多样性的先前方法，例如最大边际相关性 (MMR)，都是通过纳入明确权衡多样性和相关性的目标来实现的。我们提出了一种基于相关信息增益的新型简单优化指标，这是一种对一组检索结果的查询相关的总信息的概率度量。通过优化这个指标，多样性自然而然地从我们的系统中浮现出来。当用作 RAG 系统检索组件的直接替代品时，该方法在检索增强生成基准 (RGB) 的问答任务中产生了最先进的性能，优于直接针对相关性和多样性进行优化的现有指标。

Title: LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation

• Authors: Bunyamin Keles, Murat Gunay, Serdar I.Caglar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12126
• Pdf URL: https://arxiv.org/pdf/2407.12126
• Copy Paste: [[2407.12126]] LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation(https://arxiv.org/abs/2407.12126)
• Keywords: language model, gpt, llm, agent
• Abstract: Machine translation is indispensable in healthcare for enabling the global dissemination of medical knowledge across languages. However, complex medical terminology poses unique challenges to achieving adequate translation quality and accuracy. This study introduces a novel "LLMs-in-the-loop" approach to develop supervised neural machine translation models optimized specifically for medical texts. While large language models (LLMs) have demonstrated powerful capabilities, this research shows that small, specialized models trained on high-quality in-domain (mostly synthetic) data can outperform even vastly larger LLMs. Custom parallel corpora in six languages were compiled from scientific articles, synthetically generated clinical documents, and medical texts. Our LLMs-in-the-loop methodology employs synthetic data generation, rigorous evaluation, and agent orchestration to enhance performance. We developed small medical translation models using the MarianMT base model. We introduce a new medical translation test dataset to standardize evaluation in this domain. Assessed using BLEU, METEOR, ROUGE, and BERT scores on this test set, our MarianMT-based models outperform Google Translate, DeepL, and GPT-4-Turbo. Results demonstrate that our LLMs-in-the-loop approach, combined with fine-tuning high-quality, domain-specific data, enables specialized models to outperform general-purpose and some larger systems. This research, part of a broader series on expert small models, paves the way for future healthcare-related AI developments, including deidentification and bio-medical entity extraction models. Our study underscores the potential of tailored neural translation models and the LLMs-in-the-loop methodology to advance the field through improved data generation, evaluation, agent, and modeling techniques.
• 摘要：机器翻译在医疗保健领域必不可少，它使医学知识能够跨语言在全球传播。然而，复杂的医学术语对实现足够的翻译质量和准确性提出了独特的挑战。这项研究引入了一种新颖的“LLM 在环”方法，以开发专门针对医学文本优化的监督神经机器翻译模型。虽然大型语言模型 (LLM) 已经展示了强大的功能，但这项研究表明，在高质量领域内（主要是合成）数据上训练的小型专业模型可以胜过甚至更大的 LLM。六种语言的自定义平行语料库是从科学文章、合成生成的临床文档和医学文本中汇编而成的。我们的 LLM 在环方法采用合成数据生成、严格评估和代理编排来提高性能。我们使用 MarianMT 基础模型开发了小型医学翻译模型。我们引入了一个新的医学翻译测试数据集来标准化该领域的评估。使用 BLEU、METEOR、ROUGE 和 BERT 分数对该测试集进行评估，我们基于 MarianMT 的模型优于 Google Translate、DeepL 和 GPT-4-Turbo。结果表明，我们的 LLMs-in-the-loop 方法与微调高质量、特定领域的数据相结合，使专用模型能够胜过通用系统和一些大型系统。这项研究是关于专家小型模型的更广泛系列研究的一部分，为未来与医疗保健相关的 AI 发展铺平了道路，包括去识别和生物医学实体提取模型。我们的研究强调了量身定制的神经翻译模型和 LLMs-in-the-loop 方法通过改进数据生成、评估、代理和建模技术推动该领域发展的潜力。

Title: Predicting Emotion Intensity in Polish Political Texts: Comparing Supervised Models and Large Language Models in a Resource-Poor Language

• Authors: Hubert Plisiecki, Piotr Koc, Maria Flakus, Artur Pokropek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12141
• Pdf URL: https://arxiv.org/pdf/2407.12141
• Copy Paste: [[2407.12141]] Predicting Emotion Intensity in Polish Political Texts: Comparing Supervised Models and Large Language Models in a Resource-Poor Language(https://arxiv.org/abs/2407.12141)
• Keywords: language model, llm
• Abstract: This study explores the use of large language models (LLMs) to predict emotion intensity in Polish political texts, a resource-poor language context. The research compares the performance of several LLMs against a supervised model trained on an annotated corpus of 10,000 social media texts, evaluated for the intensity of emotions by expert judges. The findings indicate that while the supervised model generally outperforms LLMs, offering higher accuracy and lower variance, LLMs present a viable alternative, especially given the high costs associated with data annotation. The study highlights the potential of LLMs in low-resource language settings and underscores the need for further research on emotion intensity prediction and its application across different languages and continuous features. The implications suggest a nuanced decision-making process to choose the right approach to emotion prediction for researchers and practitioners based on resource availability and the specific requirements of their tasks.
• 摘要：本研究探讨了使用大型语言模型 (LLM) 预测波兰政治文本（一种资源匮乏的语言环境）中的情绪强度。该研究将几个 LLM 的性能与一个监督模型进行了比较，该模型在 10,000 个社交媒体文本的注释语料库上训练而成，并由专家评委评估情绪强度。研究结果表明，虽然监督模型通常优于 LLM，提供更高的准确性和更低的方差，但 LLM 是一种可行的替代方案，尤其是考虑到数据注释的高成本。该研究强调了 LLM 在资源匮乏的语言环境中的潜力，并强调需要进一步研究情绪强度预测及其在不同语言和连续特征中的应用。这些影响表明，研究人员和从业人员需要根据资源可用性和任务的具体要求，制定一个细致入微的决策过程，以选择正确的情绪预测方法。

Title: MASIVE: Open-Ended Affective State Identification in English and Spanish

• Authors: Nicholas Deas, Elsbeth Turcan, Iván Pérez Mejía, Kathleen McKeown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12196
• Pdf URL: https://arxiv.org/pdf/2407.12196
• Copy Paste: [[2407.12196]] MASIVE: Open-Ended Affective State Identification in English and Spanish(https://arxiv.org/abs/2407.12196)
• Keywords: llm
• Abstract: In the field of emotion analysis, much NLP research focuses on identifying a limited number of discrete emotion categories, often applied across languages. These basic sets, however, are rarely designed with textual data in mind, and culture, language, and dialect can influence how particular emotions are interpreted. In this work, we broaden our scope to a practically unbounded set of \textit{affective states}, which includes any terms that humans use to describe their experiences of feeling. We collect and publish MASIVE, a dataset of Reddit posts in English and Spanish containing over 1,000 unique affective states each. We then define the new problem of \textit{affective state identification} for language generation models framed as a masked span prediction task. On this task, we find that smaller finetuned multilingual models outperform much larger LLMs, even on region-specific Spanish affective states. Additionally, we show that pretraining on MASIVE improves model performance on existing emotion benchmarks. Finally, through machine translation experiments, we find that native speaker-written data is vital to good performance on this task.
• 摘要：在情感分析领域，许多 NLP 研究侧重于识别有限数量的离散情感类别，这些类别通常适用于各种语言。然而，这些基本集合很少考虑文本数据，而文化、语言和方言会影响特定情感的解读方式。在这项工作中，我们将范围扩大到几乎无界的 \textit{情感状态} 集合，其中包括人类用来描述其感受体验的任何术语。我们收集并发布了 MASIVE，这是一个英语和西班牙语 Reddit 帖子数据集，每个帖子包含 1,000 多个独特的情感状态。然后，我们为语言生成模型定义了 \textit{情感状态识别} 的新问题，该问题被框架为掩码跨度预测任务。在此任务中，我们发现较小的微调多语言模型的表现优于更大的 LLM，即使在特定地区的西班牙情感状态上也是如此。此外，我们表明，在 MASIVE 上进行预训练可以提高模型在现有情感基准上的表现。最后，通过机器翻译实验，我们发现母语人士编写的数据对于在此任务上取得良好表现至关重要。

Title: A Language Modeling Approach to Diacritic-Free Hebrew TTS

• Authors: Amit Roth, Arnon Turetzky, Yossi Adi
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.12206
• Pdf URL: https://arxiv.org/pdf/2407.12206
• Copy Paste: [[2407.12206]] A Language Modeling Approach to Diacritic-Free Hebrew TTS(https://arxiv.org/abs/2407.12206)
• Keywords: language model
• Abstract: We tackle the task of text-to-speech (TTS) in Hebrew. Traditional Hebrew contains Diacritics, which dictate the way individuals should pronounce given words, however, modern Hebrew rarely uses them. The lack of diacritics in modern Hebrew results in readers expected to conclude the correct pronunciation and understand which phonemes to use based on the context. This imposes a fundamental challenge on TTS systems to accurately map between text-to-speech. In this work, we propose to adopt a language modeling Diacritics-Free approach, for the task of Hebrew TTS. The model operates on discrete speech representations and is conditioned on a word-piece tokenizer. We optimize the proposed method using in-the-wild weakly supervised data and compare it to several diacritic-based TTS systems. Results suggest the proposed method is superior to the evaluated baselines considering both content preservation and naturalness of the generated speech. Samples can be found under the following link: this http URL
• 摘要：我们处理希伯来语的文本转语音 (TTS) 任务。传统希伯来语包含变音符号，这些符号规定了个人应该如何发音给定的单词，然而，现代希伯来语很少使用它们。现代希伯来语中缺乏变音符号，导致读者需要根据上下文得出正确的发音并了解要使用哪些音素。这对 TTS 系统提出了一个根本挑战，即准确地在文本转语音之间进行映射。在这项工作中，我们建议采用一种无变音符号的语言建模方法来完成希伯来语 TTS 任务。该模型以离散语音表示为基础，并以单词片段标记器为条件。我们使用野生弱监督数据优化所提出的方法，并将其与几个基于变音符号的 TTS 系统进行比较。结果表明，考虑到内容保存和生成语音的自然性，所提出的方法优于评估的基线。可以在以下链接下找到示例：此 http URL

Title: In-Context Probing Approximates Influence Function for Data Valuation

• Authors: Cathy Jiao, Gary Gao, Chenyan Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12259
• Pdf URL: https://arxiv.org/pdf/2407.12259
• Copy Paste: [[2407.12259]] In-Context Probing Approximates Influence Function for Data Valuation(https://arxiv.org/abs/2407.12259)
• Keywords: language model, llm, prompt
• Abstract: Data valuation quantifies the value of training data, and is used for data attribution (i.e., determining the contribution of training data towards model predictions), and data selection; both of which are important for curating high-quality datasets to train large language models. In our paper, we show that data valuation through in-context probing (i.e., prompting a LLM) approximates influence functions for selecting training data. We provide a theoretical sketch on this connection based on transformer models performing "implicit" gradient descent on its in-context inputs. Our empirical findings show that in-context probing and gradient-based influence frameworks are similar in how they rank training data. Furthermore, fine-tuning experiments on data selected by either method reveal similar model performance.
• 摘要：数据估值量化了训练数据的价值，并用于数据归因（即确定训练数据对模型预测的贡献）和数据选择；这两者对于策划用于训练大型语言模型的高质量数据集都很重要。在我们的论文中，我们表明通过上下文探测（即提示 LLM）进行的数据估值近似于选择训练数据的影响函数。我们基于对其上下文输入执行“隐式”梯度下降的 Transformer 模型，对这种联系提供了理论概述。我们的实证研究结果表明，上下文探测和基于梯度的影响框架在对训练数据进行排序的方式上是相似的。此外，对通过任一方法选择的数据进行的微调实验都显示出相似的模型性能。

Title: MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models

• Authors: Thao Minh Nguyen Phan, Cong-Tinh Dao, Chenwei Wu, Jian-Zhe Wang, Shun Liu, Jun-En Ding, David Restrepo, Feng Liu, Fang-Ming Hung, Wen-Chih Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12309
• Pdf URL: https://arxiv.org/pdf/2407.12309
• Copy Paste: [[2407.12309]] MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models(https://arxiv.org/abs/2407.12309)
• Keywords: language model, llm
• Abstract: Electronic health records (EHRs) are multimodal by nature, consisting of structured tabular features like lab tests and unstructured clinical notes. In real-life clinical practice, doctors use complementary multimodal EHR data sources to get a clearer picture of patients' health and support clinical decision-making. However, most EHR predictive models do not reflect these procedures, as they either focus on a single modality or overlook the inter-modality interactions/redundancy. In this work, we propose MEDFuse, a Multimodal EHR Data Fusion framework that incorporates masked lab-test modeling and large language models (LLMs) to effectively integrate structured and unstructured medical data. MEDFuse leverages multimodal embeddings extracted from two sources: LLMs fine-tuned on free clinical text and masked tabular transformers trained on structured lab test results. We design a disentangled transformer module, optimized by a mutual information loss to 1) decouple modality-specific and modality-shared information and 2) extract useful joint representation from the noise and redundancy present in clinical notes. Through comprehensive validation on the public MIMIC-III dataset and the in-house FEMH dataset, MEDFuse demonstrates great potential in advancing clinical predictions, achieving over 90% F1 score in the 10-disease multi-label classification task.
• 摘要：电子健康记录 (EHR) 本质上是多模态的，由结构化的表格特征（如实验室测试）和非结构化的临床笔记组成。在现实临床实践中，医生使用互补的多模态 EHR 数据源来更清楚地了解患者的健康状况并支持临床决策。然而，大多数 EHR 预测模型并没有反映这些过程，因为它们要么专注于单一模态，要么忽略了模态间的相互作用/冗余。在这项工作中，我们提出了 MEDFuse，这是一个多模态 EHR 数据融合框架，它结合了掩蔽实验室测试建模和大型语言模型 (LLM)，以有效地整合结构化和非结构化的医疗数据。MEDFuse 利用从两个来源提取的多模态嵌入：在自由临床文本上微调的 LLM 和在结构化实验室测试结果上训练的掩蔽表格转换器。我们设计了一个解缠转换器模块，通过相互信息损失进行优化，以 1) 解耦特定模态和共享模态的信息，以及 2) 从临床笔记中存在的噪声和冗余中提取有用的联合表示。通过对公共 MIMIC-III 数据集和内部 FEMH 数据集的全面验证，MEDFuse 在推进临床预测方面表现出巨大潜力，在 10 种疾病多标签分类任务中取得了超过 90% 的 F1 分数。

Title: The Better Angels of Machine Personality: How Personality Relates to LLM Safety

• Authors: Jie Zhang, Dongrui Liu, Chen Qian, Ziyue Gan, Yong Liu, Yu Qiao, Jing Shao
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.12344
• Pdf URL: https://arxiv.org/pdf/2407.12344
• Copy Paste: [[2407.12344]] The Better Angels of Machine Personality: How Personality Relates to LLM Safety(https://arxiv.org/abs/2407.12344)
• Keywords: language model, llm
• Abstract: Personality psychologists have analyzed the relationship between personality and safety behaviors in human society. Although Large Language Models (LLMs) demonstrate personality traits, the relationship between personality traits and safety abilities in LLMs still remains a mystery. In this paper, we discover that LLMs' personality traits are closely related to their safety abilities, i.e., toxicity, privacy, and fairness, based on the reliable MBTI-M scale. Meanwhile, the safety alignment generally increases various LLMs' Extraversion, Sensing, and Judging traits. According to such findings, we can edit LLMs' personality traits and improve their safety performance, e.g., inducing personality from ISTJ to ISTP resulted in a relative improvement of approximately 43% and 10% in privacy and fairness performance, respectively. Additionally, we find that LLMs with different personality traits are differentially susceptible to jailbreak. This study pioneers the investigation of LLM safety from a personality perspective, providing new insights into LLM safety enhancement.
• 摘要：人格心理学家已经分析了人格与人类社会安全行为之间的关系。虽然大型语言模型 (LLM) 展示了人格特质，但 LLM 中人格特质与安全能力之间的关系仍然是个谜。本文基于可靠的 MBTI-M 量表发现 LLM 的人格特质与其安全能力（即毒性、隐私和公平性）密切相关。同时，安全性调整通常会增加各种 LLM 的外向性、感觉性和判断性特质。根据这些发现，我们可以编辑 LLM 的人格特质并提高其安全性能，例如，将人格从 ISTJ 诱导为 ISTP 分别导致隐私和公平性能相对提高约 43% 和 10%。此外，我们发现具有不同人格特质的 LLM 对越狱的敏感性不同。这项研究开创了从人格角度研究 LLM 安全性的先河，为 LLM 安全性增强提供了新的见解。

Title: Conversational Query Reformulation with the Guidance of Retrieved Documents

• Authors: Jeonghyun Park, Hwanhee Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12363
• Pdf URL: https://arxiv.org/pdf/2407.12363
• Copy Paste: [[2407.12363]] Conversational Query Reformulation with the Guidance of Retrieved Documents(https://arxiv.org/abs/2407.12363)
• Keywords: language model, llm, prompt
• Abstract: Conversational search seeks to retrieve relevant passages for the given questions in Conversational QA (ConvQA). Questions in ConvQA face challenges such as omissions and coreferences, making it difficult to obtain desired search results. Conversational Query Reformulation (CQR) transforms these current queries into de-contextualized forms to resolve these issues. However, existing CQR methods focus on rewriting human-friendly queries, which may not always yield optimal search results for the retriever. To overcome this challenge, we introduce GuideCQR, a framework that utilizes guided documents to refine queries, ensuring that they are optimal for retrievers. Specifically, we augment keywords, generate expected answers from the re-ranked documents, and unify them with the filtering process. Experimental results show that queries enhanced by guided documents outperform previous CQR methods. Especially, GuideCQR surpasses the performance of Large Language Model (LLM) prompt-powered approaches and demonstrates the importance of the guided documents in formulating retriever-friendly queries across diverse setups.
• 摘要：对话式搜索旨在检索对话式问答 (ConvQA) 中给定问题的相关段落。ConvQA 中的问题面临遗漏和共指等挑战，因此很难获得所需的搜索结果。对话式查询重构 (CQR) 将这些当前查询转换为非语境化形式来解决这些问题。然而，现有的 CQR 方法专注于重写人性化查询，这可能并不总是能为检索器产生最佳搜索结果。为了克服这一挑战，我们引入了 GuideCQR，这是一个利用引导文档来优化查询的框架，确保它们对检索器来说是最佳的。具体来说，我们扩充关键字，从重新排序的文档中生成预期答案，并将它们与过滤过程统一起来。实验结果表明，通过引导文档增强的查询优于以前的 CQR 方法。特别是，GuideCQR 超越了大型语言模型 (LLM) 提示驱动方法的性能，并证明了引导文档在制定跨不同设置的检索器友好查询方面的重要性。

Title: PersLLM: A Personified Training Approach for Large Language Models

• Authors: Zheni Zeng, Jiayi Chen, Huimin Chen, Yukun Yan, Yuxuan Chen, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.12393
• Pdf URL: https://arxiv.org/pdf/2407.12393
• Copy Paste: [[2407.12393]] PersLLM: A Personified Training Approach for Large Language Models(https://arxiv.org/abs/2407.12393)
• Keywords: language model, llm, prompt, agent
• Abstract: Large language models exhibit aspects of human-level intelligence that catalyze their application as human-like agents in domains such as social simulations, human-machine interactions, and collaborative multi-agent systems. However, the absence of distinct personalities, such as displaying ingratiating behaviors, inconsistent opinions, and uniform response patterns, diminish LLMs utility in practical applications. Addressing this, the development of personality traits in LLMs emerges as a crucial area of research to unlock their latent potential. Existing methods to personify LLMs generally involve strategies like employing stylized training data for instruction tuning or using prompt engineering to simulate different personalities. These methods only capture superficial linguistic styles instead of the core of personalities and are therefore not stable. In this study, we propose PersLLM, integrating psychology-grounded principles of personality: social practice, consistency, and dynamic development, into a comprehensive training methodology. We incorporate personality traits directly into the model parameters, enhancing the model's resistance to induction, promoting consistency, and supporting the dynamic evolution of personality. Single-agent evaluation validates our method's superiority, as it produces responses more aligned with reference personalities compared to other approaches. Case studies for multi-agent communication highlight its benefits in enhancing opinion consistency within individual agents and fostering collaborative creativity among multiple agents in dialogue contexts, potentially benefiting human simulation and multi-agent cooperation. Additionally, human-agent interaction evaluations indicate that our personified models significantly enhance interactive experiences, underscoring the practical implications of our research.
• 摘要：大型语言模型展现出人类水平的智能，这些智能催化了它们作为类人智能体在社会模拟、人机交互和协作多智能体系统等领域的应用。然而，缺乏独特的个性，例如表现出讨好行为、不一致的观点和统一的反应模式，削弱了 LLM 在实际应用中的实用性。为了解决这个问题，LLM 中个性特征的发展成为释放其潜在潜力的关键研究领域。现有的拟人化 LLM 的方法通常涉及使用风格化的训练数据进行指令调整或使用提示工程来模拟不同个性等策略。这些方法只捕捉表面的语言风格，而不是个性的核心，因此不稳定。在这项研究中，我们提出了 PersLLM，将基于心理学的个性原则：社会实践、一致性和动态发展，整合到全面的训练方法中。我们将个性特征直接纳入模型参数中，增强模型的抗诱导性，促进一致性，并支持个性的动态演变。单智能体评估验证了我们方法的优越性，因为与其他方法相比，它产生的响应更符合参考个性。多智能体通信案例研究强调了其在增强个体智能体内部意见一致性和促进对话环境中多个智能体之间的协作创造力方面的优势，这可能有益于人类模拟和多智能体合作。此外，人机交互评估表明，我们的拟人化模型显著增强了交互体验，凸显了我们研究的实际意义。

Title: TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

• Authors: Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12402
• Pdf URL: https://arxiv.org/pdf/2407.12402
• Copy Paste: [[2407.12402]] TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish(https://arxiv.org/abs/2407.12402)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: this https URL.
• 摘要：多项选择问答任务评估大型语言模型 (LLM) 的推理、理解和数学能力。虽然现有基准测试采用自动翻译进行多语言评估，但这种方法容易出错，并且可能会引入文化偏见问题，尤其是在社会科学领域。我们推出了第一个多任务、多项选择土耳其语 QA 基准测试 TurkishMMLU，以评估 LLM 对土耳其语的理解。TurkishMMLU 包含 10,000 多个问题，涵盖土耳其高中教育课程的 9 个不同科目。这些问题由课程专家编写，适用于土耳其的高中课程，涵盖从自然科学和数学问题到更具文化代表性的主题，例如土耳其文学和土耳其共和国的历史。我们评估了 20 多个 LLM，包括多语言开源（例如 Gemma、Llama、MT5）、闭源（GPT 4o、Claude、Gemini）和土耳其语改编（例如 Trendyol）模型。我们提供了广泛的评估，包括对 LLM 的零样本和少量样本评估、思路链推理和问题难度分析以及模型性能。我们对土耳其语能力和当前 LLM 的局限性进行了深入分析，为未来的土耳其语 LLM 提供见解。我们公开发布了数据集和评估的代码：此 https URL。

Title: Navigating the Noisy Crowd: Finding Key Information for Claim Verification

• Authors: Haisong Gong, Huanhuan Ma, Qiang Liu, Shu Wu, Liang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12425
• Pdf URL: https://arxiv.org/pdf/2407.12425
• Copy Paste: [[2407.12425]] Navigating the Noisy Crowd: Finding Key Information for Claim Verification(https://arxiv.org/abs/2407.12425)
• Keywords: language model, llm
• Abstract: Claim verification is a task that involves assessing the truthfulness of a given claim based on multiple evidence pieces. Using large language models (LLMs) for claim verification is a promising way. However, simply feeding all the evidence pieces to an LLM and asking if the claim is factual does not yield good results. The challenge lies in the noisy nature of both the evidence and the claim: evidence passages typically contain irrelevant information, with the key facts hidden within the context, while claims often convey multiple aspects simultaneously. To navigate this "noisy crowd" of information, we propose EACon (Evidence Abstraction and Claim Deconstruction), a framework designed to find key information within evidence and verify each aspect of a claim separately. EACon first finds keywords from the claim and employs fuzzy matching to select relevant keywords for each raw evidence piece. These keywords serve as a guide to extract and summarize critical information into abstracted evidence. Subsequently, EACon deconstructs the original claim into subclaims, which are then verified against both abstracted and raw evidence individually. We evaluate EACon using two open-source LLMs on two challenging datasets. Results demonstrate that EACon consistently and substantially improve LLMs' performance in claim verification.
• 摘要：声明验证是一项基于多个证据片段评估给定声明真实性的任务。使用大型语言模型 (LLM) 进行声明验证是一种很有前途的方法。但是，简单地将所有证据片段输入 LLM 并询问声明是否属实并不能产生良好的结果。挑战在于证据和声明的嘈杂性质：证据段落通常包含不相关的信息，关键事实隐藏在上下文中，而声明通常同时传达多个方面。为了驾驭这种“嘈杂的”信息，我们提出了 EACon（证据抽象和声明解构），这是一个旨在在证据中查找关键信息并分别验证声明各个方面的框架。EACon 首先从声明中找到关键字，然后使用模糊匹配为每个原始证据片段选择相关关键字。这些关键词可作为提取和总结关键信息为抽象证据的指南。随后，EACon 将原始声明解构为子声明，然后分别根据抽象证据和原始证据进行验证。我们使用两个开源 LLM 在两个具有挑战性的数据集上对 EACon 进行了评估。结果表明，EACon 持续大幅提高了 LLM 在索赔验证方面的性能。

Title: Sharif-STR at SemEval-2024 Task 1: Transformer as a Regression Model for Fine-Grained Scoring of Textual Semantic Relations

• Authors: Seyedeh Fatemeh Ebrahimi, Karim Akhavan Azari, Amirmasoud Iravani, Hadi Alizadeh, Zeinab Sadat Taghavi, Hossein Sameti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12426
• Pdf URL: https://arxiv.org/pdf/2407.12426
• Copy Paste: [[2407.12426]] Sharif-STR at SemEval-2024 Task 1: Transformer as a Regression Model for Fine-Grained Scoring of Textual Semantic Relations(https://arxiv.org/abs/2407.12426)
• Keywords: language model
• Abstract: Semantic Textual Relatedness holds significant relevance in Natural Language Processing, finding applications across various domains. Traditionally, approaches to STR have relied on knowledge-based and statistical methods. However, with the emergence of Large Language Models, there has been a paradigm shift, ushering in new methodologies. In this paper, we delve into the investigation of sentence-level STR within Track A (Supervised) by leveraging fine-tuning techniques on the RoBERTa transformer. Our study focuses on assessing the efficacy of this approach across different languages. Notably, our findings indicate promising advancements in STR performance, particularly in Latin languages. Specifically, our results demonstrate notable improvements in English, achieving a correlation of 0.82 and securing a commendable 19th rank. Similarly, in Spanish, we achieved a correlation of 0.67, securing the 15th position. However, our approach encounters challenges in languages like Arabic, where we observed a correlation of only 0.38, resulting in a 20th rank.
• 摘要：语义文本相关性在自然语言处理中具有重要意义，可在各个领域找到应用。传统上，STR 方法依赖于基于知识和统计的方法。然而，随着大型语言模型的出现，范式发生了转变，迎来了新的方法。在本文中，我们利用 RoBERTa Transformer 上的微调技术，深入研究了 Track A（监督）中句子级 STR 的研究。我们的研究重点是评估这种方法在不同语言中的有效性。值得注意的是，我们的研究结果表明 STR 性能取得了令人鼓舞的进步，特别是在拉丁语中。具体来说，我们的结果显示英语有显着的改进，相关性达到 0.82，获得了值得称赞的第 19 名。同样，在西班牙语中，我们实现了 0.67 的相关性，获得了第 15 名。然而，我们的方法在阿拉伯语等语言中遇到了挑战，我们观察到的相关性只有 0.38，排名为第 20 位。

Title: Pretraining Data and Tokenizer for Indic LLM

• Authors: Rahul Kumar, Shubham Kakde, Divyansh Rajput, Daud Ibrahim, Rishabh Nahata, Pidathala Sowjanya, Deepak Kumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12481
• Pdf URL: https://arxiv.org/pdf/2407.12481
• Copy Paste: [[2407.12481]] Pretraining Data and Tokenizer for Indic LLM(https://arxiv.org/abs/2407.12481)
• Keywords: language model, llm
• Abstract: We present a novel approach to data preparation for developing multilingual Indic large language model. Our meticulous data acquisition spans open-source and proprietary sources, including Common Crawl, Indic books, news articles, and Wikipedia, ensuring a diverse and rich linguistic representation. For each Indic language, we design a custom preprocessing pipeline to effectively eliminate redundant and low-quality text content. Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for Indic large language models with 3B and 7B parameters, engineered for superior performance in Indic languages. We introduce a novel multilingual tokenizer training strategy, demonstrating our custom-trained Indic tokenizer outperforms the state-of-the-art OpenAI Tiktoken tokenizer, achieving a superior token-to-word ratio for Indic languages.
• 摘要：我们提出了一种用于开发多语言印度语大型语言模型的新数据准备方法。我们细致的数据采集涵盖开源和专有来源，包括 Common Crawl、印度语书籍、新闻文章和维基百科，确保语言表现形式多样而丰富。对于每种印度语，我们设计了一个自定义预处理管道，以有效消除冗余和低质量的文本内容。此外，我们对 Common Crawl 数据执行重复数据删除，以解决 70% 的抓取网页中存在的冗余问题。本研究侧重于开发高质量数据，针对具有 3B 和 7B 参数的印度语大型语言模型优化多语言数据集的标记化，这些模型专为在印度语中实现卓越性能而设计。我们引入了一种新颖的多语言标记器训练策略，证明了我们定制训练的印度语标记器优于最先进的 OpenAI Tiktoken 标记器，为印度语实现了卓越的标记与词的比率。

Title: Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning

• Authors: Mustafa Dogan, Ilker Kesen, Iacer Calixto, Aykut Erdem, Erkut Erdem
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12498
• Pdf URL: https://arxiv.org/pdf/2407.12498
• Copy Paste: [[2407.12498]] Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning(https://arxiv.org/abs/2407.12498)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: The linguistic capabilities of Multimodal Large Language Models (MLLMs) are critical for their effective application across diverse tasks. This study aims to evaluate the performance of MLLMs on the VALSE benchmark, focusing on the efficacy of few-shot In-Context Learning (ICL), and Chain-of-Thought (CoT) prompting. We conducted a comprehensive assessment of state-of-the-art MLLMs, varying in model size and pretraining datasets. The experimental results reveal that ICL and CoT prompting significantly boost model performance, particularly in tasks requiring complex reasoning and contextual understanding. Models pretrained on captioning datasets show superior zero-shot performance, while those trained on interleaved image-text data benefit from few-shot learning. Our findings provide valuable insights into optimizing MLLMs for better grounding of language in visual contexts, highlighting the importance of the composition of pretraining data and the potential of few-shot learning strategies to improve the reasoning abilities of MLLMs.
• 摘要：多模态大型语言模型 (MLLM) 的语言能力对于其在不同任务中的有效应用至关重要。本研究旨在评估 MLLM 在 VALSE 基准上的表现，重点关注少量上下文学习 (ICL) 和思维链 (CoT) 提示的有效性。我们对最先进的 MLLM 进行了全面评估，模型大小和预训练数据集各不相同。实验结果表明，ICL 和 CoT 提示显著提高了模型性能，特别是在需要复杂推理和上下文理解的任务中。在字幕数据集上预训练的模型表现出卓越的零样本性能，而在交错图像文本数据上训练的模型则受益于少量学习。我们的研究结果为优化 MLLM 以更好地将语言置于视觉环境中提供了宝贵的见解，强调了预训练数据组成的重要性以及少量学习策略提高 MLLM 推理能力的潜力。

Title: Automate or Assist? The Role of Computational Models in Identifying Gendered Discourse in US Capital Trial Transcripts

• Authors: Andrea W Wen-Yi, Kathryn Adamson, Nathalie Greenfield, Rachel Goldberg, Sandra Babcock, David Mimno, Allison Koenecke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12500
• Pdf URL: https://arxiv.org/pdf/2407.12500
• Copy Paste: [[2407.12500]] Automate or Assist? The Role of Computational Models in Identifying Gendered Discourse in US Capital Trial Transcripts(https://arxiv.org/abs/2407.12500)
• Keywords: language model
• Abstract: The language used by US courtroom actors in criminal trials has long been studied for biases. However, systematic studies for bias in high-stakes court trials have been difficult, due to the nuanced nature of bias and the legal expertise required. New large language models offer the possibility to automate annotation, saving time and cost. But validating these approaches requires both high quantitative performance as well as an understanding of how automated methods fit in existing workflows, and what they really offer. In this paper we present a case study of adding an automated system to a complex and high-stakes problem: identifying gender-biased language in US capital trials for women defendants. Our team of experienced death-penalty lawyers and NLP technologists pursued a three-phase study: first annotating manually, then training and evaluating computational models, and finally comparing human annotations to model predictions. Unlike many typical NLP tasks, annotating for gender bias in months-long capital trials was a complicated task that involves with many individual judgment calls. In contrast to standard arguments for automation that are based on efficiency and scalability, legal experts found the computational models most useful in challenging their personal bias in annotation and providing opportunities to refine and build consensus on rules for annotation. This suggests that seeking to replace experts with computational models is both unrealistic and undesirable. Rather, computational models offer valuable opportunities to assist the legal experts in annotation-based studies.
• 摘要：长期以来，人们一直在研究美国法庭人员在刑事审判中使用的语言是否存在偏见。然而，由于偏见的微妙性质和所需的法律专业知识，对高风险法庭审判中的偏见进行系统研究一直很困难。新的大型语言模型提供了自动注释的可能性，从而节省了时间和成本。但验证这些方法既需要很高的定量性能，也需要了解自动化方法如何适应现有的工作流程，以及它们真正能提供什么。在本文中，我们介绍了一个案例研究，将自动化系统添加到一个复杂且高风险的问题中：识别美国死刑审判中针对女性被告的性别偏见语言。我们经验丰富的死刑律师和 NLP 技术人员团队进行了一项三阶段研究：首先手动注释，然后训练和评估计算模型，最后将人工注释与模型预测进行比较。与许多典型的 NLP 任务不同，在长达数月的死刑审判中注释性别偏见是一项复杂的任务，涉及许多个人判断。与基于效率和可扩展性的自动化标准论点相反，法律专家发现计算模型最有用的是挑战他们在注释中的个人偏见，并提供机会完善和建立注释规则的共识。这表明，试图用计算模型取代专家既不现实也不可取。相反，计算模型提供了宝贵的机会来协助法律专家进行基于注释的研究。

Title: Case2Code: Learning Inductive Reasoning with Synthetic Data

• Authors: Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12504
• Pdf URL: https://arxiv.org/pdf/2407.12504
• Copy Paste: [[2407.12504]] Case2Code: Learning Inductive Reasoning with Synthetic Data(https://arxiv.org/abs/2407.12504)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Complex reasoning is an impressive ability shown by large language models (LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought prompting or iterative tool-using to solve challenging tasks step-by-step. In this paper, we hope to focus on evaluating and teaching LLMs to conduct inductive reasoning, that is, LLMs are supposed to infer underlying rules by observing examples or sequential transformations. However, collecting large-scale and diverse human-generated inductive data is challenging. We focus on data synthesis in the code domain and propose a \textbf{Case2Code} task by exploiting the expressiveness and correctness of programs. Specifically, we collect a diverse set of executable programs, synthesize input-output transformations for each program, and force LLMs to infer the underlying code implementations based on the synthetic I/O cases. We first evaluate representative LLMs on the synthesized Case2Code task and demonstrate that the Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale Case2Code training samples to train LLMs to perform inductive reasoning. Experimental results show that such induction training benefits not only in distribution Case2Code performance but also enhances various coding abilities of trained LLMs, demonstrating the great potential of learning inductive reasoning via synthetic data.
• 摘要：复杂推理是大型语言模型 (LLM) 所展现的一项令人印象深刻的能力。大多数 LLM 都擅长演绎推理，例如通过思路链提示或迭代使用工具逐步解决具有挑战性的任务。在本文中，我们希望专注于评估和教授 LLM 进行归纳推理，即 LLM 应该通过观​​察示例或顺序转换来推断底层规则。然而，收集大规模和多样化的人为归纳数据具有挑战性。我们专注于代码领域的数据合成，并通过利用程序的表达力和正确性提出了 \textbf{Case2Code} 任务。具体来说，我们收集一组多样化的可执行程序，为每个程序合成输入输出转换，并强制 LLM 根据合成的 I/O 案例推断底层代码实现。我们首先在合成的 Case2Code 任务上评估代表性 LLM，并证明 Case-to-code 归纳对 LLM 具有挑战性。然后，我们合成大规模 Case2Code 训练样本来训练 LLM 进行归纳推理。实验结果表明，这种归纳训练不仅有利于提高分布 Case2Code 性能，而且还能增强已训练 LLM 的各种编码能力，证明了通过合成数据学习归纳推理的巨大潜力。

Title: MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline

• Authors: Donghoon Han, Eunhwan Park, Gisang Lee, Adam Lee, Nojun Kwak
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12508
• Pdf URL: https://arxiv.org/pdf/2407.12508
• Copy Paste: [[2407.12508]] MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline(https://arxiv.org/abs/2407.12508)
• Keywords: language model, llm
• Abstract: The rapid expansion of multimedia content has made accurately retrieving relevant videos from large collections increasingly challenging. Recent advancements in text-video retrieval have focused on cross-modal interactions, large-scale foundation model training, and probabilistic modeling, yet often neglect the crucial user perspective, leading to discrepancies between user queries and the content retrieved. To address this, we introduce MERLIN (Multimodal Embedding Refinement via LLM-based Iterative Navigation), a novel, training-free pipeline that leverages Large Language Models (LLMs) for iterative feedback learning. MERLIN refines query embeddings from a user perspective, enhancing alignment between queries and video content through a dynamic question answering process. Experimental results on datasets like MSR-VTT, MSVD, and ActivityNet demonstrate that MERLIN substantially improves Recall@1, outperforming existing systems and confirming the benefits of integrating LLMs into multimodal retrieval systems for more responsive and context-aware multimedia retrieval.
• 摘要：多媒体内容的快速扩展使得从大量集合中准确检索相关视频变得越来越具有挑战性。文本视频检索的最新进展主要集中在跨模态交互、大规模基础模型训练和概率建模上，但往往忽略了至关重要的用户视角，导致用户查询和检索到的内容之间存在差异。为了解决这个问题，我们引入了 MERLIN（通过基于 LLM 的迭代导航进行多模态嵌入细化），这是一种新颖的无需训练的管道，它利用大型语言模型 (LLM) 进行迭代反馈学习。MERLIN 从用户的角度细化查询嵌入，通过动态问答过程增强查询和视频内容之间的一致性。在 MSR-VTT、MSVD 和 ActivityNet 等数据集上的实验结果表明，MERLIN 显著提高了 Recall@1，优于现有系统，并证实了将 LLM 集成到多模态检索系统中以实现响应更快、更能感知上下文的多媒体检索的好处。

Title: $\textit{GeoHard}$: Towards Measuring Class-wise Hardness through Modelling Class Semantics

• Authors: Fengyu Cai, Xinran Zhao, Hongming Zhang, Iryna Gurevych, Heinz Koeppl
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] $\textit{GeoHard}$: Towards Measuring Class-wise Hardness through Modelling Class Semantics(https://arxiv.org/abs/)
• Keywords: language model
• Abstract: Recent advances in measuring hardness-wise properties of data guide language models in sample selection within low-resource scenarios. However, class-specific properties are overlooked for task setup and learning. How will these properties influence model learning and is it generalizable across datasets? To answer this question, this work formally initiates the concept of $\textit{class-wise hardness}$. Experiments across eight natural language understanding (NLU) datasets demonstrate a consistent hardness distribution across learning paradigms, models, and human judgment. Subsequent experiments unveil a notable challenge in measuring such class-wise hardness with instance-level metrics in previous works. To address this, we propose $\textit{GeoHard}$ for class-wise hardness measurement by modeling class geometry in the semantic embedding space. $\textit{GeoHard}$ surpasses instance-level metrics by over 59 percent on $\textit{Pearson}$'s correlation on measuring class-wise hardness. Our analysis theoretically and empirically underscores the generality of $\textit{GeoHard}$ as a fresh perspective on data diagnosis. Additionally, we showcase how understanding class-wise hardness can practically aid in improving task learning.
• 摘要：在低资源场景中，测量数据硬度属性的最新进展指导了语言模型的样本选择。然而，在任务设置和学习中，类特定属性被忽略了。这些属性将如何影响模型学习，它是否可以在数据集中推广？为了回答这个问题，这项工作正式启动了$\textit{类别硬度}$的概念。在八个自然语言理解 (NLU) 数据集上进行的实验表明，在学习范式、模型和人类判断中硬度分布一致。后续实验揭示了在以前的研究中使用实例级指标来测量这种类别硬度的显著挑战。为了解决这个问题，我们提出了$\textit{GeoHard}$，通过在语义嵌入空间中建模类几何来测量类别硬度。在测量类别硬度的$\textit{Pearson}$相关性上，$\textit{GeoHard}$比实例级指标高出 59% 以上。我们的分析从理论和实证两个方面强调了 $\textit{GeoHard}$ 作为数据诊断新视角的普遍性。此外，我们还展示了理解类别难度如何实际有助于改善任务学习。

Title: On Initializing Transformers with Pre-trained Embeddings

• Authors: Ha Young Kim, Niranjan Balasubramanian, Byungkon Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12514
• Pdf URL: https://arxiv.org/pdf/2407.12514
• Copy Paste: [[2407.12514]] On Initializing Transformers with Pre-trained Embeddings(https://arxiv.org/abs/2407.12514)
• Keywords: language model
• Abstract: It has become common practice now to use random initialization schemes, rather than the pre-trained embeddings, when training transformer based models from scratch. Indeed, we find that pre-trained word embeddings from GloVe, and some sub-word embeddings extracted from language models such as T5 and mT5 fare much worse compared to random initialization. This is counter-intuitive given the well-known representational and transfer-learning advantages of pre-training. Interestingly, we also find that BERT and mBERT embeddings fare better than random initialization, showing the advantages of pre-trained representations. In this work, we posit two potential factors that contribute to these mixed results: the model sensitivity to parameter distribution and the embedding interactions with position encodings. We observe that pre-trained GloVe, T5, and mT5 embeddings have a wider distribution of values. As argued in the initialization studies, such large value initializations can lead to poor training because of saturated outputs. Further, the larger embedding values can, in effect, absorb the smaller position encoding values when added together, thus losing position information. Standardizing the pre-trained embeddings to a narrow range (e.g. as prescribed by Xavier) leads to substantial gains for Glove, T5, and mT5 embeddings. On the other hand, BERT pre-trained embeddings, while larger, are still relatively closer to Xavier initialization range which may allow it to effectively transfer the pre-trained knowledge.
• 摘要：现在，在从头开始训练基于 Transformer 的模型时，使用随机初始化方案而不是预训练的嵌入已成为一种常见做法。事实上，我们发现来自 GloVe 的预训练词嵌入以及从语言模型（如 T5 和 mT5）中提取的一些子词嵌入与随机初始化相比要差得多。考虑到预训练众所周知的表征和迁移学习优势，这是违反直觉的。有趣的是，我们还发现 BERT 和 mBERT 嵌入比随机初始化表现更好，显示了预训练表征的优势。在这项工作中，我们提出了导致这些混合结果的两个潜在因素：模型对参数分布的敏感性以及嵌入与位置编码的交互。我们观察到预训练的 GloVe、T5 和 mT5 嵌入具有更广泛的值分布。正如初始化研究中所述，如此大的值初始化可能会因为输出饱和而导致训练不佳。此外，较大的嵌入值实际上可以在加在一起时吸收较小的位置编码值，从而丢失位置信息。将预训练嵌入标准化为一个较窄的范围（例如，按照 Xavier 的规定）会为 Glove、T5 和 mT5 嵌入带来显著的收益。另一方面，BERT 预训练嵌入虽然较大，但仍然相对更接近 Xavier 初始化范围，这可能使其能够有效地传输预训练知识。

Title: Struct-X: Enhancing Large Language Models Reasoning with Structured Data

• Authors: Xiaoyu Tan, Haoyu Wang, Xihe Qiu, Yuan Cheng, Yinghui Xu, Wei Chu, Yuan Qi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12522
• Pdf URL: https://arxiv.org/pdf/2407.12522
• Copy Paste: [[2407.12522]] Struct-X: Enhancing Large Language Models Reasoning with Structured Data(https://arxiv.org/abs/2407.12522)
• Keywords: language model, llm, prompt
• Abstract: Structured data, rich in logical and relational information, has the potential to enhance the reasoning abilities of large language models (LLMs). Still, its integration poses a challenge due to the risk of overwhelming LLMs with excessive tokens and irrelevant context information. To address this, we propose Struct-X, a novel framework that operates through five key phases: ``read-model-fill-reflect-reason'' efficiently enabling LLMs to utilize structured data. It begins by encoding structured data into a topological space using graph embeddings, followed by filling in missing entity information with knowledge retrieval modules, and filtering out irrelevant tokens via a self-supervised module. The final phase involves constructing a topological network with selected tokens to further reduce the total token length for more effective LLM inference. Additionally, Struct-X includes an Auxiliary Module trained to generate prompts, aiding LLMs in analyzing structured data. Extensive experiments on benchmarks, including the knowledge graph question-answer task and the long document reading comprehension task, show that Struct-X notably improves LLM reasoning, demonstrating the effectiveness of structured data augmentation in improving LLM inference with complex input context.
• 摘要：结构化数据富含逻辑和关系信息，有可能增强大型语言模型 (LLM) 的推理能力。然而，由于过多的标记和不相关的上下文信息可能会使 LLM 不堪重负，因此其集成仍是一个挑战。为了解决这个问题，我们提出了 Struct-X，这是一个新颖的框架，它通过五个关键阶段运行：“读取-模型-填充-反映-推理”，使 LLM 能够有效地利用结构化数据。它首先使用图嵌入将结构化数据编码到拓扑空间中，然后用知识检索模块填充缺失的实体信息，并通过自监督模块过滤掉不相关的标记。最后阶段涉及使用选定的标记构建拓扑网络，以进一步减少总标记长度，从而实现更有效的 LLM 推理。此外，Struct-X 还包括一个经过训练的辅助模块来生成提示，帮助 LLM 分析结构化数据。在知识图谱问答任务和长文档阅读理解任务等基准测试上进行的大量实验表明，Struct-X 显著提高了 LLM 推理能力，证明了结构化数据增强在改善具有复杂输入上下文的 LLM 推理方面的有效性。

Title: Crafting the Path: Robust Query Rewriting for Information Retrieval

• Authors: Ingeol Baek, Jimin Lee, Joonho Yang, Hwanhee Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12529
• Pdf URL: https://arxiv.org/pdf/2407.12529
• Copy Paste: [[2407.12529]] Crafting the Path: Robust Query Rewriting for Information Retrieval(https://arxiv.org/abs/2407.12529)
• Keywords: language model, llm
• Abstract: Query rewriting aims to generate a new query that can complement the original query to improve the information retrieval system. Recent studies on query rewriting, such as query2doc (Q2D), query2expand (Q2E) and querey2cot (Q2C), rely on the internal knowledge of Large Language Models (LLMs) to generate a relevant passage to add information to the query. Nevertheless, the efficacy of these methodologies may markedly decline in instances where the requisite knowledge is not encapsulated within the model's intrinsic parameters. In this paper, we propose a novel structured query rewriting method called Crafting the Path tailored for retrieval systems. Crafting the Path involves a three-step process that crafts query-related information necessary for finding the passages to be searched in each step. Specifically, the Crafting the Path begins with Query Concept Comprehension, proceeds to Query Type Identification, and finally conducts Expected Answer Extraction. Experimental results show that our method outperforms previous rewriting methods, especially in less familiar domains for LLMs. We demonstrate that our method is less dependent on the internal parameter knowledge of the model and generates queries with fewer factual inaccuracies. Furthermore, we observe that Crafting the Path has less latency compared to the baselines.
• 摘要：查询重写旨在生成可以补充原始查询的新查询，以改进信息检索系统。查询重写的最新研究，例如 query2doc (Q2D)、query2expand (Q2E) 和 querey2cot (Q2C)，依赖于大型语言模型 (LLM) 的内部知识来生成相关段落以向查询添加信息。然而，当必要的知识未封装在模型的内在参数中时，这些方法的有效性可能会显著下降。在本文中，我们提出了一种针对检索系统量身定制的新型结构化查询重写方法，称为 Crafting the Path。Crafting the Path 涉及一个三步过程，该过程在每个步骤中制作查找要搜索的段落所需的查询相关信息。具体而言，Crafting the Path 从查询概念理解开始，然后进行查询类型识别，最后进行预期答案提取。实验结果表明，我们的方法优于以前的重写方法，尤其是在 LLM 不太熟悉的领域。我们证明，我们的方法不太依赖模型的内部参数知识，并且生成的查询具有更少的事实错误。此外，我们观察到与基线相比，Crafting the Path 的延迟更少。

Title: Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Models

• Authors: Xihe Qiu, Haoyu Wang, Xiaoyu Tan, Chao Qu, Yujie Xiong, Yuan Cheng, Yinghui Xu, Wei Chu, Yuan Qi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12532
• Pdf URL: https://arxiv.org/pdf/2407.12532
• Copy Paste: [[2407.12532]] Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Models(https://arxiv.org/abs/2407.12532)
• Keywords: language model, llm, agent
• Abstract: Effective collaboration in multi-agent systems requires communicating goals and intentions between agents. Current agent frameworks often suffer from dependencies on single-agent execution and lack robust inter-module communication, frequently leading to suboptimal multi-agent reinforcement learning (MARL) policies and inadequate task coordination. To address these challenges, we present a framework for training large language models (LLMs) as collaborative agents to enable coordinated behaviors in cooperative MARL. Each agent maintains a private intention consisting of its current goal and associated sub-tasks. Agents broadcast their intentions periodically, allowing other agents to infer coordination tasks. A propagation network transforms broadcast intentions into teammate-specific communication messages, sharing relevant goals with designated teammates. The architecture of our framework is structured into planning, grounding, and execution modules. During execution, multiple agents interact in a downstream environment and communicate intentions to enable coordinated behaviors. The grounding module dynamically adapts comprehension strategies based on emerging coordination patterns, while feedback from execution agents influnces the planning module, enabling the dynamic re-planning of sub-tasks. Results in collaborative environment simulation demonstrate intention propagation reduces miscoordination errors by aligning sub-task dependencies between agents. Agents learn when to communicate intentions and which teammates require task details, resulting in emergent coordinated behaviors. This demonstrates the efficacy of intention sharing for cooperative multi-agent RL based on LLMs.
• 摘要：多智能体系统中的有效协作需要在智能体之间传达目标和意图。当前的智能体框架通常依赖于单智能体执行，并且缺乏强大的模块间通信，这经常导致多智能体强化学习 (MARL) 策略不理想以及任务协调不足。为了应对这些挑战，我们提出了一个框架，用于训练大型语言模型 (LLM) 作为协作智能体，以实现合作 MARL 中的协调行为。每个智能体都维护一个由其当前目标和相关子任务组成的私人意图。智能体定期广播其意图，允许其他智能体推断协调任务。传播网络将广播意图转换为特定于队友的通信消息，与指定的队友共享相关目标。我们框架的架构分为规划、基础和执行模块。在执行过程中，多个智能体在下游环境中交互并传达意图以实现协调行为。基础模块根据新出现的协调模式动态调整理解策略，而执行智能体的反馈会影响规划模块，从而实现子任务的动态重新规划。协作环境模拟的结果表明，意图传播通过协调代理之间的子任务依赖关系减少了协调错误。代理了解何时传达意图以及哪些队友需要任务详细信息，从而产生协调行为。这证明了基于 LLM 的合作多代理 RL 的意图共享的有效性。

Title: E5-V: Universal Embeddings with Multimodal Large Language Models

• Authors: Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
• Subjects: cs.CL, cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.12580
• Pdf URL: https://arxiv.org/pdf/2407.12580
• Copy Paste: [[2407.12580]] E5-V: Universal Embeddings with Multimodal Large Language Models(https://arxiv.org/abs/2407.12580)
• Keywords: language model, llm, prompt
• Abstract: Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings. Our findings highlight the significant potential of MLLMs in representing multimodal inputs compared to previous approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning. We propose a single modality training approach for E5-V, where the model is trained exclusively on text pairs. This method demonstrates significant improvements over traditional multimodal training on image-text pairs, while reducing training costs by approximately 95%. Additionally, this approach eliminates the need for costly multimodal training data collection. Extensive experiments across four types of tasks demonstrate the effectiveness of E5-V. As a universal multimodal model, E5-V not only achieves but often surpasses state-of-the-art performance in each task, despite being trained on a single modality.
• 摘要：多模态大型语言模型 (MLLM) 在一般视觉和语言理解方面已显示出令人鼓舞的进步。然而，使用 MLLM 表示多模态信息仍未得到广泛探索。在这项工作中，我们引入了一个新框架 E5-V，旨在使 MLLM 适应实现通用多模态嵌入。我们的研究结果强调了与以前的方法相比，MLLM 在表示多模态输入方面的巨大潜力。通过利用带有提示的 MLLM，E5-V 有效地弥合了不同类型输入之间的模态差距，即使没有微调，在多模态嵌入中也表现出强大的性能。我们为 E5-V 提出了一种单模态训练方法，其中模型专门针对文本对进行训练。与传统的图像文本对多模态训练相比，该方法表现出显着的改进，同时将训练成本降低了约 95%。此外，这种方法消除了昂贵的多模态训练数据收集的需要。在四种类型的任务中进行的大量实验证明了 E5-V 的有效性。作为一种通用的多模态模型，尽管采用单一模态进行训练，E5-V 不仅在每项任务中达到而且经常超越最先进的性能。

Title: Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences

• Authors: Claudio Pinhanez, Paulo Cavalin, Luciana Storto, Thomas Fimbow, Alexander Cobbinah, Julio Nogima, Marisa Vasconcelos, Pedro Domingues, Priscila de Souza Mizukami, Nicole Grell, Majoí Gongora, Isabel Gonçalves
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12620
• Pdf URL: https://arxiv.org/pdf/2407.12620
• Copy Paste: [[2407.12620]] Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences(https://arxiv.org/abs/2407.12620)
• Keywords: language model, llm
• Abstract: Since 2022 we have been exploring application areas and technologies in which Artificial Intelligence (AI) and modern Natural Language Processing (NLP), such as Large Language Models (LLMs), can be employed to foster the usage and facilitate the documentation of Indigenous languages which are in danger of disappearing. We start by discussing the decreasing diversity of languages in the world and how working with Indigenous languages poses unique ethical challenges for AI and NLP. To address those challenges, we propose an alternative development AI cycle based on community engagement and usage. Then, we report encouraging results in the development of high-quality machine learning translators for Indigenous languages by fine-tuning state-of-the-art (SOTA) translators with tiny amounts of data and discuss how to avoid some common pitfalls in the process. We also present prototypes we have built in projects done in 2023 and 2024 with Indigenous communities in Brazil, aimed at facilitating writing, and discuss the development of Indigenous Language Models (ILMs) as a replicable and scalable way to create spell-checkers, next-word predictors, and similar tools. Finally, we discuss how we envision a future for language documentation where dying languages are preserved as interactive language models.
• 摘要：自 2022 年以来，我们一直在探索人工智能 (AI) 和现代自然语言处理 (NLP)（例如大型语言模型 (LLM)）的应用领域和技术，以促进濒临消失的土著语言的使用和记录。我们首先讨论世界上语言多样性的减少，以及使用土著语言对人工智能和 NLP 带来的独特道德挑战。为了应对这些挑战，我们提出了一种基于社区参与和使用的替代开发 AI 周期。然后，我们报告了通过使用少量数据对最先进的 (SOTA) 翻译器进行微调来开发土著语言高质量机器学习翻译器的令人鼓舞的结果，并讨论了如何避免此过程中的一些常见陷阱。我们还展示了我们在 2023 年和 2024 年与巴西土著社区合作开展的项目中构建的原型，旨在促进写作，并讨论了土著语言模型 (ILM) 的开发，作为创建拼写检查器、下一个单词预测器和类似工具的可复制和可扩展的方式。最后，我们讨论了我们如何设想语言文档的未来，即将消亡的语言保存为交互式语言模型。

Title: Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

• Authors: Aman Sinha, Timothee Mickus, Marianne Clausel, Mathieu Constant, Xavier Coubez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12626
• Pdf URL: https://arxiv.org/pdf/2407.12626
• Copy Paste: [[2407.12626]] Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?(https://arxiv.org/abs/2407.12626)
• Keywords: language model
• Abstract: The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in-chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.
• 摘要：预训练语言模型 (PLM) 在一系列用例中的成功，促使 NLP 社区投入大量资金来构建特定领域的基础模型。另一方面，在生物医学应用等关键任务环境中，其他方面也是首要因素，其中之一就是模型对其自身不确定性进行合理估计的能力。在本研究中，我们将从它们如何影响模型输出概率分布的熵的角度来讨论这两个要求。我们发现，领域特异性和不确定性意识通常可以成功结合，但手头的具体任务更重要。

Title: Patch-Level Training for Large Language Models

• Authors: Chenze Shao, Fandong Meng, Jie Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12665
• Pdf URL: https://arxiv.org/pdf/2407.12665
• Copy Paste: [[2407.12665]] Patch-Level Training for Large Language Models(https://arxiv.org/abs/2407.12665)
• Keywords: language model, llm
• Abstract: As Large Language Models (LLMs) achieve remarkable progress in language understanding and generation, their training efficiency has become a critical concern. Traditionally, LLMs are trained to predict the next token in a sequence. Despite the success of token-level training, it suffers from considerable computational costs due to the need to process an extensive number of tokens. To mitigate this issue, this paper introduces patch-level training for LLMs, which reduces the sequence length by compressing multiple tokens into a single patch. During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch, thereby processing the majority of the training data at a significantly reduced computational cost. Following this, the model continues token-level training on the remaining training data to align with the inference mode. Experiments on a diverse range of models (370M-2.7B parameters) demonstrate that patch-level training can reduce overall computational costs to 0.5$\times$, without compromising the model performance compared to token-level training. Source code: \url{this https URL}.
• 摘要：随着大型语言模型 (LLM) 在语言理解和生成方面取得显著进展，其训练效率已成为一个关键问题。传统上，LLM 被训练来预测序列中的下一个标记。尽管标记级训练取得了成功，但由于需要处理大量标记，因此计算成本相当高。为了缓解这个问题，本文引入了 LLM 的补丁级训练，通过将多个标记压缩为单个补丁来缩短序列长度。在补丁级训练期间，我们向语言模型提供较短的补丁序列并训练它预测下一个补丁，从而以显著降低的计算成本处理大部分训练数据。此后，模型继续对剩余的训练数据进行标记级训练，以与推理模式保持一致。在各种模型（370M-2.7B 参数）上的实验表明，与标记级训练相比，补丁级训练可以将总体计算成本降低到 0.5$\times$，而不会影响模型性能。源代码：\url{this https URL}。

Title: Subgraph-Aware Training of Text-based Methods for Knowledge Graph Completion

• Authors: Youmin Ko, Hyemin Yang, Taeuk Kim, Hyunjoon Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12703
• Pdf URL: https://arxiv.org/pdf/2407.12703
• Copy Paste: [[2407.12703]] Subgraph-Aware Training of Text-based Methods for Knowledge Graph Completion(https://arxiv.org/abs/2407.12703)
• Keywords: language model
• Abstract: Fine-tuning pre-trained language models (PLMs) has recently shown a potential to improve knowledge graph completion (KGC). However, most PLM-based methods encode only textual information, neglecting various topological structures of knowledge graphs (KGs). In this paper, we empirically validate the significant relations between the structural properties of KGs and the performance of the PLM-based methods. To leverage the structural knowledge, we propose a Subgraph-Aware Training framework for KGC (SATKGC) that combines (i) subgraph-aware mini-batching to encourage hard negative sampling, and (ii) a new contrastive learning method to focus more on harder entities and harder negative triples in terms of the structural properties. To the best of our knowledge, this is the first study to comprehensively incorporate the structural inductive bias of the subgraphs into fine-tuning PLMs. Extensive experiments on four KGC benchmarks demonstrate the superiority of SATKGC. Our code is available.
• 摘要：微调预训练语言模型 (PLM) 最近显示出改善知识图谱补全 (KGC) 的潜力。然而，大多数基于 PLM 的方法仅编码文本信息，而忽略了知识图谱 (KG) 的各种拓扑结构。在本文中，我们通过经验验证了知识图谱的结构属性与基于 PLM 的方法的性能之间的重要关系。为了利用结构知识，我们提出了一种用于 KGC 的子图感知训练框架 (SATKGC)，该框架结合了 (i) 子图感知小批量以鼓励硬负采样，以及 (ii) 一种新的对比学习方法，在结构属性方面更多地关注更难的实体和更难的负三元组。据我们所知，这是第一项将子图的结构归纳偏差全面纳入微调 PLM 的研究。在四个 KGC 基准上进行的大量实验证明了 SATKGC 的优越性。我们的代码可用。

Title: Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?

• Authors: Ben Yao, Yazhou Zhang, Qiuchi Li, Jing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12725
• Pdf URL: https://arxiv.org/pdf/2407.12725
• Copy Paste: [[2407.12725]] Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?(https://arxiv.org/abs/2407.12725)
• Keywords: language model, llm, prompt
• Abstract: Elaborating a series of intermediate reasoning steps significantly improves the ability of large language models (LLMs) to solve complex problems, as such steps would evoke LLMs to think sequentially. However, human sarcasm understanding is often considered an intuitive and holistic cognitive process, in which various linguistic, contextual, and emotional cues are integrated to form a comprehensive understanding of the speaker's true intention, which is argued not be limited to a step-by-step reasoning process. To verify this argument, we introduce a new prompting framework called SarcasmCue, which contains four prompting strategies, $viz.$ chain of contradiction (CoC), graph of cues (GoC), bagging of cues (BoC) and tensor of cues (ToC), which elicits LLMs to detect human sarcasm by considering sequential and non-sequential prompting methods. Through a comprehensive empirical comparison on four benchmarking datasets, we show that the proposed four prompting methods outperforms standard IO prompting, CoT and ToT with a considerable margin, and non-sequential prompting generally outperforms sequential prompting.
• 摘要：精心设计一系列中间推理步骤可显著提高大型语言模型 (LLM) 解决复杂问题的能力，因为这些步骤将促使 LLM 进行连续思考。然而，人类对讽刺的理解通常被认为是一种直观而整体的认知过程，其中整合了各种语言、语境和情感线索，以形成对说话者真实意图的全面理解，而这被认为不仅限于逐步的推理过程。为了验证这一论点，我们引入了一个新的提示框架 SarcasmCue，它包含四种提示策略，即矛盾链 (CoC)、线索图 (GoC)、线索装袋 (BoC) 和线索张量 (ToC)，它通过考虑顺序和非顺序提示方法来引发 LLM 检测人类的讽刺。通过对四个基准数据集进行全面的实证比较，我们发现，所提出的四种提示方法比标准 IO 提示、CoT 和 ToT 有相当大的优势，而且非顺序提示通常比顺序提示有更好的效果。

Title: A LLM Benchmark based on the Minecraft Builder Dialog Agent Task

• Authors: Chris Madge, Massimo Poesio
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12734
• Pdf URL: https://arxiv.org/pdf/2407.12734
• Copy Paste: [[2407.12734]] A LLM Benchmark based on the Minecraft Builder Dialog Agent Task(https://arxiv.org/abs/2407.12734)
• Keywords: llm, agent
• Abstract: In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying complex structures, and human written instructions. We instead attempt to provide a comprehensive synthetic benchmark for testing builder agents over a series of distinct tasks that comprise of common building operations. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math.
• 摘要：在这项工作中，我们建议将 Minecraft 建造者任务调整为适合评估 LLM 在空间导向任务中的能力的 LLM 基准，并为建造者代理设计提供信息。以前的工作提出了具有不同复杂结构的语料库和人工编写的说明。相反，我们试图提供一个全面的综合基准，用于在一系列由常见建造操作组成的不同任务中测试建造者代理。我们相信这种方法使我们能够探究不同代理的具体优势和劣势，并测试 LLM 在空间推理和基于向量的数学这一具有挑战性的领域的能力。

Title: HDLCopilot: Hardware Design Library Querying with Natural Language

• Authors: Manar Abdelatty, Sherief Reda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12749
• Pdf URL: https://arxiv.org/pdf/2407.12749
• Copy Paste: [[2407.12749]] HDLCopilot: Hardware Design Library Querying with Natural Language(https://arxiv.org/abs/2407.12749)
• Keywords: llm
• Abstract: Hardware design engineers routinely work with multiple Process Design Kits (PDKs) from various fabrication labs, each containing several standard cell libraries, optimized for specific metric such as speed, power, or density. These libraries include multiple views such as liberty files for timing information, LEF files for abstract layout details, and technology LEF for process design rules. Navigating this complex landscape to retrieve specific information about gates or design rules is often time-consuming and error-prone. To address this, we present HDLCopilot, an LLM-powered PDK query system that allows engineers to streamline interactions with PDKs in natural language format, making information retrieval accurate and more efficient. HDLCopilot achieves an accuracy of 94.23\% on an evaluation set comprised of diverse and complex natural language queries. HDLCopilot positions itself as a powerful assistant in the hardware design process, enhancing productivity and reducing potential human errors.
• 摘要：硬件设计工程师通常使用来自不同制造实验室的多个工艺设计套件 (PDK)，每个套件包含多个标准单元库，针对速度、功率或密度等特定指标进行了优化。这些库包括多个视图，例如用于时序信息的自由文件、用于抽象布局细节的 LEF 文件和用于工艺设计规则的技术 LEF。浏览这个复杂的环境以检索有关门或设计规则的特定信息通常很耗时且容易出错。为了解决这个问题，我们推出了 HDLCopilot，这是一个由 LLM 提供支持的 PDK 查询系统，它允许工程师以自然语言格式简化与 PDK 的交互，使信息检索更加准确和高效。HDLCopilot 在由各种复杂的自然语言查询组成的评估集上实现了 94.23% 的准确率。HDLCopilot 将自己定位为硬件设计过程中的强大助手，可提高生产力并减少潜在的人为错误。

Title: LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

• Authors: Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, Ziwei Liu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12772
• Pdf URL: https://arxiv.org/pdf/2407.12772
• Copy Paste: [[2407.12772]] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models(https://arxiv.org/abs/2407.12772)
• Keywords: language model
• Abstract: The advances of large foundation models necessitate wide-coverage, low-cost, and zero-contamination benchmarks. Despite continuous exploration of language model evaluations, comprehensive studies on the evaluation of Large Multi-modal Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models to promote transparent and reproducible evaluations. Although LMMS-EVAL offers comprehensive coverage, we find it still falls short in achieving low cost and zero contamination. To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency. Additionally, we present Multimodal LIVEBENCH that utilizes continuously updating news and online forums to assess models' generalization abilities in the wild, featuring a low-cost and zero-contamination evaluation approach. In summary, our work highlights the importance of considering the evaluation trilemma and provides practical solutions to navigate the trade-offs in evaluating large multi-modal models, paving the way for more effective and reliable benchmarking of LMMs. We opensource our codebase and maintain leaderboard of LIVEBENCH at this https URL and this https URL.
• 摘要：大型基础模型的进步需要广泛覆盖、低成本和零污染的基准。尽管语言模型评估不断探索，但对大型多模态模型 (LMM) 评估的全面研究仍然有限。在这项工作中，我们引入了 LMMS-EVAL，这是一个统一且标准化的多模态基准框架，具有 50 多个任务和 10 多个模型，以促进透明和可重复的评估。尽管 LMMS-EVAL 提供了全面的覆盖范围，但我们发现它在实现低成本和零污染方面仍然存在不足。为了解决这个评估难题，我们进一步介绍了 LMMS-EVAL LITE，这是一个强调覆盖范围和效率的精简评估工具包。此外，我们提出了 Multimodal LIVEBENCH，它利用不断更新的新闻和在线论坛来评估模型在野外的泛化能力，具有低成本和零污染的评估方法。总之，我们的工作强调了考虑评估三难困境的重要性，并提供了实用的解决方案来权衡评估大型多模态模型的利弊，为更有效、更可靠的 LMM 基准测试铺平了道路。我们开放了我们的代码库，并在此 https URL 和此 https URL 上维护 LIVEBENCH 的排行榜。