2024-07-09

Title: AgriLLM: Harnessing Transformers for Farmer Queries

• Authors: Krish Didwania, Pratinav Seth, Aditya Kasliwal, Amit Agarwal
• Subjects: cs.CL, cs.ET
• Abstract URL: https://arxiv.org/abs/2407.04721
• Pdf URL: https://arxiv.org/pdf/2407.04721
• Copy Paste: [[2407.04721]] AgriLLM: Harnessing Transformers for Farmer Queries(https://arxiv.org/abs/2407.04721)
• Keywords: language model, llm
• Abstract: Agriculture, vital for global sustenance, necessitates innovative solutions due to a lack of organized domain experts, particularly in developing countries where many farmers are impoverished and cannot afford expert consulting. Initiatives like Farmers Helpline play a crucial role in such countries, yet challenges such as high operational costs persist. Automating query resolution can alleviate the burden on traditional call centers, providing farmers with immediate and contextually relevant information. The integration of Agriculture and Artificial Intelligence (AI) offers a transformative opportunity to empower farmers and bridge information gaps. Language models like transformers, the rising stars of AI, possess remarkable language understanding capabilities, making them ideal for addressing information gaps in agriculture. This work explores and demonstrates the transformative potential of Large Language Models (LLMs) in automating query resolution for agricultural farmers, leveraging their expertise in deciphering natural language and understanding context. Using a subset of a vast dataset of real-world farmer queries collected in India, our study focuses on approximately 4 million queries from the state of Tamil Nadu, spanning various sectors, seasonal crops, and query types.
• 摘要：农业对全球生计至关重要，但由于缺乏有组织的领域专家，农业需要创新解决方案，特别是在发展中国家，那里许多农民都很贫困，无法负担专家咨询费用。农民帮助热线等计划在这些国家发挥着至关重要的作用，但高运营成本等挑战依然存在。自动查询解决方案可以减轻传统呼叫中心的负担，为农民提供即时且与上下文相关的信息。农业和人工智能 (AI) 的结合提供了一个变革性的机会，可以赋予农民权力并弥合信息差距。像 transformers 这样的语言模型是人工智能的后起之秀，具有非凡的语言理解能力，使其成为解决农业信息差距的理想选择。这项工作探索并展示了大型语言模型 (LLM) 在自动为农业农民解决查询方面的变革潜力，利用他们在解读自然语言和理解上下文方面的专业知识。我们的研究使用在印度收集的大量现实世界农民查询数据集的子集，重点关注来自泰米尔纳德邦的约 400 万个查询，涵盖各个部门、季节性作物和查询类型。

Title: Re-Tuning: Overcoming the Compositionality Limits of Large Language Models with Recursive Tuning

• Authors: Eric Pasewark, Kyle Montgomery, Kefei Duan, Dawn Song, Chenguang Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04787
• Pdf URL: https://arxiv.org/pdf/2407.04787
• Copy Paste: [[2407.04787]] Re-Tuning: Overcoming the Compositionality Limits of Large Language Models with Recursive Tuning(https://arxiv.org/abs/2407.04787)
• Keywords: language model
• Abstract: We present a new method for large language models to solve compositional tasks. Although they have shown strong performance on traditional language understanding tasks, large language models struggle to solve compositional tasks, where the solution depends on solving smaller instances of the same problem. We propose a natural approach to solve compositional tasks recursively. Our method, Re-Tuning, tunes models to break down a problem into subproblems, solve those subproblems, and combine the results. We show that our method significantly improves model performance on three representative compositional tasks: integer addition, dynamic programming, and parity. Compared to state-of-the-art methods that keep intermediate steps towards solving the problems, Re-Tuning achieves significantly higher accuracy and is more GPU memory efficient.
• 摘要：我们提出了一种用于大型语言模型解决组合任务的新方法。尽管大型语言模型在传统语言理解任务上表现出色，但它们在解决组合任务方面却举步维艰，因为解决方案取决于解决同一问题的较小实例。我们提出了一种自然的递归解决组合任务的方法。我们的方法，即重新调整，可以调整模型以将问题分解为子问题，解决这些子问题，并合并结果。我们表明，我们的方法显著提高了三个代表性组合任务的模型性能：整数加法、动态规划和奇偶校验。与保留解决问题的中间步骤的最先进的方法相比，重新调整实现了更高的准确率，并且更节省 GPU 内存。

Title: Toucan: Many-to-Many Translation for 150 African Language Pairs

• Authors: AbdelRahim Elmadany, Ife Adebara, Muhammad Abdul-Mageed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04796
• Pdf URL: https://arxiv.org/pdf/2407.04796
• Copy Paste: [[2407.04796]] Toucan: Many-to-Many Translation for 150 African Language Pairs(https://arxiv.org/abs/2407.04796)
• Keywords: language model
• Abstract: We address a notable gap in Natural Language Processing (NLP) by introducing a collection of resources designed to improve Machine Translation (MT) for low-resource languages, with a specific focus on African languages. First, We introduce two language models (LMs), Cheetah-1.2B and Cheetah-3.7B, with 1.2 billion and 3.7 billion parameters respectively. Next, we finetune the aforementioned models to create toucan, an Afrocentric machine translation model designed to support 156 African language pairs. To evaluate Toucan, we carefully develop an extensive machine translation benchmark, dubbed AfroLingu-MT, tailored for evaluating machine translation. Toucan significantly outperforms other models, showcasing its remarkable performance on MT for African languages. Finally, we train a new model, spBLEU-1K, to enhance translation evaluation metrics, covering 1K languages, including 614 African languages. This work aims to advance the field of NLP, fostering cross-cultural understanding and knowledge exchange, particularly in regions with limited language resources such as Africa. The GitHub repository for the Toucan project is available at this https URL.
• 摘要：我们引入了一系列资源，旨在改进资源匮乏的语言（特别是非洲语言）的机器翻译 (MT)，从而弥补了自然语言处理 (NLP) 的一个显著差距。首先，我们引入了两个语言模型 (LM)，Cheetah-1.2B 和 Cheetah-3.7B，分别具有 12 亿和 37 亿个参数。接下来，我们对上述模型进行微调，以创建 toucan，这是一个以非洲为中心的机器翻译模型，旨在支持 156 个非洲语言对。为了评估 Toucan，我们精心开发了一个广泛的机器翻译基准，称为 AfroLingu-MT，专门用于评估机器翻译。Toucan 的表现明显优于其他模型，展示了其在非洲语言机器翻译方面的出色表现。最后，我们训练了一个新模型 spBLEU-1K，以增强翻译评估指标，涵盖 1K 种语言，包括 614 种非洲语言。这项工作旨在推动 NLP 领域的发展，促进跨文化理解和知识交流，特别是在非洲等语言资源有限的地区。 Toucan 项目的 GitHub 存储库可通过此 https URL 访问。

Title: Associative Recurrent Memory Transformer

• Authors: Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, Mikhail Burtsev
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04841
• Pdf URL: https://arxiv.org/pdf/2407.04841
• Copy Paste: [[2407.04841]] Associative Recurrent Memory Transformer(https://arxiv.org/abs/2407.04841)
• Keywords: long context
• Abstract: This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our approach, Associative Recurrent Memory Transformer (ARMT), is based on transformer self-attention for local context and segment-level recurrence for storage of task specific information distributed over a long context. We demonstrate that ARMT outperfors existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%. The source code for training and evaluation is available on github.
• 摘要：本文解决了为非常长的序列创建神经架构的挑战，该架构需要在每个时间步骤中花费恒定的时间来处理新信息。我们的方法，联想循环记忆变换器 (ARMT)，基于变换器自注意力来处理局部上下文，以及基于段级循环来存储分布在长上下文中的特定任务信息。我们证明，ARMT 在联想检索任务中的表现优于现有的替代方案，并通过回答超过 5000 万个标记的单一事实问题，以 79.9% 的准确率在最近的 BABILong 多任务长上下文基准中创下了新的性能记录。训练和评估的源代码可在 github 上找到。

Title: Statistical investigations into the geometry and homology of random programs

• Authors: Jon Sporring, Ken Friis Larsen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04854
• Pdf URL: https://arxiv.org/pdf/2407.04854
• Copy Paste: [[2407.04854]] Statistical investigations into the geometry and homology of random programs(https://arxiv.org/abs/2407.04854)
• Keywords: language model, gpt, prompt, chat
• Abstract: AI-supported programming has taken giant leaps with tools such as Meta's Llama and openAI's chatGPT. These are examples of stochastic sources of programs and have already greatly influenced how we produce code and teach programming. If we consider input to such models as a stochastic source, a natural question is, what is the relation between the input and the output distributions, between the chatGPT prompt and the resulting program? In this paper, we will show how the relation between random Python programs generated from chatGPT can be described geometrically and topologically using Tree-edit distances between the program's syntax trees and without explicit modeling of the underlying space. A popular approach to studying high-dimensional samples in a metric space is to use low-dimensional embedding using, e.g., multidimensional scaling. Such methods imply errors depending on the data and dimension of the embedding space. In this article, we propose to restrict such projection methods to purely visualization purposes and instead use geometric summary statistics, methods from spatial point statistics, and topological data analysis to characterize the configurations of random programs that do not rely on embedding approximations. To demonstrate their usefulness, we compare two publicly available models: ChatGPT-4 and TinyLlama, on a simple problem related to image processing. Application areas include understanding how questions should be asked to obtain useful programs; measuring how consistently a given large language model answers; and comparing the different large language models as a programming assistant. Finally, we speculate that our approach may in the future give new insights into the structure of programming languages.
• 摘要：借助 Meta 的 Llama 和 openAI 的 chatGPT 等工具，人工智能支持的编程取得了巨大飞跃。这些是程序随机源的示例，已经极大地影响了我们编写代码和教授编程的方式。如果我们将此类模型的输入视为随机源，那么一个自然的问题是，输入和输出分布之间、chatGPT 提示和生成的程序之间的关系是什么？在本文中，我们将展示如何使用程序语法树之间的树编辑距离在几何和拓扑上描述 chatGPT 生成的随机 Python 程序之间的关系，而无需对底层空间进行显式建模。研究度量空间中高维样本的一种流行方法是使用低维嵌入，例如使用多维缩放。此类方法意味着误差取决于嵌入空间的数据和维度。在本文中，我们建议将此类投影方法限制为纯粹的可视化目的，而是使用几何汇总统计、空间点统计方法和拓扑数据分析来表征不依赖于嵌入近似的随机程序的配置。为了证明它们的实用性，我们在一个与图像处理相关的简单问题上比较了两个公开可用的模型：ChatGPT-4 和 TinyLlama。应用领域包括了解如何提出问题以获得有用的程序；衡量给定的大型语言模型回答的一致性；以及作为编程助手比较不同的大型语言模型。最后，我们推测我们的方法将来可能会为编程语言的结构提供新的见解。

Title: Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs

• Authors: Mihir Parmar, Hanieh Deilamsalehy, Franck Dernoncourt, Seunghyun Yoon, Ryan A. Rossi, Trung Bui
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04855
• Pdf URL: https://arxiv.org/pdf/2407.04855
• Copy Paste: [[2407.04855]] Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs(https://arxiv.org/abs/2407.04855)
• Keywords: language model, llm
• Abstract: Extractive summarization plays a pivotal role in natural language processing due to its wide-range applications in summarizing diverse content efficiently, while also being faithful to the original content. Despite significant advancement achieved in extractive summarization by Large Language Models (LLMs), these summaries frequently exhibit incoherence. An important aspect of the coherent summary is its readability for intended users. Although there have been many datasets and benchmarks proposed for creating coherent extractive summaries, none of them currently incorporate user intent to improve coherence in extractive summarization. Motivated by this, we propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback, offering valuable insights into how to improve coherence in extractive summaries. We utilize this dataset for aligning LLMs through supervised fine-tuning with natural language human feedback to enhance the coherence of their generated summaries. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (~10% Rouge-L) in terms of producing coherent summaries. We further utilize human feedback to benchmark results over instruction-tuned models such as FLAN-T5 which resulted in several interesting findings. Data and source code are available at this https URL.
• 摘要：抽取式摘要在自然语言处理中起着至关重要的作用，因为它在高效总结不同内容的同时还能忠实于原始内容。尽管大型语言模型 (LLM) 在抽取式摘要方面取得了重大进展，但这些摘要经常表现出不连贯性。连贯摘要的一个重要方面是它对目标用户的可读性。虽然已经提出了许多用于创建连贯的抽取式摘要的数据集和基准，但目前都没有结合用户意图来提高抽取式摘要的连贯性。受此启发，我们提出了一个系统创建的人工注释数据集，该数据集由五个公开数据集的连贯摘要和自然语言用户反馈组成，为如何提高抽取式摘要的连贯性提供了宝贵的见解。我们利用这个数据集通过有监督的微调和自然语言人工反馈来对齐 LLM，以增强其生成的摘要的连贯性。使用 Falcon-40B 和 Llama-2-13B 进行的初步实验表明，在生成连贯摘要方面，性能有显著提升（~10% Rouge-L）。我们进一步利用人工反馈对 FLAN-T5 等指令调整模型的结果进行基准测试，结果得出了一些有趣的发现。数据和源代码可在此 https URL 上找到。

Title: Automating Venture Capital: Founder assessment using LLM-powered segmentation, feature engineering and automated labeling techniques

• Authors: Ekin Ozince, Yiğit Ihlamur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04885
• Pdf URL: https://arxiv.org/pdf/2407.04885
• Copy Paste: [[2407.04885]] Automating Venture Capital: Founder assessment using LLM-powered segmentation, feature engineering and automated labeling techniques(https://arxiv.org/abs/2407.04885)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: This study explores the application of large language models (LLMs) in venture capital (VC) decision-making, focusing on predicting startup success based on founder characteristics. We utilize LLM prompting techniques, like chain-of-thought, to generate features from limited data, then extract insights through statistics and machine learning. Our results reveal potential relationships between certain founder characteristics and success, as well as demonstrate the effectiveness of these characteristics in prediction. This framework for integrating ML techniques and LLMs has vast potential for improving startup success prediction, with important implications for VC firms seeking to optimize their investment strategies.
• 摘要：本研究探讨了大型语言模型 (LLM) 在风险投资 (VC) 决策中的应用，重点是根据创始人特征预测初创企业的成功。我们利用 LLM 提示技术（如思路链）从有限的数据中生成特征，然后通过统计和机器学习提取见解。我们的结果揭示了某些创始人特征与成功之间的潜在关系，并证明了这些特征在预测中的有效性。这种将 ML 技术和 LLM 相结合的框架在改善初创企业成功预测方面具有巨大潜力，对于寻求优化投资策略的 VC 公司具有重要意义。

Title: MMSci: A Multimodal Multi-Discipline Dataset for PhD-Level Scientific Comprehension

• Authors: Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim, William Yang Wang
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.04903
• Pdf URL: https://arxiv.org/pdf/2407.04903
• Copy Paste: [[2407.04903]] MMSci: A Multimodal Multi-Discipline Dataset for PhD-Level Scientific Comprehension(https://arxiv.org/abs/2407.04903)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancement of Large Language Models (LLMs) and Large Multimodal Models (LMMs) has heightened the demand for AI-based scientific assistants capable of understanding scientific articles and figures. Despite progress, there remains a significant gap in evaluating models' comprehension of professional, graduate-level, and even PhD-level scientific content. Current datasets and benchmarks primarily focus on relatively simple scientific tasks and figures, lacking comprehensive assessments across diverse advanced scientific disciplines. To bridge this gap, we collected a multimodal, multidisciplinary dataset from open-access scientific articles published in Nature Communications journals. This dataset spans 72 scientific disciplines, ensuring both diversity and quality. We created benchmarks with various tasks and settings to comprehensively evaluate LMMs' capabilities in understanding scientific figures and content. Our evaluation revealed that these tasks are highly challenging: many open-source models struggled significantly, and even GPT-4V and GPT-4o faced difficulties. We also explored using our dataset as training resources by constructing visual instruction-following data, enabling the 7B LLaVA model to achieve performance comparable to GPT-4V/o on our benchmark. Additionally, we investigated the use of our interleaved article texts and figure images for pre-training LMMs, resulting in improvements on the material generation task. The source dataset, including articles, figures, constructed benchmarks, and visual instruction-following data, is open-sourced.
• 摘要：大型语言模型 (LLM) 和大型多模态模型 (LMM) 的快速发展增加了对能够理解科学文章和图表的基于 AI 的科学助手的需求。尽管取得了进展，但在评估模型对专业、研究生级别甚至博士级别科学内容的理解方面仍然存在巨大差距。当前的数据集和基准主要关注相对简单的科学任务和图表，缺乏对各种高级科学学科的全面评估。为了弥补这一差距，我们从《自然通讯》期刊上发表的开放获取科学文章中收集了一个多模态、多学科的数据集。该数据集涵盖 72 个科学学科，确保了多样性和质量。我们创建了具有各种任务和设置的基准，以全面评估 LMM 理解科学图表和内容的能力。我们的评估表明，这些任务极具挑战性：许多开源模型都遇到了很大困难，甚至 GPT-4V 和 GPT-4o 也遇到了困难。我们还探索了通过构建视觉指令跟踪数据将我们的数据集用作训练资源，从而使 7B LLaVA 模型在我们的基准上实现与 GPT-4V/o 相当的性能。此外，我们研究了将交错的文章文本和图片用于预训练 LMM 的情况，从而改进了材料生成任务。源数据集（包括文章、图片、构建的基准和视觉指令跟踪数据）都是开源的。

Title: Granular Privacy Control for Geolocation with Vision Language Models

• Authors: Ethan Mendes, Yang Chen, James Hays, Sauvik Das, Wei Xu, Alan Ritter
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.04952
• Pdf URL: https://arxiv.org/pdf/2407.04952
• Copy Paste: [[2407.04952]] Granular Privacy Control for Geolocation with Vision Language Models(https://arxiv.org/abs/2407.04952)
• Keywords: language model, gpt, prompt, chat
• Abstract: Vision Language Models (VLMs) are rapidly advancing in their capability to answer information-seeking questions. As these models are widely deployed in consumer applications, they could lead to new privacy risks due to emergent abilities to identify people in photos, geolocate images, etc. As we demonstrate, somewhat surprisingly, current open-source and proprietary VLMs are very capable image geolocators, making widespread geolocation with VLMs an immediate privacy risk, rather than merely a theoretical future concern. As a first step to address this challenge, we develop a new benchmark, GPTGeoChat, to test the ability of VLMs to moderate geolocation dialogues with users. We collect a set of 1,000 image geolocation conversations between in-house annotators and GPT-4v, which are annotated with the granularity of location information revealed at each turn. Using this new dataset, we evaluate the ability of various VLMs to moderate GPT-4v geolocation conversations by determining when too much location information has been revealed. We find that custom fine-tuned models perform on par with prompted API-based models when identifying leaked location information at the country or city level; however, fine-tuning on supervised data appears to be needed to accurately moderate finer granularities, such as the name of a restaurant or building.
• 摘要：视觉语言模型 (VLM) 在回答信息搜索问题的能力方面正在迅速进步。随着这些模型被广泛部署在消费者应用程序中，它们可能会因识别照片中的人、对图像进行地理定位等新兴能力而导致新的隐私风险。正如我们所展示的，令人惊讶的是，当前的开源和专有 VLM 是非常强大的图像地理定位器，这使得使用 VLM 进行广泛的地理定位成为一种直接的隐私风险，而不仅仅是理论上的未来问题。作为应对这一挑战的第一步，我们开发了一个新的基准 GPTGeoChat，以测试 VLM 调节与用户的地理定位对话的能力。我们收集了内部注释者和 GPT-4v 之间的一组 1,000 个图像地理定位对话，这些对话都标注了每次显示的位置信息的粒度。使用这个新的数据集，我们通过确定何时透露了过多的位置信息来评估各种 VLM 调节 GPT-4v 地理定位对话的能力。我们发现，在国家或城市级别识别泄露的位置信息时，自定义微调模型的表现与基于提示的 API 模型相当；然而，似乎需要对监督数据进行微调，以准确调节更细的粒度，例如餐厅或建筑物的名称。

Title: Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression

• Authors: Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, Vivek Srikumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04965
• Pdf URL: https://arxiv.org/pdf/2407.04965
• Copy Paste: [[2407.04965]] Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression(https://arxiv.org/abs/2407.04965)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly deployed in real-world scenarios with the help of recent model compression techniques. Such momentum towards local deployment means the use of compressed LLMs will widely impact a large population. However, prior analysis works often prioritize on preserving perplexity which is a direct analogy to training loss. The impact of compression method on other critical aspects of model behavior, particularly safety, still calls for a systematic assessment. To this end, we investigate the impact of model compression on four dimensions: 1) degeneration harm, i.e., bias and toxicity in generation; 2) representational harm, i.e., biases in discriminative tasks; 3) dialect bias; 4) language modeling and downstream task performance. We cover a wide spectrum of LLM compression techniques, including structured pruning, un/semi-structured ones, and quantization. Our analyses reveal that compression can lead to unexpected consequences. Although compression may unintentionally remedy LLMs' degeneration harm, it can still exacerbate on the representational harm axis. Moreover, there is a divergent impact on different protected groups as the compression rate grows. Finally, different compression methods have drastically different safety impacts, e.g., quantization mostly preserves bias while pruning degrades quickly. Our findings underscore the importance of integrating safety assessments into the development of compressed LLMs to ensure their reliability across real-world applications. Our full results are available here: \url{this https URL}
• 摘要：借助最近的模型压缩技术，大型语言模型 (LLM) 越来越多地部署在现实世界场景中。这种本地部署的势头意味着使用压缩的 LLM 将广泛影响大量人群。然而，先前的分析工作通常优先考虑保留困惑度，这与训练损失直接类似。压缩方法对模型行为其他关键方面的影响，特别是安全性，仍然需要系统评估。为此，我们研究了模型压缩对四个维度的影响：1）退化危害，即生成中的偏见和毒性；2）表征危害，即判别任务中的偏见；3）方言偏见；4）语言建模和下游任务性能。我们涵盖了广泛的 LLM 压缩技术，包括结构化修剪、非/半结构化修剪和量化。我们的分析表明，压缩可能会导致意想不到的后果。虽然压缩可能会无意中弥补 LLM 的退化危害，但它仍然会在表征危害轴上加剧。此外，随着压缩率的增加，对不同受保护组的影响也不同。最后，不同的压缩方法对安全性的影响截然不同，例如，量化大多会保留偏差，而修剪会迅速降低。我们的研究结果强调了将安全评估整合到压缩 LLM 的开发中以确保其在实际应用中的可靠性的重要性。我们的完整结果可在此处查看：\url{此 https URL}

Title: EVA-Score: Evaluation of Long-form Summarization on Informativeness through Extraction and Validation

• Authors: Yuchen Fan, Xin Zhong, Chengsi Wang, Gaoche Wu, Bowen Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04969
• Pdf URL: https://arxiv.org/pdf/2407.04969
• Copy Paste: [[2407.04969]] EVA-Score: Evaluation of Long-form Summarization on Informativeness through Extraction and Validation(https://arxiv.org/abs/2407.04969)
• Keywords: language model, gpt, llm
• Abstract: Summarization is a fundamental task in natural language processing (NLP) and since large language models (LLMs), such as GPT-4 and Claude, come out, increasing attention has been paid to long-form summarization whose input sequences are much longer, indicating more information contained. The current evaluation metrics either use similarity-based metrics like ROUGE and BERTScore which rely on similarity and fail to consider informativeness or LLM-based metrics, lacking quantitative analysis of information richness and are rather subjective. In this paper, we propose a new evaluation metric called EVA-Score using Atomic Fact Chain Generation and Document-level Relation Extraction together to automatically calculate the informativeness and give a definite number as an information score. Experiment results show that our metric shows a state-of-the-art correlation with humans. We also re-evaluate the performance of LLMs on long-form summarization comprehensively from the information aspect, forecasting future ways to use LLMs for long-form summarization.
• 摘要：摘要是自然语言处理（NLP）中的一项基本任务，自从 GPT-4 和 Claude 等大型语言模型（LLM）问世以来，长篇摘要受到了越来越多的关注，其输入序列更长，意味着包含的信息量更大。目前的评估指标要么使用基于相似度的指标，如 ROUGE 和 BERTScore，这些指标依赖于相似度而没有考虑信息量，要么使用基于 LLM 的指标，缺乏对信息丰富度的定量分析，而且比较主观。在本文中，我们提出了一种新的评估指标 EVA-Score，它结合使用原子事实链生成和文档级关系提取来自动计算信息量并给出一个确定的数字作为信息分数。实验结果表明，我们的指标与人类的相关性达到了最佳水平。我们还从信息量的角度重新全面评估了 LLM 在长篇摘要方面的表现，预测了未来使用 LLM 进行长篇摘要的方式。

Title: TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs

• Authors: Cheng Wang, Xinyang Lu, See-Kiong Ng, Bryan Kian Hsiang Low
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04981
• Pdf URL: https://arxiv.org/pdf/2407.04981
• Copy Paste: [[2407.04981]] TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs(https://arxiv.org/abs/2407.04981)
• Keywords: language model, llm
• Abstract: The rapid evolution of large language models (LLMs) represents a substantial leap forward in natural language understanding and generation. However, alongside these advancements come significant challenges related to the accountability and transparency of LLM responses. Reliable source attribution is essential to adhering to stringent legal and regulatory standards, including those set forth by the General Data Protection Regulation. Despite the well-established methods in source attribution within the computer vision domain, the application of robust attribution frameworks to natural language processing remains underexplored. To bridge this gap, we propose a novel and versatile TRansformer-based Attribution framework using Contrastive Embeddings called TRACE that, in particular, exploits contrastive learning for source attribution. We perform an extensive empirical evaluation to demonstrate the performance and efficiency of TRACE in various settings and show that TRACE significantly improves the ability to attribute sources accurately, making it a valuable tool for enhancing the reliability and trustworthiness of LLMs.
• 摘要：大型语言模型 (LLM) 的快速发展代表着自然语言理解和生成的重大飞跃。然而，伴随着这些进步而来的是与 LLM 响应的可追溯性和透明度相关的重大挑战。可靠的来源归因对于遵守严格的法律和监管标准至关重要，包括《通用数据保护条例》规定的标准。尽管在计算机视觉领域中，来源归因的方法已经很成熟，但强大的归因框架在自然语言处理中的应用仍未得到充分探索。为了弥补这一差距，我们提出了一种新颖且通用的基于 TRansformer 的归因框架，该框架使用对比嵌入，称为 TRACE，特别是利用对比学习进行来源归因。我们进行了广泛的实证评估，以展示 TRACE 在各种环境中的性能和效率，并表明 TRACE 显着提高了准确归因来源的能力，使其成为增强 LLM 可靠性和可信度的宝贵工具。

Title: Progress or Regress? Self-Improvement Reversal in Post-training

• Authors: Ting Wu, Xuefeng Li, Pengfei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05013
• Pdf URL: https://arxiv.org/pdf/2407.05013
• Copy Paste: [[2407.05013]] Progress or Regress? Self-Improvement Reversal in Post-training(https://arxiv.org/abs/2407.05013)
• Keywords: language model, llm
• Abstract: Self-improvement through post-training methods such as iterative preference learning has been acclaimed for enhancing the problem-solving capabilities (e.g., mathematical reasoning) of Large Language Models (LLMs) without human intervention. However, as exploration deepens, it becomes crucial to assess whether these improvements genuinely signify progress in solving more challenging problems or if they could lead to unintended regressions. To address this, we propose a comprehensive evaluative framework that goes beyond the superficial pass@1 metric to scrutinize the underlying enhancements of post-training paradigms for self-improvement. Through rigorous experimentation and analysis across diverse problem-solving tasks, the empirical results point out the phenomenon of \emph{self-improvement reversal}, where models showing improved performance across benchmarks will paradoxically exhibit declines in broader, essential capabilities, like output diversity and out-of-distribution (OOD) generalization. These findings indicate that current self-improvement practices through post-training are inadequate for equipping models to tackle more complex problems. Furthermore, they underscore the necessity of our critical evaluation metrics in discerning the \emph{progress or regress} dichotomy for self-improving LLMs.
• 摘要：通过迭代偏好学习等后训练方法进行自我改进，因无需人工干预即可增强大型语言模型 (LLM) 的解决问题能力（例如数学推理）而广受好评。然而，随着探索的深入，评估这些改进是否真正意味着在解决更具挑战性的问题方面取得了进展，或者是否会导致意外的回归变得至关重要。为了解决这个问题，我们提出了一个全面的评估框架，该框架超越了表面的 pass@1 指标，以仔细审查后训练范式对自我改进的根本增强。通过对各种问题解决任务进行严格的实验和分析，实证结果指出了 \emph{自我改进逆转} 现象，其中在基准测试中表现出改进性能的模型将自相矛盾地表现出更广泛、更基本的能力的下降，例如输出多样性和分布外 (OOD) 泛化。这些发现表明，目前通过后期训练进行的自我完善实践不足以使模型具备解决更复杂问题的能力。此外，它们还强调了我们的关键评估指标在辨别自我完善的 LLM 的 \emph{进步或退步} 二分法方面的必要性。

Title: How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questions

• Authors: Bojana Bašaragin, Adela Ljajić, Darija Medvecki, Lorenzo Cassano, Miloš Košprdić, Nikola Milošević
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05015
• Pdf URL: https://arxiv.org/pdf/2407.05015
• Copy Paste: [[2407.05015]] How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questions(https://arxiv.org/abs/2407.05015)
• Keywords: language model, gpt, llm, prompt, retrieval-augmented generation
• Abstract: Large language models (LLMs) have recently become the leading source of answers for users' questions online. Despite their ability to offer eloquent answers, their accuracy and reliability can pose a significant challenge. This is especially true for sensitive domains such as biomedicine, where there is a higher need for factually correct answers. This paper introduces a biomedical retrieval-augmented generation (RAG) system designed to enhance the reliability of generated responses. The system is based on a fine-tuned LLM for the referenced question-answering, where retrieved relevant abstracts from PubMed are passed to LLM's context as input through a prompt. Its output is an answer based on PubMed abstracts, where each statement is referenced accordingly, allowing the users to verify the answer. Our retrieval system achieves an absolute improvement of 23% compared to the PubMed search engine. Based on the manual evaluation on a small sample, our fine-tuned LLM component achieves comparable results to GPT-4 Turbo in referencing relevant abstracts. We make the dataset used to fine-tune the models and the fine-tuned models based on Mistral-7B-instruct-v0.1 and v0.2 publicly available.
• 摘要：大型语言模型 (LLM) 最近已成为在线用户问题答案的主要来源。尽管它们能够提供雄辩的答案，但它们的准确性和可靠性可能带来重大挑战。对于生物医学等敏感领域尤其如此，因为这些领域对事实正确答案的需求更高。本文介绍了一种生物医学检索增强生成 (RAG) 系统，旨在提高生成的响应的可靠性。该系统基于针对参考问答的微调 LLM，其中从 PubMed 检索到的相关摘要通过提示作为输入传递到 LLM 的上下文中。它的输出是基于 PubMed 摘要的答案，其中每个语句都相应地引用，允许用户验证答案。与 PubMed 搜索引擎相比，我们的检索系统实现了 23% 的绝对改进。基于对小样本的手动评估，我们微调的 LLM 组件在引用相关摘要方面取得了与 GPT-4 Turbo 相当的结果。我们公开用于微调模型的数据集以及基于 Mistral-7B-instruct-v0.1 和 v0.2 的微调模型。

Title: Enhance the Robustness of Text-Centric Multimodal Alignments

• Authors: Ting-Yu Yen, Yun-Da Tsai, Keng-Te Liao, Shou-De Lin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.05036
• Pdf URL: https://arxiv.org/pdf/2407.05036
• Copy Paste: [[2407.05036]] Enhance the Robustness of Text-Centric Multimodal Alignments(https://arxiv.org/abs/2407.05036)
• Keywords: language model, llm, prompt
• Abstract: Converting different modalities into general text, serving as input prompts for large language models (LLMs), is a common method to align multimodal models when there is limited pairwise data. This text-centric approach leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation. This enables downstream models to effectively interpret various modal inputs. This study assesses the quality and robustness of multimodal representations in the presence of missing entries, noise, or absent modalities, revealing that current text-centric alignment methods compromise downstream robustness. To address this issue, we propose a new text-centric approach that achieves superior robustness compared to previous methods across various modalities in different settings. Our findings highlight the potential of this approach to enhance the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications.
• 摘要：将不同模态转换为一般文本，作为大型语言模型 (LLM) 的输入提示，是在成对数据有限的情况下对齐多模态模型的常用方法。这种以文本为中心的方法利用文本作为模态空间的独特属性，将不同的输入转换为统一的文本表示。这使下游模型能够有效地解释各种模态输入。本研究评估了在存在缺失条目、噪声或缺失模态的情况下多模态表示的质量和稳健性，结果表明当前以文本为中心的对齐方法会损害下游的稳健性。为了解决这个问题，我们提出了一种新的以文本为中心的方法，与以前在不同设置的各种模态中的方法相比，该方法实现了更高的稳健性。我们的研究结果凸显了这种方法增强多模态表示的稳健性和适应性的潜力，为动态和现实世界的应用提供了一种有希望的解决方案。

Title: Cross-Lingual Word Alignment for ASEAN Languages with Contrastive Learning

• Authors: Jingshen Zhang, Xinying Qiu, Teng Shen, Wenyu Wang, Kailin Zhang, Wenhe Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05054
• Pdf URL: https://arxiv.org/pdf/2407.05054
• Copy Paste: [[2407.05054]] Cross-Lingual Word Alignment for ASEAN Languages with Contrastive Learning(https://arxiv.org/abs/2407.05054)
• Keywords: language model
• Abstract: Cross-lingual word alignment plays a crucial role in various natural language processing tasks, particularly for low-resource languages. Recent study proposes a BiLSTM-based encoder-decoder model that outperforms pre-trained language models in low-resource settings. However, their model only considers the similarity of word embedding spaces and does not explicitly model the differences between word embeddings. To address this limitation, we propose incorporating contrastive learning into the BiLSTM-based encoder-decoder framework. Our approach introduces a multi-view negative sampling strategy to learn the differences between word pairs in the shared cross-lingual embedding space. We evaluate our model on five bilingual aligned datasets spanning four ASEAN languages: Lao, Vietnamese, Thai, and Indonesian. Experimental results demonstrate that integrating contrastive learning consistently improves word alignment accuracy across all datasets, confirming the effectiveness of the proposed method in low-resource scenarios. We will release our data set and code to support future research on ASEAN or more low-resource word alignment.
• 摘要：跨语言词对齐在各种自然语言处理任务中起着至关重要的作用，特别是对于资源匮乏的语言。最近的研究提出了一种基于 BiLSTM 的编码器-解码器模型，该模型在资源匮乏的环境中表现优于预训练的语言模型。然而，他们的模型只考虑了词嵌入空间的相似性，并没有明确地模拟词嵌入之间的差异。为了解决这个限制，我们建议将对比学习纳入基于 BiLSTM 的编码器-解码器框架。我们的方法引入了一种多视角负采样策略来学习共享跨语言嵌入空间中单词对之间的差异。我们在五个双语对齐数据集上评估了我们的模型，这些数据集涵盖了四种东盟语言：老挝语、越南语、泰语和印尼语。实验结果表明，整合对比学习可以持续提高所有数据集的词对齐准确率，证实了所提方法在资源匮乏场景中的有效性。我们将发布我们的数据集和代码，以支持未来对东盟或更多资源匮乏的词对齐的研究。

Title: BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Records

• Authors: Weimin Lyu, Zexin Bi, Fusheng Wang, Chao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05213
• Pdf URL: https://arxiv.org/pdf/2407.05213
• Copy Paste: [[2407.05213]] BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Records(https://arxiv.org/abs/2407.05213)
• Keywords: language model
• Abstract: The advent of clinical language models integrated into electronic health records (EHR) for clinical decision support has marked a significant advancement, leveraging the depth of clinical notes for improved decision-making. Despite their success, the potential vulnerabilities of these models remain largely unexplored. This paper delves into the realm of backdoor attacks on clinical language models, introducing an innovative attention-based backdoor attack method, BadCLM (Bad Clinical Language Models). This technique clandestinely embeds a backdoor within the models, causing them to produce incorrect predictions when a pre-defined trigger is present in inputs, while functioning accurately otherwise. We demonstrate the efficacy of BadCLM through an in-hospital mortality prediction task with MIMIC III dataset, showcasing its potential to compromise model integrity. Our findings illuminate a significant security risk in clinical decision support systems and pave the way for future endeavors in fortifying clinical language models against such vulnerabilities.
• 摘要：临床语言模型被集成到电子健康记录 (EHR) 中用于临床决策支持，这标志着一项重大进步，它利用临床记录的深度来改善决策。尽管这些模型取得了成功，但其潜在漏洞在很大程度上仍未被探索。本文深入研究了临床语言模型的后门攻击领域，介绍了一种创新的基于注意力的后门攻击方法 BadCLM（不良临床语言模型）。这种技术秘密地在模型中嵌入后门，导致它们在输入中存在预定义触发器时产生错误的预测，而在其它情况下则准确运行。我们通过使用 MIMIC III 数据集的住院死亡率预测任务证明了 BadCLM 的有效性，展示了其破坏模型完整性的潜力。我们的研究结果揭示了临床决策支持系统中的重大安全风险，并为未来加强临床语言模型以抵御此类漏洞的努力铺平了道路。

Title: Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course

• Authors: Cheng-Han Chiang, Wei-Chih Chen, Chun-Yi Kuan, Chienchou Yang, Hung-yi Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05216
• Pdf URL: https://arxiv.org/pdf/2407.05216
• Copy Paste: [[2407.05216]] Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course(https://arxiv.org/abs/2407.05216)
• Keywords: language model, gpt, llm
• Abstract: Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. However, it is unclear whether these LLM-based evaluators can be applied in real-world classrooms to assess student assignments. This empirical report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students. Based on student responses, we find that LLM-based assignment evaluators are generally acceptable to students when students have free access to these LLM-based evaluators. However, students also noted that the LLM sometimes fails to adhere to the evaluation instructions. Additionally, we observe that students can easily manipulate the LLM-based evaluator to output specific strings, allowing them to achieve high scores without meeting the assignment rubric. Based on student feedback and our experience, we provide several recommendations for integrating LLM-based evaluators into future classrooms.
• 摘要：使用大型语言模型 (LLM) 进行自动评估已成为 NLP 研究中的重要评估方法。然而，这些基于 LLM 的评估器是否可以应用于现实课堂来评估学生的作业尚不清楚。这份实证报告分享了我们如何在一门有 1,028 名学生的大学课程中使用 GPT-4 作为自动作业评估器。根据学生的反馈，我们发现当学生可以免费使用这些基于 LLM 的评估器时，基于 LLM 的作业评估器通常是学生可以接受的。然而，学生们也注意到 LLM 有时不遵守评估说明。此外，我们观察到学生可以轻松操纵基于 LLM 的评估器来输出特定的字符串，从而使他们无需满足作业评分标准即可获得高分。根据学生的反馈和我们的经验，我们为将基于 LLM 的评估器集成到未来的课堂中提供了一些建议。

Title: Advancing Prompt Recovery in NLP: A Deep Dive into the Integration of Gemma-2b-it and Phi2 Models

• Authors: Jianlong Chen, Wei Xu, Zhicheng Ding, Jinxin Xu, Hao Yan, Xinyu Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05233
• Pdf URL: https://arxiv.org/pdf/2407.05233
• Copy Paste: [[2407.05233]] Advancing Prompt Recovery in NLP: A Deep Dive into the Integration of Gemma-2b-it and Phi2 Models(https://arxiv.org/abs/2407.05233)
• Keywords: language model, prompt
• Abstract: Prompt recovery, a crucial task in natural language processing, entails the reconstruction of prompts or instructions that language models use to convert input text into a specific output. Although pivotal, the design and effectiveness of prompts represent a challenging and relatively untapped field within NLP research. This paper delves into an exhaustive investigation of prompt recovery methodologies, employing a spectrum of pre-trained language models and strategies. Our study is a comparative analysis aimed at gauging the efficacy of various models on a benchmark dataset, with the goal of pinpointing the most proficient approach for prompt recovery. Through meticulous experimentation and detailed analysis, we elucidate the outstanding performance of the Gemma-2b-it + Phi2 model + Pretrain. This model surpasses its counterparts, showcasing its exceptional capability in accurately reconstructing prompts for text transformation tasks. Our findings offer a significant contribution to the existing knowledge on prompt recovery, shedding light on the intricacies of prompt design and offering insightful perspectives for future innovations in text rewriting and the broader field of natural language processing.
• 摘要：提示恢复是自然语言处理中的一项关键任务，它需要重建语言模型用于将输入文本转换为特定输出的提示或指令。尽管提示设计和有效性至关重要，但它代表了 NLP 研究中一个具有挑战性且相对尚未开发的领域。本文深入研究了提示恢复方法，采用了一系列预先训练的语言模型和策略。我们的研究是一项比较分析，旨在衡量各种模型在基准数据集上的有效性，目标是找出最熟练的提示恢复方法。通过细致的实验和详细的分析，我们阐明了 Gemma-2b-it + Phi2 模型 + Pretrain 的出色性能。该模型超越了同类模型，展示了其在准确重建文本转换任务提示方面的卓越能力。我们的研究结果对现有的提示恢复知识做出了重大贡献，揭示了提示设计的复杂性，并为文本重写和更广泛的自然语言处理领域的未来创新提供了深刻的见解。

Title: CLIMB: A Benchmark of Clinical Bias in Large Language Models

• Authors: Yubo Zhang, Shudi Hou, Mingyu Derek Ma, Wei Wang, Muhao Chen, Jieyu Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05250
• Pdf URL: https://arxiv.org/pdf/2407.05250
• Copy Paste: [[2407.05250]] CLIMB: A Benchmark of Clinical Bias in Large Language Models(https://arxiv.org/abs/2407.05250)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly applied to clinical decision-making. However, their potential to exhibit bias poses significant risks to clinical equity. Currently, there is a lack of benchmarks that systematically evaluate such clinical bias in LLMs. While in downstream tasks, some biases of LLMs can be avoided such as by instructing the model to answer "I'm not sure...", the internal bias hidden within the model still lacks deep studies. We introduce CLIMB (shorthand for A Benchmark of Clinical Bias in Large Language Models), a pioneering comprehensive benchmark to evaluate both intrinsic (within LLMs) and extrinsic (on downstream tasks) bias in LLMs for clinical decision tasks. Notably, for intrinsic bias, we introduce a novel metric, AssocMAD, to assess the disparities of LLMs across multiple demographic groups. Additionally, we leverage counterfactual intervention to evaluate extrinsic bias in a task of clinical diagnosis prediction. Our experiments across popular and medically adapted LLMs, particularly from the Mistral and LLaMA families, unveil prevalent behaviors with both intrinsic and extrinsic bias. This work underscores the critical need to mitigate clinical bias and sets a new standard for future evaluations of LLMs' clinical bias.
• 摘要：大型语言模型 (LLM) 越来越多地应用于临床决策。然而，它们表现出偏见的可能性对临床公平性构成了重大风险。目前，缺乏系统地评估 LLM 中此类临床偏见的基准。虽然在下游任务中，可以避免 LLM 的一些偏见，例如通过指示模型回答“我不确定……”，但隐藏在模型中的内部偏见仍然缺乏深入研究。我们引入了 CLIMB（大型语言模型临床偏见基准的简称），这是一种开创性的综合基准，用于评估 LLM 在临床决策任务中的内在（LLM 内）和外在（下游任务）偏见。值得注意的是，对于内在偏见，我们引入了一个新颖的指标 AssocMAD，以评估多个人口群体中 LLM 的差异。此外，我们利用反事实干预来评估临床诊断预测任务中的外在偏见。我们对热门和医学改编的 LLM（尤其是 Mistral 和 LLaMA 家族）进行了实验，揭示了普遍存在内在和外在偏见的行为。这项研究强调了减轻临床偏见的迫切需要，并为未来评估 LLM 的临床偏见树立了新标准。

Title: Beyond Binary Gender Labels: Revealing Gender Biases in LLMs through Gender-Neutral Name Predictions

• Authors: Zhiwen You, HaeJin Lee, Shubhanshu Mishra, Sullam Jeoung, Apratim Mishra, Jinseok Kim, Jana Diesner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05271
• Pdf URL: https://arxiv.org/pdf/2407.05271
• Copy Paste: [[2407.05271]] Beyond Binary Gender Labels: Revealing Gender Biases in LLMs through Gender-Neutral Name Predictions(https://arxiv.org/abs/2407.05271)
• Keywords: language model, llm
• Abstract: Name-based gender prediction has traditionally categorized individuals as either female or male based on their names, using a binary classification system. That binary approach can be problematic in the cases of gender-neutral names that do not align with any one gender, among other reasons. Relying solely on binary gender categories without recognizing gender-neutral names can reduce the inclusiveness of gender prediction tasks. We introduce an additional gender category, i.e., "neutral", to study and address potential gender biases in Large Language Models (LLMs). We evaluate the performance of several foundational and large language models in predicting gender based on first names only. Additionally, we investigate the impact of adding birth years to enhance the accuracy of gender prediction, accounting for shifting associations between names and genders over time. Our findings indicate that most LLMs identify male and female names with high accuracy (over 80%) but struggle with gender-neutral names (under 40%), and the accuracy of gender prediction is higher for English-based first names than non-English names. The experimental results show that incorporating the birth year does not improve the overall accuracy of gender prediction, especially for names with evolving gender associations. We recommend using caution when applying LLMs for gender identification in downstream tasks, particularly when dealing with non-binary gender labels.
• 摘要：基于姓名的性别预测传统上根据姓名将个人分为女性或男性，使用二元分类系统。这种二元方法在性别中立姓名与任何一种性别都不一致的情况下可能会出现问题，原因之一是这些姓名并不属于任何一种性别。仅依靠二元性别类别而不识别性别中立姓名会降低性别预测任务的包容性。我们引入了一个额外的性别类别，即“中性”，以研究和解决大型语言模型 (LLM) 中的潜在性别偏见。我们评估了几种基础和大型语言模型在仅基于名字预测性别方面的表现。此外，我们研究了添加出生年份对提高性别预测准确性的影响，考虑到姓名和性别之间的关联随时间的变化。我们的研究结果表明，大多数 LLM 可以高精度地（超过 80%）识别男性和女性姓名，但很难识别性别中立姓名（低于 40%），并且基于英语的名字的性别预测准确性高于非英语名字。实验结果表明，加入出生年份并不能提高性别预测的整体准确率，尤其是对于性别关联不断演变的姓名。我们建议在下游任务中应用 LLM 进行性别识别时要谨慎，尤其是在处理非二元性别标签时。

Title: Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition

• Authors: Zirun Guo, Tao Jin, Zhou Zhao
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.05374
• Pdf URL: https://arxiv.org/pdf/2407.05374
• Copy Paste: [[2407.05374]] Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition(https://arxiv.org/abs/2407.05374)
• Keywords: prompt
• Abstract: The development of multimodal models has significantly advanced multimodal sentiment analysis and emotion recognition. However, in real-world applications, the presence of various missing modality cases often leads to a degradation in the model's performance. In this work, we propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. These prompts enable the generation of missing modality features and facilitate the learning of intra- and inter-modality information. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters. Our proposed method outperforms other methods significantly across all evaluation metrics. Extensive experiments and ablation studies are conducted to demonstrate the effectiveness and robustness of our method, showcasing its ability to effectively handle missing modalities.
• 摘要：多模态模型的发展显著推动了多模态情绪分析和情绪识别。然而，在实际应用中，各种模态缺失情况的存在往往会导致模型性能下降。在这项工作中，我们提出了一种新颖的多模态 Transformer 框架，使用提示学习来解决模态缺失的问题。我们的方法引入了三种类型的提示：生成提示、缺失信号提示和缺失类型提示。这些提示能够生成缺失的模态特征，并有助于学习模态内和模态间信息。通过提示学习，我们大幅减少了可训练参数的数量。我们提出的方法在所有评估指标上都明显优于其他方法。我们进行了大量的实验和消融研究，以证明我们方法的有效性和鲁棒性，展示了它有效处理缺失模态的能力。

Title: IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning

• Authors: Abhinav Joshi, Shounak Paul, Akshat Sharma, Pawan Goyal, Saptarshi Ghosh, Ashutosh Modi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.05399
• Pdf URL: https://arxiv.org/pdf/2407.05399
• Copy Paste: [[2407.05399]] IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning(https://arxiv.org/abs/2407.05399)
• Keywords: llm
• Abstract: Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to streamline the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challenging. This paper addresses this challenge by proposing IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning. IL-TUR contains monolingual (English, Hindi) and multi-lingual (9 Indian languages) domain-specific tasks that address different aspects of the legal system from the point of view of understanding and reasoning over Indian legal documents. We present baseline models (including LLM-based) for each task, outlining the gap between models and the ground truth. To foster further research in the legal domain, we create a leaderboard (available at: this https URL) where the research community can upload and compare legal text understanding systems.
• 摘要：全球各地的法律体系都面临着案件和文件数量呈指数级增长的局面。迫切需要开发 NLP 和 ML 技术来自动处理和理解法律文件，以简化法律体系。然而，评估和比较专门为法律领域设计的各种 NLP 模型是一项挑战。本文通过提出 IL-TUR：印度法律文本理解和推理基准来应对这一挑战。IL-TUR 包含单语（英语、印地语）和多语（9 种印度语言）领域特定任务，从理解和推理印度法律文件的角度解决法律体系的不同方面。我们为每个任务提供了基线模型（包括基于 LLM 的模型），概述了模型与基本事实之间的差距。为了促进法律领域的进一步研究，我们创建了一个排行榜（可从此 https URL 获得），研究界可以在其中上传和比较法律文本理解系统。

Title: LTLBench: Towards Benchmarks for Evaluating Temporal Logic Reasoning in Large Language Models

• Authors: Weizhi Tang, Vaishak Belle
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05434
• Pdf URL: https://arxiv.org/pdf/2407.05434
• Copy Paste: [[2407.05434]] LTLBench: Towards Benchmarks for Evaluating Temporal Logic Reasoning in Large Language Models(https://arxiv.org/abs/2407.05434)
• Keywords: language model, llm
• Abstract: Temporal reasoning (TR) is a critical component of artificial intelligence, encompassing understanding and processing temporal information and relationships between events. To discover and study the TR ability in Large Language Models (LLMs), various datasets have been constructed in different ways for evaluating various aspects of TR ability. Our work proposes a novel approach to design and develop a pipeline for constructing datasets to evaluate the TR ability of LLMs by leveraging random directed graph generation, LTL formula, and the NuSMV model checker. Based on the pipeline, we have also constructed a dataset as a benchmark, namely LTLBench, consisting of 2,000 TR challenges and evaluated six LLMs with it. Furthermore, we have conducted additional experiments to discover the impact of increasing the number of events and formula operators on the complexity of TR problems and the performance of LLMs. We have demonstrated that although LLMs exhibit some promise in handling TR challenges, they still struggle with complex TR. We expect this work can offer insights into TR ability in LLMs while also providing a valuable tool for future TR evaluations.
• 摘要：时间推理 (TR) 是人工智能的重要组成部分，包括理解和处理时间信息以及事件之间的关系。为了发现和研究大型语言模型 (LLM) 中的 TR 能力，人们以不同的方式构建了各种数据集，以评估 TR 能力的各个方面。我们的工作提出了一种新颖的方法来设计和开发一个流程，通过利用随机有向图生成、LTL 公式和 NuSMV 模型检查器来构建数据集以评估 LLM 的 TR 能力。基于该流程，我们还构建了一个数据集作为基准，即 LTLBench，其中包含 2,000 个 TR 挑战，并用它来评估六个 LLM。此外，我们还进行了额外的实验，以发现增加事件和公式运算符的数量对 TR 问题的复杂性和 LLM 性能的影响。我们已经证明，尽管 LLM 在处理 TR 挑战方面表现出一些希望，但它们在处理复杂的 TR 方面仍然举步维艰。我们希望这项工作能够提供对 LLM 中 TR 能力的见解，同时也为未来的 TR 评估提供有价值的工具。

Title: Training Task Experts through Retrieval Based Distillation

• Authors: Jiaxin Ge, Xueying Jia, Vijay Viswanathan, Hongyin Luo, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05463
• Pdf URL: https://arxiv.org/pdf/2407.05463
• Copy Paste: [[2407.05463]] Training Task Experts through Retrieval Based Distillation(https://arxiv.org/abs/2407.05463)
• Keywords: language model, llm, chain-of-thought
• Abstract: One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by creating such data from large language models (LLMs) and then distilling such knowledge into smaller models. However, these methods are limited by the quality of the LLMs output, and tend to generate repetitive or incorrect data. In this work, we present Retrieval Based Distillation (ReBase), a method that first retrieves data from rich online sources and then transforms them into domain-specific data. This method greatly enhances data diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills the reasoning capacity of LLMs. We test our method on 4 benchmarks and results show that our method significantly improves performance by up to 7.8% on SQuAD, 1.37% on MNLI, and 1.94% on BigBench-Hard.
• 摘要：为专门任务创建可部署模型的最可靠方法之一是获取足够数量的高质量特定于任务的数据。然而，对于专门任务，通常不存在这样的数据集。现有的方法通过从大型语言模型 (LLM) 创建此类数据，然后将这些知识提炼成较小的模型来解决这个问题。然而，这些方法受到 LLM 输出质量的限制，并且往往会生成重复或不正确的数据。在这项工作中，我们提出了基于检索的提炼 (ReBase)，这种方法首先从丰富的在线来源检索数据，然后将其转换为特定于领域的数据。这种方法大大增强了数据多样性。此外，ReBase 生成了思想链推理并提炼了 LLM 的推理能力。我们在 4 个基准上测试了我们的方法，结果表明，我们的方法在 SQuAD 上显著提高了 7.8%，在 MNLI 上提高了 1.37%，在 BigBench-Hard 上提高了 1.94%。

Title: Biomedical Nested NER with Large Language Model and UMLS Heuristics

• Authors: Wenxin Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05480
• Pdf URL: https://arxiv.org/pdf/2407.05480
• Copy Paste: [[2407.05480]] Biomedical Nested NER with Large Language Model and UMLS Heuristics(https://arxiv.org/abs/2407.05480)
• Keywords: language model
• Abstract: In this paper, we present our system for the BioNNE English track, which aims to extract 8 types of biomedical nested named entities from biomedical text. We use a large language model (Mixtral 8x7B instruct) and ScispaCy NER model to identify entities in an article and build custom heuristics based on unified medical language system (UMLS) semantic types to categorize the entities. We discuss the results and limitations of our system and propose future improvements. Our system achieved an F1 score of 0.39 on the BioNNE validation set and 0.348 on the test set.
• 摘要：在本文中，我们介绍了针对 BioNNE 英语轨道的系统，该系统旨在从生物医学文本中提取 8 种类型的生物医学嵌套命名实体。我们使用大型语言模型 (Mixtral 8x7B instruct) 和 ScispaCy NER 模型来识别文章中的实体，并基于统一医学语言系统 (UMLS) 语义类型构建自定义启发式方法来对实体进行分类。我们讨论了我们系统的结果和局限性，并提出了未来的改进建议。我们的系统在 BioNNE 验证集上的 F1 得分为 0.39，在测试集上的 F1 得分为 0.348。

Title: Just read twice: closing the recall gap for recurrent language models

• Authors: Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, Christopher Ré
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.05483
• Pdf URL: https://arxiv.org/pdf/2407.05483
• Copy Paste: [[2407.05483]] Just read twice: closing the recall gap for recurrent language models(https://arxiv.org/abs/2407.05483)
• Keywords: language model, long context, prompt
• Abstract: Recurrent large language models that compete with Transformers in language modeling perplexity are emerging at a rapid rate (e.g., Mamba, RWKV). Excitingly, these architectures use a constant amount of memory during inference. However, due to the limited memory, recurrent LMs cannot recall and use all the information in long contexts leading to brittle in-context learning (ICL) quality. A key challenge for efficient LMs is selecting what information to store versus discard. In this work, we observe the order in which information is shown to the LM impacts the selection difficulty. To formalize this, we show that the hardness of information recall reduces to the hardness of a problem called set disjointness (SD), a quintessential problem in communication complexity that requires a streaming algorithm (e.g., recurrent model) to decide whether inputted sets are disjoint. We empirically and theoretically show that the recurrent memory required to solve SD changes with set order, i.e., whether the smaller set appears first in-context. Our analysis suggests, to mitigate the reliance on data order, we can put information in the right order in-context or process prompts non-causally. Towards that end, we propose: (1) JRT-Prompt, where context gets repeated multiple times in the prompt, effectively showing the model all data orders. This gives $11.0 \pm 1.3$ points of improvement, averaged across $16$ recurrent LMs and the $6$ ICL tasks, with $11.9\times$ higher throughput than FlashAttention-2 for generation prefill (length $32$k, batch size $16$, NVidia H100). We then propose (2) JRT-RNN, which uses non-causal prefix-linear-attention to process prompts and provides $99\%$ of Transformer quality at $360$M params., $30$B tokens and $96\%$ at $1.3$B params., $50$B tokens on average across the tasks, with $19.2\times$ higher throughput for prefill than FA2.
• 摘要：在语言建模困惑度方面可与 Transformers 相媲美的循环大型语言模型正在迅速涌现（例如 Mamba、RWKV）。令人兴奋的是，这些架构在推理过程中使用恒定量的内存。然而，由于内存有限，循环语言模型无法回忆和使用长上下文中的所有信息，导致上下文学习 (ICL) 质量不佳。高效语言模型面临的一个关键挑战是选择存储和丢弃哪些信息。在这项工作中，我们观察到向语言模型显示信息的顺序会影响选择难度。为了形式化这一点，我们表明信息回忆的难度降低到称为集合不相交 (SD) 的问题的难度，这是通信复杂性中的一个典型问题，需要流式算法（例如循环模型）来决定输入的集合是否不相交。我们通过经验和理论表明，解决 SD 所需的循环记忆会随着集合顺序而变化，即较小的集合是否首先出现在上下文中。我们的分析表明，为了减轻对数据顺序的依赖，我们可以在上下文中将信息按正确的顺序排列，或者以非因果的方式处理提示。为此，我们提出：(1) JRT-Prompt，其中上下文在提示中重复多次，有效地向模型显示所有数据顺序。这为 16 个循环 LM 和 6 个 ICL 任务带来了 $11.0 \pm 1.3$ 点的改进，在生成预填充方面，吞吐量比 FlashAttention-2 高 $11.9\times$（长度 $32$k，批大小 $16$，NVidia H100）。然后我们提出 (2) JRT-RNN，它使用非因果前缀线性注意力来处理提示，并在 3.6 亿个参数、300 亿个标记下提供 99% 的 Transformer 质量，在 13 亿个参数、500 亿个标记下提供 96% 的 Transformer 质量，预填充的吞吐量比 FA2 高 19.2 倍。

Title: How Effective are State Space Models for Machine Translation?

• Authors: Hugo Pitorro, Pavlo Vasylenko, Marcos Treviso, André F. T. Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05489
• Pdf URL: https://arxiv.org/pdf/2407.05489
• Copy Paste: [[2407.05489]] How Effective are State Space Models for Machine Translation?(https://arxiv.org/abs/2407.05489)
• Keywords: long context
• Abstract: Transformers are the current architecture of choice for NLP, but their attention layers do not scale well to long contexts. Recent works propose to replace attention with linear recurrent layers -- this is the case for state space models, which enjoy efficient training and inference. However, it remains unclear whether these models are competitive with transformers in machine translation (MT). In this paper, we provide a rigorous and comprehensive experimental comparison between transformers and linear recurrent models for MT. Concretely, we experiment with RetNet, Mamba, and hybrid versions of Mamba which incorporate attention mechanisms. Our findings demonstrate that Mamba is highly competitive with transformers on sentence and paragraph-level datasets, where in the latter both models benefit from shifting the training distribution towards longer sequences. Further analysis show that integrating attention into Mamba improves translation quality, robustness to sequence length extrapolation, and the ability to recall named entities.
• 摘要：Transformer 是当前 NLP 的首选架构，但它们的注意力层不能很好地扩展到长上下文。最近的研究建议用线性循环层取代注意力——这是状态空间模型的情况，它具有高效的训练和推理能力。然而，目前尚不清楚这些模型在机器翻译 (MT) 方面是否能与 Transformer 相媲美。在本文中，我们对 MT 的 Transformer 和线性循环模型进行了严格而全面的实验比较。具体来说，我们尝试了 RetNet、Mamba 和结合了注意力机制的 Mamba 的混合版本。我们的研究结果表明，Mamba 在句子和段落级数据集上与 Transformer 具有很强的竞争力，在后者中，两种模型都受益于将训练分布转向更长的序列。进一步的分析表明，将注意力集成到 Mamba 中可以提高翻译质量、对序列长度推断的鲁棒性以及回忆命名实体的能力。

Title: Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models

• Authors: Nikhil Sharma, Kenton Murray, Ziang Xiao
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.05502
• Pdf URL: https://arxiv.org/pdf/2407.05502
• Copy Paste: [[2407.05502]] Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models(https://arxiv.org/abs/2407.05502)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: With Retrieval Augmented Generation (RAG), Large Language Models (LLMs) are playing a pivotal role in information search and are being adopted globally. Although the multilingual capability of LLMs offers new opportunities to bridge the language barrier, do these capabilities translate into real-life scenarios where linguistic divide and knowledge conflicts between multilingual sources are known occurrences? In this paper, we studied LLM's linguistic preference in a RAG-based information search setting. We found that LLMs displayed systemic bias towards information in the same language as the query language in both information retrieval and answer generation. Furthermore, in scenarios where there is little information in the language of the query, LLMs prefer documents in high-resource languages, reinforcing the dominant views. Such bias exists for both factual and opinion-based queries. Our results highlight the linguistic divide within multilingual LLMs in information search systems. The seemingly beneficial multilingual capability of LLMs may backfire on information parity by reinforcing language-specific information cocoons or filter bubbles further marginalizing low-resource views.
• 摘要：借助检索增强生成 (RAG)，大型语言模型 (LLM) 在信息搜索中发挥着关键作用，并正在全球范围内被采用。虽然 LLM 的多语言能力为弥合语言障碍提供了新的机会，但这些能力是否会转化为现实生活中的场景，即多语言来源之间的语言鸿沟和知识冲突是众所周知的现象？在本文中，我们研究了基于 RAG 的信息搜索环境中的 LLM 语言偏好。我们发现，无论是在信息检索还是答案生成中，LLM 都对与查询语言相同的语言的信息表现出系统性偏见。此外，在查询语言中信息很少的情况下，LLM 更喜欢资源丰富的语言的文档，从而强化了主流观点。这种偏见既存在于事实查询中，也存在于基于观点的查询中。我们的结果强调了信息搜索系统中多语言 LLM 中的语言鸿沟。法学硕士 (LLM) 的多语言能力看似有益，但可能会对信息均等产生不利影响，因为它会强化特定语言的信息茧房或过滤泡沫，进一步边缘化低资源观点。

Title: LLMBox: A Comprehensive Library for Large Language Models

• Authors: Tianyi Tang, Yiwen Hu, Bingqian Li, Wenyang Luo, Zijing Qin, Haoxiang Sun, Jiapeng Wang, Shiyi Xu, Xiaoxue Cheng, Geyang Guo, Han Peng, Bowen Zheng, Yiru Tang, Yingqian Min, Yushuo Chen, Jie Chen, Yuanqian Zhao, Luran Ding, Yuhao Wang, Zican Dong, Chunxuan Xia, Junyi Li, Kun Zhou, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05563
• Pdf URL: https://arxiv.org/pdf/2407.05563
• Copy Paste: [[2407.05563]] LLMBox: A Comprehensive Library for Large Language Models(https://arxiv.org/abs/2407.05563)
• Keywords: language model, llm
• Abstract: To facilitate the research on large language models (LLMs), this paper presents a comprehensive and unified library, LLMBox, to ease the development, use, and evaluation of LLMs. This library is featured with three main merits: (1) a unified data interface that supports the flexible implementation of various training strategies, (2) a comprehensive evaluation that covers extensive tasks, datasets, and models, and (3) more practical consideration, especially on user-friendliness and efficiency. With our library, users can easily reproduce existing methods, train new models, and conduct comprehensive performance comparisons. To rigorously test LLMBox, we conduct extensive experiments in a diverse coverage of evaluation settings, and experimental results demonstrate the effectiveness and efficiency of our library in supporting various implementations related to LLMs. The detailed introduction and usage guidance can be found at this https URL.
• 摘要：为了促进大型语言模型 (LLM) 的研究，本文提出了一个全面统一的库 LLMBox，以简化 LLM 的开发、使用和评估。该库具有三个主要优点：（1）统一的数据接口，支持灵活实施各种训练策略；（2）全面的评估，涵盖广泛的任务、数据集和模型；（3）更实际的考虑，特别是在用户友好性和效率方面。使用我们的库，用户可以轻松复现现有方法、训练新模型并进行全面的性能比较。为了严格测试 LLMBox，我们在各种评估设置中进行了广泛的实验，实验结果证明了我们的库在支持与 LLM 相关的各种实现方面的有效性和效率。详细的介绍和使用指南可以在这个 https URL 中找到。

Title: Generative Debunking of Climate Misinformation

• Authors: Francisco Zanartu, Yulia Otmakhova, John Cook, Lea Frermann
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.05599
• Pdf URL: https://arxiv.org/pdf/2407.05599
• Copy Paste: [[2407.05599]] Generative Debunking of Climate Misinformation(https://arxiv.org/abs/2407.05599)
• Keywords: language model, gpt, llm, prompt
• Abstract: Misinformation about climate change causes numerous negative impacts, necessitating corrective responses. Psychological research has offered various strategies for reducing the influence of climate misinformation, such as the fact-myth-fallacy-fact-structure. However, practically implementing corrective interventions at scale represents a challenge. Automatic detection and correction of misinformation offers a solution to the misinformation problem. This study documents the development of large language models that accept as input a climate myth and produce a debunking that adheres to the fact-myth-fallacy-fact (``truth sandwich'') structure, by incorporating contrarian claim classification and fallacy detection into an LLM prompting framework. We combine open (Mixtral, Palm2) and proprietary (GPT-4) LLMs with prompting strategies of varying complexity. Experiments reveal promising performance of GPT-4 and Mixtral if combined with structured prompts. We identify specific challenges of debunking generation and human evaluation, and map out avenues for future work. We release a dataset of high-quality truth-sandwich debunkings, source code and a demo of the debunking system.
• 摘要：有关气候变化的错误信息会造成许多负面影响，因此需要采取纠正措施。心理学研究提供了各种减少气候错误信息影响的策略，例如事实-神话-谬误-事实结构。然而，实际实施大规模纠正干预措施是一项挑战。自动检测和纠正错误信息为错误信息问题提供了解决方案。这项研究记录了大型语言模型的发展，这些模型接受气候神话作为输入，并产生遵循事实-神话-谬误-事实（“真相三明治”）结构的揭穿，方法是将反向主张分类和谬误检测纳入 LLM 提示框架。我们将开放（Mixtral、Palm2）和专有（GPT-4）LLM 与不同复杂程度的提示策略相结合。实验表明，如果与结构化提示相结合，GPT-4 和 Mixtral 的性能将非常出色。我们确定了揭穿生成和人工评估的具体挑战，并规划了未来工作的途径。我们发布了高质量的真相三明治揭穿数据集、源代码和揭穿系统的演示。

Title: Open-world Multi-label Text Classification with Extremely Weak Supervision

• Authors: Xintong Li, Jinya Jiang, Ria Dharmani, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05609
• Pdf URL: https://arxiv.org/pdf/2407.05609
• Copy Paste: [[2407.05609]] Open-world Multi-label Text Classification with Extremely Weak Supervision(https://arxiv.org/abs/2407.05609)
• Keywords: language model, llm, prompt
• Abstract: We study open-world multi-label text classification under extremely weak supervision (XWS), where the user only provides a brief description for classification objectives without any labels or ground-truth label space. Similar single-label XWS settings have been explored recently, however, these methods cannot be easily adapted for multi-label. We observe that (1) most documents have a dominant class covering the majority of content and (2) long-tail labels would appear in some documents as a dominant class. Therefore, we first utilize the user description to prompt a large language model (LLM) for dominant keyphrases of a subset of raw documents, and then construct a (initial) label space via clustering. We further apply a zero-shot multi-label classifier to locate the documents with small top predicted scores, so we can revisit their dominant keyphrases for more long-tail labels. We iterate this process to discover a comprehensive label space and construct a multi-label classifier as a novel method, X-MLClass. X-MLClass exhibits a remarkable increase in ground-truth label space coverage on various datasets, for example, a 40% improvement on the AAPD dataset over topic modeling and keyword extraction methods. Moreover, X-MLClass achieves the best end-to-end multi-label classification accuracy.
• 摘要：我们研究极弱监督 (XWS) 下的开放世界多标签文本分类，其中用户仅提供分类目标的简短描述，没有任何标签或真实标签空间。最近已经探索了类似的单标签 XWS 设置，但是，这些方法不能轻易适应多标签。我们观察到 (1) 大多数文档都有一个覆盖大部分内容的主导类别，并且 (2) 长尾标签会在某些文档中作为主导类别出现。因此，我们首先利用用户描述来提示大型语言模型 (LLM) 以获取原始文档子集的主导关键词，然后通过聚类构建 (初始) 标签空间。我们进一步应用零样本多标签分类器来定位具有较小顶部预测分数的文档，因此我们可以重新访问它们的主导关键词以获取更多长尾标签。我们迭代此过程以发现全面的标签空间并构建多标签分类器作为一种新方法 X-MLClass。 X-MLClass 在各种数据集上的真实标签空间覆盖率均有显著提升，例如，与主题建模和关键字提取方法相比，在 AAPD 数据集上提高了 40%。此外，X-MLClass 实现了最佳的端到端多标签分类准确率。

Title: Retrieved In-Context Principles from Previous Mistakes

• Authors: Hao Sun, Yong Jiang, Bo Wang, Yingyan Hou, Yan Zhang, Pengjun Xie, Fei Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05682
• Pdf URL: https://arxiv.org/pdf/2407.05682
• Copy Paste: [[2407.05682]] Retrieved In-Context Principles from Previous Mistakes(https://arxiv.org/abs/2407.05682)
• Keywords: language model, llm, prompt
• Abstract: In-context learning (ICL) has been instrumental in adapting Large Language Models (LLMs) to downstream tasks using correct input-output examples. Recent advances have attempted to improve model performance through principles derived from mistakes, yet these approaches suffer from lack of customization and inadequate error coverage. To address these limitations, we propose Retrieved In-Context Principles (RICP), a novel teacher-student framework. In RICP, the teacher model analyzes mistakes from the student model to generate reasons and insights for preventing similar mistakes. These mistakes are clustered based on their underlying reasons for developing task-level principles, enhancing the error coverage of principles. During inference, the most relevant mistakes for each question are retrieved to create question-level principles, improving the customization of the provided guidance. RICP is orthogonal to existing prompting methods and does not require intervention from the teacher model during inference. Experimental results across seven reasoning benchmarks reveal that RICP effectively enhances performance when applied to various prompting strategies.
• 摘要：上下文学习 (ICL) 在使用正确的输入输出示例将大型语言模型 (LLM) 适应下游任务方面发挥了重要作用。最近的进展试图通过从错误中得出的原则来提高模型性能，但这些方法存在缺乏定制和错误覆盖不足的问题。为了解决这些限制，我们提出了一种新颖的师生框架——检索上下文原则 (RICP)。在 RICP 中，教师模型会分析学生模型中的错误，以产生原因和见解，以防止出现类似的错误。这些错误根据其根本原因进行聚类，以开发任务级原则，从而增强原则的错误覆盖率。在推理过程中，会检索每个问题最相关的错误以创建问题级原则，从而改善所提供指导的定制。RICP 与现有的提示方法正交，并且在推理过程中不需要教师模型的干预。七个推理基准的实验结果表明，当应用于各种提示策略时，RICP 可以有效提高性能。

Title: Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations

• Authors: Bowen Shen, Zheng Lin, Daren Zha, Wei Liu, Jian Luan, Bin Wang, Weiping Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05690
• Pdf URL: https://arxiv.org/pdf/2407.05690
• Copy Paste: [[2407.05690]] Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations(https://arxiv.org/abs/2407.05690)
• Keywords: language model, llm
• Abstract: Structured pruning fundamentally reduces computational and memory overheads of large language models (LLMs) and offers a feasible solution for end-side LLM deployment. Structurally pruned models remain dense and high-precision, highly compatible with further tuning and compression. However, as the coarse-grained structured pruning poses large damage to the highly interconnected model, achieving a high compression ratio for scaled-up LLMs remains a challenge. In this paper, we introduce a task-agnostic structured pruning approach coupled with a compact Transformer architecture design. The proposed approach, named TransAct, reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules, while preserving the inter-module activations that are sensitive to perturbations. Hence, the LLM is pruned into an intra-module low-rank architecture, significantly reducing weights, KV Cache and attention computation. TransAct is implemented on the LLaMA model and evaluated on downstream benchmarks. Results verify the optimality of our approach at high compression with respect to both efficiency and performance. Further, ablation studies reveal the strength of activation-guided iterative pruning and provide experimental analysis on the redundancy of MHA and MLP modules.
• 摘要：结构化剪枝从根本上减少了大型语言模型 (LLM) 的计算和内存开销，为端侧 LLM 部署提供了可行的解决方案。结构化剪枝后的模型保持了密集和高精度，与进一步的调整和压缩高度兼容。然而，由于粗粒度的结构化剪枝对高度互联的模型造成了很大的损害，实现扩大规模的 LLM 的高压缩比仍然是一个挑战。在本文中，我们介绍了一种与任务无关的结构化剪枝方法，并结合了紧凑的 Transformer 架构设计。所提出的方法名为 TransAct，它减少了多头注意 (MHA) 和多层感知器 (MLP) 模块内的过渡激活，同时保留了对扰动敏感的模块间激活。因此，LLM 被剪枝为模块内低秩架构，显著减少了权重、KV 缓存和注意计算。TransAct 在 LLaMA 模型上实现并在下游基准上进行了评估。结果验证了我们的方法在高压缩下在效率和性能方面的最优性。此外，消融研究揭示了激活引导迭代剪枝的强度，并对 MHA 和 MLP 模块的冗余性进行了实验分析。

Title: InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with Inverse-Instruct

• Authors: Yutong Wu, Di Huang, Wenxuan Shi, Wei Wang, Lingzhe Gao, Shihao Liu, Ziyuan Nan, Kaizhao Yuan, Rui Zhang, Xishan Zhang, Zidong Du, Qi Guo, Yewen Pu, Dawei Yin, Xing Hu, Yunji Chen
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.05700
• Pdf URL: https://arxiv.org/pdf/2407.05700
• Copy Paste: [[2407.05700]] InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with Inverse-Instruct(https://arxiv.org/abs/2407.05700)
• Keywords: language model, gpt, llm
• Abstract: Recent advancements in open-source code large language models (LLMs) have demonstrated remarkable coding abilities by fine-tuning on the data generated from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction tuning. This paper explores how to further improve an instruction-tuned code LLM by generating data from itself rather than querying closed-source LLMs. Our key observation is the misalignment between the translation of formal and informal languages: translating formal language (i.e., code) to informal language (i.e., natural language) is more straightforward than the reverse. Based on this observation, we propose INVERSE-INSTRUCT, which summarizes instructions from code snippets instead of the reverse. Specifically, given an instruction tuning corpus for code and the resulting instruction-tuned code LLM, we ask the code LLM to generate additional high-quality instructions for the original corpus through code summarization and self-evaluation. Then, we fine-tune the base LLM on the combination of the original corpus and the self-generated one, which yields a stronger instruction-tuned LLM. We present a series of code LLMs named InverseCoder, which surpasses the performance of the original code LLMs on a wide range of benchmarks, including Python text-to-code generation, multilingual coding, and data-science code generation.
• 摘要：开源代码大型语言模型 (LLM) 的最新进展通过对强大的闭源 LLM（例如 GPT-3.5 和 GPT-4）生成的数据进行微调以进行指令调整，展示了卓越的编码能力。本文探讨了如何通过从自身生成数据而不是查询闭源 LLM 来进一步改进指令调整的代码 LLM。我们的主要观察是形式语言和非正式语言的翻译之间的不一致：将形式语言（即代码）翻译成非正式语言（即自然语言）比反过来更直接。基于这一观察，我们提出了 INVERSE-INSTRUCT，它从代码片段中总结指令，而不是反过来。具体来说，给定一个代码的指令调整语料库和由此产生的指令调整的代码 LLM，我们要求代码 LLM 通过代码总结和自我评估为原始语料库生成额外的高质量指令。然后，我们在原始语料库和自生成语料库的组合上对基础 LLM 进行微调，从而产生更强大的指令调整 LLM。我们提出了一系列名为 InverseCoder 的代码 LLM，它在包括 Python 文本到代码生成、多语言编码和数据科学代码生成在内的广泛基准测试中都超越了原始代码 LLM 的性能。

Title: PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation

• Authors: Jinpeng Hu, Tengteng Dong, Hui Ma, Peng Zou, Xiao Sun, Meng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05721
• Pdf URL: https://arxiv.org/pdf/2407.05721
• Copy Paste: [[2407.05721]] PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation(https://arxiv.org/abs/2407.05721)
• Keywords: language model, llm
• Abstract: Mental health has attracted substantial attention in recent years and LLM can be an effective technology for alleviating this problem owing to its capability in text understanding and dialogue. However, existing research in this domain often suffers from limitations, such as training on datasets lacking crucial prior knowledge and evidence, and the absence of comprehensive evaluation methods. In this paper, we propose a specialized psychological large language model (LLM), named PsycoLLM, trained on a proposed high-quality psychological dataset, including single-turn QA, multi-turn dialogues enriched with prior knowledge and knowledge-based QA. Additionally, to compare the performance of PsycoLLM with other LLMs, we develop a comprehensive psychological benchmark based on authoritative psychological counseling examinations in China, which includes assessments of professional ethics, theoretical proficiency, and case analysis. The experimental results on the benchmark illustrates the effectiveness of PsycoLLM, which demonstrates superior performance compared to other LLMs.
• 摘要：近年来，心理健康问题引起了广泛关注，LLM 凭借其在文本理解和对话方面的能力，可以成为缓解这一问题的有效技术。然而，该领域的现有研究往往存在局限性，例如在缺乏关键先验知识和证据的数据集上进行训练，以及缺乏全面的评估方法。在本文中，我们提出了一个专门的心理大型语言模型 (LLM)，名为 PsycoLLM，该模型在一个拟议的高质量心理数据集上进行训练，包括单轮问答、富含先验知识的多轮对话和基于知识的问答。此外，为了将 PsycoLLM 与其他 LLM 的性能进行比较，我们根据中国权威的心理咨询考试开发了一个全面的心理基准，其中包括对职业道德、理论能力和案例分析的评估。基准上的实验结果证明了 PsycoLLM 的有效性，与其他 LLM 相比，它表现出了卓越的性能。

Title: Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition

• Authors: Seungju Kim, Meounggun Jo
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition(https://arxiv.org/abs/)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have shown promise in Automated Essay Scoring (AES), but their zero-shot and few-shot performance often falls short compared to state-of-the-art models and human raters. However, fine-tuning LLMs for each specific task is impractical due to the variety of essay prompts and rubrics used in real-world educational contexts. This study proposes a novel approach combining LLMs and Comparative Judgment (CJ) for AES, using zero-shot prompting to choose between two essays. We demonstrate that a CJ method surpasses traditional rubric-based scoring in essay scoring using LLMs.
• 摘要：大型语言模型 (LLM) 在自动论文评分 (AES) 中表现出色，但与最先进的模型和人类评分者相比，它们的零样本和少量样本表现往往不尽如人意。然而，由于现实世界的教育环境中使用的论文提示和评分标准多种多样，因此针对每项特定任务微调 LLM 是不切实际的。本研究提出了一种将 LLM 和比较判断 (CJ) 相结合的 AES 新方法，使用零样本提示在两篇论文之间进行选择。我们证明，在使用 LLM 进行论文评分时，CJ 方法优于传统的基于评分标准的评分。

Title: Empirical Study of Symmetrical Reasoning in Conversational Chatbots

• Authors: Daniela N. Rim, Heeyoul Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05734
• Pdf URL: https://arxiv.org/pdf/2407.05734
• Copy Paste: [[2407.05734]] Empirical Study of Symmetrical Reasoning in Conversational Chatbots(https://arxiv.org/abs/2407.05734)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: This work explores the capability of conversational chatbots powered by large language models (LLMs), to understand and characterize predicate symmetry, a cognitive linguistic function traditionally believed to be an inherent human trait. Leveraging in-context learning (ICL), a paradigm shift enabling chatbots to learn new tasks from prompts without re-training, we assess the symmetrical reasoning of five chatbots: ChatGPT 4, Huggingface chat AI, Microsoft's Copilot AI, LLaMA through Perplexity, and Gemini Advanced. Using the Symmetry Inference Sentence (SIS) dataset by Tanchip et al. (2020), we compare chatbot responses against human evaluations to gauge their understanding of predicate symmetry. Experiment results reveal varied performance among chatbots, with some approaching human-like reasoning capabilities. Gemini, for example, reaches a correlation of 0.85 with human scores, while providing a sounding justification for each symmetry evaluation. This study underscores the potential and limitations of LLMs in mirroring complex cognitive processes as symmetrical reasoning.
• 摘要：这项研究探索了由大型语言模型 (LLM) 驱动的对话聊天机器人理解和描述谓词对称性的能力，谓词对称性是一种认知语言功能，传统上被认为是人类固有的特征。利用情境学习 (ICL)，一种范式转变，使聊天机器人无需重新训练即可从提示中学习新任务，我们评估了五个聊天机器人的对称推理能力：ChatGPT 4、Huggingface 聊天 AI、微软的 Copilot AI、Perplexity 的 LLaMA 和 Gemini Advanced。使用 Tanchip 等人 (2020) 的对称推理句子 (SIS) 数据集，我们将聊天机器人的响应与人类评估进行比较，以衡量它们对谓词对称性的理解。实验结果显示，聊天机器人之间的性能各不相同，有些接近人类的推理能力。例如，Gemini 与人类分数的相关性达到 0.85，同时为每个对称评估提供了合理的理由。这项研究强调了法学硕士在将复杂的认知过程反映为对称推理方面的潜力和局限性。

Title: Do Multilingual Large Language Models Mitigate Stereotype Bias?

• Authors: Shangrui Nie, Michael Fromm, Charles Welch, Rebekka Görge, Akbar Karimi, Joan Plepi, Nazia Afsan Mowmita, Nicolas Flores-Herr, Mehdi Ali, Lucie Flek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05740
• Pdf URL: https://arxiv.org/pdf/2407.05740
• Copy Paste: [[2407.05740]] Do Multilingual Large Language Models Mitigate Stereotype Bias?(https://arxiv.org/abs/2407.05740)
• Keywords: language model, llm
• Abstract: While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by systematically training six LLMs of identical size (2.6B parameters) and architecture: five monolingual models (English, German, French, Italian, and Spanish) and one multilingual model trained on an equal distribution of data across these languages, all using publicly available data. To ensure robust evaluation, standard bias benchmarks were automatically translated into the five target languages and verified for both translation quality and bias preservation by human annotators. Our results consistently demonstrate that multilingual training effectively mitigates bias. Moreover, we observe that multilingual models achieve not only lower bias but also superior prediction accuracy when compared to monolingual models with the same amount of training data, model architecture, and size.
• 摘要：虽然初步结果表明，与单语 LLM 相比，多语言 LLM 表现出的偏差较小，但对多语言训练对偏差缓解的影响尚缺乏全面的了解。本研究通过系统地训练六个大小（26 亿个参数）和架构相同的 LLM 来解决这一空白：五个单语模型（英语、德语、法语、意大利语和西班牙语）和一个在这些语言的均匀数据分布上训练的多语言模型，所有模型均使用公开数据。为确保可靠的评估，标准偏差基准被自动翻译成五种目标语言，并由人工注释者验证翻译质量和偏差保留。我们的结果一致表明，多语言训练可有效缓解偏差。此外，我们观察到，与具有相同数量训练数据、模型架构和大小的单语模型相比，多语言模型不仅实现了更低的偏差，而且预测准确率更高。

Title: Large Language Models Understand Layouts

• Authors: Weiming Li, Manni Duan, Dong An, Yan Shao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.05750
• Pdf URL: https://arxiv.org/pdf/2407.05750
• Copy Paste: [[2407.05750]] Large Language Models Understand Layouts(https://arxiv.org/abs/2407.05750)
• Keywords: language model, gpt, llm, chat
• Abstract: Large language models (LLMs) demonstrate extraordinary abilities in a wide range of natural language processing (NLP) tasks. In this paper, we show that, beyond text understanding capability, LLMs are capable of processing text layouts that are denoted by spatial markers. They are able to answer questions that require explicit spatial perceiving and reasoning, while a drastic performance drop is observed when the spatial markers from the original data are excluded. We perform a series of experiments with the GPT-3.5, Baichuan2, Llama2 and ChatGLM3 models on various types of layout-sensitive datasets for further analysis. The experimental results reveal that the layout understanding ability of LLMs is mainly introduced by the coding data for pretraining, which is further enhanced at the instruction-tuning stage. In addition, layout understanding can be enhanced by integrating low-cost, auto-generated data approached by a novel text game. Finally, we show that layout understanding ability is beneficial for building efficient visual question-answering (VQA) systems.
• 摘要：大型语言模型 (LLM) 在广泛的自然语言处理 (NLP) 任务中表现出非凡的能力。在本文中，我们表明，除了文本理解能力之外，LLM 还能够处理由空间标记表示的文本布局。它们能够回答需要明确空间感知和推理的问题，而当原始数据中的空间标记被排除时，性能会急剧下降。我们在各种类型的布局敏感数据集上使用 GPT-3.5、Baichuan2、Llama2 和 ChatGLM3 模型进行了一系列实验，以进行进一步分析。实验结果表明，LLM 的布局理解能力主要由用于预训练的编码数据引入，并在指令调整阶段得到进一步增强。此外，可以通过集成新颖的文本游戏的低成本自动生成数据来增强布局理解。最后，我们表明布局理解能力有利于构建高效的视觉问答 (VQA) 系统。

Title: When is the consistent prediction likely to be a correct prediction?

• Authors: Alex Nguyen, Dheeraj Mekala, Chengyu Dong, Jingbo Shang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05778
• Pdf URL: https://arxiv.org/pdf/2407.05778
• Copy Paste: [[2407.05778]] When is the consistent prediction likely to be a correct prediction?(https://arxiv.org/abs/2407.05778)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Self-consistency (Wang et al., 2023) suggests that the most consistent answer obtained through large language models (LLMs) is more likely to be correct. In this paper, we challenge this argument and propose a nuanced correction. Our observations indicate that consistent answers derived through more computation i.e. longer reasoning texts, rather than simply the most consistent answer across all outputs, are more likely to be correct. This is predominantly because we demonstrate that LLMs can autonomously produce chain-of-thought (CoT) style reasoning with no custom prompts merely while generating longer responses, which lead to consistent predictions that are more accurate. In the zero-shot setting, by sampling Mixtral-8x7B model multiple times and considering longer responses, we achieve 86% of its self-consistency performance obtained through zero-shot CoT prompting on the GSM8K and MultiArith datasets. Finally, we demonstrate that the probability of LLMs generating a longer response is quite low, highlighting the need for decoding strategies conditioned on output length.
• 摘要：自洽性 (Wang et al., 2023) 表明，通过大型语言模型 (LLM) 获得的最一致的答案更有可能是正确的。在本文中，我们对这一论点提出质疑，并提出了细微的修正。我们的观察表明，通过更多计算（即更长的推理文本）得出的一致答案，而不是简单地在所有输出中获得的最一致的答案，更有可能是正确的。这主要是因为我们证明了 LLM 可以在生成更长的响应的同时自主地产生没有自定义提示的思路链 (CoT) 式推理，从而产生更准确的一致预测。在零样本设置中，通过多次采样 Mixtral-8x7B 模型并考虑更长的响应，我们实现了通过 GSM8K 和 MultiArith 数据集上的零样本 CoT 提示获得的 86% 的自洽性能。最后，我们证明 LLM 生成更长响应的概率非常低，这突出了需要根据输出长度进行解码策略。

Title: Large Language Models for Judicial Entity Extraction: A Comparative Study

• Authors: Atin Sakkeer Hussain, Anu Thomas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05786
• Pdf URL: https://arxiv.org/pdf/2407.05786
• Copy Paste: [[2407.05786]] Large Language Models for Judicial Entity Extraction: A Comparative Study(https://arxiv.org/abs/2407.05786)
• Keywords: language model
• Abstract: Domain-specific Entity Recognition holds significant importance in legal contexts, serving as a fundamental task that supports various applications such as question-answering systems, text summarization, machine translation, sentiment analysis, and information retrieval specifically within case law documents. Recent advancements have highlighted the efficacy of Large Language Models in natural language processing tasks, demonstrating their capability to accurately detect and classify domain-specific facts (entities) from specialized texts like clinical and financial documents. This research investigates the application of Large Language Models in identifying domain-specific entities (e.g., courts, petitioner, judge, lawyer, respondents, FIR nos.) within case law documents, with a specific focus on their aptitude for handling domain-specific language complexity and contextual variations. The study evaluates the performance of state-of-the-art Large Language Model architectures, including Large Language Model Meta AI 3, Mistral, and Gemma, in the context of extracting judicial facts tailored to Indian judicial texts. Mistral and Gemma emerged as the top-performing models, showcasing balanced precision and recall crucial for accurate entity identification. These findings confirm the value of Large Language Models in judicial documents and demonstrate how they can facilitate and quicken scientific research by producing precise, organised data outputs that are appropriate for in-depth examination.
• 摘要：领域特定实体识别在法律环境中具有重要意义，是一项基本任务，支持各种应用，例如问答系统、文本摘要、机器翻译、情感分析和案例法文件中的信息检索。最近的进展凸显了大型语言模型在自然语言处理任务中的有效性，展示了它们能够准确地从临床和财务文件等专业文本中检测和分类领域特定事实（实体）。本研究调查了大型语言模型在识别案例法文件中领域特定实体（例如法院、请愿人、法官、律师、被告、FIR 编号）中的应用，特别关注它们处理领域特定语言复杂性和上下文变化的能力。该研究评估了最先进的大型语言模型架构（包括大型语言模型 Meta AI 3、Mistral 和 Gemma）在提取针对印度司法文本的司法事实方面的性能。 Mistral 和 Gemma 成为表现最佳的模型，展现出对于准确识别实体至关重要的平衡精度和召回率。这些发现证实了大型语言模型在司法文件中的价值，并展示了它们如何通过生成适合深入研究的精确、有组织的数据输出来促进和加快科学研究。

Title: An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

• Authors: Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.05841
• Pdf URL: https://arxiv.org/pdf/2407.05841
• Copy Paste: [[2407.05841]] An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models(https://arxiv.org/abs/2407.05841)
• Keywords: language model
• Abstract: Language Models (LMs) excel in natural language processing tasks for English but show reduced performance in most other languages. This problem is commonly tackled by continually pre-training and fine-tuning these models for said languages. A significant issue in this process is the limited vocabulary coverage in the original model's tokenizer, leading to inadequate representation of new languages and necessitating an expansion of the tokenizer. The initialization of the embeddings corresponding to new vocabulary items presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods.
• 摘要：语言模型 (LM) 在英语的自然语言处理任务中表现出色，但在大多数其他语言中表现不佳。通常通过对这些语言的模型进行持续的预训练和微调来解决此问题。此过程中的一个重要问题是原始模型的标记器中的词汇覆盖范围有限，导致无法充分表示新语言，因此需要扩展标记器。与新词汇项目相对应的嵌入的初始化提出了另一个挑战。当前的策略需要跨语言嵌入，缺乏坚实的理论基础以及与强大基线的比较。在本文中，我们首先从理论上确定在现有嵌入的凸包内进行初始化是一种很好的初始化，然后采用一种新颖但简单的方法 Constrained Word2Vec (CW2V)，它不需要跨语言嵌入。我们的研究评估了在四种语言和五项任务中扩展 RoBERTa 和 LLaMA 2 的不同初始化方法。结果表明，CW2V 的表现与更先进的技术一样好甚至更好。此外，多元初始化等更简单的方法与这些先进方法的性能相当，这表明即使使用更简单的初始化方法也可以实现高效的大规模多语言持续预训练。

Title: KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions

• Authors: Yanxu Zhu, Jinlin Xiao, Yuhang Wang, Jitao Sang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05868
• Pdf URL: https://arxiv.org/pdf/2407.05868
• Copy Paste: [[2407.05868]] KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions(https://arxiv.org/abs/2407.05868)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Recent studies have demonstrated that large language models (LLMs) are susceptible to being misled by false premise questions (FPQs), leading to errors in factual knowledge, know as factuality hallucination. Existing benchmarks that assess this vulnerability primarily rely on manual construction, resulting in limited scale and lack of scalability. In this work, we introduce an automated, scalable pipeline to create FPQs based on knowledge graphs (KGs). The first step is modifying true triplets extracted from KGs to create false premises. Subsequently, utilizing the state-of-the-art capabilities of GPTs, we generate semantically rich FPQs. Based on the proposed method, we present a comprehensive benchmark, the Knowledge Graph-based False Premise Questions (KG-FPQ), which contains approximately 178k FPQs across three knowledge domains, at six levels of confusability, and in two task formats. Using KG-FPQ, we conduct extensive evaluations on several representative LLMs and provide valuable insights. The KG-FPQ dataset and code are available at~this https URL.
• 摘要：最近的研究表明，大型语言模型 (LLM) 容易受到错误前提问题 (FPQ) 的误导，从而导致事实知识出现错误，即所谓的事实幻觉。现有的评估这种脆弱性的基准主要依赖于手动构建，导致规模有限且缺乏可扩展性。在这项工作中，我们引入了一个自动化、可扩展的管道来基于知识图谱 (KG) 创建 FPQ。第一步是修改从 KG 中提取的真实三元组以创建错误前提。随后，利用 GPT 的先进功能，我们生成语义丰富的 FPQ。基于所提出的方法，我们提出了一个全面的基准，即基于知识图谱的错误前提问题 (KG-FPQ)，它包含三个知识领域、六个混淆级别和两种任务格式的大约 178k 个 FPQ。使用 KG-FPQ，我们对几个有代表性的 LLM 进行了广泛的评估，并提供了有价值的见解。 KG-FPQ 数据集和代码可在~此 https URL 上获取。

Title: Generation and De-Identification of Indian Clinical Discharge Summaries using LLMs

• Authors: Sanjeet Singh, Shreya Gupta, Niralee Gupta, Naimish Sharma, Lokesh Srivastava, Vibhu Agarwal, Ashutosh Modi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.05887
• Pdf URL: https://arxiv.org/pdf/2407.05887
• Copy Paste: [[2407.05887]] Generation and De-Identification of Indian Clinical Discharge Summaries using LLMs(https://arxiv.org/abs/2407.05887)
• Keywords: language model, llm
• Abstract: The consequences of a healthcare data breach can be devastating for the patients, providers, and payers. The average financial impact of a data breach in recent months has been estimated to be close to USD 10 million. This is especially significant for healthcare organizations in India that are managing rapid digitization while still establishing data governance procedures that align with the letter and spirit of the law. Computer-based systems for de-identification of personal information are vulnerable to data drift, often rendering them ineffective in cross-institution settings. Therefore, a rigorous assessment of existing de-identification against local health datasets is imperative to support the safe adoption of digital health initiatives in India. Using a small set of de-identified patient discharge summaries provided by an Indian healthcare institution, in this paper, we report the nominal performance of de-identification algorithms (based on language models) trained on publicly available non-Indian datasets, pointing towards a lack of cross-institutional generalization. Similarly, experimentation with off-the-shelf de-identification systems reveals potential risks associated with the approach. To overcome data scarcity, we explore generating synthetic clinical reports (using publicly available and Indian summaries) by performing in-context learning over Large Language Models (LLMs). Our experiments demonstrate the use of generated reports as an effective strategy for creating high-performing de-identification systems with good generalization capabilities.
• 摘要：医疗数据泄露的后果对患者、提供者和付款人来说可能是毁灭性的。据估计，最近几个月数据泄露的平均财务影响接近 1000 万美元。这对于印度的医疗机构来说尤其重要，这些机构正在管理快速数字化，同时仍在建立符合法律条文和精神的数据治理程序。用于去除个人信息的计算机系统容易受到数据漂移的影响，这通常会使其在跨机构环境中无效。因此，必须根据当地医疗数据集对现有的去识别化进行严格评估，以支持在印度安全采用数字医疗计划。在本文中，我们使用印度医疗机构提供的一小部分去识别化患者出院摘要，报告了在公开的非印度数据集上训练的去识别化算法（基于语言模型）的名义性能，表明缺乏跨机构泛化。同样，对现成的去识别化系统的实验揭示了与该方法相关的潜在风险。为了克服数据稀缺问题，我们探索通过对大型语言模型 (LLM) 进行上下文学习来生成合成临床报告（使用公开和印度摘要）。我们的实验表明，使用生成的报告是创建具有良好泛化能力的高性能去识别系统的有效策略。

Title: Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop

• Authors: Anum Afzal, Alexander Kowsik, Rajna Fani, Florian Matthes
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05925
• Pdf URL: https://arxiv.org/pdf/2407.05925
• Copy Paste: [[2407.05925]] Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop(https://arxiv.org/abs/2407.05925)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large Language Models have found application in various mundane and repetitive tasks including Human Resource (HR) support. We worked with the domain experts of SAP SE to develop an HR support chatbot as an efficient and effective tool for addressing employee inquiries. We inserted a human-in-the-loop in various parts of the development cycles such as dataset collection, prompt optimization, and evaluation of generated output. By enhancing the LLM-driven chatbot's response quality and exploring alternative retrieval methods, we have created an efficient, scalable, and flexible tool for HR professionals to address employee inquiries effectively. Our experiments and evaluation conclude that GPT-4 outperforms other models and can overcome inconsistencies in data through internal reasoning capabilities. Additionally, through expert analysis, we infer that reference-free evaluation metrics such as G-Eval and Prometheus demonstrate reliability closely aligned with that of human evaluation.
• 摘要：大型语言模型已应用于各种平凡而重复的任务，包括人力资源 (HR) 支持。我们与 SAP SE 的领域专家合作开发了一款 HR 支持聊天机器人，作为解决员工问询的高效工具。我们在开发周期的各个部分（例如数据集收集、提示优化和生成输出的评估）中都加入了人机交互。通过提高 LLM 驱动的聊天机器人的响应质量并探索替代检索方法，我们为 HR 专业人员创建了一种高效、可扩展且灵活的工具，可有效解决员工问询。我们的实验和评估得出结论，GPT-4 优于其他模型，并且可以通过内部推理能力克服数据不一致的问题。此外，通过专家分析，我们推断出 G-Eval 和 Prometheus 等无参考评估指标的可靠性与人工评估的可靠性非常接近。

Title: LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

• Authors: Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, Fei Yuan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.05975
• Pdf URL: https://arxiv.org/pdf/2407.05975
• Copy Paste: [[2407.05975]] LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages(https://arxiv.org/abs/2407.05975)
• Keywords: language model, llm
• Abstract: Large Language Models~(LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages. Through a comprehensive analysis of training strategies, such as vocabulary expansion and data augmentation, we develop LLaMAX. Remarkably, without sacrificing its generalization ability, LLaMAX achieves significantly higher translation performance compared to existing open-source LLMs~(by more than 10 spBLEU points) and performs on-par with specialized translation model~(M2M-100-12B) on the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve as a robust multilingual foundation model. The code~\footnote{\url{this https URL.}} and models~\footnote{\url{this https URL.}} are publicly available.
• 摘要：大型语言模型（LLM）在高资源语言任务中表现出色，但在低资源语言中的表现却因预训练期间多语言数据不足而受到阻碍。为了解决这个问题，我们投入了 35,000 个 A100-SXM4-80GB GPU 小时对 LLaMA 系列模型进行广泛的多语言持续预训练，从而支持 100 多种语言的翻译。通过全面分析词汇扩展和数据增强等训练策略，我们开发了 LLaMAX。值得注意的是，在不牺牲其泛化能力的情况下，LLaMAX 的翻译性能显著高于现有的开源 LLM（高出 10 个 spBLEU 点以上），并且在 Flores-101 基准上的表现与专门的翻译模型（M2M-100-12B）相当。大量实验表明，LLaMAX 可以作为一个强大的多语言基础模型。代码~\footnote{\url{this https URL.}} 和模型~\footnote{\url{this https URL.}} 均已公开。

Title: Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models

• Authors: Chani Jung (1), Dongkwan Kim (1), Jiho Jin (1), Jiseon Kim (1), Yeon Seonwoo (2), Yejin Choi (3 and 4), Alice Oh (1), Hyunwoo Kim (3) ((1) KAIST, (2) Amazon, (3) Allen Institute for AI, (4) University of Washington)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.06004
• Pdf URL: https://arxiv.org/pdf/2407.06004
• Copy Paste: [[2407.06004]] Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models(https://arxiv.org/abs/2407.06004)
• Keywords: language model, llm
• Abstract: While humans naturally develop theory of mind (ToM), the capability to understand other people's mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks. We posit that we can extend our understanding of LLMs' ToM abilities by evaluating key human ToM precursors -- perception inference and perception-to-belief inference -- in LLMs. We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters' perceptions on ToMi and FANToM, respectively. Our evaluation of eight state-of-the-art LLMs reveals that the models generally perform well in perception inference while exhibiting limited capability in perception-to-belief inference (e.g., lack of inhibitory control). Based on these results, we present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference. Experimental results demonstrate that PercepToM significantly enhances LLM's performance, especially in false belief scenarios.
• 摘要：虽然人类自然会发展出心智理论 (ToM)，即理解他人心理状态和信念的能力，但最先进的大型语言模型 (LLM) 在简单的 ToM 基准上表现不佳。我们假设，我们可以通过评估 LLM 中人类 ToM 的关键前体——感知推理和感知到信念推理——来扩展我们对 LLM 的 ToM 能力的理解。我们引入了两个数据集，Percept-ToMi 和 Percept-FANToM，通过分别注释角色对 ToMi 和 FANToM 的感知来评估 LLM 中 ToM 的这些前体推理。我们对八个最先进的 LLM 的评估表明，这些模型在感知推理方面通常表现良好，但在感知到信念推理方面表现出有限的能力（例如，缺乏抑制控制）。基于这些结果，我们提出了 PercepToM，这是一种新颖的 ToM 方法，它利用 LLM 强大的感知推理能力，同时补充其有限的感知到信念推理能力。实验结果表明，PercepToM 显著提高了 LLM 的性能，尤其是在错误信念场景中。

Title: Igea: a Decoder-Only Language Model for Biomedical Text Generation in Italian

• Authors: Tommaso Mario Buonocore, Simone Rancati, Enea Parimbelli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.06011
• Pdf URL: https://arxiv.org/pdf/2407.06011
• Copy Paste: [[2407.06011]] Igea: a Decoder-Only Language Model for Biomedical Text Generation in Italian(https://arxiv.org/abs/2407.06011)
• Keywords: language model
• Abstract: The development of domain-specific language models has significantly advanced natural language processing applications in various specialized fields, particularly in biomedicine. However, the focus has largely been on English-language models, leaving a gap for less-resourced languages such as Italian. This paper introduces Igea, the first decoder-only language model designed explicitly for biomedical text generation in Italian. Built on the Minerva model and continually pretrained on a diverse corpus of Italian medical texts, Igea is available in three model sizes: 350 million, 1 billion, and 3 billion parameters. The models aim to balance computational efficiency and performance, addressing the challenges of managing the peculiarities of medical terminology in Italian. We evaluate Igea using a mix of in-domain biomedical corpora and general-purpose benchmarks, highlighting its efficacy and retention of general knowledge even after the domain-specific training. This paper discusses the model's development and evaluation, providing a foundation for future advancements in Italian biomedical NLP.
• 摘要：领域特定语言模型的发展极大地推动了各个专业领域（尤其是生物医学）的自然语言处理应用。然而，人们的注意力主要集中在英语模型上，而资源较少的语言（如意大利语）则存在空白。本文介绍了 Igea，这是第一个专门为意大利语生物医学文本生成而设计的仅解码器语言模型。Igea 以 Minerva 模型为基础，并在意大利医学文本的多样化语料库上不断进行预训练，有三种模型大小：3.5 亿、10 亿和 30 亿个参数。这些模型旨在平衡计算效率和性能，解决管理意大利语医学术语特殊性的挑战。我们使用领域内生物医学语料库和通用基准对 Igea 进行评估，强调了其有效性以及即使在领域特定训练之后也能保留一般知识。本文讨论了该模型的开发和评估，为意大利生物医学 NLP 的未来发展奠定了基础。

Title: Distilling System 2 into System 1

• Authors: Ping Yu, Jing Xu, Jason Weston, Ilia Kulikov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.06023
• Pdf URL: https://arxiv.org/pdf/2407.06023
• Copy Paste: [[2407.06023]] Distilling System 2 into System 1(https://arxiv.org/abs/2407.06023)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large language models (LLMs) can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses. Since Chain-of-Thought (Wei et al., 2022), many such System 2 techniques have been proposed such as Rephrase and Respond (Deng et al., 2023a), System 2 Attention (Weston and Sukhbaatar, 2023) and Branch-Solve-Merge (Saha et al., 2023). In this work we investigate self-supervised methods to ``compile'' (distill) higher quality outputs from System 2 techniques back into LLM generations without intermediate reasoning token sequences, as this reasoning has been distilled into System 1. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that such System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities on the reasoning tasks that they cannot yet do well.
• 摘要：大型语言模型 (LLM) 可以在推理过程中花费额外的计算来生成中间想法，这有助于产生更好的最终响应。自思想链 (Wei 等人，2022) 以来，已经提出了许多此类系统 2 技术，例如重新措辞和响应 (Deng 等人，2023a)、系统 2 注意力 (Weston 和 Sukhbaatar，2023) 和分支求解合并 (Saha 等人，2023)。在这项工作中，我们研究了自监督方法，将系统 2 技术的更高质量输出“编译”（提炼）回 LLM 生成，而无需中间推理标记序列，因为这种推理已被提炼到系统 1 中。我们表明，可以成功提炼几种这样的技术，与原始系统 1 性能相比，可以得到更好的结果，并且推理成本低于系统 2。我们假设这种系统 2 提炼将成为未来持续学习 AI 系统的一个重要特征，使它们能够将系统 2 的能力集中在它们尚不能做好的推理任务上。

Title: PAS: Data-Efficient Plug-and-Play Prompt Augmentation System

• Authors: Miao Zheng, Hao Liang, Fan Yang, Haoze Sun, Tianpeng Li, Lingchu Xiong, Yan Zhang, Yozhen Wu, Kun Li, Yanjun Sheng, Mingan Lin, Tao Zhang, Guosheng Dong, Yujing Qiao, Kun Fang, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.06027
• Pdf URL: https://arxiv.org/pdf/2407.06027
• Copy Paste: [[2407.06027]] PAS: Data-Efficient Plug-and-Play Prompt Augmentation System(https://arxiv.org/abs/2407.06027)
• Keywords: language model, llm, prompt
• Abstract: In recent years, the rise of Large Language Models (LLMs) has spurred a growing demand for plug-and-play AI systems. Among the various AI techniques, prompt engineering stands out as particularly significant. However, users often face challenges in writing prompts due to the steep learning curve and significant time investment, and existing automatic prompt engineering (APE) models can be difficult to use. To address this issue, we propose PAS, an LLM-based plug-and-play APE system. PAS utilizes LLMs trained on high-quality, automatically generated prompt complementary datasets, resulting in exceptional performance. In comprehensive benchmarks, PAS achieves state-of-the-art (SoTA) results compared to previous APE models, with an average improvement of 6.09 points. Moreover, PAS is highly efficient, achieving SoTA performance with only 9000 data points. Additionally, PAS can autonomously generate prompt augmentation data without requiring additional human labor. Its flexibility also allows it to be compatible with all existing LLMs and applicable to a wide range of tasks. PAS excels in human evaluations, underscoring its suitability as a plug-in for users. This combination of high performance, efficiency, and flexibility makes PAS a valuable system for enhancing the usability and effectiveness of LLMs through improved prompt engineering.
• 摘要：近年来，大型语言模型 (LLM) 的兴起刺激了对即插即用 AI 系统的需求不断增长。在各种 AI 技术中，提示工程尤为重要。然而，由于学习难度高、时间投入大，用户在编写提示时经常面临挑战，而现有的自动提示工程 (APE) 模型可能难以使用。为了解决这个问题，我们提出了基于 LLM 的即插即用 APE 系统 PAS。PAS 利用在高质量、自动生成的提示互补数据集上训练的 LLM，从而获得卓越的性能。在综合基准测试中，与之前的 APE 模型相比，PAS 获得了最先进的 (SoTA) 结果，平均提高了 6.09 分。此外，PAS 效率很高，仅用 9000 个数据点就实现了 SoTA 性能。此外，PAS 可以自主生成提示增强数据，而无需额外的人力。它的灵活性还使其能够与所有现有的 LLM 兼容并适用于广泛的任务。 PAS 在人工评估方面表现出色，这凸显了其作为用户插件的适用性。高性能、高效率和灵活性的结合使 PAS 成为一种有价值的系统，可通过改进的快速工程来提高 LLM 的可用性和有效性。

Title: MST5 -- Multilingual Question Answering over Knowledge Graphs

• Authors: Nikit Srivastava, Mengshi Ma, Daniel Vollmers, Hamada Zahera, Diego Moussallem, Axel-Cyrille Ngonga Ngomo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MST5 -- Multilingual Question Answering over Knowledge Graphs(https://arxiv.org/abs/)
• Keywords: language model
• Abstract: Knowledge Graph Question Answering (KGQA) simplifies querying vast amounts of knowledge stored in a graph-based model using natural language. However, the research has largely concentrated on English, putting non-English speakers at a disadvantage. Meanwhile, existing multilingual KGQA systems face challenges in achieving performance comparable to English systems, highlighting the difficulty of generating SPARQL queries from diverse languages. In this research, we propose a simplified approach to enhance multilingual KGQA systems by incorporating linguistic context and entity information directly into the processing pipeline of a language model. Unlike existing methods that rely on separate encoders for integrating auxiliary information, our strategy leverages a single, pretrained multilingual transformer-based language model to manage both the primary input and the auxiliary data. Our methodology significantly improves the language model's ability to accurately convert a natural language query into a relevant SPARQL query. It demonstrates promising results on the most recent QALD datasets, namely QALD-9-Plus and QALD-10. Furthermore, we introduce and evaluate our approach on Chinese and Japanese, thereby expanding the language diversity of the existing datasets.
• 摘要：知识图谱问答 (KGQA) 简化了使用自然语言查询存储在基于图的模型中的大量知识。然而，这项研究主要集中在英语上，这对非英语人士来说不利。同时，现有的多语言 KGQA 系统在实现与英语系统相当的性能方面面临挑战，这凸显了从不同语言生成 SPARQL 查询的难度。在本研究中，我们提出了一种简化的方法，通过将语言上下文和实体信息直接合并到语言模型的处理管道中来增强多语言 KGQA 系统。与依赖单独编码器来集成辅助信息的现有方法不同，我们的策略利用单个、预训练的多语言基于转换器的语言模型来管理主要输入和辅助数据。我们的方法显著提高了语言模型将自然语言查询准确转换为相关 SPARQL 查询的能力。它在最新的 QALD 数据集（即 QALD-9-Plus 和 QALD-10）上展示了令人鼓舞的结果。此外，我们介绍并评估了我们在中文和日语上的方法，从而扩大了现有数据集的语言多样性。

Title: Variational Best-of-N Alignment

• Authors: Afra Amini, Tim Vieira, Ryan Cotterell
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.06057
• Pdf URL: https://arxiv.org/pdf/2407.06057
• Copy Paste: [[2407.06057]] Variational Best-of-N Alignment(https://arxiv.org/abs/2407.06057)
• Keywords: language model
• Abstract: Best-of-N (BoN) is a popular and effective algorithm for aligning language models to human preferences. The algorithm works as follows: at inference time, N samples are drawn from the language model, and the sample with the highest reward, as judged by a reward model, is returned as the output. Despite its effectiveness, BoN is computationally expensive; it reduces sampling throughput by a factor of N. To make BoN more efficient at inference time, one strategy is to fine-tune the language model to mimic what BoN does during inference. To achieve this, we derive the distribution induced by the BoN algorithm. We then propose to fine-tune the language model to minimize backward KL divergence to the BoN distribution. Our approach is analogous to mean-field variational inference and, thus, we term it variational BoN (vBoN). To the extent this fine-tuning is successful and we end up with a good approximation, we have reduced the inference cost by a factor of N. Our experiments on a controlled generation task suggest that while variational BoN is not as effective as BoN in aligning language models, it is close to BoN performance as vBoN appears more often on the Pareto frontier of reward and KL divergence compared to models trained with KL-constrained RL objective.
• 摘要：Best-of-N (BoN) 是一种流行且有效的算法，用于将语言模型与人类偏好对齐。该算法的工作原理如下：在推理时，从语言模型中抽取 N 个样本，并返回由奖励模型判断的奖励最高的样本作为输出。尽管 BoN 有效，但它在计算上很昂贵；它将采样吞吐量降低了 N 倍。为了使 BoN 在推理时更高效，一种策略是微调语言模型以模仿 BoN 在推理过程中所做的事情。为实现这一点，我们推导出由 BoN 算法引起的分布。然后，我们建议微调语言模型以最小化 BoN 分布的后向 KL 散度。我们的方法类似于均值场变分推理，因此我们将其称为变分 BoN (vBoN)。如果这种微调成功并且我们最终得到了一个很好的近似值，我们将推理成本降低了 N 倍。我们在受控生成任务上的实验表明，虽然变分 BoN 在对齐语言模型方面不如 BoN 有效，但它接近 BoN 性能，因为与使用 KL 约束 RL 目标训练的模型相比，vBoN 更频繁地出现在奖励和 KL 散度的帕累托前沿上。

Title: From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty

• Authors: Maor Ivgi, Ori Yoran, Jonathan Berant, Mor Geva
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.06071
• Pdf URL: https://arxiv.org/pdf/2407.06071
• Copy Paste: [[2407.06071]] From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty(https://arxiv.org/abs/2407.06071)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) often exhibit undesirable behaviors, such as hallucinations and sequence repetitions. We propose to view these behaviors as fallbacks that models exhibit under uncertainty, and investigate the connection between them. We categorize fallback behaviors -- sequence repetitions, degenerate text, and hallucinations -- and extensively analyze them in models from the same family that differ by the amount of pretraining tokens, parameter count, or the inclusion of instruction-following training. Our experiments reveal a clear and consistent ordering of fallback behaviors, across all these axes: the more advanced an LLM is (i.e., trained on more tokens, has more parameters, or instruction-tuned), its fallback behavior shifts from sequence repetitions, to degenerate text, and then to hallucinations. Moreover, the same ordering is observed throughout a single generation, even for the best-performing models; as uncertainty increases, models shift from generating hallucinations to producing degenerate text and then sequence repetitions. Lastly, we demonstrate that while common decoding techniques, such as random sampling, might alleviate some unwanted behaviors like sequence repetitions, they increase harder-to-detect hallucinations.
• 摘要：大型语言模型 (LLM) 经常表现出不良行为，例如幻觉和序列重复。我们建议将这些行为视为模型在不确定情况下表现出的后备行为，并研究它们之间的联系。我们将后备行为分类为序列重复、退化文本和幻觉，并在同一系列的模型中对它们进行广泛分析，这些模型在预训练标记数量、参数数量或包含的指令跟踪训练方面有所不同。我们的实验揭示了后备行为在所有这些轴上的清晰且一致的顺序：LLM 越先进（即在更多标记上进行训练、具有更多参数或指令调整），其后备行为就会从序列重复转变为退化文本，然后转变为幻觉。此外，即使是表现最佳的模型，在单代中也会观察到相同的顺序；随着不确定性的增加，模型从产生幻觉转变为产生退化文本，然后是序列重复。最后，我们证明，虽然常见的解码技术（例如随机采样）可能会减轻一些不良行为（例如序列重复），但它们会增加更难检测的幻觉。

Title: Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

• Authors: Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: The remarkable success of Large Language Models (LLMs) has ushered natural language processing (NLP) research into a new era. Despite their diverse capabilities, LLMs trained on different corpora exhibit varying strengths and weaknesses, leading to challenges in maximizing their overall efficiency and versatility. To address these challenges, recent studies have explored collaborative strategies for LLMs. This paper provides a comprehensive overview of this emerging research area, highlighting the motivation behind such collaborations. Specifically, we categorize collaborative strategies into three primary approaches: Merging, Ensemble, and Cooperation. Merging involves integrating multiple LLMs in the parameter space. Ensemble combines the outputs of various LLMs. Cooperation} leverages different LLMs to allow full play to their diverse capabilities for specific tasks. We provide in-depth introductions to these methods from different perspectives and discuss their potential applications. Additionally, we outline future research directions, hoping this work will catalyze further studies on LLM collaborations and paving the way for advanced NLP applications.
• 摘要：大型语言模型 (LLM) 的巨大成功将自然语言处理 (NLP) 研究带入了一个新时代。尽管 LLM 具有多种功能，但在不同语料库上训练的 LLM 表现出不同的优势和劣势，导致在最大限度地提高其整体效率和多功能性方面面临挑战。为了应对这些挑战，最近的研究探索了 LLM 的协作策略。本文全面概述了这一新兴研究领域，强调了这种协作背后的动机。具体来说，我们将协作策略分为三种主要方法：合并、集成和合作。合并涉及在参数空间中集成多个 LLM。集成结合了各种 LLM 的输出。合作利用不同的 LLM 充分发挥它们在特定任务中的不同能力。我们从不同角度深入介绍了这些方法，并讨论了它们的潜在应用。此外，我们概述了未来的研究方向，希望这项工作能够促进对 LLM 协作的进一步研究，并为高级 NLP 应用铺平道路。

Title: Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoning

• Authors: Yadong Zhang, Shaoguang Mao, Wenshan Wu, Yan Xia, Tao Ge, Man Lan, Furu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.06112
• Pdf URL: https://arxiv.org/pdf/2407.06112
• Copy Paste: [[2407.06112]] Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoning(https://arxiv.org/abs/2407.06112)
• Keywords: language model, llm, agent
• Abstract: This paper introduces BI-Directional DEliberation Reasoning (BIDDER), a novel reasoning approach to enhance the decision rationality of language models. Traditional reasoning methods typically rely on historical information and employ uni-directional (left-to-right) reasoning strategy. This lack of bi-directional deliberation reasoning results in limited awareness of potential future outcomes and insufficient integration of historical context, leading to suboptimal decisions. BIDDER addresses this gap by incorporating principles of rational decision-making, specifically managing uncertainty and predicting expected utility. Our approach involves three key processes: Inferring hidden states to represent uncertain information in the decision-making process from historical data; Using these hidden states to predict future potential states and potential outcomes; Integrating historical information (past contexts) and long-term outcomes (future contexts) to inform reasoning. By leveraging bi-directional reasoning, BIDDER ensures thorough exploration of both past and future contexts, leading to more informed and rational decisions. We tested BIDDER's effectiveness in two well-defined scenarios: Poker (Limit Texas Hold'em) and Negotiation. Our experiments demonstrate that BIDDER significantly improves the decision-making capabilities of LLMs and LLM agents.
• 摘要：本文介绍了双向深思熟虑推理 (BIDDER)，这是一种增强语言模型决策合理性的新型推理方法。传统推理方法通常依赖于历史信息并采用单向（从左到右）推理策略。这种双向深思熟虑推理的缺乏导致对未来潜在结果的认识有限，对历史背景的整合不足，从而导致决策不理想。BIDDER 通过结合理性决策原则来解决这一问题，特别是管理不确定性和预测预期效用。我们的方法涉及三个关键过程：从历史数据中推断隐藏状态以表示决策过程中的不确定信息；使用这些隐藏状态预测未来的潜在状态和潜在结果；整合历史信息（过去背景）和长期结果（未来背景）以指导推理。通过利用双向推理，BIDDER 确保彻底探索过去和未来的背景，从而做出更明智和理性的决策。我们在两个明确定义的场景中测试了 BIDDER 的有效性：扑克（限注德州扑克）和谈判。我们的实验表明，BIDDER 显著提高了 LLM 和 LLM 代理的决策能力。

Title: ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation

• Authors: Ethan Chern, Jiadi Su, Yan Ma, Pengfei Liu
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.06135
• Pdf URL: https://arxiv.org/pdf/2407.06135
• Copy Paste: [[2407.06135]] ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation(https://arxiv.org/abs/2407.06135)
• Keywords: language model, llm
• Abstract: Previous open-source large multimodal models (LMMs) have faced several limitations: (1) they often lack native integration, requiring adapters to align visual representations with pre-trained large language models (LLMs); (2) many are restricted to single-modal generation; (3) while some support multimodal generation, they rely on separate diffusion models for visual modeling and generation. To mitigate these limitations, we present Anole, an open, autoregressive, native large multimodal model for interleaved image-text generation. We build Anole from Meta AI's Chameleon, adopting an innovative fine-tuning strategy that is both data-efficient and parameter-efficient. Anole demonstrates high-quality, coherent multimodal generation capabilities. We have open-sourced our model, training framework, and instruction tuning data.
• 摘要：以前的开源大型多模态模型 (LMM) 面临几个限制：(1) 它们通常缺乏本机集成，需要适配器将视觉表示与预训练的大型语言模型 (LLM) 对齐；(2) 许多仅限于单模态生成；(3) 虽然有些支持多模态生成，但它们依赖单独的扩散模型进行视觉建模和生成。为了缓解这些限制，我们提出了 Anole，这是一个开放的、自回归的、本机的大型多模态模型，用于交错图像文本生成。我们从 Meta AI 的 Chameleon 构建了 Anole，采用了一种既节省数据又节省参数的创新微调策略。Anole 展示了高质量、连贯的多模态生成能力。我们已经开源了我们的模型、训练框架和指令调整数据。

Title: Using Grammar Masking to Ensure Syntactic Validity in LLM-based Modeling Tasks

• Authors: Lukas Netz, Jan Reimar, Bernhard Rumpe
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.06146
• Pdf URL: https://arxiv.org/pdf/2407.06146
• Copy Paste: [[2407.06146]] Using Grammar Masking to Ensure Syntactic Validity in LLM-based Modeling Tasks(https://arxiv.org/abs/2407.06146)
• Keywords: language model, llm, prompt
• Abstract: We present and evaluate a method called grammar masking, which is used to guide large language models (LLMs) toward producing syntactically correct models for a given context-free grammar. Prompt engineering methods such as few-shot learning or priming can be used to improve the chances of an LLM producing correct syntax, but the more complex the grammar, the more time-consuming and less promising these methods become. Previous work is focused primarily on the usage of either language model training or prompt engineering. In this work, a method is presented that restricts the output to a given grammar using constrained decoding to ensure the output adheres to a valid syntax. We use several DSLs built with MontiCore and task multiple LLMs to produce models with and without constrained decoding. A corresponding parser is used to confirm the syntactic correctness of each model. We show that grammar masking can dramatically improve the modeling capabilities of several LLMs, reducing the need for well-refined prompting while increasing the chance of producing correct models.
• 摘要：我们提出并评估了一种称为语法掩码的方法，该方法用于指导大型语言模型 (LLM) 为给定的上下文无关语法生成语法正确的模型。可以使用提示工程方法（例如少量学习或启动）来提高 LLM 生成正确语法的机会，但语法越复杂，这些方法就越耗时且前景越暗淡。以前的工作主要集中在语言模型训练或提示工程的使用上。在这项工作中，提出了一种方法，该方法使用约束解码将输出限制为给定的语法，以确保输出遵循有效的语法。我们使用用 MontiCore 构建的几个 DSL，并让多个 LLM 生成具有和不具有约束解码的模型。使用相应的解析器来确认每个模型的语法正确性。我们表明，语法掩码可以显著提高几个 LLM 的建模能力，减少对精细提示的需求，同时增加生成正确模型的机会。