2024-09-27

Title: REAL: Response Embedding-based Alignment for LLMs

• Authors: Honggen Zhang, Igor Molybog, June Zhang, Xufeng Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17169
• Pdf URL: https://arxiv.org/pdf/2409.17169
• Copy Paste: [[2409.17169]] REAL: Response Embedding-based Alignment for LLMs(https://arxiv.org/abs/2409.17169)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) to human preferences is a crucial step in building helpful and safe AI tools, which usually involve training on supervised datasets. Popular algorithms such as Direct Preference Optimization rely on pairs of AI-generated responses ranked according to human feedback. The labeling process is the most labor-intensive and costly part of the alignment pipeline, and improving its efficiency would have a meaningful impact on AI development. We propose a strategy for sampling a high-quality training dataset that focuses on acquiring the most informative response pairs for labeling out of a set of AI-generated responses. Experimental results on synthetic HH-RLHF benchmarks indicate that choosing dissimilar response pairs enhances the direct alignment of LLMs while reducing inherited labeling errors. We also applied our method to the real-world dataset SHP2, selecting optimal pairs from multiple responses. The model aligned on dissimilar response pairs obtained the best win rate on the dialogue task. Our findings suggest that focusing on less similar pairs can improve the efficiency of LLM alignment, saving up to 65% of annotators' work.
• 摘要：将大型语言模型 (LLM) 与人类偏好对齐是构建有用且安全的 AI 工具的关键步骤，这通常涉及对监督数据集进行训练。直接偏好优化等流行算法依赖于根据人类反馈排序的 AI 生成的响应对。标记过程是对齐流程中最耗费人力和成本的部分，提高其效率将对 AI 开发产生重大影响。我们提出了一种对高质量训练数据集进行采样的策略，该策略侧重于从一组 AI 生成的响应中获取最具信息量的响应对以进行标记。在合成 HH-RLHF 基准上的实验结果表明，选择不同的响应对可增强 LLM 的直接对齐，同时减少继承的标记错误。我们还将我们的方法应用于现实世界数据集 SHP2，从多个响应中选择最佳对。在不同响应对上对齐的模型在对话任务中获得了最佳胜率。我们的研究结果表明，专注于不太相似的对可以提高 LLM 对齐的效率，节省多达 65% 的注释者工作。

Title: Cross-Domain Content Generation with Domain-Specific Small Language Models

• Authors: Ankit Maloo Abhinav Garg
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17171
• Pdf URL: https://arxiv.org/pdf/2409.17171
• Copy Paste: [[2409.17171]] Cross-Domain Content Generation with Domain-Specific Small Language Models(https://arxiv.org/abs/2409.17171)
• Keywords: language model
• Abstract: Generating domain-specific content using small language models poses challenges, especially when dealing with multiple distinct datasets with minimal overlap. In this study, we explore methods to enable a small language model to produce coherent and relevant outputs for two different domains: stories (Dataset A) and recipes (Dataset B). Our initial experiments show that training individual models on each dataset yields satisfactory results, with each model generating appropriate content within its domain. We find that utilizing custom tokenizers tailored to each dataset significantly enhances generation quality compared to using a generic tokenizer. Attempts to adapt a single model to both domains using Low-Rank Adaptation (LoRA) or standard fine-tuning do not yield substantial results, often failing to produce meaningful outputs. Moreover, full fine-tuning without freezing the model's existing weights leads to catastrophic forgetting, where the model loses previously learned information and only retains knowledge from the new data. To overcome these challenges, we employ a knowledge expansion strategy: training only with additional parameters. This approach enables the model to generate both stories and recipes upon request, effectively handling multiple domains without suffering from catastrophic forgetting. Our findings demonstrate that knowledge expansion with frozen layers is an effective method for small language models to generate domain-specific content across distinct datasets. This work contributes to the development of efficient multi-domain language models and provides insights into managing catastrophic forgetting in small-scale architectures.
• 摘要：使用小型语言模型生成特定领域的内容存在挑战，尤其是在处理重叠最小的多个不同数据集时。在本研究中，我们探索了使小型语言模型能够为两个不同领域生成连贯且相关的输出的方法：故事（数据集 A）和食谱（数据集 B）。我们的初步实验表明，在每个数据集上训练单个模型会产生令人满意的结果，每个模型都会在其领域内生成适当的内容。我们发现，与使用通用标记器相比，使用针对每个数据集量身定制的自定义标记器可以显著提高生成质量。尝试使用低秩自适应 (LoRA) 或标准微调将单个模型适配到两个领域不会产生实质性结果，通常无法产生有意义的输出。此外，在不冻结模型现有权重的情况下进行完全微调会导致灾难性的遗忘，其中模型会丢失先前学习的信息，而只保留来自新数据的知识。为了克服这些挑战，我们采用了一种知识扩展策略：仅使用其他参数进行训练。这种方法使模型能够根据请求生成故事和食谱，从而有效地处理多个领域，而不会遭受灾难性遗忘。我们的研究结果表明，使用冻结层进行知识扩展是小型语言模型在不同数据集中生成领域特定内容的有效方法。这项工作有助于开发高效的多领域语言模型，并为管理小规模架构中的灾难性遗忘提供了见解。

Title: What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning

• Authors: Shashidhar Reddy Javaji, Zining Zhu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17172
• Pdf URL: https://arxiv.org/pdf/2409.17172
• Copy Paste: [[2409.17172]] What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning(https://arxiv.org/abs/2409.17172)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large language models (LLMs) can store a massive amount of knowledge, yet their potential to acquire new knowledge remains unknown. We propose a novel evaluation framework that evaluates this capability. This framework prompts LLMs to generate questions about a statement introducing scientific knowledge, simulating a curious person when facing the statement for the first time. We score the qualities of the generated questions, thereby evaluating the knowledge acquisition potential of the LLM. We apply controlled ablation studies to validate our scoring procedures. Additionally, we created a synthetic dataset consisting of 1101 statements in physics, chemistry, and maths with distinct levels of difficulties, 300 general knowledge statements, and 567 incorrect statements. Human evaluations were conducted to validate our model assessments, achieving an approximate weighted Cohen's kappa of 0.7 on all three metrics considered. We find that while large models like GPT-4 and Mistral 8x7b are adept at generating coherent and relevant questions, the smaller Phi-2 model is equally or more effective. This indicates that size does not solely determine a model's knowledge acquisition potential. The proposed framework quantifies a critical model capability that was commonly overlooked and opens up research opportunities for developing more knowledgeable AI systems
• 摘要：大型语言模型 (LLM) 可以存储大量知识，但它们获取新知识的潜力仍然未知。我们提出了一个评估这种能力的新型评估框架。该框架提示 LLM 生成有关介绍科学知识的陈述的问题，模拟好奇的人第一次面对该陈述时的情形。我们对生成的问题的质量进行评分，从而评估 LLM 的知识获取潜力。我们应用受控消融研究来验证我们的评分程序。此外，我们创建了一个合成数据集，其中包含 1101 条物理、化学和数学陈述，这些陈述具有不同的难度级别，300 条常识陈述和 567 条错误陈述。进行了人工评估以验证我们的模型评估，在考虑的所有三个指标上实现了近似的加权 Cohen's kappa 0.7。我们发现，虽然 GPT-4 和 Mistral 8x7b 等大型模型擅长生成连贯且相关的问题，但较小的 Phi-2 模型同样或更有效。这表明规模并不是唯一决定模型知识获取潜力的原因。所提出的框架量化了通常被忽视的关键模型能力，并为开发更具知识性的人工智能系统开辟了研究机会

Title: A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models

• Authors: Satoshi Munakata, Taku Fukui, Takao Mohri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17173
• Pdf URL: https://arxiv.org/pdf/2409.17173
• Copy Paste: [[2409.17173]] A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models(https://arxiv.org/abs/2409.17173)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large language models (LLMs) often fabricate a hallucinatory text. Several methods have been developed to detect such text by semantically comparing it with the multiple versions probabilistically regenerated. However, a significant issue is that if the storyline of each regenerated text changes, the generated texts become incomparable, which worsen detection accuracy. In this paper, we propose a hallucination detection method that incorporates a multiple-fill-in-the-blank exam approach to address this storyline-changing issue. First, our method creates a multiple-fill-in-the-blank exam by masking multiple objects from the original text. Second, prompts an LLM to repeatedly answer this exam. This approach ensures that the storylines of the exam answers align with the original ones. Finally, quantifies the degree of hallucination for each original sentence by scoring the exam answers, considering the potential for \emph{hallucination snowballing} within the original text itself. Experimental results show that our method alone not only outperforms existing methods, but also achieves clearer state-of-the-art performance in the ensembles with existing methods.
• 摘要：大型语言模型 (LLM) 通常会编造幻觉文本。已经开发了几种方法来检测此类文本，方法是将其与概率再生的多个版本进行语义比较。然而，一个重要的问题是，如果每个再生文本的故事情节发生变化，生成的文本将变得无法比较，这会降低检测准确性。在本文中，我们提出了一种幻觉检测方法，该方法结合了多重填空考试方法来解决这个故事情节变化的问题。首先，我们的方法通过屏蔽原始文本中的多个对象来创建多重填空考试。其次，提示 LLM 重复回答这个考试。这种方法确保考试答案的故事情节与原始故事情节一致。最后，通过对考试答案进行评分来量化每个原始句子的幻觉程度，考虑到原始文本本身中 \emph{幻觉滚雪球} 的可能性。实验结果表明，我们的方法不仅单独表现优于现有方法，而且在与现有方法的集成中取得了更清晰的最先进的性能。

Title: CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency

• Authors: Kangsheng Wang, Xiao Zhang, Zizheng Guo, Tianyu Hu, Huimin Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17174
• Pdf URL: https://arxiv.org/pdf/2409.17174
• Copy Paste: [[2409.17174]] CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency(https://arxiv.org/abs/2409.17174)
• Keywords: language model, llm
• Abstract: Chain-based reasoning methods like chain of thought (CoT) play a rising role in solving reasoning tasks for large language models (LLMs). However, the causal illusions between \textit{a step of reasoning} and \textit{corresponding state transitions} are becoming a significant obstacle to advancing LLMs' reasoning capabilities, especially in long-range reasoning tasks. This paper proposes a non-chain-based reasoning framework for simultaneous consideration of causal significance and consistency, i.e., the Causal Significance and Consistency Enhancer (CSCE). We customize LLM's loss function utilizing treatment effect assessments to enhance its reasoning ability from two aspects: causal significance and consistency. This ensures that the model captures essential causal relationships and maintains robust and consistent performance across various scenarios. Additionally, we transform the reasoning process from the cascading multiple one-step reasoning commonly used in Chain-Based methods, like CoT, to a causal-enhanced method that outputs the entire reasoning process in one go, further improving the model's reasoning efficiency. Extensive experiments show that our method improves both the reasoning success rate and speed. These improvements further demonstrate that non-chain-based methods can also aid LLMs in completing reasoning tasks.
• 摘要：基于链的推理方法，如思路链 (CoT)，在解决大型语言模型 (LLM) 的推理任务中发挥着越来越重要的作用。然而，\textit{推理步骤} 和 \textit{对应状态转换} 之间的因果错觉正在成为提高 LLM 推理能力的重大障碍，尤其是在长距离推理任务中。本文提出了一个非基于链的推理框架，用于同时考虑因果意义和一致性，即因果意义和一致性增强器 (CSCE)。我们利用处理效果评估定制 LLM 的损失函数，从因果意义和一致性两个方面增强其推理能力。这确保模型捕捉到必要的因果关系并在各种场景中保持稳健一致的性能。此外，我们将推理过程从链式方法（如 CoT）中常用的级联多个单步推理转变为因果增强方法，一次性输出整个推理过程，进一步提高模型的推理效率。大量实验表明，我们的方法提高了推理成功率和速度。这些改进进一步证明了非链式方法也可以帮助 LLM 完成推理任务。

Title: Fully automatic extraction of morphological traits from the Web: utopia or reality?

• Authors: Diego Marcos, Robert van de Vlasakker, Ioannis N. Athanasiadis, Pierre Bonnet, Hervé Goeau, Alexis Joly, W. Daniel Kissling, César Leblanc, André S.J. van Proosdij, Konstantinos P. Panousis
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17179
• Pdf URL: https://arxiv.org/pdf/2409.17179
• Copy Paste: [[2409.17179]] Fully automatic extraction of morphological traits from the Web: utopia or reality?(https://arxiv.org/abs/2409.17179)
• Keywords: language model, llm
• Abstract: Plant morphological traits, their observable characteristics, are fundamental to understand the role played by each species within their ecosystem. However, compiling trait information for even a moderate number of species is a demanding task that may take experts years to accomplish. At the same time, massive amounts of information about species descriptions is available online in the form of text, although the lack of structure makes this source of data impossible to use at scale. To overcome this, we propose to leverage recent advances in large language models (LLMs) and devise a mechanism for gathering and processing information on plant traits in the form of unstructured textual descriptions, without manual curation. We evaluate our approach by automatically replicating three manually created species-trait matrices. Our method managed to find values for over half of all species-trait pairs, with an F1-score of over 75%. Our results suggest that large-scale creation of structured trait databases from unstructured online text is currently feasible thanks to the information extraction capabilities of LLMs, being limited by the availability of textual descriptions covering all the traits of interest.
• 摘要：植物形态特征，即其可观察的特征，对于了解每个物种在其生态系统中所扮演的角色至关重要。然而，即使是为中等数量的物种汇编特征信息也是一项艰巨的任务，可能需要专家花费数年时间才能完成。与此同时，大量有关物种描述的信息以文本形式在线提供，尽管缺乏结构使得这种数据源无法大规模使用。为了克服这个问题，我们建议利用大型语言模型 (LLM) 的最新进展，并设计一种机制，以非结构化文本描述的形式收集和处理有关植物特征的信息，而无需人工管理。我们通过自动复制三个手动创建的物种-特征矩阵来评估我们的方法。我们的方法设法找到了超过一半的物种-特征对的值，F1 得分超过 75%。我们的结果表明，由于 LLM 的信息提取能力，目前可以从非结构化在线文本大规模创建结构化特征数据库，但受限于涵盖所有感兴趣特征的文本描述的可用性。

Title: Plurals: A System for Guiding LLMs Via Simulated Social Ensembles

• Authors: Joshua Ashkinaze, Emily Fry, Narendra Edara, Eric Gilbert, Ceren Budak
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.MA
• Abstract URL: https://arxiv.org/abs/2409.17213
• Pdf URL: https://arxiv.org/pdf/2409.17213
• Copy Paste: [[2409.17213]] Plurals: A System for Guiding LLMs Via Simulated Social Ensembles(https://arxiv.org/abs/2409.17213)
• Keywords: language model, llm, agent
• Abstract: Recent debates raised concerns that language models may favor certain viewpoints. But what if the solution is not to aim for a 'view from nowhere' but rather to leverage different viewpoints? We introduce Plurals, a system and Python library for pluralistic AI deliberation. Plurals consists of Agents (LLMs, optionally with personas) which deliberate within customizable Structures, with Moderators overseeing deliberation. Plurals is a generator of simulated social ensembles. Plurals integrates with government datasets to create nationally representative personas, includes deliberation templates inspired by democratic deliberation theory, and allows users to customize both information-sharing structures and deliberation behavior within Structures. Six case studies demonstrate fidelity to theoretical constructs and efficacy. Three randomized experiments show simulated focus groups produced output resonant with an online sample of the relevant audiences (chosen over zero-shot generation in 75% of trials). Plurals is both a paradigm and a concrete system for pluralistic AI. The Plurals library is available at this https URL and will be continually updated.
• 摘要：最近的辩论引发了人们对语言模型可能偏向某些观点的担忧。但如果解决方案不是追求“无处不在的观点”，而是利用不同的观点，情况会怎样？我们引入了 Plurals，这是一个用于多元化 AI 审议的系统和 Python 库。Plurals 由代理（LLM，可选角色）组成，它们在可定制的结构中进行审议，由主持人监督审议。Plurals 是模拟社交集合的生成器。Plurals 与政府数据集集成以创建具有全国代表性的角色，包括受民主审议理论启发的审议模板，并允许用户在结构中自定义信息共享结构和审议行为。六个案例研究证明了对理论构造和有效性的忠诚度。三项随机实验表明，模拟焦点小组产生的输出与相关受众的在线样本产生共鸣（75% 的试验中选择了零样本生成）。Plurals 既是多元化 AI 的范例，也是具体的系统。 Plurals 库可通过此 https URL 获得并将持续更新。

Title: How Transliterations Improve Crosslingual Alignment

• Authors: Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob Imani, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17326
• Pdf URL: https://arxiv.org/pdf/2409.17326
• Copy Paste: [[2409.17326]] How Transliterations Improve Crosslingual Alignment(https://arxiv.org/abs/2409.17326)
• Keywords: language model
• Abstract: Recent studies have shown that post-aligning multilingual pretrained language models (mPLMs) using alignment objectives on both original and transliterated data can improve crosslingual alignment. This improvement further leads to better crosslingual transfer performance. However, it remains unclear how and why a better crosslingual alignment is achieved, as this technique only involves transliterations, and does not use any parallel data. This paper attempts to explicitly evaluate the crosslingual alignment and identify the key elements in transliteration-based approaches that contribute to better performance. For this, we train multiple models under varying setups for two pairs of related languages: (1) Polish and Ukrainian and (2) Hindi and Urdu. To assess alignment, we define four types of similarities based on sentence representations. Our experiments show that adding transliterations alone improves the overall similarities, even for random sentence pairs. With the help of auxiliary alignment objectives, especially the contrastive objective, the model learns to distinguish matched from random pairs, leading to better alignments. However, we also show that better alignment does not always yield better downstream performance, suggesting that further research is needed to clarify the connection between alignment and performance.
• 摘要：最近的研究表明，使用对齐目标对原始数据和音译数据进行后对齐多语言预训练语言模型 (mPLM) 可以改善跨语言对齐。这种改进进一步提高了跨语言迁移性能。然而，如何以及为何实现更好的跨语言对齐仍不清楚，因为这种技术只涉及音译，而不使用任何并行数据。本文试图明确评估跨语言对齐，并确定基于音译的方法中有助于提高性能的关键要素。为此，我们在不同的设置下为两对相关语言训练了多个模型：(1) 波兰语和乌克兰语和 (2) 印地语和乌尔都语。为了评估对齐，我们根据句子表示定义了四种类型的相似性。我们的实验表明，仅添加音译就可以提高整体相似性，即使是随机句子对也是如此。在辅助对齐目标（尤其是对比目标）的帮助下，模型学会区分匹配对和随机对，从而实现更好的对齐。然而，我们也表明，更好的对齐并不总是产生更好的下游性能，这表明需要进一步研究来阐明对齐和性能之间的联系。

Title: Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM

• Authors: Robin Shing-Hei Yuen, Timothy Tin-Long Tse, Jian Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17353
• Pdf URL: https://arxiv.org/pdf/2409.17353
• Copy Paste: [[2409.17353]] Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM(https://arxiv.org/abs/2409.17353)
• Keywords: llm, chain-of-thought
• Abstract: Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research.
• 摘要：目前基于语音的 LLM 主要在大量 ASR 和 TTS 数据集上进行训练，在与这些领域相关的任务中表现出色。然而，它们处理直接语音对语音对话的能力仍然受到明显限制。这些模型通常依赖于 ASR 到 TTS 的思路链管道，将语音转换为文本进行处理，然后再生成音频响应，这会带来延迟并丢失音频特征。我们提出了一种方法，将 ASR 思路链隐式地内化到语音 LLM 中，增强其原生语音理解能力。我们的方法减少了延迟并提高了模型对语音的原生理解，为更高效、更自然的实时音频交互铺平了道路。我们还发布了一个大规模合成对话数据集，以促进进一步的研究。

Title: Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia

• Authors: Zhejian Zhou, Jiayu Wang, Dahua Lin, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17391
• Pdf URL: https://arxiv.org/pdf/2409.17391
• Copy Paste: [[2409.17391]] Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia(https://arxiv.org/abs/2409.17391)
• Keywords: language model, llm
• Abstract: Though Large Language Models (LLMs) have shown remarkable abilities in mathematics reasoning, they are still struggling with performing numeric operations accurately, such as addition and multiplication. Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance. Currently, there are two representatives: 1) Tokenize into $1$-digit, and 2) Tokenize into $1\sim 3$ digit. The difference is roughly equivalent to using different numeral systems (namely base $10$ or base $10^{3}$). In light of this, we study the scaling behavior of different numeral systems in the context of transformer-based large language models. We empirically show that a base $10$ system is consistently more data-efficient than a base $10^{2}$ or $10^{3}$ system across training data scale, model sizes under from-scratch training settings, while different number systems have very similar fine-tuning performances. We attribute this to higher token frequencies of a base $10$ system. Additionally, we reveal extrapolation behavior patterns on addition and multiplication. We identify that base $100$ and base $1000$ systems struggle on token-level discernment and token-level operations. We also sheds light on the mechanism learnt by the models.
• 摘要：尽管大型语言模型 (LLM) 在数学推理方面表现出了卓越的能力，但它们在准确执行数字运算（例如加法和乘法）方面仍然存在困难。不同的 LLM 可以通过各种方式将数字标记为 token，并影响数字运算性能。目前有两种代表：1）标记为 1 位数字，2）标记为 1\sim 3 位数字。其差异大致相当于使用不同的数字系统（即十进制或十进制）。鉴于此，我们研究了基于 Transformer 的大型语言模型中不同数字系统的缩放行为。我们通过实证表明，在从头开始的训练设置下，十进制系统在训练数据规模、模型大小方面始终比十进制或十进制系统更高效，而不同的数字系统具有非常相似的微调性能。我们将其归因于十进制系统的 token 频率更高。此外，我们还揭示了加法和乘法的外推行为模式。我们发现 100 进制和 1000 进制系统在标记级辨别和标记级操作方面存在困难。我们还阐明了模型所学到的机制。

Title: Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset

• Authors: Konstantinos Skianis, John Pavlopoulos, A. Seza Doğruöz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17397
• Pdf URL: https://arxiv.org/pdf/2409.17397
• Copy Paste: [[2409.17397]] Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset(https://arxiv.org/abs/2409.17397)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) are increasingly integrated into various medical fields, including mental health support systems. However, there is a gap in research regarding the effectiveness of LLMs in non-English mental health support applications. To address this problem, we present a novel multilingual adaptation of widely-used mental health datasets, translated from English into six languages (Greek, Turkish, French, Portuguese, German, and Finnish). This dataset enables a comprehensive evaluation of LLM performance in detecting mental health conditions and assessing their severity across multiple languages. By experimenting with GPT and Llama, we observe considerable variability in performance across languages, despite being evaluated on the same translated dataset. This inconsistency underscores the complexities inherent in multilingual mental health support, where language-specific nuances and mental health data coverage can affect the accuracy of the models. Through comprehensive error analysis, we emphasize the risks of relying exclusively on large language models (LLMs) in medical settings (e.g., their potential to contribute to misdiagnoses). Moreover, our proposed approach offers significant cost savings for multilingual tasks, presenting a major advantage for broad-scale implementation.
• 摘要：大型语言模型 (LLM) 越来越多地被整合到各种医疗领域，包括心理健康支持系统。然而，在非英语心理健康支持应用中，LLM 的有效性研究存在差距。为了解决这个问题，我们提出了一种新颖的多语言改编版，将广泛使用的心理健康数据集从英语翻译成六种语言（希腊语、土耳其语、法语、葡萄牙语、德语和芬兰语）。该数据集可以全面评估 LLM 在检测心理健康状况和评估其严重程度方面在多种语言中的表现。通过对 GPT 和 Llama 进行实验，我们观察到尽管在同一个翻译数据集上进行评估，但不同语言之间的性能存在相当大的差异。这种不一致性凸显了多语言心理健康支持固有的复杂性，其中特定于语言的细微差别和心理健康数据覆盖范围会影响模型的准确性。通过全面的错误分析，我们强调了在医疗环境中完全依赖大型语言模型 (LLM) 的风险（例如，它们可能导致误诊）。此外，我们提出的方法为多语言任务节省了显著的成本，为大规模实施带来了巨大优势。

Title: From Deception to Detection: The Dual Roles of Large Language Models in Fake News

• Authors: Dorsaf Sallami, Yuan-Chen Chang, Esma Aïmeur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17416
• Pdf URL: https://arxiv.org/pdf/2409.17416
• Copy Paste: [[2409.17416]] From Deception to Detection: The Dual Roles of Large Language Models in Fake News(https://arxiv.org/abs/2409.17416)
• Keywords: language model, llm
• Abstract: Fake news poses a significant threat to the integrity of information ecosystems and public trust. The advent of Large Language Models (LLMs) holds considerable promise for transforming the battle against fake news. Generally, LLMs represent a double-edged sword in this struggle. One major concern is that LLMs can be readily used to craft and disseminate misleading information on a large scale. This raises the pressing questions: Can LLMs easily generate biased fake news? Do all LLMs have this capability? Conversely, LLMs offer valuable prospects for countering fake news, thanks to their extensive knowledge of the world and robust reasoning capabilities. This leads to other critical inquiries: Can we use LLMs to detect fake news, and do they outperform typical detection models? In this paper, we aim to address these pivotal questions by exploring the performance of various LLMs. Our objective is to explore the capability of various LLMs in effectively combating fake news, marking this as the first investigation to analyze seven such models. Our results reveal that while some models adhere strictly to safety protocols, refusing to generate biased or misleading content, other models can readily produce fake news across a spectrum of biases. Additionally, our results show that larger models generally exhibit superior detection abilities and that LLM-generated fake news are less likely to be detected than human-written ones. Finally, our findings demonstrate that users can benefit from LLM-generated explanations in identifying fake news.
• 摘要：假新闻对信息生态系统的完整性和公众信任构成了重大威胁。大型语言模型 (LLM) 的出现为改变打击假新闻的斗争带来了巨大的希望。一般来说，LLM 在这场斗争中是一把双刃剑。一个主要问题是，LLM 很容易被用来大规模制作和传播误导性信息。这就提出了一个紧迫的问题：LLM 能轻易产生有偏见的假新闻吗？所有的 LLM 都有这种能力吗？相反，由于 LLM 对世界的广泛了解和强大的推理能力，它们为打击假新闻提供了宝贵的前景。这引出了其他关键问题：我们可以使用 LLM 来检测假新闻吗？它们是否优于典型的检测模型？在本文中，我们旨在通过探索各种 LLM 的性能来解决这些关键问题。我们的目标是探索各种 LLM 有效打击假新闻的能力，这是首次分析七种此类模型的调查。我们的结果表明，虽然有些模型严格遵守安全协议，拒绝生成有偏见或误导性的内容，但其他模型却可以轻易地生成带有各种偏见的假新闻。此外，我们的结果表明，较大的模型通常具有出色的检测能力，而且 LLM 生成的假新闻比人工编写的假新闻更不容易被发现。最后，我们的研究结果表明，用户可以从 LLM 生成的解释中受益，从而识别假新闻。

Title: Pre-Finetuning with Impact Duration Awareness for Stock Movement Prediction

• Authors: Chr-Jr Chiu, Chung-Chi Chen, Hen-Hsen Huang, Hsin-Hsi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17419
• Pdf URL: https://arxiv.org/pdf/2409.17419
• Copy Paste: [[2409.17419]] Pre-Finetuning with Impact Duration Awareness for Stock Movement Prediction(https://arxiv.org/abs/2409.17419)
• Keywords: language model
• Abstract: Understanding the duration of news events' impact on the stock market is crucial for effective time-series forecasting, yet this facet is largely overlooked in current research. This paper addresses this research gap by introducing a novel dataset, the Impact Duration Estimation Dataset (IDED), specifically designed to estimate impact duration based on investor opinions. Our research establishes that pre-finetuning language models with IDED can enhance performance in text-based stock movement predictions. In addition, we juxtapose our proposed pre-finetuning task with sentiment analysis pre-finetuning, further affirming the significance of learning impact duration. Our findings highlight the promise of this novel research direction in stock movement prediction, offering a new avenue for financial forecasting. We also provide the IDED and pre-finetuned language models under the CC BY-NC-SA 4.0 license for academic use, fostering further exploration in this field.
• 摘要：了解新闻事件对股市影响的持续时间对于有效的时间序列预测至关重要，但这一方面在当前的研究中在很大程度上被忽视了。本文通过引入一个新数据集——影响持续时间估计数据集 (IDED) 来解决这一研究空白，该数据集专门用于根据投资者意见估计影响持续时间。我们的研究表明，使用 IDED 预微调语言模型可以提高基于文本的股票走势预测的性能。此外，我们将我们提出的预微调任务与情绪分析预微调并列，进一步肯定了学习影响持续时间的重要性。我们的研究结果凸显了这一新研究方向在股票走势预测中的前景，为金融预测提供了一条新途径。我们还根据 CC BY-NC-SA 4.0 许可证提供 IDED 和预微调语言模型供学术使用，促进该领域的进一步探索。

Title: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

• Authors: Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17422
• Pdf URL: https://arxiv.org/pdf/2409.17422
• Copy Paste: [[2409.17422]] Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction(https://arxiv.org/abs/2409.17422)
• Keywords: language model, llm, long context
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\times$ speedup and 30\% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 在处理长上下文输入方面表现出了卓越的能力，但这是以增加计算资源和延迟为代价的。我们的研究为解决长上下文瓶颈引入了一种新方法，以加速 LLM 推理并减少 GPU 内存消耗。我们的研究表明，LLM 可以在生成查询答案之前识别早期层中的相关标记。利用这一见解，我们提出了一种算法，该算法使用 LLM 的早期层作为过滤器来选择和压缩输入标记，从而显着减少后续处理的上下文长度。与现有技术（例如标准注意和 SnapKV/H2O）相比，我们的方法 GemFilter 在速度和内存效率方面都有显着提高。值得注意的是，与 SOTA 方法相比，它实现了 2.4 倍的加速和 30% 的 GPU 内存使用量减少。在 Needle in a Haystack 任务上的评估表明，GemFilter 明显优于标准注意 SnapKV，并且在 LongBench 挑战中表现出相当的性能。 GemFilter 简单、无需训练，并且广泛适用于不同的 LLM。最重要的是，它允许人类检查所选的输入序列，从而提供可解释性。这些发现不仅为 LLM 部署提供了实际好处，而且还增强了我们对 LLM 内部机制的理解，为 LLM 设计和推理的进一步优化铺平了道路。我们的代码可在 \url{此 https URL} 上找到。

Title: HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows

• Authors: Wenlin Yao, Haitao Mi, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17433
• Pdf URL: https://arxiv.org/pdf/2409.17433
• Copy Paste: [[2409.17433]] HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows(https://arxiv.org/abs/2409.17433)
• Keywords: language model, llm, chain-of-thought
• Abstract: Despite recent advancements in large language models (LLMs), their performance on complex reasoning problems requiring multi-step thinking and combining various skills is still limited. To address this, we propose a novel framework HDFlow for complex reasoning with LLMs that combines fast and slow thinking modes in an adaptive manner. Our approach consists of two key components: 1) a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks and dynamically designs a workflow to assemble specialized LLM or symbolic reasoning tools to solve sub-tasks; 2) Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity. Finally, we propose an easy-to-scale method for automatically synthesizing a large-scale dataset of 27K challenging reasoning problems for complex reasoning and a hybrid thinking tuning method that trains smaller LLMs on this dataset to internalize the fast/slow hybrid reasoning strategies. Experiments on four reasoning benchmark datasets demonstrate that our slow thinking with dynamic workflows significantly outperforms Chain-of-Thought, and hybrid thinking achieves the highest accuracy while providing an effective balance between computational efficiency and performance. Fine-tuning using our hybrid thinking approach also significantly boosts the complex reasoning capabilities of open-source language models. The results showcase the promise of slow thinking, dynamic workflows, and hybrid thinking in expanding the frontier of complex problem-solving with LLMs\footnote{Code and data will be released at \url{this https URL}.}.
• 摘要：尽管大型语言模型 (LLM) 近期取得了进展，但它们在需要多步骤思考和结合各种技能的复杂推理问题上的表现仍然有限。为了解决这个问题，我们提出了一种新颖的框架 HDFlow，用于使用 LLM 进行复杂推理，该框架以自适应方式结合了快速和慢速思维模式。我们的方法由两个关键部分组成：1）一种称为动态工作流的慢速、深思熟虑推理的新方法，它自动将复杂问题分解为更易于管理的子任务，并动态设计工作流以组装专门的 LLM 或符号推理工具来解决子任务；2）混合思维，这是一种根据问题复杂性动态结合快速和慢速思维的通用框架。最后，我们提出了一种易于扩展的方法，用于自动合成 27K 个具有挑战性的复杂推理问题的大规模数据集，以及一种混合思维调整方法，该方法在此数据集上训练较小的 LLM 以内化快速/慢速混合推理策略。在四个推理基准数据集上的实验表明，我们的慢速思维和动态工作流显著优于思维链，而混合思维在计算效率和性能之间实现有效平衡的同时实现了最高的准确率。使用我们的混合思维方法进行微调也显著提升了开源语言模型的复杂推理能力。结果展示了慢速思维、动态工作流和混合思维在利用 LLM 扩展复杂问题解决领域的前景\footnote{代码和数据将在 \url{此 https URL} 上发布。}。

Title: Enhancing Financial Sentiment Analysis with Expert-Designed Hint

• Authors: Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17448
• Pdf URL: https://arxiv.org/pdf/2409.17448
• Copy Paste: [[2409.17448]] Enhancing Financial Sentiment Analysis with Expert-Designed Hint(https://arxiv.org/abs/2409.17448)
• Keywords: language model, llm
• Abstract: This paper investigates the role of expert-designed hint in enhancing sentiment analysis on financial social media posts. We explore the capability of large language models (LLMs) to empathize with writer perspectives and analyze sentiments. Our findings reveal that expert-designed hint, i.e., pointing out the importance of numbers, significantly improve performances across various LLMs, particularly in cases requiring perspective-taking skills. Further analysis on tweets containing different types of numerical data demonstrates that the inclusion of expert-designed hint leads to notable improvements in sentiment analysis performance, especially for tweets with monetary-related numbers. Our findings contribute to the ongoing discussion on the applicability of Theory of Mind in NLP and open new avenues for improving sentiment analysis in financial domains through the strategic use of expert knowledge.
• 摘要：本文探讨了专家设计的提示在增强金融社交媒体帖子情绪分析方面的作用。我们探索了大型语言模型 (LLM) 理解作者观点和分析情绪的能力。我们的研究结果表明，专家设计的提示（即指出数字的重要性）可显著提高各种 LLM 的性能，尤其是在需要换位思考技能的情况下。对包含不同类型数值数据的推文的进一步分析表明，加入专家设计的提示可显著提高情绪分析性能，尤其是对于包含与货币相关的数字的推文。我们的研究结果促进了关于心智理论在 NLP 中的适用性的持续讨论，并通过战略性地使用专家知识为改善金融领域的情绪分析开辟了新途径。

Title: Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models

• Authors: Yuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei Zhu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17455
• Pdf URL: https://arxiv.org/pdf/2409.17455
• Copy Paste: [[2409.17455]] Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models(https://arxiv.org/abs/2409.17455)
• Keywords: language model
• Abstract: Language models (LMs), despite their advances, often depend on spurious correlations, undermining their accuracy and generalizability. This study addresses the overlooked impact of subtler, more complex shortcuts that compromise model reliability beyond oversimplified shortcuts. We introduce a comprehensive benchmark that categorizes shortcuts into occurrence, style, and concept, aiming to explore the nuanced ways in which these shortcuts influence the performance of LMs. Through extensive experiments across traditional LMs, large language models, and state-of-the-art robust models, our research systematically investigates models' resilience and susceptibilities to sophisticated shortcuts. Our benchmark and code can be found at: this https URL.
• 摘要：尽管语言模型 (LM) 取得了进步，但它们通常依赖于虚假相关性，从而损害了其准确性和普遍性。本研究探讨了更微妙、更复杂的捷径所受到的忽视的影响，这些捷径除了过于简单的捷径外，还会损害模型的可靠性。我们引入了一个全面的基准，将捷径分为发生、风格和概念，旨在探索这些捷径影响 LM 性能的细微方式。通过对传统 LM、大型语言模型和最先进的稳健模型进行大量实验，我们的研究系统地研究了模型的弹性和对复杂捷径的敏感性。我们的基准和代码可以在以下位置找到：此 https URL。

Title: Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards

• Authors: Heejin Do, Sangwon Ryu, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17472
• Pdf URL: https://arxiv.org/pdf/2409.17472
• Copy Paste: [[2409.17472]] Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards(https://arxiv.org/abs/2409.17472)
• Keywords: prompt
• Abstract: Recent advances in automated essay scoring (AES) have shifted towards evaluating multiple traits to provide enriched feedback. Like typical AES systems, multi-trait AES employs the quadratic weighted kappa (QWK) to measure agreement with human raters, aligning closely with the rating schema; however, its non-differentiable nature prevents its direct use in neural network training. In this paper, we propose Scoring-aware Multi-reward Reinforcement Learning (SaMRL), which integrates actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation, as RL requires probability distributions; instead, we adopt an autoregressive score generation framework to leverage token generation probabilities for robust multi-trait score predictions. Empirical analyses demonstrate that SaMRL facilitates model training, notably enhancing scoring of previously inferior prompts.
• 摘要：自动作文评分 (AES) 的最新进展已转向评估多种特征以提供丰富的反馈。与典型的 AES 系统一样，多特征 AES 采用二次加权 kappa (QWK) 来衡量与人类评分者的一致性，与评分方案紧密结合；然而，其不可微分性质阻碍了它直接用于神经网络训练。在本文中，我们提出了评分感知多奖励强化学习 (SaMRL)，它通过为多特征 AES 设计基于 QWK 的奖励和均方误差惩罚，将实际评估方案集成到训练过程中。AES 中现有的强化学习 (RL) 应用仅限于分类模型，尽管这会导致性能下降，因为 RL 需要概率分布；相反，我们采用自回归分数生成框架来利用 token 生成概率进行稳健的多特征分数预测。实证分析表明，SaMRL 促进了模型训练，尤其是提高了以前较差提示的评分。

Title: Data Proportion Detection for Optimized Data Management for Large Language Models

• Authors: Hao Liang, Keshi Zhao, Yajie Yang, Bin Cui, Guosheng Dong, Zenan Zhou, Wentao Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17527
• Pdf URL: https://arxiv.org/pdf/2409.17527
• Copy Paste: [[2409.17527]] Data Proportion Detection for Optimized Data Management for Large Language Models(https://arxiv.org/abs/2409.17527)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results. Pre-training data typically combines information from multiple domains. To maximize performance when integrating data from various domains, determining the optimal data proportion is essential. However, state-of-the-art (SOTA) LLMs rarely disclose details about their pre-training data, making it difficult for researchers to identify ideal data proportions. In this paper, we introduce a new topic, \textit{data proportion detection}, which enables the automatic estimation of pre-training data proportions by analyzing the generated outputs of LLMs. We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection. Based on these findings, we offer valuable insights into the challenges and future directions for effective data proportion detection and data management.
• 摘要：大型语言模型 (LLM) 在广泛的任务和领域中都表现出色，而数据准备在实现这些结果方面起着至关重要的作用。预训练数据通常结合了来自多个领域的信息。为了在整合来自各个领域的数据时最大限度地提高性能，确定最佳数据比例至关重要。然而，最先进的 (SOTA) LLM 很少披露有关其预训练数据的详细信息，这使得研究人员很难确定理想的数据比例。在本文中，我们介绍了一个新主题 \textit{数据比例检测}，它通过分析 LLM 的生成输出来自动估计预训练数据比例。我们为数据比例检测提供了严格的理论证明、实用算法和初步实验结果。基于这些发现，我们对有效的数据比例检测和数据管理所面临的挑战和未来方向提供了宝贵的见解。

Title: Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

• Authors: Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Xingyu Wang, Jiaxing Wang, Hailong Yang, Jing Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17539
• Pdf URL: https://arxiv.org/pdf/2409.17539
• Copy Paste: [[2409.17539]] Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models(https://arxiv.org/abs/2409.17539)
• Keywords: language model, llm, prompt, chain-of-thought, tree-of-thought
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information from input context, and utilizes the generated logical information as an additional augmentation to the input prompts, thereby enhancing the capability of logical reasoning. The LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, the LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%; moreover, it improves Chain-of-Thought with Self-Consistency's performance on LogiQA by +5%; additionally, it boosts performance of Tree-of-Thoughts on ProofWriter dataset by +8%.
• 摘要：大型语言模型（LLM）在各种任务上都表现出色，但在复杂的逻辑推理任务中表现并不理想。虽然一些提示方法（如思维链）可以在一定程度上提高LLM的推理能力，但它们存在得出的结论可能与生成的推理链不一致的问题。为了解决这个问题，一些研究采用了命题逻辑的方法来进一步增强LLM的逻辑推理能力。然而，这些方法在提取逻辑表达式时可能存在遗漏，导致逻辑推理过程中的信息丢失，从而产生不正确的结果。为此，我们提出了思维逻辑（LoT）提示，利用命题逻辑从输入上下文中生成扩展的逻辑信息，并将生成的逻辑信息作为输入提示的额外增强，从而增强逻辑推理的能力。LoT与现有的提示方法正交，可以与它们无缝集成。大量实验表明，LoT 在五项逻辑推理任务中显著提升了各种提示方法的性能。具体来说，LoT 使 Chain-of-Thought 在 ReClor 数据集上的性能提升了 +4.35%；此外，它使 Chain-of-Thought with Self-Consistency 在 LogiQA 上的性能提升了 +5%；此外，它使 Tree-of-Thoughts 在 ProofWriter 数据集上的性能提升了 +8%。

Title: DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms

• Authors: Fuqiang Niu, Minghuan Tan, Bowen Zhang, Min Yang, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17588
• Pdf URL: https://arxiv.org/pdf/2409.17588
• Copy Paste: [[2409.17588]] DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms(https://arxiv.org/abs/2409.17588)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Idioms represent a ubiquitous vehicle for conveying sentiments in the realm of everyday discourse, rendering the nuanced analysis of idiom sentiment crucial for a comprehensive understanding of emotional expression within real-world texts. Nevertheless, the existing corpora dedicated to idiom sentiment analysis considerably limit research in text sentiment analysis. In this paper, we propose an innovative approach to automatically expand the sentiment lexicon for idioms, leveraging the capabilities of large language models through the application of Chain-of-Thought prompting. To demonstrate the effectiveness of this approach, we integrate multiple existing resources and construct an emotional idiom lexicon expansion dataset (called EmoIdiomE), which encompasses a comprehensive repository of Chinese and English idioms. Then we designed the Dual Chain-of-Thoughts (DualCoTs) method, which combines insights from linguistics and psycholinguistics, to demonstrate the effectiveness of using large models to automatically expand the sentiment lexicon for idioms. Experiments show that DualCoTs is effective in idioms sentiment lexicon expansion in both Chinese and English. For reproducibility, we will release the data and code upon acceptance.
• 摘要：成语是日常话语中表达情感的普遍工具，细致分析成语情感对于全面理解真实文本中的情感表达至关重要。然而，现有的用于成语情感分析的语料库大大限制了文本情感分析的研究。在本文中，我们提出了一种创新方法来自动扩充成语的情感词典，通过应用思维链提示来利用大型语言模型的功能。为了证明这种方法的有效性，我们整合了多种现有资源，构建了一个情感成语词典扩充数据集 (称为 EmoIdiomE)，其中包含一个全面的中英文成语库。然后，我们设计了双思维链 (DualCoTs) 方法，该方法结合了语言学和心理语言学的见解，以证明使用大型模型自动扩充成语情感词典的有效性。实验表明，DualCoTs 在中英文习语情感词库扩充方面均有效果。为了可重复性，我们将在接受后发布数据和代码。

Title: ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue

• Authors: Zhangpu Li, Changhong Zou, Suxue Ma, Zhicheng Yang, Chen Du, Youbao Tang, Zhenjie Cao, Ning Zhang, Jui-Hsin Lai, Ruei-Sung Lin, Yuan Ni, Xingzhi Sun, Jing Xiao, Kai Zhang, Mei Han
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17610
• Pdf URL: https://arxiv.org/pdf/2409.17610
• Copy Paste: [[2409.17610]] ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue(https://arxiv.org/abs/2409.17610)
• Keywords: language model, llm
• Abstract: The rocketing prosperity of large language models (LLMs) in recent years has boosted the prevalence of vision-language models (VLMs) in the medical sector. In our online medical consultation scenario, a doctor responds to the texts and images provided by a patient in multiple rounds to diagnose her/his health condition, forming a multi-turn multimodal medical dialogue format. Unlike high-quality images captured by professional equipment in traditional medical visual question answering (Med-VQA), the images in our case are taken by patients' mobile phones. These images have poor quality control, with issues such as excessive background elements and the lesion area being significantly off-center, leading to degradation of vision-language alignment in the model training phase. In this paper, we propose ZALM3, a Zero-shot strategy to improve vision-language ALignment in Multi-turn Multimodal Medical dialogue. Since we observe that the preceding text conversations before an image can infer the regions of interest (RoIs) in the image, ZALM3 employs an LLM to summarize the keywords from the preceding context and a visual grounding model to extract the RoIs. The updated images eliminate unnecessary background noise and provide more effective vision-language alignment. To better evaluate our proposed method, we design a new subjective assessment metric for multi-turn unimodal/multimodal medical dialogue to provide a fine-grained performance comparison. Our experiments across three different clinical departments remarkably demonstrate the efficacy of ZALM3 with statistical significance.
• 摘要：近年来，大型语言模型 (LLM) 的蓬勃发展，推动了视觉语言模型 (VLM) 在医疗领域的普及。在我们的在线医疗问诊场景中，医生对患者提供的文本和图片进行多轮回复，以诊断其健康状况，形成多轮多模态医疗对话格式。与传统医学视觉问答 (Med-VQA) 中专业设备采集的高质量图像不同，我们的案例中的图像是由患者的手机拍摄的。这些图像质量控制较差，存在背景元素过多、病变区域明显偏离中心等问题，导致模型训练阶段的视觉语言对齐效果下降。在本文中，我们提出了一种零样本策略 ZALM3，以改善多轮多模态医疗对话中的视觉语言对齐效果。由于我们观察到，图像之前的文本对话可以推断出图像中的感兴趣区域 (RoI)，因此 ZALM3 使用 LLM 从前面的上下文中总结关键字，并使用视觉基础模型提取 RoI。更新后的图像消除了不必要的背景噪音，并提供了更有效的视觉语言对齐。为了更好地评估我们提出的方法，我们设计了一种新的主​​观评估指标，用于多轮单模/多模医疗对话，以提供细粒度的性能比较。我们在三个不同临床部门进行的实验显著证明了 ZALM3 的有效性，具有统计学意义。

Title: T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task

• Authors: Xindi Tong, Yujin Zhu, Shijian Fan, Liang Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17640
• Pdf URL: https://arxiv.org/pdf/2409.17640
• Copy Paste: [[2409.17640]] T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task(https://arxiv.org/abs/2409.17640)
• Keywords: language model, gpt, llm
• Abstract: Long text summarization, gradually being essential for efficiently processing large volumes of information, stays challenging for Large Language Models (LLMs) such as GPT and LLaMA families because of the insufficient open-sourced training datasets and the high requirement of contextual details dealing. To address the issue, we design a novel zero-shot transfer learning framework, abbreviated as T3, to iteratively training a baseline LLM on an assistant task for the target task, where the former should own richer data resources and share structural or semantic similarity with the latter. In practice, T3 is approached to deal with the long text summarization task by utilizing question answering as the assistant task, and further validated its effectiveness on the BBC summary, NarraSum, FairytaleQA, and NLQuAD datasets, with up to nearly 14% improvement in ROUGE, 35% improvement in BLEU, and 16% improvement in Factscore compared to three baseline LLMs, demonstrating its potential for more assistant-target task combinations.
• 摘要：长文本摘要逐渐成为高效处理海量信息的关键任务，然而由于开源训练数据集不足以及对上下文细节处理的要求较高，长文本摘要对于 GPT、LLaMA 系列等大型语言模型 (LLM) 来说仍然具有挑战性。针对该问题，我们设计了一个新颖的零样本迁移学习框架，简称为 T3，用于在目标任务的辅助任务上迭代训练一个基线 LLM，其中前者应拥有更丰富的数据资源并与后者共享结构或语义相似性。在实践中，T3 被用于以问答作为辅助任务来处理长文本摘要任务，并进一步在 BBC 摘要、NarraSum、FairytaleQA 和 NLQuAD 数据集上验证了其有效性，与三个基线 LLM 相比，ROUGE 提高了近 14%，BLEU 提高了 35%，Factscore 提高了 16%，展示了其在更多辅助-目标任务组合中的潜力。

Title: Efficient In-Domain Question Answering for Resource-Constrained Environments

• Authors: Isaac Chung, Phat Vo, Arman Kizilkale, Aaron Reite
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17648
• Pdf URL: https://arxiv.org/pdf/2409.17648
• Copy Paste: [[2409.17648]] Efficient In-Domain Question Answering for Resource-Constrained Environments(https://arxiv.org/abs/2409.17648)
• Keywords: language model, gpt, llm, prompt, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) is a common method for integrating external knowledge into pretrained Large Language Models (LLMs) to enhance accuracy and relevancy in question answering (QA) tasks. However, prompt engineering and resource efficiency remain significant bottlenecks in developing optimal and robust RAG solutions for real-world QA applications. Recent studies have shown success in using fine tuning to address these problems; in particular, Retrieval Augmented Fine Tuning (RAFT) applied to smaller 7B models has demonstrated superior performance compared to RAG setups with much larger models such as GPT-3.5. The combination of RAFT with parameter-efficient fine tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), promises an even more efficient solution, yet remains an unexplored area. In this work, we combine RAFT with LoRA to reduce fine tuning and storage requirements and gain faster inference times while maintaining comparable RAG performance. This results in a more compute-efficient RAFT, or CRAFT, which is particularly useful for knowledge-intensive QA tasks in resource-constrained environments where internet access may be restricted and hardware resources limited.
• 摘要：检索增强生成 (RAG) 是一种将外部知识集成到预训练的大型语言模型 (LLM) 中的常用方法，以提高问答 (QA) 任务的准确性和相关性。然而，在为现实世界的 QA 应用程序开发最佳且强大的 RAG 解决方案时，快速工程和资源效率仍然是重大瓶颈。最近的研究表明，使用微调可以成功解决这些问题；特别是，应用于较小 7B 模型的检索增强微调 (RAFT) 与具有更大模型（例如 GPT-3.5）的 RAG 设置相比表现出了卓越的性能。RAFT 与参数高效微调 (PEFT) 技术（例如低秩自适应 (LoRA)）的结合有望提供更高效的解决方案，但这仍是一个尚未探索的领域。在这项工作中，我们将 RAFT 与 LoRA 相结合，以减少微调和存储要求并获得更快的推理时间，同时保持相当的 RAG 性能。这会产生计算效率更高的 RAFT 或 CRAFT，这对于资源受限环境中的知识密集型 QA 任务特别有用，因为互联网访问可能受到限制且硬件资源有限。

Title: Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT

• Authors: Natthanaphop Isaradech, Andrea Riedel, Wachiranun Sirikul, Markus Kreuzthaler, Stefan Schulz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17683
• Pdf URL: https://arxiv.org/pdf/2409.17683
• Copy Paste: [[2409.17683]] Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT(https://arxiv.org/abs/2409.17683)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Introduction: Medication prescriptions are often in free text and include a mix of two languages, local brand names, and a wide range of idiosyncratic formats and abbreviations. Large language models (LLMs) have shown promising ability to generate text in response to input prompts. We use ChatGPT 3.5 to automatically structure and expand medication statements in discharge summaries and thus make them easier to interpret for people and machines. Methods: Named-entity Recognition (NER) and Text Expansion (EX) are used in a zero- and few-shot setting with different prompt strategies. 100 medication statements were manually annotated and curated. NER performance was measured by using strict and partial matching. For the task EX, two experts interpreted the results by assessing semantic equivalence between original and expanded statements. The model performance was measured by precision, recall, and F1 score. Results: For NER, the best-performing prompt reached an average F1 score of 0.94 in the test set. For EX, the few-shot prompt showed superior performance among other prompts, with an average F1 score of 0.87. Conclusion: Our study demonstrates good performance for NER and EX tasks in free-text medication statements using ChatGPT. Compared to a zero-shot baseline, a few-shot approach prevented the system from hallucinating, which would be unacceptable when processing safety-relevant medication data.
• 摘要：简介：药物处方通常是自由文本，包括两种语言、本地品牌名称以及各种特殊格式和缩写。大型语言模型 (LLM) 已显示出响应输入提示生成文本的良好能力。我们使用 ChatGPT 3.5 自动构造和扩展出院摘要中的药物声明，从而使其更易于人类和机器解释。方法：在零样本和少样本设置中使用命名实体识别 (NER) 和文本扩展 (EX)，并采用不同的提示策略。100 条药物声明经过手动注释和整理。使用严格和部分匹配来衡量 NER 性能。对于任务 EX，两位专家通过评估原始语句和扩展语句之间的语义等价性来解释结果。模型性能通过精度、召回率和 F1 分数来衡量。结果：对于 NER，表现最佳的提示在测试集中的平均 F1 分数达到 0.94。对于 EX，少样本提示比其他提示表现出色，平均 F1 得分为 0.87。结论：我们的研究表明，使用 ChatGPT 在自由文本药物声明中执行 NER 和 EX 任务时表现出色。与零样本基线相比，少样本方法可防止系统产生幻觉，这在处理与安全相关的药物数据时是不可接受的。

Title: MIO: A Foundation Model on Multimodal Tokens

• Authors: Zekun Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17692
• Pdf URL: https://arxiv.org/pdf/2409.17692
• Copy Paste: [[2409.17692]] MIO: A Foundation Model on Multimodal Tokens(https://arxiv.org/abs/2409.17692)
• Keywords: language model, gpt, llm
• Abstract: In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc.
• 摘要：在本文中，我们介绍了 MIO，这是一种基于多模态标记构建的新型基础模型，能够以端到端、自回归的方式理解和生成语音、文本、图像和视频。虽然大型语言模型 (LLM) 和多模态大型语言模型 (MM-LLM) 的出现通过其多功能功能推动了人工智能的发展，但它们仍然缺乏真正的任意对任意的理解和生成。最近，GPT-4o 的发布展示了任意对任意 LLM 在复杂的现实世界任务中的巨大潜力，实现了跨图像、语音和文本的全向输入和输出。但是，它是闭源的，不支持生成多模态交错序列。为了解决这一差距，我们提出了 MIO，它使用因果多模态建模在四种模态的离散标记混合上进行训练。 MIO 经过四个阶段的训练过程：（1）对齐预训练、（2）交错预训练、（3）语音增强预训练和（4）对各种文本、视觉和语音任务进行全面监督微调。我们的实验结果表明，与之前的双模态基线、任意对任意模型基线甚至特定模态基线相比，MIO 表现出了竞争力，在某些情况下甚至更优异的性能。此外，MIO 还展示了其任意对任意功能所固有的高级功能，例如交错视频文本生成、视觉思维链推理、视觉指南生成、指导性图像编辑等。

Title: Integrating Hierarchical Semantic into Iterative Generation Model for Entailment Tree Explanation

• Authors: Qin Wang, Jianzhou Feng, Yiming Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17757
• Pdf URL: https://arxiv.org/pdf/2409.17757
• Copy Paste: [[2409.17757]] Integrating Hierarchical Semantic into Iterative Generation Model for Entailment Tree Explanation(https://arxiv.org/abs/2409.17757)
• Keywords: language model
• Abstract: Manifestly and logically displaying the line of reasoning from evidence to answer is significant to explainable question answering (QA). The entailment tree exhibits the lines structurally, which is different from the self-explanation principle in large-scale language models. Existing methods rarely consider the semantic association of sentences between and within hierarchies within the tree structure, which is prone to apparent mistakes in combinations. In this work, we propose an architecture of integrating the Hierarchical Semantics of sentences under the framework of Controller-Generator (HiSCG) to explain answers. The HiSCG designs a hierarchical mapping between hypotheses and facts, discriminates the facts involved in tree constructions, and optimizes single-step entailments. To the best of our knowledge, We are the first to notice hierarchical semantics of sentences between the same layer and adjacent layers to yield improvements. The proposed method achieves comparable performance on all three settings of the EntailmentBank dataset. The generalization results on two out-of-domain datasets also demonstrate the effectiveness of our method.
• 摘要：清晰、合乎逻辑地展示从证据到答案的推理线索对于可解释问答系统（QA）具有重要意义。蕴涵树以结构化的方式展示这些线索，不同于大规模语言模型中的自解释原则。现有方法很少考虑树结构中层次结构之间和层次结构内部的句子语义关联，这容易在组合中出现明显的错误。在本文中，我们提出了一种在控制器-生成器（HiSCG）框架下集成句子层次语义的架构来解释答案。HiSCG 设计了假设和事实之间的层次映射，区分了树构建中涉及的事实，并优化了单步蕴涵。据我们所知，我们是第一个注意到同一层和相邻层之间句子的层次语义可以带来改进的人。所提出的方法在 EntailmentBank 数据集的所有三个设置上都实现了可比的性能。在两个域外数据集上的泛化结果也证明了我们方法的有效性。

Title: Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

• Authors: Jian Li, Haojing Huang, Yujia Zhang, Pengfei Xu, Xi Chen, Rui Song, Lida Shi, Jingwen Wang, Hao Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17791
• Pdf URL: https://arxiv.org/pdf/2409.17791
• Copy Paste: [[2409.17791]] Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness(https://arxiv.org/abs/2409.17791)
• Keywords: language model, llm
• Abstract: Recently, there has been significant interest in replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs), such as Direct Preference Optimization (DPO) and its variants. These approaches commonly use a binary cross-entropy mechanism on pairwise samples, i.e., minimizing and maximizing the loss based on preferred or dis-preferred responses, respectively. However, while this training strategy omits the reward model, it also overlooks the varying preference degrees within different responses. We hypothesize that this is a key factor hindering LLMs from sufficiently understanding human preferences. To address this problem, we propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss, thereby helping LLMs improve their ability to understand the degree of preference. Extensive experiments are conducted on two widely used datasets of different tasks. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods and significantly boost their performance to achieve state-of-the-art performance. We also conduct detailed analyses to offer comprehensive insights into SPO, which verifies its effectiveness. The code is available at this https URL.
• 摘要：最近，人们对用人类反馈 (RLHF) 方法替换大型语言模型 (LLM) 中的奖励模型产生了浓厚的兴趣，例如直接偏好优化 (DPO) 及其变体。这些方法通常对成对样本使用二元交叉熵机制，即分别根据偏好或不喜欢的响应最小化和最大化损失。然而，虽然这种训练策略省略了奖励模型，但它也忽略了不同响应中不同的偏好程度。我们假设这是阻碍 LLM 充分理解人类偏好的一个关键因素。为了解决这个问题，我们提出了一种新颖的自监督偏好优化 (SPO) 框架，该框架构建了自监督偏好程度损失与对齐损失相结合，从而帮助 LLM 提高理解偏好程度的能力。在两个广泛使用的不同任务数据集上进行了广泛的实验。结果表明，SPO 可以与现有的偏好优化方法无缝集成，并显著提高其性能以实现最先进的性能。我们还进行了详细的分析，以提供对 SPO 的全面见解，从而验证了其有效性。代码可在此 https URL 上获取。

Title: Inference-Time Language Model Alignment via Integrated Value Guidance

• Authors: Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, Yu Qiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17819
• Pdf URL: https://arxiv.org/pdf/2409.17819
• Copy Paste: [[2409.17819]] Inference-Time Language Model Alignment via Integrated Value Guidance(https://arxiv.org/abs/2409.17819)
• Keywords: language model, gpt
• Abstract: Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce $\textit{Integrated Value Guidance}$ (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from $\texttt{gpt2}$-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against $\texttt{gpt-4-turbo}$ (e.g., $19.51\% \rightarrow 26.51\%$ for $\texttt{Mistral-7B-Instruct-v0.2}$ and $25.58\% \rightarrow 33.75\%$ for $\texttt{Mixtral-8x7B-Instruct-v0.1}$ with Tulu guidance).
• 摘要：大型语言模型通常会进行微调以符合人类偏好，但调整大型模型需要大量计算且非常复杂。在这项工作中，我们引入了 $\textit{集成价值指导}$ (IVG)，这种方法使用隐式和显式价值函数分别在 token 和块级别指导语言模型解码，从而有效地在推理时对齐大型语言模型。这种方法避免了直接微调的复杂性，并且优于传统方法。从经验上讲，我们证明了 IVG 在各种任务中的多功能性。在受控情绪生成和摘要任务中，我们的方法使用基于 $\texttt{gpt2}$ 的价值函数的推理时间指导显著改善了大型模型的对齐。此外，在更具挑战性的指令跟踪基准 AlpacaEval 2.0 中，我们表明，专门调整的值函数和现成的值函数都极大地提高了大型模型对抗 $\texttt{gpt-4-turbo}$ 的长度控制胜率（例如，在 Tulu 指导下，$\texttt{Mistral-7B-Instruct-v0.2}$ 为 $19.51\% \rightarrow 26.51\%$，$\texttt{Mixtral-8x7B-Instruct-v0.1}$ 为 $25.58\% \rightarrow 33.75\%$）。

Title: BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text

• Authors: Siyan Wang, Bradford Levy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17827
• Pdf URL: https://arxiv.org/pdf/2409.17827
• Copy Paste: [[2409.17827]] BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text(https://arxiv.org/abs/2409.17827)
• Keywords: language model, llm
• Abstract: Many of the recent breakthroughs in language modeling have resulted from scaling effectively the same model architecture to larger datasets. In this vein, recent work has highlighted performance gains from increasing training dataset size and quality, suggesting a need for novel sources of large-scale datasets. In this work, we introduce BeanCounter, a public dataset consisting of more than 159B tokens extracted from businesses' disclosures. We show that this data is indeed novel: less than 0.1% of BeanCounter appears in Common Crawl-based datasets and it is an order of magnitude larger than datasets relying on similar sources. Given the data's provenance, we hypothesize that BeanCounter is comparatively more factual and less toxic than web-based datasets. Exploring this hypothesis, we find that many demographic identities occur with similar prevalence in BeanCounter but with significantly less toxic context relative to other datasets. To demonstrate the utility of BeanCounter, we evaluate and compare two LLMs continually pre-trained on BeanCounter with their base models. We find an 18-33% reduction in toxic generation and improved performance within the finance domain for the continually pretrained models. Collectively, our work suggests that BeanCounter is a novel source of low-toxicity and high-quality domain-specific data with sufficient scale to train multi-billion parameter LLMs.
• 摘要：语言建模领域的许多最新突破都源于将相同的模型架构有效地扩展到更大的数据集。在这方面，最近的研究强调了通过增加训练数据集的大小和质量来提高性能，这表明需要新的大规模数据集来源。在这项工作中，我们引入了 BeanCounter，这是一个公共数据集，包含从企业披露中提取的超过 1590 亿个标记。我们表明这些数据确实是新颖的：不到 0.1% 的 BeanCounter 出现在基于 Common Crawl 的数据集中，并且比依赖类似来源的数据集大一个数量级。考虑到数据的来源，我们假设 BeanCounter 比基于​​网络的数据集更真实，毒性更小。探索这一假设后，我们发现许多人口统计身份在 BeanCounter 中出现的频率相似，但与其他数据集相比，毒性明显较小。为了证明 BeanCounter 的实用性，我们评估并比较了两个在 BeanCounter 上持续预训练的 LLM 及其基本模型。我们发现，对于持续预训练的模型，毒性生成减少了 18-33%，金融领域的性能也有所提高。总的来说，我们的工作表明，BeanCounter 是一种新型的低毒性和高质量领域特定数据来源，其规模足以训练数十亿参数的 LLM。

Title: PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification

• Authors: Tianfang Xie, Tianjing Li, Wei Zhu, Wei Han, Yi Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17834
• Pdf URL: https://arxiv.org/pdf/2409.17834
• Copy Paste: [[2409.17834]] PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification(https://arxiv.org/abs/2409.17834)
• Keywords: language model, llm, prompt
• Abstract: Due to their substantial sizes, large language models (LLMs) are typically deployed within a single-backbone multi-tenant framework. In this setup, a single instance of an LLM backbone must cater to multiple users or tasks through the application of various parameter-efficient fine-tuning (PEFT) models. Despite the availability of numerous effective PEFT techniques such as LoRA, there remains a need for a PEFT approach that achieves both high efficiency during inference and competitive performance on downstream tasks. In this research, we introduce a new and straightforward PEFT methodology named \underline{P}rompt D\underline{E}pen\underline{D}ent \underline{R}epresentation M\underline{O}dification (PEDRO). The proposed method involves integrating a lightweight vector generator into each Transformer layer, which generates vectors contingent upon the input prompts. These vectors then modify the hidden representations created by the LLM through a dot product operation, thereby influencing the semantic output and generated content of the model. Extensive experimentation across a variety of tasks indicates that: (a) PEDRO surpasses recent PEFT benchmarks when using a similar number of tunable parameters. (b) Under the single-backbone multi-tenant deployment model, PEDRO exhibits superior efficiency compared to LoRA, indicating significant industrial potential.
• 摘要：由于规模庞大，大型语言模型 (LLM) 通常部署在单主干多租户框架内。在这种设置中，LLM 主干的单个实例必须通过应用各种参数高效微调 (PEFT) 模型来满足多个用户或任务的需求。尽管有许多有效的 PEFT 技术（如 LoRA）可用，但仍然需要一种既能在推理过程中实现高效率，又能在下游任务中实现竞争性能的 PEFT 方法。在本研究中，我们介绍了一种新的、简单的 PEFT 方法，名为 \underline{P}rompt D\underline{E}pen\underline{D}ent \underline{R}epresentation M\underline{O}dification (PEDRO)。所提出的方法涉及将轻量级向量生成器集成到每个 Transformer 层中，该生成器根据输入提示生成向量。然后，这些向量通过点积运算修改 LLM 创建的隐藏表示，从而影响模型的语义输出和生成内容。在各种任务中开展的大量实验表明：（a）使用相似数量的可调参数时，PEDRO 超越了最近的 PEFT 基准。（b）在单主干多租户部署模型下，PEDRO 与 LoRA 相比表现出更高的效率，表明其具有巨大的工业潜力。

Title: EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

• Authors: Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17892
• Pdf URL: https://arxiv.org/pdf/2409.17892
• Copy Paste: [[2409.17892]] EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models(https://arxiv.org/abs/2409.17892)
• Keywords: language model
• Abstract: In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks and PolyWrite, an open-ended generation benchmark developed in this study. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability.
• 摘要：在本研究中，我们引入了 EMMA-500，这是一种大规模多语言模型，在 546 种语言的文本上进行了持续训练，旨在提高多语言性能，重点是提高资源匮乏语言的语言覆盖率。为了促进持续的预训练，我们编制了 MaLA 语料库，这是一个全面的多语言数据集，其中包含来自不同领域的精选数据集。利用这个语料库，我们对 Llama 2 7B 模型进行了广泛的持续预训练，最终得到了 EMMA-500，它在一系列基准测试中表现出了强劲的性能，包括一套全面的多语言任务和本研究开发的开放式生成基准 PolyWrite。我们的结果强调了持续预训练在扩展大型语言模型的语言容量方面的有效性，特别是对于代表性不足的语言，在跨语言迁移、任务泛化和语言适应性方面表现出显著的提升。

Title: Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

• Authors: Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis, Eric Xing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17912
• Pdf URL: https://arxiv.org/pdf/2409.17912
• Copy Paste: [[2409.17912]] Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect(https://arxiv.org/abs/2409.17912)
• Keywords: language model, gpt, llm, chat
• Abstract: We introduce Atlas-Chat, the first-ever collection of large language models specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also known as Darija, we construct our instruction dataset by consolidating existing Darija language resources, creating novel datasets both manually and synthetically, and translating English instructions with stringent quality control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit superior ability in following Darija instructions and performing standard NLP tasks. Notably, our models outperform both state-of-the-art and Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13% performance boost over a larger 13B model on DarijaMMLU, in our newly introduced evaluation suite for Darija covering both discriminative and generative tasks. Furthermore, we perform an experimental analysis of various fine-tuning strategies and base model choices to determine optimal configurations. All our resources are publicly accessible, and we believe our work offers comprehensive design methodologies of instruction-tuning for low-resource language variants, which are often neglected in favor of data-rich languages by contemporary LLMs.
• 摘要：我们推出了 Atlas-Chat，这是有史以来第一个专门为阿拉伯方言开发的大型语言模型集合。我们专注于摩洛哥阿拉伯语（也称为 Darija），通过整合现有的 Darija 语言资源、手动和合成创建新数据集以及以严格的质量控制翻译英语指令来构建我们的指令数据集。在数据集上经过微调的 Atlas-Chat-9B 和 2B 模型在遵循 Darija 指令和执行标准 NLP 任务方面表现出卓越的能力。值得注意的是，我们的模型优于最先进的和阿拉伯语专业的 LLM，如 LLaMa、Jais 和 AceGPT，例如，在我们新推出的 Darija 评估套件中，在 DarijaMMLU 上比更大的 13B 模型实现了 13% 的性能提升，该套件涵盖了判别任务和生成任务。此外，我们对各种微调策略和基本模型选择进行了实验分析，以确定最佳配置。我们所有的资源都是公开的，我们相信我们的工作提供了针对低资源语言变体的指令调整的综合设计方法，而当代的 LLM 往往忽视这些方法，而青睐数据丰富的语言。

Title: Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion

• Authors: Hengrui Gu, Kaixiong Zhou, Yili Wang, Ruobing Wang, Xin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17928
• Pdf URL: https://arxiv.org/pdf/2409.17928
• Copy Paste: [[2409.17928]] Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion(https://arxiv.org/abs/2409.17928)
• Keywords: prompt
• Abstract: During pre-training, the Text-to-Image (T2I) diffusion models encode factual knowledge into their parameters. These parameterized facts enable realistic image generation, but they may become obsolete over time, thereby misrepresenting the current state of the world. Knowledge editing techniques aim to update model knowledge in a targeted way. However, facing the dual challenges posed by inadequate editing datasets and unreliable evaluation criterion, the development of T2I knowledge editing encounter difficulties in effectively generalizing injected knowledge. In this work, we design a T2I knowledge editing framework by comprehensively spanning on three phases: First, we curate a dataset \textbf{CAKE}, comprising paraphrase and multi-object test, to enable more fine-grained assessment on knowledge generalization. Second, we propose a novel criterion, \textbf{adaptive CLIP threshold}, to effectively filter out false successful images under the current criterion and achieve reliable editing evaluation. Finally, we introduce \textbf{MPE}, a simple but effective approach for T2I knowledge editing. Instead of tuning parameters, MPE precisely recognizes and edits the outdated part of the conditioning text-prompt to accommodate the up-to-date knowledge. A straightforward implementation of MPE (Based on in-context learning) exhibits better overall performance than previous model editors. We hope these efforts can further promote faithful evaluation of T2I knowledge editing methods.
• 摘要：在预训练期间，文本到图像 (T2I) 传播模型将事实知识编码到其参数中。这些参数化的事实使得生成逼真的图像成为可能，但它们可能会随着时间的推移而过时，从而歪曲世界的现状。知识编辑技术旨在有针对性地更新模型知识。然而，面对编辑数据集不足和评估标准不可靠带来的双重挑战，T2I 知识编辑的开发在有效概括注入的知识方面遇到了困难。在这项工作中，我们全面设计了一个 T2I 知识编辑框架，涵盖三个阶段：首先，我们整理一个数据集 \textbf{CAKE}，包括释义和多对象测试，以便对知识概括进行更细粒度的评估。其次，我们提出了一个新标准 \textbf{自适应 CLIP 阈值}，以有效滤除当前标准下的虚假成功图像并实现可靠的编辑评估。最后，我们介绍了 \textbf{MPE}，这是一种简单但有效的 T2I 知识编辑方法。MPE 无需调整参数，而是精确识别和编辑条件文本提示的过时部分以适应最新知识。MPE 的直接实现（基于上下文学习）比以前的模型编辑器表现出更好的整体性能。我们希望这些努力可以进一步促进对 T2I 知识编辑方法的忠实评估。

Title: The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification

• Authors: Andreas Waldis, Joel Birrer, Anne Lauscher, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17929
• Pdf URL: https://arxiv.org/pdf/2409.17929
• Copy Paste: [[2409.17929]] The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification(https://arxiv.org/abs/2409.17929)
• Keywords: language model
• Abstract: Gender-fair language, an evolving German linguistic variation, fosters inclusion by addressing all genders or using neutral forms. Nevertheless, there is a significant lack of resources to assess the impact of this linguistic shift on classification using language models (LMs), which are probably not trained on such variations. To address this gap, we present Lou, the first dataset featuring high-quality reformulations for German text classification covering seven tasks, like stance detection and toxicity classification. Evaluating 16 mono- and multi-lingual LMs on Lou shows that gender-fair language substantially impacts predictions by flipping labels, reducing certainty, and altering attention patterns. However, existing evaluations remain valid, as LM rankings of original and reformulated instances do not significantly differ. While we offer initial insights on the effect on German text classification, the findings likely apply to other languages, as consistent patterns were observed in multi-lingual and English LMs.
• 摘要：性别平等语言是一种不断发展的德语语言变体，它通过解决所有性别或使用中性形式来促进包容性。然而，目前仍缺乏资源来评估这种语言转变对使用语言模型 (LM) 进行分类的影响，因为语言模型可能没有针对此类变体进行训练。为了解决这一差距，我们推出了 Lou，这是第一个包含高质量重构的德语文本分类数据集，涵盖立场检测和毒性分类等七项任务。在 Lou 上评估 16 个单语和多语 LM 表明，性别平等语言通过翻转标签、降低确定性和改变注意力模式对预测产生重大影响。然而，现有的评估仍然有效，因为原始实例和重构实例的 LM 排名并没有显着差异。虽然我们对德语文本分类的影响提供了初步见解，但这些发现可能适用于其他语言，因为在多语和英语 LM 中观察到了一致的模式。

Title: Predicting Anchored Text from Translation Memories for Machine Translation Using Deep Learning Methods

• Authors: Richard Yue, John E. Ortega
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17939
• Pdf URL: https://arxiv.org/pdf/2409.17939
• Copy Paste: [[2409.17939]] Predicting Anchored Text from Translation Memories for Machine Translation Using Deep Learning Methods(https://arxiv.org/abs/2409.17939)
• Keywords: gpt, chat
• Abstract: Translation memories (TMs) are the backbone for professional translation tools called computer-aided translation (CAT) tools. In order to perform a translation using a CAT tool, a translator uses the TM to gather translations similar to the desired segment to translate (s'). Many CAT tools offer a fuzzy-match algorithm to locate segments (s) in the TM that are close in distance to s'. After locating two similar segments, the CAT tool will present parallel segments (s, t) that contain one segment in the source language along with its translation in the target language. Additionally, CAT tools contain fuzzy-match repair (FMR) techniques that will automatically use the parallel segments from the TM to create new TM entries containing a modified version of the original with the idea in mind that it will be the translation of s'. Most FMR techniques use machine translation as a way of "repairing" those words that have to be modified. In this article, we show that for a large part of those words which are anchored, we can use other techniques that are based on machine learning approaches such as Word2Vec. BERT, and even ChatGPT. Specifically, we show that for anchored words that follow the continuous bag-of-words (CBOW) paradigm, Word2Vec, BERT, and GPT-4 can be used to achieve similar and, for some cases, better results than neural machine translation for translating anchored words from French to English.
• 摘要：翻译记忆库 (TM) 是专业翻译工具（称为计算机辅助翻译 (CAT) 工具）的支柱。为了使用 CAT 工具进行翻译，翻译人员使用 TM 收集与要翻译的片段 (s') 相似的翻译。许多 CAT 工具提供模糊匹配算法来定位 TM 中与 s' 距离较近的片段 (s)。在找到两个相似的片段后，CAT 工具将显示包含源语言片段的平行片段 (s, t) 及其目标语言翻译。此外，CAT 工具包含模糊匹配修复 (FMR) 技术，该技术将自动使用 TM 中的平行片段来创建新的 TM 条目，其中包含原始修改版本，并考虑到这将是 s' 的翻译。大多数 FMR 技术使用机器翻译来“修复”那些必须修改的单词。在本文中，我们展示了对于大部分锚定词，我们可以使用基于机器学习方法的其他技术，例如 Word2Vec、BERT 甚至 ChatGPT。具体来说，我们展示了对于遵循连续词袋 (CBOW) 范式的锚定词，Word2Vec、BERT 和 GPT-4 可用于实现与神经机器翻译类似的结果，在某些情况下，甚至比将锚定词从法语翻译为英语的结果更好。

Title: On Translating Technical Terminology: A Translation Workflow for Machine-Translated Acronyms

• Authors: Richard Yue, John E. Ortega, Kenneth Ward Church
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17943
• Pdf URL: https://arxiv.org/pdf/2409.17943
• Copy Paste: [[2409.17943]] On Translating Technical Terminology: A Translation Workflow for Machine-Translated Acronyms(https://arxiv.org/abs/2409.17943)
• Keywords: language model
• Abstract: The typical workflow for a professional translator to translate a document from its source language (SL) to a target language (TL) is not always focused on what many language models in natural language processing (NLP) do - predict the next word in a series of words. While high-resource languages like English and French are reported to achieve near human parity using common metrics for measurement such as BLEU and COMET, we find that an important step is being missed: the translation of technical terms, specifically acronyms. Some state-of-the art machine translation systems like Google Translate which are publicly available can be erroneous when dealing with acronyms - as much as 50% in our findings. This article addresses acronym disambiguation for MT systems by proposing an additional step to the SL-TL (FR-EN) translation workflow where we first offer a new acronym corpus for public consumption and then experiment with a search-based thresholding algorithm that achieves nearly 10% increase when compared to Google Translate and OpusMT.
• 摘要：专业翻译人员将文档从源语言 (SL) 翻译成目标语言 (TL) 的典型工作流程并不总是专注于许多自然语言处理 (NLP) 中的语言模型所做的事情 - 预测一系列单词中的下一个单词。据报道，使用 BLEU 和 COMET 等常用测量指标，英语和法语等高资源语言可以达到接近人类的水平，但我们发现缺少了一个重要步骤：技术术语的翻译，特别是首字母缩略词。一些最先进的机器翻译系统（如谷歌翻译）在处理首字母缩略词时可能会出错 - 我们发现错误率高达 50%。本文通过为 SL-TL（FR-EN）翻译工作流程提出一个额外步骤来解决机器翻译系统的首字母缩略词消歧问题，我们首先提供一个新的首字母缩略词语料库供公众使用，然后试验基于搜索的阈值算法，与谷歌翻译和 OpusMT 相比，该算法实现了近 10% 的提升。

Title: The Hard Positive Truth about Vision-Language Compositionality

• Authors: Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, Ranjay Krishna
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17958
• Pdf URL: https://arxiv.org/pdf/2409.17958
• Copy Paste: [[2409.17958]] The Hard Positive Truth about Vision-Language Compositionality(https://arxiv.org/abs/2409.17958)
• Keywords: language model
• Abstract: Several benchmarks have concluded that our best vision-language models (e.g., CLIP) are lacking in compositionality. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. In response, a surge of recent proposals show improvements by finetuning CLIP with distractors as hard negatives. Our investigations reveal that these improvements have, in fact, been significantly overstated -- because existing benchmarks do not probe whether finetuned vision-language models remain invariant to hard positives. By curating an evaluation dataset with 112,382 hard negatives and hard positives, we uncover that including hard positives decreases CLIP's performance by 12.9%, while humans perform effortlessly at 99%. CLIP finetuned with hard negatives results in an even larger decrease, up to 38.7%. With this finding, we then produce a 1,775,259 image-text training set with both hard negative and hard positive captions. By training with both, we see improvements on existing benchmarks while simultaneously improving performance on hard positives, indicating a more robust improvement in compositionality. Our work suggests the need for future research to rigorously test and improve CLIP's understanding of semantic relationships between related "positive" concepts.
• 摘要：一些基准测试已经得出结论，我们最好的视觉语言模型（例如 CLIP）缺乏组合性。给定一张图片，这些基准测试会探测模型在一系列组合干扰项中识别其相关标题的能力。作为回应，最近大量的提案显示，通过将干扰项作为硬性否定项对 CLIP 进行微调可以带来改进。我们的调查显示，这些改进实际上被大大夸大了——因为现有的基准测试并没有探测经过微调的视觉语言模型是否对硬性肯定项保持不变。通过整理包含 112,382 个硬性否定项和硬性肯定项的评估数据集，我们发现包括硬性肯定项会使 CLIP 的性能降低 12.9%，而人类可以轻松达到 99%。使用硬性否定项进行微调的 CLIP 会导致性能下降幅度更大，高达 38.7%。根据这一发现，我们随后制作了一个包含硬性否定项和硬性肯定项字幕的 1,775,259 个图像文本训练集。通过同时使用这两种方法进行训练，我们看到现有基准的改进，同时提高了硬正例的性能，这表明组合性得到了更强劲的改善。我们的工作表明，未来的研究需要严格测试和改进 CLIP 对相关“正例”概念之间语义关系的理解。

Title: BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search

• Authors: Linzhuang Sun, Hao Liang, Wentao Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17972
• Pdf URL: https://arxiv.org/pdf/2409.17972
• Copy Paste: [[2409.17972]] BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search(https://arxiv.org/abs/2409.17972)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have exhibited exceptional performance across a broad range of tasks and domains. However, they still encounter difficulties in solving mathematical problems due to the rigorous and logical nature of mathematics. Previous studies have employed techniques such as supervised fine-tuning (SFT), prompt engineering, and search-based methods to improve the mathematical problem-solving abilities of LLMs. Despite these efforts, their performance remains suboptimal and demands substantial computational resources. To address this issue, we propose a novel approach, BEATS, to enhance mathematical problem-solving abilities. Our method leverages newly designed prompts that guide the model to iteratively rewrite, advance by one step, and generate answers based on previous steps. Additionally, we introduce a new back-verification technique that uses LLMs to validate the correctness of the generated answers. Furthermore, we employ a pruning tree search to optimize search time while achieving strong performance. Notably, our method improves Qwen2-7b-Instruct's score from 36.94 to 61.52, outperforming GPT4's 42.5 on the MATH benchmark.
• 摘要：大型语言模型 (LLM) 在广泛的任务和领域中表现出色。然而，由于数学的严谨性和逻辑性，它们在解决数学问题时仍然遇到困难。先前的研究采用了监督微调 (SFT)、提示工程和基于搜索的方法等技术来提高 LLM 的数学问题解决能力。尽管做出了这些努力，但它们的性能仍然不是最优的，并且需要大量的计算资源。为了解决这个问题，我们提出了一种新方法 BEATS，以增强数学问题解决能力。我们的方法利用新设计的提示来指导模型迭代重写、前进一步并根据先前的步骤生成答案。此外，我们引入了一种新的反向验证技术，该技术使用 LLM 来验证生成的答案的正确性。此外，我们采用剪枝树搜索来优化搜索时间，同时实现强大的性能。值得注意的是，我们的方法将 Qwen2-7b-Instruct 的得分从 36.94 提高到 61.52，在 MATH 基准上优于 GPT4 的 42.5。

Title: Multilingual Evaluation of Long Context Retrieval and Reasoning

• Authors: Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel, Russell Scheinberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18006
• Pdf URL: https://arxiv.org/pdf/2409.18006
• Copy Paste: [[2409.18006]] Multilingual Evaluation of Long Context Retrieval and Reasoning(https://arxiv.org/abs/2409.18006)
• Keywords: language model, gpt, llm, long context
• Abstract: Recent large language models (LLMs) demonstrate impressive capabilities in handling long contexts, some exhibiting near-perfect recall on synthetic retrieval tasks. However, these evaluations have mainly focused on English text and involved a single target sentence within lengthy contexts. Our work investigates how LLM performance generalizes to multilingual settings with multiple hidden target sentences. We comprehensively evaluate several long-context LLMs on retrieval and reasoning tasks across five languages: English, Vietnamese, Indonesian, Swahili, and Somali. These languages share the Latin script but belong to distinct language families and resource levels. Our analysis reveals a significant performance gap between languages. The best-performing models such as Gemini-1.5 and GPT-4o, achieve around 96% accuracy in English to around 36% in Somali with a single target sentence. However, this accuracy drops to 40% in English and 0% in Somali when dealing with three target sentences. Our findings highlight the challenges long-context LLMs face when processing longer contexts, an increase in the number of target sentences, or languages of lower resource levels.
• 摘要：最近的大型语言模型 (LLM) 在处理长上下文方面表现出令人印象深刻的能力，一些模型在合成检索任务中表现出近乎完美的回忆能力。然而，这些评估主要集中在英语文本上，涉及长上下文中的单个目标句子。我们的工作研究了 LLM 性能如何推广到具有多个隐藏目标句子的多语言设置。我们全面评估了几种长上下文 LLM 在检索和推理任务中的应用，涉及五种语言：英语、越南语、印尼语、斯瓦希里语和索马里语。这些语言共享拉丁文字，但属于不同的语系和资源水平。我们的分析揭示了语言之间存在显著的性能差距。表现最佳的模型，例如 Gemini-1.5 和 GPT-4o，在单个目标句子的情况下，英语的准确率约为 96%，索马里语的准确率约为 36%。然而，当处理三个目标句子时，英语的准确率下降到 40%，索马里语的准确率下降到 0%。我们的研究结果强调了长上下文 LLM 在处理较长上下文、目标句子数量的增加或资源水平较低的语言时面临的挑战。

Title: DARE: Diverse Visual Question Answering with Robustness Evaluation

• Authors: Hannah Sterz, Jonas Pfeiffer, Ivan Vulić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18023
• Pdf URL: https://arxiv.org/pdf/2409.18023
• Copy Paste: [[2409.18023]] DARE: Diverse Visual Question Answering with Robustness Evaluation(https://arxiv.org/abs/2409.18023)
• Keywords: language model, gpt, prompt
• Abstract: Vision Language Models (VLMs) extend remarkable capabilities of text-only large language models and vision-only models, and are able to learn from and process multi-modal vision-text input. While modern VLMs perform well on a number of standard image classification and image-text matching tasks, they still struggle with a number of crucial vision-language (VL) reasoning abilities such as counting and spatial reasoning. Moreover, while they might be very brittle to small variations in instructions and/or evaluation protocols, existing benchmarks fail to evaluate their robustness (or rather the lack of it). In order to couple challenging VL scenarios with comprehensive robustness evaluation, we introduce DARE, Diverse Visual Question Answering with Robustness Evaluation, a carefully created and curated multiple-choice VQA benchmark. DARE evaluates VLM performance on five diverse categories and includes four robustness-oriented evaluations based on the variations of: prompts, the subsets of answer options, the output format and the number of correct answers. Among a spectrum of other findings, we report that state-of-the-art VLMs still struggle with questions in most categories and are unable to consistently deliver their peak performance across the tested robustness evaluations. The worst case performance across the subsets of options is up to 34% below the performance in the standard case. The robustness of the open-source VLMs such as LLaVA 1.6 and Idefics2 cannot match the closed-source models such as GPT-4 and Gemini, but even the latter remain very brittle to different variations.
• 摘要：视觉语言模型 (VLM) 扩展了纯文本大型语言模型和纯视觉模型的卓越功能，并且能够从多模态视觉文本输入中学习和处理。虽然现代 VLM 在许多标准图像分类和图像文本匹配任务上表现良好，但它们仍然难以掌握许多关键的视觉语言 (VL) 推理能力，例如计数和空间推理。此外，虽然它们可能对指令和/或评估协议的细微变化非常敏感，但现有基准无法评估它们的稳健性（或者说缺乏稳健性）。为了将具有挑战性的 VL 场景与全面的稳健性评估结合起来，我们引入了 DARE，即具有稳健性评估的多样化视觉问答，这是一个精心创建和策划的多项选择 VQA 基准。DARE 在五个不同的类别上评估 VLM 性能，并包括四个基于以下变化的稳健性导向评估：提示、答案选项的子集、输出格式和正确答案的数量。除了其他一系列发现之外，我们还报告称，最先进的 VLM 仍然难以解决大多数类别的问题，并且无法在测试的稳健性评估中始终保持最佳性能。在选项子集的最差情况下，性能比标准情况下的性能低 34%。LLaVA 1.6 和 Idefics2 等开源 VLM 的稳健性无法与 GPT-4 和 Gemini 等闭源模型相媲美，但即使是后者，在不同变体下也仍然非常脆弱。

Title: Open-World Evaluation for Retrieving Diverse Perspectives

• Authors: Hung-Ting Chen, Eunsol Choi
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.18110
• Pdf URL: https://arxiv.org/pdf/2409.18110
• Copy Paste: [[2409.18110]] Open-World Evaluation for Retrieving Diverse Perspectives(https://arxiv.org/abs/2409.18110)
• Keywords: language model, gpt, chat
• Abstract: We study retrieving a set of documents that covers various perspectives on a complex and contentious question (e.g., will ChatGPT do more harm than good?). We curate a Benchmark for Retrieval Diversity for Subjective questions (BERDS), where each example consists of a question and diverse perspectives associated with the question, sourced from survey questions and debate websites. On this data, retrievers paired with a corpus are evaluated to surface a document set that contains diverse perspectives. Our framing diverges from most retrieval tasks in that document relevancy cannot be decided by simple string matches to references. Instead, we build a language model based automatic evaluator that decides whether each retrieved document contains a perspective. This allows us to evaluate the performance of three different types of corpus (Wikipedia, web snapshot, and corpus constructed on the fly with retrieved pages from the search engine) paired with retrievers. Retrieving diverse documents remains challenging, with the outputs from existing retrievers covering all perspectives on only 33.74% of the examples. We further study the impact of query expansion and diversity-focused reranking approaches and analyze retriever sycophancy. Together, we lay the foundation for future studies in retrieval diversity handling complex queries.
• 摘要：我们研究检索一组涵盖复杂且有争议的问题的各种观点的文档（例如，ChatGPT 弊大于利吗？）。我们制定了主观问题检索多样性基准 (BERDS)，其中每个示例都包含一个问题和与该问题相关的各种观点，这些观点来自调查问题和辩论网站。根据这些数据，对与语料库配对的检索器进行评估，以显示包含不同观点的文档集。我们的框架与大多数检索任务不同，因为文档相关性不能通过与参考文献的简单字符串匹配来决定。相反，我们构建了一个基于语言模型的自动评估器，它可以决定每个检索到的文档是否包含一个观点。这使我们能够评估三种不同类型的语料库（维基百科、网络快照和使用从搜索引擎检索到的页面动态构建的语料库）与检索器配对的性能。检索多样化文档仍然具有挑战性，现有检索器的输出仅涵盖 33.74% 的示例的所有视角。我们进一步研究查询扩展和以多样性为重点的重新排序方法的影响，并分析检索器的谄媚行为。我们共同为未来处理复杂查询的检索多样性研究奠定了基础。