2025-01-14

Title: Enhancing AI Safety Through the Fusion of Low Rank Adapters

• Authors: Satya Swaroop Gudipudi, Sreeram Vipparla, Harpreet Singh, Shashwat Goel, Ponnurangam Kumaraguru
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06208
• Pdf URL: https://arxiv.org/pdf/2501.06208
• Copy Paste: [[2501.06208]] Enhancing AI Safety Through the Fusion of Low Rank Adapters(https://arxiv.org/abs/2501.06208)
• Keywords: language model, llm, prompt
• Abstract: Instruction fine-tuning of large language models (LLMs) is a powerful method for improving task-specific performance, but it can inadvertently lead to a phenomenon where models generate harmful responses when faced with malicious prompts. In this paper, we explore Low-Rank Adapter Fusion (LoRA) as a means to mitigate these risks while preserving the model's ability to handle diverse instructions effectively. Through an extensive comparative analysis against established baselines using recognized benchmark datasets, we demonstrate a 42\% reduction in the harmfulness rate by leveraging LoRA fusion between a task adapter and a safety adapter, the latter of which is specifically trained on our safety dataset. However, we also observe exaggerated safety behaviour, where the model rejects safe prompts that closely resemble unsafe ones
• 摘要：大型语言模型 (LLM) 的指令微调是提高特定任务性能的有效方法，但它可能会无意中导致模型在面对恶意提示时产生有害响应的现象。在本文中，我们探索了低秩适配器融合 (LoRA) 作为一种减轻这些风险的方法，同时保留了模型有效处理各种指令的能力。通过使用公认的基准数据集对既定基线进行广泛的比较分析，我们证明，通过利用任务适配器和安全适配器之间的 LoRA 融合，有害率降低了 42%，后者是专门针对我们的安全数据集进行训练的。然而，我们也观察到夸大的安全行为，其中模型拒绝与不安全提示非常相似的安全提示

Title: FLAME: Financial Large-Language Model Assessment and Metrics Evaluation

• Authors: Jiayu Guo, Yu Guo, Martha Li, Songtao Tan
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2501.06211
• Pdf URL: https://arxiv.org/pdf/2501.06211
• Copy Paste: [[2501.06211]] FLAME: Financial Large-Language Model Assessment and Metrics Evaluation(https://arxiv.org/abs/2501.06211)
• Keywords: language model, gpt, llm
• Abstract: LLMs have revolutionized NLP and demonstrated potential across diverse domains. More and more financial LLMs have been introduced for finance-specific tasks, yet comprehensively assessing their value is still challenging. In this paper, we introduce FLAME, a comprehensive financial LLMs evaluation system in Chinese, which includes two core evaluation benchmarks: FLAME-Cer and FLAME-Sce. FLAME-Cer covers 14 types of authoritative financial certifications, including CPA, CFA, and FRM, with a total of approximately 16,000 carefully selected questions. All questions have been manually reviewed to ensure accuracy and representativeness. FLAME-Sce consists of 10 primary core financial business scenarios, 21 secondary financial business scenarios, and a comprehensive evaluation set of nearly 100 tertiary financial application tasks. We evaluate 6 representative LLMs, including GPT-4o, GLM-4, ERNIE-4.0, Qwen2.5, XuanYuan3, and the latest Baichuan4-Finance, revealing Baichuan4-Finance excels other LLMs in most tasks. By establishing a comprehensive and professional evaluation system, FLAME facilitates the advancement of financial LLMs in Chinese contexts. Instructions for participating in the evaluation are available on GitHub: this https URL.
• 摘要：LLM 彻底改变了 NLP，并在各个领域展现出巨大潜力。越来越多的金融 LLM 被用于金融特定任务，但全面评估其价值仍然具有挑战性。在本文中，我们介绍了一个全面的中文金融 LLM 评估体系 FLAME，它包含两个核心评估基准：FLAME-Cer 和 FLAME-Sce。FLAME-Cer 涵盖 CPA、CFA 和 FRM 等 14 类权威金融认证，共计约 16,000 道精心挑选的题目。所有题目都经过人工审核以确保准确性和代表性。FLAME-Sce 包含 10 个一级核心金融业务场景、21 个二级金融业务场景以及近 100 个三级金融应用任务的综合评估集。我们评估了 6 个具有代表性的 LLM，包括 GPT-4o、GLM-4、ERNIE-4.0、Qwen2.5、XuanYuan3 和最新的 Baichuan4-Finance，结果表明 Baichuan4-Finance 在大多数任务中都优于其他 LLM。通过建立全面而专业的评估体系，FLAME 促进了中国背景下金融 LLM 的进步。参与评估的说明可在 GitHub 上找到：此 https URL。

Title: Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words

• Authors: Gouki Minegishi, Hiroki Furuta, Yusuke Iwasawa, Yutaka Matsuo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06254
• Pdf URL: https://arxiv.org/pdf/2501.06254
• Copy Paste: [[2501.06254]] Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words(https://arxiv.org/abs/2501.06254)
• Keywords: language model, llm
• Abstract: Sparse autoencoders (SAEs) have gained a lot of attention as a promising tool to improve the interpretability of large language models (LLMs) by mapping the complex superposition of polysemantic neurons into monosemantic features and composing a sparse dictionary of words. However, traditional performance metrics like Mean Squared Error and L0 sparsity ignore the evaluation of the semantic representational power of SAEs -- whether they can acquire interpretable monosemantic features while preserving the semantic relationship of words. For instance, it is not obvious whether a learned sparse feature could distinguish different meanings in one word. In this paper, we propose a suite of evaluations for SAEs to analyze the quality of monosemantic features by focusing on polysemous words. Our findings reveal that SAEs developed to improve the MSE-L0 Pareto frontier may confuse interpretability, which does not necessarily enhance the extraction of monosemantic features. The analysis of SAEs with polysemous words can also figure out the internal mechanism of LLMs; deeper layers and the Attention module contribute to distinguishing polysemy in a word. Our semantics focused evaluation offers new insights into the polysemy and the existing SAE objective and contributes to the development of more practical SAEs.
• 摘要：稀疏自编码器 (SAE) 通过将多义神经元的复杂叠加映射到单义特征并组成稀疏词典，作为一种很有前途的工具，它获得了广泛关注。然而，传统的性能指标如均方误差和 L0 稀疏性忽略了对 SAE 的语义表示能力的评估——它们是否能在保留单词的语义关系的同时获得可解释的单义特征。例如，学习到的稀疏特征是否可以区分一个单词中的不同含义并不明显。在本文中，我们提出了一套 SAE 评估方法，通过关注多义词来分析单义特征的质量。我们的研究结果表明，为改进 MSE-L0 帕累托前沿而开发的 SAE 可能会混淆可解释性，这并不一定会增强单义特征的提取。对含有多义词的 SAE 的分析也可以阐明 LLM 的内部机制；更深的层次和注意力模块有助于区分单词中的多义性。我们的语义重点评估为多义性和现有的 SAE 目标提供了新的见解，并有助于开发更实用的 SAE。

Title: What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning

• Authors: Jelena Bratulić, Sudhanshu Mittal, Christian Rupprecht, Thomas Brox
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06256
• Pdf URL: https://arxiv.org/pdf/2501.06256
• Copy Paste: [[2501.06256]] What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning(https://arxiv.org/abs/2501.06256)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance in various tasks, including In-Context Learning (ICL), where the model performs new tasks by conditioning solely on the examples provided in the context, without updating the model's weights. While prior research has explored the roles of pretraining data and model architecture, the key mechanism behind ICL remains unclear. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL. To disambiguate these factors, we conduct a study with a controlled dataset and data sequences using a deep autoregressive model. We show that conceptual repetitions in the data sequences are crucial for ICL, more so than previously indicated training data properties like burstiness or long-tail distribution. Conceptual repetitions could refer to $n$-gram repetitions in textual data or exact image copies in image sequence data. Such repetitions also offer other previously overlooked benefits such as reduced transiency in ICL performance. Furthermore, we show that the emergence of ICL depends on balancing the in-weight learning objective with the in-context solving ability during training.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出令人印象深刻的性能，包括上下文学习 (ICL)，其中模型仅根据上下文中提供的示例执行新任务，而不更新模型的权重。虽然先前的研究已经探索了预训练数据和模型架构的作用，但 ICL 背后的关键机制仍不清楚。在这项工作中，我们系统地揭示了 LLM 中支持 ICL 出现的属性。为了消除这些因素的歧义，我们使用深度自回归模型对受控数据集和数据序列进行了研究。我们表明，数据序列中的概念重复对于 ICL 至关重要，比之前指出的训练数据属性（如突发性或长尾分布）更重要。概念重复可以指文本数据中的 $n$-gram 重复或图像序列数据中的精确图像副本。这种重复还提供了其他以前被忽视的好处，例如减少 ICL 性能的瞬时性。此外，我们表明 ICL 的出现取决于在训练期间平衡权重学习目标和上下文解决能力。

Title: AgoraSpeech: A multi-annotated comprehensive dataset of political discourse through the lens of humans and AI

• Authors: Pavlos Sermpezis, Stelios Karamanidis, Eva Paraschou, Ilias Dimitriadis, Sofia Yfantidou, Filitsa-Ioanna Kouskouveli, Thanasis Troboukis, Kelly Kiki, Antonis Galanopoulos, Athena Vakali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06265
• Pdf URL: https://arxiv.org/pdf/2501.06265
• Copy Paste: [[2501.06265]] AgoraSpeech: A multi-annotated comprehensive dataset of political discourse through the lens of humans and AI(https://arxiv.org/abs/2501.06265)
• Keywords: language model, gpt, llm, chat
• Abstract: Political discourse datasets are important for gaining political insights, analyzing communication strategies or social science phenomena. Although numerous political discourse corpora exist, comprehensive, high-quality, annotated datasets are scarce. This is largely due to the substantial manual effort, multidisciplinarity, and expertise required for the nuanced annotation of rhetorical strategies and ideological contexts. In this paper, we present AgoraSpeech, a meticulously curated, high-quality dataset of 171 political speeches from six parties during the Greek national elections in 2023. The dataset includes annotations (per paragraph) for six natural language processing (NLP) tasks: text classification, topic identification, sentiment analysis, named entity recognition, polarization and populism detection. A two-step annotation was employed, starting with ChatGPT-generated annotations and followed by exhaustive human-in-the-loop validation. The dataset was initially used in a case study to provide insights during the pre-election period. However, it has general applicability by serving as a rich source of information for political and social scientists, journalists, or data scientists, while it can be used for benchmarking and fine-tuning NLP and large language models (LLMs).
• 摘要：政治话语数据集对于获得政治见解、分析传播策略或社会科学现象非常重要。尽管存在大量政治话语语料库，但全面、高质量、带注释的数据集却很少。这主要是因为对修辞策略和意识形态背景的细致注释需要大量的人工、多学科和专业知识。在本文中，我们介绍了 AgoraSpeech，这是一个精心策划的高质量数据集，包含 2023 年希腊全国大选期间来自六个政党的 171 篇政治演讲。该数据集包括六项自然语言处理 (NLP) 任务的注释（每段）：文本分类、主题识别、情绪分析、命名实体识别、两极分化和民粹主义检测。采用了两步注释，首先是 ChatGPT 生成的注释，然后是详尽的人机验证。该数据集最初用于案例研究，以在选举前提供见解。然而，它具有普遍的适用性，可以作为政治和社会科学家、记者或数据科学家的丰富信息来源，同时可以用于对 NLP 和大型语言模型 (LLM) 进行基准测试和微调。

Title: Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain

• Authors: Jing Guo, Nan Li, Ming Xu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.06277
• Pdf URL: https://arxiv.org/pdf/2501.06277
• Copy Paste: [[2501.06277]] Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain(https://arxiv.org/abs/2501.06277)
• Keywords: language model
• Abstract: Generative AI holds significant potential for ecological and environmental applications such as monitoring, data analysis, education, and policy support. However, its effectiveness is limited by the lack of a unified evaluation framework. To address this, we present the Environmental Large Language model Evaluation (ELLE) question answer (QA) dataset, the first benchmark designed to assess large language models and their applications in ecological and environmental sciences. The ELLE dataset includes 1,130 question answer pairs across 16 environmental topics, categorized by domain, difficulty, and type. This comprehensive dataset standardizes performance assessments in these fields, enabling consistent and objective comparisons of generative AI performance. By providing a dedicated evaluation tool, ELLE dataset promotes the development and application of generative AI technologies for sustainable environmental outcomes. The dataset and code are available at this https URL and this https URL.
• 摘要：生成式人工智能在生态和环境应用方面具有巨大潜力，例如监测、数据分析、教育和政策支持。然而，由于缺乏统一的评估框架，其有效性受到限制。为了解决这个问题，我们提出了环境大型语言模型评估 (ELLE) 问答 (QA) 数据集，这是第一个旨在评估大型语言模型及其在生态和环境科学中的应用的基准。ELLE 数据集包括 16 个环境主题的 1,130 个问答对，按领域、难度和类型分类。这个全面的数据集标准化了这些领域的绩效评估，从而能够对生成式人工智能的性能进行一致和客观的比较。通过提供专用的评估工具，ELLE 数据集促进了生成式人工智能技术的开发和应用，以实现可持续的环境成果。数据集和代码可在此 https URL 和此 https URL 上找到。

Title: MinMo: A Multimodal Large Language Model for Seamless Voice Interaction

• Authors: Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, Jinren Zhou
• Subjects: cs.CL, cs.AI, cs.HC, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.06282
• Pdf URL: https://arxiv.org/pdf/2501.06282
• Copy Paste: [[2501.06282]] MinMo: A Multimodal Large Language Model for Seamless Voice Interaction(https://arxiv.org/abs/2501.06282)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence lengths and insufficient pre-training. Aligned models maintain text LLM capabilities but are often limited by small datasets and a narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is this https URL, and the code and models will be released soon.
• 摘要：大型语言模型 (LLM) 和多模态语音文本模型的最新进展为无缝语音交互奠定了基础，可实现实时、自然和类似人类的对话。之前的语音交互模型分为原生模型和对齐模型。原生模型将语音和文本处理集成在一个框架中，但存在序列长度不同和预训练不足等问题。对齐模型保留了文本 LLM 功能，但通常受到数据集较小和仅关注语音任务的限制。在这项工作中，我们推出了 MinMo，这是一个多模态大型语言模型，具有大约 8B 个参数，可实现无缝语音交互。我们解决了之前对齐的多模态模型的主要局限性。我们通过语音到文本对齐、文本到语音对齐、语音到语音对齐和双工交互对齐等多个阶段对 MinMo 进行训练，训练基于 140 万小时的多样化语音数据和广泛的语音任务。经过多阶段训练后，MinMo 在语音理解和生成方面在各种基准测试中都取得了最佳表现，同时保持了文本 LLM 的功能，并且还支持全双工对话，即用户和系统之间同时进行双向通信。此外，我们提出了一种新颖而简单的语音解码器，其语音生成性能优于之前的模型。MinMo 增强的指令跟随能力支持根据用户指令控制语音生成，包括情绪、方言和语速等各种细微差别，以及模仿特定的声音。对于 MinMo 来说，语音到文本的延迟约为 100 毫秒，全双工延迟在理论上约为 600 毫秒，在实践中约为 800 毫秒。MinMo 项目网页是这个 https URL，代码和模型将很快发布。

Title: Bactrainus: Optimizing Large Language Models for Multi-hop Complex Question Answering Tasks

• Authors: Iman Barati, Arash Ghafouri, Behrouz Minaei-Bidgoli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06286
• Pdf URL: https://arxiv.org/pdf/2501.06286
• Copy Paste: [[2501.06286]] Bactrainus: Optimizing Large Language Models for Multi-hop Complex Question Answering Tasks(https://arxiv.org/abs/2501.06286)
• Keywords: language model, llm
• Abstract: In recent years, the use of large language models (LLMs) has significantly increased, and these models have demonstrated remarkable performance in a variety of general language tasks. However, the evaluation of their performance in domain-specific tasks, particularly those requiring deep natural language understanding, has received less attention. In this research, we evaluate the ability of large language models in performing domain-specific tasks, focusing on the multi-hop question answering (MHQA) problem using the HotpotQA dataset. This task, due to its requirement for reasoning and combining information from multiple textual sources, serves as a challenging benchmark for assessing the language comprehension capabilities of these models. To tackle this problem, we have designed a two-stage selector-reader architecture, where each stage utilizes an independent LLM. In addition, methods such as Chain of Thought (CoT) and question decomposition have been employed to investigate their impact on improving the model's performance. The results of the study show that the integration of large language models with these techniques can lead to up to a 4% improvement in F1 score for finding answers, providing evidence of the models' ability to handle domain-specific tasks and their understanding of complex language.
• 摘要：近年来，大型语言模型 (LLM) 的使用显著增加，这些模型在各种通用语言任务中表现出色。然而，对它们在特定领域任务中的表现的评估，尤其是那些需要深度自然语言理解的任务，却没有受到太多关注。在本研究中，我们评估了大型语言模型执行特定领域任务的能力，重点关注使用 HotpotQA 数据集的多跳问答 (MHQA) 问题。由于这项任务需要推理和结合来自多个文本源的信息，因此成为评估这些模型的语言理解能力的具有挑战性的基准。为了解决这个问题，我们设计了一个两阶段的选择器-读取器架构，其中每个阶段都使用一个独立的 LLM。此外，我们还采用了诸如思路链 (CoT) 和问题分解等方法来研究它们对提高模型性能的影响。研究结果表明，大型语言模型与这些技术的结合可使寻找答案的 F1 分数提高高达 4%，证明了模型处理特定领域任务的能力及其对复杂语言的理解。

Title: Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages

• Authors: Jannik Brinkmann, Chris Wendler, Christian Bartelt, Aaron Mueller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06346
• Pdf URL: https://arxiv.org/pdf/2501.06346
• Copy Paste: [[2501.06346]] Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages(https://arxiv.org/abs/2501.06346)
• Keywords: language model, llm
• Abstract: Human bilinguals often use similar brain regions to process multiple languages, depending on when they learned their second language and their proficiency. In large language models (LLMs), how are multiple languages learned and encoded? In this work, we explore the extent to which LLMs share representations of morphosyntactic concepts such as grammatical number, gender, and tense across languages. We train sparse autoencoders on Llama-3-8B and Aya-23-8B, and demonstrate that abstract grammatical concepts are often encoded in feature directions shared across many languages. We use causal interventions to verify the multilingual nature of these representations; specifically, we show that ablating only multilingual features decreases classifier performance to near-chance across languages. We then use these features to precisely modify model behavior in a machine translation task; this demonstrates both the generality and selectivity of these feature's roles in the network. Our findings suggest that even models trained predominantly on English data can develop robust, cross-lingual abstractions of morphosyntactic concepts.
• 摘要：人类双语者通常使用相似的大脑区域来处理多种语言，这取决于他们学习第二语言的时间和熟练程度。在大型语言模型 (LLM) 中，多种语言是如何学习和编码的？在这项工作中，我们探索了 LLM 在多大程度上共享形态句法概念（例如语法数字、性别和时态）的表示。我们在 Llama-3-8B 和 Aya-23-8B 上训练稀疏自动编码器，并证明抽象语法概念通常以多种语言共享的特征方向进行编码。我们使用因果干预来验证这些表示的多语言性质；具体而言，我们表明仅消除多语言特征会将分类器性能降低到跨语言的近似偶然水平。然后，我们使用这些特征来精确修改机器翻译任务中的模型行为；这证明了这些特征在网络中的作用的通用性和选择性。我们的研究结果表明，即使主要基于英语数据训练的模型也可以开发出稳健的、跨语言的形态句法概念抽象。

Title: Gender-Neutral Large Language Models for Medical Applications: Reducing Bias in PubMed Abstracts

• Authors: Elizabeth Schaefer, Kirk Roberts
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.06365
• Pdf URL: https://arxiv.org/pdf/2501.06365
• Copy Paste: [[2501.06365]] Gender-Neutral Large Language Models for Medical Applications: Reducing Bias in PubMed Abstracts(https://arxiv.org/abs/2501.06365)
• Keywords: language model, llm
• Abstract: This paper presents a pipeline for mitigating gender bias in large language models (LLMs) used in medical literature by neutralizing gendered occupational pronouns. A dataset of 379,000 PubMed abstracts from 1965-1980 was processed to identify and modify pronouns tied to professions. We developed a BERT-based model, ``Modern Occupational Bias Elimination with Refined Training,'' or ``MOBERT,'' trained on these neutralized abstracts, and compared its performance with ``1965Bert,'' trained on the original dataset. MOBERT achieved a 70\% inclusive replacement rate, while 1965Bert reached only 4\%. A further analysis of MOBERT revealed that pronoun replacement accuracy correlated with the frequency of occupational terms in the training data. We propose expanding the dataset and refining the pipeline to improve performance and ensure more equitable language modeling in medical applications.
• 摘要：本文介绍了一种通过中和性别职业代词来减轻医学文献中使用的大型语言模型 (LLM) 中的性别偏见的流程。我们处理了 1965 年至 1980 年的 379,000 篇 PubMed 摘要数据集，以识别和修改与职业相关的代词。我们开发了一个基于 BERT 的模型，即“现代职业偏见消除与精炼训练”或“MOBERT”，对这些中和后的摘要进行了训练，并将其性能与在原始数据集上训练的“1965Bert”进行了比较。MOBERT 实现了 70% 的包容性替换率，而 1965Bert 仅达到 4%。对 MOBERT 的进一步分析表明，代词替换准确性与训练数据中职业术语的频率相关。我们建议扩展数据集并改进流程，以提高性能并确保在医学应用中实现更公平的语言建模。

Title: AFRIDOC-MT: Document-level MT Corpus for African Languages

• Authors: Jesujoba O. Alabi, Israel Abebe Azime, Miaoran Zhang, Cristina España-Bonet, Rachel Bawden, Dawei Zhu, David Ifeoluwa Adelani, Clement Oyeleke Odoje, Idris Akinade, Iffat Maab, Davis David, Shamsuddeen Hassan Muhammad, Neo Putini, David O. Ademuyiwa, Andrew Caines, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06374
• Pdf URL: https://arxiv.org/pdf/2501.06374
• Copy Paste: [[2501.06374]] AFRIDOC-MT: Document-level MT Corpus for African Languages(https://arxiv.org/abs/2501.06374)
• Keywords: language model, gpt, llm
• Abstract: This paper introduces AFRIDOC-MT, a document-level multi-parallel translation dataset covering English and five African languages: Amharic, Hausa, Swahili, Yorùbá, and Zulu. The dataset comprises 334 health and 271 information technology news documents, all human-translated from English to these languages. We conduct document-level translation benchmark experiments by evaluating neural machine translation (NMT) models and large language models (LLMs) for translations between English and these languages, at both the sentence and pseudo-document levels. These outputs are realigned to form complete documents for evaluation. Our results indicate that NLLB-200 achieved the best average performance among the standard NMT models, while GPT-4o outperformed general-purpose LLMs. Fine-tuning selected models led to substantial performance gains, but models trained on sentences struggled to generalize effectively to longer documents. Furthermore, our analysis reveals that some LLMs exhibit issues such as under-generation, repetition of words or phrases, and off-target translations, especially for African languages.
• 摘要：本文介绍了 AFRIDOC-MT，这是一个文档级多并行翻译数据集，涵盖英语和五种非洲语言：阿姆哈拉语、豪萨语、斯瓦希里语、约鲁巴语和祖鲁语。该数据集包含 334 份健康和 271 份信息技术新闻文档，全部由人工从英语翻译成这些语言。我们通过评估神经机器翻译 (NMT) 模型和大型语言模型 (LLM) 在句子和伪文档级别对英语和这些语言之间的翻译进行文档级翻译基准测试实验。这些输出被重新调整以形成完整的文档以供评估。我们的结果表明，NLLB-200 在标准 NMT 模型中取得了最佳平均性能，而 GPT-4o 则优于通用 LLM。对选定的模型进行微调可带来显着的性能提升，但对句子进行训练的模型难以有效地推广到较长的文档。此外，我们的分析表明，一些 LLM 表现出诸如生成不足、单词或短语重复以及偏离目标的翻译等问题，尤其是对于非洲语言。

Title: Dynamics of "Spontaneous" Topic Changes in Next Token Prediction with Self-Attention

• Authors: Mumin Jia, Jairo Diaz-Rodriguez
• Subjects: cs.CL, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.06382
• Pdf URL: https://arxiv.org/pdf/2501.06382
• Copy Paste: [[2501.06382]] Dynamics of "Spontaneous" Topic Changes in Next Token Prediction with Self-Attention(https://arxiv.org/abs/2501.06382)
• Keywords: language model
• Abstract: Human cognition can spontaneously shift conversation topics, often triggered by emotional or contextual signals. In contrast, self-attention-based language models depend on structured statistical cues from input tokens for next-token prediction, lacking this spontaneity. Motivated by this distinction, we investigate the factors that influence the next-token prediction to change the topic of the input sequence. We define concepts of topic continuity, ambiguous sequences, and change of topic, based on defining a topic as a set of token priority graphs (TPGs). Using a simplified single-layer self-attention architecture, we derive analytical characterizations of topic changes. Specifically, we demonstrate that (1) the model maintains the priority order of tokens related to the input topic, (2) a topic change occurs only if lower-priority tokens outnumber all higher-priority tokens of the input topic, and (3) unlike human cognition, longer context lengths and overlapping topics reduce the likelihood of spontaneous redirection. These insights highlight differences between human cognition and self-attention-based models in navigating topic changes and underscore the challenges in designing conversational AI capable of handling "spontaneous" conversations more naturally. To our knowledge, this is the first work to address these questions in such close relation to human conversation and thought.
• 摘要：人类认知可以自发地转移对话主题，通常是由情绪或上下文信号触发的。相比之下，基于自注意力的语言模型依赖于来自输入标记的结构化统计线索来预测下一个标记，缺乏这种自发性。受此区别的启发，我们研究了影响下一个标记预测以改变输入序列主题的因素。我们定义了主题连续性、模糊序列和主题变化的概念，基于将主题定义为一组标记优先级图 (TPG)。使用简化的单层自注意力架构，我们推导出主题变化的分析特征。具体而言，我们证明 (1) 该模型保持与输入主题相关的标记的优先级顺序，(2) 仅当输入主题中优先级较低的标记数量超过所有优先级较高的标记时，才会发生主题变化，以及 (3) 与人类认知不同，较长的上下文长度和重叠主题会降低自发重定向的可能性。这些见解凸显了人类认知和基于自我注意力的模型在处理话题变化方面的差异，并强调了设计能够更自然地处理“自发”对话的对话式人工智能所面临的挑战。据我们所知，这是第一篇以与人类对话和思维如此密切的关系来解决这些问题的研究成果。

Title: Tensor Product Attention Is All You Need

• Authors: Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, Andrew Chi-Chih Yao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06425
• Pdf URL: https://arxiv.org/pdf/2501.06425
• Copy Paste: [[2501.06425]] Tensor Product Attention Is All You Need(https://arxiv.org/abs/2501.06425)
• Keywords: language model
• Abstract: Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory overhead during inference. In this paper, we propose Tensor Product Attention (TPA), a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly, significantly shrinking KV cache size at inference time. By factorizing these representations into contextual low-rank components (contextual factorization) and seamlessly integrating with RoPE, TPA achieves improved model quality alongside memory efficiency. Based on TPA, we introduce the Tensor ProducT ATTenTion Transformer (T6), a new model architecture for sequence modeling. Through extensive empirical evaluation of language modeling tasks, we demonstrate that T6 exceeds the performance of standard Transformer baselines including MHA, MQA, GQA, and MLA across various metrics, including perplexity and a range of renowned evaluation benchmarks. Notably, TPAs memory efficiency enables the processing of significantly longer sequences under fixed resource constraints, addressing a critical scalability challenge in modern language models. The code is available at this https URL.
• 摘要：扩展语言模型以处理更长的输入序列通常需要大型键值 (KV) 缓存，从而导致推理期间产生大量内存开销。在本文中，我们提出了张量积注意力 (TPA)，这是一种新颖的注意力机制，它使用张量分解来紧凑地表示查询、键和值，从而显著缩小推理时的 KV 缓存大小。通过将这些表示分解为上下文低秩组件（上下文分解）并与 RoPE 无缝集成，TPA 实现了模型质量的提高和内存效率的提高。基于 TPA，我们引入了 Tensor ProducT ATTenTion Transformer (T6)，这是一种用于序列建模的新模型架构。通过对语言建模任务进行广泛的实证评估，我们证明 T6 在各种指标（包括困惑度和一系列知名评估基准）上的表现都超过了包括 MHA、MQA、GQA 和 MLA 在内的标准 Transformer 基线。值得注意的是，TPA 的内存效率使得能够在固定资源限制下处理更长的序列，从而解决了现代语言模型中的关键可扩展性挑战。代码可在此 https URL 上获取。

Title: Synthetic Feature Augmentation Improves Generalization Performance of Language Models

• Authors: Ashok Choudhary, Cornelius Thiels, Hojjat Salehinejad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06434
• Pdf URL: https://arxiv.org/pdf/2501.06434
• Copy Paste: [[2501.06434]] Synthetic Feature Augmentation Improves Generalization Performance of Language Models(https://arxiv.org/abs/2501.06434)
• Keywords: language model, llm
• Abstract: Training and fine-tuning deep learning models, especially large language models (LLMs), on limited and imbalanced datasets poses substantial challenges. These issues often result in poor generalization, where models overfit to dominant classes and underperform on minority classes, leading to biased predictions and reduced robustness in real-world applications. To overcome these challenges, we propose augmenting features in the embedding space by generating synthetic samples using a range of techniques. By upsampling underrepresented classes, this method improves model performance and alleviates data imbalance. We validate the effectiveness of this approach across multiple open-source text classification benchmarks, demonstrating its potential to enhance model robustness and generalization in imbalanced data scenarios.
• 摘要：在有限且不平衡的数据集上训练和微调深度学习模型（尤其是大型语言模型 (LLM)）带来了巨大的挑战。这些问题通常会导致泛化能力较差，即模型对主要类别过度拟合，而对少数类别表现不佳，从而导致预测有偏差，并在实际应用中降低稳健性。为了克服这些挑战，我们建议通过使用一系列技术生成合成样本来增强嵌入空间中的特征。通过对代表性不足的类别进行上采样，此方法可以提高模型性能并缓解数据不平衡。我们在多个开源文本分类基准上验证了此方法的有效性，证明了它在数据不平衡场景中增强模型稳健性和泛化的潜力。

Title: O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoning

• Authors: Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, Xiaofan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06458
• Pdf URL: https://arxiv.org/pdf/2501.06458
• Copy Paste: [[2501.06458]] O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoning(https://arxiv.org/abs/2501.06458)
• Keywords: language model, llm
• Abstract: Building upon our previous investigations of O1 replication (Part 1: Journey Learning [Qin et al., 2024] and Part 2: Distillation [Huang et al., 2024]), this work explores the potential of inference-time scaling in large language models (LLMs) for medical reasoning tasks, ranging from diagnostic decision-making to treatment planning. Through extensive experiments on medical benchmarks of varying complexity (MedQA, Medbullets, and JAMA Clinical Challenges), our investigation reveals several key insights: (1) Increasing inference time does lead to improved performance. With a modest training set of 500 samples, our model yields substantial performance improvements of 6%-11%. (2) Task complexity directly correlates with the required length of reasoning chains, confirming the necessity of extended thought processes for challenging problems. (3) The differential diagnoses generated by our model adhere to the principles of the hypothetico-deductive method, producing a list of potential conditions that may explain a patient's symptoms and systematically narrowing these possibilities by evaluating the evidence. These findings demonstrate the promising synergy between inference-time scaling and journey learning in advancing LLMs' real-world clinical reasoning capabilities.
• 摘要：在我们之前对 O1 复制的研究（第 1 部分：旅程学习 [Qin et al., 2024] 和第 2 部分：提炼 [Huang et al., 2024]）的基础上，这项工作探索了大型语言模型 (LLM) 中推理时间扩展的潜力，可用于从诊断决策到治疗计划的医学推理任务。通过对不同复杂程度的医学基准（MedQA、Medbullets 和 JAMA Clinical Challenges）进行大量实验，我们的研究揭示了几个关键见解：（1）增加推理时间确实可以提高性能。使用 500 个样本的适度训练集，我们的模型可显着提高 6%-11% 的性能。（2）任务复杂性与推理链所需的长度直接相关，证实了扩展思维过程对于具有挑战性的问题的必要性。 (3) 我们的模型生成的鉴别诊断遵循假设演绎法的原则，列出可能解释患者症状的潜在病症，并通过评估证据系统地缩小这些可能性。这些发现表明，推理时间扩展和旅程学习在提高 LLM 的真实世界临床推理能力方面具有良好的协同作用。

Title: MedCT: A Clinical Terminology Graph for Generative AI Applications in Healthcare

• Authors: Ye Chen, Dongdong Huang, Haoyun Xu, Cong Fu, Lin Sheng, Qingli Zhou, Yuqiang Shen, Kai Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06465
• Pdf URL: https://arxiv.org/pdf/2501.06465
• Copy Paste: [[2501.06465]] MedCT: A Clinical Terminology Graph for Generative AI Applications in Healthcare(https://arxiv.org/abs/2501.06465)
• Keywords: language model, llm, hallucination
• Abstract: We introduce the world's first clinical terminology for the Chinese healthcare community, namely MedCT, accompanied by a clinical foundation model MedBERT and an entity linking model MedLink. The MedCT system enables standardized and programmable representation of Chinese clinical data, successively stimulating the development of new medicines, treatment pathways, and better patient outcomes for the populous Chinese community. Moreover, the MedCT knowledge graph provides a principled mechanism to minimize the hallucination problem of large language models (LLMs), therefore achieving significant levels of accuracy and safety in LLM-based clinical applications. By leveraging the LLMs' emergent capabilities of generativeness and expressiveness, we were able to rapidly built a production-quality terminology system and deployed to real-world clinical field within three months, while classical terminologies like SNOMED CT have gone through more than twenty years development. Our experiments show that the MedCT system achieves state-of-the-art (SOTA) performance in semantic matching and entity linking tasks, not only for Chinese but also for English. We also conducted a longitudinal field experiment by applying MedCT and LLMs in a representative spectrum of clinical tasks, including electronic health record (EHR) auto-generation and medical document search for diagnostic decision making. Our study shows a multitude of values of MedCT for clinical workflows and patient outcomes, especially in the new genre of clinical LLM applications. We present our approach in sufficient engineering detail, such that implementing a clinical terminology for other non-English societies should be readily reproducible. We openly release our terminology, models and algorithms, along with real-world clinical datasets for the development.
• 摘要：我们为华人医疗界推出了全球首个临床术语库 MedCT，以及临床基础模型 MedBERT 和实体链接模型 MedLink。MedCT 系统实现了对华人临床数据的标准化和可编程表示，从而助力华人社区开发新药、新治疗途径，改善患者治疗效果。此外，MedCT 知识图谱提供了一种原则性机制，可最大限度地减少大型语言模型 (LLM) 的幻觉问题，从而在基于 LLM 的临床应用中实现显著的准确性和安全性。通过利用 LLM 的生成性和表达性，我们能够在三个月内快速构建生产质量的术语系统并部署到现实世界的临床领域，而像 SNOMED CT 这样的经典术语库已经经历了二十多年的发展。我们的实验表明，MedCT 系统在语义匹配和实体链接任务中实现了最佳 (SOTA) 性能，不仅对中文，对英文亦是如此。我们还进行了一项纵向现场实验，将 MedCT 和 LLM 应用于一系列具有代表性的临床任务，包括电子健康记录 (EHR) 自动生成和用于诊断决策的医疗文档搜索。我们的研究表明，MedCT 对临床工作流程和患者结果具有多种价值，尤其是在新型临床 LLM 应用中。我们以足够的工程细节介绍了我们的方法，以便为其他非英语社会实施临床术语应该很容易复制。我们公开发布了我们的术语、模型和算法，以及用于开发的真实临床数据集。

Title: Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis

• Authors: Rui Liu, Zhenqi Jia, Feilong Bao, Haizhou Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06467
• Pdf URL: https://arxiv.org/pdf/2501.06467
• Copy Paste: [[2501.06467]] Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis(https://arxiv.org/abs/2501.06467)
• Keywords: agent
• Abstract: Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style. Unlike CD, stored dialogue (SD) contains preserved dialogue fragments from earlier stages of user-agent interaction, which include style expression knowledge relevant to scenarios similar to those in CD. Note that this knowledge plays a significant role in enabling the agent to synthesize expressive conversational speech that generates empathetic feedback. However, prior research has overlooked this aspect. To address this issue, we propose a novel Retrieval-Augmented Dialogue Knowledge Aggregation scheme for expressive CSS, termed RADKA-CSS, which includes three main components: 1) To effectively retrieve dialogues from SD that are similar to CD in terms of both semantic and style. First, we build a stored dialogue semantic-style database (SDSSD) which includes the text and audio samples. Then, we design a multi-attribute retrieval scheme to match the dialogue semantic and style vectors of the CD with the stored dialogue semantic and style vectors in the SDSSD, retrieving the most similar dialogues. 2) To effectively utilize the style knowledge from CD and SD, we propose adopting the multi-granularity graph structure to encode the dialogue and introducing a multi-source style knowledge aggregation mechanism. 3) Finally, the aggregated style knowledge are fed into the speech synthesizer to help the agent synthesize expressive speech that aligns with the conversational style. We conducted a comprehensive and in-depth experiment based on the DailyTalk dataset, which is a benchmarking dataset for the CSS task. Both objective and subjective evaluations demonstrate that RADKA-CSS outperforms baseline models in expressiveness rendering. Code and audio samples can be found at: this https URL.
• 摘要：对话语音合成 (CSS) 旨在以当前对话 (CD) 历史为参考，合成与对话风格相符的富有表现力的语音。与 CD 不同，存储对话 (SD) 包含来自用户代理交互早期阶段的保留对话片段，其中包括与类似于 CD 的场景相关的风格表达知识。请注意，这些知识在使代理能够合成产生共情反馈的富有表现力的对话语音方面起着重要作用。然而，之前的研究忽视了这一方面。为了解决这个问题，我们提出了一种用于富有表现力的 CSS 的新颖的检索增强对话知识聚合方案，称为 RADKA-CSS，它包括三个主要组件：1) 从 SD 中有效地检索在语义和风格方面都与 CD 相似的对话。首先，我们构建一个包含文本和音频样本的存储对话语义风格数据库 (SDSSD)。然后，我们设计了一个多属性检索方案，将 CD 的对话语义和风格向量与 SDSSD 中存储的对话语义和风格向量进行匹配，从而检索出最相似的对话。2）为了有效利用来自 CD 和 SD 的风格知识，我们建议采用多粒度图结构对对话进行编码，并引入多源风格知识聚合机制。3）最后，将聚合的风格知识输入语音合成器，帮助代理合成符合对话风格的富有表现力的语音。我们基于 DailyTalk 数据集进行了全面深入的实验，该数据集是 CSS 任务的基准数据集。客观和主观评估均表明 RADKA-CSS 在表现力渲染方面优于基线模型。代码和音频示例可以在此 https URL 中找到。

Title: First Token Probability Guided RAG for Telecom Question Answering

• Authors: Tingwei Chen, Jiayi Chen, Zijian Zhao, Haolong Chen, Liang Zhang, Guangxu Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06468
• Pdf URL: https://arxiv.org/pdf/2501.06468
• Copy Paste: [[2501.06468]] First Token Probability Guided RAG for Telecom Question Answering(https://arxiv.org/abs/2501.06468)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have garnered significant attention for their impressive general-purpose capabilities. For applications requiring intricate domain knowledge, Retrieval-Augmented Generation (RAG) has shown a distinct advantage in incorporating domain-specific information into LLMs. However, existing RAG research has not fully addressed the challenges of Multiple Choice Question Answering (MCQA) in telecommunications, particularly in terms of retrieval quality and mitigating hallucinations. To tackle these challenges, we propose a novel first token probability guided RAG framework. This framework leverages confidence scores to optimize key hyperparameters, such as chunk number and chunk window size, while dynamically adjusting the context. Our method starts by retrieving the most relevant chunks and generates a single token as the potential answer. The probabilities of all options are then normalized to serve as confidence scores, which guide the dynamic adjustment of the context. By iteratively optimizing the hyperparameters based on these confidence scores, we can continuously improve RAG performance. We conducted experiments to validate the effectiveness of our framework, demonstrating its potential to enhance accuracy in domain-specific MCQA tasks.
• 摘要：大型语言模型 (LLM) 因其令人印象深刻的通用功能而备受关注。对于需要复杂领域知识的应用程序，检索增强生成 (RAG) 在将特定领域信息纳入 LLM 方面表现出了明显的优势。然而，现有的 RAG 研究尚未完全解决电信领域多项选择题回答 (MCQA) 的挑战，特别是在检索质量和缓解幻觉方面。为了应对这些挑战，我们提出了一种新颖的第一个令牌概率引导 RAG 框架。该框架利用置信度分数来优化关键超参数，例如块数和块窗口大小，同时动态调整上下文。我们的方法首先检索最相关的块，然后生成单个令牌作为潜在答案。然后对所有选项的概率进行归一化以用作置信度分数，从而指导上下文的动态调整。通过基于这些置信度分数迭代优化超参数，我们可以不断提高 RAG 性能。我们进行了实验来验证我们框架的有效性，证明其在特定领域 MCQA 任务中提高准确性的潜力。

Title: Analyzing the Role of Context in Forecasting with Large Language Models

• Authors: Gerrit Mutschlechner, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.06496
• Pdf URL: https://arxiv.org/pdf/2501.06496
• Copy Paste: [[2501.06496]] Analyzing the Role of Context in Forecasting with Large Language Models(https://arxiv.org/abs/2501.06496)
• Keywords: language model, llm, prompt
• Abstract: This study evaluates the forecasting performance of recent language models (LLMs) on binary forecasting questions. We first introduce a novel dataset of over 600 binary forecasting questions, augmented with related news articles and their concise question-related summaries. We then explore the impact of input prompts with varying level of context on forecasting performance. The results indicate that incorporating news articles significantly improves performance, while using few-shot examples leads to a decline in accuracy. We find that larger models consistently outperform smaller models, highlighting the potential of LLMs in enhancing automated forecasting.
• 摘要：本研究评估了近期语言模型 (LLM) 对二元预测问题的预测性能。我们首先引入了一个包含 600 多个二元预测问题的新数据集，并添加了相关新闻文章及其简明的问题相关摘要。然后，我们探讨了具有不同上下文级别的输入提示对预测性能的影响。结果表明，加入新闻文章可显著提高性能，而使用少量样本会导致准确度下降。我们发现较大的模型始终优于较小的模型，这凸显了 LLM 在增强自动预测方面的潜力。

Title: PASS: Presentation Automation for Slide Generation and Speech

• Authors: Tushar Aggarwal, Aarohi Bhand
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06497
• Pdf URL: https://arxiv.org/pdf/2501.06497
• Copy Paste: [[2501.06497]] PASS: Presentation Automation for Slide Generation and Speech(https://arxiv.org/abs/2501.06497)
• Keywords: llm
• Abstract: In today's fast-paced world, effective presentations have become an essential tool for communication in both online and offline meetings. The crafting of a compelling presentation requires significant time and effort, from gathering key insights to designing slides that convey information clearly and concisely. However, despite the wealth of resources available, people often find themselves manually extracting crucial points, analyzing data, and organizing content in a way that ensures clarity and impact. Furthermore, a successful presentation goes beyond just the slides; it demands rehearsal and the ability to weave a captivating narrative to fully engage the audience. Although there has been some exploration of automating document-to-slide generation, existing research is largely centered on converting research papers. In addition, automation of the delivery of these presentations has yet to be addressed. We introduce PASS, a pipeline used to generate slides from general Word documents, going beyond just research papers, which also automates the oral delivery of the generated slides. PASS analyzes user documents to create a dynamic, engaging presentation with an AI-generated voice. Additionally, we developed an LLM-based evaluation metric to assess our pipeline across three critical dimensions of presentations: relevance, coherence, and redundancy. The data and codes are available at this https URL.
• 摘要：在当今快节奏的世界里，有效的演示已成为线上和线下会议中必不可少的沟通工具。制作一个引人注目的演示文稿需要大量的时间和精力，从收集关键见解到设计清晰简洁地传达信息的幻灯片。然而，尽管有丰富的资源可用，人们经常发现自己需要手动提取关键点、分析数据和组织内容，以确保清晰度和影响力。此外，成功的演示不仅限于幻灯片；它需要排练和编织引人入胜的故事的能力，以充分吸引观众。虽然已经有一些关于自动化文档到幻灯片生成的探索，但现有的研究主要集中在转换研究论文上。此外，这些演示文稿的交付自动化尚未得到解决。我们介绍了 PASS，这是一种用于从一般 Word 文档生成幻灯片的管道，它不仅限于研究论文，它还可以自动口头交付生成的幻灯片。PASS 分析用户文档，以创建具有 AI 生成语音的动态、引人入胜的演示文稿。此外，我们开发了基于 LLM 的评估指标，以评估我们在演示的三个关键维度上的流程：相关性、连贯性和冗余性。数据和代码可在此 https URL 上找到。

Title: Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering

• Authors: Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, Fei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06521
• Pdf URL: https://arxiv.org/pdf/2501.06521
• Copy Paste: [[2501.06521]] Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering(https://arxiv.org/abs/2501.06521)
• Keywords: language model, llm, hallucination
• Abstract: Hallucination, or the generation of incorrect or fabricated information, remains a critical challenge in large language models (LLMs), particularly in high-stake domains such as legal question answering (QA). In order to mitigate the hallucination rate in legal QA, we first introduce a benchmark called LegalHalBench and three automatic metrics to evaluate the common hallucinations when LLMs answer legal questions. We then propose a hallucination mitigation method that integrates behavior cloning and a novel Hard Sample-aware Iterative Direct Preference Optimization (HIPO). We conduct extensive real-data experiments to validate the effectiveness of our approach. Our results demonstrate remarkable improvements in various metrics, including the newly proposed Non-Hallucinated Statute Rate, Statute Relevance Rate, Legal Claim Truthfulness, as well as traditional metrics such as METEOR, BERTScore, ROUGE-L, and win rates.
• 摘要：幻觉，即产生不正确或捏造的信息，仍然是大型语言模型 (LLM) 面临的一个关键挑战，尤其是在法律问答 (QA) 等高风险领域。为了降低法律问答中的幻觉发生率，我们首先引入了一个名为 LegalHalBench 的基准和三个自动指标来评估 LLM 回答法律问题时常见的幻觉。然后，我们提出了一种幻觉缓解方法，该方法结合了行为克隆和一种新颖的硬样本感知迭代直接偏好优化 (HIPO)。我们进行了大量的真实数据实验来验证我们方法的有效性。我们的结果表明各种指标都有了显著的改善，包括新提出的非幻觉法规率、法规相关率、法律主张真实性，以及 METEOR、BERTScore、ROUGE-L 和胜率等传统指标。

Title: ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting

• Authors: Steven H. Wang, Maksim Zubkov, Kexin Fan, Sarah Harrell, Yuyang Sun, Wei Chen, Andreas Plesner, Roger Wattenhofer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06582
• Pdf URL: https://arxiv.org/pdf/2501.06582
• Copy Paste: [[2501.06582]] ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting(https://arxiv.org/abs/2501.06582)
• Keywords: llm
• Abstract: Information retrieval, specifically contract clause retrieval, is foundational to contract drafting because lawyers rarely draft contracts from scratch; instead, they locate and revise the most relevant precedent. We introduce the Atticus Clause Retrieval Dataset (ACORD), the first retrieval benchmark for contract drafting fully annotated by experts. ACORD focuses on complex contract clauses such as Limitation of Liability, Indemnification, Change of Control, and Most Favored Nation. It includes 114 queries and over 126,000 query-clause pairs, each ranked on a scale from 1 to 5 stars. The task is to find the most relevant precedent clauses to a query. The bi-encoder retriever paired with pointwise LLMs re-rankers shows promising results. However, substantial improvements are still needed to effectively manage the complex legal work typically undertaken by lawyers. As the first retrieval benchmark for contract drafting annotated by experts, ACORD can serve as a valuable IR benchmark for the NLP community.
• 摘要：信息检索，特别是合同条款检索，是合同起草的基础，因为律师很少从头开始起草合同；相反，他们会找到并修改最相关的先例。我们推出了 Atticus 条款检索数据集 (ACORD)，这是第一个由专家完全注释的合同起草检索基准。ACORD 专注于复杂的合同条款，例如责任限制、赔偿、控制权变更和最惠国待遇。它包括 114 个查询和超过 126,000 个查询条款对，每个对按 1 到 5 星的等级排名。任务是找到与查询最相关的先例条款。双编码器检索器与逐点 LLM 重新排序器配对显示出令人鼓舞的结果。然而，仍然需要进行实质性的改进才能有效管理通常由律师承担的复杂法律工作。作为第一个由专家注释的合同起草检索基准，ACORD 可以作为 NLP 社区的宝贵 IR 基准。

Title: ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning

• Authors: Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, Mark Gerstein
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06590
• Pdf URL: https://arxiv.org/pdf/2501.06590
• Copy Paste: [[2501.06590]] ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning(https://arxiv.org/abs/2501.06590)
• Keywords: language model, gpt, llm, agent
• Abstract: Chemical reasoning usually involves complex, multi-step processes that demand precise calculations, where even minor errors can lead to cascading failures. Furthermore, large language models (LLMs) encounter difficulties handling domain-specific formulas, executing reasoning steps accurately, and integrating code effectively when tackling chemical reasoning tasks. To address these challenges, we present ChemAgent, a novel framework designed to improve the performance of LLMs through a dynamic, self-updating library. This library is developed by decomposing chemical tasks into sub-tasks and compiling these sub-tasks into a structured collection that can be referenced for future queries. Then, when presented with a new problem, ChemAgent retrieves and refines pertinent information from the library, which we call memory, facilitating effective task decomposition and the generation of solutions. Our method designs three types of memory and a library-enhanced reasoning component, enabling LLMs to improve over time through experience. Experimental results on four chemical reasoning datasets from SciBench demonstrate that ChemAgent achieves performance gains of up to 46% (GPT-4), significantly outperforming existing methods. Our findings suggest substantial potential for future applications, including tasks such as drug discovery and materials science. Our code can be found at this https URL
• 摘要：化学推理通常涉及复杂的多步骤过程，需要精确计算，即使是微小的错误也会导致连锁故障。此外，大型语言模型 (LLM) 在处理化学推理任务时，在处理特定领域的公式、准确执行推理步骤和有效集成代码方面遇到困难。为了应对这些挑战，我们提出了 ChemAgent，这是一个新颖的框架，旨在通过动态、自我更新的库来提高 LLM 的性能。这个库是通过将化学任务分解为子任务并将这些子任务编译成一个结构化集合来开发的，可以在未来的查询中引用。然后，当出现新问题时，ChemAgent 会从库中检索和细化相关信息，我们称之为记忆，从而促进有效的任务分解和解决方案的生成。我们的方法设计了三种类型的记忆和一个库增强的推理组件，使 LLM 能够随着时间的推移通过经验不断改进。在 SciBench 的四个化学推理数据集上进行的实验结果表明，ChemAgent 的性能提升高达 46%（GPT-4），显著优于现有方法。我们的研究结果表明，其在未来的应用方面具有巨大潜力，包括药物发现和材料科学等任务。我们的代码可在此 https URL 中找到

Title: Dual use issues in the field of Natural Language Generation

• Authors: Emiel van Miltenburg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06636
• Pdf URL: https://arxiv.org/pdf/2501.06636
• Copy Paste: [[2501.06636]] Dual use issues in the field of Natural Language Generation(https://arxiv.org/abs/2501.06636)
• Keywords: prompt
• Abstract: This report documents the results of a recent survey in the SIGGEN community, focusing on Dual Use issues in Natural Language Generation (NLG). SIGGEN is the Special Interest Group (SIG) of the Association for Computational Linguistics (ACL) for researchers working on NLG. The survey was prompted by the ACL executive board, which asked all SIGs to provide an overview of dual use issues within their respective subfields. The survey was sent out in October 2024 and the results were processed in January 2025. With 23 respondents, the survey is presumably not representative of all SIGGEN members, but at least this document offers a helpful resource for future discussions. This report is open to feedback from the SIGGEN community. Let me know if you have any questions or comments!
• 摘要：本报告记录了 SIGGEN 社区最近一项调查的结果，重点关注自然语言生成 (NLG) 中的双重用途问题。SIGGEN 是计算语言学协会 (ACL) 为从事 NLG 工作的研究人员设立的特殊兴趣小组 (SIG)。这项调查是由 ACL 执行委员会发起的，该委员会要求所有 SIG 概述其各自子领域内的双重用途问题。调查于 2024 年 10 月发出，结果于 2025 年 1 月处理。调查有 23 名受访者，可能不能代表所有 SIGGEN 成员，但至少这份文件为未来的讨论提供了有用的资源。本报告欢迎 SIGGEN 社区的反馈。如果您有任何问题或意见，请告诉我！

Title: Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models

• Authors: Veronika Smilga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06638
• Pdf URL: https://arxiv.org/pdf/2501.06638
• Copy Paste: [[2501.06638]] Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models(https://arxiv.org/abs/2501.06638)
• Keywords: language model, prompt
• Abstract: Semantic leakage is a phenomenon recently introduced by Gonen et al. (2024). It refers to a situation in which associations learnt from the training data emerge in language model generations in an unexpected and sometimes undesired way. Prior work has focused on leakage in large language models (7B+ parameters). In this study, I use Qwen2.5 model family to explore whether smaller models, ranging from 500M to 7B parameters, demonstrate less semantic leakage due to their limited capacity for capturing complex associations. Building on the previous dataset from Gonen et al. (2024), I introduce a new dataset of color-focused prompts, categorized into specific types of semantic associations, to systematically evaluate the models' performance. Results indicate that smaller models exhibit less semantic leakage overall, although this trend is not strictly linear, with medium-sized models sometimes surpassing larger ones in leaking behavior. The dataset, the model generations, and the evaluation code are publicly available at this https URL.
• 摘要：语义泄漏是 Gonen 等人 (2024) 最近提出的一种现象。它指的是从训练数据中学习到的关联以意想不到的、有时是不希望的方式出现在语言模型生成中的情况。先前的研究主要集中在大型语言模型 (7B+ 参数) 中的泄漏。在本研究中，我使用 Qwen2.5 模型系列来探索较小的模型（从 500M 到 7B 参数不等）是否由于其捕获复杂关联的能力有限而表现出较少的语义泄漏。在 Gonen 等人 (2024) 之前的数据集的基础上，我引入了一个以颜色为重点的提示的新数据集，将其分为特定类型的语义关联，以系统地评估模型的性能。结果表明，较小的模型总体上表现出较少的语义泄漏，尽管这种趋势并非严格呈线性，中型模型的泄漏行为有时超过较大的模型。数据集、模型生成和评估代码可在此 https URL 上公开获取。

Title: FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings

• Authors: Tong Liu, Xiao Yu, Wenxuan Zhou, Jindong Gu, Volker Tresp
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06645
• Pdf URL: https://arxiv.org/pdf/2501.06645
• Copy Paste: [[2501.06645]] FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings(https://arxiv.org/abs/2501.06645)
• Keywords: language model, llm
• Abstract: Efficient preference optimization algorithms such as Direct Preference Optimization (DPO) have become a popular approach in aligning large language models (LLMs) with human preferences. These algorithms implicitly treat the LLM as a reward model, and focus on training it to correct misranked preference pairs. However, recent work~\citep{chen2024preference} empirically finds that DPO training \textit{rarely improves these misranked preference pairs}, despite its gradient emphasizing on these cases. We introduce FocalPO, a DPO variant that instead \textit{down-weighs} misranked preference pairs and prioritizes enhancing the model's understanding of pairs that it can already rank correctly. Inspired by Focal Loss used in vision tasks, FocalPO achieves this by adding a modulating factor to dynamically scale DPO loss. Our experiment demonstrates that FocalPO surpasses DPO and its variants on popular benchmarks like Alpaca Eval 2.0 using Mistral-Base-7B and Llama-3-Instruct-8B. Additionally, we empirically reveals how FocalPO affects training on correct and incorrect sample groups, further underscoring its effectiveness.
• 摘要：高效的偏好优化算法（例如直接偏好优化 (DPO)）已成为将大型语言模型 (LLM) 与人类偏好对齐的流行方法。这些算法隐式地将 LLM 视为奖励模型，并专注于训练它以纠正错误排序的偏好对。然而，最近的研究 ~\citep{chen2024preference} 通过经验发现，尽管 DPO 训练的梯度强调这些情况，但它很少改善这些错误排序的偏好对。我们引入了 FocalPO，这是 DPO 的一种变体，它反而会降低错误排序的偏好对的权重，并优先增强模型对已经可以正确排序的对的理解。受视觉任务中使用的 Focal Loss 的启发，FocalPO 通过添加调节因子来动态缩放 DPO 损失来实现这一点。我们的实验表明，在使用 Mistral-Base-7B 和 Llama-3-Instruct-8B 的 Alpaca Eval 2.0 等流行基准测试中，FocalPO 的表现优于 DPO 及其变体。此外，我们通过实证研究揭示了 FocalPO 如何影响正确和错误样本组的训练，进一步强调了其有效性。

Title: TAPO: Task-Referenced Adaptation for Prompt Optimization

• Authors: Wenxin Luo, Weirui Wang, Xiaopeng Li, Weibo Zhou, Pengyue Jia, Xiangyu Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06689
• Pdf URL: https://arxiv.org/pdf/2501.06689
• Copy Paste: [[2501.06689]] TAPO: Task-Referenced Adaptation for Prompt Optimization(https://arxiv.org/abs/2501.06689)
• Keywords: language model, llm, prompt
• Abstract: Prompt engineering can significantly improve the performance of large language models (LLMs), with automated prompt optimization (APO) gaining significant attention due to the time-consuming and laborious nature of manual prompt design. However, much of the existing work in APO overlooks task-specific characteristics, resulting in prompts that lack domain specificity and are not well-suited for task-specific optimization. In this paper, we introduce TAPO, a multitask-aware prompt optimization framework composed of three key modules. First, a task-aware metric selection module is proposed to enhance task-specific prompt generation capabilities. Second, we present a multi-metrics evaluation module to jointly evaluate prompts from multiple perspectives. Third, an evolution-based optimization framework is introduced for automatic prompt refinement, which improves adaptability across various tasks. Extensive experiments on six datasets demonstrate the effectiveness of our approach, and our code is publicly available.
• 摘要：提示工程可以显著提高大型语言模型 (LLM) 的性能，其中自动提示优化 (APO) 因手动提示设计耗时费力而受到广泛关注。然而，APO 中的许多现有工作都忽略了任务特定特性，导致提示缺乏领域特定性，不适合针对特定任务进行优化。在本文中，我们介绍了一个由三个关键模块组成的多任务感知提示优化框架 TAPO。首先，提出了一个任务感知指标选择模块来增强针对特定任务的提示生成能力。其次，我们提出了一个多指标评估模块，从多个角度联合评估提示。第三，引入了一个基于进化的优化框架来自动细化提示，从而提高了跨各种任务的适应性。在六个数据集上进行的大量实验证明了我们方法的有效性，我们的代码是公开的。

Title: ZNO-Eval: Benchmarking reasoning capabilities of large language models in Ukrainian

• Authors: Mykyta Syromiatnikov, Victoria Ruvinskaya, Anastasiya Troynina
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06715
• Pdf URL: https://arxiv.org/pdf/2501.06715
• Copy Paste: [[2501.06715]] ZNO-Eval: Benchmarking reasoning capabilities of large language models in Ukrainian(https://arxiv.org/abs/2501.06715)
• Keywords: language model, gpt
• Abstract: As the usage of large language models for problems outside of simple text understanding or generation increases, assessing their abilities and limitations becomes crucial. While significant progress has been made in this area over the last few years, most research has focused on benchmarking English, leaving other languages underexplored. This makes evaluating the reasoning and robustness level of language models in Ukrainian particularly challenging. The purpose of this work is to establish a comprehensive benchmark for the reasoning capabilities evaluation of large language models in the Ukrainian language. This paper presents the ZNO-Eval benchmark based on real exam tasks from Ukraine's standardized educational testing system: the External Independent Evaluation and the National Multi-subject Test. With single-answer options, multiple-choice, matching, and open-ended questions from diverse subjects, including Ukrainian language, mathematics, history, and geography, this dataset paves the way toward a thorough analysis of reasoning capabilities across different domains and complexities. Evaluation of several well-known language models, such as GPT-3.5-Turbo, GPT-4o, GPT-4-Turbo, Mistral Large, Claude 3 Opus, and Gemini-1.5 Pro on this benchmark demonstrated the superiority of GPT-4o in both common knowledge reasoning and intricate language tasks. At the same time, Gemini Pro and GPT-4 Turbo excelled in the arithmetic domain, leading in single-answer and open-ended math problems. While all models were close to max performance in text-only common knowledge tasks like history and geography, there still is a gap for Ukrainian language and math, thus highlighting the importance of developing specialized language benchmarks for more accurate assessments of model capabilities and limitations across different languages and contexts.
• 摘要：随着大型语言模型在简单文本理解或生成之外的问题上的使用越来越多，评估它们的能力和局限性变得至关重要。尽管过去几年该领域取得了重大进展，但大多数研究都集中在对英语进行基准测试上，而其他语言尚未得到充分探索。这使得评估乌克兰语语言模型的推理和稳健性水平尤其具有挑战性。这项工作的目的是为乌克兰语大型语言模型的推理能力评估建立一个全面的基准。本文根据乌克兰标准化教育测试系统的真实考试任务提出了 ZNO-Eval 基准：外部独立评估和国家多学科考试。该数据集包含来自乌克兰语、数学、历史和地理等不同学科的单选题、多项选择题、匹配题和开放式问题，为全面分析不同领域和复杂程度的推理能力铺平了道路。在该基准上对 GPT-3.5-Turbo、GPT-4o、GPT-4-Turbo、Mistral Large、Claude 3 Opus 和 Gemini-1.5 Pro 等几种著名语言模型的评估表明，GPT-4o 在常识推理和复杂语言任务方面都表现出色。同时，Gemini Pro 和 GPT-4 Turbo 在算术领域表现出色，在单答案和开放式数学问题中处于领先地位。虽然所有模型在历史和地理等纯文本常识任务中都接近最高性能，但乌克兰语和数学仍然存在差距，因此凸显了开发专门的语言基准的重要性，以便更准确地评估不同语言和语境中的模型能力和局限性。

Title: Measuring the Robustness of Reference-Free Dialogue Evaluation Systems

• Authors: Justin Vasselli, Adam Nohejl, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06728
• Pdf URL: https://arxiv.org/pdf/2501.06728
• Copy Paste: [[2501.06728]] Measuring the Robustness of Reference-Free Dialogue Evaluation Systems(https://arxiv.org/abs/2501.06728)
• Keywords: language model, llm, prompt
• Abstract: Advancements in dialogue systems powered by large language models (LLMs) have outpaced the development of reliable evaluation metrics, particularly for diverse and creative responses. We present a benchmark for evaluating the robustness of reference-free dialogue metrics against four categories of adversarial attacks: speaker tag prefixes, static responses, ungrammatical responses, and repeated conversational context. We analyze metrics such as DialogRPT, UniEval, and PromptEval -- a prompt-based method leveraging LLMs -- across grounded and ungrounded datasets. By examining both their correlation with human judgment and susceptibility to adversarial attacks, we find that these two axes are not always aligned; metrics that appear to be equivalent when judged by traditional benchmarks may, in fact, vary in their scores of adversarial responses. These findings motivate the development of nuanced evaluation frameworks to address real-world dialogue challenges.
• 摘要：基于大型语言模型 (LLM) 的对话系统的进步已经超过了可靠评估指标的发展，特别是对于多样化和创造性的回应。我们提出了一个基准，用于评估无参考对话指标对四类对抗性攻击的稳健性：说话人标签前缀、静态响应、不合语法的响应和重复的对话上下文。我们分析了 DialogRPT、UniEval 和 PromptEval（一种利用 LLM 的基于提示的方法）等指标，这些指标涵盖了有根据和无根据的数据集。通过研究它们与人类判断的相关性和对抗性攻击的敏感性，我们发现这两个轴并不总是一致的；按照传统基准判断时看似等同的指标实际上可能在对抗性响应的得分上有所不同。这些发现推动了细致入微的评估框架的发展，以应对现实世界的对话挑战。

Title: Better Prompt Compression Without Multi-Layer Perceptrons

• Authors: Edouardo Honig, Andrew Lizarraga, Zijun Frank Zhang, Ying Nian Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06730
• Pdf URL: https://arxiv.org/pdf/2501.06730
• Copy Paste: [[2501.06730]] Better Prompt Compression Without Multi-Layer Perceptrons(https://arxiv.org/abs/2501.06730)
• Keywords: language model, prompt
• Abstract: Prompt compression is a promising approach to speeding up language model inference without altering the generative model. Prior works compress prompts into smaller sequences of learned tokens using an encoder that is trained as a LowRank Adaptation (LoRA) of the inference language model. However, we show that the encoder does not need to keep the original language model's architecture to achieve useful compression. We introduce the Attention-Only Compressor (AOC), which learns a prompt compression encoder after removing the multilayer perceptron (MLP) layers in the Transformer blocks of a language model, resulting in an encoder with roughly 67% less parameters compared to the original model. Intriguingly we find that, across a range of compression ratios up to 480x, AOC can better regenerate prompts and outperform a baseline compression encoder that is a LoRA of the inference language model without removing MLP layers. These results demonstrate that the architecture of prompt compression encoders does not need to be identical to that of the original decoder language model, paving the way for further research into architectures and approaches for prompt compression.
• 摘要：提示压缩是一种有前途的方法，可以在不改变生成模型的情况下加速语言模型推理。先前的研究使用编码器将提示压缩成较小的学习标记序列，该编码器被训练为推理语言模型的低秩自适应 (LoRA)。但是，我们表明编码器不需要保留原始语言模型的架构即可实现有用的压缩。我们引入了仅注意压缩器 (AOC)，它在删除语言模型的 Transformer 块中的多层感知器 (MLP) 层后学习提示压缩编码器，从而产生与原始模型相比参数减少约 67% 的编码器。有趣的是，我们发现，在高达 480 倍的压缩比范围内，AOC 可以更好地重新生成提示，并且优于基线压缩编码器，后者是推理语言模型的 LoRA，无需删除 MLP 层。这些结果表明，即时压缩编码器的架构不需要与原始解码器语言模型相同，为进一步研究即时压缩的架构和方法铺平了道路。

Title: Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation

• Authors: Shunfan Zheng, Xiechi Zhang, Gerard de Melo, Xiaoling Wang, Linlin Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06741
• Pdf URL: https://arxiv.org/pdf/2501.06741
• Copy Paste: [[2501.06741]] Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation(https://arxiv.org/abs/2501.06741)
• Keywords: language model, llm
• Abstract: In the rapidly evolving landscape of large language models (LLMs) for medical applications, ensuring the reliability and accuracy of these models in clinical settings is paramount. Existing benchmarks often focus on fixed-format tasks like multiple-choice QA, which fail to capture the complexity of real-world clinical diagnostics. Moreover, traditional evaluation metrics and LLM-based evaluators struggle with misalignment, often providing oversimplified assessments that do not adequately reflect human judgment. To address these challenges, we introduce HDCEval, a Hierarchical Divide-and-Conquer Evaluation framework tailored for fine-grained alignment in medical evaluation. HDCEval is built on a set of fine-grained medical evaluation guidelines developed in collaboration with professional doctors, encompassing Patient Question Relevance, Medical Knowledge Correctness, and Expression. The framework decomposes complex evaluation tasks into specialized subtasks, each evaluated by expert models trained through Attribute-Driven Token Optimization (ADTO) on a meticulously curated preference dataset. This hierarchical approach ensures that each aspect of the evaluation is handled with expert precision, leading to a significant improvement in alignment with human evaluators.
• 摘要：在快速发展的用于医疗应用的大型语言模型 (LLM) 领域中，确保这些模型在临床环境中的可靠性和准确性至关重要。现有的基准测试通常侧重于固定格式的任务，例如多项选择问答，这些任务无法捕捉现实世界临床诊断的复杂性。此外，传统的评估指标和基于 LLM 的评估器存在不一致的问题，通常提供过于简单的评估，不能充分反映人类的判断。为了应对这些挑战，我们引入了 HDCEval，这是一个分层分而治之的评估框架，专门用于医疗评估中的细粒度对齐。HDCEval 建立在与专业医生合作开发的一套细粒度医疗评估指南之上，涵盖了患者问题的相关性、医学知识的正确性和表达。该框架将复杂的评估任务分解为专门的子任务，每个子任务都由通过属性驱动的标记优化 (ADTO) 在精心策划的偏好数据集上训练的专家模型进行评估。这种分层方法可确保评估的每个方面都以专家的精确度处理，从而显著提高与人工评估员的一致性。

Title: Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models

• Authors: Michael Toker, Ido Galil, Hadas Orgad, Rinon Gal, Yoad Tewel, Gal Chechik, Yonatan Belinkov
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06751
• Pdf URL: https://arxiv.org/pdf/2501.06751
• Copy Paste: [[2501.06751]] Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models(https://arxiv.org/abs/2501.06751)
• Keywords: prompt
• Abstract: Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. Typically, these prompts are extended to a fixed length by adding padding tokens before text encoding. Despite being a default practice, the influence of padding tokens on the image generation process has not been investigated. In this work, we conduct the first in-depth analysis of the role padding tokens play in T2I models. We develop two causal techniques to analyze how information is encoded in the representation of tokens across different components of the T2I pipeline. Using these techniques, we investigate when and how padding tokens impact the image generation process. Our findings reveal three distinct scenarios: padding tokens may affect the model's output during text encoding, during the diffusion process, or be effectively ignored. Moreover, we identify key relationships between these scenarios and the model's architecture (cross or self-attention) and its training process (frozen or trained text encoder). These insights contribute to a deeper understanding of the mechanisms of padding tokens, potentially informing future model design and training practices in T2I systems.
• 摘要：文本到图像 (T2I) 扩散模型依靠编码提示来指导图像生成过程。通常，通过在文本编码之前添加填充标记，将这些提示扩展为固定长度。尽管这是默认做法，但填充标记对图像生成过程的影响尚未得到研究。在这项工作中，我们首次深入分析了填充标记在 T2I 模型中的作用。我们开发了两种因果技术来分析信息在 T2I 管道的不同组件中如何在标记表示中进行编码。使用这些技术，我们研究了填充标记何时以及如何影响图像生成过程。我们的研究结果揭示了三种不同的情况：填充标记可能会在文本编码期间、在扩散过程中影响模型的输出，或者被有效忽略。此外，我们确定了这些场景与模型的架构（交叉或自注意力）及其训练过程（冻结或训练的文本编码器）之间的关键关系。这些见解有助于更深入地理解填充标记的机制，可能为 T2I 系统中未来的模型设计和训练实践提供参考。

Title: Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences

• Authors: Liu Yu, Ludie Guo, Ping Kuang, Fan Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06795
• Pdf URL: https://arxiv.org/pdf/2501.06795
• Copy Paste: [[2501.06795]] Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences(https://arxiv.org/abs/2501.06795)
• Keywords: language model, llm
• Abstract: Pre-trained language models (PLMs) are trained on data that inherently contains gender biases, leading to undesirable impacts. Traditional debiasing methods often rely on external corpora, which may lack quality, diversity, or demographic balance, affecting the effectiveness of debiasing. With the rise of large language models and their extensive knowledge, we propose enhancing fairness (Fair-Gender) in PLMs by absorbing coherent, attribute-balanced, and semantically rich sentences. However, these sentences cannot be directly used for debiasing due to alignment issues and the risk of negative transfer. We address this by applying causal analysis to estimate causal effects, filtering out unaligned sentences, and identifying aligned ones for incorporation into PLMs, thereby ensuring positive transfer. Experiments show that our approach significantly reduces gender biases in PLMs while preserving their language expressiveness.
• 摘要：预训练语言模型 (PLM) 是在固有包含性别偏见的数据上进行训练的，这会导致不良影响。传统的去偏见方法通常依赖于外部语料库，而外部语料库可能缺乏质量、多样性或人口平衡性，从而影响去偏见的有效性。随着大型语言模型及其广泛知识的兴起，我们建议通过吸收连贯、属性平衡和语义丰富的句子来增强 PLM 中的公平性 (Fair-Gender)。然而，由于对齐问题和负迁移风险，这些句子不能直接用于去偏见。我们通过应用因果分析来估计因果效应、过滤掉不对齐的句子并识别对齐的句子以纳入 PLM 来解决这个问题，从而确保正迁移。实验表明，我们的方法显著减少了 PLM 中的性别偏见，同时保留了其语言表达能力。

Title: Event Argument Extraction with Enriched Prompts

• Authors: Chen Liang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06825
• Pdf URL: https://arxiv.org/pdf/2501.06825
• Copy Paste: [[2501.06825]] Event Argument Extraction with Enriched Prompts(https://arxiv.org/abs/2501.06825)
• Keywords: language model, prompt
• Abstract: This work aims to delve deeper into prompt-based event argument extraction (EAE) models. We explore the impact of incorporating various types of information into the prompt on model performance, including trigger, other role arguments for the same event, and role arguments across multiple events within the same document. Further, we provide the best possible performance that the prompt-based EAE model can attain and demonstrate such models can be further optimized from the perspective of the training objective. Experiments are carried out on three small language models and two large language models in RAMS.
• 摘要：本研究旨在深入研究基于提示的事件参数提取 (EAE) 模型。我们探索将各种类型的信息纳入提示对模型性能的影响，包括触发器、同一事件的其他角色参数以及同一文档中多个事件的角色参数。此外，我们提供了基于提示的 EAE 模型可以达到的最佳性能，并证明此类模型可以从训练目标的角度进一步优化。实验在 RAMS 中的三个小型语言模型和两个大型语言模型上进行。

Title: A Comprehensive Evaluation of Large Language Models on Mental Illnesses in Arabic Context

• Authors: Noureldin Zahran, Aya E. Fouda, Radwa J. Hanafy, Mohammed E. Fouda
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06859
• Pdf URL: https://arxiv.org/pdf/2501.06859
• Copy Paste: [[2501.06859]] A Comprehensive Evaluation of Large Language Models on Mental Illnesses in Arabic Context(https://arxiv.org/abs/2501.06859)
• Keywords: language model, gpt, llm, prompt
• Abstract: Mental health disorders pose a growing public health concern in the Arab world, emphasizing the need for accessible diagnostic and intervention tools. Large language models (LLMs) offer a promising approach, but their application in Arabic contexts faces challenges including limited labeled datasets, linguistic complexity, and translation biases. This study comprehensively evaluates 8 LLMs, including general multi-lingual models, as well as bi-lingual ones, on diverse mental health datasets (such as AraDepSu, Dreaddit, MedMCQA), investigating the impact of prompt design, language configuration (native Arabic vs. translated English, and vice versa), and few-shot prompting on diagnostic performance. We find that prompt engineering significantly influences LLM scores mainly due to reduced instruction following, with our structured prompt outperforming a less structured variant on multi-class datasets, with an average difference of 14.5\%. While language influence on performance was modest, model selection proved crucial: Phi-3.5 MoE excelled in balanced accuracy, particularly for binary classification, while Mistral NeMo showed superior performance in mean absolute error for severity prediction tasks. Few-shot prompting consistently improved performance, with particularly substantial gains observed for GPT-4o Mini on multi-class classification, boosting accuracy by an average factor of 1.58. These findings underscore the importance of prompt optimization, multilingual analysis, and few-shot learning for developing culturally sensitive and effective LLM-based mental health tools for Arabic-speaking populations.
• 摘要：精神健康障碍是阿拉伯世界日益严重的公共卫生问题，这凸显了对可访问诊断和干预工具的需求。大型语言模型 (LLM) 提供了一种有前途的方法，但它们在阿拉伯语环境中的应用面临着挑战，包括有限的标记数据集、语言复杂性和翻译偏见。本研究全面评估了 8 个 LLM，包括通用多语言模型以及双语模型，在各种心理健康数据集（如 AraDepSu、Dreaddit、MedMCQA）上的表现，调查了提示设计、语言配置（母语阿拉伯语与翻译英语，反之亦然）和少量提示对诊断性能的影响。我们发现提示工程显著影响 LLM 分数，主要是由于减少了指令遵循，我们的结构化提示在多类数据集上的表现优于结构化程度较低的变体，平均差异为 14.5%。虽然语言对性能的影响不大，但模型选择至关重要：Phi-3.5 MoE 在平衡准确度方面表现出色，尤其是在二元分类方面，而 Mistral NeMo 在严重程度预测任务的平均绝对误差方面表现出色。少量提示持续提高了性能，GPT-4o Mini 在多类分类方面取得了特别显著的进步，准确度平均提高了 1.58 倍。这些发现强调了提示优化、多语言分析和少量学习对于为阿拉伯语人群开发文化敏感且有效的基于 LLM 的心理健康工具的重要性。

Title: Language Fusion for Parameter-Efficient Cross-lingual Transfer

• Authors: Philipp Borchert, Ivan Vulić, Marie-Francine Moens, Jochen De Weerdt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06892
• Pdf URL: https://arxiv.org/pdf/2501.06892
• Copy Paste: [[2501.06892]] Language Fusion for Parameter-Efficient Cross-lingual Transfer(https://arxiv.org/abs/2501.06892)
• Keywords: language model
• Abstract: Limited availability of multilingual text corpora for training language models often leads to poor performance on downstream tasks due to undertrained representation spaces for languages other than English. This 'under-representation' has motivated recent cross-lingual transfer methods to leverage the English representation space by e.g. mixing English and 'non-English' tokens at the input level or extending model parameters to accommodate new languages. However, these approaches often come at the cost of increased computational complexity. We propose Fusion forLanguage Representations (FLARE) in adapters, a novel method that enhances representation quality and downstream performance for languages other than English while maintaining parameter efficiency. FLARE integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations, maintaining parameter efficiency while improving transfer performance. A series of experiments across representative cross-lingual natural language understanding tasks, including natural language inference, question-answering and sentiment analysis, demonstrate FLARE's effectiveness. FLARE achieves performance improvements of 4.9% for Llama 3.1 and 2.2% for Gemma~2 compared to standard LoRA fine-tuning on question-answering tasks, as measured by the exact match metric.
• 摘要：用于训练语言模型的多语言文本语料库有限，这通常会导致下游任务的性能不佳，因为英语以外的语言的表示空间训练不足。这种“表示不足”促使最近的跨语言迁移方法利用英语表示空间，例如在输入级别混合英语和“非英语”标记或扩展模型参数以适应新语言。然而，这些方法往往以增加计算复杂性为代价。我们提出了适配器中的语言表示融合 (FLARE)，这是一种新方法，可在保持参数效率的同时提高英语以外语言的表示质量和下游性能。FLARE 使用轻量级线性变换在低秩 (LoRA) 适配器中集成源语言和目标语言表示，在提高传输性能的同时保持参数效率。在代表性跨语言自然语言理解任务（包括自然语言推理、问答和情感分析）中进行的一系列实验证明了 FLARE 的有效性。以精确匹配指标衡量，与标准 LoRA 在问答任务上的微调相比，FLARE 使 Llama 3.1 的性能提高了 4.9%，使 Gemma~2 的性能提高了 2.2%。

Title: Harnessing Large Language Models for Disaster Management: A Survey

• Authors: Zhenyu Lei, Yushun Dong, Weiyu Li, Rong Ding, Qi Wang, Jundong Li
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06932
• Pdf URL: https://arxiv.org/pdf/2501.06932
• Copy Paste: [[2501.06932]] Harnessing Large Language Models for Disaster Management: A Survey(https://arxiv.org/abs/2501.06932)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have revolutionized scientific research with their exceptional capabilities and transformed various fields. Among their practical applications, LLMs have been playing a crucial role in mitigating threats to human life, infrastructure, and the environment. Despite growing research in disaster LLMs, there remains a lack of systematic review and in-depth analysis of LLMs for natural disaster management. To address the gap, this paper presents a comprehensive survey of existing LLMs in natural disaster management, along with a taxonomy that categorizes existing works based on disaster phases and application scenarios. By collecting public datasets and identifying key challenges and opportunities, this study aims to guide the professional community in developing advanced LLMs for disaster management to enhance the resilience against natural disasters.
• 摘要：大型语言模型 (LLM) 以其卓越的能力彻底改变了科学研究，并改变了各个领域。在实际应用中，LLM 在减轻对人类生命、基础设施和环境的威胁方面发挥着至关重要的作用。尽管对灾难 LLM 的研究日益增多，但仍然缺乏对自然灾害管理 LLM 的系统评价和深入分析。为了弥补这一差距，本文对现有的自然灾害管理 LLM 进行了全面调查，并根据灾害阶段和应用场景对现有工作进行了分类。通过收集公共数据集并确定关键挑战和机遇，本研究旨在指导专业社区开发先进的灾害管理 LLM，以增强抵御自然灾害的能力。

Title: ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalization

• Authors: Anh Thi-Hoang Nguyen, Dung Ha Nguyen, Kiet Van Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07020
• Pdf URL: https://arxiv.org/pdf/2501.07020
• Copy Paste: [[2501.07020]] ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalization(https://arxiv.org/abs/2501.07020)
• Keywords: language model
• Abstract: ViSoLex is an open-source system designed to address the unique challenges of lexical normalization for Vietnamese social media text. The platform provides two core services: Non-Standard Word (NSW) Lookup and Lexical Normalization, enabling users to retrieve standard forms of informal language and standardize text containing NSWs. ViSoLex's architecture integrates pre-trained language models and weakly supervised learning techniques to ensure accurate and efficient normalization, overcoming the scarcity of labeled data in Vietnamese. This paper details the system's design, functionality, and its applications for researchers and non-technical users. Additionally, ViSoLex offers a flexible, customizable framework that can be adapted to various datasets and research requirements. By publishing the source code, ViSoLex aims to contribute to the development of more robust Vietnamese natural language processing tools and encourage further research in lexical normalization. Future directions include expanding the system's capabilities for additional languages and improving the handling of more complex non-standard linguistic patterns.
• 摘要：ViSoLex 是一个开源系统，旨在解决越南语社交媒体文本词汇规范化的独特挑战。该平台提供两项核心服务：非标准词 (NSW) 查找和词汇规范化，使用户能够检索非正式语言的标准形式并标准化包含 NSW 的文本。ViSoLex 的架构集成了预训练语言模型和弱监督学习技术，以确保准确、高效的规范化，克服了越南语标记数据稀缺的问题。本文详细介绍了该系统的设计、功能及其对研究人员和非技术用户的应用。此外，ViSoLex 提供了一个灵活、可定制的框架，可以适应各种数据集和研究要求。通过发布源代码，ViSoLex 旨在为开发更强大的越南语自然语言处理工具做出贡献，并鼓励进一步研究词汇规范化。未来的方向包括扩展系统对其他语言的功能，并改进对更复杂的非标准语言模式的处理。

Title: Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models

• Authors: Yongyu Mu, Hengyu Li, Junxin Wang, Xiaoxuan Zhou, Chenglong Wang, Yingfeng Luo, Qiaozhi He, Tong Xiao, Guocheng Chen, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07086
• Pdf URL: https://arxiv.org/pdf/2501.07086
• Copy Paste: [[2501.07086]] Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models(https://arxiv.org/abs/2501.07086)
• Keywords: prompt
• Abstract: Previous work on augmenting large multimodal models (LMMs) for text-to-image (T2I) generation has focused on enriching the input space of in-context learning (ICL). This includes providing a few demonstrations and optimizing image descriptions to be more detailed and logical. However, as demand for more complex and flexible image descriptions grows, enhancing comprehension of input text within the ICL paradigm remains a critical yet underexplored area. In this work, we extend this line of research by constructing parallel multilingual prompts aimed at harnessing the multilingual capabilities of LMMs. More specifically, we translate the input text into several languages and provide the models with both the original text and the translations. Experiments on two LMMs across 3 benchmarks show that our method, PMT2I, achieves superior performance in general, compositional, and fine-grained assessments, especially in human preference alignment. Additionally, with its advantage of generating more diverse images, PMT2I significantly outperforms baseline prompts when incorporated with reranking methods. Our code and parallel multilingual data can be found at this https URL.
• 摘要：之前关于增强大型多模态模型 (LMM) 以进行文本到图像 (T2I) 生成的研究主要集中在丰富上下文学习 (ICL) 的输入空间。这包括提供一些演示和优化图像描述，使其更详细、更合乎逻辑。然而，随着对更复杂、更灵活的图像描述的需求不断增长，在 ICL 范式中增强对输入文本的理解仍然是一个关键但尚未充分探索的领域。在这项工作中，我们通过构建并行多语言提示来扩展这一研究方向，旨在利用 LMM 的多语言功能。更具体地说，我们将输入文本翻译成几种语言，并为模型提供原始文本和翻译。在 3 个基准上对两个 LMM 进行的实验表明，我们的方法 PMT2I 在一般、组合和细粒度评估方面取得了卓越的表现，尤其是在人类偏好对齐方面。此外，凭借其生成更多样化图像的优势，PMT2I 在与重新排名方法结合时明显优于基线提示。我们的代码和并行多语言数据可在此 https URL 找到。

Title: ListConRanker: A Contrastive Text Reranker with Listwise Encoding

• Authors: Junlong Liu, Yue Ma, Ruihui Zhao, Junhao Zheng, Qianli Ma, Yangyang Kang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.07111
• Pdf URL: https://arxiv.org/pdf/2501.07111
• Copy Paste: [[2501.07111]] ListConRanker: A Contrastive Text Reranker with Listwise Encoding(https://arxiv.org/abs/2501.07111)
• Keywords: retrieval-augmented generation
• Abstract: Reranker models aim to re-rank the passages based on the semantics similarity between the given query and passages, which have recently received more attention due to the wide application of the Retrieval-Augmented Generation. Most previous methods apply pointwise encoding, meaning that it can only encode the context of the query for each passage input into the model. However, for the reranker model, given a query, the comparison results between passages are even more important, which is called listwise encoding. Besides, previous models are trained using the cross-entropy loss function, which leads to issues of unsmooth gradient changes during training and low training efficiency. To address these issues, we propose a novel Listwise-encoded Contrastive text reRanker (ListConRanker). It can help the passage to be compared with other passages during the encoding process, and enhance the contrastive information between positive examples and between positive and negative examples. At the same time, we use the circle loss to train the model to increase the flexibility of gradients and solve the problem of training efficiency. Experimental results show that ListConRanker achieves state-of-the-art performance on the reranking benchmark of Chinese Massive Text Embedding Benchmark, including the cMedQA1.0, cMedQA2.0, MMarcoReranking, and T2Reranking datasets.
• 摘要：重排模型旨在根据给定查询与段落之间的语义相似性对段落进行重排，近年来由于检索增强生成（Retrieval-Augmented Generation）的广泛应用而受到更多关注。之前的方法大多采用逐点编码，即对输入模型的每个段落只能编码查询的上下文。然而对于重排模型而言，给定一个查询，段落之间的比较结果更为重要，这被称为列表编码。此外，之前的模型采用交叉熵损失函数进行训练，这导致训练过程中梯度变化不平滑、训练效率低下的问题。针对这些问题，我们提出了一种新颖的列表编码对比文本重排模型（ListConRanker），它可以帮助段落在编码过程中与其他段落进行比较，增强正例之间以及正反例之间的对比信息。同时，我们使用圆损失来训练模型，增加梯度的灵活性，解决训练效率问题。实验结果表明，ListConRanker 在中文海量文本嵌入基准的重排序基准上取得了最佳性能，包括 cMedQA1.0、cMedQA2.0、MMarcoReranking 和 T2Reranking 数据集。

Title: When lies are mostly truthful: automated verbal deception detection for embedded lies

• Authors: Riccardo Loconte, Bennett Kleinberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07217
• Pdf URL: https://arxiv.org/pdf/2501.07217
• Copy Paste: [[2501.07217]] When lies are mostly truthful: automated verbal deception detection for embedded lies(https://arxiv.org/abs/2501.07217)
• Keywords: language model
• Abstract: Background: Verbal deception detection research relies on narratives and commonly assumes statements as truthful or deceptive. A more realistic perspective acknowledges that the veracity of statements exists on a continuum with truthful and deceptive parts being embedded within the same statement. However, research on embedded lies has been lagging behind. Methods: We collected a novel dataset of 2,088 truthful and deceptive statements with annotated embedded lies. Using a within-subjects design, participants provided a truthful account of an autobiographical event. They then rewrote their statement in a deceptive manner by including embedded lies, which they highlighted afterwards and judged on lie centrality, deceptiveness, and source. Results: We show that a fined-tuned language model (Llama-3-8B) can classify truthful statements and those containing embedded lies with 64% accuracy. Individual differences, linguistic properties and explainability analysis suggest that the challenge of moving the dial towards embedded lies stems from their resemblance to truthful statements. Typical deceptive statements consisted of 2/3 truthful information and 1/3 embedded lies, largely derived from past personal experiences and with minimal linguistic differences with their truthful counterparts. Conclusion: We present this dataset as a novel resource to address this challenge and foster research on embedded lies in verbal deception detection.
• 摘要：背景：言语欺骗检测研究依赖于叙述，通常假设陈述是真实的或欺骗性的。更现实的观点认为，陈述的真实性存在于一个连续体中，真实和欺骗的部分嵌入在同一个陈述中。然而，对嵌入式谎言的研究一直落后。方法：我们收集了一个包含 2,088 个真实和欺骗性陈述的新数据集，其中包含带注释的嵌入式谎言。使用受试者内设计，参与者提供了自传事件的真实描述。然后，他们通过包含嵌入式谎言以欺骗的方式重写他们的陈述，之后他们会突出显示并根据谎言的中心性、欺骗性和来源进行判断。结果：我们表明，经过微调的语言模型 (Llama-3-8B) 可以以 64% 的准确率对真实陈述和包含嵌入式谎言的陈述进行分类。个体差异、语言特性和可解释性分析表明，将刻度盘转向嵌入式谎言的挑战源于它们与真实陈述的相似性。典型的欺骗性陈述由 2/3 的真实信息和 1/3 的隐性谎言组成，这些谎言主要来自过去的个人经历，与真实的谎言在语言上几乎没有区别。结论：我们将此数据集作为应对这一挑战的新资源，并促进对言语欺骗检测中隐性谎言的研究。

Title: The Lessons of Developing Process Reward Models in Mathematical Reasoning

• Authors: Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07301
• Pdf URL: https://arxiv.org/pdf/2501.07301
• Copy Paste: [[2501.07301]] The Lessons of Developing Process Reward Models in Mathematical Reasoning(https://arxiv.org/abs/2501.07301)
• Keywords: language model, llm
• Abstract: Process Reward Models (PRMs) emerge as a promising approach for process supervision in mathematical reasoning of Large Language Models (LLMs), which aim to identify and mitigate intermediate errors in the reasoning processes. However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies. In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods. MC estimation relies on completion models to evaluate current-step correctness, leading to inaccurate step verification. Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs: (1) The unreliable policy models generate responses with correct answers but flawed processes, leading to a misalignment between the evaluation criteria of BoN and the PRM objectives of process verification. (2) The tolerance of PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a significant proportion of minimum scores concentrated on the final answer steps, revealing the shift from process to outcome-based assessment in BoN Optimized PRMs. To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task. Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research in building process supervision models.
• 摘要：过程奖励模型 (PRM) 是大型语言模型 (LLM) 数学推理过程中监督的一种有前途的方法，旨在识别和减轻推理过程中的中间错误。然而，开发有效的 PRM 面临着重大挑战，特别是在数据注释和评估方法方面。在本文中，通过大量实验，我们证明，与 LLM 作为判断和人工注释方法相比，常用的基于蒙特卡洛 (MC) 估计的 PRM 数据合成通常会产生较差的性能和泛化。MC 估计依赖于完成模型来评估当前步骤的正确性，导致步骤验证不准确。此外，我们发现 PRM 的传统 Best-of-N (BoN) 评估策略中存在潜在偏差：(1) 不可靠的策略模型会生成具有正确答案但过程有缺陷的响应，导致 BoN 的评估标准与 PRM 过程验证目标不一致。(2) PRM 对此类响应的容忍度导致 BoN 分数虚高。 (3) 现有的 PRM 中，最低分数的很大一部分集中在最后的回答步骤上，这表明 BoN 优化的 PRM 正在从过程评估转向基于结果的评估。为了应对这些挑战，我们开发了一种共识过滤机制，有效地将 MC 估计与 LLM-as-a-judge 结合起来，并提倡一种结合响应级和步骤级指标的更全面的评估框架。基于这些机制，我们在 BoN 评估和分步错误识别任务中显著提高了模型性能和数据效率。最后，我们发布了一种新的最先进的 PRM，其性能优于现有的开源替代方案，并为未来构建过程监督模型的研究提供了实用指南。

Title: FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering

• Authors: Erik Henriksson, Otto Tarkka, Filip Ginter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07314
• Pdf URL: https://arxiv.org/pdf/2501.07314
• Copy Paste: [[2501.07314]] FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering(https://arxiv.org/abs/2501.07314)
• Keywords: language model, gpt, llm
• Abstract: Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25\% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.
• 摘要：数据质量对于训练大型语言模型 (LLM) 至关重要。传统的启发式过滤器经常会漏掉低质量的文本或错误地删除有价值的内容。在本文中，我们介绍了一种基于 LLM 的行级过滤方法来提高训练数据质量。我们使用 GPT-4o mini 在行级标记来自 FineWeb 的 20,000 篇文档样本，从而使模型能够为低质量的行创建描述性标签。这些标签分为九个主要类别，我们训练了一个 DeBERTa-v3 分类器，将过滤扩展到 FineWeb 的 10B 标记子集。为了测试过滤的影响，我们在原始数据集和过滤后的数据集上训练了 GPT-2 模型。结果表明，在过滤后的数据上训练的模型在 HellaSwag 基准上实现了更高的准确率，并且更快地达到了性能目标，即使数据量减少了 25\%。这表明基于 LLM 的行级过滤可以显著提高 LLM 的数据质量和训练效率。我们发布了质量注释数据集 FinerWeb-10BT 和代码库，以支持该领域的进一步工作。

Title: Emergent effects of scaling on the functional hierarchies within large language models

• Authors: Paul C. Bogdan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07359
• Pdf URL: https://arxiv.org/pdf/2501.07359
• Copy Paste: [[2501.07359]] Emergent effects of scaling on the functional hierarchies within large language models(https://arxiv.org/abs/2501.07359)
• Keywords: language model, llm
• Abstract: Large language model (LLM) architectures are often described as functionally hierarchical: Early layers process syntax, middle layers begin to parse semantics, and late layers integrate information. The present work revisits these ideas. This research submits simple texts to an LLM (e.g., "A church and organ") and extracts the resulting activations. Then, for each layer, support vector machines and ridge regressions are fit to predict a text's label and thus examine whether a given layer encodes some information. Analyses using a small model (Llama-3.2-3b; 28 layers) partly bolster the common hierarchical perspective: Item-level semantics are most strongly represented early (layers 2-7), then two-item relations (layers 8-12), and then four-item analogies (layers 10-15). Afterward, the representation of items and simple relations gradually decreases in deeper layers that focus on more global information. However, several findings run counter to a steady hierarchy view: First, although deep layers can represent document-wide abstractions, deep layers also compress information from early portions of the context window without meaningful abstraction. Second, when examining a larger model (Llama-3.3-70b-Instruct), stark fluctuations in abstraction level appear: As depth increases, two-item relations and four-item analogies initially increase in their representation, then markedly decrease, and afterward increase again momentarily. This peculiar pattern consistently emerges across several experiments. Third, another emergent effect of scaling is coordination between the attention mechanisms of adjacent layers. Across multiple experiments using the larger model, adjacent layers fluctuate between what information they each specialize in representing. In sum, an abstraction hierarchy often manifests across layers, but large models also deviate from this structure in curious ways.
• 摘要：大型语言模型 (LLM) 架构通常被描述为功能分层的：早期层处理语法，中间层开始解析语义，后期层整合信息。本研究重新审视了这些想法。本研究将简单文本提交给 LLM（例如，“教堂和风琴”）并提取生成的激活。然后，对于每一层，支持向量机和岭回归都适合预测文本的标签，从而检查给定层是否编码某些信息。使用小型模型（Llama-3.2-3b；28 层）进行的分析部分支持了常见的分层观点：项目级语义在早期（第 2-7 层）表现最强，然后是两项关系（第 8-12 层），然后是四项类比（第 10-15 层）。之后，项目和简单关系的表示在关注更多全局信息的更深层中逐渐减少。然而，有几项发现与稳定的层次结构观点背道而驰：首先，尽管深层可以表示文档范围的抽象，但深层也会压缩上下文窗口早期部分的信息，而没有进行有意义的抽象。其次，在检查更大的模型（Llama-3.3-70b-Instruct）时，抽象级别会出现明显的波动：随着深度的增加，两项关系和四项类比的表示最初会增加，然后显着减少，之后再次瞬间增加。这种特殊的模式在多个实验中始终如一地出现。第三，缩放的另一个新兴效应是相邻层的注意机制之间的协调。在使用较大模型的多个实验中，相邻层在它们各自擅长表示的信息之间波动。总之，抽象层次结构通常跨层体现，但大型模型也会以奇怪的方式偏离这种结构。

Title: Enhancing Retrieval-Augmented Generation: A Study of Best Practices

• Authors: Siran Li, Linus Stenzel, Carsten Eickhoff, Seyed Ali Bahrainian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07391
• Pdf URL: https://arxiv.org/pdf/2501.07391
• Copy Paste: [[2501.07391]] Enhancing Retrieval-Augmented Generation: A Study of Best Practices(https://arxiv.org/abs/2501.07391)
• Keywords: language model, prompt, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems have recently shown remarkable advancements by integrating retrieval mechanisms into language models, enhancing their ability to produce more accurate and contextually relevant responses. However, the influence of various components and configurations within RAG systems remains underexplored. A comprehensive understanding of these elements is essential for tailoring RAG systems to complex retrieval tasks and ensuring optimal performance across diverse applications. In this paper, we develop several advanced RAG system designs that incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG. Our study systematically investigates key factors, including language model size, prompt design, document chunk size, knowledge base size, retrieval stride, query expansion techniques, Contrastive In-Context Learning knowledge bases, multilingual knowledge bases, and Focus Mode retrieving relevant context at sentence-level. Through extensive experimentation, we provide a detailed analysis of how these factors influence response quality. Our findings offer actionable insights for developing RAG systems, striking a balance between contextual richness and retrieval-generation efficiency, thereby paving the way for more adaptable and high-performing RAG frameworks in diverse real-world scenarios. Our code and implementation details are publicly available.
• 摘要：检索增强生成 (RAG) 系统最近通过将检索机制集成到语言模型中，增强了其生成更准确、更符合语境的响应的能力，取得了显著的进步。然而，RAG 系统中各种组件和配置的影响仍未得到充分探索。全面了解这些元素对于根据复杂的检索任务定制 RAG 系统并确保在不同应用程序中实现最佳性能至关重要。在本文中，我们开发了几种先进的 RAG 系统设计，这些设计结合了查询扩展、各种新颖的检索策略和新颖的对比语境学习 RAG。我们的研究系统地调查了关键因素，包括语言模型大小、提示设计、文档块大小、知识库大小、检索步幅、查询扩展技术、对比语境学习知识库、多语言知识库和在句子级别检索相关上下文的焦点模式。通过大量实验，我们详细分析了这些因素如何影响响​​应质量。我们的研究结果为开发 RAG 系统提供了切实可行的见解，在语境丰富性和检索生成效率之间取得平衡，从而为在各种现实场景中实现更具适应性和高性能的 RAG 框架铺平了道路。我们的代码和实施细节已公开。

Title: TiEBe: A Benchmark for Assessing the Current Knowledge of Large Language Models

• Authors: Thales Sales Almeida, Giovana Kerche Bonás, João Guilherme Alves Santos, Hugo Abonizio, Rodrigo Nogueira
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07482
• Pdf URL: https://arxiv.org/pdf/2501.07482
• Copy Paste: [[2501.07482]] TiEBe: A Benchmark for Assessing the Current Knowledge of Large Language Models(https://arxiv.org/abs/2501.07482)
• Keywords: language model, llm
• Abstract: In a rapidly evolving knowledge landscape and the increasing adoption of large language models, a need has emerged to keep these models continuously updated with current events. While existing benchmarks evaluate general factual recall, they often overlook two critical aspects: the ability of models to integrate evolving knowledge through continual learning and the significant regional disparities in their performance. To address these gaps, we introduce the Timely Events Benchmark (TiEBe), a dataset containing over 11,000 question-answer pairs focused on globally and regionally significant events. TiEBe leverages structured retrospective data from Wikipedia, enabling continuous updates to assess LLMs' knowledge of evolving global affairs and their understanding of events across different regions. Our benchmark demonstrates that LLMs exhibit substantial geographic disparities in factual recall, emphasizing the need for more balanced global knowledge representation. Furthermore, TiEBe serves as a tool for evaluating continual learning strategies, providing insights into models' ability to acquire new information without forgetting past knowledge.
• 摘要：在快速发展的知识格局和日益普及的大型语言模型中，需要让这些模型不断更新以适应当前事件。虽然现有的基准测试评估的是一般的事实回忆能力，但它们往往忽略了两个关键方面：模型通过持续学习整合不断发展的知识的能力，以及其性能中显著的地区差异。为了解决这些差距，我们引入了及时事件基准 (TiEBe)，这是一个包含超过 11,000 个问答对的数据集，重点关注全球和地区重大事件。TiEBe 利用来自维基百科的结构化回顾性数据，实现持续更新，以评估 LLM 对不断发展的全球事务的了解以及他们对不同地区事件的理解。我们的基准测试表明，LLM 在事实回忆方面表现出巨大的地理差异，强调需要更平衡的全球知识表示。此外，TiEBe 可作为评估持续学习策略的工具，深入了解模型在不忘记过去知识的情况下获取新信息的能力。

Title: Investigating Large Language Models in Inferring Personality Traits from User Conversations

• Authors: Jianfeng Zhu, Ruoming Jin, Karin G. Coifman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07532
• Pdf URL: https://arxiv.org/pdf/2501.07532
• Copy Paste: [[2501.07532]] Investigating Large Language Models in Inferring Personality Traits from User Conversations(https://arxiv.org/abs/2501.07532)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) are demonstrating remarkable human like capabilities across diverse domains, including psychological assessment. This study evaluates whether LLMs, specifically GPT-4o and GPT-4o mini, can infer Big Five personality traits and generate Big Five Inventory-10 (BFI-10) item scores from user conversations under zero-shot prompting conditions. Our findings reveal that incorporating an intermediate step--prompting for BFI-10 item scores before calculating traits--enhances accuracy and aligns more closely with the gold standard than direct trait inference. This structured approach underscores the importance of leveraging psychological frameworks in improving predictive precision. Additionally, a group comparison based on depressive symptom presence revealed differential model performance. Participants were categorized into two groups: those experiencing at least one depressive symptom and those without symptoms. GPT-4o mini demonstrated heightened sensitivity to depression-related shifts in traits such as Neuroticism and Conscientiousness within the symptom-present group, whereas GPT-4o exhibited strengths in nuanced interpretation across groups. These findings underscore the potential of LLMs to analyze real-world psychological data effectively, offering a valuable foundation for interdisciplinary research at the intersection of artificial intelligence and psychology.
• 摘要：大型语言模型 (LLM) 在心理评估等不同领域都展现出了与人类相似的卓越能力。这项研究评估了 LLM（特别是 GPT-4o 和 GPT-4o mini）是否可以在零样本提示条件下从用户对话中推断出“大五人格特质”并生成“大五人格量表 10”（BFI-10）项目分数。我们的研究结果表明，加入一个中间步骤——在计算特质之前提示 BFI-10 项目分数——可以提高准确性，并且比直接特质推断更接近黄金标准。这种结构化方法强调了利用心理框架在提高预测精度方面的重要性。此外，基于抑郁症状存在的组比较揭示了不同的模型性能。参与者被分为两组：至少经历一种抑郁症状的人和没有症状的人。 GPT-4o mini 在有症状的群体中表现出对与抑郁相关的神经质和尽责性等特征变化的高度敏感性，而 GPT-4o 在跨群体细微解释方面表现出优势。这些发现强调了法学硕士有效分析现实世界心理数据的潜力，为人工智能和心理学交叉学科研究提供了宝贵的基础。

Title: Imagine while Reasoning in Space: Multimodal Visualization-of-Thought

• Authors: Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, Furu Wei
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07542
• Pdf URL: https://arxiv.org/pdf/2501.07542
• Copy Paste: [[2501.07542]] Imagine while Reasoning in Space: Multimodal Visualization-of-Thought(https://arxiv.org/abs/2501.07542)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Chain-of-Thought (CoT) prompting has proven highly effective for enhancing complex reasoning in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). Yet, it struggles in complex spatial reasoning tasks. Nonetheless, human cognition extends beyond language alone, enabling the remarkable capability to think in both words and images. Inspired by this mechanism, we propose a new reasoning paradigm, Multimodal Visualization-of-Thought (MVoT). It enables visual thinking in MLLMs by generating image visualizations of their reasoning traces. To ensure high-quality visualization, we introduce token discrepancy loss into autoregressive MLLMs. This innovation significantly improves both visual coherence and fidelity. We validate this approach through several dynamic spatial reasoning tasks. Experimental results reveal that MVoT demonstrates competitive performance across tasks. Moreover, it exhibits robust and reliable improvements in the most challenging scenarios where CoT fails. Ultimately, MVoT establishes new possibilities for complex reasoning tasks where visual thinking can effectively complement verbal reasoning.
• 摘要：思维链 (CoT) 提示已被证明可有效增强大型语言模型 (LLM) 和多模态大型语言模型 (MLLM) 中的复杂推理能力。然而，它在复杂的空间推理任务中却举步维艰。尽管如此，人类的认知能力不仅限于语言，它还具有非凡的用文字和图像思考的能力。受此机制的启发，我们提出了一种新的推理范式，即多模态思维可视化 (MVoT)。它通过生成 MLLM 推理轨迹的图像可视化来实现视觉思维。为了确保高质量的可视化，我们在自回归 MLLM 中引入了标记差异损失。这项创新显著提高了视觉连贯性和保真度。我们通过几个动态空间推理任务验证了这种方法。实验结果表明，MVoT 在各个任务中都表现出了竞争力。此外，在 CoT 失败的最具挑战性的场景中，它表现出稳健可靠的改进。最终，MVoT 为复杂的推理任务建立了新的可能性，其中视觉思维可以有效地补充语言推理。

Title: WebWalker: Benchmarking LLMs in Web Traversal

• Authors: Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, Fei Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07572
• Pdf URL: https://arxiv.org/pdf/2501.07572
• Copy Paste: [[2501.07572]] WebWalker: Benchmarking LLMs in Web Traversal(https://arxiv.org/abs/2501.07572)
• Keywords: llm, retrieval-augmented generation, agent
• Abstract: Retrieval-augmented generation (RAG) demonstrates remarkable performance across tasks in open-domain question-answering. However, traditional search engines may retrieve shallow content, limiting the ability of LLMs to handle complex, multi-layered information. To address it, we introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal. It evaluates the capacity of LLMs to traverse a website's subpages to extract high-quality data systematically. We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm. Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.
• 摘要：检索增强生成 (RAG) 在开放域问答中的各项任务中表现出色。然而，传统搜索引擎可能会检索浅层内容，从而限制了 LLM 处理复杂、多层信息的能力。为了解决这个问题，我们引入了 WebWalkerQA，这是一个旨在评估 LLM 执行 Web 遍历能力的基准。它评估 LLM 遍历网站子页面以系统地提取高质量数据的能力。我们提出了 WebWalker，这是一个多智能体框架，它通过探索-批评范式模仿类似人类的 Web 导航。大量的实验结果表明，WebWalkerQA 具有挑战性，并通过在现实场景中的横向和纵向集成证明了 RAG 与 WebWalker 相结合的有效性。