2025-01-03

Title: A Breadth-First Catalog of Text Processing, Speech Processing and Multimodal Research in South Asian Languages

• Authors: Pranav Gupta
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00029
• Pdf URL: https://arxiv.org/pdf/2501.00029
• Copy Paste: [[2501.00029]] A Breadth-First Catalog of Text Processing, Speech Processing and Multimodal Research in South Asian Languages(https://arxiv.org/abs/2501.00029)
• Keywords: language model, llm
• Abstract: We review the recent literature (January 2022- October 2024) in South Asian languages on text-based language processing, multimodal models, and speech processing, and provide a spotlight analysis focused on 21 low-resource South Asian languages, namely Saraiki, Assamese, Balochi, Bhojpuri, Bodo, Burmese, Chhattisgarhi, Dhivehi, Gujarati, Kannada, Kashmiri, Konkani, Khasi, Malayalam, Meitei, Nepali, Odia, Pashto, Rajasthani, Sindhi, and Telugu. We identify trends, challenges, and future research directions, using a step-wise approach that incorporates relevance classification and clustering based on large language models (LLMs). Our goal is to provide a breadth-first overview of the recent developments in South Asian language technologies to NLP researchers interested in working with South Asian languages.
• 摘要：我们回顾了南亚语言中关于基于文本的语言处理、多模态模型和语音处理的最新文献（2022 年 1 月 - 2024 年 10 月），并重点分析了 21 种资源匮乏的南亚语言，即西莱基语、阿萨姆语、俾路支语、博杰普尔语、博多语、缅甸语、恰蒂斯加尔语、迪维希语、古吉拉特语、卡纳达语、克什米尔语、孔卡尼语、卡西语、马拉雅拉姆语、梅泰语、尼泊尔语、奥里亚语、普什图语、拉贾斯坦语、信德语和泰卢固语。我们使用一种结合了基于大型语言模型 (LLM) 的相关性分类和聚类的分步方法来确定趋势、挑战和未来的研究方向。我们的目标是为有兴趣研究南亚语言的 NLP 研究人员提供南亚语言技术最新发展的广度优先概述。

Title: Distilling Large Language Models for Efficient Clinical Information Extraction

• Authors: Karthik S. Vedula, Annika Gupta, Akshay Swaminathan, Ivan Lopez, Suhana Bedi, Nigam H. Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00031
• Pdf URL: https://arxiv.org/pdf/2501.00031
• Copy Paste: [[2501.00031]] Distilling Large Language Models for Efficient Clinical Information Extraction(https://arxiv.org/abs/2501.00031)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) excel at clinical information extraction but their computational demands limit practical deployment. Knowledge distillation--the process of transferring knowledge from larger to smaller models--offers a potential solution. We evaluate the performance of distilled BERT models, which are approximately 1,000 times smaller than modern LLMs, for clinical named entity recognition (NER) tasks. We leveraged state-of-the-art LLMs (Gemini and OpenAI models) and medical ontologies (RxNorm and SNOMED) as teacher labelers for medication, disease, and symptom extraction. We applied our approach to over 3,300 clinical notes spanning five publicly available datasets, comparing distilled BERT models against both their teacher labelers and BERT models fine-tuned on human labels. External validation was conducted using clinical notes from the MedAlign dataset. For disease extraction, F1 scores were 0.82 (teacher model), 0.89 (BioBERT trained on human labels), and 0.84 (BioBERT-distilled). For medication, F1 scores were 0.84 (teacher model), 0.91 (BioBERT-human), and 0.87 (BioBERT-distilled). For symptoms: F1 score of 0.73 (teacher model) and 0.68 (BioBERT-distilled). Distilled BERT models had faster inference (12x, 4x, 8x faster than GPT-4o, o1-mini, and Gemini Flash respectively) and lower costs (85x, 101x, 2x cheaper than GPT-4o, o1-mini, and Gemini Flash respectively). On the external validation dataset, the distilled BERT model achieved F1 scores of 0.883 (medication), 0.726 (disease), and 0.699 (symptom). Distilled BERT models were up to 101x cheaper and 12x faster than state-of-the-art LLMs while achieving similar performance on NER tasks. Distillation offers a computationally efficient and scalable alternative to large LLMs for clinical information extraction.
• 摘要：大型语言模型 (LLM) 擅长提取临床信息，但其计算需求限制了实际部署。知识蒸馏——将知识从较大模型转移到较小模型的过程——提供了一种潜在的解决方案。我们评估了蒸馏后的 BERT 模型在临床命名实体识别 (NER) 任务中的表现，这些模型比现代 LLM 小约 1,000 倍。我们利用最先进的 LLM（Gemini 和 OpenAI 模型）和医学本体（RxNorm 和 SNOMED）作为药物、疾病和症状提取的教师标记器。我们将我们的方法应用于五个公开数据集中的 3,300 多份临床笔记，将蒸馏后的 BERT 模型与其教师标记器和针对人工标签进行微调的 BERT 模型进行比较。使用 MedAlign 数据集中的临床笔记进行了外部验证。对于疾病提取，F1 得分分别为 0.82（教师模型）、0.89（在人类标签上训练的 BioBERT）和 0.84（BioBERT-distilled）。对于药物，F1 得分分别为 0.84（教师模型）、0.91（BioBERT-human）和 0.87（BioBERT-distilled）。对于症状：F1 得分为 0.73（教师模型）和 0.68（BioBERT-distilled）。蒸馏后的 BERT 模型具有更快的推理速度（分别比 GPT-4o、o1-mini 和 Gemini Flash 快 12 倍、4 倍、8 倍），成本更低（分别比 GPT-4o、o1-mini 和 Gemini Flash 便宜 85 倍、101 倍、2 倍）。在外部验证数据集上，蒸馏后的 BERT 模型的 F1 得分为 0.883（药物）、0.726（疾病）和 0.699（症状）。蒸馏后的 BERT 模型比最先进的 LLM 便宜 101 倍，速度快 12 倍，同时在 NER 任务上实现类似的性能。蒸馏为临床信息提取提供了一种计算效率高且可扩展的大型 LLM 替代方案。

Title: Seq2Seq Model-Based Chatbot with LSTM and Attention Mechanism for Enhanced User Interaction

• Authors: Lamya Benaddi, Charaf Ouaddi, Adnane Souha, Abdeslam Jakimi, Mohamed Rahouti, Mohammed Aledhari, Diogo Oliveira, Brahim Ouchao
• Subjects: cs.CL, cs.ET
• Abstract URL: https://arxiv.org/abs/2501.00049
• Pdf URL: https://arxiv.org/pdf/2501.00049
• Copy Paste: [[2501.00049]] Seq2Seq Model-Based Chatbot with LSTM and Attention Mechanism for Enhanced User Interaction(https://arxiv.org/abs/2501.00049)
• Keywords: gpt, chat
• Abstract: A chatbot is an intelligent software application that automates conversations and engages users in natural language through messaging platforms. Leveraging artificial intelligence (AI), chatbots serve various functions, including customer service, information gathering, and casual conversation. Existing virtual assistant chatbots, such as ChatGPT and Gemini, demonstrate the potential of AI in Natural Language Processing (NLP). However, many current solutions rely on predefined APIs, which can result in vendor lock-in and high costs. To address these challenges, this work proposes a chatbot developed using a Sequence-to-Sequence (Seq2Seq) model with an encoder-decoder architecture that incorporates attention mechanisms and Long Short-Term Memory (LSTM) cells. By avoiding predefined APIs, this approach ensures flexibility and cost-effectiveness. The chatbot is trained, validated, and tested on a dataset specifically curated for the tourism sector in Draa-Tafilalet, Morocco. Key evaluation findings indicate that the proposed Seq2Seq model-based chatbot achieved high accuracies: approximately 99.58% in training, 98.03% in validation, and 94.12% in testing. These results demonstrate the chatbot's effectiveness in providing relevant and coherent responses within the tourism domain, highlighting the potential of specialized AI applications to enhance user experience and satisfaction in niche markets.
• 摘要：聊天机器人是一种智能软件应用程序，可通过消息传递平台自动进行对话并以自然语言与用户互动。聊天机器人利用人工智能 (AI) 提供各种功能，包括客户服务、信息收集和随意交谈。现有的虚拟助手聊天机器人（如 ChatGPT 和 Gemini）展示了人工智能在自然语言处理 (NLP) 中的潜力。然而，许多当前解决方案依赖于预定义的 API，这可能导致供应商锁定和高成本。为了应对这些挑战，这项工作提出了一种使用序列到序列 (Seq2Seq) 模型开发的聊天机器人，该模型具有编码器-解码器架构，该架构结合了注意力机制和长短期记忆 (LSTM) 单元。通过避免使用预定义的 API，这种方法可确保灵活性和成本效益。聊天机器人在专门为摩洛哥 Draa-Tafilalet 旅游业策划的数据集上进行训练、验证和测试。主要评估结果表明，所提出的基于 Seq2Seq 模型的聊天机器人实现了高准确率：训练准确率约为 99.58%，验证准确率约为 98.03%，测试准确率约为 94.12%。这些结果证明了聊天机器人在旅游领域提供相关且连贯的响应方面的有效性，凸显了专业 AI 应用程序在提升小众市场用户体验和满意度方面的潜力。

Title: Large Language Models for Mathematical Analysis

• Authors: Ziye Chen, Hao Qi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00059
• Pdf URL: https://arxiv.org/pdf/2501.00059
• Copy Paste: [[2501.00059]] Large Language Models for Mathematical Analysis(https://arxiv.org/abs/2501.00059)
• Keywords: language model, llm
• Abstract: Mathematical problem-solving is a key field in artificial intelligence (AI) and a critical benchmark for evaluating the capabilities of large language models (LLMs). While extensive research has focused on mathematical problem-solving, most existing work and datasets concentrate on computational tasks, leaving gaps in areas like mathematical analysis, which demands rigorous proofs and formal reasoning. We developed the DEMI-MathAnalysis dataset, comprising proof-based problems from mathematical analysis topics such as Sequences and Limits, Infinite Series, and Convex Functions. We also designed a guiding framework to rigorously enhance LLMs' ability to solve these problems. Through fine-tuning LLMs on this dataset and employing our framework, we observed significant improvements in their capability to generate logical, complete, and elegant proofs. This work addresses critical gaps in mathematical reasoning and contributes to advancing trustworthy AI capable of handling formalized mathematical language. The code is publicly accessible at LLMs for Mathematical Analysis.
• 摘要：数学问题解决是人工智能 (AI) 的一个关键领域，也是评估大型语言模型 (LLM) 能力的关键基准。虽然大量研究都集中在数学问题解决上，但大多数现有工作和数据集都集中在计算任务上，在数学分析等领域存在空白，数学分析需要严格的证明和形式推理。我们开发了 DEMI-MathAnalysis 数据集，其中包含来自数学分析主题（例如序列和极限、无穷级数和凸函数）的基于证明的问题。我们还设计了一个指导框架，以严格提高 LLM 解决这些问题的能力。通过在这个数据集上微调 LLM 并使用我们的框架，我们观察到它们生成逻辑、完整和优雅证明的能力有了显著提高。这项工作解决了数学推理中的关键差距，并有助于推进能够处理形式化数学语言的可信 AI。该代码可在 LLM for Mathematical Analysis 上公开访问。

Title: ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis

• Authors: James P. Beno
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00062
• Pdf URL: https://arxiv.org/pdf/2501.00062
• Copy Paste: [[2501.00062]] ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis(https://arxiv.org/abs/2501.00062)
• Keywords: language model, gpt, llm, prompt
• Abstract: Bidirectional transformers excel at sentiment analysis, and Large Language Models (LLM) are effective zero-shot learners. Might they perform better as a team? This paper explores collaborative approaches between ELECTRA and GPT-4o for three-way sentiment classification. We fine-tuned (FT) four models (ELECTRA Base/Large, GPT-4o/4o-mini) using a mix of reviews from Stanford Sentiment Treebank (SST) and DynaSent. We provided input from ELECTRA to GPT as: predicted label, probabilities, and retrieved examples. Sharing ELECTRA Base FT predictions with GPT-4o-mini significantly improved performance over either model alone (82.74 macro F1 vs. 79.29 ELECTRA Base FT, 79.52 GPT-4o-mini) and yielded the lowest cost/performance ratio (\$0.12/F1 point). However, when GPT models were fine-tuned, including predictions decreased performance. GPT-4o FT-M was the top performer (86.99), with GPT-4o-mini FT close behind (86.77) at much less cost (\$0.38 vs. \$1.59/F1 point). Our results show that augmenting prompts with predictions from fine-tuned encoders is an efficient way to boost performance, and a fine-tuned GPT-4o-mini is nearly as good as GPT-4o FT at 76% less cost. Both are affordable options for projects with limited resources.
• 摘要：双向 Transformer 在情绪分析方面表现出色，而大型语言模型 (LLM) 是有效的零样本学习器。他们作为一个团队会表现得更好吗？本文探讨了 ELECTRA 和 GPT-4o 在三向情绪分类中的协作方法。我们使用来自斯坦福情绪树库 (SST) 和 DynaSent 的混合评论对四个模型 (ELECTRA Base/Large、GPT-4o/4o-mini) 进行了微调 (FT)。我们将 ELECTRA 的输入提供给 GPT：预测标签、概率和检索到的示例。与单独使用任一模型相比，将 ELECTRA Base FT 预测与 GPT-4o-mini 共享可显着提高性能（82.74 宏 F1 vs. 79.29 ELECTRA Base FT、79.52 GPT-4o-mini）并获得最低的成本/性能比（\$0.12/F1 点）。但是，当对 GPT 模型进行微调时，包括预测会降低性能。 GPT-4o FT-M 表现最佳（86.99），GPT-4o-mini FT 紧随其后（86.77），但成本低得多（0.38 美元 vs. 1.59 美元/F1 点）。我们的结果表明，使用经过微调的编码器的预测来增强提示是提高性能的有效方法，经过微调的 GPT-4o-mini 几乎与 GPT-4o FT 一样好，但成本却低 76%。对于资源有限的项目来说，两者都是经济实惠的选择。

Title: On Adversarial Robustness of Language Models in Transfer Learning

• Authors: Bohdan Turbal, Anastasiia Mazur, Jiaxu Zhao, Mykola Pechenizkiy
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00066
• Pdf URL: https://arxiv.org/pdf/2501.00066
• Copy Paste: [[2501.00066]] On Adversarial Robustness of Language Models in Transfer Learning(https://arxiv.org/abs/2501.00066)
• Keywords: language model, gpt, llm
• Abstract: We investigate the adversarial robustness of LLMs in transfer learning scenarios. Through comprehensive experiments on multiple datasets (MBIB Hate Speech, MBIB Political Bias, MBIB Gender Bias) and various model architectures (BERT, RoBERTa, GPT-2, Gemma, Phi), we reveal that transfer learning, while improving standard performance metrics, often leads to increased vulnerability to adversarial attacks. Our findings demonstrate that larger models exhibit greater resilience to this phenomenon, suggesting a complex interplay between model size, architecture, and adaptation methods. Our work highlights the crucial need for considering adversarial robustness in transfer learning scenarios and provides insights into maintaining model security without compromising performance. These findings have significant implications for the development and deployment of LLMs in real-world applications where both performance and robustness are paramount.
• 摘要：我们研究了 LLM 在迁移学习场景中的对抗鲁棒性。通过对多个数据集（MBIB 仇恨言论、MBIB 政治偏见、MBIB 性别偏见）和各种模型架构（BERT、RoBERTa、GPT-2、Gemma、Phi）进行全面实验，我们发现迁移学习虽然可以提高标准性能指标，但往往会导致对抗攻击的脆弱性增加。我们的研究结果表明，较大的模型对这种现象表现出更大的弹性，这表明模型大小、架构和适应方法之间存在复杂的相互作用。我们的工作强调了在迁移学习场景中考虑对抗鲁棒性的迫切需要，并提供了在不影响性能的情况下保持模型安全性的见解。这些发现对于在性能和鲁棒性都至关重要的实际应用中开发和部署 LLM 具有重要意义。

Title: Adversarial Negotiation Dynamics in Generative Language Models

• Authors: Arinbjörn Kolbeinsson, Benedikt Kolbeinsson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00069
• Pdf URL: https://arxiv.org/pdf/2501.00069
• Copy Paste: [[2501.00069]] Adversarial Negotiation Dynamics in Generative Language Models(https://arxiv.org/abs/2501.00069)
• Keywords: language model
• Abstract: Generative language models are increasingly used for contract drafting and enhancement, creating a scenario where competing parties deploy different language models against each other. This introduces not only a game-theory challenge but also significant concerns related to AI safety and security, as the language model employed by the opposing party can be unknown. These competitive interactions can be seen as adversarial testing grounds, where models are effectively red-teamed to expose vulnerabilities such as generating biased, harmful or legally problematic text. Despite the importance of these challenges, the competitive robustness and safety of these models in adversarial settings remain poorly understood. In this small study, we approach this problem by evaluating the performance and vulnerabilities of major open-source language models in head-to-head competitions, simulating real-world contract negotiations. We further explore how these adversarial interactions can reveal potential risks, informing the development of more secure and reliable models. Our findings contribute to the growing body of research on AI safety, offering insights into model selection and optimisation in competitive legal contexts and providing actionable strategies for mitigating risks.
• 摘要：生成语言模型越来越多地用于合同起草和增强，从而形成了一种竞争双方部署不同语言模型的场景。这不仅带来了博弈论挑战，还带来了与人工智能安全相关的重大担忧，因为对方使用的语言模型可能是未知的。这些竞争互动可以看作是对抗性试验场，模型被有效地红队化，以暴露漏洞，例如生成有偏见、有害或法律上有问题的文本。尽管这些挑战非常重要，但这些模型在对抗环境中的竞争稳健性和安全性仍然知之甚少。在这项小型研究中，我们通过评估主要开源语言模型在面对面竞争中的表现和漏洞来解决这个问题，模拟现实世界的合同谈判。我们进一步探讨了这些对抗性互动如何揭示潜在风险，为开发更安全、更可靠的模型提供信息。我们的研究结果为日益增多的人工智能安全研究做出了贡献，为竞争性法律背景下的模型选择和优化提供了见解，并提供了可行的风险缓解策略。

Title: ICLR: In-Context Learning of Representations

• Authors: Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, Hidenori Tanaka
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00070
• Pdf URL: https://arxiv.org/pdf/2501.00070
• Copy Paste: [[2501.00070]] ICLR: In-Context Learning of Representations(https://arxiv.org/abs/2501.00070)
• Keywords: language model, llm
• Abstract: Recent work has demonstrated that semantics specified by pretraining data influence how representations of different concepts are organized in a large language model (LLM). However, given the open-ended nature of LLMs, e.g., their ability to in-context learn, we can ask whether models alter these pretraining semantics to adopt alternative, context-specified ones. Specifically, if we provide in-context exemplars wherein a concept plays a different role than what the pretraining data suggests, do models reorganize their representations in accordance with these novel semantics? To answer this question, we take inspiration from the theory of conceptual role semantics and define a toy "graph tracing" task wherein the nodes of the graph are referenced via concepts seen during training (e.g., apple, bird, etc.) and the connectivity of the graph is defined via some predefined structure (e.g., a square grid). Given exemplars that indicate traces of random walks on the graph, we analyze intermediate representations of the model and find that as the amount of context is scaled, there is a sudden re-organization from pretrained semantic representations to in-context representations aligned with the graph structure. Further, we find that when reference concepts have correlations in their semantics (e.g., Monday, Tuesday, etc.), the context-specified graph structure is still present in the representations, but is unable to dominate the pretrained structure. To explain these results, we analogize our task to energy minimization for a predefined graph topology, providing evidence towards an implicit optimization process to infer context-specified semantics. Overall, our findings indicate scaling context-size can flexibly re-organize model representations, possibly unlocking novel capabilities.
• 摘要：最近的研究表明，预训练数据指定的语义会影响大型语言模型 (LLM) 中不同概念的表示的组织方式。然而，鉴于 LLM 的开放性，例如其在上下文中学习的能力，我们可以问模型是否会改变这些预训练语义以采用替代的、上下文指定的语义。具体来说，如果我们提供上下文样本，其中概念扮演的角色与预训练数据所暗示的角色不同，模型是否会根据这些新语义重新组织其表示？为了回答这个问题，我们从概念角色语义理论中汲取灵感，并定义了一个玩具“图形跟踪”任务，其中图形的节点通过训练期间看到的概念来引用（例如苹果、鸟等），图形的连接性通过一些预定义的结构（例如方形网格）来定义。给定表示图上随机游走痕迹的样本，我们分析了模型的中间表示，发现随着上下文数量的缩放，预训练的语义表示会突然重新组织为与图结构一致的上下文表示。此外，我们发现，当参考概念的语义具有相关性（例如星期一、星期二等）时，上下文指定的图结构仍然存在于表示中，但无法主导预训练的结构。为了解释这些结果，我们将我们的任务类比为预定义图拓扑的能量最小化，为推断上下文指定语义的隐式优化过程提供了证据。总体而言，我们的研究结果表明，缩放上下文大小可以灵活地重新组织模型表示，可能释放新功能。

Title: CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions

• Authors: Mourad Heddaya, Kyle MacMillan, Anup Malani, Hongyuan Mei, Chenhao Tan
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00097
• Pdf URL: https://arxiv.org/pdf/2501.00097
• Copy Paste: [[2501.00097]] CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions(https://arxiv.org/abs/2501.00097)
• Keywords: gpt, llm, hallucination
• Abstract: This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as "syllabuses." Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815. We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. Further, we find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains. CaseSumm is available at this https URL
• 摘要：本文介绍了 CaseSumm，这是法律领域中用于长上下文摘要的新型数据集，它解决了摘要评估对更长、更复杂的数据集的需求。我们收集了 25.6K 美国最高法院 (SCOTUS) 意见及其官方摘要，称为“大纲”。我们的数据集是最大的开放法律案例摘要数据集，也是第一个包含可追溯到 1815 年的 SCOTUS 判决摘要的数据集。我们还使用自动指标和专家人工评估对 LLM 生成的摘要进行了全面评估，揭示了这些评估方法之间的差异。我们的评估表明，较小的开源模型 Mistral 7b 在大多数自动指标上的表现优于较大的模型，并成功生成了类似大纲的摘要。相比之下，人类专家注释者表示 Mistral 摘要包含幻觉。注释者一致认为 GPT-4 摘要更清晰，并且表现出更高的敏感性和特异性。此外，我们发现基于 LLM 的评估与人工评估的相关性并不比传统的自动指标更高。此外，我们的分析还发现生成的摘要中存在特定的幻觉，包括先例引用错误和对案件事实的歪曲陈述。这些发现证明了当前法律摘要自动评估方法的局限性，并强调了人工评估在评估摘要质量方面的关键作用，特别是在复杂、高风险领域。CaseSumm 可在此 https URL 上找到

Title: Temporal reasoning for timeline summarisation in social media

• Authors: Jiayu Song, Mahmud Akhter, Dana Atzil Slonim, Maria Liakata
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00152
• Pdf URL: https://arxiv.org/pdf/2501.00152
• Copy Paste: [[2501.00152]] Temporal reasoning for timeline summarisation in social media(https://arxiv.org/abs/2501.00152)
• Keywords: language model, llm
• Abstract: This paper explores whether enhancing temporal reasoning capabilities in Large Language Models (LLMs) can improve the quality of timeline summarization, the task of summarising long texts containing sequences of events, particularly social media threads . We introduce \textit{NarrativeReason}, a novel dataset focused on temporal relationships among sequential events within narratives, distinguishing it from existing temporal reasoning datasets that primarily address pair-wise event relationships. Our approach then combines temporal reasoning with timeline summarization through a knowledge distillation framework, where we first fine-tune a teacher model on temporal reasoning tasks and then distill this knowledge into a student model while simultaneously training it for the task of timeline summarization. Experimental results demonstrate that our model achieves superior performance on mental health-related timeline summarization tasks, which involve long social media threads with repetitions of events and a mix of emotions, highlighting the importance of leveraging temporal reasoning to improve timeline summarisation.
• 摘要：本文探讨了增强大型语言模型 (LLM) 中的时间推理能力是否可以提高时间线摘要的质量，时间线摘要是摘要包含事件序列（尤其是社交媒体线程）的长文本的任务。我们引入了 \textit{NarrativeReason}，这是一个新颖的数据集，专注于叙述中连续事件之间的时间关系，将其与主要处理成对事件关系的现有时间推理数据集区分开来。然后，我们的方法通过知识提炼框架将时间推理与时间线摘要相结合，我们首先在时间推理任务上对教师模型进行微调，然后将这些知识提炼到学生模型中，同时对其进行训练以完成时间线摘要任务。实验结果表明，我们的模型在心理健康相关的时间线摘要任务上取得了优异的表现，这些任务涉及长社交媒体线程，其中重复事件并混合了情绪，凸显了利用时间推理来改进时间线摘要的重要性。

Title: Measuring Large Language Models Capacity to Annotate Journalistic Sourcing

• Authors: Subramaniam Vincent, Phoebe Wang, Zhan Shi, Sahas Koka, Yi Fang
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.00164
• Pdf URL: https://arxiv.org/pdf/2501.00164
• Copy Paste: [[2501.00164]] Measuring Large Language Models Capacity to Annotate Journalistic Sourcing(https://arxiv.org/abs/2501.00164)
• Keywords: language model, gpt, llm, chat
• Abstract: Since the launch of ChatGPT in late 2022, the capacities of Large Language Models and their evaluation have been in constant discussion and evaluation both in academic research and in the industry. Scenarios and benchmarks have been developed in several areas such as law, medicine and math (Bommasani et al., 2023) and there is continuous evaluation of model variants. One area that has not received sufficient scenario development attention is journalism, and in particular journalistic sourcing and ethics. Journalism is a crucial truth-determination function in democracy (Vincent, 2023), and sourcing is a crucial pillar to all original journalistic output. Evaluating the capacities of LLMs to annotate stories for the different signals of sourcing and how reporters justify them is a crucial scenario that warrants a benchmark approach. It offers potential to build automated systems to contrast more transparent and ethically rigorous forms of journalism with everyday fare. In this paper we lay out a scenario to evaluate LLM performance on identifying and annotating sourcing in news stories on a five-category schema inspired from journalism studies (Gans, 2004). We offer the use case, our dataset and metrics and as the first step towards systematic benchmarking. Our accuracy findings indicate LLM-based approaches have more catching to do in identifying all the sourced statements in a story, and equally, in matching the type of sources. An even harder task is spotting source justifications.
• 摘要：自 2022 年底推出 ChatGPT 以来，学术研究和业界一直在不断讨论和评估大型语言模型及其评估的能力。在法律、医学和数学等多个领域已经开发了场景和基准（Bommasani 等人，2023 年），并且模型变体也在不断评估。一个尚未得到足够场景开发关注的领域是新闻业，尤其是新闻采编和道德。新闻业是民主中至关重要的真相确定功能（Vincent，2023 年），而采编是所有原创新闻作品的重要支柱。评估 LLM 为不同采编信号注释故事的能力以及记者如何证明这些信号是一个需要基准方法的关键场景。它提供了建立自动化系统的潜力，以将更透明、更符合道德规范的新闻形式与日常新闻进行对比。在本文中，我们提出了一个场景来评估 LLM 在新闻报道中识别和注释来源方面的表现，该场景基于受新闻研究启发的五类模式（Gans，2004）。我们提供用例、数据集和指标，作为系统基准测试的第一步。我们的准确性结果表明，基于 LLM 的方法在识别故事中的所有来源陈述方面以及在匹配来源类型方面有更大的作用。更艰巨的任务是发现来源依据。

Title: GPT-4 on Clinic Depression Assessment: An LLM-Based Pilot Study

• Authors: Giuliano Lorenzoni, Pedro Elkind Velmovitsky, Paulo Alencar, Donald Cowan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00199
• Pdf URL: https://arxiv.org/pdf/2501.00199
• Copy Paste: [[2501.00199]] GPT-4 on Clinic Depression Assessment: An LLM-Based Pilot Study(https://arxiv.org/abs/2501.00199)
• Keywords: language model, gpt, llm, prompt
• Abstract: Depression has impacted millions of people worldwide and has become one of the most prevalent mental disorders. Early mental disorder detection can lead to cost savings for public health agencies and avoid the onset of other major comorbidities. Additionally, the shortage of specialized personnel is a critical issue because clinical depression diagnosis is highly dependent on expert professionals and is time consuming. In this study, we explore the use of GPT-4 for clinical depression assessment based on transcript analysis. We examine the model's ability to classify patient interviews into binary categories: depressed and not depressed. A comparative analysis is conducted considering prompt complexity (e.g., using both simple and complex prompts) as well as varied temperature settings to assess the impact of prompt complexity and randomness on the model's performance. Results indicate that GPT-4 exhibits considerable variability in accuracy and F1-Score across configurations, with optimal performance observed at lower temperature values (0.0-0.2) for complex prompts. However, beyond a certain threshold (temperature >= 0.3), the relationship between randomness and performance becomes unpredictable, diminishing the gains from prompt complexity. These findings suggest that, while GPT-4 shows promise for clinical assessment, the configuration of the prompts and model parameters requires careful calibration to ensure consistent results. This preliminary study contributes to understanding the dynamics between prompt engineering and large language models, offering insights for future development of AI-powered tools in clinical settings.
• 摘要：抑郁症影响了全世界数百万人，已成为最常见的精神障碍之一。早期发现精神障碍可以为公共卫生机构节省成本，并避免其他主要合并症的发生。此外，专业人员的短缺是一个关键问题，因为临床抑郁症诊断高度依赖于专家，而且非常耗时。在本研究中，我们探索了基于转录分析使用 GPT-4 进行临床抑郁症评估。我们检查了该模型将患者访谈分为二元类别的能力：抑郁和非抑郁。考虑到提示的复杂性（例如，使用简单和复杂的提示）以及不同的温度设置，进行了比较分析，以评估提示复杂性和随机性对模型性能的影响。结果表明，GPT-4 在不同配置下的准确度和 F1 分数表现出相当大的差异，对于复杂提示，在较低的温度值（0.0-0.2）下观察到最佳性能。然而，超过某个阈值（温度 >= 0.3），随机性和性能之间的关系变得不可预测，从而降低了提示复杂性带来的收益。这些发现表明，尽管 GPT-4 在临床评估方面前景光明，但提示和模型参数的配置需要仔细校准才能确保结果一致。这项初步研究有助于理解提示工程和大型语言模型之间的动态关系，为未来在临床环境中开发人工智能工具提供见解。

Title: An Empirical Evaluation of Large Language Models on Consumer Health Questions

• Authors: Moaiz Abrar, Yusuf Sermet, Ibrahim Demir
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00208
• Pdf URL: https://arxiv.org/pdf/2501.00208
• Copy Paste: [[2501.00208]] An Empirical Evaluation of Large Language Models on Consumer Health Questions(https://arxiv.org/abs/2501.00208)
• Keywords: language model, gpt, llm
• Abstract: This study evaluates the performance of several Large Language Models (LLMs) on MedRedQA, a dataset of consumer-based medical questions and answers by verified experts extracted from the AskDocs subreddit. While LLMs have shown proficiency in clinical question answering (QA) benchmarks, their effectiveness on real-world, consumer-based, medical questions remains less understood. MedRedQA presents unique challenges, such as informal language and the need for precise responses suited to non-specialist queries. To assess model performance, responses were generated using five LLMs: GPT-4o mini, Llama 3.1: 70B, Mistral-123B, Mistral-7B, and Gemini-Flash. A cross-evaluation method was used, where each model evaluated its responses as well as those of others to minimize bias. The results indicated that GPT-4o mini achieved the highest alignment with expert responses according to four out of the five models' judges, while Mistral-7B scored lowest according to three out of five models' judges. This study highlights the potential and limitations of current LLMs for consumer health medical question answering, indicating avenues for further development.
• 摘要：本研究评估了几种大型语言模型 (LLM) 在 MedRedQA 上的表现，MedRedQA 是一个由经过验证的专家从 AskDocs 子版块中提取的基于消费者的医疗问题和答案的数据集。虽然 LLM 在临床问答 (QA) 基准测试中表现出色，但它们在现实世界中基于消费者的医疗问题上的有效性仍不太为人所知。MedRedQA 带来了独特的挑战，例如非正式语言和需要针对非专业查询的精确响应。为了评估模型性能，使用五个 LLM 生成响应：GPT-4o mini、Llama 3.1：70B、Mistral-123B、Mistral-7B 和 Gemini-Flash。使用交叉评估方法，每个模型都评估自己的响应以及其他模型的响应，以尽量减少偏差。结果表明，根据五个模型中的四位评委，GPT-4o mini 与专家响应的一致性最高，而根据五个模型中的三位评委，Mistral-7B 得分最低。这项研究强调了当前法学硕士在消费者健康医疗问题解答方面的潜力和局限性，指出了进一步发展的途径。

Title: Extracting effective solutions hidden in large language models via generated comprehensive specialists: case studies in developing electronic devices

• Authors: Hikari Tomita, Nobuhiro Nakamura, Shoichi Ishida, Toshio Kamiya, Kei Terayama
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00224
• Pdf URL: https://arxiv.org/pdf/2501.00224
• Copy Paste: [[2501.00224]] Extracting effective solutions hidden in large language models via generated comprehensive specialists: case studies in developing electronic devices(https://arxiv.org/abs/2501.00224)
• Keywords: language model, llm, agent
• Abstract: Recently, many studies have increasingly explored the use of large language models (LLMs) to generate research ideas and scientific hypotheses. However, real-world research and development often require solving complex, interdisciplinary challenges where solutions may not be readily found through existing knowledge related to the problem. Therefore, it is desirable to leverage the vast, comprehensive knowledge of LLMs to generate effective, breakthrough solutions by integrating various perspectives from other disciplines. Here, we propose SELLM (Solution Enumeration via comprehensive List and LLM), a framework leveraging LLMs and structured guidance using MECE (Mutually Exclusive, Collectively Exhaustive) principles, such as International Patent Classification (IPC) and the periodic table of elements. SELLM systematically constructs comprehensive expert agents from the list to generate cross-disciplinary and effective solutions. To evaluate SELLM's practicality, we applied it to two challenges: improving light extraction in organic light-emitting diode (OLED) lighting and developing electrodes for next-generation memory materials. The results demonstrate that SELLM significantly facilitates the generation of effective solutions compared to cases without specific customization or effort, showcasing the potential of SELLM to enable LLMs to generate effective solutions even for challenging problems.
• 摘要：最近，许多研究越来越多地探索使用大型语言模型 (LLM) 来生成研究思路和科学假设。然而，现实世界的研究和开发通常需要解决复杂的跨学科挑战，而解决方案可能无法通过与问题相关的现有知识轻易找到。因此，最好利用 LLM 的广泛、全面的知识，通过整合来自其他学科的各种观点来生成有效的突破性解决方案。在这里，我们提出了 SELLM（通过综合列表和 LLM 进行解决方案枚举），这是一个利用 LLM 和使用 MECE（相互排斥、集体穷尽）原则（例如国际专利分类 (IPC) 和元素周期表）的结构化指导的框架。SELLM 系统地从列表中构建全面的专家代理，以生成跨学科和有效的解决方案。为了评估 SELLM 的实用性，我们将其应用于两个挑战：改善有机发光二极管 (OLED) 照明中的光提取和开发下一代存储材料的电极。结果表明，与无需特定定制或努力的情况相比，SELLM 显著促进了有效解决方案的生成，展示了 SELLM 的潜力，使 LLM 能够为具有挑战性的问题生成有效的解决方案。

Title: Zero-Shot Strategies for Length-Controllable Summarization

• Authors: Fabian Retkowski, Alexander Waibel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00233
• Pdf URL: https://arxiv.org/pdf/2501.00233
• Copy Paste: [[2501.00233]] Zero-Shot Strategies for Length-Controllable Summarization(https://arxiv.org/abs/2501.00233)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) struggle with precise length control, particularly in zero-shot settings. We conduct a comprehensive study evaluating LLMs' length control capabilities across multiple measures and propose practical methods to improve controllability. Our experiments with LLaMA 3 reveal stark differences in length adherence across measures and highlight inherent biases of the model. To address these challenges, we introduce a set of methods: length approximation, target adjustment, sample filtering, and automated revisions. By combining these methods, we demonstrate substantial improvements in length compliance while maintaining or enhancing summary quality, providing highly effective zero-shot strategies for precise length control without the need for model fine-tuning or architectural changes. With our work, we not only advance our understanding of LLM behavior in controlled text generation but also pave the way for more reliable and adaptable summarization systems in real-world applications.
• 摘要：大型语言模型 (LLM) 难以进行精确的长度控制，尤其是在零样本设置中。我们进行了一项全面的研究，评估了 LLM 在多个测量中的长度控制能力，并提出了提高可控性的实用方法。我们对 LLaMA 3 的实验揭示了不同测量中长度遵守情况的明显差异，并突出了模型的固有偏差。为了应对这些挑战，我们引入了一组方法：长度近似、目标调整、样本过滤和自动修订。通过结合这些方法，我们在保持或提高摘要质量的同时，展示了长度合规性的显着改进，提供了高效的零样本策略，用于精确的长度控制，而无需对模型进行微调或架构更改。通过我们的工作，我们不仅加深了对受控文本生成中 LLM 行为的理解，而且为实际应用中更可靠、适应性更强的摘要系统铺平了道路。

Title: Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT

• Authors: Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00241
• Pdf URL: https://arxiv.org/pdf/2501.00241
• Copy Paste: [[2501.00241]] Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT(https://arxiv.org/abs/2501.00241)
• Keywords: language model
• Abstract: This study evaluates fine-tuning strategies for text classification using the DistilBERT model, specifically the distilbert-base-uncased-finetuned-sst-2-english variant. Through structured experiments, we examine the influence of hyperparameters such as learning rate, batch size, and epochs on accuracy, F1-score, and loss. Polynomial regression analyses capture foundational and incremental impacts of these hyperparameters, focusing on fine-tuning adjustments relative to a baseline model. Results reveal variability in metrics due to hyperparameter configurations, showing trade-offs among performance metrics. For example, a higher learning rate reduces loss in relative analysis (p=0.027) but challenges accuracy improvements. Meanwhile, batch size significantly impacts accuracy and F1-score in absolute regression (p=0.028 and p=0.005) but has limited influence on loss optimization (p=0.170). The interaction between epochs and batch size maximizes F1-score (p=0.001), underscoring the importance of hyperparameter interplay. These findings highlight the need for fine-tuning strategies addressing non-linear hyperparameter interactions to balance performance across metrics. Such variability and metric trade-offs are relevant for tasks beyond text classification, including NLP and computer vision. This analysis informs fine-tuning strategies for large language models and promotes adaptive designs for broader model applicability.
• 摘要：本研究使用 DistilBERT 模型（特别是 distilbert-base-uncased-finetuned-sst-2-english 变体）评估文本分类的微调策略。通过结构化实验，我们研究了学习率、批次大小和时期等超参数对准确率、F1 分数和损失的影响。多项式回归分析捕捉了这些超参数的基础和增量影响，重点关注相对于基线模型的微调调整。结果揭示了由于超参数配置导致的指标变化，显示了性能指标之间的权衡。例如，较高的学习率降低了相对分析中的损失（p=0.027），但对准确率的提高提出了挑战。同时，批次大小显着影响绝对回归中的准确率和 F1 分数（p=0.028 和 p=0.005），但对损失优化的影响有限（p=0.170）。时期和批次大小之间的相互作用使 F1 分数最大化（p=0.001），强调了超参数相互作用的重要性。这些发现强调了需要微调策略来解决非线性超参数相互作用以平衡指标之间的性能。这种可变性和指标权衡对于文本分类以外的任务（包括 NLP 和计算机视觉）也很重要。该分析为大型语言模型的微调策略提供了信息，并促进了自适应设计以实现更广泛的模型适用性。

Title: Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking

• Authors: Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Shaokai Chen, Mengshu Sun, Binbin Hu, Zhiqiang Zhang, Lei Liang, Wen Zhang, Huajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00244
• Pdf URL: https://arxiv.org/pdf/2501.00244
• Copy Paste: [[2501.00244]] Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking(https://arxiv.org/abs/2501.00244)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty.
• 摘要：在当前的 NLP 研究中，大型语言模型 (LLM) 在文本生成方面表现出色。然而，缺乏事实准确性仍然是悬在 LLM 摩天大楼上空的一片乌云。结构化知识提示 (SKP) 是一种突出的范例，它通过合并结构化表示将外部知识集成到 LLM 中，在许多知识密集型任务中取得了最先进的成果。然而，现有方法通常侧重于特定问题，缺乏对 SKP 的泛化和能力边界的全面探索。本文旨在从粒度、可转移性、可扩展性和通用性四个角度评估和重新思考 SKP 范式的泛化能力。为了进行全面的评估，我们引入了一个称为 SUBARU 的新型多粒度、多级别基准，由 9 个不同任务组成，具有不同的粒度和难度级别。

Title: EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta

• Authors: Raymond Bernard, Shaina Raza (PhD), Subhabrata Das (PhD), Rahul Murugan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00257
• Pdf URL: https://arxiv.org/pdf/2501.00257
• Copy Paste: [[2501.00257]] EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta(https://arxiv.org/abs/2501.00257)
• Keywords: language model, llm
• Abstract: Despite the remarkable coherence of Large Language Models (LLMs), existing evaluation methods often suffer from fluency bias and rely heavily on multiple-choice formats, making it difficult to assess factual accuracy and complex reasoning effectively. LLMs thus frequently generate factually inaccurate responses, especially in complex reasoning tasks, highlighting two prominent challenges: (1) the inadequacy of existing methods to evaluate reasoning and factual accuracy effectively, and (2) the reliance on human evaluators for nuanced judgment, as illustrated by Williams and Huckle (2024)[1], who found manual grading indispensable despite automated grading advancements. To address evaluation gaps in open-ended reasoning tasks, we introduce the EQUATOR Evaluator (Evaluation of Question Answering Thoroughness in Open-ended Reasoning). This framework combines deterministic scoring with a focus on factual accuracy and robust reasoning assessment. Using a vector database, EQUATOR pairs open-ended questions with human-evaluated answers, enabling more precise and scalable evaluations. In practice, EQUATOR significantly reduces reliance on human evaluators for scoring and improves scalability compared to Williams and Huckle's (2004)[1] methods. Our results demonstrate that this framework significantly outperforms traditional multiple-choice evaluations while maintaining high accuracy standards. Additionally, we introduce an automated evaluation process leveraging smaller, locally hosted LLMs. We used LLaMA 3.2B, running on the Ollama binaries to streamline our assessments. This work establishes a new paradigm for evaluating LLM performance, emphasizing factual accuracy and reasoning ability, and provides a robust methodological foundation for future research.
• 摘要：尽管大型语言模型 (LLM) 具有出色的连贯性，但现有的评估方法往往存在流畅性偏差，并且严重依赖多项选择题格式，因此很难有效地评估事实准确性和复杂推理。因此，LLM 经常会产生事实不准确的答案，尤其是在复杂的推理任务中，这凸显了两个突出的挑战：(1) 现有方法不足以有效地评估推理和事实准确性，(2) 依赖人类评估者进行细微判断，正如 Williams 和 Huckle (2024)[1] 所指出的那样，他们发现尽管自动评分取得了进步，但人工评分仍然不可或缺。为了解决开放式推理任务中的评估差距，我们引入了 EQUATOR 评估器（开放式推理中问题回答彻底性的评估）。该框架将确定性评分与对事实准确性和稳健推理评估的关注相结合。使用矢量数据库，EQUATOR 将开放式问题与人工评估的答案配对，从而实现更精确和可扩展的评估。在实践中，与 Williams 和 Huckle (2004)[1] 的方法相比，EQUATOR 显著减少了对人工评估员的依赖，并提高了可扩展性。我们的结果表明，该框架在保持高准确度标准的同时，表现明显优于传统的多项选择评估。此外，我们还引入了一个利用较小的本地托管 LLM 的自动评估流程。我们使用在 Ollama 二进制文件上运行的 LLaMA 3.2B 来简化我们的评估。这项工作建立了一种评估 LLM 性能的新范式，强调事实准确性和推理能力，并为未来的研究提供了坚实的方法基础。

Title: A review of faithfulness metrics for hallucination assessment in Large Language Models

• Authors: Ben Malin, Tatiana Kalganova, Nikoloas Boulgouris
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00269
• Pdf URL: https://arxiv.org/pdf/2501.00269
• Copy Paste: [[2501.00269]] A review of faithfulness metrics for hallucination assessment in Large Language Models(https://arxiv.org/abs/2501.00269)
• Keywords: language model, llm, hallucination, prompt, retrieval augmented generation
• Abstract: This review examines the means with which faithfulness has been evaluated across open-ended summarization, question-answering and machine translation tasks. We find that the use of LLMs as a faithfulness evaluator is commonly the metric that is most highly correlated with human judgement. The means with which other studies have mitigated hallucinations is discussed, with both retrieval augmented generation (RAG) and prompting framework approaches having been linked with superior faithfulness, whilst other recommendations for mitigation are provided. Research into faithfulness is integral to the continued widespread use of LLMs, as unfaithful responses can pose major risks to many areas whereby LLMs would otherwise be suitable. Furthermore, evaluating open-ended generation provides a more comprehensive measure of LLM performance than commonly used multiple-choice benchmarking, which can help in advancing the trust that can be placed within LLMs.
• 摘要：本评论研究了在开放式总结、问答和机器翻译任务中评估忠诚度的方法。我们发现，使用 LLM 作为忠诚度评估器通常是与人类判断最相关的指标。讨论了其他研究缓解幻觉的方法，其中检索增强生成 (RAG) 和提示框架方法都与卓越的忠诚度有关，同时提供了其他缓解建议。对忠诚度的研究对于 LLM 的继续广泛使用至关重要，因为不忠诚的回答会对许多原本适合 LLM 的领域构成重大风险。此外，与常用的多项选择基准测试相比，评估开放式生成提供了更全面的 LLM 性能衡量标准，这有助于提高对 LLM 的信任。

Title: Echoes in AI: Quantifying Lack of Plot Diversity in LLM Outputs

• Authors: Weijia Xu, Nebojsa Jojic, Sudha Rao, Chris Brockett, Bill Dolan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00273
• Pdf URL: https://arxiv.org/pdf/2501.00273
• Copy Paste: [[2501.00273]] Echoes in AI: Quantifying Lack of Plot Diversity in LLM Outputs(https://arxiv.org/abs/2501.00273)
• Keywords: language model, gpt, llm
• Abstract: With rapid advances in large language models (LLMs), there has been an increasing application of LLMs in creative content ideation and generation. A critical question emerges: can current LLMs provide ideas that are diverse enough to truly bolster the collective creativity? We examine two state-of-the-art LLMs, GPT-4 and LLaMA-3, on story generation and discover that LLM-generated stories often consist of plot elements that are echoed across a number of generations. To quantify this phenomenon, we introduce the Sui Generis score, which estimates how unlikely a plot element is to appear in alternative storylines generated by the same LLM. Evaluating on 100 short stories, we find that LLM-generated stories often contain combinations of idiosyncratic plot elements echoed frequently across generations, while the original human-written stories are rarely recreated or even echoed in pieces. Moreover, our human evaluation shows that the ranking of Sui Generis scores among story segments correlates moderately with human judgment of surprise level, even though score computation is completely automatic without relying on human judgment.
• 摘要：随着大型语言模型 (LLM) 的快速发展，LLM 在创意内容构思和生成中的应用越来越多。一个关键问题出现了：当前的 LLM 能否提供足够多样化的想法来真正增强集体创造力？我们研究了两个最先进的 LLM，GPT-4 和 LLaMA-3，用于故事生成，发现 LLM 生成的故事通常由几代人重复的故事情节元素组成。为了量化这种现象，我们引入了 Sui Generis 分数，它估计了情节元素出现在同一个 LLM 生成的替代故事情节中的可能性。通过对 100 篇短篇小说进行评估，我们发现 LLM 生成的故事通常包含几代人经常重复的独特情节元素组合，而原始的人类写作故事很少被重新创作，甚至很少被片段重复。此外，我们的人工评估表明，尽管分数计算是完全自动的，不依赖于人工判断，但故事片段中 Sui Generis 分数的排名与人工对惊讶程度的判断具有中等相关性。

Title: LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts

• Authors: Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, Chris Kedzie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00274
• Pdf URL: https://arxiv.org/pdf/2501.00274
• Copy Paste: [[2501.00274]] LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts(https://arxiv.org/abs/2501.00274)
• Keywords: language model, llm, prompt
• Abstract: This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges -- indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be $\textit{combined}$ to $\textit{predict}$ each human judge's annotations on all questions, including a summary question that assesses overall quality or relevance. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-Rubric with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges' assessment of overall user satisfaction, on a scale of 1--4, with RMS error $< 0.5$, a $2\times$ improvement over the uncalibrated baseline.
• 摘要：本文介绍了一种自动评估自然语言文本的框架。手动构建的评分标准描述了如何评估多个感兴趣的维度。为了评估文本，大型语言模型 (LLM) 会针对每个评分标准问题进行提示，并生成潜在答案的分布。LLM 预测通常与人类评委的预测不一致——事实上，人类并不完全同意彼此的预测。但是，可以将多个 LLM 分布合并起来，以预测每个人类评委对所有问题的注释，包括评估整体质量或相关性的总结问题。LLM-Rubric 通过训练一个小型前馈神经网络来实现这一点，该网络既包含评委特定参数，也包含评委独立参数。在评估人机信息搜索任务中的对话系统时，我们发现，包含 9 个问题（评估自然性、简洁性和引用质量等维度）的 LLM-Rubric 可以预测人类评委对整体用户满意度的评估（等级为 1--4），RMS 误差 $< 0.5$，比未校准的基线提高了 $2\times$。

Title: MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models

• Authors: Mahir Labib Dihan, Md Tanvir Hassan, Md Tanvir Parvez, Md Hasebul Hasan, Md Almash Alam, Muhammad Aamir Cheema, Mohammed Eunus Ali, Md Rizwan Parvez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00316
• Pdf URL: https://arxiv.org/pdf/2501.00316
• Copy Paste: [[2501.00316]] MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models(https://arxiv.org/abs/2501.00316)
• Keywords: gpt, llm, agent
• Abstract: Recent advancements in foundation models have enhanced AI systems' capabilities in autonomous tool usage and reasoning. However, their ability in location or map-based reasoning - which improves daily life by optimizing navigation, facilitating resource discovery, and streamlining logistics - has not been systematically studied. To bridge this gap, we introduce MapEval, a benchmark designed to assess diverse and complex map-based user queries with geo-spatial reasoning. MapEval features three task types (textual, API-based, and visual) that require collecting world information via map tools, processing heterogeneous geo-spatial contexts (e.g., named entities, travel distances, user reviews or ratings, images), and compositional reasoning, which all state-of-the-art foundation models find challenging. Comprising 700 unique multiple-choice questions about locations across 180 cities and 54 countries, MapEval evaluates foundation models' ability to handle spatial relationships, map infographics, travel planning, and navigation challenges. Using MapEval, we conducted a comprehensive evaluation of 28 prominent foundation models. While no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and Gemini-1.5-Pro achieved competitive performance overall. However, substantial performance gaps emerged, particularly in MapEval, where agents with Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%, respectively, and the gaps became even more amplified when compared to open-source LLMs. Our detailed analyses provide insights into the strengths and weaknesses of current models, though all models still fall short of human performance by more than 20% on average, struggling with complex map images and rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in advancing general-purpose foundation models with stronger geo-spatial understanding.
• 摘要：基础模型的最新进展增强了人工智能系统在自主使用工具和推理方面的能力。然而，它们在位置或基于地图的推理方面的能力尚未得到系统研究，而位置或基于地图的推理可通过优化导航、促进资源发现和简化物流来改善日常生活。为了弥补这一差距，我们引入了 MapEval，这是一个基准测试，旨在使用地理空间推理评估多样化和复杂的基于地图的用户查询。MapEval 具有三种任务类型（基于文本、基于 API 和视觉），需要通过地图工具收集世界信息、处理异构地理空间环境（例如命名实体、旅行距离、用户评论或评分、图像）和组合推理，所有最先进的基础模型都认为这些都具有挑战性。MapEval 包含 700 个独特的多项选择题，涉及 180 个城市和 54 个国家/地区的位置，评估基础模型处理空间关系、地图信息图、旅行计划和导航挑战的能力。使用 MapEval，我们对 28 个著名的基础模型进行了全面评估。虽然没有一个模型在所有任务上都表现出色，但 Claude-3.5-Sonnet、GPT-4o 和 Gemini-1.5-Pro 总体上取得了有竞争力的表现。然而，出现了巨大的性能差距，特别是在 MapEval 中，使用 Claude-3.5-Sonnet 的代理分别比 GPT-4o 和 Gemini-1.5-Pro 分别高出 16% 和 21%，与开源 LLM 相比，差距进一步扩大。我们的详细分析提供了对当前模型的优势和劣势的洞察，尽管所有模型的平均表现仍比人类低 20% 以上，难以处理复杂的地图图像和严格的地理空间推理。这一差距凸显了 MapEval 在推进具有更强地理空间理解能力的通用基础模型方面的关键作用。

Title: Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansion

• Authors: Hebin Wang, Yangning Li, Yinghui Li, Hai-Tao Zheng, Wenhao Jiang, Hong-Gee Kim
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.00330
• Pdf URL: https://arxiv.org/pdf/2501.00330
• Copy Paste: [[2501.00330]] Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansion(https://arxiv.org/abs/2501.00330)
• Keywords: language model, llm
• Abstract: The rapid development of multimodal large language models (MLLMs) has brought significant improvements to a wide range of tasks in real-world applications. However, LLMs still exhibit certain limitations in extracting implicit semantic information. In this paper, we apply MLLMs to the Multi-modal Entity Set Expansion (MESE) task, which aims to expand a handful of seed entities with new entities belonging to the same semantic class, and multi-modal information is provided with each entity. We explore the capabilities of MLLMs to understand implicit semantic information at the entity-level granularity through the MESE task, introducing a listwise ranking method LUSAR that maps local scores to global rankings. Our LUSAR demonstrates significant improvements in MLLM's performance on the MESE task, marking the first use of generative MLLM for ESE tasks and extending the applicability of listwise ranking.
• 摘要：多模态大型语言模型 (MLLM) 的快速发展为实际应用中的广泛任务带来了显着的改进。然而，LLM 在提取隐式语义信息方面仍然存在某些局限性。在本文中，我们将 MLLM 应用于多模态实体集扩展 (MESE) 任务，该任务旨在用属于同一语义类别的新实体来扩展少量种子实体，并且每个实体都提供多模态信息。我们通过 MESE 任务探索 MLLM 在实体级粒度理解隐式语义信息的能力，引入一种将局部分数映射到全局排名的列表排名方法 LUSAR。我们的 LUSAR 展示了 MLLM 在 MESE 任务上性能的显著提升，标志着生成式 MLLM 首次用于 ESE 任务并扩展了列表排名的适用性。

Title: MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation

• Authors: Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, Na Zou
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.00332
• Pdf URL: https://arxiv.org/pdf/2501.00332
• Copy Paste: [[2501.00332]] MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation(https://arxiv.org/abs/2501.00332)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: Large Language Models (LLMs) are becoming essential tools for various natural language processing tasks but often suffer from generating outdated or incorrect information. Retrieval-Augmented Generation (RAG) addresses this issue by incorporating external, real-time information retrieval to ground LLM responses. However, the existing RAG systems frequently struggle with the quality of retrieval documents, as irrelevant or noisy documents degrade performance, increase computational overhead, and undermine response reliability. To tackle this problem, we propose Multi-Agent Filtering Retrieval-Augmented Generation (MAIN-RAG), a training-free RAG framework that leverages multiple LLM agents to collaboratively filter and score retrieved documents. Specifically, MAIN-RAG introduces an adaptive filtering mechanism that dynamically adjusts the relevance filtering threshold based on score distributions, effectively minimizing noise while maintaining high recall of relevant documents. The proposed approach leverages inter-agent consensus to ensure robust document selection without requiring additional training data or fine-tuning. Experimental results across four QA benchmarks demonstrate that MAIN-RAG consistently outperforms traditional RAG approaches, achieving a 2-11% improvement in answer accuracy while reducing the number of irrelevant retrieved documents. Quantitative analysis further reveals that our approach achieves superior response consistency and answer accuracy over baseline methods, offering a competitive and practical alternative to training-based solutions.
• 摘要：大型语言模型 (LLM) 正在成为各种自然语言处理任务的必备工具，但经常会生成过时或不正确的信息。检索增强生成 (RAG) 通过将外部实时信息检索纳入 LLM 响应来解决此问题。然而，现有的 RAG 系统经常在检索文档的质量方面遇到困难，因为不相关或嘈杂的文档会降低性能、增加计算开销并破坏响应可靠性。为了解决这个问题，我们提出了多智能体过滤检索增强生成 (MAIN-RAG)，这是一个无需训练的 RAG 框架，利用多个 LLM 智能体协作过滤和评分检索到的文档。具体而言，MAIN-RAG 引入了一种自适应过滤机制，可根据分数分布动态调整相关性过滤阈值，有效地最大限度地减少噪音，同时保持相关文档的高召回率。所提出的方法利用智能体间共识来确保稳健的文档选择，而无需额外的训练数据或微调。四个 QA 基准的实验结果表明，MAIN-RAG 始终优于传统 RAG 方法，答案准确率提高了 2-11%，同时减少了不相关的检索文档数量。定量分析进一步表明，与基线方法相比，我们的方法实现了卓越的响应一致性和答案准确率，为基于训练的解决方案提供了一种具有竞争力且实用的替代方案。

Title: Loss-Aware Curriculum Learning for Chinese Grammatical Error Correction

• Authors: Ding Zhang, Yangning Li, Lichen Bai, Hao Zhang, Yinghui Li, Haiye Lin, Hai-Tao Zheng, Xin Su, Zifei Shan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00334
• Pdf URL: https://arxiv.org/pdf/2501.00334
• Copy Paste: [[2501.00334]] Loss-Aware Curriculum Learning for Chinese Grammatical Error Correction(https://arxiv.org/abs/2501.00334)
• Keywords: language model
• Abstract: Chinese grammatical error correction (CGEC) aims to detect and correct errors in the input Chinese sentences. Recently, Pre-trained Language Models (PLMS) have been employed to improve the performance. However, current approaches ignore that correction difficulty varies across different instances and treat these samples equally, enhancing the challenge of model learning. To address this problem, we propose a multi-granularity Curriculum Learning (CL) framework. Specifically, we first calculate the correction difficulty of these samples and feed them into the model from easy to hard batch by batch. Then Instance-Level CL is employed to help the model optimize in the appropriate direction automatically by regulating the loss function. Extensive experimental results and comprehensive analyses of various datasets prove the effectiveness of our method.
• 摘要：汉语语法纠错（CGEC）旨在检测和纠正输入汉语句子中的错误。最近，预训练语言模型（PLMS）已被用于提高性能。然而，当前的方法忽略了纠正难度在不同实例之间是不同的，并对这些样本一视同仁，这增加了模型学习的挑战性。为了解决这个问题，我们提出了一个多粒度的课程学习（CL）框架。具体来说，我们首先计算这些样本的纠正难度，并将它们从易到难逐批输入模型。然后采用实例级 CL，通过调节损失函数帮助模型自动朝适当的方向优化。大量的实验结果和对各种数据集的全面分析证明了我们方法的有效性。

Title: Rethinking Layer Removal: Preserving Critical Components with Task-Aware Singular Value Decomposition

• Authors: Kainan Liu, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, Jing Xiao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00339
• Pdf URL: https://arxiv.org/pdf/2501.00339
• Copy Paste: [[2501.00339]] Rethinking Layer Removal: Preserving Critical Components with Task-Aware Singular Value Decomposition(https://arxiv.org/abs/2501.00339)
• Keywords: language model, llm
• Abstract: Layer removal has emerged as a promising approach for compressing large language models (LLMs) by leveraging redundancy within layers to reduce model size and accelerate inference. However, this technique often compromises internal consistency, leading to performance degradation and instability, with varying impacts across different model architectures. In this work, we propose Taco-SVD, a task-aware framework that retains task-critical singular value directions, preserving internal consistency while enabling efficient compression. Unlike direct layer removal, Taco-SVD preserves task-critical transformations to mitigate performance degradation. By leveraging gradient-based attribution methods, Taco-SVD aligns singular values with downstream task objectives. Extensive evaluations demonstrate that Taco-SVD outperforms existing methods in perplexity and task performance across different architectures while ensuring minimal computational overhead.
• 摘要：通过利用层内的冗余来减小模型大小并加速推理，层移除已成为一种有前途的压缩大型语言模型 (LLM) 的方法。然而，这种技术通常会损害内部一致性，导致性能下降和不稳定，并且对不同的模型架构的影响也不同。在这项工作中，我们提出了 Taco-SVD，这是一个任务感知框架，它保留了任务关键型奇异值方向，在实现高效压缩的同时保持了内部一致性。与直接层移除不同，Taco-SVD 保留了任务关键型转换以减轻性能下降。通过利用基于梯度的归因方法，Taco-SVD 将奇异值与下游任务目标对齐。广泛的评估表明，Taco-SVD 在困惑度和任务性能方面优于不同架构中的现有方法，同时确保了最小的计算开销。

Title: Chunk-Distilled Language Modeling

• Authors: Yanhong Li, Karen Livescu, Jiawei Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00343
• Pdf URL: https://arxiv.org/pdf/2501.00343
• Copy Paste: [[2501.00343]] Chunk-Distilled Language Modeling(https://arxiv.org/abs/2501.00343)
• Keywords: language model, llm
• Abstract: We introduce Chunk-Distilled Language Modeling (CD-LM), an approach to text generation that addresses two challenges in current large language models (LLMs): the inefficiency of token-level generation, and the difficulty of adapting to new data and knowledge. Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step. Our retrieval framework enables flexible construction of model- or domain-specific datastores, either leveraging the internal knowledge of existing models, or incorporating expert insights from human-annotated corpora. This adaptability allows for enhanced control over the language model's distribution without necessitating additional training. We present the CD-LM formulation along with performance metrics demonstrating its ability to improve language model performance and efficiency across a diverse set of downstream tasks. Code and data will be made publicly available.
• 摘要：我们引入了块提炼语言模型 (CD-LM)，这是一种文本生成方法，它解决了当前大型语言模型 (LLM) 中的两个挑战：token 级生成的效率低下以及难以适应新数据和知识。我们的方法将基于深度网络的 LLM 与简单的检索模块相结合，允许在单个解码步骤中生成多 token 文本块。我们的检索框架可以灵活地构建模型或领域特定的数据存储，既可以利用现有模型的内部知识，也可以结合人工注释语料库的专家见解。这种适应性允许增强对语言模型分布的控制，而无需额外的训练。我们介绍了 CD-LM 公式以及性能指标，证明了它能够在各种下游任务中提高语言模型的性能和效率。代码和数据将公开。

Title: RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions

• Authors: Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen, Benyou Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00353
• Pdf URL: https://arxiv.org/pdf/2501.00353
• Copy Paste: [[2501.00353]] RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions(https://arxiv.org/abs/2501.00353)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a key paradigm for enhancing large language models (LLMs) by incorporating external knowledge. However, current RAG methods face two limitations: (1) they only cover limited RAG scenarios. (2) They suffer from limited task diversity due to the lack of a general RAG dataset. To address these limitations, we propose RAG-Instruct, a general method for synthesizing diverse and high-quality RAG instruction data based on any source corpus. Our approach leverages (1) five RAG paradigms, which encompass diverse query-document relationships, and (2) instruction simulation, which enhances instruction diversity and quality by utilizing the strengths of existing instruction datasets. Using this method, we construct a 40K instruction dataset from Wikipedia, comprehensively covering diverse RAG scenarios and tasks. Experiments demonstrate that RAG-Instruct effectively enhances LLMs' RAG capabilities, achieving strong zero-shot performance and significantly outperforming various RAG baselines across a diverse set of tasks. RAG-Instruct is publicly available at this https URL.
• 摘要：检索增强生成 (RAG) 已成为通过整合外部知识来增强大型语言模型 (LLM) 的关键范例。然而，当前的 RAG 方法面临两个限制：(1) 它们仅涵盖有限的 RAG 场景。(2) 由于缺乏通用的 RAG 数据集，它们的任务多样性有限。为了解决这些限制，我们提出了 RAG-Instruct，这是一种基于任何源语料库合成多样化和高质量 RAG 指令数据的通用方法。我们的方法利用 (1) 五种 RAG 范例，它们涵盖了不同的查询文档关系，以及 (2) 指令模拟，它通过利用现有指令数据集的优势来增强指令多样性和质量。使用此方法，我们从 Wikipedia 构建了一个 40K 指令数据集，全面涵盖了不同的 RAG 场景和任务。实验表明，RAG-Instruct 有效地增强了 LLM 的 RAG 功能，实现了强大的零样本性能，并且在各种任务中显著优于各种 RAG 基线。 RAG-Instruct 可通过此 https URL 公开获取。

Title: Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agents

• Authors: Chengbo He, Bochao Zou, Xin Li, Jiansheng Chen, Junliang Xing, Huimin Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00430
• Pdf URL: https://arxiv.org/pdf/2501.00430
• Copy Paste: [[2501.00430]] Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agents(https://arxiv.org/abs/2501.00430)
• Keywords: language model, llm, agent
• Abstract: Agents have demonstrated their potential in scientific reasoning tasks through large language models. However, they often face challenges such as insufficient accuracy and degeneration of thought when handling complex reasoning tasks, which impede their performance. To overcome these issues, we propose the Reactive and Reflection agents with Multi-Path Reasoning (RR-MP) Framework, aimed at enhancing the reasoning capabilities of LLMs. Our approach improves scientific reasoning accuracy by employing a multi-path reasoning mechanism where each path consists of a reactive agent and a reflection agent that collaborate to prevent degeneration of thought inherent in single-agent reliance. Additionally, the RR-MP framework does not require additional training; it utilizes multiple dialogue instances for each reasoning path and a separate summarizer to consolidate insights from all paths. This design integrates diverse perspectives and strengthens reasoning across each path. We conducted zero-shot and few-shot evaluations on tasks involving moral scenarios, college-level physics, and mathematics. Experimental results demonstrate that our method outperforms baseline approaches, highlighting the effectiveness and advantages of the RR-MP framework in managing complex scientific reasoning tasks.
• 摘要：代理已经通过大型语言模型展示了其在科学推理任务中的潜力。然而，它们在处理复杂的推理任务时经常面临准确性不足和思维退化等挑战，这阻碍了它们的表现。为了克服这些问题，我们提出了具有多路径推理 (RR-MP) 框架的反应和反射代理，旨在增强 LLM 的推理能力。我们的方法通过采用多路径推理机制来提高科学推理的准确性，其中每条路径由一个反应代理和一个反射代理组成，它们协作以防止单代理依赖所固有的思维退化。此外，RR-MP 框架不需要额外的训练；它为每个推理路径使用多个对话实例，并使用单独的总结器来整合来自所有路径的见解。这种设计整合了不同的观点并加强了每条路径的推理。我们对涉及道德场景、大学物理和数学的任务进行了零样本和少量样本评估。实验结果表明，我们的方法优于基线方法，突出了 RR-MP 框架在管理复杂科学推理任务方面的有效性和优势。

Title: TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment

• Authors: Ke Yang, Volodymyr Kindratenko, ChengXiang Zhai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00522
• Pdf URL: https://arxiv.org/pdf/2501.00522
• Copy Paste: [[2501.00522]] TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment(https://arxiv.org/abs/2501.00522)
• Keywords: language model, agent
• Abstract: Training language models (LMs) and their application agents is increasingly costly due to large datasets and models, making test failures difficult to bear. Simplified language environments serve as primordial training and testing grounds, retaining essential commonsense and communication skills but in a more digestible form, potentially enhancing the learning efficiency of LMs, and thus reducing the required model size and data volume for effective training and evaluation. In these simplified language environments, workable strategies for small models, datasets, and agents may be adaptable to larger models, datasets, and agents in complex language environments. To create such environments, we focus on two aspects: i) minimizing language dataset noise and complexity, and ii) preserving the essential text distribution characteristics. Unlike previous methods, we propose a pipeline to refine text data by eliminating noise, minimizing vocabulary, and maintaining genre-specific patterns (e.g., for books, conversation, code, etc.). Implementing this pipeline with large LMs, we have created a leaner suite of LM training and evaluation datasets: 71M Leaner-Pretrain, 7M Leaner-Instruct, Leaner-Glue for assessing linguistic proficiency, and Leaner-Eval for testing instruction-following ability. Our experiments show that leaner pre-training boosts LM learning efficiency. Tiny LMs trained on these datasets outperform those trained on original datasets in instruction-following across different language granularity levels. Moreover, the Leaner-Pretrain dataset's alignment with conventional large LM training sets enables resource-optimized analysis of how learning objectives, model architectures, and training techniques impact performance on language modeling and downstream tasks. Our code and datasets are available at this https URL.
• 摘要：由于数据集和模型庞大，训练语言模型 (LM) 及其应用代理的成本越来越高，测试失败令人难以承受。简化的语言环境可作为原始的训练和测试场地，以更易于理解的形式保留必要的常识和沟通技巧，从而有可能提高 LM 的学习效率，从而减少有效训练和评估所需的模型大小和数据量。在这些简化的语言环境中，适用于小型模型、数据集和代理的可行策略可能适用于复杂语言环境中的大型模型、数据集和代理。为了创建这样的环境，我们专注于两个方面：i) 最大限度地减少语言数据集的噪音和复杂性，以及 ii) 保留基本的文本分布特征。与以前的方法不同，我们提出了一种通过消除噪音、最小化词汇量和保持特定类型模式（例如书籍、对话、代码等）来细化文本数据的流程。通过使用大型 LM 实现此管道，我们创建了一套更精简的 LM 训练和评估数据集：71M Leaner-Pretrain、7M Leaner-Instruct、用于评估语言能力的 Leaner-Glue 和用于测试指令遵循能力的 Leaner-Eval。我们的实验表明，更精简的预训练可提高 LM 的学习效率。在这些数据集上训练的微型 LM 在不同语言粒度级别的指令遵循方面优于在原始数据集上训练的微型 LM。此外，Leaner-Pretrain 数据集与传统大型 LM 训练集的一致性使得能够对学习目标、模型架构和训练技术如何影响语言建模和下游任务的性能进行资源优化分析。我们的代码和数据集可在此 https URL 上找到。

Title: Superposition in Transformers: A Novel Way of Building Mixture of Experts

• Authors: Ayoub Ben Chaliah, Hela Dellagi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00530
• Pdf URL: https://arxiv.org/pdf/2501.00530
• Copy Paste: [[2501.00530]] Superposition in Transformers: A Novel Way of Building Mixture of Experts(https://arxiv.org/abs/2501.00530)
• Keywords: language model, llm
• Abstract: Catastrophic forgetting remains a major challenge when adapting large language models (LLMs) to new tasks or domains. Conventional fine-tuning often overwrites existing knowledge, causing performance degradation on original tasks. We introduce Superposition in Transformers, a novel architecture that leverages autoencoders to superimpose the hidden representations of a base model and a fine-tuned model within a shared parameter space. By using B-spline-based blending coefficients and autoencoders that adaptively reconstruct hidden states based on the input data distribution, our method effectively mitigates catastrophic forgetting and enables a new paradigm of "in-model" superposition. This approach preserves original model capabilities while allowing compact domain-specific expertise to be added, and it supports dynamic switching between model states during inference.
• 摘要：在将大型语言模型 (LLM) 应用于新任务或新领域时，灾难性遗忘仍然是一项重大挑战。传统的微调通常会覆盖现有知识，导致原始任务的性能下降。我们在 Transformers 中引入了叠加，这是一种新颖的架构，它利用自动编码器在共享参数空间内叠加基础模型和微调模型的隐藏表示。通过使用基于 B 样条的混合系数和基于输入数据分布自适应地重建隐藏状态的自动编码器，我们的方法有效地缓解了灾难性遗忘并实现了“模型内”叠加的新范式。这种方法保留了原始模型功能，同时允许添加紧凑的领域特定专业知识，并且它支持在推理过程中在模型状态之间动态切换。

Title: AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

• Authors: Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, Mariette Awad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00559
• Pdf URL: https://arxiv.org/pdf/2501.00559
• Copy Paste: [[2501.00559]] AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects(https://arxiv.org/abs/2501.00559)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities, not only in generating human-like text, but also in acquiring knowledge. This highlights the need to go beyond the typical Natural Language Processing downstream benchmarks and asses the various aspects of LLMs including knowledge and reasoning. Numerous benchmarks have been developed to evaluate LLMs knowledge, but they predominantly focus on the English language. Given that many LLMs are multilingual, relying solely on benchmarking English knowledge is insufficient. To address this issue, we introduce AraSTEM, a new Arabic multiple-choice question dataset aimed at evaluating LLMs knowledge in STEM subjects. The dataset spans a range of topics at different levels which requires models to demonstrate a deep understanding of scientific Arabic in order to achieve high accuracy. Our findings show that publicly available models of varying sizes struggle with this dataset, and underscores the need for more localized language models. The dataset is freely accessible on Hugging Face.
• 摘要：大型语言模型 (LLM) 不仅在生成类似人类的文本方面表现出非凡的能力，而且在获取知识方面也表现出非凡的能力。这凸显了超越典型的自然语言处理下游基准并评估 LLM 的各个方面（包括知识和推理）的必要性。已经开发了许多基准来评估 LLM 的知识，但它们主要侧重于英语。鉴于许多 LLM 都是多语言的，仅依靠对英语知识进行基准测试是不够的。为了解决这个问题，我们推出了 AraSTEM，这是一个新的阿拉伯语多项选择题数据集，旨在评估 LLM 在 STEM 学科方面的知识。该数据集涵盖了不同级别的一系列主题，这需要模型展示对科学阿拉伯语的深刻理解才能实现高精度。我们的研究结果表明，不同大小的公开模型都难以处理该数据集，并强调需要更多本地化的语言模型。该数据集可在 Hugging Face 上免费访问。

Title: Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

• Authors: Mingqi Gao, Yixin Liu, Xinyu Hu, Xiaojun Wan, Jonathan Bragg, Arman Cohan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00560
• Pdf URL: https://arxiv.org/pdf/2501.00560
• Copy Paste: [[2501.00560]] Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference(https://arxiv.org/abs/2501.00560)
• Keywords: llm
• Abstract: Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences. Due to the high cost and time-consuming nature of human evaluations, an automatic LLM bencher (i.e., an automatic evaluation framework that aims to rank LLMs based on their alignment with human preferences) is indispensable. An automatic LLM bencher consists of four components: the input set (e.g., a user instruction), the evaluation model (e.g., an LLM), the evaluation type (e.g., pairwise comparison), and the aggregation method (e.g., the ELO rating system). However, previous work has not thoroughly explored how to select these components or how their different combinations influence the results. In this work, through controlled experiments, we provide a series of recommendations on how to choose each component to better automate the evaluation of LLMs. Furthermore, we discovered that when evaluating LLMs with similar performance, the performance of the automatic LLM bencher declines sharply, underscoring the limitations of current benchers and calling for future work. Lastly, we found that the evaluation models' performance at the instance level (e.g., the accuracy of selecting the best output) does not always align with their effectiveness when used as a component of a bencher, highlighting the importance of dedicated system-level evaluation of benchers.
• 摘要：评估和排名不同 LLM 的功能对于了解其性能和与人类偏好的一致性至关重要。由于人工评估成本高且耗时，自动 LLM 评测器（即旨在根据 LLM 与人类偏好的一致性对其进行排名的自动评估框架）是必不可少的。自动 LLM 评测器由四个组件组成：输入集（例如，用户指令）、评估模型（例如，LLM）、评估类型（例如，成对比较）和聚合方法（例如，ELO 评分系统）。然而，以前的研究并没有彻底探索如何选择这些组件，或者它们的不同组合如何影响结果。在这项工作中，通过受控实验，我们提供了一系列关于如何选择每个组件以更好地自动化 LLM 评估的建议。此外，我们发现，在评估具有相似性能的 LLM 时，自动 LLM 评测器的性能急剧下降，这凸显了当前评测器的局限性，并呼吁未来的研究。最后，我们发现评估模型在实例级别的性能（例如，选择最佳输出的准确性）并不总是与其用作基准测试器组件时的有效性相符，这凸显了专门的系统级基准测试器评估的重要性。

Title: An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems

• Authors: Hashmath Shaik, Alex Doboli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00562
• Pdf URL: https://arxiv.org/pdf/2501.00562
• Copy Paste: [[2501.00562]] An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems(https://arxiv.org/abs/2501.00562)
• Keywords: language model, prompt, retrieval-augmented generation
• Abstract: Large Language Models offer new opportunities to devise automated implementation generation methods that can tackle problem solving activities beyond traditional methods, which require algorithmic specifications and can use only static domain knowledge, like performance metrics and libraries of basic building blocks. Large Language Models could support creating new methods to support problem solving activities for open-ended problems, like problem framing, exploring possible solving approaches, feature elaboration and combination, more advanced implementation assessment, and handling unexpected situations. This report summarized the current work on Large Language Models, including model prompting, Reinforcement Learning, and Retrieval-Augmented Generation. Future research requirements were also discussed.
• 摘要：大型语言模型为设计自动化实现生成方法提供了新的机会，这些方法可以解决传统方法之外的问题解决活动，而传统方法需要算法规范，并且只能使用静态领域知识，例如性能指标和基本构建块库。大型语言模型可以支持创建新方法来支持开放式问题的问题解决活动，例如问题框架、探索可能的解决方法、特征细化和组合、更高级的实现评估以及处理意外情况。本报告总结了大型语言模型的当前工作，包括模型提示、强化学习和检索增强生成。还讨论了未来的研究要求。

Title: Causal Graph Guided Steering of LLM Values via Prompts and Sparse Autoencoders

• Authors: Yipeng Kang, Junqi Wang, Yexin Li, Fangwei Zhong, Xue Feng, Mengmeng Wang, Wenming Tu, Quansen Wang, Hengli Li, Zilong Zheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00581
• Pdf URL: https://arxiv.org/pdf/2501.00581
• Copy Paste: [[2501.00581]] Causal Graph Guided Steering of LLM Values via Prompts and Sparse Autoencoders(https://arxiv.org/abs/2501.00581)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) become increasingly integrated into critical applications, aligning their behavior with human values presents significant challenges. Current methods, such as Reinforcement Learning from Human Feedback (RLHF), often focus on a limited set of values and can be resource-intensive. Furthermore, the correlation between values has been largely overlooked and remains underutilized. Our framework addresses this limitation by mining a causal graph that elucidates the implicit relationships among various values within the LLMs. Leveraging the causal graph, we implement two lightweight mechanisms for value steering: prompt template steering and Sparse Autoencoder feature steering, and analyze the effects of altering one value dimension on others. Extensive experiments conducted on Gemma-2B-IT and Llama3-8B-IT demonstrate the effectiveness and controllability of our steering methods.
• 摘要：随着大型语言模型 (LLM) 越来越多地融入到关键应用中，使其行为与人类价值观保持一致面临着重大挑战。当前的方法，例如从人类反馈中进行强化学习 (RLHF)，通常专注于一组有限的价值观，并且可能耗费大量资源。此外，价值观之间的相关性在很大程度上被忽视，并且未得到充分利用。我们的框架通过挖掘因果图来解决这一限制，该因果图阐明了 LLM 中各种价值观之间的隐含关系。利用因果图，我们实现了两种轻量级的价值引导机制：提示模板引导和稀疏自动编码器特征引导，并分析了改变一个价值维度对其他维度的影响。在 Gemma-2B-IT 和 Llama3-8B-IT 上进行的大量实验证明了我们的引导方法的有效性和可控性。

Title: Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation

• Authors: M. Ali Bayram, Ali Arda Fincan, Ahmet Semih G"um"uş, Banu Diri, Savaş Yıldırım, "Oner Aytaş
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00593
• Pdf URL: https://arxiv.org/pdf/2501.00593
• Copy Paste: [[2501.00593]] Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation(https://arxiv.org/abs/2501.00593)
• Keywords: language model, llm
• Abstract: Language models have made remarkable advancements in understanding and generating human language, achieving notable success across a wide array of applications. However, evaluating these models remains a significant challenge, particularly for resource-limited languages such as Turkish. To address this gap, we introduce the Turkish MMLU (TR-MMLU) benchmark, a comprehensive evaluation framework designed to assess the linguistic and conceptual capabilities of large language models (LLMs) in Turkish. TR-MMLU is constructed from a carefully curated dataset comprising 6200 multiple-choice questions across 62 sections, selected from a pool of 280000 questions spanning 67 disciplines and over 800 topics within the Turkish education system. This benchmark provides a transparent, reproducible, and culturally relevant tool for evaluating model performance. It serves as a standard framework for Turkish NLP research, enabling detailed analyses of LLMs' capabilities in processing Turkish text and fostering the development of more robust and accurate language models. In this study, we evaluate state-of-the-art LLMs on TR-MMLU, providing insights into their strengths and limitations for Turkish-specific tasks. Our findings reveal critical challenges, such as the impact of tokenization and fine-tuning strategies, and highlight areas for improvement in model design. By setting a new standard for evaluating Turkish language models, TR-MMLU aims to inspire future innovations and support the advancement of Turkish NLP research.
• 摘要：语言模型在理解和生成人类语言方面取得了显著进步，在广泛的应用中取得了显著的成功。然而，评估这些模型仍然是一项重大挑战，特别是对于资源有限的语言，如土耳其语。为了弥补这一差距，我们推出了土耳其语 MMLU (TR-MMLU) 基准，这是一个全面的评估框架，旨在评估土耳其语大型语言模型 (LLM) 的语言和概念能力。TR-MMLU 是基于一个精心策划的数据集构建的，该数据集包含 62 个部分的 6200 个多项选择题，是从土耳其教育系统中 67 个学科和 800 多个主题的 280000 个问题中选出的。该基准提供了一个透明、可重复且与文化相关的工具来评估模型性能。它是土耳其语 NLP 研究的标准框架，可以详细分析 LLM 处理土耳其语文本的能力，并促进开发更强大、更准确的语言模型。在本研究中，我们评估了 TR-MMLU 上最先进的 LLM，深入了解了它们在土耳其语特定任务中的优势和局限性。我们的研究结果揭示了关键挑战，例如标记化和微调策略的影响，并强调了模型设计中需要改进的地方。通过为评估土耳其语语言模型设定新标准，TR-MMLU 旨在激发未来的创新并支持土耳其语 NLP 研究的发展。

Title: Efficient Standardization of Clinical Notes using Large Language Models

• Authors: Daniel B. Hier, Michael D. Carrithers, Thanh Son Do, Tayo Obafemi-Ajayi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00644
• Pdf URL: https://arxiv.org/pdf/2501.00644
• Copy Paste: [[2501.00644]] Efficient Standardization of Clinical Notes using Large Language Models(https://arxiv.org/abs/2501.00644)
• Keywords: language model
• Abstract: Clinician notes are a rich source of patient information but often contain inconsistencies due to varied writing styles, colloquialisms, abbreviations, medical jargon, grammatical errors, and non-standard formatting. These inconsistencies hinder the extraction of meaningful data from electronic health records (EHRs), posing challenges for quality improvement, population health, precision medicine, decision support, and research. We present a large language model approach to standardizing a corpus of 1,618 clinical notes. Standardization corrected an average of $4.9 +/- 1.8$ grammatical errors, $3.3 +/- 5.2$ spelling errors, converted $3.1 +/- 3.0$ non-standard terms to standard terminology, and expanded $15.8 +/- 9.1$ abbreviations and acronyms per note. Additionally, notes were re-organized into canonical sections with standardized headings. This process prepared notes for key concept extraction, mapping to medical ontologies, and conversion to interoperable data formats such as FHIR. Expert review of randomly sampled notes found no significant data loss after standardization. This proof-of-concept study demonstrates that standardization of clinical notes can improve their readability, consistency, and usability, while also facilitating their conversion into interoperable data formats.
• 摘要：临床笔记是患者信息的丰富来源，但由于写作风格、口语、缩写、医学术语、语法错误和非标准格式各异，因此经常包含不一致之处。这些不一致之处阻碍了从电子健康记录 (EHR) 中提取有意义的数据，对质量改进、人口健康、精准医疗、决策支持和研究构成挑战。我们提出了一种大型语言模型方法来标准化 1,618 份临床笔记的语料库。标准化平均纠正了 $4.9 +/- 1.8$ 个语法错误、$3.3 +/- 5.2$ 个拼写错误，将 $3.1 +/- 3.0$ 个非标准术语转换为标准术语，并扩展了每份笔记 $15.8 +/- 9.1$ 个缩写和首字母缩略词。此外，笔记被重新组织成具有标准化标题的规范部分。此过程为笔记准备了关键概念提取、医学本体映射和转换为可互操作数据格式（如 FHIR）的准备工作。专家对随机抽样笔记的审查发现，标准化后没有出现重大数据丢失。这项概念验证研究表明，临床笔记的标准化可以提高其可读性、一致性和可用性，同时也有助于将其转换为可互操作的数据格式。

Title: 2 OLMo 2 Furious

• Authors: Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, Hannaneh Hajishirzi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00656
• Pdf URL: https://arxiv.org/pdf/2501.00656
• Copy Paste: [[2501.00656]] 2 OLMo 2 Furious(https://arxiv.org/abs/2501.00656)
• Keywords: language model
• Abstract: We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes dense autoregressive models with improved architecture and training recipe, pretraining data mixtures, and instruction tuning recipes. Our modified model architecture and training recipe achieve both better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from Tülu 3 to develop OLMo 2-Instruct, focusing on permissive data and extending our final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the Pareto frontier of performance to compute, often matching or outperforming open-weight only models like Llama 3.1 and Qwen 2.5 while using fewer FLOPs and with fully transparent training data, code, and recipe. Our fully open OLMo 2-Instruct models are competitive with or surpassing open-weight only models of comparable size, including Qwen 2.5, Llama 3.1 and Gemma 2. We release all OLMo 2 artifacts openly -- models at 7B and 13B scales, both pretrained and post-trained, including their full training data, training code and recipes, training logs and thousands of intermediate checkpoints. The final instruction model is available on the Ai2 Playground as a free research demo.
• 摘要：我们推出了 OLMo 2，这是我们完全开放的语言模型的下一代。OLMo 2 包括具有改进的架构和训练配方的密集自回归模型、预训练数据混合和指令调整配方。我们修改后的模型架构和训练配方实现了更好的训练稳定性和更高的每个 token 效率。我们更新的预训练数据混合引入了一种称为 Dolmino Mix 1124 的新型专用数据混合，当通过后期课程训练（即预训练退火阶段的专用数据）引入时，它可以显著提高许多下游任务基准的模型能力。最后，我们结合 Tülu 3 的最佳实践来开发 OLMo 2-Instruct，重点关注允许数据并通过可验证奖励 (RLVR) 扩展我们最后阶段的强化学习。我们的 OLMo 2 基础模型处于计算性能的帕累托前沿，通常与 Llama 3.1 和 Qwen 2.5 等开放权重模型相当甚至更胜一筹，同时使用更少的 FLOP 和完全透明的训练数据、代码和配方。我们完全开放的 OLMo 2-Instruct 模型可与 Qwen 2.5、Llama 3.1 和 Gemma 2 等同等大小的开放权重模型相媲美甚至超越它们。我们公开发布所有 OLMo 2 成果——7B 和 13B 规模的模型，包括预训练和后训练，包括其完整的训练数据、训练代码和配方、训练日志和数千个中间检查点。最终的指导模型可在 Ai2 Playground 上作为免费研究演示获得。

Title: Labels Generated by Large Language Model Helps Measuring People's Empathy in Vitro

• Authors: Md Rakibul Hasan, Yue Yao, Md Zakir Hossain, Aneesh Krishna, Imre Rudas, Shafin Rahman, Tom Gedeon
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00691
• Pdf URL: https://arxiv.org/pdf/2501.00691
• Copy Paste: [[2501.00691]] Labels Generated by Large Language Model Helps Measuring People's Empathy in Vitro(https://arxiv.org/abs/2501.00691)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have revolutionised numerous fields, with LLM-as-a-service (LLMSaaS) having a strong generalisation ability that offers accessible solutions directly without the need for costly training. In contrast to the widely studied prompt engineering for task solving directly (in vivo), this paper explores its potential in in-vitro applications. These involve using LLM to generate labels to help the supervised training of mainstream models by (1) noisy label correction and (2) training data augmentation with LLM-generated labels. In this paper, we evaluate this approach in the emerging field of empathy computing -- automating the prediction of psychological questionnaire outcomes from inputs like text sequences. Specifically, crowdsourced datasets in this domain often suffer from noisy labels that misrepresent underlying empathy. By leveraging LLM-generated labels to train pre-trained language models (PLMs) like RoBERTa, we achieve statistically significant accuracy improvements over baselines, achieving a state-of-the-art Pearson correlation coefficient of 0.648 on NewsEmp benchmarks. In addition, we bring insightful discussions, including current challenges in empathy computing, data biases in training data and evaluation metric selection. Code and LLM-generated data are available at this https URL (available once the paper is accepted).
• 摘要：大型语言模型 (LLM) 已经彻底改变了许多领域，其中 LLM 即服务 (LLMSaaS) 具有强大的泛化能力，可直接提供可访问的解决方案，而无需昂贵的训练。与广泛研究的直接（体内）任务解决的即时工程相比，本文探讨了其在体外应用中的潜力。这涉及使用 LLM 生成标签以帮助主流模型的监督训练，通过 (1) 噪声标签校正和 (2) 使用 LLM 生成的标签进行训练数据增强。在本文中，我们评估了这种方法在新兴的同理心计算领域中的应用——自动从文本序列等输入中预测心理问卷结果。具体而言，该领域的众包数据集经常受到噪声标签的影响，这些标签会歪曲潜在的同理心。通过利用 LLM 生成的标签来训练 RoBERTa 等预训练语言模型 (PLM)，我们在基准上实现了统计上显著的准确率提升，在 NewsEmp 基准上实现了 0.648 的皮尔逊相关系数。此外，我们还带来了富有洞察力的讨论，包括共情计算的当前挑战、训练数据中的数据偏差和评估指标选择。代码和 LLM 生成的数据可在此 https URL 上获取（论文被接受后即可获取）。

Title: PANDA -- Paired Anti-hate Narratives Dataset from Asia: Using an LLM-as-a-Judge to Create the First Chinese Counterspeech Dataset

• Authors: Michael Bennie, Demi Zhang, Bushi Xiao, Jing Cao, Chryseis Xinyi Liu, Jian Meng, Alayo Tripp
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00697
• Pdf URL: https://arxiv.org/pdf/2501.00697
• Copy Paste: [[2501.00697]] PANDA -- Paired Anti-hate Narratives Dataset from Asia: Using an LLM-as-a-Judge to Create the First Chinese Counterspeech Dataset(https://arxiv.org/abs/2501.00697)
• Keywords: llm
• Abstract: Despite the global prevalence of Modern Standard Chinese language, counterspeech (CS) resources for Chinese remain virtually nonexistent. To address this gap in East Asian counterspeech research we introduce the a corpus of Modern Standard Mandarin counterspeech that focuses on combating hate speech in Mainland China. This paper proposes a novel approach of generating CS by using an LLM-as-a-Judge, simulated annealing, LLMs zero-shot CN generation and a round-robin algorithm. This is followed by manual verification for quality and contextual relevance. This paper details the methodology for creating effective counterspeech in Chinese and other non-Eurocentric languages, including unique cultural patterns of which groups are maligned and linguistic patterns in what kinds of discourse markers are programmatically marked as hate speech (HS). Analysis of the generated corpora, we provide strong evidence for the lack of open-source, properly labeled Chinese hate speech data and the limitations of using an LLM-as-Judge to score possible answers in Chinese. Moreover, the present corpus serves as the first East Asian language based CS corpus and provides an essential resource for future research on counterspeech generation and evaluation.
• 摘要：尽管现代标准汉语在全球盛行，但针对中文的反语 (CS) 资源却几乎不存在。为了弥补东亚反语研究中的这一空白，我们引入了一个现代标准普通话反语语料库，该语料库专注于打击中国大陆的仇恨言论。本文提出了一种使用 LLM-as-a-Judge、模拟退火、LLM 零样本 CN 生成和循环算法生成 CS 的新方法。然后进行手动验证以确保质量和上下文相关性。本文详细介绍了在中文和其他非欧洲中心语言中创建有效反语的方法，包括哪些群体受到诽谤的独特文化模式以及哪些话语标记被程序标记为仇恨言论 (HS) 的语言模式。通过对生成的语料库进行分析，我们为缺乏开源、正确标记的中文仇恨言论数据以及使用 LLM-as-Judge 对中文可能答案进行评分的局限性提供了强有力的证据。此外，本语料库是第一个基于东亚语言的 CS 语料库，为未来的反语音生成和评估研究提供了重要资源。

Title: Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding

• Authors: Jiajun Zhu, Peihao Wang, Ruisi Cai, Jason D. Lee, Pan Li, Zhangyang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00712
• Pdf URL: https://arxiv.org/pdf/2501.00712
• Copy Paste: [[2501.00712]] Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding(https://arxiv.org/abs/2501.00712)
• Keywords: language model
• Abstract: Transformers rely on both content-based and position-based addressing mechanisms to make predictions, but existing positional encoding techniques often diminish the effectiveness of position-based addressing. Many current methods enforce rigid patterns in attention maps, limiting the ability to model long-range dependencies and adapt to diverse tasks. Additionally, most positional encodings are learned as general biases, lacking the specialization required for different instances within a dataset. To address this, we propose con$\textbf{T}$extualized equivari$\textbf{A}$nt $\textbf{P}$osition $\textbf{E}$mbedding ($\textbf{TAPE}$), a novel framework that enhances positional embeddings by incorporating sequence content across layers. TAPE introduces dynamic, context-aware positional encodings, overcoming the constraints of traditional fixed patterns. By enforcing permutation and orthogonal equivariance, TAPE ensures the stability of positional encodings during updates, improving robustness and adaptability. Our method can be easily integrated into pre-trained transformers, offering parameter-efficient fine-tuning with minimal overhead. Extensive experiments shows that TAPE achieves superior performance in language modeling, arithmetic reasoning, and long-context retrieval tasks compared to existing positional embedding techniques.
• 摘要：Transformer 依靠基于内容和基于位置的寻址机制进行预测，但现有的位置编码技术通常会降低基于位置的寻址的有效性。许多当前方法在注意力图中强制执行刚性模式，限制了对长距离依赖关系进行建模和适应各种任务的能力。此外，大多数位置编码都是作为一般偏差来学习的，缺乏数据集内不同实例所需的专业化。为了解决这个问题，我们提出了 con$\textbf{T}$extualized equivari$\textbf{A}$nt $\textbf{P}$osition $\textbf{E}$mbedding ($\textbf{TAPE}$)，这是一种通过跨层合并序列内容来增强位置嵌入的新框架。TAPE 引入了动态的、上下文感知的位置编码，克服了传统固定模式的限制。通过强制置换和正交等变性，TAPE 确保了更新过程中位置编码的稳定性，从而提高了鲁棒性和适应性。我们的方法可以轻松集成到预先训练的转换器中，以最小的开销提供参数高效的微调。大量实验表明，与现有的位置嵌入技术相比，TAPE 在语言建模、算术推理和长上下文检索任务中取得了卓越的性能。

Title: CODEOFCONDUCT at Multilingual Counterspeech Generation: A Context-Aware Model for Robust Counterspeech Generation in Low-Resource Languages

• Authors: Michael Bennie, Bushi Xiao, Chryseis Xinyi Liu, Demi Zhang, Jian Meng, Alayo Tripp
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00713
• Pdf URL: https://arxiv.org/pdf/2501.00713
• Copy Paste: [[2501.00713]] CODEOFCONDUCT at Multilingual Counterspeech Generation: A Context-Aware Model for Robust Counterspeech Generation in Low-Resource Languages(https://arxiv.org/abs/2501.00713)
• Keywords: llm
• Abstract: This paper introduces a context-aware model for robust counterspeech generation, which achieved significant success in the MCG-COLING-2025 shared task. Our approach particularly excelled in low-resource language settings. By leveraging a simulated annealing algorithm fine-tuned on multilingual datasets, the model generates factually accurate responses to hate speech. We demonstrate state-of-the-art performance across four languages (Basque, English, Italian, and Spanish), with our system ranking first for Basque, second for Italian, and third for both English and Spanish. Notably, our model swept all three top positions for Basque, highlighting its effectiveness in low-resource scenarios. Evaluation of the shared task employs both traditional metrics (BLEU, ROUGE, BERTScore, Novelty) and JudgeLM based on LLM. We present a detailed analysis of our results, including an empirical evaluation of the model performance and comprehensive score distributions across evaluation metrics. This work contributes to the growing body of research on multilingual counterspeech generation, offering insights into developing robust models that can adapt to diverse linguistic and cultural contexts in the fight against online hate speech.
• 摘要：本文介绍了一种用于生成稳健反言的上下文感知模型，该模型在 MCG-COLING-2025 共享任务中取得了重大成功。我们的方法在资源匮乏的语言环境中尤其出色。通过利用在多语言数据集上微调的模拟退火算法，该模型可以生成对仇恨言论的事实准确回应。我们在四种语言（巴斯克语、英语、意大利语和西班牙语）中展示了最先进的性能，我们的系统在巴斯克语中排名第一，在意大利语中排名第二，在英语和西班牙语中排名第三。值得注意的是，我们的模型包揽了巴斯克语的所有三个前三名，凸显了其在资源匮乏场景中的有效性。共享任务的评估采用了传统指标（BLEU、ROUGE、BERTScore、Novelty）和基于 LLM 的 JudgeLM。我们对结果进行了详细分析，包括对模型性能的实证评估和评估指标的综合分数分布。这项工作为日益增多的多语言反言论生成研究做出了贡献，为开发能够适应不同语言和文化背景以打击网络仇恨言论的强大模型提供了见解。

Title: Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines

• Authors: Xiyang Hu
• Subjects: cs.CL, cs.AI, cs.GT, cs.IR, econ.TH
• Abstract URL: https://arxiv.org/abs/2501.00745
• Pdf URL: https://arxiv.org/pdf/2501.00745
• Copy Paste: [[2501.00745]] Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines(https://arxiv.org/abs/2501.00745)
• Keywords: language model, llm
• Abstract: The increasing integration of Large Language Model (LLM) based search engines has transformed the landscape of information retrieval. However, these systems are vulnerable to adversarial attacks, especially ranking manipulation attacks, where attackers craft webpage content to manipulate the LLM's ranking and promote specific content, gaining an unfair advantage over competitors. In this paper, we study the dynamics of ranking manipulation attacks. We frame this problem as an Infinitely Repeated Prisoners' Dilemma, where multiple players strategically decide whether to cooperate or attack. We analyze the conditions under which cooperation can be sustained, identifying key factors such as attack costs, discount rates, attack success rates, and trigger strategies that influence player behavior. We identify tipping points in the system dynamics, demonstrating that cooperation is more likely to be sustained when players are forward-looking. However, from a defense perspective, we find that simply reducing attack success probabilities can, paradoxically, incentivize attacks under certain conditions. Furthermore, defensive measures to cap the upper bound of attack success rates may prove futile in some scenarios. These insights highlight the complexity of securing LLM-based systems. Our work provides a theoretical foundation and practical insights for understanding and mitigating their vulnerabilities, while emphasizing the importance of adaptive security strategies and thoughtful ecosystem design.
• 摘要：基于大型语言模型 (LLM) 的搜索引擎的日益集成改变了信息检索的格局。然而，这些系统容易受到对抗性攻击，尤其是排名操纵攻击，攻击者会制作网页内容来操纵 LLM 的排名并推广特定内容，从而获得相对于竞争对手的不公平优势。在本文中，我们研究了排名操纵攻击的动态。我们将这个问题定义为无限重复囚徒困境，其中多个玩家战略性地决定是合作还是攻击。我们分析了合作可以持续的条件，确定了影响玩家行为的关键因素，例如攻击成本、折扣率、攻击成功率和触发策略。我们确定了系统动态中的临界点，表明当玩家具有前瞻性时，合作更有可能持续。然而，从防御的角度来看，我们发现简单地降低攻击成功率可能会在某些条件下激励攻击。此外，在某些情况下，限制攻击成功率上限的防御措施可能会徒劳无功。这些见解凸显了保护基于 LLM 的系统安全性的复杂性。我们的工作为理解和减轻其漏洞提供了理论基础和实践见解，同时强调了自适应安全策略和周到的生态系统设计的重要性。

Title: DIVE: Diversified Iterative Self-Improvement

• Authors: Yiwei Qin, Yixiu Liu, Pengfei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00747
• Pdf URL: https://arxiv.org/pdf/2501.00747
• Copy Paste: [[2501.00747]] DIVE: Diversified Iterative Self-Improvement(https://arxiv.org/abs/2501.00747)
• Keywords: language model, llm
• Abstract: Recent advances in large language models (LLMs) have demonstrated the effectiveness of Iterative Self-Improvement (ISI) techniques. However, continuous training on self-generated data leads to reduced output diversity, a limitation particularly critical in reasoning tasks where diverse solution paths are essential. We present DIVE (Diversified Iterative Self-Improvement), a novel framework that addresses this challenge through two key components: Sample Pool Expansion for broader solution exploration, and Data Selection for balancing diversity and quality in preference pairs. Experiments on MATH and GSM8k datasets show that DIVE achieves a 10% to 45% relative increase in output diversity metrics while maintaining performance quality compared to vanilla ISI. Our ablation studies confirm both components' significance in achieving these improvements. Code is available at this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展证明了迭代自我改进 (ISI) 技术的有效性。然而，对自生成数据的持续训练会导致输出多样性降低，这一限制在需要多样化解决方案路径的推理任务中尤为严重。我们提出了 DIVE（多样化迭代自我改进），这是一个新颖的框架，通过两个关键组件解决这一挑战：用于更广泛解决方案探索的样本池扩展，以及用于平衡偏好对中的多样性和质量的数据选择。在 MATH 和 GSM8k 数据集上进行的实验表明，与原始 ISI 相比，DIVE 在保持性能质量的同时，实现了输出多样性指标相对增加 10% 至 45%。我们的消融研究证实了这两个组件在实现这些改进方面的重要性。代码可在此 https URL 上找到。

Title: FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation

• Authors: Qianli Wang, Nils Feldhus, Simon Ostermann, Luis Felipe Villa-Arenas, Sebastian Möller, Vera Schmitt
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00777
• Pdf URL: https://arxiv.org/pdf/2501.00777
• Copy Paste: [[2501.00777]] FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation(https://arxiv.org/abs/2501.00777)
• Keywords: language model, llm, prompt
• Abstract: Counterfactual examples are widely used in natural language processing (NLP) as valuable data to improve models, and in explainable artificial intelligence (XAI) to understand model behavior. The automated generation of counterfactual examples remains a challenging task even for large language models (LLMs), despite their impressive performance on many tasks. In this paper, we first introduce ZeroCF, a faithful approach for leveraging important words derived from feature attribution methods to generate counterfactual examples in a zero-shot setting. Second, we present a new framework, FitCF, which further verifies aforementioned counterfactuals by label flip verification and then inserts them as demonstrations for few-shot prompting, outperforming two state-of-the-art baselines. Through ablation studies, we identify the importance of each of FitCF's core components in improving the quality of counterfactuals, as assessed through flip rate, perplexity, and similarity measures. Furthermore, we show the effectiveness of LIME and Integrated Gradients as backbone attribution methods for FitCF and find that the number of demonstrations has the largest effect on performance. Finally, we reveal a strong correlation between the faithfulness of feature attribution scores and the quality of generated counterfactuals.
• 摘要：反事实示例在自然语言处理 (NLP) 中被广泛用作改进模型的宝贵数据，并在可解释人工智能 (XAI) 中用于理解模型行为。尽管反事实示例在许多任务上的表现令人印象深刻，但即使对于大型语言模型 (LLM) 来说，自动生成反事实示例仍然是一项具有挑战性的任务。在本文中，我们首先介绍 ZeroCF，这是一种利用从特征归因方法中得出的重要词在零样本设置中生成反事实示例的可靠方法。其次，我们提出了一个新框架 FitCF，它通过标签翻转验证进一步验证上述反事实，然后将它们作为演示插入以进行少样本提示，其表现优于两个最先进的基线。通过消融研究，我们确定了 FitCF 的每个核心组件在提高反事实质量方面的重要性，通过翻转率、困惑度和相似性度量来评估。此外，我们展示了 LIME 和积分梯度作为 FitCF 的主干归因方法的有效性，并发现演示次数对性能的影响最大。最后，我们揭示了特征归因分数的真实度与生成的反事实的质量之间的高度相关性。

Title: Decoding the Flow: CauseMotion for Emotional Causality Analysis in Long-form Conversations

• Authors: Yuxuan Zhang, Yulong Li, Zichen Yu, Feilong Tang, Zhixiang Lu, Chong Li, Kang Dang, Jionglong Su
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.00778
• Pdf URL: https://arxiv.org/pdf/2501.00778
• Copy Paste: [[2501.00778]] Decoding the Flow: CauseMotion for Emotional Causality Analysis in Long-form Conversations(https://arxiv.org/abs/2501.00778)
• Keywords: language model, gpt, retrieval-augmented generation
• Abstract: Long-sequence causal reasoning seeks to uncover causal relationships within extended time series data but is hindered by complex dependencies and the challenges of validating causal links. To address the limitations of large-scale language models (e.g., GPT-4) in capturing intricate emotional causality within extended dialogues, we propose CauseMotion, a long-sequence emotional causal reasoning framework grounded in Retrieval-Augmented Generation (RAG) and multimodal fusion. Unlike conventional methods relying only on textual information, CauseMotion enriches semantic representations by incorporating audio-derived features-vocal emotion, emotional intensity, and speech rate-into textual modalities. By integrating RAG with a sliding window mechanism, it effectively retrieves and leverages contextually relevant dialogue segments, thus enabling the inference of complex emotional causal chains spanning multiple conversational turns. To evaluate its effectiveness, we constructed the first benchmark dataset dedicated to long-sequence emotional causal reasoning, featuring dialogues with over 70 turns. Experimental results demonstrate that the proposed RAG-based multimodal integrated approach, the efficacy of substantially enhances both the depth of emotional understanding and the causal inference capabilities of large-scale language models. A GLM-4 integrated with CauseMotion achieves an 8.7% improvement in causal accuracy over the original model and surpasses GPT-4o by 1.2%. Additionally, on the publicly available DiaASQ dataset, CauseMotion-GLM-4 achieves state-of-the-art results in accuracy, F1 score, and causal reasoning accuracy.
• 摘要：长序列因果推理旨在揭示扩展时间序列数据中的因果关系，但受到复杂依赖关系和验证因果关系的挑战的阻碍。为了解决大规模语言模型（例如 GPT-4）在捕捉扩展对话中复杂情感因果关系方面的局限性，我们提出了 CauseMotion，这是一个基于检索增强生成 (RAG) 和多模态融合的长序列情感因果推理框架。与仅依赖文本信息的传统方法不同，CauseMotion 通过将音频衍生特征（声音情感、情感强度和语速）合并到文本模态中来丰富语义表示。通过将 RAG 与滑动窗口机制相结合，它可以有效地检索和利用上下文相关的对话片段，从而能够推断出跨越多个对话轮次的复杂情感因果链。为了评估其有效性，我们构建了第一个专用于长序列情感因果推理的基准数据集，其中包含超过 70 个对话回合的对话。实验结果表明，所提出的基于 RAG 的多模态集成方法的有效性显著提高了情感理解的深度和大规模语言模型的因果推理能力。与 CauseMotion 集成的 GLM-4 在因果准确率上比原始模型提高了 8.7%，比 GPT-4o 高出 1.2%。此外，在公开的 DiaASQ 数据集上，CauseMotion-GLM-4 在准确率、F1 分数和因果推理准确率方面取得了最先进的结果。

Title: Navigating Nuance: In Quest for Political Truth

• Authors: Soumyadeep Sar, Dwaipayan Roy
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.00782
• Pdf URL: https://arxiv.org/pdf/2501.00782
• Copy Paste: [[2501.00782]] Navigating Nuance: In Quest for Political Truth(https://arxiv.org/abs/2501.00782)
• Keywords: language model, prompt
• Abstract: This study investigates the several nuanced rationales for countering the rise of political bias. We evaluate the performance of the Llama-3 (70B) language model on the Media Bias Identification Benchmark (MBIB), based on a novel prompting technique that incorporates subtle reasons for identifying political leaning. Our findings underscore the challenges of detecting political bias and highlight the potential of transfer learning methods to enhance future models. Through our framework, we achieve a comparable performance with the supervised and fully fine-tuned ConvBERT model, which is the state-of-the-art model, performing best among other baseline models for the political bias task on MBIB. By demonstrating the effectiveness of our approach, we contribute to the development of more robust tools for mitigating the spread of misinformation and polarization. Our codes and dataset are made publicly available in github.
• 摘要：本研究探讨了应对政治偏见崛起的几种细微原理。我们基于一种新颖的提示技术，结合了识别政治倾向的微妙原因，评估了 Llama-3 (70B) 语言模型在媒体偏见识别基准 (MBIB) 上的表现。我们的研究结果强调了检测政治偏见的挑战，并强调了迁移学习方法增强未来模型的潜力。通过我们的框架，我们实现了与监督和完全微调的 ConvBERT 模型相当的性能，该模型是最先进的模型，在 MBIB 的政治偏见任务中表现优于其他基线模型。通过展示我们方法的有效性，我们为开发更强大的工具以减轻错误信息的传播和两极分化做出了贡献。我们的代码和数据集在 github 上公开提供。

Title: Reasoning-Oriented and Analogy-Based Methods for Locating and Editing in Zero-Shot Event-Relational Reasoning

• Authors: Jingyao Tang, Lishuang Li, Liteng Mi, Haiming Wu, Hongbin Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00803
• Pdf URL: https://arxiv.org/pdf/2501.00803
• Copy Paste: [[2501.00803]] Reasoning-Oriented and Analogy-Based Methods for Locating and Editing in Zero-Shot Event-Relational Reasoning(https://arxiv.org/abs/2501.00803)
• Keywords: language model
• Abstract: Zero-shot event-relational reasoning is an important task in natural language processing, and existing methods jointly learn a variety of event-relational prefixes and inference-form prefixes to achieve such tasks. However, training prefixes consumes large computational resources and lacks interpretability. Additionally, learning various relational and inferential knowledge inefficiently exploits the connections between tasks. Therefore, we first propose a method for Reasoning-Oriented Locating and Editing (ROLE), which locates and edits the key modules of the language model for reasoning about event relations, enhancing interpretability and also resource-efficiently optimizing the reasoning ability. Subsequently, we propose a method for Analogy-Based Locating and Editing (ABLE), which efficiently exploits the similarities and differences between tasks to optimize the zero-shot reasoning capability. Experimental results show that ROLE improves interpretability and reasoning performance with reduced computational cost. ABLE achieves SOTA results in zero-shot reasoning.
• 摘要：零样本事件关系推理是自然语言处理中的重要任务，现有的方法通过联合学习各种事件关系前缀和推理形式前缀来实现此类任务。然而，训练前缀会消耗大量的计算资源并且缺乏可解释性。此外，学习各种关系和推理知识会降低任务之间联系的利用效率。因此，我们首先提出了一种面向推理的定位与编辑（ROLE）方法，该方法定位和编辑语言模型中用于推理事件关系的关键模块，增强可解释性并以资源高效的方式优化推理能力。随后，我们提出了一种基于类比的定位与编辑（ABLE）方法，该方法可以有效利用任务之间的相似性和差异性来优化零样本推理能力。实验结果表明，ROLE 在降低计算成本的同时提高了可解释性和推理性能。ABLE 在零样本推理中取得了 SOTA 效果。

Title: Embedding Style Beyond Topics: Analyzing Dispersion Effects Across Different Language Models

• Authors: Benjamin Icard, Evangelia Zve, Lila Sainero, Alice Breton, Jean-Gabriel Ganascia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00828
• Pdf URL: https://arxiv.org/pdf/2501.00828
• Copy Paste: [[2501.00828]] Embedding Style Beyond Topics: Analyzing Dispersion Effects Across Different Language Models(https://arxiv.org/abs/2501.00828)
• Keywords: language model
• Abstract: This paper analyzes how writing style affects the dispersion of embedding vectors across multiple, state-of-the-art language models. While early transformer models primarily aligned with topic modeling, this study examines the role of writing style in shaping embedding spaces. Using a literary corpus that alternates between topics and styles, we compare the sensitivity of language models across French and English. By analyzing the particular impact of style on embedding dispersion, we aim to better understand how language models process stylistic information, contributing to their overall interpretability.
• 摘要：本文分析了写作风格如何影响多个最先进的语言模型中嵌入向量的分散性。虽然早期的 Transformer 模型主要与主题建模保持一致，但本研究探讨了写作风格在塑造嵌入空间中的作用。我们使用在主题和风格之间交替的文学语料库，比较了法语和英语语言模型的敏感度。通过分析风格对嵌入分散性的特定影响，我们旨在更好地了解语言模型如何处理风格信息，从而提高其整体可解释性。

Title: LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actions

• Authors: Adam Ishay, Joohyung Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00830
• Pdf URL: https://arxiv.org/pdf/2501.00830
• Copy Paste: [[2501.00830]] LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actions(https://arxiv.org/abs/2501.00830)
• Keywords: language model, gpt, llm, chat
• Abstract: Large Language Models (LLMs) have made significant strides in various intelligent tasks but still struggle with complex action reasoning tasks that require systematic search. To address this limitation, we propose a method that bridges the natural language understanding capabilities of LLMs with the symbolic reasoning strengths of action languages. Our approach, termed "LLM+AL," leverages the LLM's strengths in semantic parsing and commonsense knowledge generation alongside the action language's proficiency in automated reasoning based on encoded knowledge. We compare LLM+AL against state-of-the-art LLMs, including ChatGPT-4, Claude 3 Opus, Gemini Ultra 1.0, and o1-preview, using benchmarks for complex reasoning about actions. Our findings indicate that, although all methods exhibit errors, LLM+AL, with relatively minimal human corrections, consistently leads to correct answers, whereas standalone LLMs fail to improve even with human feedback. LLM+AL also contributes to automated generation of action languages.
• 摘要：大型语言模型 (LLM) 在各种智能任务中取得了重大进展，但在需要系统搜索的复杂动作推理任务方面仍然举步维艰。为了解决这一限制，我们提出了一种方法，将 LLM 的自然语言理解能力与动作语言的符号推理优势结合起来。我们的方法称为“LLM+AL”，利用 LLM 在语义解析和常识知识生成方面的优势以及动作语言在基于编码知识的自动推理方面的熟练程度。我们使用复杂动作推理的基准，将 LLM+AL 与最先进的 LLM（包括 ChatGPT-4、Claude 3 Opus、Gemini Ultra 1.0 和 o1-preview）进行比较。我们的研究结果表明，尽管所有方法都存在错误，但 LLM+AL 在人工校正相对较少的情况下始终能得出正确答案，而独立的 LLM 即使有人工反馈也无法改进。LLM+AL 还有助于动作语言的自动生成。

Title: Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation

• Authors: Shoutao Guo, Shaolei Zhang, Zhengrui Ma, Yang Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00868
• Pdf URL: https://arxiv.org/pdf/2501.00868
• Copy Paste: [[2501.00868]] Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation(https://arxiv.org/abs/2501.00868)
• Keywords: language model, llm
• Abstract: Simultaneous generation models write generation results while reading streaming inputs, necessitating a policy-maker to determine the appropriate output timing. Existing simultaneous generation methods generally adopt the traditional encoder-decoder architecture and learn the generation and policy-making capabilities through complex dynamic programming techniques. Although LLMs excel at text generation, they face challenges in taking on the role of policy-makers through traditional training methods, limiting their exploration in simultaneous generation. To overcome these limitations, we propose a novel LLM-driven Simultaneous Generation (LSG) framework, which allows the off-the-shelf LLM to decide the generation timing and produce output concurrently. Specifically, LSG selects the generation policy that minimizes latency as the baseline policy. Referring to the baseline policy, LSG enables the LLM to devise an improved generation policy that better balances latency and generation quality, and writes generation results accordingly. Experiments on simultaneous translation and streaming automatic speech recognition tasks show that our method can achieve state-of-the-art performance utilizing the open-source LLMs and demonstrate practicality in real-world scenarios.
• 摘要：同步生成模型在读取流式输入的同时写入生成结果，因此需要决策者确定适当的输出时间。现有的同步生成方法通常采用传统的编码器-解码器架构，并通过复杂的动态规划技术学习生成和决策能力。尽管LLM在文本生成方面表现出色，但它们在通过传统训练方法承担决策者角色方面面临挑战，限制了它们在同步生成方面的探索。为了克服这些限制，我们提出了一种新颖的LLM驱动的同步生成（LSG）框架，该框架允许现成的LLM决定生成时间并同时产生输出。具体而言，LSG选择最小化延迟的生成策略作为基准策略。参考基准策略，LSG使LLM能够设计出一种改进的生成策略，以更好地平衡延迟和生成质量，并相应地写入生成结果。在同声翻译和流式自动语音识别任务上的实验表明，我们的方法可以利用开源LLM实现最佳性能，并在实际场景中展示实用性。

Title: LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models

• Authors: Hieu Man, Nghia Trung Ngo, Viet Dac Lai, Ryan A. Rossi, Franck Dernoncourt, Thien Huu Nguyen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.00874
• Pdf URL: https://arxiv.org/pdf/2501.00874
• Copy Paste: [[2501.00874]] LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models(https://arxiv.org/abs/2501.00874)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) based embedding models have established new state-of-the-art benchmarks for text embedding tasks, particularly in dense vector-based retrieval. However, these models predominantly focus on English, leaving multilingual embedding capabilities largely unexplored. To address this limitation, we present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision. LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks. These components are seamlessly integrated through a minimal set of trainable parameters that act as a connector, effectively transferring the multilingual encoder's language understanding capabilities to the specialized embedding model. Additionally, to comprehensively evaluate multilingual embedding performance, we introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages. Extensive experimental results demonstrate that LUSIFER significantly enhances the multilingual performance across various embedding tasks, particularly for medium and low-resource languages, without requiring explicit multilingual training data.
• 摘要：基于大型语言模型 (LLM) 的嵌入模型的最新进展为文本嵌入任务建立了新的先进基准，特别是在基于密集向量的检索中。然而，这些模型主要关注英语，而多语言嵌入功能在很大程度上尚未得到探索。为了解决这一限制，我们提出了 LUSIFER，这是一种新颖的零样本方法，它使基于 LLM 的嵌入模型适应多语言任务，而无需多语言监督。LUSIFER 的架构结合了多语言编码器（作为语言通用学习器）和基于 LLM 的嵌入模型（针对特定嵌入任务进行了优化）。这些组件通过一组最小的可训练参数无缝集成，这些参数充当连接器，有效地将多语言编码器的语言理解能力转移到专门的嵌入模型。此外，为了全面评估多语言嵌入性能，我们引入了一个新的基准，涵盖 5 个主要嵌入任务、123 个不同的数据集和 14 种语言的覆盖范围。大量实验结果表明，LUSIFER 显著提高了各种嵌入任务的多语言性能，特别是对于中低资源语言，而且不需要明确的多语言训练数据。

Title: TrustRAG: Enhancing Robustness and Trustworthiness in RAG

• Authors: Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, Yue Chen, Zhenhao Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00879
• Pdf URL: https://arxiv.org/pdf/2501.00879
• Copy Paste: [[2501.00879]] TrustRAG: Enhancing Robustness and Trustworthiness in RAG(https://arxiv.org/abs/2501.00879)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems enhance large language models (LLMs) by integrating external knowledge sources, enabling more accurate and contextually relevant responses tailored to user queries. However, these systems remain vulnerable to corpus poisoning attacks that can significantly degrade LLM performance through the injection of malicious content. To address these challenges, we propose TrustRAG, a robust framework that systematically filters compromised and irrelevant content before it reaches the language model. Our approach implements a two-stage defense mechanism: first, it employs K-means clustering to identify potential attack patterns in retrieved documents based on their semantic embeddings, effectively isolating suspicious content. Second, it leverages cosine similarity and ROUGE metrics to detect malicious documents while resolving discrepancies between the model's internal knowledge and external information through a self-assessment process. TrustRAG functions as a plug-and-play, training-free module that integrates seamlessly with any language model, whether open or closed-source, maintaining high contextual relevance while strengthening defenses against attacks. Through extensive experimental validation, we demonstrate that TrustRAG delivers substantial improvements in retrieval accuracy, efficiency, and attack resistance compared to existing approaches across multiple model architectures and datasets. We have made TrustRAG available as open-source software at \url{this https URL}.
• 摘要：检索增强生成 (RAG) 系统通过集成外部知识源来增强大型语言模型 (LLM)，从而能够根据用户查询提供更准确、更符合语境的响应。然而，这些系统仍然容易受到语料库中毒攻击，这些攻击可以通过注入恶意内容显著降低 LLM 性能。为了应对这些挑战，我们提出了 TrustRAG，这是一个强大的框架，可以在受损和不相关的内容到达语言模型之前对其进行系统性过滤。我们的方法实现了两阶段防御机制：首先，它采用 K 均值聚类，根据检索到的文档的语义嵌入来识别其中的潜在攻击模式，从而有效地隔离可疑内容。其次，它利用余弦相似度和 ROUGE 指标来检测恶意文档，同时通过自我评估过程解决模型内部知识与外部信息之间的差异。TrustRAG 是一个即插即用、无需训练的模块，可与任何语言模型（无论是开源还是闭源）无缝集成，在增强对攻击的防御能力的同时保持较高的语境相关性。通过大量的实验验证，我们证明 TrustRAG 在检索准确性、效率和抗攻击性方面与现有方法相比在多个模型架构和数据集上都有显著提升。我们已将 TrustRAG 作为开源软件在 \url{此 https URL} 上提供。

Title: Representation in large language models

• Authors: Cameron C. Yetman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.00885
• Pdf URL: https://arxiv.org/pdf/2501.00885
• Copy Paste: [[2501.00885]] Representation in large language models(https://arxiv.org/abs/2501.00885)
• Keywords: language model, llm
• Abstract: The extraordinary success of recent Large Language Models (LLMs) on a diverse array of tasks has led to an explosion of scientific and philosophical theorizing aimed at explaining how they do what they do. Unfortunately, disagreement over fundamental theoretical issues has led to stalemate, with entrenched camps of LLM optimists and pessimists often committed to very different views of how these systems work. Overcoming stalemate requires agreement on fundamental questions, and the goal of this paper is to address one such question, namely: is LLM behavior driven partly by representation-based information processing of the sort implicated in biological cognition, or is it driven entirely by processes of memorization and stochastic table look-up? This is a question about what kind of algorithm LLMs implement, and the answer carries serious implications for higher level questions about whether these systems have beliefs, intentions, concepts, knowledge, and understanding. I argue that LLM behavior is partially driven by representation-based information processing, and then I describe and defend a series of practical techniques for investigating these representations and developing explanations on their basis. The resulting account provides a groundwork for future theorizing about language models and their successors.
• 摘要：最近，大型语言模型 (LLM) 在各种任务上取得了非凡的成功，这引发了大量科学和哲学理论的涌现，这些理论旨在解释它们是如何工作的。不幸的是，对基本理论问题的分歧导致了僵局，根深蒂固的 LLM 乐观派和悲观派阵营往往对这些系统的工作原理持有截然不同的看法。打破僵局需要就基本问题达成一致，本文的目标是解决这样一个问题，即：LLM 行为是部分由与生物认知有关的基于表征的信息处理驱动，还是完全由记忆和随机表查找过程驱动？这是一个关于 LLM 实现哪种算法的问题，答案对于这些系统是否具有信念、意图、概念、知识和理解等更高层次的问题具有重要意义。我认为 LLM 行为部分由基于表征的信息处理驱动，然后描述并捍卫一系列用于研究这些表征并在其基础上开发解释的实用技术。由此得出的结论为未来关于语言模型及其后继者的理论奠定了基础。

Title: Unfolding the Headline: Iterative Self-Questioning for News Retrieval and Timeline Summarization

• Authors: Weiqi Wu, Shen Huang, Yong Jiang, Pengjun Xie, Fei Huang, Hai Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.00888
• Pdf URL: https://arxiv.org/pdf/2501.00888
• Copy Paste: [[2501.00888]] Unfolding the Headline: Iterative Self-Questioning for News Retrieval and Timeline Summarization(https://arxiv.org/abs/2501.00888)
• Keywords: language model, llm
• Abstract: In the fast-changing realm of information, the capacity to construct coherent timelines from extensive event-related content has become increasingly significant and challenging. The complexity arises in aggregating related documents to build a meaningful event graph around a central topic. This paper proposes CHRONOS - Causal Headline Retrieval for Open-domain News Timeline SummarizatiOn via Iterative Self-Questioning, which offers a fresh perspective on the integration of Large Language Models (LLMs) to tackle the task of Timeline Summarization (TLS). By iteratively reflecting on how events are linked and posing new questions regarding a specific news topic to gather information online or from an offline knowledge base, LLMs produce and refresh chronological summaries based on documents retrieved in each round. Furthermore, we curate Open-TLS, a novel dataset of timelines on recent news topics authored by professional journalists to evaluate open-domain TLS where information overload makes it impossible to find comprehensive relevant documents from the web. Our experiments indicate that CHRONOS is not only adept at open-domain timeline summarization, but it also rivals the performance of existing state-of-the-art systems designed for closed-domain applications, where a related news corpus is provided for summarization.
• 摘要：在瞬息万变的信息领域，从大量与事件相关的内容中构建连贯的时间线的能力变得越来越重要和具有挑战性。复杂性出现在聚合相关文档以围绕中心主题构建有意义的事件图时。本文提出了 CHRONOS - 通过迭代自我提问进行开放域新闻时间线摘要的因果标题检索，为集成大型语言模型 (LLM) 以解决时间线摘要 (TLS) 任务提供了新的视角。通过迭代反思事件如何链接并提出有关特定新闻主题的新问题以在线或从离线知识库收集信息，LLM 根据每轮检索到的文档生成和更新按时间顺序排列的摘要。此外，我们策划了 Open-TLS，这是一个由专业记者撰写的有关最近新闻主题的时间线的新数据集，用于评估开放域 TLS，其中信息过载使得无法从网络上找到全面的相关文档。我们的实验表明，CHRONOS 不仅擅长开放域时间线摘要，而且还能与现有的为闭域应用设计的最先进系统的性能相媲美，其中提供相关新闻语料库用于摘要。

Title: Incremental Dialogue Management: Survey, Discussion, and Implications for HRI

• Authors: Casey Kennington, Pierre Lison, David Schlangen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00953
• Pdf URL: https://arxiv.org/pdf/2501.00953
• Copy Paste: [[2501.00953]] Incremental Dialogue Management: Survey, Discussion, and Implications for HRI(https://arxiv.org/abs/2501.00953)
• Keywords: language model
• Abstract: Efforts towards endowing robots with the ability to speak have benefited from recent advancements in NLP, in particular large language models. However, as powerful as current models have become, they still operate on sentence or multi-sentence level input, not on the word-by-word input that humans operate on, affecting the degree of responsiveness that they offer, which is critical in situations where humans interact with robots using speech. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and the implications of incremental dialogue for embodied, robotic platforms.
• 摘要：赋予机器人说话能力的努力得益于 NLP 领域最近的进步，尤其是大型语言模型。然而，尽管目前的模型已经变得非常强大，但它们仍然以句子或多句级输入为基础，而不是以人类操作的逐字输入为基础，这影响了它们的响应程度，而这在人类使用语音与机器人交互的情况下至关重要。在本文中，我们回顾了以增量方式（即在单词级或以下）运行的交互式系统的文献。我们激发了对增量系统的需求，调查了语音识别和语言生成等对话重要方面的增量建模。主要关注的是系统中做出决策的部分，即对话管理器。我们发现对增量对话管理的研究很少，提出了一些实际增量对话管理的要求，以及增量对话对具身机器人平台的影响。

Title: Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice

• Authors: Federico Ravenda, Seyed Ali Bahrainian, Andrea Raballo, Antonietta Mira, Noriko Kando
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00982
• Pdf URL: https://arxiv.org/pdf/2501.00982
• Copy Paste: [[2501.00982]] Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice(https://arxiv.org/abs/2501.00982)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: In psychological practice, standardized questionnaires serve as essential tools for assessing mental constructs (e.g., attitudes, traits, and emotions) through structured questions (aka items). With the increasing prevalence of social media platforms where users share personal experiences and emotions, researchers are exploring computational methods to leverage this data for rapid mental health screening. In this study, we propose a novel adaptive Retrieval-Augmented Generation (RAG) approach that completes psychological questionnaires by analyzing social media posts. Our method retrieves the most relevant user posts for each question in a psychological survey and uses Large Language Models (LLMs) to predict questionnaire scores in a zero-shot setting. Our findings are twofold. First we demonstrate that this approach can effectively predict users' responses to psychological questionnaires, such as the Beck Depression Inventory II (BDI-II), achieving performance comparable to or surpassing state-of-the-art models on Reddit-based benchmark datasets without relying on training data. Second, we show how this methodology can be generalized as a scalable screening tool, as the final assessment is systematically derived by completing standardized questionnaires and tracking how individual item responses contribute to the diagnosis, aligning with established psychometric practices.
• 摘要：在心理学实践中，标准化问卷是通过结构化问题（又称项目）评估心理结构（例如态度、特质和情绪）的重要工具。随着社交媒体平台越来越普及，用户可以在其中分享个人经历和情感，研究人员正在探索利用计算方法利用这些数据进行快速心理健康筛查。在本研究中，我们提出了一种新颖的自适应检索增强生成 (RAG) 方法，通过分析社交媒体帖子来完成心理问卷。我们的方法检索心理调查中每个问题最相关的用户帖子，并使用大型语言模型 (LLM) 在零样本设置中预测问卷分数。我们的发现有两个方面。首先，我们证明这种方法可以有效地预测用户对心理问卷（例如贝克抑郁量表 II (BDI-II)）的反应，在不依赖训练数据的情况下，在基于 Reddit 的基准数据集上实现与最先进模型相当或超越其的性能。其次，我们展示了如何将该方法推广为一种可扩展的筛选工具，因为最终的评估是通过完成标准化问卷并跟踪单个项目反应如何有助于诊断来系统地得出的，这与既定的心理测量实践相一致。

Title: Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory

• Authors: Zhou Yang, Zhengyu Qi, Zhaochun Ren, Zhikai Jia, Haizhou Sun, Xiaofei Zhu, Xiangwen Liao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.00999
• Pdf URL: https://arxiv.org/pdf/2501.00999
• Copy Paste: [[2501.00999]] Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory(https://arxiv.org/abs/2501.00999)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks by understanding input information and predicting corresponding outputs. However, the internal mechanisms by which LLMs comprehend input and make effective predictions remain poorly understood. In this paper, we explore the working mechanism of LLMs in information processing from the perspective of Information Bottleneck Theory. We propose a non-training construction strategy to define a task space and identify the following key findings: (1) LLMs compress input information into specific task spaces (e.g., sentiment space, topic space) to facilitate task understanding; (2) they then extract and utilize relevant information from the task space at critical moments to generate accurate predictions. Based on these insights, we introduce two novel approaches: an Information Compression-based Context Learning (IC-ICL) and a Task-Space-guided Fine-Tuning (TS-FT). IC-ICL enhances reasoning performance and inference efficiency by compressing retrieved example information into the task space. TS-FT employs a space-guided loss to fine-tune LLMs, encouraging the learning of more effective compression and selection mechanisms. Experiments across multiple datasets validate the effectiveness of task space construction. Additionally, IC-ICL not only improves performance but also accelerates inference speed by over 40\%, while TS-FT achieves superior results with a minimal strategy adjustment.
• 摘要：大型语言模型 (LLM) 通过理解输入信息并预测相应的输出，在广泛的任务中表现出色。然而，人们对 LLM 理解输入和做出有效预测的内部机制仍然知之甚少。在本文中，我们从信息瓶颈理论的角度探讨了 LLM 在信息处理中的工作机制。我们提出了一种非训练构建策略来定义任务空间，并确定了以下关键发现：(1) LLM 将输入信息压缩到特定的任务空间（例如，情感空间、主题空间）以促进任务理解；(2) 然后它们在关键时刻从任务空间中提取和利用相关信息来生成准确的预测。基于这些见解，我们介绍了两种新方法：基于信息压缩的上下文学习 (IC-ICL) 和任务空间引导的微调 (TS-FT)。IC-ICL 通过将检索到的示例信息压缩到任务空间来提高推理性能和推理效率。 TS-FT 采用空间引导损失来微调 LLM，鼓励学习更有效的压缩和选择机制。跨多个数据集的实验验证了任务空间构建的有效性。此外，IC-ICL 不仅提高了性能，还将推理速度提高了 40% 以上，而 TS-FT 只需进行最少的策略调整即可获得出色的结果。

Title: MDSF: Context-Aware Multi-Dimensional Data Storytelling Framework based on Large language Model

• Authors: Chengze Zhang, Changshan Li, Shiyang Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01014
• Pdf URL: https://arxiv.org/pdf/2501.01014
• Copy Paste: [[2501.01014]] MDSF: Context-Aware Multi-Dimensional Data Storytelling Framework based on Large language Model(https://arxiv.org/abs/2501.01014)
• Keywords: language model, llm, agent
• Abstract: The exponential growth of data and advancements in big data technologies have created a demand for more efficient and automated approaches to data analysis and storytelling. However, automated data analysis systems still face challenges in leveraging large language models (LLMs) for data insight discovery, augmented analysis, and data storytelling. This paper introduces the Multidimensional Data Storytelling Framework (MDSF) based on large language models for automated insight generation and context-aware storytelling. The framework incorporates advanced preprocessing techniques, augmented analysis algorithms, and a unique scoring mechanism to identify and prioritize actionable insights. The use of fine-tuned LLMs enhances contextual understanding and generates narratives with minimal manual intervention. The architecture also includes an agent-based mechanism for real-time storytelling continuation control. Key findings reveal that MDSF outperforms existing methods across various datasets in terms of insight ranking accuracy, descriptive quality, and narrative coherence. The experimental evaluation demonstrates MDSF's ability to automate complex analytical tasks, reduce interpretive biases, and improve user satisfaction. User studies further underscore its practical utility in enhancing content structure, conclusion extraction, and richness of detail.
• 摘要：数据的指数级增长和大数据技术的进步催生了对更高效、更自动化的数据分析和叙事方法的需求。然而，自动化数据分析系统在利用大型语言模型 (LLM) 进行数据洞察发现、增强分析和数据叙事方面仍然面临挑战。本文介绍了基于大型语言模型的多维数据叙事框架 (MDSF)，用于自动生成洞察和情境感知叙事。该框架结合了先进的预处理技术、增强分析算法和独特的评分机制，以识别和优先考虑可操作的洞察。使用经过微调的 LLM 可以增强情境理解并以最少的人工干预生成叙事。该架构还包括一个基于代理的机制，用于实时叙事延续控制。主要发现表明，在洞察排名准确性、描述质量和叙事连贯性方面，MDSF 在各种数据集上的表现优于现有方法。实验评估证明了 MDSF 能够自动执行复杂的分析任务、减少解释偏差并提高用户满意度。用户研究进一步强调了其在增强内容结构、结论提取和细节丰富性方面的实用性。

Title: KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model

• Authors: Xinshuo Hu, Zifei Shan, Xinping Zhao, Zetian Sun, Zhenyu Liu, Dongfang Li, Shaolin Ye, Xinyuan Wei, Qian Chen, Baotian Hu, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01028
• Pdf URL: https://arxiv.org/pdf/2501.01028
• Copy Paste: [[2501.01028]] KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model(https://arxiv.org/abs/2501.01028)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: As retrieval-augmented generation prevails in large language models, embedding models are becoming increasingly crucial. Despite the growing number of general embedding models, prior work often overlooks the critical role of training data quality. In this work, we introduce KaLM-Embedding, a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data. Our model has been trained with key techniques proven to enhance performance: (1) persona-based synthetic data to create diversified examples distilled from LLMs, (2) ranking consistency filtering to remove less informative samples, and (3) semi-homogeneous task batch sampling to improve training efficacy. Departing from traditional BERT-like architectures, we adopt Qwen2-0.5B as the pre-trained model, facilitating the adaptation of auto-regressive language models for general embedding tasks. Extensive evaluations of the MTEB benchmark across multiple languages show that our model outperforms others of comparable size, setting a new standard for multilingual embedding models with <1B parameters.
• 摘要：随着检索增强生成在大型语言模型中盛行，嵌入模型变得越来越重要。尽管通用嵌入模型的数量不断增加，但先前的工作往往忽视了训练数据质量的关键作用。在这项工作中，我们引入了 KaLM-Embedding，这是一种通用的多语言嵌入模型，它利用大量更干净、更多样化和特定于领域的训练数据。我们的模型已经使用经过验证可以提高性能的关键技术进行了训练：(1) 基于角色的合成数据，以创建从 LLM 中提炼出的多样化示例，(2) 排名一致性过滤以删除信息量较少的样本，以及 (3) 半同质任务批量采样以提高训练效果。与传统的 BERT 类架构不同，我们采用 Qwen2-0.5B 作为预训练模型，促进自回归语言模型适应一般的嵌入任务。对多种语言的 MTEB 基准进行的广泛评估表明，我们的模型优于其他同等规模的模型，为具有 <1B 参数的多语言嵌入模型树立了新标准。

Title: ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learning

• Authors: Wonduk Seo, Zonghao Yuan, Yi Bu
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.01031
• Pdf URL: https://arxiv.org/pdf/2501.01031
• Copy Paste: [[2501.01031]] ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learning(https://arxiv.org/abs/2501.01031)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Cultural values alignment in Large Language Models (LLMs) is a critical challenge due to their tendency to embed Western-centric biases from training data, leading to misrepresentations and fairness issues in cross-cultural contexts. Recent approaches, such as role-assignment and few-shot learning, often struggle with reliable cultural alignment as they heavily rely on pre-trained knowledge, lack scalability, and fail to capture nuanced cultural values effectively. To address these issues, we propose ValuesRAG, a novel and effective framework that applies Retrieval-Augmented Generation (RAG) with in-context learning to integrate cultural and demographic knowledge dynamically during text generation. Leveraging the World Values Survey (WVS) dataset, ValuesRAG first generates summaries of values for each individual. Subsequently, we curated several representative regional datasets to serve as test datasets and retrieve relevant summaries of values based on demographic features, followed by a reranking step to select the top-k relevant summaries. ValuesRAG consistently outperforms baseline methods, both in the main experiment and in the ablation study where only the values summary was provided, highlighting ValuesRAG's potential to foster culturally aligned AI systems and enhance the inclusivity of AI-driven applications.
• 摘要：大型语言模型 (LLM) 中的文化价值观一致性是一项关键挑战，因为它们倾向于从训练数据中嵌入西方中心主义偏见，从而导致跨文化背景下的误解和公平问题。近期的方法，例如角色分配和小样本学习，通常难以实现可靠的文化一致性，因为它们严重依赖预先训练的知识、缺乏可扩展性并且无法有效捕捉细微的文化价值观。为了解决这些问题，我们提出了 ValuesRAG，这是一个新颖而有效的框架，它将检索增强生成 (RAG) 与上下文学习相结合，在文本生成过程中动态整合文化和人口统计知识。利用世界价值观调查 (WVS) 数据集，ValuesRAG 首先为每个人生成价值观摘要。随后，我们整理了几个具有代表性的区域数据集作为测试数据集，并根据人口统计特征检索相关的价值观摘要，然后进行重新排序步骤以选择前 k 个相关摘要。无论是在主要实验中还是在仅提供价值观摘要的消融研究中，ValuesRAG 的表现始终优于基线方法，这突显了 ValuesRAG 在促进文化一致的 AI 系统和增强 AI 驱动应用包容性方面的潜力。

Title: Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models

• Authors: Bin Wang, Xunlong Zou, Shuo Sun, Wenyu Zhang, Yingxu He, Zhuohan Liu, Chengwei Wei, Nancy F. Chen, AiTi Aw
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.01034
• Pdf URL: https://arxiv.org/pdf/2501.01034
• Copy Paste: [[2501.01034]] Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models(https://arxiv.org/abs/2501.01034)
• Keywords: language model, llm
• Abstract: Singlish, a Creole language rooted in English, is a key focus in linguistic research within multilingual and multicultural contexts. However, its spoken form remains underexplored, limiting insights into its linguistic structure and applications. To address this gap, we standardize and annotate the largest spoken Singlish corpus, introducing the Multitask National Speech Corpus (MNSC). These datasets support diverse tasks, including Automatic Speech Recognition (ASR), Spoken Question Answering (SQA), Spoken Dialogue Summarization (SDS), and Paralinguistic Question Answering (PQA). We release standardized splits and a human-verified test set to facilitate further research. Additionally, we propose SingAudioLLM, a multi-task multimodal model leveraging multimodal large language models to handle these tasks concurrently. Experiments reveal our models adaptability to Singlish context, achieving state-of-the-art performance and outperforming prior models by 10-30% in comparison with other AudioLLMs and cascaded solutions.
• 摘要：新加坡英语是一种源于英语的克里奥尔语，是多语言和多文化背景下语言学研究的重点。然而，其口语形式仍未得到充分探索，限制了对其语言结构和应用的深入了解。为了弥补这一差距，我们标准化并注释了最大的新加坡英语口语语料库，推出了多任务国家语音语料库 (MNSC)。这些数据集支持多种任务，包括自动语音识别 (ASR)、口语问答 (SQA)、口语对话摘要 (SDS) 和副语言问答 (PQA)。我们发布了标准化分割和人工验证的测试集，以促进进一步的研究。此外，我们提出了 SingAudioLLM，这是一种多任务多模态模型，利用多模态大型语言模型同时处理这些任务。实验表明，我们的模型对新加坡英语环境具有适应性，实现了最先进的性能，与其他 AudioLLM 和级联解决方案相比，其性能比之前的模型高出 10-30%。

Title: MSWA: Refining Local Attention with Multi-ScaleWindow Attention

• Authors: Yixing Xu, Shivank Nag, Dong Li, Lu Tian, Emad Barsoum
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01039
• Pdf URL: https://arxiv.org/pdf/2501.01039
• Copy Paste: [[2501.01039]] MSWA: Refining Local Attention with Multi-ScaleWindow Attention(https://arxiv.org/abs/2501.01039)
• Keywords: language model, llm
• Abstract: Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.
• 摘要：基于 Transformer 的 LLM 在各种 NLP 任务中都取得了出色的表现。然而，标准的自注意力机制存在二次时间复杂度和线性增加的缓存大小的问题。滑动窗口注意力 (SWA) 通过将注意力范围限制在固定大小的局部上下文窗口来解决此问题。然而，SWA 对每层的每个头部采用统一的窗口大小，使其在捕获不同尺度的上下文时效率低下。为了缓解这一限制，我们提出了多尺度窗口注意力 (MSWA)，它在 Transformer 中的头部和层之间应用不同的窗口大小。它不仅允许同一层内的头部之间使用不同的窗口大小，而且还从浅层到深层逐步增加窗口大小分配，从而使模型能够捕获具有不同长度和距离的上下文信息。语言建模和常识推理任务的实验结果证实，MSWA 在有效性和效率方面均优于传统的局部注意力。

Title: FED: Fast and Efficient Dataset Deduplication Framework with GPU Acceleration

• Authors: Youngjun Son, Chaewon Kim, Jaejin Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01046
• Pdf URL: https://arxiv.org/pdf/2501.01046
• Copy Paste: [[2501.01046]] FED: Fast and Efficient Dataset Deduplication Framework with GPU Acceleration(https://arxiv.org/abs/2501.01046)
• Keywords: llm
• Abstract: Dataset deduplication plays a crucial role in enhancing data quality, ultimately improving training performance and efficiency of LLMs. A commonly used method for data deduplication is the MinHash LSH algorithm. Recently, NVIDIA introduced a GPU-based MinHash LSH deduplication method, but it remains suboptimal, leaving room for further improvement in processing efficiency. This paper proposes a GPU-accelerated deduplication framework \sys that optimizes MinHash LSH for GPU clusters and leverages computationally efficient and partially reusable non-cryptographic hash functions. \sys significantly outperforms the CPU-based deduplication tool included in SlimPajama by up to 58.3 times and the GPU-based deduplication tool included in NVIDIA NeMo Curator by up to 8.6 times when processing 1 million documents with a node of four GPUs. Deduplication of 1.2 trillion tokens is completed in just 5.1 hours in a four-node, 16-GPU environment. The related code is publicly available on GitHub (this https URL).
• 摘要：数据集去重对提升数据质量、提升LLM的训练性能和效率起着至关重要的作用。数据去重的常用方法是MinHash LSH算法。近期，NVIDIA推出了基于GPU的MinHash LSH去重方法，但该方法仍不是最优的，在处理效率方面还有进一步提升的空间。本文提出了一种GPU加速的去重框架\sys，针对GPU集群优化了MinHash LSH，并利用了计算效率高且部分可重用的非加密哈希函数。在使用四块GPU的节点处理100万个文档时，\sys的性能显著优于SlimPajama中包含的基于CPU的去重工具高达58.3倍，优于NVIDIA NeMo Curator中包含的基于GPU的去重工具高达8.6倍。在四节点16块GPU的环境中，仅用5.1小时即可完成1.2万亿个token的去重。相关代码已在GitHub上公开（此https URL）。

Title: Dynamic Scaling of Unit Tests for Code Reward Modeling

• Authors: Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, Jie Tang
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.01054
• Pdf URL: https://arxiv.org/pdf/2501.01054
• Copy Paste: [[2501.01054]] Dynamic Scaling of Unit Tests for Code Reward Modeling(https://arxiv.org/abs/2501.01054)
• Keywords: language model, gpt, llm
• Abstract: Current large language models (LLMs) often struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation. Prior research tackles this challenge by generating multiple candidate solutions and validating them with LLM-generated unit tests. The execution results of unit tests serve as reward signals to identify correct solutions. As LLMs always confidently make mistakes, these unit tests are not reliable, thereby diminishing the quality of reward signals. Motivated by the observation that scaling the number of solutions improves LLM performance, we explore the impact of scaling unit tests to enhance reward signal quality. Our pioneer experiment reveals a positive correlation between the number of unit tests and reward signal quality, with greater benefits observed in more challenging problems. Based on these insights, we propose CodeRM-8B, a lightweight yet effective unit test generator that enables efficient and high-quality unit test scaling. Additionally, we implement a dynamic scaling mechanism that adapts the number of unit tests based on problem difficulty, further improving efficiency. Experimental results show that our approach significantly improves performance across various models on three benchmarks (e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus).
• 摘要：当前的大型语言模型 (LLM) 通常难以在第一次尝试生成代码生成等复杂推理任务时产生准确的响应。先前的研究通过生成多个候选解决方案并使用 LLM 生成的单元测试对其进行验证来解决这一挑战。单元测试的执行结果可作为识别正确解决方案的奖励信号。由于 LLM 总是会犯错，因此这些单元测试并不可靠，从而降低了奖励信号的质量。受扩展解决方案数量可提高 LLM 性能这一观察结果的启发，我们探索了扩展单元测试对增强奖励信号质量的影响。我们的先驱实验揭示了单元测试数量与奖励信号质量之间的正相关性，在更具挑战性的问题中观察到的益处更大。基于这些见解，我们提出了 CodeRM-8B，这是一种轻量级但有效的单元测试生成器，可实现高效和高质量的单元测试扩展。此外，我们实现了一种动态扩展机制，可根据问题难度调整单元测试的数量，从而进一步提高效率。实验结果表明，我们的方法在三个基准上显著提高了各种模型的性能（例如，在 HumanEval Plus 上，Llama3-8B 的性能提高了 18.43%，GPT-4o-mini 的性能提高了 3.42%）。

Title: Risks of Cultural Erasure in Large Language Models

• Authors: Rida Qadri, Aida M. Davani, Kevin Robinson, Vinodkumar Prabhakaran
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01056
• Pdf URL: https://arxiv.org/pdf/2501.01056
• Copy Paste: [[2501.01056]] Risks of Cultural Erasure in Large Language Models(https://arxiv.org/abs/2501.01056)
• Keywords: language model
• Abstract: Large language models are increasingly being integrated into applications that shape the production and discovery of societal knowledge such as search, online education, and travel planning. As a result, language models will shape how people learn about, perceive and interact with global cultures making it important to consider whose knowledge systems and perspectives are represented in models. Recognizing this importance, increasingly work in Machine Learning and NLP has focused on evaluating gaps in global cultural representational distribution within outputs. However, more work is needed on developing benchmarks for cross-cultural impacts of language models that stem from a nuanced sociologically-aware conceptualization of cultural impact or harm. We join this line of work arguing for the need of metricizable evaluations of language technologies that interrogate and account for historical power inequities and differential impacts of representation on global cultures, particularly for cultures already under-represented in the digital corpora. We look at two concepts of erasure: omission: where cultures are not represented at all and simplification i.e. when cultural complexity is erased by presenting one-dimensional views of a rich culture. The former focuses on whether something is represented, and the latter on how it is represented. We focus our analysis on two task contexts with the potential to influence global cultural production. First, we probe representations that a language model produces about different places around the world when asked to describe these contexts. Second, we analyze the cultures represented in the travel recommendations produced by a set of language model applications. Our study shows ways in which the NLP community and application developers can begin to operationalize complex socio-cultural considerations into standard evaluations and benchmarks.
• 摘要：大型语言模型越来越多地被集成到影响社会知识生产和发现的应用程序中，例如搜索、在线教育和旅行计划。因此，语言模型将影响人们学习、感知和与全球文化互动的方式，因此考虑模型中代表了谁的知识系统和观点非常重要。认识到这一重要性，机器学习和 NLP 领域的工作越来越多地集中在评估输出中全球文化代表性分布的差距。然而，需要做更多的工作来开发语言模型跨文化影响的基准，这些基准源于对文化影响或危害的细致入微的社会意识概念化。我们加入这项工作，主张需要对语言技术进行可度量的评估，以探究和解释历史上的权力不平等和代表性对全球文化的不同影响，特别是对于数字语料库中已经代表性不足的文化。我们研究了两种擦除概念：遗漏：文化完全不被呈现；简化，即通过呈现丰富文化的一维视图来消除文化复杂性。前者关注的是某事物是否被呈现，后者关注的是它如何被呈现。我们将分析重点放在两个可能影响全球文化生产的任务上下文上。首先，我们探究语言模型在被要求描述这些上下文时对世界各地不同地方产生的表征。其次，我们分析一组语言模型应用程序生成的旅行建议中所代表的文化。我们的研究展示了 NLP 社区和应用程序开发人员可以开始将复杂的社会文化考虑因素转化为标准评估和基准的方法。

Title: Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models

• Authors: Yanwen Huang, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, Jing Xiao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01059
• Pdf URL: https://arxiv.org/pdf/2501.01059
• Copy Paste: [[2501.01059]] Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models(https://arxiv.org/abs/2501.01059)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) often suffer from context faithfulness hallucinations, where outputs deviate from retrieved information due to insufficient context utilization and high output uncertainty. Our uncertainty evaluation experiments reveal a strong correlation between high uncertainty and hallucinations. We hypothesize that attention mechanisms encode signals indicative of contextual utilization, validated through probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that integrates attention distributions and uncertainty signals in a single-pass decoding process. Experiments across QA datasets demonstrate DAGCD's effectiveness, achieving significant improvements in faithfulness and robustness while maintaining computational efficiency.
• 摘要：大型语言模型 (LLM) 经常遭受上下文忠实幻觉的影响，由于上下文利用率不足和输出不确定性高，输出会偏离检索到的信息。我们的不确定性评估实验表明，高不确定性与幻觉之间存在很强的相关性。我们假设注意力机制编码了指示上下文利用的信号，并通过探测分析进行了验证。基于这些见解，我们提出了动态注意力引导上下文解码 (DAGCD)，这是一个轻量级框架，它将注意力分布和不确定性信号集成在单次解码过程中。在 QA 数据集上进行的实验证明了 DAGCD 的有效性，在保持计算效率的同时实现了忠实度和稳健性的显着提高。

Title: BeliN: A Novel Corpus for Bengali Religious News Headline Generation using Contextual Feature Fusion

• Authors: Md Osama, Ashim Dey, Kawsar Ahmed, Muhammad Ashad Kabir
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01069
• Pdf URL: https://arxiv.org/pdf/2501.01069
• Copy Paste: [[2501.01069]] BeliN: A Novel Corpus for Bengali Religious News Headline Generation using Contextual Feature Fusion(https://arxiv.org/abs/2501.01069)
• Keywords: language model
• Abstract: Automatic text summarization, particularly headline generation, remains a critical yet underexplored area for Bengali religious news. Existing approaches to headline generation typically rely solely on the article content, overlooking crucial contextual features such as sentiment, category, and aspect. This limitation significantly hinders their effectiveness and overall performance. This study addresses this limitation by introducing a novel corpus, BeliN (Bengali Religious News) - comprising religious news articles from prominent Bangladeshi online newspapers, and MultiGen - a contextual multi-input feature fusion headline generation approach. Leveraging transformer-based pre-trained language models such as BanglaT5, mBART, mT5, and mT0, MultiGen integrates additional contextual features - including category, aspect, and sentiment - with the news content. This fusion enables the model to capture critical contextual information often overlooked by traditional methods. Experimental results demonstrate the superiority of MultiGen over the baseline approach that uses only news content, achieving a BLEU score of 18.61 and ROUGE-L score of 24.19, compared to baseline approach scores of 16.08 and 23.08, respectively. These findings underscore the importance of incorporating contextual features in headline generation for low-resource languages. By bridging linguistic and cultural gaps, this research advances natural language processing for Bengali and other underrepresented languages. To promote reproducibility and further exploration, the dataset and implementation code are publicly accessible at this https URL.
• 摘要：自动文本摘要，尤其是标题生成，仍然是孟加拉宗教新闻的一个关键但尚未得到充分探索的领域。现有的标题生成方法通常仅依赖于文章内容，而忽略了情绪、类别和方面等关键的上下文特征。这种限制严重阻碍了它们的有效性和整体性能。本研究通过引入一种新的语料库 BeliN（孟加拉宗教新闻）解决了这一限制——该语料库包含来自孟加拉国著名在线报纸的宗教新闻文章，以及 MultiGen——一种上下文多输入特征融合标题生成方法。MultiGen 利用基于 Transformer 的预训练语言模型（如 BanglaT5、mBART、mT5 和 mT0），将其他上下文特征（包括类别、方面和情绪）与新闻内容相结合。这种融合使模型能够捕获传统方法经常忽略的关键上下文信息。实验结果表明，MultiGen 优于仅使用新闻内容的基线方法，BLEU 得分为 18.61，ROUGE-L 得分为 24.19，而基线方法得分分别为 16.08 和 23.08。这些发现强调了在资源匮乏的语言的标题生成中纳入语境特征的重要性。通过弥合语言和文化差距，这项研究推动了孟加拉语和其他代表性不足的语言的自然语言处理。为了促进可重复性和进一步探索，数据集和实现代码可在此 https URL 上公开访问。

Title: BlockDialect: Block-wise Fine-grained Mixed Format for Energy-Efficient LLM Inference

• Authors: Wonsuk Jang, Thierry Tambe
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01144
• Pdf URL: https://arxiv.org/pdf/2501.01144
• Copy Paste: [[2501.01144]] BlockDialect: Block-wise Fine-grained Mixed Format for Energy-Efficient LLM Inference(https://arxiv.org/abs/2501.01144)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have achieved remarkable success, but their increasing size poses significant challenges in memory usage and computational costs. Quantizing both weights and activations can address these issues, with fine-grained block-wise quantization emerging as a promising hardware-supported solution to mitigate outliers. However, existing methods struggle to capture nuanced block data distributions. To address this, we propose BlockDialect, a block-wise fine-grained mixed format technique that assigns a per-block optimal number format from formatbook for better data representation. Additionally, we introduce DialectFP4, a formatbook of FP4 variants (akin to dialects) that adapt to diverse data distributions. Importantly, DialectFP4 ensures hardware efficiency by selecting representable values as scaled integers compatible with low-precision integer arithmetic. Furthermore, we propose a two-stage approach for online DialectFP4 activation quantization. BlockDialect achieves 11.40% (6.90%) accuracy gain on the LLaMA3-8B (LLaMA2-7B) model compared to MXFP4 format with a comparable bit usage per data, while being only 5.89% (3.31%) below full precision even when quantizing full-path matrix multiplication. Focusing on how to represent over how to scale, our work presents a promising path for energy-efficient LLM inference.
• 摘要：大型语言模型 (LLM) 取得了显著的成功，但其不断增大的规模对内存使用和计算成本提出了重大挑战。量化权重和激活可以解决这些问题，细粒度分块量化正在成为一种有前途的硬件支持解决方案，以减轻异常值。然而，现有方法难以捕捉细微的块数据分布。为了解决这个问题，我们提出了 BlockDialect，这是一种分块细粒度混合格式技术，它从格式手册中为每个块分配最佳数字格式，以更好地表示数据。此外，我们引入了 DialectFP4，这是适应不同数据分布的 FP4 变体（类似于方言）的格式手册。重要的是，DialectFP4 通过将可表示的值选择为与低精度整数算法兼容的缩放整数来确保硬件效率。此外，我们提出了一种在线 DialectFP4 激活量化的两阶段方法。与 MXFP4 格式相比，BlockDialect 在 LLaMA3-8B (LLaMA2-7B) 模型上的准确率提高了 11.40% (6.90%)，每个数据的比特使用量相当，而即使在量化全路径矩阵乘法时，准确率也仅比全精度低 5.89% (3.31%)。我们的工作重点是如何表示而不是如何扩展，为节能的 LLM 推理提供了一条有希望的途径。

Title: Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets

• Authors: Mahdi Zakizadeh, Mohammad Taher Pilehvar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01168
• Pdf URL: https://arxiv.org/pdf/2501.01168
• Copy Paste: [[2501.01168]] Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets(https://arxiv.org/abs/2501.01168)
• Keywords: language model
• Abstract: The multifaceted challenge of accurately measuring gender stereotypical bias in language models is akin to discerning different segments of a broader, unseen entity. This short paper primarily focuses on intrinsic bias mitigation and measurement strategies for language models, building on prior research that demonstrates a lack of correlation between intrinsic and extrinsic approaches. We delve deeper into intrinsic measurements, identifying inconsistencies and suggesting that these benchmarks may reflect different facets of gender stereotype. Our methodology involves analyzing data distributions across datasets and integrating gender stereotype components informed by social psychology. By adjusting the distribution of two datasets, we achieve a better alignment of outcomes. Our findings underscore the complexity of gender stereotyping in language models and point to new directions for developing more refined techniques to detect and reduce bias.
• 摘要：准确测量语言模型中的性别刻板偏见是一项多方面的挑战，类似于辨别更广泛、看不见的实体的不同部分。这篇短文主要关注语言模型的内在偏见缓解和测量策略，以先前的研究表明内在方法和外在方法之间缺乏相关性为基础。我们深入研究了内在测量，发现了不一致之处，并提出这些基准可能反映了性别刻板印象的不同方面。我们的方法包括分析数据集中的数据分布，并整合社会心理学所形成的性别刻板印象成分。通过调整两个数据集的分布，我们可以更好地协调结果。我们的研究结果强调了语言模型中性别刻板印象的复杂性，并指出了开发更精细的技术来检测和减少偏见的新方向。

Title: Automated Self-Refinement and Self-Correction for LLM-based Product Attribute Value Extraction

• Authors: Alexander Brinkmann, Christian Bizer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01237
• Pdf URL: https://arxiv.org/pdf/2501.01237
• Copy Paste: [[2501.01237]] Automated Self-Refinement and Self-Correction for LLM-based Product Attribute Value Extraction(https://arxiv.org/abs/2501.01237)
• Keywords: language model, gpt, llm, prompt
• Abstract: Structured product data, in the form of attribute-value pairs, is essential for e-commerce platforms to support features such as faceted product search and attribute-based product comparison. However, vendors often provide unstructured product descriptions, making attribute value extraction necessary to ensure data consistency and usability. Large language models (LLMs) have demonstrated their potential for product attribute value extraction in few-shot scenarios. Recent research has shown that self-refinement techniques can improve the performance of LLMs on tasks such as code generation and text-to-SQL translation. For other tasks, the application of these techniques has resulted in increased costs due to processing additional tokens, without achieving any improvement in performance. This paper investigates applying two self-refinement techniques, error-based prompt rewriting and self-correction, to the product attribute value extraction task. The self-refinement techniques are evaluated across zero-shot, few-shot in-context learning, and fine-tuning scenarios using GPT-4o. The experiments show that both self-refinement techniques have only a marginal impact on the model's performance across the different scenarios, while significantly increasing processing costs. For scenarios with training data, fine-tuning yields the highest performance, while the ramp-up costs of fine-tuning are balanced out as the amount of product descriptions increases.
• 摘要：以属性-值对形式呈现的结构化产品数据对于电子商务平台支持诸如分面产品搜索和基于属性的产品比较等功能至关重要。然而，供应商通常提供非结构化的产品描述，因此必须提取属性值以确保数据的一致性和可用性。大型语言模型 (LLM) 已证明其在小样本场景中提取产品属性值的潜力。最近的研究表明，自改进技术可以提高 LLM 在代码生成和文本到 SQL 转换等任务上的性能。对于其他任务，这些技术的应用由于处理额外的标记而导致成本增加，而性能却没有任何提高。本文研究了将两种自改进技术（基于错误的提示重写和自我纠正）应用于产品属性值提取任务。使用 GPT-4o 在零样本、小样本上下文学习和微调场景中评估了自改进技术。实验表明，两种自我优化技术对模型在不同场景下的表现影响甚微，但处理成本却显著增加。对于有训练数据的场景，微调可获得最高性能，而微调的提升成本会随着产品描述数量的增加而得到平衡。

Title: Large Language Model-Enhanced Symbolic Reasoning for Knowledge Base Completion

• Authors: Qiyuan He, Jianfei Yu, Wenya Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01246
• Pdf URL: https://arxiv.org/pdf/2501.01246
• Copy Paste: [[2501.01246]] Large Language Model-Enhanced Symbolic Reasoning for Knowledge Base Completion(https://arxiv.org/abs/2501.01246)
• Keywords: language model, llm, hallucination
• Abstract: Integrating large language models (LLMs) with rule-based reasoning offers a powerful solution for improving the flexibility and reliability of Knowledge Base Completion (KBC). Traditional rule-based KBC methods offer verifiable reasoning yet lack flexibility, while LLMs provide strong semantic understanding yet suffer from hallucinations. With the aim of combining LLMs' understanding capability with the logical and rigor of rule-based approaches, we propose a novel framework consisting of a Subgraph Extractor, an LLM Proposer, and a Rule Reasoner. The Subgraph Extractor first samples subgraphs from the KB. Then, the LLM uses these subgraphs to propose diverse and meaningful rules that are helpful for inferring missing facts. To effectively avoid hallucination in LLMs' generations, these proposed rules are further refined by a Rule Reasoner to pinpoint the most significant rules in the KB for Knowledge Base Completion. Our approach offers several key benefits: the utilization of LLMs to enhance the richness and diversity of the proposed rules and the integration with rule-based reasoning to improve reliability. Our method also demonstrates strong performance across diverse KB datasets, highlighting the robustness and generalizability of the proposed framework.
• 摘要：将大型语言模型 (LLM) 与基于规则的推理相结合，为提高知识库补全 (KBC) 的灵活性和可靠性提供了强大的解决方案。传统的基于规则的 KBC 方法提供可验证的推理，但缺乏灵活性，而 LLM 提供强大的语义理解，但存在幻觉。为了将 LLM 的理解能力与基于规则的方法的逻辑性和严谨性相结合，我们提出了一个由子图提取器、LLM 提议器和规则推理器组成的新框架。子图提取器首先从知识库中抽取子图。然后，LLM 使用这些子图提出多样化且有意义的规则，这些规则有助于推断缺失的事实。为了有效避免 LLM 生成中的幻觉，规则推理器进一步细化这些提议的规则，以精确定位知识库补全中最重要的规则。我们的方法提供了几个关键好处：利用 LLM 来增强提议规则的丰富性和多样性，并与基于规则的推理相结合以提高可靠性。我们的方法还在不同的 KB 数据集上表现出了强大的性能，凸显了所提出框架的稳健性和通用性。

Title: Digital Guardians: Can GPT-4, Perspective API, and Moderation API reliably detect hate speech in reader comments of German online newspapers?

• Authors: Manuel Weber, Moritz Huber, Maximilian Auch, Alexander Döschl, Max-Emanuel Keller, Peter Mandl
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01256
• Pdf URL: https://arxiv.org/pdf/2501.01256
• Copy Paste: [[2501.01256]] Digital Guardians: Can GPT-4, Perspective API, and Moderation API reliably detect hate speech in reader comments of German online newspapers?(https://arxiv.org/abs/2501.01256)
• Keywords: language model, gpt, prompt
• Abstract: In recent years, toxic content and hate speech have become widespread phenomena on the internet. Moderators of online newspapers and forums are now required, partly due to legal regulations, to carefully review and, if necessary, delete reader comments. This is a labor-intensive process. Some providers of large language models already offer solutions for automated hate speech detection or the identification of toxic content. These include GPT-4o from OpenAI, Jigsaw's (Google) Perspective API, and OpenAI's Moderation API. Based on the selected German test dataset HOCON34k, which was specifically created for developing tools to detect hate speech in reader comments of online newspapers, these solutions are compared with each other and against the HOCON34k baseline. The test dataset contains 1,592 annotated text samples. For GPT-4o, three different promptings are used, employing a Zero-Shot, One-Shot, and Few-Shot approach. The results of the experiments demonstrate that GPT-4o outperforms both the Perspective API and the Moderation API, and exceeds the HOCON34k baseline by approximately 5 percentage points, as measured by a combined metric of MCC and F2-score.
• 摘要：近年来，恶意内容和仇恨言论已成为互联网上的普遍现象。部分由于法律规定，在线报纸和论坛的版主现在需要仔细审查并在必要时删除读者评论。这是一个劳动密集型的过程。一些大型语言模型提供商已经提供了自动检测仇恨言论或识别恶意内容的解决方案。这些包括 OpenAI 的 GPT-4o、Jigsaw（谷歌）的 Perspective API 和 OpenAI 的 Moderation API。基于选定的德国测试数据集 HOCON34k（该数据集专门用于开发检测在线报纸读者评论中的仇恨言论的工具），这些解决方案相互比较并与 HOCON34k 基线进行比较。测试数据集包含 1,592 个带注释的文本样本。对于 GPT-4o，使用了三种不同的提示，采用零样本、单样本和少样本方法。实验结果表明，GPT-4o 的表现优于 Perspective API 和 Moderation API，并且比 HOCON34k 基线高出约 5 个百分点，以 MCC 和 F2 分数的综合指标来衡量。

Title: CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings

• Authors: Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, Junyang Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01257
• Pdf URL: https://arxiv.org/pdf/2501.01257
• Copy Paste: [[2501.01257]] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings(https://arxiv.org/abs/2501.01257)
• Keywords: language model, llm
• Abstract: With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 20 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.
• 摘要：随着现有大型语言模型（LLM）代码推理能力的不断提升，以及 OpenAI o1 和 o3 等推理模型的突破，开发更具挑战性和综合性的基准测试以有效测试其复杂的竞赛级编码能力的需求日益增加。现有的基准测试，如 LiveCodeBench 和 USACO，由于缺乏私有测试用例、缺乏对特殊评委的支持以及执行环境不一致而未能达到要求。为了弥补这一差距，我们推出了 CodeElo，这是一个标准化的竞赛级代码生成基准测试，首次有效地解决了所有这些挑战。CodeElo 基准测试主要基于官方 CodeForces 平台，并尽可能与平台保持一致。我们汇编了 CodeForces 上最近六个月的竞赛问题，其中包含竞赛部门、问题难度等级和问题算法标签等详细信息。我们引入了一种独特的评判方法，其中问题直接提交到平台，并开发了一个可靠的 Elo 评级计算系统，该系统与平台保持一致，与人类参与者相当但方差较低。通过在我们的 CodeElo 上进行测试，我们首次提供了 30 个现有的热门开源和 3 个专有 LLM 的 Elo 评级。结果表明，o1-mini 和 QwQ-32B-Preview 表现突出，分别获得了 1578 和 1261 的 Elo 评级，而其他模型甚至连最简单的问题都很难解决，在所有人类参与者中排名最低的 20%。我们还进行了详细的分析实验，以深入了解算法之间的性能以及使用 C++ 和 Python 之间的比较，这可以为未来的研究提供方向。

Title: ProgCo: Program Helps Self-Correction of Large Language Models

• Authors: Xiaoshuai Song, Yanan Wu, Weixun Wang, Jiaheng Liu, Wenbo Su, Bo Zheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01264
• Pdf URL: https://arxiv.org/pdf/2501.01264
• Copy Paste: [[2501.01264]] ProgCo: Program Helps Self-Correction of Large Language Models(https://arxiv.org/abs/2501.01264)
• Keywords: language model, llm
• Abstract: Self-Correction aims to enable large language models (LLMs) to self-verify and self-refine their initial responses without external feedback. However, LLMs often fail to effectively self-verify and generate correct feedback, further misleading refinement and leading to the failure of self-correction, especially in complex reasoning tasks. In this paper, we propose Program-driven Self-Correction (ProgCo). First, program-driven verification (ProgVe) achieves complex verification logic and extensive validation through self-generated, self-executing verification pseudo-programs. Then, program-driven refinement (ProgRe) receives feedback from ProgVe, conducts dual reflection and refinement on both responses and verification programs to mitigate misleading of incorrect feedback in complex reasoning tasks. Experiments on three instruction-following and mathematical benchmarks indicate that ProgCo achieves effective self-correction, and can be further enhance performance when combined with real program tools.
• 摘要：自我纠正旨在使大型语言模型（LLM）能够在没有外界反馈的情况下自我验证和自我细化其初始响应。然而，LLM 往往无法有效地自我验证和生成正确的反馈，从而进一步误导细化并导致自我纠正失败，尤其是在复杂的推理任务中。在本文中，我们提出了程序驱动的自我纠正（ProgCo）。首先，程序驱动验证（ProgVe）通过自生成、自执行的验证伪程序实现复杂的验证逻辑和广泛的验证。然后，程序驱动细化（ProgRe）从 ProgVe 接收反馈，对响应和验证程序进行双重反思和细化，以减轻复杂推理任务中错误反馈的误导。在三个指令跟踪和数学基准上的实验表明，ProgCo 实现了有效的自我纠正，并且与真实程序工具结合使用时可以进一步提高性能。

Title: Does a Large Language Model Really Speak in Human-Like Language?

• Authors: Mose Park, Yunjin Choi, Jong-June Jeon
• Subjects: cs.CL, stat.AP
• Abstract URL: https://arxiv.org/abs/2501.01273
• Pdf URL: https://arxiv.org/pdf/2501.01273
• Copy Paste: [[2501.01273]] Does a Large Language Model Really Speak in Human-Like Language?(https://arxiv.org/abs/2501.01273)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have recently emerged, attracting considerable attention due to their ability to generate highly natural, human-like text. This study compares the latent community structures of LLM-generated text and human-written text within a hypothesis testing procedure. Specifically, we analyze three text sets: original human-written texts ($\mathcal{O}$), their LLM-paraphrased versions ($\mathcal{G}$), and a twice-paraphrased set ($\mathcal{S}$) derived from $\mathcal{G}$. Our analysis addresses two key questions: (1) Is the difference in latent community structures between $\mathcal{O}$ and $\mathcal{G}$ the same as that between $\mathcal{G}$ and $\mathcal{S}$? (2) Does $\mathcal{G}$ become more similar to $\mathcal{O}$ as the LLM parameter controlling text variability is adjusted? The first question is based on the assumption that if LLM-generated text truly resembles human language, then the gap between the pair ($\mathcal{O}$, $\mathcal{G}$) should be similar to that between the pair ($\mathcal{G}$, $\mathcal{S}$), as both pairs consist of an original text and its paraphrase. The second question examines whether the degree of similarity between LLM-generated and human text varies with changes in the breadth of text generation. To address these questions, we propose a statistical hypothesis testing framework that leverages the fact that each text has corresponding parts across all datasets due to their paraphrasing relationship. This relationship enables the mapping of one dataset's relative position to another, allowing two datasets to be mapped to a third dataset. As a result, both mapped datasets can be quantified with respect to the space characterized by the third dataset, facilitating a direct comparison between them. Our results indicate that GPT-generated text remains distinct from human-authored text.
• 摘要：大型语言模型 (LLM) 最近出现，因其能够生成高度自然、类似人类的文本而备受关注。本研究在假设检验程序中比较了 LLM 生成的文本和人类书写文本的潜在社区结构。具体来说，我们分析了三个文本集：原始人类书写文本 ($\mathcal{O}$)、其 LLM 释义版本 ($\mathcal{G}$) 以及从 $\mathcal{G}$ 派生的两次释义集 ($\mathcal{S}$)。我们的分析解决了两个关键问题：(1) $\mathcal{O}$ 和 $\mathcal{G}$ 之间的潜在社区结构差异是否与 $\mathcal{G}$ 和 $\mathcal{S}$ 之间的差异相同？(2) 随着控制文本可变性的 LLM 参数的调整，$\mathcal{G}$ 是否会变得更类似于 $\mathcal{O}$？第一个问题基于这样的假设：如果 LLM 生成的文本确实类似于人类语言，那么对 ($\mathcal{O}$, $\mathcal{G}$) 之间的差距应该与对 ($\mathcal{G}$, $\mathcal{S}$) 之间的差距相似，因为这两对都由原始文本及其释义组成。第二个问题检查 LLM 生成的文本和人类文本之间的相似度是否随着文本生成广度的变化而变化。为了解决这些问题，我们提出了一个统计假设检验框架，该框架利用了由于释义关系，每个文本在所有数据集中都有对应部分的事实。这种关系使得一个数据集的相对位置可以映射到另一个数据集，从而允许将两个数据集映射到第三个数据集。因此，两个映射的数据集都可以根据第三个数据集所表征的空间进行量化，从而便于直接比较它们。我们的结果表明，GPT 生成的文本与人类编写的文本仍然不同。

Title: NeutraSum: A Language Model can help a Balanced Media Diet by Neutralizing News Summaries

• Authors: Xi Luo, Junjie Liu, Sirong Wu, Yuhui Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01284
• Pdf URL: https://arxiv.org/pdf/2501.01284
• Copy Paste: [[2501.01284]] NeutraSum: A Language Model can help a Balanced Media Diet by Neutralizing News Summaries(https://arxiv.org/abs/2501.01284)
• Keywords: language model
• Abstract: Media bias in news articles arises from the political polarisation of media outlets, which can reinforce societal stereotypes and beliefs. Reporting on the same event often varies significantly between outlets, reflecting their political leanings through polarised language and focus. Although previous studies have attempted to generate bias-free summaries from multiperspective news articles, they have not effectively addressed the challenge of mitigating inherent media bias. To address this gap, we propose \textbf{NeutraSum}, a novel framework that integrates two neutrality losses to adjust the semantic space of generated summaries, thus minimising media bias. These losses, designed to balance the semantic distances across polarised inputs and ensure alignment with expert-written summaries, guide the generation of neutral and factually rich summaries. To evaluate media bias, we employ the political compass test, which maps political leanings based on economic and social dimensions. Experimental results on the Allsides dataset demonstrate that NeutraSum not only improves summarisation performance but also achieves significant reductions in media bias, offering a promising approach for neutral news summarisation.
• 摘要：新闻文章中的媒体偏见源于媒体的政治两极分化，这会强化社会刻板印象和信念。不同媒体对同一事件的报道往往存在很大差异，通过两极化的语言和焦点反映出他们的政治倾向。尽管之前的研究试图从多视角新闻文章中生成无偏见的摘要，但它们并没有有效解决减轻固有媒体偏见的挑战。为了解决这一差距，我们提出了 \textbf{NeutraSum}，这是一个新颖的框架，它整合了两个中立性损失来调整生成的摘要的语义空间，从而最大限度地减少媒体偏见。这些损失旨在平衡两极化输入之间的语义距离并确保与专家撰写的摘要保持一致，指导生成中立且事实丰富的摘要。为了评估媒体偏见，我们采用了政治指南针测试，该测试根据经济和社会维度绘制政治倾向。在 Allsides 数据集上的实验结果表明，NeutraSum 不仅提高了摘要性能，而且还显著减少了媒体偏见，为中立新闻摘要提供了一种有前途的方法。

Title: ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

• Authors: Vaskar Nath, Pranav Raja, Claire Yoon, Sean Hendryx
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01290
• Pdf URL: https://arxiv.org/pdf/2501.01290
• Copy Paste: [[2501.01290]] ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark(https://arxiv.org/abs/2501.01290)
• Keywords: prompt
• Abstract: Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only the final answer but also the intermediate steps is important for evaluation, development, and identifying failures during inference time. To bridge this gap, we introduce ToolComp, a comprehensive benchmark designed to evaluate multi-step tool-use reasoning. ToolComp is developed through a collaboration between models and human annotators, featuring human-edited/verified prompts, final answers, and process supervision labels, allowing for the evaluation of both final outcomes and intermediate reasoning. Evaluation across six different model families demonstrates the challenging nature of our dataset, with the majority of models achieving less than 50% accuracy. Additionally, we generate synthetic training data to compare the performance of outcome-supervised reward models (ORMs) with process-supervised reward models (PRMs) to assess their ability to improve complex tool-use reasoning as evaluated by ToolComp. Our results show that PRMs generalize significantly better than ORMs, achieving a 19% and 11% improvement in rank@1 accuracy for ranking base and fine-tuned model trajectories, respectively. These findings highlight the critical role of process supervision in both the evaluation and training of AI models, paving the way for more robust and capable systems in complex, multi-step tool-use tasks.
• 摘要：尽管人工智能最近取得了进展，但开发能够执行涉及多种工具的复杂、多步骤推理任务的系统仍然是一项重大挑战。当前的基准测试无法捕捉工具使用推理的现实世界复杂性，其中验证最终答案和中间步骤的正确性对于评估、开发和识别推理期间的失败非常重要。为了弥补这一差距，我们引入了 ToolComp，这是一个旨在评估多步骤工具使用推理的综合基准测试。ToolComp 是通过模型和人工注释者之间的合作开发的，具有人工编辑/验证的提示、最终答案和过程监督标签，可以评估最终结果和中间推理。对六个不同模型系列的评估表明了我们数据集的挑战性，大多数模型的准确率不到 50%。此外，我们生成合成训练数据来比较结果监督奖励模型 (ORM) 和过程监督奖励模型 (PRM) 的性能，以评估它们改进复杂工具使用推理的能力，正如 ToolComp 评估的那样。我们的结果表明，PRM 的泛化能力明显优于 ORM，在对基础和微调模型轨迹进行排名时，rank@1 准确率分别提高了 19% 和 11%。这些发现凸显了过程监督在 AI 模型评估和训练中的关键作用，为在复杂、多步骤的工具使用任务中打造更强大、更强大的系统铺平了道路。

Title: Citations and Trust in LLM Generated Responses

• Authors: Yifan Ding, Matthew Facciani, Amrit Poudel, Ellen Joyce, Salvador Aguinaga, Balaji Veeramani, Sanmitra Bhattacharya, Tim Weninger
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01303
• Pdf URL: https://arxiv.org/pdf/2501.01303
• Copy Paste: [[2501.01303]] Citations and Trust in LLM Generated Responses(https://arxiv.org/abs/2501.01303)
• Keywords: llm, chat
• Abstract: Question answering systems are rapidly advancing, but their opaque nature may impact user trust. We explored trust through an anti-monitoring framework, where trust is predicted to be correlated with presence of citations and inversely related to checking citations. We tested this hypothesis with a live question-answering experiment that presented text responses generated using a commercial Chatbot along with varying citations (zero, one, or five), both relevant and random, and recorded if participants checked the citations and their self-reported trust in the generated responses. We found a significant increase in trust when citations were present, a result that held true even when the citations were random; we also found a significant decrease in trust when participants checked the citations. These results highlight the importance of citations in enhancing trust in AI-generated content.
• 摘要：问答系统正在迅速发展，但其不透明性可能会影响用户的信任。我们通过反监控框架探索了信任，其中信任预计与引用的存在相关，与检查引用成反比。我们通过实时问答实验测试了这一假设，该实验展示了使用商业聊天机器人生成的文本响应以及不同的引用（零、一或五），既有相关的也有随机的，并记录参与者是否检查了引用以及他们对生成的响应的自我报告信任。我们发现当引用存在时，信任度显著增加，即使引用是随机的，这一结果也是如此；我们还发现当参与者检查引用时，信任度显著下降。这些结果强调了引用在增强对人工智能生成内容的信任方面的重要性。

Title: Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case

• Authors: Kaushik Roy, Harshul Surana, Darssan Eswaramoorthi, Yuxin Zi, Vedant Palit, Ritvik Garimella, Amit Sheth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01305
• Pdf URL: https://arxiv.org/pdf/2501.01305
• Copy Paste: [[2501.01305]] Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case(https://arxiv.org/abs/2501.01305)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large language models (LLMs) are increasingly attracting the attention of healthcare professionals for their potential to assist in diagnostic assessments, which could alleviate the strain on the healthcare system caused by a high patient load and a shortage of providers. For LLMs to be effective in supporting diagnostic assessments, it is essential that they closely replicate the standard diagnostic procedures used by clinicians. In this paper, we specifically examine the diagnostic assessment processes described in the Patient Health Questionnaire-9 (PHQ-9) for major depressive disorder (MDD) and the Generalized Anxiety Disorder-7 (GAD-7) questionnaire for generalized anxiety disorder (GAD). We investigate various prompting and fine-tuning techniques to guide both proprietary and open-source LLMs in adhering to these processes, and we evaluate the agreement between LLM-generated diagnostic outcomes and expert-validated ground truth. For fine-tuning, we utilize the Mentalllama and Llama models, while for prompting, we experiment with proprietary models like GPT-3.5 and GPT-4o, as well as open-source models such as llama-3.1-8b and mixtral-8x7b.
• 摘要：大型语言模型 (LLM) 越来越多地吸引医疗专业人士的关注，因为它们有助于诊断评估，从而减轻因患者负担过重和提供者短缺而给医疗保健系统带来的压力。为了使 LLM 有效地支持诊断评估，必须使其紧密复制临床医生使用的标准诊断程序。在本文中，我们专门研究了针对重度抑郁症 (MDD) 的患者健康问卷-9 (PHQ-9) 和针对广泛性焦虑症 (GAD) 的广泛性焦虑症-7 (GAD-7) 问卷中描述的诊断评估过程。我们研究了各种提示和微调技术，以指导专有和开源 LLM 遵守这些过程，并评估了 LLM 生成的诊断结果与专家验证的基本事实之间的一致性。对于微调，我们利用 Mentalllama 和 Llama 模型，而对于提示，我们尝试了 GPT-3.5 和 GPT-4o 等专有模型，以及 llama-3.1-8b 和 mixtral-8x7b 等开源模型。

Title: Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking

• Authors: Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01306
• Pdf URL: https://arxiv.org/pdf/2501.01306
• Copy Paste: [[2501.01306]] Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking(https://arxiv.org/abs/2501.01306)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) demonstrate exceptional capabilities, yet still face the hallucination issue. Typical text generation approaches adopt an auto-regressive generation without deliberate reasoning, which often results in untrustworthy and factually inaccurate responses. In this paper, we propose HaluSearch, a novel framework that incorporates tree search-based algorithms (e.g. MCTS) to enable an explicit slow thinking generation process for mitigating hallucinations of LLMs during inference. Specifically, HaluSearch frames text generation as a step-by-step reasoning process, using a self-evaluation reward model to score each generation step and guide the tree search towards the most reliable generation pathway for fully exploiting the internal knowledge of LLMs. To balance efficiency and quality, we introduce a hierarchical thinking system switch mechanism inspired by the dual process theory in cognitive science, which dynamically alternates between fast and slow thinking modes at both the instance and step levels, adapting to the complexity of questions and reasoning states. We conduct extensive experiments on both English and Chinese datasets and the results show that our approach significantly outperforms baseline approaches.
• 摘要：大型语言模型 (LLM) 表现出卓越的能力，但仍面临幻觉问题。典型的文本生成方法采用自回归生成，没有经过深思熟虑的推理，这通常会导致不可信且事实不准确的响应。在本文中，我们提出了 HaluSearch，这是一个新颖的框架，它结合了基于树搜索的算法（例如 MCTS），以实现明确的慢速思维生成过程，以减轻 LLM 在推理过程中的幻觉。具体来说，HaluSearch 将文本生成定义为一个循序渐进的推理过程，使用自我评估奖励模型对每个生成步骤进行评分，并引导树搜索走向最可靠的生成路径，以充分利用 LLM 的内部知识。为了平衡效率和质量，我们引入了一种受认知科学中的双重过程理论启发的分层思维系统切换机制，该机制在实例和步骤级别上动态地在快速和慢速思维模式之间切换，以适应问题和推理状态的复杂性。我们对英语和中文数据集进行了大量的实验，结果表明我们的方法明显优于基线方法。

Title: Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension

• Authors: Yanbo Fang, Ruixiang Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01332
• Pdf URL: https://arxiv.org/pdf/2501.01332
• Copy Paste: [[2501.01332]] Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension(https://arxiv.org/abs/2501.01332)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Understanding how large language models (LLMs) acquire, retain, and apply knowledge remains an open challenge. This paper introduces a novel framework, K-(CSA)^2, which categorizes LLM knowledge along two dimensions: correctness and confidence. The framework defines six categories of knowledge, ranging from highly confident correctness to confidently held misconceptions, enabling a nuanced evaluation of model comprehension beyond binary accuracy. Using this framework, we demonstrate how techniques like chain-of-thought prompting and reinforcement learning with human feedback fundamentally alter the knowledge structures of internal (pre-trained) and external (context-dependent) knowledge in LLMs. CoT particularly enhances base model performance and shows synergistic benefits when applied to aligned LLMs. Moreover, our layer-wise analysis reveals that higher layers in LLMs encode more high-confidence knowledge, while low-confidence knowledge tends to emerge in middle-to-lower layers.
• 摘要：了解大型语言模型 (LLM) 如何获取、保留和应用知识仍然是一个悬而未决的挑战。本文介绍了一个新框架 K-(CSA)^2，该框架将 LLM 知识分为两个维度：正确性和置信度。该框架定义了六类知识，从高度自信的正确性到自信的误解，从而能够对模型理解进行超越二进制准确性的细致评估。使用这个框架，我们展示了诸如思路提示和带有人工反馈的强化学习等技术如何从根本上改变 LLM 中内部（预训练）和外部（上下文相关）知识的知识结构。CoT 特别增强了基础模型的性能，并且在应用于对齐的 LLM 时显示出协同效益。此外，我们的分层分析表明，LLM 中的较高层编码了更多高置信度知识，而低置信度知识往往出现在中层到较低层。

Title: Aligning Large Language Models for Faithful Integrity Against Opposing Argument

• Authors: Yong Zhao, Yang Deng, See-Kiong Ng, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01336
• Pdf URL: https://arxiv.org/pdf/2501.01336
• Copy Paste: [[2501.01336]] Aligning Large Language Models for Faithful Integrity Against Opposing Argument(https://arxiv.org/abs/2501.01336)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks. However, they can be easily misled by unfaithful arguments during conversations, even when their original statements are correct. To this end, we investigate the problem of maintaining faithful integrity in LLMs. This involves ensuring that LLMs adhere to their faithful statements in the face of opposing arguments and are able to correct their incorrect statements when presented with faithful arguments. In this work, we propose a novel framework, named Alignment for Faithful Integrity with Confidence Estimation (AFICE), which aims to align the LLM responses with faithful integrity. Specifically, AFICE first designs a Bilateral Confidence Estimation (BCE) approach for estimating the uncertainty of each response generated by the LLM given a specific context, which simultaneously estimate the model's confidence to the question based on the internal states during decoding as well as to the answer based on cumulative probability ratios. With the BCE, we construct a conversational preference dataset composed of context, original statement, and argument, which is adopted for aligning the LLM for faithful integrity using Direct Preference Optimization (DPO). Extensive experimental results on a wide range of benchmarks demonstrate significant improvements in the LLM's ability to maintain faithful responses when encountering opposing arguments, ensuring both the practical utility and trustworthiness of LLMs in complex interactive settings. Code and data will be released via this https URL
• 摘要：大型语言模型 (LLM) 在复杂的推理任务中表现出了令人印象深刻的能力。然而，在对话过程中，即使原始陈述是正确的，它们也很容易被不忠实的论点误导。为此，我们研究了在 LLM 中保持忠实完整性的问题。这涉及确保 LLM 在面对反对论点时坚持其忠实陈述，并能够在出现忠实论点时纠正其不正确的陈述。在这项工作中，我们提出了一个新框架，名为“带置信度估计的忠实完整性对齐 (AFICE)”，旨在使 LLM 响应与忠实完整性对齐。具体而言，AFICE 首先设计了一种双边置信度估计 (BCE) 方法来估计给定特定上下文时 LLM 生成的每个响应的不确定性，该方法同时根据解码过程中的内部状态估计模型对问题的置信度，并根据累积概率比估计模型对答案的置信度。利用 BCE，我们构建了一个由上下文、原始陈述和论点组成的对话偏好数据集，该数据集用于使用直接偏好优化 (DPO) 对齐 LLM 以实现忠实完整性。在各种基准上进行的大量实验结果表明，LLM 在遇到反对论点时保持忠实响应的能力得到了显著提高，从而确保了 LLM 在复杂交互环境中的实用性和可信度。代码和数据将通过此 https URL 发布

Title: Training Medical Large Vision-Language Models with Abnormal-Aware Feedback

• Authors: Yucheng Zhou, Lingran Song, Jianbing Shen
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01377
• Pdf URL: https://arxiv.org/pdf/2501.01377
• Copy Paste: [[2501.01377]] Training Medical Large Vision-Language Models with Abnormal-Aware Feedback(https://arxiv.org/abs/2501.01377)
• Keywords: language model, gpt, prompt
• Abstract: Existing Medical Large Vision-Language Models (Med-LVLMs), which encapsulate extensive medical knowledge, demonstrate excellent capabilities in understanding medical images and responding to human queries based on these images. However, there remain challenges in visual localization in medical images, which is crucial for abnormality detection and interpretation. To address these issues, we propose a novel UMed-LVLM designed with Unveiling Medical abnormalities. Specifically, we collect a Medical Abnormalities Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM training. To collect MAU dataset, we propose a prompt method utilizing the GPT-4V to generate diagnoses based on identified abnormal areas in medical images. Moreover, the two-stage training method includes Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, comprising Abnormal Localization Rewarding and Vision Relevance Rewarding. Experimental results demonstrate that our UMed-LVLM surpasses existing Med-LVLMs in identifying and understanding medical abnormality. In addition, this work shows that enhancing the abnormality detection capabilities of Med-LVLMs significantly improves their understanding of medical images and generalization capability.
• 摘要：现有的医学大型视觉语言模型 (Med-LVLM) 囊括了广泛的医学知识，在理解医学图像和响应基于这些图像的人类查询方面表现出色。然而，在医学图像中的视觉定位方面仍然存在挑战，这对于异常检测和解释至关重要。为了解决这些问题，我们提出了一种新型的 UMed-LVLM，其设计采用了揭示医学异常的方法。具体来说，我们收集了一个医学异常揭示 (MAU) 数据集，并提出了一种两阶段的 UMed-LVLM 训练方法。为了收集 MAU 数据集，我们提出了一种利用 GPT-4V 根据医学图像中识别出的异常区域生成诊断的快速方法。此外，两阶段训练方法包括异常感知指令调整和异常感知奖励，包括异常定位奖励和视觉相关性奖励。实验结果表明，我们的 UMed-LVLM 在识别和理解医学异常方面超越了现有的 Med-LVLM。此外，这项工作表明，增强 Med-LVLM 的异常检测能力可显著提高其对医学图像的理解和泛化能力。

Title: OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios

• Authors: Xize Cheng, Dongjie Fu, Xiaoda Yang, Minghui Fang, Ruofan Hu, Jingyu Lu, Bai Jionghao, Zehan Wang, Shengpeng Ji, Rongjie Huang, Linjun Li, Yu Chen, Tao Jin, Zhou Zhao
• Subjects: cs.CL, cs.HC, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.01384
• Pdf URL: https://arxiv.org/pdf/2501.01384
• Copy Paste: [[2501.01384]] OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios(https://arxiv.org/abs/2501.01384)
• Keywords: language model, chat
• Abstract: With the rapid development of large language models, researchers have created increasingly advanced spoken dialogue systems that can naturally converse with humans. However, these systems still struggle to handle the full complexity of real-world conversations, including audio events, musical contexts, and emotional expressions, mainly because current dialogue datasets are constrained in both scale and scenario diversity. In this paper, we propose leveraging synthetic data to enhance the dialogue models across diverse scenarios. We introduce ShareChatX, the first comprehensive, large-scale dataset for spoken dialogue that spans diverse scenarios. Based on this dataset, we introduce OmniChat, a multi-turn dialogue system with a heterogeneous feature fusion module, designed to optimize feature selection in different dialogue contexts. In addition, we explored critical aspects of training dialogue systems using synthetic data. Through comprehensive experimentation, we determined the ideal balance between synthetic and real data, achieving state-of-the-art results on the real-world dialogue dataset DailyTalk. We also highlight the crucial importance of synthetic data in tackling diverse, complex dialogue scenarios, especially those involving audio and music. For more details, please visit our demo page at \url{this https URL}.
• 摘要：随着大型语言模型的快速发展，研究人员已经创建了越来越先进的口语对话系统，可以自然地与人类交谈。然而，这些系统仍然难以处理现实世界对话的全部复杂性，包括音频事件、音乐背景和情感表达，主要是因为当前的对话数据集在规模和场景多样性方面都受到限制。在本文中，我们建议利用合成数据来增强不同场景中的对话模型。我们推出了 ShareChatX，这是第一个涵盖不同场景的全面、大规模口语对话数据集。基于此数据集，我们推出了 OmniChat，这是一个具有异构特征融合模块的多轮对话系统，旨在优化不同对话环境中的特征选择。此外，我们探索了使用合成数据训练对话系统的关键方面。通过全面的实验，我们确定了合成数据和真实数据之间的理想平衡，并在现实世界对话数据集 DailyTalk 上取得了最先进的结果。我们还强调了合成数据在处理多样化、复杂的对话场景（尤其是涉及音频和音乐的对话场景）方面至关重要。欲了解更多详情，请访问我们的演示页面 \url{this https URL}。