2025-02-06

Title: A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

• Authors: Yan Li, Tianyi Zhang, Zechuan Li, Soyeon Caren Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02659
• Pdf URL: https://arxiv.org/pdf/2502.02659
• Copy Paste: [[2502.02659]] A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)(https://arxiv.org/abs/2502.02659)
• Keywords: language model, llm
• Abstract: Transformer-based Large Language Models (LLMs) struggle to process inputs exceeding their training context window, with performance degrading due to positional out-of-distribution (O.O.D.) that disrupt attention computations. Existing solutions, fine-tuning and training-free methods, are limited by computational inefficiency, attention logit outliers or loss of local positional information. To address this, we propose Greedy Attention Logit Interpolation (GALI), a training-free length extrapolation method that maximizes the utilization of pretrained positional intervals while avoiding attention logit outliers through attention logit interpolation. The result demonstrates that GALI consistently outperforms state-of-the-art training-free methods. Our findings reveal that LLMs interpret positional intervals unevenly within their training context window, suggesting that extrapolating within a smaller positional interval range yields superior results-even for short-context tasks. GALI represents a significant step toward resolving the positional O.O.D. challenge, enabling more reliable long-text understanding in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at this https URL.
• 摘要：基于 Transformer 的大型语言模型 (LLM) 难以处理超出其训练上下文窗口的输入，由于位置失配 (O.O.D.) 会扰乱注意力计算，导致性能下降。现有的解决方案、微调和免训练方法受到计算效率低下、注意力 logit 异常值或局部位置信息丢失的限制。为了解决这个问题，我们提出了贪婪注意力 logit 插值 (GALI)，这是一种免训练长度外推方法，可最大限度地利用预训练的位置间隔，同时通过注意力 logit 插值避免注意力 logit 异常值。结果表明，GALI 始终优于最先进的免训练方法。我们的研究结果表明，LLM 在其训练上下文窗口内对位置间隔的解释不均匀，这表明在较小的位置间隔范围内进行外推会产生更好的结果——即使对于短上下文任务也是如此。GALI 代表着朝着解决位置 O.O.D. 迈出的重要一步。挑战，使 LLM 中的长文本理解更加可靠。我们对 GALI 的实现以及我们论文中的实验都在此 https URL 上开源。

Title: Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes

• Authors: Mayuka Jayawardhana (1), Renbo Tu (2), Samuel Dooley (3), Valeriia Cherepanova (4), Andrew Gordon Wilson (5), Frank Hutter (6), Colin White (7), Tom Goldstein (1), Micah Goldblum (8) ((1) University of Maryland, (2) University of Toronto, (3) Meta, (4) Amazon, (5) New York University, (6) University of Freiburg, (7) this http URL, (8) Columbia University)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02672
• Pdf URL: https://arxiv.org/pdf/2502.02672
• Copy Paste: [[2502.02672]] Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes(https://arxiv.org/abs/2502.02672)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) perform remarkably well on tabular datasets in zero- and few-shot settings, since they can extract meaning from natural language column headers that describe features and labels. Similarly, TabPFN, a recent non-LLM transformer pretrained on numerous tables for in-context learning, has demonstrated excellent performance for dataset sizes up to a thousand samples. In contrast, gradient-boosted decision trees (GBDTs) are typically trained from scratch on each dataset without benefiting from pretraining data and must learn the relationships between columns from their entries alone since they lack natural language understanding. LLMs and TabPFN excel on small tabular datasets where a strong prior is essential, yet they are not competitive with GBDTs on medium or large datasets, since their context lengths are limited. In this paper, we propose a simple and lightweight approach for fusing large language models and TabPFN with gradient-boosted decision trees, which allows scalable GBDTs to benefit from the natural language capabilities and pretraining of transformers. We name our fusion methods LLM-Boost and PFN-Boost, respectively. While matching or surpassing the performance of the transformer at sufficiently small dataset sizes and GBDTs at sufficiently large sizes, LLM-Boost and PFN-Boost outperform both standalone components on a wide range of dataset sizes in between. We demonstrate state-of-the-art performance against numerous baselines and ensembling algorithms. We find that PFN-Boost achieves the best average performance among all methods we test for all but very small dataset sizes. We release our code at this http URL .
• 摘要：大型语言模型 (LLM) 在零样本和少样本设置下的表格数据集上表现非常出色，因为它们可以从描述特征和标签的自然语言列标题中提取含义。同样，TabPFN 是一种最近的非 LLM 转换器，在大量表格上进行预训练以进行上下文学习，在高达一千个样本的数据集大小下表现出色。相比之下，梯度提升决策树 (GBDT) 通常在每个数据集上从头开始训练，没有从预训练数据中受益，并且必须仅从其条目中学习列之间的关系，因为它们缺乏自然语言理解。LLM 和 TabPFN 在小型表格数据集上表现出色，其中强大的先验是必不可少的，但它们在中型或大型数据集上无法与 GBDT 竞争，因为它们的上下文长度有限。在本文中，我们提出了一种简单而轻量级的方法，用于将大型语言模型和 TabPFN 与梯度提升决策树融合，这使得可扩展的 GBDT 能够从自然语言功能和转换器的预训练中受益。我们将融合方法分别命名为 LLM-Boost 和 PFN-Boost。虽然 LLM-Boost 和 PFN-Boost 在足够小的数据集大小下可匹敌或超越 Transformer 的性能，在足够大的数据集大小下可匹敌或超越 GBDT 的性能，但在介于两者之间的各种数据集大小上，它们的表现均优于两个独立组件。我们针对众多基线和集成算法展示了最先进的性能。我们发现，除了非常小的数据集大小外，PFN-Boost 在我们测试的所有方法中都实现了最佳平均性能。我们在此 http URL 上发布了我们的代码。

Title: How Inclusively do LMs Perceive Social and Moral Norms?

• Authors: Michael Galarnyk, Agam Shah, Dipanwita Guhathakurta, Poojitha Nandigam, Sudheer Chava
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02696
• Pdf URL: https://arxiv.org/pdf/2502.02696
• Copy Paste: [[2502.02696]] How Inclusively do LMs Perceive Social and Moral Norms?(https://arxiv.org/abs/2502.02696)
• Keywords: language model, prompt
• Abstract: This paper discusses and contains offensive content. Language models (LMs) are used in decision-making systems and as interactive assistants. However, how well do these models making judgements align with the diversity of human values, particularly regarding social and moral norms? In this work, we investigate how inclusively LMs perceive norms across demographic groups (e.g., gender, age, and income). We prompt 11 LMs on rules-of-thumb (RoTs) and compare their outputs with the existing responses of 100 human annotators. We introduce the Absolute Distance Alignment Metric (ADA-Met) to quantify alignment on ordinal questions. We find notable disparities in LM responses, with younger, higher-income groups showing closer alignment, raising concerns about the representation of marginalized perspectives. Our findings highlight the importance of further efforts to make LMs more inclusive of diverse human values. The code and prompts are available on GitHub under the CC BY-NC 4.0 license.
• 摘要：本文讨论并包含令人反感的内容。语言模型 (LM) 用于决策系统和交互式助手。然而，这些做出判断的模型与人类价值观的多样性，尤其是社会和道德规范的多样性，有多好的契合度？在这项工作中，我们调查了 LM 如何包容不同人口群体（例如性别、年龄和收入）的规范。我们提示 11 个 LM 遵循经验法则 (RoT)，并将其输出与 100 名人类注释者的现有响应进行比较。我们引入了绝对距离对齐度量 (ADA-Met) 来量化序数问题的对齐。我们发现 LM 的响应存在明显差异，年轻、高收入群体的对齐程度更高，这引发了人们对边缘化观点的代表性的担忧。我们的研究结果强调了进一步努力使 LM 更具包容性的重要性，以包容不同的人类价值观。代码和提示可在 GitHub 上获得，根据 CC BY-NC 4.0 许可。

Title: Cross-Lingual Transfer for Low-Resource Natural Language Processing

• Authors: Iker García-Ferrero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02722
• Pdf URL: https://arxiv.org/pdf/2502.02722
• Copy Paste: [[2502.02722]] Cross-Lingual Transfer for Low-Resource Natural Language Processing(https://arxiv.org/abs/2502.02722)
• Keywords: language model
• Abstract: Natural Language Processing (NLP) has seen remarkable advances in recent years, particularly with the emergence of Large Language Models that have achieved unprecedented performance across many tasks. However, these developments have mainly benefited a small number of high-resource languages such as English. The majority of languages still face significant challenges due to the scarcity of training data and computational resources. To address this issue, this thesis focuses on cross-lingual transfer learning, a research area aimed at leveraging data and models from high-resource languages to improve NLP performance for low-resource languages. Specifically, we focus on Sequence Labeling tasks such as Named Entity Recognition, Opinion Target Extraction, and Argument Mining. The research is structured around three main objectives: (1) advancing data-based cross-lingual transfer learning methods through improved translation and annotation projection techniques, (2) developing enhanced model-based transfer learning approaches utilizing state-of-the-art multilingual models, and (3) applying these methods to real-world problems while creating open-source resources that facilitate future research in low-resource NLP. More specifically, this thesis presents a new method to improve data-based transfer with T-Projection, a state-of-the-art annotation projection method that leverages text-to-text multilingual models and machine translation systems. T-Projection significantly outperforms previous annotation projection methods by a wide margin. For model-based transfer, we introduce a constrained decoding algorithm that enhances cross-lingual Sequence Labeling in zero-shot settings using text-to-text models. Finally, we develop Medical mT5, the first multilingual text-to-text medical model, demonstrating the practical impact of our research on real-world applications.
• 摘要：近年来，自然语言处理 (NLP) 取得了显著进展，尤其是大型语言模型的出现，它们在许多任务中取得了前所未有的性能。然而，这些发展主要惠及少数资源丰富的语言，如英语。由于训练数据和计算资源的稀缺，大多数语言仍然面临重大挑战。为了解决这个问题，本论文重点研究跨语言迁移学习，这是一个研究领域，旨在利用资源丰富的语言的数据和模型来提高资源匮乏语言的 NLP 性能。具体来说，我们专注于序列标记任务，如命名实体识别、意见目标提取和论据挖掘。研究围绕三个主要目标展开：(1) 通过改进的翻译和注释投影技术推进基于数据的跨语言迁移学习方法，(2) 利用最先进的多语言模型开发增强的基于模型的迁移学习方法，以及 (3) 将这些方法应用于实际问题，同时创建开源资源，以促进未来对资源匮乏的 NLP 的研究。更具体地说，本论文提出了一种使用 T-Projection 改进基于数据的传输的新方法，T-Projection 是一种最先进的注释投影方法，利用文本到文本的多语言模型和机器翻译系统。T-Projection 的表现远远优于以前的注释投影方法。对于基于模型的传输，我们引入了一种受限解码算法，该算法使用文本到文本模型增强了零样本设置中的跨语言序列标记。最后，我们开发了 Medical mT5，这是第一个多语言文本到文本医学模型，展示了我们的研究对实际应用的实际影响。

Title: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

• Authors: Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, Thomas Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02737
• Pdf URL: https://arxiv.org/pdf/2502.02737
• Copy Paste: [[2502.02737]] SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model(https://arxiv.org/abs/2502.02737)
• Keywords: language model, llm
• Abstract: While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.
• 摘要：虽然大型语言模型促进了人工智能许多应用的突破，但它们固有的庞大性使其在计算上成本高昂，并且在资源受限的环境中难以部署。在本文中，我们记录了 SmolLM2 的开发，这是一种最先进的“小型”（17 亿个参数）语言模型 (LM)。为了获得强大的性能，我们使用多阶段训练过程对约 11 万亿个标记的数据进行了 SmolLM2 过度训练，该过程将网络文本与专门的数学、代码和指令跟踪数据混合在一起。我们还在发现现有数据集过小或质量低下的阶段引入了新的专用数据集（FineMath、Stack-Edu 和 SmolTalk）。为了为我们的设计决策提供信息，我们执行小规模消融以及手动细化过程，根据前一阶段的性能更新每个阶段的数据集混合率。最终，我们证明了 SmolLM2 的表现优于其他近期的小型 LM，包括 Qwen2.5-1.5B 和 Llama3.2-1B。为了促进未来对 LM 开发以及小型 LM 应用的研究，我们发布了 SmolLM2 以及我们在此项目过程中准备的所有数据集。

Title: SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models

• Authors: Amirhossein Dabiriaghdam, Lele Wang
• Subjects: cs.CL, cs.CR, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02787
• Pdf URL: https://arxiv.org/pdf/2502.02787
• Copy Paste: [[2502.02787]] SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models(https://arxiv.org/abs/2502.02787)
• Keywords: language model, llm
• Abstract: The rapid proliferation of large language models (LLMs) has created an urgent need for reliable methods to detect whether a text is generated by such models. In this paper, we propose SimMark, a posthoc watermarking algorithm that makes LLMs' outputs traceable without requiring access to the model's internal logits, enabling compatibility with a wide range of LLMs, including API-only models. By leveraging the similarity of semantic sentence embeddings and rejection sampling to impose detectable statistical patterns imperceptible to humans, and employing a soft counting mechanism, SimMark achieves robustness against paraphrasing attacks. Experimental results demonstrate that SimMark sets a new benchmark for robust watermarking of LLM-generated content, surpassing prior sentence-level watermarking techniques in robustness, sampling efficiency, and applicability across diverse domains, all while preserving the text quality.
• 摘要：大型语言模型 (LLM) 的快速普及迫切需要可靠的方法来检测文本是否由此类模型生成。在本文中，我们提出了一种事后水印算法 SimMark，该算法无需访问模型的内部逻辑即可跟踪 LLM 的输出，从而能够与各种 LLM（包括仅限 API 的模型）兼容。通过利用语义句子嵌入的相似性和拒绝采样来施加人类无法察觉的可检测统计模式，并采用软计数机制，SimMark 实现了对释义攻击的鲁棒性。实验结果表明，SimMark 为 LLM 生成内容的鲁棒水印设定了新的基准，在鲁棒性、采样效率和跨不同领域的适用性方面超越了之前的句子级水印技术，同时保持了文本质量。

Title: Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

• Authors: Jingyu Liu, Beidi Chen, Ce Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02789
• Pdf URL: https://arxiv.org/pdf/2502.02789
• Copy Paste: [[2502.02789]] Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation(https://arxiv.org/abs/2502.02789)
• Keywords: language model, llm, prompt
• Abstract: Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Because optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is purely compute-bounded and the performance bottleneck shifts from the self-attention to the MLP part. We present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to still preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to $7\times$ maximal end-to-end QPS on real downstream tasks and $7.66\times$ TTFT improvement during benchmarking.
• 摘要：改进第一个标记时间 (TTFT) 是现代大型语言模型 (LLM) 推理引擎中一个非常重要的目标。因为优化 TTFT 可直接提高最大 QPS 并满足许多关键应用程序的要求。然而，提升 TTFT 非常具有挑战性，因为它完全受计算限制，并且性能瓶颈从自注意力转移到 MLP 部分。我们提出了 SpecPrefill，这是一个无需训练的框架，它基于以下见解加速了长上下文查询和中上下文查询的推理 TTFT：LLM 足够通用，仅给出精心选择的提示标记子集即可保持质量。SpecPrefill 的核心是利用轻量级模型根据上下文推测本地重要标记。然后，这些标记连同必要的位置信息一起发送到主模型进行处理。我们通过一组不同的任务评估 SpecPrefill，然后在真实的端到端设置和消融研究中对性能改进进行全面的基准测试。 SpecPrefill 成功为 Llama-3.1-405B-Instruct-FP8 提供服务，在实际下游任务上实现高达 $7\times$ 的最大端到端 QPS，并在基准测试期间实现 $7.66\times$ 的 TTFT 改进。

Title: CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration

• Authors: Yizhe Yang, Palakorn Achananuparp, Heyan Huang, Jing Jiang, Kit Phey Leng, Nicholas Gabriel Lim, Cameron Tan Shi Ern, Ee-peng Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02807
• Pdf URL: https://arxiv.org/pdf/2502.02807
• Copy Paste: [[2502.02807]] CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration(https://arxiv.org/abs/2502.02807)
• Keywords: language model, llm, agent
• Abstract: Conversational counselor agents have become essential tools for addressing the rising demand for scalable and accessible mental health support. This paper introduces CAMI, a novel automated counselor agent grounded in Motivational Interviewing (MI) -- a client-centered counseling approach designed to address ambivalence and facilitate behavior change. CAMI employs a novel STAR framework, consisting of client's state inference, motivation topic exploration, and response generation modules, leveraging large language models (LLMs). These components work together to evoke change talk, aligning with MI principles and improving counseling outcomes for clients from diverse backgrounds. We evaluate CAMI's performance through both automated and manual evaluations, utilizing simulated clients to assess MI skill competency, client's state inference accuracy, topic exploration proficiency, and overall counseling success. Results show that CAMI not only outperforms several state-of-the-art methods but also shows more realistic counselor-like behavior. Additionally, our ablation study underscores the critical roles of state inference and topic exploration in achieving this performance.
• 摘要：对话咨询代理已成为满足日益增长的可扩展和可访问心理健康支持需求的重要工具。本文介绍了 CAMI，这是一种基于动机访谈 (MI) 的新型自动化咨询代理——这是一种以客户为中心的咨询方法，旨在解决矛盾心理并促进行为改变。CAMI 采用一种新颖的 STAR 框架，包括客户状态推理、动机主题探索和响应生成模块，利用大型语言模型 (LLM)。这些组件共同作用以引发改变谈话，符合 MI 原则并改善来自不同背景的客户的咨询结果。我们通过自动和手动评估来评估 CAMI 的性能，利用模拟客户来评估 MI 技能能力、客户状态推理准确性、主题探索能力和整体咨询成功率。结果表明，CAMI 不仅优于几种最先进的方法，而且还表现出更现实的咨询师行为。此外，我们的消融研究强调了状态推理和主题探索在实现这一表现方面的关键作用。

Title: Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning

• Authors: Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S. Yu, Carla Gomes, Bart Selman, Qingsong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02871
• Pdf URL: https://arxiv.org/pdf/2502.02871
• Copy Paste: [[2502.02871]] Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning(https://arxiv.org/abs/2502.02871)
• Keywords: language model, llm
• Abstract: Scientific reasoning, the process through which humans apply logic, evidence, and critical thinking to explore and interpret scientific phenomena, is essential in advancing knowledge reasoning across diverse fields. However, despite significant progress, current scientific reasoning models still struggle with generalization across domains and often fall short of multimodal perception. Multimodal Large Language Models (MLLMs), which integrate text, images, and other modalities, present an exciting opportunity to overcome these limitations and enhance scientific reasoning. Therefore, this position paper argues that MLLMs can significantly advance scientific reasoning across disciplines such as mathematics, physics, chemistry, and biology. First, we propose a four-stage research roadmap of scientific reasoning capabilities, and highlight the current state of MLLM applications in scientific reasoning, noting their ability to integrate and reason over diverse data types. Second, we summarize the key challenges that remain obstacles to achieving MLLM's full potential. To address these challenges, we propose actionable insights and suggestions for the future. Overall, our work offers a novel perspective on MLLM integration with scientific reasoning, providing the LLM community with a valuable vision for achieving Artificial General Intelligence (AGI).
• 摘要：科学推理是人类运用逻辑、证据和批判性思维来探索和解释科学现象的过程，它对于推进不同领域的知识推理至关重要。然而，尽管取得了重大进展，当前的科学推理模型仍然难以实现跨领域的泛化，而且往往达不到多模态感知的水平。多模态大型语言模型 (MLLM) 集成了文本、图像和其他模态，为克服这些限制并增强科学推理提供了令人兴奋的机会。因此，本立场文件认为，MLLM 可以显著推进数学、物理、化学和生物等学科的科学推理。首先，我们提出了科学推理能力的四阶段研究路线图，并重点介绍了 MLLM 在科学推理中的应用现状，并指出了它们整合和推理各种数据类型的能力。其次，我们总结了阻碍 MLLM 充分发挥潜力的关键挑战。为了应对这些挑战，我们提出了可行的见解和未来建议。总的来说，我们的工作为 MLLM 与科学推理的结合提供了新颖的视角，为 LLM 社区提供了实现通用人工智能 (AGI) 的宝贵愿景。

Title: Lowering the Barrier of Machine Learning: Achieving Zero Manual Labeling in Review Classification Using LLMs

• Authors: Yejian Zhang, Shingo Takada
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02893
• Pdf URL: https://arxiv.org/pdf/2502.02893
• Copy Paste: [[2502.02893]] Lowering the Barrier of Machine Learning: Achieving Zero Manual Labeling in Review Classification Using LLMs(https://arxiv.org/abs/2502.02893)
• Keywords: language model, gpt, llm
• Abstract: With the internet's evolution, consumers increasingly rely on online reviews for service or product choices, necessitating that businesses analyze extensive customer feedback to enhance their offerings. While machine learning-based sentiment classification shows promise in this realm, its technical complexity often bars small businesses and individuals from leveraging such advancements, which may end up making the competitive gap between small and large businesses even bigger in terms of improving customer satisfaction. This paper introduces an approach that integrates large language models (LLMs), specifically Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT)-based models, making it accessible to a wider audience. Our experiments across various datasets confirm that our approach retains high classification accuracy without the need for manual labeling, expert knowledge in tuning and data annotation, or substantial computational power. By significantly lowering the barriers to applying sentiment classification techniques, our methodology enhances competitiveness and paves the way for making machine learning technology accessible to a broader audience.
• 摘要：随着互联网的发展，消费者越来越依赖在线评论来选择服务或产品，这就要求企业分析大量客户反馈以改进其产品。虽然基于机器学习的情绪分类在这一领域前景光明，但其技术复杂性往往会阻碍小型企业和个人利用此类进步，这最终可能会进一步拉大小型企业在提高客户满意度方面的竞争差距。本文介绍了一种集成大型语言模型 (LLM) 的方法，特别是基于生成式预训练 Transformer (GPT) 和 Transformer 双向编码器表示 (BERT) 的模型，使其可供更广泛的受众使用。我们在各种数据集上进行的实验证实，我们的方法无需手动标记、调整和数据注释方面的专业知识或大量计算能力即可保持较高的分类准确率。通过显著降低应用情绪分类技术的门槛，我们的方法提高了竞争力，并为让更广泛的受众能够使用机器学习技术铺平了道路。

Title: A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs

• Authors: Bradley P. Allen, Paul T. Groth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02896
• Pdf URL: https://arxiv.org/pdf/2502.02896
• Copy Paste: [[2502.02896]] A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs(https://arxiv.org/abs/2502.02896)
• Keywords: language model, llm, agent
• Abstract: Evaluating large language models (LLMs) for tasks like fact extraction in support of knowledge graph construction frequently involves computing accuracy metrics using a ground truth benchmark based on a knowledge graph (KG). These evaluations assume that errors represent factual disagreements. However, human discourse frequently features metalinguistic disagreement, where agents differ not on facts but on the meaning of the language used to express them. Given the complexity of natural language processing and generation using LLMs, we ask: do metalinguistic disagreements occur between LLMs and KGs? Based on an investigation using the T-REx knowledge alignment dataset, we hypothesize that metalinguistic disagreement does in fact occur between LLMs and KGs, with potential relevance for the practice of knowledge graph engineering. We propose a benchmark for evaluating the detection of factual and metalinguistic disagreements between LLMs and KGs. An initial proof of concept of such a benchmark is available on Github.
• 摘要：评估大型语言模型 (LLM) 是否可用于执行事实提取等任务以支持知识图谱构建，通常需要使用基于知识图谱 (KG) 的地面实况基准来计算准确度指标。这些评估假设错误代表事实分歧。然而，人类话语中经常存在元语言分歧，代理之间的分歧不在于事实，而在于用于表达事实的语言的含义。鉴于使用 LLM 进行自然语言处理和生成的复杂性，我们要问：LLM 和 KG 之间是否存在元语言分歧？基于使用 T-REx 知识对齐数据集进行的调查，我们假设 LLM 和 KG 之间确实存在元语言分歧，这可能与知识图谱工程实践相关。我们提出了一个基准来评估 LLM 和 KG 之间事实和元语言分歧的检测。此类基准的初步概念证明可在 Github 上找到。

Title: LLaVAC: Fine-tuning LLaVA as a Multimodal Sentiment Classifier

• Authors: T. Chay-intr, Y. Chen, K. Viriyayudhakorn, T. Theeramunkong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02938
• Pdf URL: https://arxiv.org/pdf/2502.02938
• Copy Paste: [[2502.02938]] LLaVAC: Fine-tuning LLaVA as a Multimodal Sentiment Classifier(https://arxiv.org/abs/2502.02938)
• Keywords: prompt
• Abstract: We present LLaVAC, a method for constructing a classifier for multimodal sentiment analysis. This method leverages fine-tuning of the Large Language and Vision Assistant (LLaVA) to predict sentiment labels across both image and text modalities. Our approach involves designing a structured prompt that incorporates both unimodal and multimodal labels to fine-tune LLaVA, enabling it to perform sentiment classification effectively. Experiments on the MVSA-Single dataset demonstrate that LLaVAC outperforms existing methods in multimodal sentiment analysis across three data processing procedures. The implementation of LLaVAC is publicly available at this https URL.
• 摘要：我们提出了一种构建多模态情绪分析分类器的方法 LLaVAC。该方法利用大型语言和视觉助手 (LLaVA) 的微调来预测图像和文本模态中的情绪标签。我们的方法涉及设计一个结构化的提示，结合单模态和多模态标签来微调 LLaVA，使其能够有效地执行情绪分类。在 MVSA-Single 数据集上进行的实验表明，LLaVAC 在三个数据处理过程中的多模态情绪分析中优于现有方法。LLaVAC 的实现可在此 https URL 上公开获取。

Title: LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction

• Authors: Ziwei Wang, Jie Zhou, Qin Chen, Min Zhang, Bo Jiang, Aimin Zhou, Qinchun Bai, Liang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02945
• Pdf URL: https://arxiv.org/pdf/2502.02945
• Copy Paste: [[2502.02945]] LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction(https://arxiv.org/abs/2502.02945)
• Keywords: language model, llm, long context
• Abstract: The knowledge tracing (KT) problem is an extremely important topic in personalized education, which aims to predict whether students can correctly answer the next question based on their past question-answer records. Prior work on this task mainly focused on learning the sequence of behaviors based on the IDs or textual information. However, these studies usually fail to capture students' sufficient behavioral patterns without reasoning with rich world knowledge about questions. In this paper, we propose a large language models (LLMs)-based framework for KT, named \texttt{\textbf{LLM-KT}}, to integrate the strengths of LLMs and traditional sequence interaction models. For task-level alignment, we design Plug-and-Play instruction to align LLMs with KT, leveraging LLMs' rich knowledge and powerful reasoning capacity. For modality-level alignment, we design the plug-in context and sequence to integrate multiple modalities learned by traditional methods. To capture the long context of history records, we present a plug-in context to flexibly insert the compressed context embedding into LLMs using question-specific and concept-specific tokens. Furthermore, we introduce a plug-in sequence to enhance LLMs with sequence interaction behavior representation learned by traditional sequence models using a sequence adapter. Extensive experiments show that \texttt{\textbf{LLM-KT}} obtains state-of-the-art performance on four typical datasets by comparing it with approximately 20 strong baselines.
• 摘要：知识追踪（KT）问题是个性化教育中极其重要的课题，其目的是根据学生过去的问答记录预测学生是否能够正确回答下一个问题。之前关于此任务的工作主要集中于根据ID或文本信息学习行为序列。然而，这些研究通常无法捕捉到学生的充分行为模式，而无需使用关于问题的丰富世界知识进行推理。在本文中，我们提出了一个基于大型语言模型（LLM）的 KT 框架，名为 \texttt{\textbf{LLM-KT}}，以整合 LLM 和传统序列交互模型的优势。对于任务级对齐，我们设计了即插即用指令将 LLM 与 KT 对齐，利用 LLM 丰富的知识和强大的推理能力。对于模态级对齐，我们设计了插件上下文和序列来整合传统方法学习到的多种模态。为了捕捉历史记录的长上下文，我们提出了一个插件上下文，使用问题特定和概念特定的标记将压缩的上下文嵌入灵活地插入到 LLM 中。此外，我们引入了一个插件序列，以使用序列适配器通过传统序列模型学习的序列交互行为表示来增强 LLM。大量实验表明，通过将 \texttt{\textbf{LLM-KT}} 与大约 20 个强基线进行比较，它在四个典型数据集上获得了最佳性能。

Title: ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation

• Authors: Qinzhuo Wu, Wei Liu, Jian Luan, Bin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02955
• Pdf URL: https://arxiv.org/pdf/2502.02955
• Copy Paste: [[2502.02955]] ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation(https://arxiv.org/abs/2502.02955)
• Keywords: agent
• Abstract: Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the IoU Acc and Text Acc by 7.12% and 7.69% on the step-level and 4.72% and 4.63% on the task-level compared to the SOTA agent. Our data and code will be released upon acceptance.
• 摘要：最近，移动 AI 代理越来越受到关注。给定一个任务，移动 AI 代理可以通过多个步骤与移动设备交互，最终形成解决任务的 GUI 流程。然而，现有的代理倾向于在每个步骤中关注与任务最相关的元素，从而导致局部最优解并忽略整体 GUI 流程。为了解决这个问题，我们构建了一个名为 MobileReach 的训练数据集，将任务分解为页面到达和操作子任务。此外，我们提出了 ReachAgent，这是一个两阶段框架，专注于提高其任务完成能力。它利用页面到达和页面操作子任务以及基于奖励的偏好 GUI 流程来进一步增强代理。实验结果表明，与 SOTA 代理相比，ReachAgent 在步骤级别上显著提高了 IoU Acc 和 Text Acc，分别提高了 7.12% 和 7.69%，在任务级别上显著提高了 4.72% 和 4.63%。我们的数据和代码将在接受后发布。

Title: Position: Editing Large Language Models Poses Serious Safety Risks

• Authors: Paul Youssef, Zhixue Zhao, Daniel Braun, Jörg Schlötterer, Christin Seifert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.02958
• Pdf URL: https://arxiv.org/pdf/2502.02958
• Copy Paste: [[2502.02958]] Position: Editing Large Language Models Poses Serious Safety Risks(https://arxiv.org/abs/2502.02958)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) contain large amounts of facts about the world. These facts can become outdated over time, which has led to the development of knowledge editing methods (KEs) that can change specific facts in LLMs with limited side effects. This position paper argues that editing LLMs poses serious safety risks that have been largely overlooked. First, we note the fact that KEs are widely available, computationally inexpensive, highly performant, and stealthy makes them an attractive tool for malicious actors. Second, we discuss malicious use cases of KEs, showing how KEs can be easily adapted for a variety of malicious purposes. Third, we highlight vulnerabilities in the AI ecosystem that allow unrestricted uploading and downloading of updated models without verification. Fourth, we argue that a lack of social and institutional awareness exacerbates this risk, and discuss the implications for different stakeholders. We call on the community to (i) research tamper-resistant models and countermeasures against malicious model editing, and (ii) actively engage in securing the AI ecosystem.
• 摘要：大型语言模型 (LLM) 包含大量有关世界的事实。这些事实可能会随着时间的推移而过时，这导致了知识编辑方法 (KE) 的发展，这些方法可以更改 LLM 中的特定事实，而副作用有限。本立场文件认为，编辑 LLM 会带来严重的安全风险，而这些风险在很大程度上被忽视了。首先，我们注意到 KE 广泛可用、计算成本低、性能高且隐秘，这使它们成为恶意行为者的诱人工具。其次，我们讨论了 KE 的恶意用例，展示了如何将 KE 轻松用于各种恶意目的。第三，我们强调了 AI 生态系统中的漏洞，这些漏洞允许不受限制地上传和下载更新的模型而无需验证。第四，我们认为缺乏社会和机构意识加剧了这种风险，并讨论了对不同利益相关者的影响。我们呼吁社区 (i) 研究防篡改模型和针对恶意模型编辑的对策，以及 (ii) 积极参与保护 AI 生态系统。

Title: Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons

• Authors: Renjun Hu, Yi Cheng, Libin Meng, Jiaxin Xia, Yi Zong, Xing Shi, Wei Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02988
• Pdf URL: https://arxiv.org/pdf/2502.02988
• Copy Paste: [[2502.02988]] Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons(https://arxiv.org/abs/2502.02988)
• Keywords: language model, llm, prompt
• Abstract: The rapid advancement of large language models (LLMs) has opened new possibilities for their adoption as evaluative judges. This paper introduces Themis, a fine-tuned LLM judge that delivers sophisticated context-aware evaluations. We provide a comprehensive overview of the development pipeline for Themis, highlighting its scenario-dependent evaluation prompts and two novel methods for controlled instruction generation. These designs enable Themis to effectively distill evaluative skills from teacher models, while retaining flexibility for continuous development. We introduce two human-labeled benchmarks for meta-evaluation, demonstrating that Themis can achieve high alignment with human preferences in an economical manner. Additionally, we explore insights into the LLM-as-a-judge paradigm, revealing nuances in performance and the varied effects of reference answers. Notably, we observe that pure knowledge distillation from strong LLMs, though common, does not guarantee performance improvement through scaling. We propose a mitigation strategy based on instruction-following difficulty. Furthermore, we provide practical guidelines covering data balancing, prompt customization, multi-objective training, and metric aggregation. We aim for our method and findings, along with the fine-tuning data, benchmarks, and model checkpoints, to support future research and development in this area.
• 摘要：大型语言模型 (LLM) 的快速发展为其作为评估评判者的应用开辟了新的可能性。本文介绍了 Themis，这是一款经过微调的 LLM 评判者，可提供复杂的上下文感知评估。我们全面概述了 Themis 的开发流程，重点介绍了其基于场景的评估提示和两种用于受控指令生成的新方法。这些设计使 Themis 能够有效地从教师模型中提取评估技能，同时保留了持续开发的灵活性。我们引入了两个人工标记的元评估基准，表明 Themis 可以以经济的方式实现与人类偏好的高度一致。此外，我们探索了对 LLM 作为评判者范式的见解，揭示了性能的细微差别和参考答案的不同影响。值得注意的是，我们观察到，从强大的 LLM 中进行纯知识提取虽然很常见，但并不能保证通过扩展来提高性能。我们提出了一种基于指令跟随难度的缓解策略。此外，我们还提供实用指南，涵盖数据平衡、快速定制、多目标训练和指标聚合。我们的目标是让我们的方法和发现以及微调数据、基准和模型检查点支持该领域的未来研究和开发。

Title: MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation

• Authors: Seonok Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03004
• Pdf URL: https://arxiv.org/pdf/2502.03004
• Copy Paste: [[2502.03004]] MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation(https://arxiv.org/abs/2502.03004)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across natural language processing tasks. However, their application to specialized domains such as medicine and biology requires further optimization to ensure factual accuracy, reliability, and contextual depth. We introduce MedBioLM, a domain-adapted biomedical question-answering model designed to enhance both short-form and long-form queries. By integrating fine-tuning and retrieval-augmented generation (RAG), MedBioLM dynamically incorporates domain-specific knowledge, improving reasoning abilities and factual accuracy. To evaluate its effectiveness, we fine-tuned the model on diverse biomedical QA datasets, covering structured multiple-choice assessments and complex clinical reasoning tasks. Fine-tuning significantly improves accuracy on benchmark datasets, while RAG enhances factual consistency. These results highlight the potential of domain-optimized LLMs in advancing biomedical research, medical education, and clinical decision support.
• 摘要：大型语言模型 (LLM) 在自然语言处理任务中表现出了令人印象深刻的能力。然而，它们在医学和生物学等专业领域的应用需要进一步优化，以确保事实准确性、可靠性和上下文深度。我们引入了 MedBioLM，这是一种领域适应的生物医学问答模型，旨在增强短格式和长格式查询。通过集成微调和检索增强生成 (RAG)，MedBioLM 动态地整合了领域特定的知识，提高了推理能力和事实准确性。为了评估其有效性，我们在各种生物医学 QA 数据集上对该模型进行了微调，涵盖了结构化的多项选择评估和复杂的临床推理任务。微调显著提高了基准数据集的准确性，而 RAG 则提高了事实一致性。这些结果凸显了领域优化的 LLM 在推进生物医学研究、医学教育和临床决策支持方面的潜力。

Title: Knowledge Distillation from Large Language Models for Household Energy Modeling

• Authors: Mohannad Takrouri, Nicolás M. Cuadrado, Martin Takáč
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03034
• Pdf URL: https://arxiv.org/pdf/2502.03034
• Copy Paste: [[2502.03034]] Knowledge Distillation from Large Language Models for Household Energy Modeling(https://arxiv.org/abs/2502.03034)
• Keywords: language model, llm, prompt
• Abstract: Machine learning (ML) is increasingly vital for smart-grid research, yet restricted access to realistic, diverse data - often due to privacy concerns - slows progress and fuels doubts within the energy sector about adopting ML-based strategies. We propose integrating Large Language Models (LLMs) in energy modeling to generate realistic, culturally sensitive, and behavior-specific data for household energy usage across diverse geographies. In this study, we employ and compare five different LLMs to systematically produce family structures, weather patterns, and daily consumption profiles for households in six distinct countries. A four-stage methodology synthesizes contextual daily data, including culturally nuanced activities, realistic weather ranges, HVAC operations, and distinct `energy signatures' that capture unique consumption footprints. Additionally, we explore an alternative strategy where external weather datasets can be directly integrated, bypassing intermediate weather modeling stages while ensuring physically consistent data inputs. The resulting dataset provides insights into how cultural, climatic, and behavioral factors converge to shape carbon emissions, offering a cost-effective avenue for scenario-based energy optimization. This approach underscores how prompt engineering, combined with knowledge distillation, can advance sustainable energy research and climate mitigation efforts. Source code is available at this https URL .
• 摘要：机器学习 (ML) 对智能电网研究越来越重要，但对现实、多样化数据的访问受限（通常是出于隐私考虑）会减缓进展，并加剧能源部门对采用基于 ML 的策略的怀疑。我们建议将大型语言模型 (LLM) 集成到能源建模中，以生成现实、文化敏感且行为特定的数据，用于不同地区的家庭能源使用情况。在本研究中，我们使用并比较了五种不同的 LLM，以系统地生成六个不同国家/地区家庭的家庭结构、天气模式和日常消费概况。四阶段方法综合了上下文日常数据，包括文化细微差别的活动、现实的天气范围、暖通空调操作和捕捉独特消费足迹的独特“能源特征”。此外，我们探索了一种替代策略，其中可以直接集成外部天气数据集，绕过中间天气建模阶段，同时确保物理上一致的数据输入。由此产生的数据集提供了有关文化、气候和行为因素如何汇聚以影响碳排放的见解，为基于场景的能源优化提供了一种经济有效的途径。这种方法强调了快速工程与知识提炼相结合如何推动可持续能源研究和气候缓解努力。源代码可从此 https URL 获取。

Title: IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates

• Authors: Aissatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03080
• Pdf URL: https://arxiv.org/pdf/2502.03080
• Copy Paste: [[2502.03080]] IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates(https://arxiv.org/abs/2502.03080)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: While Large Language Models (LLMs) demonstrate impressive reasoning capabilities, understanding and validating their knowledge utilization remains challenging. Chain-of-thought (CoT) prompting partially addresses this by revealing intermediate reasoning steps, but the knowledge flow and application remain implicit. We introduce IAO (Input-Action-Output) prompting, a structured template-based method that explicitly models how LLMs access and apply their knowledge during complex reasoning tasks. IAO decomposes problems into sequential steps, each clearly identifying the input knowledge being used, the action being performed, and the resulting output. This structured decomposition enables us to trace knowledge flow, verify factual consistency, and identify potential knowledge gaps or misapplications. Through experiments across diverse reasoning tasks, we demonstrate that IAO not only improves zero-shot performance but also provides transparency in how LLMs leverage their stored knowledge. Human evaluation confirms that this structured approach enhances our ability to verify knowledge utilization and detect potential hallucinations or reasoning errors. Our findings provide insights into both knowledge representation within LLMs and methods for more reliable knowledge application.
• 摘要：虽然大型语言模型 (LLM) 表现出令人印象深刻的推理能力，但理解和验证它们的知识利用仍然具有挑战性。思路链 (CoT) 提示通过揭示中间推理步骤部分解决了这个问题，但知识流和应用仍然是隐含的。我们引入了 IAO（输入-动作-输出）提示，这是一种基于模板的结构化方法，可明确模拟 LLM 在复杂推理任务中如何访问和应用其知识。IAO 将问题分解为连续步骤，每个步骤都清楚地标识正在使用的输入知识、正在执行的操作以及结果输出。这种结构化分解使我们能够追踪知识流、验证事实一致性并识别潜在的知识差距或误用。通过跨不同推理任务的实验，我们证明 IAO 不仅可以提高零样本性能，还可以提供 LLM 如何利用其存储知识的透明度。人工评估证实，这种结构化方法增强了我们验证知识利用率和检测潜在幻觉或推理错误的能力。我们的研究结果为 LLM 中的知识表示和更可靠的知识应用方法提供了见解。

Title: Structured Token Retention and Computational Memory Paths in Large Language Models

• Authors: Jonathan Delena, Augustin Moreau, Dominic Ravensdale, Frederick Chatterton
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03102
• Pdf URL: https://arxiv.org/pdf/2502.03102
• Copy Paste: [[2502.03102]] Structured Token Retention and Computational Memory Paths in Large Language Models(https://arxiv.org/abs/2502.03102)
• Keywords: language model
• Abstract: Memory retention mechanisms play a central role in determining the efficiency of computational architectures designed for processing extended sequences. Conventional methods for token management often impose fixed retention thresholds or rely on uniform attention weight distributions, leading to inefficient memory utilization and premature information loss in extended sequence modeling. Structured Token Retention (STR) introduces a probabilistic selection framework that dynamically adjusts token persistence based on contextual significance, ensuring that computational resources are allocated to semantically relevant elements. Computational Memory Paths (CMP) extend this framework through hierarchical memory allocation, refining retention efficiency through structured reallocation of token embeddings. Comparative assessments against baseline models demonstrate that STR and CMP improve token survival rates across long input sequences while reducing cumulative error propagation across processing layers. Experimental results further indicate reductions in computational overhead, improving inference speed without degrading contextual coherence. Token distribution analyses reveal that structured memory allocation prevents excessive redundancy in attention weight calculations, optimizing information retrieval efficiency in large-scale generative architectures. The integration of STR and CMP into an open-source model illustrates the adaptability of structured memory retention methodologies, highlighting their applicability in generative text processing, long-context comprehension, and scalable sequence modeling.
• 摘要：内存保留机制在确定用于处理扩展序列的计算架构的效率方面起着核心作用。传统的 token 管理方法通常会施加固定的保留阈值或依赖于统一的注意力权重分布，从而导致扩展序列建模中的内存利用率低下和信息过早丢失。结构化 token 保留 (STR) 引入了一个概率选择框架，该框架可根据上下文重要性动态调整 token 持久性，确保将计算资源分配给语义相关的元素。计算内存路径 (CMP) 通过分层内存分配扩展了此框架，通过结构化重新分配 token 嵌入来提高保留效率。与基线模型的比较评估表明，STR 和 CMP 提高了长输入序列中的 token 存活率，同时减少了跨处理层的累积错误传播。实验结果进一步表明计算开销减少，提高了推理速度，而不会降低上下文连贯性。token 分布分析表明，结构化内存分配可防止注意力权重计算中的过度冗余，从而优化大规模生成架构中的信息检索效率。 STR 和 CMP 集成到开源模型中说明了结构化记忆保留方法的适应性，突出了它们在生成文本处理、长上下文理解和可扩展序列建模中的适用性。

Title: Teaching Large Language Models Number-Focused Headline Generation With Key Element Rationales

• Authors: Zhen Qian, Xiuzhen Zhang, Xiaofei Xu, Feng Xia
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03129
• Pdf URL: https://arxiv.org/pdf/2502.03129
• Copy Paste: [[2502.03129]] Teaching Large Language Models Number-Focused Headline Generation With Key Element Rationales(https://arxiv.org/abs/2502.03129)
• Keywords: language model, llm, chain-of-thought
• Abstract: Number-focused headline generation is a summarization task requiring both high textual quality and precise numerical accuracy, which poses a unique challenge for Large Language Models (LLMs). Existing studies in the literature focus only on either textual quality or numerical reasoning and thus are inadequate to address this challenge. In this paper, we propose a novel chain-of-thought framework for using rationales comprising key elements of the Topic, Entities, and Numerical reasoning (TEN) in news articles to enhance the capability for LLMs to generate topic-aligned high-quality texts with precise numerical accuracy. Specifically, a teacher LLM is employed to generate TEN rationales as supervision data, which are then used to teach and fine-tune a student LLM. Our approach teaches the student LLM automatic generation of rationales with enhanced capability for numerical reasoning and topic-aligned numerical headline generation. Experiments show that our approach achieves superior performance in both textual quality and numerical accuracy.
• 摘要：以数字为中心的标题生成是一项摘要任务，需要高文本质量和精确的数字准确性，这对大型语言模型 (LLM) 提出了独特的挑战。现有文献中的研究仅关注文本质量或数字推理，因此不足以应对这一挑战。在本文中，我们提出了一种新颖的思路框架，使用由新闻文章中的主题、实体和数字推理 (TEN) 的关键元素组成的原理来增强 LLM 生成具有精确数字准确性的主题对齐高质量文本的能力。具体来说，聘请教师 LLM 生成 TEN 原理作为监督数据，然后使用这些数据来教授和微调学生 LLM。我们的方法教会学生 LLM 自动生成具有增强的数字推理能力和主题对齐数字标题生成能力的原理。实验表明，我们的方法在文本质量和数字准确性方面都取得了优异的表现。

Title: Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models

• Authors: Xumeng Wen, Shun Zheng, Zhen Xu, Yiming Sun, Jiang Bian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03147
• Pdf URL: https://arxiv.org/pdf/2502.03147
• Copy Paste: [[2502.03147]] Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2502.03147)
• Keywords: language model, llm
• Abstract: Recent studies have shown that large language models (LLMs), when customized with post-training on tabular data, can acquire general tabular in-context learning (TabICL) capabilities. These models are able to transfer effectively across diverse data schemas and different task domains. However, existing LLM-based TabICL approaches are constrained to few-shot scenarios due to the sequence length limitations of LLMs, as tabular instances represented in plain text consume substantial tokens. To address this limitation and enable scalable TabICL for any data size, we propose retrieval-augmented LLMs tailored to tabular data. Our approach incorporates a customized retrieval module, combined with retrieval-guided instruction-tuning for LLMs. This enables LLMs to effectively leverage larger datasets, achieving significantly improved performance across 69 widely recognized datasets and demonstrating promising scaling behavior. Extensive comparisons with state-of-the-art tabular models reveal that, while LLM-based TabICL still lags behind well-tuned numeric models in overall performance, it uncovers powerful algorithms under limited contexts, enhances ensemble diversity, and excels on specific datasets. These unique properties underscore the potential of language as a universal and accessible interface for scalable tabular data learning.
• 摘要：最近的研究表明，大型语言模型 (LLM) 在对表格数据进行后期训练后，可以获得通用的表格上下文学习 (TabICL) 功能。这些模型能够有效地在不同的数据模式和不同的任务域之间迁移。然而，由于 LLM 的序列长度限制，现有的基于 LLM 的 TabICL 方法仅限于少样本场景，因为以纯文本表示的表格实例会消耗大量标记。为了解决这一限制并实现适用于任何数据大小的可扩展 TabICL，我们提出了针对表格数据量身定制的检索增强 LLM。我们的方法结合了定制的检索模块，并结合了针对 LLM 的检索引导指令调整。这使 LLM 能够有效地利用更大的数据集，在 69 个广泛认可的数据集上实现显着的性能提升，并展示了有希望的扩展行为。与最先进的表格模型进行广泛比较后发现，虽然基于 LLM 的 TabICL 在整体性能上仍然落后于经过良好调整的数值模型，但它在有限的背景下发现了强大的算法，增强了集合多样性，并在特定数据集上表现出色。这些独特的属性凸显了语言作为可扩展表格数据学习的通用且可访问的界面的潜力。

Title: Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models

• Authors: Jialiang Wu, Yi Shen, Sijia Liu, Yi Tang, Sen Song, Xiaoyi Wang, Longjun Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03199
• Pdf URL: https://arxiv.org/pdf/2502.03199
• Copy Paste: [[2502.03199]] Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models(https://arxiv.org/abs/2502.03199)
• Keywords: language model, llm, hallucination
• Abstract: Despite their impressive capacities, Large language models (LLMs) often struggle with the hallucination issue of generating inaccurate or fabricated content even when they possess correct knowledge. In this paper, we extend the exploration of the correlation between hidden-state prediction changes and output factuality into a deeper, token-wise level. Based on the insights , we propose cross-layer Entropy eNhanced Decoding (END), a decoding method that mitigates hallucinations without requiring extra training. END leverages inner probability changes across layers to individually quantify the factual knowledge required for each candidate token, and adjusts the final predicting distribution to prioritize tokens with higher factuality. Experiments on both hallucination and QA benchmarks demonstrate that END significantly enhances the truthfulness and informativeness of generated content while maintaining robust QA accuracy. Moreover, our work provides a deeper perspective on understanding the correlations between inherent knowledge and output factuality.
• 摘要：尽管大型语言模型 (LLM) 具有令人印象深刻的能力，但它们常常会遇到幻觉问题，即即便拥有正确的知识，也会生成不准确或虚构的内容。在本文中，我们将隐藏状态预测变化与输出事实性之间的相关性的探索扩展到更深的、逐个标记的级别。基于这些见解，我们提出了跨层熵增强解码 (END)，这是一种无需额外训练即可减轻幻觉的解码方法。END 利用跨层的内部概率变化来单独量化每个候选标记所需的事实知识，并调整最终的预测分布以优先考虑具有更高事实性的标记。在幻觉和 QA 基准上的实验表明，END 显着提高了生成内容的真实性和信息性，同时保持了强大的 QA 准确性。此外，我们的工作为理解固有知识和输出事实性之间的相关性提供了更深入的视角。

Title: iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs

• Authors: Julius Mayer, Mohamad Ballout, Serwan Jassim, Farbod Nosrat Nezami, Elia Bruni
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.03214
• Pdf URL: https://arxiv.org/pdf/2502.03214
• Copy Paste: [[2502.03214]] iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs(https://arxiv.org/abs/2502.03214)
• Keywords: language model, agent
• Abstract: Vision-Language Models (VLMs) are known to struggle with spatial reasoning and visual alignment. To help overcome these limitations, we introduce iVISPAR, an interactive multi-modal benchmark designed to evaluate the spatial reasoning capabilities of VLMs acting as agents. iVISPAR is based on a variant of the sliding tile puzzle-a classic problem that demands logical planning, spatial awareness, and multi-step reasoning. The benchmark supports visual 2D, 3D, and text-based input modalities, enabling comprehensive assessments of VLMs' planning and reasoning skills. We evaluate a broad suite of state-of-the-art open-source and closed-source VLMs, comparing their performance while also providing optimal path solutions and a human baseline to assess the task's complexity and feasibility for humans. Results indicate that while some VLMs perform well on simple spatial tasks, they encounter difficulties with more complex configurations and problem properties. Notably, while VLMs generally perform better in 2D vision compared to 3D or text-based representations, they consistently fall short of human performance, illustrating the persistent challenge of visual alignment. This highlights critical gaps in current VLM capabilities, highlighting their limitations in achieving human-level cognition.
• 摘要：众所周知，视觉语言模型 (VLM) 在空间推理和视觉对齐方面存在困难。为了帮助克服这些限制，我们引入了 iVISPAR，这是一种交互式多模式基准，旨在评估作为代理的 VLM 的空间推理能力。iVISPAR 基于滑动拼图的变体 - 这是一个需要逻辑规划、空间意识和多步推理的经典问题。该基准支持视觉 2D、3D 和基于文本的输入模式，可以全面评估 VLM 的规划和推理能力。我们评估了一系列最先进的开源和闭源 VLM，比较了它们的性能，同时还提供了最佳路径解决方案和人类基线，以评估任务的复杂性和对人类的可行性。结果表明，虽然一些 VLM 在简单的空间任务上表现良好，但它们在更复杂的配置和问题属性方面遇到了困难。值得注意的是，虽然 VLM 在 2D 视觉方面的表现通常比 3D 或基于文本的表示更好，但它们始终无法达到人类的表现，这说明视觉对齐的挑战一直存在。这凸显了当前 VLM 功能中的关键差距，凸显了它们在实现人类水平认知方面的局限性。

Title: How do Humans and Language Models Reason About Creativity? A Comparative Analysis

• Authors: Antonio Laverghetta Jr., Tuhin Chakrabarty, Tom Hope, Jimmy Pronchick, Krupa Bhawsar, Roger E. Beaty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03253
• Pdf URL: https://arxiv.org/pdf/2502.03253
• Copy Paste: [[2502.03253]] How do Humans and Language Models Reason About Creativity? A Comparative Analysis(https://arxiv.org/abs/2502.03253)
• Keywords: language model, llm
• Abstract: Creativity assessment in science and engineering is increasingly based on both human and AI judgment, but the cognitive processes and biases behind these evaluations remain poorly understood. We conducted two experiments examining how including example solutions with ratings impact creativity evaluation, using a finegrained annotation protocol where raters were tasked with explaining their originality scores and rating for the facets of remoteness (whether the response is "far" from everyday ideas), uncommonness (whether the response is rare), and cleverness. In Study 1, we analyzed creativity ratings from 72 experts with formal science or engineering training, comparing those who received example solutions with ratings (example) to those who did not (no example). Computational text analysis revealed that, compared to experts with examples, no-example experts used more comparative language (e.g., "better/worse") and emphasized solution uncommonness, suggesting they may have relied more on memory retrieval for comparisons. In Study 2, parallel analyses with state-of-the-art LLMs revealed that models prioritized uncommonness and remoteness of ideas when rating originality, suggesting an evaluative process rooted around the semantic similarity of ideas. In the example condition, while LLM accuracy in predicting the true originality scores improved, the correlations of remoteness, uncommonness, and cleverness with originality also increased substantially - to upwards of 0.99 - suggesting a homogenization in the LLMs evaluation of the individual facets. These findings highlight important implications for how humans and AI reason about creativity and suggest diverging preferences for what different populations prioritize when rating.
• 摘要：科学和工程领域的创造力评估越来越多地基于人类和人工智能的判断，但这些评估背后的认知过程和偏见仍然不太为人所知。我们进行了两项实验，研究了包含带有评级的示例解决方案如何影响创造力评估，使用细粒度注释协议，其中评分者的任务是解释他们的原创性分数和对遥远性（响应是否与日常想法“相差甚远”），不常见性（响应是否罕见）和聪明性等方面的评级。在研究 1 中，我们分析了 72 位接受过正规科学或工程培训的专家的创造力评级，将收到带有评级的示例解决方案（示例）的专家与未收到评级的示例解决方案（无示例）的专家进行比较。计算文本分析显示，与有示例的专家相比，没有示例的专家使用更多的比较语言（例如，“更好/更差”）并强调解决方案的不常见性，这表明他们可能更多地依赖记忆检索进行比较。在研究 2 中，与最先进的 LLM 进行的平行分析表明，模型在评估原创性时优先考虑想法的不常见性和遥远性，这表明评估过程以想法的语义相似性为根基。在示例条件下，虽然 LLM 预测真实原创性分数的准确性有所提高，但遥远性、不常见性和巧妙性与原创性的相关性也大幅提高 - 达到 0.99 以上 - 表明 LLM 对各个方面的评估存在同质化。这些发现突出了人类和人工智能如何推理创造力的重要意义，并表明不同人群在评级时优先考虑的偏好不同。

Title: Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning

• Authors: DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, Qinqing Zheng
• Subjects: cs.CL, cs.AI, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2502.03275
• Pdf URL: https://arxiv.org/pdf/2502.03275
• Copy Paste: [[2502.03275]] Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning(https://arxiv.org/abs/2502.03275)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel at reasoning and planning when trained on chainof-thought (CoT) data, where the step-by-step thought process is explicitly outlined by text tokens. However, this results in lengthy inputs where many words support textual coherence rather than core reasoning information, and processing these inputs consumes substantial computation resources. In this work, we propose a hybrid representation of the reasoning process, where we partially abstract away the initial reasoning steps using latent discrete tokens generated by VQ-VAE, significantly reducing the length of reasoning traces. We explore the use of latent trace abstractions in two scenarios: 1) training the model from scratch for the Keys-Finding Maze problem, 2) fine-tuning LLMs on this hybrid data with an extended vocabulary including unseen latent tokens, for both logical and mathematical reasoning problems. To facilitate effective learning, we introduce a simple training procedure that randomly mixes latent and text tokens, which enables fast adaptation to new latent tokens. Our approach consistently outperforms the baselines methods in various benchmarks.
• 摘要：大型语言模型 (LLM) 在对思维链 (CoT) 数据进行训练时，擅长推理和规划，其中逐步的思维过程由文本标记明确概述。然而，这会导致输入冗长，其中许多单词支持文本连贯性而不是核心推理信息，并且处理这些输入会消耗大量计算资源。在这项工作中，我们提出了一种推理过程的混合表示，其中我们使用 VQ-VAE 生成的潜在离散标记部分抽象出初始推理步骤，从而显著缩短了推理轨迹的长度。我们探索了在两种情况下使用潜在轨迹抽象：1) 从头开始​​训练模型以解决钥匙寻找迷宫问题，2) 使用包括看不见的潜在标记在内的扩展词汇表对这种混合数据上的 LLM 进行微调，以解决逻辑和数学推理问题。为了促进有效学习，我们引入了一种简单的训练程序，随机混合潜在标记和文本标记，从而能够快速适应新的潜在标记。我们的方法在各种基准测试中始终优于基线方法。

Title: MeDiSumQA: Patient-Oriented Question-Answer Generation from Discharge Letters

• Authors: Amin Dada, Osman Alperen Koras, Marie Bauer, Amanda Butler, Kaleb E. Smith, Jens Kleesiek, Julian Friedrich
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03298
• Pdf URL: https://arxiv.org/pdf/2502.03298
• Copy Paste: [[2502.03298]] MeDiSumQA: Patient-Oriented Question-Answer Generation from Discharge Letters(https://arxiv.org/abs/2502.03298)
• Keywords: language model, llm
• Abstract: While increasing patients' access to medical documents improves medical care, this benefit is limited by varying health literacy levels and complex medical terminology. Large language models (LLMs) offer solutions by simplifying medical information. However, evaluating LLMs for safe and patient-friendly text generation is difficult due to the lack of standardized evaluation resources. To fill this gap, we developed MeDiSumQA. MeDiSumQA is a dataset created from MIMIC-IV discharge summaries through an automated pipeline combining LLM-based question-answer generation with manual quality checks. We use this dataset to evaluate various LLMs on patient-oriented question-answering. Our findings reveal that general-purpose LLMs frequently surpass biomedical-adapted models, while automated metrics correlate with human judgment. By releasing MeDiSumQA on PhysioNet, we aim to advance the development of LLMs to enhance patient understanding and ultimately improve care outcomes.
• 摘要：虽然增加患者获取医疗文件的渠道可以改善医疗保健，但这种好处受到不同健康素养水平和复杂医学术语的限制。大型语言模型 (LLM) 通过简化医疗信息提供解决方案。然而，由于缺乏标准化的评估资源，评估 LLM 是否安全且对患者友好，这很困难。为了填补这一空白，我们开发了 MeDiSumQA。MeDiSumQA 是一个数据集，它通过结合基于 LLM 的问答生成和手动质量检查的自动化流程从 MIMIC-IV 出院摘要中创建。我们使用此数据集评估各种 LLM 在面向患者的问答方面的表现。我们的研究结果表明，通用 LLM 经常超越生物医学适应模型，而自动化指标与人类判断相关。通过在 PhysioNet 上发布 MeDiSumQA，我们旨在推动 LLM 的发展，以增强患者理解并最终改善护理结果。

Title: Out-of-Distribution Detection using Synthetic Data Generation

• Authors: Momin Abbas, Muneeza Azmat, Raya Horesh, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03323
• Pdf URL: https://arxiv.org/pdf/2502.03323
• Copy Paste: [[2502.03323]] Out-of-Distribution Detection using Synthetic Data Generation(https://arxiv.org/abs/2502.03323)
• Keywords: language model, llm
• Abstract: Distinguishing in- and out-of-distribution (OOD) inputs is crucial for reliable deployment of classification systems. However, OOD data is typically unavailable or difficult to collect, posing a significant challenge for accurate OOD detection. In this work, we present a method that harnesses the generative capabilities of Large Language Models (LLMs) to create high-quality synthetic OOD proxies, eliminating the dependency on any external OOD data source. We study the efficacy of our method on classical text classification tasks such as toxicity detection and sentiment classification as well as classification tasks arising in LLM development and deployment, such as training a reward model for RLHF and detecting misaligned generations. Extensive experiments on nine InD-OOD dataset pairs and various model sizes show that our approach dramatically lowers false positive rates (achieving a perfect zero in some cases) while maintaining high accuracy on in-distribution tasks, outperforming baseline methods by a significant margin.
• 摘要：区分分布内和分布外 (OOD) 输入对于分类系统的可靠部署至关重要。然而，OOD 数据通常不可用或难以收集，这对准确的 OOD 检测构成了重大挑战。在这项工作中，我们提出了一种方法，利用大型语言模型 (LLM) 的生成能力来创建高质量的合成 OOD 代理，消除对任何外部 OOD 数据源的依赖。我们研究了我们的方法对经典文本分类任务（例如毒性检测和情绪分类）以及 LLM 开发和部署中出现的分类任务（例如训练 RLHF 的奖励模型和检测错位的生成）的有效性。对九个 InD-OOD 数据集对和各种模型大小进行的大量实验表明，我们的方法显着降低了假阳性率（在某些情况下达到完美的零），同时在分布内任务上保持了高准确率，远远优于基线方法。

Title: ECM: A Unified Electronic Circuit Model for Explaining the Emergence of In-Context Learning and Chain-of-Thought in Large Language Model

• Authors: Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, Ting Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03325
• Pdf URL: https://arxiv.org/pdf/2502.03325
• Copy Paste: [[2502.03325]] ECM: A Unified Electronic Circuit Model for Explaining the Emergence of In-Context Learning and Chain-of-Thought in Large Language Model(https://arxiv.org/abs/2502.03325)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Recent advancements in large language models (LLMs) have led to significant successes across various applications, where the most noticeable is to a series of emerging capabilities, particularly in the areas of In-Context Learning (ICL) and Chain-of-Thought (CoT). To better understand and control model performance, many studies have begun investigating the underlying causes of these phenomena and their impact on task outcomes. However, existing explanatory frameworks predominantly focus on isolating and explaining ICL and CoT independently, leading to an incomplete understanding of their combined influence on model performance. To address this gap, we propose the Electronic Circuit Model (ECM), which provides a foundation for developing scalable, learnable policies and improving the management of AI-generated content. Specifically, ECM conceptualizes model behavior as an electronic circuit: ICL is represented as semantic magnetic field to providing an additional voltage following Faraday's Law, while CoT is modeled as series resistors to constrain the model output performance following Ohm's Law. Experimental results demonstrate that the ECM effectively predicts and explains LLM performance across a variety of prompting strategies. Furthermore, we apply ECM to advanced reasoning strategy optimization on a series of tasks, such as the International Olympiad in Informatics (IOI) and the International Mathematical Olympiad (IMO), achieving competitive performance that surpasses nearly 80% of top human competitors.
• 摘要：大型语言模型 (LLM) 的最新进展已在各种应用中取得了重大成功，其中最引人注目的是一系列新兴功能，特别是在上下文学习 (ICL) 和思维链 (CoT) 领域。为了更好地理解和控制模型性能，许多研究已经开始调查这些现象的根本原因及其对任务结果的影响。然而，现有的解释框架主要侧重于独立地隔离和解释 ICL 和 CoT，导致对它们对模型性能的综合影响的理解不完整。为了解决这一差距，我们提出了电子电路模型 (ECM)，它为开发可扩展、可学习的策略和改进 AI 生成内容的管理提供了基础。具体来说，ECM 将模型行为概念化为电子电路：ICL 表示为语义磁场，按照法拉第定律提供额外的电压，而 CoT 被建模为串联电阻，以按照欧姆定律限制模型输出性能。实验结果表明，ECM 可以有效地预测和解释各种提示策略中的 LLM 性能。此外，我们将ECM应用于国际信息学奥林匹克（IOI）、国际数学奥林匹克（IMO）等一系列任务的高级推理策略优化，取得了超越近80%人类顶尖选手的竞技成绩。

Title: Minerva: A Programmable Memory Test Benchmark for Language Models

• Authors: Menglin Xia, Victor Ruehle, Saravan Rajmohan, Reza Shokri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03358
• Pdf URL: https://arxiv.org/pdf/2502.03358
• Copy Paste: [[2502.03358]] Minerva: A Programmable Memory Test Benchmark for Language Models(https://arxiv.org/abs/2502.03358)
• Keywords: language model, llm
• Abstract: How effectively can LLM-based AI assistants utilize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from several limitations: they are static, susceptible to overfitting, difficult to interpret, and lack actionable insights--failing to pinpoint the specific capabilities a model lacks when it does not pass a test. In this paper, we present a framework for automatically generating a comprehensive set of tests to evaluate models' abilities to use their memory effectively. Our framework extends the range of capability tests beyond the commonly explored (passkey, key-value, needle in the haystack) search, a dominant focus in the literature. Specifically, we evaluate models on atomic tasks such as searching, recalling, editing, matching, comparing information in context memory, and performing basic operations when inputs are structured into distinct blocks, simulating real-world data. Additionally, we design composite tests to investigate the models' ability to maintain state while operating on memory. Our benchmark enables an interpretable, detailed assessment of memory capabilities of LLMs.
• 摘要：基于 LLM 的 AI 助手如何有效地利用其记忆（上下文）来执行各种任务？传统的数据基准测试通常是手工制作的，存在一些局限性：它们是静态的、容易过度拟合、难以解释并且缺乏可操作的洞察力——无法确定模型在未通过测​​试时所缺乏的特定能力。在本文中，我们提出了一个框架，用于自动生成一组全面的测试来评估模型有效使用其内存的能力。我们的框架将能力测试的范围扩展到了文献中主要关注的常见搜索（密码、键值、大海捞针）之外。具体来说，我们评估模型在原子任务上的表现，例如搜索、调用、编辑、匹配、比较上下文内存中的信息，以及在输入被构造成不同的块时执行基本操作，模拟真实世界的数据。此外，我们设计了复合测试来调查模型在操作内存时保持状态的能力。我们的基准测试可以对 LLM 的内存能力进行可解释的详细评估。

Title: Demystifying Long Chain-of-Thought Reasoning in LLMs

• Authors: Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, Xiang Yue
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03373
• Pdf URL: https://arxiv.org/pdf/2502.03373
• Copy Paste: [[2502.03373]] Demystifying Long Chain-of-Thought Reasoning in LLMs(https://arxiv.org/abs/2502.03373)
• Keywords: language model, llm, chain-of-thought
• Abstract: Scaling inference compute enhances reasoning in large language models (LLMs), with long chains-of-thought (CoTs) enabling strategies like backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which long CoTs emerge remain unclear, and RL training requires careful design choices. In this study, we systematically investigate the mechanics of long CoT reasoning, identifying the key factors that enable models to generate long CoT trajectories. Through extensive supervised fine-tuning (SFT) and RL experiments, we present four main findings: (1) While SFT is not strictly necessary, it simplifies training and improves efficiency; (2) Reasoning capabilities tend to emerge with increased training compute, but their development is not guaranteed, making reward shaping crucial for stabilizing CoT length growth; (3) Scaling verifiable reward signals is critical for RL. We find that leveraging noisy, web-extracted solutions with filtering mechanisms shows strong potential, particularly for out-of-distribution (OOD) tasks such as STEM reasoning; and (4) Core abilities like error correction are inherently present in base models, but incentivizing these skills effectively for complex tasks via RL demands significant compute, and measuring their emergence requires a nuanced approach. These insights provide practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs. Our code is available at: this https URL.
• 摘要：扩展推理计算可增强大型语言模型 (LLM) 中的推理能力，而长链思维 (CoT) 可实现回溯和纠错等策略。强化学习 (RL) 已成为开发这些能力的重要方法，但长链思维出现的条件仍不清楚，而且强化学习训练需要谨慎的设计选择。在本研究中，我们系统地研究了长链思维推理的机制，确定了使模型能够生成长链思维轨迹的关键因素。通过大量的监督微调 (SFT) 和强化学习实验，我们得出了四个主要发现：(1) 虽然 SFT 并非绝对必要，但它简化了训练并提高了效率；(2) 推理能力往往会随着训练计算的增加而出现，但它们的发展并不能得到保证，因此奖励塑造对于稳定 CoT 长度增长至关重要；(3) 扩展可验证的奖励信号对强化学习至关重要。我们发现，利用具有过滤机制的嘈杂、网络提取的解决方案具有强大的潜力，特别是对于 STEM 推理等分布外 (OOD) 任务；(4) 纠错等核心能力固有地存在于基础模型中，但通过 RL 有效地激励这些技能以完成复杂任务需要大量计算，而衡量它们的出现需要采取细致入微的方法。这些见解为优化训练策略以增强 LLM 中的长 CoT 推理提供了实用指导。我们的代码可在以下网址获取：此 https URL。

Title: Integrating automatic speech recognition into remote healthcare interpreting: A pilot study of its impact on interpreting quality

• Authors: Shiyi Tan, Constantin Orăsan, Sabine Braun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03381
• Pdf URL: https://arxiv.org/pdf/2502.03381
• Copy Paste: [[2502.03381]] Integrating automatic speech recognition into remote healthcare interpreting: A pilot study of its impact on interpreting quality(https://arxiv.org/abs/2502.03381)
• Keywords: gpt, chat
• Abstract: This paper reports on the results from a pilot study investigating the impact of automatic speech recognition (ASR) technology on interpreting quality in remote healthcare interpreting settings. Employing a within-subjects experiment design with four randomised conditions, this study utilises scripted medical consultations to simulate dialogue interpreting tasks. It involves four trainee interpreters with a language combination of Chinese and English. It also gathers participants' experience and perceptions of ASR support through cued retrospective reports and semi-structured interviews. Preliminary data suggest that the availability of ASR, specifically the access to full ASR transcripts and to ChatGPT-generated summaries based on ASR, effectively improved interpreting quality. Varying types of ASR output had different impacts on the distribution of interpreting error types. Participants reported similar interactive experiences with the technology, expressing their preference for full ASR transcripts. This pilot study shows encouraging results of applying ASR to dialogue-based healthcare interpreting and offers insights into the optimal ways to present ASR output to enhance interpreter experience and performance. However, it should be emphasised that the main purpose of this study was to validate the methodology and that further research with a larger sample size is necessary to confirm these findings.
• 摘要：本文报告了一项初步研究的结果，该研究调查了自动语音识别 (ASR) 技术对远程医疗口译环境中的口译质量的影响。本研究采用具有四种随机条件的受试者内实验设计，利用脚本化的医疗咨询来模拟对话口译任务。它涉及四名语言组合为中文和英文的实习口译员。它还通过提示性回顾报告和半结构化访谈收集参与者对 ASR 支持的经验和看法。初步数据表明，ASR 的可用性，特别是对完整 ASR 记录的访问以及基于 ASR 的 ChatGPT 生成的摘要的访问，有效地提高了口译质量。不同类型的 ASR 输出对口译错误类型的分布有不同的影响。参与者报告了类似的技术互动体验，表达了他们对完整 ASR 记录的偏好。这项初步研究显示了将 ASR 应用于基于对话的医疗口译的令人鼓舞的结果，并提供了有关呈现 ASR 输出的最佳方式以增强口译员体验和表现的见解。但需要强调的是，本研究的主要目的是验证方法论，需要进一步进行更大样本量的研究来证实这些发现。

Title: High-Fidelity Simultaneous Speech-To-Speech Translation

• Authors: Tom Labiausse, Laurent Mazaré, Edouard Grave, Patrick Pérez, Alexandre Défossez, Neil Zeghidour
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2502.03382
• Pdf URL: https://arxiv.org/pdf/2502.03382
• Copy Paste: [[2502.03382]] High-Fidelity Simultaneous Speech-To-Speech Translation(https://arxiv.org/abs/2502.03382)
• Keywords: language model
• Abstract: We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation. We furthermore address the fundamental challenge of simultaneous interpretation, which unlike its consecutive counterpart, where one waits for the end of the source utterance to start translating, adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. To do so, we introduce a weakly-supervised method that leverages the perplexity of an off-the-shelf text translation system to identify optimal delays on a per-word basis and create aligned synthetic data. After supervised training, Hibiki performs adaptive, simultaneous speech translation with vanilla temperature sampling. On a French-English simultaneous speech translation task, Hibiki demonstrates state-of-the-art performance in translation quality, speaker fidelity and naturalness. Moreover, the simplicity of its inference process makes it compatible with batched translation and even real-time on-device deployment. We provide examples as well as models and inference code.
• 摘要：我们推出了 Hibiki，一种仅使用解码器的同声传译模型。Hibiki 利用多流语言模型同步处理源语音和目标语音，并联合生成文本和音频标记以执行语音到文本和语音到语音的翻译。此外，我们还解决了同声传译的基本挑战，与等待源话语结束才开始翻译的连续翻译不同，同声传译会调整其流程以积累足够的上下文，从而逐块实时生成正确的翻译。为此，我们引入了一种弱监督方法，该方法利用现成的文本翻译系统的困惑度来确定每个单词的最佳延迟并创建对齐的合成数据。经过监督训练后，Hibiki 会使用原始温度采样执行自适应同声传译。在法语-英语同声传译任务中，Hibiki 在翻译质量、说话者保真度和自然度方面表现出色。此外，其推理过程简单，可兼容批量翻译，甚至实时设备部署。我们提供示例以及模型和推理代码。

Title: LIMO: Less is More for Reasoning

• Authors: Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, Pengfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03387
• Pdf URL: https://arxiv.org/pdf/2502.03387
• Copy Paste: [[2502.03387]] LIMO: Less is More for Reasoning(https://arxiv.org/abs/2502.03387)
• Keywords: language model
• Abstract: We present a fundamental discovery that challenges our understanding of how complex reasoning emerges in large language models. While conventional wisdom suggests that sophisticated reasoning tasks demand extensive training data (>100,000 examples), we demonstrate that complex mathematical reasoning abilities can be effectively elicited with surprisingly few examples. Through comprehensive experiments, our proposed model LIMO demonstrates unprecedented performance in mathematical reasoning. With merely 817 curated training samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of the training data required by previous approaches. LIMO demonstrates exceptional out-of-distribution generalization, achieving 40.5% absolute improvement across 10 diverse benchmarks, outperforming models trained on 100x more data, challenging the notion that SFT leads to memorization rather than generalization. Based on these results, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes. This hypothesis posits that the elicitation threshold for complex reasoning is determined by two key factors: (1) the completeness of the model's encoded knowledge foundation during pre-training, and (2) the effectiveness of post-training examples as "cognitive templates" that show the model how to utilize its knowledge base to solve complex reasoning tasks. To facilitate reproducibility and future research in data-efficient reasoning, we release LIMO as a comprehensive open-source suite at this https URL.
• 摘要：我们提出了一项根本性的发现，挑战了我们对大型语言模型中复杂推理如何出现的理解。虽然传统观点认为复杂的推理任务需要大量的训练数据（>100,000 个示例），但我们证明，只需极少的示例就可以有效地激发复杂的数学推理能力。通过全面的实验，我们提出的模型 LIMO 在数学推理方面表现出了前所未有的性能。仅使用 817 个精选的训练样本，LIMO 在 AIME 上的准确率就达到了 57.1%，在 MATH 上的准确率达到了 94.8%，分别高于之前基于 SFT 的模型的 6.5% 和 59.2%，同时仅使用了之前方法所需的 1% 的训练数据。LIMO 表现出卓越的分布外泛化能力，在 10 个不同的基准测试中实现了 40.5% 的绝对改进，优于在 100 倍以上数据上训练的模型，挑战了 SFT 导致记忆而不是泛化的观念。基于这些结果，我们提出了少即是多推理假设（LIMO 假设）：在基础模型中，领域知识在预训练期间已被全面编码，复杂的推理能力可以通过最少但精确协调的认知过程演示出现。该假设认为，复杂推理的引出阈值由两个关键因素决定：（1）模型在预训练期间编码知识基础的完整性，以及（2）训练后示例作为“认知模板”的有效性，这些模板向模型展示了如何利用其知识库来解决复杂的推理任务。为了促进数据高效推理的可重复性和未来研究，我们在此 https URL 上发布了 LIMO 作为一套全面的开源套件。

Title: SPRI: Aligning Large Language Models with Context-Situated Principles

• Authors: Hongli Zhan, Muneeza Azmat, Raya Horesh, Junyi Jessy Li, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03397
• Pdf URL: https://arxiv.org/pdf/2502.03397
• Copy Paste: [[2502.03397]] SPRI: Aligning Large Language Models with Context-Situated Principles(https://arxiv.org/abs/2502.03397)
• Keywords: language model, llm
• Abstract: Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at this https URL.
• 摘要：调整大型语言模型以整合和反映人类价值观是艰巨的，尤其是对于需要复杂人工监督的任务，因为依靠人类专业知识进行特定于上下文的指导需要大量资源和时间。先前的工作已经利用预定义的规则或原则集来指导模型的行为（Bai 等人，2022 年；Sun 等人，2023 年）。然而，这些原则往往是通用的，因此很难使它们适应每个单独的输入查询或上下文。在这项工作中，我们提出了情景原则 (SPRI)，这是一个几乎不需要人工干预的框架，旨在为每个输入查询自动实时生成指导原则并利用它们来调整每个响应。我们在三个任务上评估了 SPRI，并表明 1) SPRI 可以在复杂的特定领域任务中推导出原则，从而产生与专家制作的原则相当的性能；2) SPRI 生成的原则导致特定于实例的评分标准优于之前的 LLM-as-a-judge 框架； 3) 使用 SPRI 生成合成 SFT 数据可显著提高真实性。我们在此 https URL 上发布了我们的代码和模型生成。

Title: Think or Step-by-Step? UnZIPping the Black Box in Zero-Shot Prompts

• Authors: Nikta Gohari Sadr, Sangmitra Madhusudan, Ali Emami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03418
• Pdf URL: https://arxiv.org/pdf/2502.03418
• Copy Paste: [[2502.03418]] Think or Step-by-Step? UnZIPping the Black Box in Zero-Shot Prompts(https://arxiv.org/abs/2502.03418)
• Keywords: language model, llm, prompt
• Abstract: Zero-shot prompting techniques have significantly improved the performance of Large Language Models (LLMs). However, we lack a clear understanding of why zero-shot prompts are so effective. For example, in the prompt "Let's think step-by-step," is "think" or "step-by-step" more crucial to its success? Existing interpretability methods, such as gradient-based and attention-based approaches, are computationally intensive and restricted to open-source models. We introduce the ZIP score (Zero-shot Importance of Perturbation score), a versatile metric applicable to both open and closed-source models, based on systematic input word perturbations. Our experiments across four recent LLMs, seven widely-used prompts, and several tasks, reveal interesting patterns in word importance. For instance, while both 'step-by-step' and 'think' show high ZIP scores, which one is more influential depends on the model and task. We validate our method using controlled experiments and compare our results with human judgments, finding that proprietary models align more closely with human intuition regarding word significance. These findings enhance our understanding of LLM behavior and contribute to developing more effective zero-shot prompts and improved model analysis.
• 摘要：零样本提示技术显著提高了大型语言模型 (LLM) 的性能。然而，我们不清楚零样本提示为何如此有效。例如，在提示“让我们一步一步思考”中，“思考”还是“一步一步”对其成功更重要？现有的可解释性方法，例如基于梯度和基于注意力的方法，都是计算密集型的，并且仅限于开源模型。我们引入了 ZIP 分数（零样本扰动重要性分数），这是一种基于系统输入词扰动的通用指标，适用于开源和闭源模型。我们在最近的四个 LLM、七个广泛使用的提示和几个任务中进行了实验，揭示了单词重要性的有趣模式。例如，虽然“一步一步”和“思考”都显示出较高的 ZIP 分数，但哪一个更有影响力取决于模型和任务。我们通过受控实验验证了我们的方法，并将我们的结果与人类判断进行了比较，发现专有模型与人类对单词重要性的直觉更接近。这些发现增强了我们对 LLM 行为的理解，并有助于开发更有效的零样本提示和改进的模型分析。

Title: On Fairness of Unified Multimodal Large Language Model for Image Generation

• Authors: Ming Liu, Hao Chen, Jindong Wang, Liwen Wang, Bhiksha Raj Ramakrishnan, Wensheng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03429
• Pdf URL: https://arxiv.org/pdf/2502.03429
• Copy Paste: [[2502.03429]] On Fairness of Unified Multimodal Large Language Model for Image Generation(https://arxiv.org/abs/2502.03429)
• Keywords: language model, llm
• Abstract: Unified multimodal large language models (U-MLLMs) have demonstrated impressive performance in visual understanding and generation in an end-to-end pipeline. Compared with generation-only models (e.g., Stable Diffusion), U-MLLMs may raise new questions about bias in their outputs, which can be affected by their unified capabilities. This gap is particularly concerning given the under-explored risk of propagating harmful stereotypes. In this paper, we benchmark the latest U-MLLMs and find that most exhibit significant demographic biases, such as gender and race bias. To better understand and mitigate this issue, we propose a locate-then-fix strategy, where we audit and show how the individual model component is affected by bias. Our analysis shows that bias originates primarily from the language model. More interestingly, we observe a "partial alignment" phenomenon in U-MLLMs, where understanding bias appears minimal, but generation bias remains substantial. Thus, we propose a novel balanced preference model to balance the demographic distribution with synthetic data. Experiments demonstrate that our approach reduces demographic bias while preserving semantic fidelity. We hope our findings underscore the need for more holistic interpretation and debiasing strategies of U-MLLMs in the future.
• 摘要：统一多模态大型语言模型 (U-MLLM) 在端到端管道中已在视觉理解和生成方面表现出色。与仅生成模型（例如稳定扩散）相比，U-MLLM 可能会引发有关其输出偏差的新问题，这些偏差可能会受到其统一功能的影响。鉴于传播有害刻板印象的风险尚未得到充分探索，这一差距尤其令人担忧。在本文中，我们对最新的 U-MLLM 进行了基准测试，发现大多数 U-MLLM 都表现出明显的人口偏见，例如性别和种族偏见。为了更好地理解和缓解这个问题，我们提出了一种先定位后修复的策略，在该策略中，我们审核并展示单个模型组件如何受到偏见的影响。我们的分析表明，偏见主要源于语言模型。更有趣的是，我们在 U-MLLM 中观察到“部分对齐”现象，其中理解偏见似乎很小，但生成偏见仍然很大。因此，我们提出了一种新颖的平衡偏好模型，以平衡人口分布与合成数据。实验表明，我们的方法可以减少人口偏见，同时保持语义保真度。我们希望我们的研究结果能够强调未来对 U-MLLM 进行更全面的解释和消除偏见策略的必要性。