2025-10-31

Title: StreetMath: Study of LLMs' Approximation Behaviors

• Authors: Chiung-Yi Tseng, Somshubhra Roy, Maisha Thasin, Danyang Zhang, Blessing Effiong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.25776
• Pdf URL: https://arxiv.org/pdf/2510.25776
• Copy Paste: [[2510.25776]] StreetMath: Study of LLMs' Approximation Behaviors(https://arxiv.org/abs/2510.25776)
• Keywords: language model, gpt, llm
• Abstract: There is a substantial body of literature examining the mathematical reasoning capabilities of large language models (LLMs), particularly their performance on precise arithmetic operations in autoregressive architectures. However, their ability to perform approximate reasoning in informal, fast-paced mathematical operations has received far less attention, especially among non-autoregressive decoder models. Our work addresses this gap by introducing StreetMath, a benchmark designed to evaluate models' approximation abilities under real-world approximation scenarios. We conduct extensive evaluations across different LLM architectures: Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, Dream-v0-Instruct-7B, Falcon-Mamba-7B-Instruct, and Mamba-GPT-3B. Furthermore, we apply mechanistic interpretability techniques to probe their internal computational states. Our analysis reveals that LLMs generally attempt to compute exact values or invoke external tools even in tasks that call for approximation. Moreover, while models sometimes reach the correct answer in early layers or steps, they still consume more tokens when solving approximation tasks. Additional experiments indicate that exact and approximate arithmetic operations rely on largely separate neural components. Drawing upon research on cognitive psychology, we argue that LLMs do not exhibit cognitive miserliness in the same way humans do in street math settings. We open source our work this https URL
• 摘要：有大量文献研究了大型语言模型 (LLM) 的数学推理能力，特别是它们在自回归架构中的精确算术运算上的性能。然而，它们在非正式、快节奏的数学运算中执行近似推理的能力受到的关注要少得多，特别是在非自回归解码器模型中。我们的工作通过引入 StreetMath 来解决这一差距，这是一个旨在评估模型在现实近似场景下的近似能力的基准。我们对不同的 LLM 架构进行了广泛的评估：Qwen3-4B-Instruct-2507、Qwen3-4B-Thinking-2507、Dream-v0-Instruct-7B、Falcon-Mamba-7B-Instruct 和 Mamba-GPT-3B。此外，我们应用机械可解释性技术来探测它们的内部计算状态。我们的分析表明，法学硕士通常会尝试计算精确值或调用外部工具，即使在需要近似值的任务中也是如此。此外，虽然模型有时会在早期层或步骤中达到正确答案，但它们在解决近似任务时仍然会消耗更多标记。其他实验表明，精确和近似算术运算很大程度上依赖于独立的神经组件。根据认知心理学的研究，我们认为法学硕士不会像人类在街头数学环境中那样表现出认知吝啬。我们通过此 https URL 开源我们的工作

Title: zFLoRA: Zero-Latency Fused Low-Rank Adapters

• Authors: Dhananjaya Gowda, Seoha Song, Harshith Goka, Junhyun Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.25784
• Pdf URL: https://arxiv.org/pdf/2510.25784
• Copy Paste: [[2510.25784]] zFLoRA: Zero-Latency Fused Low-Rank Adapters(https://arxiv.org/abs/2510.25784)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly deployed with task-specific adapters catering to multiple downstream applications. In such a scenario, the additional compute associated with these apparently insignificant number of adapter parameters (typically less than 1% of the base model) turns out to be disproportionately significant during inference time (upto 2.5x times that of the base model). In this paper, we propose a new zero-latency fused low-rank adapter (zFLoRA) that introduces zero or negligible latency overhead on top of the base model. Experimental results on LLMs of size 1B, 3B and 7B show that zFLoRA compares favorably against the popular supervised fine-tuning benchmarks including low-rank adapters (LoRA) as well as full fine-tuning (FFT). Experiments are conducted on 18 different tasks across three different categories namely commonsense reasoning, math reasoning and summary-dialogue. Latency measurements made on NPU (Samsung Galaxy S25+) as well as GPU (NVIDIA H100) platforms show that the proposed zFLoRA adapters introduce zero to negligible latency overhead.
• 摘要：大型语言模型 (LLM) 越来越多地部署特定于任务的适配器，以满足多个下游应用程序的需求。在这种情况下，与这些明显微不足道的适配器参数数量（通常小于基本模型的 1%）相关的额外计算在推理时间内变得异常重要（高达基本模型的 2.5 倍）。在本文中，我们提出了一种新的零延迟融合低秩适配器（zFLoRA），它在基本模型之上引入了零或可以忽略不计的延迟开销。大小为 1B、3B 和 7B 的 LLM 的实验结果表明，zFLoRA 与流行的监督微调基准（包括低秩适配器 (LoRA) 以及完全微调 (FFT)）相比具有优势。实验针对三个不同类别（即常识推理、数学推理和摘要对话）的 18 种不同任务进行。在 NPU（三星 Galaxy S25+）和 GPU（NVIDIA H100）平台上进行的延迟测量表明，所提出的 zFLoRA 适配器带来的延迟开销为零甚至可以忽略不计。

Title: LISTEN to Your Preferences: An LLM Framework for Multi-Objective Selection

• Authors: Adam S. Jovine, Tinghan Ye, Francis Bahk, Jingjing Wang, David B. Shmoys, Peter I. Frazier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25799
• Pdf URL: https://arxiv.org/pdf/2510.25799
• Copy Paste: [[2510.25799]] LISTEN to Your Preferences: An LLM Framework for Multi-Objective Selection(https://arxiv.org/abs/2510.25799)
• Keywords: language model, llm
• Abstract: Human experts often struggle to select the best option from a large set of items with multiple competing objectives, a process bottlenecked by the difficulty of formalizing complex, implicit preferences. To address this, we introduce LISTEN, a framework that leverages a Large Language Model (LLM) as a zero-shot preference oracle, guided only by an expert's high-level priorities in natural language. To operate within LLM constraints like context windows and inference costs, we propose two iterative algorithms: LISTEN-U, which uses the LLM to refine a parametric utility function, and LISTEN-T, a non-parametric method that performs tournament-style selections over small batches of solutions. Evaluated on diverse tasks including flight booking, shopping, and exam scheduling, our results show LISTEN-U excels when preferences are parametrically aligned (a property we measure with a novel concordance metric), while LISTEN-T offers more robust performance. This work explores a promising direction for steering complex multi-objective decisions directly with natural language, reducing the cognitive burden of traditional preference elicitation.
• 摘要：人类专家常常难以从大量具有多个相互竞争目标的项目中选择最佳选项，这一过程因难以形式化复杂、隐含的偏好而受到瓶颈。为了解决这个问题，我们引入了 LISTEN，这是一个利用大型语言模型 (LLM) 作为零样本偏好预言机的框架，仅由专家在自然语言中的高级优先级引导。为了在 LLM 约束（如上下文窗口和推理成本）内运行，我们提出了两种迭代算法：LISTEN-U，它使用 LLM 来细化参数效用函数；LISTEN-T，一种非参数方法，对小批量解决方案执行锦标赛式选择。对包括航班预订、购物和考试安排在内的各种任务进行评估，我们的结果表明，当偏好参数一致时（我们用新颖的一致性指标来衡量的属性），LISTEN-U 表现出色，而 LISTEN-T 则提供更强大的性能。这项工作探索了一个有希望的方向，即直接用自然语言引导复杂的多目标决策，减少传统偏好诱导的认知负担。

Title: Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data

• Authors: Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yi-An Ma, Yeyun Gong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25804
• Pdf URL: https://arxiv.org/pdf/2510.25804
• Copy Paste: [[2510.25804]] Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data(https://arxiv.org/abs/2510.25804)
• Keywords: language model, llm
• Abstract: Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across extended spans of text. However, a significant portion of readily available long-text data lacks meaningful long-distance dependencies; most spans can be predicted using only local context. Training on such data is inefficient, making careful data selection crucial. Therefore, we introduce LongFilter, a framework for curating training data tailored to long-context pretraining. LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings, thereby identifying samples where long-range dependencies are essential. Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.
• 摘要：长上下文语言模型通过利用跨扩展文本范围的依赖关系来解锁推理、代码生成和文档摘要方面的高级功能。然而，很大一部分容易获得的长文本数据缺乏有意义的长距离依赖性；大多数跨度可以仅使用本地上下文来预测。对此类数据的训练效率低下，因此仔细的数据选择至关重要。因此，我们引入了 LongFilter，这是一个针对长上下文预训练定制训练数据的框架。 LongFilter 通过对比长上下文和短上下文设置下的模型预测来测量扩展上下文提供的信息增益，从而识别长期依赖性至关重要的样本。 LLaMA-3-8B 的实验将其上下文长度从 8K 扩展到 64K，结果表明 LongFilter 可以有效地选择高质量数据，并对 HELMET、LongBench 和 RULER 等基准测试产生显着改进。

Title: Ideology-Based LLMs for Content Moderation

• Authors: Stefano Civelli, Pietro Bernardelle, Nardiena A. Pratama, Gianluca Demartini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25805
• Pdf URL: https://arxiv.org/pdf/2510.25805
• Copy Paste: [[2510.25805]] Ideology-Based LLMs for Content Moderation(https://arxiv.org/abs/2510.25805)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly used in content moderation systems, where ensuring fairness and neutrality is essential. In this study, we examine how persona adoption influences the consistency and fairness of harmful content classification across different LLM architectures, model sizes, and content modalities (language vs. vision). At first glance, headline performance metrics suggest that personas have little impact on overall classification accuracy. However, a closer analysis reveals important behavioral shifts. Personas with different ideological leanings display distinct propensities to label content as harmful, showing that the lens through which a model "views" input can subtly shape its judgments. Further agreement analyses highlight that models, particularly larger ones, tend to align more closely with personas from the same political ideology, strengthening within-ideology consistency while widening divergence across ideological groups. To show this effect more directly, we conducted an additional study on a politically targeted task, which confirmed that personas not only behave more coherently within their own ideology but also exhibit a tendency to defend their perspective while downplaying harmfulness in opposing views. Together, these findings highlight how persona conditioning can introduce subtle ideological biases into LLM outputs, raising concerns about the use of AI systems that may reinforce partisan perspectives under the guise of neutrality.
• 摘要：大型语言模型 (LLM) 越来越多地用于内容审核系统，其中确保公平性和中立性至关重要。在这项研究中，我们研究了人物角色的采用如何影响不同法学硕士架构、模型大小和内容模式（语言与视觉）中有害内容分类的一致性和公平性。乍一看，总体性能指标表明角色对整体分类准确性影响不大。然而，更仔细的分析揭示了重要的行为转变。具有不同意识形态倾向的角色显示出将内容标记为有害的不同倾向，这表明模型“查看”输入的镜头可以巧妙地塑造其判断。进一步的一致性分析强调，模型，尤其是较大的模型，往往与来自同一政治意识形态的人物角色更加紧密地结合在一起，加强了意识形态内部的一致性，同时扩大了意识形态群体之间的分歧。为了更直接地展示这种效果，我们对一项政治目标任务进行了一项额外的研究，该研究证实人物角色不仅在自己的意识形态中表现得更加连贯，而且还表现出捍卫自己观点的倾向，同时淡化反对观点的危害性。总之，这些发现凸显了角色调节如何在法学硕士输出中引入微妙的意识形态偏见，引发人们对人工智能系统的使用的担忧，这些系统可能会在中立的幌子下强化党派观点。

Title: Beyond Long Context: When Semantics Matter More than Tokens

• Authors: Tarun Kumar Chawdhury, Jon D. Duke
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.25816
• Pdf URL: https://arxiv.org/pdf/2510.25816
• Copy Paste: [[2510.25816]] Beyond Long Context: When Semantics Matter More than Tokens(https://arxiv.org/abs/2510.25816)
• Keywords: long context, retrieval augmented generation
• Abstract: Electronic Health Records (EHR) store clinical documentation as base64 encoded attachments in FHIR DocumentReference resources, which makes semantic question answering difficult. Traditional vector database methods often miss nuanced clinical relationships. The Clinical Entity Augmented Retrieval (CLEAR) method, introduced by Lopez et al. 2025, uses entity aware retrieval and achieved improved performance with an F1 score of 0.90 versus 0.86 for embedding based retrieval, while using over 70 percent fewer tokens. We developed a Clinical Notes QA Evaluation Platform to validate CLEAR against zero shot large context inference and traditional chunk based retrieval augmented generation. The platform was tested on 12 clinical notes ranging from 10,000 to 65,000 tokens representing realistic EHR content. CLEAR achieved a 58.3 percent win rate, an average semantic similarity of 0.878, and used 78 percent fewer tokens than wide context processing. The largest performance gains occurred on long notes, with a 75 percent win rate for documents exceeding 65,000 tokens. These findings confirm that entity aware retrieval improves both efficiency and accuracy in clinical natural language processing. The evaluation framework provides a reusable and transparent benchmark for assessing clinical question answering systems where semantic precision and computational efficiency are critical.
• 摘要：电子健康记录 (EHR) 将临床文档存储为 FHIR DocumentReference 资源中的 Base64 编码附件，这使得语义问答变得困难。传统的矢量数据库方法常常会忽略微妙的临床关系。 Lopez 等人提出的临床实体增强检索 (CLEAR) 方法。 2025 年，使用实体感知检索并实现了性能改进，F1 分数为 0.90，而基于嵌入的检索为 0.86，同时使用的令牌减少了 70% 以上。我们开发了一个 Clinical Notes QA 评估平台，以针对零样本大上下文推理和传统的基于块的检索增强生成来验证 CLEAR。该平台在 12 个临床记录上进行了测试，范围从 10,000 到 65,000 个代表真实 EHR 内容的代币。 CLEAR 的胜率达到了 58.3%，平均语义相似度为 0.878，并且比宽上下文处理使用的标记少了 78%。最大的性能提升出现在长笔记上，超过 65,000 个标记的文档的获胜率为 75%。这些发现证实实体感知检索提高了临床自然语言处理的效率和准确性。该评估框架为评估语义精度和计算效率至关重要的临床问答系统提供了可重用且透明的基准。

Title: A Survey on Efficient Large Language Model Training: From Data-centric Perspectives

• Authors: Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25817
• Pdf URL: https://arxiv.org/pdf/2510.25817
• Copy Paste: [[2510.25817]] A Survey on Efficient Large Language Model Training: From Data-centric Perspectives(https://arxiv.org/abs/2510.25817)
• Keywords: language model, llm
• Abstract: Post-training of Large Language Models (LLMs) is crucial for unlocking their task generalization potential and domain-specific capabilities. However, the current LLM post-training paradigm faces significant data challenges, including the high costs of manual annotation and diminishing marginal returns on data scales. Therefore, achieving data-efficient post-training has become a key research question. In this paper, we present the first systematic survey of data-efficient LLM post-training from a data-centric perspective. We propose a taxonomy of data-efficient LLM post-training methods, covering data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. We summarize representative approaches in each category and outline future research directions. By examining the challenges in data-efficient LLM post-training, we highlight open problems and propose potential research avenues. We hope our work inspires further exploration into maximizing the potential of data utilization in large-scale model training. Paper List: this https URL
• 摘要：大型语言模型 (LLM) 的后训练对于释放其任务泛化潜力和特定领域的能力至关重要。然而，当前的法学硕士后培训范式面临着重大的数据挑战，包括手动注释的高成本和数据规模的边际收益递减。因此，实现数据高效的后训练已成为一个关键的研究问题。在本文中，我们从以数据为中心的角度对数据高效的法学硕士培训后进行了首次系统调查。我们提出了数据高效的法学硕士后培训方法的分类，涵盖数据选择、数据质量增强、合成数据生成、数据蒸馏和压缩以及自我进化的数据生态系统。我们总结了每个类别的代表性方法并概述了未来的研究方向。通过研究数据高效的法学硕士培训后的挑战，我们强调了未解决的问题并提出了潜在的研究途径。我们希望我们的工作能够激发进一步探索，最大限度地发挥大规模模型训练中数据利用的潜力。论文列表：此 https URL

Title: Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation

• Authors: Frederico Belcavello, Ely Matos, Arthur Lorenzi, Lisandra Bonoto, Lívia Ruiz, Luiz Fernando Pereira, Victor Herbst, Yulla Navarro, Helen de Andrade Abreu, Lívia Dutra, Tiago Timponi Torrent
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.25904
• Pdf URL: https://arxiv.org/pdf/2510.25904
• Copy Paste: [[2510.25904]] Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation(https://arxiv.org/abs/2510.25904)
• Keywords: llm
• Abstract: The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time.
• 摘要：使用基于法学硕士的应用程序作为加速和/或替代人类劳动创建语言资源和数据集的手段已成为现实。尽管如此，尽管此类工具在语言研究方面具有潜力，但仍然缺乏对其性能和对注释数据集创建的影响的全面评估，特别是在 NLP 的透视方法下。本文通过报告使用基于 LLM 的语义角色标签器对类 FrameNet 语义注释的（半）自动化进行了广泛的评估，为缩小这一差距做出了贡献。所采用的方法比较了手动、自动和半自动注释三种实验设置中的注释时间、覆盖范围和多样性。结果表明，与仅人类设置相比，混合半自动注释设置导致帧多样性增加和相似的注释覆盖率，而自动设置在除注释时间之外的所有指标中表现均较差。

Title: RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline

• Authors: André V. Duarte, Xuying li, Bin Zeng, Arlindo L. Oliveira, Lei Li, Zhuo Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25941
• Pdf URL: https://arxiv.org/pdf/2510.25941
• Copy Paste: [[2510.25941]] RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline(https://arxiv.org/abs/2510.25941)
• Keywords: language model, gpt, llm, agent
• Abstract: If we cannot inspect the training data of a large language model (LLM), how can we ever know what it has seen? We believe the most compelling evidence arises when the model itself freely reproduces the target content. As such, we propose RECAP, an agentic pipeline designed to elicit and verify memorized training data from LLM outputs. At the heart of RECAP is a feedback-driven loop, where an initial extraction attempt is evaluated by a secondary language model, which compares the output against a reference passage and identifies discrepancies. These are then translated into minimal correction hints, which are fed back into the target model to guide subsequent generations. In addition, to address alignment-induced refusals, RECAP includes a jailbreaking module that detects and overcomes such barriers. We evaluate RECAP on EchoTrace, a new benchmark spanning over 30 full books, and the results show that RECAP leads to substantial gains over single-iteration approaches. For instance, with GPT-4.1, the average ROUGE-L score for the copyrighted text extraction improved from 0.38 to 0.47 - a nearly 24% increase.
• 摘要：如果我们无法检查大型语言模型（LLM）的训练数据，我们如何知道它看到了什么？我们相信，当模型本身自由地再现目标内容时，就会出现最令人信服的证据。因此，我们提出 RECAP，这是一种代理管道，旨在从 LLM 输出中引出并验证记忆的训练数据。 RECAP 的核心是一个反馈驱动的循环，其中初始提取尝试由第二语言模型进行评估，该模型将输出与参考段落进行比较并识别差异。然后将它们转化为最小的校正提示，并将其反馈到目标模型中以指导后续生成。此外，为了解决对齐引起的拒绝问题，RECAP 包含一个越狱模块，可以检测并克服此类障碍。我们在 EchoTrace 上评估 RECAP，这是一个涵盖 30 多本完整书籍的新基准，结果表明 RECAP 比单次迭代方法带来了巨大的收益。例如，使用 GPT-4.1，受版权保护的文本提取的平均 ROUGE-L 分数从 0.38 提高到 0.47，提高了近 24%。

Title: Revisiting Multilingual Data Mixtures in Language Model Pretraining

• Authors: Negar Foroutan, Paul Teiletche, Ayush Kumar Tarun, Antoine Bosselut
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.25947
• Pdf URL: https://arxiv.org/pdf/2510.25947
• Copy Paste: [[2510.25947]] Revisiting Multilingual Data Mixtures in Language Model Pretraining(https://arxiv.org/abs/2510.25947)
• Keywords: language model, llm
• Abstract: The impact of different multilingual data mixtures in pretraining large language models (LLMs) has been a topic of ongoing debate, often raising concerns about potential trade-offs between language coverage and model performance (i.e., the curse of multilinguality). In this work, we investigate these assumptions by training 1.1B and 3B parameter LLMs on diverse multilingual corpora, varying the number of languages from 25 to 400. Our study challenges common beliefs surrounding multilingual training. First, we find that combining English and multilingual data does not necessarily degrade the in-language performance of either group, provided that languages have a sufficient number of tokens included in the pretraining corpus. Second, we observe that using English as a pivot language (i.e., a high-resource language that serves as a catalyst for multilingual generalization) yields benefits across language families, and contrary to expectations, selecting a pivot language from within a specific family does not consistently improve performance for languages within that family. Lastly, we do not observe a significant "curse of multilinguality" as the number of training languages increases in models at this scale. Our findings suggest that multilingual data, when balanced appropriately, can enhance language model capabilities without compromising performance, even in low-resource settings
• 摘要：不同多语言数据混合对预训练大型语言模型 (LLM) 的影响一直是一个持续争论的话题，经常引起人们对语言覆盖范围和模型性能之间潜在权衡的担忧（即多语言诅咒）。在这项工作中，我们通过在不同的多语言语料库上训练 1.1B 和 3B 参数法学硕士来研究这些假设，语言数量从 25 种到 400 种不等。我们的研究挑战了围绕多语言训练的常见信念。首先，我们发现，只要语言在预训练语料库中包含足够数量的标记，结合英语和多语言数据并不一定会降低任一组的语言表现。其次，我们观察到，使用英语作为枢轴语言（即作为多语言泛化催化剂的高资源语言）可以在各个语言家族中带来好处，但与预期相反，从特定家族中选择枢轴语言并不能持续提高该家族中语言的性能。最后，随着模型中训练语言数量的增加，我们并没有观察到明显的“多语言诅咒”。我们的研究结果表明，即使在资源匮乏的环境中，多语言数据如果适当平衡，也可以在不影响性能的情况下增强语言模型的能力

Title: Semantic Label Drift in Cross-Cultural Translation

• Authors: Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Polydoros Giannouris, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25967
• Pdf URL: https://arxiv.org/pdf/2510.25967
• Copy Paste: [[2510.25967]] Semantic Label Drift in Cross-Cultural Translation(https://arxiv.org/abs/2510.25967)
• Keywords: language model, llm
• Abstract: Machine Translation (MT) is widely employed to address resource scarcity in low-resource languages by generating synthetic data from high-resource counterparts. While sentiment preservation in translation has long been studied, a critical but underexplored factor is the role of cultural alignment between source and target languages. In this paper, we hypothesize that semantic labels are drifted or altered during MT due to cultural divergence. Through a series of experiments across culturally sensitive and neutral domains, we establish three key findings: (1) MT systems, including modern Large Language Models (LLMs), induce label drift during translation, particularly in culturally sensitive domains; (2) unlike earlier statistical MT tools, LLMs encode cultural knowledge, and leveraging this knowledge can amplify label drift; and (3) cultural similarity or dissimilarity between source and target languages is a crucial determinant of label preservation. Our findings highlight that neglecting cultural factors in MT not only undermines label fidelity but also risks misinterpretation and cultural conflict in downstream applications.
• 摘要：机器翻译 (MT) 被广泛用于通过从高资源对应语言生成合成数据来解决低资源语言的资源稀缺问题。虽然翻译中的情感保存长期以来一直被研究，但一个关键但尚未充分探索的因素是源语言和目标语言之间文化一致性的作用。在本文中，我们假设语义标签在机器翻译过程中由于文化差异而发生漂移或改变。通过跨文化敏感和中立领域的一系列实验，我们得出了三个关键发现：（1）机器翻译系统，包括现代大型语言模型（LLM），会在翻译过程中引起标签漂移，特别是在文化敏感领域； （2）与早期的统计机器翻译工具不同，法学硕士编码文化知识，利用这些知识可以放大标签漂移； (3)源语言和目标语言之间的文化相似性或差异性是标签保存的关键决定因素。我们的研究结果强调，忽视机器翻译中的文化因素不仅会损害标签保真度，而且还会在下游应用中带来误解和文化冲突的风险。

Title: SymCode: A Neurosymbolic Approach to Mathematical Reasoning via Verifiable Code Generation

• Authors: Sina Bagheri Nezhad, Yao Li, Ameeta Agrawal
• Subjects: cs.CL, cs.PL
• Abstract URL: https://arxiv.org/abs/2510.25975
• Pdf URL: https://arxiv.org/pdf/2510.25975
• Copy Paste: [[2510.25975]] SymCode: A Neurosymbolic Approach to Mathematical Reasoning via Verifiable Code Generation(https://arxiv.org/abs/2510.25975)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) often struggle with complex mathematical reasoning, where prose-based generation leads to unverified and arithmetically unsound solutions. Current prompting strategies like Chain of Thought still operate within this unreliable medium, lacking a mechanism for deterministic verification. To address these limitations, we introduce SymCode, a neurosymbolic framework that reframes mathematical problem-solving as a task of verifiable code generation using the SymPy library. We evaluate SymCode on challenging benchmarks, including MATH-500 and OlympiadBench, demonstrating significant accuracy improvements of up to 13.6 percentage points over baselines. Our analysis shows that SymCode is not only more token-efficient but also fundamentally shifts model failures from opaque logical fallacies towards transparent, programmatic errors. By grounding LLM reasoning in a deterministic symbolic engine, SymCode represents a key step towards more accurate and trustworthy AI in formal domains.
• 摘要：大型语言模型（LLM）经常与复杂的数学推理作斗争，其中基于散文的生成会导致未经验证且算术上不健全的解决方案。当前的诸如思想链之类的提示策略仍然在这种不可靠的介质中运行，缺乏确定性验证的机制。为了解决这些限制，我们引入了 SymCode，这是一种神经符号框架，它将数学问题解决重新构建为使用 SymPy 库生成可验证代码的任务。我们在具有挑战性的基准（包括 MATH-500 和 OlympiadBench）上对 SymCode 进行了评估，结果显示准确度比基准显着提高了 13.6 个百分点。我们的分析表明，SymCode 不仅具有更高的令牌效率，而且从根本上将模型故障从不透明的逻辑谬误转变为透明的程序错误。通过将 LLM 推理建立在确定性符号引擎中，SymCode 代表了在正式领域中迈向更准确、更值得信赖的人工智能的关键一步。

Title: NeuronMM: High-Performance Matrix Multiplication for LLM Inference on AWS Trainium

• Authors: Dinghong Song (1), Jierui Xu (2), Weichu Yang (2), Pengfei Su (1), Dong Li (1) ((1) University of California, Merced, (2) University of Wisconsin, Madison)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.25977
• Pdf URL: https://arxiv.org/pdf/2510.25977
• Copy Paste: [[2510.25977]] NeuronMM: High-Performance Matrix Multiplication for LLM Inference on AWS Trainium(https://arxiv.org/abs/2510.25977)
• Keywords: llm
• Abstract: AI accelerators, customized to AI workloads, provide cost-effective and high-performance solutions for training and inference. Trainium, an AI accelerator recently developed by Amazon Web Services (AWS), provides an attractive option for LLM training and inference through its heterogeneous architecture. However, leveraging Trainium architecture for high performance can be challenging because of its systolic array architecture and special requirement on data layout. In this paper, we design high-performance matrix multiplication (matmul), a critical compute kernel, for LLM inference on Trainium. We introduce a series of techniques customized to Trainium based on kernel fusion and novel caching strategies to reduce data movement across the software-managed memory hierarchy, maximize SRAM bandwidth, and avoid expensive matrix transpose. Evaluating with nine datasets and four recent LLMs, we show that our system largely outperforms the state-of-the-art matmul implemented by AWS on Trainium: at the level of matmul kernel, it achieves an average 1.35x speedup (up to 2.22x), which translates to an average 1.66x speedup (up to 2.49x) for end-to-end LLM inference.
• 摘要：针对 AI 工作负载定制的 AI 加速器可为训练和推理提供经济高效且高性能的解决方案。 Trainium 是 Amazon Web Services (AWS) 最近开发的人工智能加速器，通过其异构架构为 LLM 训练和推理提供了一个有吸引力的选择。然而，由于其脉动阵列架构和对数据布局的特殊要求，利用 Trainium 架构实现高性能可能具有挑战性。在本文中，我们设计了高性能矩阵乘法（matmul），这是一个关键的计算内核，用于 Trainium 上的 LLM 推理。我们引入了一系列基于内核融合和新颖的缓存策略为 Trainium 定制的技术，以减少软件管理的内存层次结构中的数据移动、最大化 SRAM 带宽并避免昂贵的矩阵转置。通过对九个数据集和四个最近的 LLM 进行评估，我们发现我们的系统在很大程度上优于 AWS 在 Trainium 上实现的最先进的 matmul：在 matmul 内核级别，它实现了平均 1.35 倍的加速（高达 2.22 倍），这意味着端到端 LLM 推理的平均加速为 1.66 倍（高达 2.49 倍）。

Title: AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache

• Authors: Dinghong Song (1), Yuan Feng (1), Yiwei Wang (1), Shangye Chen (1), Cyril Guyot (2), Filip Blagojevic (2), Hyeran Jeon (1), Pengfei Su (1), Dong Li (1) ((1) University of California, Merced, USA, (2) Western Digital Research, USA)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.25979
• Pdf URL: https://arxiv.org/pdf/2510.25979
• Copy Paste: [[2510.25979]] AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache(https://arxiv.org/abs/2510.25979)
• Keywords: language model, llm, chat
• Abstract: Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification, question answering, recommendation, and text embedding rely solely on the prefill stage of inference, where the model encodes input sequences without performing autoregressive decoding. In these prefill only scenarios, the self-attention computation becomes the primary performance bottleneck due to its quadratic complexity with respect to sequence length. In this paper, we observe that semantically different sentences often produce similar attention maps across layers and heads. Building on this insight, we propose AttnCache, a framework that accelerates the prefill stage of LLM inference by retrieving and reusing similar attention maps. Based on an attention map memorization database, AttnCache employs efficient caching and similarity search techniques to identify and reuse pre-cached attention maps during inference, thereby reducing the computational overhead of self-attention. Experimental results show that AttnCache achieves an average of 1.2x end-to-end and 2x attention speedup on CPU, and 1.6x end-to-end and 3x attention speedup on GPU, with negligible accuracy degradation.
• 摘要：大型语言模型 (LLM) 广泛应用于生成应用程序，例如聊天、代码生成和推理。然而，许多现实世界的工作负载（例如分类、问答、推荐和文本嵌入）仅依​​赖于推理的预填充阶段，其中模型对输入序列进行编码而不执行自回归解码。在这些仅预填充的场景中，自注意力计算由于其相对于序列长度的二次复杂度而成为主要性能瓶颈。在本文中，我们观察到语义不同的句子通常会在层和头之间产生相似的注意力图。基于这一见解，我们提出了 AttnCache，这是一个通过检索和重用类似注意力图来加速 LLM 推理的预填充阶段的框架。 AttnCache基于注意力图记忆数据库，采用高效的缓存和相似性搜索技术来在推理过程中识别和重用预先缓存的注意力图，从而减少自注意力的计算开销。实验结果表明，AttnCache 在 CPU 上平均实现 1.2 倍的端到端加速和 2 倍的注意力加速，在 GPU 上平均实现 1.6 倍的端到端和 3 倍的注意力加速，且精度下降可以忽略不计。

Title: Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

• Authors: Yihe Deng, I-Hung Hsu, Jun Yan, Zifeng Wang, Rujun Han, Gufeng Zhang, Yanfei Chen, Wei Wang, Tomas Pfister, Chen-Yu Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.25992
• Pdf URL: https://arxiv.org/pdf/2510.25992
• Copy Paste: [[2510.25992]] Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning(https://arxiv.org/abs/2510.25992)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) often struggle with problems that require multi-step reasoning. For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to overfit long demonstrations through rigid token-by-token imitation. To address this gap, we propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions". SRL trains the model to generate an internal reasoning monologue before committing to each action. It provides smoother rewards based on the similarity between the model's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As a result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning-oriented LLMs.
• 摘要：大型语言模型 (LLM) 经常会遇到需要多步骤推理的问题。对于小规模开源模型，可验证奖励强化学习（RLVR）在多次尝试后也很少采样到正确的解决方案时会失败，而监督微调（SFT）往往会通过严格的逐个令牌模仿来过度拟合长时间的演示。为了解决这一差距，我们提出了监督强化学习（SRL），这是一个将问题解决重新表述为生成一系列逻辑“动作”的框架。 SRL 训练模型在执行每个操作之前生成内部推理独白。它根据模型动作与从 SFT 数据集中提取的专家动作之间的相似性，以逐步的方式提供更平滑的奖励。即使所有的展示都不正确，这种监督也能提供更丰富的学习信号，同时鼓励在专家演示的指导下进行灵活的推理。因此，SRL 使小型模型能够学习以前 SFT 或 RLVR 无法学习的挑战性问题。此外，在使用 RLVR 进行优化之前使用 SRL 进行初始化训练可以产生最强的整体性能。除了推理基准之外，SRL 还有效地推广到代理软件工程任务，将其建立为面向推理的法学硕士的强大且多功能的培训框架。

Title: PORTool: Tool-Use LLM Training with Rewarded Tree

• Authors: Feijie Wu, Weiwu Zhu, Yuxiang Zhang, Soumya Chatterjee, Jiarong Zhu, Fan Mo, Rodin Luo, Jing Gao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26020
• Pdf URL: https://arxiv.org/pdf/2510.26020
• Copy Paste: [[2510.26020]] PORTool: Tool-Use LLM Training with Rewarded Tree(https://arxiv.org/abs/2510.26020)
• Keywords: language model, llm
• Abstract: Current tool-use large language models (LLMs) are trained on static datasets, enabling them to interact with external tools and perform multi-step, tool-integrated reasoning, which produces tool-call trajectories. However, these models imitate how a query is resolved in a generic tool-call routine, thereby failing to explore possible solutions and demonstrating limited performance in an evolved, dynamic tool-call environment. In this work, we propose PORTool, a reinforcement learning (RL) method that encourages a tool-use LLM to explore various trajectories yielding the correct answer. Specifically, this method starts with generating multiple rollouts for a given query, and some of them share the first few tool-call steps, thereby forming a tree-like structure. Next, we assign rewards to each step, based on its ability to produce a correct answer and make successful tool calls. A shared step across different trajectories receives the same reward, while different steps under the same fork receive different rewards. Finally, these step-wise rewards are used to calculate fork-relative advantages, blended with trajectory-relative advantages, to train the LLM for tool use. The experiments utilize 17 tools to address user queries, covering both time-sensitive and time-invariant topics. We conduct ablation studies to systematically justify the necessity and the design robustness of step-wise rewards. Furthermore, we compare the proposed PORTool with other training approaches and demonstrate significant improvements in final accuracy and the number of tool-call steps.
• 摘要：当前的工具使用大型语言模型（LLM）是在静态数据集上进行训练的，使它们能够与外部工具交互并执行多步骤、工具集成推理，从而产生工具调用轨迹。然而，这些模型模仿了如何在通用工具调用例程中解决查询，因此无法探索可能的解决方案，并且在不断发展的动态工具调用环境中表现出有限的性能。在这项工作中，我们提出了 PORTool，这是一种强化学习 (RL) 方法，鼓励使用工具的法学硕士探索各种轨迹，从而产生正确的答案。具体来说，该方法首先为给定查询生成多个推出，其中一些共享前几个工具调用步骤，从而形成树状结构。接下来，我们根据每个步骤产生正确答案和成功调用工具的能力为其分配奖励。不同轨迹上的共享步骤会获得相同的奖励，而同一分叉下的不同步骤会获得不同的奖励。最后，这些逐步奖励用于计算分叉相对优势，与轨迹相对优势相结合，以训练法学硕士的工具使用。实验利用 17 种工具来解决用户查询，涵盖时间敏感和时间不变的主题。我们进行消融研究，以系统地证明逐步奖励的必要性和设计的稳健性。此外，我们将所提出的 PORTool 与其他训练方法进行了比较，并证明了最终准确性和工具调用步骤数量的显着改进。

Title: Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs

• Authors: HyoJung Han, Sweta Agrawal, Eleftheria Briakou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26024
• Pdf URL: https://arxiv.org/pdf/2510.26024
• Copy Paste: [[2510.26024]] Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs(https://arxiv.org/abs/2510.26024)
• Keywords: language model, llm
• Abstract: Cross-lingual alignment (CLA) aims to align multilingual representations, enabling Large Language Models (LLMs) to seamlessly transfer knowledge across languages. While intuitive, we hypothesize, this pursuit of representational convergence can inadvertently cause "cultural erasure", the functional loss of providing culturally-situated responses that should diverge based on the query language. In this work, we systematically analyze this trade-off by introducing a holistic evaluation framework, the transfer-localization plane, which quantifies both desirable knowledge transfer and undesirable cultural erasure. Using this framework, we re-evaluate recent CLA approaches and find that they consistently improve factual transfer at the direct cost of cultural localization across all six languages studied. Our investigation into the internal representations of these models reveals a key insight: universal factual transfer and culturally-specific knowledge are optimally steerable at different model layers. Based on this finding, we propose Surgical Steering, a novel inference-time method that disentangles these two objectives. By applying targeted activation steering to distinct layers, our approach achieves a better balance between the two competing dimensions, effectively overcoming the limitations of current alignment techniques.
• 摘要：跨语言对齐 (CLA) 旨在对齐多语言表示，使大型语言模型 (LLM) 能够跨语言无缝传输知识。我们假设，虽然直观，但这种对表征趋同的追求可能会无意中导致“文化擦除”，即提供应根据查询语言而有所不同的文化情境响应的功能损失。在这项工作中，我们通过引入整体评估框架（转移本地化平面）系统地分析了这种权衡，该框架量化了理想的知识转移和不良的文化擦除。使用这个框架，我们重新评估了最近的 CLA 方法，发现它们在所研究的所有六种语言中以文化本地化的直接成本为代价，持续改善了事实转移。我们对这些模型的内部表示的调查揭示了一个关键的见解：普遍的事实转移和文化特定的知识在不同的模型层上是最佳可操纵的。基于这一发现，我们提出了 Surgical Steering，这是一种新颖的推理时间方法，可以解开这两个目标。通过将目标激活控制应用于不同的层，我们的方法在两个竞争维度之间实现了更好的平衡，有效克服了当前对齐技术的局限性。

Title: QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback

• Authors: Taku Mikuriya, Tatsuya Ishigaki, Masayuki Kawarada, Shunya Minami, Tadashi Kadowaki, Yohichi Suzuki, Soshun Naito, Shunya Takata, Takumi Kato, Tamotsu Basseda, Reo Yamada, Hiroya Takamura
• Subjects: cs.CL, cs.PL, quant-ph
• Abstract URL: https://arxiv.org/abs/2510.26101
• Pdf URL: https://arxiv.org/pdf/2510.26101
• Copy Paste: [[2510.26101]] QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback(https://arxiv.org/abs/2510.26101)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have increasingly been applied to automatic programming code generation. This task can be viewed as a language generation task that bridges natural language, human knowledge, and programming logic. However, it remains underexplored in domains that require interaction with hardware devices, such as quantum programming, where human coders write Python code that is executed on a quantum computer. To address this gap, we introduce QCoder Benchmark, an evaluation framework that assesses LLMs on quantum programming with feedback from simulated hardware devices. Our benchmark offers two key features. First, it supports evaluation using a quantum simulator environment beyond conventional Python execution, allowing feedback of domain-specific metrics such as circuit depth, execution time, and error classification, which can be used to guide better generation. Second, it incorporates human-written code submissions collected from real programming contests, enabling both quantitative comparisons and qualitative analyses of LLM outputs against human-written codes. Our experiments reveal that even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the benchmark. In contrast, reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes (39.98%). We release the QCoder Benchmark dataset and public evaluation API to support further research.
• 摘要：大型语言模型（LLM）越来越多地应用于自动编程代码生成。该任务可以被视为连接自然语言、人类知识和编程逻辑的语言生成任务。然而，在需要与硬件设备交互的领域，例如量子编程，人类编码人员编写在量子计算机上执行的 Python 代码，它仍然没有得到充分探索。为了弥补这一差距，我们引入了 QCoder Benchmark，这是一个评估框架，可根据模拟硬件设备的反馈来评估 LLM 的量子编程能力。我们的基准测试提供了两个关键功能。首先，它支持使用传统 Python 执行之外的量子模拟器环境进行评估，允许反馈特定领域的指标，例如电路深度、执行时间和错误分类，这些指标可用于指导更好的生成。其次，它结合了从真实编程竞赛中收集的人类编写的代码提交，可以对法学硕士输出与人类编写的代码进行定量比较和定性分析。我们的实验表明，即使是像 GPT-4o 这样的先进模型，准确率也只能达到 18.97% 左右，凸显了基准测试的难度。相比之下，o3 等基于推理的模型准确率高达 78%，超过了人类编写代码的平均成功率 (39.98%)。我们发布了 QCoder Benchmark 数据集和公共评估 API 以支持进一步的研究。

Title: Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking

• Authors: Feng Ju, Zeyu Qin, Rui Min, Zhitao He, Lingpeng Kong, Yi R. Fung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26122
• Pdf URL: https://arxiv.org/pdf/2510.26122
• Copy Paste: [[2510.26122]] Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking(https://arxiv.org/abs/2510.26122)
• Keywords: language model, llm, chain-of-thought
• Abstract: While Test-Time Scaling (TTS) has proven effective in improving the reasoning ability of large language models (LLMs), low diversity in model outputs often becomes a bottleneck; this is partly caused by the common "one problem, one solution" (1P1S) training practice, which provides a single canonical answer and can push models toward a narrow set of reasoning paths. To address this, we propose a "one problem, multiple solutions" (1PNS) training paradigm that exposes the model to a variety of valid reasoning trajectories and thus increases inference diversity. A core challenge for 1PNS is reliably measuring semantic differences between multi-step chains of thought, so we introduce Reasoning Path Divergence (RPD), a step-level metric that aligns and scores Long Chain-of-Thought solutions to capture differences in intermediate reasoning. Using RPD, we curate maximally diverse solution sets per problem and fine-tune Qwen3-4B-Base. Experiments show that RPD-selected training yields more varied outputs and higher pass@k, with an average +2.80% gain in pass@16 over a strong 1P1S baseline and a +4.99% gain on AIME24, demonstrating that 1PNS further amplifies the effectiveness of TTS. Our code is available at this https URL .
• 摘要：虽然测试时间缩放（TTS）已被证明可以有效提高大型语言模型（LLM）的推理能力，但模型输出的低多样性往往成为瓶颈；这部分是由于常见的“一个问题，一个解决方案”（1P1S）训练实践造成的，它提供了单一的规范答案，并且可以将模型推向一组狭窄的推理路径。为了解决这个问题，我们提出了一种“一个问题，多个解决方案”（1PNS）训练范式，使模型暴露于各种有效的推理轨迹，从而增加推理多样性。 1PNS 的核心挑战是可靠地测量多步骤思想链之间的语义差异，因此我们引入了推理路径发散 (RPD)，这是一种步骤级度量，可对长思想链解决方案进行对齐和评分，以捕获中间推理中的差异。使用 RPD，我们为每个问题策划最大程度多样化的解决方案集并微调 Qwen3-4B-Base。实验表明，RPD 选择的训练产生更多样的输出和更高的 pass@k，与强大的 1P1S 基线相比，pass@16 平均增益 +2.80%，在 AIME24 上增益 +4.99%，这表明 1PNS 进一步增强了 TTS 的有效性。我们的代码可在此 https URL 获取。

Title: On the Influence of Discourse Relations in Persuasive Texts

• Authors: Nawar Turk, Sevag Kaspar, Leila Kosseim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26124
• Pdf URL: https://arxiv.org/pdf/2510.26124
• Copy Paste: [[2510.26124]] On the Influence of Discourse Relations in Persuasive Texts(https://arxiv.org/abs/2510.26124)
• Keywords: language model, llm, prompt
• Abstract: This paper investigates the relationship between Persuasion Techniques (PTs) and Discourse Relations (DRs) by leveraging Large Language Models (LLMs) and prompt engineering. Since no dataset annotated with both PTs and DRs exists, we took the SemEval 2023 Task 3 dataset labelled with 19 PTs as a starting point and developed LLM-based classifiers to label each instance of the dataset with one of the 22 PDTB 3.0 level-2 DRs. In total, four LLMs were evaluated using 10 different prompts, resulting in 40 unique DR classifiers. Ensemble models using different majority-pooling strategies were used to create 5 silver datasets of instances labelled with both persuasion techniques and level-2 PDTB senses. The silver dataset sizes vary from 1,281 instances to 204 instances, depending on the majority pooling technique used. Statistical analysis of these silver datasets shows that six discourse relations (namely Cause, Purpose, Contrast, Cause+Belief, Concession, and Condition) play a crucial role in persuasive texts, especially in the use of Loaded Language, Exaggeration/Minimisation, Repetition and to cast Doubt. This insight can contribute to detecting online propaganda and misinformation, as well as to our general understanding of effective communication.
• 摘要：本文利用大型语言模型 (LLM) 和提示工程研究说服技术 (PT) 和话语关系 (DR) 之间的关系。由于不存在同时标注 PT 和 DR 的数据集，因此我们以标注有 19 个 PT 的 SemEval 2023 Task 3 数据集为起点，开发了基于 LLM 的分类器，用 22 个 PDTB 3.0 2 级 DR 之一来标记数据集的每个实例。总共，使用 10 种不同的提示对 4 名法学硕士进行了评估，产生了 40 个独特的 DR 分类器。使用不同的多数池化策略的集成模型被用来创建 5 个实例银数据集，并用说服技术和 2 级 PDTB 意义进行标记。白银数据集大小从 1,281 个实例到 204 个实例不等，具体取决于所使用的多数池技术。对这些银数据集的统计分析表明，六种话语关系（即原因、目的、对比、原因+信念、让步和条件）在说服性文本中发挥着至关重要的作用，特别是在负载语言、夸张/最小化、重复和质疑的使用中。这种洞察力有助于检测在线宣传和错误信息，以及我们对有效沟通的一般理解。

Title: MossNet: Mixture of State-Space Experts is a Multi-Head Attention

• Authors: Shikhar Tuli, James Seale Smith, Haris Jeelani, Chi-Heng Lin, Abhishek Patel, Vasili Ramanishka, Yen-Chang Hsu, Hongxia Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26182
• Pdf URL: https://arxiv.org/pdf/2510.26182
• Copy Paste: [[2510.26182]] MossNet: Mixture of State-Space Experts is a Multi-Head Attention(https://arxiv.org/abs/2510.26182)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have significantly advanced generative applications in natural language processing (NLP). Recent trends in model architectures revolve around efficient variants of transformers or state-space/gated-recurrent models (SSMs, GRMs). However, prevailing SSM/GRM-based methods often emulate only a single attention head, potentially limiting their expressiveness. In this work, we propose MossNet, a novel mixture-of-state-space-experts architecture that emulates a linear multi-head attention (MHA). MossNet leverages a mixture-of-experts (MoE) implementation not only in channel-mixing multi-layered perceptron (MLP) blocks but also in the time-mixing SSM kernels to realize multiple "attention heads." Extensive experiments on language modeling and downstream evaluations show that MossNet outperforms both transformer- and SSM-based architectures of similar model size and data budgets. Larger variants of MossNet, trained on trillions of tokens, further confirm its scalability and superior performance. In addition, real-device profiling on a Samsung Galaxy S24 Ultra and an Nvidia A100 GPU demonstrate favorable runtime speed and resource usage compared to similarly sized baselines. Our results suggest that MossNet is a compelling new direction for efficient, high-performing recurrent LLM architectures.
• 摘要：大型语言模型 (LLM) 在自然语言处理 (NLP) 中具有显着先进的生成应用。模型架构的最新趋势围绕着变压器或状态空间/门控循环模型（SSM、GRM）的高效变体。然而，流行的基于 SSM/GRM 的方法通常仅模拟单个注意力头，这可能限制了它们的表达能力。在这项工作中，我们提出了 MossNet，一种新颖的状态空间专家混合架构，可模拟线性多头注意力（MHA）。 MossNet 不仅在通道混合多层感知器 (MLP) 块中利用专家混合 (MoE) 实现，还在时间混合 SSM 内核中利用来实现多个“注意力头”。关于语言建模和下游评估的大量实验表明，MossNet 的性能优于模型大小和数据预算相似的基于 Transformer 和 SSM 的架构。 MossNet 的更大变体经过数万亿代币的训练，进一步证实了其可扩展性和卓越的性能。此外，与类似大小的基准相比，三星 Galaxy S24 Ultra 和 Nvidia A100 GPU 上的真实设备分析显示出良好的运行速度和资源使用情况。我们的结果表明，MossNet 是高效、高性能循环 LLM 架构的一个引人注目的新方向。

Title: Similarity-Distance-Magnitude Language Models

• Authors: Allen Schmaltz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26183
• Pdf URL: https://arxiv.org/pdf/2510.26183
• Copy Paste: [[2510.26183]] Similarity-Distance-Magnitude Language Models(https://arxiv.org/abs/2510.26183)
• Keywords: language model
• Abstract: We introduce Similarity-Distance-Magnitude (SDM) language models (LMs), which are sequence prediction models fine-tuned to maximize the proportion of generations in the well-calibrated, high-probability region partitioned by a final-layer SDM activation layer used for binary classification of instruction-following. We demonstrate that existing pre-trained decoder-only Transformer LMs can be readily converted into SDM LMs via supervised fine-tuning, using the final-layer SDM activation layer during training to estimate a change-of-base for a supervised next-token loss over a contrastive input encoding scheme, with additional hard negative examples generated online during training. This results in reduced abstentions (i.e., improved statistical efficiency) compared to strong supervised baselines.
• 摘要：我们引入了相似性-距离-幅度（SDM）语言模型（LM），它们是经过微调的序列预测模型，以最大化经过良好校准的高概率区域中的代比例，该区域由用于指令跟踪的二进制分类的最终层 SDM 激活层划分。我们证明，现有的仅预训练解码器的 Transformer LM 可以通过监督微调轻松转换为 SDM LM，在训练期间使用最后层 SDM 激活层来估计对比输入编码方案中监督下一个令牌损失的基数变化，并在训练期间在线生成额外的硬负例。与强有力的监督基线相比，这会减少弃权率（即提高统计效率）。

Title: RCScore: Quantifying Response Consistency in Large Language Models

• Authors: Dongjun Jang, Youngchae Ahn, Hyopil Shin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26193
• Pdf URL: https://arxiv.org/pdf/2510.26193
• Copy Paste: [[2510.26193]] RCScore: Quantifying Response Consistency in Large Language Models(https://arxiv.org/abs/2510.26193)
• Keywords: language model, llm
• Abstract: Current LLM evaluations often rely on a single instruction template, overlooking models' sensitivity to instruction style-a critical aspect for real-world deployments. We present RCScore, a multi-dimensional framework quantifying how instruction formulation affects model responses. By systematically transforming benchmark problems into multiple instruction styles, RCScore reveals performance variations undetected by conventional metrics. Our experiments across ten LLMs on four reasoning benchmarks demonstrate that instruction style can shift accuracy by up to 16.7% points. We introduce Cross-Response Similarity (CRS), a method applying RCScore metrics to measure stylistic self-consistency, and establish its strong correlation with task accuracy, suggesting consistency as a valuable proxy for model reliability. Additional findings show that deterministic decoding produces more stylistically stable outputs, and model scale correlates positively with cross-style consistency. RCScore offers a principled approach to assess instruction robustness.
• 摘要：当前的法学硕士评估通常依赖于单一指令模板，忽视了模型对指令风格的敏感性——这是现实世界部署的一个关键方面。我们提出了 RCScore，一个量化指令制定如何影响模型响应的多维框架。通过系统地将基准测试问题转化为多种指令风格，RCScore 揭示了传统指标无法检测到的性能变化。我们针对 10 个法学硕士在四个推理基准上进行的实验表明，教学风格可以将准确性提高高达 16.7%。我们引入了交叉响应相似度（CRS），这是一种应用 RCScore 指标来衡量风格自我一致性的方法，并建立了其与任务准确性的强相关性，表明一致性是模型可靠性的一个有价值的代理。其他研究结果表明，确定性解码会产生风格上更稳定的输出，并且模型规模与跨风格一致性呈正相关。 RCScore 提供了评估指令稳健性的原则性方法。

Title: Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation

• Authors: Woojin Kim, Jaeyoung Do
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26200
• Pdf URL: https://arxiv.org/pdf/2510.26200
• Copy Paste: [[2510.26200]] Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation(https://arxiv.org/abs/2510.26200)
• Keywords: language model
• Abstract: While diffusion language models (DLMs) enable fine-grained refinement, their practical controllability remains fragile. We identify and formally characterize a central failure mode called update forgetting, in which uniform and context agnostic updates induce token level fluctuations across timesteps, erasing earlier semantic edits and disrupting the cumulative refinement process, thereby degrading fluency and coherence. As this failure originates in uniform and context agnostic updates, effective control demands explicit token ordering. We propose Token Timestep Allocation (TTA), which realizes soft and semantic token ordering via per token timestep schedules: critical tokens are frozen early, while uncertain tokens receive continued refinement. This timestep based ordering can be instantiated as either a fixed policy or an adaptive policy driven by task signals, thereby supporting a broad spectrum of refinement strategies. Because it operates purely at inference time, it applies uniformly across various DLMs and naturally extends to diverse supervision sources. Empirically, TTA improves controllability and fluency: on sentiment control, it yields more than 20 percent higher accuracy and nearly halves perplexity using less than one fifth the steps; in detoxification, it lowers maximum toxicity (12.2 versus 14.5) and perplexity (26.0 versus 32.0). Together, these results demonstrate that softened ordering via timestep allocation is the critical lever for mitigating update forgetting and achieving stable and controllable diffusion text generation.
• 摘要：虽然扩散语言模型（DLM）可以进行细粒度的细化，但它们的实际可控性仍然脆弱。我们识别并正式描述了一种称为更新遗忘的中心故障模式，其中统一和上下文无关的更新会导致跨时间步的标记级别波动，擦除早期的语义编辑并扰乱累积的细化过程，从而降低流畅性和连贯性。由于这种失败源于统一且与上下文无关的更新，因此有效的控制需要显式的令牌排序。我们提出令牌时间步分配（TTA），它通过每个令牌时间步计划实现软和语义令牌排序：关键令牌被提前冻结，而不确定的令牌则得到持续的细化。这种基于时间步长的排序可以实例化为固定策略或由任务信号驱动的自适应策略，从而支持广泛的细化策略。因为它纯粹在推理时运行，所以它统一适用于各种 DLM，并自然地扩展到不同的监督源。从经验上看，TTA 提高了可控性和流畅性：在情绪控制方面，它的准确率提高了 20% 以上，并且使用不到五分之一的步骤将困惑度降低了近一半；在解毒过程中，它降低了最大毒性（12.2 与 14.5）和困惑度（26.0 与 32.0）。总之，这些结果表明，通过时间步分配的软排序是减轻更新遗忘和实现稳定可控的扩散文本生成的关键杠杆。

Title: What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

• Authors: Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26202
• Pdf URL: https://arxiv.org/pdf/2510.26202
• Copy Paste: [[2510.26202]] What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data(https://arxiv.org/abs/2510.26202)
• Keywords: language model
• Abstract: Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.
• 摘要：人类反馈可能会以不可预测和不良的方式改变语言模型，因为从业者对反馈数据编码的内容缺乏清晰的了解。虽然之前的工作研究了对某些属性（例如长度或阿谀奉承）的偏好，但在不预先指定假设的情况下自动提取相关特征仍然具有挑战性。我们介绍我的人类反馈中有什么？ （WIMHF），一种使用稀疏自动编码器解释反馈数据的方法。 WIMHF 表征了 (1) 数据集能够测量的偏好和 (2) 注释者实际表达的偏好。在 7 个数据集中，WIMHF 识别出了少量人类可解释的特征，这些特征占黑盒模型实现的偏好预测信号的大部分。这些特征揭示了人类偏好的广泛多样性，以及数据集级上下文的作用：例如，Reddit 上的用户喜欢非正式内容和笑话，而 HH-RLHF 和 PRISM 中的注释者不喜欢它们。 WIMHF 还暴露了潜在的不安全偏好，例如 LMArena 用户倾向于投票反对拒绝，而往往支持有毒内容。学习到的功能可以实现有效的数据管理：在 Arena 中重新标记有害示例可以带来巨大的安全收益 (+37%)，并且不会影响总体性能。它们还允许细粒度的个性化：在社区对齐数据集上，我们学习注释者特定的权重而不是主观特征，从而改善偏好预测。 WIMHF为从业者提供了一种以人为本的分析方法，以更好地理解和使用偏好数据。

Title: Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning

• Authors: Qi Luo, Xiaonan Li, Tingshuo Fan, Xinchi Chen, Xipeng Qiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26205
• Pdf URL: https://arxiv.org/pdf/2510.26205
• Copy Paste: [[2510.26205]] Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning(https://arxiv.org/abs/2510.26205)
• Keywords: language model, llm, hallucination, retrieval augmented generation, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) has emerged as a leading approach to reducing hallucinations in large language models (LLMs). Current RAG evaluation benchmarks primarily focus on what we call local RAG: retrieving relevant chunks from a small subset of documents to answer queries that require only localized understanding within specific text chunks. However, many real-world applications require a fundamentally different capability -- global RAG -- which involves aggregating and analyzing information across entire document collections to derive corpus-level insights (for example, "What are the top 10 most cited papers in 2023?"). In this paper, we introduce GlobalQA -- the first benchmark specifically designed to evaluate global RAG capabilities, covering four core task types: counting, extremum queries, sorting, and top-k extraction. Through systematic evaluation across different models and baselines, we find that existing RAG methods perform poorly on global tasks, with the strongest baseline achieving only 1.51 F1 score. To address these challenges, we propose GlobalRAG, a multi-tool collaborative framework that preserves structural coherence through chunk-level retrieval, incorporates LLM-driven intelligent filters to eliminate noisy documents, and integrates aggregation modules for precise symbolic computation. On the Qwen2.5-14B model, GlobalRAG achieves 6.63 F1 compared to the strongest baseline's 1.51 F1, validating the effectiveness of our method.
• 摘要：检索增强生成（RAG）已成为减少大语言模型（LLM）中幻觉的主要方法。当前的 RAG 评估基准主要关注我们所说的本地 RAG：从一小部分文档中检索相关块，以回答仅需要在特定文本块中进行本地化理解的查询。然而，许多现实世界的应用程序需要一种根本不同的功能——全局 RAG——它涉及聚合和分析整个文档集合中的信息，以得出语料库级别的见解（例如，“2023 年被引用次数最多的 10 篇论文是什么？”）。在本文中，我们介绍了 GlobalQA——第一个专门用于评估全局 RAG 能力的基准测试，涵盖四种核心任务类型：计数、极值查询、排序和 top-k 提取。通过对不同模型和基线的系统评估，我们发现现有的 RAG 方法在全局任务上表现不佳，最强的基线仅达到 1.51 F1 分数。为了应对这些挑战，我们提出了 GlobalRAG，这是一个多工具协作框架，它通过块级检索保持结构一致性，结合 LLM 驱动的智能过滤器来消除噪声文档，并集成聚合模块以进行精确的符号计算。在 Qwen2.5-14B 模型上，GlobalRAG 实现了 6.63 F1，而最强基线的 F1 为 1.51，验证了我们方法的有效性。

Title: Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs

• Authors: Takuma Sato, Seiya Kawano, Koichiro Yoshino
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26253
• Pdf URL: https://arxiv.org/pdf/2510.26253
• Copy Paste: [[2510.26253]] Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs(https://arxiv.org/abs/2510.26253)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: The ability to accurately interpret implied meanings plays a crucial role in human communication and language use, and language models are also expected to possess this capability. This study demonstrates that providing language models with pragmatic theories as prompts is an effective in-context learning approach for tasks to understand implied meanings. Specifically, we propose an approach in which an overview of pragmatic theories, such as Gricean pragmatics and Relevance Theory, is presented as a prompt to the language model, guiding it through a step-by-step reasoning process to derive a final interpretation. Experimental results showed that, compared to the baseline, which prompts intermediate reasoning without presenting pragmatic theories (0-shot Chain-of-Thought), our methods enabled language models to achieve up to 9.6\% higher scores on pragmatic reasoning tasks. Furthermore, we show that even without explaining the details of pragmatic theories, merely mentioning their names in the prompt leads to a certain performance improvement (around 1-3%) in larger models compared to the baseline.
• 摘要：准确解释隐含意义的能力在人类交流和语言使用中起着至关重要的作用，语言模型也有望具备这种能力。这项研究表明，为语言模型提供语用理论作为提示是理解隐含意义任务的一种有效的上下文学习方法。具体来说，我们提出了一种方法，其中对格莱斯语用学和相关性理论等语用理论的概述作为对语言模型的提示，引导其通过逐步推理过程得出最终解释。实验结果表明，与在不提供语用理论（0-shot Chain-of-Thought）的情况下进行中间推理的基线相比，我们的方法使语言模型在语用推理任务上的得分提高了 9.6%。此外，我们表明，即使不解释实用理论的细节，仅在提示中提及它们的名称，与基线相比，在较大模型中也会带来一定的性能提升（大约 1-3%）。

Title: Language Models Are Borrowing-Blind: A Multilingual Evaluation of Loanword Identification across 10 Languages

• Authors: Mérilin Sousa Silva, Sina Ahmadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26254
• Pdf URL: https://arxiv.org/pdf/2510.26254
• Copy Paste: [[2510.26254]] Language Models Are Borrowing-Blind: A Multilingual Evaluation of Loanword Identification across 10 Languages(https://arxiv.org/abs/2510.26254)
• Keywords: language model
• Abstract: Throughout language history, words are borrowed from one language to another and gradually become integrated into the recipient's lexicon. Speakers can often differentiate these loanwords from native vocabulary, particularly in bilingual communities where a dominant language continuously imposes lexical items on a minority language. This paper investigates whether pretrained language models, including large language models, possess similar capabilities for loanword identification. We evaluate multiple models across 10 languages. Despite explicit instructions and contextual information, our results show that models perform poorly in distinguishing loanwords from native ones. These findings corroborate previous evidence that modern NLP systems exhibit a bias toward loanwords rather than native equivalents. Our work has implications for developing NLP tools for minority languages and supporting language preservation in communities under lexical pressure from dominant languages.
• 摘要：纵观语言历史，单词从一种语言借用到另一种语言，并逐渐融入接受者的词典中。说话者通常可以将这些外来词与母语词汇区分开来，特别是在主导语言不断将词汇项目强加于少数语言的双语社区中。本文研究了预训练的语言模型（包括大型语言模型）是否具有类似的外来词识别功能。我们评估了 10 种语言的多个模型。尽管有明确的说明和上下文信息，我们的结果表明模型在区分外来词和母语词方面表现不佳。这些发现证实了先前的证据，即现代 NLP 系统表现出对外来词而不是本地对等词的偏见。我们的工作对于开发针对少数语言的 NLP 工具以及支持在主流语言的词汇压力下保护社区的语言具有重要意义。

Title: Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual

• Authors: Sukrit Sriratanawilai, Jhayahgrit Thongwat, Romrawin Chumpu, Patomporn Payoungkhamdee, Sarana Nutanong, Peerat Limkonchotiwat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26271
• Pdf URL: https://arxiv.org/pdf/2510.26271
• Copy Paste: [[2510.26271]] Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual(https://arxiv.org/abs/2510.26271)
• Keywords: language model
• Abstract: Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferring knowledge from larger to smaller VLMs, applying KD in multilingualism is an underexplored area. This paper presents a controlled empirical study of KD behavior across five distillation approaches, isolating their effects on cross-lingual representation consistency and downstream performance stability under model compression. We study five distillation formulations across CLIP and SigLIP2, and evaluate them on in-domain retrieval and out-of-domain visual QA. We find that some configurations preserve or even improve multilingual retrieval robustness despite halving model size, but others fail to maintain cross-task stability, exposing design-sensitive trade-offs that aggregate accuracy alone does not reveal.
• 摘要：视觉语言模型（VLM）在不同语言之间表现出不均匀的性能，当模型大小减小时，这个问题往往会加剧。虽然知识蒸馏 (KD) 在将知识从较大的 VLM 转移到较小的 VLM 方面显示出良好的结果，但在多语言中应用 KD 仍是一个尚未充分探索的领域。本文对五种蒸馏方法的 KD 行为进行了受控实证研究，隔离了它们对模型压缩下的跨语言表示一致性和下游性能稳定性的影响。我们研究了 CLIP 和 SigLIP2 的五种蒸馏公式，并在域内检索和域外视觉 QA 上评估它们。我们发现，尽管模型大小减半，一些配置仍保留甚至提高了多语言检索的鲁棒性，但其他配置无法保持跨任务稳定性，暴露了设计敏感的权衡，而仅聚合准确性并不能揭示这一点。

Title: Do LLMs Signal When They're Right? Evidence from Neuron Agreement

• Authors: Kang Chen, Yaoning Wang, Kai Xiong, Zhuoka Feng, Wenhe Sun, Haotian Chen, Yixin Cao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26277
• Pdf URL: https://arxiv.org/pdf/2510.26277
• Copy Paste: [[2510.26277]] Do LLMs Signal When They're Right? Evidence from Neuron Agreement(https://arxiv.org/abs/2510.26277)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) commonly boost reasoning via sample-evaluate-ensemble decoders, achieving label free gains without ground truth. However, prevailing strategies score candidates using only external outputs such as token probabilities, entropies, or self evaluations, and these signals can be poorly calibrated after post training. We instead analyze internal behavior based on neuron activations and uncover three findings: (1) external signals are low dimensional projections of richer internal dynamics; (2) correct responses activate substantially fewer unique neurons than incorrect ones throughout generation; and (3) activations from correct responses exhibit stronger cross sample agreement, whereas incorrect ones diverge. Motivated by these observations, we propose Neuron Agreement Decoding (NAD), an unsupervised best-of-N method that selects candidates using activation sparsity and cross sample neuron agreement, operating solely on internal signals and without requiring comparable textual outputs. NAD enables early correctness prediction within the first 32 generated tokens and supports aggressive early stopping. Across math and science benchmarks with verifiable answers, NAD matches majority voting; on open ended coding benchmarks where majority voting is inapplicable, NAD consistently outperforms Avg@64. By pruning unpromising trajectories early, NAD reduces token usage by 99% with minimal loss in generation quality, showing that internal signals provide reliable, scalable, and efficient guidance for label free ensemble decoding.
• 摘要：大型语言模型 (LLM) 通常通过样本评估集成解码器来增强推理，从而在没有基本事实的情况下实现无标签增益。然而，流行的策略仅使用外部输出（例如令牌概率、熵或自我评估）对候选人进行评分，并且这些信号在训练后可能很难校准。相反，我们根据神经元激活来分析内部行为，并发现三个发现：（1）外部信号是更丰富的内部动态的低维投影； (2) 在整个世代中，正确的反应激活的独特神经元比错误的神经元要少得多； （3）正确响应的激活表现出更强的跨样本一致性，而错误的响应则出现分歧。受这些观察的启发，我们提出了神经元一致性解码（NAD），这是一种无监督的 N 最佳方法，它使用激活稀疏性和跨样本神经元一致性来选择候选者，仅对内部信号进行操作，而不需要可比较的文本输出。 NAD 能够在前 32 个生成的令牌中实现早期正确性预测，并支持积极的早期停止。在具有可验证答案的数学和科学基准中，NAD 与多数投票相匹配；在不适用多数投票的开放式编码基准上，NAD 始终优于 Avg@64。通过尽早修剪无希望的轨迹，NAD 将令牌使用量减少了 99%，同时生成质量的损失最小，这表明内部信号为无标签集成解码提供了可靠、可扩展且高效的指导。

Title: Unravelling the Mechanisms of Manipulating Numbers in Language Models

• Authors: Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2510.26285
• Pdf URL: https://arxiv.org/pdf/2510.26285
• Copy Paste: [[2510.26285]] Unravelling the Mechanisms of Manipulating Numbers in Language Models(https://arxiv.org/abs/2510.26285)
• Keywords: language model, llm
• Abstract: Recent work has shown that different large language models (LLMs) converge to similar and accurate input embedding representations for numbers. These findings conflict with the documented propensity of LLMs to produce erroneous outputs when dealing with numeric information. In this work, we aim to explain this conflict by exploring how language models manipulate numbers and quantify the lower bounds of accuracy of these mechanisms. We find that despite surfacing errors, different language models learn interchangeable representations of numbers that are systematic, highly accurate and universal across their hidden states and the types of input contexts. This allows us to create universal probes for each LLM and to trace information -- including the causes of output errors -- to specific layers. Our results lay a fundamental understanding of how pre-trained LLMs manipulate numbers and outline the potential of more accurate probing techniques in addressed refinements of LLMs' architectures.
• 摘要：最近的工作表明，不同的大型语言模型（LLM）会收敛到相似且准确的数字输入嵌入表示。这些发现与法学硕士在处理数字信息时产生错误输出的记录倾向相冲突。在这项工作中，我们的目标是通过探索语言模型如何操纵数字并量化这些机制的准确性下限来解释这种冲突。我们发现，尽管存在错误，不同的语言模型仍然可以学习可互换的数字表示，这些表示在其隐藏状态和输入上下文类型中是系统的、高度准确的和通用的。这使我们能够为每个法学硕士创建通用探针，并将信息（包括输出错误的原因）跟踪到特定层。我们的结果奠定了对预先训练的法学硕士如何操纵数字的基本理解，并概述了更准确的探测技术在法学硕士架构的改进中的潜力。

Title: Can Agent Conquer Web? Exploring the Frontiers of ChatGPT Atlas Agent in Web Games

• Authors: Jingran Zhang, Ning Li, Justin Cui
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26298
• Pdf URL: https://arxiv.org/pdf/2510.26298
• Copy Paste: [[2510.26298]] Can Agent Conquer Web? Exploring the Frontiers of ChatGPT Atlas Agent in Web Games(https://arxiv.org/abs/2510.26298)
• Keywords: gpt, chat, agent
• Abstract: OpenAI's ChatGPT Atlas introduces new capabilities for web interaction, enabling the model to analyze webpages, process user intents, and execute cursor and keyboard inputs directly within the browser. While its capacity for information retrieval tasks has been demonstrated, its performance in dynamic, interactive environments remains less explored. In this study, we conduct an early evaluation of Atlas's web interaction capabilities using browser-based games as test scenarios, including Google's T-Rex Runner, Sudoku, Flappy Bird, and this http URL. We employ in-game performance scores as quantitative metrics to assess performance across different task types. Our results show that Atlas performs strongly in logical reasoning tasks like Sudoku, completing puzzles significantly faster than human baselines, but struggles substantially in real-time games requiring precise timing and motor control, often failing to progress beyond initial obstacles. These findings suggest that while Atlas demonstrates capable analytical processing, there remain notable limitations in dynamic web environments requiring real-time interaction. The website of our project can be found at this https URL.
• 摘要：OpenAI 的 ChatGPT Atlas 引入了新的 Web 交互功能，使模型能够分析网页、处理用户意图以及直接在浏览器中执行光标和键盘输入。虽然它的信息检索任务能力已经得到证明，但它在动态、交互式环境中的性能仍然很少被探索。在本研究中，我们使用基于浏览器的游戏作为测试场景，对 Atlas 的网络交互能力进行了早期评估，包括 Google 的 T-Rex Runner、Sudoku、Flappy Bird 和这个 http URL。我们使用游戏中的表现得分作为定量指标来评估不同任务类型的表现。我们的结果表明，Atlas 在数独等逻辑推理任务中表现出色，完成谜题的速度明显快于人类基线，但在需要精确计时和运动控制的实时游戏中表现不佳，常常无法超越最初的障碍。这些发现表明，虽然 Atlas 展示了强大的分析处理能力，但在需要实时交互的动态 Web 环境中仍然存在明显的局限性。我们项目的网站可以在此 https URL 找到。

Title: SCRIBE: Structured Chain Reasoning for Interactive Behaviour Explanations using Tool Calling

• Authors: Fares Fawzi, Vinitra Swamy, Dominik Glandorf, Tanya Nazaretsky, Tanja Käser
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26322
• Pdf URL: https://arxiv.org/pdf/2510.26322
• Copy Paste: [[2510.26322]] SCRIBE: Structured Chain Reasoning for Interactive Behaviour Explanations using Tool Calling(https://arxiv.org/abs/2510.26322)
• Keywords: language model, gpt
• Abstract: Language models can be used to provide interactive, personalized student feedback in educational settings. However, real-world deployment faces three key challenges: privacy concerns, limited computational resources, and the need for pedagogically valid responses. These constraints require small, open-source models that can run locally and reliably ground their outputs in correct information. We introduce SCRIBE, a framework for multi-hop, tool-augmented reasoning designed to generate valid responses to student questions about feedback reports. SCRIBE combines domain-specific tools with a self-reflective inference pipeline that supports iterative reasoning, tool use, and error recovery. We distil these capabilities into 3B and 8B models via two-stage LoRA fine-tuning on synthetic GPT-4o-generated data. Evaluation with a human-aligned GPT-Judge and a user study with 108 students shows that 8B-SCRIBE models achieve comparable or superior quality to much larger models in key dimensions such as relevance and actionability, while being perceived on par with GPT-4o and Llama-3.3 70B by students. These findings demonstrate the viability of SCRIBE for low-resource, privacy-sensitive educational applications.
• 摘要：语言模型可用于在教育环境中提供交互式、个性化的学生反馈。然而，现实世界的部署面临三个关键挑战：隐私问题、有限的计算资源以及对教学有效响应的需求。这些限制需要小型开源模型，这些模型可以在本地运行并可靠地根据正确的信息输出。我们引入了 SCRIBE，这是一个多跳、工具增强推理的框架，旨在针对学生有关反馈报告的问题生成有效的答案。 SCRIBE 将特定领域的工具与支持迭代推理、工具使用和错误恢复的自反射推理管道相结合。我们通过对合成 GPT-4o 生成的数据进行两阶段 LoRA 微调，将这些功能提炼为 3B 和 8B 模型。通过人性化的 GPT-Judge 评估和 108 名学生的用户研究表明，8B-SCRIBE 模型在相关性和可操作性等关键维度上达到了与更大的模型相当或更高的质量，同时被学生认为与 GPT-4o 和 Llama-3.3 70B 相当。这些发现证明了 SCRIBE 对于资源匮乏、隐私敏感的教育应用程序的可行性。

Title: From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learning

• Authors: Nishit Neema, Srinjoy Mukherjee, Sapan Shah, Gokul Ramakrishnan, Ganesh Venkatesh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26336
• Pdf URL: https://arxiv.org/pdf/2510.26336
• Copy Paste: [[2510.26336]] From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learning(https://arxiv.org/abs/2510.26336)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel at general tasks but underperform in specialized domains like economics and psychology, which require deep, principled understanding. To address this, we introduce ACER (Automated Curriculum-Enhanced Regimen) that transforms generalist models into domain experts without sacrificing their broad capabilities. ACER first synthesizes a comprehensive, textbook-style curriculum by generating a table of contents for a subject and then creating question-answer (QA) pairs guided by Bloom's taxonomy. This ensures systematic topic coverage and progressively increasing difficulty. The resulting synthetic corpus is used for continual pretraining with an interleaved curriculum schedule, aligning learning across both content and cognitive dimensions. Experiments with Llama 3.2 (1B and 3B) show significant gains in specialized MMLU subsets. In challenging domains like microeconomics, where baselines struggle, ACER boosts accuracy by 5 percentage points. Across all target domains, we observe a consistent macro-average improvement of 3 percentage points. Notably, ACER not only prevents catastrophic forgetting but also facilitates positive cross-domain knowledge transfer, improving performance on non-target domains by 0.7 points. Beyond MMLU, ACER enhances performance on knowledge-intensive benchmarks like ARC and GPQA by over 2 absolute points, while maintaining stable performance on general reasoning tasks. Our results demonstrate that ACER offers a scalable and effective recipe for closing critical domain gaps in LLMs.
• 摘要：大型语言模型（LLM）擅长一般任务，但在经济学和心理学等需要深入、原则性理解的专业领域表现不佳。为了解决这个问题，我们引入了 ACER（自动化课程增强方案），它将通才模型转变为领域专家，而不会牺牲其广泛的能力。 ACER 首先生成一个主题的目录，然后根据 Bloom 分类法创建问答 (QA) 对，从而综合综合全面的教科书式课程。这确保了系统的主题覆盖和逐渐增加的难度。由此产生的合成语料库用于通过交错的课程安排进行持续的预训练，从而使内容和认知维度的学习保持一致。 Llama 3.2（1B 和 3B）的实验显示在专门的 MMLU 子集中有显着的增益。在微观经济学等具有挑战性的领域，基线存在困难，ACER 将准确性提高了 5 个百分点。在所有目标领域，我们观察到宏观平均改善一致为 3 个百分点。值得注意的是，ACER 不仅可以防止灾难性遗忘，还可以促进积极的跨领域知识转移，将非目标领域的性能提高 0.7 个百分点。除了 MMLU 之外，ACER 将 ARC 和 GPQA 等知识密集型基准测试的性能提高了 2 个以上绝对点，同时在一般推理任务上保持稳定的性能。我们的结果表明，ACER 为缩小法学硕士的关键领域差距提供了可扩展且有效的方法。

Title: MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data

• Authors: Mykhailo Poliakov, Nadiya Shvai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26345
• Pdf URL: https://arxiv.org/pdf/2510.26345
• Copy Paste: [[2510.26345]] MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data(https://arxiv.org/abs/2510.26345)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Health-related misinformation is very prevalent and potentially harmful. It is difficult to identify, especially when claims distort or misinterpret scientific findings. We investigate the impact of synthetic data generation and lightweight fine-tuning techniques on the ability of large language models (LLMs) to recognize fallacious arguments using the MISSCI dataset and framework. In this work, we propose MisSynth, a pipeline that applies retrieval-augmented generation (RAG) to produce synthetic fallacy samples, which are then used to fine-tune an LLM model. Our results show substantial accuracy gains with fine-tuned models compared to vanilla baselines. For instance, the LLaMA 3.1 8B fine-tuned model achieved an over 35% F1-score absolute improvement on the MISSCI test split over its vanilla baseline. We demonstrate that introducing synthetic fallacy data to augment limited annotated resources can significantly enhance zero-shot LLM classification performance on real-world scientific misinformation tasks, even with limited computational resources. The code and synthetic dataset are available on this https URL.
• 摘要：与健康相关的错误信息非常普遍且具有潜在危害。这是很难识别的，尤其是当声明歪曲或误解了科学发现时。我们研究了合成数据生成和轻量级微调技术对大型语言模型 (LLM) 使用 MISSCI 数据集和框架识别错误论点的能力的影响。在这项工作中，我们提出了 MisSynth，这是一个应用检索增强生成（RAG）来生成合成谬误样本的管道，然后将其用于微调 LLM 模型。我们的结果显示，与普通基线相比，微调模型的准确度大幅提高。例如，LLaMA 3.1 8B 微调模型在 MISSCI 测试分割上比其普通基线实现了超过 35% 的 F1 分数绝对改进。我们证明，即使计算资源有限，引入合成谬误数据来增强有限的注释资源也可以显着提高现实世界科学错误信息任务的零样本 LLM 分类性能。代码和合成数据集可在此 https URL 上获取。

Title: The Geometry of Dialogue: Graphing Language Models to Reveal Synergistic Teams for Multi-Agent Collaboration

• Authors: Kotaro Furuya, Yuichi Kitagawa
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2510.26352
• Pdf URL: https://arxiv.org/pdf/2510.26352
• Copy Paste: [[2510.26352]] The Geometry of Dialogue: Graphing Language Models to Reveal Synergistic Teams for Multi-Agent Collaboration(https://arxiv.org/abs/2510.26352)
• Keywords: language model, llm, agent
• Abstract: While a multi-agent approach based on large language models (LLMs) represents a promising strategy to surpass the capabilities of single models, its success is critically dependent on synergistic team composition. However, forming optimal teams is a significant challenge, as the inherent opacity of most models obscures the internal characteristics necessary for effective collaboration. In this paper, we propose an interaction-centric framework for automatic team composition that does not require any prior knowledge including their internal architectures, training data, or task performances. Our method constructs a "language model graph" that maps relationships between models from the semantic coherence of pairwise conversations, and then applies community detection to identify synergistic model clusters. Our experiments with diverse LLMs demonstrate that the proposed method discovers functionally coherent groups that reflect their latent specializations. Priming conversations with specific topics identified synergistic teams which outperform random baselines on downstream benchmarks and achieve comparable accuracy to that of manually-curated teams based on known model specializations. Our findings provide a new basis for the automated design of collaborative multi-agent LLM teams.
• 摘要：虽然基于大语言模型 (LLM) 的多智能体方法代表了一种超越单一模型能力的有前景的策略，但其成功关键取决于协同团队的组成。然而，组建最佳团队是一项重大挑战，因为大多数模型固有的不透明性掩盖了有效协作所需的内部特征。在本文中，我们提出了一种以交互为中心的自动团队组成框架，不需要任何先验知识，包括内部架构、训练数据或任务性能。我们的方法构建了一个“语言模型图”，该图根据成对对话的语义一致性映射模型之间的关系，然后应用社区检测来识别协同模型集群。我们对不同法学硕士的实验表明，所提出的方法发现了反映其潜在专业的功能一致的群体。与特定主题的启动对话确定了协同团队，这些团队在下游基准上优于随机基线，并达到与基于已知模型专业化的手动策划团队相当的准确性。我们的研究结果为协作多代理法学硕士团队的自动化设计提供了新的基础。

Title: On the Role of Context for Discourse Relation Classification in Scientific Writing

• Authors: Stephen Wan, Wei Liu, Michael Strube
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26354
• Pdf URL: https://arxiv.org/pdf/2510.26354
• Copy Paste: [[2510.26354]] On the Role of Context for Discourse Relation Classification in Scientific Writing(https://arxiv.org/abs/2510.26354)
• Keywords: language model, llm
• Abstract: With the increasing use of generative Artificial Intelligence (AI) methods to support science workflows, we are interested in the use of discourse-level information to find supporting evidence for AI generated scientific claims. A first step towards this objective is to examine the task of inferring discourse structure in scientific writing. In this work, we present a preliminary investigation of pretrained language model (PLM) and Large Language Model (LLM) approaches for Discourse Relation Classification (DRC), focusing on scientific publications, an under-studied genre for this task. We examine how context can help with the DRC task, with our experiments showing that context, as defined by discourse structure, is generally helpful. We also present an analysis of which scientific discourse relation types might benefit most from context.
• 摘要：随着越来越多地使用生成人工智能（AI）方法来支持科学工作流程，我们对使用话语级信息来寻找人工智能生成的科学主张的支持证据感兴趣。实现这一目标的第一步是检查科学写作中推断话语结构的任务。在这项工作中，我们对用于话语关系分类（DRC）的预训练语言模型（PLM）和大型语言模型（LLM）方法进行了初步研究，重点关注科学出版物，这是该任务的一个尚未得到充分研究的类型。我们研究了上下文如何帮助完成 DRC 任务，我们的实验表明，由话语结构定义的上下文通常是有帮助的。我们还分析了哪些科学话语关系类型可能从上下文中受益最多。

Title: OmniEduBench: A Comprehensive Chinese Benchmark for Evaluating Large Language Models in Education

• Authors: Min Zhang, Hao Chen, Hao Chen, Wenqi Zhang, Didi Zhu, Xin Lin, Bo Jiang, Aimin Zhou, Fei Wu, Kun Kuang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26422
• Pdf URL: https://arxiv.org/pdf/2510.26422
• Copy Paste: [[2510.26422]] OmniEduBench: A Comprehensive Chinese Benchmark for Evaluating Large Language Models in Education(https://arxiv.org/abs/2510.26422)
• Keywords: language model, llm
• Abstract: With the rapid development of large language models (LLMs), various LLM-based works have been widely applied in educational fields. However, most existing LLMs and their benchmarks focus primarily on the knowledge dimension, largely neglecting the evaluation of cultivation capabilities that are essential for real-world educational scenarios. Additionally, current benchmarks are often limited to a single subject or question type, lacking sufficient diversity. This issue is particularly prominent within the Chinese context. To address this gap, we introduce OmniEduBench, a comprehensive Chinese educational benchmark. OmniEduBench consists of 24.602K high-quality question-answer pairs. The data is meticulously divided into two core dimensions: the knowledge dimension and the cultivation dimension, which contain 18.121K and 6.481K entries, respectively. Each dimension is further subdivided into 6 fine-grained categories, covering a total of 61 different subjects (41 in the knowledge and 20 in the cultivation). Furthermore, the dataset features a rich variety of question formats, including 11 common exam question types, providing a solid foundation for comprehensively evaluating LLMs' capabilities in education. Extensive experiments on 11 mainstream open-source and closed-source LLMs reveal a clear performance gap. In the knowledge dimension, only Gemini-2.5 Pro surpassed 60\% accuracy, while in the cultivation dimension, the best-performing model, QWQ, still trailed human intelligence by nearly 30\%. These results highlight the substantial room for improvement and underscore the challenges of applying LLMs in education.
• 摘要：随着大语言模型（LLM）的快速发展，各种基于LLM的作品在教育领域得到了广泛的应用。然而，大多数现有的法学硕士及其基准主要关注知识维度，很大程度上忽视了对现实世界教育场景至关重要的培养能力的评估。此外，当前的基准通常仅限于单一主题或问题类型，缺乏足够的多样性。这个问题在中国背景下尤为突出。为了弥补这一差距，我们推出了 OmniEduBench，一个综合性的中国教育基准。 OmniEduBench 由 24.602K 个高质量问答对组成。数据被精心划分为两个核心维度：知识维度和修养维度，分别包含18.121K和6.481K条目。每个维度又细分为6个细粒度类别，共涵盖61个不同科目（知识41个，修养20个）。此外，数据集题型丰富，包括11种常见考试题型，为全面评估LLM的教育能力提供了坚实的基础。对 11 个主流开源和闭源 LLM 进行的大量实验揭示了明显的性能差距。在知识维度上，只有Gemini-2.5 Pro的准确率超过了60%，而在修炼维度上，表现最好的模型QWQ仍然落后人类智力近30%。这些结果凸显了巨大的改进空间，并强调了在教育中应用法学硕士所面临的挑战。

Title: 1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models

• Authors: Zeliang Zong, Kai Zhang, Zheyang Li, Wenming Tan, Ye Ren, Yiyan Zhai, Jilin Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26446
• Pdf URL: https://arxiv.org/pdf/2510.26446
• Copy Paste: [[2510.26446]] 1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models(https://arxiv.org/abs/2510.26446)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in language comprehension and generation; however, their widespread adoption is constrained by substantial bandwidth and computational demands. While pruning and low-rank approximation have each demonstrated promising performance individually, their synergy for LLMs remains underexplored. We introduce \underline{S}ynergistic \underline{S}parse and \underline{L}ow-Rank \underline{C}ompression (SSLC) methods for LLMs, which leverages the strengths of both techniques: low-rank approximation compresses the model by retaining its essential structure with minimal information loss, whereas sparse optimization eliminates non-essential weights, preserving those crucial for generalization. Based on theoretical analysis, we first formulate the low-rank approximation and sparse optimization as a unified problem and solve it by iterative optimization algorithm. Experiments on LLaMA and Qwen2.5 models (7B-70B) show that SSLC, without any additional training steps, consistently surpasses standalone methods, achieving state-of-the-arts results. Notably, SSLC compresses Qwen2.5 by 50\% with no performance drop and achieves at least 1.63$\times$ speedup, offering a practical solution for efficient LLM deployment.
• 摘要：大型语言模型（LLM）在语言理解和生成方面表现出了卓越的熟练程度；然而，它们的广泛采用受到大量带宽和计算需求的限制。虽然剪枝和低秩近似各自都表现出了有希望的性能，但它们对法学硕士的协同作用仍未得到充分探索。我们为 LLM 引入了 \underline{S}ynergistic \underline{S}parse 和 \underline{L}ow-Rank \underline{C}compression (SSLC) 方法，该方法利用了这两种技术的优势：低秩近似通过以最小的信息损失保留其基本结构来压缩模型，而稀疏优化消除了非必要的权重，保留了对泛化至关重要的权重。基于理论分析，我们首先将低秩近似和稀疏优化作为一个统一的问题，并通过迭代优化算法进行求解。 LLaMA 和 Qwen2.5 模型 (7B-70B) 上的实验表明，SSLC 在没有任何额外训练步骤的情况下，始终超越独立方法，取得了最先进的结果。值得注意的是，SSLC 将 Qwen2.5 压缩了 50%，性能没有下降，并实现了至少 1.63$\times$ 的加速，为高效的 LLM 部署提供了实用的解决方案。

Title: Bayesian Network Fusion of Large Language Models for Sentiment Analysis

• Authors: Rasoul Amirzadeh, Dhananjay Thiruvady, Fatemeh Shiri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26484
• Pdf URL: https://arxiv.org/pdf/2510.26484
• Copy Paste: [[2510.26484]] Bayesian Network Fusion of Large Language Models for Sentiment Analysis(https://arxiv.org/abs/2510.26484)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) continue to advance, with an increasing number of domain-specific variants tailored for specialised tasks. However, these models often lack transparency and explainability, can be costly to fine-tune, require substantial prompt engineering, yield inconsistent results across domains, and impose significant adverse environmental impact due to their high computational demands. To address these challenges, we propose the Bayesian network LLM fusion (BNLF) framework, which integrates predictions from three LLMs, including FinBERT, RoBERTa, and BERTweet, through a probabilistic mechanism for sentiment analysis. BNLF performs late fusion by modelling the sentiment predictions from multiple LLMs as probabilistic nodes within a Bayesian network. Evaluated across three human-annotated financial corpora with distinct linguistic and contextual characteristics, BNLF demonstrates consistent gains of about six percent in accuracy over the baseline LLMs, underscoring its robustness to dataset variability and the effectiveness of probabilistic fusion for interpretable sentiment classification.
• 摘要：大型语言模型 (LLM) 不断发展，针对专门任务定制的特定领域变体数量不断增加。然而，这些模型通常缺乏透明度和可解释性，微调成本高昂，需要大量的即时工程，跨领域产生不一致的结果，并且由于其高计算需求而造成严重的不利环境影响。为了应对这些挑战，我们提出了贝叶斯网络 LLM 融合（BNLF）框架，该框架通过情感分析的概率机制集成了 FinBERT、RoBERTa 和 BERTweet 等三个 LLM 的预测。 BNLF 通过将多个 LLM 的情绪预测建模为贝叶斯网络中的概率节点来执行后期融合。通过对三个具有不同语言和语境特征的人工注释金融语料库进行评估，BNLF 的准确性比基线法学硕士持续提高约 6%，强调了其对数据集可变性的鲁棒性以及可解释情感分类的概率融合的有效性。

Title: A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool

• Authors: Adam E. Flanders, Yifan Peng, Luciano Prevedello, Robyn Ball, Errol Colak, Prahlad Menon, George Shih, Hui-Ming Lin, Paras Lakhani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26498
• Pdf URL: https://arxiv.org/pdf/2510.26498
• Copy Paste: [[2510.26498]] A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool(https://arxiv.org/abs/2510.26498)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Purpose: The purpose of this study was to determine if an ensemble of multiple LLM agents could be used collectively to provide a more reliable assessment of a pixel-based AI triage tool than a single LLM. Methods: 29,766 non-contrast CT head exams from fourteen hospitals were processed by a commercial intracranial hemorrhage (ICH) AI detection tool. Radiology reports were analyzed by an ensemble of eight open-source LLM models and a HIPAA compliant internal version of GPT-4o using a single multi-shot prompt that assessed for presence of ICH. 1,726 examples were manually reviewed. Performance characteristics of the eight open-source models and consensus were compared to GPT-4o. Three ideal consensus LLM ensembles were tested for rating the performance of the triage tool. Results: The cohort consisted of 29,766 head CTs exam-report pairs. The highest AUC performance was achieved with llama3.3:70b and GPT-4o (AUC= 0.78). The average precision was highest for Llama3.3:70b and GPT-4o (AP=0.75 & 0.76). Llama3.3:70b had the highest F1 score (0.81) and recall (0.85), greater precision (0.78), specificity (0.72), and MCC (0.57). Using MCC (95% CI) the ideal combination of LLMs were: Full-9 Ensemble 0.571 (0.552-0.591), Top-3 Ensemble 0.558 (0.537-0.579), Consensus 0.556 (0.539-0.574), and GPT4o 0.522 (0.500-0.543). No statistically significant differences were observed between Top-3, Full-9, and Consensus (p > 0.05). Conclusion: An ensemble of medium to large sized open-source LLMs provides a more consistent and reliable method to derive a ground truth retrospective evaluation of a clinical AI triage tool over a single LLM alone.
• 摘要：目的：本研究的目的是确定多个 LLM 代理的集合是否可以共同使用，以便为基于像素的 AI 分类工具提供比单个 LLM 更可靠的评估。方法：使用商用颅内出血 (ICH) AI 检测工具处理来自 14 家医院的 29,766 例非增强 CT 头部检查。放射学报告由八个开源 LLM 模型和符合 HIPAA 的 GPT-4o 内部版本组成，使用单个多镜头提示来评估 ICH 的存在。人工审核了 1,726 个示例。将八个开源模型的性能特征和共识与 GPT-4o 进行了比较。对三个理想的共识 LLM 整体进行了测试，以评估分类工具的性能。结果：该队列由 29,766 组头部 CT 检查-报告对组成。 llama3.3:70b 和 GPT-4o 实现了最高 AUC 性能（AUC= 0.78）。 Llama3.3:70b 和 GPT-4o 的平均精度最高（AP=0.75 和 0.76）。 Llama3.3:70b 具有最高的 F1 分数 (0.81) 和召回率 (0.85)、更高的精确度 (0.78)、特异性 (0.72) 和 MCC (0.57)。使用 MCC (95% CI)，LLM 的理想组合为：Full-9 Ensemble 0.571 (0.552-0.591)、Top-3 Ensemble 0.558 (0.537-0.579)、Consensus 0.556 (0.539-0.574) 和 GPT4o 0.522 （0.500-0.543）。 Top-3、Full-9 和 Consensus 之间没有观察到统计学上的显着差异 (p > 0.05)。结论：中型到大型开源法学硕士的集合提供了一种更加一致和可靠的方法，可以比单独的法学硕士对临床 AI 分类工具进行地面实况回顾性评估。

Title: Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs

• Authors: Dipak Meher, Carlotta Domeniconi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26512
• Pdf URL: https://arxiv.org/pdf/2510.26512
• Copy Paste: [[2510.26512]] Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs(https://arxiv.org/abs/2510.26512)
• Keywords: llm, prompt
• Abstract: Human smuggling networks are increasingly adaptive and difficult to analyze. Legal case documents offer critical insights but are often unstructured, lexically dense, and filled with ambiguous or shifting references, which pose significant challenges for automated knowledge graph (KG) construction. While recent LLM-based approaches improve over static templates, they still generate noisy, fragmented graphs with duplicate nodes due to the absence of guided extraction and coreference resolution. The recently proposed CORE-KG framework addresses these limitations by integrating a type-aware coreference module and domain-guided structured prompts, significantly reducing node duplication and legal noise. In this work, we present a systematic ablation study of CORE-KG to quantify the individual contributions of its two key components. Our results show that removing coreference resolution results in a 28.32% increase in node duplication and a 4.32% increase in noisy nodes, while removing structured prompts leads to a 4.34% increase in node duplication and a 73.33% increase in noisy nodes. These findings offer empirical insights for designing robust LLM-based pipelines for extracting structured representations from complex legal texts.
• 摘要：人口走私网络的适应性越来越强且难以分析。法律案例文档提供了重要的见解，但通常是非结构化的、词汇密集的，并且充满了不明确或变化的引用，这给自动化知识图（KG）构建带来了重大挑战。虽然最近基于 LLM 的方法比静态模板有所改进，但由于缺乏引导提取和共指解析，它们仍然会生成带有重复节点的嘈杂、碎片化的图。最近提出的 CORE-KG 框架通过集成类型感知共指模块和领域引导的结构化提示来解决这些限制，从而显着减少节点重复和合法噪音。在这项工作中，我们提出了 CORE-KG 的系统消融研究，以量化其两个关键组成部分的单独贡献。我们的结果表明，删除共指解析会导致节点重复增加 28.32%，噪声节点增加 4.32%，而删除结构化提示会导致节点重复增加 4.34%，噪声节点增加 73.33%。这些发现为设计强大的基于法学硕士的管道以从复杂的法律文本中提取结构化表示提供了经验见解。

Title: Hebrew Diacritics Restoration using Visual Representation

• Authors: Yair Elboher, Yuval Pinter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26521
• Pdf URL: https://arxiv.org/pdf/2510.26521
• Copy Paste: [[2510.26521]] Hebrew Diacritics Restoration using Visual Representation(https://arxiv.org/abs/2510.26521)
• Keywords: language model
• Abstract: Diacritics restoration in Hebrew is a fundamental task for ensuring accurate word pronunciation and disambiguating textual meaning. Despite the language's high degree of ambiguity when unvocalized, recent machine learning approaches have significantly advanced performance on this task. In this work, we present DIVRIT, a novel system for Hebrew diacritization that frames the task as a zero-shot classification problem. Our approach operates at the word level, selecting the most appropriate diacritization pattern for each undiacritized word from a dynamically generated candidate set, conditioned on the surrounding textual context. A key innovation of DIVRIT is its use of a Hebrew Visual Language Model, which processes undiacritized text as an image, allowing diacritic information to be embedded directly within the input's vector representation. Through a comprehensive evaluation across various configurations, we demonstrate that the system effectively performs diacritization without relying on complex, explicit linguistic analysis. Notably, in an ``oracle'' setting where the correct diacritized form is guaranteed to be among the provided candidates, DIVRIT achieves a high level of accuracy. Furthermore, strategic architectural enhancements and optimized training methodologies yield significant improvements in the system's overall generalization capabilities. These findings highlight the promising potential of visual representations for accurate and automated Hebrew diacritization.
• 摘要：希伯来语变音符号恢复是确保单词发音准确和消除文本含义歧义的一项基本任务。尽管该语言在无声时具有高度的歧义性，但最近的机器学习方法在这项任务上的性能显着提高。在这项工作中，我们提出了 DIVRIT，这是一种用于希伯来语变音的新颖系统，它将任务构建为零样本分类问题。我们的方法在单词级别运行，根据周围的文本上下文，从动态生成的候选集中为每个未区分的单词选择最合适的变音模式。 DIVRIT 的一项关键创新是它使用了希伯来语视觉语言模型，该模型将未经区分的文本处理为图像，从而允许将变音符号信息直接嵌入到输入的向量表示中。通过对各种配置的综合评估，我们证明系统可以有效地执行变音符号，而无需依赖复杂、明确的语言分析。值得注意的是，在保证正确的变音形式存在于所提供的候选中的“oracle”环境中，DIVRIT 实现了很高的准确性。此外，战略架构的增强和优化的训练方法显着提高了系统的整体泛化能力。这些发现凸显了视觉表示在准确和自动化希伯来语变音符号方面的巨大潜力。

Title: The Structure of Relation Decoding Linear Operators in Large Language Models

• Authors: Miranda Anna Christ, Adrián Csiszárik, Gergely Becsó, Dániel Varga
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26543
• Pdf URL: https://arxiv.org/pdf/2510.26543
• Copy Paste: [[2510.26543]] The Structure of Relation Decoding Linear Operators in Large Language Models(https://arxiv.org/abs/2510.26543)
• Keywords: language model
• Abstract: This paper investigates the structure of linear operators introduced in Hernandez et al. [2023] that decode specific relational facts in transformer language models. We extend their single-relation findings to a collection of relations and systematically chart their organization. We show that such collections of relation decoders can be highly compressed by simple order-3 tensor networks without significant loss in decoding accuracy. To explain this surprising redundancy, we develop a cross-evaluation protocol, in which we apply each linear decoder operator to the subjects of every other relation. Our results reveal that these linear maps do not encode distinct relations, but extract recurring, coarse-grained semantic properties (e.g., country of capital city and country of food are both in the country-of-X property). This property-centric structure clarifies both the operators' compressibility and highlights why they generalize only to new relations that are semantically close. Our findings thus interpret linear relational decoding in transformer language models as primarily property-based, rather than relation-specific.
• 摘要：本文研究了 Hernandez 等人引入的线性算子的结构。 [2023] 解码 Transformer 语言模型中的特定关系事实。我们将他们的单一关系发现扩展到关系集合，并系统地绘制他们的组织图表。我们证明了这种关系解码器的集合可以通过简单的三阶张量网络进行高度压缩，而不会显着损失解码精度。为了解释这种令人惊讶的冗余，我们开发了一个交叉评估协议，其中我们将每个线性解码器运算符应用于每个其他关系的主题。我们的结果表明，这些线性映射并不编码不同的关系，而是提取重复出现的、粗粒度的语义属性（例如，首都国家和食品国家都在 X 国家属性中）。这种以属性为中心的结构阐明了运算符的可压缩性，并强调了为什么它们仅推广到语义上接近的新关系。因此，我们的研究结果将 Transformer 语言模型中的线性关系解码解释为主要基于属性，而不是特定于关系。

Title: InfoFlow: Reinforcing Search Agent Via Reward Density Optimization

• Authors: Kun Luo, Hongjin Qian, Zheng Liu, Ziyi Xia, Shitao Xiao, Siqi Bao, Jun Zhao, Kang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26575
• Pdf URL: https://arxiv.org/pdf/2510.26575
• Copy Paste: [[2510.26575]] InfoFlow: Reinforcing Search Agent Via Reward Density Optimization(https://arxiv.org/abs/2510.26575)
• Keywords: llm, agent
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a promising approach for enhancing agentic deep search. However, its application is often hindered by low \textbf{Reward Density} in deep search scenarios, where agents expend significant exploratory costs for infrequent and often null final rewards. In this paper, we formalize this challenge as the \textbf{Reward Density Optimization} problem, which aims to improve the reward obtained per unit of exploration cost. This paper introduce \textbf{InfoFlow}, a systematic framework that tackles this problem from three aspects. 1) \textbf{Subproblem decomposition}: breaking down long-range tasks to assign process rewards, thereby providing denser learning signals. 2) \textbf{Failure-guided hints}: injecting corrective guidance into stalled trajectories to increase the probability of successful outcomes. 3) \textbf{Dual-agent refinement}: employing a dual-agent architecture to offload the cognitive burden of deep exploration. A refiner agent synthesizes the search history, which effectively compresses the researcher's perceived trajectory, thereby reducing exploration cost and increasing the overall reward density. We evaluate InfoFlow on multiple agentic search benchmarks, where it significantly outperforms strong baselines, enabling lightweight LLMs to achieve performance comparable to advanced proprietary LLMs.
• 摘要：具有可验证奖励的强化学习（RLVR）是增强代理深度搜索的一种有前途的方法。然而，它的应用常常受到深度搜索场景中低奖励密度的阻碍，在这种场景中，代理花费大量的探索成本来获得不频繁且通常无效的最终奖励。在本文中，我们将这一挑战形式化为奖励密度优化问题，其目的是提高每单位探索成本获得的奖励。本文介绍了\textbf{InfoFlow}，一个从三个方面解决这个问题的系统框架。 1）\textbf{子问题分解}：分解远程任务以分配过程奖励，从而提供更密集的学习信号。 2) \textbf{失败引导提示}：向停滞的轨迹注入纠正指导，以增加成功结果的可能性。 3）\textbf{双代理细化}：采用双代理架构来减轻深度探索的认知负担。精炼代理综合搜索历史，有效压缩研究人员的感知轨迹，从而降低探索成本并提高整体奖励密度。我们在多个代理搜索基准上评估 InfoFlow，它的性能显着优于强大的基准，使轻量级法学硕士能够实现与高级专有法学硕士相当的性能。

Title: Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models

• Authors: Yinrong Hong, Zhiquan Tan, Kai Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26577
• Pdf URL: https://arxiv.org/pdf/2510.26577
• Copy Paste: [[2510.26577]] Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models(https://arxiv.org/abs/2510.26577)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) face significant inference latency challenges stemming from their autoregressive design and large size. To address this, speculative decoding emerges as a solution, enabling the simultaneous generation and validation of multiple tokens. While recent approaches like EAGLE-2 and EAGLE-3 improve speculative decoding using dynamic tree structures, they often neglect the impact of crucial system variables such as GPU devices and batch sizes. Therefore, we introduce a new dynamic tree decoding approach called CAST that takes into account inference costs, including factors such as GPU configurations and batch sizes, to dynamically refine the tree structure. Through comprehensive experimentation across six diverse tasks and utilizing six distinct LLMs, our methodology demonstrates remarkable results, achieving speeds up to 5.2 times faster than conventional decoding methods. Moreover, it generally outperforms existing state-of-the-art techniques from 5% to 20%.
• 摘要：大型语言模型 (LLM) 面临着源自其自回归设计和大尺寸的重大推理延迟挑战。为了解决这个问题，推测性解码作为一种解决方案出现，能够同时生成和验证多个令牌。虽然 EAGLE-2 和 EAGLE-3 等最新方法使用动态树结构改进了推测解码，但它们经常忽略关键系统变量（例如 GPU 设备和批量大小）的影响。因此，我们引入了一种称为 CAST 的新动态树解码方法，该方法考虑了推理成本，包括 GPU 配置和批量大小等因素，以动态细化树结构。通过对六种不同任务的综合实验并利用六种不同的法学硕士，我们的方法展示了显着的结果，速度比传统解码方法快 5.2 倍。此外，它的性能通常比现有最先进技术高出 5% 到 20%。

Title: SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding

• Authors: Yiqiao Jin, Rachneet Kaur, Zhen Zeng, Sumitra Ganesh, Srijan Kumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26615
• Pdf URL: https://arxiv.org/pdf/2510.26615
• Copy Paste: [[2510.26615]] SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding(https://arxiv.org/abs/2510.26615)
• Keywords: language model, llm, agent
• Abstract: Multi-page visual documents such as manuals, brochures, presentations, and posters convey key information through layout, colors, icons, and cross-slide references. While large language models (LLMs) offer opportunities in document understanding, current systems struggle with complex, multi-page visual documents, particularly in fine-grained reasoning over elements and pages. We introduce SlideAgent, a versatile agentic framework for understanding multi-modal, multi-page, and multi-layout documents, especially slide decks. SlideAgent employs specialized agents and decomposes reasoning into three specialized levels-global, page, and element-to construct a structured, query-agnostic representation that captures both overarching themes and detailed visual or textual cues. During inference, SlideAgent selectively activates specialized agents for multi-level reasoning and integrates their outputs into coherent, context-aware answers. Extensive experiments show that SlideAgent achieves significant improvement over both proprietary (+7.9 overall) and open-source models (+9.8 overall).
• 摘要：多页视觉文档（例如手册、小册子、演示文稿和海报）通过布局、颜色、图标和交叉幻灯片引用传达关键信息。虽然大型语言模型 (LLM) 提供了文档理解的机会，但当前的系统难以处理复杂的多页可视文档，特别是在元素和页面的细粒度推理方面。我们介绍 SlideAgent，这是一个多功能代理框架，用于理解多模式、多页面和多布局文档，尤其是幻灯片。 SlideAgent 采用专门的代理，并将推理分解为三个专门的级别（全局、页面和元素），以构建结构化的、与查询无关的表示，该表示捕获总体主题和详细的视觉或文本提示。在推理过程中，SlideAgent 有选择地激活专门的代理进行多级推理，并将其输出集成为连贯的、上下文感知的答案。大量实验表明，SlideAgent 比专有模型（总体+7.9）和开源模型（总体+9.8）都有显着改进。

Title: Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model

• Authors: Biao Zhang, Yong Cheng, Siamak Shakeri, Xinyi Wang, Min Ma, Orhan Firat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26622
• Pdf URL: https://arxiv.org/pdf/2510.26622
• Copy Paste: [[2510.26622]] Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model(https://arxiv.org/abs/2510.26622)
• Keywords: language model, llm
• Abstract: Recent large language model (LLM) research has undergone an architectural shift from encoder-decoder modeling to nowadays the dominant decoder-only modeling. This rapid transition, however, comes without a rigorous comparative analysis especially \textit{from the scaling perspective}, raising concerns that the potential of encoder-decoder models may have been overlooked. To fill this gap, we revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM). We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales, ranging from $\sim$150M to $\sim$8B. Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance. While DecLLM is overall more compute-optimal during pretraining, RedLLM demonstrates comparable scaling and context length extrapolation capabilities. After instruction tuning, RedLLM achieves comparable and even better results on various downstream tasks while enjoying substantially better inference efficiency. We hope our findings could inspire more efforts on re-examining RedLLM, unlocking its potential for developing powerful and efficient LLMs.
• 摘要：最近的大语言模型（LLM）研究经历了从编码器-解码器建模到当今占主导地位的仅解码器建模的架构转变。然而，这种快速转变没有经过严格的比较分析，尤其是 \textit{从缩放角度}，这引发了人们对编码器-解码器模型的潜力可能被忽视的担忧。为了填补这一空白，我们重新审视编码器-解码器 LLM (RedLLM)，并使用仅解码器 LLM (DecLLM) 的最新配方对其进行了增强。我们对使用前缀语言模型（LM）预训练的 RedLLM 和使用因果 LM 预训练的 DecLLM 在不同模型规模（从 $\sim$150M 到 $\sim$8B）进行了全面比较。使用 RedPajama V1（1.6T 令牌）进行预训练，并使用 FLAN 进行指令调整，我们的实验表明 RedLLM 具有引人注目的扩展特性和令人惊讶的强大性能。虽然 DecLLM 在预训练期间总体上计算更优化，但 RedLLM 展示了可比的缩放和上下文长度外推功能。经过指令调整后，RedLLM 在各种下游任务上取得了可比甚至更好的结果，同时享有显着提高的推理效率。我们希望我们的发现能够激发更多人重新审视 RedLLM，释放其开发强大而高效的 LLM 的潜力。

Title: Evontree: Ontology Rule-Guided Self-Evolution of Large Language Models

• Authors: Mingchen Tu, Zhiqiang Liu, Juan Li, Liangyurui Liu, Junjie Wang, Lei Liang, Wen Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26683
• Pdf URL: https://arxiv.org/pdf/2510.26683
• Copy Paste: [[2510.26683]] Evontree: Ontology Rule-Guided Self-Evolution of Large Language Models(https://arxiv.org/abs/2510.26683)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated exceptional capabilities across multiple domains by leveraging massive pre-training and curated fine-tuning data. However, in data-sensitive fields such as healthcare, the lack of high-quality, domain-specific training corpus hinders LLMs' adaptation for specialized applications. Meanwhile, domain experts have distilled domain wisdom into ontology rules, which formalize relationships among concepts and ensure the integrity of knowledge management repositories. Viewing LLMs as implicit repositories of human knowledge, we propose Evontree, a novel framework that leverages a small set of high-quality ontology rules to systematically extract, validate, and enhance domain knowledge within LLMs, without requiring extensive external datasets. Specifically, Evontree extracts domain ontology from raw models, detects inconsistencies using two core ontology rules, and reinforces the refined knowledge via self-distilled fine-tuning. Extensive experiments on medical QA benchmarks with Llama3-8B-Instruct and Med42-v2 demonstrate consistent outperformance over both unmodified models and leading supervised baselines, achieving up to a 3.7% improvement in accuracy. These results confirm the effectiveness, efficiency, and robustness of our approach for low-resource domain adaptation of LLMs.
• 摘要：大型语言模型 (LLM) 通过利用大量预训练和精心策划的微调数据，展示了跨多个领域的卓越功能。然而，在医疗保健等数据敏感领域，缺乏高质量、特定领域的训练语料库阻碍了法学硕士对专业应用的适应。同时，领域专家将领域智慧提炼为本体规则，使概念之间的关系形式化并确保知识管理存储库的完整性。将法学硕士视为人类知识的隐式存储库，我们提出了 Evontree，这是一种新颖的框架，它利用一小组高质量的本体规则来系统地提取、验证和增强法学硕士内的领域知识，而不需要大量的外部数据集。具体来说，Evontree 从原始模型中提取领域本体，使用两个核心本体规则检测不一致，并通过自蒸馏微调强化细化知识。使用 Llama3-8B-Instruct 和 Med42-v2 对医疗 QA 基准进行的广泛实验表明，其性能始终优于未修改的模型和领先的监督基线，准确率提高了 3.7%。这些结果证实了我们的法学硕士低资源领域适应方法的有效性、效率和稳健性。

Title: Kimi Linear: An Expressive, Efficient Attention Architecture

• Authors: Kimi Team: Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T.Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, Yulun Du
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26692
• Pdf URL: https://arxiv.org/pdf/2510.26692
• Copy Paste: [[2510.26692]] Kimi Linear: An Expressive, Efficient Attention Architecture(https://arxiv.org/abs/2510.26692)
• Keywords: llm
• Abstract: We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
• 摘要：我们引入了 Kimi Linear，这是一种混合线性注意力架构，它首次在各种场景（包括短上下文、长上下文和强化学习 (RL) 缩放机制）的公平比较下表现优于完全注意力。其核心在于 Kimi Delta Attention (KDA)，这是一个富有表现力的线性注意力模块，它通过更细粒度的门控机制扩展了 Gated DeltaNet，从而能够更有效地使用有限的有限状态 RNN 内存。我们定制的分块算法通过对角加低秩 (DPLR) 转换矩阵的特殊变体实现了高硬件效率，与一般 DPLR 公式相比，它大大减少了计算量，同时与经典的 delta 规则保持更加一致。我们基于 KDA 和多头潜在注意力 (MLA) 的分层混合，预训练具有 3B 个激活参数和 48B 个总参数的 Kimi 线性模型。我们的实验表明，使用相同的训练方案，Kimi Linear 在所有评估的任务中均优于完整的 MLA，同时将 KV 缓存使用量减少高达 75%，并在 1M 上下文中实现高达 6 倍的解码吞吐量。这些结果表明，Kimi Linear 可以直接替代全注意力架构，具有卓越的性能和效率，包括具有较长输入和输出长度的任务。为了支持进一步的研究，我们开源了 KDA 内核和 vLLM 实现，并发布了预训练和指令调整的模型检查点。

Title: The End of Manual Decoding: Towards Truly End-to-End Language Models

• Authors: Zhichao Wang, Dongyang Ma, Xinting Huang, Deng Cai, Tian Lan, Jiahao Xu, Haitao Mi, Xiaoying Tang, Yan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26697
• Pdf URL: https://arxiv.org/pdf/2510.26697
• Copy Paste: [[2510.26697]] The End of Manual Decoding: Towards Truly End-to-End Language Models(https://arxiv.org/abs/2510.26697)
• Keywords: language model, llm
• Abstract: The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyperparameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end" generation by learning to control its own decoding strategy. We augment the standard transformer with lightweight heads that, at each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into a parametric, token-level process, allowing the model to self-regulate its sampling strategy within a single forward pass. Through extensive experiments on eight benchmarks, we demonstrate that AutoDeco not only significantly outperforms default decoding strategies but also achieves performance comparable to an oracle-tuned baseline derived from "hacking the test set"-a practical upper bound for any static method. Crucially, we uncover an emergent capability for instruction-based decoding control: the model learns to interpret natural language commands (e.g., "generate with low randomness") and adjusts its predicted temperature and top-p on a token-by-token basis, opening a new paradigm for steerable and interactive LLM decoding.
• 摘要：法学硕士的“端到端”标签是用词不当。在实践中，它们依赖于不可微的解码过程，该过程需要费力地手动调整温度和 top-p 等超参数。本文介绍了 AutoDeco，这是一种新颖的架构，它通过学习控制自己的解码策略来实现真正的“端到端”生成。我们用轻量级头增强了标准变压器，在每一步中，动态预测上下文特定的温度和 top-p 值以及下一个令牌 logits。这种方法将解码转换为参数化的、令牌级的过程，允许模型在单个前向传递中自我调节其采样策略。通过对八个基准测试的大量实验，我们证明 AutoDeco 不仅显着优于默认解码策略，而且还实现了与“破解测试集”（任何静态方法的实用上限）导出的 Oracle 调整基线相当的性能。至关重要的是，我们发现了基于指令的解码控制的新兴功能：该模型学习解释自然语言命令（例如，“以低随机性生成”）并在逐个令牌的基础上调整其预测温度和 top-p，为可操纵和交互式 LLM 解码开辟了新范例。

Title: Value Drifts: Tracing Value Alignment During LLM Post-Training

• Authors: Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, Siva Reddy
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.26707
• Pdf URL: https://arxiv.org/pdf/2510.26707
• Copy Paste: [[2510.26707]] Value Drifts: Tracing Value Alignment During LLM Post-Training(https://arxiv.org/abs/2510.26707)
• Keywords: llm
• Abstract: As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.
• 摘要：随着法学硕士在社会中扮演着越来越重要的角色，他们面临着越来越多的问题，要求他们不仅要利用一般知识，还要与某些人类价值体系保持一致。因此，研究法学硕士与人类价值观的一致性已成为一个重要的研究领域。然而，之前的工作主要集中在评估经过充分训练的模型的一致性，忽略了模型学习表达人类价值观的训练动态。在这项工作中，我们研究了模型后期训练过程中如何以及在哪个阶段出现价值调整。我们的分析理清了训练后算法和数据集的影响，测量了训练期间价值漂移的幅度和时间。通过对不同大小的 Llama-3 和 Qwen-3 模型以及流行的监督微调 (SFT) 和偏好优化数据集和算法进行实验，我们发现 SFT 阶段通常会建立模型的值，而后续的偏好优化很少重新对齐这些值。此外，使用能够控制值操作的综合偏好数据集，我们发现不同的偏好优化算法会导致不同的值对齐结果，即使偏好数据保持不变。我们的研究结果为如何在训练后学习价值观提供了可行的见解，并有助于为数据管理提供信息，以及选择用于偏好优化的模型和算法，以提高模型与人类价值观的一致性。

Title: AMO-Bench: Large Language Models Still Struggle in High School Math Competitions

• Authors: Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma, Xuanlin Wang, Ziwen Wang, Shuang Zhou (Alphabetical order by last name)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26768
• Pdf URL: https://arxiv.org/pdf/2510.26768
• Copy Paste: [[2510.26768]] AMO-Bench: Large Language Models Still Struggle in High School Math Competitions(https://arxiv.org/abs/2510.26768)
• Keywords: language model, llm
• Abstract: We present AMO-Bench, an Advanced Mathematical reasoning benchmark with Olympiad level or even higher difficulty, comprising 50 human-crafted problems. Existing benchmarks have widely leveraged high school math competitions for evaluating mathematical reasoning capabilities of large language models (LLMs). However, many existing math competitions are becoming less effective for assessing top-tier LLMs due to performance saturation (e.g., AIME24/25). To address this, AMO-Bench introduces more rigorous challenges by ensuring all 50 problems are (1) cross-validated by experts to meet at least the International Mathematical Olympiad (IMO) difficulty standards, and (2) entirely original problems to prevent potential performance leakages from data memorization. Moreover, each problem in AMO-Bench requires only a final answer rather than a proof, enabling automatic and robust grading for evaluation. Experimental results across 26 LLMs on AMO-Bench show that even the best-performing model achieves only 52.4% accuracy on AMO-Bench, with most LLMs scoring below 40%. Beyond these poor performances, our further analysis reveals a promising scaling trend with increasing test-time compute on AMO-Bench. These results highlight the significant room for improving the mathematical reasoning in current LLMs. We release AMO-Bench to facilitate further research into advancing the reasoning abilities of language models. this https URL
• 摘要：我们推出了 AMO-Bench，这是一种具有奥林匹克级别甚至更高难度的高级数学推理基准，包含 50 个人工设计的问题。现有基准广泛利用高中数学竞赛来评估大型语言模型 (LLM) 的数学推理能力。然而，由于表现饱和（例如 AIME24/25），许多现有的数学竞赛对于评估顶级法学硕士的效果正在降低。为了解决这个问题，AMO-Bench 引入了更严格的挑战，确保所有 50 个问题（1）经过专家交叉验证，至少满足国际数学奥林匹克（IMO）难度标准，（2）完全原创的问题，以防止数据记忆中潜在的性能泄漏。此外，AMO-Bench 中的每个问题只需要最终答案而不需要证明，从而实现自动且稳健的评估评分。 AMO-Bench 上 26 个法学硕士的实验结果表明，即使是性能最好的模型，在 AMO-Bench 上也只能达到 52.4% 的准确率，大多数法学硕士的得分低于 40%。除了这些糟糕的性能之外，我们的进一步分析揭示了随着 AMO-Bench 上测试时间计算的增加，一个有希望的扩展趋势。这些结果凸显了当前法学硕士数学推理的巨大改进空间。我们发布 AMO-Bench 是为了促进进一步研究提高语言模型的推理能力。这个 https 网址

Title: Gistify! Codebase-Level Understanding via Runtime Execution

• Authors: Hyunji Lee, Minseon Kim, Chinmay Singh, Matheus Pereira, Atharv Sonwane, Isadora White, Elias Stengel-Eskin, Mohit Bansal, Zhengyan Shi, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan, Lucas Caccia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26790
• Pdf URL: https://arxiv.org/pdf/2510.26790
• Copy Paste: [[2510.26790]] Gistify! Codebase-Level Understanding via Runtime Execution(https://arxiv.org/abs/2510.26790)
• Keywords: llm, agent
• Abstract: As coding agents are increasingly deployed in large codebases, the need to automatically design challenging, codebase-level evaluation is central. We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase. The coding LLM is given full access to a codebase along with a specific entrypoint (e.g., a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential components necessary to execute the provided command. Success on Gistify requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions traces.
• 摘要：随着编码代理越来越多地部署在大型代码库中，自动设计具有挑战性的代码库级评估的需求变得至关重要。我们提出了 Gistify，这是一项编码 LLM 必须创建单个、最小、独立的文件的任务，该文件可以重现代码库的特定功能。编码 LLM 具有对代码库以及特定入口点（例如 python 命令）的完全访问权限，并且生成的文件必须复制在完整代码库下运行的同一命令的输出，同时仅包含执行所提供命令所需的基本组件。 Gistify 的成功需要对代码库的结构理解、对其执行流程的准确建模以及生成潜在的大型代码补丁的能力。我们的研究结果表明，当前最先进的模型很难可靠地解决 Gistify 任务，尤其是那些执行轨迹较长的任务。