2025-11-03

Title: Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling

• Authors: Hyunji Lee, Wenhao Yu, Hongming Zhang, Kaixin Ma, Jiyeon Kim, Dong Yu, Minjoon Seo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.26912
• Pdf URL: https://arxiv.org/pdf/2510.26912
• Copy Paste: [[2510.26912]] Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling(https://arxiv.org/abs/2510.26912)
• Keywords: language model
• Abstract: Hybrid models that combine state space models (SSMs) with attention mechanisms have shown strong performance by leveraging the efficiency of SSMs and the high recall ability of attention. However, the architectural design choices behind these hybrid models remain insufficiently understood. In this work, we analyze hybrid architectures through the lens of memory utilization and overall performance, and propose a complementary method to further enhance their effectiveness. We first examine the distinction between sequential and parallel integration of SSM and attention layers. Our analysis reveals several interesting findings, including that sequential hybrids perform better on shorter contexts, whereas parallel hybrids are more effective for longer contexts. We also introduce a data-centric approach of continually training on datasets augmented with paraphrases, which further enhances recall while preserving other capabilities. It generalizes well across different base models and outperforms architectural modifications aimed at enhancing recall. Our findings provide a deeper understanding of hybrid SSM-attention models and offer practical guidance for designing architectures tailored to various use cases. Our findings provide a deeper understanding of hybrid SSM-attention models and offer practical guidance for designing architectures tailored to various use cases.
• 摘要：将状态空间模型（SSM）与注意力机制相结合的混合模型通过利用 SSM 的效率和注意力的高召回能力，表现出了强大的性能。然而，这些混合模型背后的架构设计选择仍然没有得到足够的了解。在这项工作中，我们从内存利用率和整体性能的角度分析混合架构，并提出一种补充方法来进一步提高其有效性。我们首先研究 SSM 和注意力层的顺序集成和并行集成之间的区别。我们的分析揭示了一些有趣的发现，包括顺序混合在较短的上下文中表现更好，而并行混合对于较长的上下文更有效。我们还引入了一种以数据为中心的方法，对通过释义增强的数据集进行持续训练，这进一步增强了召回率，同时保留了其他功能。它可以很好地概括不同的基础模型，并且优于旨在增强召回率的架构修改。我们的研究结果提供了对混合 SSM 注意力模型的更深入的理解，并为设计适合各种用例的架构提供了实用指导。我们的研究结果提供了对混合 SSM 注意力模型的更深入的理解，并为设计适合各种用例的架构提供了实用指导。

Title: Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations

• Authors: Jean-Philippe Corbeil, Asma Ben Abacha, Jerome Tremblay, Phillip Swazinna, Akila Jeeson Daniel, Miguel Del-Agua, Francois Beaulieu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.26974
• Pdf URL: https://arxiv.org/pdf/2510.26974
• Copy Paste: [[2510.26974]] Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations(https://arxiv.org/abs/2510.26974)
• Keywords: language model, llm
• Abstract: Clinical documentation increasingly uses automatic speech recognition and summarization, yet converting conversations into actionable medical orders for Electronic Health Records remains unexplored. A solution to this problem can significantly reduce the documentation burden of clinicians and directly impact downstream patient care. We introduce the MEDIQA-OE 2025 shared task, the first challenge on extracting medical orders from doctor-patient conversations. Six teams participated in the shared task and experimented with a broad range of approaches, and both closed- and open-weight large language models (LLMs). In this paper, we describe the MEDIQA-OE task, dataset, final leaderboard ranking, and participants' solutions.
• 摘要：临床文档越来越多地使用自动语音识别和摘要，但将对话转换为电子健康记录的可操作医疗指令仍有待探索。该问题的解决方案可以显着减轻临床医生的文档负担，并直接影响下游患者护理。我们介绍 MEDIQA-OE 2025 共享任务，这是从医患对话中提取医疗指令的第一个挑战。六个团队参与了这项共同任务，并尝试了多种方法，以及封闭式和开放式权重大语言模型 (LLM)。在本文中，我们描述了 MEDIQA-OE 任务、数据集、最终排行榜排名以及参与者的解决方案。

Title: Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services

• Authors: Jayden Serenari, Stephen Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27016
• Pdf URL: https://arxiv.org/pdf/2510.27016
• Copy Paste: [[2510.27016]] Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services(https://arxiv.org/abs/2510.27016)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: With the increasing use of conversational AI systems, there is growing concern over privacy leaks, especially when users share sensitive personal data in interactions with Large Language Models (LLMs). Conversations shared with these models may contain Personally Identifiable Information (PII), which, if exposed, could lead to security breaches or identity theft. To address this challenge, we present the Local Optimizations for Pseudonymization with Semantic Integrity Directed Entity Detection (LOPSIDED) framework, a semantically-aware privacy agent designed to safeguard sensitive PII data when using remote LLMs. Unlike prior work that often degrade response quality, our approach dynamically replaces sensitive PII entities in user prompts with semantically consistent pseudonyms, preserving the contextual integrity of conversations. Once the model generates its response, the pseudonyms are automatically depseudonymized, ensuring the user receives an accurate, privacy-preserving output. We evaluate our approach using real-world conversations sourced from ShareGPT, which we further augment and annotate to assess whether named entities are contextually relevant to the model's response. Our results show that LOPSIDED reduces semantic utility errors by a factor of 5 compared to baseline techniques, all while enhancing privacy.
• 摘要：随着对话式人工智能系统的使用越来越多，人们越来越担心隐私泄露，特别是当用户在与大型语言模型（LLM）交互时共享敏感的个人数据时。与这些模型共享的对话可能包含个人身份信息 (PII)，这些信息一旦泄露，可能会导致安全漏洞或身份盗用。为了应对这一挑战，我们提出了使用语义完整性定向实体检测 (LOPSIDED) 框架进行假名化的本地优化，这是一种语义感知的隐私代理，旨在在使用远程 LLM 时保护敏感的 PII 数据。与之前经常降低响应质量的工作不同，我们的方法用语义一致的假名动态替换用户提示中的敏感 PII 实体，从而保留对话的上下文完整性。一旦模型生成响应，假名就会自动去假名化，确保用户收到准确、保护隐私的输出。我们使用来自 ShareGPT 的真实世界对话来评估我们的方法，我们进一步对其进行增强和注释，以评估命名实体是否与模型响应的上下文相关。我们的结果表明，与基线技术相比，LOPSIDED 将语义效用错误减少了 5 倍，同时增强了隐私性。

Title: Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral

• Authors: Ayoub Hammal, Pierre Zweigenbaum, Caio Corro
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27017
• Pdf URL: https://arxiv.org/pdf/2510.27017
• Copy Paste: [[2510.27017]] Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral(https://arxiv.org/abs/2510.27017)
• Keywords: language model, llm
• Abstract: Several previous works concluded that the largest part of generation capabilities of large language models (LLM) are learned (early) during pre-training. However, LLMs still require further alignment to adhere to downstream task requirements and stylistic preferences, among other desired properties. As LLMs continue to scale in terms of size, the computational cost of alignment procedures increase prohibitively. In this work, we propose a novel approach to circumvent these costs via proxy-based test-time alignment, i.e. using guidance from a small aligned model. Our approach can be described as token-specific cascading method, where the token-specific deferral rule is reduced to 0-1 knapsack problem. In this setting, we derive primal and dual approximations of the optimal deferral decision. We experimentally show the benefits of our method both in task performance and speculative decoding speed.
• 摘要：之前的几项工作得出的结论是，大型语言模型（LLM）的生成能力的大部分是在预训练期间（早期）学习的。然而，法学硕士仍然需要进一步调整，以遵守下游任务要求和风格偏好以及其他所需的属性。随着法学硕士规模的不断扩大，比对程序的计算成本急剧增加。在这项工作中，我们提出了一种通过基于代理的测试时间对齐来规避这些成本的新方法，即使用小型对齐模型的指导。我们的方法可以描述为特定于代币的级联方法，其中特定于代币的延迟规则被简化为 0-1 背包问题。在这种情况下，我们得出最佳推迟决策的原始近似和对偶近似。我们通过实验证明了我们的方法在任务性能和推测解码速度方面的优势。

Title: Elastic Architecture Search for Efficient Language Models

• Authors: Shang Wang
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2510.27037
• Pdf URL: https://arxiv.org/pdf/2510.27037
• Copy Paste: [[2510.27037]] Elastic Architecture Search for Efficient Language Models(https://arxiv.org/abs/2510.27037)
• Keywords: language model
• Abstract: As large pre-trained language models become increasingly critical to natural language understanding (NLU) tasks, their substantial computational and memory requirements have raised significant economic and environmental concerns. Addressing these challenges, this paper introduces the Elastic Language Model (ELM), a novel neural architecture search (NAS) method optimized for compact language models. ELM extends existing NAS approaches by introducing a flexible search space with efficient transformer blocks and dynamic modules for dimension and head number adjustment. These innovations enhance the efficiency and flexibility of the search process, which facilitates more thorough and effective exploration of model architectures. We also introduce novel knowledge distillation losses that preserve the unique characteristics of each block, in order to improve the discrimination between architectural choices during the search process. Experiments on masked language modeling and causal language modeling tasks demonstrate that models discovered by ELM significantly outperform existing methods.
• 摘要：随着大型预训练语言模型对于自然语言理解 (NLU) 任务变得越来越重要，其大量的计算和内存需求引发了重大的经济和环境问题。为了解决这些挑战，本文介绍了弹性语言模型（ELM），这是一种针对紧凑语言模型优化的新型神经架构搜索（NAS）方法。 ELM 扩展了现有的 NAS 方法，引入了灵活的搜索空间，该搜索空间具有高效的变压器块和用于尺寸和磁头数量调整的动态模块。这些创新提高了搜索过程的效率和灵活性，有利于对模型架构进行更彻底、更有效的探索。我们还引入了新颖的知识蒸馏损失，以保留每个块的独特特征，以提高搜索过程中架构选择之间的区分度。掩码语言建模和因果语言建模任务的实验表明，ELM 发现的模型显着优于现有方法。

Title: Dataset Creation and Baseline Models for Sexism Detection in Hausa

• Authors: Fatima Adam Muhammad, Shamsuddeen Muhammad Hassan, Isa Inuwa-Dutse
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27038
• Pdf URL: https://arxiv.org/pdf/2510.27038
• Copy Paste: [[2510.27038]] Dataset Creation and Baseline Models for Sexism Detection in Hausa(https://arxiv.org/abs/2510.27038)
• Keywords: language model
• Abstract: Sexism reinforces gender inequality and social exclusion by perpetuating stereotypes, bias, and discriminatory norms. Noting how online platforms enable various forms of sexism to thrive, there is a growing need for effective sexism detection and mitigation strategies. While computational approaches to sexism detection are widespread in high-resource languages, progress remains limited in low-resource languages where limited linguistic resources and cultural differences affect how sexism is expressed and perceived. This study introduces the first Hausa sexism detection dataset, developed through community engagement, qualitative coding, and data augmentation. For cultural nuances and linguistic representation, we conducted a two-stage user study (n=66) involving native speakers to explore how sexism is defined and articulated in everyday discourse. We further experiment with both traditional machine learning classifiers and pre-trained multilingual language models and evaluating the effectiveness few-shot learning in detecting sexism in Hausa. Our findings highlight challenges in capturing cultural nuance, particularly with clarification-seeking and idiomatic expressions, and reveal a tendency for many false positives in such cases.
• 摘要：性别歧视通过延续陈规定型观念、偏见和歧视性规范，加剧了性别不平等和社会排斥。注意到在线平台如何助长各种形式的性别歧视，人们越来越需要有效的性别歧视检测和缓解策略。虽然性别歧视检测的计算方法在高资源语言中很普遍，但在低资源语言中进展仍然有限，因为有限的语言资源和文化差异影响了性别歧视的表达和感知方式。本研究介绍了第一个豪萨性别歧视检测数据集，该数据集是通过社区参与、定性编码和数据增强开发的。对于文化差异和语言表征，我们进行了一项涉及母语人士的两阶段用户研究 (n=66)，以探讨如何在日常话语中定义和表达性别歧视。我们进一步实验了传统的机器学习分类器和预先训练的多语言语言模型，并评估了小样本学习在检测豪萨语性别歧视方面的有效性。我们的研究结果强调了捕捉文化细微差别的挑战，特别是在寻求澄清和惯用表达方面，并揭示了在此类情况下出现许多误报的趋势。

Title: VISTA Score: Verification In Sequential Turn-based Assessment

• Authors: Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier, Michael White
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27052
• Pdf URL: https://arxiv.org/pdf/2510.27052
• Copy Paste: [[2510.27052]] VISTA Score: Verification In Sequential Turn-based Assessment(https://arxiv.org/abs/2510.27052)
• Keywords: language model, llm, hallucination
• Abstract: Hallucination--defined here as generating statements unsupported or contradicted by available evidence or conversational context--remains a major obstacle to deploying conversational AI systems in settings that demand factual reliability. Existing metrics either evaluate isolated responses or treat unverifiable content as errors, limiting their use for multi-turn dialogue. We introduce VISTA (Verification In Sequential Turn-based Assessment), a framework for evaluating conversational factuality through claim-level verification and sequential consistency tracking. VISTA decomposes each assistant turn into atomic factual claims, verifies them against trusted sources and dialogue history, and categorizes unverifiable statements (subjective, contradicted, lacking evidence, or abstaining). Across eight large language models and four dialogue factuality benchmarks (AIS, BEGIN, FAITHDIAL, and FADE), VISTA substantially improves hallucination detection over FACTSCORE and LLM-as-Judge baselines. Human evaluation confirms that VISTA's decomposition improves annotator agreement and reveals inconsistencies in existing benchmarks. By modeling factuality as a dynamic property of conversation, VISTA offers a more transparent, human-aligned measure of truthfulness in dialogue systems.
• 摘要：幻觉（此处定义为生成不受现有证据或对话上下文支持或矛盾的陈述）仍然是在需要事实可靠性的环境中部署对话式人工智能系统的主要障碍。现有的指标要么评估孤立的响应，要么将无法验证的内容视为错误，从而限制了它们在多轮对话中的使用。我们引入 VISTA（基于回合的顺序评估验证），这是一个通过声明级验证和顺序一致性跟踪来评估对话事实性的框架。 VISTA 将每个助手分解为原子事实主张，根据可信来源和对话历史对其进行验证，并对无法验证的陈述进行分类（主观、矛盾、缺乏证据或弃权）。在八个大型语言模型和四个对话事实性基准（AIS、BEGIN、FAITHDIAL 和 FADE）中，VISTA 相对于 FACTSCORE 和 LLM-as-Judge 基准大幅改进了幻觉检测。人工评估证实 VISTA 的分解提高了注释器一致性并揭示了现有基准中的不一致之处。通过将真实性建模为对话的动态属性，VISTA 提供了一种更透明、更人性化的对话系统真实性衡量标准。

Title: LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraints

• Authors: Xiaofan Guo, Yaxuan Luan, Yue Kang, Xiangchen Song, Jinxu Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27054
• Pdf URL: https://arxiv.org/pdf/2510.27054
• Copy Paste: [[2510.27054]] LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraints(https://arxiv.org/abs/2510.27054)
• Keywords: llm, retrieval-augmented generation
• Abstract: This paper addresses the issues of insufficient coverage, unstable results, and limited reliability in retrieval-augmented generation under complex knowledge environments, and proposes a confidence control method that integrates multi-granularity memory indexing with uncertainty estimation. The method builds a hierarchical memory structure that divides knowledge representations into different levels of granularity, enabling dynamic indexing and retrieval from local details to global context, and thus establishing closer semantic connections between retrieval and generation. On this basis, an uncertainty estimation mechanism is introduced to explicitly constrain and filter low-confidence paths during the generation process, allowing the model to maintain information coverage while effectively suppressing noise and false content. The overall optimization objective consists of generation loss, entropy constraints, and variance regularization, forming a unified confidence control framework. In the experiments, comprehensive sensitivity tests and comparative analyses were designed, covering hyperparameters, environmental conditions, and data structures, to verify the stability and robustness of the proposed method across different scenarios. The results show that the method achieves superior performance over existing models in QA accuracy, retrieval recall, ranking quality, and factual consistency, demonstrating the effectiveness of combining multi-granularity indexing with confidence control. This study not only provides a new technical pathway for retrieval-augmented generation but also offers practical evidence for improving the reliability and controllability of large models in complex contexts.
• 摘要：针对复杂知识环境下检索增强生成覆盖不足、结果不稳定、可靠性有限的问题，提出一种多粒度内存索引与不确定性估计相结合的置信度控制方法。该方法构建了分层记忆结构，将知识表示划分为不同粒度级别，实现从局部细节到全局上下文的动态索引和检索，从而在检索和生成之间建立更紧密的语义联系。在此基础上，引入不确定性估计机制，在生成过程中显式约束和过滤低置信度路径，使模型在保持信息覆盖的同时有效抑制噪声和虚假内容。整体优化目标由生成损失、熵约束和方差正则化组成，形成统一的置信度控制框架。在实验中，设计了全面的敏感性测试和比较分析，涵盖超参数、环境条件和数据结构，以验证所提方法在不同场景下的稳定性和鲁棒性。结果表明，该方法在 QA 准确性、检索召回率、排名质量和事实一致性方面均优于现有模型，证明了多粒度索引与置信度控制相结合的有效性。该研究不仅为检索增强生成提供了一条新的技术途径，而且为提高复杂环境下大型模型的可靠性和可控性提供了实践证据。

Title: Detecting Data Contamination in LLMs via In-Context Learning

• Authors: Michał Zawalski, Meriem Boubdir, Klaudia Bałazy, Besmira Nushi, Pablo Ribalta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27055
• Pdf URL: https://arxiv.org/pdf/2510.27055
• Copy Paste: [[2510.27055]] Detecting Data Contamination in LLMs via In-Context Learning(https://arxiv.org/abs/2510.27055)
• Keywords: language model, llm
• Abstract: We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in-context examples typically boost confidence for unseen datasets but may reduce it when the dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and dataset-agnostic, making it easy to integrate with benchmark evaluations.
• 摘要：我们提出了通过上下文进行污染检测（CoDeC），这是一种实用且准确的方法，用于检测和量化大型语言模型中的训练数据污染。 CoDeC 通过测量上下文学习如何影响模型性能来区分训练期间记忆的数据和训练分布之外的数据。我们发现，上下文中的示例通常会增强对未见过的数据集的信心，但当数据集是训练的一部分时，由于记忆模式被破坏，信心可能会降低。实验表明，CoDeC 产生可解释的污染分数，可以清楚地区分可见和未见的数据集，并揭示了使用未公开的训练语料库的开放权重模型中的记忆的有力证据。该方法简单、自动化，且与模型和数据集无关，因此可以轻松与基准评估集成。

Title: Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models

• Authors: Jiasen Zheng, Huajun Zhang, Xu Yan, Ran Hao, Chong Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27077
• Pdf URL: https://arxiv.org/pdf/2510.27077
• Copy Paste: [[2510.27077]] Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models(https://arxiv.org/abs/2510.27077)
• Keywords: language model
• Abstract: This paper addresses the limitations of large-scale language models in safety alignment and robustness by proposing a fine-tuning method that combines contrastive distillation with noise-robust training. The method freezes the backbone model and transfers the knowledge boundaries of the teacher model to the student model through distillation, thereby improving semantic consistency and alignment accuracy. At the same time, noise perturbations and robust optimization constraints are introduced during training to ensure that the model maintains stable predictive outputs under noisy and uncertain inputs. The overall framework consists of distillation loss, robustness loss, and a regularization term, forming a unified optimization objective that balances alignment ability with resistance to interference. To systematically validate its effectiveness, the study designs experiments from multiple perspectives, including distillation weight sensitivity, stability analysis under computation budgets and mixed-precision environments, and the impact of data noise and distribution shifts on model performance. Results show that the method significantly outperforms existing baselines in knowledge transfer, robustness, and overall safety, achieving the best performance across several key metrics. This work not only enriches the theoretical system of parameter-efficient fine-tuning but also provides a new solution for building safer and more trustworthy alignment mechanisms.
• 摘要：本文通过提出一种将对比蒸馏与噪声鲁棒训练相结合的微调方法，解决了大规模语言模型在安全对齐和鲁棒性方面的局限性。该方法冻结骨干模型，通过蒸馏将教师模型的知识边界转移到学生模型，从而提高语义一致性和对齐精度。同时，在训练过程中引入噪声扰动和鲁棒优化约束，以确保模型在噪声和不确定输入下保持稳定的预测输出。整体框架由蒸馏损失、鲁棒性损失和正则化项组成，形成统一的优化目标，平衡对齐能力和抗干扰能力。为了系统地验证其有效性，该研究从多个角度设计了实验，包括蒸馏权重敏感性、计算预算和混合精度环境下的稳定性分析，以及数据噪声和分布偏移对模型性能的影响。结果表明，该方法在知识转移、稳健性和整体安全性方面显着优于现有基线，在多个关键指标上实现了最佳性能。这项工作不仅丰富了参数高效微调的理论体系，而且为构建更安全、更值得信赖的对齐机制提供了新的解决方案。

Title: Characterizing Selective Refusal Bias in Large Language Models

• Authors: Adel Khorramrouz, Sharon Levy
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2510.27087
• Pdf URL: https://arxiv.org/pdf/2510.27087
• Copy Paste: [[2510.27087]] Characterizing Selective Refusal Bias in Large Language Models(https://arxiv.org/abs/2510.27087)
• Keywords: language model, llm
• Abstract: Safety guardrails in large language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implications via an indirect attack, where we target previously refused groups. Our findings emphasize the need for more equitable and robust performance in safety guardrails across demographic groups.
• 摘要：大型语言模型（LLM）中的安全护栏是为了防止恶意用户大规模生成有毒内容而开发的。然而，这些措施可能会无意中引入或反映新的偏见，因为法学硕士可能拒绝生成针对某些人口群体而不是其他人口群体的有害内容。我们通过目标个人和交叉人口群体的拒绝率、LLM 响应类型以及产生拒绝的持续时间来探索 LLM 护栏中的这种选择性拒绝偏差。我们的结果显示了跨性别、性取向、国籍和宗教属性的选择性拒绝偏见的证据。这导致我们通过间接攻击来调查额外的安全影响，其中我们针对以前被拒绝的群体。我们的研究结果强调，需要在不同人口群体的安全护栏方面实现更加公平和稳健的表现。

Title: Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks

• Authors: Rajarshi Haldar, Julia Hockenmaier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27106
• Pdf URL: https://arxiv.org/pdf/2510.27106
• Copy Paste: [[2510.27106]] Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks(https://arxiv.org/abs/2510.27106)
• Keywords: language model, llm
• Abstract: As Natural Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large language models (LLMs) for evaluating these generations has gained traction, as they tend to align more closely with human preferences than conventional n-gram or embedding-based metrics. In our experiments, we show that LLM judges have low intra-rater reliability in their assigned scores across different runs. This variance makes their ratings inconsistent, almost arbitrary in the worst case, making it difficult to measure how good their judgments actually are. We quantify this inconsistency across different NLG tasks and benchmarks and see if judicious use of LLM judges can still be useful following proper guidelines.
• 摘要：随着自然语言生成（NLG）继续被广泛采用，对其进行正确评估已变得相当困难。最近，使用大型语言模型 (LLM) 来评估这些代已经受到关注，因为与传统的 n 元语法或基于嵌入的指标相比，它们往往更符合人类的偏好。在我们的实验中，我们表明法学硕士法官在不同的运行中分配的分数的评分者内部可靠性较低。这种差异使得他们的评级不一致，在最坏的情况下几乎是任意的，因此很难衡量他们的判断实际上有多好。我们量化了不同 NLG 任务和基准中的这种不一致，并看看遵循适当的指导方针，明智地使用法学硕士法官是否仍然有用。

Title: Probability Distributions Computed by Hard-Attention Transformers

• Authors: Andy Yang, Anej Svete, Jiaoda Li, Anthony Widjaja Lin, Jonathan Rawski, Ryan Cotterell, David Chiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27118
• Pdf URL: https://arxiv.org/pdf/2510.27118
• Copy Paste: [[2510.27118]] Probability Distributions Computed by Hard-Attention Transformers(https://arxiv.org/abs/2510.27118)
• Keywords: language model
• Abstract: Most expressivity results for transformers treat them as language recognizers (which accept or reject strings), and not as they are used in practice, as language models (which generate strings autoregressively and probabilistically). Here, we characterize the probability distributions that transformer language models can express. We show that making transformer language recognizers autoregressive can sometimes increase their expressivity, and that making them probabilistic can break equivalences that hold in the non-probabilistic case. Our overall contribution is to tease apart what functions transformers are capable of expressing, in their most common use-case as language models.
• 摘要：大多数 Transformer 的表现力结果将它们视为语言识别器（接受或拒绝字符串），而不是像实践中使用的语言模型（以自回归和概率方式生成字符串）。在这里，我们描述了 Transformer 语言模型可以表达的概率分布。我们证明，使 Transformer 语言识别器成为自回归有时可以增加其表达能力，而使它们成为概率性可以打破非概率情况下的等价性。我们的总体贡献是梳理变压器在最常见的用例（即语言模型）中能够表达哪些功能。

Title: MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models

• Authors: Zixin Chen, Hongzhan Lin, Kaixin Li, Ziyang Luo, Yayue Deng, Jing Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27196
• Pdf URL: https://arxiv.org/pdf/2510.27196
• Copy Paste: [[2510.27196]] MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models(https://arxiv.org/abs/2510.27196)
• Keywords: language model, llm, agent
• Abstract: The proliferation of memes on social media necessitates the capabilities of multimodal Large Language Models (mLLMs) to effectively understand multimodal harmfulness. Existing evaluation approaches predominantly focus on mLLMs' detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. In this paper, we propose MemeArena, an agent-based arena-style evaluation framework that provides a context-aware and unbiased assessment for mLLMs' understanding of multimodal harmfulness. Specifically, MemeArena simulates diverse interpretive contexts to formulate evaluation tasks that elicit perspective-specific analyses from mLLMs. By integrating varied viewpoints and reaching consensus among evaluators, it enables fair and unbiased comparisons of mLLMs' abilities to interpret multimodal harmfulness. Extensive experiments demonstrate that our framework effectively reduces the evaluation biases of judge agents, with judgment results closely aligning with human preferences, offering valuable insights into reliable and comprehensive mLLM evaluations in multimodal harmfulness understanding. Our code and data are publicly available at this https URL.
• 摘要：社交媒体上表情包的激增需要多模态大语言模型 (mLLM) 的能力来有效理解多模态危害性。现有的评估方法主要关注 mLLM 对二元分类任务的检测准确性，这往往无法反映不同背景下有害性的深入解释细微差别。在本文中，我们提出了 MemeArena，这是一种基于代理的竞技场式评估框架，为 mLLM 对多模式危害性的理解提供上下文感知和公正的评估。具体来说，MemeArena 模拟不同的解释上下文来制定评估任务，从而从 mLLM 中引出特定观点的分析。通过整合不同的观点并在评估者之间达成共识，它可以对 mLLM 解释多模式危害性的能力进行公平和公正的比较。大量实验表明，我们的框架有效减少了判断代理的评估偏差，判断结果与人类偏好紧密结合，为多模态危害性理解中可靠且全面的mLLM评估提供了有价值的见解。我们的代码和数据可通过此 https URL 公开获取。

Title: Identifying the Periodicity of Information in Natural Language

• Authors: Yulin Ou, Yu Wang, Yang Xu, Hendrik Buschmeier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27241
• Pdf URL: https://arxiv.org/pdf/2510.27241
• Copy Paste: [[2510.27241]] Identifying the Periodicity of Information in Natural Language(https://arxiv.org/abs/2510.27241)
• Keywords: llm
• Abstract: Recent theoretical advancement of information density in natural language has brought the following question on desk: To what degree does natural language exhibit periodicity pattern in its encoded information? We address this question by introducing a new method called AutoPeriod of Surprisal (APS). APS adopts a canonical periodicity detection algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document. By applying the algorithm to a set of corpora, we have obtained the following interesting results: Firstly, a considerable proportion of human language demonstrates a strong pattern of periodicity in information; Secondly, new periods that are outside the distributions of typical structural units in text (e.g., sentence boundaries, elementary discourse units, etc.) are found and further confirmed via harmonic regression modeling. We conclude that the periodicity of information in language is a joint outcome from both structured factors and other driving factors that take effect at longer distances. The advantages of our periodicity detection method and its potentials in LLM-generation detection are further discussed.
• 摘要：最近自然语言信息密度的理论进展提出了以下问题：自然语言在其编码信息中表现出周期性模式到什么程度？我们通过引入一种称为 AutoPeriod of Surprisal (APS) 的新方法来解决这个问题。 APS采用规范周期性检测算法，能够识别单个文档的意外序列中存在的任何重要周期。通过将该算法应用于一组语料库，我们得到了以下有趣的结果：首先，相当一部分的人类语言表现出很强的信息周期性模式；其次，通过调和回归模型发现并进一步确认文本中典型结构单元（例如句子边界、基本话语单元等）分布之外的新周期。我们的结论是，语言中信息的周期性是结构因素和其他在较长距离下生效的驱动因素的共同结果。进一步讨论了我们的周期性检测方法的优点及其在 LLM 生成检测中的潜力。

Title: Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

• Authors: Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani, J Ross Mitchell
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2510.27246
• Pdf URL: https://arxiv.org/pdf/2510.27246
• Copy Paste: [[2510.27246]] Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs(https://arxiv.org/abs/2510.27246)
• Keywords: language model, llm
• Abstract: Evaluating the abilities of large language models (LLMs) for tasks that require long-term memory and thus long-context reasoning, for example in conversational settings, is hampered by the existing benchmarks, which often lack narrative coherence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces a comprehensive solution to these challenges. First, we present a novel framework for automatically generating long (up to 10M tokens), coherent, and topically diverse conversations, accompanied by probing questions targeting a wide range of memory abilities. From this, we construct BEAM, a new benchmark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT-a framework inspired by human cognition that equips LLMs with three complementary memory systems: a long-term episodic memory, a short-term working memory, and a scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with 1M token context windows (with and without retrieval-augmentation) struggle as dialogues lengthen. In contrast, LIGHT consistently improves performance across various models, achieving an average improvement of 3.5%-12.69% over the strongest baselines, depending on the backbone LLM. An ablation study further confirms the contribution of each memory component.
• 摘要：评估大语言模型 (LLM) 执行需要长期记忆和长上下文推理的任务（例如在对话环境中）的能力受到现有基准的阻碍，这些基准通常缺乏叙述连贯性、涵盖狭窄的领域，并且仅测试简单的面向回忆的任务。本文介绍了应对这些挑战的全面解决方案。首先，我们提出了一个新颖的框架，用于自动生成长（最多 10M 个标记）、连贯且主题多样的对话，并附带针对广泛记忆能力的探究性问题。由此，我们构建了 BEAM，一个包含 100 个对话和 2,000 个经过验证的问题的新基准。其次，为了提高模型性能，我们提出了 LIGHT——一个受人类认知启发的框架，为法学硕士配备了三个互补的记忆系统：长期情景记忆、短期工作记忆和用于积累显着事实的便签本。我们在 BEAM 上的实验表明，即使是具有 1M 个 token 上下文窗口（有或没有检索增强）的法学硕士，随着对话的延长，也会遇到困难。相比之下，LIGHT 持续提高各种模型的性能，与最强基线相比平均提高 3.5%-12.69%，具体取决于骨干 LLM。消融研究进一步证实了每个记忆成分的贡献。

Title: Languages are Modalities: Cross-Lingual Alignment via Encoder Injection

• Authors: Rajan Agarwal, Aarush Gupta
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.27254
• Pdf URL: https://arxiv.org/pdf/2510.27254
• Copy Paste: [[2510.27254]] Languages are Modalities: Cross-Lingual Alignment via Encoder Injection(https://arxiv.org/abs/2510.27254)
• Keywords: language model, llm
• Abstract: Instruction-tuned Large Language Models (LLMs) underperform on low resource, non-Latin scripts due to tokenizer fragmentation and weak cross-lingual coupling. We present LLINK (Latent Language Injection for Non-English Knowledge), a compute efficient language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder. First, we align sentence embeddings from a frozen multilingual encoder to the decoder's latent embedding space at a reserved position via a lightweight contrastive projector. Second, the vector is expanded into K soft slots and trained with minimal adapters so the frozen decoder consumes the signal. LLINK substantially improves bilingual retrieval and achieves 81.3% preference over the base model and 63.6% over direct fine-tuning in LLM-judged Q&A evaluations. We further find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment, despite the model having residual weaknesses in numeric fidelity. Treating low resource languages as a modality offers a practical path to stronger cross-lingual alignment in lightweight LLMs.
• 摘要：由于标记器碎片和跨语言耦合较弱，指令调整的大型语言模型 (LLM) 在低资源、非拉丁脚本上表现不佳。我们提出了 LLINK（非英语知识的潜在语言注入），这是一种计算高效的语言模态方法，可以调节指令调整的解码器，而无需更改分词器或重新训练解码器。首先，我们通过轻量级对比投影仪将来自冻结多语言编码器的句子嵌入与解码器的潜在嵌入空间在保留位置对齐。其次，将向量扩展为 K 个软槽，并使用最少的适配器进行训练，以便冻结的解码器消耗信号。 LLINK 极大地改善了双语检索，在 LLM 评审的问答评估中，比基本模型获得了 81.3% 的偏好，比直接微调获得了 63.6% 的偏好。我们进一步发现，改进可以归因于标记化膨胀的减少和更强的跨语言对齐，尽管该模型在数字保真度方面仍然存在缺陷。将低资源语言视为一种模式为轻量级法学硕士提供了更强的跨语言对齐的实用途径。

Title: MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models

• Authors: Kangkun Mao, Jinru Ding, Jiayuan Chen, Mouxiao Bian, Ruiyao Chen, Xinwei Peng, Sijie Ren, Linyang Li, Jie Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27267
• Pdf URL: https://arxiv.org/pdf/2510.27267
• Copy Paste: [[2510.27267]] MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models(https://arxiv.org/abs/2510.27267)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) enter the medical domain, most benchmarks evaluate them on question answering or descriptive reasoning, overlooking quantitative reasoning critical to clinical decision-making. Existing datasets like MedCalc-Bench cover few calculation tasks and fail to reflect real-world computational scenarios. We introduce MedCalc-Eval, the largest benchmark for assessing LLMs' medical calculation abilities, comprising 700+ tasks across two types: equation-based (e.g., Cockcroft-Gault, BMI, BSA) and rule-based scoring systems (e.g., Apgar, Glasgow Coma Scale). These tasks span diverse specialties including internal medicine, surgery, pediatrics, and cardiology, offering a broader and more challenging evaluation setting. To improve performance, we further develop MedCalc-Env, a reinforcement learning environment built on the InternBootcamp framework, enabling multi-step clinical reasoning and planning. Fine-tuning a Qwen2.5-32B model within this environment achieves state-of-the-art results on MedCalc-Eval, with notable gains in numerical sensitivity, formula selection, and reasoning robustness. Remaining challenges include unit conversion, multi-condition logic, and contextual understanding. Code and datasets are available at this https URL.
• 摘要：随着大型语言模型（LLM）进入医学领域，大多数基准测试都根据问题回答或描述性推理来评估它们，而忽略了对临床决策至关重要的定量推理。 MedCalc-Bench 等现有数据集涵盖的计算任务很少，无法反映现实世界的计算场景。我们推出了 MedCalc-Eval，这是评估法学硕士医学计算能力的最大基准，包含两种类型的 700 多个任务：基于方程的（例如 Cockcroft-Gault、BMI、BSA）和基于规则的评分系统（例如 Apgar、格拉斯哥昏迷量表）。这些任务跨越不同的专业，包括内科、外科、儿科和心脏病学，提供了更广泛和更具挑战性的评估环境。为了提高性能，我们进一步开发了 MedCalc-Env，这是一个基于 InternBootcamp 框架构建的强化学习环境，可实现多步骤临床推理和规划。在此环境中微调 Qwen2.5-32B 模型在 MedCalc-Eval 上取得了最先进的结果，在数值灵敏度、公式选择和推理稳健性方面取得了显着的进步。剩下的挑战包括单位转换、多条件逻辑和上下文理解。代码和数据集可从此 https URL 获取。

Title: Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?

• Authors: Deokhyung Kang, Seonjeong Hwang, Daehui Kim, Hyounghun Kim, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.27269
• Pdf URL: https://arxiv.org/pdf/2510.27269
• Copy Paste: [[2510.27269]] Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?(https://arxiv.org/abs/2510.27269)
• Keywords: language model
• Abstract: Reasoning language models (RLMs) achieve strong performance on complex reasoning tasks, yet they still suffer from a multilingual reasoning gap, performing better in high-resource languages than in low-resource ones. While recent efforts have reduced this gap, its underlying causes remain largely unexplored. In this paper, we address this by showing that the multilingual reasoning gap largely stems from failures in language understanding-the model's inability to represent the multilingual input meaning into the dominant language (i.e., English) within its reasoning trace. This motivates us to examine whether understanding failures can be detected, as this ability could help mitigate the multilingual reasoning gap. To this end, we evaluate a range of detection methods and find that understanding failures can indeed be identified, with supervised approaches performing best. Building on this, we propose Selective Translation, a simple yet effective strategy that translates the multilingual input into English only when an understanding failure is detected. Experimental results show that Selective Translation bridges the multilingual reasoning gap, achieving near full-translation performance while using translation for only about 20% of inputs. Together, our work demonstrates that understanding failures are the primary cause of the multilingual reasoning gap and can be detected and selectively mitigated, providing key insight into its origin and a promising path toward more equitable multilingual reasoning. Our code and data are publicly available at this https URL.
• 摘要：推理语言模型（RLM）在复杂的推理任务上取得了很强的性能，但它们仍然存在多语言推理差距，在高资源语言中的表现比在低资源语言中的表现更好。尽管最近的努力缩小了这一差距，但其根本原因在很大程度上仍未得到探索。在本文中，我们通过表明多语言推理差距很大程度上源于语言理解的失败来解决这个问题——模型无法在其推理轨迹中将多语言输入含义表示为主导语言（即英语）。这促使我们检查是否可以检测到理解失败，因为这种能力可以帮助缩小多语言推理差距。为此，我们评估了一系列检测方法，发现理解失败确实可以被识别，其中监督方法表现最好。在此基础上，我们提出了选择性翻译，这是一种简单而有效的策略，仅在检测到理解失败时才将多语言输入翻译成英语。实验结果表明，选择性翻译弥合了多语言推理差距，实现了接近完整的翻译性能，同时仅使用约 20% 的输入进行翻译。我们的工作共同表明，理解失败是多语言推理差距的主要原因，并且可以被检测到并有选择地缓解，从而提供对其起源的关键洞察，并为更公平的多语言推理提供一条有希望的道路。我们的代码和数据可通过此 https URL 公开获取。

Title: A Unified Representation Underlying the Judgment of Large Language Models

• Authors: Yi-Long Lu, Jiajun Song, Wei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27328
• Pdf URL: https://arxiv.org/pdf/2510.27328
• Copy Paste: [[2510.27328]] A Unified Representation Underlying the Judgment of Large Language Models(https://arxiv.org/abs/2510.27328)
• Keywords: language model, llm, hallucination
• Abstract: A central architectural question for both biological and artificial intelligence is whether judgment relies on specialized modules or a unified, domain-general resource. While the discovery of decodable neural representations for distinct concepts in Large Language Models (LLMs) has suggested a modular architecture, whether these representations are truly independent systems remains an open question. Here we provide evidence for a convergent architecture. Across a range of LLMs, we find that diverse evaluative judgments are computed along a dominant dimension, which we term the Valence-Assent Axis (VAA). This axis jointly encodes subjective valence ("what is good") and the model's assent to factual claims ("what is true"). Through direct interventions, we show this unified representation creates a critical dependency: the VAA functions as a control signal that steers the generative process to construct a rationale consistent with its evaluative state, even at the cost of factual accuracy. This mechanism, which we term the subordination of reasoning, shifts the process of reasoning from impartial inference toward goal-directed justification. Our discovery offers a mechanistic account for systemic bias and hallucination, revealing how an architecture that promotes coherent judgment can systematically undermine faithful reasoning.
• 摘要：生物智能和人工智能的一个核心架构问题是判断是否依赖于专门的模块或统一的、通用领域的资源。虽然大型语言模型（LLM）中不同概念的可解码神经表征的发现提出了模块化架构，但这些表征是否是真正独立的系统仍然是一个悬而未决的问题。在这里，我们提供了聚合架构的证据。在一系列法学硕士中，我们发现不同的评价判断是沿着一个主导维度计算的，我们将其称为效价-同意轴（VAA）。该轴联合编码主观效价（“什么是好的”）和模型对事实主张的同意（“什么是真的”）。通过直接干预，我们表明这种统一的表征产生了一种关键的依赖性：VAA 作为控制信号发挥作用，引导生成过程构建与其评估状态一致的基本原理，即使以牺牲事实准确性为代价。这种机制，我们称之为推理的从属性，将推理过程从公正的推理转向目标导向的论证。我们的发现为系统偏见和幻觉提供了机械解释，揭示了促进连贯判断的架构如何系统地破坏忠实推理。

Title: TransAlign: Machine Translation Encoders are Strong Word Aligners, Too

• Authors: Benedikt Ebing, Christian Goldschmied, Goran Glavaš
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27337
• Pdf URL: https://arxiv.org/pdf/2510.27337
• Copy Paste: [[2510.27337]] TransAlign: Machine Translation Encoders are Strong Word Aligners, Too(https://arxiv.org/abs/2510.27337)
• Keywords: language model
• Abstract: In the absence of sizable training data for most world languages and NLP tasks, translation-based strategies such as translate-test -- evaluating on noisy source language data translated from the target language -- and translate-train -- training on noisy target language data translated from the source language -- have been established as competitive approaches for cross-lingual transfer (XLT). For token classification tasks, these strategies require label projection: mapping the labels from each token in the original sentence to its counterpart(s) in the translation. To this end, it is common to leverage multilingual word aligners (WAs) derived from encoder language models such as mBERT or LaBSE. Despite obvious associations between machine translation (MT) and WA, research on extracting alignments with MT models is largely limited to exploiting cross-attention in encoder-decoder architectures, yielding poor WA results. In this work, in contrast, we propose TransAlign, a novel word aligner that utilizes the encoder of a massively multilingual MT model. We show that TransAlign not only achieves strong WA performance but substantially outperforms popular WA and state-of-the-art non-WA-based label projection methods in MT-based XLT for token classification.
• 摘要：由于大多数世界语言和 NLP 任务缺乏大量训练数据，基于翻译的策略，例如翻译测试（评估从目标语言翻译而来的嘈杂源语言数据）和翻译训练（对从源语言翻译而来的嘈杂目标语言数据进行训练）已被确立为跨语言迁移（XLT）的竞争方法。对于标记分类任务，这些策略需要标签投影：将原始句子中每个标记的标签映射到翻译中的对应标记。为此，通常利用从 mBERT 或 LaBSE 等编码器语言模型派生的多语言单词对齐器 (WA)。尽管机器翻译 (MT) 和 WA 之间存在明显的关联，但使用 MT 模型提取对齐的研究在很大程度上仅限于利用编码器-解码器架构中的交叉注意力，从而产生较差的 WA 结果。相比之下，在这项工作中，我们提出了 TransAlign，一种新颖的单词对齐器，它利用大规模多语言 MT 模型的编码器。我们表明，TransAlign 不仅实现了强大的 WA 性能，而且在基于 MT 的 XLT 标记分类中大大优于流行的 WA 和最先进的非基于 WA 的标签投影方法。

Title: ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations

• Authors: Zijian Wang, Chang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27355
• Pdf URL: https://arxiv.org/pdf/2510.27355
• Copy Paste: [[2510.27355]] ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations(https://arxiv.org/abs/2510.27355)
• Keywords: language model, llm
• Abstract: This paper introduces ThoughtProbe, a novel inference time framework that leverages the hidden reasoning features of Large Language Models (LLMs) to improve their reasoning performance. Unlike previous works that manipulate the hidden representations to steer LLM generation, we harness them as discriminative signals to guide the tree structured response space exploration. In each node expansion, a classifier serves as a scoring and ranking mechanism that efficiently allocates computational resources by prioritizing higher score candidates for continuation. After completing the tree expansion, we collect answers from all branches to form a candidate answer pool. We then propose a branch aggregation method that marginalizes over all supporting branches by aggregating their CoT scores, thereby identifying the optimal answer from the pool. Experimental results show that our framework's comprehensive exploration not only covers valid reasoning chains but also effectively identifies them, achieving significant improvements across multiple arithmetic reasoning benchmarks.
• 摘要：本文介绍了 ThoughtProbe，这是一种新颖的推理时间框架，它利用大型语言模型 (LLM) 的隐藏推理功能来提高其推理性能。与之前操纵隐藏表示来引导 LLM 生成的工作不同，我们利用它们作为判别信号来指导树结构响应空间探索。在每个节点扩展中，分类器充当评分和排名机制，通过优先考虑分数较高的候选者以进行继续，从而有效地分配计算资源。完成树扩展后，我们从所有分支收集答案，形成候选答案池。然后，我们提出了一种分支聚合方法，通过聚合所有支持分支的 CoT 分数来边缘化它们，从而从池中识别最佳答案。实验结果表明，我们的框架的全面探索不仅涵盖了有效的推理链，而且还有效地识别了它们，在多个算术推理基准上取得了显着的改进。

Title: From the Rock Floor to the Cloud: A Systematic Survey of State-of-the-Art NLP in Battery Life Cycle

• Authors: Tosin Adewumi, Martin Karlsson, Marcus Liwicki, Mikael Sjödahl, Lama Alkhaled, Rihab Gargouri, Nudrat Habib, Franz Hennie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27369
• Pdf URL: https://arxiv.org/pdf/2510.27369
• Copy Paste: [[2510.27369]] From the Rock Floor to the Cloud: A Systematic Survey of State-of-the-Art NLP in Battery Life Cycle(https://arxiv.org/abs/2510.27369)
• Keywords: prompt, agent
• Abstract: We present a comprehensive systematic survey of the application of natural language processing (NLP) along the entire battery life cycle, instead of one stage or method, and introduce a novel technical language processing (TLP) framework for the EU's proposed digital battery passport (DBP) and other general battery predictions. We follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method and employ three reputable databases or search engines, including Google Scholar, Institute of Electrical and Electronics Engineers Xplore (IEEE Xplore), and Scopus. Consequently, we assessed 274 scientific papers before the critical review of the final 66 relevant papers. We publicly provide artifacts of the review for validation and reproducibility. The findings show that new NLP tasks are emerging in the battery domain, which facilitate materials discovery and other stages of the life cycle. Notwithstanding, challenges remain, such as the lack of standard benchmarks. Our proposed TLP framework, which incorporates agentic AI and optimized prompts, will be apt for tackling some of the challenges.
• 摘要：我们对自然语言处理 (NLP) 在整个电池生命周期中的应用进行了全面的系统调查，而不是单一阶段或方法，并为欧盟提出的数字电池护照 (DBP) 和其他一般电池预测引入了一种新颖的技术语言处理 (TLP) 框架。我们遵循系统评价和荟萃分析的首选报告项目 (PRISMA) 方法，并采用三个信誉良好的数据库或搜索引擎，包括 Google Scholar、电气和电子工程师协会 Xplore (IEEE Xplore) 和 Scopus。因此，在对最终 66 篇相关论文进行严格审查之前，我们评估了 274 篇科学论文。我们公开提供审核的工件以进行验证和可重复性。研究结果表明，新的 NLP 任务正在电池领域出现，这有助于材料发现和生命周期的其他阶段。尽管如此，挑战仍然存在，例如缺乏标准基准。我们提出的 TLP 框架结合了代理人工智能和优化提示，将适合解决一些挑战。

Title: Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs

• Authors: Jiahao Liu, Zijian Wang, Kuo Zhao, Dong Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27400
• Pdf URL: https://arxiv.org/pdf/2510.27400
• Copy Paste: [[2510.27400]] Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs(https://arxiv.org/abs/2510.27400)
• Keywords: language model, llm
• Abstract: Knowledge editing has emerged as an efficient approach for updating factual knowledge in large language models (LLMs). It typically locates knowledge storage modules and then modifies their parameters. However, most existing methods focus on the weights of multilayer perceptron (MLP) modules, which are often identified as the main repositories of factual information. Other components, such as attention (Attn) modules, are often ignored during editing. This imbalance can leave residual outdated knowledge and limit editing effectiveness. We perform comprehensive knowledge localization experiments on advanced LLMs and find that Attn modules play a substantial role in factual knowledge storage and retrieval, especially in earlier layers. Based on these insights, we propose IntAttn-Edit, a method that extends the associative memory paradigm to jointly update both MLP and Attn modules. Our approach uses a knowledge balancing strategy that allocates update magnitudes in proportion to each module's measured contribution to knowledge storage. Experiments on standard benchmarks show that IntAttn-Edit achieves higher edit success, better generalization, and stronger knowledge preservation than prior methods. Further analysis shows that the balancing strategy keeps editing performance within an optimal range across diverse settings.
• 摘要：知识编辑已成为更新大型语言模型（LLM）中事实知识的有效方法。它通常定位知识存储模块，然后修改其参数。然而，大多数现有方法都关注多层感知器（MLP）模块的权重，这些模块通常被认为是事实信息的主要存储库。其他组件，例如注意力（Attn）模块，在编辑过程中经常被忽略。这种不平衡可能会留下残留的过时知识并限制编辑效率。我们对高级 LLM 进行了全面的知识本地化实验，发现 Attn 模块在事实知识存储和检索中发挥着重要作用，尤其是在早期层。基于这些见解，我们提出了 IntAttn-Edit，一种扩展关联记忆范式以联合更新 MLP 和 Attn 模块的方法。我们的方法使用知识平衡策略，根据每个模块对知识存储的测量贡献按比例分配更新幅度。标准基准测试的实验表明，与之前的方法相比，IntAttn-Edit 实现了更高的编辑成功率、更好的泛化性和更强的知识保存性。进一步的分析表明，平衡策略可以在不同的设置下使编辑性能保持在最佳范围内。

Title: Dynamic Affective Memory Management for Personalized LLM Agents

• Authors: Junfeng Lu, Yueyan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27418
• Pdf URL: https://arxiv.org/pdf/2510.27418
• Copy Paste: [[2510.27418]] Dynamic Affective Memory Management for Personalized LLM Agents(https://arxiv.org/abs/2510.27418)
• Keywords: language model, llm, agent
• Abstract: Advances in large language models are making personalized AI agents a new research focus. While current agent systems primarily rely on personalized external memory databases to deliver customized experiences, they face challenges such as memory redundancy, memory staleness, and poor memory-context integration, largely due to the lack of effective memory updates during interaction. To tackle these issues, we propose a new memory management system designed for affective scenarios. Our approach employs a Bayesian-inspired memory update algorithm with the concept of memory entropy, enabling the agent to autonomously maintain a dynamically updated memory vector database by minimizing global entropy to provide more personalized services. To better evaluate the system's effectiveness in this context, we propose DABench, a benchmark focusing on emotional expression and emotional change toward objects. Experimental results demonstrate that, our system achieves superior performance in personalization, logical coherence, and accuracy. Ablation studies further validate the effectiveness of the Bayesian-inspired update mechanism in alleviating memory bloat. Our work offers new insights into the design of long-term memory systems.
• 摘要：大语言模型的进步正在使个性化人工智能代理成为新的研究焦点。虽然当前的代理系统主要依靠个性化的外部内存数据库来提供定制体验，但它们面临着内存冗余、内存陈旧和内存上下文集成不佳等挑战，这很大程度上是由于交互过程中缺乏有效的内存更新。为了解决这些问题，我们提出了一种专为情感场景设计的新内存管理系统。我们的方法采用贝叶斯启发的内存更新算法和内存熵的概念，使代理能够通过最小化全局熵来自主维护动态更新的内存向量数据库，以提供更加个性化的服务。为了更好地评估系统在这种情况下的有效性，我们提出了 DABench，这是一个专注于对象的情感表达和情感变化的基准。实验结果表明，我们的系统在个性化、逻辑连贯性和准确性方面取得了优异的性能。消融研究进一步验证了贝叶斯启发的更新机制在缓解内存膨胀方面的有效性。我们的工作为长期记忆系统的设计提供了新的见解。

Title: VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision

• Authors: Xuan Gong, Senmiao Wang, Hanbo Huang, Ruoyu Sun, Shiyu Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27462
• Pdf URL: https://arxiv.org/pdf/2510.27462
• Copy Paste: [[2510.27462]] VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision(https://arxiv.org/abs/2510.27462)
• Keywords: language model, llm, chain-of-thought
• Abstract: Supervised fine-tuning (SFT) on long chain-of-thought (CoT) trajectories has emerged as a crucial technique for enhancing the reasoning abilities of large language models (LLMs). However, the standard cross-entropy loss treats all tokens equally, ignoring their heterogeneous contributions across a reasoning trajectory. This uniform treatment leads to misallocated supervision and weak generalization, especially in complex, long-form reasoning tasks. To address this, we introduce \textbf{V}ariance-\textbf{C}ontrolled \textbf{O}ptimization-based \textbf{RE}weighting (VCORE), a principled framework that reformulates CoT supervision as a constrained optimization problem. By adopting an optimization-theoretic perspective, VCORE enables a principled and adaptive allocation of supervision across tokens, thereby aligning the training objective more closely with the goal of robust reasoning generalization. Empirical evaluations demonstrate that VCORE consistently outperforms existing token reweighting methods. Across both in-domain and out-of-domain settings, VCORE achieves substantial performance gains on mathematical and coding benchmarks, using models from the Qwen3 series (4B, 8B, 32B) and LLaMA-3.1-8B-Instruct. Moreover, we show that VCORE serves as a more effective initialization for subsequent reinforcement learning, establishing a stronger foundation for advancing the reasoning capabilities of LLMs. The Code will be released at this https URL.
• 摘要：长思想链 (CoT) 轨迹上的监督微调 (SFT) 已成为增强大型语言模型 (LLM) 推理能力的关键技术。然而，标准交叉熵损失平等对待所有标记，忽略它们在推理轨迹上的异质贡献。这种统一的处理会导致监督分配不当和泛化能力弱，尤其是在复杂、长形式的推理任务中。为了解决这个问题，我们引入了\textbf{V}ariance-\textbf{C}控制的\textbf{O}基于优化的\textbf{RE}权重（VCORE），这是一个原则框架，将CoT监督重新表述为约束优化问题。通过采用优化理论的视角，VCORE 能够在代币之间进行有原则的、自适应的监督分配，从而使训练目标与鲁棒推理泛化的目标更加紧密地结合起来。实证评估表明 VCORE 始终优于现有的代币重新加权方法。在域内和域外设置中，VCORE 使用 Qwen3 系列（4B、8B、32B）和 LLaMA-3.1-8B-Instruct 的模型在数学和编码基准上实现了显着的性能提升。此外，我们表明 VCORE 可以为后续强化学习提供更有效的初始化，为提升法学硕士的推理能力奠定更坚实的基础。该代码将在此 https URL 发布。

Title: Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning

• Authors: Chenyang Shao, Sijian Ren, Fengli Xu, Yong Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27469
• Pdf URL: https://arxiv.org/pdf/2510.27469
• Copy Paste: [[2510.27469]] Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning(https://arxiv.org/abs/2510.27469)
• Keywords: language model, llm
• Abstract: In recent years, large language models (LLMs) have witnessed remarkable advancements, with the test-time scaling law consistently enhancing the reasoning capabilities. Through systematic evaluation and exploration of a diverse spectrum of intermediate thoughts, LLMs demonstrate the potential to generate deliberate reasoning steps, thereby substantially enhancing reasoning accuracy. However, LLMs' autoregressive generation paradigm results in reasoning performance scaling sub-optimally with test-time computation, often requiring excessive computational overhead to propose thoughts while yielding only marginal performance gains. In contrast, diffusion language models (DLMs) can efficiently produce diverse samples through parallel denoising in a single forward pass, inspiring us to leverage them for proposing intermediate thoughts, thereby alleviating the computational burden associated with autoregressive generation while maintaining quality. In this work, we propose an efficient collaborative reasoning framework, leveraging DLMs to generate candidate thoughts and LLMs to evaluate their quality. Experiments across diverse benchmarks demonstrate that our framework achieves strong performance in complex reasoning tasks, offering a promising direction for future research. Our code is open-source at this https URL.
• 摘要：近年来，大型语言模型（LLM）取得了显着的进步，测试时间缩放法则不断增强推理能力。通过对各种中间思想的系统评估和探索，法学硕士展示了产生深思熟虑的推理步骤的潜力，从而大大提高了推理的准确性。然而，法学硕士的自回归生成范式导致推理性能在测试时计算中扩展得不够理想，通常需要过多的计算开销来提出想法，同时只能产生边际性能增益。相比之下，扩散语言模型（DLM）可以通过在一次前向传递中并行去噪来有效地生成不同的样本，启发我们利用它们来提出中间想法，从而减轻与自回归生成相关的计算负担，同时保持质量。在这项工作中，我们提出了一个高效的协作推理框架，利用 DLM 生成候选想法，并利用 LLM 评估其质量。跨不同基准的实验表明，我们的框架在复杂的推理任务中取得了强劲的性能，为未来的研究提供了一个有希望的方向。我们的代码在此 https URL 上开源。

Title: Patient-Centered Summarization Framework for AI Clinical Summarization: A Mixed-Methods Design

• Authors: Maria Lizarazo Jimenez, Ana Gabriela Claros, Kieran Green, David Toro-Tobon, Felipe Larios, Sheena Asthana, Camila Wenczenovicz, Kerly Guevara Maldonado, Luis Vilatuna-Andrango, Cristina Proano-Velez, Satya Sai Sri Bandi, Shubhangi Bagewadi, Megan E. Branda, Misk Al Zahidy, Saturnino Luz, Mirella Lapata, Juan P. Brito, Oscar J. Ponce-Ponte
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27535
• Pdf URL: https://arxiv.org/pdf/2510.27535
• Copy Paste: [[2510.27535]] Patient-Centered Summarization Framework for AI Clinical Summarization: A Mixed-Methods Design(https://arxiv.org/abs/2510.27535)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are increasingly demonstrating the potential to reach human-level performance in generating clinical summaries from patient-clinician conversations. However, these summaries often focus on patients' biology rather than their preferences, values, wishes, and concerns. To achieve patient-centered care, we propose a new standard for Artificial Intelligence (AI) clinical summarization tasks: Patient-Centered Summaries (PCS). Our objective was to develop a framework to generate PCS that capture patient values and ensure clinical utility and to assess whether current open-source LLMs can achieve human-level performance in this task. We used a mixed-methods process. Two Patient and Public Involvement groups (10 patients and 8 clinicians) in the United Kingdom participated in semi-structured interviews exploring what personal and contextual information should be included in clinical summaries and how it should be structured for clinical use. Findings informed annotation guidelines used by eight clinicians to create gold-standard PCS from 88 atrial fibrillation consultations. Sixteen consultations were used to refine a prompt aligned with the guidelines. Five open-source LLMs (Llama-3.2-3B, Llama-3.1-8B, Mistral-8B, Gemma-3-4B, and Qwen3-8B) generated summaries for 72 consultations using zero-shot and few-shot prompting, evaluated with ROUGE-L, BERTScore, and qualitative metrics. Patients emphasized lifestyle routines, social support, recent stressors, and care values. Clinicians sought concise functional, psychosocial, and emotional context. The best zero-shot performance was achieved by Mistral-8B (ROUGE-L 0.189) and Llama-3.1-8B (BERTScore 0.673); the best few-shot by Llama-3.1-8B (ROUGE-L 0.206, BERTScore 0.683). Completeness and fluency were similar between experts and models, while correctness and patient-centeredness favored human PCS.
• 摘要：大型语言模型 (LLM) 越来越多地展示了在从患者与临床医生的对话中生成临床摘要方面达到人类水平性能的潜力。然而，这些总结通常侧重于患者的生物学而不是他们的偏好、价值观、愿望和担忧。为了实现以患者为中心的护理，我们提出了人工智能（AI）临床总结任务的新标准：以患者为中心的总结（PCS）。我们的目标是开发一个框架来生成 PCS，捕捉患者价值并确保临床实用性，并评估当前的开源法学硕士是否可以在此任务中实现人类水平的表现。我们使用了混合方法流程。英国的两个患者和公众参与小组（10 名患者和 8 名临床医生）参加了半结构化访谈，探讨临床摘要中应包含哪些个人和背景信息以及如何构建临床摘要以供临床使用。研究结果为 8 名临床医生根据 88 次房颤咨询创建金标准 PCS 所使用的注释指南提供了信息。经过十六次磋商，完善了符合指导方针的提示。五个开源法学硕士（Llama-3.2-3B、Llama-3.1-8B、Mistral-8B、Gemma-3-4B 和 Qwen3-8B）使用零样本和少样本提示生成了 72 次咨询的摘要，并使用 ROUGE-L、BERTScore 和定性指标进行评估。患者强调生活方式、社会支持、近期压力源和护理价值观。临床医生寻求简洁的功能、社会心理和情感背景。 Mistral-8B (ROUGE-L 0.189) 和 Llama-3.1-8B (BERTScore 0.673) 实现了最佳零样本性能； Llama-3.1-8B 的最佳少数镜头（ROUGE-L 0.206，BERTScore 0.683）。专家和模型之间的完整性和流畅性相似，而正确性和以患者为中心有利于人类 PCS。

Title: DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models

• Authors: Malik H. Altakrori, Nizar Habash, Abdelhakim Freihat, Younes Samih, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Teresa Lynn, Preslav Nakov, Alham Fikri Aji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.27543
• Pdf URL: https://arxiv.org/pdf/2510.27543
• Copy Paste: [[2510.27543]] DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models(https://arxiv.org/abs/2510.27543)
• Keywords: language model, llm
• Abstract: We present DialectalArabicMMLU, a new benchmark for evaluating the performance of large language models (LLMs) across Arabic dialects. While recently developed Arabic and multilingual benchmarks have advanced LLM evaluation for Modern Standard Arabic (MSA), dialectal varieties remain underrepresented despite their prevalence in everyday communication. DialectalArabicMMLU extends the MMLU-Redux framework through manual translation and adaptation of 3K multiple-choice question-answer pairs into five major dialects (Syrian, Egyptian, Emirati, Saudi, and Moroccan), yielding a total of 15K QA pairs across 32 academic and professional domains (22K QA pairs when also including English and MSA). The benchmark enables systematic assessment of LLM reasoning and comprehension beyond MSA, supporting both task-based and linguistic analysis. We evaluate 19 open-weight Arabic and multilingual LLMs (1B-13B parameters) and report substantial performance variation across dialects, revealing persistent gaps in dialectal generalization. DialectalArabicMMLU provides the first unified, human-curated resource for measuring dialectal understanding in Arabic, thus promoting more inclusive evaluation and future model development.
• 摘要：我们推出了 DialectalArabicMMLU，这是一个用于评估跨阿拉伯语方言的大型语言模型 (LLM) 性能的新基准。虽然最近制定的阿拉伯语和多语言基准已经推进了现代标准阿拉伯语（MSA）的法学硕士评估，但方言变体尽管在日常交流中很普遍，但代表性仍然不足。 DialectalArabicMMLU 通过手动翻译和将 3K 多项选择题答案对改编为五种主要方言（叙利亚语、埃及语、阿联酋语、沙特语和摩洛哥语），扩展了 MMLU-Redux 框架，在 32 个学术和专业领域产生了总共 15K 个 QA 对（22K 个 QA 对，还包括英语和 MSA）。该基准能够对 MSA 之外的 LLM 推理和理解进行系统评估，支持基于任务的分析和语言分析。我们评估了 19 个开放式阿拉伯语和多语言法学硕士（1B-13B 参数），并报告了不同方言之间的巨大性能差异，揭示了方言泛化方面持续存在的差距。 DialectalArabicMMLU 提供了第一个统一的、人工管理的资源，用于测量阿拉伯语方言理解，从而促进更具包容性的评估和未来模型的开发。

Title: Multilingual BERT language model for medical tasks: Evaluation on domain-specific adaptation and cross-linguality

• Authors: Yinghao Luo (1 and 2), Lang Zhou (1 and 2), Amrish Jhingoer (1 and 2), Klaske Vliegenthart Jongbloed (3 and 4), Carlijn Jordans (4), Ben Werkhoven (5), Tom Seinen (6), Erik van Mulligen (6), Casper Rokx (3 and 4), Yunlei Li (1) ((1) Department of Pathology & Clinical Bioinformatics, Erasmus University Medical Center Rotterdam, (2) Department of Computer Science, Vrije Universiteit Amsterdam, (3) Department of Internal Medicine, Erasmus University Medical Center Rotterdam, (4) Department of Medical Microbiology and Infectious Diseases, Erasmus University Medical Center Rotterdam, (5) Department of Data and Analytics, Erasmus University Medical Center Rotterdam, (6) Department of Medical Informatics, Erasmus University Medical Center Rotterdam)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27552
• Pdf URL: https://arxiv.org/pdf/2510.27552
• Copy Paste: [[2510.27552]] Multilingual BERT language model for medical tasks: Evaluation on domain-specific adaptation and cross-linguality(https://arxiv.org/abs/2510.27552)
• Keywords: language model
• Abstract: In multilingual healthcare applications, the availability of domain-specific natural language processing(NLP) tools is limited, especially for low-resource languages. Although multilingual bidirectional encoder representations from transformers (BERT) offers a promising motivation to mitigate the language gap, the medical NLP tasks in low-resource languages are still underexplored. Therefore, this study investigates how further pre-training on domain-specific corpora affects model performance on medical tasks, focusing on three languages: Dutch, Romanian and Spanish. In terms of further pre-training, we conducted four experiments to create medical domain models. Then, these models were fine-tuned on three downstream tasks: Automated patient screening in Dutch clinical notes, named entity recognition in Romanian and Spanish clinical notes. Results show that domain adaptation significantly enhanced task performance. Furthermore, further differentiation of domains, e.g. clinical and general biomedical domains, resulted in diverse performances. The clinical domain-adapted model outperformed the more general biomedical domain-adapted model. Moreover, we observed evidence of cross-lingual transferability. Moreover, we also conducted further investigations to explore potential reasons contributing to these performance differences. These findings highlight the feasibility of domain adaptation and cross-lingual ability in medical NLP. Within the low-resource language settings, these findings can provide meaningful guidance for developing multilingual medical NLP systems to mitigate the lack of training data and thereby improve the model performance.
• 摘要：在多语言医疗保健应用中，特定领域的自然语言处理 (NLP) 工具的可用性是有限的，特别是对于资源匮乏的语言。尽管来自 Transformer 的多语言双向编码器表示 (BERT) 为缩小语言差距提供了有希望的动力，但低资源语言的医学 NLP 任务仍未得到充分探索。因此，本研究调查了特定领域语料库的进一步预训练如何影响医疗任务的模型性能，重点关注三种语言：荷兰语、罗马尼亚语和西班牙语。在进一步的预训练方面，我们进行了四次实验来创建医学领域模型。然后，这些模型在三个下游任务上进行了微调：荷兰语临床记录中的自动患者筛查、罗马尼亚语和西班牙语临床记录中的命名实体识别。结果表明，领域适应显着提高了任务绩效。此外，域的进一步区分，例如临床和一般生物医学领域，导致了不同的表现。临床领域适应模型优于更一般的生物医学领域适应模型。此外，我们观察到跨语言可迁移性的证据。此外，我们还进行了进一步的调查，以探讨造成这些性能差异的潜在原因。这些发现凸显了领域适应和跨语言能力在医学 NLP 中的可行性。在资源匮乏的语言环境中，这些发现可以为开发多语言医学 NLP 系统提供有意义的指导，以缓解训练数据的缺乏，从而提高模型性能。

Title: Data-Efficient Domain Adaptation for LLM-based MT using Contrastive Preference Optimization

• Authors: Inacio Vieira, Antonio Castaldo, James O'Doherty, Sheila Castilho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27556
• Pdf URL: https://arxiv.org/pdf/2510.27556
• Copy Paste: [[2510.27556]] Data-Efficient Domain Adaptation for LLM-based MT using Contrastive Preference Optimization(https://arxiv.org/abs/2510.27556)
• Keywords: llm
• Abstract: LLMs often require adaptation to domain-specific requirements, a process that can be expensive when relying solely on SFT. We present an empirical study on applying CPO to simulate a post-editing workflow for data-efficient domain adaptation. Our approach synthesizes preference pairs by treating the base model's own raw output as the 'rejected' translation and the human-approved TM entry as the 'chosen' one. This method provides direct feedback on the model's current knowledge, guiding it to align with domain-specific standards. Experiments in English-Brazilian Portuguese and English-Korean show that, by using just 14.7k preference pairs, the model achieves performance close to that of a model trained on 160k+ samples with SFT, demonstrating significant data efficiency. Although we showcase its effectiveness in MT, this application of CPO naturally generalizes to other generative tasks where a model's initial drafts can serve as a contrastive signal against a golden reference.
• 摘要：法学硕士通常需要适应特定领域的要求，如果仅依靠 SFT，这一过程的成本可能会很高。我们提出了一项关于应用 CPO 来模拟后期编辑工作流程以实现数据高效领域适应的实证研究。我们的方法通过将基本模型自身的原始输出视为“拒绝”翻译，将人类批准的 TM 条目视为“选择”翻译来合成偏好对。此方法提供有关模型当前知识的直接反馈，指导其与特定领域的标准保持一致。英语-巴西葡萄牙语和英语-韩语实验表明，仅使用 14.7k 个偏好对，该模型的性能就接近于使用 SFT 训练超过 160k 个样本的模型，展示了显着的数据效率。尽管我们展示了它在 MT 中的有效性，但 CPO 的这种应用自然可以推广到其他生成任务，其中模型的初始草稿可以作为与黄金参考的对比信号。

Title: MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval

• Authors: Qi Luo, Xiaonan Li, Yuxin Wang, Tingshuo Fan, Yuan Li, Xinchi Chen, Xipeng Qiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27569
• Pdf URL: https://arxiv.org/pdf/2510.27569
• Copy Paste: [[2510.27569]] MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval(https://arxiv.org/abs/2510.27569)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: Large Language Models (LLMs) excel at reasoning and generation but are inherently limited by static pretraining data, resulting in factual inaccuracies and weak adaptability to new information. Retrieval-Augmented Generation (RAG) addresses this issue by grounding LLMs in external knowledge; However, the effectiveness of RAG critically depends on whether the model can adequately access relevant information. Existing RAG systems rely on a single retriever with fixed top-k selection, restricting access to a narrow and static subset of the corpus. As a result, this single-retriever paradigm has become the primary bottleneck for comprehensive external information acquisition, especially in tasks requiring corpus-level reasoning. To overcome this limitation, we propose MARAG-R1, a reinforcement-learned multi-tool RAG framework that enables LLMs to dynamically coordinate multiple retrieval mechanisms for broader and more precise information access. MARAG-R1 equips the model with four retrieval tools -- semantic search, keyword search, filtering, and aggregation -- and learns both how and when to use them through a two-stage training process: supervised fine-tuning followed by reinforcement learning. This design allows the model to interleave reasoning and retrieval, progressively gathering sufficient evidence for corpus-level synthesis. Experiments on GlobalQA, HotpotQA, and 2WikiMultiHopQA demonstrate that MARAG-R1 substantially outperforms strong baselines and achieves new state-of-the-art results in corpus-level reasoning tasks.
• 摘要：大型语言模型（LLM）擅长推理和生成，但本质上受到静态预训练数据的限制，导致事实不准确且对新信息的适应性较弱。检索增强生成（RAG）通过让法学硕士扎根于外部知识来解决这个问题；然而，RAG 的有效性关键取决于模型是否能够充分访问相关信息。现有的 RAG 系统依赖于具有固定 top-k 选择的单个检索器，限制对语料库的狭窄静态子集的访问。因此，这种单检索器范式已成为全面外部信息获取的主要瓶颈，特别是在需要语料库级推理的任务中。为了克服这一限制，我们提出了 MARAG-R1，这是一种强化学习的多工具 RAG 框架，使法学硕士能够动态协调多种检索机制，以实现更广泛、更精确的信息访问。 MARAG-R1 为模型配备了四种检索工具——语义搜索、关键词搜索、过滤和聚合——并通过两阶段训练过程学习如何以及何时使用它们：监督微调和强化学习。这种设计允许模型将推理和检索交织在一起，逐步收集足够的证据以进行语料库级别的合成。 GlobalQA、HotpotQA 和 2WikiMultiHopQA 上的实验表明，MARAG-R1 的性能大大优于强大的基线，并在语料库级推理任务中取得了新的最先进的结果。

Title: SpecAttn: Speculating Sparse Attention

• Authors: Harsh Shah
• Subjects: cs.CL, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2510.27641
• Pdf URL: https://arxiv.org/pdf/2510.27641
• Copy Paste: [[2510.27641]] SpecAttn: Speculating Sparse Attention(https://arxiv.org/abs/2510.27641)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) face significant computational bottlenecks during inference due to the quadratic complexity of self-attention mechanisms, particularly as context lengths increase. We introduce SpecAttn, a novel training-free approach that seamlessly integrates with existing speculative decoding techniques to enable efficient sparse attention in pre-trained transformers. Our key insight is to exploit the attention weights already computed by the draft model during speculative decoding to identify important tokens for the target model, eliminating redundant computation while maintaining output quality. SpecAttn employs three core techniques: KL divergence-based layer alignment between draft and target models, a GPU-optimized sorting-free algorithm for top-p token selection from draft attention patterns, and dynamic key-value cache pruning guided by these predictions. By leveraging the computational work already performed in standard speculative decoding pipelines, SpecAttn achieves over 75% reduction in key-value cache accesses with a mere 15.29% increase in perplexity on the PG-19 dataset, significantly outperforming existing sparse attention methods. Our approach demonstrates that speculative execution can be enhanced to provide approximate verification without significant performance degradation.
• 摘要：由于自注意力机制的二次复杂度，大型语言模型（LLM）在推理过程中面临着严重的计算瓶颈，特别是随着上下文长度的增加。我们引入了 SpecAttn，这是一种新颖的免训练方法，它与现有的推测解码技术无缝集成，以在预训练的 Transformer 中实现高效的稀疏注意力。我们的主要见解是利用草稿模型在推测解码过程中已经计算出的注意力权重来识别目标模型的重要标记，消除冗余计算，同时保持输出质量。 SpecAttn 采用三种核心技术：草稿模型和目标模型之间基于 KL 散度的层对齐、用于从草稿注意模式中选择 top-p 标记的 GPU 优化的无排序算法，以及由这些预测引导的动态键值缓存修剪。通过利用标准推测解码管道中已经执行的计算工作，SpecAttn 在 PG-19 数据集上的键值缓存访问量减少了 75% 以上，而困惑度仅增加了 15.29%，显着优于现有的稀疏注意力方法。我们的方法表明，可以增强推测执行以提供近似验证，而不会显着降低性能。

Title: Culture Cartography: Mapping the Landscape of Cultural Knowledge

• Authors: Caleb Ziems, William Held, Jane Yu, Amir Goldberg, David Grusky, Diyi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.27672
• Pdf URL: https://arxiv.org/pdf/2510.27672
• Copy Paste: [[2510.27672]] Culture Cartography: Mapping the Landscape of Cultural Knowledge(https://arxiv.org/abs/2510.27672)
• Keywords: gpt, llm
• Abstract: To serve global users safely and productively, LLMs need culture-specific knowledge that might not be learned during pre-training. How do we find such knowledge that is (1) salient to in-group users, but (2) unknown to LLMs? The most common solutions are single-initiative: either researchers define challenging questions that users passively answer (traditional annotation), or users actively produce data that researchers structure as benchmarks (knowledge extraction). The process would benefit from mixed-initiative collaboration, where users guide the process to meaningfully reflect their cultures, and LLMs steer the process towards more challenging questions that meet the researcher's goals. We propose a mixed-initiative methodology called CultureCartography. Here, an LLM initializes annotation with questions for which it has low-confidence answers, making explicit both its prior knowledge and the gaps therein. This allows a human respondent to fill these gaps and steer the model towards salient topics through direct edits. We implement this methodology as a tool called CultureExplorer. Compared to a baseline where humans answer LLM-proposed questions, we find that CultureExplorer more effectively produces knowledge that leading models like DeepSeek R1 and GPT-4o are missing, even with web search. Fine-tuning on this data boosts the accuracy of Llama-3.1-8B by up to 19.2% on related culture benchmarks.
• 摘要：为了安全、高效地为全球用户提供服务，法学硕士需要在预培训期间可能无法学到的文化特定知识。我们如何找到这样的知识：（1）对于组内用户来说很重要，但（2）对于法学硕士来说是未知的？最常见的解决方案是单一主动的：要么研究人员定义用户被动回答的挑战性问题（传统注释），要么用户主动生成研究人员构建为基准的数据（知识提取）。该过程将受益于混合主动合作，用户指导该过程有意义地反映他们的文化，而法学硕士则引导该过程解决更具挑战性的问题，以满足研究人员的目标。我们提出了一种称为文化制图学的混合倡议方法。在这里，法学硕士用其低置信度答案的问题来初始化注释，明确其先验知识和其中的差距。这使得人类受访者能够填补这些空白，并通过直接编辑引导模型转向突出主题。我们将这种方法作为一个名为 CultureExplorer 的工具来实现。与人类回答 LLM 提出的问题的基线相比，我们发现 CultureExplorer 更有效地产生 DeepSeek R1 和 GPT-4o 等领先模型所缺少的知识，即使是通过网络搜索也是如此。对这些数据进行微调可将 Llama-3.1-8B 在相关文化基准上的准确性提高高达 19.2%。

Title: Continuous Autoregressive Language Models

• Authors: Chenze Shao, Darren Li, Fandong Meng, Jie Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.27688
• Pdf URL: https://arxiv.org/pdf/2510.27688
• Copy Paste: [[2510.27688]] Continuous Autoregressive Language Models(https://arxiv.org/abs/2510.27688)
• Keywords: language model, llm
• Abstract: The efficiency of large language models (LLMs) is fundamentally limited by their sequential, token-by-token generation process. We argue that overcoming this bottleneck requires a new design axis for LLM scaling: increasing the semantic bandwidth of each generative step. To this end, we introduce Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9\% accuracy. This allows us to model language as a sequence of continuous vectors instead of discrete tokens, which reduces the number of generative steps by a factor of K. The paradigm shift necessitates a new modeling toolkit; therefore, we develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling in the continuous domain. Experiments show that CALM significantly improves the performance-compute trade-off, achieving the performance of strong discrete baselines at a significantly lower computational cost. More importantly, these findings establish next-vector prediction as a powerful and scalable pathway towards ultra-efficient language models. Code: this https URL. Project: this https URL.
• 摘要：大型语言模型 (LLM) 的效率从根本上受到其顺序、逐个标记生成过程的限制。我们认为，克服这一瓶颈需要 LLM 扩展的新设计轴：增加每个生成步骤的语义带宽。为此，我们引入连续自回归语言模型（CALM），这是从离散下一个标记预测到连续下一个向量预测的范式转变。 CALM 使用高保真自动编码器将 K 个标记块压缩为单个连续向量，从中可以以超过 99.9% 的准确度重建原始标记。这使我们能够将语言建模为连续向量序列而不是离散标记，从而将生成步骤的数量减少了 K 倍。范式转变需要一个新的建模工具包；因此，我们开发了一个全面的无似然框架，可以在连续域中实现稳健的训练、评估和可控采样。实验表明，CALM 显着改善了性能与计算的权衡，以显着降低的计算成本实现了强离散基线的性能。更重要的是，这些发现将下一个向量预测确立为实现超高效语言模型的强大且可扩展的途径。代码：此 https URL。项目：此 https URL。