2025-10-22

Title: Modeling Layered Consciousness with Multi-Agent Large Language Models

• Authors: Sang Hun Kim, Jongmin Lee, Dongkyu Park, So Young Lee, Yosep Chong
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2510.17844
• Pdf URL: https://arxiv.org/pdf/2510.17844
• Copy Paste: [[2510.17844]] Modeling Layered Consciousness with Multi-Agent Large Language Models(https://arxiv.org/abs/2510.17844)
• Keywords: language model, llm, agent
• Abstract: We propose a multi-agent framework for modeling artificial consciousness in large language models (LLMs), grounded in psychoanalytic theory. Our \textbf{Psychodynamic Model} simulates self-awareness, preconsciousness, and unconsciousness through agent interaction, guided by a Personalization Module combining fixed traits and dynamic needs. Using parameter-efficient fine-tuning on emotionally rich dialogues, the system was evaluated across eight personalized conditions. An LLM as a judge approach showed a 71.2\% preference for the fine-tuned model, with improved emotional depth and reduced output variance, demonstrating its potential for adaptive, personalized cognition.
• 摘要：我们提出了一个基于精神分析理论的多智能体框架，用于在大语言模型（LLM）中建模人工意识。我们的 \textbf{心理动力学模型} 在结合固定特征和动态需求的个性化模块的指导下，通过代理交互模拟自我意识、前意识和无意识。通过对情感丰富的对话进行参数有效的微调，系统在八种个性化条件下进行了评估。以法学硕士作为评判方法，71.2% 的人倾向于微调模型，该模型具有改善的情感深度和减少的输出方差，展示了其自适应、个性化认知的潜力。

Title: Outraged AI: Large language models prioritise emotion over cost in fairness enforcement

• Authors: Hao Liu, Yiqing Dai, Haotian Tan, Yu Lei, Yujia Zhou, Zhen Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17880
• Pdf URL: https://arxiv.org/pdf/2510.17880
• Copy Paste: [[2510.17880]] Outraged AI: Large language models prioritise emotion over cost in fairness enforcement(https://arxiv.org/abs/2510.17880)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Emotions guide human decisions, but whether large language models (LLMs) use emotion similarly remains unknown. We tested this using altruistic third-party punishment, where an observer incurs a personal cost to enforce fairness, a hallmark of human morality and often driven by negative emotion. In a large-scale comparison of 4,068 LLM agents with 1,159 adults across 796,100 decisions, LLMs used emotion to guide punishment, sometimes even more strongly than humans did: Unfairness elicited stronger negative emotion that led to more punishment; punishing unfairness produced more positive emotion than accepting; and critically, prompting self-reports of emotion causally increased punishment. However, mechanisms diverged: LLMs prioritized emotion over cost, enforcing norms in an almost all-or-none manner with reduced cost sensitivity, whereas humans balanced fairness and cost. Notably, reasoning models (o3-mini, DeepSeek-R1) were more cost-sensitive and closer to human behavior than foundation models (GPT-3.5, DeepSeek-V3), yet remained heavily emotion-driven. These findings provide the first causal evidence of emotion-guided moral decisions in LLMs and reveal deficits in cost calibration and nuanced fairness judgements, reminiscent of early-stage human responses. We propose that LLMs progress along a trajectory paralleling human development; future models should integrate emotion with context-sensitive reasoning to achieve human-like emotional intelligence.
• 摘要：情感指导人类决策，但大型语言模型 (LLM) 是否同样使用情感仍然未知。我们使用利他的第三方惩罚来测试这一点，观察者为执行公平而付出个人成本，公平是人类道德的标志，并且常常受到负面情绪的驱动。在对 4,068 名法学硕士代理人与 1,159 名成年人进行的 796,100 项决策的大规模比较中，法学硕士使用情绪来指导惩罚，有时甚至比人类更强烈：不公平会引发更强烈的负面情绪，从而导致更多的惩罚；惩罚不公平比接受不公平产生更积极的情绪；更重要的是，促使自我报告情绪会导致惩罚的增加。然而，机制存在分歧：法学硕士优先考虑情感而非成本，以几乎全有或全无的方式执行规范，降低成本敏感性，而人类则平衡公平和成本。值得注意的是，推理模型（o3-mini、DeepSeek-R1）比基础模型（GPT-3.5、DeepSeek-V3）对成本更加敏感，更接近人类行为，但仍然严重受情感驱动。这些发现为法学硕士中情绪引导的道德决策提供了第一个因果证据，并揭示了成本校准和细致入微的公平判断方面的缺陷，让人想起早期人类的反应。我们建议法学硕士沿着与人类发展平行的轨迹前进；未来的模型应该将情感与情境敏感推理相结合，以实现类似人类的情商。

Title: POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

• Authors: Yizhuo Chen, Xin Liu, Ruijie Wang, Zheng Li, Pei Chen, Changlong Yu, Priyanka Nigam, Meng Jiang, Bing Yin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17881
• Pdf URL: https://arxiv.org/pdf/2510.17881
• Copy Paste: [[2510.17881]] POPI: Personalizing LLMs via Optimized Natural Language Preference Inference(https://arxiv.org/abs/2510.17881)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) achieve strong benchmark performance, yet user experiences remain inconsistent due to diverse preferences in style, tone, and reasoning mode. Nevertheless, existing alignment techniques such as reinforcement learning from human feedback (RLHF) or Direct Preference Optimization (DPO) largely optimize toward population-level averages and overlook individual variation. Naive personalization strategies like per-user fine-tuning are computationally prohibitive, and in-context approaches that prepend raw user signals often suffer from inefficiency and noise. To address these challenges, we propose POPI, a general framework that introduces a preference inference model to distill heterogeneous user signals into concise natural language summaries. These summaries act as transparent, compact, and transferable personalization representations that condition a shared generation model to produce personalized responses. POPI jointly optimizes both preference inference and personalized generation under a unified objective using reinforcement learning, ensuring summaries maximally encode useful preference information. Extensive experiments across four personalization benchmarks demonstrate that POPI consistently improves personalization accuracy while reducing context overhead by a large margin. Moreover, optimized summaries seamlessly transfer to frozen off-the-shelf LLMs, enabling plug-and-play personalization without weight updates.
• 摘要：大型语言模型 (LLM) 实现了强大的基准性能，但由于风格、语气和推理模式的偏好不同，用户体验仍然不一致。然而，现有的对齐技术，例如来自人类反馈的强化学习（RLHF）或直接偏好优化（DPO），很大程度上针对群体水平的平均值进行优化，并忽略了个体差异。像针对每个用户进行微调这样简单的个性化策略在计算上是令人望而却步的，而预先考虑原始用户信号的上下文方法通常效率低下且存在噪音。为了应对这些挑战，我们提出了 POPI，这是一个通用框架，它引入了偏好推理模型，将异构用户信号提炼成简洁的自然语言摘要。这些摘要充当透明、紧凑且可转移的个性化表示，调节共享生成模型以产生个性化响应。 POPI 使用强化学习在统一目标下联合优化偏好推断和个性化生成，确保摘要最大限度地编码有用的偏好信息。跨越四个个性化基准的广泛实验表明，POPI 持续提高个性化准确性，同时大幅减少上下文开销。此外，优化的摘要可以无缝转移到冻结的现成法学硕士，从而实现即插即用的个性化，无需更新体重。

Title: Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review

• Authors: Zhyar Rzgar K. Rostam, Gábor Kertész
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.17892
• Pdf URL: https://arxiv.org/pdf/2510.17892
• Copy Paste: [[2510.17892]] Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review(https://arxiv.org/abs/2510.17892)
• Keywords: language model, llm
• Abstract: The exponential increase in scientific literature and online information necessitates efficient methods for extracting knowledge from textual data. Natural language processing (NLP) plays a crucial role in addressing this challenge, particularly in text classification tasks. While large language models (LLMs) have achieved remarkable success in NLP, their accuracy can suffer in domain-specific contexts due to specialized vocabulary, unique grammatical structures, and imbalanced data distributions. In this systematic literature review (SLR), we investigate the utilization of pre-trained language models (PLMs) for domain-specific text classification. We systematically review 41 articles published between 2018 and January 2024, adhering to the PRISMA statement (preferred reporting items for systematic reviews and meta-analyses). This review methodology involved rigorous inclusion criteria and a multi-step selection process employing AI-powered tools. We delve into the evolution of text classification techniques and differentiate between traditional and modern approaches. We emphasize transformer-based models and explore the challenges and considerations associated with using LLMs for domain-specific text classification. Furthermore, we categorize existing research based on various PLMs and propose a taxonomy of techniques used in the field. To validate our findings, we conducted a comparative experiment involving BERT, SciBERT, and BioBERT in biomedical sentence classification. Finally, we present a comparative study on the performance of LLMs in text classification tasks across different domains. In addition, we examine recent advancements in PLMs for domain-specific text classification and offer insights into future directions and limitations in this rapidly evolving domain.
• 摘要：科学文献和在线信息的指数级增长需要有效的方法从文本数据中提取知识。自然语言处理（NLP）在应对这一挑战中发挥着至关重要的作用，特别是在文本分类任务中。虽然大型语言模型 (LLM) 在 NLP 领域取得了显着的成功，但由于专门的词汇、独特的语法结构和不平衡的数据分布，它们的准确性在特定领域的上下文中可能会受到影响。在这篇系统文献综述（SLR）中，我们研究了预训练语言模型（PLM）在特定领域文本分类中的应用。我们遵循 PRISMA 声明（系统评价和荟萃分析的首选报告项目），系统地评价了 2018 年至 2024 年 1 月期间发表的 41 篇文章。这种审查方法涉及严格的纳入标准和采用人工智能工具的多步骤选择过程。我们深入研究文本分类技术的演变，并区分传统方法和现代方法。我们强调基于 Transformer 的模型，并探讨与使用 LLM 进行特定领域文本分类相关的挑战和注意事项。此外，我们根据各种 PLM 对现有研究进行分类，并提出该领域使用的技术的分类法。为了验证我们的发现，我们在生物医学句子分类中进行了 BERT、SciBERT 和 BioBERT 的比较实验。最后，我们对法学硕士在不同领域的文本分类任务中的表现进行了比较研究。此外，我们还研究了 PLM 在特定领域文本分类方面的最新进展，并深入了解这个快速发展的领域的未来方向和局限性。

Title: Atomic Literary Styling: Mechanistic Manipulation of Prose Generation in Neural Language Models

• Authors: Tsogt-Ochir Enkhbayar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.17909
• Pdf URL: https://arxiv.org/pdf/2510.17909
• Copy Paste: [[2510.17909]] Atomic Literary Styling: Mechanistic Manipulation of Prose Generation in Neural Language Models(https://arxiv.org/abs/2510.17909)
• Keywords: language model, gpt
• Abstract: We present a mechanistic analysis of literary style in GPT-2, identifying individual neurons that discriminate between exemplary prose and rigid AI-generated text. Using Herman Melville's Bartleby, the Scrivener as a corpus, we extract activation patterns from 355 million parameters across 32,768 neurons in late layers. We find 27,122 statistically significant discriminative neurons ($p < 0.05$), with effect sizes up to $|d| = 1.4$. Through systematic ablation studies, we discover a paradoxical result: while these neurons correlate with literary text during analysis, removing them often improves rather than degrades generated prose quality. Specifically, ablating 50 high-discriminating neurons yields a 25.7% improvement in literary style metrics. This demonstrates a critical gap between observational correlation and causal necessity in neural networks. Our findings challenge the assumption that neurons which activate on desirable inputs will produce those outputs during generation, with implications for mechanistic interpretability research and AI alignment.
• 摘要：我们对 GPT-2 中的文学风格进行了机械分析，识别出区分典范散文和严格的人工智能生成文本的单个神经元。使用 Herman Melville 的 Bartleby（Scrivener）作为语料库，我们从后期层 32,768 个神经元的 3.55 亿个参数中提取激活模式。我们发现 27,122 个统计上显着的判别神经元 ($p < 0.05$)，效应大小高达 $|d| = 1.4 美元。通过系统的消融研究，我们发现了一个矛盾的结果：虽然这些神经元在分析过程中与文学文本相关，但删除它们通常会提高而不是降低生成的散文质量。具体来说，消除 50 个高辨别神经元可使文学风格指标提高 25.7%。这表明神经网络中观察相关性和因果必然性之间存在重大差距。我们的研究结果挑战了这样一种假设，即根据所需输入激活的神经元将在生成过程中产生这些输出，这对机械可解释性研究和人工智能对齐具有影响。

Title: JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMs

• Authors: Junlan Feng, Fanyu Meng, Chong Long, Pengyu Cong, Duqing Wang, Yan Zheng, Yuyao Zhang, Xuanchang Gao, Ye Yuan, Yunfei Ma, Zhijie Ren, Fan Yang, Na Wu, Di Jin, Chao Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17918
• Pdf URL: https://arxiv.org/pdf/2510.17918
• Copy Paste: [[2510.17918]] JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMs(https://arxiv.org/abs/2510.17918)
• Keywords: language model, llm, hallucination
• Abstract: The hallucination and credibility concerns of large language models (LLMs) are global challenges that the industry is collectively addressing. Recently, a significant amount of advances have been made on post-training and inference techniques to mitigate these challenges. However, it is widely agreed that unsafe and hallucinations of LLMs intrinsically originate from pre-training, involving pre-training data and the next-token prediction learning mechanism. In this paper, we focus on enhancing pre-training data to improve the trustworthiness and safety of LLMs. Since the data is vast, it's almost impossible to entirely purge the data of factual errors, logical inconsistencies, or distributional biases. Moreover, the pre-training data lack grounding in real-world knowledge. Each piece of data is treated as a sequence of tokens rather than as a representation of a part of the world. To overcome these issues, we propose approaches to enhancing our pre-training data with its context in the world and increasing a substantial amount of data reflecting industrial scenarios. We argue that most source data are created by the authors for specific purposes in a certain spatial-temporal context. They have played a role in the real world. By incorporating related world context information, we aim to better anchor pre-training data within real-world scenarios, thereby reducing uncertainty in model training and enhancing the model's safety and trustworthiness. We refer to our Data with World Context as DWC. We continue pre-training an earlier checkpoint of JT-35B-Base with 1.5 trillion of DWC tokens. We introduce our post-training procedures to activate the potentials of DWC. Compared with the Qwen model of a similar scale, JT-Safe-35B achieves an average performance improvement of 1.79% on the Safety and Trustworthy evaluation benchmarks, while being pretrained with only 6.2 trillion tokens.
• 摘要：大语言模型（LLM）的幻觉和可信度问题是业界正在共同解决的全球挑战。最近，在后训练和推理技术方面取得了显着的进展，以缓解这些挑战。然而，人们普遍认为LLM的不安全和幻觉本质上源于预训练，涉及预训练数据和下一个token预测学习机制。在本文中，我们重点关注增强预训练数据以提高法学硕士的可信度和安全性。由于数据庞大，几乎不可能完全清除数据中的事实错误、逻辑不一致或分布偏差。此外，预训练数据缺乏现实世界知识的基础。每条数据都被视为令牌序列，而不是世界一部分的表示。为了克服这些问题，我们提出了一些方法，根据世界背景增强我们的预训练数据，并增加大量反映工业场景的数据。我们认为，大多数源数据是作者在特定时空背景下出于特定目的而创建的。他们在现实世界中发挥了作用。通过整合相关的世界上下文信息，我们的目标是更好地将预训练数据锚定在现实场景中，从而减少模型训练的不确定性，增强模型的安全性和可信度。我们将具有世界背景的数据称为 DWC。我们继续使用 1.5 万亿个 DWC 代币对 JT-35B-Base 的早期检查点进行预训练。我们引入培训后程序来激活 DWC 的潜力。与类似规模的 Qwen 模型相比，JT-Safe-35B 在仅使用 6.2 万亿代币进行预训练的情况下，在安全可信评估基准上实现了 1.79% 的平均性能提升。

Title: CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections

• Authors: Keuntae Kim, Eunhye Jeong, Sehyeon Lee, Seohee Yoon, Yong Suk Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17921
• Pdf URL: https://arxiv.org/pdf/2510.17921
• Copy Paste: [[2510.17921]] CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections(https://arxiv.org/abs/2510.17921)
• Keywords: language model, llm, prompt
• Abstract: Recent advances in enhancing the reasoning ability of large language models (LLMs) have been remarkably successful. LLMs trained with reinforcement learning (RL) for reasoning demonstrate strong performance in challenging tasks such as mathematics and coding, even with relatively small model sizes. However, despite these improvements in task accuracy, the assessment of creativity in LLM generations has been largely overlooked in reasoning tasks, in contrast to writing tasks. The lack of research on creativity assessment in reasoning primarily stems from two challenges: (1) the difficulty of defining the range of creativity, and (2) the necessity of human evaluation in the assessment process. To address these challenges, we propose CLAWS, a method that defines and classifies mathematical solutions into typical, creative, and hallucinated categories without human evaluation, by leveraging attention weights across prompt sections and output. CLAWS outperforms five existing white-box detection methods (Perplexity, Logit Entropy, Window Entropy, Hidden Score, and Attention Score) on five 7-8B math RL models (DeepSeek, Qwen, Mathstral, OpenMath2, and Oreal). We validate CLAWS on 4545 math problems collected from 181 math contests (AJHSME, AMC, AIME).
• 摘要：最近在增强大型语言模型（LLM）推理能力方面取得的进展非常成功。通过强化学习 (RL) 进行推理训练的法学硕士在数学和编码等具有挑战性的任务中表现出强大的性能，即使模型规模相对较小。然而，尽管任务准确性有所提高，但与写作任务相比，法学硕士世代的创造力评估在推理任务中很大程度上被忽视了。推理创造力评估研究的缺乏主要源于两个挑战：（1）创造力范围难以界定；（2）评估过程中人类评估的必要性。为了应对这些挑战，我们提出了 CLAWS，这是一种通过利用提示部分和输出的注意力权重来定义数学解决方案并将其分类为典型、创造性和幻觉类别的方法，无需人工评估。 CLAWS 在 5 个 7-8B 数学 RL 模型（DeepSeek、Qwen、Mathstral、OpenMath2 和 Oreal）上的性能优于现有的 5 种白盒检测方法（Perplexity、Logit Entropy、Window Entropy、Hidden Score 和 Attention Score）。我们对从 181 场数学竞赛（AJHSME、AMC、AIME）中收集的 4545 个数学问题进行了 CLAWS 验证。

Title: Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models

• Authors: Shuodi Liu, Yingzhuo Liu, Zi Wang, Yusheng Wang, Huijia Wu, Liuyu Xiang, Zhaofeng He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17922
• Pdf URL: https://arxiv.org/pdf/2510.17922
• Copy Paste: [[2510.17922]] Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models(https://arxiv.org/abs/2510.17922)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable reasoning and planning capabilities, driving extensive research into task decomposition. Existing task decomposition methods focus primarily on memory, tool usage, and feedback mechanisms, achieving notable success in specific domains, but they often overlook the trade-off between performance and cost. In this study, we first conduct a comprehensive investigation on task decomposition, identifying six categorization schemes. Then, we perform an empirical analysis of three factors that influence the performance and cost of task decomposition: categories of approaches, characteristics of tasks, and configuration of decomposition and execution models, uncovering three critical insights and summarizing a set of practical principles. Building on this analysis, we propose the Select-Then-Decompose strategy, which establishes a closed-loop problem-solving process composed of three stages: selection, execution, and verification. This strategy dynamically selects the most suitable decomposition approach based on task characteristics and enhances the reliability of the results through a verification module. Comprehensive evaluations across multiple benchmarks show that the Select-Then-Decompose consistently lies on the Pareto frontier, demonstrating an optimal balance between performance and cost. Our code is publicly available at this https URL.
• 摘要：大型语言模型 (LLM) 已展现出卓越的推理和规划能力，推动了对任务分解的广泛研究。现有的任务分解方法主要关注内存、工具使用和反馈机制，在特定领域取得了显着的成功，但它们往往忽视了性能和成本之间的权衡。在本研究中，我们首先对任务分解进行了全面的研究，确定了六种分类方案。然后，我们对影响任务分解性能和成本的三个因素进行了实证分析：方法类别、任务特征以及分解和执行模型的配置，揭示了三个关键见解并总结了一组实用原则。在此分析的基础上，我们提出了“选择然后分解”策略，该策略建立了一个由选择、执行和验证三个阶段组成的闭环问题解决流程。该策略根据任务特征动态选择最合适的分解方法，并通过验证模块增强结果的可靠性。跨多个基准的综合评估表明，选择然后分解始终位于帕累托前沿，展示了性能和成本之间的最佳平衡。我们的代码可通过此 https URL 公开获取。

Title: Efficient Toxicity Detection in Gaming Chats: A Comparative Study of Embeddings, Fine-Tuned Transformers and LLMs

• Authors: Yehor Tereshchenko, Mika Hämäläinen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17924
• Pdf URL: https://arxiv.org/pdf/2510.17924
• Copy Paste: [[2510.17924]] Efficient Toxicity Detection in Gaming Chats: A Comparative Study of Embeddings, Fine-Tuned Transformers and LLMs(https://arxiv.org/abs/2510.17924)
• Keywords: language model, llm, prompt, chat, retrieval-augmented generation
• Abstract: This paper presents a comprehensive comparative analysis of Natural Language Processing (NLP) methods for automated toxicity detection in online gaming chats. Traditional machine learning models with embeddings, large language models (LLMs) with zero-shot and few-shot prompting, fine-tuned transformer models, and retrieval-augmented generation (RAG) approaches are evaluated. The evaluation framework assesses three critical dimensions: classification accuracy, processing speed, and computational costs. A hybrid moderation system architecture is proposed that optimizes human moderator workload through automated detection and incorporates continuous learning mechanisms. The experimental results demonstrate significant performance variations across methods, with fine-tuned DistilBERT achieving optimal accuracy-cost trade-offs. The findings provide empirical evidence for deploying cost-effective, efficient content moderation systems in dynamic online gaming environments.
• 摘要：本文对在线游戏聊天中自动毒性检测的自然语言处理 (NLP) 方法进行了全面的比较分析。评估了具有嵌入的传统机器学习模型、具有零样本和少样本提示的大型语言模型 (LLM)、微调 Transformer 模型以及检索增强生成 (RAG) 方法。该评估框架评估三个关键维度：分类精度、处理速度和计算成本。提出了一种混合审核系统架构，通过自动检测优化人工审核工作量并结合持续学习机制。实验结果表明，不同方法之间存在显着的性能差异，经过微调的 DistilBERT 实现了最佳的准确性与成本权衡。研究结果为在动态在线游戏环境中部署具有成本效益、高效的内容审核系统提供了经验证据。

Title: AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM

• Authors: Haoyu Huang, Hong Ting Tsang, Jiaxin Bai, Xi Peng, Gong Zhang, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17934
• Pdf URL: https://arxiv.org/pdf/2510.17934
• Copy Paste: [[2510.17934]] AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM(https://arxiv.org/abs/2510.17934)
• Keywords: language model, llm, long context, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) has shown some success in augmenting large language models (LLMs) with external knowledge. However, as a non-parametric knowledge integration paradigm for LLMs, RAG methods heavily rely on external retrieval modules and the retrieved textual context prior. Especially for very large scale knowledge augmentation, they would introduce substantial inference latency due to expensive searches and much longer relevant context. In this paper, we propose a parametric knowledge integration method, called \textbf{AtlasKV}, a scalable, effective, and general way to augment LLMs with billion-scale knowledge graphs (KGs) (e.g. 1B triples) using very little GPU memory cost (e.g. less than 20GB VRAM). In AtlasKV, we introduce KG2KV and HiKVP to integrate KG triples into LLMs at scale with sub-linear time and memory complexity. It maintains strong knowledge grounding and generalization performance using the LLMs' inherent attention mechanism, and requires no external retrievers, long context priors, or retraining when adapting to new knowledge.
• 摘要：检索增强生成（RAG）在使用外部知识增强大型语言模型（LLM）方面取得了一些成功。然而，作为法学硕士的非参数知识集成范式，RAG 方法严重依赖外部检索模块和检索到的文本上下文先验。特别是对于非常大规模的知识增强，由于昂贵的搜索和更长的相关上下文，它们会引入大量的推理延迟。在本文中，我们提出了一种名为 \textbf{AtlasKV} 的参数化知识集成方法，这是一种可扩展、有效且通用的方法，可以使用很少的 GPU 内存成本（例如小于 20GB VRAM）通过十亿级知识图（KG）（例如 1B 三元组）来增强 LLM。在AtlasKV中，我们引入了KG2KV和HiKVP，以亚线性时间和内存复杂度将KG三元组大规模集成到LLM中。它利用法学硕士固有的注意力机制保持强大的知识基础和泛化性能，并且在适应新知识时不需要外部检索器、长上下文先验或再训练。

Title: Believe It or Not: How Deeply do LLMs Believe Implanted Facts?

• Authors: Stewart Slocum, Julian Minder, Clément Dumas, Henry Sleight, Ryan Greenblatt, Samuel Marks, Rowan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.17941
• Pdf URL: https://arxiv.org/pdf/2510.17941
• Copy Paste: [[2510.17941]] Believe It or Not: How Deeply do LLMs Believe Implanted Facts?(https://arxiv.org/abs/2510.17941)
• Keywords: language model, llm, prompt
• Abstract: Knowledge editing techniques promise to implant new factual knowledge into large language models (LLMs). But do LLMs really believe these facts? We develop a framework to measure belief depth and use it to evaluate the success of knowledge editing techniques. We operationalize belief depth as the extent to which implanted knowledge 1) generalizes to related contexts (e.g. Fermi estimates several logical steps removed), 2) is robust to self-scrutiny and direct challenge, and 3) is represented similarly to genuine knowledge (as measured by linear probes). Our evaluations show that simple prompting and mechanistic editing techniques fail to implant knowledge deeply. In contrast, Synthetic Document Finetuning (SDF) - where models are trained on LLM-generated documents consistent with a fact - often succeeds at implanting beliefs that behave similarly to genuine knowledge. However, SDF's success is not universal, as implanted beliefs that contradict basic world knowledge are brittle and representationally distinct from genuine knowledge. Overall, our work introduces measurable criteria for belief depth and enables the rigorous evaluation necessary for deploying knowledge editing in real-world applications.
• 摘要：知识编辑技术有望将新的事实知识植入大型语言模型（LLM）中。但法学硕士真的相信这些事实吗？我们开发了一个框架来衡量信念深度，并用它来评估知识编辑技术的成功。我们将信念深度操作为植入知识的程度：1）推广到相关上下文（例如费米估计删除了几个逻辑步骤），2）对自我审查和直接挑战具有鲁棒性，3）与真实知识类似地表示（通过线性探针测量）。我们的评估表明，简单的提示和机械编辑技术无法深入植入知识。相比之下，综合文档微调（SDF）——模型在法学硕士生成的与事实一致的文档上进行训练——通常能够成功地植入与真实知识类似的信念。然而，SDF 的成功并不普遍，因为与基本世界知识相矛盾的植入信念是脆弱的，并且在代表性上与真正的知识不同。总的来说，我们的工作引入了可测量的信念深度标准，并实现了在现实应用程序中部署知识编辑所需的严格评估。

Title: SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone

• Authors: Nishant Subramani, Alfredo Gomez, Mona Diab
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.17998
• Pdf URL: https://arxiv.org/pdf/2510.17998
• Copy Paste: [[2510.17998]] SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone(https://arxiv.org/abs/2510.17998)
• Keywords: language model
• Abstract: Modern language models are evaluated on large benchmarks, which are difficult to make sense of, especially for model selection. Looking at the raw evaluation numbers themselves using a model-centric lens, we propose SimBA, a three phase framework to Simplify Benchmark Analysis. The three phases of SimBA are: stalk, where we conduct dataset & model comparisons, prowl, where we discover a representative subset, and pounce, where we use the representative subset to predict performance on a held-out set of models. Applying SimBA to three popular LM benchmarks: HELM, MMLU, and BigBenchLite reveals that across all three benchmarks, datasets and models relate strongly to one another (stalk). We develop an representative set discovery algorithm which covers a benchmark using raw evaluation scores alone. Using our algorithm, we find that with 6.25% (1/16), 1.7% (1/58), and 28.4% (21/74) of the datasets for HELM, MMLU, and BigBenchLite respectively, we achieve coverage levels of at least 95% (prowl). Additionally, using just these representative subsets, we can both preserve model ranks and predict performance on a held-out set of models with near zero mean-squared error (pounce). Taken together, SimBA can help model developers improve efficiency during model training and dataset creators validate whether their newly created dataset differs from existing datasets in a benchmark. Our code is open source, available at this https URL.
• 摘要：现代语言模型是在大型基准上进行评估的，这很难理解，尤其是对于模型选择而言。使用以模型为中心的镜头查看原始评估数据本身，我们提出了 SimBA，这是一个简化基准分析的三阶段框架。 SimBA 的三个阶段是：跟踪（我们进行数据集和模型比较）、徘徊（我们发现代表性子集）和突袭（我们使用代表性子集来预测一组模型的性能）。将 SimBA 应用于三个流行的 LM 基准：HELM、MMLU 和 BigBenchLite 表明，在所有三个基准中，数据集和模型彼此密切相关（茎）。我们开发了一种代表性的集合发现算法，该算法涵盖了仅使用原始评估分数的基准。使用我们的算法，我们发现 HELM、MMLU 和 BigBenchLite 的数据集分别为 6.25% (1/16)、1.7% (1/58) 和 28.4% (21/74)，我们实现了至少 95% 的覆盖率水平（徘徊）。此外，仅使用这些代表性子集，我们既可以保留模型排名，又可以以接近零均方误差（pounce）的方式预测一组模型的性能。总而言之，SimBA 可以帮助模型开发人员提高模型训练期间的效率，并帮助数据集创建者验证其新创建的数据集是否与基准中的现有数据集不同。我们的代码是开源的，可通过此 https URL 获取。

Title: Is Multilingual LLM Watermarking Truly Multilingual? A Simple Back-Translation Solution

• Authors: Asim Mohamed, Martin Gubri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18019
• Pdf URL: https://arxiv.org/pdf/2510.18019
• Copy Paste: [[2510.18019]] Is Multilingual LLM Watermarking Truly Multilingual? A Simple Back-Translation Solution(https://arxiv.org/abs/2510.18019)
• Keywords: language model, llm
• Abstract: Multilingual watermarking aims to make large language model (LLM) outputs traceable across languages, yet current methods still fall short. Despite claims of cross-lingual robustness, they are evaluated only on high-resource languages. We show that existing multilingual watermarking methods are not truly multilingual: they fail to remain robust under translation attacks in medium- and low-resource languages. We trace this failure to semantic clustering, which fails when the tokenizer vocabulary contains too few full-word tokens for a given language. To address this, we introduce STEAM, a back-translation-based detection method that restores watermark strength lost through translation. STEAM is compatible with any watermarking method, robust across different tokenizers and languages, non-invasive, and easily extendable to new languages. With average gains of +0.19 AUC and +40%p TPR@1% on 17 languages, STEAM provides a simple and robust path toward fairer watermarking across diverse languages.
• 摘要：多语言水印旨在使大型语言模型 (LLM) 输出可跨语言追踪，但目前的方法仍然存在不足。尽管声称具有跨语言稳健性，但它们仅在高资源语言上进行评估。我们表明，现有的多语言水印方法并不是真正的多语言：它们在中低资源语言的翻译攻击下无法保持鲁棒性。我们将这种失败追溯到语义聚类，当标记器词汇表包含给定语言的全词标记太少时，语义聚类就会失败。为了解决这个问题，我们引入了 STEAM，这是一种基于反向翻译的检测方法，可以恢复翻译过程中丢失的水印强度。 STEAM 与任何水印方法兼容，在不同的标记器和语言中具有鲁棒性，非侵入性，并且可以轻松扩展到新语言。 STEAM 在 17 种语言上的平均增益为 +0.19 AUC 和 +40%p TPR@1%，为跨不同语言实现更公平的水印提供了简单而强大的途径。

Title: From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models

• Authors: Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Minwoo Lee, Shu-ping Yeh, Evgeny Stupachenko, Hao Feng, Li Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.18030
• Pdf URL: https://arxiv.org/pdf/2510.18030
• Copy Paste: [[2510.18030]] From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models(https://arxiv.org/abs/2510.18030)
• Keywords: language model, llm
• Abstract: Structured pruning is a practical approach to deploying large language models (LLMs) efficiently, as it yields compact, hardware-friendly architectures. However, the dominant local paradigm is task-agnostic: by optimizing layer-wise reconstruction rather than task objectives, it tends to preserve perplexity or generic zero-shot behavior but fails to capitalize on modest task-specific calibration signals, often yielding limited downstream gains. We revisit global structured pruning and present GISP-Global Iterative Structured Pruning-a post-training method that removes attention heads and MLP channels using first-order, loss-based important weights aggregated at the structure level with block-wise normalization. An iterative schedule, rather than one-shot pruning, stabilizes accuracy at higher sparsity and mitigates perplexity collapse without requiring intermediate fine-tuning; the pruning trajectory also forms nested subnetworks that support a "prune-once, deploy-many" workflow. Furthermore, because importance is defined by a model-level loss, GISP naturally supports task-specific objectives; we instantiate perplexity for language modeling and a margin-based objective for decision-style tasks. Extensive experiments show that across Llama2-7B/13B, Llama3-8B, and Mistral-0.3-7B, GISP consistently lowers WikiText-2 perplexity and improves downstream accuracy, with especially strong gains at 40-50% sparsity; on DeepSeek-R1-Distill-Llama-3-8B with GSM8K, task-aligned calibration substantially boosts exact-match accuracy.
• 摘要：结构化剪枝是一种有效部署大型语言模型 (LLM) 的实用方法，因为它可以产生紧凑、硬件友好的架构。然而，占主导地位的局部范式是与任务无关的：通过优化分层重建而不是任务目标，它往往会保留困惑或通用的零样本行为，但无法利用适度的特定于任务的校准信号，通常会产生有限的下游收益。我们重新审视全局结构化剪枝，并提出 GISP（全局迭代结构化剪枝），这是一种训练后方法，使用在结构级别聚合的一阶、基于损失的重要权重和逐块归一化来删除注意力头和 MLP 通道。迭代计划，而不是一次性剪枝，可以在较高稀疏性下稳定准确性，并减轻困惑崩溃，而不需要中间微调；修剪轨迹还形成嵌套子网络，支持“修剪一次，部署多次”工作流程。此外，由于重要性是由模型级损失定义的，GISP 自然支持特定于任务的目标；我们实例化了语言建模的困惑度和决策型任务的基于边际的目标。大量实验表明，在 Llama2-7B/13B、Llama3-8B 和 Mistral-0.3-7B 中，GISP 持续降低 WikiText-2 的困惑度并提高下游准确性，尤其是在稀疏度为 40-50% 时获得了显着的提升；在具有 GSM8K 的 DeepSeek-R1-Distill-Llama-3-8B 上，任务对齐校准大大提高了精确匹配精度。

Title: Language Models as Semantic Augmenters for Sequential Recommenders

• Authors: Mahsa Valizadeh, Xiangjue Dong, Rui Tuo, James Caverlee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18046
• Pdf URL: https://arxiv.org/pdf/2510.18046
• Copy Paste: [[2510.18046]] Language Models as Semantic Augmenters for Sequential Recommenders(https://arxiv.org/abs/2510.18046)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel at capturing latent semantics and contextual relationships across diverse modalities. However, in modeling user behavior from sequential interaction data, performance often suffers when such semantic context is limited or absent. We introduce LaMAR, a LLM-driven semantic enrichment framework designed to enrich such sequences automatically. LaMAR leverages LLMs in a few-shot setting to generate auxiliary contextual signals by inferring latent semantic aspects of a user's intent and item relationships from existing metadata. These generated signals, such as inferred usage scenarios, item intents, or thematic summaries, augment the original sequences with greater contextual depth. We demonstrate the utility of this generated resource by integrating it into benchmark sequential modeling tasks, where it consistently improves performance. Further analysis shows that LLM-generated signals exhibit high semantic novelty and diversity, enhancing the representational capacity of the downstream models. This work represents a new data-centric paradigm where LLMs serve as intelligent context generators, contributing a new method for the semi-automatic creation of training data and language resources.
• 摘要：大型语言模型 (LLM) 擅长捕获不同模式中的潜在语义和上下文关系。然而，在根据顺序交互数据对用户行为进行建模时，当此类语义上下文有限或不存在时，性能通常会受到影响。我们引入了 LaMAR，这是一个 LLM 驱动的语义丰富框架，旨在自动丰富此类序列。 LaMAR 在几次镜头设置中利用 LLM，通过从现有元数据推断用户意图和项目关系的潜在语义方面来生成辅助上下文信号。这些生成的信号（例如推断的使用场景、项目意图或主题摘要）以更大的上下文深度增强了原始序列。我们通过将生成的资源集成到基准顺序建模任务中来展示其实用性，从而持续提高性能。进一步分析表明，LLM 生成的信号表现出高度的语义新颖性和多样性，增强了下游模型的表示能力。这项工作代表了一种新的以数据为中心的范例，其中法学硕士充当智能上下文生成器，为半自动创建训练数据和语言资源提供了一种新方法。

Title: Chain-of-Thought Reasoning Improves Context-Aware Translation with Large Language Models

• Authors: Shabnam Ataee, Andrei Popescu-Belis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18077
• Pdf URL: https://arxiv.org/pdf/2510.18077
• Copy Paste: [[2510.18077]] Chain-of-Thought Reasoning Improves Context-Aware Translation with Large Language Models(https://arxiv.org/abs/2510.18077)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: This paper assesses the capacity of large language models (LLMs) to translate texts that include inter-sentential dependencies. We use the English-French DiscEvalMT benchmark (Bawden et al., 2018) with pairs of sentences containing translation challenges either for pronominal anaphora or for lexical cohesion. We evaluate 12 LLMs from the DeepSeek-R1, GPT, Llama, Mistral and Phi families on two tasks: (1) distinguishing a correct translation from a wrong but plausible one; (2) generating a correct translation. We compare prompts that encourage chain-of-thought reasoning with those that do not. The best models take advantage of reasoning and reach about 90% accuracy on the first task, and COMET scores of about 92% on the second task, with GPT-4, GPT-4o and Phi standing out. Moreover, we observe a "wise get wiser" effect: the improvements through reasoning are positively correlated with the scores of the models without reasoning.
• 摘要：本文评估了大型语言模型 (LLM) 翻译包含句子间依赖关系的文本的能力。我们使用英语-法语 DiscEvalMT 基准（Bawden 等人，2018）以及包含代词照应或词汇衔接的翻译挑战的句子对。我们在两项任务上评估了来自 DeepSeek-R1、GPT、Llama、Mistral 和 Phi 系列的 12 名法学硕士：（1）区分正确的翻译和错误但合理的翻译； (2) 生成正确的翻译。我们将鼓励链式推理的提示与不鼓励链式推理的提示进行比较。最好的模型利用推理，在第一个任务上达到约 90% 的准确率，在第二个任务上 COMET 得分约为 92%，其中 GPT-4、GPT-4o 和 Phi 表现突出。此外，我们观察到“明智变得更明智”的效果：通过推理的改进与没有推理的模型的分数正相关。

Title: Na Prática, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jurídica

• Authors: Marina Soares Marinho, Daniela Vianna, Livy Real, Altigran da Silva, Gabriela Migliorini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18108
• Pdf URL: https://arxiv.org/pdf/2510.18108
• Copy Paste: [[2510.18108]] Na Prática, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jurídica(https://arxiv.org/abs/2510.18108)
• Keywords: gpt, chat
• Abstract: This study presents the Jusbrasil Study on the Use of General-Purpose AIs in Law, proposing an experimental evaluation protocol combining legal theory, such as material correctness, systematic coherence, and argumentative integrity, with empirical assessment by 48 legal professionals. Four systems (JusIA, ChatGPT Free, ChatGPT Pro, and Gemini) were tested in tasks simulating lawyers' daily work. JusIA, a domain-specialized model, consistently outperformed the general-purpose systems, showing that both domain specialization and a theoretically grounded evaluation are essential for reliable legal AI outputs.
• 摘要：本研究介绍了 Jusbrasil 关于通用人工智能在法律中的使用的研究，提出了一种实验性评估方案，将材料正确性、系统连贯性和论证完整性等法律理论与 48 名法律专业人士的实证评估相结合。四个系统（JusIA、ChatGPT Free、ChatGPT Pro 和 Gemini）在模拟律师日常工作的任务中进行了测试。 JusIA 是一种领域专业化模型，其性能始终优于通用系统，这表明领域专业化和基于理论的评估对于可靠的法律人工智能输出至关重要。

Title: Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment

• Authors: Patricia Delafuente, Arya Honraopatil, Lara J. Martin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18112
• Pdf URL: https://arxiv.org/pdf/2510.18112
• Copy Paste: [[2510.18112]] Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment(https://arxiv.org/abs/2510.18112)
• Keywords: language model, llm, prompt, agent
• Abstract: This paper explores the application of Large Language Models (LLMs) and reasoning to predict Dungeons & Dragons (DnD) player actions and format them as Avrae Discord bot commands. Using the FIREBALL dataset, we evaluated a reasoning model, DeepSeek-R1-Distill-LLaMA-8B, and an instruct model, LLaMA-3.1-8B-Instruct, for command generation. Our findings highlight the importance of providing specific instructions to models, that even single sentence changes in prompts can greatly affect the output of models, and that instruct models are sufficient for this task compared to reasoning models.
• 摘要：本文探讨了大型语言模型 (LLM) 和推理的应用，以预测龙与地下城 (DnD) 玩家操作并将其格式化为 Avrae Discord 机器人命令。使用 FIREBALL 数据集，我们评估了用于命令生成的推理模型 DeepSeek-R1-Distill-LLaMA-8B 和指令模型 LLaMA-3.1-8B-Instruct。我们的研究结果强调了为模型提供具体指令的重要性，即使提示中的单个句子变化也会极大地影响模型的输出，并且与推理模型相比，指令模型足以完成这项任务。

Title: LLMs Encode How Difficult Problems Are

• Authors: William Lugoloobi, Chris Russell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18147
• Pdf URL: https://arxiv.org/pdf/2510.18147
• Copy Paste: [[2510.18147]] LLMs Encode How Difficult Problems Are(https://arxiv.org/abs/2510.18147)
• Keywords: language model, llm, hallucination
• Abstract: Large language models exhibit a puzzling inconsistency: they solve complex problems yet frequently fail on seemingly simpler ones. We investigate whether LLMs internally encode problem difficulty in a way that aligns with human judgment, and whether this representation tracks generalization during reinforcement learning post-training. We train linear probes across layers and token positions on 60 models, evaluating on mathematical and coding subsets of Easy2HardBench. We find that human-labeled difficulty is strongly linearly decodable (AMC: $\rho \approx 0.88$) and exhibits clear model-size scaling, whereas LLM-derived difficulty is substantially weaker and scales poorly. Steering along the difficulty direction reveals that pushing models toward "easier" representations reduces hallucination and improves accuracy. During GRPO training on Qwen2.5-Math-1.5B, the human-difficulty probe strengthens and positively correlates with test accuracy across training steps, while the LLM-difficulty probe degrades and negatively correlates with performance. These results suggest that human annotations provide a stable difficulty signal that RL amplifies, while automated difficulty estimates derived from model performance become misaligned precisely as models improve. We release probe code and evaluation scripts to facilitate replication.
• 摘要：大型语言模型表现出令人费解的不一致：它们解决了复杂的问题，但在看似简单的问题上却经常失败。我们研究法学硕士是否以符合人类判断的方式在内部编码问题难度，以及这种表示是否在强化学习训练后跟踪泛化。我们在 60 个模型上跨层和标记位置训练线性探针，对 Easy2HardBench 的数学和编码子集进行评估。我们发现，人类标记的难度具有很强的线性可解码性（AMC：$\rho \约 0.88$），并且表现出清晰的模型大小缩放，而 LLM 衍生的难度则明显较弱，且缩放性较差。沿着难度方向引导表明，将模型推向“更简单”的表示可以减少幻觉并提高准确性。在 Qwen2.5-Math-1.5B 上的 GRPO 训练期间，人类难度探针在整个训练步骤中增强并与测试准确性呈正相关，而 LLM 难度探针则降低并与性能呈负相关。这些结果表明，人类注释提供了 RL 放大的稳定难度信号，而从模型性能得出的自动难度估计随着模型的改进而精确地错位。我们发布探测代码和评估脚本以方便复制。

Title: Extracting Rule-based Descriptions of Attention Features in Transformers

• Authors: Dan Friedman, Adithya Bhaskar, Alexander Wettig, Danqi Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.18148
• Pdf URL: https://arxiv.org/pdf/2510.18148
• Copy Paste: [[2510.18148]] Extracting Rule-based Descriptions of Attention Features in Transformers(https://arxiv.org/abs/2510.18148)
• Keywords: gpt
• Abstract: Mechanistic interpretability strives to explain model behavior in terms of bottom-up primitives. The leading paradigm is to express hidden states as a sparse linear combination of basis vectors, called features. However, this only identifies which text sequences (exemplars) activate which features; the actual interpretation of features requires subjective inspection of these exemplars. This paper advocates for a different solution: rule-based descriptions that match token patterns in the input and correspondingly increase or decrease the likelihood of specific output tokens. Specifically, we extract rule-based descriptions of SAE features trained on the outputs of attention layers. While prior work treats the attention layers as an opaque box, we describe how it may naturally be expressed in terms of interactions between input and output features, of which we study three types: (1) skip-gram rules of the form "[Canadian city]... speaks --> English", (2) absence rules of the form "[Montreal]... speaks -/-> English," and (3) counting rules that toggle only when the count of a word exceeds a certain value or the count of another word. Absence and counting rules are not readily discovered by inspection of exemplars, where manual and automatic descriptions often identify misleading or incomplete explanations. We then describe a simple approach to extract these types of rules automatically from a transformer, and apply it to GPT-2 small. We find that a majority of features may be described well with around 100 skip-gram rules, though absence rules are abundant even as early as the first layer (in over a fourth of features). We also isolate a few examples of counting rules. This paper lays the groundwork for future research into rule-based descriptions of features by defining them, showing how they may be extracted, and providing a preliminary taxonomy of some of the behaviors they represent.
• 摘要：机械可解释性致力于用自下而上的原语来解释模型行为。主要范例是将隐藏状态表示为基向量（称为特征）的稀疏线性组合。然而，这只能识别哪些文本序列（范例）激活哪些功能；对特征的实际解释需要对这些示例进行主观检查。本文主张采用不同的解决方案：基于规则的描述，匹配输入中的标记模式，并相应地增加或减少特定输出标记的可能性。具体来说，我们提取在注意力层输出上训练的 SAE 特征的基于规则的描述。虽然之前的工作将注意力层视为一个不透明的盒子，但我们描述了如何自然地用输入和输出特征之间的交互来表达它，我们研究了三种类型：（1）“[加拿大城市]...说话->英语”形式的skip-gram规则，（2）“[蒙特利尔]...说话-/->英语”形式的缺席规则，以及（3）仅当单词计数超过特定值时才切换的计数规则 或另一个单词的计数。通过检查示例并不容易发现缺失和计数规则，其中手动和自动描述经常识别出误导性或不完整的解释。然后，我们描述了一种简单的方法，可以从变压器中自动提取这些类型的规则，并将其应用于小型 GPT-2。我们发现，大多数特征可以用大约 100 个skip-gram 规则来很好地描述，尽管缺席规则甚至早在第一层就很丰富（在超过四分之一的特征中）。我们还分离出一些计数规则的示例。本文通过定义特征、展示如何提取特征以及提供它们所代表的一些行为的初步分类，为未来研究基于规则的特征描述奠定了基础。

Title: Automatic Prompt Generation via Adaptive Selection of Prompting Techniques

• Authors: Yohei Ikenoue, Hitomi Tashiro, Shigeru Kuroyanagi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18162
• Pdf URL: https://arxiv.org/pdf/2510.18162
• Copy Paste: [[2510.18162]] Automatic Prompt Generation via Adaptive Selection of Prompting Techniques(https://arxiv.org/abs/2510.18162)
• Keywords: language model, llm, prompt
• Abstract: Prompt engineering is crucial for achieving reliable and effective outputs from large language models (LLMs), but its design requires specialized knowledge of prompting techniques and a deep understanding of target tasks. To address this challenge, we propose a novel method that adaptively selects task-appropriate prompting techniques based on users' abstract task descriptions and automatically generates high-quality prompts without relying on pre-existing templates or frameworks. The proposed method constructs a knowledge base that associates task clusters, characterized by semantic similarity across diverse tasks, with their corresponding prompting techniques. When users input task descriptions, the system assigns them to the most relevant task cluster and dynamically generates prompts by integrating techniques drawn from the knowledge base. An experimental evaluation of the proposed method on 23 tasks from BIG-Bench Extra Hard (BBEH) demonstrates superior performance compared with standard prompts and existing automatic prompt-generation tools, as measured by both arithmetic and harmonic mean scores. This research establishes a foundation for streamlining and standardizing prompt creation, enabling non-experts to effectively leverage LLMs.
• 摘要：提示工程对于从大型语言模型 (LLM) 获得可靠且有效的输出至关重要，但其设计需要提示技术的专业知识和对目标任务的深入理解。为了应对这一挑战，我们提出了一种新颖的方法，该方法可以根据用户的抽象任务描述自适应地选择适合任务的提示技术，并自动生成高质量的提示，而无需依赖预先存在的模板或框架。所提出的方法构建了一个知识库，将任务集群与其相应的提示技术相关联，其特征是不同任务之间的语义相似性。当用户输入任务描述时，系统将其分配给最相关的任务集群，并通过集成从知识库中提取的技术动态生成提示。对 BIG-Bench Extra Hard (BBEH) 的 23 项任务进行的实验评估表明，与标准提示和现有自动提示生成工具相比，该方法具有更优越的性能（通过算术平均分数和调和平均分数来衡量）。这项研究为简化和标准化提示创建奠定了基础，使非专家能够有效利用法学硕士。

Title: CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models

• Authors: Ritam Upadhyay, Naman Ahuja, Rishabh Baral, Aparna Garimella, Vivek Gupta
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18173
• Pdf URL: https://arxiv.org/pdf/2510.18173
• Copy Paste: [[2510.18173]] CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models(https://arxiv.org/abs/2510.18173)
• Keywords: language model, llm, prompt
• Abstract: LLM Driven text-to-table (T2T) systems often rely on extensive prompt-engineering or iterative event extraction in code-parsable formats, which boosts scores but are computationally expensive and obscure how models actually reason over temporal evolving narratives to summarise key information. We present CMT-Bench, a diagnostic benchmark built from live cricket commentary that requires dynamic table generation across two evolving schemas under a dense, rule-governed policy. CMT-Bench is designed to probe robustness via three semantics-preserving dimensions: (i) extractive-cue ablation to separate extractive shortcuts from state tracking, (ii) temporal prefixing to test long-context stability, and (iii) entity-form perturbations (anonymization, outof-distribution substitutions, role-entangling paraphrases) to assess sensitivity to surface variation. Across diverse long-context stateof-the-art LLMs, we find large drops without extractive summaries, monotonic degradation with input length, and consistent accuracy drop under entity-form changes. Complementary distributional tests confirm significant shifts in numeric error patterns, indicating drift in reasoning rather than mere noise. Our results show that current LLMs are brittle in dynamic Textto-table generation, motivating robustness-first evaluation as a prerequisite for developing efficient and scalable approaches for this task.
• 摘要：LLM 驱动的文本到表格 (T2T) 系统通常依赖于代码可解析格式的广泛提示工程或迭代事件提取，这会提高分数，但计算成本很高，并且模糊模型如何实际推理时间演变的叙述以总结关键信息。我们提出了 CMT-Bench，这是一种根据现场板球评论构建的诊断基准，需要在密集的、规则管理的策略下跨两个不断发展的模式生成动态表。 CMT-Bench 旨在通过三个保留语义的维度来探测鲁棒性：(i) 提取线索消融，将提取快捷方式与状态跟踪分开；(ii) 时间前缀，用于测试长上下文稳定性；(iii) 实体形式扰动（匿名化、分布外替换、角色纠缠释义），以评估对表面变化的敏感性。在各种长上下文最先进的法学硕士中，我们发现没有提取摘要的情况下大幅下降，输入长度的单调退化，以及实体形式变化下一致的准确性下降。补充分布测试证实了数字错误模式的显着变化，表明推理的漂移而不仅仅是噪音。我们的结果表明，当前的法学硕士在动态文本到表生成方面很脆弱，因此将鲁棒性优先评估作为为此任务开发高效和可扩展方法的先决条件。

Title: Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge

• Authors: Yoshinari Fujinuma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18196
• Pdf URL: https://arxiv.org/pdf/2510.18196
• Copy Paste: [[2510.18196]] Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge(https://arxiv.org/abs/2510.18196)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are commonly used as evaluators in various applications, but the reliability of the outcomes remains a challenge. One such challenge is using LLMs-as-judges for direct assessment, i.e., assigning scores from a specified range without any references. We first show that this challenge stems from LLM judge outputs being associated with score range bias, i.e., LLM judge outputs are highly sensitive to pre-defined score ranges, preventing the search for optimal score ranges. We also show that similar biases exist among models from the same family. We then mitigate this bias through contrastive decoding, achieving up to 11.3% relative improvement on average in Spearman correlation with human judgments across different score ranges.
• 摘要：大型语言模型（LLM）通常在各种应用中用作评估器，但结果的可靠性仍然是一个挑战。其中一项挑战是使用法学硕士作为评委进行直接评估，即在没有任何参考资料的情况下从指定范围内分配分数。我们首先表明，这一挑战源于法学硕士法官的输出与分数范围偏差相关，即法学硕士法官的输出对预定义的分数范围高度敏感，从而阻止了对最佳分数范围的搜索。我们还表明，同一家族的模型之间也存在类似的偏差。然后，我们通过对比解码来减轻这种偏差，在不同分数范围内，Spearman 相关性与人类判断的平均相对改善高达 11.3%。

Title: DelvePO: Direction-Guided Self-Evolving Framework for Flexible Prompt Optimization

• Authors: Tao Tao, Guanghui Zhu, Lang Guo, Hongyi Chen, Chunfeng Yuan, Yihua Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18257
• Pdf URL: https://arxiv.org/pdf/2510.18257
• Copy Paste: [[2510.18257]] DelvePO: Direction-Guided Self-Evolving Framework for Flexible Prompt Optimization(https://arxiv.org/abs/2510.18257)
• Keywords: language model, gpt, llm, prompt
• Abstract: Prompt Optimization has emerged as a crucial approach due to its capabilities in steering Large Language Models to solve various tasks. However, current works mainly rely on the random rewriting ability of LLMs, and the optimization process generally focus on specific influencing factors, which makes it easy to fall into local optimum. Besides, the performance of the optimized prompt is often unstable, which limits its transferability in different tasks. To address the above challenges, we propose $\textbf{DelvePO}$ ($\textbf{D}$irection-Guid$\textbf{e}$d Se$\textbf{l}$f-E$\textbf{v}$olving Framework for Fl$\textbf{e}$xible $\textbf{P}$rompt $\textbf{O}$ptimization), a task-agnostic framework to optimize prompts in self-evolve manner. In our framework, we decouple prompts into different components that can be used to explore the impact that different factors may have on various tasks. On this basis, we introduce working memory, through which LLMs can alleviate the deficiencies caused by their own uncertainties and further obtain key insights to guide the generation of new prompts. Extensive experiments conducted on different tasks covering various domains for both open- and closed-source LLMs, including DeepSeek-R1-Distill-Llama-8B, Qwen2.5-7B-Instruct and GPT-4o-mini. Experimental results show that DelvePO consistently outperforms previous SOTA methods under identical experimental settings, demonstrating its effectiveness and transferability across different tasks.
• 摘要：即时优化因其引导大型语言模型解决各种任务的能力而成为一种重要的方法。然而，目前的工作主要依赖于LLM的随机重写能力，优化过程一般集中于特定的影响因素，这使得很容易陷入局部最优。此外，优化后的提示的性能往往不稳定，这限制了其在不同任务中的可移植性。为了解决上述挑战，我们提出 $\textbf{DelvePO}$ ($\textbf{D}$direction-Guid$\textbf{e}$d Se$\textbf{l}$f-E$\textbf{v}$olving Framework for Fl$\textbf{e}$xible $\textbf{P}$rompt $\textbf{O}$ptimization)，一个与任务无关的框架，以自我进化的方式优化提示。在我们的框架中，我们将提示解耦到不同的组件中，这些组件可用于探索不同因素可能对各种任务产生的影响。在此基础上，我们引入工作记忆，法学硕士可以通过工作记忆缓解自身不确定性带来的不足，并进一步获得关键见解来指导新提示的生成。针对开源和闭源 LLM 的各个领域的不同任务进行了广泛的实验，包括 DeepSeek-R1-Distill-Llama-8B、Qwen2.5-7B-Instruct 和 GPT-4o-mini。实验结果表明，在相同的实验设置下，DelvePO 始终优于以前的 SOTA 方法，证明了其在不同任务中的有效性和可迁移性。

Title: Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs

• Authors: Yanhong Li, Zixuan Lan, Jiawei Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18279
• Pdf URL: https://arxiv.org/pdf/2510.18279
• Copy Paste: [[2510.18279]] Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs(https://arxiv.org/abs/2510.18279)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) and their multimodal variants can now process visual inputs, including images of text. This raises an intriguing question: can we compress textual inputs by feeding them as images to reduce token usage while preserving performance? In this paper, we show that visual text representations are a practical and surprisingly effective form of input compression for decoder LLMs. We exploit the idea of rendering long text inputs as a single image and provide it directly to the model. This leads to dramatically reduced number of decoder tokens required, offering a new form of input compression. Through experiments on two distinct benchmarks RULER (long-context retrieval) and CNN/DailyMail (document summarization) we demonstrate that this text-as-image method yields substantial token savings (often nearly half) without degrading task performance.
• 摘要：大型语言模型 (LLM) 及其多模式变体现在可以处理视觉输入，包括文本图像。这就提出了一个有趣的问题：我们能否通过将文本输入作为图像提供来压缩它们，以减少令牌使用，同时保持性能？在本文中，我们证明视觉文本表示对于解码器 LLM 来说是一种实用且令人惊讶的有效输入压缩形式。我们利用将长文本输入渲染为单个图像的想法，并将其直接提供给模型。这导致所需的解码器令牌数量大幅减少，从而提供了一种新形式的输入压缩。通过对两个不同基准 RULER（长上下文检索）和 CNN/DailyMail（文档摘要）的实验，我们证明这种文本即图像方法可以节省大量令牌（通常接近一半），而不会降低任务性能。

Title: BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks

• Authors: Tianyuan Huang, Zepeng Zhu, Hangdi Xing, Zirui Shao, Zhi Yu, Chaoxiong Yang, Jiaxian He, Xiaozhong Liu, Jiajun Bu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18288
• Pdf URL: https://arxiv.org/pdf/2510.18288
• Copy Paste: [[2510.18288]] BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks(https://arxiv.org/abs/2510.18288)
• Keywords: language model, llm
• Abstract: Braille plays a vital role in education and information accessibility for visually impaired individuals. However, Braille information processing faces challenges such as data scarcity and ambiguities in mixed-text contexts. We construct English and Chinese Braille Mixed Datasets (EBMD/CBMD) with mathematical formulas to support diverse Braille domain research, and propose a syntax tree-based augmentation method tailored for Braille data. To address the underperformance of traditional fine-tuning methods in Braille-related tasks, we investigate Braille Knowledge-Based Fine-Tuning (BKFT), which reduces the learning difficulty of Braille contextual features. BrailleLLM employs BKFT via instruction tuning to achieve unified Braille translation, formula-to-Braille conversion, and mixed-text translation. Experiments demonstrate that BKFT achieves significant performance improvements over conventional fine-tuning in Braille translation scenarios. Our open-sourced datasets and methodologies establish a foundation for low-resource multilingual Braille research.
• 摘要：盲文在视障人士的教育和信息获取方面发挥着至关重要的作用。然而，盲文信息处理面临着数据稀缺和混合文本上下文中的歧义等挑战。我们用数学公式构建了英语和中文盲文混合数据集（EBMD/CBMD）以支持多样化的盲文领域研究，并提出了一种针对盲文数据量身定制的基于语法树的增强方法。为了解决传统微调方法在盲文相关任务中表现不佳的问题，我们研究了盲文基于知识的微调（BKFT），它降低了盲文上下文特征的学习难度。 BrailleLLM 通过指令调优，采用 BKFT 实现统一盲文翻译、公式到盲文转换以及混合文本翻译。实验表明，在盲文翻译场景中，BKFT 比传统微调取得了显着的性能提升。我们的开源数据集和方法为资源匮乏的多语言盲文研究奠定了基础。

Title: Food4All: A Multi-Agent Framework for Real-time Free Food Discovery with Integrated Nutritional Metadata

• Authors: Zhengqing Yuan, Yiyang Li, Weixiang Sun, Zheyuan Zhang, Kaiwen Shi, Keerthiram Murugesan, Yanfang Ye
• Subjects: cs.CL, cs.CY, cs.MA
• Abstract URL: https://arxiv.org/abs/2510.18289
• Pdf URL: https://arxiv.org/pdf/2510.18289
• Copy Paste: [[2510.18289]] Food4All: A Multi-Agent Framework for Real-time Free Food Discovery with Integrated Nutritional Metadata(https://arxiv.org/abs/2510.18289)
• Keywords: llm, chat, agent
• Abstract: Food insecurity remains a persistent public health emergency in the United States, tightly interwoven with chronic disease, mental illness, and opioid misuse. Yet despite the existence of thousands of food banks and pantries, access remains fragmented: 1) current retrieval systems depend on static directories or generic search engines, which provide incomplete and geographically irrelevant results; 2) LLM-based chatbots offer only vague nutritional suggestions and fail to adapt to real-world constraints such as time, mobility, and transportation; and 3) existing food recommendation systems optimize for culinary diversity but overlook survival-critical needs of food-insecure populations, including immediate proximity, verified availability, and contextual barriers. These limitations risk leaving the most vulnerable individuals, those experiencing homelessness, addiction, or digital illiteracy, unable to access urgently needed resources. To address this, we introduce Food4All, the first multi-agent framework explicitly designed for real-time, context-aware free food retrieval. Food4All unifies three innovations: 1) heterogeneous data aggregation across official databases, community platforms, and social media to provide a continuously updated pool of food resources; 2) a lightweight reinforcement learning algorithm trained on curated cases to optimize for both geographic accessibility and nutritional correctness; and 3) an online feedback loop that dynamically adapts retrieval policies to evolving user needs. By bridging information acquisition, semantic analysis, and decision support, Food4All delivers nutritionally annotated and guidance at the point of need. This framework establishes an urgent step toward scalable, equitable, and intelligent systems that directly support populations facing food insecurity and its compounding health risks.
• 摘要：粮食不安全仍然是美国持续存在的公共卫生紧急情况，与慢性病、精神疾病和阿片类药物滥用密切相关。然而，尽管存在数千个食品银行和食品储藏室，但访问仍然分散：1）当前的检索系统依赖于静态目录或通用搜索引擎，这些引擎提供的结果不完整且与地理无关； 2）基于LLM的聊天机器人仅提供模糊的营养建议，无法适应现实世界的时间、流动性和交通等限制； 3) 现有的食品推荐系统针对烹饪多样性进行了优化，但忽视了食品不安全人群的生存关键需求，包括邻近性、经验证的可用性和背景障碍。这些限制可能会使最脆弱的个人、无家可归、成瘾或数字文盲的人无法获得急需的资源。为了解决这个问题，我们引入了 Food4All，这是第一个专门为实时、上下文感知的免费食物检索而设计的多代理框架。 Food4All统一了三项创新：1）跨官方数据库、社区平台和社交媒体的异构数据聚合，提供持续更新的食物资源池； 2) 轻量级强化学习算法，根据精选案例进行训练，以优化地理可及性和营养正确性； 3) 在线反馈循环，动态调整检索策略以适应不断变化的用户需求。通过连接信息获取、语义分析和决策支持，Food4All 在需要时提供营养注释和指导。该框架为实现可扩展、公平和智能系统迈出了紧急一步，直接支持面临粮食不安全及其复杂健康风险的人群。

Title: From Retrieval to Generation: Unifying External and Parametric Knowledge for Medical Question Answering

• Authors: Lei Li, Xiao Zhou, Yingying Zhang, Xian Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18297
• Pdf URL: https://arxiv.org/pdf/2510.18297
• Copy Paste: [[2510.18297]] From Retrieval to Generation: Unifying External and Parametric Knowledge for Medical Question Answering(https://arxiv.org/abs/2510.18297)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Medical question answering (QA) requires extensive access to domain-specific knowledge. A promising direction is to enhance large language models (LLMs) with external knowledge retrieved from medical corpora or parametric knowledge stored in model parameters. Existing approaches typically fall into two categories: Retrieval-Augmented Generation (RAG), which grounds model reasoning on externally retrieved evidence, and Generation-Augmented Generation (GAG), which depends solely on the models internal knowledge to generate contextual documents. However, RAG often suffers from noisy or incomplete retrieval, while GAG is vulnerable to hallucinated or inaccurate information due to unconstrained generation. Both issues can mislead reasoning and undermine answer reliability. To address these challenges, we propose MedRGAG, a unified retrieval-generation augmented framework that seamlessly integrates external and parametric knowledge for medical QA. MedRGAG comprises two key modules: Knowledge-Guided Context Completion (KGCC), which directs the generator to produce background documents that complement the missing knowledge revealed by retrieval; and Knowledge-Aware Document Selection (KADS), which adaptively selects an optimal combination of retrieved and generated documents to form concise yet comprehensive evidence for answer generation. Extensive experiments on five medical QA benchmarks demonstrate that MedRGAG achieves a 12.5% improvement over MedRAG and a 4.5% gain over MedGENIE, highlighting the effectiveness of unifying retrieval and generation for knowledge-intensive reasoning. Our code and data are publicly available at this https URL
• 摘要：医学问答 (QA) 需要广泛访问特定领域的知识。一个有前途的方向是利用从医学语料库检索的外部知识或存储在模型参数中的参数知识来增强大型语言模型（LLM）。现有的方法通常分为两类：检索增强生成（RAG），其基于外部检索的证据进行模型推理；生成增强生成（GAG），其仅依赖于模型内部知识来生成上下文文档。然而，RAG 经常遭受噪声或不完整的检索，而 GAG 由于不受约束的生成而容易受到幻觉或不准确信息的影响。这两个问题都会误导推理并破坏答案的可靠性。为了应对这些挑战，我们提出了 MedRGAG，这是一个统一的检索生成增强框架，可以无缝集成用于医学 QA 的外部知识和参数知识。 MedRGAG 包含两个关键模块： 知识引导上下文补全 (KGCC)，它指导生成器生成背景文档，以补充检索所揭示的缺失知识；知识感知文档选择（KADS），它自适应地选择检索和生成的文档的最佳组合，以形成简洁而全面的答案生成证据。对五个医学 QA 基准的广泛实验表明，MedRGAG 比 MedRAG 提高了 12.5%，比 MedGENIE 提高了 4.5%，凸显了统一检索和生成知识密集型推理的有效性。我们的代码和数据可通过此 https URL 公开获取

Title: ECG-LLM-- training and evaluation of domain-specific large language models for electrocardiography

• Authors: Lara Ahrens, Wilhelm Haverkamp, Nils Strodthoff
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.18339
• Pdf URL: https://arxiv.org/pdf/2510.18339
• Copy Paste: [[2510.18339]] ECG-LLM-- training and evaluation of domain-specific large language models for electrocardiography(https://arxiv.org/abs/2510.18339)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Domain-adapted open-weight large language models (LLMs) offer promising healthcare applications, from queryable knowledge bases to multimodal assistants, with the crucial advantage of local deployment for privacy preservation. However, optimal adaptation strategies, evaluation methodologies, and performance relative to general-purpose LLMs remain poorly characterized. We investigated these questions in electrocardiography, an important area of cardiovascular medicine, by finetuning open-weight models on domain-specific literature and implementing a multi-layered evaluation framework comparing finetuned models, retrieval-augmented generation (RAG), and Claude Sonnet 3.7 as a representative general-purpose model. Finetuned Llama 3.1 70B achieved superior performance on multiple-choice evaluations and automatic text metrics, ranking second to Claude 3.7 in LLM-as-a-judge assessments. Human expert evaluation favored Claude 3.7 and RAG approaches for complex queries. Finetuned models significantly outperformed their base counterparts across nearly all evaluation modes. Our findings reveal substantial performance heterogeneity across evaluation methodologies, underscoring assessment complexity. Nevertheless, domain-specific adaptation through finetuning and RAG achieves competitive performance with proprietary models, supporting the viability of privacy-preserving, locally deployable clinical solutions.
• 摘要：领域适应的开放权重大语言模型（LLM）提供了有前途的医疗保健应用程序，从可查询的知识库到多模式助手，具有本地部署以保护隐私的关键优势。然而，相对于通用法学硕士的最佳适应策略、评估方法和表现仍然知之甚少。我们通过对特定领域文献的开放权重模型进行微调，并实施多层评估框架来比较微调模型、检索增强生成（RAG）和作为代表性通用模型的 Claude Sonnet 3.7，研究了心电图（心血管医学的一个重要领域）中的这些问题。 Finetuned Llama 3.1 70B 在多项选择评估和自动文本指标方面取得了优异的表现，在法学硕士法官评估中排名第二，仅次于 Claude 3.7。人类专家评估倾向于使用 Claude 3.7 和 RAG 方法来处理复杂查询。经过微调的模型在几乎所有评估模式中都显着优于其基础模型。我们的研究结果揭示了评估方法之间存在显着的绩效异质性，强调了评估的复杂性。尽管如此，通过微调和 RAG 进行的特定领域的适应与专有模型相比实现了具有竞争力的性能，支持了隐私保护、本地可部署的临床解决方案的可行性。

Title: Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction

• Authors: Vipul Rathore, Malik Hammad Faisal, Parag Singla, Mausam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18344
• Pdf URL: https://arxiv.org/pdf/2510.18344
• Copy Paste: [[2510.18344]] Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction(https://arxiv.org/abs/2510.18344)
• Keywords: language model, llm, prompt
• Abstract: Distantly Supervised Relation Extraction (DSRE) remains a long-standing challenge in NLP, where models must learn from noisy bag-level annotations while making sentence-level predictions. While existing state-of-the-art (SoTA) DSRE models rely on task-specific training, their integration with in-context learning (ICL) using large language models (LLMs) remains underexplored. A key challenge is that the LLM may not learn relation semantics correctly, due to noisy annotation. In response, we propose HYDRE -- HYbrid Distantly Supervised Relation Extraction framework. It first uses a trained DSRE model to identify the top-k candidate relations for a given test sentence, then uses a novel dynamic exemplar retrieval strategy that extracts reliable, sentence-level exemplars from training data, which are then provided in LLM prompt for outputting the final relation(s). We further extend HYDRE to cross-lingual settings for RE in low-resource languages. Using available English DSRE training data, we evaluate all methods on English as well as a newly curated benchmark covering four diverse low-resource Indic languages -- Oriya, Santali, Manipuri, and Tulu. HYDRE achieves up to 20 F1 point gains in English and, on average, 17 F1 points on Indic languages over prior SoTA DSRE models. Detailed ablations exhibit HYDRE's efficacy compared to other prompting strategies.
• 摘要：远程监督关系提取 (DSRE) 仍然是 NLP 中长期存在的挑战，其中模型必须从嘈杂的包级注释中学习，同时进行句子级预测。虽然现有最先进的 (SoTA) DSRE 模型依赖于特定于任务的训练，但它们与使用大语言模型 (LLM) 的上下文学习 (ICL) 的集成仍未得到充分探索。一个关键的挑战是，由于注释噪音，法学硕士可能无法正确学习关系语义。为此，我们提出了 HYDRE——混合远程监督关系提取框架。它首先使用经过训练的 DSRE 模型来识别给定测试句子的 top-k 候选关系，然后使用新颖的动态样本检索策略从训练数据中提取可靠的句子级样本，然后在 LLM 提示中提供输出最终关系。我们进一步将 HYDRE 扩展到资源匮乏语言中 RE 的跨语言设置。使用可用的英语 DSRE 训练数据，我们评估了所有英语方法以及新策划的基准，涵盖四种不同的低资源印度语言——奥里亚语、桑塔利语、曼尼普里语和图鲁语。与之前的 SoTA DSRE 模型相比，HYDRE 在英语方面的 F1 分数提高了多达 20 个，在印度语方面的 F1 分数平均提高了 17 个。与其他提示策略相比，详细的消融展示了 HYDRE 的功效。

Title: KrishokBondhu: A Retrieval-Augmented Voice-Based Agricultural Advisory Call Center for Bengali Farmers

• Authors: Mohd Ruhul Ameen, Akif Islam, Farjana Aktar, M. Saifuzzaman Rafat
• Subjects: cs.CL, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2510.18355
• Pdf URL: https://arxiv.org/pdf/2510.18355
• Copy Paste: [[2510.18355]] KrishokBondhu: A Retrieval-Augmented Voice-Based Agricultural Advisory Call Center for Bengali Farmers(https://arxiv.org/abs/2510.18355)
• Keywords: language model, retrieval-augmented generation
• Abstract: In Bangladesh, many farmers continue to face challenges in accessing timely, expert-level agricultural guidance. This paper presents KrishokBondhu, a voice-enabled, call-centre-integrated advisory platform built on a Retrieval-Augmented Generation (RAG) framework, designed specifically for Bengali-speaking farmers. The system aggregates authoritative agricultural handbooks, extension manuals, and NGO publications; applies Optical Character Recognition (OCR) and document-parsing pipelines to digitize and structure the content; and indexes this corpus in a vector database for efficient semantic retrieval. Through a simple phone-based interface, farmers can call the system to receive real-time, context-aware advice: speech-to-text converts the Bengali query, the RAG module retrieves relevant content, a large language model (Gemma 3-4B) generates a context-grounded response, and text-to-speech delivers the answer in natural spoken Bengali. In a pilot evaluation, KrishokBondhu produced high-quality responses for 72.7% of diverse agricultural queries covering crop management, disease control, and cultivation practices. Compared to the KisanQRS benchmark, the system achieved a composite score of 4.53 (vs. 3.13) on a 5-point scale, a 44.7% improvement, with especially large gains in contextual richness (+367%) and completeness (+100.4%), while maintaining comparable relevance and technical specificity. Semantic similarity analysis further revealed a strong correlation between retrieved context and answer quality, emphasizing the importance of grounding generative responses in curated documentation. KrishokBondhu demonstrates the feasibility of integrating call-centre accessibility, multilingual voice interaction, and modern RAG techniques to deliver expert-level agricultural guidance to remote Bangladeshi farmers, paving the way toward a fully AI-driven agricultural advisory ecosystem.
• 摘要：在孟加拉国，许多农民在获得及时的专家级农业指导方面继续面临挑战。本文介绍了 KrishokBondhu，这是一个基于检索增强生成 (RAG) 框架的语音支持呼叫中心集成咨询平台，专为孟加拉语农民设计。系统聚合权威农业手册、推广手册、NGO出版物；应用光学字符识别 (OCR) 和文档解析管道对内容进行数字化和结构化；并将该语料库索引到向量数据库中以进行有效的语义检索。通过一个简单的基于电话的界面，农民可以调用系统来接收实时的上下文感知建议：语音到文本转换孟加拉语查询，RAG 模块检索相关内容，大型语言模型 (Gemma 3-4B) 生成基于上下文的响应，文本到语音以自然孟加拉语口语提供答案。在一项试点评估中，KrishokBondhu 对 72.7% 的各种农业问题做出了高质量的答复，涵盖作物管理、疾病控制和耕作实践。与 KisanQRS 基准相比，该系统在 5 分制上的综合得分为 4.53（对比 3.13），提高了 44.7%，在上下文丰富性 (+367%) 和完整性 (+100.4%) 方面取得了特别大的进步，同时保持了可比较的相关性和技术特异性。语义相似性分析进一步揭示了检索到的上下文和答案质量之间的强相关性，强调了将生成性响应扎根于策划文档中的重要性。 KrishokBondhu 展示了整合呼叫中心可访问性、多语言语音交互和现代 RAG 技术为偏远的孟加拉国农民提供专家级农业指导的可行性，为完全由人工智能驱动的农业咨询生态系统铺平了道路。

Title: KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

• Authors: Donghyeon Ko, Yeguk Jin, Kyubyung Chae, Byungwook Lee, Chansong Jo, Sookyo In, Jaehong Lee, Taesup Kim, Donghyun Kwak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18368
• Pdf URL: https://arxiv.org/pdf/2510.18368
• Copy Paste: [[2510.18368]] KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs(https://arxiv.org/abs/2510.18368)
• Keywords: language model, llm
• Abstract: We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 1,000 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 33.7% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, our analysis of reasoning LLMs shows that engaging reasoning capabilities in the factual QA task can both help models better elicit their latent knowledge and improve their ability to abstain when uncertain. KoSimpleQA can be found at this https URL.
• 摘要：我们提出了 $\textbf{Korean SimpleQA (KoSimpleQA)}$，这是一个评估大型语言模型 (LLM) 真实性的基准，重点关注韩国文化知识。 KoSimpleQA 的设计具有挑战性，但易于评分，由 1,000 个简短的事实调查问题组成，并提供明确的答案。我们对支持韩语的不同规模的不同开源法学硕士进行了全面评估，发现即使是最强的模型也只有 33.7% 的时间生成正确答案，这凸显了 KoSimpleQA 的挑战性。值得注意的是，KoSimpleQA 上的性能排名与 English SimpleQA 上的性能排名有很大不同，凸显了我们数据集的独特价值。此外，我们对推理法学硕士的分析表明，在事实问答任务中引入推理能力既可以帮助模型更好地引出其潜在知识，又可以提高模型在不确定时放弃的能力。 KoSimpleQA 可以在此 https URL 找到。

Title: Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning

• Authors: Monorama Swain, Bubai Maji, Jagabandhu Mishra, Markus Schedl, Anders Søgaard, Jesper Rindom Jensen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18374
• Pdf URL: https://arxiv.org/pdf/2510.18374
• Copy Paste: [[2510.18374]] Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning(https://arxiv.org/abs/2510.18374)
• Keywords: prompt
• Abstract: In this work, we address the challenge of building fair English ASR systems for second-language speakers. Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 accent groups, indicating significant fairness gaps. To mitigate this, we propose fairness-prompted finetuning with lightweight adapters, incorporating Spectral Decoupling (SD), Group Distributionally Robust Optimization (Group-DRO), and Invariant Risk Minimization (IRM). Our proposed fusion of traditional empirical risk minimization (ERM) with cross-entropy and fairness-driven objectives (SD, Group DRO, and IRM) enhances fairness across accent groups while maintaining overall recognition accuracy. In terms of macro-averaged word error rate, our approach achieves a relative improvement of 58.7% and 58.5% over the large pretrained Whisper and SeamlessM4T, and 9.7% and 7.8% over them, finetuning with standard empirical risk minimization with cross-entropy loss.
• 摘要：在这项工作中，我们解决了为第二语言使用者构建公平的英语 ASR 系统的挑战。我们对广泛使用的 ASR 模型 Whisper 和 Seamless-M4T 的分析表明，26 个口音组的单词错误率 (WER) 存在较大波动，表明存在显着的公平性差距。为了缓解这一问题，我们提出使用轻量级适配器进行公平促进的微调，结合频谱解耦（SD）、组分布鲁棒优化（Group-DRO）和不变风险最小化（IRM）。我们提出的传统经验风险最小化 (ERM) 与交叉熵和公平驱动目标（SD、Group DRO 和 IRM）的融合增强了跨口音群体的公平性，同时保持了整体识别准确性。就宏观平均错误率而言，我们的方法比大型预训练 Whisper 和 SeamlessM4T 分别提高了 58.7% 和 58.5%，比它们分别提高了 9.7% 和 7.8%，并通过交叉熵损失的标准经验风险最小化进行微调。

Title: MENTOR: A Reinforcement Learning Framework for Model Enhancement via Teacher-Optimized Rewards in Small Models

• Authors: ChangSu Choi, Hoyun Song, Dongyeon Kim, WooHyeon Jung, Minkyung Cho, Sunjin Park, NohHyeob Bae, Seona Yu, KyungTae Lim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18383
• Pdf URL: https://arxiv.org/pdf/2510.18383
• Copy Paste: [[2510.18383]] MENTOR: A Reinforcement Learning Framework for Model Enhancement via Teacher-Optimized Rewards in Small Models(https://arxiv.org/abs/2510.18383)
• Keywords: language model, llm
• Abstract: Distilling the tool-using capabilities of large language models (LLMs) into smaller, more efficient small language models (SLMs) is a key challenge for their practical application. The predominant approach, supervised fine-tuning (SFT), suffers from poor generalization as it trains models to imitate a static set of teacher trajectories rather than learn a robust methodology. While reinforcement learning (RL) offers an alternative, the standard RL using sparse rewards fails to effectively guide SLMs, causing them to struggle with inefficient exploration and adopt suboptimal strategies. To address these distinct challenges, we propose MENTOR, a framework that synergistically combines RL with teacher-guided distillation. Instead of simple imitation, MENTOR employs an RL-based process to learn a more generalizable policy through exploration. In addition, to solve the problem of reward sparsity, it uses a teacher's reference trajectory to construct a dense, composite teacher-guided reward that provides fine-grained guidance. Extensive experiments demonstrate that MENTOR significantly improves the cross-domain generalization and strategic competence of SLMs compared to both SFT and standard sparse-reward RL baselines.
• 摘要：将大语言模型（LLM）的工具使用能力提炼成更小、更高效的小语言模型（SLM）是其实际应用的关键挑战。主要方法监督微调（SFT）泛化能力较差，因为它训练模型来模仿一组静态的教师轨迹，而不是学习稳健的方法。虽然强化学习 (RL) 提供了一种替代方案，但使用稀疏奖励的标准 RL 无法有效指导 SLM，导致它们陷入低效探索并采用次优策略。为了应对这些独特的挑战，我们提出了 MENTOR，这是一个将强化学习与教师指导的精炼协同结合的框架。 MENTOR 不是简单的模仿，而是采用基于强化学习的过程，通过探索来学习更通用的策略。此外，为了解决奖励稀疏的问题，它使用教师的参考轨迹构建密集的、复合的教师引导奖励，提供细粒度的指导。大量实验表明，与 SFT 和标准稀疏奖励 RL 基线相比，MENTOR 显着提高了 SLM 的跨域泛化和策略能力。

Title: Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

• Authors: Siyuan Yan, Guo-Qing Jiang, Yuchen Zhang, Xiaoxing Ma, Ran Zhu, Chun Cao, Jingwei Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18413
• Pdf URL: https://arxiv.org/pdf/2510.18413
• Copy Paste: [[2510.18413]] Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference(https://arxiv.org/abs/2510.18413)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods alleviate these costs but rely on heuristic patterns that struggle to recall critical key-value (KV) pairs for each query, resulting in accuracy degradation. We introduce Adamas, a lightweight yet highly accurate sparse attention mechanism designed for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and leverages Manhattan-distance estimation for efficient top-k selections. Experiments show that Adamas matches the accuracy of full attention with only a 64-token budget, achieves near-lossless performance at 128, and supports up to 8x higher sparsity than prior state-of-the-art (SOTA) methods while delivering up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences. Remarkably, Adamas attains comparable or even lower perplexity than full attention, underscoring its effectiveness in maintaining accuracy under aggressive sparsity.
• 摘要：大型语言模型 (LLM) 现在支持数十万到数百万个标记的上下文窗口，从而支持长文档摘要、大规模代码合成、多文档问答和持久多轮对话等应用。然而，这种扩展的上下文加剧了自注意力的二次成本，导致自回归解码中出现严重的延迟。现有的稀疏注意力方法可以减轻这些成本，但依赖于启发式模式，很难回忆起每个查询的关键键值 (KV) 对，从而导致准确性下降。我们引入了 Adamas，这是一种轻量级但高度准确的稀疏注意力机制，专为长上下文推理而设计。 Adamas 应用 Hadamard 变换、分桶化和 2 位压缩来生成紧凑的表示，并利用曼哈顿距离估计来实现高效的 top-k 选择。实验表明，Adamas 仅用 64 个令牌预算即可达到完全注意力的准确性，在 128 时实现近乎无损的性能，并支持比先前最先进 (SOTA) 方法高出 8 倍的稀疏度，同时在 32K 长度的序列上提供高达 4.4 倍的自注意力和 1.5 倍的端到端加速。值得注意的是，Adamas 获得了与完全注意力相当甚至更低的困惑度，这突显了它在极度稀疏的情况下保持准确性的有效性。

Title: Chain-of-Conceptual-Thought: Eliciting the Agent to Deeply Think within the Response

• Authors: Qingqing Gu, Dan Wang, Yue Zhao, Xiaoyu Wang, Zhonglin Jiang, Yong Chen, Hongyan Li, Luo Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18434
• Pdf URL: https://arxiv.org/pdf/2510.18434
• Copy Paste: [[2510.18434]] Chain-of-Conceptual-Thought: Eliciting the Agent to Deeply Think within the Response(https://arxiv.org/abs/2510.18434)
• Keywords: llm, prompt, chain-of-thought, agent
• Abstract: Chain-of-Thought (CoT) is widely applied to improve the LLM capability in math, coding and reasoning tasks. However, its performance is limited for open-domain tasks since there are no clearly defined reasoning steps or logical transitions. To mitigate such challenges, we propose another prompt-based paradigm called Chain of Conceptual Thought (CoCT), where the LLM first tags a concept, then generates the detailed content. The chain of concepts is allowed within the utterance, encouraging the LLM's deep and strategic thinking. We experiment with this paradigm in daily and emotional support conversations where the concept is comprised of emotions, strategies and topics. Automatic, human and model evaluations suggest that CoCT surpasses baselines such as Self-Refine, ECoT, ToT, SoT and RAG, suggesting a potential effective prompt-based paradigm of LLM for a wider scope of tasks.
• 摘要：思想链（CoT）广泛应用于提高法学硕士在数学、编码和推理任务中的能力。然而，它的性能对于开放域任务来说是有限的，因为没有明确定义的推理步骤或逻辑转换。为了缓解此类挑战，我们提出了另一种基于提示的范例，称为概念思维链（CoCT），其中法学硕士首先标记一个概念，然后生成详细内容。话语中允许出现概念链，鼓励法学硕士进行深入的战略思考。我们在日常和情感支持对话中尝试这种范例，其中概念由情感、策略和主题组成。自动、人工和模型评估表明，CoCT 超越了 Self-Refine、ECoT、ToT、SoT 和 RAG 等基线，这表明 LLM 的潜在有效范式基于提示，适用于更广泛的任务。

Title: Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation

• Authors: Yasser Hamidullah, Koel Dutta Chowdury, Yusser Al-Ghussin, Shakib Yazdani, Cennet Oguz, Josef van Genabith, Cristina España-Bonet
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18439
• Pdf URL: https://arxiv.org/pdf/2510.18439
• Copy Paste: [[2510.18439]] Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation(https://arxiv.org/abs/2510.18439)
• Keywords: language model, hallucination
• Abstract: Hallucination, where models generate fluent text unsupported by visual evidence, remains a major flaw in vision-language models and is particularly critical in sign language translation (SLT). In SLT, meaning depends on precise grounding in video, and gloss-free models are especially vulnerable because they map continuous signer movements directly into natural language without intermediate gloss supervision that serves as alignment. We argue that hallucinations arise when models rely on language priors rather than visual input. To capture this, we propose a token-level reliability measure that quantifies how much the decoder uses visual information. Our method combines feature-based sensitivity, which measures internal changes when video is masked, with counterfactual signals, which capture probability differences between clean and altered video inputs. These signals are aggregated into a sentence-level reliability score, providing a compact and interpretable measure of visual grounding. We evaluate the proposed measure on two SLT benchmarks (PHOENIX-2014T and CSL-Daily) with both gloss-based and gloss-free models. Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations. Beyond these quantitative trends, we also find that reliability distinguishes grounded tokens from guessed ones, allowing risk estimation without references; when combined with text-based signals (confidence, perplexity, or entropy), it further improves hallucination risk estimation. Qualitative analysis highlights why gloss-free models are more susceptible to hallucinations. Taken together, our findings establish reliability as a practical and reusable tool for diagnosing hallucinations in SLT, and lay the groundwork for more robust hallucination detection in multimodal generation.
• 摘要：幻觉（模型在没有视觉证据支持的情况下生成流畅的文本）仍然是视觉语言模型的一个主要缺陷，在手语翻译（SLT）中尤其重要。在 SLT 中，意义取决于视频中的精确基础，而无光泽模型尤其容易受到攻击，因为它们将连续的手语动作直接映射到自然语言中，而无需中间光泽监督作为对齐。我们认为，当模型依赖于语言先验而不是视觉输入时，就会出现幻觉。为了捕捉这一点，我们提出了一种令牌级可靠性度量，该度量可以量化解码器使用视觉信息的程度。我们的方法结合了基于特征的灵敏度（测量视频被屏蔽时的内部变化）与反事实信号（捕获干净和更改的视频输入之间的概率差异）。这些信号被聚合成句子级的可靠性得分，提供紧凑且可解释的视觉基础测量。我们使用基于光泽和无光泽的模型在两个 SLT 基准（PHOENIX-2014T 和 CSL-Daily）上评估所提出的措施。我们的结果表明，可靠性可以预测幻觉率，在数据集和架构中进行概括，并在视觉退化的情况下降低。除了这些定量趋势之外，我们还发现可靠性将真实的代币与猜测的代币区分开来，允许在没有参考的情况下进行风险评估；当与基于文本的信号（置信度、困惑度或熵）相结合时，它可以进一步改善幻觉风险估计。定性分析强调了为什么无光泽模型更容易产生幻觉。总而言之，我们的研究结果确立了作为 SLT 中诊断幻觉的实用且可重复使用的工具的可靠性，并为多模式生成中更强大的幻觉检测奠定了基础。

Title: Engagement Undermines Safety: How Stereotypes and Toxicity Shape Humor in Language Models

• Authors: Atharvan Dogra, Soumya Suvra Ghosal, Ameet Deshpande, Ashwin Kalyan, Dinesh Manocha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18454
• Pdf URL: https://arxiv.org/pdf/2510.18454
• Copy Paste: [[2510.18454]] Engagement Undermines Safety: How Stereotypes and Toxicity Shape Humor in Language Models(https://arxiv.org/abs/2510.18454)
• Keywords: language model, llm, prompt
• Abstract: Large language models are increasingly used for creative writing and engagement content, raising safety concerns about the outputs. Therefore, casting humor generation as a testbed, this work evaluates how funniness optimization in modern LLM pipelines couples with harmful content by jointly measuring humor, stereotypicality, and toxicity. This is further supplemented by analyzing incongruity signals through information-theoretic metrics. Across six models, we observe that harmful outputs receive higher humor scores which further increase under role-based prompting, indicating a bias amplification loop between generators and evaluators. Information-theoretic analyses show harmful cues widen predictive uncertainty and surprisingly, can even make harmful punchlines more expected for some models, suggesting structural embedding in learned humor distributions. External validation on an additional satire-generation task with human perceived funniness judgments shows that LLM satire increases stereotypicality and typically toxicity, including for closed models. Quantitatively, stereotypical/toxic jokes gain $10-21\%$ in mean humor score, stereotypical jokes appear $11\%$ to $28\%$ more often among the jokes marked funny by LLM-based metric and up to $10\%$ more often in generations perceived as funny by humans.
• 摘要：大型语言模型越来越多地用于创意写作和参与内容，引发了对输出的安全担忧。因此，这项工作将幽默生成作为测试平台，通过联合测量幽默、刻板性和毒性来评估现代法学硕士管道中的搞笑优化如何与有害内容相结合。通过信息论指标分析不协调信号进一步补充了这一点。在六个模型中，我们观察到有害的输出会获得更高的幽默分数，并且在基于角色的提示下进一步增加，这表明生成器和评估器之间存在偏差放大循环。信息论分析表明，有害线索扩大了预测的不确定性，令人惊讶的是，甚至可以使某些模型更容易预期有害的妙语，这表明学习幽默分布中存在结构嵌入。通过人类感知的搞笑判断对额外的讽刺生成任务进行的外部验证表明，LLM 讽刺会增加刻板印象和典型毒性，包括封闭模型。从数量上看，刻板/有毒笑话的平均幽默得分增加了 $10-21\%$，刻板笑话在基于 LLM 的指标标记为有趣的笑话中出现的频率更高，为 $11\%$ 到 $28\%$，在被人类认为有趣的几代人中出现频率高达 $10\%$。

Title: ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks

• Authors: Liyang He, Yuren Zhang, Ziwei Zhu, Zhenghui Li, Shiwei Tong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18455
• Pdf URL: https://arxiv.org/pdf/2510.18455
• Copy Paste: [[2510.18455]] ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks(https://arxiv.org/abs/2510.18455)
• Keywords: retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics: the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such a benchmark introduces a critical requirement for player-centric authenticity to ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and a dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games to create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Code is avaliable at: this https URL.
• 摘要：检索增强生成 (RAG) 系统在在线游戏等动态领域中变得越来越重要，但缺乏专用基准阻碍了该领域的标准化评估。核心难点在于双重动力：游戏内容更新和玩家群体焦点转移之间的持续相互作用。此外，自动化此类基准的必要性引入了以玩家为中心的真实性的关键要求，以确保生成的问题是现实的。为了解决这一综合挑战，我们引入了 ChronoPlay，这是一种用于自动、连续生成游戏 RAG 基准的新颖框架。 ChronoPlay 利用双动态更新机制来跟踪这两种形式的变化，并利用来自官方来源和玩家社区的双源合成引擎来确保事实的正确性和真实的查询模式。我们在三种不同的游戏上实例化我们的框架，为游戏领域创建第一个动态 RAG 基准，为这些复杂而现实的条件下的模型性能提供新的见解。代码可在以下位置获取：此 https URL。

Title: CEFR-Annotated WordNet: LLM-Based Proficiency-Guided Semantic Database for Language Learning

• Authors: Masato Kikuchi, Masatsugu Ono, Toshioki Soga, Tetsu Tanabe, Tadachika Ozono
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18466
• Pdf URL: https://arxiv.org/pdf/2510.18466
• Copy Paste: [[2510.18466]] CEFR-Annotated WordNet: LLM-Based Proficiency-Guided Semantic Database for Language Learning(https://arxiv.org/abs/2510.18466)
• Keywords: language model, llm
• Abstract: Although WordNet is a valuable resource owing to its structured semantic networks and extensive vocabulary, its fine-grained sense distinctions can be challenging for second-language learners. To address this, we developed a WordNet annotated with the Common European Framework of Reference for Languages (CEFR), integrating its semantic networks with language-proficiency levels. We automated this process using a large language model to measure the semantic similarity between sense definitions in WordNet and entries in the English Vocabulary Profile Online. To validate our method, we constructed a large-scale corpus containing both sense and CEFR-level information from our annotated WordNet and used it to develop contextual lexical classifiers. Our experiments demonstrate that models fine-tuned on our corpus perform comparably to those trained on gold-standard annotations. Furthermore, by combining our corpus with the gold-standard data, we developed a practical classifier that achieves a Macro-F1 score of 0.81, indicating the high accuracy of our annotations. Our annotated WordNet, corpus, and classifiers are publicly available to help bridge the gap between natural language processing and language education, thereby facilitating more effective and efficient language learning.
• 摘要：尽管 WordNet 因其结构化语义网络和广泛的词汇而成为宝贵的资源，但其细粒度的语义区分对于第二语言学习者来说可能具有挑战性。为了解决这个问题，我们开发了一个带有欧洲共同语言参考框架 (CEFR) 注释的 WordNet，将其语义网络与语言熟练程度相结合。我们使用大型语言模型来自动化此过程，以测量 WordNet 中的语义定义与在线英语词汇表中的条目之间的语义相似性。为了验证我们的方法，我们构建了一个大规模语料库，其中包含来自带注释的 WordNet 的语义和 CEFR 级别信息，并用它来开发上下文词汇分类器。我们的实验表明，在我们的语料库上进行微调的模型的性能与在黄金标准注释上训练的模型相当。此外，通过将我们的语料库与黄金标准数据相结合，我们开发了一个实用的分类器，其 Macro-F1 得分为 0.81，表明我们的注释具有很高的准确性。我们的带注释的 WordNet、语料库和分类器是公开可用的，有助于弥合自然语言处理和语言教育之间的差距，从而促进更有效和高效的语言学习。

Title: IMB: An Italian Medical Benchmark for Question Answering

• Authors: Antonio Romano, Giuseppe Riccio, Mariano Barone, Marco Postiglione, Vincenzo Moscato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18468
• Pdf URL: https://arxiv.org/pdf/2510.18468
• Copy Paste: [[2510.18468]] IMB: An Italian Medical Benchmark for Question Answering(https://arxiv.org/abs/2510.18468)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Online medical forums have long served as vital platforms where patients seek professional healthcare advice, generating vast amounts of valuable knowledge. However, the informal nature and linguistic complexity of forum interactions pose significant challenges for automated question answering systems, especially when dealing with non-English languages. We present two comprehensive Italian medical benchmarks: \textbf{IMB-QA}, containing 782,644 patient-doctor conversations from 77 medical categories, and \textbf{IMB-MCQA}, comprising 25,862 multiple-choice questions from medical specialty examinations. We demonstrate how Large Language Models (LLMs) can be leveraged to improve the clarity and consistency of medical forum data while retaining their original meaning and conversational style, and compare a variety of LLM architectures on both open and multiple-choice question answering tasks. Our experiments with Retrieval Augmented Generation (RAG) and domain-specific fine-tuning reveal that specialized adaptation strategies can outperform larger, general-purpose models in medical question answering tasks. These findings suggest that effective medical AI systems may benefit more from domain expertise and efficient information retrieval than from increased model scale. We release both datasets and evaluation frameworks in our GitHub repository to support further research on multilingual medical question answering: this https URL.
• 摘要：在线医疗论坛长期以来一直是患者寻求专业医疗建议的重要平台，产生了大量有价值的知识。然而，论坛互动的非正式性质和语言复杂性给自动问答系统带来了重大挑战，特别是在处理非英语语言时。我们提出了两个全面的意大利医学基准：\textbf{IMB-QA}，包含来自 77 个医学类别的 782,644 条医患对话，以及 \textbf{IMB-MCQA}，包含来自医学专业考试的 25,862 个多项选择题。我们演示了如何利用大型语言模型 (LLM) 来提高医学论坛数据的清晰度和一致性，同时保留其原始含义和对话风格，并在开放式和多项选择题回答任务上比较各种 LLM 架构。我们的检索增强生成（RAG）和特定领域微调的实验表明，在医学问答任务中，专门的适应策略可以胜过更大的通用模型。这些发现表明，有效的医疗人工智能系统可能更多地受益于领域专业知识和高效的信息检索，而不是增加的模型规模。我们在 GitHub 存储库中发布了数据集和评估框架，以支持多语言医学问答的进一步研究：此 https URL。

Title: DART: A Structured Dataset of Regulatory Drug Documents in Italian for Clinical NLP

• Authors: Mariano Barone, Antonio Laudante, Giuseppe Riccio, Antonio Romano, Marco Postiglione, Vincenzo Moscato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18475
• Pdf URL: https://arxiv.org/pdf/2510.18475
• Copy Paste: [[2510.18475]] DART: A Structured Dataset of Regulatory Drug Documents in Italian for Clinical NLP(https://arxiv.org/abs/2510.18475)
• Keywords: language model, llm
• Abstract: The extraction of pharmacological knowledge from regulatory documents has become a key focus in biomedical natural language processing, with applications ranging from adverse event monitoring to AI-assisted clinical decision support. However, research in this field has predominantly relied on English-language corpora such as DrugBank, leaving a significant gap in resources tailored to other healthcare systems. To address this limitation, we introduce DART (Drug Annotation from Regulatory Texts), the first structured corpus of Italian Summaries of Product Characteristics derived from the official repository of the Italian Medicines Agency (AIFA). The dataset was built through a reproducible pipeline encompassing web-scale document retrieval, semantic segmentation of regulatory sections, and clinical summarization using a few-shot-tuned large language model with low-temperature decoding. DART provides structured information on key pharmacological domains such as indications, adverse drug reactions, and drug-drug interactions. To validate its utility, we implemented an LLM-based drug interaction checker that leverages the dataset to infer clinically meaningful interactions. Experimental results show that instruction-tuned LLMs can accurately infer potential interactions and their clinical implications when grounded in the structured textual fields of DART. We publicly release our code on GitHub: this https URL.
• 摘要：从监管文件中提取药理学知识已成为生物医学自然语言处理的重点，其应用范围从不良事件监测到人工智能辅助临床决策支持。然而，该领域的研究主要依赖于 DrugBank 等英语语料库，导致针对其他医疗保健系统的资源存在巨大缺口。为了解决这一限制，我们引入了 DART（来自监管文本的药物注释），这是第一个源自意大利药品管理局 (AIFA) 官方存储库的意大利产品特性摘要结构化语料库。该数据集是通过可重复的管道构建的，包括网络规模的文档检索、监管部分的语义分割以及使用带有低温解码的几次镜头调整的大型语言模型进行的临床总结。 DART 提供关键药理学领域的结构化信息，例如适应症、药物不良反应和药物间相互作用。为了验证其实用性，我们实施了一个基于法学硕士的药物相互作用检查器，该检查器利用数据集来推断有临床意义的相互作用。实验结果表明，当基于 DART 的结构化文本字段时，经过指令调整的法学硕士可以准确推断潜在的相互作用及其临床意义。我们在 GitHub 上公开发布我们的代码：此 https URL。

Title: How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices

• Authors: Han Peng, Peiyu Liu, Zican Dong, Daixuan Cheng, Junyi Li, Yiru Tang, Shuo Wang, Wayne Xin Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18480
• Pdf URL: https://arxiv.org/pdf/2510.18480
• Copy Paste: [[2510.18480]] How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices(https://arxiv.org/abs/2510.18480)
• Keywords: language model
• Abstract: Diffusion language models (DLMs) have emerged as a promising alternative to the long-dominant autoregressive (AR) paradigm, offering a parallelable decoding process that could yield greater efficiency. Yet, in practice, current open-source DLMs often underperform their AR counterparts in speed, limiting their real-world utility. This work presents a systematic study of DLM efficiency, identifying key issues in prior evaluation methods. Through empirical benchmarking and a roofline-based theoretical analysis, we demonstrate that AR models generally achieve higher throughput, while DLMs consistently lag. We also investigate acceleration strategies, finding that techniques like dual cache and parallel decoding mainly offer gains at small batch sizes, with their benefits diminishing upon scaling. Our findings underscore the necessity of robust evaluation methods and improved acceleration strategies to advance research on DLMs.
• 摘要：扩散语言模型 (DLM) 已成为长期主导的自回归 (AR) 范式的有前途的替代方案，提供可并行的解码过程，可以产生更高的效率。然而，在实践中，当前的开源 DLM 在速度上往往不如 AR 同类产品，限制了它们在现实世界中的实用性。这项工作对 DLM 效率进行了系统研究，确定了先前评估方法中的关键问题。通过实证基准测试和基于屋顶线的理论分析，我们证明 AR 模型通常可以实现更高的吞吐量，而 DLM 始终滞后。我们还研究了加速策略，发现双缓存和并行解码等技术主要在小批量大小下提供增益，而它们的优势在扩展时会减弱。我们的研究结果强调了稳健的评估方法和改进的加速策略的必要性，以推进 DLM 的研究。

Title: Identity-Aware Large Language Models require Cultural Reasoning

• Authors: Alistair Plum, Anne-Marie Lutgen, Christoph Purschke, Achim Rettinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18510
• Pdf URL: https://arxiv.org/pdf/2510.18510
• Copy Paste: [[2510.18510]] Identity-Aware Large Language Models require Cultural Reasoning(https://arxiv.org/abs/2510.18510)
• Keywords: language model
• Abstract: Large language models have become the latest trend in natural language processing, heavily featuring in the digital tools we use every day. However, their replies often reflect a narrow cultural viewpoint that overlooks the diversity of global users. This missing capability could be referred to as cultural reasoning, which we define here as the capacity of a model to recognise culture-specific knowledge values and social norms, and to adjust its output so that it aligns with the expectations of individual users. Because culture shapes interpretation, emotional resonance, and acceptable behaviour, cultural reasoning is essential for identity-aware AI. When this capacity is limited or absent, models can sustain stereotypes, ignore minority perspectives, erode trust, and perpetuate hate. Recent empirical studies strongly suggest that current models default to Western norms when judging moral dilemmas, interpreting idioms, or offering advice, and that fine-tuning on survey data only partly reduces this tendency. The present evaluation methods mainly report static accuracy scores and thus fail to capture adaptive reasoning in context. Although broader datasets can help, they cannot alone ensure genuine cultural competence. Therefore, we argue that cultural reasoning must be treated as a foundational capability alongside factual accuracy and linguistic coherence. By clarifying the concept and outlining initial directions for its assessment, a foundation is laid for future systems to be able to respond with greater sensitivity to the complex fabric of human culture.
• 摘要：大型语言模型已成为自然语言处理的最新趋势，在我们日常使用的数字工具中占有重要地位。然而，他们的回复往往反映了一种狭隘的文化观点，忽视了全球用户的多样性。这种缺失的能力可以称为文化推理，我们在这里将其定义为模型识别特定文化的知识价值观和社会规范并调整其输出以使其符合个人用户的期望的能力。由于文化塑造解释、情感共鸣和可接受的行为，因此文化推理对于身份感知人工智能至关重要。当这种能力有限或缺失时，模型就会维持刻板印象，忽视少数群体的观点，削弱信任，并使仇恨永久化。最近的实证研究强烈表明，当前的模型在判断道德困境、解释习语或提供建议时默认了西方规范，而对调查数据的微调只能部分减少这种趋势。目前的评估方法主要报告静态准确性分数，因此无法捕获上下文中的自适应推理。尽管更广泛的数据集可以有所帮助，但仅靠它们并不能确保真正的文化能力。因此，我们认为文化推理必须被视为与事实准确性和语言连贯性并列的基本能力。通过澄清概念并概述其评估的初步方向，为未来的系统能够以更高的灵敏度对人类文化的复杂结构做出反应奠定了基础。

Title: Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency

• Authors: Svetlana Maslenkova, Clement Christophe, Marco AF Pimentel, Tathagata Raha, Muhammad Umar Salman, Ahmed Al Mahrooqi, Avani Gupta, Shadab Khan, Ronnie Rajan, Praveenkumar Kanithi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18556
• Pdf URL: https://arxiv.org/pdf/2510.18556
• Copy Paste: [[2510.18556]] Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency(https://arxiv.org/abs/2510.18556)
• Keywords: language model, llm
• Abstract: Large language models offer transformative potential for healthcare, yet their responsible and equitable development depends critically on a deeper understanding of how training data characteristics influence model behavior, including the potential for bias. Current practices in dataset curation and bias assessment often lack the necessary transparency, creating an urgent need for comprehensive evaluation frameworks to foster trust and guide improvements. In this study, we present an in-depth analysis of potential downstream biases in clinical language models, with a focus on differential opioid prescription tendencies across diverse demographic groups, such as ethnicity, gender, and age. As part of this investigation, we introduce HC4: Healthcare Comprehensive Commons Corpus, a novel and extensively curated pretraining dataset exceeding 89 billion tokens. Our evaluation leverages both established general benchmarks and a novel, healthcare-specific methodology, offering crucial insights to support fairness and safety in clinical AI applications.
• 摘要：大型语言模型为医疗保健提供了变革潜力，但其负责任和公平的发展关键取决于对训练数据特征如何影响模型行为（包括潜在偏差）的更深入理解。当前数据集管理和偏见评估的实践往往缺乏必要的透明度，迫切需要全面的评估框架来促进信任和指导改进。在这项研究中，我们对临床语言模型中潜在的下游偏差进行了深入分析，重点关注不同人口群体（例如种族、性别和年龄）的阿片类药物处方倾向差异。作为本次调查的一部分，我们引入了 HC4：医疗保健综合共享语料库，这是一个新颖且精心策划的预训练数据集，包含超过 890 亿个代币。我们的评估利用了既定的通用基准和新颖的医疗保健特定方法，为支持临床人工智能应用的公平性和安全性提供了重要的见解。

Title: Large language models for folktale type automation based on motifs: Cinderella case study

• Authors: Tjaša Arčon, Marko Robnik-Šikonja, Polona Tratnik
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18561
• Pdf URL: https://arxiv.org/pdf/2510.18561
• Copy Paste: [[2510.18561]] Large language models for folktale type automation based on motifs: Cinderella case study(https://arxiv.org/abs/2510.18561)
• Keywords: language model
• Abstract: Artificial intelligence approaches are being adapted to many research areas, including digital humanities. We built a methodology for large-scale analyses in folkloristics. Using machine learning and natural language processing, we automatically detected motifs in a large collection of Cinderella variants and analysed their similarities and differences with clustering and dimensionality reduction. The results show that large language models detect complex interactions in tales, enabling computational analysis of extensive text collections and facilitating cross-lingual comparisons.
• 摘要：人工智能方法正在适应许多研究领域，包括数字人文。我们建立了一种用于民俗学大规模分析的方法。使用机器学习和自然语言处理，我们自动检测大量灰姑娘变体中的主题，并通过聚类和降维分析它们的异同。结果表明，大型语言模型可以检测故事中复杂的交互，从而能够对大量文本集合进行计算分析并促进跨语言比较。

Title: MLMA: Towards Multilingual with Mamba Based Architectures

• Authors: Mohamed Nabih Ali, Daniele Falavigna, Alessio Brutti
• Subjects: cs.CL, cs.SD
• Abstract URL: https://arxiv.org/abs/2510.18684
• Pdf URL: https://arxiv.org/pdf/2510.18684
• Copy Paste: [[2510.18684]] MLMA: Towards Multilingual with Mamba Based Architectures(https://arxiv.org/abs/2510.18684)
• Keywords: language model
• Abstract: Multilingual automatic speech recognition (ASR) remains a challenging task, especially when balancing performance across high- and low-resource languages. Recent advances in sequence modeling suggest that architectures beyond Transformers may offer better scalability and efficiency. In this work, we introduce MLMA (Multilingual Language Modeling with Mamba for ASR), a new approach that leverages the Mamba architecture--an efficient state-space model optimized for long-context sequence processing--for multilingual ASR. Using Mamba, MLMA implicitly incorporates language-aware conditioning and shared representations to support robust recognition across diverse languages. Experiments on standard multilingual benchmarks show that MLMA achieves competitive performance compared to Transformer-based architectures. These results highlight Mamba's potential as a strong backbone for scalable, efficient, and accurate multilingual speech recognition.
• 摘要：多语言自动语音识别 (ASR) 仍然是一项具有挑战性的任务，尤其是在平衡高资源语言和低资源语言之间的性能时。序列建模的最新进展表明 Transformer 之外的架构可以提供更好的可扩展性和效率。在这项工作中，我们介绍了 MLMA（使用 Mamba for ASR 进行多语言语言建模），这是一种利用 Mamba 架构（一种针对长上下文序列处理而优化的高效状态空间模型）进行多语言 ASR 的新方法。使用 Mamba，MLMA 隐式地结合了语言感知调节和共享表示，以支持跨不同语言的稳健识别。标准多语言基准测试表明，与基于 Transformer 的架构相比，MLMA 实现了具有竞争力的性能。这些结果凸显了 Mamba 作为可扩展、高效和准确的多语言语音识别的强大支柱的潜力。

Title: Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering

• Authors: Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18691
• Pdf URL: https://arxiv.org/pdf/2510.18691
• Copy Paste: [[2510.18691]] Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering(https://arxiv.org/abs/2510.18691)
• Keywords: llm, long context
• Abstract: This study is the first to investigate LLM comprehension capabilities over long-context (LC) medical QA of clinical relevance. Our comprehensive assessment spans a range of content-inclusion settings based on their relevance, LLM models of varying capabilities and datasets across task formulations, revealing insights on model size effects, limitations, underlying memorization issues and the benefits of reasoning models. Importantly, we examine the effect of RAG on medical LC comprehension, uncover best settings in single versus multi-document reasoning datasets and showcase RAG strategies for improvements over LC. We shed light into some of the evaluation aspects using a multi-faceted approach. Our qualitative and error analyses address open questions on when RAG is beneficial over LC, revealing common failure cases.
• 摘要：这项研究是第一个调查法学硕士对临床相关的长上下文（LC）医学质量保证的理解能力的研究。我们的综合评估涵盖了一系列基于相关性的内容包含设置、不同功能的法学硕士模型和跨任务制定的数据集，揭示了对模型大小影响、局限性、潜在记忆问题和推理模型的好处的见解。重要的是，我们研究了 RAG 对医学 LC 理解的影响，发现单文档推理数据集与多文档推理数据集中的最佳设置，并展示了改进 LC 的 RAG 策略。我们使用多方面的方法阐明了一些评估方面。我们的定性和误差分析解决了 RAG 何时比 LC 更有利的悬而未决的问题，揭示了常见的失败案例。

Title: SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish

• Authors: Josh McGiff, Nikola S. Nikolov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18725
• Pdf URL: https://arxiv.org/pdf/2510.18725
• Copy Paste: [[2510.18725]] SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish(https://arxiv.org/abs/2510.18725)
• Keywords: language model
• Abstract: Fine-tuning is widely used to tailor large language models for specific tasks such as neural machine translation (NMT). However, leveraging transfer learning is computationally expensive when fine-tuning large multilingual models with billions of parameters, thus creating a barrier to entry for researchers working on low-resource domains such as Irish translation. Parameter-efficient fine-tuning (PEFT) bridges this gap by training on a fraction of the original model parameters, with the Low-Rank Adaptation (LoRA) approach introducing small, trainable adapter layers. We introduce SemiAdapt and SemiLoRA as semi-supervised inference-efficient approaches that strengthen domain adaptation and lead to improved overall performance in NMT. We demonstrate that SemiAdapt can outperform full-domain fine-tuning, while most notably, SemiLoRA can propel PEFT methods to match or even outperform full-model fine-tuning. We further evaluate domain-by-dataset fine-tuning and demonstrate that our embedding-based inference methods perform especially well on larger and noisier corpora. All Irish translation models developed in this work are released as open resources. These methods aim to make high-quality domain adaptation and fine-tuning more accessible to researchers working with low-resource languages.
• 摘要：微调广泛用于为神经机器翻译（NMT）等特定任务定制大型语言模型。然而，在对具有数十亿参数的大型多语言模型进行微调时，利用迁移学习的计算成本很高，从而为爱尔兰翻译等低资源领域的研究人员设置了进入障碍。参数高效微调 (PEFT) 通过对原始模型参数的一小部分进行训练，并通过引入小型可训练适配器层的低秩适应 (LoRA) 方法来弥补这一差距。我们引入 SemiAdapt 和 SemiLoRA 作为半监督推理高效方法，增强领域适应并提高 NMT 的整体性能。我们证明了 SemiAdapt 可以胜过全域微调，而最值得注意的是，SemiLoRA 可以推动 PEFT 方法匹配甚至胜过全模型微调。我们进一步评估逐个数据集的微调，并证明我们的基于嵌入的推理方法在更大、噪声更大的语料库上表现得特别好。这项工作中开发的所有爱尔兰语翻译模型均作为开放资源发布。这些方法旨在使使用低资源语言的研究人员更容易获得高质量的领域适应和微调。

Title: Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation

• Authors: Ming Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.18731
• Pdf URL: https://arxiv.org/pdf/2510.18731
• Copy Paste: [[2510.18731]] Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation(https://arxiv.org/abs/2510.18731)
• Keywords: language model, llm
• Abstract: Large Language Models demonstrate strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC), a degradation in performance as information is revealed progressively in multi-turn settings. Motivated by the current progress on Reinforcement Learning with Verifiable Rewards (RLVR), we propose Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR), a framework that encourages models not only to generate correct answers, but also to judge the solvability of questions in the multi-turn conversation setting. Our approach employs a competence-gated curriculum that incrementally increases dialogue difficulty (in terms of instruction shards), stabilizing training while promoting reliability. Using multi-turn, on-policy rollouts and a mixed-reward system, RLAAR teaches models to balance problem-solving with informed abstention, reducing premature answering behaviors that cause LiC. Evaluated on LiC benchmarks, RLAAR significantly mitigates LiC performance decay (62.6% to 75.1%) and improves calibrated abstention rates (33.5% to 73.4%). Together, these results provide a practical recipe for building multi-turn reliable and trustworthy LLMs.
• 摘要：大型语言模型在单轮指令跟踪方面表现出强大的能力，但会遭受对话丢失（LiC）的困扰，即随着信息在多轮设置中逐渐显示而导致性能下降。受当前可验证奖励强化学习（RLVR）进展的推动，我们提出了可验证准确性和弃权奖励的课程强化学习（RLAAR），这是一个框架，鼓励模型不仅生成正确答案，而且在多轮对话环境中判断问题的可解决性。我们的方法采用能力门控课程，逐步增加对话难度（就指令碎片而言），稳定培训，同时提高可靠性。 RLAAR 使用多轮、按策略推出和混合奖励系统，教导模型在解决问题与知情弃权之间取得平衡，减少导致 LiC 的过早回答行为。根据 LiC 基准进行评估，RLAAR 显着减轻了 LiC 性能衰减（62.6% 至 75.1%），并提高了校准弃权率（33.5% 至 73.4%）。总之，这些结果为构建多轮可靠且值得信赖的法学硕士提供了实用的方法。

Title: Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweighting

• Authors: Taha Binhuraib, Greta Tuckute, Nicholas Blauch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18745
• Pdf URL: https://arxiv.org/pdf/2510.18745
• Copy Paste: [[2510.18745]] Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweighting(https://arxiv.org/abs/2510.18745)
• Keywords: language model, llm
• Abstract: Spatial functional organization is a hallmark of biological brains: neurons are arranged topographically according to their response properties, at multiple scales. In contrast, representations within most machine learning models lack spatial biases, instead manifesting as disorganized vector spaces that are difficult to visualize and interpret. Here, we propose a novel form of self-attention that turns Transformers into "Topoformers" with topographic organization. We introduce spatial querying - where keys and queries are arranged on 2D grids, and local pools of queries are associated with a given key - and spatial reweighting, where we convert the standard fully connected layer of self-attention into a locally connected layer. We first demonstrate the feasibility of our approach by training a 1-layer Topoformer on a sentiment classification task. Training with spatial querying encourages topographic organization in the queries and keys, and spatial reweighting separately encourages topographic organization in the values and self-attention outputs. We then apply the Topoformer motifs at scale, training a BERT architecture with a masked language modeling objective. We find that the topographic variant performs on par with a non-topographic control model on NLP benchmarks, yet produces interpretable topographic organization as evaluated via eight linguistic test suites. Finally, analyzing an fMRI dataset of human brain responses to a large set of naturalistic sentences, we demonstrate alignment between low-dimensional topographic variability in the Topoformer model and human brain language network. Scaling up Topoformers further holds promise for greater interpretability in NLP research, and for more accurate models of the organization of linguistic information in the human brain.
• 摘要：空间功能组织是生物大脑的一个标志：神经元根据其反应特性在多个尺度上进行拓扑排列。相比之下，大多数机器学习模型中的表示缺乏空间偏差，而是表现为难以可视化和解释的无序向量空间。在这里，我们提出了一种新颖的自注意力形式，将 Transformers 转变为具有拓扑组织的“Topoformers”。我们引入了空间查询（其中键和查询排列在 2D 网格上，并且本地查询池与给定键相关联）和空间重新加权（其中我们将标准的自注意力全连接层转换为本地连接层）。我们首先通过在情感分类任务上训练 1 层 Topoformer 来证明我们方法的可行性。空间查询训练鼓励查询和键中的拓扑组织，空间重新加权分别鼓励值和自注意力输出中的拓扑组织。然后，我们大规模应用 Topoformer 主题，训练具有掩码语言建模目标的 BERT 架构。我们发现，地形变体在 NLP 基准上的表现与非地形控制模型相当，但通过八个语言测试套件评估，可以产生可解释的地形组织。最后，通过分析人脑对大量自然句子的反应的功能磁共振成像数据集，我们证明了 Topoformer 模型中的低维地形变异性与人脑语言网络之间的一致性。扩大拓扑形成器的规模进一步有望提高 NLP 研究的可解释性，并为人脑中的语言信息组织提供更准确的模型。

Title: KAT-Coder Technical Report

• Authors: Zizheng Zhan, Ken Deng, Xiaojiang Zhang, Jinghui Wang, Huaixi Tang, Zhiyi Lai, Haoyang Huang, Wen Xiang, Kun Wu, Wenhao Zhuang, Minglei Zhang, Shaojie Wang, Shangpeng Yan, Kepeng Lei, Zongxian Feng, Huiming Wang, Zheng Lin, Mengtong Li, Mengfei Xie, Yinghan Cui, Xuxing Chen, Chao Wang, Weihao Li, Wenqiang Zhu, Jiarong Zhang, Jingxuan Xu, Songwei Yu, Yifan Yao, Xinping Lei, Han Li, Junqi Xiong, Zuchen Gao, Dailin Li, Haimo Li, Jiaheng Liu, Yuqun Zhang, Junyi Peng, Haotian Zhang, Bin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18779
• Pdf URL: https://arxiv.org/pdf/2510.18779
• Copy Paste: [[2510.18779]] KAT-Coder Technical Report(https://arxiv.org/abs/2510.18779)
• Keywords: language model, llm, agent
• Abstract: Recent advances in large language models (LLMs) have enabled progress in agentic coding, where models autonomously reason, plan, and act within interactive software development workflows. However, bridging the gap between static text-based training and dynamic real-world agentic execution remains a core challenge. In this technical report, we present KAT-Coder, a large-scale agentic code model trained through a multi-stage curriculum encompassing Mid-Term Training, Supervised Fine-Tuning (SFT), Reinforcement Fine-Tuning (RFT), and Reinforcement-to-Deployment Adaptation. The Mid-Term stage enhances reasoning, planning, and reflection capabilities through a corpus of real software engineering data and synthetic agentic interactions. The SFT stage constructs a million-sample dataset balancing twenty programming languages, ten development contexts, and ten task archetypes. The RFT stage introduces a novel multi-ground-truth reward formulation for stable and sample-efficient policy optimization. Finally, the Reinforcement-to-Deployment phase adapts the model to production-grade IDE environments using Error-Masked SFT and Tree-Structured Trajectory Training. In summary, these stages enable KAT-Coder to achieve robust tool-use reliability, instruction alignment, and long-context reasoning, forming a deployable foundation for real-world intelligent coding agents. Our KAT series 32B model, KAT-Dev, has been open-sourced on this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展推动了代理编码的进步，其中模型在交互式软件开发工作流程中自主推理、计划和操作。然而，弥合基于静态文本的训练和动态现实世界代理执行之间的差距仍然是一个核心挑战。在这份技术报告中，我们介绍了 KAT-Coder，这是一种通过多阶段课程进行训练的大规模代理代码模型，包括中期培训、监督微调 (SFT)、强化微调 (RFT) 和强化部署适应。中期阶段通过真实软件工程数据和综合代理交互的语料库增强推理、规划和反思能力。 SFT 阶段构建了一个包含 100 万个样本的数据集，平衡了 20 种编程语言、10 种开发环境和 10 种任务原型。 RFT 阶段引入了一种新颖的多真实奖励公式，用于稳定且样本高效的策略优化。最后，强化到部署阶段使用错误屏蔽 SFT 和树结构轨迹训练使模型适应生产级 IDE 环境。总之，这些阶段使 KAT-Coder 能够实现强大的工具使用可靠性、指令对齐和长上下文推理，为现实世界的智能编码代理奠定了可部署的基础。我们的 KAT 系列 32B 型号 KAT-Dev 已在此 https URL 上开源。

Title: WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection

• Authors: Guanzhong He, Zhen Yang, Jinxin Liu, Bin Xu, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2510.18798
• Pdf URL: https://arxiv.org/pdf/2510.18798
• Copy Paste: [[2510.18798]] WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection(https://arxiv.org/abs/2510.18798)
• Keywords: agent
• Abstract: Search agents have achieved significant advancements in enabling intelligent information retrieval and decision-making within interactive environments. Although reinforcement learning has been employed to train agentic models capable of more dynamic interactive retrieval, existing methods are limited by shallow tool-use depth and the accumulation of errors over multiple iterative interactions. In this paper, we present WebSeer, a more intelligent search agent trained via reinforcement learning enhanced with a self-reflection mechanism. Specifically, we construct a large dataset annotated with reflection patterns and design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm for real-world web-based environments, which enables the model to generate longer and more reflective tool-use trajectories. Our approach substantially extends tool-use chains and improves answer accuracy. Using a single 14B model, we achieve state-of-the-art results on HotpotQA and SimpleQA, with accuracies of 72.3% and 90.0%, respectively, and demonstrate strong generalization to out-of-distribution datasets. The code is available at this https URL
• 摘要：搜索代理在交互式环境中实现智能信息检索和决策方面取得了重大进展。尽管强化学习已被用来训练能够进行更动态的交互式检索的代理模型，但现有方法受到工具使用深度浅和多次迭代交互过程中错误累积的限制。在本文中，我们提出了 WebSeer，这是一种更智能的搜索代理，通过强化学习进行训练，并具有自我反思机制。具体来说，我们构建了一个用反射模式注释的大型数据集，并设计了一个两阶段训练框架，该框架将冷启动和强化学习统一在现实网络环境的自我反射范式中，这使得模型能够生成更长、更具反射性的工具使用轨迹。我们的方法大大扩展了工具使用链并提高了答案准确性。使用单个 14B 模型，我们在 HotpotQA 和 SimpleQA 上取得了最先进的结果，准确率分别为 72.3% 和 90.0%，并展示了对分布外数据集的强大泛化能力。该代码可在此 https URL 获取

Title: Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoring

• Authors: Shuxin Lin, Dhaval Patel, Christodoulos Constantinides
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18817
• Pdf URL: https://arxiv.org/pdf/2510.18817
• Copy Paste: [[2510.18817]] Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoring(https://arxiv.org/abs/2510.18817)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Small Language Models (SLMs) are becoming increasingly popular in specialized fields, such as industrial applications, due to their efficiency, lower computational requirements, and ability to be fine-tuned for domain-specific tasks, enabling accurate and cost-effective solutions. However, performing complex reasoning using SLMs in specialized fields such as Industry 4.0 remains challenging. In this paper, we propose a knowledge distillation framework for industrial asset health, which transfers reasoning capabilities via Chain-of-Thought (CoT) distillation from Large Language Models (LLMs) to smaller, more efficient models (SLMs). We discuss the advantages and the process of distilling LLMs using multi-choice question answering (MCQA) prompts to enhance reasoning and refine decision-making. We also perform in-context learning to verify the quality of the generated knowledge and benchmark the performance of fine-tuned SLMs with generated knowledge against widely used LLMs. The results show that the fine-tuned SLMs with CoT reasoning outperform the base models by a significant margin, narrowing the gap to their LLM counterparts. Our code is open-sourced at: this https URL.
• 摘要：小语言模型 (SLM) 在工业应用等专业领域越来越受欢迎，因为它们效率高、计算要求低，并且能够针对特定领域的任务进行微调，从而实现准确且经济高效的解决方案。然而，在工业 4.0 等专业领域使用 SLM 执行复杂推理仍然具有挑战性。在本文中，我们提出了一种工业资产健康的知识蒸馏框架，该框架通过思想链（CoT）蒸馏将推理能力从大型语言模型（LLM）转移到更小、更高效的模型（SLM）。我们讨论了使用多选题回答（MCQA）提示来增强推理和完善决策的优势和提炼法学硕士的过程。我们还执行上下文学习来验证生成知识的质量，并根据广泛使用的法学硕士对使用生成知识进行微调的 SLM 的性能进行基准测试。结果表明，经过 CoT 推理的微调 SLM 明显优于基本模型，缩小了与 LLM 模型的差距。我们的代码是开源的：此 https URL。

Title: MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training

• Authors: Wenxuan Li, Chengruidong Zhang, Huiqiang Jiang, Yucheng Li, Yuqing Yang, Lili Qiu
• Subjects: cs.CL, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2510.18830
• Pdf URL: https://arxiv.org/pdf/2510.18830
• Copy Paste: [[2510.18830]] MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training(https://arxiv.org/abs/2510.18830)
• Keywords: language model, llm, long context
• Abstract: The adoption of long context windows has become a standard feature in Large Language Models (LLMs), as extended contexts significantly enhance their capacity for complex reasoning and broaden their applicability across diverse scenarios. Dynamic sparse attention is a promising approach for reducing the computational cost of long-context. However, efficiently training LLMs with dynamic sparse attention on ultra-long contexts-especially in distributed settings-remains a significant challenge, due in large part to worker- and step-level imbalance. This paper introduces MTraining, a novel distributed methodology leveraging dynamic sparse attention to enable efficient training for LLMs with ultra-long contexts. Specifically, MTraining integrates three key components: a dynamic sparse training pattern, balanced sparse ring attention, and hierarchical sparse ring attention. These components are designed to synergistically address the computational imbalance and communication overheads inherent in dynamic sparse attention mechanisms during the training of models with extensive context lengths. We demonstrate the efficacy of MTraining by training Qwen2.5-3B, successfully expanding its context window from 32K to 512K tokens on a cluster of 32 A100 GPUs. Our evaluations on a comprehensive suite of downstream tasks, including RULER, PG-19, InfiniteBench, and Needle In A Haystack, reveal that MTraining achieves up to a 6x higher training throughput while preserving model accuracy. Our code is available at this https URL.
• 摘要：采用长上下文窗口已成为大型语言模型 (LLM) 的标准功能，因为扩展上下文显着增强了其复杂推理的能力，并扩大了其在不同场景中的适用性。动态稀疏注意力是降低长上下文计算成本的一种有前途的方法。然而，在超长上下文中（尤其是在分布式环境中）有效地训练具有动态稀疏注意力的法学硕士仍然是一个重大挑战，这在很大程度上是由于工人和步骤级别的不平衡。本文介绍了 MTraining，这是一种利用动态稀疏注意力的新型分布式方法，能够对具有超长上下文的法学硕士进行高效训练。具体来说，MTraining 集成了三个关键组件：动态稀疏训练模式、平衡稀疏环注意力和分层稀疏环注意力。这些组件旨在协同解决在训练具有广泛上下文长度的模型期间动态稀疏注意机制固有的计算不平衡和通信开销。我们通过训练 Qwen2.5-3B 展示了 MTraining 的功效，在 32 个 A100 GPU 的集群上成功地将其上下文窗口从 32K 令牌扩展到 512K 令牌。我们对一整套下游任务（包括 RULER、PG-19、InfiniteBench 和 Needle In A Haystack）的评估表明，MTraining 在保持模型准确性的同时实现了高达 6 倍的训练吞吐量。我们的代码可以在这个 https URL 上找到。

Title: Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning

• Authors: Chenghao Zhu, Meiling Tao, Tiannan Wang, Dongyi Ding, Yuchen Eleanor Jiang, Wangchunshu Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18849
• Pdf URL: https://arxiv.org/pdf/2510.18849
• Copy Paste: [[2510.18849]] Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning(https://arxiv.org/abs/2510.18849)
• Keywords: language model, gpt, llm
• Abstract: Faithfully personalizing large language models (LLMs) to align with individual user preferences is a critical but challenging task. While supervised fine-tuning (SFT) quickly reaches a performance plateau, standard reinforcement learning from human feedback (RLHF) also struggles with the nuances of personalization. Scalar-based reward models are prone to reward hacking which leads to verbose and superficially personalized responses. To address these limitations, we propose Critique-Post-Edit, a robust reinforcement learning framework that enables more faithful and controllable personalization. Our framework integrates two key components: (1) a Personalized Generative Reward Model (GRM) that provides multi-dimensional scores and textual critiques to resist reward hacking, and (2) a Critique-Post-Edit mechanism where the policy model revises its own outputs based on these critiques for more targeted and efficient learning. Under a rigorous length-controlled evaluation, our method substantially outperforms standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses the performance of GPT-4.1. These results demonstrate a practical path to faithful, efficient, and controllable personalization.
• 摘要：忠实地个性化大型语言模型（LLM）以符合个人用户的偏好是一项关键但具有挑战性的任务。虽然监督微调 (SFT) 很快达到了性能稳定水平，但来自人类反馈的标准强化学习 (RLHF) 也在应对个性化的细微差别。基于标量的奖励模型很容易受到奖励黑客攻击，从而导致冗长且表面上个性化的响应。为了解决这些限制，我们提出了 Critique-Post-Edit，这是一个强大的强化学习框架，可以实现更忠实和可控的个性化。我们的框架集成了两个关键组件：（1）个性化生成奖励模型（GRM），提供多维分数和文本批评以抵御奖励黑客；（2）批评后编辑机制，其中政策模型根据这些批评修改自己的输出，以实现更有针对性和更高效的学习。在严格的长度控制评估下，我们的方法在个性化基准上大大优于标准 PPO。个性化Qwen2.5-7B实现了平均11%的胜率提升，个性化Qwen2.5-14B模型超越了GPT-4.1的性能。这些结果展示了一条实现忠实、高效和可控个性化的实用途径。

Title: LightMem: Lightweight and Efficient Memory-Augmented Generation

• Authors: Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2510.18866
• Pdf URL: https://arxiv.org/pdf/2510.18866
• Copy Paste: [[2510.18866]] LightMem: Lightweight and Efficient Memory-Augmented Generation(https://arxiv.org/abs/2510.18866)
• Keywords: language model, gpt, llm
• Abstract: Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. Experiments on LongMemEval with GPT and Qwen backbones show that LightMem outperforms strong baselines in accuracy (up to 10.9% gains) while reducing token usage by up to 117x, API calls by up to 159x, and runtime by over 12x. The code is available at this https URL.
• 摘要：尽管大型语言模型 (LLM) 具有卓越的功能，但它仍难以在动态和复杂的环境中有效利用历史交互信息。内存系统通过引入持久性信息存储、检索和利用机制，使法学硕士能够超越无状态交互。然而，现有的存储器系统经常引入大量的时间和计算开销。为此，我们引入了一种名为LightMem的新内存系统，它在内存系统的性能和效率之间取得了平衡。受人类记忆 Atkinson-Shiffrin 模型的启发，LightMem 将记忆组织为三个互补的阶段。首先，受认知启发的感觉记忆通过轻量级压缩快速过滤不相关的信息，并根据主题对信息进行分组。接下来，主题感知短期记忆会整合这些基于主题的组，组织和总结内容以实现更结构化的访问。最后，具有睡眠时间更新的长期记忆采用离线程序，将整合与在线推理分离。在具有 GPT 和 Qwen 主干的 LongMemEval 上进行的实验表明，LightMem 在准确性方面优于强大的基线（增益高达 10.9%），同时将令牌使用量减少高达 117 倍，API 调用减少高达 159 倍，运行时间减少超过 12 倍。该代码可从此 https URL 获取。

Title: How Do LLMs Use Their Depth?

• Authors: Akshat Gupta, Jay Yeung, Gopala Anumanchipalli, Anna Ivanova
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2510.18871
• Pdf URL: https://arxiv.org/pdf/2510.18871
• Copy Paste: [[2510.18871]] How Do LLMs Use Their Depth?(https://arxiv.org/abs/2510.18871)
• Keywords: language model, llm
• Abstract: Growing evidence suggests that large language models do not use their depth uniformly, yet we still lack a fine-grained understanding of their layer-wise prediction dynamics. In this paper, we trace the intermediate representations of several open-weight models during inference and reveal a structured and nuanced use of depth. Specifically, we propose a "Guess-then-Refine" framework that explains how LLMs internally structure their computations to make predictions. We first show that the top-ranked predictions in early LLM layers are composed primarily of high-frequency tokens, which act as statistical guesses proposed by the model early on due to the lack of appropriate contextual information. As contextual information develops deeper into the model, these initial guesses get refined into contextually appropriate tokens. Even high-frequency token predictions from early layers get refined >70% of the time, indicating that correct token prediction is not "one-and-done". We then go beyond frequency-based prediction to examine the dynamic usage of layer depth across three case studies. (i) Part-of-speech analysis shows that function words are, on average, the earliest to be predicted correctly. (ii) Fact recall task analysis shows that, in a multi-token answer, the first token requires more computational depth than the rest. (iii) Multiple-choice task analysis shows that the model identifies the format of the response within the first half of the layers, but finalizes its response only toward the end. Together, our results provide a detailed view of depth usage in LLMs, shedding light on the layer-by-layer computations that underlie successful predictions and providing insights for future works to improve computational efficiency in transformer-based models.
• 摘要：越来越多的证据表明，大型语言模型并没有统一使用它们的深度，但我们仍然缺乏对其分层预测动态的细粒度理解。在本文中，我们追踪了几个开放权重模型在推理过程中的中间表示，并揭示了深度的结构化和细致入微的使用。具体来说，我们提出了一个“猜测然后优化”框架，该框架解释了法学硕士如何在内部构建其计算以进行预测。我们首先表明，早期 LLM 层中排名靠前的预测主要由高频标记组成，由于缺乏适当的上下文信息，这些标记充当模型早期提出的统计猜测。随着上下文信息深入到模型中，这些最初的猜测会被细化为适合上下文的标记。即使来自早期层的高频令牌预测也有超过 70% 的时间得到改进，这表明正确的令牌预测不是“一劳永逸”的。然后，我们超越基于频率的预测来检查三个案例研究中层深度的动态使用。 (i) 词性分析表明，平均而言，功能词是最早被正确预测的。 (ii) 事实回忆任务分析表明，在多标记答案中，第一个标记比其他标记需要更多的计算深度。 (iii) 多项选择任务分析表明，该模型识别了前半层内响应的格式，但仅在最后确定其响应。总之，我们的结果提供了法学硕士中深度使用的详细视图，揭示了成功预测背后的逐层计算，并为未来提高基于变压器的模型的计算效率的工作提供了见解。