2025-11-27

Title: Democratizing LLM Efficiency: From Hyperscale Optimizations to Universal Deployability

• Authors: Hen-Hsen Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20662
• Pdf URL: https://arxiv.org/pdf/2511.20662
• Copy Paste: [[2511.20662]] Democratizing LLM Efficiency: From Hyperscale Optimizations to Universal Deployability(https://arxiv.org/abs/2511.20662)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large language models (LLMs) have become indispensable, but the most celebrated efficiency methods -- mixture-of-experts (MoE), speculative decoding, and complex retrieval-augmented generation (RAG) -- were built for hyperscale providers with vast infrastructure and elite teams. Outside that context, their benefits collapse into overhead, fragility, and wasted carbon. The result is that a handful of Big Tech companies benefit, while thousands of hospitals, schools, governments, and enterprises are left without viable options. We argue that the next frontier is not greater sophistication at scale, but robust simplicity: efficiency that thrives under modest resources and minimal expertise. We propose a new research agenda: retrofitting pretrained models with more efficient architectures without retraining, inventing lightweight fine-tuning that preserves alignment, making reasoning economical despite long chains of thought, enabling dynamic knowledge management without heavy RAG pipelines, and adopting Overhead-Aware Efficiency (OAE) as a standard benchmark. By redefining efficiency to include adoption cost, sustainability, and fairness, we can democratize LLM deployment -- ensuring that optimization reduces inequality and carbon waste rather than amplifying them.
• 摘要：大型语言模型 (LLM) 已变得不可或缺，但最著名的效率方法——专家混合 (MoE)、推测性解码和复杂检索增强生成 (RAG)——是为拥有庞大基础设施和精英团队的超大规模提供商构建的。在这种背景之外，它们的好处就会转化为开销、脆弱性和碳浪费。结果是少数大型科技公司受益，而数千家医院、学校、政府和企业却没有可行的选择。我们认为，下一个前沿不是大规模的复杂性，而是强大的简单性：在有限的资源和最少的专业知识下蓬勃发展的效率。我们提出了一个新的研究议程：在无需重新训练的情况下使用更高效的架构改造预训练模型，发明保持对齐的轻量级微调，在思想链长的情况下使推理变得经济，在没有重型 RAG 管道的情况下实现动态知识管理，并采用开销感知效率（OAE）作为标准基准。通过重新定义效率，将采用成本、可持续性和公平性纳入其中，我们可以实现法学硕士部署的民主化——确保优化减少而不是加剧不平等和碳浪费。

Title: PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation

• Authors: Yongfu Xue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20668
• Pdf URL: https://arxiv.org/pdf/2511.20668
• Copy Paste: [[2511.20668]] PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation(https://arxiv.org/abs/2511.20668)
• Keywords: language model, llm
• Abstract: Reward models are crucial for aligning Large Language Models (LLMs) with human preferences but face two representative challenges. First, traditional discriminative reward models usually concatenate questions and responses directly as input, resulting in low data efficiency. Second, reward models are vulnerable to reward overoptimization. We propose PIRA, a training paradigm addressing these issues through three strategies: (1) Reformulating question-answer pairs into preference-based instructions for clearer and more explicit task specification, (2) aggregating rewards from diverse preference tasks to reduce bias and improve robustness, and (3) averaging value-head outputs under varying dropout rates to stabilize rewards. Extensive experiments have demonstrated the effectiveness of PIRA.
• 摘要：奖励模型对于使大型语言模型 (LLM) 与人类偏好保持一致至关重要，但面临两个代表性挑战。首先，传统的歧视性奖励模型通常将问题和回答直接串联起来作为输入，导致数据效率低下。其次，奖励模型容易受到奖励过度优化的影响。我们提出了 PIRA，一种通过三种策略解决这些问题的训练范式：（1）将问答对重新调整为基于偏好的指令，以实现更清晰、更明确的任务规范；（2）聚合来自不同偏好任务的奖励，以减少偏差并提高鲁棒性；（3）在不同的退出率下平均价值头输出以稳定奖励。大量实验证明了 PIRA 的有效性。

Title: Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data

• Authors: Mann Khatri, Mirza Yusuf, Rajiv Ratn Shah, Ponnurangam Kumaraguru
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.20669
• Pdf URL: https://arxiv.org/pdf/2511.20669
• Copy Paste: [[2511.20669]] Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data(https://arxiv.org/abs/2511.20669)
• Keywords: language model, llm, long context
• Abstract: Large Language Models (LLMs), trained on extensive datasets from the web, exhibit remarkable general reasoning skills. Despite this, they often struggle in specialized areas like law, mainly because they lack domain-specific pretraining. The legal field presents unique challenges, as legal documents are generally long and intricate, making it hard for models to process the full text efficiently. Previous studies have examined in-context approaches to address the knowledge gap, boosting model performance in new domains without full domain alignment. In our paper, we analyze model behavior on legal tasks by conducting experiments in three areas: (i) reorganizing documents based on rhetorical roles to assess how structured information affects long context processing and model decisions, (ii) defining rhetorical roles to familiarize the model with legal terminology, and (iii) emulating the step-by-step reasoning of courts regarding rhetorical roles to enhance model reasoning. These experiments are conducted in a zero-shot setting across three Indian legal judgment prediction datasets. Our results reveal that organizing data or explaining key legal terms significantly boosts model performance, with a minimum increase of ~1.5% and a maximum improvement of 4.36% in F1 score compared to the baseline.
• 摘要：大型语言模型 (LLM) 在网络上的大量数据集上进行训练，表现出卓越的一般推理能力。尽管如此，他们经常在法律等专业领域陷入困境，主要是因为他们缺乏特定领域的预训练。法律领域面临着独特的挑战，因为法律文件通常又长又复杂，使得模型很难有效地处理全文。之前的研究已经研究了解决知识差距的上下文方法，在没有完全领域对齐的情况下提高新领域中的模型性能。在我们的论文中，我们通过在三个领域进行实验来分析法律任务的模型行为：（i）根据修辞角色重新组织文档，以评估结构化信息如何影响长上下文处理和模型决策，（ii）定义修辞角色以使模型熟悉法律术语，以及（iii）模拟法院关于修辞角色的逐步推理以增强模型推理。这些实验是在三个印度法律判决预测数据集的零样本设置中进行的。我们的结果表明，组织数据或解释关键法律术语可显着提高模型性能，与基线相比，F1 分数最小提高约 1.5%，最大提高 4.36%。

Title: MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data

• Authors: Saad Mankarious, Ayah Zirikly, Daniel Wiechmann, Elma Kerz, Edward Kempa, Yu Qiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.20672
• Pdf URL: https://arxiv.org/pdf/2511.20672
• Copy Paste: [[2511.20672]] MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data(https://arxiv.org/abs/2511.20672)
• Keywords: language model
• Abstract: Social media data has become a vital resource for studying mental health, offering real-time insights into thoughts, emotions, and behaviors that traditional methods often miss. Progress in this area has been facilitated by benchmark datasets for mental health analysis; however, most existing benchmarks have become outdated due to limited data availability, inadequate cleaning, and the inherently diverse nature of social media content (e.g., multilingual and harmful material). We present a new benchmark dataset, \textbf{MindSET}, curated from Reddit using self-reported diagnoses to address these limitations. The annotated dataset contains over \textbf{13M} annotated posts across seven mental health conditions, more than twice the size of previous benchmarks. To ensure data quality, we applied rigorous preprocessing steps, including language filtering, and removal of Not Safe for Work (NSFW) and duplicate content. We further performed a linguistic analysis using LIWC to examine psychological term frequencies across the eight groups represented in the dataset. To demonstrate the dataset utility, we conducted binary classification experiments for diagnosis detection using both fine-tuned language models and Bag-of-Words (BoW) features. Models trained on MindSET consistently outperformed those trained on previous benchmarks, achieving up to an \textbf{18-point} improvement in F1 for Autism detection. Overall, MindSET provides a robust foundation for researchers exploring the intersection of social media and mental health, supporting both early risk detection and deeper analysis of emerging psychological trends.
• 摘要：社交媒体数据已成为研究心理健康的重要资源，提供传统方法经常忽略的思想、情绪和行为的实时洞察。心理健康分析基准数据集促进了这一领域的进展；然而，由于数据可用性有限、清理不充分以及社交媒体内容固有的多样性（例如多语言和有害材料），大多数现有基准已经过时。我们提出了一个新的基准数据集 \textbf{MindSET}，它是从 Reddit 中使用自我报告的诊断来解决这些限制的。带注释的数据集包含超过 \textbf{13M} 个针对七种心理健康状况的带注释的帖子，是之前基准大小的两倍多。为了确保数据质量，我们应用了严格的预处理步骤，包括语言过滤以及删除不安全工作 (NSFW) 和重复内容。我们进一步使用 LIWC 进行语言分析，以检查数据集中代表的八个组的心理术语频率。为了演示数据集的实用性，我们使用微调语言模型和词袋（BoW）特征进行了用于诊断检测的二元分类实验。在 MindSET 上训练的模型始终优于在之前基准测试中训练的模型，在自闭症检测的 F1 方面实现了 \textbf{18-point} 的改进。总体而言，MindSET 为研究人员探索社交媒体和心理健康的交叉点提供了坚实的基础，支持早期风险检测和对新兴心理趋势的更深入分析。

Title: Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic

• Authors: Saleh Almohaimeed, May Alsofyani, Saad Almohaimeed, Mansour Al Ghanim, Liqiang Wang
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2511.20677
• Pdf URL: https://arxiv.org/pdf/2511.20677
• Copy Paste: [[2511.20677]] Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic(https://arxiv.org/abs/2511.20677)
• Keywords: language model, gpt, prompt
• Abstract: In recent years, the task of cross-domain, context-dependent text-to-SQL has received significant attention. Enables users with no prior knowledge of SQL to have a conversation with databases using natural language. However, most of the available datasets and research have been conducted in English, along with some work in Chinese. To this date, no effort has been made to address this task in the Arabic language. In this paper, we introduce Ar-SParC, the first Arabic cross-domain, context-dependent text-to-SQL dataset. The dataset consists of 3,450 sequences of interrelated questions, each sequence containing an average of approximately three questions, which results in a total of 10225 questions along with their corresponding SQL queries. We conducted 40 experiments on the Ar-SParC dataset using two large language models, GPT-3.5-turbo and GPT-4.5-turbo, applying 10 different prompt engineering techniques, including four question representation methods and six in-context learning techniques. Furthermore, we developed a novel approach named GAT corrector, which enhanced the performance across all 40 experiments, yielding an average improvement of 1.9% in execution accuracy (EX) and 1.9% in interaction accuracy (IX) under zero-shot settings, and an average increase of 1.72% EX and 0.92% IX under in-context learning settings. Finally, we conducted an ablation study with two more experiments to explain why the GAT corrector outperformed the previous GAT verifier technique, particularly for the Arabic language.
• 摘要：近年来，跨域、上下文相关的文本到 SQL 的任务受到了极大的关注。使不具备 SQL 知识的用户能够使用自然语言与数据库进行对话。然而，大多数可用的数据集和研究都是用英语进行的，还有一些工作是用中文进行的。迄今为止，尚未采取任何措施以阿拉伯语解决这项任务。在本文中，我们介绍了 Ar-SParC，这是第一个阿拉伯语跨域、上下文相关的文本到 SQL 数据集。该数据集由 3,450 个相互关联的问题序列组成，每个序列平均包含大约三个问题，总共有 10225 个问题及其相应的 SQL 查询。我们使用两种大型语言模型 GPT-3.5-turbo 和 GPT-4.5-turbo 在 Ar-SParC 数据集上进行了 40 次实验，应用了 10 种不同的提示工程技术，包括四种问题表示方法和六种上下文学习技术。此外，我们开发了一种名为 GAT 校正器的新颖方法，它增强了所有 40 个实验的性能，在零样本设置下，执行精度 (EX) 平均提高了 1.9%，交互精度 (IX) 平均提高了 1.9%，在上下文学习设置下，EX 平均提高了 1.72%，交互精度 (IX) 平均提高了 0.92%。最后，我们通过另外两个实验进行了消融研究，以解释为什么 GAT 校正器优于之前的 GAT 验证器技术，特别是对于阿拉伯语。

Title: Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes

• Authors: Matthew W. Kenaston (1), Umair Ayub (1), Mihir Parmar (2), Muhammad Umair Anjum (1), Syed Arsalan Ahmed Naqvi (1), Priya Kumar (1), Samarth Rawal (1), Aadel A. Chaudhuri (4), Yousef Zakharia (3), Elizabeth I. Heath (5), Tanios S. Bekaii-Saab (3), Cui Tao (6), Eliezer M. Van Allen (7), Ben Zhou (2), YooJung Choi (2), Chitta Baral (2), Irbaz Bin Riaz (1 and 3 and 6) ((1) Mayo Clinic College of Medicine and Science, Phoenix, AZ, (2) School of Computing and AI, Arizona State University, Tempe, AZ, (3) Mayo Clinic Comprehensive Cancer Center, Phoenix, AZ, (4) Department of Radiation Oncology, Mayo Clinic, Rochester, MN, (5) Department of Oncology, Mayo Clinic, Rochester, MN, (6) Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, (7) Dana-Farber Cancer Institute, Harvard Medical School, Boston, MA)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.20680
• Pdf URL: https://arxiv.org/pdf/2511.20680
• Copy Paste: [[2511.20680]] Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes(https://arxiv.org/abs/2511.20680)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: Despite high performance on clinical benchmarks, large language models may reach correct conclusions through faulty reasoning, a failure mode with safety implications for oncology decision support that is not captured by accuracy-based evaluation. In this two-cohort retrospective study, we developed a hierarchical taxonomy of reasoning errors from GPT-4 chain-of-thought responses to real oncology notes and tested its clinical relevance. Using breast and pancreatic cancer notes from the CORAL dataset, we annotated 600 reasoning traces to define a three-tier taxonomy mapping computational failures to cognitive bias frameworks. We validated the taxonomy on 822 responses from prostate cancer consult notes spanning localized through metastatic disease, simulating extraction, analysis, and clinical recommendation tasks. Reasoning errors occurred in 23 percent of interpretations and dominated overall errors, with confirmation bias and anchoring bias most common. Reasoning failures were associated with guideline-discordant and potentially harmful recommendations, particularly in advanced disease management. Automated evaluators using state-of-the-art language models detected error presence but could not reliably classify subtypes. These findings show that large language models may provide fluent but clinically unsafe recommendations when reasoning is flawed. The taxonomy provides a generalizable framework for evaluating and improving reasoning fidelity before clinical deployment.
• 摘要：尽管在临床基准上表现出色，但大型语言模型可能会通过错误的推理得出正确的结论，这是一种对肿瘤学决策支持具有安全影响的故障模式，而基于准确性的评估无法捕获这种故障模式。在这项两队列回顾性研究中，我们开发了一种从 GPT-4 思维链反应到真实肿瘤学笔记的推理错误的分层分类法，并测试了其临床相关性。使用 CORAL 数据集中的乳腺癌和胰腺癌注释，我们注释了 600 条推理轨迹，以定义将计算故障映射到认知偏差框架的三层分类法。我们验证了来自前列腺癌咨询笔记的 822 个响应的分类，涵盖局部转移性疾病、模拟提取、分析和临床推荐任务。 23% 的解释出现推理错误，并且主导了总体错误，其中确认偏差和锚定偏差最为常见。推理失败与指南不一致和潜在有害的建议有关，特别是在高级疾病管理方面。使用最先进的语言模型的自动评估器可以检测到错误的存在，但无法可靠地对子类型进行分类。这些发现表明，当推理有缺陷时，大型语言模型可能会提供流畅但临床上不安全的建议。该分类法提供了一个通用框架，用于在临床部署之前评估和提高推理保真度。

Title: Dynamic Template Selection for Output Token Generation Optimization: MLP-Based and Transformer Approaches

• Authors: Bharadwaj Yadavalli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20683
• Pdf URL: https://arxiv.org/pdf/2511.20683
• Copy Paste: [[2511.20683]] Dynamic Template Selection for Output Token Generation Optimization: MLP-Based and Transformer Approaches(https://arxiv.org/abs/2511.20683)
• Keywords: language model, gpt, llm, prompt
• Abstract: Contemporary large language model deployments typically employ uniform prompting strategies across diverse query types, applying verbose response patterns to both complex analytical tasks and straightforward factual questions. This one-size-fits-all methodology leads to substantial token inefficiency, a concern amplified by the significant cost differential between input and output tokens--the latter commanding 4-8x higher prices across major providers. We present Dynamic Template Selection (DTS), which adaptively matches response templates to query complexity, achieving significant cost reductions without compromising response quality. We compared two routing approaches: a simple MLP that uses pre-computed embeddings and a more complex fine-tuned RoBERTa transformer. Through comprehensive evaluation on 1,000 MMLU questions, we find that the MLP router achieves 90.5% routing accuracy on held-out test data, marginally exceeding RoBERTa's performance (89.5%) despite utilizing 125M fewer parameters. Notably, our empirical analysis reveals provider-agnostic behavior in template selection--routing decisions generalize effectively across 3 major LLM providers (OpenAI GPT-4, Google Gemini, and Anthropic Claude), as validated through 9,000 production API calls. While routing accuracy remains consistent at 90.5% across providers, observed token reductions vary from 32.6% to 33.9%, reflecting provider-specific generation characteristics. This work contributes several key elements: formal problem formulation with theoretical grounding in machine learning, four algorithms with corresponding complexity analyses, and extensive empirical validation across production systems.
• 摘要：当代大型语言模型部署通常在不同的查询类型中采用统一的提示策略，将详细的响应模式应用于复杂的分析任务和简单的事实问题。这种一刀切的方法导致代币效率严重低下，输入和输出代币之间巨大的成本差异加剧了这一问题——后者在主要提供商中的价格高出 4-8 倍。我们提出了动态模板选择 (DTS)，它可以自适应地将响应模板与查询复杂性相匹配，从而在不影响响应质量的情况下显着降低成本。我们比较了两种路由方法：使用预先计算的嵌入的简单 MLP 和更复杂的微调 RoBERTa 变压器。通过对 1,000 个 MMLU 问题的综合评估，我们发现 MLP 路由器在保留的测试数据上实现了 90.5% 的路由准确率，尽管使用的参数少了 125M，但仍略微超过 RoBERTa 的性能（89.5%）。值得注意的是，我们的实证分析揭示了模板选择中与提供商无关的行为——路由决策在 3 个主要 LLM 提供商（OpenAI GPT-4、Google Gemini 和 Anthropic Claude）中有效地推广，通过 9,000 个生产 API 调用进行了验证。虽然各个提供商的路由准确性保持一致为 90.5%，但观察到的代币减少量从 32.6% 到 33.9% 不等，反映了提供商特定的生成特征。这项工作贡献了几个关键要素：具有机器学习理论基础的正式问题表述、具有相应复杂性分析的四种算法以及跨生产系统的广泛经验验证。

Title: LLMs-Powered Accurate Extraction, Querying and Intelligent Management of Literature derived 2D Materials Data

• Authors: Lijun Shang, Yadong Yu, Wenqiang Kang, Jian Zhou, Dongyue Gao, Pan Xiang, Zhe Liu, Mengyan Dai, Zhonglu Guo, Zhimei Sun
• Subjects: cs.CL, cond-mat.mtrl-sci, cs.DB
• Abstract URL: https://arxiv.org/abs/2511.20691
• Pdf URL: https://arxiv.org/pdf/2511.20691
• Copy Paste: [[2511.20691]] LLMs-Powered Accurate Extraction, Querying and Intelligent Management of Literature derived 2D Materials Data(https://arxiv.org/abs/2511.20691)
• Keywords: llm
• Abstract: Two-dimensional (2D) materials have showed widespread applications in energy storage and conversion owning to their unique physicochemical, and electronic properties. Most of the valuable information for the materials, such as their properties and preparation methods, is included in the published research papers. However, due to the dispersion of synthe
• 摘要：二维（2D）材料由于其独特的物理化学和电子特性，在能量存储和转换方面显示出广泛的应用。该材料的大部分有价值的信息，例如其性质和制备方法，都包含在已发表的研究论文中。但由于合成物分散

Title: Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models

• Authors: Trung Cuong Dang, David Mohaisen
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.20799
• Pdf URL: https://arxiv.org/pdf/2511.20799
• Copy Paste: [[2511.20799]] Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models(https://arxiv.org/abs/2511.20799)
• Keywords: language model, llm, chat
• Abstract: Large language models, trained on massive corpora, are prone to verbatim memorization of training data, creating significant privacy and copyright risks. While previous works have proposed various definitions for memorization, many exhibit shortcomings in comprehensively capturing this phenomenon, especially in aligned models. To address this, we introduce a novel framework: multi-prefix memorization. Our core insight is that memorized sequences are deeply encoded and thus retrievable via a significantly larger number of distinct prefixes than non-memorized content. We formalize this by defining a sequence as memorized if an external adversarial search can identify a target count of distinct prefixes that elicit it. This framework shifts the focus from single-path extraction to quantifying the robustness of a memory, measured by the diversity of its retrieval paths. Through experiments on open-source and aligned chat models, we demonstrate that our multi-prefix definition reliably distinguishes memorized from non-memorized data, providing a robust and practical tool for auditing data leakage in LLMs.
• 摘要：在海量语料库上训练的大型语言模型很容易逐字记忆训练数据，从而产生重大的隐私和版权风险。虽然之前的工作提出了记忆的各种定义，但许多工作在全面捕捉这种现象方面表现出缺陷，特别是在对齐模型中。为了解决这个问题，我们引入了一种新颖的框架：多前缀记忆。我们的核心见解是，记忆的序列经过深度编码，因此可以通过比未记忆的内容多得多的不同前缀进行检索。如果外部对抗性搜索可以识别引发该序列的不同前缀的目标计数，我们可以通过将序列定义为记忆来形式化这一点。该框架将焦点从单路径提取转移到量化记忆的稳健性，通过其检索路径的多样性来衡量。通过对开源和对齐的聊天模型的实验，我们证明了我们的多前缀定义能够可靠地区分记忆数据和非记忆数据，为审计法学硕士中的数据泄漏提供了一个强大而实用的工具。

Title: SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models

• Authors: Jiaojiao Han, Wujiang Xu, Mingyu Jin, Mengnan Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20820
• Pdf URL: https://arxiv.org/pdf/2511.20820
• Copy Paste: [[2511.20820]] SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models(https://arxiv.org/abs/2511.20820)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have achieved remarkable progress, yet their internal mechanisms remain largely opaque, posing a significant challenge to their safe and reliable deployment. Sparse autoencoders (SAEs) have emerged as a promising tool for decomposing LLM representations into more interpretable features, but explaining the features captured by SAEs remains a challenging task. In this work, we propose SAGE (SAE AGentic Explainer), an agent-based framework that recasts feature interpretation from a passive, single-pass generation task into an active, explanation-driven process. SAGE implements a rigorous methodology by systematically formulating multiple explanations for each feature, designing targeted experiments to test them, and iteratively refining explanations based on empirical activation feedback. Experiments on features from SAEs of diverse language models demonstrate that SAGE produces explanations with significantly higher generative and predictive accuracy compared to state-of-the-art this http URL agent-based framework that recasts feature interpretation from a passive, single-pass generation task into an active, explanationdriven process. SAGE implements a rigorous methodology by systematically formulating multiple explanations for each feature, designing targeted experiments to test them, and iteratively refining explanations based on empirical activation feedback. Experiments on features from SAEs of diverse language models demonstrate that SAGE produces explanations with significantly higher generative and predictive accuracy compared to state-of-the-art baselines.
• 摘要：大型语言模型（LLM）已经取得了显着的进步，但其内部机制仍然很大程度上不透明，对其安全可靠的部署构成了重大挑战。稀疏自动编码器 (SAE) 已成为一种很有前途的工具，用于将 LLM 表示分解为更可解释的特征，但解释 SAE 捕获的特征仍然是一项具有挑战性的任务。在这项工作中，我们提出了 SAGE（SAE AGenticExplainer），这是一种基于代理的框架，它将特征解释从被动的单遍生成任务重新转变为主动的、解释驱动的过程。 SAGE 通过系统地为每个特征制定多种解释，设计有针对性的实验来测试它们，并根据经验激活反馈迭代地完善解释，从而实现了严格的方法论。对不同语言模型的 SAE 特征进行的实验表明，与最先进的基于 http URL 代理的框架相比，SAGE 生成的解释具有显着更高的生成和预测准确性，该框架将特征解释从被动的单遍生成任务重新转变为主动的解释驱动过程。 SAGE 通过系统地为每个特征制定多种解释，设计有针对性的实验来测试它们，并根据经验激活反馈迭代地完善解释，从而实现了严格的方法论。对不同语言模型的 SAE 特征进行的实验表明，与最先进的基线相比，SAGE 生成的解释具有显着更高的生成和预测准确性。

Title: Structured Prompting Enables More Robust, Holistic Evaluation of Language Models

• Authors: Asad Aali, Muhammad Ahmed Mohsin, Vasiliki Bikia, Arnav Singhvi, Richard Gaus, Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Yifan Mai, Jordan Cahoon, Michael Pfeffer, Roxana Daneshjou, Sanmi Koyejo, Emily Alsentzer, Percy Liang, Christopher Potts, Nigam H. Shah, Akshay S. Chaudhari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.20836
• Pdf URL: https://arxiv.org/pdf/2511.20836
• Copy Paste: [[2511.20836]] Structured Prompting Enables More Robust, Holistic Evaluation of Language Models(https://arxiv.org/abs/2511.20836)
• Keywords: language model, prompt, chain-of-thought
• Abstract: As language models (LMs) are increasingly adopted across domains, high-quality benchmarking frameworks that accurately estimate performance are essential for guiding deployment decisions. While frameworks such as Holistic Evaluation of Language Models (HELM) enable broad evaluation across tasks, they often rely on fixed prompts that fail to generalize across LMs, yielding unrepresentative performance estimates. Unless we estimate each LM's ceiling (maximum achievable via changes to the prompt), we risk underestimating performance. Declarative prompting frameworks, such as DSPy, offer a scalable alternative to manual prompt engineering by crafting structured prompts that can be optimized per task. However, such frameworks have not been systematically evaluated across established benchmarks. We present a reproducible DSPy+HELM framework that introduces structured prompting methods which elicit reasoning, enabling more accurate LM benchmarking. Using four prompting methods, we evaluate four frontier LMs across seven benchmarks (general/medical domain) against existing HELM baseline scores. We find that without structured prompting: (i) HELM underestimates LM performance (by 4% average), (ii) performance estimates vary more across benchmarks (+2% standard deviation), (iii) performance gaps are misrepresented (leaderboard rankings flip on 3/7 benchmarks), and (iv) introducing reasoning (chain-of-thought) reduces LM sensitivity to prompt design (smaller {\Delta} across prompts). To our knowledge, this is the first large-scale benchmarking study to empirically characterize LM behavior across benchmarks and prompting methods, showing that scalable performance ceiling estimation enables more decision-useful benchmarks. We open-source (i) DSPy+HELM Integration (this https URL) and (ii) Prompt Optimization Pipeline (this https URL).
• 摘要：随着语言模型 (LM) 越来越多地被跨领域采用，准确评估性能的高质量基准测试框架对于指导部署决策至关重要。虽然语言模型整体评估 (HELM) 等框架可以跨任务进行广泛评估，但它们通常依赖于无法在 LM 之间泛化的固定提示，从而产生不具有代表性的性能估计。除非我们估计每个 LM 的上限（通过更改提示可实现的最大值），否则我们可能会低估性能。声明性提示框架（例如 DSPy）通过制作可针对每个任务进行优化的结构化提示，为手动提示工程提供了可扩展的替代方案。然而，此类框架尚未根据既定基准进行系统评估。我们提出了一个可重复的 DSPy+HELM 框架，该框架引入了引发推理的结构化提示方法，从而实现更准确的 LM 基准测试。使用四种提示方法，我们根据现有 HELM 基线分数评估了七个基准（普通/医学领域）的四个前沿 LM。我们发现，如果没有结构化提示：(i) HELM 会低估 LM 性能（平均降低 4%），(ii) 不同基准之间的性能估计差异更大（+2% 标准差），(iii) 性能差距被歪曲（排行榜排名在 3/7 基准上翻转），以及 (iv) 引入推理（思维链）会降低 LM 对提示设计的敏感性（不同提示之间的 {\Delta} 更小）。据我们所知，这是第一个大规模基准研究，以实证方式描述跨基准和提示方法的 LM 行为，表明可扩展的性能上限估计可以实现更有用的基准。我们开源 (i) DSPy+HELM 集成（此 https URL）和 (ii) 提示优化管道（此 https URL）。

Title: Length-MAX Tokenizer for Language Models

• Authors: Dong Dong, Weijie Su
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.20849
• Pdf URL: https://arxiv.org/pdf/2511.20849
• Copy Paste: [[2511.20849]] Length-MAX Tokenizer for Language Models(https://arxiv.org/abs/2511.20849)
• Keywords: language model, gpt
• Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference. Our method, which we refer to as the Length-MAX tokenizer, obtains its vocabulary by casting a length-weighted objective maximization as a graph partitioning problem and developing a greedy approximation algorithm. On FineWeb and diverse domains, it yields 14--18\% fewer tokens than Byte Pair Encoding (BPE) across vocabulary sizes from 10K to 50K, and the reduction is 13.0\% when the size is 64K. Training GPT-2 models at 124M, 355M, and 1.3B parameters from scratch with five runs each shows 18.5\%, 17.2\%, and 18.5\% fewer steps, respectively, to reach a fixed validation loss, and 13.7\%, 12.7\%, and 13.7\% lower inference latency, together with a 16\% throughput gain at 124M, while consistently improving on downstream tasks including reducing LAMBADA perplexity by 11.7\% and enhancing HellaSwag accuracy by 4.3\%. Moreover, the Length-MAX tokenizer achieves 99.62\% vocabulary coverage and the out-of-vocabulary rate remains low at 0.12\% on test sets. These results demonstrate that optimizing for average token length, rather than frequency alone, offers an effective approach to more efficient language modeling without sacrificing -- and often improving -- downstream performance. The tokenizer is compatible with production systems and reduces embedding and KV-cache memory by 18\% at inference.
• 摘要：我们为语言模型引入了一种新的标记器，它可以最大限度地减少每个字符的平均标记，从而减少在训练期间表示文本和在推理期间生成文本所需的标记数量。我们的方法（我们将其称为“长度最大分词器”）通过将长度加权目标最大化转换为图划分问题并开发贪婪近似算法来获取其词汇表。在 FineWeb 和不同的领域中，在词汇大小从 10K 到 50K 的范围内，它产生的标记比字节对编码 (BPE) 少 14--18\%，当词汇大小为 64K 时，减少量为 13.0\%。从头开始以 124M、355M 和 1.3B 参数训练 GPT-2 模型，每次运行五次，显示达到固定验证损失的步骤分别减少了 18.5%、17.2% 和 18.5%，推理延迟降低了 13.7%、12.7% 和 13.7%，并且在 124M 时吞吐量增益提高了 16%，同时不断改进下游任务，包括将 LAMBADA 困惑度降低 11.7%，将 HellaSwag 准确度提高 4.3%。此外，Length-MAX 分词器在测试集上实现了 99.62% 的词汇覆盖率，词汇外率保持在 0.12% 的低水平。这些结果表明，优化平均标记长度而不是单独优化频率，可以提供一种有效的方法来实现更高效的语言建模，而不会牺牲（并且通常会提高）下游性能。分词器与生产系统兼容，并在推理时将嵌入和 KV 缓存内存减少 18%。

Title: Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory

• Authors: Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.20857
• Pdf URL: https://arxiv.org/pdf/2511.20857
• Copy Paste: [[2511.20857]] Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory(https://arxiv.org/abs/2511.20857)
• Keywords: language model, llm, agent
• Abstract: Statefulness is essential for large language model (LLM) agents to perform long-term planning and problem-solving. This makes memory a critical component, yet its management and evolution remain largely underexplored. Existing evaluations mostly focus on static conversational settings, where memory is passively retrieved from dialogue to answer queries, overlooking the dynamic ability to accumulate and reuse experience across evolving task streams. In real-world environments such as interactive problem assistants or embodied agents, LLMs are required to handle continuous task streams, yet often fail to learn from accumulated interactions, losing valuable contextual insights, a limitation that calls for test-time evolution, where LLMs retrieve, integrate, and update memory continuously during deployment. To bridge this gap, we introduce Evo-Memory, a comprehensive streaming benchmark and framework for evaluating self-evolving memory in LLM agents. Evo-Memory structures datasets into sequential task streams, requiring LLMs to search, adapt, and evolve memory after each interaction. We unify and implement over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets. To better benchmark experience reuse, we provide a baseline method, ExpRAG, for retrieving and utilizing prior experience, and further propose ReMem, an action-think-memory refine pipeline that tightly integrates reasoning, task actions, and memory updates to achieve continual improvement.
• 摘要：有状态对于大型语言模型 (LLM) 代理执行长期规划和解决问题至关重要。这使得内存成为一个关键组成部分，但其管理和进化在很大程度上仍未得到充分探索。现有的评估主要集中在静态对话设置上，其中记忆是从对话中被动检索以回答查询，忽视了在不断发展的任务流中积累和重用经验的动态能力。在交互式问题助手或实体代理等现实环境中，LLM 需要处理连续的任务流，但通常无法从累积的交互中学习，从而丢失有价值的上下文见解，这是需要测试时演化的限制，其中 LLM 在部署期间不断检索、集成和更新内存。为了弥补这一差距，我们引入了 Evo-Memory，这是一个全面的流基准测试和框架，用于评估 LLM 代理中的自我进化内存。 Evo-Memory 将数据集构建为连续的任务流，要求法学硕士在每次交互后搜索、适应和进化记忆。我们统一并实现了十多个代表性内存模块，并在 10 个不同的多轮目标导向和单轮推理和 QA 数据集上对它们进行评估。为了更好地对经验重用进行基准测试，我们提供了一种基线方法 ExpRAG，用于检索和利用先前的经验，并进一步提出了 ReMem，一种紧密集成推理、任务操作和记忆更新以实现持续改进的行动-思考-记忆细化管道。

Title: Winning with Less for Low Resource Languages: Advantage of Cross-Lingual English_Persian Argument Mining Model over LLM Augmentation

• Authors: Ali Jahan, Masood Ghayoomi, Annette Hautli-Janisz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20872
• Pdf URL: https://arxiv.org/pdf/2511.20872
• Copy Paste: [[2511.20872]] Winning with Less for Low Resource Languages: Advantage of Cross-Lingual English_Persian Argument Mining Model over LLM Augmentation(https://arxiv.org/abs/2511.20872)
• Keywords: language model, llm
• Abstract: Argument mining is a subfield of natural language processing to identify and extract the argument components, like premises and conclusions, within a text and to recognize the relations between them. It reveals the logical structure of texts to be used in tasks like knowledge extraction. This paper aims at utilizing a cross-lingual approach to argument mining for low-resource languages, by constructing three training scenarios. We examine the models on English, as a high-resource language, and Persian, as a low-resource language. To this end, we evaluate the models based on the English Microtext corpus \citep{PeldszusStede2015}, and its parallel Persian translation. The learning scenarios are as follow: (i) zero-shot transfer, where the model is trained solely with the English data, (ii) English-only training enhanced by synthetic examples generated by Large Language Models (LLMs), and (iii) a cross-lingual model that combines the original English data with manually translated Persian sentences. The zero-shot transfer model attains F1 scores of 50.2\% on the English test set and 50.7\% on the Persian test set. LLM-based augmentation model improves the performance up to 59.2\% on English and 69.3\% on Persian. The cross-lingual model, trained on both languages but evaluated solely on the Persian test set, surpasses the LLM-based variant, by achieving a F1 of 74.8\%. Results indicate that a lightweight cross-lingual blend can outperform considerably the more resource-intensive augmentation pipelines, and it offers a practical pathway for the argument mining task to overcome data resource shortage on low-resource languages.
• 摘要：论证挖掘是自然语言处理的一个子领域，用于识别和提取文本中的论证成分（例如前提和结论）并识别它们之间的关系。它揭示了用于知识提取等任务的文本的逻辑结构。本文旨在通过构建三个训练场景，利用跨语言方法对低资源语言进行论证挖掘。我们研究了英语（高资源语言）和波斯语（低资源语言）的模型。为此，我们基于英语缩微文本语料库 \citep{PeldszusStede2015} 及其并行波斯语翻译来评估模型。学习场景如下：（i）零样本迁移，仅使用英语数据训练模型；（ii）通过大型语言模型（LLM）生成的综合示例增强纯英语训练；（iii）将原始英语数据与手动翻译的波斯语句子相结合的跨语言模型。零样本迁移模型在英语测试集上获得了 50.2% 的 F1 分数，在波斯语测试集上获得了 50.7% 的 F1 分数。基于 LLM 的增强模型将英语的性能提高了 59.2%，波斯语的性能提高了 69.3%。该跨语言模型使用两种语言进行训练，但仅在波斯语测试集上进行评估，其 F1 达到 74.8%，超越了基于 LLM 的变体。结果表明，轻量级跨语言混合可以大大优于资源密集型增强管道，并且它为参数挖掘任务克服低资源语言上的数据资源短缺问题提供了一条实用途径。

Title: Emergence and Localisation of Semantic Role Circuits in LLMs

• Authors: Nura Aljaafari, Danilo S. Carvalho, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20910
• Pdf URL: https://arxiv.org/pdf/2511.20910
• Copy Paste: [[2511.20910]] Emergence and Localisation of Semantic Role Circuits in LLMs(https://arxiv.org/abs/2511.20910)
• Keywords: language model, llm
• Abstract: Despite displaying semantic competence, large language models' internal mechanisms that ground abstract semantic structure remain insufficiently characterised. We propose a method integrating role-cross minimal pairs, temporal emergence analysis, and cross-model comparison to study how LLMs implement semantic roles. Our analysis uncovers: (i) highly concentrated circuits (89-94% attribution within 28 nodes); (ii) gradual structural refinement rather than phase transitions, with larger models sometimes bypassing localised circuits; and (iii) moderate cross-scale conservation (24-59% component overlap) alongside high spectral similarity. These findings suggest that LLMs form compact, causally isolated mechanisms for abstract semantic structure, and these mechanisms exhibit partial transfer across scales and architectures.
• 摘要：尽管显示了语义能力，但大型语言模型支撑抽象语义结构的内部机制仍然没有得到充分的表征。我们提出了一种集成角色跨最小对、时间涌现分析和跨模型比较的方法来研究法学硕士如何实现语义角色。我们的分析揭示了：(i) 高度集中的电路（89-94% 归因于 28 个节点内）； (ii) 逐步结构细化而不是相变，较大的模型有时会绕过局部电路； (iii) 适度的跨尺度保护（24-59% 成分重叠）以及高光谱相似性。这些发现表明法学硕士为抽象语义结构形成了紧凑的、因果隔离的机制，并且这些机制表现出跨尺度和架构的部分迁移。

Title: Chatty-KG: A Multi-Agent AI System for On-Demand Conversational Question Answering over Knowledge Graphs

• Authors: Reham Omar, Abdelghny Orogat, Ibrahim Abdelaziz, Omij Mangukiya, Panos Kalnis, Essam Mansour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.20940
• Pdf URL: https://arxiv.org/pdf/2511.20940
• Copy Paste: [[2511.20940]] Chatty-KG: A Multi-Agent AI System for On-Demand Conversational Question Answering over Knowledge Graphs(https://arxiv.org/abs/2511.20940)
• Keywords: language model, gpt, llm, chat, retrieval-augmented generation, agent
• Abstract: Conversational Question Answering over Knowledge Graphs (KGs) combines the factual grounding of KG-based QA with the interactive nature of dialogue systems. KGs are widely used in enterprise and domain applications to provide structured, evolving, and reliable knowledge. Large language models (LLMs) enable natural and context-aware conversations, but lack direct access to private and dynamic KGs. Retrieval-augmented generation (RAG) systems can retrieve graph content but often serialize structure, struggle with multi-turn context, and require heavy indexing. Traditional KGQA systems preserve structure but typically support only single-turn QA, incur high latency, and struggle with coreference and context tracking. To address these limitations, we propose Chatty-KG, a modular multi-agent system for conversational QA over KGs. Chatty-KG combines RAG-style retrieval with structured execution by generating SPARQL queries through task-specialized LLM agents. These agents collaborate for contextual interpretation, dialogue tracking, entity and relation linking, and efficient query planning, enabling accurate and low-latency translation of natural questions into executable queries. Experiments on large and diverse KGs show that Chatty-KG significantly outperforms state-of-the-art baselines in both single-turn and multi-turn settings, achieving higher F1 and P@1 scores. Its modular design preserves dialogue coherence and supports evolving KGs without fine-tuning or pre-processing. Evaluations with commercial (e.g., GPT-4o, Gemini-2.0) and open-weight (e.g., Phi-4, Gemma 3) LLMs confirm broad compatibility and stable performance. Overall, Chatty-KG unifies conversational flexibility with structured KG grounding, offering a scalable and extensible approach for reliable multi-turn KGQA.
• 摘要：基于知识图谱 (KG) 的会话式问答将基于知识图谱的 QA 的事实基础与对话系统的交互性质相结合。知识图谱广泛应用于企业和领域应用程序中，以提供结构化、不断发展且可靠的知识。大型语言模型 (LLM) 可实现自然且上下文感知的对话，但缺乏对私有和动态知识图谱的直接访问。检索增强生成（RAG）系统可以检索图形内容，但通常会序列化结构，难以应对多轮上下文，并且需要大量索引。传统的 KGQA 系统保留结构，但通常仅支持单轮 QA，会产生高延迟，并且难以处理共指和上下文跟踪。为了解决这些限制，我们提出了 Chatty-KG，这是一种模块化多代理系统，用于通过 KG 进行对话 QA。 Chatty-KG 通过任务专用 LLM 代理生成 SPARQL 查询，将 RAG 式检索与结构化执行相结合。这些代理协作进行上下文解释、对话跟踪、实体和关系链接以及高效的查询规划，从而实现将自然问题准确且低延迟地转换为可执行查询。对大型且多样化的 KG 进行的实验表明，Chatty-KG 在单轮和多轮设置中均显着优于最先进的基线，实现了更高的 F1 和 P@1 分数。其模块化设计保留了对话的连贯性，并支持不断发展的知识图谱，无需微调或预处理。对商业（例如 GPT-4o、Gemini-2.0）和开放权重（例如 Phi-4、Gemma 3）LLM 的评估证实了广泛的兼容性和稳定的性能。总体而言，Chatty-KG 将会话灵活性与结构化 KG 基础相结合，为可靠的多轮 KGQA 提供了可扩展且可扩展的方法。

Title: TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models

• Authors: Ioana Buhnila, Aman Sinha, Mathieu Constant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21006
• Pdf URL: https://arxiv.org/pdf/2511.21006
• Copy Paste: [[2511.21006]] TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models(https://arxiv.org/abs/2511.21006)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have proven efficient in giving definition-type answers to user input queries. While for humans giving various types of answers, such as examples and paraphrases, is an easy task, LLMs struggle to provide correct answers for other than definition-type queries. In this study, we evaluated this drop in performance using TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also introduce RefoMed-EN, an English dataset consisting of 6170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We studied whether the high frequency of a concept (head) or low frequency (tail) impacts the language model's performance. We evaluated the quality of the LLM's output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM's task performance for definition type questions is the highest, while for the exemplification type it is the lowest. Additionally, we showed that for definition-type questions, large language models are prone to paraphrase more on popular and frequent knowledge and less on tail and technical knowledge, especially in the expert texts.
• 摘要：事实证明，大型语言模型 (LLM) 可以有效地为用户输入查询提供定义类型的答案。虽然对于人类来说给出各种类型的答案（例如示例和释义）是一项简单的任务，但法学硕士很难为定义类型查询之外的其他问题提供正确的答案。在本研究中，我们使用 TrackList（一种细粒度的语言和统计分析管道）评估了性能下降的情况，以调查预训练数据对法学硕士回答各种语言查询的影响。我们还介绍了 RefoMed-EN，这是一个英语数据集，由 6170 个人工注释的医学术语及其相应的定义、名称、例证、解释或释义组成。我们研究了概念的高频（头）或低频（尾）是否会影响语言模型的性能。我们使用句法和语义相似性指标、统计相关性和嵌入来评估法学硕士输出的质量。结果显示，法学硕士对于定义类型问题的任务绩效最高，而对于示例类型问题的任务绩效最低。此外，我们还表明，对于定义类型的问题，大型语言模型倾向于更多地解释流行和频繁的知识，而较少解释尾部和技术知识，尤其是在专家文本中。

Title: Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels

• Authors: Anantha Padmanaban Krishna Kumar (Boston University)
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.21038
• Pdf URL: https://arxiv.org/pdf/2511.21038
• Copy Paste: [[2511.21038]] Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels(https://arxiv.org/abs/2511.21038)
• Keywords: llm, prompt
• Abstract: Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \emph{natural} demonstrations (with correct labels) and \emph{inverted} demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: this https URL.
• 摘要：上下文学习 (ICL) 是否可以覆盖预先训练的标签语义，或者只是细化现有的语义主干？我们通过将 LLM 视为提示诱导分类器并对比它们在 \emph{natural} 演示（具有正确标签）和 \emph{inverted} 演示（系统地翻转标签含义）下的行为来解决这个问题。我们将 ICL 行为分解为三个对齐指标（真实对齐、先验对齐和提示对齐），并引入语义覆盖率，定义为翻转语义下的正确性。在八个分类任务和八个开源法学硕士（1--12B 参数）中，我们找到了语义锚视图的一致证据。通过自然演示，ICL 提高了准确性，同时保持了强大的先前对准；大多数正确的预测与零样本行为一致，即使先验很弱。通过反向演示，模型无法学习连贯的反语义分类器：只有通过牺牲准确性才能提高提示对齐，并且在我们的少样本 1--12B 设置中，语义覆盖率始终保持为零。 ICL 不是灵活地重新映射标签含义，而是主要调整输入如何投射到预训练期间学习的稳定语义方向上，澄清小样本提示的基本限制，并建议在这些尺度上覆盖标签语义需要 ICL 之外的干预措施。所有代码均可在以下位置获取：此 https URL。

Title: Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection

• Authors: Michael Iskandardinata, William Christian, Derwin Suhartono
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.21066
• Pdf URL: https://arxiv.org/pdf/2511.21066
• Copy Paste: [[2511.21066]] Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection(https://arxiv.org/abs/2511.21066)
• Keywords: language model, llm, prompt
• Abstract: Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: this https URL.
• 摘要：尽管神经网络方法最近取得了进展，但在自然语言处理（NLP）领域，检测讽刺仍然是一项具有挑战性的任务。目前，预训练语言模型 (PLM) 和大型语言模型 (LLM) 是讽刺检测的首选方法。然而，讽刺文本的复杂性，加上不同社区的语言多样性和文化差异，甚至对于 PLM 和 LLM 来说，也使得这项任务变得更加困难。除此之外，这些模型还表现出对单词或标记的不可靠检测，需要额外的分析基础。基于法学硕士中用于讽刺检测的最先进的提示方法（称为语用元认知提示（PMP）），我们引入了一种检索感知方法，该方法结合了每个目标文本检索到的上下文信息。我们的流程探索了两种互补的方式来提供上下文：当模型缺乏必要的背景时，使用基于网络的检索添加非参数知识，并引出模型自身的内部知识以实现自我知识意识策略。我们使用三个数据集评估了我们的方法，例如 Twitter Indonesia Sarcastic、SemEval-2018 Task 3 和 MUStARD。与原始 PMP 方法相比，非参数检索使 Twitter Indonesia Sarcastic 的宏观 F1 显着提高了 9.87%。自我知识检索将 Semeval 上的宏观 F1 提高了 3.29%，将 MUStARD 上的宏观 F1 提高了 4.08%。这些发现强调了背景对于提高法学硕士在讽刺检测任务中的表现的重要性，特别是涉及法学硕士的文化特定俚语、参考文献或未知术语。未来的工作将集中于优化相关上下文信息的检索并检查检索质量如何影响性能。实验代码位于：此 https URL。

Title: Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models

• Authors: Bryan E. Tuck, Rakesh M. Verma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21086
• Pdf URL: https://arxiv.org/pdf/2511.21086
• Copy Paste: [[2511.21086]] Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models(https://arxiv.org/abs/2511.21086)
• Keywords: language model, gpt
• Abstract: Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-architecture evaluation remains limited. We evaluate 28 configurations spanning three model families (Qwen3, Claude Haiku-4.5, GPT-5-mini) on 58 word puzzles requiring character-level constraint satisfaction. Architectural differences produce substantially larger performance gaps (2.0-2.2x, F1=0.761 vs. 0.343) than parameter scaling within families (83% gain from eightfold scaling), suggesting that constraint satisfaction may require specialized architectural features or training objectives beyond standard language model scaling. Thinking budget sensitivity proves heterogeneous: high-capacity models show strong returns (+0.102 to +0.136 F1), while mid-sized variants saturate or degrade. These patterns are inconsistent with uniform compute benefits. Using difficulty ratings from 10,000 human solvers per puzzle, we establish modest but consistent calibration (r=0.24-0.38) across all families, yet identify systematic failures on common words with unusual orthography ("data", "poop", "loll": 86-95% human success, 89-96% model miss rate). These failures reveal over-reliance on distributional plausibility that penalizes orthographically atypical but constraint-valid patterns, suggesting architectural innovations may be required beyond simply scaling parameters or computational budgets.
• 摘要：大型语言模型必须在受控文本生成过程中满足严格的拼写约束，但系统的跨架构评估仍然有限。我们在需要字符级约束满足的 58 个单词谜题上评估了跨越三个模型系列（Qwen3、Claude Haiku-4.5、GPT-5-mini）的 28 个配置。架构差异产生的性能差距（2.0-2.2x，F1=0.761 vs. 0.343）比族内的参数缩放（八倍缩放获得 83% 的收益）要大得多，这表明约束满足可能需要超出标准语言模型缩放的专门架构特征或训练目标。事实证明，思维预算敏感性存在差异：高容量型号显示出强劲的回报（+0.102 至 +0.136 F1），而中型型号则饱和或下降。这些模式与统一的计算优势不一致。使用每个谜题 10,000 名人类解决者的难度评级，我们在所有系列中建立了适度但一致的校准 (r=0.24-0.38)，同时识别出具有不寻常拼写法的常见单词的系统性失败（“data”、“poop”、“loll”：人类成功率为 86-95%，模型失误率为 89-96%）。这些失败揭示了对分布合理性的过度依赖，这会惩罚拼写上非典型但约束有效的模式，这表明可能需要的架构创新不仅仅是简单地缩放参数或计算预算。

Title: MortgageLLM: Domain-Adaptive Pretraining with Residual Instruction Transfer, Alignment Tuning, and Task-Specific Routing

• Authors: Manish Jain, Satheesh Kumar Ponnambalam, Salman Faroz, Chandrakanth Lns, Vinay Sharma
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.21101
• Pdf URL: https://arxiv.org/pdf/2511.21101
• Copy Paste: [[2511.21101]] MortgageLLM: Domain-Adaptive Pretraining with Residual Instruction Transfer, Alignment Tuning, and Task-Specific Routing(https://arxiv.org/abs/2511.21101)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) demonstrate exceptional capabilities across general domains, yet their application to specialized sectors such as mortgage finance requires domain-specific knowledge augmentation while preserving instruction-following fidelity. We present MortgageLLM, a novel domain-specific large language model that addresses this dual challenge. It is developed using a dual-track specialization framework from a single base model (LLaMA-3.1-8B). We opted for this dual-expert approach as a single multi-task model suffers from performance trade-offs, where optimizing for structured tasks (via SFT) degrades conversational fidelity (via DPO). Our dual-track method solves this by creating two specialists, allowing each to be optimally trained for its distinct capability. Our approach applies the instruction residual technique to restore instruction-following capabilities post-domain adaptation without supervised fine-tuning. We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself. We validate our approach on domain-specific benchmarks, where our final model (MLM v2) significantly outperforms the base LLaMA-3.1-8B-Instruct, achieving an LLM-as-a-Judge summarization score of 4.58 (vs. 3.99), a Q&A score of 4.09 (vs. 4.0), and a classification score of 2.6 (vs. 1.2). On semantic similarity, our model achieved a BERTScore of 0.77 for summarization (vs. 0.74), 0.68 for Q&A (vs. 0.58), and 0.75 for classification (vs. 0.73), substantially outperforming baseline approaches.
• 摘要：大型语言模型 (LLM) 在一般领域展示了卓越的能力，但它们在抵押贷款金融等专业领域的应用需要特定领域的知识增强，同时保持指令遵循的保真度。我们提出了 MortgageLLM，这是一种新颖的特定领域大语言模型，可以解决这一双重挑战。它是使用基于单一基础模型 (LLaMA-3.1-8B) 的双轨专业化框架开发的。我们选择这种双专家方法是因为单个多任务模型会受到性能权衡的影响，其中对结构化任务的优化（通过 SFT）会降低对话保真度（通过 DPO）。我们的双轨方法通过创建两名专家来解决这个问题，使每个专家都能够针对其独特的能力进行最佳培训。我们的方法应用指令残差技术来恢复域后适应的指令跟随能力，而无需监督微调。我们的贡献：（1）将这种剩余技术应用于高度专业化的抵押贷款金融领域； （2）结合对话式问答模型和用于分类和摘要的结构化任务模型的双专家架构； (3) 一种智能任务路由机制，使用由专家模型之一本身执行的少样本分类。我们在特定领域的基准上验证了我们的方法，其中我们的最终模型 (MLM v2) 显着优于基础 LLaMA-3.1-8B-Instruct，LLM 作为法官的总结得分为 4.58（对比 3.99），问答得分为 4.09（对比 4.0），分类得分为 2.6（对比 1.2）。在语义相似性方面，我们的模型在摘要方面的 BERTScore 为 0.77（相对于 0.74），问答为 0.68（相对于 0.58），分类为 0.75（相对于 0.73），大大优于基线方法。

Title: Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines

• Authors: Yuhang Wang, Yanxu Zhu, Dongyuan Lu, Jitao Sang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.21214
• Pdf URL: https://arxiv.org/pdf/2511.21214
• Copy Paste: [[2511.21214]] Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines(https://arxiv.org/abs/2511.21214)
• Keywords: prompt
• Abstract: Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
• 摘要：推理模型在复杂的推理任务中表现出了卓越的能力。然而，确保他们的安全免受对抗性越狱提示仍然是一个严峻的挑战。由于此类提示的隐蔽性和欺骗性，它们通常可以逃避内置的安全机制并导致有害内容的生成。这强调了对自适应安全调整方法的需求，该方法使模型能够自主加强防御以响应对抗性输入。本文介绍了基于综合指南的自适应安全调整（SGASA）框架，该框架将模型生成的安全指南内化，以增强模型增强针对有害对抗性提示的鲁棒性的能力，同时最大限度地减少对良性请求的不必要拒绝。 SGASA 包含两个关键阶段： 数据预合成，生成安全指南和增强提示；对齐微调，利用监督微调 (SFT) 和直接偏好优化 (DPO) 将这些准则嵌入到模型中。跨多个数据集的广泛实验表明，SGASA 显着提高了模型安全性，验证了其自适应和可扩展的有效性。

Title: Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?

• Authors: Steven Wang, Kyle Hunt, Shaojie Tang, Kenneth Joseph
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21218
• Pdf URL: https://arxiv.org/pdf/2511.21218
• Copy Paste: [[2511.21218]] Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?(https://arxiv.org/abs/2511.21218)
• Keywords: language model, llm
• Abstract: There is ongoing debate about whether large language models (LLMs) can serve as substitutes for human participants in survey and experimental research. While recent work in fields such as marketing and psychology has explored the potential of LLM-based simulation, a growing body of evidence cautions against this practice: LLMs often fail to align with real human behavior, exhibiting limited diversity, systematic misalignment for minority subgroups, insufficient within-group variance, and discrepancies between stated beliefs and actions. This study examines an important and distinct question in this domain: whether fine-tuning on a small subset of human survey data, such as that obtainable from a pilot study, can mitigate these issues and yield realistic simulated outcomes. Using a behavioral experiment on information disclosure, we compare human and LLM-generated responses across multiple dimensions, including distributional divergence, subgroup alignment, belief-action coherence, and the recovery of regression coefficients. We find that fine-tuning on small human samples substantially improves heterogeneity, alignment, and belief-action coherence relative to the base model. However, even the best-performing fine-tuned models fail to reproduce the regression coefficients of the original study, suggesting that LLM-generated data remain unsuitable for replacing human participants in formal inferential analyses.
• 摘要：关于大型语言模型（LLM）是否可以替代人类参与者参与调查和实验研究，一直存在争论。虽然最近在市场营销和心理学等领域的工作已经探索了基于法学硕士的模拟的潜力，但越来越多的证据警告人们不要这种做法：法学硕士往往无法与真实的人类行为保持一致，表现出有限的多样性，少数群体的系统性失调，群体内差异不足，以及陈述的信念和行动之间的差异。本研究探讨了该领域中一个重要且独特的问题：对一小部分人类调查数据（例如从试点研究中获得的数据）进行微调是否可以缓解这些问题并产生真实的模拟结果。通过信息披露的行为实验，我们在多个维度上比较了人类和法学硕士产生的反应，包括分布分歧、亚组对齐、信念-行动一致性和回归系数的恢复。我们发现，相对于基本模型，对小型人类样本进行微调可以显着改善异质性、一致性和信念-行动一致性。然而，即使是性能最好的微调模型也无法重现原始研究的回归系数，这表明法学硕士生成的数据仍然不适合在正式的推理分析中替代人类参与者。

Title: PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark

• Authors: Robert Belanec, Branislav Pecher, Ivan Srba, Maria Bielikova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21285
• Pdf URL: https://arxiv.org/pdf/2511.21285
• Copy Paste: [[2511.21285]] PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark(https://arxiv.org/abs/2511.21285)
• Keywords: language model, llm
• Abstract: Despite the state-of-the-art performance of Large Language Models (LLMs) achieved on many tasks, their massive scale often leads to high computational and environmental costs, limiting their accessibility. Parameter-efficient fine-tuning (PEFT) methods address this challenge by reducing the number of trainable parameters while maintaining strong downstream performance. Despite the increased development in PEFT methods, current evaluations remain limited (in terms of evaluated models and datasets) and difficult to reproduce. To bridge this gap, we introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs. We demonstrate its usage across 27 NLP datasets and 6 PEFT methods. To account for different PEFT training and inference factors, we also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.
• 摘要：尽管大型语言模型 (LLM) 在许多任务上实现了最先进的性能，但其大规模通常会导致高昂的计算和环境成本，从而限制了其可访问性。参数高效微调（PEFT）方法通过减少可训练参数的数量，同时保持强大的下游性能来解决这一挑战。尽管 PEFT 方法不断发展，但当前的评估仍然有限（在评估模型和数据集方面）并且难以重现。为了弥补这一差距，我们引入了 PEFT-Bench，这是一个统一的端到端基准，用于评估自回归 LLM 上的各种 PEFT 方法。我们展示了它在 27 个 NLP 数据集和 6 种 PEFT 方法中的用法。为了考虑不同的 PEFT 训练和推理因素，我们还引入了 PEFT 软评分惩罚 (PSCP) 指标，该指标考虑了可训练参数、推理速度和训练内存使用情况。

Title: Emergent Lexical Semantics in Neural Language Models: Testing Martin's Law on LLM-Generated Text

• Authors: Kai Kugler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21334
• Pdf URL: https://arxiv.org/pdf/2511.21334
• Copy Paste: [[2511.21334]] Emergent Lexical Semantics in Neural Language Models: Testing Martin's Law on LLM-Generated Text(https://arxiv.org/abs/2511.21334)
• Keywords: language model, llm
• Abstract: We present the first systematic investigation of Martin's Law - the empirical relationship between word frequency and polysemy - in text generated by neural language models during training. Using DBSCAN clustering of contextualized embeddings as an operationalization of word senses, we analyze four Pythia models (70M-1B parameters) across 30 training checkpoints. Our results reveal a non-monotonic developmental trajectory: Martin's Law emerges around checkpoint 100, reaches peak correlation (r > 0.6) at checkpoint 104, then degrades by checkpoint 105. Smaller models (70M, 160M) experience catastrophic semantic collapse at late checkpoints, while larger models (410M, 1B) show graceful degradation. The frequency-specificity trade-off remains stable (r $\approx$ -0.3) across all models. These findings suggest that compliance with linguistic regularities in LLM-generated text is not monotonically increasing with training, but instead follows a balanced trajectory with an optimal semantic window. This work establishes a novel methodology for evaluating emergent linguistic structure in neural language models.
• 摘要：我们首次系统地研究了马丁定律——词频和一词多义之间的经验关系——在神经语言模型在训练过程中生成的文本中。使用上下文嵌入的 DBSCAN 聚类作为词义的操作化，我们分析了 30 个训练检查点的 4 个 Pythia 模型（70M-1B 参数）。我们的结果揭示了非单调的发展轨迹：马丁定律在检查点 100 左右出现，在检查点 104 处达到峰值相关性（r > 0.6），然后在检查点 105 处退化。较小的模型（70M、160M）在后期检查点经历灾难性的语义崩溃，而较大的模型（410M、1B）则表现出优雅的退化。在所有模型中，频率特异性权衡保持稳定（r $\approx$ -0.3）。这些发现表明，法学硕士生成的文本中语言规律的符合性并不是随着训练而单调增加，而是遵循具有最佳语义窗口的平衡轨迹。这项工作建立了一种评估神经语言模型中的新兴语言结构的新颖方法。

Title: Training Introspective Behavior: Fine-Tuning Induces Reliable Internal State Detection in a 7B Model

• Authors: Joshua Fonseca Rivera
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.21399
• Pdf URL: https://arxiv.org/pdf/2511.21399
• Copy Paste: [[2511.21399]] Training Introspective Behavior: Fine-Tuning Induces Reliable Internal State Detection in a 7B Model(https://arxiv.org/abs/2511.21399)
• Keywords: language model
• Abstract: Lindsey (2025) investigates introspective awareness in language models through four experiments, finding that models can sometimes detect and identify injected activation patterns -- but unreliably (~20% success in the best model). We focus on the first of these experiments -- self-report of injected "thoughts" -- and ask whether this capability can be directly trained rather than waiting for emergence. Through fine-tuning on transient single-token injections, we transform a 7B parameter model from near-complete failure (0.4% accuracy, 6.7% false positive rate) to reliable detection (85% accuracy on held-out concepts at {\alpha}=40, 0% false positives). Our model detects fleeting "thoughts" injected at a single token position, retains that information, and reports the semantic content across subsequent generation steps. On this task, our trained model satisfies three of Lindsey's criteria: accuracy (correct identification), grounding (0/60 false positives), and internality (detection precedes verbalization). Generalization to unseen concept vectors (7.5pp gap) demonstrates the model learns a transferable skill rather than memorizing specific vectors, though this does not establish metacognitive representation in Lindsey's sense. These results address an open question raised by Lindsey: whether "training for introspection would help eliminate cross-model differences." We show that at least one component of introspective behavior can be directly induced, offering a pathway to built-in AI transparency.
• 摘要：Lindsey (2025) 通过四项实验研究了语言模型中的内省意识，发现模型有时可以检测和识别注入的激活模式，但不可靠（最佳模型成功率约为 20%）。我们关注第一个实验——注入“想法”的自我报告——并询问这种能力是否可以直接训练，而不是等待出现。通过对瞬态单令牌注入进行微调，我们将 7B 参数模型从接近完全失败（0.4% 准确率，6.7% 误报率）转变为可靠检测（在 {\alpha}=40 时保留概念的准确率达到 85%，0% 误报率）。我们的模型检测在单个标记位置注入的转瞬即逝的“想法”，保留该信息，并在后续生成步骤中报告语义内容。在这项任务中，我们训练的模型满足 Lindsey 的三个标准：准确性（正确识别）、基础性（0/60 误报）和内在性（检测先于言语）。对看不见的概念向量的泛化（7.5pp 差距）表明该模型学习了可转移的技能，而不是记住特定的向量，尽管这并没有建立 Lindsey 意义上的元认知表示。这些结果解决了 Lindsey 提出的一个悬而未决的问题：“内省训练是否有助于消除跨模型差异”。我们证明，至少内省行为的一个组成部分可以被直接诱导，从而为内置人工智能透明度提供了一条途径。

Title: Can LLMs extract human-like fine-grained evidence for evidence-based fact-checking?

• Authors: Antonín Jarolím, Martin Fajčík, Lucia Makaiová
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21401
• Pdf URL: https://arxiv.org/pdf/2511.21401
• Copy Paste: [[2511.21401]] Can LLMs extract human-like fine-grained evidence for evidence-based fact-checking?(https://arxiv.org/abs/2511.21401)
• Keywords: language model, gpt, llm
• Abstract: Misinformation frequently spreads in user comments under online news articles, highlighting the need for effective methods to detect factually incorrect information. To strongly support or refute claims extracted from such comments, it is necessary to identify relevant documents and pinpoint the exact text spans that justify or contradict each claim. This paper focuses on the latter task -- fine-grained evidence extraction for Czech and Slovak claims. We create new dataset, containing two-way annotated fine-grained evidence created by paid annotators. We evaluate large language models (LLMs) on this dataset to assess their alignment with human annotations. The results reveal that LLMs often fail to copy evidence verbatim from the source text, leading to invalid outputs. Error-rate analysis shows that the {llama3.1:8b model achieves a high proportion of correct outputs despite its relatively small size, while the gpt-oss-120b model underperforms despite having many more parameters. Furthermore, the models qwen3:14b, deepseek-r1:32b, and gpt-oss:20b demonstrate an effective balance between model size and alignment with human annotations.
• 摘要：错误信息经常在在线新闻文章下的用户评论中传播，这突出表明需要有效的方法来检测事实上不正确的信息。为了强烈支持或反驳从此类评论中提取的主张，有必要识别相关文件并查明证明或反驳每个主张的确切文本范围。本文重点关注后一个任务——捷克和斯洛伐克索赔的细粒度证据提取。我们创建新的数据集，包含由付费注释者创建的双向注释细粒度证据。我们在此数据集上评估大型语言模型（LLM），以评估它们与人类注释的一致性。结果表明，法学硕士常常无法从源文本中逐字复制证据，从而导致无效的输出。错误率分析表明，{llama3.1:8b 模型尽管尺寸相对较小，但仍实现了较高比例的正确输出，而 gpt-oss-120b 模型尽管参数较多，但表现不佳。此外，模型 qwen3:14b、deepseek-r1:32b 和 gpt-oss:20b 展示了模型大小和与人类注释对齐之间的有效平衡。

Title: Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation

• Authors: Zhifeng Hao, Qibin Song, Ruichu Cai, Boyan Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21402
• Pdf URL: https://arxiv.org/pdf/2511.21402
• Copy Paste: [[2511.21402]] Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation(https://arxiv.org/abs/2511.21402)
• Keywords: language model, llm, chain-of-thought
• Abstract: Recent divide-and-conquer reasoning approaches, particularly those based on Chain-of-Thought (CoT), have substantially improved the Text-to-SQL capabilities of Large Language Models (LLMs). However, when applied to complex enterprise databases, such methods struggle to maintain coherent reasoning due to limited context capacity, unreliable schema linking, and weak grounding in database semantics. To overcome these issues, we introduce DSR-SQL, a \textbf{D}ual-\textbf{S}tate \textbf{R}easoning framework that models Text-to-SQL as an interaction between an adaptive context state and a progressive generation state. The first constructs a compact, semantically faithful environment by refining large schemas and selecting relevant structures, while the second formalizes SQL synthesis as feedback-guided state transitions, enabling the model to self-correct and align with user intent. Without any post-training or in-context examples, DSR-SQL achieves competitive performance, reaching 35.28\% execution accuracy on Spider 2.0-Snow and 68.32\% on BIRD development set. Our implementation will be open-sourced at: this https URL.
• 摘要：最近的分而治之的推理方法，特别是基于思想链 (CoT) 的方法，极大地提高了大型语言模型 (LLM) 的文本到 SQL 的能力。然而，当应用于复杂的企业数据库时，由于上下文容量有限、模式链接不可靠以及数据库语义基础薄弱，此类方法很难保持连贯的推理。为了克服这些问题，我们引入了 DSR-SQL，一个 \textbf{D}ual-\textbf{S}tate \textbf{R} 推理框架，它将 Text-to-SQL 建模为自适应上下文状态和渐进生成状态之间的交互。第一个通过细化大型模式和选择相关结构构建一个紧凑的、语义上忠实的环境，而第二个将 SQL 合成形式化为反馈引导的状态转换，使模型能够自我纠正并与用户意图保持一致。在没有任何训练后或上下文示例的情况下，DSR-SQL 实现了具有竞争力的性能，在 Spider 2.0-Snow 上达到 35.28% 的执行精度，在 BIRD 开发集上达到 68.32% 的执行精度。我们的实现将在以下位置开源：此 https URL。

Title: A Systematic Study of Model Merging Techniques in Large Language Models

• Authors: Oğuz Kağan Hitit, Leander Girrbach, Zeynep Akata
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.21437
• Pdf URL: https://arxiv.org/pdf/2511.21437
• Copy Paste: [[2511.21437]] A Systematic Study of Model Merging Techniques in Large Language Models(https://arxiv.org/abs/2511.21437)
• Keywords: language model, llm
• Abstract: Model merging combines multiple fine-tuned checkpoints into a single model without additional training, offering an attractive approach to reusing models and efficiently improving performance. However, it remains unclear whether the advantages reported for smaller models and classifiers generalize to LLMs. We present a large-scale, systematic evaluation of six state-of-the-art merging methods, including recent subspace methods, across four open-weight LLMs, twelve fine-tuned checkpoints per base model, and sixteen standard LLM benchmarks. Evaluating through standardized benchmarks, we measure both the probability that a merged model outperforms the base model and relative gains over the best individual checkpoint. Our results show that the oldest and simplest method, Task Arithmetic, is the only approach that reliably yields performance gains on LLMs. Other interference-aware and subspace merging methods typically result in significant performance drops. Our findings indicate that current merging techniques do not directly transfer to modern LLMs. This motivates the design of LLM-specific merging algorithms and merging-aware fine-tuning methods. Code will be released upon acceptance of this paper.
• 摘要：模型合并将多个微调检查点合并到一个模型中，无需额外训练，为重用模型和有效提高性能提供了一种有吸引力的方法。然而，目前尚不清楚较小模型和分类器所报告的优势是否适用于法学硕士。我们对六种最先进的合并方法（包括最近的子空间方法）进行了大规模、系统的评估，涵盖四个开放权重 LLM、每个基本模型的 12 个微调检查点和 16 个标准 LLM 基准。通过标准化基准进行评估，我们测量合并模型优于基本模型的概率以及相对于最佳单个检查点的相对增益。我们的结果表明，最古老和最简单的方法，即任务算术，是唯一能够可靠地提高法学硕士成绩的方法。其他干扰感知和子空间合并方法通常会导致性能显着下降。我们的研究结果表明，当前的合并技术并没有直接转移到现代法学硕士。这激发了 LLM 特定合并算法和合并感知微调方法的设计。代码将在本文被接受后发布。

Title: Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation

• Authors: Lina Conti, Dennis Fucci, Marco Gaido, Matteo Negri, Guillaume Wisniewski, Luisa Bentivogli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.21517
• Pdf URL: https://arxiv.org/pdf/2511.21517
• Copy Paste: [[2511.21517]] Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation(https://arxiv.org/abs/2511.21517)
• Keywords: language model
• Abstract: Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bias concerns. For example, in speech translation (ST), when translating from languages with notional gender, such as English, into languages where gender-ambiguous terms referring to the speaker are assigned grammatical gender, the speaker's vocal characteristics may play a role in gender assignment. This risks misgendering speakers, whether through masculine defaults or vocal-based assumptions. Yet, how ST models make these decisions remains poorly understood. We investigate the mechanisms ST models use to assign gender to speaker-referring terms across three language pairs (en-es/fr/it), examining how training data patterns, internal language model (ILM) biases, and acoustic information interact. We find that models do not simply replicate term-specific gender associations from training data, but learn broader patterns of masculine prevalence. While the ILM exhibits strong masculine bias, models can override these preferences based on acoustic input. Using contrastive feature attribution on spectrograms, we reveal that the model with higher gender accuracy relies on a previously unknown mechanism: using first-person pronouns to link gendered terms back to the speaker, accessing gender information distributed across the frequency spectrum rather than concentrated in pitch.
• 摘要：与文本不同，语音通过音调等声音提示传达有关说话者的信息，例如性别。这引起了特定于模态的偏见问题。例如，在语音翻译 (ST) 中，当从具有概念性别的语言（例如英语）翻译为指代说话者的性别模糊术语被分配语法性别的语言时，说话者的声音特征可能在性别分配中发挥作用。无论是通过男性默认还是基于声音的假设，这都有可能导致说话者性别错误。然而，人们对 ST 模型如何做出这些决策仍然知之甚少。我们研究了 ST 模型用于将性别分配给三种语言对 (en-es/fr/it) 的说话人指称术语的机制，检查训练数据模式、内部语言模型 (ILM) 偏差和声学信息如何相互作用。我们发现模型不仅仅从训练数据中复制特定术语的性别关联，而且还了解更广泛的男性流行模式。虽然 ILM 表现出强烈的男性偏见，但模型可以根据声学输入覆盖这些偏好。使用声谱图上的对比特征归因，我们揭示了具有更高性别准确性的模型依赖于以前未知的机制：使用第一人称代词将性别术语链接回说话者，访问分布在频谱上而不是集中在音调上的性别信息。

Title: RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions

• Authors: Minjoon Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21568
• Pdf URL: https://arxiv.org/pdf/2511.21568
• Copy Paste: [[2511.21568]] RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions(https://arxiv.org/abs/2511.21568)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) often exhibit inconsistent behavior when answering paraphrased questions, suggesting a reliance on surface-level patterns rather than true semantic understanding. To address this limitation, we introduce RoParQ, a benchmark specifically constructed to evaluate cross-paraphrase consistency in closed-book multiple-choice QA. This benchmark is derived from standard datasets by generating paraphrases via proprietary models and selectively retaining examples that elicit inconsistent confidence from a judge model. We further propose XParaCon, a novel evaluation metric that quantifies a model's robustness by measuring the standard deviation of accuracies across question variants. Additionally, we implement a reasoning-based, paraphrase-aware Supervised Fine-Tuning (SFT) strategy designed to align models toward semantic invariance. Our experiments demonstrate that this targeted alignment significantly enhances robustness. Notably, fine-tuned lightweight models achieved consistency levels comparable to much larger pre-trained models. These results highlight the efficacy of our approach in mitigating superficial memorization and fostering more robust, reliable LLMs.
• 摘要：大型语言模型 (LLM) 在回答释义问题时经常表现出不一致的行为，这表明依赖于表面模式而不是真正的语义理解。为了解决这个限制，我们引入了 RoParQ，这是一个专门为评估闭卷多项选择 QA 中的交叉释义一致性而构建的基准。该基准源自标准数据集，通过专有模型生成释义，并有选择地保留引起法官模型置信度不一致的示例。我们进一步提出了 XParaCon，这是一种新颖的评估指标，通过测量问题变体的准确度的标准差来量化模型的稳健性。此外，我们还实施了一种基于推理、释义感知的监督微调（SFT）策略，旨在使模型保持语义不变性。我们的实验表明，这种有针对性的比对显着增强了鲁棒性。值得注意的是，经过微调的轻量级模型达到了与更大的预训练模型相当的一致性水平。这些结果凸显了我们的方法在减少肤浅记忆和培养更强大、更可靠的法学硕士方面的有效性。

Title: Auxiliary Metrics Help Decoding Skill Neurons in the Wild

• Authors: Yixiu Zhao, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21610
• Pdf URL: https://arxiv.org/pdf/2511.21610
• Copy Paste: [[2511.21610]] Auxiliary Metrics Help Decoding Skill Neurons in the Wild(https://arxiv.org/abs/2511.21610)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) exhibit remarkable capabilities across a wide range of tasks, yet their internal mechanisms remain largely opaque. In this paper, we introduce a simple, lightweight, and broadly applicable method with a focus on isolating neurons that encode specific skills. Building upon prior work that identified "skill neurons" via soft prompt training on classification tasks, our approach extends the analysis to complex scenarios involving multiple skills. We correlate neuron activations with auxiliary metrics -- such as external labels and the model's own confidence score -- thereby uncovering interpretable and task-specific behaviors without the need for manual token aggregation. We empirically validate our method on tasks spanning open-ended text generation and natural language inference, demonstrating its ability to detect neurons that not only drive known skills but also reveal previously unidentified shortcuts in arithmetic reasoning on BigBench.
• 摘要：大型语言模型（LLM）在广泛的任务中表现出卓越的能力，但其内部机制在很大程度上仍然不透明。在本文中，我们介绍了一种简单、轻量且广泛适用的方法，重点是隔离编码特定技能的神经元。基于之前通过分类任务的软提示训练识别“技能神经元”的工作，我们的方法将分析扩展到涉及多种技能的复杂场景。我们将神经元激活与辅助指标（例如外部标签和模型自身的置信度得分）相关联，从而发现可解释的和特定于任务的行为，而无需手动标记聚合。我们在涵盖开放式文本生成和自然语言推理的任务中实证验证了我们的方法，证明了其检测神经元的能力，这些神经元不仅驱动已知的技能，而且还揭示了 BigBench 上算术推理中先前未识别的捷径。

Title: Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining

• Authors: Dongyang Fan, Diba Hashemi, Sai Praneeth Karimireddy, Martin Jaggi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.21613
• Pdf URL: https://arxiv.org/pdf/2511.21613
• Copy Paste: [[2511.21613]] Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining(https://arxiv.org/abs/2511.21613)
• Keywords: language model, llm
• Abstract: Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal-URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.
• 摘要：最近，将元数据纳入大型语言模型 (LLM) 预训练已成为加速训练的一种有前途的方法。然而，之前的工作只强调了一个有用的信号 URL，而其他形式的元数据是否可以产生更大的好处仍然是一个悬而未决的问题。在这项研究中，我们研究了更广泛的元数据类型，并找到了其他类型的元数据，例如文档质量的细粒度指标，这些指标在前置时也可以加速预训练。我们发现有效元数据的一个共同特征：它们以更细的粒度编码信息。我们进一步引入元数据附加作为提高训练效率的一种手段，其中预测适当的元数据作为辅助任务可以帮助加快预训练速度。此外，经过掩蔽损失训练的可学习元令牌可以通过诱导质量感知的潜在结构来恢复部分加速。通过探索，我们分析潜在表征以了解元数据如何塑造学习。总之，这些结果为整合元数据提供了实用指南，以提高法学硕士预训练的效率和有效性。

Title: The author is dead, but what if they never lived? A reception experiment on Czech AI- and human-authored poetry

• Authors: Anna Marklová, Ondřej Vinš, Martina Vokáčová, Jiří Milička
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.21629
• Pdf URL: https://arxiv.org/pdf/2511.21629
• Copy Paste: [[2511.21629]] The author is dead, but what if they never lived? A reception experiment on Czech AI- and human-authored poetry(https://arxiv.org/abs/2511.21629)
• Keywords: language model
• Abstract: Large language models are increasingly capable of producing creative texts, yet most studies on AI-generated poetry focus on English -- a language that dominates training data. In this paper, we examine the perception of AI- and human-written Czech poetry. We ask if Czech native speakers are able to identify it and how they aesthetically judge it. Participants performed at chance level when guessing authorship (45.8\% correct on average), indicating that Czech AI-generated poems were largely indistinguishable from human-written ones. Aesthetic evaluations revealed a strong authorship bias: when participants believed a poem was AI-generated, they rated it as less favorably, even though AI poems were in fact rated equally or more favorably than human ones on average. The logistic regression model uncovered that the more the people liked a poem, the less probable was that they accurately assign the authorship. Familiarity with poetry or literary background had no effect on recognition accuracy. Our findings show that AI can convincingly produce poetry even in a morphologically complex, low-resource (with respect of the training data of AI models) Slavic language such as Czech. The results suggest that readers' beliefs about authorship and the aesthetic evaluation of the poem are interconnected.
• 摘要：大型语言模型越来越有能力生成创造性文本，但大多数关于人工智能生成诗歌的研究都集中在英语——一种在训练数据中占主导地位的语言。在本文中，我们研究了人工智能和人类撰写的捷克诗歌的感知。我们询问以捷克语为母语的人是否能够识别它以及他们如何审美地判断它。参与者在猜测作者身份时表现出随机水平（平均正确率为 45.8％），这表明捷克人工智能生成的诗歌在很大程度上与人类写的诗歌没有区别。审美评估揭示了强烈的作者偏见：当参与者相信一首诗是人工智能生成的时，他们对它的评价较差，尽管人工智能诗歌实际上与人类诗歌的平均评价相同或更有利。逻辑回归模型发现，人们越喜欢一首诗，他们准确分配作者身份的可能性就越小。对诗歌或文学背景的熟悉程度对识别准确性没有影响。我们的研究结果表明，即使是形态复杂、资源匮乏（就人工智能模型的训练数据而言）的斯拉夫语言（例如捷克语），人工智能也可以令人信服地创作诗歌。结果表明，读者对作者身份的信念与诗歌的审美评价是相互关联的。

Title: Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework

• Authors: Dong Wang, Yang Li, Ansong Ni, Ching-Feng Yeh, Youssef Emad, Xinjie Lei, Liam Robbins, Karthik Padthe, Hu Xu, Xian Li, Asli Celikyilmaz, Ramya Raghavendra, Lifei Huang, Carole-Jean Wu, Shang-Wen Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.21686
• Pdf URL: https://arxiv.org/pdf/2511.21686
• Copy Paste: [[2511.21686]] Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework(https://arxiv.org/abs/2511.21686)
• Keywords: language model, llm, agent
• Abstract: Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality.
• 摘要：合成数据对于训练大型语言模型变得越来越重要，特别是当真实数据稀缺、昂贵或隐私敏感时。许多此类生成任务需要协调的多智能体工作流程，其中专门的智能体协作生成更高质量、更多样化且结构更丰富的数据。然而，现有的多代理综合框架通常依赖于集中式编排器，从而产生可扩展性瓶颈，或者针对特定领域进行硬编码，从而限制了灵活性。我们提出 \textbf{Matrix}，一个分散的框架，它将控制流和数据流表示为通过分布式队列传递的序列化消息。这种点对点设计消除了中央协调器。每个任务都通过轻量级代理独立进行，而计算密集型操作（例如 LLM 推理或容器化环境）则由分布式服务处理。 Matrix 基于 Ray 构建，可扩展到数以万计的并发代理工作流程，并提供模块化、可配置的设计，可以轻松适应各种数据生成工作流程。我们在不同的综合场景中评估 Matrix，例如多智能体协作对话、基于网络的推理数据提取以及客户服务环境中的工具使用轨迹生成。在所有情况下，Matrix 在相同的硬件资源下实现了 $2$--$15\times 更高的数据生成吞吐量，且不影响输出质量。

Title: ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration

• Authors: Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2511.21689
• Pdf URL: https://arxiv.org/pdf/2511.21689
• Copy Paste: [[2511.21689]] ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration(https://arxiv.org/abs/2511.21689)
• Keywords: language model, gpt, agent
• Abstract: Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity's Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools can both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks. We introduce ToolOrchestra, a method for training small orchestrators that coordinate intelligent tools. ToolOrchestra explicitly uses reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1%, outperforming GPT-5 (35.1%) while being 2.5x more efficient. On tau2-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to unseen tools. These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems.
• 摘要：大型语言模型是强大的多面手，但解决诸如人类期末考试 (HLE) 之类的深刻而复杂的问题在概念上仍然具有挑战性，而且计算成本高昂。我们证明，管理其他模型和各种工具的小型协调器既可以推动智能的上限，又可以提高解决困难的代理任务的效率。我们介绍了 ToolOrchestra，这是一种训练协调智能工具的小型编排器的方法。 ToolOrchestra 明确使用强化学习，并具有结果、效率和用户偏好感知奖励。使用 ToolOrchestra，我们生产了 Orchestrator，这是一种 8B 模型，与以前的工具使用代理相比，它以更低的成本实现了更高的准确性，同时与用户对给定查询使用哪些工具的偏好保持一致。在 HLE 上，Orchestrator 的得分为 37.1%，优于 GPT-5 (35.1%)，同时效率提高了 2.5 倍。在 tau2-Bench 和 FRAMES 上，Orchestrator 大幅超越了 GPT-5，而成本仅为 GPT-5 的 30% 左右。广泛的分析表明，Orchestrator 在多个指标下实现了性能和成本之间的最佳权衡，并且可以稳健地推广到未见过的工具。这些结果表明，使用轻量级编排模型组合多种工具比现有方法更高效、更有效，为实用且可扩展的工具增强推理系统铺平了道路。

Title: Revisiting Generalization Across Difficulty Levels: It's Not So Easy

• Authors: Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.21692
• Pdf URL: https://arxiv.org/pdf/2511.21692
• Copy Paste: [[2511.21692]] Revisiting Generalization Across Difficulty Levels: It's Not So Easy(https://arxiv.org/abs/2511.21692)
• Keywords: language model, llm
• Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.
• 摘要：我们研究大型语言模型 (LLM) 在不同任务难度上的泛化能力，这是有效数据管理和评估的关键问题。关于使用更简单还是更难的数据进行训练是否会带来更好的结果，以及这些收益是否来自于更简单或更难的测试数据，现有的研究结果不一。我们通过对法学硕士跨模型、数据集和示例难度的细粒度组的泛化进行系统评估来解决这个问题。我们使用数千个不同法学硕士的输出和项目反应理论（IRT）（教育测试中公认的难度指标）对六个数据集中的示例进行排名。因此，与之前的工作不同，我们的难度评级完全取决于许多不同法学硕士的能力，不包括人类对难度的看法。通过更客观、更大规模、更细粒度的分析，我们发现跨难度泛化往往是有限的；无论是简单数据还是硬数据训练都无法在所有困难上实现一致的改进。这些结果表明，法学硕士的培训和评估数据中存在一系列困难的重要性，而在难度方面采取捷径是有风险的。