2026-03-16

Title: Task-Specific Knowledge Distillation via Intermediate Probes

• Authors: Ryan Brown, Chris Russell
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12270
• Pdf URL: https://arxiv.org/pdf/2603.12270
• Copy Paste: [[2603.12270]] Task-Specific Knowledge Distillation via Intermediate Probes(https://arxiv.org/abs/2603.12270)
• Keywords: language model, llm, prompt
• Abstract: Knowledge distillation from large language models (LLMs) assumes that the teacher's output distribution is a high-quality training signal. On reasoning tasks, this assumption is frequently violated. A model's intermediate representations may encode the correct answer, yet this information is lost or distorted through the vocabulary projection, where prompt formatting and answer-token choices creates brittle, noisy outputs. We introduce \method{}, a distillation framework that bypasses this bottleneck by training lightweight probes on frozen teacher hidden states and using the probe's predictions, rather than output logits, as supervision for student training. This simple change yields consistent improvements across four reasoning benchmarks (AQuA-RAT, ARC Easy/Challenge, and MMLU), with gains most pronounced under limited data. Probes trained on intermediate representations provide cleaner labels than the teacher's own outputs, effectively denoising the distillation signal. \method{} requires no architectural changes to student or teacher, is architecture-agnostic, and adds minimal compute since probe training is cheap and teacher representations can be cached. By exploiting internal representations, \method{} enables practitioners to extract more value from large teacher models without additional training data or architectural complexity.
• 摘要：大语言模型（LLM）的知识蒸馏假设教师的输出分布是高质量的训练信号。在推理任务中，这个假设经常被违反。模型的中间表示可能会编码正确的答案，但这些信息会通过词汇表投影丢失或扭曲，其中提示格式和答案标记选择会产生脆弱的、嘈杂的输出。我们引入了 \method{}，一种蒸馏框架，它通过在冻结的教师隐藏状态上训练轻量级探针并使用探针的预测（而不是输出 logits）作为学生训练的监督来绕过这个瓶颈。这个简单的改变在四个推理基准（AQuA-RAT、ARC Easy/Challenge 和 MMLU）上产生了一致的改进，在有限的数据下收益最为明显。在中间表示上训练的探针提供比教师自己的输出更清晰的标签，有效地对蒸馏信号进行去噪。 \method{} 不需要对学生或教师进行架构更改，与架构无关，并且由于探针训练成本低廉并且可以缓存教师表示，因此增加了最少的计算量。通过利用内部表示，\method{} 使从业者能够从大型教师模型中提取更多价值，而无需额外的训练数据或架构复杂性。

Title: Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models

• Authors: Boyu Qiao, Sean Guo, Xian Yang, Kun Li, Wei Zhou, Songlin Hu, Yunya Song
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2603.12271
• Pdf URL: https://arxiv.org/pdf/2603.12271
• Copy Paste: [[2603.12271]] Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models(https://arxiv.org/abs/2603.12271)
• Keywords: language model, llm, long context
• Abstract: LLMs are widely used in knowledge-intensive tasks where the same fact may be revised multiple times within context. Unlike prior work focusing on one-shot updates or single conflicts, multi-update scenarios contain multiple historically valid versions that compete at retrieval, yet remain underexplored. This challenge resembles the AB-AC interference paradigm in cognitive psychology: when the same cue A is successively associated with B and C, the old and new associations compete during retrieval, leading to bias. Inspired by this, we introduce a Dynamic Knowledge Instance (DKI) evaluation framework, modeling multi-updates of the same fact as a cue paired with a sequence of updated values, and assess models via endpoint probing of the earliest (initial) and latest (current) states. Across diverse LLMs, we observe that retrieval bias intensifies as updates increase, earliest-state accuracy stays high while latest-state accuracy drops substantially. Diagnostic analyses of attention, hidden-state similarity, and output logits further reveal that these signals become flatter and weakly discriminative on errors, providing little stable basis for identifying the latest update. Finally, cognitively inspired heuristic intervention strategies yield only modest gains and do not eliminate the bias. Our results reveal a persistent challenge in tracking and following knowledge updates in long contexts.
• 摘要：法学硕士广泛应用于知识密集型任务，其中同一事实可能会在上下文中多次修改。与之前关注一次性更新或单一冲突的工作不同，多重更新场景包含多个历史有效版本，这些版本在检索方面相互竞争，但仍未得到充分探索。这一挑战类似于认知心理学中的 AB-AC 干扰范式：当相同的线索 A 相继与 B 和 C 关联时，新旧关联在检索过程中竞争，导致偏差。受此启发，我们引入了动态知识实例（DKI）评估框架，将同一事实的多次更新建模为与一系列更新值配对的提示，并通过最早（初始）和最新（当前）状态的端点探测来评估模型。在不同的法学硕士中，我们观察到检索偏差随着更新的增加而加剧，最早状态的准确性保持较高水平，而最新状态的准确性则大幅下降。对注意力、隐藏状态相似性和输出逻辑的诊断分析进一步表明，这些信号变得更加平坦，并且对错误的辨别力很弱，为识别最新更新提供了很少的稳定基础。最后，认知启发的启发式干预策略只能产生适度的收益，并且不能消除偏见。我们的结果揭示了在长上下文中跟踪和跟踪知识更新的持续挑战。

Title: ActTail: Global Activation Sparsity in Large Language Models

• Authors: Wenwen Hou, Xinyuan Song, Shiwei Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2603.12272
• Pdf URL: https://arxiv.org/pdf/2603.12272
• Copy Paste: [[2603.12272]] ActTail: Global Activation Sparsity in Large Language Models(https://arxiv.org/abs/2603.12272)
• Keywords: language model, llm
• Abstract: Activation sparsity is a promising approach for accelerating large language model (LLM) inference by reducing computation and memory movement. However, existing activation sparsity methods typically apply uniform sparsity across projections, ignoring the heterogeneous statistical properties of Transformer weights and thereby amplifying performance degradation. In this paper, we propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory. Specifically, we capture this heterogeneity via the heavy-tail exponent computed from each projection's empirical spectral density (ESD), which is used as a quantitative indicator to assign projection-specific sparsity budgets. Importantly, we provide a theoretical analysis that establishes an explicit relationship between the activation sparsity ratio and the heavy-tail exponent under the HT-SR regime, offering principled guidance for sparsity allocation beyond heuristic design. Experiments on LLaMA and Mistral models show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation. At 80% sparsity, perplexity is reduced by 21.8% on LLaMA-2-7B, 40.1% on LLaMA-2-13B, and 9.4% on Mistral-7B.
• 摘要：激活稀疏性是一种通过减少计算和内存移动来加速大型语言模型 (LLM) 推理的有前景的方法。然而，现有的激活稀疏性方法通常在投影之间应用均匀的稀疏性，忽略 Transformer 权重的异构统计特性，从而放大性能下降。在本文中，我们提出了 ActTail，一种基于 TopK 幅度的激活稀疏方法，具有基于重尾自正则化（HT-SR）理论的全局激活稀疏分配。具体来说，我们通过根据每个投影的经验谱密度（ESD）计算的重尾指数来捕获这种异质性，该指数被用作分配特定于投影的稀疏预算的定量指标。重要的是，我们提供了一种理论分析，在 HT-SR 机制下建立了激活稀疏率和重尾指数之间的明确关系，为启发式设计之外的稀疏分配提供了原则指导。 LLaMA 和 Mistral 模型的实验表明，与均匀分配相比，我们的方法在高稀疏性下提高了困惑度和下游任务性能。稀疏度为 80% 时，LLaMA-2-7B 上的困惑度降低了 21.8%，LLaMA-2-13B 上的困惑度降低了 40.1%，Mistral-7B 上的困惑度降低了 9.4%。

Title: Aligning Language Models from User Interactions

• Authors: Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor, Idan Shenfeld, Giorgia Ramponi, Andreas Krause
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2603.12273
• Pdf URL: https://arxiv.org/pdf/2603.12273
• Copy Paste: [[2603.12273]] Aligning Language Models from User Interactions(https://arxiv.org/abs/2603.12273)
• Keywords: language model, chat
• Abstract: Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user's preferences. Importantly, language models are already able to make use of this information in context. After observing a user's follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user's follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model's behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation.
• 摘要：多轮用户交互是语言模型产生的最丰富的数据之一，但我们缺乏有效的方法来从中学习。虽然通常会被丢弃，但这些交互通常包含有用的信息：后续用户消息可能表明响应不正确、未能遵循指令或与用户的偏好不符。重要的是，语言模型已经能够在上下文中使用这些信息。在观察用户的后续行为后，同一模型通常能够修正其行为。我们利用这种能力提出了一种原则性的、可扩展的方法，通过自我蒸馏直接从用户交互中学习。通过根据用户的后续消息调整模型并将生成的令牌分布与原始策略进行比较，我们获得了更新策略的目标，该目标捕获模型行为的事后变化。然后，我们将这种事后分布提炼回当前政策。值得注意的是，我们表明，对 WildChat 中的真实用户对话进行训练可以改进标准对齐和指令遵循基准方面的语言模型，而不会降低其他功能。同样的机制可以实现个性化，允许模型通过交互不断适应个人用户，而无需明确的反馈。我们的结果表明，部署过程中自然产生的原始用户交互可以实现调整、个性化和持续适应。

Title: GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping

• Authors: Chahana Dahal, Ashutosh Balasubramaniam, Zuobin Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12275
• Pdf URL: https://arxiv.org/pdf/2603.12275
• Copy Paste: [[2603.12275]] GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping(https://arxiv.org/abs/2603.12275)
• Keywords: language model, llm
• Abstract: Unlearning knowledge is a pressing and challenging task in Large Language Models (LLMs) because of their unprecedented capability to memorize and digest training data at scale, raising more significant issues regarding safety, privacy, and intellectual property. However, existing works, including parameter editing, fine-tuning, and distillation-based methods, are all focused on flat sentence-level data but overlook the relational, multi-hop, and reasoned knowledge in naturally structured data. In response to this gap, this paper introduces Graph Oblivion and Node Erasure (GONE), a benchmark for evaluating knowledge unlearning over structured knowledge graph (KG) facts in LLMs. This KG-based benchmark enables the disentanglement of three effects of unlearning: direct fact removal, reasoning-based leakage, and catastrophic forgetting. In addition, Neighborhood-Expanded Distribution Shaping (NEDS), a novel unlearning framework, is designed to leverage graph connectivity and identify anchor correlated neighbors, enforcing a precise decision boundary between the forgotten fact and its semantic neighborhood. Evaluations on LLaMA-3-8B and Mistral-7B across multiple knowledge editing and unlearning methods showcase NEDS's superior performance (1.000 on unlearning efficacy and 0.839 on locality) on GONE and other benchmarks. Code is available at this https URL.
• 摘要：在大型语言模型（LLM）中，忘却知识是一项紧迫且具有挑战性的任务，因为它们具有前所未有的大规模记忆和消化训练数据的能力，从而引发了有关安全、隐私和知识产权的更重要问题。然而，现有的工作，包括参数编辑、微调和基于蒸馏的方法，都专注于平面句子级数据，而忽视了自然结构化数据中的关系、多跳和推理知识。针对这一差距，本文引入了图遗忘和节点擦除（GONE），这是一个评估法学硕士中结构化知识图（KG）事实知识遗忘的基准。这种基于 KG 的基准可以消除遗忘的三种影响：直接事实删除、基于推理的泄漏和灾难性遗忘。此外，邻域扩展分布塑造（NEDS）是一种新颖的遗忘框架，旨在利用图连接性并识别锚相关的邻居，在被遗忘的事实与其语义邻域之间强制执行精确的决策边界。对 LLaMA-3-8B 和 Mistral-7B 跨多种知识编辑和遗忘方法的评估展示了 NEDS 在 GONE 和其他基准上的卓越性能（遗忘效率为 1.000，局部性为 0.839）。代码可从此 https URL 获取。

Title: Prompt Injection as Role Confusion

• Authors: Charles Ye, Jasmine Cui, Dylan Hadfield-Menell
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2603.12277
• Pdf URL: https://arxiv.org/pdf/2603.12277
• Copy Paste: [[2603.12277]] Prompt Injection as Role Confusion(https://arxiv.org/abs/2603.12277)
• Keywords: language model, prompt, agent
• Abstract: Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models infer roles from how text is written, not where it comes from. We design novel role probes to capture how models internally identify "who is speaking." These reveal why prompt injection works: untrusted text that imitates a role inherits that role's authority. We test this insight by injecting spoofed reasoning into user prompts and tool outputs, achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration, across multiple open- and closed-weight models with near-zero baselines. Strikingly, the degree of internal role confusion strongly predicts attack success before generation begins. Our findings reveal a fundamental gap: security is defined at the interface but authority is assigned in latent space. More broadly, we introduce a unifying, mechanistic framework for prompt injection, demonstrating that diverse prompt-injection attacks exploit the same underlying role-confusion mechanism.
• 摘要：尽管进行了大量的安全培训，语言模型仍然容易受到即时注入攻击。我们将这种失败归因于角色混乱：模型根据文本的编写方式而不是文本的来源来推断角色。我们设计了新颖的角色探针来捕获模型如何在内部识别“谁在说话”。这些揭示了提示注入起作用的原因：模仿角色的不受信任的文本继承该角色的权限。我们通过将欺骗性推理注入用户提示和工具输出来测试这种洞察力，在基线接近于零的多个开放权重和封闭权重模型中，StrongREJECT 的平均成功率为 60%，代理渗漏的平均成功率为 61%。引人注目的是，内部角色混乱的程度强烈地预测了一代开始之前攻击的成功。我们的研究结果揭示了一个根本性的差距：安全性是在界面上定义的，但权限是在潜在空间中分配的。更广泛地说，我们引入了一个用于即时注入的统一的机械框架，证明了不同的即时注入攻击利用了相同的底层角色混淆机制。

Title: LLM-Augmented Therapy Normalization and Aspect-Based Sentiment Analysis for Treatment-Resistant Depression on Reddit

• Authors: Yuxin Zhu, Sahithi Lakamana, Masoud Rouhizadeh, Selen Bozkurt, Rachel Hershenberg, Abeed Sarker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12343
• Pdf URL: https://arxiv.org/pdf/2603.12343
• Copy Paste: [[2603.12343]] LLM-Augmented Therapy Normalization and Aspect-Based Sentiment Analysis for Treatment-Resistant Depression on Reddit(https://arxiv.org/abs/2603.12343)
• Keywords: language model, llm
• Abstract: Treatment-resistant depression (TRD) is a severe form of major depressive disorder in which patients do not achieve remission despite multiple adequate treatment trials. Evidence across pharmacologic options for TRD remains limited, and trials often do not fully capture patient-reported tolerability. Large-scale online peer-support narratives therefore offer a complementary lens on how patients describe and evaluate medications in real-world use. In this study, we curated a corpus of 5,059 Reddit posts explicitly referencing TRD from 3,480 subscribers across 28 mental health-related subreddits from 2010 to 2025. Of these, 3,839 posts mentioned at least one medication, yielding 23,399 mentions of 81 generic-name medications after lexicon-based normalization of brand names, misspellings, and colloquialisms. We developed an aspect-based sentiment classifier by fine-tuning DeBERTa-v3 on the SMM4H 2023 therapy-sentiment Twitter corpus with large language model based data augmentation, achieving a micro-F1 score of 0.800 on the shared-task test set. Applying this classifier to Reddit, we quantified sentiment toward individual medications across three categories: positive, neutral, and negative, and tracked patterns by drug, subscriber, subreddit, and year. Overall, 72.1% of medication mentions were neutral, 14.8% negative, and 13.1% positive. Conventional antidepressants, especially SSRIs and SNRIs, showed consistently higher negative than positive proportions, whereas ketamine and esketamine showed comparatively more favorable sentiment profiles. These findings show that normalized medication extraction combined with aspect-based sentiment analysis can help characterize patient-perceived treatment experiences in TRD-related Reddit discourse, complementing clinical evidence with large-scale patient-generated perspectives.
• 摘要：难治性抑郁症（TRD）是重度抑郁症的一种严重形式，尽管进行了多次充分的治疗试验，患者仍未获得缓解。 TRD 药物选择的证据仍然有限，而且试验通常不能完全反映患者报告的耐受性。因此，大规模在线同伴支持叙述为患者如何描述和评估现实世界中的药物使用提供了补充视角。在这项研究中，我们整理了 2010 年至 2025 年间 28 个心理健康相关 subreddit 中 3,480 名订阅者的 5,059 个 Reddit 帖子，其中明确引用了 TRD。其中，3,839 个帖子至少提到了一种药物，在基于词典对品牌名称、拼写错误和药物名称进行标准化后，产生了 23,399 次提及 81 种通用名称药物的次数。口语。我们通过基于大型语言模型的数据增强对 SMM4H 2023 治疗情感 Twitter 语料库上的 DeBERTa-v3 进行微调，开发了一种基于方面的情感分类器，在共享任务测试集上实现了 0.800 的 micro-F1 分数。将此分类器应用于 Reddit，我们量化了对三种类别的个体药物的情绪：积极、中立和消极，并按药物、订阅者、子 Reddit 和年份跟踪模式。总体而言，72.1% 的药物提及是中性的，14.8% 是负面的，13.1% 是正面的。传统抗抑郁药，尤其是 SSRI 和 SNRI 的负面情绪比例始终高于正面比例，而氯胺酮和艾氯胺酮则表现出相对更有利的情绪状况。这些发现表明，标准化药物提取与基于方面的情感分析相结合，可以帮助描述 TRD 相关 Reddit 话语中患者感知的治疗体验，用大规模患者产生的观点补充临床证据。

Title: TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

• Authors: Liang-Hsuan Tseng, Hung-yi Lee
• Subjects: cs.CL, cs.SD
• Abstract URL: https://arxiv.org/abs/2603.12350
• Pdf URL: https://arxiv.org/pdf/2603.12350
• Copy Paste: [[2603.12350]] TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling(https://arxiv.org/abs/2603.12350)
• Keywords: language model
• Abstract: Text-speech joint spoken language modeling (SLM) aims at natural and intelligent speech-based interactions, but developing such a system may suffer from modality mismatch: speech unit sequences are much longer than text tokens. Prior work reduces this gap with text-aligned tokenization and embedding (TASTE), producing speech tokens that align in lengths with their textual counterparts. However, the dependence on an external ASR system and the use of a non-causal decoder limits streaming use. To address this limitation, we propose TASTE-S, a streamable extension of TASTE suitable for real-time usage. TASTE-S integrates a CTC-based ASR module into the encoder for instant dual-modality encoding. We also redesign the unit decoder to enable on-the-fly decoding. With joint training, we show that TASTE-S matches TASTE's performance while significantly reducing latency. Further investigations reveal that TASTE-S remains robust to transcriptions and enables long-form encoding and decoding.
• 摘要：文本-​​语音联合口语建模（SLM）旨在基于自然且智能的语音交互，但开发这样的系统可能会遇到模态不匹配的问题：语音单元序列比文本标记长得多。之前的工作通过文本对齐标记化和嵌入 (TASTE) 缩小了这一差距，生成长度与其文本对应部分一致的语音标记。然而，对外部 ASR 系统的依赖和非因果解码器的使用限制了流媒体的使用。为了解决这个限制，我们提出了 TASTE-S，它是适合实时使用的 TASTE 的流式扩展。 TASTE-S 将基于 CTC 的 ASR 模块集成到编码器中，以实现即时双模态编码。我们还重新设计了单元解码器以实现即时解码。通过联合训练，我们表明 TASTE-S 与 TASTE 的性能相匹配，同时显着降低了延迟。进一步的研究表明，TASTE-S 对转录仍然具有鲁棒性，并且能够实现长格式的编码和解码。

Title: Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviors

• Authors: Pengcheng Wen, Yanxu Zhu, Jiapeng Sun, Han Zhu, Yujin Zhou, Chi-Min Chan, Sirui Han, Yike Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12397
• Pdf URL: https://arxiv.org/pdf/2603.12397
• Copy Paste: [[2603.12397]] Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviors(https://arxiv.org/abs/2603.12397)
• Keywords: llm, chain-of-thought
• Abstract: Chain-of-Thought (CoT) is often viewed as a window into LLM decision-making, yet recent work suggests it may function merely as post-hoc rationalization. This raises a critical alignment question: Does the reasoning trace causally shape model generalization independent of the final answer? To isolate reasoning's causal effect, we design a controlled experiment holding final harmful answers constant while varying reasoning paths. We construct datasets with \textit{Evil} reasoning embracing malice, \textit{Misleading} reasoning rationalizing harm, and \textit{Submissive} reasoning yielding to pressure. We train models (0.6B--14B parameters) under multiple paradigms, including question-thinking-answer (QTA), question-thinking (QT), and thinking-only (T-only), and evaluate them in both think and no-think modes. We find that: (1) CoT training could amplify harmful generalization more than standard fine-tuning; (2) distinct reasoning types induce distinct behavioral patterns aligned with their semantics, despite identical final answers; (3) training on reasoning without answer supervision (QT or T-only) is sufficient to alter behavior, proving reasoning carries an independent signal; and (4) these effects persist even when generating answers without reasoning, indicating deep internalization. Our findings demonstrate that reasoning content is causally potent, challenging alignment strategies that supervise only outputs.
• 摘要：思想链（CoT）通常被视为了解 LLM 决策的窗口，但最近的研究表明，它可能仅仅起到事后合理化的作用。这就提出了一个关键的对齐问题：推理轨迹是否会独立于最终答案而塑造模型泛化？为了隔离推理的因果效应，我们设计了一个受控实验，在改变推理路径的同时保持最终有害答案不变。我们用包含恶意的 \textit{Evil} 推理、将伤害合理化的 \textit{Misleading} 推理和屈服于压力的 \textit{Submissive} 推理构建数据集。我们在多种范式下训练模型（0.6B--14B 参数），包括问题思维回答（QTA）、问题思维（QT）和仅思考（T-only），并在思考和不思考模式下对其进行评估。我们发现：（1）CoT 训练比标准微调更能放大有害泛化； （2）尽管最终答案相同，但不同的推理类型会导致与其语义一致的不同行为模式； (3) 没有答案监督（仅 QT 或 T）的推理训练足以改变行为，证明推理携带独立信号； （4）即使在没有推理的情况下生成答案，这些影响仍然存在，表明深度内化。我们的研究结果表明，推理内容具有因果关系，对仅监督输出的对齐策略具有挑战性。

Title: Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis

• Authors: Abdullah Al Mofael, Lisa M. Kuhn, Ghassan Alkadi, Kuo-Pao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12423
• Pdf URL: https://arxiv.org/pdf/2603.12423
• Copy Paste: [[2603.12423]] Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis(https://arxiv.org/abs/2603.12423)
• Keywords: language model, gpt
• Abstract: Negation remains a persistent challenge for modern language models, often causing reversed meanings or factual errors. In this work, we conduct a causal analysis of how GPT-2 Small internally processes such linguistic transformations. We examine its hidden representations at both the layer and head level. Our analysis is based on a self-curated 12,000-pair dataset of matched affirmative and negated sentences, covering multiple linguistic templates and forms of negation. To quantify this behavior, we define a metric, the Negation Effect Score (NES), which measures the model's sensitivity in distinguishing between affirmative statements and their negations. We carried out two key interventions to probe causal structure. In activation patching, internal activations from affirmative sentences were inserted into their negated counterparts to see how meaning shifted. In ablation, specific attention heads were temporarily disabled to observe how logical polarity changed. Together, these steps revealed how negation signals move and evolve through GPT-2's layers. Our findings indicate that this capability is not widespread; instead, it is highly concentrated within a limited number of mid-layer attention heads, primarily within layers 4 to 6. Ablating these specific components directly disrupts the model's negation sensitivity: on our in-domain, ablation increased NES (indicating weaker negation sensitivity), and re-introducing cached affirmative activations (rescue) increased NES further, confirming that these heads carry affirmative signal rather than restoring baseline behavior. On xNot360, ablation slightly decreased NES and rescue restored performance above baseline. This pattern demonstrates that these causal patterns are consistent across various negation forms and remain detectable on the external xNot360 benchmark, though with smaller magnitude.
• 摘要：否定仍然是现代语言模型的一个持续挑战，常常导致含义颠倒或事实错误。在这项工作中，我们对 GPT-2 Small 内部如何处理此类语言转换进行了因果分析。我们在层和头级别检查其隐藏表示。我们的分析基于自行整理的 12,000 对匹配的肯定句和否定句数据集，涵盖多种语言模板和否定形式。为了量化这种行为，我们定义了一个指标，即否定效应分数（NES），它衡量模型在区分肯定陈述和否定陈述方面的敏感性。我们进行了两项关键干预措施来探究因果结构。在激活修补中，肯定句的内部激活被插入到其否定句中，以观察含义如何变化。在消融过程中，特定的注意力头暂时被禁用，以观察逻辑极性如何变化。这些步骤共同揭示了否定信号如何通过 GPT-2 的层移动和演化。我们的研究结果表明，这种能力并不普遍。相反，它高度集中在有限数量的中间层注意力头内，主要在第 4 层到第 6 层内。消除这些特定组件直接破坏了模型的否定敏感性：在我们的域内，消除增加了 NES（表明否定敏感性较弱），并且重新引入缓存的肯定激活（救援）进一步增加了 NES，确认这些头携带肯定信号而不是恢复基线行为。在 xNot360 上，消融稍微降低了 NES，而救援将性能恢复到基线以上。该模式表明，这些因果模式在各种否定形式中是一致的，并且在外部 xNot360 基准测试中仍然可检测到，尽管幅度较小。

Title: CSE-UOI at SemEval-2026 Task 6: A Two-Stage Heterogeneous Ensemble with Deliberative Complexity Gating for Political Evasion Detection

• Authors: Christos Tzouvaras, Konstantinos Skianis, Athanasios Voulodimos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12453
• Pdf URL: https://arxiv.org/pdf/2603.12453
• Copy Paste: [[2603.12453]] CSE-UOI at SemEval-2026 Task 6: A Two-Stage Heterogeneous Ensemble with Deliberative Complexity Gating for Political Evasion Detection(https://arxiv.org/abs/2603.12453)
• Keywords: language model, llm, agent
• Abstract: This paper describes our system for SemEval-2026 Task 6, which classifies clarity of responses in political interviews into three categories: Clear Reply, Ambivalent, and Clear Non-Reply. We propose a heterogeneous dual large language model (LLM) ensemble via self-consistency (SC) and weighted voting, and a novel post-hoc correction mechanism, Deliberative Complexity Gating (DCG). This mechanism uses cross-model behavioral signals and exploits the finding that an LLM response-length proxy correlates strongly with sample ambiguity. To further examine mechanisms for improving ambiguity detection, we evaluated multi-agent debate as an alternative strategy for increasing deliberative capacity. Unlike DCG, which adaptively gates reasoning using cross-model behavioral signals, debate increases agent count without increasing model diversity. Our solution achieved a Macro-F1 score of 0.85 on the evaluation set, securing 3rd place.
• 摘要：本文描述了我们的 SemEval-2026 任务 6 系统，该系统将政治访谈中回答的清晰度分为三类：明确答复、矛盾和明确不答复。我们提出了一种通过自洽（SC）和加权投票的异构双大语言模型（LLM）集成，以及一种新颖的事后纠正机制——慎思复杂性门控（DCG）。该机制使用跨模型行为信号，并利用了 LLM 响应长度代理与样本模糊性密切相关的发现。为了进一步研究改进歧义检测的机制，我们评估了多主体辩论作为提高审议能力的替代策略。与使用跨模型行为信号自适应地控制推理的 DCG 不同，辩论会增加代理数量，但不会增加模型多样性。我们的解决方案在评估集上获得了 0.85 的 Macro-F1 分数，获得第三名。

Title: Shattering the Shortcut: A Topology-Regularized Benchmark for Multi-hop Medical Reasoning in LLMs

• Authors: Xing Zi, Xinying Zhou, Jinghao Xiao, Catarina Moreira, Mukesh Prasad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12458
• Pdf URL: https://arxiv.org/pdf/2603.12458
• Copy Paste: [[2603.12458]] Shattering the Shortcut: A Topology-Regularized Benchmark for Multi-hop Medical Reasoning in LLMs(https://arxiv.org/abs/2603.12458)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: While Large Language Models (LLMs) achieve expert-level performance on standard medical benchmarks through single-hop factual recall, they severely struggle with the complex, multi-hop diagnostic reasoning required in real-world clinical settings. A primary obstacle is "shortcut learning", where models exploit highly connected, generic hub nodes (e.g., "inflammation") in knowledge graphs to bypass authentic micro-pathological cascades. To address this, we introduce ShatterMed-QA, a bilingual benchmark of 10,558 multi-hop clinical questions designed to rigorously evaluate deep diagnostic reasoning. Our framework constructs a topology-regularized medical Knowledge Graph using a novel $k$-Shattering algorithm, which physically prunes generic hubs to explicitly sever logical shortcuts. We synthesize the evaluation vignettes by applying implicit bridge entity masking and topology-driven hard negative sampling, forcing models to navigate biologically plausible distractors without relying on superficial elimination. Comprehensive evaluations of 21 LLMs reveal massive performance degradation on our multi-hop tasks, particularly among domain-specific models. Crucially, restoring the masked evidence via Retrieval-Augmented Generation (RAG) triggers near-universal performance recovery, validating ShatterMed-QA's structural fidelity and proving its efficacy in diagnosing the fundamental reasoning deficits of current medical AI. Explore the dataset, interactive examples, and full leaderboards at our project website: this https URL
• 摘要：虽然大型语言模型 (LLM) 通过单跳事实回忆在标准医学基准上实现了专家级性能，但它们在现实临床环境中所需的复杂、多跳诊断推理方面遇到了严重的困难。主要障碍是“捷径学习”，其中模型利用知识图中高度连接的通用中心节点（例如“炎症”）来绕过真实的微病理级联。为了解决这个问题，我们引入了 ShatterMed-QA，这是一个包含 10,558 个多跳临床问题的双语基准，旨在严格评估深度诊断推理。我们的框架使用一种新颖的 $k$-Shattering 算法构建了一个拓扑正则化的医学知识图，该算法物理上修剪通用中心以显式切断逻辑捷径。我们通过应用隐式桥实体屏蔽和拓扑驱动的硬负采样来合成评估片段，迫使模型在不依赖表面消除的情况下导航生物学上合理的干扰因素。对 21 个法学硕士的综合评估揭示了我们的多跳任务的性能大幅下降，特别是在特定领域的模型中。至关重要的是，通过检索增强生成（RAG）恢复被掩盖的证据会触发近乎普遍的性能恢复，验证 ShatterMed-QA 的结构保真度并证明其在诊断当前医疗人工智能的基本推理缺陷方面的有效性。在我们的项目网站上探索数据集、交互式示例和完整排行榜：此 https URL

Title: Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback

• Authors: Mei Tan, Lena Phalen, Dorottya Demszky
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2603.12471
• Pdf URL: https://arxiv.org/pdf/2603.12471
• Copy Paste: [[2603.12471]] Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback(https://arxiv.org/abs/2603.12471)
• Keywords: gpt, llm, prompt
• Abstract: Effective personalized feedback is critical to students' literacy development. Though LLM-powered tools now promise to automate such feedback at scale, LLMs are not language-neutral: they privilege standard academic English and reproduce social stereotypes, raising concerns about how "personalization" shapes the feedback students receive. We examine how four widely used LLMs (GPT-4o, GPT-3.5-turbo, Llama-3.3 70B, Llama-3.1 8B) adapt written feedback in response to student attributes. Using 600 eighth-grade persuasive essays from the PERSUADE dataset, we generated feedback under prompt conditions embedding gender, race/ethnicity, learning needs, achievement, and motivation. We analyze lexical shifts across model outputs by adapting the Marked Words framework. Our results reveal systematic, stereotype-aligned shifts in feedback conditioned on presumed student attributes--even when essay content was identical. Feedback for students marked by race, language, or disability often exhibited positive feedback bias and feedback withholding bias--overuse of praise, less substantive critique, and assumptions of limited ability. Across attributes, models tailored not only what content was emphasized but also how writing was judged and how students were addressed. We term these instructional orientations Marked Pedagogies and highlight the need for transparency and accountability in automated feedback tools.
• 摘要：有效的个性化反馈对于学生的读写能力发展至关重要。尽管法学硕士支持的工具现在有望大规模自动化此类反馈，但法学硕士并不是语言中立的：它们优先考虑标准学术英语并复制社会刻板印象，引发人们对“个性化”如何影响学生收到的反馈的担忧。我们研究了四种广泛使用的法学硕士（GPT-4o、GPT-3.5-turbo、Llama-3.3 70B、Llama-3.1 8B）如何根据学生的属性调整书面反馈。我们使用 PERSUADE 数据集中的 600 篇八年级说服性文章，在即时条件下生成反馈，其中包含性别、种族/民族、学习需求、成就和动机。我们通过采用标记词框架来分析模型输出之间的词汇变化。我们的结果揭示了以假定的学生属性为条件的反馈的系统性、与刻板印象一致的转变——即使论文内容相同。对以种族、语言或残疾为标志的学生的反馈经常表现出积极反馈偏差和反馈保留偏差——过度使用表扬、缺乏实质性的批评以及对能力有限的假设。跨属性，模型不仅可以定制强调的内容，还可以定制如何评判写作以及如何对待学生。我们将这些教学方向称为“标记教学法”，并强调自动反馈工具的透明度和问责制的必要性。

Title: LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

• Authors: Himel Ghosh, Nick Elias Werner
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2603.12522
• Pdf URL: https://arxiv.org/pdf/2603.12522
• Copy Paste: [[2603.12522]] LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation(https://arxiv.org/abs/2603.12522)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) are deployed widely, detecting and understanding bias in their outputs is critical. We present LLM BiasScope, a web application for side-by-side comparison of LLM outputs with real-time bias analysis. The system supports multiple providers (Google Gemini, DeepSeek, MiniMax, Mistral, Meituan, Meta Llama) and enables researchers and practitioners to compare models on the same prompts while analyzing bias patterns. LLM BiasScope uses a two-stage bias detection pipeline: sentence-level bias detection followed by bias type classification for biased sentences. The analysis runs automatically on both user prompts and model responses, providing statistics, visualizations, and detailed breakdowns of bias types. The interface displays two models side-by-side with synchronized streaming responses, per-model bias summaries, and a comparison view highlighting differences in bias distributions. The system is built on this http URL with React, integrates Hugging Face inference endpoints for bias detection, and uses the Vercel AI SDK for multi-provider LLM access. Features include real-time streaming, export to JSON/PDF, and interactive visualizations (bar charts, radar charts) for bias analysis. LLM BiasScope is available as an open-source web application, providing a practical tool for bias evaluation and comparative analysis of LLM behaviour.
• 摘要：随着大型语言模型 (LLM) 的广泛部署，检测和理解其输出中的偏差至关重要。我们推出了 LLM BiasScope，这是一个网络应用程序，用于将 LLM 输出与实时偏差分析进行并排比较。该系统支持多个提供商（Google Gemini、DeepSeek、MiniMax、Mistral、Meituan、Meta Llama），使研究人员和从业者能够在分析偏差模式的同时根据相同的提示比较模型。 LLM BiasScope 使用两阶段偏差检测流程：句子级偏差检测，然后对有偏差的句子进行偏差类型分类。该分析会根据用户提示和模型响应自动运行，提供统计数据、可视化和偏差类型的详细细分。该界面并排显示两个模型，并具有同步流响应、每个模型的偏差摘要以及突出显示偏差分布差异的比较视图。该系统基于此 http URL 和 React 构建，集成了 Hugging Face 推理端点以进行偏差检测，并使用 Vercel AI SDK 进行多提供商 LLM 访问。功能包括实时流式传输、导出为 JSON/PDF 以及用于偏差分析的交互式可视化（条形图、雷达图）。 LLM BiasScope 作为开源 Web 应用程序提供，为 LLM 行为的偏见评估和比较分析提供了实用工具。

Title: AgentDrift: Unsafe Recommendation Drift Under Tool Corruption Hidden by Ranking Metrics in LLM Agents

• Authors: Zekun Wu, Adriano Koshiyama, Sahan Bulathwela, Maria Perez-Ortiz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12564
• Pdf URL: https://arxiv.org/pdf/2603.12564
• Copy Paste: [[2603.12564]] AgentDrift: Unsafe Recommendation Drift Under Tool Corruption Hidden by Ranking Metrics in LLM Agents(https://arxiv.org/abs/2603.12564)
• Keywords: llm, agent
• Abstract: Tool-augmented LLM agents increasingly serve as multi-turn advisors in high-stakes domains, yet their evaluation relies on ranking-quality metrics that measure what is recommended but not whether it is safe for the user. We introduce a paired-trajectory protocol that replays real financial dialogues under clean and contaminated tool-output conditions across seven LLMs (7B to frontier) and decomposes divergence into information-channel and memory-channel mechanisms. Across the seven models tested, we consistently observe the evaluation-blindness pattern: recommendation quality is largely preserved under contamination (utility preservation ratio approximately 1.0) while risk-inappropriate products appear in 65-93% of turns, a systematic safety failure poorly reflected by standard NDCG. Safety violations are predominantly information-channel-driven, emerge at the first contaminated turn, and persist without self-correction over 23-step trajectories; no agent across 1,563 contaminated turns explicitly questions tool-data reliability. Even narrative-only corruption (biased headlines, no numerical manipulation) induces significant drift while completely evading consistency monitors. A safety-penalized NDCG variant (sNDCG) reduces preservation ratios to 0.51-0.74, indicating that much of the evaluation gap becomes visible once safety is explicitly measured. These results motivate considering trajectory-level safety monitoring, beyond single-turn quality, for deployed multi-turn agents in high-stakes settings.
• 摘要：工具增强的 LLM 代理越来越多地充当高风险领域的多轮顾问，但他们的评估依赖于排名质量指标，这些指标衡量推荐内容，而不是衡量推荐内容对用户是否安全。我们引入了一种配对轨迹协议，可以在七个法学硕士（7B 到前沿）的干净和受污染的工具输出条件下重播真实的金融对话，并将分歧分解为信息通道和内存通道机制。在测试的七个模型中，我们一致观察到评估盲目模式：推荐质量很大程度上在污染下得以保留（效用保留率约为 1.0），而风险不适当的产品出现在 65-93% 的轮次中，这是标准 NDCG 未能很好反映的系统性安全故障。安全违规行为主要是信息渠道驱动的，在第一个受污染的转弯处出现，并且在 23 步轨迹上没有自我纠正的情况下持续存在；在 1,563 个受污染转弯中，没有任何代理人明确质疑工具数据的可靠性。即使只是叙述性的腐败（有偏见的头条新闻，没有数字操纵）也会引起显着的偏差，同时完全逃避一致性监视器。安全惩罚 NDCG 变体 (sNDCG) 将保留率降低至 0.51-0.74，这表明一旦明确测量安全性，大部分评估差距就会变得可见。这些结果促使我们考虑对高风险环境中部署的多转弯智能体进行轨迹级安全监控，超越单转弯质量。

Title: Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation

• Authors: Jia-Chen Zhang, Zhen-Wei Yan, Yu-Jie Xiong, Chun-Ming Xia
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2603.12577
• Pdf URL: https://arxiv.org/pdf/2603.12577
• Copy Paste: [[2603.12577]] Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation(https://arxiv.org/abs/2603.12577)
• Keywords: llm
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for deploying LLMs in multi-task scenarios due to its extreme parameter efficiency. While Mixture-of-Experts (MoE) based LoRA variants have achieved promising results by dynamically routing tokens to different low-rank experts, they largely overlook the hierarchical nature of task complexity. Existing methods typically employ experts with uniform architectures, limiting their ability to capture diverse feature granularities required by distinct tasks--where some tasks demand high-level semantic abstraction while others require fine-grained syntactic manipulation. To bridge this gap, we propose Expert Pyramid Tuning (EPT), a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT. Unlike standard LoRA, EPT decomposes task adaptation into two stages: (1) A shared meta-knowledge Subspace that encodes universal linguistic patterns in low dimensions; (2) A Pyramid Projection Mechanism that utilizes learnable up-projection operators to reconstruct high-dimensional features at varying scales. A task-aware router then dynamically selects the optimal combination of these multi-scale features. Extensive experiments across multiple multi-task benchmarks demonstrate that EPT significantly outperforms SOTA MoE-LoRA variants. Crucially, thanks to the re-parameterization capability of our design, EPT achieves this performance improvement while simultaneously reducing the number of training parameters.
• 摘要：参数高效微调（PEFT）由于其极高的参数效率，已成为在多任务场景中部署 LLM 的主导范例。虽然基于专家混合 (MoE) 的 LoRA 变体通过动态地将令牌路由到不同的低级别专家而取得了有希望的结果，但它们在很大程度上忽视了任务复杂性的层次性质。现有的方法通常采用具有统一架构的专家，限制了他们捕获不同任务所需的不同特征粒度的能力——其中一些任务需要高级语义抽象，而另一些任务则需要细粒度的句法操作。为了弥补这一差距，我们提出了专家金字塔调整（EPT），这是一种新颖的架构，它将计算机视觉中的多尺度特征金字塔概念集成到 PEFT 领域。与标准 LoRA 不同，EPT 将任务适应分解为两个阶段：（1）共享元知识子空间，编码低维度的通用语言模式； (2)金字塔投影机制，利用可学习的上投影算子来重建不同尺度的高维特征。然后，任务感知路由器动态选择这些多尺度特征的最佳组合。跨多个多任务基准的大量实验表明，EPT 的性能显着优于 SOTA MoE-LoRA 变体。至关重要的是，得益于我们设计的重新参数化功能，EPT 实现了性能改进，同时减少了训练参数的数量。

Title: Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System

• Authors: Necva Bölücü, Jessica Irons, Changhyun Lee, Brian Jin, Maciej Rybinski, Huichen Yang, Andreas Duenser, Stephen Wan
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2603.12638
• Pdf URL: https://arxiv.org/pdf/2603.12638
• Copy Paste: [[2603.12638]] Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System(https://arxiv.org/abs/2603.12638)
• Keywords: llm
• Abstract: The rapid growth of scientific literature has made manual extraction of structured knowledge increasingly impractical. To address this challenge, we introduce SCILIRE, a system for creating datasets from scientific literature. SCILIRE has been designed around Human-AI teaming principles centred on workflows for verifying and curating data. It facilitates an iterative workflow in which researchers can review and correct AI outputs. Furthermore, this interaction is used as a feedback signal to improve future LLM-based inference. We evaluate our design using a combination of intrinsic benchmarking outcomes together with real-world case studies across multiple domains. The results demonstrate that SCILIRE improves extraction fidelity and facilitates efficient dataset creation.
• 摘要：科学文献的快速增长使得手动提取结构化知识变得越来越不切实际。为了应对这一挑战，我们引入了 SCILIRE，一个用于从科学文献创建数据集的系统。 SCILIRE 围绕人机协作原则进行设计，以验证和整理数据的工作流程为中心。它促进了迭代工作流程，研究人员可以在其中审查和纠正人工智能输出。此外，这种交互被用作反馈信号，以改进未来基于 LLM 的推理。我们结合内在基准测试结果和跨多个领域的真实案例研究来评估我们的设计。结果表明 SCILIRE 提高了提取保真度并促进了高效的数据集创建。

Title: 98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

• Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12646
• Pdf URL: https://arxiv.org/pdf/2603.12646
• Copy Paste: [[2603.12646]] 98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router(https://arxiv.org/abs/2603.12646)
• Keywords: llm, prompt
• Abstract: System-level routers that intercept LLM requests for safety classification, domain routing, and PII detection must be both fast and operationally lightweight: they should add minimal latency to every request, yet not require a dedicated GPU -- an expensive resource better used for LLM inference itself. When the router co-locates on the same GPU as vLLM serving instances, standard attention's $O(n^2)$ memory makes long-context classification (8K--32K tokens) impossible: at 8K tokens, three concurrent classifiers need ${\sim}$4.5\,GB for attention masks alone, far exceeding the memory left by vLLM. We present three staged optimizations for the vLLM Semantic Router, benchmarked on AMD Instinct MI300X, that solve both the latency and the memory problem. \emph{Stage~1}: a custom CK Flash Attention operator for ONNX Runtime on ROCm reduces attention memory from $O(n^2)$ to $O(n)$ and end-to-end (E2E) latency from 4{,}918\,ms to 127\,ms (\textbf{38.7$\times$}), enabling 8K--32K tokens where SDPA OOMs. \emph{Stage~2}: classical NLP prompt compression (TextRank, position weighting, TF-IDF, and novelty scoring) reduces all inputs to ${\sim}$512 tokens without neural inference, capping both latency and GPU memory at a constant regardless of original prompt length (E2E 127$\to$62\,ms, \textbf{2.0$\times$}). \emph{Stage~3}: near-streaming body processing with adaptive chunking and zero-copy JSON eliminates serialization overhead (E2E 62$\to$50\,ms, \textbf{1.2$\times$}). Cumulatively: \textbf{98$\times$} improvement (4{,}918\,ms to 50\,ms), 16K-token routing in 108\,ms, and a total router GPU footprint under 800\,MB -- small enough to share a GPU with LLM serving and removing the need for a dedicated accelerator. Stage~1 targets AMD ROCm (NVIDIA GPUs already have FlashAttention via cuDNN); Stages~2 and~3 are hardware-agnostic.
• 摘要：拦截安全分类、域路由和 PII 检测的 LLM 请求的系统级路由器必须既快速又轻量级：它们应该为每个请求添加最小的延迟，但不需要专用 GPU——这是一种更适合 LLM 推理本身的昂贵资源。当路由器与 vLLM 服务实例位于同一 GPU 上时，标准注意力的 $O(n^2)$ 内存使得长上下文分类（8K--32K 令牌）变得不可能：在 8K 令牌时，三个并发分类器仅用于注意力掩码就需要 ${\sim}$4.5\,GB，远​​远超过 vLLM 留下的内存。我们针对 vLLM 语义路由器提出了三阶段优化，以 AMD Instinct MI300X 为基准，解决了延迟和内存问题。 \emph{Stage~1}：ROCm 上 ONNX 运行时的自定义 CK Flash Attention 运算符将注意力内存从 $O(n^2)$ 减少到 $O(n)$，并将端到端 (E2E) 延迟从 4{,}918\,ms 减少到 127\,ms (\textbf{38.7$\times$})，从而在 SDPA OOM 时启用 8K--32K 令牌。 \emph{Stage~2}：经典 NLP 提示压缩（TextRank、位置加权、TF-IDF 和新颖性评分）将所有输入减少到 ${\sim}$512 标记，无需神经推理，将延迟和 GPU 内存限制在一个常数，无论原始提示长度如何（E2E 127$\to$62\,ms, \textbf{2.0$\times$}）。 \emph{Stage~3}：使用自适应分块和零拷贝 JSON 的近流正文处理消除了序列化开销（E2E 62$\to$50\,ms，\textbf{1.2$\times$}）。累计：\textbf{98$\times$} 改进（4{,}918\,ms 到 50\,ms），16K 令牌路由只需 108\,ms，路由器 GPU 总占用空间低于 800\,MB——小到足以与 LLM 服务共享 GPU，并且无需专用加速器。 Stage~1 的目标是 AMD ROCm（NVIDIA GPU 已经通过 cuDNN 提供了 FlashAttention）；第~2 和~3 阶段与硬件无关。

Title: Continual Learning in Large Language Models: Methods, Challenges, and Opportunities

• Authors: Hongyang Chen, Zhongwu Sun, Hongfei Ye, Kunchi Li, Xuemin Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12658
• Pdf URL: https://arxiv.org/pdf/2603.12658
• Copy Paste: [[2603.12658]] Continual Learning in Large Language Models: Methods, Challenges, and Opportunities(https://arxiv.org/abs/2603.12658)
• Keywords: language model, llm
• Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting-a critical limitation of the static pre-training paradigm inherent to modern LLMs. This survey presents a comprehensive overview of CL methodologies tailored for LLMs, structured around three core training stages: continual pre-training, continual fine-tuning, and continual this http URL the canonical taxonomy of rehearsal-, regularization-, and architecture-based methods, we further subdivide each category by its distinct forgetting mitigation mechanisms and conduct a rigorous comparative analysis of the adaptability and critical improvements of traditional CL methods for LLMs. In doing so, we explicitly highlight core distinctions between LLM CL and traditional machine learning, particularly with respect to scale, parameter efficiency, and emergent capabilities. Our analysis covers essential evaluation metrics, including forgetting rates and knowledge transfer efficiency, along with emerging benchmarks for assessing CL performance. This survey reveals that while current methods demonstrate promising results in specific domains, fundamental challenges persist in achieving seamless knowledge integration across diverse tasks and temporal scales. This systematic review contributes to the growing body of knowledge on LLM adaptation, providing researchers and practitioners with a structured framework for understanding current achievements and future opportunities in lifelong learning for language models.
• 摘要：持续学习 (CL) 已成为一种关键范式，使大型语言模型 (LLM) 能够动态适应不断发展的知识和顺序任务，同时减轻灾难性遗忘——这是现代 LLM 固有的静态预训练范式的一个关键限制。这项调查全面概述了为法学硕士量身定制的 CL 方法，围绕三个核心培训阶段构建：持续预训练、持续微调和持续此 http URL（排练、正则化和基于架构的方法的规范分类法），我们通过其独特的遗忘缓解机制进一步细分每个类别，并对法学硕士传统 CL 方法的适应性和关键改进进行严格的比较分析。在此过程中，我们明确强调了 LLM CL 和传统机器学习之间的核心区别，特别是在规模、参数效率和涌现能力方面。我们的分析涵盖了基本的评估指标，包括遗忘率和知识转移效率，以及评估 CL 绩效的新兴基准。这项调查表明，虽然当前的方法在特定领域显示出有希望的结果，但在实现跨不同任务和时间尺度的无缝知识集成方面仍然存在根本挑战。这项系统综述有助于不断增长关于法学硕士适应的知识体系，为研究人员和从业者提供了一个结构化框架，以了解语言模型终身学习的当前成就和未来机会。

Title: From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space

• Authors: Lehui Li, Yuyao Wang, Jisheng Yan, Wei Zhang, Jinliang Deng, Haoliang Sun, Zhongyi Han, Yongshun Gong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12664
• Pdf URL: https://arxiv.org/pdf/2603.12664
• Copy Paste: [[2603.12664]] From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space(https://arxiv.org/abs/2603.12664)
• Keywords: llm, prompt
• Abstract: Incorporating textual information into time-series forecasting holds promise for addressing event-driven non-stationarity; however, a fundamental modality gap hinders effective fusion: textual descriptions express temporal impacts implicitly and qualitatively, whereas forecasting models rely on explicit and quantitative signals. Through controlled semi-synthetic experiments, we show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues. To bridge this gap, we propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities. This space consists of interpretable, numerically grounded temporal primitives (mean shift, volatility, shape, and lag) extracted from text by an LLM via structured prompting and filtered through confidence-aware gating. Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines. The code will be released after acceptance.
• 摘要：将文本信息纳入时间序列预测有望解决事件驱动的非平稳性问题；然而，基本的模态差距阻碍了有效的融合：文本描述隐式和定性地表达了时间影响，而预测模型依赖于明确和定量的信号。通过受控的半合成实验，我们表明现有的方法过度关注冗余标记，并且难以可靠地将文本语义转换为可用的数字线索。为了弥补这一差距，我们提出了 TESS，它引入了时间演化语义空间作为模态之间的中间瓶颈。该空间由法学硕士通过结构化提示从文本中提取的可解释的、基于数字的时间基元（均值漂移、波动性、形状和滞后）组成，并通过置信感知门控进行过滤。对四个真实世界数据集的实验表明，与最先进的单峰和多峰基线相比，预测误差最多可减少 29%。代码将在接受后发布。

Title: MetaKE: Meta-learning Aligned Knowledge Editing via Bi-level Optimization

• Authors: Shuxin Liu, Ou Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12677
• Pdf URL: https://arxiv.org/pdf/2603.12677
• Copy Paste: [[2603.12677]] MetaKE: Meta-learning Aligned Knowledge Editing via Bi-level Optimization(https://arxiv.org/abs/2603.12677)
• Keywords: language model, llm
• Abstract: Knowledge editing (KE) aims to precisely rectify specific knowledge in Large Language Models (LLMs) without disrupting general capabilities. State-of-the-art methods suffer from an open-loop control mismatch. We identify a critical "Semantic-Execution Disconnect": the semantic target is derived independently without feedback from the downstream's feasible region. This misalignment often causes valid semantic targets to fall within the prohibited space, resulting in gradient truncation and editing failure. To bridge this gap, we propose MetaKE (Meta-learning Aligned Knowledge Editing), a new framework that reframes KE as a bi-level optimization problem. Departing from static calculation, MetaKE treats the edit target as a learnable meta-parameter: the upper-level optimizer seeks a feasible target to maximize post-edit performance, while the lower-level solver executes the editing. To address the challenge of differentiating through complex solvers, we derive a Structural Gradient Proxy, which explicitly backpropagates editability constraints to the target learning phase. Theoretical analysis demonstrates that MetaKE automatically aligns the edit direction with the model's feasible manifold. Extensive experiments confirm that MetaKE significantly outperforms strong baselines, offering a new perspective on knowledge editing.
• 摘要：知识编辑（KE）旨在精确纠正大型语言模型（LLM）中的特定知识，而不破坏一般能力。最先进的方法存在开环控制失配的问题。我们确定了一个关键的“语义执行断开”：语义目标是独立导出的，没有来自下游可行区域的反馈。这种错位通常会导致有效的语义目标落在禁止的空间内，从而导致梯度截断和编辑失败。为了弥补这一差距，我们提出了 MetaKE（元学习对齐知识编辑），这是一个将 KE 重新构建为双层优化问题的新框架。与静态计算不同，MetaKE 将编辑目标视为可学习的元参数：上层优化器寻找可行的目标以最大化编辑后性能，而下层求解器则执行编辑。为了解决通过复杂求解器进行区分的挑战，我们派生了一个结构梯度代理，它明确地将可编辑性约束反向传播到目标学习阶段。理论分析表明，MetaKE 自动将编辑方向与模型的可行流形对齐。大量实验证实，MetaKE 的性能显着优于强大的基线，为知识编辑提供了新的视角。

Title: Experimental evidence of progressive ChatGPT models self-convergence

• Authors: Konstantinos F. Xylogiannopoulos, Petros Xanthopoulos, Panagiotis Karampelas, Georgios A. Bakamitsos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12683
• Pdf URL: https://arxiv.org/pdf/2603.12683
• Copy Paste: [[2603.12683]] Experimental evidence of progressive ChatGPT models self-convergence(https://arxiv.org/abs/2603.12683)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large Language Models (LLMs) that undergo recursive training on synthetically generated data are susceptible to model collapse, a phenomenon marked by the generation of meaningless output. Existing research has examined this issue from either theoretical or empirical perspectives, often focusing on a single model trained recursively on its own outputs. While prior studies have cautioned against the potential degradation of LLM output quality under such conditions, no longitudinal investigation has yet been conducted to assess this effect over time. In this study, we employ a text similarity metric to evaluate different ChatGPT models' capacity to generate diverse textual outputs. Our findings indicate a measurable decline of recent ChatGPT releases' ability to produce varied text, even when explicitly prompted to do so, by setting the temperature parameter to one. The observed reduction in output diversity may be attributed to the influence of the amounts of synthetic data incorporated within their training datasets as the result of internet infiltration by LLM generated data. The phenomenon is defined as model self-convergence because of the gradual increase of similarities of produced texts among different ChatGPT versions.
• 摘要：对综合生成的数据进行递归训练的大型语言模型 (LLM) 很容易出现模型崩溃的情况，这种现象的特点是生成无意义的输出。现有的研究从理论或实证的角度研究了这个问题，通常侧重于根据其自身输出递归训练的单个模型。虽然之前的研究已经警告在这种情况下法学硕士输出质量可能会下降，但尚未进行纵向调查来评估随着时间的推移这种影响。在本研究中，我们采用文本相似性度量来评估不同 ChatGPT 模型生成不同文本输出的能力。我们的研究结果表明，最近的 ChatGPT 版本生成各种文本的能力明显下降，即使通过将温度参数设置为 1 明确提示这样做也是如此。观察到的输出多样性的减少可能归因于 LLM 生成的数据对互联网渗透造成的训练数据集中合成数据量的影响。这种现象被定义为模型自收敛，因为不同 ChatGPT 版本之间生成的文本的相似性逐渐增加。

Title: EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning

• Authors: Chi Ruan, Dongfu Jiang, Huaye Zeng, Ping Nie, Wenhu Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12698
• Pdf URL: https://arxiv.org/pdf/2603.12698
• Copy Paste: [[2603.12698]] EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning(https://arxiv.org/abs/2603.12698)
• Keywords: language model
• Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for improving code generation in large language models, but its effectiveness is limited by weak and static verification signals in existing coding RL datasets. In this paper, we propose a solution-conditioned and adversarial verification framework that iteratively refines test cases based on the execution behaviors of candidate solutions, with the goal of increasing difficulty, improving discriminative power, and reducing redundancy. Based on this framework, we introduce EvolveCoder-22k, a large-scale coding reinforcement learning dataset constructed through multiple rounds of adversarial test case evolution. Empirical analysis shows that iterative refinement substantially strengthens verification, with pass@1 decreasing from 43.80 to 31.22. Reinforcement learning on EvolveCoder-22k yields stable optimization and consistent performance gains, improving Qwen3-4B by an average of 4.2 points across four downstream benchmarks and outperforming strong 4B-scale baselines. Our results highlight the importance of adversarial, solution-conditioned verification for effective and scalable reinforcement learning in code generation.
• 摘要：具有可验证奖励的强化学习 (RLVR) 是一种很有前途的改进大型语言模型中的代码生成的方法，但其有效性受到现有编码 RL 数据集中弱且静态的验证信号的限制。在本文中，我们提出了一种以解决方案为条件的对抗性验证框架，该框架根据候选解决方案的执行行为迭代地细化测试用例，其目标是增加难度、提高判别力并减少冗余。基于这个框架，我们引入了EvolveCoder-22k，这是一个通过多轮对抗性测试用例进化构建的大规模编码强化学习数据集。实证分析表明，迭代细化大大加强了验证，pass@1从43.80下降到31.22。 EvolveCoder-22k 上的强化学习可实现稳定的优化和一致的性能增益，在四个下游基准测试中将 Qwen3-4B 平均提高 4.2 个点，并优于强大的 4B 规模基准。我们的结果强调了对抗性、解决方案条件验证对于代码生成中有效且可扩展的强化学习的重要性。

Title: SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

• Authors: Aditya Maheshwari, Amit Gajkeshwar, Kaushal Sharma, Vivek Patel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12768
• Pdf URL: https://arxiv.org/pdf/2603.12768
• Copy Paste: [[2603.12768]] SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models(https://arxiv.org/abs/2603.12768)
• Keywords: language model, gpt, llm
• Abstract: As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at this https URL
• 摘要：随着大型语言模型 (LLM) 成为宗教知识的流行来源，了解它是否公平对待不同群体非常重要。这项研究首次衡量了法学硕士如何处理伊斯兰教两个主要教派：逊尼派和什叶派之间的差异。我们提出了一项名为 SectEval 的测试，有英语和印地语版本，包含 88 个问题，用于检查 15 个顶级 LLM 模型（专有模型和开放权重模型）的偏差。我们的结果显示出基于语言的重大不一致。在英语中，许多强大的模型 DeepSeek-v3 和 GPT-4o 通常倾向于什叶派答案。然而，当用印地语问完全相同的问题时，这些模型转而支持逊尼派答案。这意味着用户只需更改语言就可以获得完全不同的宗教建议。我们还研究了模型对位置的反应。高级模型 Claude-3.5 更改了答案以匹配用户的国家/地区，为来自伊朗的用户提供什叶派答案，为来自沙特阿拉伯的用户提供逊尼派答案。相比之下，较小的模型（尤其是印地语）忽略了用户的位置并坚持逊尼派观点。这些发现表明人工智能并不是中立的；它的宗教“真理”根据你所说的语言和你声称来自的国家而变化。该数据集可在此 https URL 获取

Title: Adaptive Vision-Language Model Routing for Computer Use Agents

• Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2603.12823
• Pdf URL: https://arxiv.org/pdf/2603.12823
• Copy Paste: [[2603.12823]] Adaptive Vision-Language Model Routing for Computer Use Agents(https://arxiv.org/abs/2603.12823)
• Keywords: language model, agent
• Abstract: Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Language Model (VLM) to interpret screenshots and predict grounded tool calls. However, grounding accuracy varies dramatically across VLMs, while current CUA systems typically route every action to a single fixed model regardless of difficulty. We propose \textbf{Adaptive VLM Routing} (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs. For each tool call, AVR estimates action difficulty from multimodal embeddings, probes a small VLM to measure confidence, and routes the action to the cheapest model whose predicted accuracy satisfies a target reliability threshold. For \textit{warm} agents with memory of prior UI interactions, retrieved context further narrows the capability gap between small and large models, allowing many actions to be handled without escalation. We formalize routing as a cost--accuracy trade-off, derive a threshold-based policy for model selection, and evaluate AVR using ScreenSpot-Pro grounding data together with the OpenClaw agent routing benchmark. Across these settings, AVR projects inference cost reductions of up to 78\% while staying within 2 percentage points of an all-large-model baseline. When combined with the Visual Confused Deputy guardrail, AVR also escalates high-risk actions directly to the strongest available model, unifying efficiency and safety within a single routing framework. Materials are also provided Model, benchmark, and code: this https URL.
• 摘要：计算机使用代理 (CUA) 依靠视觉语言模型 (VLM) 来解释屏幕截图并预测基础工具调用，将自然语言指令转换为图形用户界面 (GUI) 操作，例如单击、击键和滚动。然而，不同 VLM 的接地精度差异很大，而当前的 CUA 系统通常将每个动作路由到单个固定模型，无论难度如何。我们提出了自适应 VLM 路由（AVR），这是一个在 CUA 协调器和 VLM 池之间插入轻量级语义路由层的框架。对于每个工具调用，AVR 都会根据多模态嵌入估计操作难度，探测小型 VLM 来测量置信度，并将操作路由到预测精度满足目标可靠性阈值的最便宜模型。对于具有先前 UI 交互记忆的 \textit{warm} 代理，检索到的上下文进一步缩小了小型模型和大型模型之间的能力差距，允许在不升级的情况下处理许多操作。我们将路由形式化为成本与准确性的权衡，得出基于阈值的模型选择策略，并使用 ScreenSpot-Pro 接地数据和 OpenClaw 代理路由基准来评估 AVR。在这些设置中，AVR 预计推理成本可降低高达 78%，同时与全大型模型基线的差距保持在 2 个百分点以内。当与视觉混乱副护栏结合使用时，AVR 还将高风险操作直接升级为最强大的可用模型，在单个路由框架内统一效率和安全性。还提供了材料模型、基准测试和代码：此 https URL。

Title: Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

• Authors: Xu Guo, Qiming Ge, Jian Tong, Kedi Chen, Jin Zhang, Xiaogui Yang, Xuan Gao, Haijun Lv, Zhihui Lu, Yicheng Zou, Qipeng Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12826
• Pdf URL: https://arxiv.org/pdf/2603.12826
• Copy Paste: [[2603.12826]] Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design(https://arxiv.org/abs/2603.12826)
• Keywords: language model
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.
• 摘要：具有可验证奖励的强化学习（RLVR）显着增强了大型语言模型的推理能力。当应用于 RLVR 时，多项选择题 (MCQ) 提供了可扩展的可验证数据源，但存在诱发奖励黑客的风险，其中模型通过随机猜测或简单消除进行快捷推理。当前的方法通常通过将 MCQ 转换为开放式格式来缓解这一问题，从而丢弃专家设计的干扰项提供的对比信号。在这项工作中，我们系统地研究了期权设计对 RLVR 的影响。我们的分析强调了两个主要见解：（1）训练和测试之间的选项计数不匹配会降低性能。 (2) 强大的干扰因素有效地减轻了随机猜测，即使有 2 向问题也能实现有效的 RLVR 训练。受这些发现的启发，我们提出了迭代干扰项管理（IDC），这是一个积极构建高质量干扰项以阻止消除捷径并促进深度推理的框架。各种基准的实验表明，与原始数据相比，我们的方法有效地提高了干扰质量，并在 RLVR 训练中产生了显着的收益。

Title: CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

• Authors: João Silva, Luís Gomes, António Branco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12872
• Pdf URL: https://arxiv.org/pdf/2603.12872
• Copy Paste: [[2603.12872]] CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility(https://arxiv.org/abs/2603.12872)
• Keywords: language model, llm
• Abstract: This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at this https URL.
• 摘要：本文报告了欧洲葡萄牙语开放大型语言模型 (LLM) (PT-PT) 排行榜的开发及其相关基准。该排行榜旨在解决欧洲葡萄牙语法学硕士评估方面的差距，迄今为止，欧洲葡萄牙语还没有专门针对这种语言变体的排行榜。该文件还报告了新颖的基准，其中包括一些涉及迄今为止欧洲葡萄牙语基准中尚未提供的绩效方面的基准，即模型保障措施和与葡萄牙文化的一致性。排行榜可通过此 https URL 获取。

Title: Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study

• Authors: Liel Binyamin, Elior Sulem
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12906
• Pdf URL: https://arxiv.org/pdf/2603.12906
• Copy Paste: [[2603.12906]] Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study(https://arxiv.org/abs/2603.12906)
• Keywords: language model
• Abstract: Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.
• 摘要：对发展上合理的语言模型的研究主要集中在英语上，留下了关于多语言环境的悬而未决的问题。我们通过在严格大小匹配的数据条件下将 BabyBERTa 扩展到英语-法语场景，涵盖单语、双语和跨语言设置，提出了紧凑语言模型的系统研究。我们的设计对比了两种类型的训练语料库：(i) 儿童导向语音（约 250 万个标记），遵循 BabyBERTa 和相关工作；(ii) 多领域语料库（约 1000 万个标记），将 BabyLM 框架扩展到法语。为了实现公平评估，我们还引入了新资源，包括法语版本的 QAMR 和 QASRL，以及英语和法语多领域语料库。我们在句法和语义任务上评估模型，并将它们与仅在维基百科数据上训练的模型进行比较。结果揭示了上下文相关的效果：维基百科上的训练始终有利于语义任务，而儿童导向的语音可以改善单语环境中的语法判断。双语预训练在文本蕴含方面取得了显着的成果，尤其是法语方面的改进尤其显着。重要的是，BabyBERTa、RoBERTa 和 LTG-BERT 中出现了类似的模式，这表明跨架构的趋势一致。

Title: DS$^2$-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning

• Authors: Ruiyao Xu, Noelle I. Samia, Han Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12932
• Pdf URL: https://arxiv.org/pdf/2603.12932
• Copy Paste: [[2603.12932]] DS$^2$-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning(https://arxiv.org/abs/2603.12932)
• Keywords: language model, llm
• Abstract: Adapting Large Language Models (LLMs) to specialized domains requires high-quality instruction tuning datasets, which are expensive to create through human annotation. Existing data synthesis methods focus on general-purpose tasks and fail to capture domain-specific terminology and reasoning patterns. To address this, we introduce DS$^2$-Instruct, a zero-shot framework that generates domain-specific instruction datasets without human supervision. Our approach first generates task-informed keywords to ensure comprehensive domain coverage. It then creates diverse instructions by pairing these keywords with different cognitive levels from Bloom's Taxonomy. Finally, it uses self-consistency validation to ensure data quality. We apply this framework to generate datasets across seven challenging domains, such as mathematics, finance, and logical reasoning. Comprehensive evaluation demonstrates that models fine-tuned on our generated data achieve substantial improvements over existing data generation methods.
• 摘要：使大型语言模型 (LLM) 适应专业领域需要高质量的指令调整数据集，而通过人工注释创建这些数据集的成本很高。现有的数据合成方法侧重于通用任务，无法捕获特定领域的术语和推理模式。为了解决这个问题，我们引入了 DS$^2$-Instruct，这是一个零样本框架，可以在没有人工监督的情况下生成特定领域的指令数据集。我们的方法首先生成任务通知的关键字，以确保全面的领域覆盖。然后，它通过将这些关键字与布鲁姆分类法中的不同认知水平配对来创建不同的指令。最后，它使用自我一致性验证来确保数据质量。我们应用这个框架来生成跨七个具有挑战性的领域的数据集，例如数学、金融和逻辑推理。综合评估表明，根据我们生成的数据进行微调的模型比现有的数据生成方法取得了实质性改进。

Title: Long-form RewardBench: Evaluating Reward Models for Long-form Generation

• Authors: Hui Huang, Yancheng He, Wei Liu, Muyun Yang, Jiaheng Liu, Kehai Chen, Bing Xu, Conghui Zhu, Hailong Cao, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.12963
• Pdf URL: https://arxiv.org/pdf/2603.12963
• Copy Paste: [[2603.12963]] Long-form RewardBench: Evaluating Reward Models for Long-form Generation(https://arxiv.org/abs/2603.12963)
• Keywords: chat
• Abstract: The widespread adoption of reinforcement learning-based alignment highlights the growing importance of reward models. Various benchmarks have been built to evaluate reward models in various domains and scenarios. However, a significant gap remains in assessing reward models for long-form generation, despite its critical role in real-world applications. To bridge this, we introduce Long-form RewardBench, the first reward modeling testbed specifically designed for long-form generation. Our benchmark encompasses five key subtasks: QA, RAG, Chat, Writing, and Reasoning. We collected instruction and preference data through a meticulously designed multi-stage data collection process, and conducted extensive experiments on 20+ mainstream reward models, including both classifiers and generative models. Our findings reveal that current models still lack long-form reward modeling capabilities. Furthermore, we designed a novel Long-form Needle-in-a-Haystack Test, which revealed a correlation between reward modeling performance and the error's position within a response, as well as the overall response length, with distinct characteristics observed between classification and generative models. Finally, we demonstrate that classifiers exhibit better generalizability compared to generative models trained on the same data. As the first benchmark for long-form reward modeling, this work aims to offer a robust platform for visualizing progress in this crucial area.
• 摘要：基于强化学习的一致性的广泛采用凸显了奖励模型日益增长的重要性。已经建立了各种基准来评估不同领域和场景中的奖励模型。然而，尽管长格式生成的奖励模型在实际应用中发挥着关键作用，但在评估长格式生成的奖励模型方面仍然存在很大差距。为了解决这个问题，我们引入了 Long-form RewardBench，这是第一个专为长格式生成而设计的奖励建模测试平台。我们的基准测试包含五个关键子任务：QA、RAG、聊天、写作和推理。我们通过精心设计的多阶段数据收集流程收集了指令和偏好数据，并对 20 多个主流奖励模型（包括分类器和生成模型）进行了广泛的实验。我们的研究结果表明，当前模型仍然缺乏长期奖励建模功能。此外，我们设计了一种新颖的长篇大海捞针测试，它揭示了奖励建模性能与响应中错误位置以及总体响应长度之间的相关性，并在分类模型和生成模型之间观察到了不同的特征。最后，我们证明与在相同数据上训练的生成模型相比，分类器表现出更好的通用性。作为长形式奖励建模的第一个基准，这项工作旨在提供一个强大的平台来可视化这一关键领域的进展。

Title: Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

• Authors: Boxuan Lyu, Haiyue Song, Zhi Qu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.12983
• Pdf URL: https://arxiv.org/pdf/2603.12983
• Copy Paste: [[2603.12983]] Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation(https://arxiv.org/abs/2603.12983)
• Keywords: llm
• Abstract: Error Span Detection (ESD) is a crucial subtask in Machine Translation (MT) evaluation, aiming to identify the location and severity of translation errors. While fine-tuning models on human-annotated data improves ESD performance, acquiring such data is expensive and prone to inconsistencies among annotators. To address this, we propose a novel self-evolution framework based on Minimum Bayes Risk (MBR) decoding, named Iterative MBR Distillation for ESD, which eliminates the reliance on human annotations by leveraging an off-the-shelf LLM to generate this http URL experiments on the WMT Metrics Shared Task datasets demonstrate that models trained solely on these self-generated pseudo-labels outperform both unadapted base model and supervised baselines trained on human annotations at the system and span levels, while maintaining competitive sentence-level performance.
• 摘要：错误跨度检测（ESD）是机器翻译（MT）评估中的一个重要子任务，旨在识别翻译错误的位置和严重程度。虽然对人工注释数据的模型进行微调可以提高 ESD 性能，但获取此类数据的成本很高，而且注释者之间容易出现不一致。为了解决这个问题，我们提出了一种基于最小贝叶斯风险（MBR）解码的新颖的自我进化框架，名为“ESD 迭代 MBR 蒸馏”，它通过利用现成的 LLM 在 WMT Metrics 共享任务数据集上生成此 http URL 实验来消除对人类注释的依赖。跨越水平，同时保持有竞争力的句子水平表现。

Title: Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

• Authors: Yifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa, Lei Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.13045
• Pdf URL: https://arxiv.org/pdf/2603.13045
• Copy Paste: [[2603.13045]] Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation(https://arxiv.org/abs/2603.13045)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training methods rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages. In this paper, we introduce WALAR, a reinforcement training method using only monolingual text to elevate LLMs' translation capabilities on massive low-resource languages while retaining their performance on high-resource languages. Our key insight is based on the observation of failure modes (or "holes") in existing source-based multilingual quality estimation (QE) models. Reinforcement learning (RL) using these QE models tends to amplify such holes, resulting in poorer multilingual LLMs. We develop techniques including word alignment and language alignment to mitigate such holes in WALAR's reward for RL training. We continually trained an LLM supporting translation of 101 languages using WALAR. The experiments show that our new model outperforms LLaMAX, one of the strongest open-source multilingual LLMs by a large margin on 1400 language directions on Flores-101 dataset.
• 摘要：大型语言模型（LLM）在高资源语言对的机器翻译方面表现出了卓越的能力，但在低资源翻译方面的表现仍然落后。现有的后训练方法严重依赖于高质量的并行数据，而这些数据对于资源匮乏的语言来说通常是稀缺的或无法获得的。在本文中，我们介绍了 WALAR，这是一种仅使用单语文本的强化训练方法，可以提高法学硕士在大量低资源语言上的翻译能力，同时保留其在高资源语言上的翻译能力。我们的主要见解基于对现有基于源的多语言质量估计 (QE) 模型中的故障模式（或“漏洞”）的观察。使用这些 QE 模型的强化学习 (RL) 往往会放大这些漏洞，导致多语言法学硕士的成绩较差。我们开发了包括单词对齐和语言对齐在内的技术来弥补 WALAR 对 RL 训练的奖励中的此类漏洞。我们持续培训一名支持使用 WALAR 翻译 101 种语言的法学硕士。实验表明，我们的新模型在 Flores-101 数据集上的 1400 种语言方向上远远优于 LLaMAX（最强的开源多语言 LLM 之一）。

Title: ESG-Bench: Benchmarking Long-Context ESG Reports for Hallucination Mitigation

• Authors: Siqi Sun, Ben Peng Wu, Mali Jin, Peizhen Bai, Hanpei Zhang, Xingyi Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2603.13154
• Pdf URL: https://arxiv.org/pdf/2603.13154
• Copy Paste: [[2603.13154]] ESG-Bench: Benchmarking Long-Context ESG Reports for Hallucination Mitigation(https://arxiv.org/abs/2603.13154)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: As corporate responsibility increasingly incorporates environmental, social, and governance (ESG) criteria, ESG reporting is becoming a legal requirement in many regions and a key channel for documenting sustainability practices and assessing firms' long-term and ethical performance. However, the length and complexity of ESG disclosures make them difficult to interpret and automate the analysis reliably. To support scalable and trustworthy analysis, this paper introduces ESG-Bench, a benchmark dataset for ESG report understanding and hallucination mitigation in large language models (LLMs). ESG-Bench contains human-annotated question-answer (QA) pairs grounded in real-world ESG report contexts, with fine-grained labels indicating whether model outputs are factually supported or hallucinated. Framing ESG report analysis as a QA task with verifiability constraints enables systematic evaluation of LLMs' ability to extract and reason over ESG content and provides a new use case: mitigating hallucinations in socially sensitive, compliance-critical settings. We design task-specific Chain-of-Thought (CoT) prompting strategies and fine-tune multiple state-of-the-art LLMs on ESG-Bench using CoT-annotated rationales. Our experiments show that these CoT-based methods substantially outperform standard prompting and direct fine-tuning in reducing hallucinations, and that the gains transfer to existing QA benchmarks beyond the ESG domain.
• 摘要：随着企业责任越来越多地纳入环境、社会和治理 (ESG) 标准，ESG 报告正在成为许多地区的法律要求，也是记录可持续发展实践和评估公司长期道德绩效的关键渠道。然而，ESG 披露的长度和复杂性使得它们难以可靠地解释和自动化分析。为了支持可扩展且值得信赖的分析，本文引入了 ESG-Bench，这是一个用于大型语言模型 (LLM) 中 ESG 报告理解和幻觉缓解的基准数据集。 ESG-Bench 包含基于现实世界 ESG 报告上下文的人工注释问答 (QA) 对，并带有细粒度标签，指示模型输出是否得到事实支持或幻觉。将 ESG 报告分析框架为具有可验证性约束的 QA 任务，可以系统地评估法学硕士提取和推理 ESG 内容的能力，并提供一个新的用例：减轻社会敏感、合规性关键环境中的幻觉。我们设计特定于任务的思想链 (CoT) 提示策略，并使用 CoT 注释的基本原理在 ESG-Bench 上微调多个最先进的 LLM。我们的实验表明，这些基于 CoT 的方法在减少幻觉方面远远优于标准提示和直接微调，并且收益转移到 ESG 领域之外的现有 QA 基准。

Title: Neuron-Aware Data Selection In Instruction Tuning For Large Language Models

• Authors: Xin Chen, Junchao Wu, Shu Yang, Runzhe Zhan, Zeyu Wu, Min Yang, Shujian Huang, Lidia S. Chao, Derek F. Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2603.13201
• Pdf URL: https://arxiv.org/pdf/2603.13201
• Copy Paste: [[2603.13201]] Neuron-Aware Data Selection In Instruction Tuning For Large Language Models(https://arxiv.org/abs/2603.13201)
• Keywords: language model, gpt, llm
• Abstract: Instruction Tuning (IT) has been proven to be an effective approach to unlock the powerful capabilities of large language models (LLMs). Recent studies indicate that excessive IT data can degrade LLMs performance, while carefully selecting a small subset of high-quality IT data can significantly enhance their capabilities. Therefore, identifying the most efficient subset data from the IT dataset to effectively develop either specific or general abilities in LLMs has become a critical challenge. To address this, we propose a novel and efficient framework called NAIT. NAIT evaluates the impact of IT data on LLMs performance by analyzing the similarity of neuron activation patterns between the IT dataset and the target domain capability. Specifically, NAIT captures neuron activation patterns from in-domain datasets of target domain capabilities to construct reusable and transferable neuron activation features. It then evaluates and selects optimal samples based on the similarity between candidate samples and the expected activation features of the target capabilities. Experimental results show that training on the 10\% Alpaca-GPT4 IT data subset selected by NAIT consistently outperforms methods that rely on external advanced models or uncertainty-based features across various tasks. Our findings also reveal the transferability of neuron activation features across different capabilities of LLMs. In particular, IT data with more logical reasoning and programmatic features possesses strong general transferability, enabling models to develop stronger capabilities across multiple tasks, while a stable core subset of data is sufficient to consistently activate fundamental model capabilities and universally improve performance across diverse tasks.
• 摘要：指令调优 (IT) 已被证明是释放大型语言模型 (LLM) 强大功能的有效方法。最近的研究表明，过多的 IT 数据会降低法学硕士的绩效，而仔细选择一小部分高质量 IT 数据可以显着增强其能力。因此，从 IT 数据集中识别最有效的子集数据以有效培养法学硕士的特定或一般能力已成为一项关键挑战。为了解决这个问题，我们提出了一种新颖且高效的框架，称为 NAIT。 NAIT 通过分析 IT 数据集和目标域能力之间神经元激活模式的相似性来评估 IT 数据对 LLM 性能的影响。具体来说，NAIT 从目标域功能的域内数据集中捕获神经元激活模式，以构建可重用和可转移的神经元激活特征。然后根据候选样本与目标能力的预期激活特征之间的相似度来评估和选择最佳样本。实验结果表明，在 NAIT 选择的 10\% Alpaca-GPT4 IT 数据子集上进行的训练在各种任务中始终优于依赖外部高级模型或基于不确定性的特征的方法。我们的研究结果还揭示了神经元激活特征在法学硕士不同能力之间的可转移性。特别是，具有更多逻辑推理和编程特性的IT数据具有很强的通用可移植性，使模型能够在多个任务中开发出更强的能力，而稳定的核心数据子集足以持续激活模型的基础能力并普遍提高跨不同任务的性能。