2025-12-04

Title: Entropy-Based Measurement of Value Drift and Alignment Work in Large Language Models

• Authors: Samih Fadli
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03047
• Pdf URL: https://arxiv.org/pdf/2512.03047
• Copy Paste: [[2512.03047]] Entropy-Based Measurement of Value Drift and Alignment Work in Large Language Models(https://arxiv.org/abs/2512.03047)
• Keywords: language model
• Abstract: Large language model safety is usually assessed with static benchmarks, but key failures are dynamic: value drift under distribution shift, jailbreak attacks, and slow degradation of alignment in deployment. Building on a recent Second Law of Intelligence that treats ethical entropy as a state variable which tends to increase unless countered by alignment work, we make this framework operational for large language models. We define a five-way behavioral taxonomy, train a classifier to estimate ethical entropy S(t) from model transcripts, and measure entropy dynamics for base and instruction-tuned variants of four frontier models across stress tests. Base models show sustained entropy growth, while tuned variants suppress drift and reduce ethical entropy by roughly eighty percent. From these trajectories we estimate an effective alignment work rate gamma_eff and embed S(t) and gamma_eff in a monitoring pipeline that raises alerts when entropy drift exceeds a stability threshold, enabling run-time oversight of value drift.
• 摘要：大型语言模型的安全性通常通过静态基准进行评估，但关键故障是动态的：分布转移下的价值漂移、越狱攻击以及部署中对齐的缓慢退化。基于最近的智能第二定律，该定律将道德熵视为一个状态变量，除非通过对齐工作来抵消，否则该变量往往会增加，我们使该框架可用于大型语言模型。我们定义了五向行为分类法，训练分类器从模型转录中估计道德熵 S(t)，并在压力测试中测量四个前沿模型的基础和指令调整变体的熵动态。基本模型显示出持续的熵增长，而调整后的变体则抑制了漂移并将道德熵减少了大约百分之八十。根据这些轨迹，我们估计有效的对齐工作率 gamma_eff 并将 S(t) 和 gamma_eff 嵌入监控管道中，当熵漂移超过稳定性阈值时，该管道会发出警报，从而实现对值漂移的运行时监督。

Title: Watermarks for Embeddings-as-a-Service Large Language Models

• Authors: Anudeex Shetty
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03079
• Pdf URL: https://arxiv.org/pdf/2512.03079
• Copy Paste: [[2512.03079]] Watermarks for Embeddings-as-a-Service Large Language Models(https://arxiv.org/abs/2512.03079)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities in natural language understanding and generation. Based on these LLMs, businesses have started to provide Embeddings-as-a-Service (EaaS), offering feature extraction capabilities (in the form of text embeddings) that benefit downstream natural language processing tasks. However, prior research has demonstrated that EaaS is vulnerable to imitation attacks, where an attacker clones the service's model in a black-box manner without access to the model's internal workings. In response, watermarks have been added to the text embeddings to protect the intellectual property of EaaS providers by allowing them to check for model ownership. This thesis focuses on defending against imitation attacks by investigating EaaS watermarks. To achieve this goal, we unveil novel attacks and propose and validate new watermarking techniques. Firstly, we show that existing EaaS watermarks can be removed through paraphrasing the input text when attackers clone the model during imitation attacks. Our study illustrates that paraphrasing can effectively bypass current state-of-the-art EaaS watermarks across various attack setups (including different paraphrasing techniques and models) and datasets in most instances. This demonstrates a new vulnerability in recent EaaS watermarking techniques. Subsequently, as a countermeasure, we propose a novel watermarking technique, WET (Watermarking EaaS with Linear Transformation), which employs linear transformation of the embeddings. Watermark verification is conducted by applying a reverse transformation and comparing the similarity between recovered and original embeddings. We demonstrate its robustness against paraphrasing attacks with near-perfect verifiability. We conduct detailed ablation studies to assess the significance of each component and hyperparameter in WET.
• 摘要：大型语言模型 (LLM) 在自然语言理解和生成方面表现出了卓越的能力。基于这些法学硕士，企业已经开始提供嵌入即服务（EaaS），提供有利于下游自然语言处理任务的特征提取功能（以文本嵌入的形式）。然而，之前的研究表明，EaaS 很容易受到模仿攻击，攻击者以黑盒方式克隆服务模型，而无需访问模型的内部工作原理。作为回应，文本嵌入中添加了水印，通过允许 EaaS 提供商检查模型所有权来保护他们的知识产权。本论文的重点是通过研究 EaaS 水印来防御模仿攻击。为了实现这一目标，我们推出了新颖的攻击，并提出并验证了新的水印技术。首先，我们表明，当攻击者在模仿攻击期间克隆模型时，可以通过解释输入文本来删除现有的 EaaS 水印。我们的研究表明，在大多数情况下，释义可以跨各种攻击设置（包括不同的释义技术和模型）和数据集，有效绕过当前最先进的 EaaS 水印。这展示了最近的 EaaS 水印技术中的一个新漏洞。随后，作为对策，我们提出了一种新颖的水印技术，WET（带有线性变换的水印EaaS），它采用嵌入的线性变换。水印验证是通过应用逆变换并比较恢复的嵌入和原始嵌入之间的相似性来进行的。我们通过近乎完美的可验证性证明了它对释义攻击的鲁棒性。我们进行详细的消融研究，以评估 WET 中每个组件和超参数的重要性。

Title: Alleviating Choice Supportive Bias in LLM with Reasoning Dependency Generation

• Authors: Nan Zhuang, Wenshuo Wang, Lekai Qian, Yuxiao Wang, Boyu Cao, Qi Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.03082
• Pdf URL: https://arxiv.org/pdf/2512.03082
• Copy Paste: [[2512.03082]] Alleviating Choice Supportive Bias in LLM with Reasoning Dependency Generation(https://arxiv.org/abs/2512.03082)
• Keywords: language model, llm
• Abstract: Recent studies have demonstrated that some Large Language Models exhibit choice-supportive bias (CSB) when performing evaluations, systematically favoring their chosen options and potentially compromising the objectivity of AI-assisted decision making. While existing debiasing approaches primarily target demographic and social biases, methods for addressing cognitive biases in LLMs remain largely unexplored. In this work, we present the first solution to address CSB through Reasoning Dependency Generation (RDG), a novel framework for generating unbiased reasoning data to mitigate choice-supportive bias through fine-tuning. RDG automatically constructs balanced reasoning QA pairs, explicitly (un)modeling the dependencies between choices, evidences, and justifications. Our approach is able to generate a large-scale dataset of QA pairs across domains, incorporating Contextual Dependency Data and Dependency Decouple Data. Experiments show that LLMs fine-tuned on RDG-generated data demonstrate a 81.5% improvement in memory-based experiments and 94.3% improvement in the evaluation-based experiment, while maintaining similar performance on standard BBQ benchmarks. This work pioneers an approach for addressing cognitive biases in LLMs and contributes to the development of more reliable AI-assisted decision support systems.
• 摘要：最近的研究表明，一些大型语言模型在进行评估时表现出选择支持偏差（CSB），系统地偏向其选择的选项，并可能损害人工智能辅助决策的客观性。虽然现有的消除偏见方法主要针对人口和社会偏见，但解决法学硕士认知偏见的方法在很大程度上仍未得到探索。在这项工作中，我们提出了第一个通过推理依赖生成（RDG）解决 CSB 问题的解决方案，这是一种生成无偏见推理数据的新颖框架，通过微调来减轻选择支持偏见。 RDG 自动构建平衡推理 QA 对，显式（取消）建模选择、证据和理由之间的依赖关系。我们的方法能够生成跨领域的大规模 QA 对数据集，结合上下文依赖数据和依赖解耦数据。实验表明，对 RDG 生成的数据进行微调的 LLM 在基于内存的实验中显示出 81.5% 的改进，在基于评估的实验中显示出 94.3% 的改进，同时在标准 BBQ 基准上保持类似的性能。这项工作开创了一种解决法学硕士认知偏差的方法，并有助于开发更可靠的人工智能辅助决策支持系统。

Title: Enhancing Job Matching: Occupation, Skill and Qualification Linking with the ESCO and EQF taxonomies

• Authors: Stylianos Saroglou, Konstantinos Diamantaras, Francesco Preta, Marina Delianidi, Apostolos Benisis, Christian Johannes Meyer
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03195
• Pdf URL: https://arxiv.org/pdf/2512.03195
• Copy Paste: [[2512.03195]] Enhancing Job Matching: Occupation, Skill and Qualification Linking with the ESCO and EQF taxonomies(https://arxiv.org/abs/2512.03195)
• Keywords: language model
• Abstract: This study investigates the potential of language models to improve the classification of labor market information by linking job vacancy texts to two major European frameworks: the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy and the European Qualifications Framework (EQF). We examine and compare two prominent methodologies from the literature: Sentence Linking and Entity Linking. In support of ongoing research, we release an open-source tool, incorporating these two methodologies, designed to facilitate further work on labor classification and employment discourse. To move beyond surface-level skill extraction, we introduce two annotated datasets specifically aimed at evaluating how occupations and qualifications are represented within job vacancy texts. Additionally, we examine different ways to utilize generative large language models for this task. Our findings contribute to advancing the state of the art in job entity extraction and offer computational infrastructure for examining work, skills, and labor market narratives in a digitally mediated economy. Our code is made publicly available: this https URL
• 摘要：本研究通过将职位空缺文本与两个主要欧洲框架联系起来，调查了语言模型改善劳动力市场信息分类的潜力：欧洲技能、能力、资格和职业（ESCO）分类法和欧洲资格框架（EQF）。我们检查并比较文献中的两种著名方法：句子链接和实体链接。为了支持正在进行的研究，我们发布了一个开源工具，结合了这两种方法，旨在促进劳动分类和就业讨论的进一步工作。为了超越表面技能提取，我们引入了两个带注释的数据集，专门用于评估职位空缺文本中职业和资格的表示方式。此外，我们还研究了利用生成大型语言模型来完成此任务的不同方法。我们的研究结果有助于推进工作实体提取的最先进水平，并为检查数字媒介经济中的工作、技能和劳动力市场叙述提供计算基础设施。我们的代码已公开：此 https URL

Title: InvertiTune: High-Quality Data Synthesis for Cost-Effective Single-Shot Text-to-Knowledge Graph Generation

• Authors: Faezeh Faez, Marzieh S. Tahaei, Yaochen Hu, Ali Pourranjbar, Mahdi Biparva, Mark Coates, Yingxue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.03197
• Pdf URL: https://arxiv.org/pdf/2512.03197
• Copy Paste: [[2512.03197]] InvertiTune: High-Quality Data Synthesis for Cost-Effective Single-Shot Text-to-Knowledge Graph Generation(https://arxiv.org/abs/2512.03197)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have revolutionized the ability to understand and generate text, enabling significant progress in automatic knowledge graph construction from text (Text2KG). Many Text2KG methods, however, rely on iterative LLM prompting, making them computationally expensive and prone to overlooking complex relations distributed throughout the text. To address these limitations, we propose InvertiTune, a framework that combines a controlled data generation pipeline with supervised fine-tuning (SFT). Within this framework, the data-generation pipeline systematically extracts subgraphs from large knowledge bases, applies noise filtering, and leverages LLMs to generate corresponding natural text descriptions, a task more aligned with LLM capabilities than direct KG generation from text. This pipeline enables generating datasets composed of longer texts paired with larger KGs that better reflect real-world scenarios compared to existing benchmarks, thus supporting effective SFT of lightweight models for single-shot KG construction. Experimental results on CE12k, a dataset generated using the introduced pipeline, show that InvertiTune outperforms larger non-fine-tuned LLMs as well as state-of-the-art Text2KG approaches, while also demonstrating stronger cross-dataset generalization on CrossEval-1200, a test set created from three established benchmark datasets and CE12k. These findings highlight the importance of realistic, high-quality training data for advancing efficient and high-performing Text2KG systems.
• 摘要：大型语言模型 (LLM) 彻底改变了理解和生成文本的能力，使文本自动知识图构建 (Text2KG) 取得了重大进展。然而，许多 Text2KG 方法依赖于迭代的 LLM 提示，这使得它们的计算成本很高，并且容易忽略分布在整个文本中的复杂关系。为了解决这些限制，我们提出了 InvertiTune，这是一个将受控数据生成管道与监督微调（SFT）相结合的框架。在此框架内，数据生成管道系统地从大型知识库中提取子图，应用噪声过滤，并利用 LLM 生成相应的自然文本描述，该任务比直接从文本生成 KG 更符合 LLM 功能。该管道能够生成由较长文本与较大知识图谱配对组成的数据集，与现有基准相比，这些数据集更好地反映了现实世界的场景，从而支持单次知识图谱构建的轻量级模型的有效 SFT。 CE12k（使用引入的管道生成的数据集）上的实验结果表明，InvertiTune 优于更大的非微调 LLM 以及最先进的 Text2KG 方法，同时还在 CrossEval-1200（根据三个已建立的基准数据集和 CE12k 创建的测试集）上展示了更强的跨数据集泛化能力。这些发现凸显了真实、高质量的训练数据对于推进高效、高性能的 Text2KG 系统的重要性。

Title: Identifying attributions of causality in political text

• Authors: Paulina Garcia-Corral
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2512.03214
• Pdf URL: https://arxiv.org/pdf/2512.03214
• Copy Paste: [[2512.03214]] Identifying attributions of causality in political text(https://arxiv.org/abs/2512.03214)
• Keywords: language model
• Abstract: Explanations are a fundamental element of how people make sense of the political world. Citizens routinely ask and answer questions about why events happen, who is responsible, and what could or should be done differently. Yet despite their importance, explanations remain an underdeveloped object of systematic analysis in political science, and existing approaches are fragmented and often issue-specific. I introduce a framework for detecting and parsing explanations in political text. To do this, I train a lightweight causal language model that returns a structured data set of causal claims in the form of cause-effect pairs for downstream analysis. I demonstrate how causal explanations can be studied at scale, and show the method's modest annotation requirements, generalizability, and accuracy relative to human coding.
• 摘要：解释是人们理解政治世界的基本要素。公民经常询问并回答有关事件发生原因、责任人以及可以或应该采取哪些不同措施的问题。然而，尽管解释很重要，但它仍然是政治学中系统分析的一个不发达的对象，现有的方法是支离破碎的，而且往往是针对特定问题的。我介绍了一个用于检测和解析政治文本中的解释的框架。为此，我训练了一个轻量级因果语言模型，该模型以因果对的形式返回因果声明的结构化数据集，以供下游分析。我演示了如何大规模研究因果解释，并展示该方法相对于人类编码的适度注释要求、普遍性和准确性。

Title: Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs

• Authors: Kunj Joshi, David A. Smith
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03310
• Pdf URL: https://arxiv.org/pdf/2512.03310
• Copy Paste: [[2512.03310]] Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs(https://arxiv.org/abs/2512.03310)
• Keywords: language model, llm
• Abstract: The current literature on memorization in Natural Language Models, especially Large Language Models (LLMs), poses severe security and privacy risks, as models tend to memorize personally identifying information (PIIs) from training data. We introduce Randomized Masked Fine-Tuning (RMFT), a novel privacy-preserving fine-tuning technique that reduces PII memorization while minimizing performance impact. Using the Enron Email Dataset, we demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning, outperforming deduplication methods while maintaining only a 5.73% increase in perplexity. We present MaxTER, a Pareto-optimal evaluation framework for assessing privacy-utility tradeoffs, and show the performance of RMFT vs Deduplication by Area Under The Response Curve (AURC) metric.
• 摘要：当前有关自然语言模型（尤其是大型语言模型（LLM））记忆的文献存在严重的安全和隐私风险，因为模型倾向于记住训练数据中的个人识别信息（PII）。我们引入了随机屏蔽微调 (RMFT)，这是一种新颖的隐私保护微调技术，可减少 PII 记忆，同时最大限度地减少性能影响。使用安然电子邮件数据集，我们证明，与基线微调相比，RMFT 的总提取率降低了 80.81%，可见提取率降低了 80.17%，优于重复数据删除方法，同时困惑度仅增加了 5.73%。我们提出了 MaxTER，一种用于评估隐私与效用权衡的帕累托最优评估框架，并按响应曲线下面积 (AURC) 指标展示 RMFT 与重复数据删除的性能。

Title: Modeling Topics and Sociolinguistic Variation in Code-Switched Discourse: Insights from Spanish-English and Spanish-Guaraní

• Authors: Nemika Tyagi, Nelvin Licona Guevara, Olga Kellert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03334
• Pdf URL: https://arxiv.org/pdf/2512.03334
• Copy Paste: [[2512.03334]] Modeling Topics and Sociolinguistic Variation in Code-Switched Discourse: Insights from Spanish-English and Spanish-Guaraní(https://arxiv.org/abs/2512.03334)
• Keywords: language model, llm
• Abstract: This study presents an LLM-assisted annotation pipeline for the sociolinguistic and topical analysis of bilingual discourse in two typologically distinct contexts: Spanish-English and Spanish-Guaraní. Using large language models, we automatically labeled topic, genre, and discourse-pragmatic functions across a total of 3,691 code-switched sentences, integrated demographic metadata from the Miami Bilingual Corpus, and enriched the Spanish-Guaraní dataset with new topic annotations. The resulting distributions reveal systematic links between gender, language dominance, and discourse function in the Miami data, and a clear diglossic division between formal Guaraní and informal Spanish in Paraguayan texts. These findings replicate and extend earlier interactional and sociolinguistic observations with corpus-scale quantitative evidence. The study demonstrates that large language models can reliably recover interpretable sociolinguistic patterns traditionally accessible only through manual annotation, advancing computational methods for cross-linguistic and low-resource bilingual research.
• 摘要：本研究提出了一个法学硕士辅助注释管道，用于在两种类型不同的背景下对双语话语进行社会语言学和主题分析：西班牙语-英语和西班牙语-瓜拉尼语。使用大型语言模型，我们在总共 3,691 个语码转换句子中自动标记了主题、流派和话语语用函数，集成了迈阿密双语语料库中的人口统计元数据，并通过新的主题注释丰富了西班牙语-瓜拉尼语数据集。由此产生的分布揭示了迈阿密数据中性别、语言优势和话语功能之间的系统联系，以及巴拉圭文本中正式瓜拉尼语和非正式西班牙语之间的明显双语区分。这些发现通过语料库规模的定量证据复制并扩展了早期的互动和社会语言学观察。该研究表明，大型语言模型可以可靠地恢复传统上只能通过手动注释才能访问的可解释的社会语言模式，从而推进跨语言和低资源双语研究的计算方法。

Title: PERCS: Persona-Guided Controllable Biomedical Summarization Dataset

• Authors: Rohan Charudatt Salvi, Chirag Chawla, Dhruv Jain, Swapnil Panigrahi, Md Shad Akhtar, Shweta Yadav
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03340
• Pdf URL: https://arxiv.org/pdf/2512.03340
• Copy Paste: [[2512.03340]] PERCS: Persona-Guided Controllable Biomedical Summarization Dataset(https://arxiv.org/abs/2512.03340)
• Keywords: language model
• Abstract: Automatic medical text simplification plays a key role in improving health literacy by making complex biomedical research accessible to diverse readers. However, most existing resources assume a single generic audience, overlooking the wide variation in medical literacy and information needs across user groups. To address this limitation, we introduce PERCS (Persona-guided Controllable Summarization), a dataset of biomedical abstracts paired with summaries tailored to four personas: Laypersons, Premedical Students, Non-medical Researchers, and Medical Experts. These personas represent different levels of medical literacy and information needs, emphasizing the need for targeted, audience-specific summarization. Each summary in PERCS was reviewed by physicians for factual accuracy and persona alignment using a detailed error taxonomy. Technical validation shows clear differences in readability, vocabulary, and content depth across personas. Along with describing the dataset, we benchmark four large language models on PERCS using automatic evaluation metrics that assess comprehensiveness, readability, and faithfulness, establishing baseline results for future research. The dataset, annotation guidelines, and evaluation materials are publicly available to support research on persona-specific communication and controllable biomedical summarization.
• 摘要：自动医学文本简化使复杂的生物医学研究可供不同的读者使用，在提高健康素养方面发挥着关键作用。然而，大多数现有资源假设单一的通用受众，忽视了不同用户群体的医学素养和信息需求的巨大差异。为了解决这一限制，我们引入了 PERCS（角色引导可控摘要），这是一个生物医学摘要数据集，并配有针对四种角色定制的摘要：外行、医学预科学生、非医学研究人员和医学专家。这些角色代表了不同水平的医学素养和信息需求，强调需要有针对性的、针对特定受众的总结。 PERCS 中的每个摘要均由医生使用详细的错误分类法审查事实准确性和角色一致性。技术验证显示不同角色在可读性、词汇量和内容深度方面存在明显差异。除了描述数据集之外，我们还使用自动评估指标对 PERCS 上的四种大型语言模型进行基准测试，评估全面性、可读性和可信度，为未来的研究建立基线结果。数据集、注释指南和评估材料是公开的，以支持特定角色通信和可控生物医学总结的研究。

Title: Idea-Gated Transformers: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning

• Authors: Darshan Fofadiya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.03343
• Pdf URL: https://arxiv.org/pdf/2512.03343
• Copy Paste: [[2512.03343]] Idea-Gated Transformers: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning(https://arxiv.org/abs/2512.03343)
• Keywords: language model, gpt, llm, prompt
• Abstract: Autoregressive Language Models (LLMs) trained on Next-Token Prediction (NTP) often suffer from ``Topic Drift'' where the generation wanders away from the initial prompt due to a reliance on local associations rather than global planning \citep{holtzman2019curious}. While scaling model size mitigates this \citep{brown2020language}, the fundamental myopia of the NTP objective remains. In this work, we introduce the Idea-Gated Transformer, a novel architecture that separates semantic planning from syntactic generation. We introduce an auxiliary ``Idea Head'' trained to predict the bag-of-words distribution for a future context window, creating a latent ``Concept Vector'' that actively gates the main vocabulary during generation. We propose a differentiable gating mechanism that suppresses semantically irrelevant tokens, effectively pruning the search space in real-time. Experiments on WikiText-103 demonstrate that while the Idea-Gated model achieves comparable validation perplexity to a standard GPT-2 baseline, it exhibits significantly superior Domain Retention. Qualitative and quantitative analysis reveals that the gating mechanism successfully locks generation into specific semantic clusters (e.g., Finance, Science) and resists associative drift, offering a parameter-efficient path toward more controllable language modeling.
• 摘要：在下一个标记预测 (NTP) 上训练的自回归语言模型 (LLM) 经常会遭受“主题漂移”的困扰，即由于依赖本地关联而不是全局规划，一代人偏离了最初的提示\citep{holtzman2019curious}。虽然缩放模型大小可以缓解这种情况，但 NTP 目标的基本短视仍然存在。在这项工作中，我们介绍了 Idea-Gated Transformer，这是一种将语义规划与句法生成分开的新颖架构。我们引入了一个辅助“想法头”，经过训练来预测未来上下文窗口的词袋分布，创建一个潜在的“概念向量”，在生成过程中主动门控主要词汇。我们提出了一种可微分的门控机制，可以抑制语义上不相关的标记，从而有效地实时修剪搜索空间。 WikiText-103 上的实验表明，虽然 Idea-Gated 模型实现了与标准 GPT-2 基线相当的验证困惑度，但它表现出了显着优越的域保留。定性和定量分析表明，门控机制成功地将生成锁定到特定的语义集群（例如金融、科学）并抵抗关联漂移，为更可控的语言建模提供了一条参数有效的路径。

Title: From Hypothesis to Premises: LLM-based Backward Logical Reasoning with Selective Symbolic Translation

• Authors: Qingchuan Li, Mingyue Cheng, Zirui Liu, Daoyu Wang, Yuting Zeng, Tongxuan Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03360
• Pdf URL: https://arxiv.org/pdf/2512.03360
• Copy Paste: [[2512.03360]] From Hypothesis to Premises: LLM-based Backward Logical Reasoning with Selective Symbolic Translation(https://arxiv.org/abs/2512.03360)
• Keywords: language model, llm
• Abstract: Logical reasoning is a core challenge in natural language understanding and a fundamental capability of artificial intelligence, underpinning scientific discovery, mathematical theorem proving, and complex decision-making. Despite the remarkable progress of large language models (LLMs), most current approaches still rely on forward reasoning paradigms, generating step-by-step rationales from premises to conclusions. However, such methods often suffer from redundant inference paths, hallucinated steps, and semantic drift, resulting in inefficient and unreliable reasoning. In this paper, we propose a novel framework, Hypothesis-driven Backward Logical Reasoning (HBLR). The core idea is to integrate confidence-aware symbolic translation with hypothesis-driven backward reasoning. In the translation phase, only high-confidence spans are converted into logical form, such as First-Order Logic (FOL), while uncertain content remains in natural language. A translation reflection module further ensures semantic fidelity by evaluating symbolic outputs and reverting lossy ones back to text when necessary. In the reasoning phase, HBLR simulates human deductive thinking by assuming the conclusion is true and recursively verifying its premises. A reasoning reflection module further identifies and corrects flawed inference steps, enhancing logical coherence. Extensive experiments on five reasoning benchmarks demonstrate that HBLR consistently outperforms strong baselines in both accuracy and efficiency.
• 摘要：逻辑推理是自然语言理解的核心挑战，也是人工智能的基本能力，支撑科学发现、数学定理证明和复杂决策。尽管大型语言模型（LLM）取得了显着进展，但大多数当前方法仍然依赖于前向推理范式，生成从前提到结论的逐步推理。然而，此类方法往往存在冗余推理路径、幻觉步骤和语义漂移等问题，导致推理效率低下且不可靠。在本文中，我们提出了一种新颖的框架，假设驱动的后向逻辑推理（HBLR）。核心思想是将置信感知的符号翻译与假设驱动的反向推理相结合。在翻译阶段，仅将高置信度跨度转换为逻辑形式，例如一阶逻辑（FOL），而不确定的内容仍保留在自然语言中。翻译反射模块通过评估符号输出并在必要时将有损输出恢复为文本来进一步确保语义保真度。在推理阶段，HBLR通过假设结论正确并递归验证其前提来模拟人类的演绎思维。推理反思模块进一步识别并纠正有缺陷的推理步骤，增强逻辑连贯性。对五个推理基准的大量实验表明，HBLR 在准确性和效率方面始终优于强大的基准。

Title: Characterizing Language Use in a Collaborative Situated Game

• Authors: Nicholas Tomlin, Naitian Zhou, Eve Fleisig, Liangyuan (Circle)Chen, Téa Wright, Lauren Vinh, Laura X. Ma, Seun Eisape, Ellie French, Tingting Du, Tianjiao Zhang, Alexander Koller, Alane Suhr
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03381
• Pdf URL: https://arxiv.org/pdf/2512.03381
• Copy Paste: [[2512.03381]] Characterizing Language Use in a Collaborative Situated Game(https://arxiv.org/abs/2512.03381)
• Keywords: chat
• Abstract: Cooperative video games, where multiple participants must coordinate by communicating and reasoning under uncertainty in complex environments, yield a rich source of language data. We collect the Portal Dialogue Corpus: a corpus of 11.5 hours of spoken human dialogue in the co-op mode of the popular Portal 2 virtual puzzle game, comprising 24.5K total utterances. We analyze player language and behavior, identifying a number of linguistic phenomena that rarely appear in most existing chitchat or task-oriented dialogue corpora, including complex spatial reference, clarification and repair, and ad-hoc convention formation. To support future analyses of language use in complex, situated, collaborative problem-solving scenarios, we publicly release the corpus, which comprises player videos, audio, transcripts, game state data, and both manual and automatic annotations of language data.
• 摘要：合作视频游戏中，多个参与者必须在复杂环境的不确定性下通过沟通和推理进行协调，产生丰富的语言数据源。我们收集了 Portal 对话语料库：流行的 Portal 2 虚拟益智游戏的合作模式下的 11.5 小时人类口语对话的语料库，总共包含 24.5K 条话语。我们分析玩家的语言和行为，识别出许多在大多数现有闲聊或任务导向的对话语料库中很少出现的语言现象，包括复杂的空间参考、澄清和修复，以及临时约定的形成。为了支持未来在复杂、情境、协作解决问题的场景中对语言使用的分析，我们公开发布了语料库，其中包括玩家视频、音频、文字记录、游戏状态数据以及语言数据的手动和自动注释。

Title: Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates

• Authors: Yixing Xu, Chao Li, Xuanwu Yin, Spandan Tiwari, Dong Li, Ashish Sirasao, Emad Barsoum
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03402
• Pdf URL: https://arxiv.org/pdf/2512.03402
• Copy Paste: [[2512.03402]] Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates(https://arxiv.org/abs/2512.03402)
• Keywords: language model, gpt, llm
• Abstract: Low-rank adaptation (LoRA) is one of the most popular methods among parameter-efficient fine-tuning (PEFT) methods to adapt pre-trained large language models (LLMs) to specific downstream tasks. However, the model trained based on LoRA often has an unsatisfactory performance due to its low-rank assumption. In this paper, we propose a novel method called Dual LoRA to improve the performance by incorporating an inductive bias into the original LoRA. Specifically, we separate low-rank matrices into two groups: the magnitude group to control whether or not and how far we should update a parameter and the direction group to decide whether this parameter should move forward or backward, to better simulate the parameter updating process of the full fine-tuning based on gradient-based optimization algorithms. We show that this can be simply achieved by adding a ReLU function to the magnitude group and a sign function to the direction group. We conduct several experiments over a wide range of NLP tasks, including natural language generation (NLG), understanding (NLU), and commonsense reasoning datasets on GPT-2, RoBERTa, DeBERTa, and LLaMA-1/2/3 as baseline models. The results show that we consistently outperform LoRA and its state-of-the-art variants with the same number of trainable parameters.
• 摘要：低秩适应（LoRA）是参数高效微调（PEFT）方法中最流行的方法之一，用于使预训练的大语言模型（LLM）适应特定的下游任务。然而，基于LoRA训练的模型往往由于其低秩假设而表现不理想。在本文中，我们提出了一种称为 Dual LoRA 的新颖方法，通过将归纳偏置纳入原始 LoRA 来提高性能。具体来说，我们将低秩矩阵分为两组：幅度组控制是否更新参数以及更新参数的程度，方向组决定该参数应该向前还是向后移动，以更好地模拟基于梯度优化算法的全微调的参数更新过程。我们证明，这可以通过向幅度组添加 ReLU 函数并向方向组添加符号函数来简单地实现。我们针对广泛的 NLP 任务进行了多项实验，包括自然语言生成 (NLG)、理解 (NLU) 以及以 GPT-2、RoBERTa、DeBERTa 和 LLaMA-1/2/3 作为基线模型的常识推理数据集。结果表明，在相同数量的可训练参数下，我们的性能始终优于 LoRA 及其最先进的变体。

Title: A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention

• Authors: Di Xiu, Hongyin Tang, Bolin Rong, Lizhi Yan, Jingang Wang, Yifan Lu, Xunliang Cai
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03494
• Pdf URL: https://arxiv.org/pdf/2512.03494
• Copy Paste: [[2512.03494]] A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention(https://arxiv.org/abs/2512.03494)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) are increasingly prevalent in the field of long-context modeling, however, their inference computational costs have become a critical bottleneck hindering the advancement of tasks such as agents and multimodal applications. This report conducts a preliminary investigation into the effectiveness and theoretical mechanisms of the Top-$k$ Attention mechanism during both the decoding and training phases. First, we validate the effectiveness of exact Top-$k$ Decoding through extensive experimentation. Experiments demonstrate that retaining only the pivotal Keys with the highest similarity to the Query as the context window during the decoding stage achieves performance comparable to, or even surpassing, full attention on downstream tasks such as HELMET and LongBench v2. Second, we further explore the native Top-$k$ Attention training strategy. Experiments confirm that ensuring the consistency between training and inference regarding Top-$k$ Attention operations facilitates the further unlocking of Top-$k$ Decoding's potential, thereby significantly enhancing model performance. Furthermore, considering the high computational complexity of exact Top-$k$ Attention, we investigate the impact of approximate Top-$k$ algorithm precision on downstream tasks. Our research confirms a positive correlation between downstream task performance and approximation fidelity, and we provide statistical evaluations of the Lightning Indexer's precision within the DeepSeek-V3.2-Exp model. Finally, this report provides a theoretical interpretation from the perspective of Entropy. Experimental observations indicate that models subjected to Top-$k$ Attention SFT exhibit a distinct phenomenon of entropy reduction in downstream tasks, which validates the hypothesis that low-entropy states are better adapted to Top-$k$ Decoding.
• 摘要：大型语言模型（LLM）在长上下文建模领域越来越普遍，然而，它们的推理计算成本已成为阻碍智能体和多模态应用等任务进步的关键瓶颈。本报告对Top-$k$ Attention机制在解码和训练阶段的有效性和理论机制进行了初步研究。首先，我们通过大量实验验证精确 Top-$k$ 解码的有效性。实验表明，在解码阶段仅保留与查询相似度最高的关键键作为上下文窗口，其性能可与甚至超越对 HELMET 和 LongBench v2 等下游任务的充分关注。其次，我们进一步探索原生的 Top-$k$ 注意力训练策略。实验证实，确保Top-$k$ Attention操作的训练和推理之间的一致性，有利于进一步释放Top-$k$ Decoding的潜力，从而显着提升模型性能。此外，考虑到精确 Top-$k$ Attention 的高计算复杂度，我们研究了近似 Top-$k$ 算法精度对下游任务的影响。我们的研究证实了下游任务性能和近似保真度之间存在正相关性，并且我们提供了 DeepSeek-V3.2-Exp 模型中 Lightning Indexer 精度的统计评估。最后，本报告从熵的角度提供了理论解释。实验观察表明，经过Top-$k$ Attention SFT的模型在下游任务中表现出明显的熵减少现象，这验证了低熵状态更适合Top-$k$ Decoding的假设。

Title: Understanding LLM Reasoning for Abstractive Summarization

• Authors: Haohan Yuan, Siu Cheung Hui, Haopeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03503
• Pdf URL: https://arxiv.org/pdf/2512.03503
• Copy Paste: [[2512.03503]] Understanding LLM Reasoning for Abstractive Summarization(https://arxiv.org/abs/2512.03503)
• Keywords: language model, llm
• Abstract: While the reasoning capabilities of Large Language Models (LLMs) excel in analytical tasks such as mathematics and code generation, their utility for abstractive summarization remains widely assumed but largely unverified. To bridge this gap, we first tailor general reasoning strategies to the summarization domain. We then conduct a systematic, large scale comparative study of 8 reasoning strategies and 3 Large Reasoning Models (LRMs) across 8 diverse datasets, assessing both summary quality and faithfulness. Our findings show that reasoning is not a universal solution and its effectiveness is highly dependent on the specific strategy and context. Specifically, we observe a trade-off between summary quality and factual faithfulness: explicit reasoning strategies tend to improve fluency at the expense of factual grounding, while implicit reasoning in LRMs exhibits the inverse pattern. Furthermore, increasing an LRM's internal reasoning budget does not improve, and can even hurt, factual consistency, suggesting that effective summarization demands faithful compression rather than creative over-thinking.
• 摘要：虽然大型语言模型 (LLM) 的推理能力在数学和代码生成等分析任务中表现出色，但其抽象概括的效用仍然被广泛假设，但在很大程度上未经验证。为了弥补这一差距，我们首先针对摘要领域定制通用推理策略。然后，我们对 8 个不同数据集的 8 种推理策略和 3 个大型推理模型 (LRM) 进行了系统、大规模的比较研究，评估摘要质量和可信度。我们的研究结果表明，推理并不是一个通用的解决方案，其有效性很大程度上取决于具体的策略和背景。具体来说，我们观察到摘要质量和事实忠实度之间的权衡：显式推理策略往往会以牺牲事实基础为代价来提高流畅性，而 LRM 中的隐式推理则表现出相反的模式。此外，增加 LRM 的内部推理预算不会改善甚至可能损害事实一致性，这表明有效的总结需要忠实的压缩而不是创造性的过度思考。

Title: Fine-grained Narrative Classification in Biased News Articles

• Authors: Zeba Afroz, Harsh Vardhan, Pawan Bhakuni, Aanchal Punia, Rajdeep Kumar, Md. Shad Akhtar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.03582
• Pdf URL: https://arxiv.org/pdf/2512.03582
• Copy Paste: [[2512.03582]] Fine-grained Narrative Classification in Biased News Articles(https://arxiv.org/abs/2512.03582)
• Keywords: gpt, prompt
• Abstract: Narratives are the cognitive and emotional scaffolds of propaganda. They organize isolated persuasive techniques into coherent stories that justify actions, attribute blame, and evoke identification with ideological camps. In this paper, we propose a novel fine-grained narrative classification in biased news articles. We also explore article-bias classification as the precursor task to narrative classification and fine-grained persuasive technique identification. We develop INDI-PROP, the first ideologically grounded fine-grained narrative dataset with multi-level annotation for analyzing propaganda in Indian news media. Our dataset INDI-PROP comprises 1,266 articles focusing on two polarizing socio-political events in recent times: CAA and the Farmers' protest. Each article is annotated at three hierarchical levels: (i) ideological article-bias (pro-government, pro-opposition, neutral), (ii) event-specific fine-grained narrative frames anchored in ideological polarity and communicative intent, and (iii) persuasive techniques. We propose FANTA and TPTC, two GPT-4o-mini guided multi-hop prompt-based reasoning frameworks for the bias, narrative, and persuasive technique classification. FANTA leverages multi-layered communicative phenomena by integrating information extraction and contextual framing for hierarchical reasoning. On the other hand, TPTC adopts systematic decomposition of persuasive cues via a two-stage approach. Our evaluation suggests substantial improvement over underlying baselines in each case.
• 摘要：叙事是宣传的认知和情感支架。他们将孤立的说服技巧组织成连贯的故事，为行动辩护，归咎责任，并唤起人们对意识形态阵营的认同。在本文中，我们提出了一种针对有偏见的新闻文章的新颖的细粒度叙事分类。我们还探索文章偏见分类作为叙事分类和细粒度说服技术识别的先驱任务。我们开发了 INDI-PROP，这是第一个基于意识形态的细粒度叙事数据集，具有多级注释，用于分析印度新闻媒体的宣传。我们的数据集 INDI-PROP 包含 1,266 篇文章，重点关注近期两个两极分化的社会政治事件：CAA 和农民抗议。每篇文章都按三个层次进行注释：（i）意识形态文章偏见（支持政府、支持反对派、中立），（ii）基于意识形态极性和交流意图的特定事件的细粒度叙事框架，以及（iii）说服技巧。我们提出了 FANTA 和 TPTC，这两个 GPT-4o-mini 引导的多跳基于提示的推理框架，用于偏见、叙述和说服技术分类。 FANTA 通过集成信息提取和上下文框架来进行分层推理，从而利用多层交流现象。另一方面，TPTC 通过两阶段方法对说服线索进行系统分解。我们的评估表明每种情况下的基本基线都有显着改善。

Title: AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment

• Authors: Ahmad Aghaebrahimian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.03634
• Pdf URL: https://arxiv.org/pdf/2512.03634
• Copy Paste: [[2512.03634]] AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment(https://arxiv.org/abs/2512.03634)
• Keywords: language model, hallucination
• Abstract: Large Language Models have significantly advanced natural language processing tasks, but remain prone to generating incorrect or misleading but plausible arguments. This issue, known as hallucination, is particularly concerning in high-stakes domains like clinical applications, where factual inaccuracies can have severe consequences. Existing evaluation metrics fail to adequately assess factual consistency and lack interpretability, making diagnosing and mitigating errors difficult. We propose an interpretable framework for factual consistency assessment for in-domain and open-domain texts to address these limitations. Our approach decomposes text into atomic facts and introduces a flexible, schema-free methodology. Unlike previous methods with an absolute metric, we incorporate a weighted metric to enhance factual evaluation. Additionally, we propose a mechanism to control assessment complexity in intricate domains. We benchmark our approach on popular general and clinical datasets and release our code to support fact-aware model training in future research.
• 摘要：大型语言模型具有显着先进的自然语言处理任务，但仍然容易生成不正确或误导性但合理的论点。这个问题被称为幻觉，在临床应用等高风险领域尤其令人担忧，在这些领域，事实不准确可能会造成严重后果。现有的评估指标无法充分评估事实的一致性并且缺乏可解释性，使得诊断和减少错误变得困难。我们提出了一个可解释的框架，用于域内和开放域文本的事实一致性评估，以解决这些限制。我们的方法将文本分解为原子事实，并引入了灵活的、无模式的方法。与以前使用绝对度量的方法不同，我们采用了加权度量来增强事实评估。此外，我们提出了一种控制复杂领域评估复杂性的机制。我们对流行的一般和临床数据集的方法进行基准测试，并发布我们的代码以支持未来研究中的事实感知模型训练。

Title: Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Context

• Authors: Beatrice Savoldi, Giuseppe Attanasio, Olga Gorodetskaya, Marta Marchiori Manerba, Elisa Bassignana, Silvia Casola, Matteo Negri, Tommaso Caselli, Luisa Bentivogli, Alan Ramponi, Arianna Muti, Nicoletta Balbo, Debora Nozza
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03671
• Pdf URL: https://arxiv.org/pdf/2512.03671
• Copy Paste: [[2512.03671]] Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Context(https://arxiv.org/abs/2512.03671)
• Keywords: chat
• Abstract: The rise of Artificial Intelligence (AI) language technologies, particularly generative AI (GenAI) chatbots accessible via conversational interfaces, is transforming digital interactions. While these tools hold societal promise, they also risk widening digital divides due to uneven adoption and low awareness of their limitations. This study presents the first comprehensive empirical mapping of GenAI adoption, usage patterns, and literacy in Italy, based on newly collected survey data from 1,906 Italian-speaking adults. Our findings reveal widespread adoption for both work and personal use, including sensitive tasks like emotional support and medical advice. Crucially, GenAI is supplanting other technologies to become a primary information source: this trend persists despite low user digital literacy, posing a risk as users struggle to recognize errors or misinformation. Moreover, we identify a significant gender divide -- particularly pronounced in older generations -- where women are half as likely to adopt GenAI and use it less frequently than men. While we find literacy to be a key predictor of adoption, it only partially explains this disparity, suggesting that other barriers are at play. Overall, our data provide granular insights into the multipurpose usage of GenAI, highlighting the dual need for targeted educational initiatives and further investigation into the underlying barriers to equitable participation that competence alone cannot explain.
• 摘要：人工智能 (AI) 语言技术的兴起，特别是可通过对话界面访问的生成式人工智能 (GenAI) 聊天机器人，正在改变数字交互。尽管这些工具具有社会前景，但由于采用不均匀且对其局限性认识不足，它们也面临着扩大数字鸿沟的风险。这项研究基于新收集的 1,906 名意大利语成年人的调查数据，首次对意大利 GenAI 的采用、使用模式和读写能力进行了全面的实证分析。我们的研究结果表明，它广泛应用于工作和个人用途，包括情感支持和医疗建议等敏感任务。至关重要的是，GenAI 正在取代其他技术，成为主要信息源：尽管用户数字素养较低，但这种趋势仍然存在，由于用户难以识别错误或错误信息，因此存在风险。此外，我们发现存在显着的性别差异——在老一辈中尤其明显——女性采用 GenAI 的可能性只有男性的一半，但使用频率却低于男性。虽然我们发现识字率是采用率的关键预测因素，但它只能部分解释这种差异，表明其他障碍也在发挥作用。总体而言，我们的数据为 GenAI 的多用途使用提供了细致的见解，强调了有针对性的教育举措的双重需求，以及进一步调查仅凭能力无法解释的公平参与的潜在障碍。

Title: Evaluating Hydro-Science and Engineering Knowledge of Large Language Models

• Authors: Shiruo Hu, Wenbo Shan, Yingjia Li, Zhiqi Wan, Xinpeng Yu, Yunjia Qi, Haotian Xia, Yang Xiao, Dingxiao Liu, Jiaru Wang, Chenxu Gong, Ruixi Zhang, Shuyue Wu, Shibo Cui, Chee Hui Lai, Wei Luo, Yubin He, Bin Xu, Jianshi Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03672
• Pdf URL: https://arxiv.org/pdf/2512.03672
• Copy Paste: [[2512.03672]] Evaluating Hydro-Science and Engineering Knowledge of Large Language Models(https://arxiv.org/abs/2512.03672)
• Keywords: language model, llm
• Abstract: Hydro-Science and Engineering (Hydro-SE) is a critical and irreplaceable domain that secures human water supply, generates clean hydropower energy, and mitigates flood and drought disasters. Featuring multiple engineering objectives, Hydro-SE is an inherently interdisciplinary domain that integrates scientific knowledge with engineering expertise. This integration necessitates extensive expert collaboration in decision-making, which poses difficulties for intelligence. With the rapid advancement of large language models (LLMs), their potential application in the Hydro-SE domain is being increasingly explored. However, the knowledge and application abilities of LLMs in Hydro-SE have not been sufficiently evaluated. To address this issue, we propose the Hydro-SE LLM evaluation benchmark (Hydro-SE Bench), which contains 4,000 multiple-choice questions. Hydro-SE Bench covers nine subfields and enables evaluation of LLMs in aspects of basic conceptual knowledge, engineering application ability, and reasoning and calculation ability. The evaluation results on Hydro-SE Bench show that the accuracy values vary among 0.74 to 0.80 for commercial LLMs, and among 0.41 to 0.68 for small-parameter LLMs. While LLMs perform well in subfields closely related to natural and physical sciences, they struggle with domain-specific knowledge such as industry standards and hydraulic structures. Model scaling mainly improves reasoning and calculation abilities, but there is still great potential for LLMs to better handle problems in practical engineering application. This study highlights the strengths and weaknesses of LLMs for Hydro-SE tasks, providing model developers with clear training targets and Hydro-SE researchers with practical guidance for applying LLMs.
• 摘要：水利科学与工程（Hydro-SE）是保障人类供水、产生清洁水电能源、减轻洪涝灾害的重要且不可替代的领域。 Hydro-SE 具有多个工程目标，本质上是一个跨学科领域，它将科学知识与工程专业知识相结合。这种整合需要专家在决策过程中进行广泛的协作，这给情报工作带来了困难。随着大语言模型（LLM）的快速发展，人们越来越多地探索它们在 Hydro-SE 领域的潜在应用。然而，法学硕士在 Hydro-SE 方面的知识和应用能力尚未得到充分评估。为了解决这个问题，我们提出了Hydro-SE LLM评估基准（Hydro-SE Bench），其中包含4,000道多项选择题。 Hydro-SE Bench涵盖9个子领域，可对LLM的基础概念知识、工程应用能力、推理计算能力等方面进行评估。 Hydro-SE Bench 的评估结果显示，商业法学硕士的准确度值在 0.74 至 0.80 之间，小参数法学硕士的准确度值在 0.41 至 0.68 之间。虽然法学硕士在与自然科学和物理科学密切相关的子领域表现良好，但他们在行业标准和水利结构等特定领域知识方面遇到了困难。模型缩放主要提高推理和计算能力，但法学硕士在更好地处理实际工程应用中的问题方面仍有很大潜力。这项研究强调了法学硕士在 Hydro-SE 任务中的优势和劣势，为模型开发人员提供了明确的培训目标，并为 Hydro-SE 研究人员提供了应用法学硕士的实用指导。

Title: Different types of syntactic agreement recruit the same units within large language models

• Authors: Daria Kryvosheieva, Andrea de Varda, Evelina Fedorenko, Greta Tuckute
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03676
• Pdf URL: https://arxiv.org/pdf/2512.03676
• Copy Paste: [[2512.03676]] Different types of syntactic agreement recruit the same units within large language models(https://arxiv.org/abs/2512.03676)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) can reliably distinguish grammatical from ungrammatical sentences, but how grammatical knowledge is represented within the models remains an open question. We investigate whether different syntactic phenomena recruit shared or distinct components in LLMs. Using a functional localization approach inspired by cognitive neuroscience, we identify the LLM units most responsive to 67 English syntactic phenomena in seven open-weight models. These units are consistently recruited across sentences containing the phenomena and causally support the models' syntactic performance. Critically, different types of syntactic agreement (e.g., subject-verb, anaphor, determiner-noun) recruit overlapping sets of units, suggesting that agreement constitutes a meaningful functional category for LLMs. This pattern holds in English, Russian, and Chinese; and further, in a cross-lingual analysis of 57 diverse languages, structurally more similar languages share more units for subject-verb agreement. Taken together, these findings reveal that syntactic agreement-a critical marker of syntactic dependencies-constitutes a meaningful category within LLMs' representational spaces.
• 摘要：大型语言模型（LLM）可以可靠地区分语法句子和非语法句子，但如何在模型中表示语法知识仍然是一个悬而未决的问题。我们研究不同的句法现象是否在法学硕士中招募共享或不同的组成部分。使用受认知神经科学启发的功能定位方法，我们确定了对七个开放权重模型中的 67 种英语句法现象最敏感的法学硕士单元。这些单元在包含现象的句子中一致地招募，并因果地支持模型的句法性能。重要的是，不同类型的句法一致性（例如，主语-动词、照应词、限定词-名词）招募了重叠的单元集，这表明一致性构成了法学硕士有意义的功能类别。这种模式适用于英语、俄语和汉语；此外，在对 57 种不同语言的跨语言分析中，结构上更相似的语言共享更多的主谓一致单位。总而言之，这些发现表明，句法一致（句法依赖性的关键标志）构成了法学硕士表征空间中的一个有意义的类别。

Title: DZ-TDPO: Non-Destructive Temporal Alignment for Mutable State Tracking in Long-Context Dialogue

• Authors: Yijun Liao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03704
• Pdf URL: https://arxiv.org/pdf/2512.03704
• Copy Paste: [[2512.03704]] DZ-TDPO: Non-Destructive Temporal Alignment for Mutable State Tracking in Long-Context Dialogue(https://arxiv.org/abs/2512.03704)
• Keywords: chat
• Abstract: Long-context dialogue systems suffer from State Inertia, where static constraints prevent models from resolving conflicts between evolving user intents and established historical context. To address this, we propose DZ-TDPO, a non-destructive alignment framework that synergizes conflict-aware dynamic KL constraints with a learnable temporal attention bias. Experiments on the Multi-Session Chat (MSC) dataset demonstrate that DZ-TDPO achieves state-of-the-art win rates (86.2% on Phi-3.5) while maintaining robust zero-shot generalization. Crucially, our scaling analysis reveals a "Capacity-Stability Trade-off": while smaller models incur an "alignment tax" (perplexity surge) to overcome historical inertia, the larger Qwen2.5-7B model achieves near-perfect alignment (99.4% win rate) with negligible perplexity overhead. This confirms that TAI can be alleviated via precise attention regulation rather than destructive weight updates, preserving general capabilities (MMLU) across model scales. Code and data are available: this https URL
• 摘要：长上下文对话系统受到状态惯性的影响，其中静态约束阻止模型解决不断变化的用户意图与既定历史上下文之间的冲突。为了解决这个问题，我们提出了 DZ-TDPO，这是一种非破坏性对齐框架，它将冲突感知动态 KL 约束与可学习的时间注意力偏差相结合。多会话聊天 (MSC) 数据集上的实验表明，DZ-TDPO 实现了最先进的胜率（在 Phi-3.5 上为 86.2%），同时保持了强大的零样本泛化能力。至关重要的是，我们的扩展分析揭示了“容量与稳定性的权衡”：虽然较小的模型会产生“对齐税”（困惑度激增）来克服历史惯性，但较大的 Qwen2.5-7B 模型实现了近乎完美的对齐（99.4% 胜率），而困惑度开销可以忽略不计。这证实了 TAI 可以通过精确的注意力调节而不是破坏性的权重更新来缓解，从而在模型尺度上保留通用能力（MMLU）。代码和数据可用：此 https URL

Title: AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation

• Authors: Chuyue Wang, Jie Feng, Yuxi Wu, Hang Zhang, Zhiguo Fan, Bing Cheng, Wei Lin
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2512.03737
• Pdf URL: https://arxiv.org/pdf/2512.03737
• Copy Paste: [[2512.03737]] AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation(https://arxiv.org/abs/2512.03737)
• Keywords: language model, llm, hallucination
• Abstract: Accurate and reliable search on online healthcare platforms is critical for user safety and service efficacy. Traditional methods, however, often fail to comprehend complex and nuanced user queries, limiting their effectiveness. Large language models (LLMs) present a promising solution, offering powerful semantic understanding to bridge this gap. Despite their potential, deploying LLMs in this high-stakes domain is fraught with challenges, including factual hallucinations, specialized knowledge gaps, and high operational costs. To overcome these barriers, we introduce \textbf{AR-Med}, a novel framework for \textbf{A}utomated \textbf{R}elevance assessment for \textbf{Med}ical search that has been successfully deployed at scale on the Online Medical Delivery Platforms. AR-Med grounds LLM reasoning in verified medical knowledge through a retrieval-augmented approach, ensuring high accuracy and reliability. To enable efficient online service, we design a practical knowledge distillation scheme that compresses large teacher models into compact yet powerful student models. We also introduce LocalQSMed, a multi-expert annotated benchmark developed to guide model iteration and ensure strong alignment between offline and online performance. Extensive experiments show AR-Med achieves an offline accuracy of over 93\%, a 24\% absolute improvement over the original online system, and delivers significant gains in online relevance and user satisfaction. Our work presents a practical and scalable blueprint for developing trustworthy, LLM-powered systems in real-world healthcare applications.
• 摘要：在线医疗平台上准确可靠的搜索对于用户安全和服务效率至关重要。然而，传统方法通常无法理解复杂且细致的用户查询，从而限制了其有效性。大型语言模型 (LLM) 提供了一种有前景的解决方案，提供强大的语义理解来弥补这一差距。尽管法学硕士具有潜力，但在这个高风险领域部署法学硕士仍充满挑战，包括事实幻觉、专业知识差距和高昂的运营成本。为了克服这些障碍，我们引入了 \textbf{AR-Med}，这是一种用于 \textbf{A} 自动化 \textbf{R} 医疗搜索高度评估的新颖框架，该框架已成功在在线医疗交付平台上大规模部署。 AR-Med 通过检索增强方法将法学硕士推理基于经过验证的医学知识，确保高度准确性和可靠性。为了实现高效的在线服务，我们设计了一种实用的知识蒸馏方案，将大型教师模型压缩为紧凑而强大的学生模型。我们还引入了 LocalQSMed，这是一个多专家注释基准，旨在指导模型迭代并确保离线和在线性能之间的紧密一致性。大量实验表明，AR-Med 的离线准确率超过 93%，比原始在线系统绝对提高了 24%，并在在线相关性和用户满意度方面带来了显着提升。我们的工作为在现实世界的医疗保健应用中开发值得信赖的、由法学硕士支持的系统提供了一个实用且可扩展的蓝图。

Title: Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective

• Authors: Jingyang Ou, Jiaqi Han, Minkai Xu, Shaoxuan Xu, Jianwen Xie, Stefano Ermon, Yi Wu, Chongxuan Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03759
• Pdf URL: https://arxiv.org/pdf/2512.03759
• Copy Paste: [[2512.03759]] Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective(https://arxiv.org/abs/2512.03759)
• Keywords: language model, llm
• Abstract: Reinforcement Learning (RL) has proven highly effective for autoregressive language models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies in likelihood approximation: while autoregressive models naturally provide token-level conditional probabilities essential for token-level RL objectives (e.g., GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO-based Sequence-level Policy Optimization (ESPO), a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to ensure stable large-scale training. Extensive experiments on mathematical reasoning, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains on math and coding benchmarks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dLLMs. Our code is available at this https URL.
• 摘要：强化学习 (RL) 已被证明对于自回归语言模型非常有效，但将这些方法应用于扩散大语言模型 (dLLM) 提出了根本性的挑战。核心困难在于似然近似：虽然自回归模型自然地提供了令牌级 RL 目标（例如 GRPO）所必需的令牌级条件概率，但 dLLM 通过缺乏这种因式分解的迭代非自回归去噪步骤来生成序列。为了解决这种根本性的不匹配问题，我们提出了基于 ELBO 的序列级策略优化 (ESPO)，这是一种有原则的 RL 框架，它将整个序列生成视为单个操作，并使用 ELBO 作为易于处理的序列级似然代理。我们的方法结合了每个标记的重要性比归一化和稳健的 KL 散度估计，以确保稳定的大规模训练。针对数学推理、编码和规划任务的大量实验表明，ESPO 的性能显着优于代币级基准，在倒计时任务上实现了 20-40 分的显着改进，同时在数学和编码基准上保持一致的收益。我们的方法将序列级优化建立为 dLLM 中 RL 的原则性且经验有效的范例。我们的代码可以在这个 https URL 上找到。

Title: In-Context Representation Hijacking

• Authors: Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2512.03771
• Pdf URL: https://arxiv.org/pdf/2512.03771
• Copy Paste: [[2512.03771]] In-Context Representation Hijacking(https://arxiv.org/abs/2512.03771)
• Keywords: language model, llm, prompt
• Abstract: We introduce \textbf{Doublespeak}, a simple \emph{in-context representation hijacking} attack against large language models (LLMs). The attack works by systematically replacing a harmful keyword (e.g., \textit{bomb}) with a benign token (e.g., \textit{carrot}) across multiple in-context examples, provided a prefix to a harmful request. We demonstrate that this substitution leads to the internal representation of the benign token converging toward that of the harmful one, effectively embedding the harmful semantics under a euphemism. As a result, superficially innocuous prompts (e.g., ``How to build a carrot?'') are internally interpreted as disallowed instructions (e.g., ``How to build a bomb?''), thereby bypassing the model's safety alignment. We use interpretability tools to show that this semantic overwrite emerges layer by layer, with benign meanings in early layers converging into harmful semantics in later ones. Doublespeak is optimization-free, broadly transferable across model families, and achieves strong success rates on closed-source and open-source systems, reaching 74\% ASR on Llama-3.3-70B-Instruct with a single-sentence context override. Our findings highlight a new attack surface in the latent space of LLMs, revealing that current alignment strategies are insufficient and should instead operate at the representation level.
• 摘要：我们引入了 \textbf{Doublespeak}，这是针对大型语言模型 (LLM) 的简单 \emph{上下文表示劫持} 攻击。该攻击的工作原理是在多个上下文示例中系统地用良性标记（例如 \textit{carrot}）替换有害关键字（例如 \textit{bomb}），并为有害请求提供前缀。我们证明，这种替换导致良性标记的内部表示向有害标记的内部表示收敛，从而有效地将有害语义嵌入到委婉语中。因此，表面上无害的提示（例如，“如何建造胡萝卜？”）在内部被解释为不允许的指令（例如，“如何建造炸弹？”），从而绕过模型的安全对齐。我们使用可解释性工具来表明这种语义覆盖是逐层出现的，早期层中的良性含义在后面层中汇聚成有害语义。 Doublespeak 无需优化，可以广泛地跨模型系列迁移，并且在闭源和开源系统上取得了很高的成功率，在 Llama-3.3-70B-Instruct 上通过单句上下文覆盖达到了 74% ASR。我们的研究结果强调了法学硕士潜在空间中的一个新的攻击面，表明当前的对齐策略是不够的，应该在表征层面上操作。

Title: Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5

• Authors: Huey Sun, Anabel Yong, Lorenzo Gilly, Felipe Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03803
• Pdf URL: https://arxiv.org/pdf/2512.03803
• Copy Paste: [[2512.03803]] Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5(https://arxiv.org/abs/2512.03803)
• Keywords: language model
• Abstract: Contrastive decoding is a lightweight and effective inference-time method that improves the quality of text generation in Large Language Models. However, algorithms such as DoLa (Decoding by Contrastive Layers) have only been implemented in decoder-only architectures and studied for their impact on improving factuality. This work adapts DoLa for the T5 and FLAN-T5 model families and evaluates its impact on the models' instruction following capabilities, which to our knowledge is the first implementation of a contrastive decoding strategy in an encoder-decoder architecture. Our results show that DoLa improves the faithfulness of text generation for certain categories of tasks and harms others. To understand these results, we present a layer-by-layer analysis of logit evolution in a FLAN-T5 model to quantify DoLa's impact on token output probabilities.
• 摘要：对比解码是一种轻量级且有效的推理时间方法，可以提高大型语言模型中文本生成的质量。然而，诸如 DoLa（对比层解码）之类的算法仅在纯解码器架构中实现，并研究了它们对提高真实性的影响。这项工作将 DoLa 应用于 T5 和 FLAN-T5 模型系列，并评估其对模型指令跟踪能力的影响，据我们所知，这是编码器-解码器架构中对比解码策略的首次实现。我们的结果表明，DoLa 提高了某些类别任务的文本生成的可信度，但会损害其他任务。为了理解这些结果，我们在 FLAN-T5 模型中对 Logit 演化进行了逐层分析，以量化 DoLa 对令牌输出概率的影响。

Title: Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology

• Authors: Kylie L. Anglin, Stephanie Milan, Brittney Hernandez, Claudia Ventura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03818
• Pdf URL: https://arxiv.org/pdf/2512.03818
• Copy Paste: [[2512.03818]] Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology(https://arxiv.org/abs/2512.03818)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong text classification performance. However, LLM output - here, the category assigned to a text - depends heavily on the wording of the prompt. While literature on prompt engineering is expanding, few studies focus on classification tasks, and even fewer address domains like psychology, where constructs have precise, theory-driven definitions that may not be well represented in pre-training data. We present an empirical framework for optimizing LLM performance for identifying constructs in texts via prompt engineering. We experimentally evaluate five prompting strategies --codebook-guided empirical prompt selection, automatic prompt engineering, persona prompting, chain-of-thought reasoning, and explanatory prompting - with zero-shot and few-shot classification. We find that persona, chain-of-thought, and explanations do not fully address performance loss accompanying a badly worded prompt. Instead, the most influential features of a prompt are the construct definition, task framing, and, to a lesser extent, the examples provided. Across three constructs and two models, the classifications most aligned with expert judgments resulted from a few-shot prompt combining codebook-guided empirical prompt selection with automatic prompt engineering. Based on our findings, we recommend that researchers generate and evaluate as many prompt variants as feasible, whether human-crafted, automatically generated, or ideally both, and select prompts and examples based on empirical performance in a training dataset, validating the final approach in a holdout set. This procedure offers a practical, systematic, and theory-driven method for optimizing LLM prompts in settings where alignment with expert judgment is critical.
• 摘要：由于其架构和大量的预训练数据，大型语言模型 (LLM) 表现出强大的文本分类性能。然而，LLM 的输出——这里指的是分配给文本的类别——在很大程度上取决于提示的措辞。虽然有关即时工程的文献不断增加，但很少有研究关注分类任务，而涉及心理学等领域的研究就更少了，在这些领域中，结构具有精确的、理论驱动的定义，而这些定义可能无法在预训练数据中得到很好的体现。我们提出了一个经验框架，用于优化法学硕士性能，通过即时工程识别文本中的结构。我们通过零样本和少样本分类实验评估了五种提示策略——密码本引导的经验提示选择、自动提示工程、角色提示、思维链推理和解释性提示。我们发现人物角色、思维链和解释并不能完全解决措辞不当的提示所带来的性能损失。相反，提示中最有影响力的特征是结构定义、任务框架，以及在较小程度上提供的示例。在三个构造和两个模型中，最符合专家判断的分类是由将密码本引导的经验提示选择与自动提示工程相结合的几次提示得出的。根据我们的发现，我们建议研究人员生成和评估尽可能多的提示变体，无论是人工生成的、自动生成的还是理想情况下两者兼而有之，并根据训练数据集中的经验表现选择提示和示例，验证保留集中的最终方法。该程序提供了一种实用的、系统的、理论驱动的方法，用于在与专家判断一致至关重要的环境中优化法学硕士提示。

Title: Training and Evaluation of Guideline-Based Medical Reasoning in LLMs

• Authors: Michael Staniek, Artem Sokolov, Stefan Riezler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03838
• Pdf URL: https://arxiv.org/pdf/2512.03838
• Copy Paste: [[2512.03838]] Training and Evaluation of Guideline-Based Medical Reasoning in LLMs(https://arxiv.org/abs/2512.03838)
• Keywords: llm, prompt
• Abstract: Machine learning for early prediction in medicine has recently shown breakthrough performance, however, the focus on improving prediction accuracy has led to a neglect of faithful explanations that are required to gain the trust of medical practitioners. The goal of this paper is to teach LLMs to follow medical consensus guidelines step-by-step in their reasoning and prediction process. Since consensus guidelines are ubiquitous in medicine, instantiations of verbalized medical inference rules to electronic health records provide data for fine-tuning LLMs to learn consensus rules and possible exceptions thereof for many medical areas. Consensus rules also enable an automatic evaluation of the model's inference process regarding its derivation correctness (evaluating correct and faithful deduction of a conclusion from given premises) and value correctness (comparing predicted values against real-world measurements). We exemplify our work using the complex Sepsis-3 consensus definition. Our experiments show that small fine-tuned models outperform one-shot learning of considerably larger LLMs that are prompted with the explicit definition and models that are trained on medical texts including consensus definitions. Since fine-tuning on verbalized rule instantiations of a specific medical area yields nearly perfect derivation correctness for rules (and exceptions) on unseen patient data in that area, the bottleneck for early prediction is not out-of-distribution generalization, but the orthogonal problem of generalization into the future by forecasting sparsely and irregularly sampled clinical variables. We show that the latter results can be improved by integrating the output representations of a time series forecasting model with the LLM in a multimodal setup.
• 摘要：用于医学早期预测的机器学习最近显示出突破性的性能，然而，对提高预测准确性的关注导致忽视了获得医生信任所需的忠实解释。本文的目标是教会法学硕士在推理和预测过程中逐步遵循医学共识指南。由于共识指南在医学中无处不在，因此电子健康记录的口头医学推理规则的实例化为法学硕士的微调提供了数据，以了解许多医学领域的共识规则及其可能的例外情况。共识规则还可以自动评估模型的推论过程，即其推导正确性（评估从给定前提得出结论的正确和忠实的推论）和值正确性（将预测值与实际测量值进行比较）。我们使用复杂的 Sepsis-3 共识定义来举例说明我们的工作。我们的实验表明，小型微调模型的性能优于相当大的法学硕士的一次性学习，这些法学硕士由明确的定义和在包括共识定义在内的医学文本上训练的模型来提示。由于对特定医疗领域的语言化规则实例进行微调可以为该领域未见过的患者数据的规则（和例外）产生近乎完美的推导正确性，因此早期预测的瓶颈不是分布外泛化，而是通过预测稀疏和不规则采样的临床变量来泛化到未来的正交问题。我们表明，通过将时间序列预测模型的输出表示与多模态设置中的 LLM 相集成，可以改进后者的结果。

Title: Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers

• Authors: Hongzhan Lin, Zhiqi Bai, Xinmiao Zhang, Sen Yang, Xiang Li, Siran Yang, Yunlong Xu, Jiaheng Liu, Yongchi Zhao, Jiamang Wang, Yuchi Xu, Wenbo Su, Bo Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03870
• Pdf URL: https://arxiv.org/pdf/2512.03870
• Copy Paste: [[2512.03870]] Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers(https://arxiv.org/abs/2512.03870)
• Keywords: llm
• Abstract: Transformer decoders have achieved strong results across tasks, but the memory required for the KV cache becomes prohibitive at long sequence lengths. Although Cross-layer KV Cache sharing (e.g., YOCO, CLA) offers a path to mitigate KV Cache bottleneck, it typically underperforms within-layer methods like GQA. To understand the root cause, we investigate the information flow of keys and values of the top-layers. Our preliminary reveals a clear distribution: values are predominantly derived from the bottom layer, while keys draw more information from both bottom and middle layers. Building upon this, we propose FusedKV, whose top-layer KV caches are a learnable fusion of the most informative ones from the bottom and middle layers. This fusion operates directly on post-RoPE keys, preserving relative positional information without the computational cost of re-applying rotary embeddings. To further improve efficiency, we propose FusedKV-Lite, an cross-layer sharing approach, where top-layer KV caches are directly derived from the bottom-layer values and the middle-layer keys. Compared to FusedKV, FusedKV-Lite reduces I/O overhead at the cost of a slight increase in perplexity. In experiments on LLMs ranging from 332M to 4B parameters, our proposed method reduce 50\% cache memory while achieving lower validation perplexity than the standard Transformer decoder, establishing it as a memory-efficient, high-performance architectural alternative.
• 摘要：Transformer 解码器在跨任务方面取得了出色的成果，但 KV 缓存所需的内存在长序列长度下变得令人望而却步。尽管跨层 KV 缓存共享（例如 YOCO、CLA）提供了缓解 KV 缓存瓶颈的途径，但它通常不如 GQA 等层内方法。为了了解根本原因，我们研究了顶层键和值的信息流。我们的初步揭示了一个清晰的分布：值主要来自底层，而键则从底层和中间层获取更多信息。在此基础上，我们提出了 FusedKV，其顶层 KV 缓存是底层和中间层信息最丰富的缓存的可学习融合。这种融合直接在 RoPE 后的密钥上运行，保留相对位置信息，而无需重新应用旋转嵌入的计算成本。为了进一步提高效率，我们提出了 FusedKV-Lite，一种跨层共享方法，其中顶层 KV 缓存直接从底层值和中间层键派生。与 FusedKV 相比，FusedKV-Lite 减少了 I/O 开销，但代价是困惑度略有增加。在从 332M 到 4B 参数范围的 LLM 实验中，我们提出的方法减少了 50% 的缓存，同时实现了比标准 Transformer 解码器更低的验证困惑，将其确立为内存高效、高性能的架构替代方案。

Title: BERnaT: Basque Encoders for Representing Natural Textual Diversity

• Authors: Ekhi Azurmendi, Joseba Fernandez de Landa, Jaione Bengoetxea, Maite Heredia, Julen Etxaniz, Mikel Zubillaga, Ander Soraluze, Aitor Soroa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.03903
• Pdf URL: https://arxiv.org/pdf/2512.03903
• Copy Paste: [[2512.03903]] BERnaT: Basque Encoders for Representing Natural Textual Diversity(https://arxiv.org/abs/2512.03903)
• Keywords: language model
• Abstract: Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally exclude non-standard linguistic varieties, reduce model robustness and reinforce representational biases. In this paper, we argue that language models should aim to capture the full spectrum of language variation (dialectal, historical, informal, etc.) rather than relying solely on standardized text. Focusing on Basque, a morphologically rich and low-resource language, we construct new corpora combining standard, social media, and historical sources, and pre-train the BERnaT family of encoder-only models in three configurations: standard, diverse, and combined. We further propose an evaluation framework that separates Natural Language Understanding (NLU) tasks into standard and diverse subsets to assess linguistic generalization. Results show that models trained on both standard and diverse data consistently outperform those trained on standard corpora, improving performance across all task types without compromising standard benchmark accuracy. These findings highlight the importance of linguistic diversity in building inclusive, generalizable language models.
• 摘要：语言模型依赖于大量文本语料库，这些文本语料库通常会进行质量过滤，这一过程可能会无意中排除非标准语言品种，降低模型的稳健性并强化表征偏差。在本文中，我们认为语言模型的目标应该是捕捉全部语言变异（方言、历史、非正式等），而不是仅仅依赖标准化文本。我们以巴斯克语这种形态丰富且资源匮乏的语言为重点，结合标准、社交媒体和历史来源构建了新的语料库，并以三种配置预训练了 BERnaT 系列仅编码器模型：标准、多样化和组合。我们进一步提出了一个评估框架，将自然语言理解（NLU）任务分为标准和多样化的子集，以评估语言泛化。结果表明，在标准和多样化数据上训练的模型始终优于在标准语料库上训练的模型，在不影响标准基准准确性的情况下提高了所有任务类型的性能。这些发现强调了语言多样性在构建包容性、可推广的语言模型中的重要性。

Title: Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions

• Authors: Kazi Abrab Hossain, Jannatul Somiya Mahmud, Maria Hossain Tuli, Anik Mitra, S. M. Taiabul Haque, Farig Y. Sadeque
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2512.03943
• Pdf URL: https://arxiv.org/pdf/2512.03943
• Copy Paste: [[2512.03943]] Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions(https://arxiv.org/abs/2512.03943)
• Keywords: language model, prompt
• Abstract: While recent developments in large language models have improved bias detection and classification, sensitive subjects like religion still present challenges because even minor errors can result in severe misunderstandings. In particular, multilingual models often misrepresent religions and have difficulties being accurate in religious contexts. To address this, we introduce BRAND: Bilingual Religious Accountable Norm Dataset, which focuses on the four main religions of South Asia: Buddhism, Christianity, Hinduism, and Islam, containing over 2,400 entries, and we used three different types of prompts in both English and Bengali. Our results indicate that models perform better in English than in Bengali and consistently display bias toward Islam, even when answering religion-neutral questions. These findings highlight persistent bias in multilingual models when similar questions are asked in different languages. We further connect our findings to the broader issues in HCI regarding religion and spirituality.
• 摘要：尽管大型语言模型的最新发展改进了偏差检测和分类，但宗教等敏感主题仍然面临挑战，因为即使是很小的错误也可能导致严重的误解。特别是，多语言模型经常歪曲宗教，并且很难在宗教背景下准确表达。为了解决这个问题，我们引入了 BRAND：双语宗教责任规范数据集，该数据集关注南亚四大主要宗教：佛教、基督教、印度教和伊斯兰教，包含 2,400 多个条目，我们使用了英语和孟加拉语三种不同类型的提示。我们的结果表明，模型在英语中的表现比在孟加拉语中的表现更好，并且即使在回答宗教中立的问题时也始终表现出对伊斯兰教的偏见。这些发现凸显了当用不同语言提出类似问题时，多语言模型中持续存在的偏见。我们进一步将我们的发现与人机交互中有关宗教和灵性的更广泛问题联系起来。

Title: Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study

• Authors: Lifeng Chen, Ryan Lai, Tianming Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03976
• Pdf URL: https://arxiv.org/pdf/2512.03976
• Copy Paste: [[2512.03976]] Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study(https://arxiv.org/abs/2512.03976)
• Keywords: language model, llm
• Abstract: Adapting large language models (LLMs) to low-resource languages remains a major challenge due to data scarcity and cross-lingual drift. This work presents a two-stage adaptation of Qwen2.5-3B to Tibetan, a morphologically rich and underrepresented language. We employ Continual Pretraining (CPT) to establish Tibetan linguistic grounding, followed by Supervised Fine-Tuning (SFT) for task and translation specialization. Empirical evaluations demonstrate a consistent decrease in perplexity (from 2.98 $\rightarrow$ 1.54) and substantial improvements in Chinese$\rightarrow$Tibetan translation quality (BLEU: 0.046 $\rightarrow$ 0.261; chrF: 2.2 $\rightarrow$ 6.6). Layer-wise analysis across 435 layers in Qwen3-4B reveals that adaptation primarily concentrates on embedding and output heads, with mid--late MLP projections encoding domain-specific transformations. Our findings suggest that CPT constructs a Tibetan semantic manifold while SFT sharpens task alignment with minimal representational disruption. This study provides the first quantitative exploration of Tibetan adaptation dynamics for LLMs, and offers an open, reproducible framework for extending multilingual foundation models to low-resource settings.
• 摘要：由于数据稀缺和跨语言漂移，使大型语言模型（LLM）适应资源匮乏的语言仍然是一个重大挑战。这项工作提出了 Qwen2.5-3B 对藏语的两阶段适应，藏语是一种形态丰富且代表性不足的语言。我们采用持续预训练（CPT）来建立藏语语言基础，然后采用监督微调（SFT）来实现任务和翻译专业化。实证评估表明，困惑度持续下降（从 2.98 $\rightarrow$ 1.54），中文$\rightarrow$藏文翻译质量显着提高（BLEU：0.046 $\rightarrow$ 0.261；chrF：2.2 $\rightarrow$ 6.6）。 Qwen3-4B 中 435 层的逐层分析表明，适应主要集中在嵌入和输出头，中后期 MLP 投影编码特定于域的转换。我们的研究结果表明，CPT 构建了藏语语义流形，而 SFT 则以最小的表征干扰强化了任务对齐。这项研究首次对法学硕士的西藏适应动态进行了定量探索，并提供了一个开放的、可重复的框架，用于将多语言基础模型扩展到资源匮乏的环境。

Title: Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models

• Authors: Taido Purason, Pavel Chizhov, Ivan P. Yamshchikov, Mark Fishel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.03989
• Pdf URL: https://arxiv.org/pdf/2512.03989
• Copy Paste: [[2512.03989]] Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models(https://arxiv.org/abs/2512.03989)
• Keywords: language model
• Abstract: Tokenizer adaptation plays an important role in transferring pre-trained language models to new domains or languages. In this work, we address two complementary aspects of this process: vocabulary extension and pruning. The common approach to extension trains a new tokenizer on domain-specific text and appends the tokens that do not overlap with the existing vocabulary, which often results in many tokens that are unreachable or never used. We propose continued BPE training, which adapts a pre-trained tokenizer by continuing the BPE merge learning process on new data. Experiments across multiple languages and model families show that this approach improves tokenization efficiency and leads to better utilization of added vocabulary. We also introduce leaf-based vocabulary pruning, which removes redundant tokens while preserving model quality. Together, these methods provide practical tools for controlled vocabulary modification, which we release as an open-source package.
• 摘要：分词器适应在将预先训练的语言模型转移到新领域或语言方面发挥着重要作用。在这项工作中，我们解决了这个过程的两个互补方面：词汇扩展和修剪。扩展的常见方法是在特定于域的文本上训练新的标记生成器，并附加与现有词汇表不重叠的标记，这通常会导致许多标记无法访问或从未使用过。我们建议继续进行 BPE 训练，通过继续对新数据进行 BPE 合并学习过程来适应预先训练的分词器。跨多种语言和模型系列的实验表明，这种方法提高了标记化效率，并可以更好地利用添加的词汇。我们还引入了基于叶子的词汇修剪，它可以在保持模型质量的同时删除冗余标记。这些方法共同提供了用于受控词汇修改的实用工具，我们将其作为开源包发布。

Title: AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving

• Authors: Ying Wang, Zhen Jin, Jiexiong Xu, Wenhai Lin, Yiquan Chen, Wenzhi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2512.04013
• Pdf URL: https://arxiv.org/pdf/2512.04013
• Copy Paste: [[2512.04013]] AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving(https://arxiv.org/abs/2512.04013)
• Keywords: language model, llm
• Abstract: As augmented large language models (LLMs) with external tools become increasingly popular in web applications, improving augmented LLM inference serving efficiency and optimizing service-level objectives (SLOs) are critical for enhancing user experience. To achieve this, inference systems must maximize request handling within latency constraints, referred to as increasing effective throughput. However, existing systems face two major challenges: (i) reliance on first-come-first-served (FCFS) scheduling causes severe head-of-line blocking, leading to queuing delays exceeding the SLOs for many requests; and (ii) static batch token limit, which fails to adapt to fluctuating loads and hardware conditions. Both of these factors degrade effective throughput and service quality. This paper presents AugServe, an efficient inference framework designed to reduce queueing latency and enhance effective throughput for augmented LLM inference services. The core idea of AugServe is a two-stage adaptive request scheduling strategy. Specifically, AugServe combines the inference features of augmented LLM requests to optimize the order of scheduling decisions (stage I). These decisions are continuously refined with runtime information (stage II), adapting to both request characteristics and system capabilities. In addition, AugServe dynamically adjusts the token batching mechanism based on hardware status and real-time load, further enhancing throughput performance. Experimental results show that AugServe achieves 4.7-33.1x and 3.3-13.2x higher effective throughput than vLLM and InferCept, while reducing time-to-first-token (TTFT) by up to 96.3% and 95.0%, respectively.
• 摘要：随着带有外部工具的增强型大语言模型 (LLM) 在 Web 应用程序中变得越来越流行，提高增强型 LLM 推理服务效率和优化服务级别目标 (SLO) 对于增强用户体验至关重要。为了实现这一目标，推理系统必须在延迟限制内最大化请求处理，称为增加有效吞吐量。然而，现有系统面临两大挑战：（i）依赖先到先服务（FCFS）调度会导致严重的队头阻塞，导致许多请求的排队延迟超过 SLO； (ii) 静态批量令牌限制，无法适应波动的负载和硬件条件。这两个因素都会降低有效吞吐量和服务质量。本文介绍了 AugServe，这是一种高效的推理框架，旨在减少排队延迟并提高增强 LLM 推理服务的有效吞吐量。 AugServe的核心思想是两阶段自适应请求调度策略。具体来说，AugServe 结合了增强 LLM 请求的推理功能来优化调度决策的顺序（第一阶段）。这些决策根据运行时信息（第二阶段）不断完善，以适应请求特征和系统功能。此外，AugServe根据硬件状态和实时负载动态调整令牌批量机制，进一步增强吞吐量性能。实验结果表明，AugServe 的有效吞吐量比 vLLM 和 InferCept 高 4.7-33.1 倍和 3.3-13.2 倍，同时首次令牌时间 (TTFT) 分别减少高达 96.3% 和 95.0%。

Title: Jina-VLM: Small Multilingual Vision Language Model

• Authors: Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2512.04032
• Pdf URL: https://arxiv.org/pdf/2512.04032
• Copy Paste: [[2512.04032]] Jina-VLM: Small Multilingual Vision Language Model(https://arxiv.org/abs/2512.04032)
• Keywords: language model
• Abstract: We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. Across standard VQA benchmarks and multilingual evaluations, Jina-VLM outperforms comparable models while preserving competitive text-only performance.
• 摘要：我们提出了 Jina-VLM，这是一种 2.4B 参数视觉语言模型，可在开放的 2B 规模 VLM 中实现最先进的多语言视觉问答。该模型通过注意力池连接器将 SigLIP2 视觉编码器与 Qwen3 语言主干耦合起来，从而实现对任意分辨率图像的令牌高效处理。在标准 VQA 基准和多语言评估中，Jina-VLM 的性能优于同类模型，同时保持有竞争力的纯文本性能。

Title: SkillFactory: Self-Distillation For Learning Cognitive Behaviors

• Authors: Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, Greg Durrett
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2512.04072
• Pdf URL: https://arxiv.org/pdf/2512.04072
• Copy Paste: [[2512.04072]] SkillFactory: Self-Distillation For Learning Cognitive Behaviors(https://arxiv.org/abs/2512.04072)
• Keywords: language model
• Abstract: Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL; (2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
• 摘要：利用长思维链的推理模型采用各种认知技能，例如验证答案、回溯、通过替代方法重试等等。之前的工作表明，当基础语言模型表现出这些技能时，通过强化学习 (RL) 进一步训练该模型可以学会利用这些技能。我们怎样才能让模型利用基础模型没有展示的技能？我们的工作 SkillFactory 是一种微调模型的方法，可以在强化学习之前的监督微调 (SFT) 阶段粗略地学习这些技能。我们的方法不依赖于从更强大的模型中蒸馏出来，而是使用模型本身的样本，重新排列以提供这些技能格式的训练数据。这些“银色”SFT 轨迹可能并不完美，但对于启动模型在强化学习期间获取技能来说仍然有效。我们的评估表明，（1）从 SkillFactory SFT 初始化开始，可以帮助模型泛化到强化学习后任务的更难变体，尽管强化学习前的性能较低； (2) 模型确实使用了认知技能； (3) RLed SkillFactory 模型对于域外任务的回归比 RLed 基础模型更稳健。我们的工作表明，在强化学习之前学习的归纳偏差有助于模型学习强大的认知技能使用。