2025-11-24

Title: Towards Hyper-Efficient RAG Systems in VecDBs: Distributed Parallel Multi-Resolution Vector Search

• Authors: Dong Liu, Yanxuan Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16681
• Pdf URL: https://arxiv.org/pdf/2511.16681
• Copy Paste: [[2511.16681]] Towards Hyper-Efficient RAG Systems in VecDBs: Distributed Parallel Multi-Resolution Vector Search(https://arxiv.org/abs/2511.16681)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems have become a dominant approach to augment large language models (LLMs) with external knowledge. However, existing vector database (VecDB) retrieval pipelines rely on flat or single-resolution indexing structures, which cannot adapt to the varying semantic granularity required by diverse user queries. This limitation leads to suboptimal trade-offs between retrieval speed and contextual relevance. To address this, we propose \textbf{Semantic Pyramid Indexing (SPI)}, a novel multi-resolution vector indexing framework that introduces query-adaptive resolution control for RAG in VecDBs. Unlike existing hierarchical methods that require offline tuning or separate model training, SPI constructs a semantic pyramid over document embeddings and dynamically selects the optimal resolution level per query through a lightweight classifier. This adaptive approach enables progressive retrieval from coarse-to-fine representations, significantly accelerating search while maintaining semantic coverage. We implement SPI as a plugin for both FAISS and Qdrant backends and evaluate it across multiple RAG tasks including MS MARCO, Natural Questions, and multimodal retrieval benchmarks. SPI achieves up to \textbf{5.7$\times$} retrieval speedup and \textbf{1.8$\times$} memory efficiency gain while improving end-to-end QA F1 scores by up to \textbf{2.5 points} compared to strong baselines. Our theoretical analysis provides guarantees on retrieval quality and latency bounds, while extensive ablation studies validate the contribution of each component. The framework's compatibility with existing VecDB infrastructures makes it readily deployable in production RAG systems. Code is availabe at \href{this https URL}{this https URL\_VecDB}.
• 摘要：检索增强生成（RAG）系统已成为利用外部知识增强大型语言模型（LLM）的主要方法。然而，现有的矢量数据库（VecDB）检索管道依赖于平面或单分辨率索引结构，无法适应不同用户查询所需的不同语义粒度。这种限制导致检索速度和上下文相关性之间的权衡不理想。为了解决这个问题，我们提出了 \textbf{语义金字塔索引（SPI）}，这是一种新颖的多分辨率向量索引框架，为 VecDB 中的 RAG 引入了查询自适应分辨率控制。与需要离线调整或单独模型训练的现有分层方法不同，SPI 在文档嵌入上构建语义金字塔，并通过轻量级分类器动态选择每个查询的最佳分辨率级别。这种自适应方法可以实现从粗到细的表示的渐进检索，显着加速搜索，同时保持语义覆盖。我们将 SPI 作为 FAISS 和 Qdrant 后端的插件来实现，并在多个 RAG 任务中对其进行评估，包括 MS MARCO、自然问题和多模式检索基准。与强基线相比，SPI 实现了高达 \textbf{5.7$\times$} 的检索加速和 \textbf{1.8$\times$} 内存效率增益，同时将端到端 QA F1 分数提高了高达 \textbf{2.5 分}。我们的理论分析为检索质量和延迟范围提供了保证，而广泛的消融研究则验证了每个组件的贡献。该框架与现有 VecDB 基础设施的兼容性使其可以轻松部署在生产 RAG 系统中。代码可在 \href{此 https URL}{此 https URL\_VecDB} 处获取。

Title: Bench360: Benchmarking Local LLM Inference from 360°

• Authors: Linus Stuhlmann, Mauricio Fadel Argerich, Jonathan Fürst
• Subjects: cs.CL, cs.AI, cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2511.16682
• Pdf URL: https://arxiv.org/pdf/2511.16682
• Copy Paste: [[2511.16682]] Bench360: Benchmarking Local LLM Inference from 360°(https://arxiv.org/abs/2511.16682)
• Keywords: language model, llm
• Abstract: Running large language models (LLMs) locally is becoming increasingly common. While the growing availability of small open-source models and inference engines has lowered the entry barrier, users now face an overwhelming number of configuration choices. Identifying an optimal configuration -- balancing functional and non-functional requirements -- requires substantial manual effort. While several benchmarks target LLM inference, they are designed for narrow evaluation goals and not user-focused. They fail to integrate relevant system and task-specific metrics into a unified, easy-to-use benchmark that supports multiple inference engines, usage scenarios, and quantization levels. To address this gap, we present Bench360 -- Benchmarking Local LLM Inference from 360°. Bench360 allows users to easily define their own custom tasks along with datasets and relevant task-specific metrics and then automatically benchmarks selected LLMs, inference engines, and quantization levels across different usage scenarios (single stream, batch & server). Bench360 tracks a wide range of metrics, including (1) system metrics -- such as Computing Performance (e.g., latency, throughput), Resource Usage (e.g., energy per query), and Deployment (e.g., cold start time) -- and (2) task-specific metrics such as ROUGE, F1 score or accuracy. We demonstrate Bench360 on four common LLM tasks -- General Knowledge & Reasoning, QA, Summarization and Text-to-SQL -- across three hardware platforms and four state of the art inference engines. Our results reveal several interesting trade-offs between task performance and system-level efficiency, highlighting the differences in inference engines and models. Most importantly, there is no single best setup for local inference, which strongly motivates the need for a framework such as Bench360.
• 摘要：在本地运行大型语言模型 (LLM) 变得越来越普遍。虽然小型开源模型和推理引擎的可用性不断增加降低了进入门槛，但用户现在面临着大量的配置选择。确定最佳配置（平衡功能和非功能需求）需要大量的手动工作。虽然有几个基准针对 LLM 推理，但它们是为狭窄的评估目标而设计的，而不是以用户为中心。他们无法将相关系统和特定于任务的指标集成到一个统一的、易于使用的基准中，该基准支持多个推理引擎、使用场景和量化级别。为了解决这一差距，我们推出了 Bench360——360° 本地 LLM 推理基准测试。 Bench360 允许用户轻松定义自己的自定义任务以及数据集和相关任务特定指标，然后自动对不同使用场景（单流、批处理和服务器）中选定的 LLM、推理引擎和量化级别进行基准测试。 Bench360 跟踪广泛的指标，包括 (1) 系统指标——例如计算性能（例如延迟、吞吐量）、资源使用情况（例如每次查询的能量）和部署（例如冷启动时间）——以及 (2) 特定于任务的指标，例如 ROUGE、F1 分数或准确性。我们在三个硬件平台和四个最先进的推理引擎上演示了 Bench360 的四个常见 LLM 任务——一般知识和推理、质量保证、摘要和文本到 SQL。我们的结果揭示了任务性能和系统级效率之间的一些有趣的权衡，突出了推理引擎和模型的差异。最重要的是，本地推理没有单一的最佳设置，这强烈激发了对 Bench360 等框架的需求。

Title: How Well Do LLMs Understand Tunisian Arabic?

• Authors: Mohamed Mahdi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16683
• Pdf URL: https://arxiv.org/pdf/2511.16683
• Copy Paste: [[2511.16683]] How Well Do LLMs Understand Tunisian Arabic?(https://arxiv.org/abs/2511.16683)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) are the engines driving today's AI agents. The better these models understand human languages, the more natural and user-friendly the interaction with AI becomes, from everyday devices like computers and smartwatches to any tool that can act intelligently. Yet, the ability of industrial-scale LLMs to comprehend low-resource languages, such as Tunisian Arabic (Tunizi), is often overlooked. This neglect risks excluding millions of Tunisians from fully interacting with AI in their own language, pushing them toward French or English. Such a shift not only threatens the preservation of the Tunisian dialect but may also create challenges for literacy and influence younger generations to favor foreign languages. In this study, we introduce a novel dataset containing parallel Tunizi, standard Tunisian Arabic, and English translations, along with sentiment labels. We benchmark several popular LLMs on three tasks: transliteration, translation, and sentiment analysis. Our results reveal significant differences between models, highlighting both their strengths and limitations in understanding and processing Tunisian dialects. By quantifying these gaps, this work underscores the importance of including low-resource languages in the next generation of AI systems, ensuring technology remains accessible, inclusive, and culturally grounded.
• 摘要：大型语言模型 (LLM) 是驱动当今人工智能代理的引擎。这些模型对人类语言的理解越好，与人工智能的交互就越自然和用户友好，从计算机和智能手表等日常设备到任何可以智能操作的工具。然而，工业规模的法学硕士理解突尼斯阿拉伯语（Tunizi）等低资源语言的能力却常常被忽视。这种忽视可能会导致数百万突尼斯人无法用自己的语言与人工智能充分互动，从而将他们推向法语或英语。这种转变不仅威胁到突尼斯方言的保存，还可能给识字能力带来挑战，并影响年轻一代偏爱外语。在这项研究中，我们引入了一个新颖的数据集，其中包含并行的突尼斯语、标准突尼斯阿拉伯语和英语翻译，以及情感标签。我们在三个任务上对几个流行的法学硕士进行了基准测试：音译、翻译和情感分析。我们的结果揭示了模型之间的显着差异，突出了它们在理解和处理突尼斯方言方面的优势和局限性。通过量化这些差距，这项工作强调了将低资源语言纳入下一代人工智能系统的重要性，以确保技术保持可访问性、包容性和文化基础。

Title: Prompt-Based Value Steering of Large Language Models

• Authors: Giulio Antonio Abbo, Tony Belpaeme
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16688
• Pdf URL: https://arxiv.org/pdf/2511.16688
• Copy Paste: [[2511.16688]] Prompt-Based Value Steering of Large Language Models(https://arxiv.org/abs/2511.16688)
• Keywords: language model, prompt
• Abstract: Large language models are increasingly used in applications where alignment with human values is critical. While model fine-tuning is often employed to ensure safe responses, this technique is static and does not lend itself to everyday situations involving dynamic values and preferences. In this paper, we present a practical, reproducible, and model-agnostic procedure to evaluate whether a prompt candidate can effectively steer generated text toward specific human values, formalising a scoring method to quantify the presence and gain of target values in generated responses. We apply our method to a variant of the Wizard-Vicuna language model, using Schwartz's theory of basic human values and a structured evaluation through a dialogue dataset. With this setup, we compare a baseline prompt to one explicitly conditioned on values, and show that value steering is possible even without altering the model or dynamically optimising prompts.
• 摘要：大型语言模型越来越多地用于与人类价值观保持一致至关重要的应用程序中。虽然模型微调通常用于确保安全响应，但这种技术是静态的，不适用于涉及动态价值观和偏好的日常情况。在本文中，我们提出了一种实用的、可重复的、与模型无关的程序来评估提示候选者是否可以有效地将生成的文本引向特定的人类价值观，并形式化一种评分方法来量化生成的响应中目标值的存在和增益。我们将我们的方法应用于 Wizard-Vicuna 语言模型的变体，使用施瓦茨的基本人类价值观理论和通过对话数据集进行的结构化评估。通过这种设置，我们将基线提示与明确以值为条件的提示进行比较，并表明即使不更改模型或动态优化提示，也可以实现价值引导。

Title: Falsely Accused: How AI Detectors Misjudge Slightly Polished Arabic Articles

• Authors: Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16690
• Pdf URL: https://arxiv.org/pdf/2511.16690
• Copy Paste: [[2511.16690]] Falsely Accused: How AI Detectors Misjudge Slightly Polished Arabic Articles(https://arxiv.org/abs/2511.16690)
• Keywords: language model, llm
• Abstract: Many AI detection models have been developed to counter the presence of articles created by artificial intelligence (AI). However, if a human-authored article is slightly polished by AI, a shift will occur in the borderline decision of these AI detection models, leading them to consider it AI-generated article. This misclassification may result in falsely accusing authors of AI plagiarism and harm the credibility of AI detector models. In English, some efforts were made to meet this challenge, but not in Arabic. In this paper, we generated two datasets. The first dataset contains 800 Arabic articles, half AI-generated and half human-authored. We used it to evaluate 14 Large Language models (LLMs) and commercial AI detectors to assess their ability in distinguishing between human-authored and AI-generated articles. The best 8 models were chosen to act as detectors for our primary concern, which is whether they would consider slightly polished human text as AI-generated. The second dataset, Ar-APT, contains 400 Arabic human-authored articles polished by 10 LLMs using 4 polishing settings, totaling 16400 samples. We use it to evaluate the 8 nominated models and determine whether slight polishing will affect their performance. The results reveal that all AI detectors incorrectly attribute a significant number of articles to AI. The best performing LLM, Claude-4 Sonnet, achieved 83.51%, their performance decreased to 57.63% for articles slightly polished by LLaMA-3. Whereas for the best performing commercial model, this http URL, that achieves 92% accuracy, dropped to 12% for articles slightly polished by Mistral or Gemma-3.
• 摘要：人们已经开发了许多人工智能检测模型来对抗人工智能 (AI) 创建的文章的存在。然而，如果一篇人类撰写的文章被人工智能稍微修饰一下，这些人工智能检测模型的边界决策就会发生变化，导致他们认为这是人工智能生成的文章。这种错误分类可能会导致作者错误地指责人工智能抄袭，并损害人工智能检测器模型的可信度。在英语中，人们做出了一些努力来应对这一挑战，但在阿拉伯语中则不然。在本文中，我们生成了两个数据集。第一个数据集包含 800 篇阿拉伯语文章，其中一半是人工智能生成的，一半是人类撰写的。我们用它来评估 14 个大型语言模型 (LLM) 和商业 AI 检测器，以评估它们区分人类撰写的文章和 AI 生成的文章的能力。选择最好的 8 个模型作为我们主要关心的检测器，即它们是否会将稍微修饰过的人类文本视为人工智能生成的。第二个数据集 Ar-APT 包含 400 篇阿拉伯人撰写的文章，由 10 名法学硕士使用 4 种抛光设置进行抛光，总共 16400 个样本。我们用它来评估 8 款提名机型，并确定轻微的打磨是否会影响它们的性能。结果表明，所有人工智能检测器都错误地将大量文章归因于人工智能。表现最好的法学硕士 Claude-4 Sonnet 达到了 83.51%，对于经过 LLaMA-3 稍微润色的文章，其表现下降至 57.63%。而对于性能最佳的商业模型，这个 http URL 的准确率达到 92%，而对于经过 Mistral 或 Gemma-3 稍微修饰的文章，准确率下降到 12%。

Title: Reproducibility Report: Test-Time Training on Nearest Neighbors for Large Language Models

• Authors: Boyang Zhou, Johan Lindqvist, Lindsey Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16691
• Pdf URL: https://arxiv.org/pdf/2511.16691
• Copy Paste: [[2511.16691]] Reproducibility Report: Test-Time Training on Nearest Neighbors for Large Language Models(https://arxiv.org/abs/2511.16691)
• Keywords: language model, gpt
• Abstract: We reproduce the central claims of Test-Time Training on Nearest Neighbors for Large Language Models (Hardt and Sun, 2024), which proposes adapting a language model at inference time by fine-tuning on retrieved nearest-neighbor sequences. Using pretrained RoBERTa embeddings indexed with Faiss, we retrieve 20 neighbors per test input and apply one gradient update per neighbor across GPT-2 (117M, 774M), GPT-Neo (1.3B), and R1-Distilled-Qwen2.5-1.5B. Our experiments confirm that test-time training significantly reduces perplexity and bits-per-byte metrics across diverse domains from The Pile, with the largest improvements in structured or specialized datasets such as GitHub and EuroParl. We further validate that models not pretrained on The Pile benefit more from this adaptation than models already trained on similar data, allowing smaller models to approach the performance of larger ones. Due to infrastructure limitations, we introduce a memory-efficient retrieval implementation that loads only required line offsets rather than entire files, reducing RAM requirements from over 128 GB per server to 32 GB. We also extend the original study by evaluating R1-Distilled-Qwen2.5-1.5B, showing that test-time training yields consistent gains even for modern reasoning-optimized architectures. Overall, our results support the robustness and generality of nearest-neighbor test-time training while highlighting practical considerations for reproducing large-scale retrieval-augmented adaptation.
• 摘要：我们重现了大型语言模型最近邻的测试时训练（Hardt 和 Sun，2024）的核心主张，该主张提出通过对检索到的最近邻序列进行微调来在推理时调整语言模型。使用 Faiss 索引的预训练 RoBERTa 嵌入，我们为每个测试输入检索 20 个邻居，并在 GPT-2 (117M, 774M)、GPT-Neo (1.3B) 和 R1-Distilled-Qwen2.5-1.5B 上对每个邻居应用一个梯度更新。我们的实验证实，测试时训练显着降低了 The Pile 中不同领域的困惑度和每字节位数指标，其中结构化或专业数据集（例如 GitHub 和 EuroParl）的改进最大。我们进一步验证了未在 The Pile 上进行预训练的模型比已经在类似数据上训练的模型从这种适应中受益更多，从而允许较小的模型接近较大模型的性能。由于基础设施的限制，我们引入了一种内存高效的检索实现，它仅加载所需的行偏移量而不是整个文件，从而将 RAM 需求从每台服务器的 128 GB 以上减少到 32 GB。我们还通过评估 R1-Distilled-Qwen2.5-1.5B 扩展了原始研究，表明即使对于现代推理优化架构，测试时训练也能产生一致的增益。总体而言，我们的结果支持最近邻测试时间训练的稳健性和通用性，同时强调了再现大规模检索增强适应的实际考虑因素。

Title: How Language Directions Align with Token Geometry in Multilingual LLMs

• Authors: JaeSeong Kim, Suan Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16693
• Pdf URL: https://arxiv.org/pdf/2511.16693
• Copy Paste: [[2511.16693]] How Language Directions Align with Token Geometry in Multilingual LLMs(https://arxiv.org/abs/2511.16693)
• Keywords: llm
• Abstract: Multilingual LLMs demonstrate strong performance across diverse languages, yet there has been limited systematic analysis of how language information is structured within their internal representation space and how it emerges across layers. We conduct a comprehensive probing study on six multilingual LLMs, covering all 268 transformer layers, using linear and nonlinear probes together with a new Token--Language Alignment analysis to quantify the layer-wise dynamics and geometric structure of language encoding. Our results show that language information becomes sharply separated in the first transformer block (+76.4$\pm$8.2 percentage points from Layer 0 to 1) and remains almost fully linearly separable throughout model depth. We further find that the alignment between language directions and vocabulary embeddings is strongly tied to the language composition of the training data. Notably, Chinese-inclusive models achieve a ZH Match@Peak of 16.43\%, whereas English-centric models achieve only 3.90\%, revealing a 4.21$\times$ structural imprinting effect. These findings indicate that multilingual LLMs distinguish languages not by surface script features but by latent representational structures shaped by the training corpus. Our analysis provides practical insights for data composition strategies and fairness in multilingual representation learning. All code and analysis scripts are publicly available at: this https URL.
• 摘要：多语言法学硕士在不同语言上表现出强大的表现，但对语言信息如何在其内部表示空间内构建以及如何跨层出现的系统分析有限。我们对 6 个多语言 LLM 进行了全面的探索研究，涵盖了所有 268 个转换器层，使用线性和非线性探针以及新的 Token——语言对齐分析来量化语言编码的逐层动态和几何结构。我们的结果表明，语言信息在第一个 Transformer 块中变得急剧分离（从第 0 层到第 1 层+76.4$\pm$8.2 个百分点），并且在整个模型深度中保持几乎完全线性可分离。我们进一步发现，语言方向和词汇嵌入之间的一致性与训练数据的语言构成密切相关。值得注意的是，包含中文的模型的 ZH Match@Peak 为 16.43\%，而以英语为中心的模型仅达到 3.90\%，揭示了 4.21$\times$ 的结构印记效应。这些发现表明，多语言法学硕士不是通过表面文字特征来区分语言，而是通过训练语料库塑造的潜在表征结构来区分语言。我们的分析为多语言表示学习中的数据构成策略和公平性提供了实用的见解。所有代码和分析脚本均可在以下网址公开获取：此 https URL。

Title: Hierarchical Retrieval with Out-Of-Vocabulary Queries: A Case Study on SNOMED CT

• Authors: Jonathon Dilworth, Hui Yang, Jiaoyan Chen, Yongsheng Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16698
• Pdf URL: https://arxiv.org/pdf/2511.16698
• Copy Paste: [[2511.16698]] Hierarchical Retrieval with Out-Of-Vocabulary Queries: A Case Study on SNOMED CT(https://arxiv.org/abs/2511.16698)
• Keywords: language model
• Abstract: SNOMED CT is a biomedical ontology with a hierarchical representation of large-scale concepts. Knowledge retrieval in SNOMED CT is critical for its application, but often proves challenging due to language ambiguity, synonyms, polysemies and so on. This problem is exacerbated when the queries are out-of-vocabulary (OOV), i.e., having no equivalent matchings in the ontology. In this work, we focus on the problem of hierarchical concept retrieval from SNOMED CT with OOV queries, and propose an approach based on language model-based ontology embeddings. For evaluation, we construct OOV queries annotated against SNOMED CT concepts, testing the retrieval of the most direct subsumers and their less relevant ancestors. We find that our method outperforms the baselines including SBERT and two lexical matching methods. While evaluated against SNOMED CT, the approach is generalisable and can be extended to other ontologies. We release code, tools, and evaluation datasets at this https URL.
• 摘要：SNOMED CT 是一种生物医学本体，具有大规模概念的分层表示。 SNOMED CT 中的知识检索对其应用至关重要，但由于语言歧义、同义词、一词多义等原因，往往具有挑战性。当查询超出词汇表（OOV）时，即本体中没有等效匹配时，这个问题会加剧。在这项工作中，我们重点关注使用 OOV 查询从 SNOMED CT 进行分层概念检索的问题，并提出了一种基于语言模型的本体嵌入的方法。为了进行评估，我们构建了针对 SNOMED CT 概念注释的 OOV 查询，测试最直接的子用户及其不太相关的祖先的检索。我们发现我们的方法优于包括 SBERT 和两种词汇匹配方法在内的基线。在针对 SNOMED CT 进行评估时，该方法具有普适性，可以扩展到其他本体。我们在此 https URL 发布代码、工具和评估数据集。

Title: Detecting and Steering LLMs' Empathy in Action

• Authors: Juan P. Cadile
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16699
• Pdf URL: https://arxiv.org/pdf/2511.16699
• Copy Paste: [[2511.16699]] Detecting and Steering LLMs' Empathy in Action(https://arxiv.org/abs/2511.16699)
• Keywords: llm, prompt
• Abstract: We investigate empathy-in-action -- the willingness to sacrifice task efficiency to address human needs -- as a linear direction in LLM activation space. Using contrastive prompts grounded in the Empathy-in-Action (EIA) benchmark, we test detection and steering across Phi-3-mini-4k (3.8B), Qwen2.5-7B (safety-trained), and Dolphin-Llama-3.1-8B (uncensored). Detection: All models show AUROC 0.996-1.00 at optimal layers. Uncensored Dolphin matches safety-trained models, demonstrating empathy encoding emerges independent of safety training. Phi-3 probes correlate strongly with EIA behavioral scores (r=0.71, p<0.01). Cross-model probe agreement is limited (Qwen: r=-0.06, Dolphin: r=0.18), revealing architecture-specific implementations despite convergent detection. Steering: Qwen achieves 65.3% success with bidirectional control and coherence at extreme interventions. Phi-3 shows 61.7% success with similar coherence. Dolphin exhibits asymmetric steerability: 94.4% success for pro-empathy steering but catastrophic breakdown for anti-empathy (empty outputs, code artifacts). Implications: The detection-steering gap varies by model. Qwen and Phi-3 maintain bidirectional coherence; Dolphin shows robustness only for empathy enhancement. Safety training may affect steering robustness rather than preventing manipulation, though validation across more models is needed.
• 摘要：我们研究行动中的同理心——愿意牺牲任务效率来满足人类需求——作为法学硕士激活空间的线性方向。使用基于 Empathy-in-Action (EIA) 基准的对比提示，我们测试了 Phi-3-mini-4k (3.8B)、Qwen2.5-7B（经过安全训练）和 Dolphin-Llama-3.1-8B（未经审查）的检测和转向。检测：所有模型在最佳层均显示 AUROC 0.996-1.00。未经审查的海豚与经过安全训练的模型相匹配，证明同理心编码的出现独立于安全训练。 Phi-3 探针与 EIA 行为评分密切相关（r=0.71，p<0.01）。跨模型探测一致性有限（Qwen：r=-0.06，Dolphin：r=0.18），尽管有收敛检测，但仍揭示了特定于架构的实现。转向：Qwen 在极端干预下通过双向控制和连贯性实现了 65.3% 的成功率。 Phi-3 显示 61.7% 的成功率，并且具有相似的一致性。 Dolphin 表现出不对称的可操纵性：支持同理心的引导成功率为 94.4%，但反同理心的灾难性崩溃（空输出、代码伪影）。含义：检测-引导差距因模型而异。 Qwen 和 Phi-3 保持双向一致性；海豚仅在增强同理心方面表现出鲁棒性。尽管需要在更多模型上进行验证，但安全培训可能会影响转向稳健性而不是防止操纵。

Title: NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generation

• Authors: Hossain Shaikh Saadi, Faria Alam, Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2511.16787
• Pdf URL: https://arxiv.org/pdf/2511.16787
• Copy Paste: [[2511.16787]] NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generation(https://arxiv.org/abs/2511.16787)
• Keywords: agent
• Abstract: This paper presents JGU Mainz's winning system for the BLP-2025 Shared Task on Code Generation from Bangla Instructions. We propose a multi-agent-based pipeline. First, a code-generation agent produces an initial solution from the input instruction. The candidate program is then executed against the provided unit tests (pytest-style, assert-based). Only the failing cases are forwarded to a debugger agent, which reruns the tests, extracts error traces, and, conditioning on the error messages, the current program, and the relevant test cases, generates a revised solution. Using this approach, our submission achieved first place in the shared task with a $Pass@1$ score of 95.4. We also make our code public.
• 摘要：本文介绍了 JGU Mainz 在 BLP-2025 孟加拉指令代码生成共享任务中获胜的系统。我们提出了一个基于多代理的管道。首先，代码生成代理根据输入指令生成初始解决方案。然后针对提供的单元测试（pytest 样式、基于断言）执行候选程序。只有失败的案例才会被转发到调试器代理，调试器代理会重新运行测试，提取错误跟踪，并根据错误消息、当前程序和相关测试案例生成修改后的解决方案。使用这种方法，我们提交的内容在共享任务中获得了第一名，$Pass@1$ 得分为 95.4。我们还将我们的代码公开。

Title: Interpretable dimensions support an effect of agentivity and telicity on split intransitivity

• Authors: Eva Neu, Brian Dillon, Katrin Erk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16824
• Pdf URL: https://arxiv.org/pdf/2511.16824
• Copy Paste: [[2511.16824]] Interpretable dimensions support an effect of agentivity and telicity on split intransitivity(https://arxiv.org/abs/2511.16824)
• Keywords: agent
• Abstract: Intransitive verbs fall into two different syntactic classes, unergatives and unaccusatives. It has long been argued that verbs describing an agentive action are more likely to appear in an unergative syntax, and those describing a telic event to appear in an unaccusative syntax. However, recent work by Kim et al. (2024) found that human ratings for agentivity and telicity were a poor predictor of the syntactic behavior of intransitives. Here we revisit this question using interpretable dimensions, computed from seed words on opposite poles of the agentive and telic scales. Our findings support the link between unergativity/unaccusativity and agentivity/telicity, and demonstrate that using interpretable dimensions in conjunction with human judgments can offer valuable evidence for semantic properties that are not easily evaluated in rating tasks.
• 摘要：不及物动词分为两个不同的句法类别：非作格和非宾格。长期以来，人们一直认为描述施事动作的动词更有可能出现在非作格语法中，而描述目的事件的动词更可能出现在非宾格语法中。然而，Kim 等人最近的工作。 (2024) 发现人类对能动性和动态性的评分不能很好地预测不及物动词的句法行为。在这里，我们使用可解释的维度重新审视这个问题，这些维度是根据施动尺度和目的尺度相反两极的种子词计算出来的。我们的研究结果支持非作格性/非宾格性和能动性/灵敏性之间的联系，并证明将可解释的维度与人类判断结合使用可以为在评级任务中不易评估的语义属性提供有价值的证据。

Title: PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models

• Authors: Oscar Chew, Po-Yi Lu, Jayden Lin, Kuan-Hao Huang, Hsuan-Tien Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16830
• Pdf URL: https://arxiv.org/pdf/2511.16830
• Copy Paste: [[2511.16830]] PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models(https://arxiv.org/abs/2511.16830)
• Keywords: prompt
• Abstract: Recent studies show that text to image (T2I) diffusion models are vulnerable to backdoor attacks, where a trigger in the input prompt can steer generation toward harmful or unintended content. To address this, we introduce PEPPER (PErcePtion Guided PERturbation), a backdoor defense that rewrites the caption into a semantically distant yet visually similar caption while adding unobstructive elements. With this rewriting strategy, PEPPER disrupt the trigger embedded in the input prompt, dilute the influence of trigger tokens and thereby achieve enhanced robustness. Experiments show that PEPPER is particularly effective against text encoder based attacks, substantially reducing attack success while preserving generation quality. Beyond this, PEPPER can be paired with any existing defenses yielding consistently stronger and generalizable robustness than any standalone method. Our code will be released on Github.
• 摘要：最近的研究表明，文本到图像（T2I）扩散模型很容易受到后门攻击，其中输入提示中的触发器可以引导生成有害或意外的内容。为了解决这个问题，我们引入了 PEPPER（PErcePtion Guided PERturbation），这是一种后门防御，它将标题重写为语义上遥远但视觉上相似的标题，同时添加无障碍元素。通过这种重写策略，PEPPER破坏了嵌入在输入提示中的触发器，削弱了触发器令牌的影响，从而增强了鲁棒性。实验表明，PEPPER 对于基于文本编码器的攻击特别有效，可显着降低攻击成功率，同时保持生成质量。除此之外，PEPPER 可以与任何现有的防御措施配合使用，比任何独立方法都具有更强大和通用的鲁棒性。我们的代码将在 Github 上发布。

Title: ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answers

• Authors: Seyed Mohssen Ghafari, Ronny Kol, Juan C. Quiroz, Nella Luan, Monika Patial, Chanaka Rupasinghe, Herman Wandabwa, Luiz Pizzato
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.16846
• Pdf URL: https://arxiv.org/pdf/2511.16846
• Copy Paste: [[2511.16846]] ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answers(https://arxiv.org/abs/2511.16846)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) frequently generate responses that are lengthy and verbose, filled with redundant or unnecessary details. This diminishes clarity and user satisfaction, and it increases costs for model developers, especially with well-known proprietary models that charge based on the number of output tokens. In this paper, we introduce a novel reference-free metric for evaluating the conciseness of responses generated by LLMs. Our method quantifies non-essential content without relying on gold standard references and calculates the average of three calculations: i) a compression ratio between the original response and an LLM abstractive summary; ii) a compression ratio between the original response and an LLM extractive summary; and iii) wordremoval compression, where an LLM removes as many non-essential words as possible from the response while preserving its meaning, with the number of tokens removed indicating the conciseness score. Experimental results demonstrate that our proposed metric identifies redundancy in LLM outputs, offering a practical tool for automated evaluation of response brevity in conversational AI systems without the need for ground truth human annotations.
• 摘要：大型语言模型 (LLM) 经常生成冗长且冗长的响应，其中充满了冗余或不必要的细节。这降低了清晰度和用户满意度，并增加了模型开发人员的成本，特别是对于根据输出代币数量收费的众所周知的专有模型。在本文中，我们介绍了一种新颖的无参考指标，用于评估法学硕士生成的答案的简洁性。我们的方法在不依赖黄金标准参考的情况下量化非必要内容，并计算三个计算的平均值：i) 原始响应和 LLM 摘要摘要之间的压缩比； ii) 原始回复和法学硕士摘要摘要之间的压缩比； iii) 单词删除压缩，其中法学硕士从响应中删除尽可能多的非必要单词，同时保留其含义，删除的标记数量表示简洁性得分。实验结果表明，我们提出的指标可以识别 LLM 输出中的冗余，为自动评估对话式 AI 系统中的响应简洁性提供了实用的工具，而无需地面真实的人类注释。

Title: Improving Latent Reasoning in LLMs via Soft Concept Mixing

• Authors: Kang Wang, Xiangyu Duan, Tianyi Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16885
• Pdf URL: https://arxiv.org/pdf/2511.16885
• Copy Paste: [[2511.16885]] Improving Latent Reasoning in LLMs via Soft Concept Mixing(https://arxiv.org/abs/2511.16885)
• Keywords: language model, llm
• Abstract: Unlike human reasoning in abstract conceptual spaces, large language models (LLMs) typically reason by generating discrete tokens, which potentially limit their expressive power. The recent work Soft Thinking has shown that LLMs' latent reasoning via soft concepts is a promising direction, but LLMs are trained on discrete tokens. To reduce this gap between the soft concepts in reasoning and the discrete tokens in training, we propose Soft Concept Mixing (SCM), a soft concept aware training scheme that directly exposes the model to soft representations during training. Specifically, SCM constructs a soft concept vector by forming a probability-weighted average of embeddings. Then, this vector is mixed into the model's hidden states, which embody rich contextual information. Finally, the entire latent reasoning process is optimized with Reinforcement Learning (RL). Experiments on five reasoning benchmarks demonstrate that SCM improves the reasoning performance of LLMs, and simultaneously maintains a stable training dynamic.
• 摘要：与抽象概念空间中的人类推理不同，大型语言模型 (LLM) 通常通过生成离散标记进行推理，这可能会限制其表达能力。最近的工作 Soft Thinking 表明，法学硕士通过软概念进行潜在推理是一个有前途的方向，但法学硕士是在离散标记上进行训练的。为了缩小推理中的软概念与训练中的离散标记之间的差距，我们提出了软概念混合（SCM），这是一种软概念感知训练方案，可在训练期间直接将模型暴露给软表示。具体来说，SCM 通过形成嵌入的概率加权平均值来构造软概念向量。然后，该向量被混合到模型的隐藏状态中，其中体现了丰富的上下文信息。最后，通过强化学习（RL）优化整个潜在推理过程。五个推理基准的实验表明，SCM 提高了 LLM 的推理性能，同时保持稳定的训练动态。

Title: Deep Improvement Supervision

• Authors: Arip Asadulaev, Rayan Banerjee, Fakhri Karray, Martin Takac
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.16886
• Pdf URL: https://arxiv.org/pdf/2511.16886
• Copy Paste: [[2511.16886]] Deep Improvement Supervision(https://arxiv.org/abs/2511.16886)
• Keywords: language model, llm
• Abstract: Recently, it was shown that small, looped architectures, such as Tiny Recursive Models (TRMs), can outperform Large Language Models (LLMs) on complex reasoning tasks, including the Abstraction and Reasoning Corpus (ARC). In this work, we investigate a core question: how can we further improve the efficiency of these methods with minimal changes? To address this, we frame the latent reasoning of TRMs as a form of classifier-free guidance and implicit policy improvement algorithm. Building on these insights, we propose a novel training scheme that provides a target for each loop during training. We demonstrate that our approach significantly enhances training efficiency. Our method reduces the total number of forward passes by 18x and eliminates halting mechanisms, while maintaining quality comparable to standard TRMs. Notably, we achieve 24% accuracy on ARC-1 with only 0.8M parameters, outperforming most LLMs.
• 摘要：最近，研究表明，小型循环架构，例如微型递归模型（TRM），在复杂的推理任务（包括抽象和推理语料库（ARC））上可以胜过大型语言模型（LLM）。在这项工作中，我们研究了一个核心问题：如何以最小的改变进一步提高这些方法的效率？为了解决这个问题，我们将 TRM 的潜在推理构建为无分类器指导和隐式策略改进算法的一种形式。基于这些见解，我们提出了一种新颖的训练方案，为训练期间的每个循环提供目标。我们证明我们的方法显着提高了培训效率。我们的方法将前向传递的总数减少了 18 倍，并消除了停止机制，同时保持了与标准 TRM 相当的质量。值得注意的是，我们仅用 0.8M 个参数就在 ARC-1 上实现了 24% 的准确率，优于大多数法学硕士。

Title: Predicting the Formation of Induction Heads

• Authors: Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16893
• Pdf URL: https://arxiv.org/pdf/2511.16893
• Copy Paste: [[2511.16893]] Predicting the Formation of Induction Heads(https://arxiv.org/abs/2511.16893)
• Keywords: language model
• Abstract: Arguably, specialized attention heads dubbed induction heads (IHs) underlie the remarkable in-context learning (ICL) capabilities of modern language models (LMs); yet, a precise characterization of their formation remains unclear. In this study, we investigate the relationship between statistical properties of training data (for both natural and synthetic data) and IH formation. We show that (1) a simple equation combining batch size and context size predicts the point at which IHs form; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find a precise Pareto frontier in terms of these two values; and (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but when the frequency and reliability are low, categoriality and the shape of the marginal distribution matter.
• 摘要：可以说，被称为归纳头（IH）的专门注意力头是现代语言模型（LM）卓越的上下文学习（ICL）能力的基础；然而，它们形成的精确特征仍不清楚。在本研究中，我们研究了训练数据（自然数据和合成数据）的统计特性与 IH 形成之间的关系。我们证明（1）一个结合批量大小和上下文大小的简单方程可以预测 IH 的形成点； (2)表面二元重复频率和可靠性强烈影响IH的形成，我们根据这两个值找到了精确的Pareto前沿； (3) 具有高二元组重复频率和可靠性的局部依赖性足以形成 IH，但当频率和可靠性较低时，边缘分布的类别和形状就很重要。

Title: ARQUSUMM: Argument-aware Quantitative Summarization of Online Conversations

• Authors: An Quang Tang, Xiuzhen Zhang, Minh Ngoc Dinh, Zhuang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.16985
• Pdf URL: https://arxiv.org/pdf/2511.16985
• Copy Paste: [[2511.16985]] ARQUSUMM: Argument-aware Quantitative Summarization of Online Conversations(https://arxiv.org/abs/2511.16985)
• Keywords: llm
• Abstract: Online conversations have become more prevalent on public discussion platforms (e.g. Reddit). With growing controversial topics, it is desirable to summarize not only diverse arguments, but also their rationale and justification. Early studies on text summarization focus on capturing general salient information in source documents, overlooking the argumentative nature of online conversations. Recent research on conversation summarization although considers the argumentative relationship among sentences, fail to explicate deeper argument structure within sentences for summarization. In this paper, we propose a novel task of argument-aware quantitative summarization to reveal the claim-reason structure of arguments in conversations, with quantities measuring argument strength. We further propose ARQUSUMM, a novel framework to address the task. To reveal the underlying argument structure within sentences, ARQUSUMM leverages LLM few-shot learning grounded in the argumentation theory to identify propositions within sentences and their claim-reason relationships. For quantitative summarization, ARQUSUMM employs argument structure-aware clustering algorithms to aggregate arguments and quantify their support. Experiments show that ARQUSUMM outperforms existing conversation and quantitative summarization models and generate summaries representing argument structures that are more helpful to users, of high textual quality and quantification accuracy.
• 摘要：在线对话在公共讨论平台（例如 Reddit）上变得更加普遍。随着争议话题越来越多，我们不仅需要总结不同的论点，还需要总结它们的基本原理和理由。早期对文本摘要的研究侧重于捕获源文档中的一般显着信息，忽视了在线对话的争论本质。最近关于会话摘要的研究虽然考虑了句子之间的议论文关系，但未能阐明句子中更深层次的议论文结构以进行摘要。在本文中，我们提出了一种新的论证感知定量总结任务，以揭示对话中论证的主张-理由结构，并用量来衡量论证强度。我们进一步提出 ARQUSUMM，一个解决该任务的新颖框架。为了揭示句子中潜在的论证结构，ARQUSUMM 利用基于论证理论的法学硕士少样本学习来识别句子中的命题及其主张与理由的关系。对于定量总结，ARQUSUMM 采用参数结构感知聚类算法来聚合参数并量化它们的支持度。实验表明，ARQUSUMM 优于现有的对话和定量摘要模型，生成的摘要代表对用户更有帮助的论点结构，具有较高的文本质量和量化准确性。

Title: Supervised Fine Tuning of Large Language Models for Domain Specific Knowledge Graph Construction:A Case Study on Hunan's Historical Celebrities

• Authors: Junjie Hao, Chun Wang, Ying Qiao, Qiuyue Zuo, Qiya Song, Hua Ma, Xieping Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.17012
• Pdf URL: https://arxiv.org/pdf/2511.17012
• Copy Paste: [[2511.17012]] Supervised Fine Tuning of Large Language Models for Domain Specific Knowledge Graph Construction:A Case Study on Hunan's Historical Celebrities(https://arxiv.org/abs/2511.17012)
• Keywords: language model
• Abstract: Large language models and knowledge graphs offer strong potential for advancing research on historical culture by supporting the extraction, analysis, and interpretation of cultural heritage. Using Hunan's modern historical celebrities shaped by Huxiang culture as a case study, pre-trained large models can help researchers efficiently extract key information, including biographical attributes, life events, and social relationships, from textual sources and construct structured knowledge graphs. However, systematic data resources for Hunan's historical celebrities remain limited, and general-purpose models often underperform in domain knowledge extraction and structured output generation in such low-resource settings. To address these issues, this study proposes a supervised fine-tuning approach for enhancing domain-specific information extraction. First, we design a fine-grained, schema-guided instruction template tailored to the Hunan historical celebrities domain and build an instruction-tuning dataset to mitigate the lack of domain-specific training corpora. Second, we apply parameter-efficient instruction fine-tuning to four publicly available large language models - Qwen2.5-7B, Qwen3-8B, DeepSeek-R1-Distill-Qwen-7B, and Llama-3.1-8B-Instruct - and develop evaluation criteria for assessing their extraction performance. Experimental results show that all models exhibit substantial performance gains after fine-tuning. Among them, Qwen3-8B achieves the strongest results, reaching a score of 89.3866 with 100 samples and 50 training iterations. This study provides new insights into fine-tuning vertical large language models for regional historical and cultural domains and highlights their potential for cost-effective applications in cultural heritage knowledge extraction and knowledge graph construction.
• 摘要：大型语言模型和知识图通过支持文化遗产的提取、分析和解释，为推进历史文化研究提供了强大的潜力。以湖湘文化塑造的湖南近代历史名人为例，预训练的大型模型可以帮助研究人员高效地从文本来源中提取关键信息，包括传记属性、生活事件和社会关系，并构建结构化知识图谱。然而，湖南历史名人的系统数据资源仍然有限，在资源匮乏的情况下，通用模型在领域知识提取和结构化输出生成方面往往表现不佳。为了解决这些问题，本研究提出了一种有监督的微调方法来增强特定领域的信息提取。首先，我们设计了一个针对湖南历史名人领域的细粒度、模式引导的指令模板，并构建了一个指令调优数据集，以缓解特定领域训练语料库的缺乏。其次，我们将参数高效的指令微调应用于四种公开可用的大型语言模型 - Qwen2.5-7B、Qwen3-8B、DeepSeek-R1-Distill-Qwen-7B 和 Llama-3.1-8B-Instruct - 并制定评估其提取性能的评估标准。实验结果表明，所有模型在微调后都表现出显着的性能提升。其中，Qwen3-8B 取得了最强的成绩，在 100 个样本和 50 次训练迭代的情况下达到了 89.3866 的分数。这项研究为微调区域历史和文化领域的垂直大语言模型提供了新的见解，并强调了它们在文化遗产知识提取和知识图谱构建中具有成本效益的应用潜力。

Title: Do Vision-Language Models Understand Visual Persuasiveness?

• Authors: Gyuwon Park
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2511.17036
• Pdf URL: https://arxiv.org/pdf/2511.17036
• Copy Paste: [[2511.17036]] Do Vision-Language Models Understand Visual Persuasiveness?(https://arxiv.org/abs/2511.17036)
• Keywords: language model
• Abstract: Recent advances in vision-language models (VLMs) have enabled impressive multi-modal reasoning and understanding. Yet, whether these models truly grasp visual persuasion-how visual cues shape human attitudes and decisions-remains unclear. To probe this question, we construct a high-consensus dataset for binary persuasiveness judgment and introduce the taxonomy of Visual Persuasive Factors (VPFs), encompassing low-level perceptual, mid-level compositional, and high-level semantic cues. We also explore cognitive steering and knowledge injection strategies for persuasion-relevant reasoning. Empirical analysis across VLMs reveals a recall-oriented bias-models over-predict high persuasiveness-and weak discriminative power for low/mid-level features. In contrast, high-level semantic alignment between message and object presence emerges as the strongest predictor of human judgment. Among intervention strategies, simple instruction or unguided reasoning scaffolds yield marginal or negative effects, whereas concise, object-grounded rationales significantly improve precision and F1 scores. These results indicate that VLMs core limitation lies not in recognizing persuasive objects but in linking them to communicative intent.
• 摘要：视觉语言模型 (VLM) 的最新进展实现了令人印象深刻的多模式推理和理解。然而，这些模型是否真正掌握了视觉说服力——视觉线索如何塑造人类的态度和决策——仍不清楚。为了探讨这个问题，我们构建了一个用于二元说服力判断的高共识数据集，并引入了视觉说服因素（VPF）的分类法，包括低级感知、中级构图和高级语义线索。我们还探索与说服相关的推理的认知指导和知识注入策略。跨 VLM 的实证分析揭示了以召回为导向的偏差模型过度预测了高说服力和低/中级特征的弱判别力。相比之下，消息和对象存在之间的高级语义对齐成为人类判断的最强预测因素。在干预策略中，简单的指导或无指导的推理支架会产生边际或负面影响，而简洁、基于对象的基本原理可显着提高精度和 F1 分数。这些结果表明，VLM 的核心局限性不在于识别有说服力的对象，而在于将它们与交流意图联系起来。

Title: Principled Design of Interpretable Automated Scoring for Large-Scale Educational Assessments

• Authors: Yunsung Kim, Mike Hardy, Joseph Tey, Candace Thille, Chris Piech
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17069
• Pdf URL: https://arxiv.org/pdf/2511.17069
• Copy Paste: [[2511.17069]] Principled Design of Interpretable Automated Scoring for Large-Scale Educational Assessments(https://arxiv.org/abs/2511.17069)
• Keywords: llm
• Abstract: AI-driven automated scoring systems offer scalable and efficient means of evaluating complex student-generated responses. Yet, despite increasing demand for transparency and interpretability, the field has yet to develop a widely accepted solution for interpretable automated scoring to be used in large-scale real-world assessments. This work takes a principled approach to address this challenge. We analyze the needs and potential benefits of interpretable automated scoring for various assessment stakeholders and develop four principles of interpretability -- Faithfulness, Groundedness, Traceability, and Interchangeability (FGTI) -- targeted at those needs. To illustrate the feasibility of implementing these principles, we develop the AnalyticScore framework for short answer scoring as a baseline reference framework for future research. AnalyticScore operates by (1) extracting explicitly identifiable elements of the responses, (2) featurizing each response into human-interpretable values using LLMs, and (3) applying an intuitive ordinal logistic regression model for scoring. In terms of scoring accuracy, AnalyticScore outperforms many uninterpretable scoring methods, and is within only 0.06 QWK of the uninterpretable SOTA on average across 10 items from the ASAP-SAS dataset. By comparing against human annotators conducting the same featurization task, we further demonstrate that the featurization behavior of AnalyticScore aligns well with that of humans.
• 摘要：人工智能驱动的自动评分系统提供了可扩展且高效的方法来评估复杂的学生生成的答案。然而，尽管对透明度和可解释性的需求不断增加，该领域尚未开发出一种广泛接受的解决方案，用于可解释的自动评分，以用于大规模的现实世界评估。这项工作采用原则性方法来应对这一挑战。我们分析了各种评估利益相关者的可解释性自动评分的需求和潜在好处，并针对这些需求制定了四个可解释性原则——忠实性、接地性、可追溯性和可互换性 (FGTI)。为了说明实施这些原则的可行性，我们开发了用于简答评分的 AnalyticScore 框架，作为未来研究的基线参考框架。 AnalyticScore 的运作方式如下：(1) 提取响应中明确可识别的元素，(2) 使用法学硕士将每个响应特征化为人类可解释的值，以及 (3) 应用直观的序数逻辑回归模型进行评分。在评分准确性方面，AnalyticScore 优于许多不可解释的评分方法，并且在 ASAP-SAS 数据集中的 10 个项目中，平均与不可解释的 SOTA 的误差仅在 0.06 QWK 以内。通过与执行相同特征化任务的人类注释者进行比较，我们进一步证明 AnalyticScore 的特征化行为与人类的特征化行为非常一致。

Title: MUCH: A Multilingual Claim Hallucination Benchmark

• Authors: Jérémie Dentan, Alexi Canesse, Davide Buscaldi, Aymen Shabou, Sonia Vanier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17081
• Pdf URL: https://arxiv.org/pdf/2511.17081
• Copy Paste: [[2511.17081]] MUCH: A Multilingual Claim Hallucination Benchmark(https://arxiv.org/abs/2511.17081)
• Keywords: language model, llm, hallucination
• Abstract: Claim-level Uncertainty Quantification (UQ) is a promising approach to mitigate the lack of reliability in Large Language Models (LLMs). We introduce MUCH, the first claim-level UQ benchmark designed for fair and reproducible evaluation of future methods under realistic conditions. It includes 4,873 samples across four European languages (English, French, Spanish, and German) and four instruction-tuned open-weight LLMs. Unlike prior claim-level benchmarks, we release 24 generation logits per token, facilitating the development of future white-box methods without re-generating data. Moreover, in contrast to previous benchmarks that rely on manual or LLM-based segmentation, we propose a new deterministic algorithm capable of segmenting claims using as little as 0.2% of the LLM generation time. This makes our segmentation approach suitable for real-time monitoring of LLM outputs, ensuring that MUCH evaluates UQ methods under realistic deployment constraints. Finally, our evaluations show that current methods still have substantial room for improvement in both performance and efficiency.
• 摘要：声明级不确定性量化 (UQ) 是缓解大型语言模型 (LLM) 可靠性缺乏的一种很有前途的方法。我们推出了 MUCH，这是第一个声明级 UQ 基准，旨在在现实条件下对未来方法进行公平且可重复的评估。它包括四种欧洲语言（英语、法语、西班牙语和德语）的 4,873 个样本和四个经过指令调整的开放重量法学硕士。与之前的声明级基准不同，我们为每个代币发布 24 代 logits，促进未来白盒方法的开发，而无需重新生成数据。此外，与之前依赖手动或基于 LLM 的分割的基准相比，我们提出了一种新的确定性算法，能够使用低至 0.2% 的 LLM 生成时间来分割索赔。这使得我们的分割方法适合实时监控 LLM 输出，确保 MUCH 在实际部署约束下评估 UQ 方法。最后，我们的评估表明，当前的方法在性能和效率方面仍有很大的改进空间。

Title: Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation

• Authors: Yeqin Zhang, Yizheng Zhao, Chen Hu, Binxing Jiao, Daxin Jiang, Ruihang Miao, Cam-Tu Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.17129
• Pdf URL: https://arxiv.org/pdf/2511.17129
• Copy Paste: [[2511.17129]] Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation(https://arxiv.org/abs/2511.17129)
• Keywords: language model, llm
• Abstract: Text representation plays a critical role in tasks like clustering, retrieval, and other downstream applications. With the emergence of large language models (LLMs), there is increasing interest in harnessing their capabilities for this purpose. However, most of the LLMs are inherently causal and optimized for next-token prediction, making them suboptimal for producing holistic representations. To address this, recent studies introduced pretext tasks to adapt LLMs for text representation. Most of these tasks, however, rely on token-level prediction objectives, such as the masked next-token prediction (MNTP) used in LLM2Vec. In this work, we explore the untapped potential of context compression as a pretext task for unsupervised adaptation of LLMs. During compression pre-training, the model learns to generate compact memory tokens, which substitute the whole context for downstream sequence prediction. Experiments demonstrate that a well-designed compression objective can significantly enhance LLM-based text representations, outperforming models trained with token-level pretext tasks. Further improvements through contrastive learning produce a strong representation model (LLM2Comp) that outperforms contemporary LLM-based text encoders on a wide range of tasks while being more sample-efficient, requiring significantly less training data.
• 摘要：文本表示在聚类、检索和其他下游应用程序等任务中起着至关重要的作用。随着大型语言模型 (LLM) 的出现，人们越来越有兴趣利用其功能来实现此目的。然而，大多数 LLM 本质上都是因果关系，并且针对下一个标记预测进行了优化，这使得它们在生成整体表示方面不是最佳的。为了解决这个问题，最近的研究引入了借口任务来使法学硕士适应文本表示。然而，这些任务中的大多数都依赖于令牌级别的预测目标，例如 LLM2Vec 中使用的屏蔽下一个令牌预测 (MNTP)。在这项工作中，我们探索了上下文压缩的未开发潜力，作为法学硕士无监督适应的借口任务。在压缩预训练期间，模型学习生成紧凑的记忆标记，用整个上下文代替下游序列预测。实验表明，精心设计的压缩目标可以显着增强基于 LLM 的文本表示，优于使用令牌级借口任务训练的模型。通过对比学习的进一步改进产生了强大的表示模型 (LLM2Comp)，该模型在广泛的任务上优于当代基于 LLM 的文本编码器，同时样本效率更高，需要的训练数据显着减少。

Title: LangMark: A Multilingual Dataset for Automatic Post-Editing

• Authors: Diego Velazquez, Mikaela Grace, Konstantinos Karageorgos, Lawrence Carin, Aaron Schliem, Dimitrios Zaikis, Roger Wechsler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17153
• Pdf URL: https://arxiv.org/pdf/2511.17153
• Copy Paste: [[2511.17153]] LangMark: A Multilingual Dataset for Automatic Post-Editing(https://arxiv.org/abs/2511.17153)
• Keywords: language model, llm, prompt
• Abstract: Automatic post-editing (APE) aims to correct errors in machine-translated text, enhancing translation quality, while reducing the need for human intervention. Despite advances in neural machine translation (NMT), the development of effective APE systems has been hindered by the lack of large-scale multilingual datasets specifically tailored to NMT outputs. To address this gap, we present and release LangMark, a new human-annotated multilingual APE dataset for English translation to seven languages: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish. The dataset has 206,983 triplets, with each triplet consisting of a source segment, its NMT output, and a human post-edited translation. Annotated by expert human linguists, our dataset offers both linguistic diversity and scale. Leveraging this dataset, we empirically show that Large Language Models (LLMs) with few-shot prompting can effectively perform APE, improving upon leading commercial and even proprietary machine translation systems. We believe that this new resource will facilitate the future development and evaluation of APE systems.
• 摘要：自动译后编辑 (APE) 旨在纠正机器翻译文本中的错误，提高翻译质量，同时减少人工干预的需要。尽管神经机器翻译 (NMT) 取得了进步，但由于缺乏专门针对 NMT 输出定制的大规模多语言数据集，有效的 APE 系统的开发受到了阻碍。为了弥补这一差距，我们提出并发布了 LangMark，这是一个新的人工注释多语言 APE 数据集，用于将英语翻译为七种语言：巴西葡萄牙语、法语、德语、意大利语、日语、俄语和西班牙语。该数据集有 206,983 个三元组，每个三元组由源片段、其 NMT 输出和人工后期编辑翻译组成。我们的数据集由人类语言学家专家注释，提供了语言多样性和规模。利用该数据集，我们凭经验证明，具有少量提示的大型语言模型 (LLM) 可以有效地执行 APE，从而改进领先的商业甚至专有机器翻译系统。我们相信这一新资源将有助于 APE 系统的未来开发和评估。

Title: The PLLuM Instruction Corpus

• Authors: Piotr Pęzik, Filip Żarnecki, Konrad Kaczyński, Anna Cichosz, Zuzanna Deckert, Monika Garnys, Izabela Grabarczyk, Wojciech Janowski, Sylwia Karasińska, Aleksandra Kujawiak, Piotr Misztela, Maria Szymańska, Karolina Walkusz, Igor Siek, Maciej Chrabąszcz, Anna Kołos, Agnieszka Karlińska, Karolina Seweryn, Aleksandra Krasnodębska, Paula Betscher, Zofia Cieślińska, Katarzyna Kowol, Artur Wilczek, Maciej Trzciński, Katarzyna Dziewulska, Roman Roszko, Tomasz Bernaś, Jurgita Vaičenonienė, Danuta Roszko, Paweł Levchuk, Paweł Kowalski, Irena Prawdzic-Jankowska, Marek Kozłowski, Sławomir Dadas, Rafał Poświata, Alina Wróblewska, Katarzyna Krasnowska-Kieraś, Maciej Ogrodniczuk, Michał Rudolf, Piotr Rybak, Karolina Saputa, Joanna Wołoszyn, Marcin Oleksy, Bartłomiej Koptyra, Teddy Ferdinan, Stanisław Woźniak, Maciej Piasecki, Paweł Walkowiak, Konrad Wojtasik, Arkadiusz Janz, Przemysław Kazienko, Julia Moska, Jan Kocoń
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.17161
• Pdf URL: https://arxiv.org/pdf/2511.17161
• Copy Paste: [[2511.17161]] The PLLuM Instruction Corpus(https://arxiv.org/abs/2511.17161)
• Keywords: language model, llm
• Abstract: This paper describes the instruction dataset used to fine-tune a set of transformer-based large language models (LLMs) developed in the PLLuM (Polish Large Language Model) project. We present a functional typology of the organic, converted, and synthetic instructions used in PLLuM and share some observations about the implications of using human-authored versus synthetic instruction datasets in the linguistic adaptation of base LLMs. Additionally, we release the first representative subset of the PLLuM instruction corpus (PLLuMIC), which we believe to be useful in guiding and planning the development of similar datasets for other LLMs.
• 摘要：本文描述了用于微调 PLLuM（波兰大语言模型）项目中开发的一组基于 Transformer 的大语言模型 (LLM) 的指令数据集。我们提出了 PLLuM 中使用的有机指令、转换指令和合成指令的功能类型学，并分享了一些关于在基础法学硕士语言适应中使用人类编写的指令数据集与合成指令数据集的影响的观察结果。此外，我们还发布了 PLLuM 指令语料库 (PLLuMIC) 的第一个代表性子集，我们相信它对于指导和规划其他法学硕士类似数据集的开发很有用。

Title: Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models

• Authors: Vy Nguyen, Ziqi Xu, Jeffrey Chan, Estrid He, Feng Xia, Xiuzhen Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.17170
• Pdf URL: https://arxiv.org/pdf/2511.17170
• Copy Paste: [[2511.17170]] Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models(https://arxiv.org/abs/2511.17170)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) often produce fluent but factually incorrect responses, a phenomenon known as hallucination. Abstention, where the model chooses not to answer and instead outputs phrases such as "I don't know", is a common safeguard. However, existing abstention methods typically rely on post-generation signals, such as generation variations or feedback, which limits their ability to prevent unreliable responses in advance. In this paper, we introduce Aspect-Based Causal Abstention (ABCA), a new framework that enables early abstention by analysing the internal diversity of LLM knowledge through causal inference. This diversity reflects the multifaceted nature of parametric knowledge acquired from various sources, representing diverse aspects such as disciplines, legal contexts, or temporal frames. ABCA estimates causal effects conditioned on these aspects to assess the reliability of knowledge relevant to a given query. Based on these estimates, we enable two types of abstention: Type-1, where aspect effects are inconsistent (knowledge conflict), and Type-2, where aspect effects consistently support abstention (knowledge insufficiency). Experiments on standard benchmarks demonstrate that ABCA improves abstention reliability, achieves state-of-the-art performance, and enhances the interpretability of abstention decisions.
• 摘要：大型语言模型 (LLM) 通常会产生流畅但实际上不正确的响应，这种现象称为幻觉。弃权，即模型选择不回答，而是输出诸如“我不知道”之类的短语，是一种常见的保护措施。然而，现有的弃权方法通常依赖于生成后信号，例如生成变化或反馈，这限制了它们提前防止不可靠响应的能力。在本文中，我们介绍了基于方面的因果弃权（ABCA），这是一种新框架，通过因果推理分析LLM知识的内部多样性，从而实现早期弃权。这种多样性反映了从不同来源获取的参数知识的多面性，代表了学科、法律背景或时间框架等不同方面。 ABCA 估计以这些方面为条件的因果效应，以评估与给定查询相关的知识的可靠性。基于这些估计，我们启用两种类型的弃权：类型 1，方面效果不一致（知识冲突）；类型 2，方面效果一致支持弃权（知识不足）。标准基准的实验表明，ABCA 提高了弃权可靠性，实现了最先进的性能，并增强了弃权决策的可解释性。

Title: AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale

• Authors: Ziyang Wang, Yuanlei Zheng, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Pei Fu, Zhenbo Luo, Wei Chen, Xiang Bai
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2511.17190
• Pdf URL: https://arxiv.org/pdf/2511.17190
• Copy Paste: [[2511.17190]] AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale(https://arxiv.org/abs/2511.17190)
• Keywords: language model, llm, agent
• Abstract: For industrial-scale text-to-SQL, supplying the entire database schema to Large Language Models (LLMs) is impractical due to context window limits and irrelevant noise. Schema linking, which filters the schema to a relevant subset, is therefore critical. However, existing methods incur prohibitive costs, struggle to trade off recall and noise, and scale poorly to large databases. We present \textbf{AutoLink}, an autonomous agent framework that reformulates schema linking as an iterative, agent-driven process. Guided by an LLM, AutoLink dynamically explores and expands the linked schema subset, progressively identifying necessary schema components without inputting the full database schema. Our experiments demonstrate AutoLink's superior performance, achieving state-of-the-art strict schema linking recall of \textbf{97.4\%} on Bird-Dev and \textbf{91.2\%} on Spider-2.0-Lite, with competitive execution accuracy, i.e., \textbf{68.7\%} EX on Bird-Dev (better than CHESS) and \textbf{34.9\%} EX on Spider-2.0-Lite (ranking 2nd on the official leaderboard). Crucially, AutoLink exhibits \textbf{exceptional scalability}, \textbf{maintaining high recall}, \textbf{efficient token consumption}, and \textbf{robust execution accuracy} on large schemas (e.g., over 3,000 columns) where existing methods severely degrade-making it a highly scalable, high-recall schema-linking solution for industrial text-to-SQL systems.
• 摘要：对于工业规模的文本到 SQL，由于上下文窗口限制和不相关的噪声，向大型语言模型 (LLM) 提供整个数据库模式是不切实际的。因此，模式链接（将模式过滤到相关子集）至关重要。然而，现有的方法会产生高昂的成本，难以权衡召回率和噪音，并且很难扩展到大型数据库。我们提出了 \textbf{AutoLink}，一个自主代理框架，它将模式链接重新表述为迭代的、代理驱动的过程。在法学硕士的指导下，AutoLink 动态探索和扩展链接的模式子集，逐步识别必要的模式组件，而无需输入完整的数据库模式。我们的实验证明了 AutoLink 的卓越性能，实现了 Bird-Dev 上的 \textbf{97.4\%} 和 Spider-2.0-Lite 上的 \textbf{91.2\%} 的最先进的严格模式链接召回，具有竞争性的执行准确性，即 Bird-Dev 上的 \textbf{68.7\%} EX（优于 CHESS）和 \textbf{34.9\%} EX Spider-2.0-Lite（官方排行榜第二名）。至关重要的是，AutoLink 在现有方法严重退化的大型模式（例如，超过 3,000 列）上展示了 \textbf{卓越的可扩展性}、\textbf{保持高召回率}、\textbf{有效的令牌消耗}和 \textbf{鲁棒的执行准确性}，使其成为工业文本到 SQL 系统的高度可扩展、高召回率的模式链接解决方案。

Title: E$^3$-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models

• Authors: Tao Yuan, Haoli Bai, Yinfei Pan, Xuyang Cao, Tianyu Zhang, Lu Hou, Ting Hu, Xianzhi Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17205
• Pdf URL: https://arxiv.org/pdf/2511.17205
• Copy Paste: [[2511.17205]] E$^3$-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models(https://arxiv.org/abs/2511.17205)
• Keywords: language model
• Abstract: With the increasing size of large language models, layer pruning has gained increased attention as a hardware-friendly approach for model compression. However, existing layer pruning methods struggle to simultaneously address key practical deployment challenges, including performance degradation, high training costs, and limited acceleration. To overcome these limitations, we propose \name, a task-\underline{E}ffective, training-\underline{E}conomical and inference-\underline{E}fficient layer pruning framework. \namespace introduces two key innovations: (1) a differentiable mask optimization method using a Gumbel-TopK sampler, enabling efficient and precise pruning mask search; and (2) an entropy-aware adaptive knowledge distillation strategy that enhances task performance. Extensive experiments over diverse model architectures and benchmarks demonstrate the superiority of our method over state-of-the-art approaches. Notably, \namespace achieves 96\% accuracy, a mere 0.8\% drop from the original model (96.8\%) on MATH-500 when pruning 25\% layers of Qwen3-32B, outperforming existing SOTA (95\%), with a 1.33$\times$ inference speedup by consuming merely 0.5B tokens (0.5\% of the post-training data volume).
• 摘要：随着大型语言模型规模的不断增大，层剪枝作为一种硬件友好的模型压缩方法越来越受到关注。然而，现有的层剪枝方法很难同时解决关键的实际部署挑战，包括性能下降、训练成本高和加速有限。为了克服这些限制，我们提出了一个任务\underline{E}有效、训练\underline{E}经济且推理\underline{E}高效的层剪枝框架。 \namespace 引入了两个关键创新：（1）使用 Gumbel-TopK 采样器的可微掩码优化方法，实现高效、精确的修剪掩码搜索； (2) 一种增强任务性能的熵感知自适应知识蒸馏策略。对不同模型架构和基准的广泛实验证明了我们的方法相对于最先进方法的优越性。值得注意的是，当修剪 Qwen3-32B 的 25% 层时，\namespace 达到了 96\% 的准确率，仅比 MATH-500 上的原始模型 (96.8\%) 下降了 0.8\%，优于现有的 SOTA (95\%)，仅消耗 0.5B 令牌（训练后数据的 0.5\%），推理速度提高了 1.33$\times$量）。

Title: A Simple Yet Strong Baseline for Long-Term Conversational Memory of LLM Agents

• Authors: Sizhe Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17208
• Pdf URL: https://arxiv.org/pdf/2511.17208
• Copy Paste: [[2511.17208]] A Simple Yet Strong Baseline for Long-Term Conversational Memory of LLM Agents(https://arxiv.org/abs/2511.17208)
• Keywords: llm, agent
• Abstract: LLM-based conversational agents still struggle to maintain coherent, personalized interaction over many sessions: fixed context windows limit how much history can be kept in view, and most external memory approaches trade off between coarse retrieval over large chunks and fine-grained but fragmented views of the dialogue. Motivated by neo-Davidsonian event semantics, we propose an event-centric alternative that represents conversational history as short, event-like propositions which bundle together participants, temporal cues, and minimal local context, rather than as independent relation triples or opaque summaries. In contrast to work that aggressively compresses or forgets past content, our design aims to preserve information in a non-compressive form and make it more accessible, rather than more lossy. Concretely, we instruct an LLM to decompose each session into enriched elementary discourse units (EDUs) -- self-contained statements with normalized entities and source turn attributions -- and organize sessions, EDUs, and their arguments in a heterogeneous graph that supports associative recall. On top of this representation we build two simple retrieval-based variants that use dense similarity search and LLM filtering, with an optional graph-based propagation step to connect and aggregate evidence across related EDUs. Experiments on the LoCoMo and LongMemEval$_S$ benchmarks show that these event-centric memories match or surpass strong baselines, while operating with much shorter QA contexts. Our results suggest that structurally simple, event-level memory provides a principled and practical foundation for long-horizon conversational agents. Our code and data will be released at this https URL.
• 摘要：基于 LLM 的对话代理仍然难以在许多会话中保持连贯、个性化的交互：固定的上下文窗口限制了可以保留的历史记录量，并且大多数外部记忆方法在大块的粗略检索和细粒度但碎片化的对话视图之间进行权衡。受新戴维森事件语义学的启发，我们提出了一种以事件为中心的替代方案，它将对话历史表示为简短的、类似事件的命题，将参与者、时间线索和最小的局部上下文捆绑在一起，而不是作为独立的关系三元组或不透明的摘要。与积极压缩或忘记过去内容的工作相比，我们的设计旨在以非压缩形式保存信息并使其更易于访问，而不是更有损失。具体来说，我们指导法学硕士将每个会话分解为丰富的基本话语单元（EDU）——具有规范化实体和来源转向归因的独立陈述——并将会话、EDU 及其论点组织在支持联想回忆的异构图中。在此表示之上，我们构建了两个简单的基于检索的变体，它们使用密集相似性搜索和 LLM 过滤，并具有可选的基于图的传播步骤来连接和聚合相关 EDU 的证据。 LoCoMo 和 LongMemEval$_S$ 基准的实验表明，这些以事件为中心的记忆匹配或超过了强大的基线，同时在更短的 QA 上下文中运行。我们的结果表明，结构简单的事件级记忆为长视野会话代理提供了原则性和实用性的基础。我们的代码和数据将在此 https URL 发布。

Title: Parrot: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs

• Authors: Yusuf Çelebi, Mahmoud El Hussieni, Özay Ezerceli
• Subjects: cs.CL, cs.AI, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.17220
• Pdf URL: https://arxiv.org/pdf/2511.17220
• Copy Paste: [[2511.17220]] Parrot: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs(https://arxiv.org/abs/2511.17220)
• Keywords: language model, gpt, llm
• Abstract: This study presents PARROT (Persuasion and Agreement Robustness Rating of Output Truth), a robustness focused framework designed to measure the degradation in accuracy that occurs under social pressure exerted on users through authority and persuasion in large language models (LLMs) the phenomenon of sycophancy (excessive conformity). PARROT (i) isolates causal effects by comparing the neutral version of the same question with an authoritatively false version using a double-blind evaluation, (ii) quantifies confidence shifts toward the correct and imposed false responses using log-likelihood-based calibration tracking, and (iii) systematically classifies failure modes (e.g., robust correct, sycophantic agreement, reinforced error, stubborn error, self-correction, etc.) using an eight-state behavioral taxonomy. We evaluated 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates. Findings show marked heterogeneity: advanced models (e.g., GPT-5, GPT-4.1, Claude Sonnet 4.5) exhibit low "follow rates" ($\leq 11\%$, GPT-5: 4\%) and minimal accuracy loss, while older/smaller models show severe epistemic collapse (GPT-4: 80\%, Qwen 2.5-1.5B: 94\%). The danger is not limited to response changes; weak models reduce confidence in the correct response while increasing confidence in the imposed incorrect response. While international law and global knowledge at the domain level exhibit high fragility, elementary mathematics is relatively resilient. Consequently, we argue that the goal of "resistance to overfitting pressure" should be addressed as a primary objective alongside accuracy, harm avoidance, and privacy for safe deployment in the real world.
• 摘要：本研究提出了 PARROT（输出真相的说服和协议鲁棒性评级），这是一个以鲁棒性为重点的框架，旨在衡量在大型语言模型（LLM）中通过权威和说服对用户施加的社会压力下发生的准确性下降，以及阿谀奉承（过度从众）的现象。 PARROT (i) 通过使用双盲评估将同一问题的中性版本与权威错误版本进行比较来隔离因果效应，(ii) 使用基于对数似然的校准跟踪来量化向正确和强加的错误响应的置信度转变，以及 (iii) 使用八状态行为系统对故障模式进行系统分类（例如，稳健正确、阿谀奉承、强化错误、顽固错误、自我纠正等）分类学。我们使用跨 13 个领域的 1,302 个 MMLU 式多项选择问题和特定领域的权威模板评估了 22 个模型。研究结果显示出明显的异质性：高级模型（例如，GPT-5、GPT-4.1、Claude Sonnet 4.5）表现出较低的“跟随率”（$\leq 11\%$，GPT-5：4\%）和最小的准确性损失，而较旧/较小的模型则表现出严重的认知崩溃（GPT-4：80\%，Qwen 2.5-1.5B：94\%）。危险不仅限于响应变化；弱模型会降低对正确响应的置信度，同时增加对强加的错误响应的置信度。虽然领域层面的国际法和全球知识表现出高度的脆弱性，但基础数学相对具有弹性。因此，我们认为，“抵抗过度拟合压力”的目标应该与准确性、避免伤害和隐私一起作为主要目标，以便在现实世界中安全部署。

Title: Social-Media Based Personas Challenge: Hybrid Prediction of Common and Rare User Actions on Bluesky

• Authors: Benjamin White, Anastasia Shimorina
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17241
• Pdf URL: https://arxiv.org/pdf/2511.17241
• Copy Paste: [[2511.17241]] Social-Media Based Personas Challenge: Hybrid Prediction of Common and Rare User Actions on Bluesky(https://arxiv.org/abs/2511.17241)
• Keywords: llm
• Abstract: Understanding and predicting user behavior on social media platforms is crucial for content recommendation and platform design. While existing approaches focus primarily on common actions like retweeting and liking, the prediction of rare but significant behaviors remains largely unexplored. This paper presents a hybrid methodology for social media user behavior prediction that addresses both frequent and infrequent actions across a diverse action vocabulary. We evaluate our approach on a large-scale Bluesky dataset containing 6.4 million conversation threads spanning 12 distinct user actions across 25 persona clusters. Our methodology combines four complementary approaches: (i) a lookup database system based on historical response patterns; (ii) persona-specific LightGBM models with engineered temporal and semantic features for common actions; (iii) a specialized hybrid neural architecture fusing textual and temporal representations for rare action classification; and (iv) generation of text replies. Our persona-specific models achieve an average macro F1-score of 0.64 for common action prediction, while our rare action classifier achieves 0.56 macro F1-score across 10 rare actions. These results demonstrate that effective social media behavior prediction requires tailored modeling strategies recognizing fundamental differences between action types. Our approach achieved first place in the SocialSim: Social-Media Based Personas challenge organized at the Social Simulation with LLMs workshop at COLM 2025.
• 摘要：理解和预测社交媒体平台上的用户行为对于内容推荐和平台设计至关重要。虽然现有方法主要关注转发和点赞等常见行为，但对罕见但重要行为的预测在很大程度上仍未得到探索。本文提出了一种社交媒体用户行为预测的混合方法，该方法可以解决不同动作词汇中的频繁和不频繁动作。我们在大规模 Bluesky 数据集上评估我们的方法，该数据集包含 640 万个对话线程，涵盖 25 个角色集群中的 12 个不同的用户操作。我们的方法结合了四种互补的方法：（i）基于历史响应模式的查找数据库系统； (ii) 具有针对常见动作的工程时间和语义特征的特定角色 LightGBM 模型； (iii) 一种专门的混合神经架构，融合了文本和时间表示，用于罕见动作分类； (iv) 生成文本回复。我们的特定角色模型在常见动作预测方面的平均宏观 F1 得分为 0.64，而我们的罕见动作分类器在 10 个罕见动作中实现了 0.56 的宏观 F1 得分。这些结果表明，有效的社交媒体行为预测需要定制的建模策略，以识别行为类型之间的根本差异。我们的方法在 COLM 2025 法学硕士社交模拟研讨会上组织的 SocialSim：基于社交媒体的角色挑战赛中获得第一名。

Title: Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation

• Authors: Marii Ojastu, Hele-Andra Kuulmets, Aleksei Dorkin, Marika Borovikova, Dage Särg, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17290
• Pdf URL: https://arxiv.org/pdf/2511.17290
• Copy Paste: [[2511.17290]] Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation(https://arxiv.org/abs/2511.17290)
• Keywords: language model, llm, prompt
• Abstract: In this paper, we present a localized and culturally adapted Estonian translation of the test set from the widely used commonsense reasoning benchmark, WinoGrande. We detail the translation and adaptation process carried out by translation specialists and evaluate the performance of both proprietary and open source models on the human translated benchmark. Additionally, we explore the feasibility of achieving high-quality machine translation by incorporating insights from the manual translation process into the design of a detailed prompt. This prompt is specifically tailored to address both the linguistic characteristics of Estonian and the unique translation challenges posed by the WinoGrande dataset. Our findings show that model performance on the human translated Estonian dataset is slightly lower than on the original English test set, while performance on machine-translated data is notably worse. Additionally, our experiments indicate that prompt engineering offers limited improvement in translation quality or model accuracy, and highlight the importance of involving language specialists in dataset translation and adaptation to ensure reliable and interpretable evaluations of language competency and reasoning in large language models.
• 摘要：在本文中，我们提出了来自广泛使用的常识推理基准 WinoGrande 的测试集的本地化和文化适应的爱沙尼亚语翻译。我们详细介绍了翻译专家执行的翻译和改编过程，并评估了专有模型和开源模型在人工翻译基准上的性能。此外，我们还探索了通过将手动翻译过程中的见解融入到详细提示的设计中来实现高质量机器翻译的可行性。该提示是专门为解决爱沙尼亚语的语言特征和 WinoGrande 数据集带来的独特翻译挑战而定制的。我们的研究结果表明，人工翻译的爱沙尼亚语数据集上的模型性能略低于原始英语测试集，而机器翻译数据上的性能则明显较差。此外，我们的实验表明，即时工程对翻译质量或模型准确性的改善有限，并强调了让语言专家参与数据集翻译和适应的重要性，以确保对大型语言模型中的语言能力和推理进行可靠且可解释的评估。

Title: Large Language Models for Sentiment Analysis to Detect Social Challenges: A Use Case with South African Languages

• Authors: Koena Ronny Mabokela, Tim Schlippe, Matthias Wölfel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.17301
• Pdf URL: https://arxiv.org/pdf/2511.17301
• Copy Paste: [[2511.17301]] Large Language Models for Sentiment Analysis to Detect Social Challenges: A Use Case with South African Languages(https://arxiv.org/abs/2511.17301)
• Keywords: language model, gpt, llm
• Abstract: Sentiment analysis can aid in understanding people's opinions and emotions on social issues. In multilingual communities sentiment analysis systems can be used to quickly identify social challenges in social media posts, enabling government departments to detect and address these issues more precisely and effectively. Recently, large-language models (LLMs) have become available to the wide public and initial analyses have shown that they exhibit magnificent zero-shot sentiment analysis abilities in English. However, there is no work that has investigated to leverage LLMs for sentiment analysis on social media posts in South African languages and detect social challenges. Consequently, in this work, we analyse the zero-shot performance of the state-of-the-art LLMs GPT-3.5, GPT-4, LlaMa 2, PaLM 2, and Dolly 2 to investigate the sentiment polarities of the 10 most emerging topics in English, Sepedi and Setswana social media posts that fall within the jurisdictional areas of 10 South African government departments. Our results demonstrate that there are big differences between the various LLMs, topics, and languages. In addition, we show that a fusion of the outcomes of different LLMs provides large gains in sentiment classification performance with sentiment classification errors below 1%. Consequently, it is now feasible to provide systems that generate reliable information about sentiment analysis to detect social challenges and draw conclusions about possible needs for actions on specific topics and within different language groups.
• 摘要：情绪分析可以帮助了解人们对社会问题的看法和情绪。在多语言社区中，情感分析系统可用于快速识别社交媒体帖子中的社会挑战，使政府部门能够更准确、更有效地发现和解决这些问题。最近，大语言模型（LLM）已向公众开放，初步分析表明它们在英语中表现出出色的零样本情感分析能力。然而，还没有研究利用法学硕士对南非语言的社交媒体帖子进行情绪分析并检测社会挑战。因此，在这项工作中，我们分析了最先进的 LLM GPT-3.5、GPT-4、LlaMa 2、PaLM 2 和 Dolly 2 的零样本性能，以调查英语、Sepedi 和 Setswana 社交媒体帖子中 10 个最新兴主题的情绪极性，这些主题属于南非 10 个政府部门的管辖范围。我们的结果表明，不同的法学硕士、主题和语言之间存在很大差异。此外，我们还发现，不同法学硕士成果的融合可以在情感分类性能方面带来巨大提升，情感分类错误率低于 1%。因此，现在可以提供能够生成有关情绪分析的可靠信息的系统，以检测社会挑战并得出有关特定主题和不同语言组内可能采取行动的结论。

Title: Humanlike Multi-user Agent (HUMA): Designing a Deceptively Human AI Facilitator for Group Chats

• Authors: Mateusz Jacniacki, Martí Carmona Serrat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.17315
• Pdf URL: https://arxiv.org/pdf/2511.17315
• Copy Paste: [[2511.17315]] Humanlike Multi-user Agent (HUMA): Designing a Deceptively Human AI Facilitator for Group Chats(https://arxiv.org/abs/2511.17315)
• Keywords: language model, llm, chat, agent
• Abstract: Conversational agents built on large language models (LLMs) are becoming increasingly prevalent, yet most systems are designed for one-on-one, turn-based exchanges rather than natural, asynchronous group chats. As AI assistants become widespread throughout digital platforms, from virtual assistants to customer service, developing natural and humanlike interaction patterns seems crucial for maintaining user trust and engagement. We present the Humanlike Multi-user Agent (HUMA), an LLM-based facilitator that participates in multi-party conversations using human-like strategies and timing. HUMA extends prior multi-user chatbot work with an event-driven architecture that handles messages, replies, reactions and introduces realistic response-time simulation. HUMA comprises three components-Router, Action Agent, and Reflection-which together adapt LLMs to group conversation dynamics. We evaluate HUMA in a controlled study with 97 participants in four-person role-play chats, comparing AI and human community managers (CMs). Participants classified CMs as human at near-chance rates in both conditions, indicating they could not reliably distinguish HUMA agents from humans. Subjective experience was comparable across conditions: community-manager effectiveness, social presence, and engagement/satisfaction differed only modestly with small effect sizes. Our results suggest that, in natural group chat settings, an AI facilitator can match human quality while remaining difficult to identify as nonhuman.
• 摘要：基于大型语言模型 (LLM) 构建的会话代理正变得越来越普遍，但大多数系统都是为一对一、回合制交流而设计的，而不是自然的异步群聊。随着人工智能助手在从虚拟助手到客户服务的整个数字平台上广泛应用，开发自然且人性化的交互模式对于维持用户信任和参与度似乎至关重要。我们提出了类人多用户代理（HUMA），这是一种基于法学硕士的协调器，它使用类人策略和计时参与多方对话。 HUMA 通过事件驱动的架构扩展了先前的多用户聊天机器人工作，该架构可处理消息、回复、反应并引入真实的响应时间模拟。 HUMA 由三个组件组成：路由器、操作代理和反射，它们共同使 LLM 适应群组对话动态。我们在一项对照研究中评估了 HUMA，该研究有 97 名参与者参与四人角色扮演聊天，比较人工智能和人类社区管理者 (CM)。在这两种情况下，参与者将 CM 归类为人类，这表明他们无法可靠地区分 HUMA 药剂和人类。各种条件下的主观体验具有可比性：社区管理者的有效性、社会存在感和参与度/满意度仅略有不同，且效应规模较小。我们的结果表明，在自然的群聊环境中，人工智能协调员可以与人类质量相匹配，同时仍然难以被识别为非人类。

Title: Selective Rotary Position Embedding

• Authors: Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.17388
• Pdf URL: https://arxiv.org/pdf/2511.17388
• Copy Paste: [[2511.17388]] Selective Rotary Position Embedding(https://arxiv.org/abs/2511.17388)
• Keywords: language model
• Abstract: Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce \textit{Selective RoPE}, an \textit{input-dependent} rotary embedding mechanism, that generalizes \textit{RoPE}, and enables rotation in \textit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with \textit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.
• 摘要：位置信息对于语言建模至关重要。在 softmax 转换器中，旋转位置嵌入 (\textit{RoPE}) 通过 \textit{fixed-angle} 旋转对位置进行编码，而在线性转换器中，顺序是通过依赖于输入的（选择性）门控来处理的，该门控会衰减过去的键值关联。选择性通常被证明可以改善与语言相关的任务。受此启发，我们引入了 \textit{Selective RoPE}，一种 \textit{input-dependent} 旋转嵌入机制，它概括了 \textit{RoPE}，并为线性和 softmax 变换器实现了 \textit{任意角度} 的旋转。我们表明，softmax 注意力已经在查询键对上执行了这些旋转的隐藏形式，揭示了隐式位置结构。我们进一步表明，在状态空间模型和门控线性变换器中，实部管理遗忘，而虚部通过旋转对位置进行编码。我们通过为门控变压器配备 \textit{Selective RoPE} 来验证我们的方法，证明其依赖于输入的旋转提高了语言建模和复制、状态跟踪和检索等困难序列任务的性能。

Title: Beyond Multiple Choice: A Hybrid Framework for Unifying Robust Evaluation and Verifiable Reasoning Training

• Authors: Yesheng Liu, Hao Li, Haiyu Xu, Baoqi Pei, Jiahao Wang, Mingxuan Zhao, Jingshu Zheng, Zheqi He, JG Yao, Bowen Qin, Xi Yang, Jiajun Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.17405
• Pdf URL: https://arxiv.org/pdf/2511.17405
• Copy Paste: [[2511.17405]] Beyond Multiple Choice: A Hybrid Framework for Unifying Robust Evaluation and Verifiable Reasoning Training(https://arxiv.org/abs/2511.17405)
• Keywords: language model, llm
• Abstract: Multiple-choice question answering (MCQA) has been a popular format for evaluating and reinforcement fine-tuning (RFT) of modern multimodal language models. Its constrained output format allows for simplified, deterministic automatic verification. However, we find that the options may leak exploitable signals, which makes the accuracy metrics unreliable for indicating real capabilities and encourages explicit or implicit answer guessing behaviors during RFT. We propose ReVeL (Rewrite and Verify by LLM), a framework that rewrites multiple-choice questions into open-form questions while keeping answers verifiable whenever possible. The framework categorizes questions according to different answer types, apply different rewriting and verification schemes, respectively. When applied for RFT, we converted 20k MCQA examples and use GRPO to finetune Qwen2.5-VL models. Models trained on ReVeL-OpenQA match MCQA accuracy on multiple-choice benchmarks and improve OpenQA accuracy by about six percentage points, indicating better data efficiency and more robust reward signals than MCQA-based training. When used for evaluation, ReVeL also reveals up to 20 percentage points of score inflation in MCQA benchmarks (relative to OpenQA), improves judging accuracy, and reduces both cost and latency. We will release code and data publicly.
• 摘要：多项选择题回答（MCQA）已成为现代多模态语言模型评估和强化微调（RFT）的流行格式。其受限的输出格式允许简化、确定性的自动验证。然而，我们发现这些选项可能会泄漏可利用的信号，这使得准确度指标无法可靠地指示真实功能，并在 RFT 期间鼓励显式或隐式答案猜测行为。我们提出了 ReVeL（LLM 重写和验证），这是一个将多项选择题重写为开放式问题的框架，同时尽可能保持答案可验证。该框架根据不同的答案类型对问题进行分类，分别应用不同的重写和验证方案。当应用于RFT时，我们转换了20k MCQA示例并使用GRPO来微调Qwen2.5-VL模型。在 ReVeL-OpenQA 上训练的模型在多项选择基准上与 MCQA 准确性相匹配，并将 OpenQA 准确性提高了约六个百分点，这表明比基于 MCQA 的训练有更好的数据效率和更强大的奖励信号。当用于评估时，ReVeL 还揭示了 MCQA 基准中高达 20 个百分点的分数膨胀（相对于 OpenQA），提高了判断准确性，并降低了成本和延迟。我们将公开发布代码和数据。

Title: SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation

• Authors: Shrikant Kendre, Austin Xu, Honglu Zhou, Michael Ryoo, Shafiq Joty, Juan Carlos Niebles
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2511.17432
• Pdf URL: https://arxiv.org/pdf/2511.17432
• Copy Paste: [[2511.17432]] SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation(https://arxiv.org/abs/2511.17432)
• Keywords: language model, llm, hallucination
• Abstract: Traditional evaluation metrics for textual and visual question answering, like ROUGE, METEOR, and Exact Match (EM), focus heavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While measures like BERTScore and MoverScore leverage contextual embeddings to address this limitation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive evaluation. Extensive benchmarks across text, image, and video QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation.
• 摘要：文本和视觉问答的传统评估指标，如 ROUGE、METEOR 和精确匹配 (EM)，主要关注基于 n-gram 的词汇相似性，往往缺少准确评估所需的更深入的语义理解。虽然 BERTScore 和 MoverScore 等指标利用上下文嵌入来解决这一限制，但它们在平衡句子级和关键字级语义方面缺乏灵活性，并且忽略了词汇相似性，而词汇相似性仍然很重要。基于大型语言模型 (LLM) 的评估器虽然功能强大，但也存在成本高、偏差、不一致和幻觉等缺点。为了解决这些问题，我们引入了 SMILE：集成词汇准确性的语义度量，这是一种将句子级语义理解与关键字级语义理解和简单关键字匹配相结合的新颖方法。这种综合方法平衡了词汇精度和语义相关性，提供了综合评价。跨文本、图像和视频 QA 任务的广泛基准测试表明，SMILE 与人类判断高度相关，并且计算量轻，弥合了词汇和语义评估之间的差距。

Title: Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards

• Authors: Zhen Wang, Zhifeng Gao, Guolin Ke
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.17473
• Pdf URL: https://arxiv.org/pdf/2511.17473
• Copy Paste: [[2511.17473]] Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards(https://arxiv.org/abs/2511.17473)
• Keywords: language model, llm
• Abstract: Test-time scaling has been shown to substantially improve large language models' (LLMs) mathematical reasoning. However, for a large portion of mathematical corpora, especially theorem proving, RLVR's scalability is limited: intermediate reasoning is crucial, while final answers are difficult to directly and reliably verify. Meanwhile, token-level SFT often degenerates into rote memorization rather than inducing longer chains of thought. Inspired by BERT's self-supervised tasks, we propose MR-RLVR (Masked-and-Reordered RLVR), which constructs process-level self-supervised rewards via "masked-then-fill" and "step reordering" to extract learnable signals from intermediate reasoning. Our training pipeline comprises two stages: we first perform self-supervised training on sampled mathematical calculation and proof data; we then conduct RLVR fine-tuning on mathematical calculation datasets where only outcomes are verifiable. We implement MR-RLVR on Qwen2.5-3B and DeepSeek-R1-Distill-Qwen-1.5B, and evaluate on AIME24, AIME25, AMC23, and MATH500. Under a fixed sampling and decoding budget, MR-RLVR achieves average relative gains over the original RLVR of +9.86% Pass@1, +5.27% Pass@5, and +4.00% Pass@8. These results indicate that incorporating process-aware self-supervised signals can effectively enhance RLVR's scalability and performance in only outcome-verifiable settings.
• 摘要：测试时间缩放已被证明可以显着改善大型语言模型（LLM）的数学推理。然而，对于很大一部分数学语料库，尤其是定理证明，RLVR 的可扩展性是有限的：中间推理至关重要，而最终答案很难直接可靠地验证。与此同时，代币级别的 SFT 往往会退化为死记硬背，而不是引发更长的思维链。受 BERT 自监督任务的启发，我们提出了 MR-RLVR（Masked-and-Reordered RLVR），它通过“masked-then-fill”和“step reordering”构建过程级自监督奖励，从中间推理中提取可学习的信号。我们的训练流程包括两个阶段：首先对采样的数学计算和证明数据进行自监督训练；然后，我们对数学计算数据集进行 RLVR 微调，其中只有结果是可验证的。我们在 Qwen2.5-3B 和 DeepSeek-R1-Distill-Qwen-1.5B 上实现 MR-RLVR，并在 AIME24、AIME25、AMC23 和 MATH500 上进行评估。在固定采样和解码预算下，MR-RLVR 相对于原始 RLVR 实现了 +9.86% Pass@1、+5.27% Pass@5 和 +4.00% Pass@8 的平均相对增益。这些结果表明，结合过程感知的自监督信号可以有效增强 RLVR 仅在结果可验证的设置中的可扩展性和性能。