2025-08-08

Title: Enhancing Dialogue Annotation with Speaker Characteristics Leveraging a Frozen LLM

• Authors: Thomas Thebaud, Yen-Ju Lu, Matthew Wiesner, Peter Viechnicki, Najim Dehak
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2508.04795
• Pdf URL: https://arxiv.org/pdf/2508.04795
• Copy Paste: [[2508.04795]] Enhancing Dialogue Annotation with Speaker Characteristics Leveraging a Frozen LLM(https://arxiv.org/abs/2508.04795)
• Keywords: language model, llm
• Abstract: In dialogue transcription pipelines, Large Language Models (LLMs) are frequently employed in post-processing to improve grammar, punctuation, and readability. We explore a complementary post-processing step: enriching transcribed dialogues by adding metadata tags for speaker characteristics such as age, gender, and emotion. Some of the tags are global to the entire dialogue, while some are time-variant. Our approach couples frozen audio foundation models, such as Whisper or WavLM, with a frozen LLAMA language model to infer these speaker attributes, without requiring task-specific fine-tuning of either model. Using lightweight, efficient connectors to bridge audio and language representations, we achieve competitive performance on speaker profiling tasks while preserving modularity and speed. Additionally, we demonstrate that a frozen LLAMA model can compare x-vectors directly, achieving an Equal Error Rate of 8.8% in some scenarios.
• 摘要：在对话转录管道中，大型语言模型（LLMS）经常用于后处理以改善语法，标点符号和可读性。我们探讨了一个互补的后处理步骤：通过为诸如年龄，性别和情感等扬声器特征添加元数据标签来丰富转录的对话。有些标签是整个对话的全球性，而有些标签是时间变化的。我们的方法与冻结的Llama语言模型冻结了音频基础模型，例如低语或Wavlm，以推断这些说话者的属性，而无需对任何一个模型进行特定于任务的微调调整。使用轻巧，高效的连接器来桥接音频和语言表示，我们可以在扬声器分析任务上实现竞争性能，同时保持模块化和速度。此外，我们证明了冷冻的骆驼模型可以直接比较X-向量，在某些情况下达到8.8％的错误率。

Title: Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History

• Authors: Tommaso Tosato, Saskia Helbling, Yorguin-Jose Mantilla-Ramos, Mahmood Hegazy, Alberto Tosato, David John Lemay, Irina Rish, Guillaume Dumas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.04826
• Pdf URL: https://arxiv.org/pdf/2508.04826
• Copy Paste: [[2508.04826]] Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History(https://arxiv.org/abs/2508.04826)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models require consistent behavioral patterns for safe deployment, yet their personality-like traits remain poorly understood. We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25+ open-source models (1B-671B parameters) across 500,000+ responses. Using traditional (BFI-44, SD3) and novel LLM-adapted personality instruments, we systematically vary question order, paraphrasing, personas, and reasoning modes. Our findings challenge fundamental deployment assumptions: (1) Even 400B+ models exhibit substantial response variability (SD > 0.4); (2) Minor prompt reordering alone shifts personality measurements by up to 20%; (3) Interventions expected to stabilize behavior, such as chain-of-thought reasoning, detailed personas instruction, inclusion of conversation history, can paradoxically increase variability; (4) LLM-adapted instruments show equal instability to human-centric versions, confirming architectural rather than translational limitations. This persistent instability across scales and mitigation strategies suggests current LLMs lack the foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that personality-based alignment strategies may be fundamentally inadequate.
• 摘要：大型语言模型需要一致的行为模式才能安全部署，但是它们的人格状性状仍然知之甚少。我们介绍了持续存在（合成文本中的人格稳定性），这是一个全面的评估框架测试25多个开源模型（1B-671B参数），遍及500,000多个响应。使用传统（BFI-44，SD3）和新型LLM适应性的个性工具，我们会系统地改变问题顺序，释义，角色和推理模式。我们的发现挑战基本部署假设：（1）甚至400B+模型也表现出很大的响应变异性（SD> 0.4）； （2）单独迅速重新排序的次要重新排序将人格测量最多转移20％； （3）预计将稳定行为的干预措施，例如经过思考的推理，详细的人格教学，包括对话历史的历史，可以矛盾地增加变异性； （4）LLM适应的工具表现出与以人为中心的版本相等的不稳定性，确认建筑而不是翻译局限性。跨尺度和缓解策略的这种持续不稳定表明，当前的LLM缺乏真正的行为一致性的基础。对于需要可预测行为的安全至关重要的应用，这些发现表明基于人格的一致性策略在根本上可能是不足的。

Title: RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory

• Authors: Jun Liu, Zhenglun Kong, Changdi Yang, Fan Yang, Tianqi Li, Peiyan Dong, Joannah Nanjekye, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Pu Zhao, Xue Lin, Dong Huang, Yanzhi Wang
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2508.04903
• Pdf URL: https://arxiv.org/pdf/2508.04903
• Copy Paste: [[2508.04903]] RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory(https://arxiv.org/abs/2508.04903)
• Keywords: language model, llm, agent
• Abstract: Multi-agent large language model (LLM) systems have shown strong potential in complex reasoning and collaborative decision-making tasks. However, most existing coordination schemes rely on static or full-context routing strategies, which lead to excessive token consumption, redundant memory exposure, and limited adaptability across interaction rounds. We introduce RCR-Router, a modular and role-aware context routing framework designed to enable efficient, adaptive collaboration in multi-agent LLMs. To our knowledge, this is the first routing approach that dynamically selects semantically relevant memory subsets for each agent based on its role and task stage, while adhering to a strict token budget. A lightweight scoring policy guides memory selection, and agent outputs are iteratively integrated into a shared memory store to facilitate progressive context refinement. To better evaluate model behavior, we further propose an Answer Quality Score metric that captures LLM-generated explanations beyond standard QA accuracy. Experiments on three multi-hop QA benchmarks -- HotPotQA, MuSiQue, and 2WikiMultihop -- demonstrate that RCR-Router reduces token usage (up to 30%) while improving or maintaining answer quality. These results highlight the importance of structured memory routing and output-aware evaluation in advancing scalable multi-agent LLM systems.
• 摘要：多机构大语言模型（LLM）系统在复杂的推理和协作决策任务中表现出强大的潜力。但是，大多数现有的协调方案都依赖于静态或全文路由策略，这些路由策略会导致代币消耗过多，冗余内存曝光以及在互动回合中的适应性有限。我们介绍了RCR-Router，这是一个模块化和角色感知的上下文路由框架，旨在在多代理LLM中实现高效的自适应协作。据我们所知，这是第一种路由方法，它根据每个代理的角色和任务阶段动态选择语义相关的内存子集，同时遵守严格的令牌预算。轻巧的评分策略指导内存选择，并且将代理输出迭代地集成到共享内存存储中，以促进渐进的上下文改进。为了更好地评估模型行为，我们进一步提出了一个答案质量评分指标，该指标可捕获超出标准质量检查精度的LLM生成的解释。在三个多跳跃质量检查基准测试基准（HotPotQA，Musique和2Wikimultihop）上进行的实验表明，RCR-Router可以减少令牌使用率（最高30％），同时提高或维持答案质量。这些结果突出了结构化内存路由和输出感知评估在推进可扩展的多代理LLM系统中的重要性。

Title: I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations

• Authors: Julia Kharchenko, Tanya Roosta, Aman Chadha, Chirag Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.04939
• Pdf URL: https://arxiv.org/pdf/2508.04939
• Copy Paste: [[2508.04939]] I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations(https://arxiv.org/abs/2508.04939)
• Keywords: language model, llm
• Abstract: This paper introduces a comprehensive benchmark for evaluating how Large Language Models (LLMs) respond to linguistic shibboleths: subtle linguistic markers that can inadvertently reveal demographic attributes such as gender, social class, or regional background. Through carefully constructed interview simulations using 100 validated question-response pairs, we demonstrate how LLMs systematically penalize certain linguistic patterns, particularly hedging language, despite equivalent content quality. Our benchmark generates controlled linguistic variations that isolate specific phenomena while maintaining semantic equivalence, which enables the precise measurement of demographic bias in automated evaluation systems. We validate our approach along multiple linguistic dimensions, showing that hedged responses receive 25.6% lower ratings on average, and demonstrate the benchmark's effectiveness in identifying model-specific biases. This work establishes a foundational framework for detecting and measuring linguistic discrimination in AI systems, with broad applications to fairness in automated decision-making contexts.
• 摘要：本文介绍了一个全面的基准，用于评估大型语言模型（LLMS）对语言shibboleths的反应：微妙的语言标记，可以无意间揭示人口统计学属性，例如性别，社会阶层或区域背景。通过使用100个经过验证的问题响应对仔细构建的访谈模拟，我们演示了尽管内容质量等效的内容，但LLMS如何系统地惩罚某些语言模式，尤其是对冲语言。我们的基准测试产生受控的语言变化，可以在保持语义等效性的同时隔离特定现象，从而实现自动化评估系统中人口统计学偏见的精确度量。我们沿多个语言维度验证了我们的方法，表明对冲响应的平均评级降低了25.6％，并证明了基准在识别模型特异性偏见方面的有效性。这项工作建立了一个基本框架，用于检测和衡量AI系统中语言歧视，并在自动决策背景下公平地应用了广泛的应用。

Title: A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health

• Authors: Song Wang, Yishu Wei, Haotian Ma, Max Lovitt, Kelly Deng, Yuan Meng, Zihan Xu, Jingze Zhang, Yunyu Xiao, Ying Ding, Xuhai Xu, Joydeep Ghosh, Yifan Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05003
• Pdf URL: https://arxiv.org/pdf/2508.05003
• Copy Paste: [[2508.05003]] A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health(https://arxiv.org/abs/2508.05003)
• Keywords: language model, gpt
• Abstract: Background: Understanding social determinants of health (SDoH) factors contributing to suicide incidents is crucial for early intervention and prevention. However, data-driven approaches to this goal face challenges such as long-tailed factor distributions, analyzing pivotal stressors preceding suicide incidents, and limited model explainability. Methods: We present a multi-stage large language model framework to enhance SDoH factor extraction from unstructured text. Our approach was compared to other state-of-the-art language models (i.e., pre-trained BioBERT and GPT-3.5-turbo) and reasoning models (i.e., DeepSeek-R1). We also evaluated how the model's explanations help people annotate SDoH factors more quickly and accurately. The analysis included both automated comparisons and a pilot user study. Results: We show that our proposed framework demonstrated performance boosts in the overarching task of extracting SDoH factors and in the finer-grained tasks of retrieving relevant context. Additionally, we show that fine-tuning a smaller, task-specific model achieves comparable or better performance with reduced inference costs. The multi-stage design not only enhances extraction but also provides intermediate explanations, improving model explainability. Conclusions: Our approach improves both the accuracy and transparency of extracting suicide-related SDoH from unstructured texts. These advancements have the potential to support early identification of individuals at risk and inform more effective prevention strategies.
• 摘要：背景：了解造成自杀事件的健康因素（SDOH）因素（SDOH）因素对于早期干预和预防至关重要。但是，该目标的数据驱动方法面临的挑战，例如长尾因子分布，分析自杀事件之前的关键压力源以及有限的模型解释性。方法：我们提出了一个多阶段的大语言模型框架，以增强从非结构化文本中提取SDOH因子。将我们的方法与其他最先进的语言模型（即预先培训的生物Biobert和GPT-3.5-Turbo）和推理模型（即DeepSeek-R1）进行了比较。我们还评估了该模型的解释如何帮助人们更快，准确地注释SDOH因素。分析包括自动比较和试点用户研究。结果：我们表明，我们提出的框架在提取SDOH因素以及检索相关上下文的精细粒度任务中表明了绩效的提高。此外，我们表明，通过降低推理成本，微调较小的，特定于任务的模型可相当或更好地性能。多阶段设计不仅可以增强提取性，而且还提供了中间解释，从而提高了模型的解释性。结论：我们的方法提高了从非结构化文本中提取自杀相关的SDOH的准确性和透明度。这些进步有可能支持对处于危险中的个人的早期识别并为更有效的预防策略提供信息。

Title: Evaluation of LLMs in AMR Parsing

• Authors: Shu Han Ho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05028
• Pdf URL: https://arxiv.org/pdf/2508.05028
• Copy Paste: [[2508.05028]] Evaluation of LLMs in AMR Parsing(https://arxiv.org/abs/2508.05028)
• Keywords: language model, llm
• Abstract: Meaning Representation (AMR) is a semantic formalism that encodes sentence meaning as rooted, directed, acyclic graphs, where nodes represent concepts and edges denote semantic relations. Finetuning decoder only Large Language Models (LLMs) represent a promising novel straightfoward direction for AMR parsing. This paper presents a comprehensive evaluation of finetuning four distinct LLM architectures, Phi 3.5, Gemma 2, LLaMA 3.2, and DeepSeek R1 LLaMA Distilled using the LDC2020T02 Gold AMR3.0 test set. Our results have shown that straightfoward finetuning of decoder only LLMs can achieve comparable performance to complex State of the Art (SOTA) AMR parsers. Notably, LLaMA 3.2 demonstrates competitive performance against SOTA AMR parsers given a straightforward finetuning approach. We achieved SMATCH F1: 0.804 on the full LDC2020T02 test split, on par with APT + Silver (IBM) at 0.804 and approaching Graphene Smatch (MBSE) at 0.854. Across our analysis, we also observed a consistent pattern where LLaMA 3.2 leads in semantic performance while Phi 3.5 excels in structural validity.
• 摘要：含义表示（AMR）是一种语义形式主义，它编码句子含义为根，定向，无环图，其中节点代表概念和边缘表示语义关系。仅芬太尼解码器仅大型语言模型（LLMS）代表了一个有前途的新型直接方向，用于解析AMR。本文对使用LDC202020202 Gold AMR3.0测试集的四个不同的LLM体系结构，PHI 3.5，Gemma 2，Llama 3.2和DeepSeek R1 Llama进行了全面评估。我们的结果表明，Decoder的笔直填充只有LLM可以实现与复杂状态（SOTA）AMR解析器相当的性能。值得注意的是，Llama 3.2表现出对SOTA AMR解析器的竞争性能，鉴于一种直接的冠军方法。我们在完整的LDC2020T02测试拆分上实现了Smatch F1：0.804，在0.804时与APT +银（IBM）相当，并在0.854接近Graphene Smatch（MBSE）。在我们的分析中，我们还观察到了一种一致的模式，其中骆驼3.2在语义性能方面引导，而PHI 3.5在结构有效性方面表现出色。

Title: Align, Don't Divide: Revisiting the LoRA Architecture in Multi-Task Learning

• Authors: Jinda Liu, Bo Cheng, Yi Chang, Yuan Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05078
• Pdf URL: https://arxiv.org/pdf/2508.05078
• Copy Paste: [[2508.05078]] Align, Don't Divide: Revisiting the LoRA Architecture in Multi-Task Learning(https://arxiv.org/abs/2508.05078)
• Keywords: language model, llm
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models (LLMs). In practice, LLMs are often required to handle a diverse set of tasks from multiple domains, a scenario naturally addressed by multi-task learning (MTL). Within this MTL context, a prevailing trend involves LoRA variants with multiple adapters or heads, which advocate for structural diversity to capture task-specific knowledge. Our findings present a direct challenge to this paradigm. We first show that a simplified multi-head architecture with high inter-head similarity substantially outperforms complex multi-adapter and multi-head systems. This leads us to question the multi-component paradigm itself, and we further demonstrate that a standard single-adapter LoRA, with a sufficiently increased rank, also achieves highly competitive performance. These results lead us to a new hypothesis: effective MTL generalization hinges on learning robust shared representations, not isolating task-specific features. To validate this, we propose Align-LoRA, which incorporates an explicit loss to align task representations within the shared adapter space. Experiments confirm that Align-LoRA significantly surpasses all baselines, establishing a simpler yet more effective paradigm for adapting LLMs to multiple tasks. The code is available at this https URL.
• 摘要：参数有效的微调（PEFT）对于调整大语言模型（LLMS）至关重要。在实践中，通常需要LLM来处理来自多个域的各种任务，这是多任务学习（MTL）自然解决的情况。在这种MTL上下文中，主要的趋势涉及具有多个适配器或头部的Lora变体，这些变体主张结构多样性以捕获特定于任务的知识。我们的发现对此范式提出了直接挑战。我们首先表明具有高头间相似性的简化多头体系结构基本上优于复杂的多适配器和多头系统。这使我们质疑多组分范式本身，我们进一步证明，标准的单套件洛拉（Lora）具有足够的等级，也取得了高度竞争性的表现。这些结果使我们提出了一个新的假设：有效的MTL概括取决于学习强大的共享表示形式，而不是隔离特定于任务的特征。为了验证这一点，我们提出了Align-Lora，该Align-lora在共享适配器空间内包含对对齐任务表示的明显损失。实验证实，Align-Lora显着超过了所有基准，从而建立了更简单，更有效的范式，以使LLM适应多个任务。该代码可在此HTTPS URL上找到。

Title: BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation

• Authors: Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Hua Wu, Haifeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05100
• Pdf URL: https://arxiv.org/pdf/2508.05100
• Copy Paste: [[2508.05100]] BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation(https://arxiv.org/abs/2508.05100)
• Keywords: language model, llm, long context, retrieval-augmented generation
• Abstract: With the rapid advancement of large language models (LLMs), retrieval-augmented generation (RAG) has emerged as a critical approach to supplement the inherent knowledge limitations of LLMs. However, due to the typically large volume of retrieved information, RAG tends to operate with long context lengths. From the perspective of entropy engineering, we identify unconstrained entropy growth and attention dilution due to long retrieval context as significant factors affecting RAG performance. In this paper, we propose the balanced entropy-engineered RAG (BEE-RAG) framework, which improves the adaptability of RAG systems to varying context lengths through the principle of entropy invariance. By leveraging balanced context entropy to reformulate attention dynamics, BEE-RAG separates attention sensitivity from context length, ensuring a stable entropy level. Building upon this, we introduce a zero-shot inference strategy for multi-importance estimation and a parameter-efficient adaptive fine-tuning mechanism to obtain the optimal balancing factor for different settings. Extensive experiments across multiple RAG tasks demonstrate the effectiveness of BEE-RAG.
• 摘要：随着大语言模型（LLM）的快速发展，检索型发电（RAG）已成为补充LLM固有的知识限制的关键方法。但是，由于通常检索到的信息通常很大，RAG倾向于以较长的上下文长度运行。从熵工程的角度来看，我们确定由于长期检索环境而导致的无约束的熵增长和注意力稀释是影响抹布性能的重要因素。在本文中，我们提出了平衡的熵工程抹布（Bee-rag）框架，该框架通过熵不变性原理提高了抹布系统来改变上下文长度的适应性。通过利用平衡的上下文熵来重新重新注意注意力动态，蜜蜂rag将注意力灵敏度与上下文长度分开，从而确保稳定的熵水平。在此基础上，我们引入了一种零射的推理策略，以进行多元化估计和参数有效的自适应微调机制，以获得不同设置的最佳平衡因子。跨多个抹布任务的广泛实验证明了蜜蜂抹布的有效性。

Title: Attention Basin: Why Contextual Position Matters in Large Language Models

• Authors: Zihao Yi, Delong Zeng, Zhenqing Ling, Haohao Luo, Zhe Xu, Wei Liu, Jian Luan, Wanxia Cao, Ying Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05128
• Pdf URL: https://arxiv.org/pdf/2508.05128
• Copy Paste: [[2508.05128]] Attention Basin: Why Contextual Position Matters in Large Language Models(https://arxiv.org/abs/2508.05128)
• Keywords: language model, llm
• Abstract: The performance of Large Language Models (LLMs) is significantly sensitive to the contextual position of information in the input. To investigate the mechanism behind this positional bias, our extensive experiments reveal a consistent phenomenon we term the attention basin: when presented with a sequence of structured items (e.g., retrieved documents or few-shot examples), models systematically assign higher attention to the items at the beginning and end of the sequence, while neglecting those in the middle. Crucially, our analysis further reveals that allocating higher attention to critical information is key to enhancing model performance. Based on these insights, we introduce Attention-Driven Reranking (AttnRank), a two-stage framework that (i) estimates a model's intrinsic positional attention preferences using a small calibration set, and (ii) reorders retrieved documents or few-shot examples to align the most salient content with these high-attention positions. AttnRank is a model-agnostic, training-free, and plug-and-play method with minimal computational overhead. Experiments on multi-hop QA and few-shot in-context learning tasks demonstrate that AttnRank achieves substantial improvements across 10 large language models of varying architectures and scales, without modifying model parameters or training procedures.
• 摘要：大语言模型（LLMS）的性能对输入中信息的上下文位置非常敏感。为了研究这种位置偏见的机制，我们的广泛实验揭示了一种一致的现象，我们将注意力盆地称为：当出现一系列结构化项目（例如，检索到的文档或几个示例）时，模型在序列开始和结束时有系统地分配了更高的注意力，同时忽略了中间。至关重要的是，我们的分析进一步表明，将更高的关注分配给关键信息是提高模型性能的关键。基于这些见解，我们介绍了注意力驱动的重新管理（ATTNRANK），这是一个两阶段的框架，（i）使用较小的校准集估算模型的固有位置注意力偏好，（ii）回收文档或几个示例的示例以使最明显的含量与这些高度关注位置保持一致。 ATTNRANK是一种模型不足的，无训练和插件的方法，其计算开销最少。关于多跳质量质量质量检查和少数遗传的内在学习任务的实验表明，attnrank在不修改模型参数或培训程序的情况下，在不同的架构和量表的10个大语言模型上实现了实质性改进。

Title: Towards Assessing Medical Ethics from Knowledge to Practice

• Authors: Chang Hong, Minghao Wu, Qingying Xiao, Yuchi Wang, Xiang Wan, Guangjun Yu, Benyou Wang, Yan Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05132
• Pdf URL: https://arxiv.org/pdf/2508.05132
• Copy Paste: [[2508.05132]] Towards Assessing Medical Ethics from Knowledge to Practice(https://arxiv.org/abs/2508.05132)
• Keywords: language model, llm
• Abstract: The integration of large language models into healthcare necessitates a rigorous evaluation of their ethical reasoning, an area current benchmarks often overlook. We introduce PrinciplismQA, a comprehensive benchmark with 3,648 questions designed to systematically assess LLMs' alignment with core medical ethics. Grounded in Principlism, our benchmark features a high-quality dataset. This includes multiple-choice questions curated from authoritative textbooks and open-ended questions sourced from authoritative medical ethics case study literature, all validated by medical experts. Our experiments reveal a significant gap between models' ethical knowledge and their practical application, especially in dynamically applying ethical principles to real-world scenarios. Most LLMs struggle with dilemmas concerning Beneficence, often over-emphasizing other principles. Frontier closed-source models, driven by strong general capabilities, currently lead the benchmark. Notably, medical domain fine-tuning can enhance models' overall ethical competence, but further progress requires better alignment with medical ethical knowledge. PrinciplismQA offers a scalable framework to diagnose these specific ethical weaknesses, paving the way for more balanced and responsible medical AI.
• 摘要：大型语言模型纳入医疗保健需要对其道德推理进行严格的评估，当前的基准通常会忽略。我们介绍了综合基准，这是一个综合基准，其中有3,648个问题，旨在系统地评估LLMS与核心医学伦理的一致性。我们的基准以原则为基础，具有高质量的数据集。这包括从权威教科书中策划的多项选择问题以及从医学伦理学案例研究文献中提出的开放式问题，所有问题均由医学专家验证。我们的实验揭示了模型的道德知识与其实际应用之间的显着差距，尤其是在将道德原则动态应用于现实世界的情况时。大多数LLM都面临有关福利的困境，通常过度强调其他原则。 Frontier闭合源模型在强大的一般能力驱动下，目前领先于基准。值得注意的是，医学领域的微调可以增强模型的整体道德能力，但进一步的进步需要更好地与医学道德知识保持一致。 PrackiplismQA提供了一个可扩展的框架来诊断这些特定的道德弱点，为更加平衡和负责任的医学AI铺平了道路。

Title: ATLANTIS at SemEval-2025 Task 3: Detecting Hallucinated Text Spans in Question Answering

• Authors: Catherine Kobus, François Lancelot, Marion-Cécile Martin, Nawal Ould Amer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05179
• Pdf URL: https://arxiv.org/pdf/2508.05179
• Copy Paste: [[2508.05179]] ATLANTIS at SemEval-2025 Task 3: Detecting Hallucinated Text Spans in Question Answering(https://arxiv.org/abs/2508.05179)
• Keywords: language model, llm, hallucination, prompt
• Abstract: This paper presents the contributions of the ATLANTIS team to SemEval-2025 Task 3, focusing on detecting hallucinated text spans in question answering systems. Large Language Models (LLMs) have significantly advanced Natural Language Generation (NLG) but remain susceptible to hallucinations, generating incorrect or misleading content. To address this, we explored methods both with and without external context, utilizing few-shot prompting with a LLM, token-level classification or LLM fine-tuned on synthetic data. Notably, our approaches achieved top rankings in Spanish and competitive placements in English and German. This work highlights the importance of integrating relevant context to mitigate hallucinations and demonstrate the potential of fine-tuned models and prompt engineering.
• 摘要：本文介绍了Atlantis团队对Semeval-2025任务3的贡献，重点是检测有关答案系统中的幻觉文本跨度。大型语言模型（LLM）具有显着高级的自然语言生成（NLG），但仍容易受到幻觉的影响，产生了错误或误导性的内容。为了解决这个问题，我们探索了有或没有外部上下文的方法，使用LLM，令牌级别的分类或对合成数据进行了微调的LLM进行微调。值得注意的是，我们的方法在西班牙语和英语和德语中获得了最高排名。这项工作突出了整合相关环境以减轻幻觉的重要性，并证明了微调模型和及时工程的潜力。

Title: Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation

• Authors: Haonan Shangguan, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, Ge Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05234
• Pdf URL: https://arxiv.org/pdf/2508.05234
• Copy Paste: [[2508.05234]] Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation(https://arxiv.org/abs/2508.05234)
• Keywords: language model, llm, chain-of-thought
• Abstract: The surge in rich multimodal content on social media platforms has greatly advanced Multimodal Sentiment Analysis (MSA), with Large Language Models (LLMs) further accelerating progress in this field. Current approaches primarily leverage the knowledge and reasoning capabilities of parameter-heavy (Multimodal) LLMs for sentiment classification, overlooking autonomous multimodal sentiment reasoning generation in resource-constrained environments. Therefore, we focus on the Resource-Limited Joint Multimodal Sentiment Reasoning and Classification task, JMSRC, which simultaneously performs multimodal sentiment reasoning chain generation and sentiment classification only with a lightweight model. We propose a Multimodal Chain-of-Thought Reasoning Distillation model, MulCoT-RD, designed for JMSRC that employs a "Teacher-Assistant-Student" distillation paradigm to address deployment constraints in resource-limited environments. We first leverage a high-performance Multimodal Large Language Model (MLLM) to generate the initial reasoning dataset and train a medium-sized assistant model with a multi-task learning mechanism. A lightweight student model is jointly trained to perform efficient multimodal sentiment reasoning generation and classification. Extensive experiments on four datasets demonstrate that MulCoT-RD with only 3B parameters achieves strong performance on JMSRC, while exhibiting robust generalization and enhanced interpretability.
• 摘要：社交媒体平台上丰富的多模式内容的激增具有极大的高级多模式分析（MSA），大型语言模型（LLMS）进一步加速了该领域的进展。当前的方法主要利用参数重（多模式）LLM的知识和推理能力进行情感分类，从而忽略了在资源受限环境中忽略自动多模式情感推理的生成。因此，我们专注于资源有限的联合多模式情感推理和分类任务JMSRC，同时仅使用轻量级模型执行多模式情感推理链的产生和情感分类。我们建议使用专为JMSRC设计的多模式链推理蒸馏模型Mulcot-RD，该模型采用了“教师协助的”蒸馏范式来解决资源有限环境中的部署约束。我们首先利用高性能的多模式大型语言模型（MLLM）来生成初始推理数据集，并培训具有多任务学习机制的中型助手模型。轻巧的学生模型经过共同培训，以执行高效的多模式推理产生和分类。在四个数据集上进行的广泛实验表明，只有3B参数的Mulcot-RD在JMSRC上实现了强劲的性能，同时表现出强大的概括和增强的可解释性。

Title: Pruning Large Language Models by Identifying and Preserving Functional Networks

• Authors: Yiheng Liu, Junhao Ning, Sichen Xia, Xiaohui Gao, Ning Qiang, Bao Ge, Junwei Han, Xintao Hu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.05239
• Pdf URL: https://arxiv.org/pdf/2508.05239
• Copy Paste: [[2508.05239]] Pruning Large Language Models by Identifying and Preserving Functional Networks(https://arxiv.org/abs/2508.05239)
• Keywords: language model, llm
• Abstract: Structured pruning is one of the representative techniques for compressing large language models (LLMs) to reduce GPU memory consumption and accelerate inference speed. It offers significant practical value in improving the efficiency of LLMs in real-world applications. Current structured pruning methods typically rely on assessment of the importance of the structure units and pruning the units with less importance. Most of them overlooks the interaction and collaboration among artificial neurons that are crucial for the functionalities of LLMs, leading to a disruption in the macro functional architecture of LLMs and consequently a pruning performance degradation. Inspired by the inherent similarities between artificial neural networks and functional neural networks in the human brain, we alleviate this challenge and propose to prune LLMs by identifying and preserving functional networks within LLMs in this study. To achieve this, we treat an LLM as a digital brain and decompose the LLM into functional networks, analogous to identifying functional brain networks in neuroimaging data. Afterwards, an LLM is pruned by preserving the key neurons within these functional networks. Experimental results demonstrate that the proposed method can successfully identify and locate functional networks and key neurons in LLMs, enabling efficient model pruning. Our code is available at this https URL.
• 摘要：结构化修剪是压缩大语言模型（LLM）的代表性技术之一，可降低GPU记忆消耗并加速推理速度。它在提高LLM在现实世界应用中的效率方面具有重要的实用价值。当前的结构化修剪方法通常依赖于评估结构单位的重要性，并以较少的意义修剪单元。他们中的大多数人忽略了对LLMS功能至关重要的人工神经元之间的相互作用和协作，从而导致LLM的宏观功能架构的破坏，从而导致性能降低。受到人工神经网络与人脑功能神经网络之间固有的相似性的启发，我们通过在本研究中识别和保留LLMS中的功能网络来减轻这一挑战，并建议修剪LLM。为了实现这一目标，我们将LLM视为数字大脑，并将LLM分解为功能网络，类似于在神经影像数据中识别功能性脑网络。之后，通过保留这些功能网络中的关键神经元来修剪LLM。实验结果表明，所提出的方法可以成功识别和定位LLM中的功能网络和关键神经元，从而实现有效的模型修剪。我们的代码可在此HTTPS URL上找到。

Title: CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL

• Authors: Sijie Wang, Quanjiang Guo, Kai Zhao, Yawei Zhang, Xin Li, Xiang Li, Siqi Li, Rui She, Shangshu Yu, Wee Peng Tay
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05242
• Pdf URL: https://arxiv.org/pdf/2508.05242
• Copy Paste: [[2508.05242]] CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL(https://arxiv.org/abs/2508.05242)
• Keywords: language model, llm
• Abstract: Code large language models (LLMs) have become indispensable tools for building efficient and automated coding pipelines. Existing models are typically post-trained using reinforcement learning (RL) from general-purpose LLMs using "human instruction-final answer" pairs, where the instructions are usually from manual annotations. However, collecting high-quality coding instructions is both labor-intensive and difficult to scale. On the other hand, code snippets are abundantly available from various sources. This imbalance presents a major bottleneck in instruction-based post-training. We propose CodeBoost, a post-training framework that enhances code LLMs purely from code snippets, without relying on human-annotated instructions. CodeBoost introduces the following key components: (1) maximum-clique curation, which selects a representative and diverse training corpus from code; (2) bi-directional prediction, which enables the model to learn from both forward and backward prediction objectives; (3) error-aware prediction, which incorporates learning signals from both correct and incorrect outputs; (4) heterogeneous augmentation, which diversifies the training distribution to enrich code semantics; and (5) heterogeneous rewarding, which guides model learning through multiple reward types including format correctness and execution feedback from both successes and failures. Extensive experiments across several code LLMs and benchmarks verify that CodeBoost consistently improves performance, demonstrating its effectiveness as a scalable and effective training pipeline.
• 摘要：代码大语言模型（LLM）已成为建立高效和自动化编码管道的必不可少的工具。现有模型通常是使用加强学习（RL）从通用LLMS进行训练的，使用“人体指导 - 最终答案”对，其中指令通常来自手动注释。但是，收集高质量的编码说明既劳动密集型又难以扩展。另一方面，代码片段可从各种来源获得。这种不平衡给基于指导的后培训带来了主要的瓶颈。我们提出了CodeBoost，这是一个训练后框架，它纯粹是从代码片段增强代码LLM，而无需依赖于人类通知的说明。 CodeBoost介绍了以下关键组件：（1）最大关联策展，从代码中选择代表和多样化的培训语料库； （2）双向预测，使模型能够从前进和向后的预测目标中学习； （3）错误感知到的预测，该预测包含了来自正确和不正确输出的学习信号； （4）异质增强，这使训练分布多样化以丰富代码语义； （5）异构奖励，它通过多种奖励类型来指导模型学习，包括格式的正确性和来自成功和失败的执行反馈。在几个代码LLM和基准测试中进行的广泛实验证明了Codebost始终如一地提高了性能，证明了其有效性是可扩展有效的培训管道。

Title: ASCoT: An Adaptive Self-Correction Chain-of-Thought Method for Late-Stage Fragility in LLMs

• Authors: Dongxu Zhang, Ning Yang, Jihua Zhu, Jinnan Yang, Miao Xin, Baoliang Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05282
• Pdf URL: https://arxiv.org/pdf/2508.05282
• Copy Paste: [[2508.05282]] ASCoT: An Adaptive Self-Correction Chain-of-Thought Method for Late-Stage Fragility in LLMs(https://arxiv.org/abs/2508.05282)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet the reliability of these reasoning chains remains a critical challenge. A widely held "cascading failure" hypothesis suggests that errors are most detrimental when they occur early in the reasoning process. This paper challenges that assumption through systematic error-injection experiments, revealing a counter-intuitive phenomenon we term "Late-Stage Fragility": errors introduced in the later stages of a CoT chain are significantly more likely to corrupt the final answer than identical errors made at the beginning. To address this specific vulnerability, we introduce the Adaptive Self-Correction Chain-of-Thought (ASCoT) method. ASCoT employs a modular pipeline in which an Adaptive Verification Manager (AVM) operates first, followed by the Multi-Perspective Self-Correction Engine (MSCE). The AVM leverages a Positional Impact Score function I(k) that assigns different weights based on the position within the reasoning chains, addressing the Late-Stage Fragility issue by identifying and prioritizing high-risk, late-stage steps. Once these critical steps are identified, the MSCE applies robust, dual-path correction specifically to the failure parts. Extensive experiments on benchmarks such as GSM8K and MATH demonstrate that ASCoT achieves outstanding accuracy, outperforming strong baselines, including standard CoT. Our work underscores the importance of diagnosing specific failure modes in LLM reasoning and advocates for a shift from uniform verification strategies to adaptive, vulnerability-aware correction mechanisms.
• 摘要：经过思考链（COT）提示已显着提高了大语言模型（LLM）的推理能力，但是这些推理链的可靠性仍然是一个至关重要的挑战。一个广泛持有的“级联失败”假设表明，在推理过程的早期发生时，错误是最有害的。本文挑战了通过系统的错误注射实验假设，揭示了违反直觉现象，我们称为“后期脆弱性”：在COT链的后期中引入的错误比一开始时的相同错误更有可能损坏最终答案。为了解决这一特定漏洞，我们介绍了自适应自我纠正链（ASCOT）方法。 Ascot采用了模块化管道，其中自适应验证管理器（AVM）首先运行，然后是多人自我校正引擎（MSCE）。 AVM利用位置影响分数函数I（K），该功能根据推理链中的位置分配不同的权重，通过识别和优先级高风险，晚期步骤来解决晚期脆弱性问题。一旦确定了这些关键步骤，MSCE便会适用于专门针对故障零件的双路径校正。对GSM8K和MATH等基准测试的广泛实验表明，Ascot可以达到出色的精度，超过了包括标准COT在内的强大基准。我们的工作强调了诊断LLM推理中特定故障模式的重要性，并提倡从统一验证策略转变为适应性，脆弱性感知的校正机制。

Title: Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue

• Authors: Sukannya Purkayastha, Nils Dycke, Anne Lauscher, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05283
• Pdf URL: https://arxiv.org/pdf/2508.05283
• Copy Paste: [[2508.05283]] Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue(https://arxiv.org/abs/2508.05283)
• Keywords: language model, llm, agent
• Abstract: Meta-reviewing is a pivotal stage in the peer-review process, serving as the final step in determining whether a paper is recommended for acceptance. Prior research on meta-reviewing has treated this as a summarization problem over review reports. However, complementary to this perspective, meta-reviewing is a decision-making process that requires weighing reviewer arguments and placing them within a broader context. Prior research has demonstrated that decision-makers can be effectively assisted in such scenarios via dialogue agents. In line with this framing, we explore the practical challenges for realizing dialog agents that can effectively assist meta-reviewers. Concretely, we first address the issue of data scarcity for training dialogue agents by generating synthetic data using Large Language Models (LLMs) based on a self-refinement strategy to improve the relevance of these dialogues to expert domains. Our experiments demonstrate that this method produces higher-quality synthetic data and can serve as a valuable resource towards training meta-reviewing assistants. Subsequently, we utilize this data to train dialogue agents tailored for meta-reviewing and find that these agents outperform \emph{off-the-shelf} LLM-based assistants for this task. Finally, we apply our agents in real-world meta-reviewing scenarios and confirm their effectiveness in enhancing the efficiency of meta-reviewing.\footnote{Code and Data: this https URL
• 摘要：元评估是同行评审过程中的关键阶段，是确定是否建议接受论文的最后一步。先前对元评估的研究将此视为审查报告中的摘要问题。但是，互补的观点，元评估是一个决策过程，需要权衡审稿人的论点并将其置于更广泛的背景下。先前的研究表明，在这种情况下，可以通过对话代理有效地协助决策者。与此框架相一致，我们探讨了实现可以有效帮助元评论者的对话代理所面临的实际挑战。具体而言，我们首先通过基于自我注册策略生成综合数据来解决培训对话代理的数据稀缺问题，以提高这些对话与专家领域的相关性。我们的实验表明，这种方法会产生更高质量的合成数据，并可以作为培训元评估助手的宝贵资源。随后，我们利用这些数据来训练针对元评估的对话代理，并发现这些代理商的表现优于基于该任务的LLM助手。最后，我们将代理商应用于现实世界的元评估场景，并确认它们在提高元评估效率的有效性。\ footNote {代码和数据：此https url

Title: SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens

• Authors: Nikita Dragunov, Temurbek Rahmatullaev, Elizaveta Goncharova, Andrey Kuznetsov, Anton Razzhigaev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05305
• Pdf URL: https://arxiv.org/pdf/2508.05305
• Copy Paste: [[2508.05305]] SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens(https://arxiv.org/abs/2508.05305)
• Keywords: llm
• Abstract: The recently proposed Large Concept Model (LCM) generates text by predicting a sequence of sentence-level embeddings and training with either mean-squared error or diffusion objectives. We present SONAR-LLM, a decoder-only transformer that "thinks" in the same continuous SONAR embedding space, yet is supervised through token-level cross-entropy propagated via the frozen SONAR decoder. This hybrid objective retains the semantic abstraction of LCM while eliminating its diffusion sampler and restoring a likelihood-based training signal. Across model sizes from 39M to 1.3B parameters, SONAR-LLM attains competitive generation quality. We report scaling trends, ablations, benchmark results, and release the complete training code and all pretrained checkpoints to foster reproducibility and future research.
• 摘要：最近提出的大型概念模型（LCM）通过预测具有均值错误或扩散目标的一系列句子级嵌入和训练来生成文本。我们介绍了Sonar-Llm，这是一种仅解码器的变压器，在同一连续的声纳嵌入空间中“思考”，但通过通过冷冻声纳解码器传播的令牌跨透明镜进行了监督。该混合物镜保留了LCM的语义抽象，同时消除了其扩散采样器并恢复基于可能性的训练信号。在39m到1.3B参数的型号尺寸之间，Sonar-Llm具有竞争力的生成质量。我们报告缩放趋势，消融，基准结果，并发布完整的培训代码和所有验证的检查站，以促进可重复性和未来的研究。

Title: Efficient Reasoning for Large Reasoning Language Models via Certainty-Guided Reflection Suppression

• Authors: Jiameng Huang, Baijiong Lin, Guhao Feng, Jierun Chen, Di He, Lu Hou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.05337
• Pdf URL: https://arxiv.org/pdf/2508.05337
• Copy Paste: [[2508.05337]] Efficient Reasoning for Large Reasoning Language Models via Certainty-Guided Reflection Suppression(https://arxiv.org/abs/2508.05337)
• Keywords: language model, chain-of-thought
• Abstract: Recent Large Reasoning Language Models (LRLMs) employ long chain-of-thought reasoning with complex reflection behaviors, typically signaled by specific trigger words (e.g., "Wait" and "Alternatively") to enhance performance. However, these reflection behaviors can lead to the overthinking problem where the generation of redundant reasoning steps that unnecessarily increase token usage, raise inference costs, and reduce practical utility. In this paper, we propose Certainty-Guided Reflection Suppression (CGRS), a novel method that mitigates overthinking in LRLMs while maintaining reasoning accuracy. CGRS operates by dynamically suppressing the model's generation of reflection triggers when it exhibits high confidence in its current response, thereby preventing redundant reflection cycles without compromising output quality. Our approach is model-agnostic, requires no retraining or architectural modifications, and can be integrated seamlessly with existing autoregressive generation pipelines. Extensive experiments across four reasoning benchmarks (i.e., AIME24, AMC23, MATH500, and GPQA-D) demonstrate CGRS's effectiveness: it reduces token usage by an average of 18.5% to 41.9% while preserving accuracy. It also achieves the optimal balance between length reduction and performance compared to state-of-the-art baselines. These results hold consistently across model architectures (e.g., DeepSeek-R1-Distill series, QwQ-32B, and Qwen3 family) and scales (4B to 32B parameters), highlighting CGRS's practical value for efficient reasoning.
• 摘要：最近的大型推理语言模型（LRLMS）采用长期的经过思考的推理，具有复杂的反射行为，通常以特定的触发词（例如“ wait”和“替代”）来提高性能。但是，这些反思行为可能导致过度思考的问题，在这种问题中，产生不必要地增加令牌使用，提高推理成本并减少实际实用性的冗余推理步骤的产生。在本文中，我们提出了确定性引导的反射抑制（CGRS），这是一种新颖的方法，可以减轻LRLMS中的过度思考，同时保持推理精度。 CGRS通过动态抑制模型的反射触发器的生成，在其当前响应上表现出较高的信心，从而防止冗余反射周期而不会损害输出质量，从而可以动态抑制模型的反射触发器。我们的方法是模型不合时宜的，不需要重新训练或架构修改，并且可以与现有自回归的生成管道无缝集成。对四个推理基准测试（即AIME24，AMC23，MATH500和GPQA-D）进行的广泛实验证明了CGRS的有效性：它平均将令牌用法降低了18.5％至41.9％，同时保留准确性。与最先进的基线相比，它还可以达到长度降低和性能之间的最佳平衡。这些结果在模型体系结构（例如DeepSeek-R1-Distill Series，QWQ-32B和QWEN3家族）和量表（4B至32B参数）之间始终如一地保持，突出了CGRS的实践价值。

Title: Can Language Models Critique Themselves? Investigating Self-Feedback for Retrieval Augmented Generation at BioASQ 2025

• Authors: Samy Ateia, Udo Kruschwitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05366
• Pdf URL: https://arxiv.org/pdf/2508.05366
• Copy Paste: [[2508.05366]] Can Language Models Critique Themselves? Investigating Self-Feedback for Retrieval Augmented Generation at BioASQ 2025(https://arxiv.org/abs/2508.05366)
• Keywords: language model, llm, retrieval augmented generation, agent
• Abstract: Agentic Retrieval Augmented Generation (RAG) and 'deep research' systems aim to enable autonomous search processes where Large Language Models (LLMs) iteratively refine outputs. However, applying these systems to domain-specific professional search, such as biomedical research, presents challenges, as automated systems may reduce user involvement and misalign with expert information needs. Professional search tasks often demand high levels of user expertise and transparency. The BioASQ CLEF 2025 challenge, using expert-formulated questions, can serve as a platform to study these issues. We explored the performance of current reasoning and nonreasoning LLMs like Gemini-Flash 2.0, o3-mini, o4-mini and DeepSeek-R1. A key aspect of our methodology was a self-feedback mechanism where LLMs generated, evaluated, and then refined their outputs for query expansion and for multiple answer types (yes/no, factoid, list, ideal). We investigated whether this iterative self-correction improves performance and if reasoning models are more capable of generating useful feedback. Preliminary results indicate varied performance for the self-feedback strategy across models and tasks. This work offers insights into LLM self-correction and informs future work on comparing the effectiveness of LLM-generated feedback with direct human expert input in these search systems.
• 摘要：代理检索增强发电（RAG）和“深入研究”系统旨在实现自主搜索过程，其中大型语言模型（LLMS）迭代地改进了输出。但是，将这些系统应用于特定领域的专业搜索，例如生物医学研究，提出了挑战，因为自动化系统可能会减少用户参与并与专家信息需求不一致。专业的搜索任务通常需要高水平的用户专业知识和透明度。使用专家形成的问题的Bioasq Clef 2025挑战可以作为研究这些问题的平台。我们探讨了当前推理和非策划LLM的性能，例如Gemini-Flash 2.0，O3-Mini，O4-Mini和DeepSeek-R1。我们方法论的一个关键方面是一种自我反馈机制，LLMS生成，评估并完善其输出以进行查询扩展和多种答案类型（是/否，FACTOID，list，list，理想）。我们研究了这种迭代自我纠正是否可以提高性能以及推理模型是否能够产生有用的反馈。初步结果表明跨模型和任务的自我反馈策略的性能各不相同。这项工作提供了有关LLM自我纠正的见解，并为将LLM生成的反馈与直接人类专家的有效性与这些搜索系统中的直接人类专家投入进行比较有关未来的工作。

Title: MyCulture: Exploring Malaysia's Diverse Culture under Low-Resource Language Constraints

• Authors: Zhong Ken Hew, Jia Xin Low, Sze Jue Yang, Chee Seng chan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05429
• Pdf URL: https://arxiv.org/pdf/2508.05429
• Copy Paste: [[2508.05429]] MyCulture: Exploring Malaysia's Diverse Culture under Low-Resource Language Constraints(https://arxiv.org/abs/2508.05429)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) often exhibit cultural biases due to training data dominated by high-resource languages like English and Chinese. This poses challenges for accurately representing and evaluating diverse cultural contexts, particularly in low-resource language settings. To address this, we introduce MyCulture, a benchmark designed to comprehensively evaluate LLMs on Malaysian culture across six pillars: arts, attire, customs, entertainment, food, and religion presented in Bahasa Melayu. Unlike conventional benchmarks, MyCulture employs a novel open-ended multiple-choice question format without predefined options, thereby reducing guessing and mitigating format bias. We provide a theoretical justification for the effectiveness of this open-ended structure in improving both fairness and discriminative power. Furthermore, we analyze structural bias by comparing model performance on structured versus free-form outputs, and assess language bias through multilingual prompt variations. Our evaluation across a range of regional and international LLMs reveals significant disparities in cultural comprehension, highlighting the urgent need for culturally grounded and linguistically inclusive benchmarks in the development and assessment of LLMs.
• 摘要：大型语言模型（LLMS）经常由于培训数据以英语和汉语（例如英语和中文）为主导，因此通常会出现文化偏见。这对准确代表和评估各种文化背景的挑战提出了挑战，尤其是在低资源语言环境中。为了解决这个问题，我们介绍了Myculture，这是一种基准测试，旨在全面评估六个支柱的马来西亚文化LLM：艺术，服装，海关，娱乐，娱乐，食物和宗教信仰。与传统的基准不同，Myculture采用了一种新型的开放式多项选择问题格式，而无需预定义的选项，从而减少了猜测和缓解格式的偏见。我们为这种开放式结构在改善公平和歧视能力方面的有效性提供了理论上的理由。此外，我们通过比较结构化和自由形式输出的模型性能来分析结构偏差，并通过多种语言及时变化评估语言偏见。我们在一系列区域和国际LLM的评估中揭示了文化理解的显着差异，强调了对LLMS的开发和评估中对文化基础和语言上包容的基准的迫切需求。

Title: LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

• Authors: Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Yue Zhang, Junzhe Wang, Shichun Liu, Shihan Dou, Huayu Sha, Qiyuan Peng, Changhao Jiang, Jingqi Tong, Yilong Wu, Zhihao Zhang, Mingqi Wu, Zhiheng Xi, Mingxu Chai, Tao Liang, Zhihui Fei, Zhen Wang, Mingyang Wan, Guojun Ma, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05452
• Pdf URL: https://arxiv.org/pdf/2508.05452
• Copy Paste: [[2508.05452]] LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models(https://arxiv.org/abs/2508.05452)
• Keywords: language model, llm
• Abstract: Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-3, a framework for dynamic evaluation of LLMs. LLMEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, complemented by a relative ranking system for fair comparison. An 20-month longitudinal study of nearly 50 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-3 offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards.
• 摘要：对静态基准测试的大型语言模型（LLM）的现有评估容易受到数据污染和排行榜过度拟合，构成真正模型功能的关键问题。为了解决这个问题，我们介绍了LLMEVAL-3，这是一个用于LLMS动态评估的框架。 LLMEVAL-3建立在由220K研究生级问题的专有银行建立，从中为每个评估运行动态采样了看不见的测试集。其自动化管道可确保通过耐污染的数据策展，一种新型的反实际体系结构以及校准的LLM-AS-A-A-Gudge流程与人类专家达成90％一致的校准，并得到相对排名系统的补充，以进行公平比较。对近50个领先模型的20个月纵向研究揭示了有关知识记忆的表现上限，并揭示了静态基准无法检测到的数据污染漏洞。该框架在排名稳定性和一致性方面表现出了出色的鲁棒性，为动态评估范式提供了强有力的经验验证。 LLMEVAL-3提供了一种可靠和可信的方法，用于评估LLM超出排行榜分数的真正能力，从而促进了更值得信赖的评估标准的发展。

Title: TASE: Token Awareness and Structured Evaluation for Multilingual Language Models

• Authors: Chenzhuo Zhao, Xinda Wang, Yue Huang, Junting Lu, Ziqian Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05468
• Pdf URL: https://arxiv.org/pdf/2508.05468
• Copy Paste: [[2508.05468]] TASE: Token Awareness and Structured Evaluation for Multilingual Language Models(https://arxiv.org/abs/2508.05468)
• Keywords: language model, llm
• Abstract: While large language models (LLMs) have demonstrated remarkable performance on high-level semantic tasks, they often struggle with fine-grained, token-level understanding and structural reasoning--capabilities that are essential for applications requiring precision and control. We introduce TASE, a comprehensive benchmark designed to evaluate LLMs' ability to perceive and reason about token-level information across languages. TASE covers 10 tasks under two core categories: token awareness and structural understanding, spanning Chinese, English, and Korean, with a 35,927-instance evaluation set and a scalable synthetic data generation pipeline for training. Tasks include character counting, token alignment, syntactic structure parsing, and length constraint satisfaction. We evaluate over 30 leading commercial and open-source LLMs, including O3, Claude 4, Gemini 2.5 Pro, and DeepSeek-R1, and train a custom Qwen2.5-14B model using the GRPO training method. Results show that human performance significantly outpaces current LLMs, revealing persistent weaknesses in token-level reasoning. TASE sheds light on these limitations and provides a new diagnostic lens for future improvements in low-level language understanding and cross-lingual generalization. Our code and dataset are publicly available at this https URL .
• 摘要：尽管大型语言模型（LLMS）在高级语义任务上表现出了出色的表现，但他们经常在精细的，代币级别的理解和结构推理中挣扎 - 可靠性对于需要精确和控制的应用至关重要。我们介绍了TASE，这是一种综合基准，旨在评估LLMS跨语言的信息和理性信息的能力。 Tase涵盖了两个核心类别下的10个任务：象征意识和结构性理解，涵盖了中文，英语和韩语，具有35,927个实体评估集，以及可扩展的合成数据生成培训管道。任务包括字符计数，令牌对齐，句法结构解析和长度约束满意度。我们评估了30多个领先的商业和开源LLM，包括O3，Claude 4，Gemini 2.5 Pro和DeepSeek-R1，并使用GRPO培训方法训练自定义QWEN2.5-14B模型。结果表明，人类绩效显着超过了当前的LLM，揭示了令牌级别推理的持续弱点。 Tase阐明了这些局限性，并为低级语言理解和跨语性概括提供了新的诊断镜头。我们的代码和数据集可在此HTTPS URL上公开获得。

Title: Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations

• Authors: Li-Chun Lu, Miri Liu, Pin-Chun Lu, Yufei Tian, Shao-Hua Sun, Nanyun Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05470
• Pdf URL: https://arxiv.org/pdf/2508.05470
• Copy Paste: [[2508.05470]] Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations(https://arxiv.org/abs/2508.05470)
• Keywords: llm
• Abstract: We systematically examine, analyze, and compare representative creativity measures--creativity index, perplexity, syntactic templates, and LLM-as-a-Judge--across diverse creative domains, including creative writing, unconventional problem-solving, and research ideation. Our analyses reveal that these metrics exhibit limited consistency, capturing different dimensions of creativity. We highlight key limitations, including the creativity index's focus on lexical diversity, perplexity's sensitivity to model confidence, and syntactic templates' inability to capture conceptual creativity. Additionally, LLM-as-a-Judge shows instability and bias. Our findings underscore the need for more robust, generalizable evaluation frameworks that better align with human judgments of creativity.
• 摘要：我们系统地检查，分析和比较了代表性的创造力度量 - 创造性指数，困惑，句法模板和LLM-AS-A-A-a-as-Audge-a-Abross多样化的创意领域，包括创造性写作，非常常规的问题解决问题和研究基础。我们的分析表明，这些指标表现出有限的一致性，从而捕获了不同的创造力。我们重点介绍了关键局限性，包括创造力指数对词汇多样性的关注，对建模信心的敏感性以及句法模板无法捕获概念创造力的敏感性。此外，LLM-AS-A-Gudge显示出不稳定和偏见。我们的发现强调了需要更加可靠，可推广的评估框架，以更好地与人类的创造力判断保持一致。

Title: LAG: Logic-Augmented Generation from a Cartesian Perspective

• Authors: Yilin Xiao, Chuang Zhou, Qinggang Zhang, Su Dong, Shengyuan Chen, Xiao Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05509
• Pdf URL: https://arxiv.org/pdf/2508.05509
• Copy Paste: [[2508.05509]] LAG: Logic-Augmented Generation from a Cartesian Perspective(https://arxiv.org/abs/2508.05509)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, yet exhibit critical limitations in knowledge-intensive tasks, often generating hallucinations when faced with questions requiring specialized expertise. While retrieval-augmented generation (RAG) mitigates this by integrating external knowledge, it struggles with complex reasoning scenarios due to its reliance on direct semantic retrieval and lack of structured logical organization. Inspired by Cartesian principles from \textit{Discours de la méthode}, this paper introduces Logic-Augmented Generation (LAG), a novel paradigm that reframes knowledge augmentation through systematic question decomposition and dependency-aware reasoning. Specifically, LAG first decomposes complex questions into atomic sub-questions ordered by logical dependencies. It then resolves these sequentially, using prior answers to guide context retrieval for subsequent sub-questions, ensuring stepwise grounding in logical chain. To prevent error propagation, LAG incorporates a logical termination mechanism that halts inference upon encountering unanswerable sub-questions and reduces wasted computation on excessive reasoning. Finally, it synthesizes all sub-resolutions to generate verified responses. Experiments on four benchmark datasets demonstrate that LAG significantly enhances reasoning robustness, reduces hallucination, and aligns LLM problem-solving with human cognition, offering a principled alternative to existing RAG systems.
• 摘要：大型语言模型（LLMS）在各种任务中都表现出了出色的功能，但在知识密集型任务中表现出关键的局限性，当面对需要专业知识的问题时，通常会产生幻觉。尽管检索型发电（RAG）通过整合外部知识来减轻这种情况，但由于其依赖直接的语义检索和缺乏结构化逻辑组织的依赖，它在复杂的推理方案中挣扎。受\ textit {Discours delaMéthode}的笛卡尔原理的启发，本文介绍了逻辑增强的一代（lag），这是一种新颖的范式，该范式通过系统的问题分解和依赖性意识到的推理来逆转知识增强。具体而言，滞后首先将复杂的问题分解为逻辑依赖性排序的原子亚问题。然后，它使用先前的答案依次解决这些问题，以指导上下文检索后续子问题，从而确保逻辑链中的逐步接地。为了防止误差传播，滞后结合了逻辑终止机制，该机制在遇到无法解答的子问题和减少的计算时停止了推断。最后，它综合了所有子分辨率以生成经过验证的响应。四个基准数据集的实验表明，滞后可显着增强推理的鲁棒性，降低幻觉，并使LLM问题解决与人类认知相结合，并提供了现有抹布系统的原则性替代方案。

Title: The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities

• Authors: Harsh Nishant Lalai, Raj Sanjay Shah, Jiaxin Pei, Sashank Varma, Yi-Chia Wang, Ali Emami
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05525
• Pdf URL: https://arxiv.org/pdf/2508.05525
• Copy Paste: [[2508.05525]] The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities(https://arxiv.org/abs/2508.05525)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have been extensively tuned to mitigate explicit biases, yet they often exhibit subtle implicit biases rooted in their pre-training data. Rather than directly probing LLMs with human-crafted questions that may trigger guardrails, we propose studying how models behave when they proactively ask questions themselves. The 20 Questions game, a multi-turn deduction task, serves as an ideal testbed for this purpose. We systematically evaluate geographic performance disparities in entity deduction using a new dataset, Geo20Q+, consisting of both notable people and culturally significant objects (e.g., foods, landmarks, animals) from diverse regions. We test popular LLMs across two gameplay configurations (canonical 20-question and unlimited turns) and in seven languages (English, Hindi, Mandarin, Japanese, French, Spanish, and Turkish). Our results reveal geographic disparities: LLMs are substantially more successful at deducing entities from the Global North than the Global South, and the Global West than the Global East. While Wikipedia pageviews and pre-training corpus frequency correlate mildly with performance, they fail to fully explain these disparities. Notably, the language in which the game is played has minimal impact on performance gaps. These findings demonstrate the value of creative, free-form evaluation frameworks for uncovering subtle biases in LLMs that remain hidden in standard prompting setups. By analyzing how models initiate and pursue reasoning goals over multiple turns, we find geographic and cultural disparities embedded in their reasoning processes. We release the dataset (Geo20Q+) and code at this https URL.
• 摘要：大型语言模型（LLM）已被广泛调整以减轻明确的偏见，但它们经常表现出植根于训练数据的微妙隐式偏见。我们没有直接探测可能引发护栏的人工问题的问题，而是建议研究模型自己主动提出问题时的行为。为此目的，这20个问题游戏是一项多重扣除任务，是理想的测试台。我们使用新的数据集（GEO20Q+）系统地评估实体扣除的地理性能差异，该数据集由来自不同地区的著名人物和具有文化意义的物体（例如食品，地标，动物）组成。我们在两种游戏配置（规范的20个问题和无限的转弯）和七种语言（英语，印地语，印地语，普通话，日语，法语，西班牙语和土耳其语）中测试了流行的LLM。我们的结果表明，地理差异：LLMS比全球南部的全球北部和全球西部比全球东部的实体更为成功。尽管Wikipedia pageviews和预训练语料库频率与性能温和相关，但他们无法完全解释这些差异。值得注意的是，游戏的语言对性能差距的影响最小。这些发现证明了创意，自由形式评估框架的价值对于发现LLM中的微妙偏见仍然隐藏在标准提示设置中。通过分析模型如何启动和通过多个转弯实现推理目标，我们发现地理和文化差异嵌入了其推理过程中。我们在此HTTPS URL上发布数据集（GEO20Q+）和代码。

Title: CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation

• Authors: Santosh T.Y.S.S, Youssef Tarek Elkhayat, Oana Ichim, Pranav Shetty, Dongsheng Wang, Zhiqiang Ma, Armineh Nourbakhsh, Xiaomo Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05534
• Pdf URL: https://arxiv.org/pdf/2508.05534
• Copy Paste: [[2508.05534]] CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation(https://arxiv.org/abs/2508.05534)
• Keywords: llm, retrieval-augmented generation
• Abstract: Due to their ability to process long and complex contexts, LLMs can offer key benefits to the Legal domain, but their adoption has been hindered by their tendency to generate unfaithful, ungrounded, or hallucinatory outputs. While Retrieval-Augmented Generation offers a promising solution by grounding generations in external knowledge, it offers no guarantee that the provided context will be effectively integrated. To address this, context-aware decoding strategies have been proposed to amplify the influence of relevant context, but they usually do not explicitly enforce faithfulness to the context. In this work, we introduce Confidence-guided Copy-based Decoding for Legal Text Generation (CoCoLex)-a decoding strategy that dynamically interpolates the model produced vocabulary distribution with a distribution derived based on copying from the context. CoCoLex encourages direct copying based on the model's confidence, ensuring greater fidelity to the source. Experimental results on five legal benchmarks demonstrate that CoCoLex outperforms existing context-aware decoding methods, particularly in long-form generation tasks.
• 摘要：由于其处理漫长而复杂的环境的能力，LLM可以为法律领域提供关键的好处，但是由于它们产生不忠，不可思议或幻觉的产出的倾向，他们的采用受到了阻碍。虽然检索型的一代通过将几代人扎根于外部知识提供了有希望的解决方案，但它不能保证提供的上下文将有效地整合。为了解决这个问题，已经提出了上下文感知的解码策略来扩大相关背景的影响，但它们通常不会明确地对上下文实现忠诚。在这项工作中，我们介绍了针对法律文本生成（Cocolex）的基于信心引导的基于副本的解码 - 一种解码策略，该策略会通过基于从上下文的复制而得出的分布来动态地插入模型产生的词汇分布。 Cocolex会根据模型的信心鼓励直接复制，从而确保对来源的更大忠诚。五个法律基准的实验结果表明，Cocolex的表现优于现有的上下文感知解码方法，尤其是在长期生成任务中。

Title: Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees

• Authors: Guang Yang, Xinyang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05544
• Pdf URL: https://arxiv.org/pdf/2508.05544
• Copy Paste: [[2508.05544]] Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees(https://arxiv.org/abs/2508.05544)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have shown remarkable progress in multiple-choice question answering (MCQA), but their inherent unreliability, such as hallucination and overconfidence, limits their application in high-risk domains. To address this, we propose a frequency-based uncertainty quantification method under black-box settings, leveraging conformal prediction (CP) to ensure provable coverage guarantees. Our approach involves multiple independent samplings of the model's output distribution for each input, with the most frequent sample serving as a reference to calculate predictive entropy (PE). Experimental evaluations across six LLMs and four datasets (MedMCQA, MedQA, MMLU, MMLU-Pro) demonstrate that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions, as measured by AUROC. Furthermore, the method effectively controls the empirical miscoverage rate under user-specified risk levels, validating that sampling frequency can serve as a viable substitute for logit-based probabilities in black-box scenarios. This work provides a distribution-free model-agnostic framework for reliable uncertainty quantification in MCQA with guaranteed coverage, enhancing the trustworthiness of LLMs in practical applications.
• 摘要：大型语言模型（LLMS）在多项选择问题答案（MCQA）中表现出了很大的进步，但是它们固有的不可靠性（例如幻觉和过度自信）限制了它们在高风险域中的应用。为了解决这个问题，我们提出了一种基于频率的不确定性量化方法在黑框设置下，利用保形预测（CP）来确保可证明的覆盖范围保证。我们的方法涉及对每个输入的模型输出分布的多个独立采样，最频繁的样本用作计算预测熵（PE）的参考。跨六个LLM和四个数据集（MEDMCQA，MEDQA，MMLU，MMLU-PRO）进行的实验评估表明，基于频率的PE优于基于logit的PE，可以通过AUROC测量，以区分正确和错误的预测。此外，该方法有效地控制了用户指定风险水平下的经验误差率，从而证明了采样频率可以作为黑盒情景中基于logit的概率的可行替代品。这项工作为MCQA中可靠的不确定性量化提供了一个无分配模型的框架框架，并保证了覆盖范围，从而增强了LLM在实际应用中的可信度。

Title: Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs

• Authors: Franziska Weeber, Tanise Ceron, Sebastian Padó
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2508.05553
• Pdf URL: https://arxiv.org/pdf/2508.05553
• Copy Paste: [[2508.05553]] Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs(https://arxiv.org/abs/2508.05553)
• Keywords: language model, llm, prompt
• Abstract: Public opinion surveys show cross-cultural differences in political opinions between socio-cultural contexts. However, there is no clear evidence whether these differences translate to cross-lingual differences in multilingual large language models (MLLMs). We analyze whether opinions transfer between languages or whether there are separate opinions for each language in MLLMs of various sizes across five Western languages. We evaluate MLLMs' opinions by prompting them to report their (dis)agreement with political statements from voting advice applications. To better understand the interaction between languages in the models, we evaluate them both before and after aligning them with more left or right views using direct preference optimization and English alignment data only. Our findings reveal that unaligned models show only very few significant cross-lingual differences in the political opinions they reflect. The political alignment shifts opinions almost uniformly across all five languages. We conclude that in Western language contexts, political opinions transfer between languages, demonstrating the challenges in achieving explicit socio-linguistic, cultural, and political alignment of MLLMs.
• 摘要：公众舆论调查显示社会文化背景之间的政治意见跨文化差异。但是，尚无明确的证据，这些差异是否转化为多语言大语模型（MLLM）的跨语化差异。我们分析语言之间的意见转移，还是在五种西方语言的各种规模的MLLM中对每种语言都有单独的意见。我们通过促使他们报告其（DIS）协议，并通过投票建议申请的政治陈述来评估MLLM的意见。为了更好地理解模型中语言之间的相互作用，我们仅使用直接偏好优化和英语对齐数据对它们进行更左右视图对齐它们。我们的发现表明，未对齐的模型在它们所反映的政治观点中仅显示很少有重大的跨语性差异。政治一致性几乎在所有五种语言中都统一地转移了意见。我们得出的结论是，在西方语言的背景下，政治观点在语言之间转移，表明在实现MLLM的明确社会语言，文化和政治统一方面所面临的挑战。

Title: MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy

• Authors: Shaoxiong Zhan, Yanlin Lai, Ziyu Lu, Dahua Lin, Ziqing Yang, Fei Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05592
• Pdf URL: https://arxiv.org/pdf/2508.05592
• Copy Paste: [[2508.05592]] MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy(https://arxiv.org/abs/2508.05592)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models have achieved substantial progress in mathematical reasoning, yet their advancement is limited by the scarcity of high-quality, high-difficulty training data. Existing synthesis methods largely rely on transforming human-written templates, limiting both diversity and scalability. We propose MathSmith, a novel framework for synthesizing challenging mathematical problems to enhance LLM reasoning. Rather than modifying existing problems, MathSmith constructs new ones from scratch by randomly sampling concept-explanation pairs from PlanetMath, ensuring data independence and avoiding contamination. To increase difficulty, we design nine predefined strategies as soft constraints during rationales. We further adopts reinforcement learning to jointly optimize structural validity, reasoning complexity, and answer consistency. The length of the reasoning trace generated under autoregressive prompting is used to reflect cognitive complexity, encouraging the creation of more demanding problems aligned with long-chain-of-thought reasoning. Experiments across five benchmarks, categorized as easy & medium (GSM8K, MATH-500) and hard (AIME2024, AIME2025, OlympiadBench), show that MathSmith consistently outperforms existing baselines under both short and long CoT settings. Additionally, a weakness-focused variant generation module enables targeted improvement on specific concepts. Overall, MathSmith exhibits strong scalability, generalization, and transferability, highlighting the promise of high-difficulty synthetic data in advancing LLM reasoning capabilities.
• 摘要：大型语言模型在数学推理方面取得了重大进展，但是它们的进步受到高质量，高缺乏培训数据的稀缺的限制。现有的综合方法在很大程度上依赖于转换人写的模板，从而限制了多样性和可扩展性。我们提出了Mathsmith，这是一个新颖的框架，用于综合具有挑战性的数学问题以增强LLM推理。 Mathsmith并没有修改现有问题，而是通过从星球上随机采样概念解释对来构建新问题，从而确保数据独立性并避免污染。为了增加难度，我们将九种预定义策略设计为理由期间的软限制。我们进一步采用强化学习来共同优化结构有效性，推理的复杂性和答案一致性。在自回归提示下产生的推理痕迹的长度用于反映认知复杂性，鼓励创建与长期链链的推理相一致的更苛刻的问题。跨五个基准测试的实验，分类为Easy＆Medium（GSM8K，Math-500）和Hard（Aime2024，Aime2025，OlympiaDbench），表明Mathsmith在短和长cot设置下始终超过现有的基准。此外，以弱点为重点的变体生成模块可以针对特定概念进行针对性的改进。总体而言，数学史密斯具有强大的可扩展性，概括性和可传递性，突出了高缺陷综合数据在提高LLM推理能力方面的希望。

Title: Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models

• Authors: Haitao Hong, Yuchen Yan, Xingyu Wu, Guiyang Hou, Wenqi Zhang, Weiming Lu, Yongliang Shen, Jun Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05613
• Pdf URL: https://arxiv.org/pdf/2508.05613
• Copy Paste: [[2508.05613]] Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models(https://arxiv.org/abs/2508.05613)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable performance in reasoning tasks, where reinforcement learning (RL) serves as a key algorithm for enhancing their reasoning capabilities. Currently, there are two mainstream reward paradigms: model-based rewards and rule-based rewards. However, both approaches suffer from limitations: rule-based rewards lack robustness, while model-based rewards are vulnerable to reward hacking. To address these issues, we propose Cooper(Co-optimizing Policy Model and Reward Model), a RL framework that jointly optimizes both the policy model and the reward model. Cooper leverages the high precision of rule-based rewards when identifying correct responses, and dynamically constructs and selects positive-negative sample pairs for continued training the reward model. This design enhances robustness and mitigates the risk of reward hacking. To further support Cooper, we introduce a hybrid annotation strategy that efficiently and accurately generates training data for the reward model. We also propose a reference-based reward modeling paradigm, where the reward model takes a reference answer as input. Based on this design, we train a reward model named VerifyRM, which achieves higher accuracy on VerifyBench compared to other models of the same size. We conduct reinforcement learning using both VerifyRM and Cooper. Our experiments show that Cooper not only alleviates reward hacking but also improves end-to-end RL performance, for instance, achieving a 0.54% gain in average accuracy on Qwen2.5-1.5B-Instruct. Our findings demonstrate that dynamically updating reward model is an effective way to combat reward hacking, providing a reference for better integrating reward models into RL.
• 摘要：大型语言模型（LLMS）在推理任务中表现出了出色的表现，在推理任务中，强化学习（RL）是增强其推理能力的关键算法。当前，有两个主流奖励范式：基于模型的奖励和基于规则的奖励。但是，两种方法都遭受了局限性：基于规则的奖励缺乏鲁棒性，而基于模型的奖励容易受到奖励黑客的影响。为了解决这些问题，我们提出了库珀（优化政策模型和奖励模型），这是一个共同优化政策模型和奖励模型的RL框架。库珀在确定正确的响应时利用基于规则的奖励的高精度，并动态构建并选择正面的样本对继续训练奖励模型。这种设计增强了鲁棒性，并减轻了奖励黑客的风险。为了进一步支持库珀，我们引入了一种混合注释策略，该策略有效，准确地为奖励模型生成了培训数据。我们还提出了一个基于参考的奖励建模范式，其中奖励模型将参考答案作为输入。基于此设计，我们训练一个名为VerifyRM的奖励模型，该模型与其他相同尺寸的模型相比，该模型在验证基地上的准确性更高。我们使用VerifyRM和Cooper进行增强学习。我们的实验表明，库珀不仅减轻了奖励黑客攻击，还可以改善端到端的RL性能，例如，在QWEN2.5-1.5B-Instruct上的平均准确性达到0.54％。我们的发现表明，动态更新奖励模型是对抗奖励黑客入侵的有效方法，为将奖励模型更好地整合到RL中提供了参考。

Title: OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks

• Authors: Zixuan Wang, Dingming Li, Hongxing Li, Shuo Chen, Yuchen Yan, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, Yueting Zhuang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05614
• Pdf URL: https://arxiv.org/pdf/2508.05614
• Copy Paste: [[2508.05614]] OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks(https://arxiv.org/abs/2508.05614)
• Keywords: language model, agent
• Abstract: Large language models excel at abstract reasoning but their capacity for embodied agent reasoning remains largely unexplored. We present OmniEAR, a comprehensive framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks. Unlike existing benchmarks that provide predefined tool sets or explicit collaboration directives, OmniEAR requires agents to dynamically acquire capabilities and autonomously determine coordination strategies based on task demands. Through text-based environment representation, we model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains. Our systematic evaluation reveals severe performance degradation when models must reason from constraints: while achieving 85-96% success with explicit instructions, performance drops to 56-85% for tool reasoning and 63-85% for implicit collaboration, with compound tasks showing over 50% failure rates. Surprisingly, complete environmental information degrades coordination performance, indicating models cannot filter task-relevant constraints. Fine-tuning improves single-agent tasks dramatically (0.6% to 76.3%) but yields minimal multi-agent gains (1.5% to 5.5%), exposing fundamental architectural limitations. These findings demonstrate that embodied reasoning poses fundamentally different challenges than current models can address, establishing OmniEAR as a rigorous benchmark for evaluating and advancing embodied AI systems. Our code and data are included in the supplementary materials and will be open-sourced upon acceptance.
• 摘要：大型语言模型在抽象推理方面表现出色，但其体现的代理推理的能力仍然在很大程度上没有探索。我们提出了Omniear，这是一个综合框架，用于评估语言模型如何理解体现任务中的物理互动，工具使用和多代理协调。与提供预定义工具集或显式协作指令的现有基准不同，Omniear要求代理商以动态获取功能并根据任务需求自主确定协调策略。通过基于文本的环境表示，我们在跨越家庭和工业领域的1,500个场景中建模了连续的物理特性和复杂的空间关系。当模型必须从约束中推理时，我们的系统评估显示出严重的性能下降：通过明确指示获得85-96％的成功，工具推理的绩效下降到56-85％，而对于隐式协作而言63-85％，复合任务显示出超过50％的失败率。令人惊讶的是，完整的环境信息降低了协调性能，表明模型无法过滤与任务相关的约束。微调可显着提高单一代理任务（0.6％至76.3％），但产生最小的多代理收益（1.5％至5.5％），暴露了基本的建筑限制。这些发现表明，体现的推理与当前模型所能解决的根本不同的挑战构成了不同的挑战，并确立了无所不知的基准，以评估和前进体现的AI系统。我们的代码和数据包含在补充材料中，并将在接受后开源。

Title: Learning to Reason for Factuality

• Authors: Xilun Chen, Ilia Kulikov, Vincent-Pierre Berges, Barlas Oğuz, Rulin Shao, Gargi Ghosh, Jason Weston, Wen-tau Yih
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.05618
• Pdf URL: https://arxiv.org/pdf/2508.05618
• Copy Paste: [[2508.05618]] Learning to Reason for Factuality(https://arxiv.org/abs/2508.05618)
• Keywords: language model, llm, hallucination
• Abstract: Reasoning Large Language Models (R-LLMs) have significantly advanced complex reasoning tasks but often struggle with factuality, generating substantially more hallucinations than their non-reasoning counterparts on long-form factuality benchmarks. However, extending online Reinforcement Learning (RL), a key component in recent R-LLM advancements, to the long-form factuality setting poses several unique challenges due to the lack of reliable verification methods. Previous work has utilized automatic factuality evaluation frameworks such as FActScore to curate preference data in the offline RL setting, yet we find that directly leveraging such methods as the reward in online RL leads to reward hacking in multiple ways, such as producing less detailed or relevant responses. We propose a novel reward function that simultaneously considers the factual precision, response detail level, and answer relevance, and applies online RL to learn high quality factual reasoning. Evaluated on six long-form factuality benchmarks, our factual reasoning model achieves an average reduction of 23.1 percentage points in hallucination rate, a 23% increase in answer detail level, and no degradation in the overall response helpfulness.
• 摘要：推理大语言模型（R-LLS）具有明显高级的复杂推理任务，但通常在事实上挣扎，比在长期的事实基准上产生的幻觉要大得多。但是，扩展在线加强学习（RL）是最近R-LLM进步中的关键组成部分，因为缺乏可靠的验证方法，长期化的事实设置构成了一些独特的挑战。先前的工作已经利用了自动事实评估框架，例如FactScore来策划离线RL设置中的偏好数据，但是我们发现，直接利用在线RL中的奖励等方法会导致以多种方式奖励黑客奖励，例如产生较少详细或相关的响应。我们提出了一种新颖的奖励功能，同时考虑了事实的精度，响应细节级别和回答相关性，并应用在线RL来学习高质量的事实推理。在六个长形式的事实基准测试中，我们的事实推理模型的平均幻觉率平均降低了23.1个百分点，答案细节水平增加了23％，并且整体响应有益的帮助没有降解。

Title: How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn Conversations

• Authors: Brandon Jaipersaud, David Krueger, Ekdeep Singh Lubana
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.05625
• Pdf URL: https://arxiv.org/pdf/2508.05625
• Copy Paste: [[2508.05625]] How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn Conversations(https://arxiv.org/abs/2508.05625)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have started to demonstrate the ability to persuade humans, yet our understanding of how this dynamic transpires is limited. Recent work has used linear probes, lightweight tools for analyzing model representations, to study various LLM skills such as the ability to model user sentiment and political perspective. Motivated by this, we apply probes to study persuasion dynamics in natural, multi-turn conversations. We leverage insights from cognitive science to train probes on distinct aspects of persuasion: persuasion success, persuadee personality, and persuasion strategy. Despite their simplicity, we show that they capture various aspects of persuasion at both the sample and dataset levels. For instance, probes can identify the point in a conversation where the persuadee was persuaded or where persuasive success generally occurs across the entire dataset. We also show that in addition to being faster than expensive prompting-based approaches, probes can do just as well and even outperform prompting in some settings, such as when uncovering persuasion strategy. This suggests probes as a plausible avenue for studying other complex behaviours such as deception and manipulation, especially in multi-turn settings and large-scale dataset analysis where prompting-based methods would be computationally inefficient.
• 摘要：大型语言模型（LLM）已经开始证明说服人类的能力，但是我们对这种动态反相是有限的理解。最近的工作使用了线性探针，轻量级工具来分析模型表示，研究各种LLM技能，例如对用户情感和政治观点进行建模的能力。在此激励的情况下，我们将探针应用于自然，多转向对话中的说服力。我们利用认知科学的见解来培训有关说服力的不同方面的探针：说服成功，说服力的性格和说服策略。尽管它们很简单，但我们表明它们在样本和数据集级别上都捕捉了说服力的各个方面。例如，探针可以识别说服说服的对话中的点，或者说服力成功在整个数据集中通常发生。我们还表明，除了比基于昂贵的提示方法快的速度外，探针还可以做得同样甚至超越某些设置的提示，例如在发现说服力策略时。这表明探针是研究其他复杂行为（例如欺骗和操纵）的合理途径，尤其是在多转弯设置和大规模数据集分析中，基于促进的方法将在计算上效率低下。

Title: H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages

• Authors: Mehrdad Zakershahrak, Samira Ghodratnama
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.05628
• Pdf URL: https://arxiv.org/pdf/2508.05628
• Copy Paste: [[2508.05628]] H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages(https://arxiv.org/abs/2508.05628)
• Keywords: language model, gpt
• Abstract: Byte-level language models eliminate fragile tokenizers but face computational challenges in morphologically-rich languages (MRLs), where words span many bytes. We propose H-NET++, a hierarchical dynamic-chunking model that learns linguistically-informed segmentation through end-to-end training. Key innovations include: (1) a lightweight Transformer context-mixer (1.9M parameters) for cross-chunk attention, (2) a two-level latent hyper-prior for document-level consistency, (3) specialized handling of orthographic artifacts (e.g. Persian ZWNJ), and (4) curriculum-based training with staged sequence lengths. On a 1.4B-token Persian corpus, H-NET++ achieves state-of-the-art results: 0.159 BPB reduction versus BPE-based GPT-2-fa (12% better compression), 5.4pp gain on ParsGLUE, 53% improved robustness to ZWNJ corruption, and 73.8% F1 on gold morphological boundaries. Our learned chunks align with Persian morphology without explicit supervision, demonstrating that hierarchical dynamic chunking provides an effective tokenizer-free solution for MRLs while maintaining computational efficiency.
• 摘要：字节级的语言模型消除了脆弱的引物，但面临形态上富裕语言（MRLS）的计算挑战，其中单词涵盖了许多字节。我们提出了H-NET ++，这是一个分层动态型造型模型，该模型通过端到端培训学习语言信息的分割。关键创新包括：（1）轻巧的变压器上下文混合物（190万参数），用于跨块注意，（2）具有文档级一致性的两级潜在超级优先级，（3）对拼字法的专业处理（例如，波斯语ZWNJ），以及（4）基于基于仪表式序列的课程培训。在1.4b token的波斯语料库上，H-NET ++取得了最先进的结果：0.159 BPB减少与基于BPE的GPT-2-FA（增强压缩更好12％），Parsglue的5.4pp增长，53％的增长率为53％，提高了ZWNJ腐败的稳健性，对金色Merphodicalies的稳健性提高了73.8％F1。我们学到的块与波斯的形态保持一致，没有明确的监督，这表明层次的动态块为MRL提供了有效的无令牌解决方案，同时保持计算效率。