2025-05-14

Title: Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces

• Authors: Michael Pichat, William Pogrund, Paloma Pichat, Judicael Poumay, Armanouche Gasparian, Samuel Demarchi, Martin Corbet, Alois Georgeon, Michael Veillet-Guillem
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.07831
• Pdf URL: https://arxiv.org/pdf/2505.07831
• Copy Paste: [[2505.07831]] Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces(https://arxiv.org/abs/2505.07831)
• Keywords: language model
• Abstract: The polysemantic nature of synthetic neurons in artificial intelligence language models is currently understood as the result of a necessary superposition of distributed features within the latent space. We propose an alternative approach, geometrically defining a neuron in layer n as a categorical vector space with a non-orthogonal basis, composed of categorical sub-dimensions extracted from preceding neurons in layer n-1. This categorical vector space is structured by the activation space of each neuron and enables, via an intra-neuronal attention process, the identification and utilization of a critical categorical zone for the efficiency of the language model - more homogeneous and located at the intersection of these different categorical sub-dimensions.
• 摘要：当前，人工智能语言模型中合成神经元的多性性性质被理解为潜在空间内分布式特征必要的叠加的结果。我们提出了一种替代方法，几何方法将n层的神经元定义为具有非正交基础的分类矢量空间，该空间由从N-1层中的前一个神经元中提取的分类子量措施组成。该分类矢量空间是由每个神经元的激活空间结构的，并通过神经内的注意力过程，鉴定和利用语言模型效率的关键分类区域的识别和利用 - 更均匀，位于这些不同的分类子测量的交叉点。

Title: A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas

• Authors: Pranav Narayanan Venkit, Jiayi Li, Yingfan Zhou, Sarah Rajtmajer, Shomir Wilson
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2505.07850
• Pdf URL: https://arxiv.org/pdf/2505.07850
• Copy Paste: [[2505.07850]] A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas(https://arxiv.org/abs/2505.07850)
• Keywords: language model, gpt, llm
• Abstract: As LLMs (large language models) are increasingly used to generate synthetic personas particularly in data-limited domains such as health, privacy, and HCI, it becomes necessary to understand how these narratives represent identity, especially that of minority communities. In this paper, we audit synthetic personas generated by 3 LLMs (GPT4o, Gemini 1.5 Pro, Deepseek 2.5) through the lens of representational harm, focusing specifically on racial identity. Using a mixed methods approach combining close reading, lexical analysis, and a parameterized creativity framework, we compare 1512 LLM generated personas to human-authored responses. Our findings reveal that LLMs disproportionately foreground racial markers, overproduce culturally coded language, and construct personas that are syntactically elaborate yet narratively reductive. These patterns result in a range of sociotechnical harms, including stereotyping, exoticism, erasure, and benevolent bias, that are often obfuscated by superficially positive narrations. We formalize this phenomenon as algorithmic othering, where minoritized identities are rendered hypervisible but less authentic. Based on these findings, we offer design recommendations for narrative-aware evaluation metrics and community-centered validation protocols for synthetic identity generation.
• 摘要：由于LLM（大型语言模型）越来越多地用于产生合成角色，尤其是在诸如健康，隐私和HCI之类的数据限制领域中，因此有必要了解这些叙述如何代表身份，尤其是少数群体的身份。在本文中，我们通过代表性危害的镜头审核3个LLM（GPT4O，Gemini 1.5 Pro，DeepSeek 2.5）产生的合成角色，专门针对种族认同。使用混合方法方法，结合了近距离阅读，词汇分析和参数化的创造力框架，我们将1512 LLM生成的角色与人为实现的响应进行比较。我们的发现表明，LLMS不成比例地前景种族标记，过度生产文化编码的语言以及构建句法详尽但叙事上还原性的角色。这些模式导致一系列社会技术危害，包括刻板印象，外来主义，擦除和仁慈的偏见，这些偏见通常被表面上积极的叙述所困扰。我们将这种现象形式化为算法其他，在这种情况下，少数化的身份被变得过度允许但不那么真实。基于这些发现，我们为叙事感知的评估指标和以社区为中心的验证协议提供了设计建议。

Title: Joint Detection of Fraud and Concept Drift inOnline Conversations with LLM-Assisted Judgment

• Authors: Ali Senol, Garima Agrawal, Huan Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.07852
• Pdf URL: https://arxiv.org/pdf/2505.07852
• Copy Paste: [[2505.07852]] Joint Detection of Fraud and Concept Drift inOnline Conversations with LLM-Assisted Judgment(https://arxiv.org/abs/2505.07852)
• Keywords: language model, llm, chat
• Abstract: Detecting fake interactions in digital communication platforms remains a challenging and insufficiently addressed problem. These interactions may appear as harmless spam or escalate into sophisticated scam attempts, making it difficult to flag malicious intent early. Traditional detection methods often rely on static anomaly detection techniques that fail to adapt to dynamic conversational shifts. One key limitation is the misinterpretation of benign topic transitions referred to as concept drift as fraudulent behavior, leading to either false alarms or missed threats. We propose a two stage detection framework that first identifies suspicious conversations using a tailored ensemble classification model. To improve the reliability of detection, we incorporate a concept drift analysis step using a One Class Drift Detector (OCDD) to isolate conversational shifts within flagged dialogues. When drift is detected, a large language model (LLM) assesses whether the shift indicates fraudulent manipulation or a legitimate topic change. In cases where no drift is found, the behavior is inferred to be spam like. We validate our framework using a dataset of social engineering chat scenarios and demonstrate its practical advantages in improving both accuracy and interpretability for real time fraud detection. To contextualize the trade offs, we compare our modular approach against a Dual LLM baseline that performs detection and judgment using different language models.
• 摘要：在数字通信平台中检测虚假互动仍然是一个具有挑战性且不足以解决的问题。这些相互作用可能是无害的垃圾邮件或升级为复杂的骗局尝试，因此很难尽早提出恶意意图。传统的检测方法通常依赖于无法适应动态对话转移的静态异常检测技术。一个关键的限制是误解了良性主题过渡，称为概念漂移是欺诈行为，导致错误警报或错过威胁。我们提出了一个两个阶段检测框架，该框架首先使用量身定制的集合分类模型来识别可疑对话。为了提高检测的可靠性，我们使用一个类漂移检测器（OCDD）结合了一个概念漂移分析步骤，以在标记的对话中隔离对话转移。当检测到漂移时，大型语言模型（LLM）会评估班次是否表示欺诈性操纵还是合法的主题更改。如果没有发现漂移，则可以推断出这种行为为垃圾邮件。我们使用社会工程聊天方案的数据集验证我们的框架，并证明了其在提高实时欺诈检测的准确性和可解释性方面的实际优势。为了使交易折扣的背景与之相关，我们将模块化方法与使用不同语言模型执行检测和判断的双LLM基线进行了比较。

Title: CrashSage: A Large Language Model-Centered Framework for Contextual and Interpretable Traffic Crash Analysis

• Authors: Hao Zhen, Jidong J. Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07853
• Pdf URL: https://arxiv.org/pdf/2505.07853
• Copy Paste: [[2505.07853]] CrashSage: A Large Language Model-Centered Framework for Contextual and Interpretable Traffic Crash Analysis(https://arxiv.org/abs/2505.07853)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Road crashes claim over 1.3 million lives annually worldwide and incur global economic losses exceeding \$1.8 trillion. Such profound societal and financial impacts underscore the urgent need for road safety research that uncovers crash mechanisms and delivers actionable insights. Conventional statistical models and tree ensemble approaches typically rely on structured crash data, overlooking contextual nuances and struggling to capture complex relationships and underlying semantics. Moreover, these approaches tend to incur significant information loss, particularly in narrative elements related to multi-vehicle interactions, crash progression, and rare event characteristics. This study presents CrashSage, a novel Large Language Model (LLM)-centered framework designed to advance crash analysis and modeling through four key innovations. First, we introduce a tabular-to-text transformation strategy paired with relational data integration schema, enabling the conversion of raw, heterogeneous crash data into enriched, structured textual narratives that retain essential structural and relational context. Second, we apply context-aware data augmentation using a base LLM model to improve narrative coherence while preserving factual integrity. Third, we fine-tune the LLaMA3-8B model for crash severity inference, demonstrating superior performance over baseline approaches, including zero-shot, zero-shot with chain-of-thought prompting, and few-shot learning, with multiple models (GPT-4o, GPT-4o-mini, LLaMA3-70B). Finally, we employ a gradient-based explainability technique to elucidate model decisions at both the individual crash level and across broader risk factor dimensions. This interpretability mechanism enhances transparency and enables targeted road safety interventions by providing deeper insights into the most influential factors.
• 摘要：道路撞车事故每年在全球范围内有超过130万人的生活，全球经济损失超过1.8万亿美元。如此深刻的社会和财务影响强调了对道路安全研究的迫切需求，该研究发现了崩溃机制并提供了可行的见解。常规的统计模型和树集合方法通常依赖结构化的崩溃数据，忽略上下文的细微差别以及努力捕获复杂的关系和基础语义的努力。此外，这些方法倾向于引起大量信息丢失，尤其是在与多车相互作用，崩溃进展和罕见事件特征有关的叙事元素中。这项研究介绍了CrashSage，这是一种新型的大型语言模型（LLM）以中心的框架，旨在通过四个关键创新来推动崩溃分析和建模。首先，我们介绍了一种与关系数据集成模式配对的表格到文本转换策略，从而使原始的，异构的崩溃数据转换为保留必要的结构和关系环境的丰富，结构化的文本叙述。其次，我们使用基本LLM模型应用上下文感知的数据增强来提高叙事连贯性，同时保持事实完整性。第三，我们对碰撞严重性推断的Llama3-8B模型进行了微调，表明了优于基线方法的性能，包括零射击，零射击，带有经过三通链的提示，以及几个型号学习，具有多种模型（GPT-4O，GPT-4O，GPT-4O-MINI，LLAMA 3-70B）。最后，我们采用一种基于梯度的可解释性技术来阐明单个崩溃水平和更广泛的风险因素维度的模型决策。这种可解释性机制可以提高透明度，并通过提供对最有影响力的因素的深入了解，从而实现有针对性的道路安全干预措施。

Title: Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines

• Authors: Faiza Hassan, Summra Saleem, Kashif Javed, Muhammad Nabeel Asim, Abdur Rehman, Andreas Dengel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07857
• Pdf URL: https://arxiv.org/pdf/2505.07857
• Copy Paste: [[2505.07857]] Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines(https://arxiv.org/abs/2505.07857)
• Keywords: language model, llm
• Abstract: Multifarious intent detection predictors are developed for different languages, including English, Chinese and French, however, the field remains underdeveloped for Urdu, the 10th most spoken language. In the realm of well-known languages, intent detection predictors utilize the strategy of few-shot learning and prediction of unseen classes based on the model training on seen classes. However, Urdu language lacks few-shot strategy based intent detection predictors and traditional predictors are focused on prediction of the same classes which models have seen in the train set. To empower Urdu language specific intent detection, this introduces a unique contrastive learning approach that leverages unlabeled Urdu data to re-train pre-trained language models. This re-training empowers LLMs representation learning for the downstream intent detection task. Finally, it reaps the combined potential of pre-trained LLMs and the prototype-informed attention mechanism to create a comprehensive end-to-end LLMPIA intent detection pipeline. Under the paradigm of proposed predictive pipeline, it explores the potential of 6 distinct language models and 13 distinct similarity computation methods. The proposed framework is evaluated on 2 public benchmark datasets, namely ATIS encompassing 5836 samples and Web Queries having 8519 samples. Across ATIS dataset under 4-way 1 shot and 4-way 5 shot experimental settings LLMPIA achieved 83.28% and 98.25% F1-Score and on Web Queries dataset produced 76.23% and 84.42% F1-Score, respectively. In an additional case study on the Web Queries dataset under same classes train and test set settings, LLMPIA outperformed state-of-the-art predictor by 53.55% F1-Score.
• 摘要：多种意图检测预测因素是针对包括英语，中文和法语在内的不同语言开发的，但是，该领域仍然以乌尔都语为乌尔都语，这是第十大口语的乌尔都语。在众所周知的语言领域中，意图检测预测因素利用了基于对可见类的模型培训的几乎没有发现的学习和对看不见类的策略。但是，乌尔都语的语言缺乏基于策略的策略检测预测指标，传统预测指标的重点是对火车集合中相同类别的相同类别的预测。为了赋予乌尔都语语言特定意图检测能力，这引入了一种独特的对比学习方法，该方法利用未标记的乌尔都语数据重新培训了预训练的语言模型。这种重新训练使LLMS表示为下游意图检测任务学习。最后，它收获了预先训练的LLM的综合潜力和原型的注意机制，以创建全面的端到端LLMPIA意图检测管道。在提议的预测管道的范式下，它探讨了6种不同的语言模型和13种不同的相似性计算方法的潜力。提出的框架在2个公共基准数据集上进行了评估，即ATIS包含5836个样本和具有8519个样本的Web查询。在ATIS数据集以下的ATIS数据集中，LLMPIA的4次射击和4射击实验设置的实验设置分别达到83.28％和98.25％的F1得分，并且在Web查询数据集上分别产生了76.23％和84.42％的F1-SCORE。在同一类中的Web查询数据集和测试设置设置下的其他案例研究中，LLMPIA的表现优于53.55％的F1得分。

Title: Scaling Laws for Speculative Decoding

• Authors: Siyuan Yan, Mo Zhu, Guo-qing Jiang, Jianfei Wang, Jiaxing Chen, Wentai Zhang, Xiang Liao, Xiao Cui, Chen Zhang, Zhuoran Song, Ran Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07858
• Pdf URL: https://arxiv.org/pdf/2505.07858
• Copy Paste: [[2505.07858]] Scaling Laws for Speculative Decoding(https://arxiv.org/abs/2505.07858)
• Keywords: language model, llm, chain-of-thought
• Abstract: The escalating demand for efficient decoding in large language models (LLMs) is particularly critical for reasoning-intensive architectures like OpenAI-o3 and DeepSeek-R1, which depend on extended chain-of-thought reasoning. This study investigates speculative decoding techniques through dense LLM architectures to establish foundational insights for accelerating reasoning tasks. While speculative decoding methods leveraging parallel draft-verification cycles have emerged as promising acceleration techniques, the scaling laws governing decoding efficiency remain under-explored compared to conventional backbone LLMs developed through Pretraining->SFT->RLHF training paradigms. In this work, we discover Log-linear Scaling Laws (Theorem 1.1, 1.2 and 1.3) governing draft model acceptance rate (or decoding speed) across three dimensions: pretraining token volume, draft model capacity, and decoding batch size. Building on these laws, we achieve Scylla, which coordinates multi-dimensional scaling for popular LLMs (Llama2/3, Qwen2.5). Empirical validation shows Scylla achieves 1.5-2.2 higher acceptance rate than EAGLE2 and 0.3 higher than EAGLE3 at temperature T = 0, with peak performance gains on summarization and QA tasks (Figure 2). Industrial inference engine deployments demonstrate 2X decoding throughput improvements over EAGLE2 (Table 5), validating the transformative potential of systematic scaling for efficient LLM inference. Code will be released later.
• 摘要：大语模型（LLM）对有效解码的不断升级对于依赖于扩展的思想推理的推理密集型体系结构（如OpenAI-O3和DeepSeek-R1）尤其重要。这项研究通过密集的LLM体系结构调查了投机解码技术，以建立加速推理任务的基础见解。尽管利用平行草稿验证周期的投机解码方法已成为有前途的加速技术，但与通过预处理 - > sft-> sft->>>> rlHF训练范式相比，控制解码效率的缩放定律仍然不足以探索。在这项工作中，我们发现了跨三个维度的模型接受率（或解码速度）的对数线性缩放定律（定理1.1、1.2和1.3）：预处理令牌量，模型容量和解码批处理大小。在这些法律的基础上，我们实现了Scylla，它可以协调流行LLM的多维缩放（Llama2/3，Qwen2.5）。经验验证显示，在温度t = 0时，Scylla的接受率比EAGLE3高1.5-2.2高于EAGE3的接受率高0.3，而摘要和质量检查任务的峰值性能提高（图2）。工业推理引擎部署证明了2倍解码对EAGLE2的解码改进（表5），验证了系统缩放对有效LLM推理的变革潜力。代码将稍后发布。

Title: Boosting Performance on ARC is a Matter of Perspective

• Authors: Daniel Franzen, Jan Disselhoff, David Hartmann
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.07859
• Pdf URL: https://arxiv.org/pdf/2505.07859
• Copy Paste: [[2505.07859]] Boosting Performance on ARC is a Matter of Perspective(https://arxiv.org/abs/2505.07859)
• Keywords: language model, llm
• Abstract: The Abstraction and Reasoning Corpus (ARC-AGI) poses a significant challenge for large language models (LLMs), exposing limitations in their abstract reasoning abilities. In this work, we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algorithm to generate diverse, high-probability candidate solutions. Furthermore, we utilize the LLM not only as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly available approaches. While concurrent closed-source work has reported higher scores, our method distinguishes itself through its transparency, reproducibility, and remarkably low inference cost, averaging only around 2ct per task on readily available hardware (we assume a price of 36ct/hour for a Nvidia 4090 GPU).
• 摘要：抽象和推理语料库（ARC-AGI）对大型语言模型（LLM）构成了重大挑战，在其抽象推理能力中暴露了限制。在这项工作中，我们在整个培训，生成和评分阶段中利用特定于任务的数据增强，并采用深度优先的搜索算法来生成多样化的高概率候选解决方案。此外，我们不仅利用LLM作为发电机，而且还利用其输出概率来选择最有前途的解决方案。我们的方法在公共ARC-AGI评估集上达到了71.6％（286.5/400的解决任务），证明了公开可用方法之间的最新性能。尽管并发封闭式工作报告的分数较高，但我们的方法通过其透明度，可重复性和明显低的推理成本来区分自己，而在易于使用的硬件上，每项任务的平均每任务仅为2ct约2ct（我们假设NVIDIA 4090 GPU的价格为36ct/小时）。

Title: Scalable LLM Math Reasoning Acceleration with Low-rank Distillation

• Authors: Harry Dong, Bilge Acun, Beidi Chen, Yuejie Chi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.07861
• Pdf URL: https://arxiv.org/pdf/2505.07861
• Copy Paste: [[2505.07861]] Scalable LLM Math Reasoning Acceleration with Low-rank Distillation(https://arxiv.org/abs/2505.07861)
• Keywords: language model, llm
• Abstract: Due to long generations, large language model (LLM) math reasoning demands significant computational resources and time. While many existing efficient inference methods have been developed with excellent performance preservation on language tasks, they often severely degrade math performance. In this paper, we propose Caprese, a low-cost distillation method to recover lost capabilities from deploying efficient inference methods, focused primarily in feedforward blocks. With original weights unperturbed, roughly 1% of additional parameters, and only 20K synthetic training samples, we are able to recover much if not all of the math capabilities lost from efficient inference for thinking LLMs and without harm to language tasks for instruct LLMs. Moreover, Caprese slashes the number of active parameters (~2B cut for Gemma 2 9B and Llama 3.1 8B) and integrates cleanly into existing model layers to reduce latency (>11% reduction to generate 2048 tokens with Qwen 2.5 14B) while encouraging response brevity.
• 摘要：由于长期，大型语言模型（LLM）数学推理需要大量的计算资源和时间。尽管已经开发了许多现有的有效推理方法，并在语言任务上具有出色的性能保护，但它们通常会严重降低数学性能。在本文中，我们提出了Caprese，这是一种低成本蒸馏方法，以从部署有效的推理方法中恢复损失的功能，主要集中在前馈块上。由于原始权重不受干扰，大约有1％的其他参数，只有20K合成训练样本，我们将能够恢复太多，即使不是全部的数学功能，由于对LLM的有效推断而丧失了所有的数学功能，并且对LLM的语言任务而不会损害LLM的语言任务。此外，Caprese削减了活动参数的数量（Gemma 2 9b和Llama 3.1 8B的降低〜2B），并将清晰集成到现有模型层中以减少延迟（降低11％以减少QWEN 2.5 14b的2048代币），同时又鼓励响应简洁。

Title: QoSBERT: An Uncertainty-Aware Approach based on Pre-trained Language Models for Service Quality Prediction

• Authors: Ziliang Wang, Xiaohong Zhang, Ze Shi Li, Meng Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.07863
• Pdf URL: https://arxiv.org/pdf/2505.07863
• Copy Paste: [[2505.07863]] QoSBERT: An Uncertainty-Aware Approach based on Pre-trained Language Models for Service Quality Prediction(https://arxiv.org/abs/2505.07863)
• Keywords: language model
• Abstract: Accurate prediction of Quality of Service (QoS) metrics is fundamental for selecting and managing cloud based services. Traditional QoS models rely on manual feature engineering and yield only point estimates, offering no insight into the confidence of their predictions. In this paper, we propose QoSBERT, the first framework that reformulates QoS prediction as a semantic regression task based on pre trained language models. Unlike previous approaches relying on sparse numerical features, QoSBERT automatically encodes user service metadata into natural language descriptions, enabling deep semantic understanding. Furthermore, we integrate a Monte Carlo Dropout based uncertainty estimation module, allowing for trustworthy and risk-aware service quality prediction, which is crucial yet underexplored in existing QoS models. QoSBERT applies attentive pooling over contextualized embeddings and a lightweight multilayer perceptron regressor, fine tuned jointly to minimize absolute error. We further exploit the resulting uncertainty estimates to select high quality training samples, improving robustness in low resource settings. On standard QoS benchmark datasets, QoSBERT achieves an average reduction of 11.7% in MAE and 6.7% in RMSE for response time prediction, and 6.9% in MAE for throughput prediction compared to the strongest baselines, while providing well calibrated confidence intervals for robust and trustworthy service quality estimation. Our approach not only advances the accuracy of service quality prediction but also delivers reliable uncertainty quantification, paving the way for more trustworthy, data driven service selection and optimization.
• 摘要：准确的服务质量预测（QOS）指标对于选择和管理基于云的服务至关重要。传统的QoS模型依靠手动功能工程和仅产生点估计，从而没有深入了解其预测的信心。在本文中，我们提出了Qosbert，这是第一个基于训练有素的语言模型将QoS预测作为语义回归任务进行重新定义的框架。与以前的方法依靠稀疏数值特征不同，Qosbert会自动将用户服务元数据编码为自然语言描述，从而使语义理解深刻。此外，我们整合了基于蒙特卡洛辍学的不确定性估计模块，从而允许值得信赖和风险感知的服务质量预测，这在现有的QoS模型中至关重要却又没有忽视。 Qosbert应用于上下文化的嵌入和轻质多层感知器回归剂的细心合并，进行了微调，以最大程度地减少绝对误差。我们进一步利用了最终的不确定性估计来选择高质量的培训样本，从而改善了低资源设置的鲁棒性。在标准的QoS基准数据集上，Qosbert在响应时间预测中的平均降低为11.7％，RMSE的平均降低为6.7％，与最强的基准相比，响应时间预测的平均降低为6.7％，MAE的平均含量为6.9％，同时为稳健和可信赖的服务质量估算提供了良好的置信区间。我们的方法不仅提高了服务质量预测的准确性，而且还提供了可靠的不确定性量化，为更值得信赖，数据驱动的服务选择和优化铺平了道路。

Title: Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection

• Authors: Suavis Giramata, Madhusudan Srinivasan, Venkat Naidu Gudivada, Upulee Kanewala
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2505.07870
• Pdf URL: https://arxiv.org/pdf/2505.07870
• Copy Paste: [[2505.07870]] Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection(https://arxiv.org/abs/2505.07870)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs. This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs. Given the exponential growth of possible test cases, exhaustive testing is impractical; therefore, prioritizing MRs based on their effectiveness in detecting fairness violations is crucial. We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection. Experimental results demonstrate that our proposed prioritization approach improves fault detection rates by 22% compared to random prioritization and 12% compared to distance-based prioritization, while reducing the time to the first failure by 15% and 8%, respectively. Furthermore, our approach performs within 5% of fault-based prioritization in effectiveness, while significantly reducing the computational cost associated with fault labeling. These results validate the effectiveness of diversity-based MR prioritization in enhancing fairness testing for LLMs.
• 摘要：大型语言模型（LLM）越来越多地在各种应用程序中部署，引起了对公平性和产出潜在偏见的关键关注。本文探讨了变质测试中变质关系（MRS）的优先级，作为有效检测LLM中公平问题的策略。鉴于可能的测试用例的指数增长，详尽的测试是不切实际的。因此，基于MRS在检测公平违规行为方面的有效性的优先级至关重要。我们采用基于句子多样性的方法来计算和对MRS进行排名以优化故障检测。实验结果表明，与基于距离的优先级相比，与随机优先级相比，我们提出的优先级方法可将断层检测率提高22％，而12％的优先级相比，同时将时间降低到第一次失败的时间分别将其减少15％和8％。此外，我们的方法在有效性的基于故障优先级的5％以内，同时大大降低了与故障标记相关的计算成本。这些结果证明了基于多样性的MR优先级在增强LLM的公平测试中的有效性。

Title: Evaluating Financial Sentiment Analysis with Annotators Instruction Assisted Prompting: Enhancing Contextual Interpretation and Stock Prediction Accuracy

• Authors: A M Muntasir Rahman, Ajim Uddin, Guiling "Grace" Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07871
• Pdf URL: https://arxiv.org/pdf/2505.07871
• Copy Paste: [[2505.07871]] Evaluating Financial Sentiment Analysis with Annotators Instruction Assisted Prompting: Enhancing Contextual Interpretation and Stock Prediction Accuracy(https://arxiv.org/abs/2505.07871)
• Keywords: llm, prompt
• Abstract: Financial sentiment analysis (FSA) presents unique challenges to LLMs that surpass those in typical sentiment analysis due to the nuanced language used in financial contexts. The prowess of these models is often undermined by the inherent subjectivity of sentiment classifications in existing benchmark datasets like Financial Phrasebank. These datasets typically feature undefined sentiment classes that reflect the highly individualized perspectives of annotators, leading to significant variability in annotations. This variability results in an unfair expectation for LLMs during benchmarking, where they are tasked to conjecture the subjective viewpoints of human annotators without sufficient context. In this paper, we introduce the Annotators' Instruction Assisted Prompt, a novel evaluation prompt designed to redefine the task definition of FSA for LLMs. By integrating detailed task instructions originally intended for human annotators into the LLMs' prompt framework, AIAP aims to standardize the understanding of sentiment across both human and machine interpretations, providing a fair and context-rich foundation for sentiment analysis. We utilize a new dataset, WSBS, derived from the WallStreetBets subreddit to demonstrate how AIAP significantly enhances LLM performance by aligning machine operations with the refined task definitions. Experimental results demonstrate that AIAP enhances LLM performance significantly, with improvements up to 9.08. This context-aware approach not only yields incremental gains in performance but also introduces an innovative sentiment-indexing method utilizing model confidence scores. This method enhances stock price prediction models and extracts more value from the financial sentiment analysis, underscoring the significance of WSB as a critical source of financial text. Our research offers insights into both improving FSA through better evaluation methods.
• 摘要：财务情感分析（FSA）对LLM提出了独特的挑战，这些挑战由于财务环境中使用的细微差异而超过典型的情感分析。这些模型的能力通常被现有基准数据集（如金融词班克）中的情感分类的固有主观性所破坏。这些数据集通常具有未定义的情感类别，反映了注释者高度个性化的观点，从而导致注释的显着差异。这种可变性导致对基准测试期间LLM的不公平期望，在该基准测试过程中，他们的任务是猜测人类注释者的主观观点而没有足够的背景。在本文中，我们介绍了注释者的指示辅助提示，这是一个新颖的评估提示，旨在重新定义LLM的FSA任务定义。通过将最初旨在针对人类注释者的详细任务指令集成到LLMS的及时框架中，AIAP旨在标准化人类和机器解释中对情感的理解，为情感分析提供了公平和上下文丰富的基础。我们利用源自Wallstreetbets子Reddit的新数据集WSB，以证明AIAP如何通过使机器操作与精制的任务定义对齐来显着增强LLM性能。实验结果表明，AIAP显着提高了LLM的性能，改善了9.08。这种上下文感知的方法不仅可以在绩效方面产生增量增长，而且还引入了一种创新的情感索引方法，利用模型置信度得分。该方法增强了股票价格预测模型，并从财务情感分析中提取了更多价值，强调了WSB作为财务文本的关键来源的重要性。我们的研究提供了通过更好的评估方法来改善FSA的见解。

Title: The Sound of Populism: Distinct Linguistic Features Across Populist Variants

• Authors: Yu Wang, Runxi Yu, Zhongyuan Wang, Jing He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.07874
• Pdf URL: https://arxiv.org/pdf/2505.07874
• Copy Paste: [[2505.07874]] The Sound of Populism: Distinct Linguistic Features Across Populist Variants(https://arxiv.org/abs/2505.07874)
• Keywords: language model
• Abstract: This study explores the sound of populism by integrating the classic Linguistic Inquiry and Word Count (LIWC) features, which capture the emotional and stylistic tones of language, with a fine-tuned RoBERTa model, a state-of-the-art context-aware language model trained to detect nuanced expressions of populism. This approach allows us to uncover the auditory dimensions of political rhetoric in U.S. presidential inaugural and State of the Union addresses. We examine how four key populist dimensions (i.e., left-wing, right-wing, anti-elitism, and people-centrism) manifest in the linguistic markers of speech, drawing attention to both commonalities and distinct tonal shifts across these variants. Our findings reveal that populist rhetoric consistently features a direct, assertive ``sound" that forges a connection with ``the people'' and constructs a charismatic leadership persona. However, this sound is not simply informal but strategically calibrated. Notably, right-wing populism and people-centrism exhibit a more emotionally charged discourse, resonating with themes of identity, grievance, and crisis, in contrast to the relatively restrained emotional tones of left-wing and anti-elitist expressions.
• 摘要：这项研究通过整合经典的语言探究和单词计数（LIWC）特征来探讨民粹主义的声音，这些特征捕捉了语言的情感和风格色调，并与精心调整的Roberta模型（一种最先进的上下文意识到的语言模型，训练有素，训练有素训练，可检测到可见的民粹主义表达方式。这种方法使我们能够在美国总统就职典礼上讲话中揭示政治言论的听觉维度。我们研究了四个关键的民粹主义维度（即左翼，右翼，反盛会和以人为中心主义）如何在语言的语言标记中表现出来，引起人们对共同点的关注以及这些变体之间的独特音调转变。 Our findings reveal that populist rhetoric consistently features a direct, assertive ``sound" that forges a connection with ``the people'' and constructs a charismatic leadership persona. However, this sound is not simply informal but strategically calibrated. Notably, right-wing populism and people-centrism exhibit a more emotionally charged discourse, resonating with themes of identity, grievance, and crisis, in contrast to the左翼和反专业主义者表达的相对限制的情感音调。

Title: Recovering Event Probabilities from Large Language Model Embeddings via Axiomatic Constraints

• Authors: Jian-Qiao Zhu, Haijiang Yan, Thomas L. Griffiths
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07883
• Pdf URL: https://arxiv.org/pdf/2505.07883
• Copy Paste: [[2505.07883]] Recovering Event Probabilities from Large Language Model Embeddings via Axiomatic Constraints(https://arxiv.org/abs/2505.07883)
• Keywords: language model, llm
• Abstract: Rational decision-making under uncertainty requires coherent degrees of belief in events. However, event probabilities generated by Large Language Models (LLMs) have been shown to exhibit incoherence, violating the axioms of probability theory. This raises the question of whether coherent event probabilities can be recovered from the embeddings used by the models. If so, those derived probabilities could be used as more accurate estimates in events involving uncertainty. To explore this question, we propose enforcing axiomatic constraints, such as the additive rule of probability theory, in the latent space learned by an extended variational autoencoder (VAE) applied to LLM embeddings. This approach enables event probabilities to naturally emerge in the latent space as the VAE learns to both reconstruct the original embeddings and predict the embeddings of semantically related events. We evaluate our method on complementary events (i.e., event A and its complement, event not-A), where the true probabilities of the two events must sum to 1. Experiment results on open-weight language models demonstrate that probabilities recovered from embeddings exhibit greater coherence than those directly reported by the corresponding models and align closely with the true probabilities.
• 摘要：不确定性下的理性决策需要对事件的信念连贯。但是，大语模型（LLM）产生的事件概率已显示出表现出不一致的性，违反了概率理论的公理。这就提出了一个问题，即是否可以从模型使用的嵌入中恢复相干事件概率。如果是这样，这些衍生概率可以用作涉及不确定性事件的更准确的估计。为了探讨这个问题，我们提出了在由应用于LLM嵌入的扩展变异自动编码器（VAE）学到的潜在空间中的公理约束，例如概率理论的加性规则。这种方法使事件概率可以自然出现在潜在空间中，因为VAE学会了重建原始的嵌入并预测语义相关事件的嵌入。我们评估了我们的互补事件（即事件A及其补充事件，不是A）的方法，其中两个事件的真实概率必须总计为1。开放权重语言模型的实验结果表明，从嵌入式中恢复的概率比相应模型直接报道并与真实的概率紧密相关。

Title: PLHF: Prompt Optimization with Few-Shot Human Feedback

• Authors: Chun-Pai Yang, Kan Zheng, Shou-De Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07886
• Pdf URL: https://arxiv.org/pdf/2505.07886
• Copy Paste: [[2505.07886]] PLHF: Prompt Optimization with Few-Shot Human Feedback(https://arxiv.org/abs/2505.07886)
• Keywords: language model, llm, prompt
• Abstract: Automatic prompt optimization frameworks are developed to obtain suitable prompts for large language models (LLMs) with respect to desired output quality metrics. Although existing approaches can handle conventional tasks such as fixed-solution question answering, defining the metric becomes complicated when the output quality cannot be easily assessed by comparisons with standard golden samples. Consequently, optimizing the prompts effectively and efficiently without a clear metric becomes a critical challenge. To address the issue, we present PLHF (which stands for "P"rompt "L"earning with "H"uman "F"eedback), a few-shot prompt optimization framework inspired by the well-known RLHF technique. Different from naive strategies, PLHF employs a specific evaluator module acting as the metric to estimate the output quality. PLHF requires only a single round of human feedback to complete the entire prompt optimization process. Empirical results on both public and industrial datasets show that PLHF outperforms prior output grading strategies for LLM prompt optimizations.
• 摘要：开发自动及时优化框架，以获得有关所需的输出质量指标的大型语言模型（LLM）的合适提示。尽管现有方法可以处理常规任务，例如固定解决问题，但是当无法通过与标准金样本进行比较来评估输出质量时，定义度量会变得复杂。因此，在没有明确指标的情况下，有效，有效地优化提示将成为一个关键的挑战。为了解决这个问题，我们提出了PLHF（代表“ p“ rompt” l“使用“ H” UMAN“ F” EEDBACK赚取），这是一个受众所周知的RLHF技术启发的几个及时及时的优化框架。与天真的策略不同，PLHF采用了一个特定的评估器模块，该模块充当估计产量质量的指标。 PLHF仅需要一轮人反馈即可完成整个及时的优化过程。公共数据集和工业数据集的经验结果表明，PLHF的表现优于LLM迅速优化的先前输出分级策略。

Title: Implementing Long Text Style Transfer with LLMs through Dual-Layered Sentence and Paragraph Structure Extraction and Mapping

• Authors: Yusen Wu, Xiaotie Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07888
• Pdf URL: https://arxiv.org/pdf/2505.07888
• Copy Paste: [[2505.07888]] Implementing Long Text Style Transfer with LLMs through Dual-Layered Sentence and Paragraph Structure Extraction and Mapping(https://arxiv.org/abs/2505.07888)
• Keywords: language model, llm, prompt
• Abstract: This paper addresses the challenge in long-text style transfer using zero-shot learning of large language models (LLMs), proposing a hierarchical framework that combines sentence-level stylistic adaptation with paragraph-level structural coherence. We argue that in the process of effective paragraph-style transfer, to preserve the consistency of original syntactic and semantic information, it is essential to perform style transfer not only at the sentence level but also to incorporate paragraph-level semantic considerations, while ensuring structural coherence across inter-sentential relationships. Our proposed framework, ZeroStylus, operates through two systematic phases: hierarchical template acquisition from reference texts and template-guided generation with multi-granular matching. The framework dynamically constructs sentence and paragraph template repositories, enabling context-aware transformations while preserving inter-sentence logical relationships. Experimental evaluations demonstrate significant improvements over baseline methods, with structured rewriting achieving 6.90 average score compared to 6.70 for direct prompting approaches in tri-axial metrics assessing style consistency, content preservation, and expression quality. Ablation studies validate the necessity of both template hierarchies during style transfer, showing higher content preservation win rate against sentence-only approaches through paragraph-level structural encoding, as well as direct prompting method through sentence-level pattern extraction and matching. The results establish new capabilities for coherent long-text style transfer without requiring parallel corpora or LLM fine-tuning.
• 摘要：本文使用大型语言模型（LLMS）的零拍学习解决了长篇文本样式转移的挑战，并提出了一个层次结构框架，将句子级的风格适应与段落级别的结构连贯性结合在一起。我们认为，在有效的段落风格转移过程中，为了保留原始句法和语义信息的一致性，不仅要在句子级别上执行样式转移，而且要整合段落级别的语义考虑，同时确保跨项间关系的结构相干性。我们提出的框架Zerostylus通过两个系统的阶段运行：从参考文本和模板引导的生成中，具有多粒子匹配的生成。该框架动态构建句子和段落模板存储库，从而实现上下文感知的转换，同时保持句子间的逻辑关系。实验评估表明，基线方法的显着改善，结构化的重写达到6.90平均得分，而三轴指标的直接提示方法可以评估样式一致性，内容保存和表达质量，而直接提示方法的平均得分为6.70。消融研究验证了样式转移过程中两个模板层次结构的必要性，通过段落级的结构编码显示了较高的内容保存率，并且通过句子级的结构编码进行了较高的句子，并通过句子级的模式提取和匹配来直接提示方法。结果为连贯的长文本风格转移建立了新的功能，而无需平行语料库或LLM微调。

Title: BioProBench: Comprehensive Dataset and Benchmark in Biological Protocol Understanding and Reasoning

• Authors: Yuyang Liu, Liuzhenghao Lv, Xiancheng Zhang, Li Yuan, Yonghong Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.07889
• Pdf URL: https://arxiv.org/pdf/2505.07889
• Copy Paste: [[2505.07889]] BioProBench: Comprehensive Dataset and Benchmark in Biological Protocol Understanding and Reasoning(https://arxiv.org/abs/2505.07889)
• Keywords: llm
• Abstract: Biological protocols are fundamental to reproducible and safe life science research. While LLMs excel on general tasks, their systematic evaluation on these highly specialized, accuracy-critical, and inherently procedural texts remains limited. In this work, we present BioProBench, the first large-scale, integrated multi-task benchmark for biological protocol understanding and reasoning. While limited benchmarks have touched upon specific aspects like protocol QA, BioProBench provides a comprehensive suite of five core tasks: Protocol Question Answering, Step Ordering, Error Correction, Protocol Generation, and Protocol Reasoning, enabling a holistic evaluation of LLMs on procedural biological texts. Built upon 27K original protocols, it yields nearly 556K high-quality structured instances. We evaluate 12 mainstream open/closed-source LLMs on BioProBench. Experimental results reveal that while top models preform well on surface understanding tasks, struggle significantly with deep reasoning and structured generation tasks like ordering and generation. Furthermore, model comparisons reveal diverse performance: certain open-source models approach closed-source levels on some tasks, yet bio-specific small models lag behind general LLMs, indicating limitations on complex procedural content. Overall, our findings underscore that procedural reasoning within biological protocols represents a significant challenge for current LLMs. BioProBench serves as a standardized framework to diagnose these specific limitations and guide the development of AI systems better equipped for safely automating complex scientific procedures. The code and data are available at: this https URL and this https URL.
• 摘要：生物方案对于可重复且安全的生活科学研究至关重要。尽管LLM在一般任务上表现出色，但它们对这些高度专业，准确性和固有的程序文本的系统评估仍然有限。在这项工作中，我们提出了生物探测，这是第一个大型，集成的多任务基准，用于生物协议理解和推理。尽管有限的基准测试涉及协议QA等特定方面，但BioProbench提供了五个核心任务的综合套件：协议问题回答，步骤订购，错误校正，协议生成和协议推理，并对程序生物学文本进行LLMS的整体评估。基于27k原始协议，它产生了近556K高质量的结构化实例。我们评估了生物探测器上的12个主流开放/封闭式LLM。实验结果表明，尽管顶级模型在理解任务方面很好地预成立了，但在深层推理和结构化的生成任务中遇到了重大斗争，例如订购和生成。此外，模型比较揭示了各种性能：某些开源模型在某些任务上接近封闭源水平，但生物特异性的小型模型落后于一般LLM，表明对复杂的程序内容的限制。总体而言，我们的发现强调了生物协议中的程序推理代表了当前LLM的重大挑战。 BioProbench是诊断这些特定局限性的标准框架，并指导AI系统的开发更好地可以安全地自动化复杂的科学程序。代码和数据可在以下网址提供：此HTTPS URL和此HTTPS URL。

Title: TrumorGPT: Graph-Based Retrieval-Augmented Large Language Model for Fact-Checking

• Authors: Ching Nam Hang, Pei-Duo Yu, Chee Wei Tan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07891
• Pdf URL: https://arxiv.org/pdf/2505.07891
• Copy Paste: [[2505.07891]] TrumorGPT: Graph-Based Retrieval-Augmented Large Language Model for Fact-Checking(https://arxiv.org/abs/2505.07891)
• Keywords: language model, gpt, llm, hallucination, retrieval-augmented generation
• Abstract: In the age of social media, the rapid spread of misinformation and rumors has led to the emergence of infodemics, where false information poses a significant threat to society. To combat this issue, we introduce TrumorGPT , a novel generative artificial intelligence solution designed for fact-checking in the health domain. TrumorGPT aims to distinguish "trumors", which are health-related rumors that turn out to be true, providing a crucial tool in differentiating between mere speculation and verified facts. This framework leverages a large language model (LLM) with few-shot learning for semantic health knowledge graph construction and semantic reasoning. TrumorGPT incorporates graph-based retrieval-augmented generation (GraphRAG) to address the hallucination issue common in LLMs and the limitations of static training data. GraphRAG involves accessing and utilizing information from regularly updated semantic health knowledge graphs that consist of the latest medical news and health information, ensuring that fact-checking by TrumorGPT is based on the most recent data. Evaluating with extensive healthcare datasets, TrumorGPT demonstrates superior performance in fact-checking for public health claims. Its ability to effectively conduct fact-checking across various platforms marks a critical step forward in the fight against health-related misinformation, enhancing trust and accuracy in the digital information age.
• 摘要：在社交媒体时代，错误信息和谣言的迅速传播导致了InfoDemics的出现，因为虚假信息对社会构成了重大威胁。为了解决这个问题，我们介绍了Trumorgpt，这是一种新颖的生成人工智能解决方案，旨在在健康领域进行事实检查。 Trumorgpt旨在区分“ Trumors”，这是与健康相关的谣言，事实证明是真实的，为区分仅仅猜测和经过验证的事实提供了重要的工具。该框架利用了一个大型语言模型（LLM），而语义健康知识图形构建和语义推理几乎没有学习。 Trumorgpt结合了基于图的检索演示生成（GraphRag），以解决LLMS中常见的幻觉问题以及静态训练数据的局限性。 GraphRag涉及从定期更新的语义健康知识图中访问和利用信息，这些信息包括最新的医疗新闻和健康信息，以确保Trumorgpt对事实进行检查是基于最新数据。 Trumorgpt通过广泛的医疗保健数据集进行评估，在对​​公共卫生索赔的事实检查方面表现出了卓越的表现。它有效地跨各个平台进行事实检查的能力标志着与健康相关的错误信息的斗争，提高了数字信息时代的信任和准确性。

Title: LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

• Authors: Stefano Rando, Luca Romani, Alessio Sampieri, Yuta Kyuragi, Luca Franco, Fabio Galasso, Tatsunori Hashimoto, John Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07897
• Pdf URL: https://arxiv.org/pdf/2505.07897
• Copy Paste: [[2505.07897]] LongCodeBench: Evaluating Coding LLMs at 1M Context Windows(https://arxiv.org/abs/2505.07897)
• Keywords: llm
• Abstract: Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5.
• 摘要：在短短几年内，模型的上下文长度已迅速增长，从数千到数百万个令牌。现代长篇小说模型的极端上下文大小使得很难构建现实的长篇小说基准 - 不仅是由于收集了数百万个秘密任务的成本，而且还因为确定需要重要背景的现实场景。我们将代码理解和维修确定为长篇小说模型的自然测试床和挑战任务，并引入LongCodebench（LCB），这是在长篇小说方案中测试LLM编码能力的基准。我们的基准测试通过从实际github问题绘制和构建QA（LongCodeQA）和错误固定（Longswe-Bench）任务来测试LCLM在现实和重要设置中的理解和维修功能。我们仔细地分层了基准的复杂性，使我们能够评估不同尺度的模型 - 从QWEN2.5 14B指示到Google的旗舰双子座模型。我们发现，对于所有模型而言，长篇小说仍然是一个弱点，Claude 3.5十四行诗的性能下降，例如QWEN2.5的70.2％至40％。

Title: DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise

• Authors: Ding Cao, Yuchen Cai, Rongxi Guo, Xuesong He, Guiquan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07899
• Pdf URL: https://arxiv.org/pdf/2505.07899
• Copy Paste: [[2505.07899]] DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise(https://arxiv.org/abs/2505.07899)
• Keywords: language model
• Abstract: Sequential knowledge editing techniques aim to continuously update the knowledge in large language models at a low cost, preventing the models from generating outdated or incorrect information. However, existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing. Through theoretical analysis and experiments, we identify that as the number of edits increases, the model's output increasingly deviates from the desired target, leading to a drop in editing success rates. We refer to this issue as the accumulation of superimposed noise problem. To address this, we identify the factors contributing to this deviation and propose DeltaEdit, a novel method that optimizes update parameters through a dynamic orthogonal constraints strategy, effectively reducing interference between edits to mitigate deviation. Experimental results demonstrate that DeltaEdit significantly outperforms existing methods in edit success rates and the retention of generalization capabilities, ensuring stable and reliable model performance even under extensive sequential editing.
• 摘要：顺序知识编辑技术旨在以低成本不断地以大语言模型更新知识，从而阻止模型产生过时或不正确的信息。但是，现有的顺序编辑方法在长期编辑后的编辑成功率显着下降。通过理论分析和实验，我们确定随着编辑数量的增加，该模型的输出越来越偏离所需目标，从而导致编辑成功率下降。我们将此问题称为叠加噪声问题的积累。为了解决这个问题，我们确定了导致这种偏差的因素，并提出了Deltaedit，这是一种新颖的方法，通过动态正交约束策略来优化更新参数，从而有效地减少了编辑之间的干扰以减轻偏差。实验结果表明，Deltaedit在编辑成功率和概括能力的保留方面显着优于现有方法，即使在广泛的顺序编辑下，也可以确保稳定且可靠的模型性能。

Title: SEM: Reinforcement Learning for Search-Efficient Large Language Models

• Authors: Zeyang Sha, Shiwen Cui, Weiqiang Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.07903
• Pdf URL: https://arxiv.org/pdf/2505.07903
• Copy Paste: [[2505.07903]] SEM: Reinforcement Learning for Search-Efficient Large Language Models(https://arxiv.org/abs/2505.07903)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models(LLMs) have demonstrated their capabilities not only in reasoning but also in invoking external tools, particularly search engines. However, teaching models to discern when to invoke search and when to rely on their internal knowledge remains a significant challenge. Existing reinforcement learning approaches often lead to redundant search behaviors, resulting in inefficiencies and over-cost. In this paper, we propose SEM, a novel post-training reinforcement learning framework that explicitly trains LLMs to optimize search usage. By constructing a balanced dataset combining MuSiQue and MMLU, we create scenarios where the model must learn to distinguish between questions it can answer directly and those requiring external retrieval. We design a structured reasoning template and employ Group Relative Policy Optimization(GRPO) to post-train the model's search behaviors. Our reward function encourages accurate answering without unnecessary search while promoting effective retrieval when needed. Experimental results demonstrate that our method significantly reduces redundant search operations while maintaining or improving answer accuracy across multiple challenging benchmarks. This framework advances the model's reasoning efficiency and extends its capability to judiciously leverage external knowledge.
• 摘要：大型语言模型（LLMS）的最新进步不仅在推理方面，而且在调用外部工具（尤其是搜索引擎）方面证明了它们的能力。但是，教学模型以辨别何时调用搜索以及何时依靠他们的内部知识仍然是一个重大挑战。现有的强化学习方法通​​常会导致多余的搜索行为，从而导致效率低下和过度成本。在本文中，我们提出了SEM，这是一种新型的训练后加固学习框架，该框架明确训练LLMS以优化搜索使用情况。通过构建将Musique和MMLU结合的平衡数据集，我们创建了模型必须学会区分它可以直接回答的问题和需要外部检索的方案。我们设计一个结构化的推理模板，并采用小组相对策略优化（GRPO）来培训模型的搜索行为。我们的奖励功能鼓励在没有不必要的搜索的情况下进行准确的回答，同时在需要时促进有效的检索。实验结果表明，我们的方法大大减少了冗余搜索操作，同时维持或提高了多个具有挑战性的基准的答案准确性。该框架提高了模型的推理效率，并扩展了其能力，以明智地利用外部知识。

Title: Re$^2$: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions

• Authors: Daoze Zhang, Zhijian Bao, Sihang Du, Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, Yang Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.07920
• Pdf URL: https://arxiv.org/pdf/2505.07920
• Copy Paste: [[2505.07920]] Re$^2$: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions(https://arxiv.org/abs/2505.07920)
• Keywords: language model, llm
• Abstract: Peer review is a critical component of scientific progress in the fields like AI, but the rapid increase in submission volume has strained the reviewing system, which inevitably leads to reviewer shortages and declines review quality. Besides the growing research popularity, another key factor in this overload is the repeated resubmission of substandard manuscripts, largely due to the lack of effective tools for authors to self-evaluate their work before submission. Large Language Models (LLMs) show great promise in assisting both authors and reviewers, and their performance is fundamentally limited by the quality of the peer review data. However, existing peer review datasets face three major limitations: (1) limited data diversity, (2) inconsistent and low-quality data due to the use of revised rather than initial submissions, and (3) insufficient support for tasks involving rebuttal and reviewer-author interactions. To address these challenges, we introduce the largest consistency-ensured peer review and rebuttal dataset named Re^2, which comprises 19,926 initial submissions, 70,668 review comments, and 53,818 rebuttals from 24 conferences and 21 workshops on OpenReview. Moreover, the rebuttal and discussion stage is framed as a multi-turn conversation paradigm to support both traditional static review tasks and dynamic interactive LLM assistants, providing more practical guidance for authors to refine their manuscripts and helping alleviate the growing review burden. Our data and code are available in this https URL.
• 摘要：同行评审是AI等领域科学进步的关键组成部分，但是提交量的迅速增加使审查系统扭曲了，这不可避免地导致审阅者短缺并降低了审查质量。除了日益增长的研究流行度外，这种超负荷的另一个关键因素是重复重新提交不合格手稿，这在很大程度上是由于缺乏有效的工具来使作者在提交之前自我评估他们的工作。大型语言模型（LLMS）在协助作者和审稿人方面表现出了巨大的希望，并且其性能从根本上受到同行评审数据的质量的限制。但是，现有的同行评审数据集面临三个主要局限性：（1）由于使用修订而不是初步提交而导致的数据多样性，（2）不一致和低质量数据，以及（3）对涉及反驳和审阅者 - 授权人互动的任务的足够支持。为了应对这些挑战，我们介绍了名为Re^2的最大一致性确保的同行评审和反驳数据集，其中包括19,926个初步提交，70,668个评论评论和53,818个反驳，来自24个会议和21个Open Repreview的研讨会。此外，反驳和讨论阶段被构成了多头对话范式，以支持传统的静态审查任务和动态的互动LLM助手，为作者提供了更实用的指导，以完善其手稿并帮助减轻日益增长的审查负担。我们的数据和代码可在此HTTPS URL中找到。

Title: Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models

• Authors: Weiyi Wu, Xinwen Xu, Chongyang Gao, Xingjian Diao, Siting Li, Lucas A. Salas, Jiang Gui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.07968
• Pdf URL: https://arxiv.org/pdf/2505.07968
• Copy Paste: [[2505.07968]] Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models(https://arxiv.org/abs/2505.07968)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have great potential in the field of health care, yet they face great challenges in adapting to rapidly evolving medical knowledge. This can lead to outdated or contradictory treatment suggestions. This study investigated how LLMs respond to evolving clinical guidelines, focusing on concept drift and internal inconsistencies. We developed the DriftMedQA benchmark to simulate guideline evolution and assessed the temporal reliability of various LLMs. Our evaluation of seven state-of-the-art models across 4,290 scenarios demonstrated difficulties in rejecting outdated recommendations and frequently endorsing conflicting guidance. Additionally, we explored two mitigation strategies: Retrieval-Augmented Generation and preference fine-tuning via Direct Preference Optimization. While each method improved model performance, their combination led to the most consistent and reliable results. These findings underscore the need to improve LLM robustness to temporal shifts to ensure more dependable applications in clinical practice.
• 摘要：大型语言模型（LLM）在医疗保健领域具有巨大的潜力，但是在适应快速发展的医学知识方面，它们面临着巨大的挑战。这可能导致过时或矛盾的治疗建议。这项研究调查了LLM对不断发展的临床准则的反应，重点是概念漂移和内部不一致。我们开发了DriftMedQA基准，以模拟指南的演变，并评估了各种LLM的时间可靠性。我们对4,290个场景中七个最先进模型的评估表明，在拒绝过时的建议和经常认可相互矛盾的指导方面遇到了困难。此外，我们探讨了两种缓解策略：通过直接偏好优化的检索增强生成和偏好微调。尽管每种方法都改善了模型性能，但它们的组合导致了最一致，最可靠的结果。这些发现强调了提高LLM鲁棒性到时间变化的必要性，以确保在临床实践中更可靠的应用。

Title: Task-Adaptive Semantic Communications with Controllable Diffusion-based Data Regeneration

• Authors: Fupei Guo, Achintha Wijesinghe, Songyang Zhang, Zhi Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.07980
• Pdf URL: https://arxiv.org/pdf/2505.07980
• Copy Paste: [[2505.07980]] Task-Adaptive Semantic Communications with Controllable Diffusion-based Data Regeneration(https://arxiv.org/abs/2505.07980)
• Keywords: prompt
• Abstract: Semantic communications represent a new paradigm of next-generation networking that shifts bit-wise data delivery to conveying the semantic meanings for bandwidth efficiency. To effectively accommodate various potential downstream tasks at the receiver side, one should adaptively convey the most critical semantic information. This work presents a novel task-adaptive semantic communication framework based on diffusion models that is capable of dynamically adjusting the semantic message delivery according to various downstream tasks. Specifically, we initialize the transmission of a deep-compressed general semantic representation from the transmitter to enable diffusion-based coarse data reconstruction at the receiver. The receiver identifies the task-specific demands and generates textual prompts as feedback. Integrated with the attention mechanism, the transmitter updates the semantic transmission with more details to better align with the objectives of the intended receivers. Our test results demonstrate the efficacy of the proposed method in adaptively preserving critical task-relevant information for semantic communications while preserving high compression efficiency.
• 摘要：语义通信代表了下一代网络的新范式，该范式将数据传递转移到传达带宽效率的语义含义。为了有效地适应接收方的各种潜在下游任务，应该适应传达最关键的语义信息。这项工作提出了一个基于扩散模型的新型任务自适应语义通信框架，该框架能够根据各种下游任务动态调整语义消息传递。具体而言，我们初始化了从发射器中深压的一般语义表示，以实现基于扩散的粗数据重建。接收器确定特定任务的需求并生成文本提示作为反馈。发射器与注意机制集成在一起，更新语义传输，并提供更多详细信息，以更好地与预期接收器的目标保持一致。我们的测试结果证明了所提出的方法在适应性地保留较关键的任务信息以进行语义通信的同时保留高压缩效率的功效。

Title: Large Language Models and Arabic Content: A Review

• Authors: Haneh Rhel, Dmitri Roussinov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08004
• Pdf URL: https://arxiv.org/pdf/2505.08004
• Copy Paste: [[2505.08004]] Large Language Models and Arabic Content: A Review(https://arxiv.org/abs/2505.08004)
• Keywords: language model, llm, prompt
• Abstract: Over the past three years, the rapid advancement of Large Language Models (LLMs) has had a profound impact on multiple areas of Artificial Intelligence (AI), particularly in Natural Language Processing (NLP) across diverse languages, including Arabic. Although Arabic is considered one of the most widely spoken languages across 27 countries in the Arabic world and used as a second language in some other non-Arabic countries as well, there is still a scarcity of Arabic resources, datasets, and tools. Arabic NLP tasks face various challenges due to the complexities of the Arabic language, including its rich morphology, intricate structure, and diverse writing standards, among other factors. Researchers have been actively addressing these challenges, demonstrating that pre-trained Large Language Models (LLMs) trained on multilingual corpora achieve significant success in various Arabic NLP tasks. This study provides an overview of using large language models (LLMs) for the Arabic language, highlighting early pre-trained Arabic Language models across various NLP applications and their ability to handle diverse Arabic content tasks and dialects. It also provides an overview of how techniques like finetuning and prompt engineering can enhance the performance of these models. Additionally, the study summarizes common Arabic benchmarks and datasets while presenting our observations on the persistent upward trend in the adoption of LLMs.
• 摘要：在过去的三年中，大型语言模型（LLM）的快速发展对人工智能（AI）的多个领域产生了深远的影响，尤其是在包括阿拉伯语在内的各种语言的自然语言处理（NLP）中。尽管阿拉伯语被认为是阿拉伯世界27个国家 /地区最广泛的语言之一，并且在其他一些非阿拉伯国家也被用作第二语言，但仍然存在阿拉伯资源，数据集和工具的稀缺性。由于阿拉伯语的复杂性，包括其丰富的形态，错综复杂的结构和多样化的写作标准以及其他因素，阿拉伯NLP任务面临各种挑战。研究人员一直在积极应对这些挑战，表明接受过多语言语料库培训的预先培训的大语言模型（LLMS）在各种阿拉伯NLP任务中取得了重大成功。这项研究提供了有关阿拉伯语使用大语言模型（LLM）的概述，强调了各种NLP应用程序中早期训练的阿拉伯语模型及其处理多种阿拉伯内容任务和方言的能力。它还概述了Finetuning和及时工程等技术如何增强这些模型的性能。此外，该研究总结了普通的阿拉伯基准和数据集，同时介绍了我们对LLM采用持续向上趋势的观察。

Title: FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning

• Authors: Zhehao Zhang, Weijie Xu, Fanyou Wu, Chandan K. Reddy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08054
• Pdf URL: https://arxiv.org/pdf/2505.08054
• Copy Paste: [[2505.08054]] FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning(https://arxiv.org/abs/2505.08054)
• Keywords: language model, llm, prompt, agent
• Abstract: Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities.
• 摘要：大语言模型（LLM）中的安全对准方法通常会导致良性查询过度杂乱无章，从而大大降低了它们在敏感情况下的效用。为了应对这一挑战，我们引入了虚假项目，这是一种综合资源，其中包含16K看似有毒的查询，并伴随着44个与安全相关类别的结构化响应。我们提出了一个图形信息的对抗性多代理交互框架，以生成多样化和复杂的提示，同时用明确的推理构建响应，以帮助模型准确区分不安全的环境。虚拟设备包括针对标准指令调节模型和面向推理的模型的培训数据集，以及人类宣称的基准测试集。我们对29个最先进（SOTA）LLM的广泛基准测试表明，持续的过度挑战。经验结果表明，通过虚假的监督填充大大减少了不必要的拒绝，而不会损害总体安全性或一般语言能力。

Title: HYPERNYM MERCURY: Token Optimization through Semantic Field Constriction and Reconstruction from Hypernyms. A New Text Compression Method

• Authors: Chris Forrester, Octavia Sulea
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08058
• Pdf URL: https://arxiv.org/pdf/2505.08058
• Copy Paste: [[2505.08058]] HYPERNYM MERCURY: Token Optimization through Semantic Field Constriction and Reconstruction from Hypernyms. A New Text Compression Method(https://arxiv.org/abs/2505.08058)
• Keywords: llm, prompt, agent
• Abstract: Compute optimization using token reduction of LLM prompts is an emerging task in the fields of NLP and next generation, agentic AI. In this white paper, we introduce a novel (patent pending) text representation scheme and a first-of-its-kind word-level semantic compression of paragraphs that can lead to over 90\% token reduction, while retaining high semantic similarity to the source text. We explain how this novel compression technique can be lossless and how the detail granularity is controllable. We discuss benchmark results over open source data (i.e. Bram Stoker's Dracula available through Project Gutenberg) and show how our results hold at the paragraph level, across multiple genres and models.
• 摘要：使用LLM提示的令牌减少的计算优化是NLP和下一代Agesic AI领域的一项新任务。在这份白皮书中，我们介绍了一种小说（专利）文本表示方案，以及对段落的首个单词级别的语义压缩，可以导致超过90 \％的令牌减少，同时保持与源文本的高语义相似性。我们解释了这种新颖的压缩技术如何无损，细节粒度如何可控。我们讨论了开源数据（即通过Project Gutenberg获得的Bram Stoker的Dracula）讨论基准结果，并展示我们的结果如何在多种流派和模型的段落级别上保持。

Title: Are LLMs complicated ethical dilemma analyzers?

• Authors: Jiashen (Jason)Du, Jesse Yao, Allen Liu, Zhekai Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08106
• Pdf URL: https://arxiv.org/pdf/2505.08106
• Copy Paste: [[2505.08106]] Are LLMs complicated ethical dilemma analyzers?(https://arxiv.org/abs/2505.08106)
• Keywords: language model, gpt, llm
• Abstract: One open question in the study of Large Language Models (LLMs) is whether they can emulate human ethical reasoning and act as believable proxies for human judgment. To investigate this, we introduce a benchmark dataset comprising 196 real-world ethical dilemmas and expert opinions, each segmented into five structured components: Introduction, Key Factors, Historical Theoretical Perspectives, Resolution Strategies, and Key Takeaways. We also collect non-expert human responses for comparison, limited to the Key Factors section due to their brevity. We evaluate multiple frontier LLMs (GPT-4o-mini, Claude-3.5-Sonnet, Deepseek-V3, Gemini-1.5-Flash) using a composite metric framework based on BLEU, Damerau-Levenshtein distance, TF-IDF cosine similarity, and Universal Sentence Encoder similarity. Metric weights are computed through an inversion-based ranking alignment and pairwise AHP analysis, enabling fine-grained comparison of model outputs to expert responses. Our results show that LLMs generally outperform non-expert humans in lexical and structural alignment, with GPT-4o-mini performing most consistently across all sections. However, all models struggle with historical grounding and proposing nuanced resolution strategies, which require contextual abstraction. Human responses, while less structured, occasionally achieve comparable semantic similarity, suggesting intuitive moral reasoning. These findings highlight both the strengths and current limitations of LLMs in ethical decision-making.
• 摘要：大型语言模型（LLM）研究中的一个开放问题是，它们是否可以模仿人类的道德推理并充当可信的人类判断的代理。为了调查这一点，我们介绍了一个基准数据集，其中包括196个现实的道德困境和专家意见，分为五个结构化组成部分：简介，关键因素，历史理论观点，解决方案策略和关键要点。我们还收集了非专业人类的反应以进行比较，由于其简洁而仅限于关键因素部分。我们使用基于BLEU，Damerau-Levenshtein距离，TF-IDF cesine相似性和Universal Sente Sododer相似性的复合度量框架评估了多个边界LLM（GPT-4O-MINI，Claude-3.5-Sonnet，DeepSeek-V3，Gemini-1.5-flash）。度量权重通过基于反演的排名对齐和成对AHP分析来计算，从而可以将模型输出与专家响应进行细粒度比较。我们的结果表明，LLM在词汇和结构对齐中的表现通常优于非专家人类，而GPT-4O-Mini在所有部分中表现最一致。但是，所有模型都在历史基础上挣扎，并提出了细微的分辨率策略，这些策略需要上下文抽象。人类的反应虽然结构较低，但偶尔会达到可比的语义相似性，这表明直观的道德推理。这些发现突出了LLM在道德决策中的优势和当前局限性。

Title: Putting It All into Context: Simplifying Agents with LCLMs

• Authors: Mingjian Jiang, Yangjun Ruan, Luis Lastras, Pavan Kapanipathi, Tatsunori Hashimoto
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.08120
• Pdf URL: https://arxiv.org/pdf/2505.08120
• Copy Paste: [[2505.08120]] Putting It All into Context: Simplifying Agents with LCLMs(https://arxiv.org/abs/2505.08120)
• Keywords: language model, long context, prompt, agent
• Abstract: Recent advances in language model (LM) agents have demonstrated significant potential for automating complex real-world tasks. To make progress on these difficult tasks, LM agent architectures have become increasingly complex, often incorporating multi-step retrieval tools, multiple agents, and scaffolding adapted to the underlying LM. In this work, we investigate whether all of this complexity is necessary, or if parts of these scaffolds can be removed on challenging tasks like SWE-bench. We show that in the case of SWE-bench, simply putting the entire environment into the context of a long context language model (LCLM) and properly prompting the model makes it competitive with carefully tuned, complex agent scaffolds. We show that a Gemini-1.5-Pro model without any scaffolding or tools achieves 38% on SWE-Bench-Verified, comparable with approaches using carefully tuned agent scaffolds (32%). While the unscaffolded approach with Gemini-1.5-Pro falls short of the strongest agentic architectures, we demonstrate that the more capable Gemini-2.5-Pro using the same unscaffolded approach directly attains a 50.8% solve rate. Additionally, a two-stage approach combining Gemini-1.5-Pro with Claude-3.7 achieves a competitive 48.6% solve rate.
• 摘要：语言模型（LM）代理的最新进展表现出了自动化复杂现实世界任务的巨大潜力。为了在这些艰巨的任务上取得进展，LM代理体系结构已经变得越来越复杂，通常结合了多个步骤检索工具，多个代理和适应基础LM的脚手架。在这项工作中，我们研究了所有这些复杂性是否必要，或者是否可以在SWE-Bench（例如SWE-Bench）上删除这些脚手架的一部分。我们表明，在SWE-Bench的情况下，只需将整个环境置于长上下文语言模型（LCLM）的上下文中，并正确提示该模型使其与经过精心调整，复杂的代理脚手架的竞争力。我们表明，没有任何脚手架或工具的Gemini-1.5-Pro模型在SWE-Bench验证的，与使用精心调谐的剂支架（32％）的方法相媲美，可在38％上获得38％。虽然双子座1.5-PRO的无肢体方法远低于最强的代理体系结构，但我们证明了使用相同的无障碍方法的更有能力的Gemini-2.5-Pro直接达到了50.8％的求解率。此外，将双子座1.5-Pro与Claude-3.7结合的两阶段方法实现了48.6％的求解率。

Title: ALOHA: Empowering Multilingual Agent for University Orientation with Hierarchical Retrieval

• Authors: Mingxu Tao, Bowen Tang, Mingxuan Ma, Yining Zhang, Hourun Li, Feifan Wen, Hao Ma, Jia Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08130
• Pdf URL: https://arxiv.org/pdf/2505.08130
• Copy Paste: [[2505.08130]] ALOHA: Empowering Multilingual Agent for University Orientation with Hierarchical Retrieval(https://arxiv.org/abs/2505.08130)
• Keywords: language model, llm, chat, agent
• Abstract: The rise of Large Language Models~(LLMs) revolutionizes information retrieval, allowing users to obtain required answers through complex instructions within conversations. However, publicly available services remain inadequate in addressing the needs of faculty and students to search campus-specific information. It is primarily due to the LLM's lack of domain-specific knowledge and the limitation of search engines in supporting multilingual and timely scenarios. To tackle these challenges, we introduce ALOHA, a multilingual agent enhanced by hierarchical retrieval for university orientation. We also integrate external APIs into the front-end interface to provide interactive service. The human evaluation and case study show our proposed system has strong capabilities to yield correct, timely, and user-friendly responses to the queries in multiple languages, surpassing commercial chatbots and search engines. The system has been deployed and has provided service for more than 12,000 people.
• 摘要：大语言模型〜（LLM）的兴起彻底改变了信息检索，使用户可以通过对话中的复杂说明获得所需的答案。但是，公开可用的服务在满足教师和学生搜索特定校园特定信息的需求方面仍然不足。这主要是由于LLM缺乏特定领域的知识以及搜索引擎支持多语言和及时场景的局限性。为了应对这些挑战，我们介绍了Aloha，Aloha是一种多种语言的代理人，可通过层次结构检索进行大学定向。我们还将外部API集成到前端接口中以提供交互式服务。人类评估和案例研究表明，我们提出的系统具有强大的功能，可以对多种语言的查询产生正确，及时和用户友好的响应，超过商业聊天机器人和搜索引擎。该系统已部署，并为12,000多人提供了服务。

Title: Fusing Bidirectional Chains of Thought and Reward Mechanisms A Method for Enhancing Question-Answering Capabilities of Large Language Models for Chinese Intangible Cultural Heritage

• Authors: Ruilin Liu, Zhixiao Zhao, Jieqiong Li, Chang Liu, Dongbo Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08167
• Pdf URL: https://arxiv.org/pdf/2505.08167
• Copy Paste: [[2505.08167]] Fusing Bidirectional Chains of Thought and Reward Mechanisms A Method for Enhancing Question-Answering Capabilities of Large Language Models for Chinese Intangible Cultural Heritage(https://arxiv.org/abs/2505.08167)
• Keywords: language model, llm
• Abstract: The rapid development of large language models (LLMs) has provided significant support and opportunities for the advancement of domain-specific LLMs. However, fine-tuning these large models using Intangible Cultural Heritage (ICH) data inevitably faces challenges such as bias, incorrect knowledge inheritance, and catastrophic forgetting. To address these issues, we propose a novel training method that integrates a bidirectional chains of thought and a reward mechanism. This method is built upon ICH-Qwen, a large language model specifically designed for the field of intangible cultural heritage. The proposed method enables the model to not only perform forward reasoning but also enhances the accuracy of the generated answers by utilizing reverse questioning and reverse reasoning to activate the model's latent knowledge. Additionally, a reward mechanism is introduced during training to optimize the decision-making process. This mechanism improves the quality of the model's outputs through structural and content evaluations with different weighting schemes. We conduct comparative experiments on ICH-Qwen, with results demonstrating that our method outperforms 0-shot, step-by-step reasoning, knowledge distillation, and question augmentation methods in terms of accuracy, Bleu-4, and Rouge-L scores on the question-answering task. Furthermore, the paper highlights the effectiveness of combining the bidirectional chains of thought and reward mechanism through ablation experiments. In addition, a series of generalizability experiments are conducted, with results showing that the proposed method yields improvements on various domain-specific datasets and advanced models in areas such as Finance, Wikidata, and StrategyQA. This demonstrates that the method is adaptable to multiple domains and provides a valuable approach for model training in future applications across diverse fields.
• 摘要：大型语言模型（LLM）的快速发展为域特异性LLM的发展提供了重大支持和机会。但是，使用无形的文化遗产（ICH）数据对这些大型模型进行微调不可避免地面临挑战，例如偏见，不正确的知识继承和灾难性的遗忘。为了解决这些问题，我们提出了一种新颖的培训方法，该方法整合了思想的双向链和奖励机制。该方法建立在Ich-Qwen的基础上，这是一种专门为无形文化遗产领域设计的大型语言模型。所提出的方法使模型不仅可以执行正向推理，还可以利用反向质疑和反向推理来激活模型的潜在知识，从而提高了生成的答案的准确性。此外，在培训期间引入了奖励机制，以优化决策过程。这种机制通过使用不同的加权方案进行结构和内容评估来提高模型输出的质量。我们对ICH-QWEN进行了比较实验，结果表明我们的方法在准确性，BLEU-4和Rouge-l分数方面优于0次，逐步推理，知识蒸馏和问题增强方法。此外，本文强调了通过消融实验结合思想和奖励机制的双向链的有效性。此外，还进行了一系列的概括性实验，结果表明该方法在诸如金融，Wikidata和stragityqa等领域的各种领域特定数据集和高级模型上产生了改进。这表明该方法适用于多个领域，并为跨不同领域的未来应用中的模型培训提供了宝贵的方法。

Title: Exploiting Text Semantics for Few and Zero Shot Node Classification on Text-attributed Graph

• Authors: Yuxiang Wang, Xiao Yan, Shiyu Jin, Quanqing Xu, Chuang Hu, Yuanyuan Zhu, Bo Du, Jia Wu, Jiawei Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08168
• Pdf URL: https://arxiv.org/pdf/2505.08168
• Copy Paste: [[2505.08168]] Exploiting Text Semantics for Few and Zero Shot Node Classification on Text-attributed Graph(https://arxiv.org/abs/2505.08168)
• Keywords: prompt
• Abstract: Text-attributed graph (TAG) provides a text description for each graph node, and few- and zero-shot node classification on TAGs have many applications in fields such as academia and social networks. Existing work utilizes various graph-based augmentation techniques to train the node and text embeddings, while text-based augmentations are largely unexplored. In this paper, we propose Text Semantics Augmentation (TSA) to improve accuracy by introducing more text semantic supervision signals. Specifically, we design two augmentation techniques, i.e., positive semantics matching and negative semantics contrast, to provide more reference texts for each graph node or text description. Positive semantic matching retrieves texts with similar embeddings to match with a graph node. Negative semantic contrast adds a negative prompt to construct a text description with the opposite semantics, which is contrasted with the original node and text. We evaluate TSA on 5 datasets and compare with 13 state-of-the-art baselines. The results show that TSA consistently outperforms all baselines, and its accuracy improvements over the best-performing baseline are usually over 5%.
• 摘要：Text-Attribed Graph（TAG）为每个图节点提供了一个文本说明，并且标签上的很少和零弹的节点分类在学术界和社交网络等字段中都有许多应用程序。现有的工作利用各种基于图的增强技术来训练节点和文本嵌入，而基于文本的增强物在很大程度上没有探索。在本文中，我们提出了文本语义扩展（TSA），以通过引入更多文本语义监督信号来提高准确性。具体而言，我们设计了两种增强技术，即正面语义匹配和负面语义对比，以为每个图节点或文本描述提供更多参考文本。正面语义匹配可以检索具有相似嵌入的文本，以与图节点匹配。负面语义对比度增加了一个负面的提示，可以用相反的语义构建文本描述，该语义与原始节点和文本形成对比。我们在5个数据集上评估TSA，并与13个最先进的基线进行比较。结果表明，TSA始终胜过所有基线，并且其与表现最佳基线相比的准确性提高通常超过5％。

Title: A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs

• Authors: Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08200
• Pdf URL: https://arxiv.org/pdf/2505.08200
• Copy Paste: [[2505.08200]] A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs(https://arxiv.org/abs/2505.08200)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) have the tendency to hallucinate, i.e., to sporadically generate false or fabricated information. This presents a major challenge, as hallucinations often appear highly convincing and users generally lack the tools to detect them. Uncertainty quantification (UQ) provides a framework for assessing the reliability of model outputs, aiding in the identification of potential hallucinations. In this work, we introduce pre-trained UQ heads: supervised auxiliary modules for LLMs that substantially enhance their ability to capture uncertainty compared to unsupervised UQ methods. Their strong performance stems from the powerful Transformer architecture in their design and informative features derived from LLM attention maps. Experimental evaluation shows that these heads are highly robust and achieve state-of-the-art performance in claim-level hallucination detection across both in-domain and out-of-domain prompts. Moreover, these modules demonstrate strong generalization to languages they were not explicitly trained on. We pre-train a collection of UQ heads for popular LLM series, including Mistral, Llama, and Gemma 2. We publicly release both the code and the pre-trained heads.
• 摘要：大型语言模型（LLMS）具有幻觉的趋势，即零星产生虚假或捏造的信息。这是一个重大挑战，因为幻觉通常看起来很有说服力，用户通常缺乏检测它们的工具。不确定性定量（UQ）提供了一个框架，用于评估模型输出的可靠性，并有助于识别潜在的幻觉。在这项工作中，我们介绍了预训练的UQ头：LLMS的监督辅助模块，这些模块与无监督的UQ方法相比大大提高了其捕获不确定性的能力。它们的强大性能源于其设计中强大的变压器体系结构以及LLM注意图衍生出的内容丰富的功能。实验评估表明，这些头部非常稳健，并且在跨域内和外域提示的索赔级幻觉检测中实现了最先进的性能。此外，这些模块表现出对他们未经明确训练的语言的强烈概括。我们为受欢迎的LLM系列的UQ Heads收集了，包括Mistral，Llama和Gemma 2。我们公开发布了代码和预培训的头部。

Title: Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement

• Authors: Haoran Ye, Jing Jin, Yuhang Xie, Xin Zhang, Guojie Song
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2505.08245
• Pdf URL: https://arxiv.org/pdf/2505.08245
• Copy Paste: [[2505.08245]] Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement(https://arxiv.org/abs/2505.08245)
• Keywords: language model, llm
• Abstract: The rapid advancement of large language models (LLMs) has outpaced traditional evaluation methodologies. It presents novel challenges, such as measuring human-like psychological constructs, navigating beyond static and task-specific benchmarks, and establishing human-centered evaluation. These challenges intersect with Psychometrics, the science of quantifying the intangible aspects of human psychology, such as personality, values, and intelligence. This survey introduces and synthesizes an emerging interdisciplinary field of LLM Psychometrics, which leverages psychometric instruments, theories, and principles to evaluate, understand, and enhance LLMs. We systematically explore the role of Psychometrics in shaping benchmarking principles, broadening evaluation scopes, refining methodologies, validating results, and advancing LLM capabilities. This paper integrates diverse perspectives to provide a structured framework for researchers across disciplines, enabling a more comprehensive understanding of this nascent field. Ultimately, we aim to provide actionable insights for developing future evaluation paradigms that align with human-level AI and promote the advancement of human-centered AI systems for societal benefit. A curated repository of LLM psychometric resources is available at this https URL.
• 摘要：大型语言模型（LLM）的快速发展已经超过了传统评估方法。它提出了新的挑战，例如测量类似人类的心理结构，超越静态和特定于任务的基准，并建立以人为本的评估。这些挑战与心理计量学相交，即量化人类心理学的无形方面的科学，例如人格，价值观和智力。这项调查介绍并综合了LLM心理计量学的新兴跨学科领域，该领域利用心理学工具，理论和原则来评估，理解和增强LLMS。我们系统地探讨了心理计量学在塑造基准原理，扩大评估范围，炼油方法，验证结果以及提高LLM功能的作用。本文整合了各种观点，为跨学科的研究人员提供结构化框架，从而使对这个新生领域的更全面地了解。最终，我们旨在提供可行的见解，以开发与人级AI保持一致的未来评估范例，并促进以人为中心的AI系统的进步以获得社会利益。 LLM心理测量资源的策划存储库可在此HTTPS URL上获得。

Title: Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration

• Authors: Rishabh Agrawal, Himanshu Kumar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08261
• Pdf URL: https://arxiv.org/pdf/2505.08261
• Copy Paste: [[2505.08261]] Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration(https://arxiv.org/abs/2505.08261)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The rapid progress in large language models (LLMs) has paved the way for novel approaches in knowledge-intensive tasks. Among these, Cache-Augmented Generation (CAG) has emerged as a promising alternative to Retrieval-Augmented Generation (RAG). CAG minimizes retrieval latency and simplifies system design by preloading knowledge into the model's context. However, challenges persist in scaling CAG to accommodate large and dynamic knowledge bases effectively. This paper introduces Adaptive Contextual Compression (ACC), an innovative technique designed to dynamically compress and manage context inputs, enabling efficient utilization of the extended memory capabilities of modern LLMs. To further address the limitations of standalone CAG, we propose a Hybrid CAG-RAG Framework, which integrates selective retrieval to augment preloaded contexts in scenarios requiring additional information. Comprehensive evaluations on diverse datasets highlight the proposed methods' ability to enhance scalability, optimize efficiency, and improve multi-hop reasoning performance, offering practical solutions for real-world knowledge integration challenges.
• 摘要：大语言模型（LLM）的快速进步为知识密集型任务的新方法铺平了道路。其中，cache augment Edenation（CAG）已成为检索效果（RAG）的有前途的替代品。 CAG通过将知识预加载到模型上下文中来最大程度地减少检索延迟并简化系统设计。但是，挑战一直在扩展CAG以有效地适应大型和动态的知识基础。本文介绍了自适应上下文压缩（ACC），这是一种创新技术，旨在动态压缩和管理上下文输入，从而有效利用现代LLM的扩展内存功能。为了进一步解决独立CAG的局限性，我们提出了一个混合CAG-rag框架，该框架集成了选择性检索，以在需要其他信息的情况下，将预定的上下文进行了增强。对各种数据集的全面评估突出了提出的方法提高可扩展性，优化效率和提高多跳推理性能的能力，从而为现实世界知识整合挑战提供了实用解决方案。

Title: Evaluating the Effectiveness of Black-Box Prompt Optimization as the Scale of LLMs Continues to Grow

• Authors: Ziyu Zhou, Yihang Wu, Jingyuan Yang, Zhan Xiao, Rongjun Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08303
• Pdf URL: https://arxiv.org/pdf/2505.08303
• Copy Paste: [[2505.08303]] Evaluating the Effectiveness of Black-Box Prompt Optimization as the Scale of LLMs Continues to Grow(https://arxiv.org/abs/2505.08303)
• Keywords: language model, gpt, llm, prompt
• Abstract: Black-Box prompt optimization methods have emerged as a promising strategy for refining input prompts to better align large language models (LLMs), thereby enhancing their task performance. Although these methods have demonstrated encouraging results, most studies and experiments have primarily focused on smaller-scale models (e.g., 7B, 14B) or earlier versions (e.g., GPT-3.5) of LLMs. As the scale of LLMs continues to increase, such as with DeepSeek V3 (671B), it remains an open question whether these black-box optimization techniques will continue to yield significant performance improvements for models of such scale. In response to this, we select three well-known black-box optimization methods and evaluate them on large-scale LLMs (DeepSeek V3 and Gemini 2.0 Flash) across four NLU and NLG datasets. The results show that these black-box prompt optimization methods offer only limited improvements on these large-scale LLMs. Furthermore, we hypothesize that the scale of the model is the primary factor contributing to the limited benefits observed. To explore this hypothesis, we conducted experiments on LLMs of varying sizes (Qwen 2.5 series, ranging from 7B to 72B) and observed an inverse scaling law, wherein the effectiveness of black-box optimization methods diminished as the model size increased.
• 摘要：Black-box提示优化方法已成为提高输入提示以更好地对齐大型语言模型（LLM）的有前途的策略，从而增强了其任务性能。尽管这些方法证明了令人鼓舞的结果，但大多数研究和实验主要集中在LLMS的较小规模模型（例如7b，14b）或更早版本（例如GPT-3.5）上。随着LLM的规模继续增加，例如DeepSeek V3（671b），这些黑盒优化技术是否会继续为这种规模的模型带来重大的性能改进，这仍然是一个悬而未决的问题。为此，我们选择了三种著名的黑盒优化方法，并在四个NLU和NLG数据集中在大规模LLMS（DeepSeek V3和Gemini 2.0 Flash）上评估它们。结果表明，这些Black-Box提示优化方法仅对这些大规模LLM的改进有限。此外，我们假设该模型的规模是导致观察到的有限收益的主要因素。为了探讨这一假设，我们对不同尺寸的LLM（QWEN 2.5系列，范围从7b到72b）进行了实验，并观察到了逆缩放定律，其中黑盒优化方法的有效性随着模型尺寸的增加而降低。

Title: AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scale

• Authors: Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, Xiangang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08311
• Pdf URL: https://arxiv.org/pdf/2505.08311
• Copy Paste: [[2505.08311]] AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scale(https://arxiv.org/abs/2505.08311)
• Keywords: language model
• Abstract: We present AM-Thinking-v1, a 32B dense language model that advances the frontier of reasoning, embodying the collaborative spirit of open-source innovation. Outperforming DeepSeek-R1 and rivaling leading Mixture-of-Experts (MoE) models like Qwen3-235B-A22B and Seed1.5-Thinking, AM-Thinking-v1 achieves impressive scores of 85.3 on AIME 2024, 74.4 on AIME 2025, and 70.3 on LiveCodeBench, showcasing state-of-the-art mathematical and coding capabilities among open-source models of similar scale. Built entirely from the open-source Qwen2.5-32B base model and publicly available queries, AM-Thinking-v1 leverages a meticulously crafted post-training pipeline - combining supervised fine-tuning and reinforcement learning - to deliver exceptional reasoning capabilities. This work demonstrates that the open-source community can achieve high performance at the 32B scale, a practical sweet spot for deployment and fine-tuning. By striking a balance between top-tier performance and real-world usability, we hope AM-Thinking-v1 inspires further collaborative efforts to harness mid-scale models, pushing reasoning boundaries while keeping accessibility at the core of innovation. We have open-sourced our model on \href{this https URL}{Hugging Face}.
• 摘要：我们提出了AM-INCKING-V1，这是一种32B密集的语言模型，它促进了推理的前沿，体现了开源创新的协作精神。 Outperforming DeepSeek-R1 and rivaling leading Mixture-of-Experts (MoE) models like Qwen3-235B-A22B and Seed1.5-Thinking, AM-Thinking-v1 achieves impressive scores of 85.3 on AIME 2024, 74.4 on AIME 2025, and 70.3 on LiveCodeBench, showcasing state-of-the-art mathematical and coding capabilities among类似规模的开源模型。完全由开源QWEN2.5-32B基本模型和公开可用的查询建立，AM-INCKING-V1利用精心制作的训练后管道 - 结合了监督的微调和加强学习 - 以提供出色的推理能力。这项工作表明，开源社区可以在32B量表上实现高性能，这是部署和微调的实用最佳位置。通过达到顶级性能和现实世界可用性之间的平衡，我们希望AM-INCKINGING-V1激发了进一步的协作努力，以利用中级模型，推动推理界限，同时保持创新核心的可及性。我们已经在\ href {this HTTPS url} {拥抱face}上开源了模型。

Title: On the Geometry of Semantics in Next-token Prediction

• Authors: Yize Zhao, Christos Thrampoulidis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08348
• Pdf URL: https://arxiv.org/pdf/2505.08348
• Copy Paste: [[2505.08348]] On the Geometry of Semantics in Next-token Prediction(https://arxiv.org/abs/2505.08348)
• Keywords: language model
• Abstract: Modern language models demonstrate a remarkable ability to capture linguistic meaning despite being trained solely through next-token prediction (NTP). We investigate how this conceptually simple training objective leads models to extract and encode latent semantic and grammatical concepts. Our analysis reveals that NTP optimization implicitly guides models to encode concepts via singular value decomposition (SVD) factors of a centered data-sparsity matrix that captures next-word co-occurrence patterns. While the model never explicitly constructs this matrix, learned word and context embeddings effectively factor it to capture linguistic structure. We find that the most important SVD factors are learned first during training, motivating the use of spectral clustering of embeddings to identify human-interpretable semantics, including both classical k-means and a new orthant-based method directly motivated by our interpretation of concepts. Overall, our work bridges distributional semantics, neural collapse geometry, and neural network training dynamics, providing insights into how NTP's implicit biases shape the emergence of meaning representations in language models.
• 摘要：现代语言模型表现出非凡的能力，尽管仅通过下一步的预测（NTP）接受了训练，但仍具有捕捉语言意义的能力。我们研究了这个概念上简单的训练目标如何导致模型提取和编码潜在的语义和语法概念。我们的分析表明，NTP优化隐式指导模型通过捕获下一个字的共发生模式的中心数据符号矩阵的奇异值分解（SVD）因子编码概念。尽管该模型从未明确构建此矩阵，但学到的单词和上下文嵌入有效地将其构成了语言结构。我们发现，最重要的SVD因素首先是在训练期间学习的，激发了嵌入的光谱聚类的使用来识别人类可介入的语义，包括经典K-均值和一种直接动机的基于我们对概念的解释的新基于矫形的方法。总体而言，我们的工作桥梁分布语义，神经崩溃的几何形状和神经网络训练动力学，提供了有关NTP隐式偏见如何影响语言模型中意义表示的出现的见解。

Title: Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring

• Authors: Mina Almasi, Ross Deans Kristensen-McLachlan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08351
• Pdf URL: https://arxiv.org/pdf/2505.08351
• Copy Paste: [[2505.08351]] Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring(https://arxiv.org/abs/2505.08351)
• Keywords: language model, llm, prompt, chat
• Abstract: This paper investigates the potentials of Large Language Models (LLMs) as adaptive tutors in the context of second-language learning. In particular, we evaluate whether system prompting can reliably constrain LLMs to generate only text appropriate to the student's competence level. We simulate full teacher-student dialogues in Spanish using instruction-tuned, open-source LLMs ranging in size from 7B to 12B parameters. Dialogues are generated by having an LLM alternate between tutor and student roles with separate chat histories. The output from the tutor model is then used to evaluate the effectiveness of CEFR-based prompting to control text difficulty across three proficiency levels (A1, B1, C1). Our findings suggest that while system prompting can be used to constrain model outputs, prompting alone is too brittle for sustained, long-term interactional contexts - a phenomenon we term alignment drift. Our results provide insights into the feasibility of LLMs for personalized, proficiency-aligned adaptive tutors and provide a scalable method for low-cost evaluation of model performance without human participants.
• 摘要：本文在第二语言学习的背景下研究了大语模型（LLM）的潜力作为自适应导师。特别是，我们评估系统提示是否可以可靠地约束LLM，以生成适合学生能力水平的文本。我们使用指令调整的开源LLMS在西班牙语中模拟完整的教师学生对话，范围从7b到12B参数。对话是通过在教师和学生角色之间具有单独的聊天历史的LLM替代方式来生成对话的。然后，使用导师模型的输出来评估基于CEFR的提示的有效性，以控制三个能力水平（A1，B1，C1）的文本难度。我们的发现表明，尽管可以使用系统提示来限制模型输出，但单独提示对于持续的，长期的互动环境来说太脆了 - 我们术语对齐漂移的现象。我们的结果提供了对LLM对个性化，熟练统一的适应性导师的可行性的见解，并为没有人类参与者的模型绩效提供了可扩展的方法。

Title: Towards Contamination Resistant Benchmarks

• Authors: Rahmatullah Musawi, Sheng Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08389
• Pdf URL: https://arxiv.org/pdf/2505.08389
• Copy Paste: [[2505.08389]] Towards Contamination Resistant Benchmarks(https://arxiv.org/abs/2505.08389)
• Keywords: language model, llm
• Abstract: The rapid development of large language models (LLMs) has transformed the landscape of natural language processing. Evaluating LLMs properly is crucial for understanding their potential and addressing concerns such as safety. However, LLM evaluation is confronted by various factors, among which contamination stands out as a key issue that undermines the reliability of evaluations. In this work, we introduce the concept of contamination resistance to address this challenge. We propose a benchmark based on Caesar ciphers (e.g., "ab" to "bc" when the shift is 1), which, despite its simplicity, is an excellent example of a contamination resistant benchmark. We test this benchmark on widely used LLMs under various settings, and we find that these models struggle with this benchmark when contamination is controlled. Our findings reveal issues in current LLMs and raise important questions regarding their true capabilities. Our work contributes to the development of contamination resistant benchmarks, enabling more rigorous LLM evaluation and offering insights into the true capabilities and limitations of LLMs.
• 摘要：大型语言模型（LLM）的快速发展改变了自然语言处理的景观。正确评估LLMS对于了解其潜力并解决安全等问题至关重要。但是，LLM评估面临各种因素，其中污染是破坏评估可靠性的关键问题。在这项工作中，我们介绍了污染抵抗的概念，以应对这一挑战。我们提出了一个基于凯撒密码（例如，当移动为1时的“ ab”为“ bc”）的基准，尽管它很简单，但它是耐污染基准测试的一个很好的例子。我们在各种设置下在广泛使用的LLMS上测试了此基准测试，我们发现在控制污染时，这些模型与该基准测试。我们的发现揭示了当前LLM中的问题，并提出了有关其真正能力的重要问题。我们的工作有助于开发抗污染基准，从而使LLM更严格的评估以及对LLM的真正能力和局限性的见解。

Title: Accelerating Chain-of-Thought Reasoning: When Goal-Gradient Importance Meets Dynamic Skipping

• Authors: Ren Zhuang, Ben Wang, Shuifa Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08392
• Pdf URL: https://arxiv.org/pdf/2505.08392
• Copy Paste: [[2505.08392]] Accelerating Chain-of-Thought Reasoning: When Goal-Gradient Importance Meets Dynamic Skipping(https://arxiv.org/abs/2505.08392)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Large Language Models leverage Chain-of-Thought (CoT) prompting for complex tasks, but their reasoning traces are often excessively verbose and inefficient, leading to significant computational costs and latency. Current CoT compression techniques typically rely on generic importance metrics and static compression rates, which may inadvertently remove functionally critical tokens or fail to adapt to varying reasoning complexity. To overcome these limitations, we propose Adaptive GoGI-Skip, a novel framework learning dynamic CoT compression via supervised fine-tuning. This approach introduces two synergistic innovations: (1) Goal-Gradient Importance (GoGI), a novel metric accurately identifying functionally relevant tokens by measuring the gradient influence of their intermediate representations on the final answer loss, and (2) Adaptive Dynamic Skipping (ADS), a mechanism dynamically regulating the compression rate based on runtime model uncertainty while ensuring local coherence through an adaptive N-token constraint. To our knowledge, this is the first work unifying a goal-oriented, gradient-based importance metric with dynamic, uncertainty-aware skipping for CoT compression. Trained on compressed MATH data, Adaptive GoGI-Skip demonstrates strong cross-domain generalization across diverse reasoning benchmarks including AIME, GPQA, and GSM8K. It achieves substantial efficiency gains - reducing CoT token counts by over 45% on average and delivering 1.6-2.0 times inference speedups - while maintaining high reasoning accuracy. Notably, it significantly outperforms existing baselines by preserving accuracy even at high effective compression rates, advancing the state of the art in the CoT reasoning efficiency-accuracy trade-off.
• 摘要：大型语言模型利用了促使复杂任务的经验链（COT），但是它们的推理痕迹通常过于冗长且效率低下，从而导致了巨大的计算成本和潜伏期。当前的COT压缩技术通常依赖于通用重要性指标和静态压缩率，这些指标和静态压缩率可能会无意中删除功能上关键的令牌或无法适应不同的推理复杂性。为了克服这些局限性，我们提出了自适应Gogi-Skip，这是一种新型的框架学习动态COT通过监督的微调压缩。这种方法引入了两种协同创新：（1）目标梯度重要性（GOGI），一种新颖的指标，通过测量其中间表示对最终答案损失的梯度影响，准确地识别与功能相关的代币，以及（2）自适应动态跳过（ADS），一种基于运行时构图的机制，在运行时构建了一种机制，以实现运行的构图，并在运行时构建了良好的态度。据我们所知，这是统一一个面向目标的，基于梯度的重要性指标，具有动态，不确定性感知到COT压缩的工作。自适应GOGI-SKIP经过压缩数学数据的培训，在包括AIME，GPQA和GSM8K在内的各种推理基准之间表现出强大的跨域泛化。它实现了可观的效率增长 - 将COT令牌计数平均降低了45％以上，并提供了1.6-2.0倍的推理速度 - 同时保持高推理精度。值得注意的是，它即使以高效的压缩率保持准确性，它可以大大优于现有基准，从而在COT推理效率 - 清晰度的权衡中提高了最新的状态。

Title: TUMS: Enhancing Tool-use Abilities of LLMs with Multi-structure Handlers

• Authors: Aiyao He, Sijia Cui, Shuai Xu, Yanna Wang, Bo Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08402
• Pdf URL: https://arxiv.org/pdf/2505.08402
• Copy Paste: [[2505.08402]] TUMS: Enhancing Tool-use Abilities of LLMs with Multi-structure Handlers(https://arxiv.org/abs/2505.08402)
• Keywords: language model, llm
• Abstract: Recently, large language models(LLMs) have played an increasingly important role in solving a wide range of NLP tasks, leveraging their capabilities of natural language understanding and generating. Integration with external tools further enhances LLMs' effectiveness, providing more precise, timely, and specialized responses. However, LLMs still encounter difficulties with non-executable actions and improper actions, which are primarily attributed to incorrect parameters. The process of generating parameters by LLMs is confined to the tool level, employing the coarse-grained strategy without considering the different difficulties of various tools. To address this issue, we propose TUMS, a novel framework designed to enhance the tool-use capabilities of LLMs by transforming tool-level processing into parameter-level processing. Specifically, our framework consists of four key components: (1) an intent recognizer that identifies the user's intent to help LLMs better understand the task; (2) a task decomposer that breaks down complex tasks into simpler subtasks, each involving a tool call; (3) a subtask processor equipped with multi-structure handlers to generate accurate parameters; and (4) an executor. Our empirical studies have evidenced the effectiveness and efficiency of the TUMS framework with an average of 19.6\% and 50.6\% improvement separately on easy and hard benchmarks of ToolQA, meanwhile, we demonstrated the key contribution of each part with ablation experiments, offering more insights and stimulating future research on Tool-augmented LLMs.
• 摘要：最近，大型语言模型（LLMS）在解决广泛的NLP任务中发挥了越来越重要的作用，利用其自然语言理解和产生的能力。与外部工具的集成进一步增强了LLM的有效性，提供了更精确，及时和专业的响应。但是，LLMS仍然遇到难以执行的动作和不当动作的困难，这主要归因于错误的参数。 LLMS生成参数的过程仅限于工具级别，采用粗粒策略，而无需考虑各种工具的不同困难。为了解决这个问题，我们提出了TUMS，这是一个新颖的框架，旨在通过将工具级处理转换为参数级处理来增强LLM的工具使用功能。具体来说，我们的框架由四个关键组成部分组成：（1）识别用户有助于更好地了解任务的意图的意图识别器； （2）将复杂任务分解为更简单的子任务的任务分解器，每个任务涉及工具调用； （3）配备多结构处理程序的子任务处理器，以生成准确的参数； （4）执行人。我们的实证研究已经证明了TUMS框架的有效性和效率，平均在易于且硬的基准上分别改善了19.6 \％和50.6 \％的改进，同时，我们证明了每个部分的关键贡献。

Title: Hakim: Farsi Text Embedding Model

• Authors: Mehran Sarmadi, Morteza Alikhani, Erfan Zinvandi, Zahra Pourbahman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.08435
• Pdf URL: https://arxiv.org/pdf/2505.08435
• Copy Paste: [[2505.08435]] Hakim: Farsi Text Embedding Model(https://arxiv.org/abs/2505.08435)
• Keywords: language model, chat, retrieval-augmented generation
• Abstract: Recent advancements in text embedding have significantly improved natural language understanding across many languages, yet Persian remains notably underrepresented in large-scale embedding research. In this paper, we present Hakim, a novel state-of-the-art Persian text embedding model that achieves a 8.5% performance improvement over existing approaches on the FaMTEB benchmark, outperforming all previously developed Persian language models. As part of this work, we introduce three new datasets - Corpesia, Pairsia-sup, and Pairsia-unsup - to support supervised and unsupervised training scenarios. Additionally, Hakim is designed for applications in chatbots and retrieval-augmented generation (RAG) systems, particularly addressing retrieval tasks that require incorporating message history within these systems. We also propose a new baseline model built on the BERT architecture. Our language model consistently achieves higher accuracy across various Persian NLP tasks, while the RetroMAE-based model proves particularly effective for textual information retrieval applications. Together, these contributions establish a new foundation for advancing Persian language understanding.
• 摘要：文本嵌入的最新进展已大大提高了许多语言的自然语言理解，但波斯语中的大规模嵌入研究中的代表人数尤其不足。在本文中，我们介绍了Hakim，这是一种新型的波斯文本嵌入模型，比FAMTEB基准上的现有方法实现了8.5％的性能提高，表现优于所有先前开发的波斯语模型。作为这项工作的一部分，我们介绍了三个新数据集 - Corpesia，Pairsia -Sup和Pairsia -Unsup-以支持受监督和无监督的培训场景。此外，Hakim是为聊天机器人和检索功能生成（RAG）系统的应用而设计的，尤其是解决需要将消息历史记录合并到这些系统中的检索任务。我们还提出了建立在BERT体系结构上的新基线模型。我们的语言模型在各种波斯NLP任务中始终达到更高的准确性，而基于逆转录的模型则证明对文本信息检索应用特别有效。这些贡献共同为推进波斯语言理解建立了新的基础。

Title: A document processing pipeline for the construction of a dataset for topic modeling based on the judgments of the Italian Supreme Court

• Authors: Matteo Marulli, Glauco Panattoni, Marco Bertini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08439
• Pdf URL: https://arxiv.org/pdf/2505.08439
• Copy Paste: [[2505.08439]] A document processing pipeline for the construction of a dataset for topic modeling based on the judgments of the Italian Supreme Court(https://arxiv.org/abs/2505.08439)
• Keywords: language model
• Abstract: Topic modeling in Italian legal research is hindered by the lack of public datasets, limiting the analysis of legal themes in Supreme Court judgments. To address this, we developed a document processing pipeline that produces an anonymized dataset optimized for topic modeling. The pipeline integrates document layout analysis (YOLOv8x), optical character recognition, and text anonymization. The DLA module achieved a mAP@50 of 0.964 and a mAP@50-95 of 0.800. The OCR detector reached a mAP@50-95 of 0.9022, and the text recognizer (TrOCR) obtained a character error rate of 0.0047 and a word error rate of 0.0248. Compared to OCR-only methods, our dataset improved topic modeling with a diversity score of 0.6198 and a coherence score of 0.6638. We applied BERTopic to extract topics and used large language models to generate labels and summaries. Outputs were evaluated against domain expert interpretations. Claude Sonnet 3.7 achieved a BERTScore F1 of 0.8119 for labeling and 0.9130 for summarization.
• 摘要：意大利法律研究中的主题建模受到缺乏公共数据集的阻碍，从而限制了最高法院判决中法律主题的分析。为了解决这个问题，我们开发了一个文档处理管道，该管道生成了针对主题建模优化的匿名数据集。管道集成了文档布局分析（Yolov8x），光学字符识别和文本匿名化。 DLA模块获得了0.964中的50个地图和0.800的50-95的地图。 OCR探测器达到了0.9022的50-95的地图，文本识别器（TRORC）获得了0.0047的字符错误率，单词错误率为0.0248。与仅OCR方法相比，我们的数据集改进了主题建模，多样性得分为0.6198，一致性得分为0.6638。我们应用了伯托式提取主题，并使用大型语言模型来生成标签和摘要。对域专家解释评估了输出。 Claude Sonnet 3.7的标签达到了0.8119的Bertscore F1，用于汇总的Bertscore F1达到了0.8119。

Title: IterKey: Iterative Keyword Generation with LLMs for Enhanced Retrieval Augmented Generation

• Authors: Kazuki Hayashi, Hidetaka Kamigaito, Shinya Kouda, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08450
• Pdf URL: https://arxiv.org/pdf/2505.08450
• Copy Paste: [[2505.08450]] IterKey: Iterative Keyword Generation with LLMs for Enhanced Retrieval Augmented Generation(https://arxiv.org/abs/2505.08450)
• Keywords: language model, llm, retrieval augmented generation, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a way to complement the in-context knowledge of Large Language Models (LLMs) by integrating external documents. However, real-world applications demand not only accuracy but also interpretability. While dense retrieval methods provide high accuracy, they lack interpretability; conversely, sparse retrieval methods offer transparency but often fail to capture the full intent of queries due to their reliance on keyword matching. To address these issues, we introduce IterKey, an LLM-driven iterative keyword generation framework that enhances RAG via sparse retrieval. IterKey consists of three LLM-driven stages: generating keywords for retrieval, generating answers based on retrieved documents, and validating the answers. If validation fails, the process iteratively repeats with refined keywords. Across four QA tasks, experimental results show that IterKey achieves 5% to 20% accuracy improvements over BM25-based RAG and simple baselines. Its performance is comparable to dense retrieval-based RAG and prior iterative query refinement methods using dense models. In summary, IterKey is a novel BM25-based approach leveraging LLMs to iteratively refine RAG, effectively balancing accuracy with interpretability.
• 摘要：通过集成外部文档来补充大型语言模型（LLMS）的内在知识知识的一种方法，它已出现了检索增强的生成（RAG）。但是，现实世界的应用不仅需要准确性，还需要解释性。尽管密集的检索方法具有很高的准确性，但它们缺乏解释性。相反，稀疏的检索方法具有透明度，但由于依赖关键字匹配的依赖，通常无法捕获查询的全部意图。为了解决这些问题，我们介绍了Iterkey，这是一个LLM驱动的迭代关键字生成框架，可通过稀疏检索增强抹布。 iterkey由三个LLM驱动阶段组成：生成用于检索的关键字，根据检索文档生成答案并验证答案。如果验证失败，则该过程以精致的关键字迭代重复。在四个质量检查任务中，实验结果表明，ITERKEY可在基于BM25的抹布和简单基线方面取得5％至20％的精度提高。它的性能可与使用密集模型的基于密集的基于检索的抹布和先前的迭代查询方法相媲美。总而言之，ITERKEY是一种基于BM25的新型方法，利用LLM的迭代性完善抹布，有效地平衡了准确性和可解释性。

Title: RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models

• Authors: Fujun Zhang, XiangDong Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08463
• Pdf URL: https://arxiv.org/pdf/2505.08463
• Copy Paste: [[2505.08463]] RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models(https://arxiv.org/abs/2505.08463)
• Keywords: language model, llm
• Abstract: Fine-tuning pre-trained language models (PLMs) has become a dominant paradigm in applying PLMs to downstream tasks. However, with limited fine-tuning, PLMs still struggle with the discrepancies between the representation obtained from the PLMs' encoder and the optimal input to the PLMs' decoder. This paper tackles this challenge by learning to calibrate the representation of PLMs in the latent space. In the proposed representation calibration method (RepCali), we integrate a specific calibration block to the latent space after the encoder and use the calibrated output as the decoder input. The merits of the proposed RepCali include its universality to all PLMs with encoder-decoder architectures, its plug-and-play nature, and ease of implementation. Extensive experiments on 25 PLM-based models across 8 tasks (including both English and Chinese datasets) demonstrate that the proposed RepCali offers desirable enhancements to PLMs (including LLMs) and significantly improves the performance of downstream tasks. Comparison experiments across 4 benchmark tasks indicate that RepCali is superior to the representative fine-tuning baselines.
• 摘要：微调预训练的语言模型（PLM）已成为将PLM应用于下游任务的主要范式。但是，在微调有限的情况下，PLM仍然在从PLMS的编码器获得的表示与PLMS解码器的最佳输入之间的差异。本文通过学习校准潜在空间中PLM的表示来应对这一挑战。在提出的表示校准方法（REPCALI）中，我们将特定的校准块集成到编码后的潜在空间，并将校准的输出用作解码器输入。拟议的Repcali的优点包括其对所有具有编码器架构的PLM的通用性，其插件的性质和易于实现。对8个任务（包括英语和中文数据集）进行了25个基于PLM的模型的广泛实验表明，拟议的Repcali为PLM（包括LLMS）提供了理想的增强功能，并显着提高了下游任务的性能。跨4个基准任务的比较实验表明，Repcali优于代表性的微调基线。

Title: Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directions

• Authors: Lata Pangtey, Anukriti Bhatnagar, Shubhi Bansal, Shahid Shafi Dar, Nagendra Kumar
• Subjects: cs.CL, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2505.08464
• Pdf URL: https://arxiv.org/pdf/2505.08464
• Copy Paste: [[2505.08464]] Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directions(https://arxiv.org/abs/2505.08464)
• Keywords: language model, llm
• Abstract: Stance detection is essential for understanding subjective content across various platforms such as social media, news articles, and online reviews. Recent advances in Large Language Models (LLMs) have revolutionized stance detection by introducing novel capabilities in contextual understanding, cross-domain generalization, and multimodal analysis. Despite these progressions, existing surveys often lack comprehensive coverage of approaches that specifically leverage LLMs for stance detection. To bridge this critical gap, our review article conducts a systematic analysis of stance detection, comprehensively examining recent advancements of LLMs transforming the field, including foundational concepts, methodologies, datasets, applications, and emerging challenges. We present a novel taxonomy for LLM-based stance detection approaches, structured along three key dimensions: 1) learning methods, including supervised, unsupervised, few-shot, and zero-shot; 2) data modalities, such as unimodal, multimodal, and hybrid; and 3) target relationships, encompassing in-target, cross-target, and multi-target scenarios. Furthermore, we discuss the evaluation techniques and analyze benchmark datasets and performance trends, highlighting the strengths and limitations of different architectures. Key applications in misinformation detection, political analysis, public health monitoring, and social media moderation are discussed. Finally, we identify critical challenges such as implicit stance expression, cultural biases, and computational constraints, while outlining promising future directions, including explainable stance reasoning, low-resource adaptation, and real-time deployment frameworks. Our survey highlights emerging trends, open challenges, and future directions to guide researchers and practitioners in developing next-generation stance detection systems powered by large language models.
• 摘要：立场检测对于理解各种平台（例如社交媒体，新闻文章和在线评论）的主观内容至关重要。大型语言模型（LLM）的最新进展通过在上下文理解，跨域概括和多模态分析中引入新的能力来彻底改变了立场检测。尽管进行了这些进展，但现有的调查通常缺乏对专门利用LLMS立场检测的方法的全面覆盖。为了弥合这一关键差距，我们的评论文章对立场检测进行了系统的分析，全面研究了LLM的最新进步，包括基础概念，方法论，数据集，应用程序和新兴挑战。我们为基于LLM的立场检测方法提供了一种新的分类法，该方法沿着三个关键维度结构：1）学习方法，包括受监督，无监督，很少射击和零射； 2）数据模式，例如单峰，多模式和混合动力； 3）目标关系，包括目标内部目标，跨目标和多目标方案。此外，我们讨论评估技术并分析基准数据集和性能趋势，突出了不同体系结构的优势和局限性。讨论了错误信息检测，政治分析，公共卫生监测和社交媒体审核中的关键应用。最后，我们确定了诸如隐式立场表达，文化偏见和计算限制之类的关键挑战，同时概述了有希望的未来方向，包括可解释的立场推理，低资源适应和实时部署框架。我们的调查强调了新兴趋势，开放挑战以及未来的方向，以指导研究人员和从业人员开发由大语言模型提供支持的下一代立场检测系统。

Title: Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?

• Authors: Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2505.08468
• Pdf URL: https://arxiv.org/pdf/2505.08468
• Copy Paste: [[2505.08468]] Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?(https://arxiv.org/abs/2505.08468)
• Keywords: language model, gpt
• Abstract: Charts are ubiquitous as they help people understand and reason with data. Recently, various downstream tasks, such as chart question answering, chart2text, and fact-checking, have emerged. Large Vision-Language Models (LVLMs) show promise in tackling these tasks, but their evaluation is costly and time-consuming, limiting real-world deployment. While using LVLMs as judges to assess the chart comprehension capabilities of other LVLMs could streamline evaluation processes, challenges like proprietary datasets, restricted access to powerful models, and evaluation costs hinder their adoption in industrial settings. To this end, we present a comprehensive evaluation of 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. We design both pairwise and pointwise evaluation tasks covering criteria like factual correctness, informativeness, and relevancy. Additionally, we analyze LVLM judges based on format adherence, positional consistency, length bias, and instruction-following. We focus on cost-effective LVLMs (<10B parameters) suitable for both research and commercial use, following a standardized evaluation protocol and rubric to measure the LVLM judge's accuracy. Experimental results reveal notable variability: while some open LVLM judges achieve GPT-4-level evaluation performance (about 80% agreement with GPT-4 judgments), others struggle (below ~10% agreement). Our findings highlight that state-of-the-art open-source LVLMs can serve as cost-effective automatic evaluators for chart-related tasks, though biases such as positional preference and length bias persist.
• 摘要：图表无处不在，因为它们可以帮助人们通过数据理解和推理。最近，出现了各种下游任务，例如图表问题回答，图表2文本和事实检查。大型视觉模型（LVLMS）在解决这些任务方面表明了有希望，但是它们的评估既昂贵又耗时，从而限制了现实世界的部署。尽管使用LVLM作为法官来评估其他LVLM的图表理解能力可能会简化评估过程，但诸如专有数据集之类的挑战，限制对强大模型的访问以及评估成本阻碍了他们在工业环境中的采用。为此，我们介绍了13个开源LVLM的全面评估，作为各种图表理解和推理任务的法官。我们设计了涵盖事实正确性，信息性和相关性等标准的成对评估任务。此外，我们根据格式依从性，位置一致性，长度偏差和指导跟随指导分析LVLM法官。我们专注于经济高效的LVLM（<10b参数），适用于研究和商业用途，遵循标准化的评估协议和标准化协议，以衡量LVLM法官的准确性。实验结果显示出明显的可变性：虽然一些开放的LVLM法官达到了GPT-4级评估绩效（与GPT-4判断的80％一致），而另一些则挣扎（低于10％的一致性）。我们的发现强调，最新的开源LVLM可以作为与图表相关的任务的具有成本效益的自动评估者，尽管诸如位置偏好和长度偏见之类的偏见持续存在。

Title: LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models

• Authors: Takumi Shibata, Yuichi Miyamura
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08498
• Pdf URL: https://arxiv.org/pdf/2505.08498
• Copy Paste: [[2505.08498]] LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models(https://arxiv.org/abs/2505.08498)
• Keywords: language model, llm
• Abstract: Recent advances in large language models (LLMs) have enabled zero-shot automated essay scoring (AES), providing a promising way to reduce the cost and effort of essay scoring in comparison with manual grading. However, most existing zero-shot approaches rely on LLMs to directly generate absolute scores, which often diverge from human evaluations owing to model biases and inconsistent scoring. To address these limitations, we propose LLM-based Comparative Essay Scoring (LCES), a method that formulates AES as a pairwise comparison task. Specifically, we instruct LLMs to judge which of two essays is better, collect many such comparisons, and convert them into continuous scores. Considering that the number of possible comparisons grows quadratically with the number of essays, we improve scalability by employing RankNet to efficiently transform LLM preferences into scalar scores. Experiments using AES benchmark datasets show that LCES outperforms conventional zero-shot methods in accuracy while maintaining computational efficiency. Moreover, LCES is robust across different LLM backbones, highlighting its applicability to real-world zero-shot AES.
• 摘要：大型语言模型（LLM）的最新进展使零击自动论文评分（AES）提供了一种有希望的方法，可与手动分级相比，降低论文评分的成本和精力。但是，大多数现有的零射方法依赖于LLM来直接生成绝对得分，由于模型偏见和不一致的评分，这通常与人类评估不同。为了解决这些局限性，我们提出了基于LLM的比较论文评分（LCE），该方法将AES作为成对比较任务提出。具体来说，我们指示LLMS判断哪两篇论文中的哪个更好，收集许多此类比较，然后将其转换为连续的分数。考虑到可能的比较数量随论文的数量二次增长，我们通过使用RankNet有效地将LLM偏好转换为标量分数来提高可扩展性。使用AES基准数据集的实验表明，LCES在保持计算效率的同时，LCES的准确性优于常规的零击方法。此外，LCE在不同的LLM骨架上具有强大的功能，突出了其对现实世界中AES的适用性。

Title: Small but Significant: On the Promise of Small Language Models for Accessible AIED

• Authors: Yumou Wei, Paulo Carvalho, John Stamper
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2505.08588
• Pdf URL: https://arxiv.org/pdf/2505.08588
• Copy Paste: [[2505.08588]] Small but Significant: On the Promise of Small Language Models for Accessible AIED(https://arxiv.org/abs/2505.08588)
• Keywords: language model, gpt, llm, prompt
• Abstract: GPT has become nearly synonymous with large language models (LLMs), an increasingly popular term in AIED proceedings. A simple keyword-based search reveals that 61% of the 76 long and short papers presented at AIED 2024 describe novel solutions using LLMs to address some of the long-standing challenges in education, and 43% specifically mention GPT. Although LLMs pioneered by GPT create exciting opportunities to strengthen the impact of AI on education, we argue that the field's predominant focus on GPT and other resource-intensive LLMs (with more than 10B parameters) risks neglecting the potential impact that small language models (SLMs) can make in providing resource-constrained institutions with equitable and affordable access to high-quality AI tools. Supported by positive results on knowledge component (KC) discovery, a critical challenge in AIED, we demonstrate that SLMs such as Phi-2 can produce an effective solution without elaborate prompting strategies. Hence, we call for more attention to developing SLM-based AIED approaches.
• 摘要：GPT已成为大型语言模型（LLM）的代名词，这是AIED程序中日益流行的术语。一个简单的基于关键字的搜索表明，在AIED 2024上发表的76篇长篇小说中有61％描述了使用LLMS解决教育中一些长期挑战的新颖解决方案，而43％的人则具体提及GPT。尽管GPT开创了LLMS创造了激动人心的机会来增强AI对教育的影响，但我们认为该领域的主要关注的焦点是GPT和其他资源密集型LLM（具有超过10B参数）的风险忽略了小型语言（SLM）在提供资源施加的机构方面可以提供可公平访问和可提供高价工具的潜在影响。在AID中的一个关键挑战的知识成分（KC）发现的积极结果的支持下，我们证明了SLMS（例如PHI-2）可以在不精心提示策略的情况下产生有效的解决方案。因此，我们呼吁更多地关注开发基于SLM的AIED方法。

Title: Enhancing Thyroid Cytology Diagnosis with RAG-Optimized LLMs and Pa-thology Foundation Models

• Authors: Hussien Al-Asi, Jordan P Reynolds, Shweta Agarwal, Bryan J Dangott, Aziza Nassar, Zeynettin Akkus
• Subjects: cs.CL, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2505.08590
• Pdf URL: https://arxiv.org/pdf/2505.08590
• Copy Paste: [[2505.08590]] Enhancing Thyroid Cytology Diagnosis with RAG-Optimized LLMs and Pa-thology Foundation Models(https://arxiv.org/abs/2505.08590)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Advancements in artificial intelligence (AI) are transforming pathology by integrat-ing large language models (LLMs) with retrieval-augmented generation (RAG) and domain-specific foundation models. This study explores the application of RAG-enhanced LLMs coupled with pathology foundation models for thyroid cytology diagnosis, addressing challenges in cytological interpretation, standardization, and diagnostic accuracy. By leveraging a curated knowledge base, RAG facilitates dy-namic retrieval of relevant case studies, diagnostic criteria, and expert interpreta-tion, improving the contextual understanding of LLMs. Meanwhile, pathology foun-dation models, trained on high-resolution pathology images, refine feature extrac-tion and classification capabilities. The fusion of these AI-driven approaches en-hances diagnostic consistency, reduces variability, and supports pathologists in dis-tinguishing benign from malignant thyroid lesions. Our results demonstrate that integrating RAG with pathology-specific LLMs significantly improves diagnostic efficiency and interpretability, paving the way for AI-assisted thyroid cytopathology, with foundation model UNI achieving AUC 0.73-0.93 for correct prediction of surgi-cal pathology diagnosis from thyroid cytology samples.
• 摘要：人工智能（AI）的进步正在通过集成大型语言模型（LLMS）与检索增强生成（RAG）和特定领域的基础模型来改变病理。这项研究探讨了rag增强的LLM的应用，以及用于甲状腺细胞学诊断的病理基础模型，解决了细胞学解释，标准化和诊断准确性的挑战。通过利用精心策划的知识库，抹布促进了相关案例研究，诊断标准和专家解释的染色性检索，从而改善了对LLM的上下文理解。同时，经过高分辨率病理图像训练的病理FOUN-DATION模型，精炼特征外推力和分类功能。这些AI驱动的方法的融合诊断一致性，降低了变异性，并支持病理学家与恶性甲状腺病变的良性降解。我们的结果表明，将抹布与病理特异性LLM相结合可显着提高诊断效率和解释性，为AI辅助甲状腺细胞病理学铺平道路，以及基础模型UNI实现AUC 0.73-0.93，以正确预测甲状腺病毒细胞学样品的手术性病理学诊断。

Title: Automatic Task Detection and Heterogeneous LLM Speculative Decoding

• Authors: Danying Ge, Jianhua Gao, Qizhi Jiang, Yifei Feng, Weixing Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08600
• Pdf URL: https://arxiv.org/pdf/2505.08600
• Copy Paste: [[2505.08600]] Automatic Task Detection and Heterogeneous LLM Speculative Decoding(https://arxiv.org/abs/2505.08600)
• Keywords: language model, llm, prompt
• Abstract: Speculative decoding, which combines a draft model with a target model, has emerged as an effective approach to accelerate large language model (LLM) inference. However, existing methods often face a trade-off between the acceptance rate and decoding speed in downstream tasks due to the limited capacity of the draft model, making it difficult to ensure efficiency across diverse tasks. To address this problem, we propose a speculative decoding algorithm tailored for downstream task optimization. It includes an automatic task partitioning and assigning method, which automatically categorizes downstream tasks into different sub-tasks and assigns them to a set of heterogeneous draft models. Each draft model is aligned with the target model using task-specific data, thereby enhancing the consistency of inference results. In addition, our proposed method incorporates an online lightweight prompt classifier to dynamically route prompts to the appropriate draft model. Experimental results demonstrate that the proposed method improves draft accuracy by 6% to 50% over vanilla speculative decoding, while achieving a speedup of 1.10x to 2.64x in LLM inference.
• 摘要：将草案模型与目标模型结合在一起的投机解码已成为加速大型语言模型（LLM）推断的有效方法。但是，由于模型草案的能力有限，现有方法通常会在下游任务中的接受率和解码速度之间的权衡面临，因此很难确保各种任务的效率。为了解决这个问题，我们提出了一种针对下游任务优化的投机解码算法。它包括一个自动任务分区和分配方法，该方法将自动将下游任务分为不同的子任务，并将其分配给一组异构草稿模型。每个草稿模型都使用特定于任务的数据与目标模型对齐，从而增强了推理结果的一致性。此外，我们提出的方法还将在线轻质提示分类器包含到动态路由提示到适当的草稿模型。实验结果表明，所提出的方法比香草投机解码提高了6％至50％，同时在LLM推理中达到了1.10倍至2.64倍的速度。

Title: Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing

• Authors: Chen Wu, Yin Song
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.08651
• Pdf URL: https://arxiv.org/pdf/2505.08651
• Copy Paste: [[2505.08651]] Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing(https://arxiv.org/abs/2505.08651)
• Keywords: language model
• Abstract: We present MegaBeam-Mistral-7B, a language model that supports 512K-token context length. Our work addresses practical limitations in long-context training, supporting real-world tasks such as compliance monitoring and verification. Evaluated on three long-context benchmarks, our 7B-parameter model demonstrates superior in-context learning performance on HELMET and robust retrieval and tracing capability on RULER. It is currently the only open model to achieve competitive long-range reasoning on BABILong at 512K context length without RAG or targeted fine-tuning. Released as fully open source under the Apache 2.0 license, the model has been downloaded over 100,000 times on Hugging Face. Model available at: this https URL
• 摘要：我们提出Megabeam-Mistral-7b，这是一个支持512k tonding上下文长度的语言模型。我们的工作解决了长篇文化培训中的实际限制，支持了现实世界中的任务，例如合规性监视和验证。我们在三个长篇小写基准测试中进行了评估，我们的7B参数模型在头盔上表现出了出色的秘密学习表现，并在标尺上的鲁棒检索和追踪能力。目前，它是唯一以512K上下文长度上的Babilong实现竞争性远程推理的开放模型，而无需抹布或针对性的微调。在Apache 2.0许可下以完全开源的方式发布，该模型在拥抱面前已下载了100,000多次。型号可用：此HTTPS URL

Title: Revealing economic facts: LLMs know more than they say

• Authors: Marcus Buckmann, Quynh Anh Nguyen, Edward Hill
• Subjects: cs.CL, cs.LG, econ.GN
• Abstract URL: https://arxiv.org/abs/2505.08662
• Pdf URL: https://arxiv.org/pdf/2505.08662
• Copy Paste: [[2505.08662]] Revealing economic facts: LLMs know more than they say(https://arxiv.org/abs/2505.08662)
• Keywords: language model, llm
• Abstract: We investigate whether the hidden states of large language models (LLMs) can be used to estimate and impute economic and financial statistics. Focusing on county-level (e.g. unemployment) and firm-level (e.g. total assets) variables, we show that a simple linear model trained on the hidden states of open-source LLMs outperforms the models' text outputs. This suggests that hidden states capture richer economic information than the responses of the LLMs reveal directly. A learning curve analysis indicates that only a few dozen labelled examples are sufficient for training. We also propose a transfer learning method that improves estimation accuracy without requiring any labelled data for the target variable. Finally, we demonstrate the practical utility of hidden-state representations in super-resolution and data imputation tasks.
• 摘要：我们研究是否可以使用大语言模型（LLM）的隐藏状态来估计和估算经济和财务统计。专注于县级（例如失业）和公司级别（例如总资产）变量，我们表明，在开源LLMS的隐藏状态下训练的简单线性模型优于模型的文本输出。这表明，隐藏的国家捕获了比直接揭示的LLM的回应更丰富的经济信息。学习曲线分析表明，只有几十个标记的例子足以训练。我们还提出了一种转移学习方法，该方法可提高估计精度，而无需为目标变量标记的数据。最后，我们证明了隐藏状态表示在超分辨率和数据插图任务中的实际实用性。

Title: Adaptive Schema-aware Event Extraction with Retrieval-Augmented Generation

• Authors: Sheng Liang, Hang Lv, Zhihao Wen, Yaxiong Wu, Yongyue Zhang, Hao Wang, Yong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08690
• Pdf URL: https://arxiv.org/pdf/2505.08690
• Copy Paste: [[2505.08690]] Adaptive Schema-aware Event Extraction with Retrieval-Augmented Generation(https://arxiv.org/abs/2505.08690)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Event extraction (EE) is a fundamental task in natural language processing (NLP) that involves identifying and extracting event information from unstructured text. Effective EE in real-world scenarios requires two key steps: selecting appropriate schemas from hundreds of candidates and executing the extraction process. Existing research exhibits two critical gaps: (1) the rigid schema fixation in existing pipeline systems, and (2) the absence of benchmarks for evaluating joint schema matching and extraction. Although large language models (LLMs) offer potential solutions, their schema hallucination tendencies and context window limitations pose challenges for practical deployment. In response, we propose Adaptive Schema-aware Event Extraction (ASEE), a novel paradigm combining schema paraphrasing with schema retrieval-augmented generation. ASEE adeptly retrieves paraphrased schemas and accurately generates targeted structures. To facilitate rigorous evaluation, we construct the Multi-Dimensional Schema-aware Event Extraction (MD-SEE) benchmark, which systematically consolidates 12 datasets across diverse domains, complexity levels, and language settings. Extensive evaluations on MD-SEE show that our proposed ASEE demonstrates strong adaptability across various scenarios, significantly improving the accuracy of event extraction.
• 摘要：事件提取（EE）是自然语言处理（NLP）的基本任务，涉及从非结构化文本中识别和提取事件信息。在实际情况下，有效的EE需要两个关键步骤：从数百个候选人中选择适当的模式并执行提取过程。现有研究表明了两个关键差距：（1）现有管道系统中的刚性模式固定，以及（2）缺乏评估关节模式匹配和提取的基准。尽管大型语言模型（LLMS）提供了潜在的解决方案，但他们的模式幻觉趋势和上下文窗口限制对实际部署构成了挑战。作为回应，我们提出了自适应模式感知事件提取（ASEE），这是一种新颖的范式，将模式释义与模式检索效果的一代相结合。 ASEE熟练检索释义模式，并准确地生成靶向结构。为了促进严格的评估，我们构建了多维模式感知事件提取（MD-SEE）基准，该基准系统地合并了跨不同域，复杂性级别和语言设置的12个数据集。对MD-SEE的广泛评估表明，我们提出的ASEE在各种情况下都表现出强大的适应性，从而显着提高了事件提取的准确性。

Title: NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context

• Authors: Ben Yao, Qiuchi Li, Yazhou Zhang, Siyu Yang, Bohan Zhang, Prayag Tiwari, Jing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08734
• Pdf URL: https://arxiv.org/pdf/2505.08734
• Copy Paste: [[2505.08734]] NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context(https://arxiv.org/abs/2505.08734)
• Keywords: language model, llm
• Abstract: This work introduces the first benchmark for nursing value alignment, consisting of five core value dimensions distilled from international nursing codes: Altruism, Human Dignity, Integrity, Justice, and Professionalism. The benchmark comprises 1,100 real-world nursing behavior instances collected through a five-month longitudinal field study across three hospitals of varying tiers. These instances are annotated by five clinical nurses and then augmented with LLM-generated counterfactuals with reversed ethic polarity. Each original case is paired with a value-aligned and a value-violating version, resulting in 2,200 labeled instances that constitute the Easy-Level dataset. To increase adversarial complexity, each instance is further transformed into a dialogue-based format that embeds contextual cues and subtle misleading signals, yielding a Hard-Level dataset. We evaluate 23 state-of-the-art (SoTA) LLMs on their alignment with nursing values. Our findings reveal three key insights: (1) DeepSeek-V3 achieves the highest performance on the Easy-Level dataset (94.55), where Claude 3.5 Sonnet outperforms other models on the Hard-Level dataset (89.43), significantly surpassing the medical LLMs; (2) Justice is consistently the most difficult nursing value dimension to evaluate; and (3) in-context learning significantly improves alignment. This work aims to provide a foundation for value-sensitive LLMs development in clinical settings. The dataset and the code are available at this https URL.
• 摘要：这项工作介绍了护理价值一致性的第一个基准，包括从国际护理法规中提取的五个核心价值维度：利他主义，人类尊严，正直，正义和专业精神。该基准包括通过三个不同层次的三个医院进行的五个月纵向野外研究收集的1,100个现实的护理行为实例。这些实例由五名临床护士注释，然后用LLM生成的反事实与伦理极性相反。每个原始情况都与一个值和价值竞争的版本配对，导致构成Easy级数据集的2,200个标记的实例。为了提高对抗性的复杂性，每个实例都会进一步转换为基于对话的格式，该格式嵌入上下文提示和微妙的误导信号，产生硬级数据集。我们评估了23个最新的LLM与护理价值的一致性。我们的发现揭示了三个关键见解：（1）DeepSeek-V3在Easy级数据集（94.55）上取得了最高的性能，其中Claude 3.5 SONNet在硬级数据集（89.43）上的其他模型胜过其他模型，显着超过了医疗LLMS； （2）正义始终是要评估的最困难的护理价值维度； （3）中文学习可显着改善对齐方式。这项工作旨在为临床环境中的价值敏感LLMS开发提供基础。该数据集和代码可在此HTTPS URL上找到。

Title: Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies

• Authors: Xiaoliang Luo, Xinyi Xu, Michael Ramscar, Bradley C. Love
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08739
• Pdf URL: https://arxiv.org/pdf/2505.08739
• Copy Paste: [[2505.08739]] Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies(https://arxiv.org/abs/2505.08739)
• Keywords: language model, gpt, llm
• Abstract: Can autoregressive large language models (LLMs) learn consistent probability distributions when trained on sequences in different token orders? We prove formally that for any well-defined probability distribution, sequence perplexity is invariant under any factorization, including forward, backward, or arbitrary permutations. This result establishes a rigorous theoretical foundation for studying how LLMs learn from data and defines principled protocols for empirical evaluation. Applying these protocols, we show that prior studies examining ordering effects suffer from critical methodological flaws. We retrain GPT-2 models across forward, backward, and arbitrary permuted orders on scientific text. We find systematic deviations from theoretical invariance across all orderings with arbitrary permutations strongly deviating from both forward and backward models, which largely (but not completely) agreed with one another. Deviations were traceable to differences in self-attention, reflecting positional and locality biases in processing. Our theoretical and empirical results provide novel avenues for understanding positional biases in LLMs and suggest methods for detecting when LLMs' probability distributions are inconsistent and therefore untrustworthy.
• 摘要：自回归大型语言模型（LLMS）是否可以在不同令牌订单中接受序列培训时学习一致的概率分布？我们正式证明，对于任何定义明确的概率分布，在任何分解（包括向前，后退或任意排列）下，序列的困惑是不变的。该结果为研究LLM如何从数据中学习并定义了经验评估的原则方案，为研究LLM的学习方式建立了严格的理论基础。应用这些方案时，我们表明，先前研究有序效应的研究受到关键的方法论缺陷。我们对科学文本进行了跨前进，向后和任意排列的订单探讨GPT-2模型。我们发现，在所有顺序中，系统偏差与所有秩序的理论不变性有着与前进模型的强烈偏差，这在很大程度上（但不是完全）彼此同意。偏差可以追溯到自我注意力的差异，反映了处理中的位置和位置偏见。我们的理论和经验结果为理解LLM中的位置偏见提供了新的途径，并提出了检测LLMS概率分布何时不一致的方法，因此不可信。

Title: AC-Reason: Towards Theory-Guided Actual Causality Reasoning with Large Language Models

• Authors: Yanxi Zhang, Xin Cong, Zhong Zhang, Xiao Liu, Dongyan Zhao, Yesai Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08750
• Pdf URL: https://arxiv.org/pdf/2505.08750
• Copy Paste: [[2505.08750]] AC-Reason: Towards Theory-Guided Actual Causality Reasoning with Large Language Models(https://arxiv.org/abs/2505.08750)
• Keywords: language model, gpt, llm
• Abstract: Actual causality (AC), a fundamental aspect of causal reasoning (CR), is responsible for attribution and responsibility assignment in real-world scenarios. However, existing LLM-based methods lack grounding in formal AC theory, resulting in limited interpretability. Therefore, we propose AC-Reason, a semi-formal reasoning framework that identifies causally relevant events within an AC scenario, infers the values of their formal causal factors (e.g., sufficiency, necessity, and normality), and answers AC queries via a theory-guided algorithm with explanations. While AC-Reason does not explicitly construct a causal graph, it operates over variables in the underlying causal structure to support principled reasoning. To enable comprehensive evaluation, we introduce AC-Bench, a new benchmark built upon and substantially extending Big-Bench Hard Causal Judgment (BBH-CJ). AC-Bench comprises ~1K carefully annotated samples, each with detailed reasoning steps and focuses solely on actual causation. The case study shows that synthesized samples in AC-Bench present greater challenges for LLMs. Extensive experiments on BBH-CJ and AC-Bench show that AC-Reason consistently improves LLM performance over baselines. On BBH-CJ, all tested LLMs surpass the average human rater accuracy of 69.60%, with GPT-4 + AC-Reason achieving 75.04%. On AC-Bench, GPT-4 + AC-Reason again achieves the highest accuracy of 71.82%. AC-Bench further enables fine-grained analysis of reasoning faithfulness, revealing that only Qwen-2.5-72B-Instruct, Claude-3.5-Sonnet, and GPT-4o exhibit faithful reasoning, whereas GPT-4 tends to exploit shortcuts. Finally, our ablation study proves that integrating AC theory into LLMs is highly effective, with the proposed algorithm contributing the most significant performance gains.
• 摘要：实际因果关系（AC）是因果推理的基本方面（CR），负责在现实世界中的归因和责任分配。但是，现有的基于LLM的方法缺乏正式AC理论的基础，从而导致有限的解释性。因此，我们提出了AC-REASAN，这是一个半正式的推理框架，可以在AC场景中识别因果关系事件，并渗透其形式因果因素的值（例如，足够，必要性和正常性），并通过理论指导的算法和解释来回答AC查询。虽然AC-REASAN并未明确构建因果图，但它在基本因果结构中的变量上运行以支持原则上的推理。为了实现全面的评估，我们介绍了AC Bench，这是建立在大基础强硬因果判断（BBH-CJ）的基础上的新基准。 AC Bench包括〜1K仔细注释的样本，每个样本都有详细的推理步骤，仅关注实际因果关系。案例研究表明，AC板台中的合成样品对LLM提出了更大的挑战。对BBH-CJ和AC板台的广泛实验表明，AC-REASAN始终提高基准的LLM性能。在BBH-CJ上，所有测试的LLMS都超过了人类平均评估者的准确性69.60％，而GPT-4 + AC-REASON的平均评估者的平均准确度达到75.04％。在AC-Bench上，GPT-4 + AC-REASAN再次获得71.82％的最高准确度。 Ac-Bench进一步实现了对推理忠诚的细粒度分析，这表明只有QWEN-2.5-72b-Instruct，Claude-3.5-Sonnet和GPT-4O表现出忠实的推理，而GPT-4倾向于利用捷径。最后，我们的消融研究证明，将AC理论整合到LLM中是非常有效的，所提出的算法促进了最重要的性能增长。

Title: Aya Vision: Advancing the Frontier of Multilingual Multimodality

• Authors: Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, Jeremy Pekmez, Jason Ozuzu, Pierre Richemond, Acyr Locatelli, Nick Frosst, Phil Blunsom, Aidan Gomez, Ivan Zhang, Marzieh Fadaee, Manoj Govindassamy, Sudip Roy, Matthias Gallé, Beyza Ermis, Ahmet Üstün, Sara Hooker
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.08751
• Pdf URL: https://arxiv.org/pdf/2505.08751
• Copy Paste: [[2505.08751]] Aya Vision: Advancing the Frontier of Multilingual Multimodality(https://arxiv.org/abs/2505.08751)
• Keywords: language model
• Abstract: Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degradation of existing text-only capabilities once vision is introduced. These difficulties are further magnified in the multilingual setting, where the need for multimodal data in different languages exacerbates existing data scarcity, machine translation often distorts meaning, and catastrophic forgetting is more pronounced. To address the aforementioned challenges, we introduce novel techniques spanning both data and modeling. First, we develop a synthetic annotation framework that curates high-quality, diverse multilingual multimodal instruction data, enabling Aya Vision models to produce natural, human-preferred responses to multimodal inputs across many languages. Complementing this, we propose a cross-modal model merging technique that mitigates catastrophic forgetting, effectively preserving text-only capabilities while simultaneously enhancing multimodal generative performance. Aya-Vision-8B achieves best-in-class performance compared to strong multimodal models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which outperforms models more than twice its size, such as Molmo-72B and LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the multi-modal frontier, and provides insights into techniques that effectively bend the need for compute while delivering extremely high performance.
• 摘要：构建多模式模型从根本上具有挑战性：它需要对齐视觉和语言方式，策划高质量的教学数据，并避免一旦引入视觉，避免了现有的仅文本功能的降级。这些困难在多语言环境中进一步放大，在这种设置中，对不同语言的多模式数据的需求加剧了现有数据稀缺，机器翻译通常会扭曲含义，而灾难性的遗忘更为明显。为了应对上述挑战，我们介绍了涵盖数据和建模的新技术。首先，我们开发了一个合成的注释框架，该框架策划了高质量的多语言多模式指令数据，从而使AYA视觉模型能够对跨多种语言产生自然的，人类偏爱的响应。在此方面，我们提出了一种跨模型合并技术，以减轻灾难性遗忘，有效地保留仅文本功能，同时增强多模式生成性能。与QWEN-2.5-VL-7B，PIXTRAL-12B，甚至更大的Llama-3.2-90B-Vision相比，AYA-Vision-8b具有一流的性能。我们使用AYA-Vision-32B进一步扩展这种方法，该方法的大小超过两倍，例如Molmo-72b和Llama-3.2-90B-Vision。我们的工作在多模式边界上取得了多种语言的进步，并提供了对有效弯曲计算需求的技术的见解，同时提供了极高的性能。

Title: HealthBench: Evaluating Large Language Models Towards Improved Human Health

• Authors: Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.08775
• Pdf URL: https://arxiv.org/pdf/2505.08775
• Copy Paste: [[2505.08775]] HealthBench: Evaluating Large Language Models Towards Improved Human Health(https://arxiv.org/abs/2505.08775)
• Keywords: language model, gpt
• Abstract: We present HealthBench, an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32%. We hope that HealthBench grounds progress towards model development and applications that benefit human health.
• 摘要：我们提供HealthBench，这是一种开源基准测试，衡量了医疗保健中大语言模型的性能和安全性。 HealthBench由模型与个人用户或医疗保健专业人员之间的5,000个多转向对话组成。使用由262位医生创建的特定于对话的标题评估响应。与以前的多项选择或短期基准不同，HealthBench可以通过48,562个独特的专栏标准来实现现实的开放式评估，这些标准涵盖了几种健康环境（例如，紧急情况，转化临床数据，全球健康）和行为维度（例如，准确性，指导，遵循，通信）。在过去两年中，HealthBench的表现反映了稳定的初始进展（比较GPT-3.5 Turbo的16％与GPT-4O的32％）和最近的更快改善（O3得分为60％）。较小的型号特别改进：GPT-4.1纳米的表现优于GPT-4O，便宜25倍。我们还发布了两个HealthBench的变化：HealthBench的共识，其中包括34个通过医师共识验证的模型行为的特别重要的维度，而HealthBench Hard Hard Health Bench在当前最高分数为32％。我们希望HealthBench的基础朝着建模开发和有益于人类健康的应用方面的进展。