2025-05-01

Title: Waking Up an AI: A Quantitative Framework for Prompt-Induced Phase Transition in Large Language Models

• Authors: Makoto Sato
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21012
• Pdf URL: https://arxiv.org/pdf/2504.21012
• Copy Paste: [[2504.21012]] Waking Up an AI: A Quantitative Framework for Prompt-Induced Phase Transition in Large Language Models(https://arxiv.org/abs/2504.21012)
• Keywords: language model, llm, prompt
• Abstract: What underlies intuitive human thinking? One approach to this question is to compare the cognitive dynamics of humans and large language models (LLMs). However, such a comparison requires a method to quantitatively analyze AI cognitive behavior under controlled conditions. While anecdotal observations suggest that certain prompts can dramatically change LLM behavior, these observations have remained largely qualitative. Here, we propose a two-part framework to investigate this phenomenon: a Transition-Inducing Prompt (TIP) that triggers a rapid shift in LLM responsiveness, and a Transition Quantifying Prompt (TQP) that evaluates this change using a separate LLM. Through controlled experiments, we examined how LLMs react to prompts embedding two semantically distant concepts (e.g., mathematical aperiodicity and traditional crafts)--either fused together or presented separately--by changing their linguistic quality and affective tone. Whereas humans tend to experience heightened engagement when such concepts are meaningfully blended producing a novel concept--a form of conceptual fusion--current LLMs showed no significant difference in responsiveness between semantically fused and non-fused prompts. This suggests that LLMs may not yet replicate the conceptual integration processes seen in human intuition. Our method enables fine-grained, reproducible measurement of cognitive responsiveness, and may help illuminate key differences in how intuition and conceptual leaps emerge in artificial versus human minds.
• 摘要：什么是直觉的人类思维？解决这个问题的一种方法是比较人类的认知动态和大型语言模型（LLMS）。但是，这种比较需要一种在受控条件下定量分析AI认知行为的方法。尽管轶事观察表明某些提示可以大大改变LLM的行为，但这些观察结果仍然在很大程度上是定性的。在这里，我们提出了一个分为两部分的框架来研究这种现象：诱导过渡的提示（提示），它触发了LLM响应能力的快速变化，以及使用单独的LLM评估此变化的过渡量化提示（TQP）。通过受控的实验，我们检查了LLM对嵌入两个语义上遥远概念的提示（例如，数学上的多个学术和传统手工艺品）的反应 - 要么通过改变语言质量和情感色调，要么分别融合或分别融合在一起。当这种概念是有意义地混合产生一种新颖概念时，人类往往会经历更高的参与度 - 一种概念融合的形式 - 流动性LLM在语义融合和非融合提示之间的响应性差异没有显着差异。这表明LLM可能尚未复制人类直觉中看到的概念整合过程。我们的方法可以实现对认知反应能力的细粒度，可重复的测量，并可能有助于阐明在人工和人类思想中直觉和概念飞跃如何出现的关键差异。

Title: HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization

• Authors: Enes Özeren, Yihong Liu, Hinrich Schütze
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21018
• Pdf URL: https://arxiv.org/pdf/2504.21018
• Copy Paste: [[2504.21018]] HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization(https://arxiv.org/abs/2504.21018)
• Keywords: language model, llm
• Abstract: Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages, largely due to limited exposure to these languages during pre-training. A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data. Among such methods, OFA (Liu et al., 2024a) proposes a similarity-based subword embedding initialization heuristic that is both effective and efficient. However, OFA restricts target-language token embeddings to be convex combinations of a fixed number of source-language embeddings, which may limit expressiveness. To overcome this limitation, we propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding initialization. The hypernetwork is trained to map from an external multilingual word vector space to the PLMs token embedding space using source-language tokens. Once trained, it can generate flexible embeddings for target-language tokens, serving as a good starting point for continual pretraining. Experiments demonstrate that HYPEROFA consistently outperforms random initialization baseline and matches or exceeds the performance of OFA in both continual pre-training convergence and downstream task performance. We make the code publicly available.
• 摘要：许多预训练的语言模型（PLM）在中低资源语言上表现出次优的性能，这在很大程度上是由于预训练期间对这些语言的曝光率有限。解决此问题的一种常见策略是引入针对目标语言的新代币，初始化其嵌入，并在目标语言数据上进行连续的预培训。在此类方法中，OFA（Liu等，2024a）提出了一种基于相似性的子字嵌入初始化启发式，既有效又有效。但是，OFA限制了目标令牌嵌入为固定数量源语言嵌入的凸组合，这可能会限制表达性。为了克服这一限制，我们提出了Hyperofa，这是一种基于超网的方法，用于更适应性令牌嵌入初始化。使用源语言令牌训练了超网络从外部多语言矢量空间映射到PLMS令牌嵌入空间。一旦受过训练，它就可以为目标语言令牌生成柔性嵌入，这是连续预处理的良好起点。实验表明，HyperOFA始终优于随机初始化基线，并匹配或超过持续的预训练收敛和下游任务性能中OFA的性能。我们使代码公开可用。

Title: Kill two birds with one stone: generalized and robust AI-generated text detection via dynamic perturbations

• Authors: Yinghan Zhou, Juan Wen, Wanli Peng, Yiming Xue, Ziwei Zhang, Zhengxian Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21019
• Pdf URL: https://arxiv.org/pdf/2504.21019
• Copy Paste: [[2504.21019]] Kill two birds with one stone: generalized and robust AI-generated text detection via dynamic perturbations(https://arxiv.org/abs/2504.21019)
• Keywords: language model
• Abstract: The growing popularity of large language models has raised concerns regarding the potential to misuse AI-generated text (AIGT). It becomes increasingly critical to establish an excellent AIGT detection method with high generalization and robustness. However, existing methods either focus on model generalization or concentrate on robustness. The unified mechanism, to simultaneously address the challenges of generalization and robustness, is less explored. In this paper, we argue that robustness can be view as a specific form of domain shift, and empirically reveal an intrinsic mechanism for model generalization of AIGT detection task. Then, we proposed a novel AIGT detection method (DP-Net) via dynamic perturbations introduced by a reinforcement learning with elaborated reward and action. Experimentally, extensive results show that the proposed DP-Net significantly outperforms some state-of-the-art AIGT detection methods for generalization capacity in three cross-domain scenarios. Meanwhile, the DP-Net achieves best robustness under two text adversarial attacks. The code is publicly available at this https URL.
• 摘要：大型语言模型的日益普及引起了人们对滥用AI生成的文本（AIGT）的潜力的担忧。建立具有高概括和鲁棒性的出色AIGT检测方法变得越来越重要。但是，现有方法要么集中于模型概括，要么集中于鲁棒性。不太探索统一的机制，同时应对概括和鲁棒性的挑战。在本文中，我们认为鲁棒性可以视为域移动的特定形式，并从经验上揭示了用于模型AIGT检测任务的固有机制。然后，我们通过强化学习带来的动态扰动提出了一种新颖的AIGT检测方法（DP-NET），并具有详尽的奖励和动作。在实验上，广泛的结果表明，在三种跨域场景中，提出的DP-NET明显优于某些最先进的AIGT检测方法，用于泛化能力。同时，DP-NET在两次文本对抗攻击下实现了最佳鲁棒性。该代码在此HTTPS URL上公开可用。

Title: Context-Enhanced Contrastive Search for Improved LLM Text Generation

• Authors: Jaydip Sen, Rohit Pandey, Hetvi Waghela
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21020
• Pdf URL: https://arxiv.org/pdf/2504.21020
• Copy Paste: [[2504.21020]] Context-Enhanced Contrastive Search for Improved LLM Text Generation(https://arxiv.org/abs/2504.21020)
• Keywords: language model, llm, chat
• Abstract: Recently, Large Language Models (LLMs) have demonstrated remarkable advancements in Natural Language Processing (NLP). However, generating high-quality text that balances coherence, diversity, and relevance remains challenging. Traditional decoding methods, such as bean search and top-k sampling, often struggle with either repetitive or incoherent outputs, particularly in tasks that require long-form text generation. To address these limitations, the paper proposes a novel enhancement of the well-known Contrastive Search algorithm, Context-Enhanced Contrastive Search (CECS) with contextual calibration. The proposed scheme introduces several novelties including dynamic contextual importance weighting, multi-level Contrastive Search, and adaptive temperature control, to optimize the balance between fluency, creativity, and precision. The performance of CECS is evaluated using several standard metrics such as BLEU, ROUGE, and semantic similarity. Experimental results demonstrate significant improvements in both coherence and relevance of the generated texts by CECS outperforming the existing Contrastive Search techniques. The proposed algorithm has several potential applications in the real world including legal document drafting, customer service chatbots, and content marketing.
• 摘要：最近，大型语言模型（LLMS）在自然语言处理（NLP）方面取得了显着进步。但是，生成平衡连贯性，多样性和相关性的高质量文本仍然具有挑战性。传统的解码方法，例如豆类搜索和TOP-K采样，通常会在重复或不连贯的输出中遇到困难，尤其是在需要长期文本生成的任务中。为了解决这些局限性，本文提出了对众所周知的对比搜索算法，上下文增强对比度搜索（CEC）的新颖性，并具有上下文校准。提出的方案介绍了几个新颖性，包括动态上下文重要性加权，多级对比度搜索和自适应温度控制，以优化流利，创造力和精度之间的平衡。使用几种标准指标（例如BLEU，rouge和语义相似性）评估CEC的性能。实验结果表明，通过CEC的表现优于现有的对比搜索技术，生成的文本的相干性和相关性都有显着提高。拟议的算法在现实世界中具有多个潜在的应用程序，包括法律文件起草，客户服务聊天机器人和内容营销。

Title: ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees

• Authors: Jun Wang, David Smith Sundarsingh, Jyotirmoy V. Deshmukh, Yiannis Kantaros
• Subjects: cs.CL, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2504.21022
• Pdf URL: https://arxiv.org/pdf/2504.21022
• Copy Paste: [[2504.21022]] ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees(https://arxiv.org/abs/2504.21022)
• Keywords: llm
• Abstract: Linear Temporal Logic (LTL) has become a prevalent specification language for robotic tasks. To mitigate the significant manual effort and expertise required to define LTL-encoded tasks, several methods have been proposed for translating Natural Language (NL) instructions into LTL formulas, which, however, lack correctness guarantees. To address this, we introduce a new NL-to-LTL translation method, called ConformalNL2LTL, that can achieve user-defined translation success rates over unseen NL commands. Our method constructs LTL formulas iteratively by addressing a sequence of open-vocabulary Question-Answering (QA) problems with LLMs. To enable uncertainty-aware translation, we leverage conformal prediction (CP), a distribution-free uncertainty quantification tool for black-box models. CP enables our method to assess the uncertainty in LLM-generated answers, allowing it to proceed with translation when sufficiently confident and request help otherwise. We provide both theoretical and empirical results demonstrating that ConformalNL2LTL achieves user-specified translation accuracy while minimizing help rates.
• 摘要：线性时间逻辑（LTL）已成为机器人任务的普遍规范语言。为了减轻定义LTL编码任务所需的重大手动努力和专业知识，已经提出了几种将自然语言（NL）说明转化为LTL公式的方法，但是，这些方法缺乏正确的保证。为了解决这个问题，我们引入了一种新的NL-TO-LTL翻译方法，称为Conformalnl2LTL，该方法可以通过看不见的NL命令来实现用户定义的转换成功率。我们的方法通过解决了LLMS的一系列开放式问题避开（QA）问题来构建LTL公式。为了启用不确定性感知的翻译，我们利用了保形预测（CP），这是黑盒模型的无分配不确定性量化工具。 CP使我们的方法能够评估LLM生成的答案中的不确定性，从而使其在充分自信时进行翻译并要求提供帮助。我们提供理论和经验结果，表明Condomalnl2LTL可实现用户指定的翻译精度，同时最大程度地降低帮助率。

Title: Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost

• Authors: Sheng Cao, Mingrui Wu, Karthik Prasad, Yuandong Tian, Zechun Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21023
• Pdf URL: https://arxiv.org/pdf/2504.21023
• Copy Paste: [[2504.21023]] Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost(https://arxiv.org/abs/2504.21023)
• Keywords: language model
• Abstract: The post-training phase of large language models is essential for enhancing capabilities such as instruction-following, reasoning, and alignment with human preferences. However, it demands extensive high-quality data and poses risks like overfitting, alongside significant computational costs due to repeated post-training and evaluation after each base model update. This paper introduces $Param\Delta$, a novel method that streamlines post-training by transferring knowledge from an existing post-trained model to a newly updated base model with ZERO additional training. By computing the difference between post-trained model weights ($\Theta_\text{post}$) and base model weights ($\Theta_\text{base}$), and adding this to the updated base model ($\Theta'_\text{base}$), we define $Param\Delta$ Model as: $\Theta_{\text{Param}\Delta} = \Theta_\text{post} - \Theta_\text{base} + \Theta'_\text{base}$. This approach surprisingly equips the new base model with post-trained capabilities, achieving performance comparable to direct post-training. We did analysis on LLama3, Llama3.1, Qwen, and DeepSeek-distilled models. Results indicate $Param\Delta$ Model effectively replicates traditional post-training. For example, the $Param\Delta$ Model obtained from 70B Llama3-inst, Llama3-base, Llama3.1-base models attains approximately 95\% of Llama3.1-inst model's performance on average. $Param\Delta$ brings a new perspective on how to fully leverage models in the open-weight community, where checkpoints for base and instruct models are readily available and frequently updated, by providing a cost-free framework to accelerate the iterative cycle of model development.
• 摘要：大语言模型的训练阶段对于增强能力，例如跟踪，推理和与人类偏好的一致性。但是，它需要广泛的高质量数据，并带来过度拟合之类的风险，以及由于每个基本模型更新后反复培训和评估而导致的大量计算成本。本文介绍了$ param \ delta $，这是一种新颖的方法，通过将知识从现有的后培训模型转移到新更新的基本模型和零额外培训的新更新基础模型来简化培训。通过计算训练后的模型权重（$ \ theta_ \ text {post} $）和基本模型权重（$ \ theta_ \ text {base} $）之间的差异，并将其添加到更新的基本模型（$ \ \ theta'_ _ _ \ \ text {base} $），我们定义$ \ theta _ {\ text {param} \ delta} = \ theta_ \ text {这种方法令人惊讶地使新的基本模型具有后训练的能力，从而达到了与直接训练相当的性能。我们对Llama3，Llama3.1，Qwen和DeepSeek-Distildistled模型进行了分析。结果表明$ param \ delta $模型有效地复制了传统的训练后训练。例如，从70B llama3-inst，llama3-base，llama3.1基本型号获得的$ param \ delta $模型达到了llama3.1-inst模型的平均表现的约95％。 $ param \ delta $带来了如何在开放式社区中充分利用模型的新观点，在该社区中，基本和指导模型的检查点可以通过提供一个免费的框架来加速模型开发的迭代循环，从而很容易获得并经常更新。

Title: WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

• Authors: Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21024
• Pdf URL: https://arxiv.org/pdf/2504.21024
• Copy Paste: [[2504.21024]] WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model(https://arxiv.org/abs/2504.21024)
• Keywords: language model, llm, agent
• Abstract: Agent self-improvement, where the backbone Large Language Model (LLM) of the agent are trained on trajectories sampled autonomously based on their own policies, has emerged as a promising approach for enhancing performance. Recent advancements, particularly in web environments, face a critical limitation: their performance will reach a stagnation point during autonomous learning cycles, hindering further improvement. We argue that this stems from limited exploration of the web environment and insufficient exploitation of pre-trained web knowledge in LLMs. To improve the performance of self-improvement, we propose a novel framework that introduces a co-evolving World Model LLM. This world model predicts the next observation based on the current observation and action within the web environment. Leveraging LLMs' pretrained knowledge of abundant web content, the World Model serves dual roles: (1) as a virtual web server generating self-instructed training data to continuously refine the agent's policy, and (2) as an imagination engine during inference, enabling look-ahead simulation to guide action selection for the agent LLM. Experiments in real-world web environments (Mind2Web-Live, WebVoyager, and GAIA-web) show a 10% performance gain over existing self-evolving agents, demonstrating the efficacy and generalizability of our approach, without using any distillation from more powerful close-sourced models. Our work establishes the necessity of integrating world models into autonomous agent frameworks to unlock sustained adaptability.
• 摘要：代理人的自我完善，代理的主链大语言模型（LLM）是根据自己的政策自动采样的轨迹进行培训的，它已成为提高性能的一种有希望的方法。最近的进步，尤其是在网络环境中，面临着一个关键的局限性：它们的性能将在自主学习周期期间达到停滞点，从而阻碍进一步的进步。我们认为，这源于对Web环境的探索有限，以及对LLMS中预训练的Web知识的利用不足。为了提高自我完善的表现，我们提出了一个新型框架，该框架引入了共同发展的世界模型LLM。这个世界模型根据网络环境中的当前观察和行动预测下一个观察。通过利用LLM的知识了解丰富的Web内容知识，世界模型发挥了双重作用：（1）作为虚拟Web服务器生成自我实施的培训数据，以连续完善代理商的策略，以及（2）作为推理期间的想象引擎，启用Look-Ahead模拟代理人的仿真动作选择Agent Agent LLM。在现实世界中的Web环境（Mind2Web-Live，WebVoyager和Gaia-Web）中的实验表明，与现有自我发展的代理相比，性能增长了10％，表明了我们方法的效果和概括性，而无需使用更强大的更近距离的模型的任何蒸馏。我们的工作确定了将世界模型纳入自治剂框架以解锁持续适应性的必要性。

Title: Durghotona GPT: A Web Scraping and Large Language Model Based Framework to Generate Road Accident Dataset Automatically in Bangladesh

• Authors: MD Thamed Bin Zaman Chowdhury, Moazzem Hossain, Md. Ridwanul Islam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21025
• Pdf URL: https://arxiv.org/pdf/2504.21025
• Copy Paste: [[2504.21025]] Durghotona GPT: A Web Scraping and Large Language Model Based Framework to Generate Road Accident Dataset Automatically in Bangladesh(https://arxiv.org/abs/2504.21025)
• Keywords: language model, gpt, llm
• Abstract: Road accidents pose significant concerns globally. They lead to large financial losses, injuries, disabilities, and societal challenges. Accurate and timely accident data is essential for predicting and mitigating these events. This paper presents a novel framework named 'Durghotona GPT' that integrates web scraping and Large Language Models (LLMs) to automate the generation of comprehensive accident datasets from prominent national dailies in Bangladesh. The authors collected accident reports from three major newspapers: Prothom Alo, Dhaka Tribune, and The Daily Star. The collected news was then processed using the newest available LLMs: GPT-4, GPT-3.5, and Llama-3. The framework efficiently extracts relevant information, categorizes reports, and compiles detailed datasets. Thus, this framework overcomes limitations of manual data collection methods such as delays, errors, and communication gaps. The authors' evaluation demonstrates that Llama-3, an open-source model, performs comparably to GPT-4. It achieved 89% accuracy in the authors' evaluation. Therefore, it can be considered a cost-effective alternative for similar tasks. The results suggest that the framework developed by the authors can drastically enhance the quality and availability of accident data. As a result, it can support critical applications in traffic safety analysis, urban planning, and public health. The authors also developed an interface for 'Durghotona GPT' for ease of use as part of this paper. Future work will focus on expanding data collection methods and refining LLMs to further increase dataset accuracy and applicability.
• 摘要：道路事故在全球引起了重大关注。它们导致财务损失，伤害，残疾和社会挑战。准确及时的事故数据对于预测和减轻这些事件至关重要。本文介绍了一个名为“ Durghotona GPT”的新颖框架，该框架集成了网络刮擦和大型语言模型（LLMS），以自动化孟加拉国著名国家日报的综合事故数据集的生成。作者收集了三家主要报纸的事故报告：普罗托姆·阿罗，达卡论坛报和《每日星报》。然后使用最新可用的LLM处理收集的新闻：GPT-4，GPT-3.5和Llama-3。该框架有效提取相关信息，对报告进行分类并编译详细的数据集。因此，该框架克服了手动数据收集方法的局限性，例如延迟，错误和通信差距。作者的评估表明，一种开源模型Llama-3的性能与GPT-4相当。它在作者的评估中达到了89％的精度。因此，可以将其视为类似任务的具有成本效益的替代方法。结果表明，作者开发的框架可以大大提高事故数据的质量和可用性。结果，它可以支持交通安全分析，城市规划和公共卫生中的关键应用程序。作者还为本文的一部分开发了一个用于易用性的“ Durghotona GPT”的界面。未来的工作将着重于扩展数据收集方法和完善LLM，以进一步提高数据集准确性和适用性。

Title: Creating and Evaluating Code-Mixed Nepali-English and Telugu-English Datasets for Abusive Language Detection Using Traditional and Deep Learning Models

• Authors: Manish Pandey, Nageshwar Prasad Yadav, Mokshada Adduru, Sawan Rai
• Subjects: cs.CL, cs.AI, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2504.21026
• Pdf URL: https://arxiv.org/pdf/2504.21026
• Copy Paste: [[2504.21026]] Creating and Evaluating Code-Mixed Nepali-English and Telugu-English Datasets for Abusive Language Detection Using Traditional and Deep Learning Models(https://arxiv.org/abs/2504.21026)
• Keywords: language model, llm
• Abstract: With the growing presence of multilingual users on social media, detecting abusive language in code-mixed text has become increasingly challenging. Code-mixed communication, where users seamlessly switch between English and their native languages, poses difficulties for traditional abuse detection models, as offensive content may be context-dependent or obscured by linguistic blending. While abusive language detection has been extensively explored for high-resource languages like English and Hindi, low-resource languages such as Telugu and Nepali remain underrepresented, leaving gaps in effective moderation. In this study, we introduce a novel, manually annotated dataset of 2 thousand Telugu-English and 5 Nepali-English code-mixed comments, categorized as abusive and non-abusive, collected from various social media platforms. The dataset undergoes rigorous preprocessing before being evaluated across multiple Machine Learning (ML), Deep Learning (DL), and Large Language Models (LLMs). We experimented with models including Logistic Regression, Random Forest, Support Vector Machines (SVM), Neural Networks (NN), LSTM, CNN, and LLMs, optimizing their performance through hyperparameter tuning, and evaluate it using 10-fold cross-validation and statistical significance testing (t-test). Our findings provide key insights into the challenges of detecting abusive language in code-mixed settings and offer a comparative analysis of computational approaches. This study contributes to advancing NLP for low-resource languages by establishing benchmarks for abusive language detection in Telugu-English and Nepali-English code-mixed text. The dataset and insights can aid in the development of more robust moderation strategies for multilingual social media environments.
• 摘要：随着社交媒体上多语言用户的日益增长，在代码混合文本中检测滥用语言变得越来越具有挑战性。用户在英语及其母语之间无缝切换代码混合通信为传统的滥用检测模型带来了困难，因为进攻内容可能与语言融合有关上下文依赖或掩盖。尽管对英语和印地语（例如英语）语言进行了广泛探索滥用语言检测，但泰卢固语和尼泊尔等低资源语言的代表性仍然不足，使差距有效地节制。在这项研究中，我们介绍了一本小说，手动注释的数据集，其中包含来自各种社交媒体平台收集的2,000个泰卢固语 - 英语和5个尼泊尔英语代码混合评论，被归类为滥用和非虐待。在对多个机器学习（ML），深度学习（DL）和大语言模型（LLMS）进行评估之前，数据集经过严格的预处理。我们尝试了包括逻辑回归，随机森林，支持向量机（SVM），神经网络（NN），LSTM，CNN和LLM的模型，通过超参数调整优化其性能，并使用10倍的Cross-validation和统计显着性测试（T-t-Test）对其进行评估。我们的发现提供了有关在代码混合设置中检测滥用语言的挑战的关键见解，并提供了对计算方法的比较分析。这项研究通过在泰卢固语 - 英语和尼泊尔英语代码混合文本中为滥用语言检测建立基准，从而有助于推进低资源语言的NLP。数据集和洞察力可以帮助制定多语言社交媒体环境的更强大的节制策略。

Title: UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models

• Authors: Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21027
• Pdf URL: https://arxiv.org/pdf/2504.21027
• Copy Paste: [[2504.21027]] UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models(https://arxiv.org/abs/2504.21027)
• Keywords: language model, llm
• Abstract: The advent of Large Language Models (LLMs) holds promise for revolutionizing various fields traditionally dominated by human expertise. Urban planning, a professional discipline that fundamentally shapes our daily surroundings, is one such field heavily relying on multifaceted domain knowledge and experience of human experts. The extent to which LLMs can assist human practitioners in urban planning remains largely unexplored. In this paper, we introduce a comprehensive benchmark, UrbanPlanBench, tailored to evaluate the efficacy of LLMs in urban planning, which encompasses fundamental principles, professional knowledge, and management and regulations, aligning closely with the qualifications expected of human planners. Through extensive evaluation, we reveal a significant imbalance in the acquisition of planning knowledge among LLMs, with even the most proficient models falling short of meeting professional standards. For instance, we observe that 70% of LLMs achieve subpar performance in understanding planning regulations compared to other aspects. Besides the benchmark, we present the largest-ever supervised fine-tuning (SFT) dataset, UrbanPlanText, comprising over 30,000 instruction pairs sourced from urban planning exams and textbooks. Our findings demonstrate that fine-tuned models exhibit enhanced performance in memorization tests and comprehension of urban planning knowledge, while there exists significant room for improvement, particularly in tasks requiring domain-specific terminology and reasoning. By making our benchmark, dataset, and associated evaluation and fine-tuning toolsets publicly available at this https URL, we aim to catalyze the integration of LLMs into practical urban planning, fostering a symbiotic collaboration between human expertise and machine intelligence.
• 摘要：大型语言模型（LLM）的出现有望革新传统上以人类专业知识为主的各个领域。城市规划是一门从根本上塑造我们日常周围环境的专业学科，是这样的领域，严重依靠人类专家的多方面领域知识和经验。 LLM可以帮助人类从业者进行城市规划的程度，在很大程度上尚未探索。在本文中，我们介绍了一个全面的基准，Urbanplanbench量身定制，旨在评估LLM在城市规划中的疗效，该计划涵盖了基本原则，专业知识以及管理和法规，与人类计划者的预期资格紧密一致。通过广泛的评估，我们揭示了在LLM中获得计划知识的严重失衡，甚至最精通的模型也没有达到专业标准。例如，我们观察到，与其他方面相比，有70％的LLMS在理解计划法规方面取得了不同意的表现。除了基准之外，我们还介绍了有史以来最大的监督微调（SFT）数据集，即UrbanPlantext，包括来自城市规划考试和教科书的30,000多个指令对。我们的发现表明，微型模型在记忆测试和城市规划知识的理解中表现出增强的性能，同时存在着很大的改进空间，尤其是在需要特定领域的术语和推理的任务中。通过在此HTTPS URL上公开提供基准，数据集以及相关的评估和微调工具集，我们旨在将LLMS整合到实用的城市规划中，从而促进人类专业知识与机器智能之间的共生协作。

Title: Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts

• Authors: Hanhua Hong, Chenghao Xiao, Yang Wang, Yiqi Liu, Wenge Rong, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21117
• Pdf URL: https://arxiv.org/pdf/2504.21117
• Copy Paste: [[2504.21117]] Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts(https://arxiv.org/abs/2504.21117)
• Keywords: llm, prompt
• Abstract: Evaluating natural language generation (NLG) systems is challenging due to the diversity of valid outputs. While human evaluation is the gold standard, it suffers from inconsistencies, lack of standardisation, and demographic biases, limiting reproducibility. LLM-based evaluation offers a scalable alternative but is highly sensitive to prompt design, where small variations can lead to significant discrepancies. In this work, we propose an inversion learning method that learns effective reverse mappings from model outputs back to their input instructions, enabling the automatic generation of highly effective, model-specific evaluation prompts. Our method requires only a single evaluation sample and eliminates the need for time-consuming manual prompt engineering, thereby improving both efficiency and robustness. Our work contributes toward a new direction for more robust and efficient LLM-based evaluation.
• 摘要：由于有效产出的多样性，评估自然语言产生（NLG）系统是具有挑战性的。尽管人类评估是黄金标准，但它遭受了不一致，缺乏标准化和人口偏见的困扰，从而限制了可重复性。基于LLM的评估提供了可扩展的替代方案，但对及时设计非常敏感，在这种设计中，小型变化会导致重大差异。在这项工作中，我们提出了一种反演学习方法，该方法将有效的反向映射从模型输出回到其输入指令，从而可以自动生成高效，特定于模型的评估提示。我们的方法仅需要一个评估样本，而无需进行耗时的手动及时工程，从而提高了效率和鲁棒性。我们的工作朝着新的方向做出了贡献，以实现更强大，有效的基于LLM的评估。

Title: LLM Enhancer: Merged Approach using Vector Embedding for Reducing Large Language Model Hallucinations with External Knowledge

• Authors: Naheed Rayhan, Md. Ashrafuzzaman
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21132
• Pdf URL: https://arxiv.org/pdf/2504.21132
• Copy Paste: [[2504.21132]] LLM Enhancer: Merged Approach using Vector Embedding for Reducing Large Language Model Hallucinations with External Knowledge(https://arxiv.org/abs/2504.21132)
• Keywords: language model, gpt, llm, hallucination, chat, agent
• Abstract: Large Language Models (LLMs), such as ChatGPT, have demonstrated the capability to generate human like, natural responses across a range of tasks, including task oriented dialogue and question answering. However, their application in real world, critical scenarios is often hindered by a tendency to produce inaccurate information and a limited ability to leverage external knowledge sources. This paper introduces the LLM ENHANCER system, designed to integrate multiple online sources such as Google, Wikipedia, and DuckDuckGo to enhance data accuracy. The LLMs employed within this system are open source. The data acquisition process for the LLM ENHANCER system operates in parallel, utilizing custom agent tools to manage the flow of information. Vector embeddings are used to identify the most pertinent information, which is subsequently supplied to the LLM for user interaction. The LLM ENHANCER system mitigates hallucinations in chat based LLMs while preserving response naturalness and accuracy.
• 摘要：大型语言模型（LLM），例如ChatGpt，已经证明了在一系列任务中产生人类自然反应的能力，包括面向任务的对话和问题回答。但是，它们在现实世界中的应用，通常会因产生不准确的信息和利用外部知识来源的能力而受到阻碍。本文介绍了LLM增强器系统，旨在集成多个在线资源，例如Google，Wikipedia和DuckDuckgo，以提高数据准确性。该系统中使用的LLM是开源的。 LLM增强器系统的数据采集过程并行运行，利用自定义代理工具来管理信息流。向量嵌入用于识别最相关的信息，随后将其提供给LLM进行用户交互。 LLM增强剂系统可减轻基于聊天的LLM中的幻觉，同时保持自然性和准确性。

Title: Detecting Manipulated Contents Using Knowledge-Grounded Inference

• Authors: Mark Huasong Meng, Ruizhe Wang, Meng Xu, Chuan Yan, Guangdong Bai
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2504.21165
• Pdf URL: https://arxiv.org/pdf/2504.21165
• Copy Paste: [[2504.21165]] Detecting Manipulated Contents Using Knowledge-Grounded Inference(https://arxiv.org/abs/2504.21165)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The detection of manipulated content, a prevalent form of fake news, has been widely studied in recent years. While existing solutions have been proven effective in fact-checking and analyzing fake news based on historical events, the reliance on either intrinsic knowledge obtained during training or manually curated context hinders them from tackling zero-day manipulated content, which can only be recognized with real-time contextual information. In this work, we propose Manicod, a tool designed for detecting zero-day manipulated content. Manicod first sources contextual information about the input claim from mainstream search engines, and subsequently vectorizes the context for the large language model (LLM) through retrieval-augmented generation (RAG). The LLM-based inference can produce a "truthful" or "manipulated" decision and offer a textual explanation for the decision. To validate the effectiveness of Manicod, we also propose a dataset comprising 4270 pieces of manipulated fake news derived from 2500 recent real-world news headlines. Manicod achieves an overall F1 score of 0.856 on this dataset and outperforms existing methods by up to 1.9x in F1 score on their benchmarks on fact-checking and claim verification.
• 摘要：近年来，对操纵内容的检测是一种普遍的假新闻形式。尽管现有的解决方案已被证明在事实检查和基于历史事件的虚假新闻方面有效，但依赖在培训期间获得的固有知识或手动策划的上下文阻碍了他们解决零日操纵的内容，这只能通过实时上下文信息来识别。在这项工作中，我们提出了Manicod，该工具旨在检测零日操纵的内容。 MANICOD首先从主流搜索引擎中获取有关输入声明的上下文信息，然后通过检索功能增强的生成（RAG）对大语言模型（LLM）的上下文进行了矢量。基于LLM的推论可以产生“真实”或“操纵”的决定，并为该决定提供文本解释。为了验证Manicod的有效性，我们还提出了一个数据集，其中包括4270件被操纵的假新闻，这些新闻源自2500个现实世界新闻头条。 Manicod在该数据集上达到了0.856的总体F1分数，并且在事实检查和索赔验证方面的基准测试上，其基准测试中的F1得分高达1.9倍。

Title: Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare

• Authors: Lovedeep Gondara, Jonathan Simkin, Graham Sayle, Shebnum Devji, Gregory Arbour, Raymond Ng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.21191
• Pdf URL: https://arxiv.org/pdf/2504.21191
• Copy Paste: [[2504.21191]] Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare(https://arxiv.org/abs/2504.21191)
• Keywords: language model, llm
• Abstract: This study aims to guide language model selection by investigating: 1) the necessity of finetuning versus zero-shot usage, 2) the benefits of domain-adjacent versus generic pretrained models, 3) the value of further domain-specific pretraining, and 4) the continued relevance of Small Language Models (SLMs) compared to Large Language Models (LLMs) for specific tasks. Using electronic pathology reports from the British Columbia Cancer Registry (BCCR), three classification scenarios with varying difficulty and data size are evaluated. Models include various SLMs and an LLM. SLMs are evaluated both zero-shot and finetuned; the LLM is evaluated zero-shot only. Finetuning significantly improved SLM performance across all scenarios compared to their zero-shot results. The zero-shot LLM outperformed zero-shot SLMs but was consistently outperformed by finetuned SLMs. Domain-adjacent SLMs generally performed better than the generic SLM after finetuning, especially on harder tasks. Further domain-specific pretraining yielded modest gains on easier tasks but significant improvements on the complex, data-scarce task. The results highlight the critical role of finetuning for SLMs in specialized domains, enabling them to surpass zero-shot LLM performance on targeted classification tasks. Pretraining on domain-adjacent or domain-specific data provides further advantages, particularly for complex problems or limited finetuning data. While LLMs offer strong zero-shot capabilities, their performance on these specific tasks did not match that of appropriately finetuned SLMs. In the era of LLMs, SLMs remain relevant and effective, offering a potentially superior performance-resource trade-off compared to LLMs.
• 摘要：这项研究旨在通过调查来指导语言模型选择：1）填充与零摄像用法的必要性，2）域名与通用预验证的模型的好处，3）与大语言模型（LLMS相比）相比，其他域特异性预期的价值； 4）与小语言模型（SLMS）的持续相关性（SLMS）的持续相关性。使用不列颠哥伦比亚癌症登记处（BCCR）的电子病理报告，评估了三种具有不同难度和数据大小的分类方案。模型包括各种SLM和LLM。对SLM进行零拍和填充评估； LLM仅评估零射击。与零射击结果相比，在所有方案中，芬特琴显着提高了SLM的性能。零射门LLM的表现优于零射击SLM，但始终超过了FineTuned SLM的表现。固定后，域 - 染色器SLM的性能通常比通用SLM更好，尤其是在更艰难的任务上。进一步的特定领域预处理在更容易的任务上得出了适度的收益，但对复杂的数据筛选任务进行了重大改进。结果突出了SLM在专用域中登录的关键作用，从而使它们能够在目标分类任务上超过零击的LLM性能。在域名或特定领域的数据上进行预处理提供了进一步的优势，尤其是对于复杂问题或有限的芬特数据。尽管LLMS具有强大的零射击功能，但它们在这些特定任务上的性能与适当的FineTununed SLM不符。在LLMS时代，SLM仍然相关和有效，与LLMS相比，SLM提供了潜在的卓越性能资源权衡。

Title: Automatic Legal Writing Evaluation of LLMs

• Authors: Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21202
• Pdf URL: https://arxiv.org/pdf/2504.21202
• Copy Paste: [[2504.21202]] Automatic Legal Writing Evaluation of LLMs(https://arxiv.org/abs/2504.21202)
• Keywords: language model, llm
• Abstract: Despite the recent advances in Large Language Models, benchmarks for evaluating legal writing remain scarce due to the inherent complexity of assessing open-ended responses in this domain. One of the key challenges in evaluating language models on domain-specific tasks is finding test datasets that are public, frequently updated, and contain comprehensive evaluation guidelines. The Brazilian Bar Examination meets these requirements. We introduce oab-bench, a benchmark comprising 105 questions across seven areas of law from recent editions of the exam. The benchmark includes comprehensive evaluation guidelines and reference materials used by human examiners to ensure consistent grading. We evaluate the performance of four LLMs on oab-bench, finding that Claude-3.5 Sonnet achieves the best results with an average score of 7.93 out of 10, passing all 21 exams. We also investigated whether LLMs can serve as reliable automated judges for evaluating legal writing. Our experiments show that frontier models like OpenAI's o1 achieve a strong correlation with human scores when evaluating approved exams, suggesting their potential as reliable automated evaluators despite the inherently subjective nature of legal writing assessment. The source code and the benchmark -- containing questions, evaluation guidelines, model-generated responses, and their respective automated evaluations -- are publicly available.
• 摘要：尽管大型语言模型最近取得了进步，但由于评估该领域的开放式响应的固有复杂性，用于评估法律写作的基准仍然很少。评估特定领域任务的语言模型的关键挑战之一是找到公共，经常更新并包含全面评估指南的测试数据集。巴西律师考试符合这些要求。我们介绍了OAB Bench，这是一个基准，其中包括最近的考试中七个法律领域的105个问题。基准包括人类检查人员使用的全面评估指南和参考材料，以确保一致的评分。我们评估了OAB板凳上四个LLM的性能，发现Claude-3.5十四行诗在10分中的平均得分为7.93，通过了所有21次考试，可以取得最佳成绩。我们还调查了LLM是否可以作为评估法律写作的可靠自动化法官。我们的实验表明，诸如OpenAI的O1之类的边境模型在评估批准的考试时与人类得分有着密切的相关性，尽管法律写作评估的固有主观性质，但它们还是作为可靠的自动化评估者的潜力。源代码和基准（包含问题，评估指南，模型生成的响应及其各自的自动化评估）是公开的。

Title: Pretraining Large Brain Language Model for Active BCI: Silent Speech

• Authors: Jinzhao Zhou, Zehong Cao, Yiqun Duan, Connor Barkley, Daniel Leong, Xiaowei Jiang, Quoc-Toan Nguyen, Ziyi Zhao, Thomas Do, Yu-Cheng Chang, Sheng-Fu Liang, Chin-teng Lin
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2504.21214
• Pdf URL: https://arxiv.org/pdf/2504.21214
• Copy Paste: [[2504.21214]] Pretraining Large Brain Language Model for Active BCI: Silent Speech(https://arxiv.org/abs/2504.21214)
• Keywords: language model
• Abstract: This paper explores silent speech decoding in active brain-computer interface (BCI) systems, which offer more natural and flexible communication than traditional BCI applications. We collected a new silent speech dataset of over 120 hours of electroencephalogram (EEG) recordings from 12 subjects, capturing 24 commonly used English words for language model pretraining and decoding. Following the recent success of pretraining large models with self-supervised paradigms to enhance EEG classification performance, we propose Large Brain Language Model (LBLM) pretrained to decode silent speech for active BCI. To pretrain LBLM, we propose Future Spectro-Temporal Prediction (FSTP) pretraining paradigm to learn effective representations from unlabeled EEG data. Unlike existing EEG pretraining methods that mainly follow a masked-reconstruction paradigm, our proposed FSTP method employs autoregressive modeling in temporal and frequency domains to capture both temporal and spectral dependencies from EEG signals. After pretraining, we finetune our LBLM on downstream tasks, including word-level and semantic-level classification. Extensive experiments demonstrate significant performance gains of the LBLM over fully-supervised and pretrained baseline models. For instance, in the difficult cross-session setting, our model achieves 47.0\% accuracy on semantic-level classification and 39.6\% in word-level classification, outperforming baseline methods by 5.4\% and 7.3\%, respectively. Our research advances silent speech decoding in active BCI systems, offering an innovative solution for EEG language model pretraining and a new dataset for fundamental research.
• 摘要：本文探讨了主动脑机界面（BCI）系统中的无声语音解码，该系统比传统的BCI应用程序提供了更自然和灵活的通信。我们收集了一个新的无声语音数据集，其中包括120多个主题的120个小时的脑电图记录（EEG）录音，捕获了24个常用的英语单词用于语言模型预处理和解码。在最新的具有自律范式的大型模型以增强脑电图分类绩效之后，我们提出了预计的大脑语言模型（LBLM）来解码主动BCI的无声语音。为了预识LBLM，我们提出了未来的光谱时间预测（FSTP）预处理，以从未标记的EEG数据中学习有效表示。与主要遵循蒙版重建范式的现有脑电图预处理方法不同，我们提出的FSTP方法在时间和频域中采用自动回归建模，以从EEG信号中捕获时间和频谱依赖性。预处理后，我们在下游任务（包括单词级别和语义级别的分类）上对LBLM进行了验证。广泛的实验表明，LBLM在完全监督和预处理的基线模型上的显着性能提高。例如，在困难的跨课程环境中，我们的模型在语义级别的分类方面达到了47.0 \％的准确性，在单词级别的分类中达到了39.6 \％，分别优于5.4 \％\％和7.3 \％的基线方法。我们的研究推进了主动BCI系统中的无声语音解码，为EEG语言模型预处理提供了创新的解决方案，以及用于基础研究的新数据集。

Title: Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math

• Authors: Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, Shuohang Wang, Weijian Xu, Jianfeng Gao, Weizhu Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21233
• Pdf URL: https://arxiv.org/pdf/2504.21233
• Copy Paste: [[2504.21233]] Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math(https://arxiv.org/abs/2504.21233)
• Keywords: language model, llm, chain-of-thought
• Abstract: Chain-of-Thought (CoT) significantly enhances formal reasoning capabilities in Large Language Models (LLMs) by training them to explicitly generate intermediate reasoning steps. While LLMs readily benefit from such techniques, improving reasoning in Small Language Models (SLMs) remains challenging due to their limited model capacity. Recent work by Deepseek-R1 demonstrates that distillation from LLM-generated synthetic data can substantially improve the reasoning ability of SLM. However, the detailed modeling recipe is not disclosed. In this work, we present a systematic training recipe for SLMs that consists of four steps: (1) large-scale mid-training on diverse distilled long-CoT data, (2) supervised fine-tuning on high-quality long-CoT data, (3) Rollout DPO leveraging a carefully curated preference dataset, and (4) Reinforcement Learning (RL) with Verifiable Reward. We apply our method on Phi-4-Mini, a compact 3.8B-parameter model. The resulting Phi-4-Mini-Reasoning model exceeds, on math reasoning tasks, much larger reasoning models, e.g., outperforming DeepSeek-R1-Distill-Qwen-7B by 3.2 points and DeepSeek-R1-Distill-Llama-8B by 7.7 points on Math-500. Our results validate that a carefully designed training recipe, with large-scale high-quality CoT data, is effective to unlock strong reasoning capabilities even in resource-constrained small models.
• 摘要：通过训练它们明确生成中间的推理步骤，可以在大语言模型（LLMS）中显着增强正式的推理能力（COT）。尽管LLM很容易从此类技术中受益，但由于其模型有限的能力，小语言模型（SLM）的推理（SLM）仍然具有挑战性。 DeepSeek-R1的最新工作表明，从LLM生成的合成数据蒸馏可以大大提高SLM的推理能力。但是，详细的建模配方尚未披露。 In this work, we present a systematic training recipe for SLMs that consists of four steps: (1) large-scale mid-training on diverse distilled long-CoT data, (2) supervised fine-tuning on high-quality long-CoT data, (3) Rollout DPO leveraging a carefully curated preference dataset, and (4) Reinforcement Learning (RL) with Verifiable Reward.我们将方法应用于Phi-4-Mini，这是一种紧凑的3.8B参数模型。在数学推理任务上，最终的PHI-4-MINI-RENOSING模型超过了更大的推理模型，例如，在数学上优于DeepSeek-R1-Distill-Qwen-7b，胜过3.2分，而DeepSeek-R1-Distill-distill-dillama-8b则超过7.7点，在数学500上以7.7分。我们的结果证明，具有大规模高质量COT数据的精心设计的培训配方即使在资源受限的小型模型中也有效地解锁了强大的推理能力。

Title: Memorization and Knowledge Injection in Gated LLMs

• Authors: Xu Pan, Ely Hahami, Zechen Zhang, Haim Sompolinsky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21239
• Pdf URL: https://arxiv.org/pdf/2504.21239
• Copy Paste: [[2504.21239]] Memorization and Knowledge Injection in Gated LLMs(https://arxiv.org/abs/2504.21239)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) currently struggle to sequentially add new memories and integrate new knowledge. These limitations contrast with the human ability to continuously learn from new experiences and acquire knowledge throughout life. Most existing approaches add memories either through large context windows or external memory buffers (e.g., Retrieval-Augmented Generation), and studies on knowledge injection rarely test scenarios resembling everyday life events. In this work, we introduce a continual learning framework, Memory Embedded in Gated LLMs (MEGa), which injects event memories directly into the weights of LLMs. Each memory is stored in a dedicated set of gated low-rank weights. During inference, a gating mechanism activates relevant memory weights by matching query embeddings to stored memory embeddings. This enables the model to both recall entire memories and answer related questions. On two datasets - fictional characters and Wikipedia events - MEGa outperforms baseline approaches in mitigating catastrophic forgetting. Our model draws inspiration from the complementary memory system of the human brain.
• 摘要：大型语言模型（LLM）目前很难依次添加新的记忆并整合新知识。这些局限性与人类不断学习新经验并掌握知识的能力形成鲜明对比。大多数现有方法通过大型上下文窗口或外部记忆缓冲区（例如，检索出来的一代）增加了记忆，并且有关知识注入的研究很少测试类似日常生活事件的情况。在这项工作中，我们引入了一个持续的学习框架，内存嵌入了门控LLMS（Mega）中，该框架将事件记忆直接注入LLMS的权重。每个内存都存储在一组专用的封闭式低级重量中。在推断过程中，门控机制通过将查询嵌入与存储的存储器嵌入匹配来激活相关的内存权重。这使模型既可以回忆整个记忆，又可以回答相关问题。在两个数据集（虚构的角色和Wikipedia事件）上，Mega在减轻灾难性遗忘方面的表现优于基线方法。我们的模型从人脑的互补记忆系统中汲取灵感。

Title: Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QA

• Authors: Xuanzhao Dong, Wenhui Zhu, Hao Wang, Xiwen Chen, Peijie Qiu, Rui Yin, Yi Su, Yalin Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21252
• Pdf URL: https://arxiv.org/pdf/2504.21252
• Copy Paste: [[2504.21252]] Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QA(https://arxiv.org/abs/2504.21252)
• Keywords: language model, llm, hallucination, retrieval-augmented generation, agent
• Abstract: Medical question answering (QA) is a reasoning-intensive task that remains challenging for large language models (LLMs) due to hallucinations and outdated domain knowledge. Retrieval-Augmented Generation (RAG) provides a promising post-training solution by leveraging external knowledge. However, existing medical RAG systems suffer from two key limitations: (1) a lack of modeling for human-like reasoning behaviors during information retrieval, and (2) reliance on suboptimal medical corpora, which often results in the retrieval of irrelevant or noisy snippets. To overcome these challenges, we propose Discuss-RAG, a plug-and-play module designed to enhance the medical QA RAG system through collaborative agent-based reasoning. Our method introduces a summarizer agent that orchestrates a team of medical experts to emulate multi-turn brainstorming, thereby improving the relevance of retrieved content. Additionally, a decision-making agent evaluates the retrieved snippets before their final integration. Experimental results on four benchmark medical QA datasets show that Discuss-RAG consistently outperforms MedRAG, especially significantly improving answer accuracy by up to 16.67% on BioASQ and 12.20% on PubMedQA. The code is available at: this https URL.
• 摘要：医疗问题回答（QA）是一项重要的密集型任务，由于幻觉和过时的领域知识，大型语言模型（LLMS）仍然具有挑战性。通过利用外部知识，检索提升的生成（RAG）提供了有希望的训练后解决方案。但是，现有的医学抹布系统遭受了两个关键的局限性：（1）在信息检索过程中缺乏类似人类的推理行为的建模，以及（2）依靠次优医学公司，这通常会导致检索不相关或嘈杂或嘈杂的鞋类。为了克服这些挑战，我们提出了讨论Rag，这是一个旨在通过基于协作代理的推理增强医学质量质量质量抹布系统的插件模块。我们的方法介绍了一个摘要代理，该摘要代理人精心策划了一组医学专家，以模拟多头集思广益，从而改善了检索到的内容的相关性。此外，决策代理在最终集成之前评估了所检索的摘要。四个基准医疗质量检查数据集的实验结果表明，讨论rag始终超过MedRag，尤其是在Bioasq上显着提高答案准确性高达16.67％，而PubMedQA的答案准确性则高达16.67％。该代码可用：此HTTPS URL。

Title: BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models

• Authors: Zhiting Fan, Ruizhe Chen, Zuozhu Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21299
• Pdf URL: https://arxiv.org/pdf/2504.21299
• Copy Paste: [[2504.21299]] BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models(https://arxiv.org/abs/2504.21299)
• Keywords: language model, llm
• Abstract: Identifying bias in LLM-generated content is a crucial prerequisite for ensuring fairness in LLMs. Existing methods, such as fairness classifiers and LLM-based judges, face limitations related to difficulties in understanding underlying intentions and the lack of criteria for fairness judgment. In this paper, we introduce BiasGuard, a novel bias detection tool that explicitly analyzes inputs and reasons through fairness specifications to provide accurate judgments. BiasGuard is implemented through a two-stage approach: the first stage initializes the model to explicitly reason based on fairness specifications, while the second stage leverages reinforcement learning to enhance its reasoning and judgment capabilities. Our experiments, conducted across five datasets, demonstrate that BiasGuard outperforms existing tools, improving accuracy and reducing over-fairness misjudgments. We also highlight the importance of reasoning-enhanced decision-making and provide evidence for the effectiveness of our two-stage optimization pipeline.
• 摘要：确定LLM生成的内容中的偏见是确保LLM公平性的关键先决条件。现有方法，例如公平分类器和基于LLM的法官，面临与理解潜在意图的困难以及缺乏公平判断的标准有关的限制。在本文中，我们介绍了BiasGuard，这是一种新型的偏见检测工具，该工具通过公平规格明确分析输入和原因，以提供准确的判断。 BiasGuard是通过两阶段的方法实施的：第一阶段将模型初始化以根据公平规格明确理性，而第二阶段则利用强化学习来增强其推理和判断能力。我们在五个数据集中进行的实验表明，偏心者的表现优于现有工具，提高了准确性并降低了过度判断性的误解。我们还强调了推理增强决策的重要性，并为我们两阶段优化管道的有效性提供了证据。

Title: Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges

• Authors: Xiao Xiao, Yu Su, Sijing Zhang, Zhang Chen, Yadong Chen, Tian Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21303
• Pdf URL: https://arxiv.org/pdf/2504.21303
• Copy Paste: [[2504.21303]] Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges(https://arxiv.org/abs/2504.21303)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) exhibit probabilistic output characteristics, yet conventional evaluation frameworks rely on deterministic scalar metrics. This study introduces a Bayesian approach for LLM capability assessment that integrates prior knowledge through probabilistic inference, addressing limitations under limited-sample regimes. By treating model capabilities as latent variables and leveraging a curated query set to induce discriminative responses, we formalize model ranking as a Bayesian hypothesis testing problem over mutually exclusive capability intervals. Experimental evaluations with GPT-series models demonstrate that the proposed method achieves superior discrimination compared to conventional evaluation methods. Results indicate that even with reduced sample sizes, the approach maintains statistical robustness while providing actionable insights, such as probabilistic statements about a model's likelihood of surpassing specific baselines. This work advances LLM evaluation methodologies by bridging Bayesian inference with practical constraints in real-world deployment scenarios.
• 摘要：大型语言模型（LLMS）具有概率的输出特征，但常规的评估框架依赖于确定性标量指标。这项研究介绍了一种用于LLM功能评估的贝叶斯方法，该方法通过概率推断整合了先验知识，解决了有限样本制度下的局限性。通过将模型功能视为潜在变量，并利用策划的查询来诱导歧视性响应，我们将模型排名正式为贝叶斯假说测试问题，而不是相互排斥的能力间隔。 GPT系列模型的实验评估表明，与常规评估方法相比，所提出的方法可以实现优越的歧视。结果表明，即使样本量减少，该方法也保持统计鲁棒性，同时提供可行的见解，例如有关模型超过特定基线的可能性的概率陈述。这项工作通过将贝叶斯推断与现实部署方案中的实际约束桥接来推进LLM评估方法。

Title: Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?

• Authors: Kaixun Yang, Mladen Raković, Dragan Gašević, Guanliang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21330
• Pdf URL: https://arxiv.org/pdf/2504.21330
• Copy Paste: [[2504.21330]] Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?(https://arxiv.org/abs/2504.21330)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large Language Models (LLMs) are widely used in Automated Essay Scoring (AES) due to their ability to capture semantic meaning. Traditional fine-tuning approaches required technical expertise, limiting accessibility for educators with limited technical backgrounds. However, prompt-based tools like ChatGPT have made AES more accessible, enabling educators to obtain machine-generated scores using natural-language prompts (i.e., the prompt-based paradigm). Despite advancements, prior studies have shown bias in fine-tuned LLMs, particularly against disadvantaged groups. It remains unclear whether such biases persist or are amplified in the prompt-based paradigm with cutting-edge tools. Since such biases are believed to stem from the demographic information embedded in pre-trained models (i.e., the ability of LLMs' text embeddings to predict demographic attributes), this study explores the relationship between the model's predictive power of students' demographic attributes based on their written works and its predictive bias in the scoring task in the prompt-based paradigm. Using a publicly available dataset of over 25,000 students' argumentative essays, we designed prompts to elicit demographic inferences (i.e., gender, first-language background) from GPT-4o and assessed fairness in automated scoring. Then we conducted multivariate regression analysis to explore the impact of the model's ability to predict demographics on its scoring outcomes. Our findings revealed that (i) prompt-based LLMs can somewhat infer students' demographics, particularly their first-language backgrounds, from their essays; (ii) scoring biases are more pronounced when the LLM correctly predicts students' first-language background than when it does not; and (iii) scoring error for non-native English speakers increases when the LLM correctly identifies them as non-native.
• 摘要：大型语言模型（LLM）由于能够捕获语义含义的能力而广泛用于自动化论文评分（AES）。传统的微调方法需要技术专长，这限制了技术背景有限的教育者的可访问性。但是，诸如CHATGPT之类的及时基于的工具使AES更容易访问，使教育工作者能够使用自然语言提示（即基于及时的范式）获得机器生成的分数。尽管有进步，但先前的研究表明，通过微调的LLM，特别是针对不利的群体的偏见。目前尚不清楚这种偏见是否持续或在具有尖端工具的基于及时的范式中放大。由于认为这种偏见源于预先训练模型中的人口统计信息（即LLMS文本嵌入预测人口统计学属性的能力），因此本研究探讨了基于迅速基于迅速的Paradigm的计分任务中学生人口统计学属性的预测能力的预测能力。我们使用了25,000多个学生的辩论论文的公开数据集，我们设计了提示，以引起来自GPT-4O的人群推断（即性别，第一语言背景），并在自动评分中评估了公平性。然后，我们进行了多元回归分析，以探讨该模型预测人口统计学评分结果的能力的影响。我们的发现表明，（i）基于迅速的LLM可以从论文中推断出学生的人口统计，尤其是他们的第一语言背景； （ii）当LLM正确预测学生的第一语言背景时，评分偏见比没有的偏见更为明显； （iii）当LLM正确地将其识别为非本地人时，非母语说话者的评分错误会增加。

Title: Retrieval-Enhanced Few-Shot Prompting for Speech Event Extraction

• Authors: Máté Gedeon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21372
• Pdf URL: https://arxiv.org/pdf/2504.21372
• Copy Paste: [[2504.21372]] Retrieval-Enhanced Few-Shot Prompting for Speech Event Extraction(https://arxiv.org/abs/2504.21372)
• Keywords: language model, gpt, llm, prompt
• Abstract: Speech Event Extraction (SpeechEE) is a challenging task that lies at the intersection of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP), requiring the identification of structured event information from spoken language. In this work, we present a modular, pipeline-based SpeechEE framework that integrates high-performance ASR with semantic search-enhanced prompting of Large Language Models (LLMs). Our system first classifies speech segments likely to contain events using a hybrid filtering mechanism including rule-based, BERT-based, and LLM-based models. It then employs few-shot LLM prompting, dynamically enriched via semantic similarity retrieval, to identify event triggers and extract corresponding arguments. We evaluate the pipeline using multiple LLMs (Llama3-8B, GPT-4o-mini, and o1-mini) highlighting significant performance gains with o1-mini, which achieves 63.3% F1 on trigger classification and 27.8% F1 on argument classification, outperforming prior benchmarks. Our results demonstrate that pipeline approaches, when empowered by retrieval-augmented LLMs, can rival or exceed end-to-end systems while maintaining interpretability and modularity. This work provides practical insights into LLM-driven event extraction and opens pathways for future hybrid models combining textual and acoustic features.
• 摘要：语音事件提取（演讲者）是一项具有挑战性的任务，是自动语音识别（ASR）和自然语言处理（NLP）的交集，需要从口语中识别结构化事件信息。在这项工作中，我们提出了一个基于管道的演讲者框架，该框架将高性能ASR与大型语言模型（LLMS）的语义搜索增强提示（LLMS）集成在一起。我们的系统首先将语音段分类可能包含使用混合过滤机制，包括基于规则，基于BERT和LLM的模型的事件。然后，它采用了很少的LLM提示，通过语义相似性检索动态丰富，以识别事件触发器并提取相应的参数。我们使用多个LLM（LLAMA3-8B，GPT-4O-MINI和O1-MINI）评估管道，强调了O1-Mini的显着性能提高，O1-Mini在触发分类上实现了63.3％的F1，而在参数分类上却胜过了27.8％的F1，超过了先前的Benchmarks。我们的结果表明，当通过检索授权的LLM授权时，管道方法可以竞争或超过端到端系统，同时保持可解释性和模块化。这项工作为LLM驱动的事件提取提供了实用的见解，并为结合文本和声学特征结合的未来混合模型打开了途径。

Title: RWKV-X: A Linear Complexity Hybrid Language Model

• Authors: Haowen Hou, Zhiyi Huang, Kaifeng Tan, Rongchang Lu, Fei Richard Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21463
• Pdf URL: https://arxiv.org/pdf/2504.21463
• Copy Paste: [[2504.21463]] RWKV-X: A Linear Complexity Hybrid Language Model(https://arxiv.org/abs/2504.21463)
• Keywords: language model
• Abstract: In this paper, we introduce \textbf{RWKV-X}, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture long-range context. Unlike previous hybrid approaches that rely on full attention layers and retain quadratic complexity, RWKV-X achieves linear-time complexity in training and constant-time complexity in inference decoding. We demonstrate that RWKV-X, when continually pretrained on 64K-token sequences, achieves near-perfect accuracy on the 64K passkey retrieval benchmark. It consistently outperforms prior RWKV-7 models on long-context benchmarks, while maintaining strong performance on short-context tasks. These results highlight RWKV-X as a scalable and efficient backbone for general-purpose language modeling, capable of decoding sequences up to 1 million tokens with stable speed and memory usage. To facilitate further research and analysis, we have made the checkpoints and the associated code publicly accessible at: this https URL.
• 摘要：在本文中，我们介绍了\ textbf {rwkv-x}，这是一种新型混合体系结构，将RWKV对于短距离建模的效率与旨在捕获长期环境的稀疏注意机制相结合。与以前的混合方法依赖于全部注意层并保留二次复杂性不同，RWKV-X在训练和推理解码中的恒定时间复杂性中实现了线性时间的复杂性。我们证明，RWKV-X在64k token序列上不断预估计时，可以在64K Passkey检索基准上实现几乎完美的精度。它始终在长篇小写基准测试上胜过先前的RWKV-7模型，同时保持在短篇小说任务上的强劲性能。这些结果突出了RWKV-X作为通用语言建模的可扩展且有效的主链，能够以稳定的速度和内存使用情况来解码高达100万个令牌的序列。为了促进进一步的研究和分析，我们在以下位置进行了检查点和相关的代码：此HTTPS URL。

Title: Homa at SemEval-2025 Task 5: Aligning Librarian Records with OntoAligner for Subject Tagging

• Authors: Hadi Bayrami Asl Tekanlou, Jafar Razmara, Mahsa Sanaei, Mostafa Rahgouy, Hamed Babaei Giglou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21474
• Pdf URL: https://arxiv.org/pdf/2504.21474
• Copy Paste: [[2504.21474]] Homa at SemEval-2025 Task 5: Aligning Librarian Records with OntoAligner for Subject Tagging(https://arxiv.org/abs/2504.21474)
• Keywords: retrieval-augmented generation
• Abstract: This paper presents our system, Homa, for SemEval-2025 Task 5: Subject Tagging, which focuses on automatically assigning subject labels to technical records from TIBKAT using the Gemeinsame Normdatei (GND) taxonomy. We leverage OntoAligner, a modular ontology alignment toolkit, to address this task by integrating retrieval-augmented generation (RAG) techniques. Our approach formulates the subject tagging problem as an alignment task, where records are matched to GND categories based on semantic similarity. We evaluate OntoAligner's adaptability for subject indexing and analyze its effectiveness in handling multilingual records. Experimental results demonstrate the strengths and limitations of this method, highlighting the potential of alignment techniques for improving subject tagging in digital libraries.
• 摘要：本文介绍了我们的系统，用于Semeval-2025任务5：主题标签，该标记的重点是使用Gemeinsame Normdatei（GND）分类自动将主题标签分配给Tibkat的技术记录。我们利用模块化本体对准工具包上的Aligner来解决此任务，通过集成检索型生成（RAG）技术。我们的方法将主题标记问题作为一项对齐任务提出，其中记录基于语义相似性匹配GND类别。我们评估了Alagigner的适应性，以索引并分析其在处理多语言记录中的有效性。实验结果证明了这种方法的优势和局限性，突出了对齐技术改善数字库中受试者标记的潜力。

Title: Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models

• Authors: Lucas Maisonnave, Cyril Moineau, Olivier Bichler, Fabrice Rastello
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21553
• Pdf URL: https://arxiv.org/pdf/2504.21553
• Copy Paste: [[2504.21553]] Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models(https://arxiv.org/abs/2504.21553)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, their size presents significant challenges for deployment and inference. This paper investigates the quantization of LLMs, focusing on the LLaMA architecture and its derivatives. We challenge existing assumptions about activation outliers in LLMs and propose a novel mixed-precision quantization approach tailored for LLaMA-like models. Our method leverages the observation that activation spikes in LLaMA architectures are predominantly concentrated in specific projection layers. By applying higher precision (FP16 or FP8) to these layers while quantizing the rest of the model to lower bit-widths, we achieve superior performance compared to existing quantization techniques. Experimental results on LLaMA2, LLaMA3, and Mistral models demonstrate significant improvements in perplexity and zero-shot accuracy, particularly for 8-bit per-tensor quantization. Our approach outperforms general-purpose methods designed to handle outliers across all architecture types, highlighting the benefits of architecture-specific quantization strategies. This research contributes to the ongoing efforts to make LLMs more efficient and deployable, potentially enabling their use in resource-constrained environments. Our findings emphasize the importance of considering model-specific characteristics in developing effective quantization pipelines for state-of-the-art language models by identifying and targeting a small number of projections that concentrate activation spikes.
• 摘要：大型语言模型（LLM）在各种自然语言处理任务中表现出了显着的功能。但是，它们的规模面临部署和推理的重大挑战。本文研究了LLMS的量化，重点介绍了Llama建筑及其衍生物。我们挑战了有关LLMS中激活异常值的现有假设，并提出了一种针对Llama样模型量身定制的新型混合精确量化方法。我们的方法利用了这样的观察结果，即在特定投影层中，骆驼架构中的激活尖峰主要集中在特定的投影层中。通过将较高的精度（FP16或FP8）应用于这些层，同时将模型其余部分量化以降低位宽度，与现有的量化技术相比，我们实现了卓越的性能。 Llama2，Llama3和Mistral模型的实验结果表明，困惑性和零照片的精度有显着改善，尤其是对于8位每张量量化而言。我们的方法优于旨在处理所有体系结构类型的离群值的通用方法，突出了特定于建筑特定的量化策略的好处。这项研究有助于持续的努力，使LLMS更有效，可部署，并有可能在资源受限的环境中使用。我们的发现强调了通过识别和定位少数集中激活尖峰的投影来考虑为最先进的语言模型开发有效量化管道时考虑特定模型特征的重要性。

Title: DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing

• Authors: Lisa Kluge, Maximilian Kähler
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2504.21589
• Pdf URL: https://arxiv.org/pdf/2504.21589
• Copy Paste: [[2504.21589]] DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing(https://arxiv.org/abs/2504.21589)
• Keywords: llm, prompt
• Abstract: This paper presents our system developed for the SemEval-2025 Task 5: LLMs4Subjects: LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog. Our system relies on prompting a selection of LLMs with varying examples of intellectually annotated records and asking the LLMs to similarly suggest keywords for new records. This few-shot prompting technique is combined with a series of post-processing steps that map the generated keywords to the target vocabulary, aggregate the resulting subject terms to an ensemble vote and, finally, rank them as to their relevance to the record. Our system is fourth in the quantitative ranking in the all-subjects track, but achieves the best result in the qualitative ranking conducted by subject indexing experts.
• 摘要：本文介绍了我们为Semeval-2025任务开发的系统5：LLMS4Subjects：基于LLM的自动化受试者标记国家技术图书馆的开放访问目录。我们的系统依赖于促使LLM的选择，其中具有不同的智力注释记录示例，并要求LLMS类似地建议新记录的关键字。这种几次启动的提示技术与一系列的后处理步骤相结合，将生成的关键字映射到目标词汇中，将结果主题汇总为合奏投票，最后对它们与记录的相关性进行排名。我们的系统在全受试者轨道的定量排名中排名第四，但在主题索引专家进行的定性排名中取得了最佳成果。

Title: RDF-Based Structured Quality Assessment Representation of Multilingual LLM Evaluations

• Authors: Jonas Gwozdz, Andreas Both
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2504.21605
• Pdf URL: https://arxiv.org/pdf/2504.21605
• Copy Paste: [[2504.21605]] RDF-Based Structured Quality Assessment Representation of Multilingual LLM Evaluations(https://arxiv.org/abs/2504.21605)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) increasingly serve as knowledge interfaces, yet systematically assessing their reliability with conflicting information remains difficult. We propose an RDF-based framework to assess multilingual LLM quality, focusing on knowledge conflicts. Our approach captures model responses across four distinct context conditions (complete, incomplete, conflicting, and no-context information) in German and English. This structured representation enables the comprehensive analysis of knowledge leakage-where models favor training data over provided context-error detection, and multilingual consistency. We demonstrate the framework through a fire safety domain experiment, revealing critical patterns in context prioritization and language-specific performance, and demonstrating that our vocabulary was sufficient to express every assessment facet encountered in the 28-question study.
• 摘要：大型语言模型（LLMS）越来越多地用作知识界面，但是系统地评估其可靠性与相互矛盾的信息仍然很困难。我们提出了一个基于RDF的框架，以评估多语言LLM质量，重点是知识冲突。我们的方法在德语和英语中捕获了四个不同的上下文条件（完整，不完整，相互冲突和无文本信息）的模型响应。该结构化表示可以对知识泄漏进行全面分析，其中模型比提供的上下文检测和多语言一致性有利于培训数据。我们通过消防安全域实验来展示框架，揭示了上下文优先级和语言特定表现中的临界模式，并证明我们的词汇足以表达在28个问题研究中遇到的每个评估方面。

Title: Meeseeks: An Iterative Benchmark Evaluating LLMs Multi-Turn Instruction-Following Ability

• Authors: Jiaming Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21625
• Pdf URL: https://arxiv.org/pdf/2504.21625
• Copy Paste: [[2504.21625]] Meeseeks: An Iterative Benchmark Evaluating LLMs Multi-Turn Instruction-Following Ability(https://arxiv.org/abs/2504.21625)
• Keywords: language model, llm, agent
• Abstract: The ability to follow instructions accurately is fundamental for Large Language Models (LLMs) to serve as reliable agents in real-world applications. While existing instruction-following benchmarks are either single-turn or introduce new requirements in each turn without allowing self-correction, Meeseeks simulates realistic human-LLM interactions through an iterative feedback process. This design enables models to self-correct based on specific requirement failures, better reflecting real-world user-end usage patterns. The benchmark implements a comprehensive evaluation system with 38 capability tags organized across three dimensions: Intent Recognition, Granular Content Validation, and Output Structure Validation. Through rigorous evaluation across LLMs, Meeseeks provides valuable insights into LLMs' instruction-following capabilities in practical applications.
• 摘要：准确遵循指令的能力对于大语模型（LLM）是现实世界应用中可靠代理的基础。虽然现有的遵循基准是单转的，或者在每个回合中都不允许自我纠正引入新的要求，但Meeseeks通过迭代反馈过程模拟现实的人-LLM交互。该设计使模型能够根据特定要求失败进行自我纠正，更好地反映现实世界的用户端使用模式。基准测试实现了一个全面的评估系统，该系统具有38个能力标签，该系统在三个维度上组织：意图识别，颗粒内容验证和输出结构验证。通过跨LLM的严格评估，Meeseeks对LLMS在实际应用中的指导遵循功能提供了宝贵的见解。

Title: Sadeed: Advancing Arabic Diacritization Through Small Language Model

• Authors: Zeina Aldallal, Sara Chrouf, Khalil Hennara, Mohamed Motaism Hamed, Muhammad Hreden, Safwan AlModhayan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21635
• Pdf URL: https://arxiv.org/pdf/2504.21635
• Copy Paste: [[2504.21635]] Sadeed: Advancing Arabic Diacritization Through Small Language Model(https://arxiv.org/abs/2504.21635)
• Keywords: language model
• Abstract: Arabic text diacritization remains a persistent challenge in natural language processing due to the language's morphological richness. In this paper, we introduce Sadeed, a novel approach based on a fine-tuned decoder-only language model adapted from Kuwain 1.5B Hennara et al. [2025], a compact model originally trained on diverse Arabic corpora. Sadeed is fine-tuned on carefully curated, high-quality diacritized datasets, constructed through a rigorous data-cleaning and normalization pipeline. Despite utilizing modest computational resources, Sadeed achieves competitive results compared to proprietary large language models and outperforms traditional models trained on similar domains. Additionally, we highlight key limitations in current benchmarking practices for Arabic diacritization. To address these issues, we introduce SadeedDiac-25, a new benchmark designed to enable fairer and more comprehensive evaluation across diverse text genres and complexity levels. Together, Sadeed and SadeedDiac-25 provide a robust foundation for advancing Arabic NLP applications, including machine translation, text-to-speech, and language learning tools.
• 摘要：由于语言的形态丰富性，阿拉伯文本的数字化仍然是自然语言处理中的持续挑战。在本文中，我们介绍了一种基于纯正调整解码器的语言模型的新颖方法，该模型改编自Kuwain 1.5B Hennara等人。 [2025]，一种紧凑型模型，最初是在多种阿拉伯语料库中培训的。通过严格的数据清洁和归一化管道构建的精心策划，高质量的数字数据集进行了微调。尽管利用适中的计算资源，但与专有的大语言模型相比，Sadeed取得了竞争成果，并且优于在类似领域训练的传统模型。此外，我们重点介绍了当前基准测试实践的关键限制。为了解决这些问题，我们介绍了SadeedDiac-25，这是一种新的基准测试，旨在实现各种文本类型和复杂性水平的更公平，更全面的评估。共同，Sadeed和SadeedDiac-25为推进阿拉伯NLP应用程序（包括机器翻译，文本到语音学习和语言学习工具）提供了强大的基础。

Title: Investigating Literary Motifs in Ancient and Medieval Novels with Large Language Models

• Authors: Emelie Hallenberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.21742
• Pdf URL: https://arxiv.org/pdf/2504.21742
• Copy Paste: [[2504.21742]] Investigating Literary Motifs in Ancient and Medieval Novels with Large Language Models(https://arxiv.org/abs/2504.21742)
• Keywords: language model
• Abstract: The Greek fictional narratives often termed love novels or romances, ranging from the first century CE to the middle of the 15th century, have long been considered as similar in many ways, not least in the use of particular literary motifs. By applying the use of fine-tuned large language models, this study aims to investigate which motifs exactly that the texts in this corpus have in common, and in which ways they differ from each other. The results show that while some motifs persist throughout the corpus, others fluctuate in frequency, indicating certain trends or external influences. Conclusively, the method proves to adequately extract literary motifs according to a set definition, providing data for both quantitative and qualitative analyses.
• 摘要：希腊虚构的叙事经常被称为“爱小说或浪漫史”，从一世纪的公元到15世纪中叶，长期以来一直被认为是相似的，尤其是在使用特定的文学主题中。通过应用微调的大语言模型，本研究的目的是研究哪些基于该语料库中的文本具有共同点，以及它们彼此之间有所不同。结果表明，虽然某些主题在整个语料库中都存在，但其他主题却在频率上波动，表明某些趋势或外部影响。结论性地，该方法证明是根据设定的定义充分提取文学基础的，为定量和定性分析提供了数据。

Title: MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness

• Authors: Junsheng Huang, Zhitao He, Sandeep Polisetty, Qingyun Wang, May Fung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21773
• Pdf URL: https://arxiv.org/pdf/2504.21773
• Copy Paste: [[2504.21773]] MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness(https://arxiv.org/abs/2504.21773)
• Keywords: language model, llm, hallucination
• Abstract: With the widespread application of large language models (LLMs), the issue of generating non-existing facts, known as hallucination, has garnered increasing attention. Previous research in enhancing LLM confidence estimation mainly focuses on the single problem setting. However, LLM awareness of its internal parameterized knowledge boundary under the more challenging multi-problem setting, which requires answering multiple problems accurately simultaneously, remains underexplored. To bridge this gap, we introduce a novel method, Multiple Answers and Confidence Stepwise Tuning (MAC-Tuning), that separates the learning of answer prediction and confidence estimation during fine-tuning on instruction data. Extensive experiments demonstrate that our method outperforms baselines by up to 25% in average precision.
• 摘要：随着大语言模型（LLM）的广泛应用，生成不存在的事实（称为幻觉）的问题引起了人们越来越多的关注。提高LLM置信度估计的先前研究主要集中在单个问题设置上。但是，LLM在更具挑战性的多问题设置下对其内部参数化知识边界的认识，该设置需要同时准确地回答多个问题，但仍未得到充实。为了弥合这一差距，我们介绍了一种新颖的方法，多个答案和置信度逐步调整（MAC-调整），该方法将学习答案预测和置信度估计的学习分开。广泛的实验表明，我们的方法的平均精度最高超过25％。

Title: WebThinker: Empowering Large Reasoning Models with Deep Research Capability

• Authors: Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, Zhicheng Dou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2504.21776
• Pdf URL: https://arxiv.org/pdf/2504.21776
• Copy Paste: [[2504.21776]] WebThinker: Empowering Large Reasoning Models with Deep Research Capability(https://arxiv.org/abs/2504.21776)
• Keywords: agent
• Abstract: Large reasoning models (LRMs), such as OpenAI-o1 and DeepSeek-R1, demonstrate impressive long-horizon reasoning capabilities. However, their reliance on static internal knowledge limits their performance on complex, knowledge-intensive tasks and hinders their ability to produce comprehensive research reports requiring synthesis of diverse web information. To address this, we propose \textbf{WebThinker}, a deep research agent that empowers LRMs to autonomously search the web, navigate web pages, and draft research reports during the reasoning process. WebThinker integrates a \textbf{Deep Web Explorer} module, enabling LRMs to dynamically search, navigate, and extract information from the web when encountering knowledge gaps. It also employs an \textbf{Autonomous Think-Search-and-Draft strategy}, allowing the model to seamlessly interleave reasoning, information gathering, and report writing in real time. To further enhance research tool utilization, we introduce an \textbf{RL-based training strategy} via iterative online Direct Preference Optimization (DPO). Extensive experiments on complex reasoning benchmarks (GPQA, GAIA, WebWalkerQA, HLE) and scientific report generation tasks (Glaive) demonstrate that WebThinker significantly outperforms existing methods and strong proprietary systems. Our approach enhances LRM reliability and applicability in complex scenarios, paving the way for more capable and versatile deep research systems. The code is available at this https URL.
• 摘要：大型推理模型（LRMS），例如OpenAI-O1和DeepSeek-R1，表现出令人印象深刻的长途推理能力。但是，他们对静态内部知识的依赖将其表现限制在复杂，知识密集的任务上，并阻碍了他们生成需要综合不同Web信息的全面研究报告的能力。为了解决这个问题，我们建议\ textbf {WebThinker}，这是一位深入的研究代理，授权LRMS自主搜索网络，导航网页和在推理过程中的研究报告草案草案。 WebThinker集成了A \ TextBf {Deep Web Explorer}模块，使LRMS能够在遇到知识差距时动态搜索，导航和提取信息。它还采用\ textbf {自主思维 - 搜索策略}，允许模型无缝地交织推理，信息收集和实时报告写作。为了进一步增强研究工具的利用，我们通过迭代在线直接偏好优化（DPO）引入了\ textbf {rl的培训策略}。关于复杂推理基准（GPQA，Gaia，Webwalkerqa，HLE）和科学报告生成任务（GLAIVE）的广泛实验表明，WebThinker的表现会大大优于现有方法和强大的专有系统。我们的方法在复杂的场景中增强了LRM的可靠性和适用性，为更有能力和多功能的深层研究系统铺平了道路。该代码可在此HTTPS URL上找到。

Title: DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition

• Authors: Z.Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z.F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao, Daya Guo, Chong Ruan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21801
• Pdf URL: https://arxiv.org/pdf/2504.21801
• Copy Paste: [[2504.21801]] DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition(https://arxiv.org/abs/2504.21801)
• Keywords: language model, prompt, chain-of-thought
• Abstract: We introduce DeepSeek-Prover-V2, an open-source large language model designed for formal theorem proving in Lean 4, with initialization data collected through a recursive theorem proving pipeline powered by DeepSeek-V3. The cold-start training procedure begins by prompting DeepSeek-V3 to decompose complex problems into a series of subgoals. The proofs of resolved subgoals are synthesized into a chain-of-thought process, combined with DeepSeek-V3's step-by-step reasoning, to create an initial cold start for reinforcement learning. This process enables us to integrate both informal and formal mathematical reasoning into a unified model. The resulting model, DeepSeek-Prover-V2-671B, achieves state-of-the-art performance in neural theorem proving, reaching 88.9% pass ratio on the MiniF2F-test and solving 49 out of 658 problems from PutnamBench. In addition to standard benchmarks, we introduce ProverBench, a collection of 325 formalized problems, to enrich our evaluation, including 15 selected problems from the recent AIME competitions (years 24-25). Further evaluation on these 15 AIME problems shows that the model successfully solves 6 of them. In comparison, DeepSeek-V3 solves 8 of these problems using majority voting, highlighting that the gap between formal and informal mathematical reasoning in large language models is substantially narrowing.
• 摘要：我们介绍了DeepSeek-Prover-V2，这是一种旨在正式定理的开源大语言模型，该模型在LEAN 4中证明，通过递归定理收集的初始化数据证明了由DeepSeek-V3供电的管道。冷启动训练程序首先促使DeepSeek-V3将复杂的问题分解为一系列子目标。解决的子目标的证明被合成为一个经过思考的过程，并结合了DeepSeek-V3的逐步推理，为增强学习创造了最初的冷启动。这个过程使我们能够将非正式和正式的数学推理同时整合到统一的模型中。最终的模型DeepSeek-Prover-V2-671b在神经定理中实现了最先进的表现，在MiniF2F检验中达到了88.9％的通行率，并在Putnambench的658个问题中解决了49个问题。除标准基准测试外，我们还介绍了proverbench，这是325个形式化问题的集合，以丰富我们的评估，其中包括最近的Aime竞赛中的15个选定问题（24-25岁）。对这15个AIME问题的进一步评估表明，该模型成功解决了其中的6个。相比之下，DeepSeek-V3使用多数投票解决了其中的8个问题，这强调了大语言模型中正式和非正式数学推理之间的差距大大缩小。

Title: TRUST: An LLM-Based Dialogue System for Trauma Understanding and Structured Assessments

• Authors: Sichang Tu, Abigail Powers, Stephen Doogan, Jinho D. Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.21851
• Pdf URL: https://arxiv.org/pdf/2504.21851
• Copy Paste: [[2504.21851]] TRUST: An LLM-Based Dialogue System for Trauma Understanding and Structured Assessments(https://arxiv.org/abs/2504.21851)
• Keywords: language model, llm, agent
• Abstract: Objectives: While Large Language Models (LLMs) have been widely used to assist clinicians and support patients, no existing work has explored dialogue systems for standard diagnostic interviews and assessments. This study aims to bridge the gap in mental healthcare accessibility by developing an LLM-powered dialogue system that replicates clinician behavior. Materials and Methods: We introduce TRUST, a framework of cooperative LLM modules capable of conducting formal diagnostic interviews and assessments for Post-Traumatic Stress Disorder (PTSD). To guide the generation of appropriate clinical responses, we propose a Dialogue Acts schema specifically designed for clinical interviews. Additionally, we develop a patient simulation approach based on real-life interview transcripts to replace time-consuming and costly manual testing by clinicians. Results: A comprehensive set of evaluation metrics is designed to assess the dialogue system from both the agent and patient simulation perspectives. Expert evaluations by conversation and clinical specialists show that TRUST performs comparably to real-life clinical interviews. Discussion: Our system performs at the level of average clinicians, with room for future enhancements in communication styles and response appropriateness. Conclusions: Our TRUST framework shows its potential to facilitate mental healthcare availability.
• 摘要：目标：尽管大型语言模型（LLM）已被广泛用于协助临床医生和支持患者，但没有现有的工作探索了对话系统，用于标准诊断访谈和评估。这项研究旨在通过开发复制临床医生行为的LLM驱动的对话系统来弥合心理保健可及性的差距。材料和方法：我们引入信任，这是一个合作LLM模块的框架，能够进行创伤后应激障碍（PTSD）的正式诊断访谈和评估。为了指导生成适当的临床反应，我们提出了专门为临床访谈设计的对话法案。此外，我们根据现实生活中的访谈记录开发了一种患者模拟方法，以替代临床医生的耗时且昂贵的手动测试。结果：一组全面的评估指标旨在从代理和患者模拟的角度评估对话系统。对话和临床专家的专家评估表明，信任与现实生活中的临床访谈相当。讨论：我们的系统在普通临床医生的水平上表现，并为未来的沟通风格和响应适当性提供了空间。结论：我们的信任框架显示了其促进心理保健可用性的潜力。