2025-03-28

Title: ECLAIR: Enhanced Clarification for Interactive Responses in an Enterprise AI Assistant

• Authors: John Murzaku, Zifan Liu, Vaishnavi Muppala, Md Mehrab Tanjim, Xiang Chen, Yunyao Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20791
• Pdf URL: https://arxiv.org/pdf/2503.20791
• Copy Paste: [[2503.20791]] ECLAIR: Enhanced Clarification for Interactive Responses in an Enterprise AI Assistant(https://arxiv.org/abs/2503.20791)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have shown remarkable progress in understanding and generating natural language across various applications. However, they often struggle with resolving ambiguities in real-world, enterprise-level interactions, where context and domain-specific knowledge play a crucial role. In this demonstration, we introduce ECLAIR (Enhanced CLArification for Interactive Responses), a multi-agent framework for interactive disambiguation. ECLAIR enhances ambiguous user query clarification through an interactive process where custom agents are defined, ambiguity reasoning is conducted by the agents, clarification questions are generated, and user feedback is leveraged to refine the final response. When tested on real-world customer data, ECLAIR demonstrates significant improvements in clarification question generation compared to standard few-shot methods.
• 摘要：大型语言模型（LLMS）在理解和生成各种应用程序的自然语言方面表现出色。但是，他们经常在解决现实世界中的企业层次互动中解决歧义，在这种互动中，上下文和特定领域的知识起着至关重要的作用。在此演示中，我们介绍了Eclair（增强了交互式响应的澄清），这是一个用于交互式歧义的多代理框架。 Eclair通过定义自定义代理，歧义推理的交互过程来增强歧义用户查询澄清，由代理进行歧义推理，生成澄清问题，并利用用户反馈来完善最终响应。在对现实世界的客户数据进行测试时，Eclair与标准的几杆方法相比，澄清问题的产生显着改善。

Title: Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text DeIdentification?

• Authors: Veysel Kocaman, Muhammed Santas, Yigit Gul, Mehmet Butgul, David Talby
• Subjects: cs.CL, cs.CR, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.20794
• Pdf URL: https://arxiv.org/pdf/2503.20794
• Copy Paste: [[2503.20794]] Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text DeIdentification?(https://arxiv.org/abs/2503.20794)
• Keywords: gpt
• Abstract: We systematically assess the performance of three leading API-based de-identification systems - Azure Health Data Services, AWS Comprehend Medical, and OpenAI GPT-4o - against our de-identification systems on a ground truth dataset of 48 clinical documents annotated by medical experts. Our analysis, conducted at both entity-level and token-level, demonstrates that our solution, Healthcare NLP, achieves the highest accuracy, with a 96% F1-score in protected health information (PHI) detection, significantly outperforming Azure (91%), AWS (83%), and GPT-4o (79%). Beyond accuracy, Healthcare NLP is also the most cost-effective solution, reducing processing costs by over 80% compared to Azure and GPT-4o. Its fixed-cost local deployment model avoids the escalating per-request fees of cloud-based services, making it a scalable and economical choice. Our results underscore a critical limitation: zero-shot commercial APIs fail to meet the accuracy, adaptability, and cost-efficiency required for regulatory-grade clinical de-identification. Healthcare NLP's superior performance, customization capabilities, and economic advantages position it as the more viable solution for healthcare organizations seeking compliance and scalability in clinical NLP workflows.
• 摘要：我们系统地评估了三个领先的基于API的去识别系统的性能 - Azure Health Data Services，AWS了解医疗和OpenAI GPT-4O-对我们在48个由医学专家注释的48个临床文档的地面真实数据集上的去识别系统。我们在实体级和令牌级进行的分析表明，我们的解决方案NLP实现了最高的精度，在受保护的健康信息（PHI）检测中的F1得分为96％，显着超过了Azure（91％），AWS（83％）和GPT-4O（GPT-4O）（79％）（79％）。除了准确性之外，医疗保健NLP也是最具成本效益的解决方案，与Azure和GPT-4O相比，将处理成本降低了80％以上。其固定成本的本地部署模型避免了基于云的服务的每次要求升级，这使其成为可扩展且经济的选择。我们的结果强调了一个关键的限制：零拍商业API无法满足监管级临床去识别所需的准确性，适应性和成本效益。医疗保健NLP的出色绩效，定制能力和经济优势将其定位为寻求临床NLP工作流程合规性和可扩展性的医疗组织更可行的解决方案。

Title: "Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection

• Authors: Muhammad Haroon, Magdalena Wojcieszak, Anshuman Chhabra
• Subjects: cs.CL, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2503.20797
• Pdf URL: https://arxiv.org/pdf/2503.20797
• Copy Paste: [[2503.20797]] "Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection(https://arxiv.org/abs/2503.20797)
• Keywords: language model, llm
• Abstract: The rapid growth of social media platforms has led to concerns about radicalization, filter bubbles, and content bias. Existing approaches to classifying ideology are limited in that they require extensive human effort, the labeling of large datasets, and are not able to adapt to evolving ideological contexts. This paper explores the potential of Large Language Models (LLMs) for classifying the political ideology of online content in the context of the two-party US political spectrum through in-context learning (ICL). Our extensive experiments involving demonstration selection in label-balanced fashion, conducted on three datasets comprising news articles and YouTube videos, reveal that our approach significantly outperforms zero-shot and traditional supervised methods. Additionally, we evaluate the influence of metadata (e.g., content source and descriptions) on ideological classification and discuss its implications. Finally, we show how providing the source for political and non-political content influences the LLM's classification.
• 摘要：社交媒体平台的快速增长导致人们对激进化，过滤气泡和内容偏见的关注。现有的对意识形态进行分类的方法是有限的，因为它们需要大量的人类努力，大型数据集的标签，并且无法适应不断发展的意识形态环境。本文探讨了大语模型（LLMS）通过在美国两方政治范围​​的背景下通过秘密学习（ICL）对在线内容的政治意识形态进行分类的潜力。我们的广泛的实验涉及以标签平衡方式进行演示选择的，该实验在包含新闻文章和YouTube视频的三个数据集上进行，揭示了我们的方法极大地超过了零击和传统监督方法。此外，我们评估了元数据（例如内容来源和描述）对意识形态分类的影响，并讨论其含义。最后，我们展示了如何提供政治和非政治内容的来源影响LLM的分类。

Title: Comprehensive Manuscript Assessment with Text Summarization Using 69707 articles

• Authors: Qichen Sun, Yuxing Lu, Kun Xia, Li Chen, He Sun, Jinzhuo Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20835
• Pdf URL: https://arxiv.org/pdf/2503.20835
• Copy Paste: [[2503.20835]] Comprehensive Manuscript Assessment with Text Summarization Using 69707 articles(https://arxiv.org/abs/2503.20835)
• Keywords: language model
• Abstract: Rapid and efficient assessment of the future impact of research articles is a significant concern for both authors and reviewers. The most common standard for measuring the impact of academic papers is the number of citations. In recent years, numerous efforts have been undertaken to predict citation counts within various citation windows. However, most of these studies focus solely on a specific academic field or require early citation counts for prediction, rendering them impractical for the early-stage evaluation of papers. In this work, we harness Scopus to curate a significantly comprehensive and large-scale dataset of information from 69707 scientific articles sourced from 99 journals spanning multiple disciplines. We propose a deep learning methodology for the impact-based classification tasks, which leverages semantic features extracted from the manuscripts and paper metadata. To summarize the semantic features, such as titles and abstracts, we employ a Transformer-based language model to encode semantic features and design a text fusion layer to capture shared information between titles and abstracts. We specifically focus on the following impact-based prediction tasks using information of scientific manuscripts in pre-publication stage: (1) The impact of journals in which the manuscripts will be published. (2) The future impact of manuscripts themselves. Extensive experiments on our datasets demonstrate the superiority of our proposed model for impact-based prediction tasks. We also demonstrate potentials in generating manuscript's feedback and improvement suggestions.
• 摘要：对研究文章的未来影响的快速有效评估对作者和审稿人来说都是一个重要的关注。衡量学术论文影响的最常见标准是引用的数量。近年来，已经采取了许多努力来预测各种引文窗口中的引文数量。但是，这些研究中的大多数仅着眼于特定的学术领域，或者需要提前引用计数来预测，这使它们在论文的早期评估中不切实际。在这项工作中，我们利用Scopus策划了来自69707个科学文章的大量全面和大规模的信息数据集，这些科学文章来自99个期刊，这些期刊涉及多个学科。我们为基于影响的分类任务提出了一种深度学习方法，该方法利用了从手稿和纸元数据中提取的语义特征。为了总结语义特征，例如标题和摘要，我们采用基于变压器的语言模型来编码语义特征并设计文本融合层，以捕获标题和摘要之间的共享信息。我们专门针对以下基于影响的预测任务，使用出版前阶段的科学手稿信息：（1）发表手稿的期刊的影响。 （2）手稿本身的未来影响。我们数据集上的广泛实验证明了我们提出的模型对基于影响的预测任务的优越性。我们还展示了产生手稿的反馈和改进建议的潜力。

Title: Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models

• Authors: Qing Yao, Kanishka Misra, Leonie Weissweiler, Kyle Mahowald
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20850
• Pdf URL: https://arxiv.org/pdf/2503.20850
• Copy Paste: [[2503.20850]] Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models(https://arxiv.org/abs/2503.20850)
• Keywords: language model
• Abstract: Language models (LMs) tend to show human-like preferences on a number of syntactic phenomena, but the extent to which these are attributable to direct exposure to the phenomena or more general properties of language is unclear. We explore this with the English dative alternation (DO: "gave Y the X" vs. PO: "gave the X to Y"), using a controlled rearing paradigm wherein we iteratively train small LMs on systematically manipulated input. We focus on properties that affect the choice of alternant: length and animacy. Both properties are directly present in datives but also reflect more global tendencies for shorter elements to precede longer ones and animates to precede inanimates. First, by manipulating and ablating datives for these biases in the input, we show that direct evidence of length and animacy matters, but easy-first preferences persist even without such evidence. Then, using LMs trained on systematically perturbed datasets to manipulate global length effects (re-linearizing sentences globally while preserving dependency structure), we find that dative preferences can emerge from indirect evidence. We conclude that LMs' emergent syntactic preferences come from a mix of direct and indirect sources.
• 摘要：语言模型（LMS）倾向于在许多句法现象上表现出类似人类的偏好，但是这些偏好是直接暴露于语言现象或更一般性的语言属性的程度尚不清楚。我们使用受控的饲养范式使用英语的交替探索这一点（DO：“给予Y给X” vs. Po：“将X给Y”），其中我们在系统地操纵输入的情况下进行迭代训练小的LMS。我们专注于影响备用选择的属性：长度和动画。两种属性都直接存在于数据中，但也反映了更短的元素在更长的元素之前和动画以先于无动物之前的动画趋势。首先，通过在输入中操纵和烧毁这些偏见的数据，我们表明了长度和动画事务的直接证据，但即使没有这样的证据，但即使没有这种证据，却持续了一首偏好。然后，使用经过系统扰动数据集训练的LMS来操纵全球长度效应（在全球范围内重新定性句子，同时保留依赖关系结构），我们发现，从间接的证据中可以出现效果。我们得出的结论是，LMS的新兴句法偏好来自直接和间接来源的混合。

Title: Hacia la interpretabilidad de la detección anticipada de riesgos de depresión utilizando grandes modelos de lenguaje

• Authors: Horacio Thompson, Maximiliano Sapino, Edgardo Ferretti, Marcelo Errecalde
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20939
• Pdf URL: https://arxiv.org/pdf/2503.20939
• Copy Paste: [[2503.20939]] Hacia la interpretabilidad de la detección anticipada de riesgos de depresión utilizando grandes modelos de lenguaje(https://arxiv.org/abs/2503.20939)
• Keywords: language model, llm
• Abstract: Early Detection of Risks (EDR) on the Web involves identifying at-risk users as early as possible. Although Large Language Models (LLMs) have proven to solve various linguistic tasks efficiently, assessing their reasoning ability in specific domains is crucial. In this work, we propose a method for solving depression-related EDR using LLMs on Spanish texts, with responses that can be interpreted by humans. We define a reasoning criterion to analyze users through a specialist, apply in-context learning to the Gemini model, and evaluate its performance both quantitatively and qualitatively. The results show that accurate predictions can be obtained, supported by explanatory reasoning, providing a deeper understanding of the solution. Our approach offers new perspectives for addressing EDR problems by leveraging the power of LLMs.
• 摘要：网络上的风险（EDR）的早期检测涉及尽早识别高危用户。尽管大型语言模型（LLM）已被证明可以有效地解决各种语言任务，但评估其在特定领域的推理能力至关重要。在这项工作中，我们提出了一种在西班牙文本上使用LLMS解决与抑郁症相关的EDR的方法，其反应可以由人类解释。我们定义了一个推理标准，可以通过专家分析用户，将其应用于双子座模型，并在定量和质量上评估其性能。结果表明，可以通过解释性推理获得准确的预测，从而更深入地了解解决方案。我们的方法通过利用LLM的力量来解决EDR问题的新观点。

Title: Clean & Clear: Feasibility of Safe LLM Clinical Guidance

• Authors: Julia Ive, Felix Jozsa, Nick Jackson, Paulina Bondaronek, Ciaran Scott Hill, Richard Dobson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20953
• Pdf URL: https://arxiv.org/pdf/2503.20953
• Copy Paste: [[2503.20953]] Clean & Clear: Feasibility of Safe LLM Clinical Guidance(https://arxiv.org/abs/2503.20953)
• Keywords: llm, chat
• Abstract: Background: Clinical guidelines are central to safe evidence-based medicine in modern healthcare, providing diagnostic criteria, treatment options and monitoring advice for a wide range of illnesses. LLM-empowered chatbots have shown great promise in Healthcare Q&A tasks, offering the potential to provide quick and accurate responses to medical inquiries. Our main objective was the development and preliminary assessment of an LLM-empowered chatbot software capable of reliably answering clinical guideline questions using University College London Hospital (UCLH) clinical guidelines. Methods: We used the open-weight Llama-3.1-8B LLM to extract relevant information from the UCLH guidelines to answer questions. Our approach highlights the safety and reliability of referencing information over its interpretation and response generation. Seven doctors from the ward assessed the chatbot's performance by comparing its answers to the gold standard. Results: Our chatbot demonstrates promising performance in terms of relevance, with ~73% of its responses rated as very relevant, showcasing a strong understanding of the clinical context. Importantly, our chatbot achieves a recall of 0.98 for extracted guideline lines, substantially minimising the risk of missing critical information. Approximately 78% of responses were rated satisfactory in terms of completeness. A small portion (~14.5%) contained minor unnecessary information, indicating occasional lapses in precision. The chatbot' showed high efficiency, with an average completion time of 10 seconds, compared to 30 seconds for human respondents. Evaluation of clinical reasoning showed that 72% of the chatbot's responses were without flaws. Our chatbot demonstrates significant potential to speed up and improve the process of accessing locally relevant clinical information for healthcare professionals.
• 摘要：背景：临床指南对于现代医疗保健中的安全循证医学至关重要，提供诊断标准，治疗选择和监测各种疾病的建议。 LLM授权的聊天机器人在医疗保健问答任务中表现出了巨大的希望，从而提供了对医疗查询的快速准确回应的潜力。我们的主要目标是对LLM授权的聊天机器人软件的开发和初步评估，该软件能够使用伦敦大学学院医院（UCLH）临床指南可靠地回答临床指南问题。方法：我们使用开放式Llama-3.1-8B LLM从UCLH指南中提取相关信息以回答问题。我们的方法强调了有关其解释和响应产生的信息的安全性和可靠性。病房的七名医生通过比较其对黄金标准的答案来评估聊天机器人的表现。结果：我们的聊天机器人在相关性方面表现出了有希望的表现，其响应的约73％被评为非常相关，表明对临床环境有深刻的了解。重要的是，我们的聊天机器人的召回机器人召回了0.98的指南，从而最大程度地降低了缺少关键信息的风险。大约78％的响应在完整性方面被评为令人满意。一小部分（〜14.5％）包含少量不必要的信息，表明精确的偶尔失误。聊天机器人的效率很高，平均完成时间为10秒，而人类受访者为30秒。对临床推理的评估表明，聊天机器人的反应中有72％没有缺陷。我们的聊天机器人具有加快和改善访问当地相关临床信息的医疗保健专业人员的巨大潜力。

Title: Sociotechnical Effects of Machine Translation

• Authors: Joss Moorkens, Andy Way, Séamus Lankford
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20959
• Pdf URL: https://arxiv.org/pdf/2503.20959
• Copy Paste: [[2503.20959]] Sociotechnical Effects of Machine Translation(https://arxiv.org/abs/2503.20959)
• Keywords: language model, llm
• Abstract: While the previous chapters have shown how machine translation (MT) can be useful, in this chapter we discuss some of the side-effects and risks that are associated, and how they might be mitigated. With the move to neural MT and approaches using Large Language Models (LLMs), there is an associated impact on climate change, as the models built by multinational corporations are massive. They are hugely expensive to train, consume large amounts of electricity, and output huge volumes of kgCO2 to boot. However, smaller models which still perform to a high level of quality can be built with much lower carbon footprints, and tuning pre-trained models saves on the requirement to train from scratch. We also discuss the possible detrimental effects of MT on translators and other users. The topics of copyright and ownership of data are discussed, as well as ethical considerations on data and MT use. Finally, we show how if done properly, using MT in crisis scenarios can save lives, and we provide a method of how this might be done.
• 摘要：虽然前几章已经显示了机器翻译（MT）的有用方式，但在本章中，我们讨论了一些相关的副作用和风险，以及如何缓解它们。随着使用大语言模型（LLM）转向神经MT和方法，对气候变化产生了相关的影响，因为跨国公司建立的模型巨大。他们的训练，消耗大量电力并输出大量KGCO2启动。但是，可以使用较低的碳足迹来构建仍然具有高质量的较小型号，并根据从头开始训练的要求进行调整预训练的模型。我们还讨论了MT对翻译人员和其他用户的可能有害影响。讨论了版权和数据所有权的主题，以及有关数据和MT使用的道德考虑。最后，我们展示了如何正确地完成危机场景中的MT可以挽救生命，并且我们提供了一种可以做到这一点的方法。

Title: Multi-Modal Framing Analysis of News

• Authors: Arnav Arora, Srishti Yadav, Maria Antoniak, Serge Belongie, Isabelle Augenstein
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.20960
• Pdf URL: https://arxiv.org/pdf/2503.20960
• Copy Paste: [[2503.20960]] Multi-Modal Framing Analysis of News(https://arxiv.org/abs/2503.20960)
• Keywords: language model
• Abstract: Automated frame analysis of political communication is a popular task in computational social science that is used to study how authors select aspects of a topic to frame its reception. So far, such studies have been narrow, in that they use a fixed set of pre-defined frames and focus only on the text, ignoring the visual contexts in which those texts appear. Especially for framing in the news, this leaves out valuable information about editorial choices, which include not just the written article but also accompanying photographs. To overcome such limitations, we present a method for conducting multi-modal, multi-label framing analysis at scale using large (vision-)language models. Grounding our work in framing theory, we extract latent meaning embedded in images used to convey a certain point and contrast that to the text by comparing the respective frames used. We also identify highly partisan framing of topics with issue-specific frame analysis found in prior qualitative work. We demonstrate a method for doing scalable integrative framing analysis of both text and image in news, providing a more complete picture for understanding media bias.
• 摘要：政治交流的自动框架分析是计算社会科学中的一项流行任务，用于研究作者如何选择主题的方面来构建其接收。到目前为止，此类研究一直很狭窄，因为它们使用了固定的预定框架，并且仅专注于文本，而忽略了这些文本出现的视觉上下文。特别是对于新闻中的框架，这遗漏了有关编辑选择的宝贵信息，其中不仅包括书面文章，还包括随附的照片。为了克服此类局限性，我们提出了一种使用大型（视觉）语言模型进行大规模进行多模式的多模式框架分析的方法。我们将框架理论的工作扎根，我们提取了嵌入在用于传达某个点的图像中的潜在含义，并通过比较所使用的各个框架与文本进行对比。我们还通过在先前的定性工作中发现的特定于发行的框架分析来确定主题的高度党派框架。我们展示了一种在新闻中对文本和图像进行可扩展的集成框架分析的方法，为理解媒体偏见提供了更完整的图片。

Title: ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction

• Authors: Yiqiao Jin, Stefano Petrangeli, Yu Shen, Gang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20978
• Pdf URL: https://arxiv.org/pdf/2503.20978
• Copy Paste: [[2503.20978]] ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction(https://arxiv.org/abs/2503.20978)
• Keywords: language model, llm, agent
• Abstract: Graphical User Interface (GUI) agents are autonomous systems that interpret and generate actions, enabling intelligent user assistance and automation. Effective training of these agent presents unique challenges, such as sparsity in supervision signals, scalability for large datasets, and the need for nuanced user understanding. We propose stateful screen schema, an efficient representation of GUI interactions that captures key user actions and intentions over time. Building on this foundation, we introduce ScreenLLM, a set of multimodal large language models (MLLMs) tailored for advanced UI understanding and action prediction. Extensive experiments on both open-source and proprietary models show that ScreenLLM accurately models user behavior and predicts actions. Our work lays the foundation for scalable, robust, and intelligent GUI agents that enhance user interaction in diverse software environments.
• 摘要：图形用户界面（GUI）代理是解释和生成动作的自主系统，从而实现智能用户帮助和自动化。对这些代理的有效培训提出了独特的挑战，例如监督信号的稀疏性，大型数据集的可扩展性以及对用户理解的需求。我们提出了有效的屏幕架构，这是GUI交互的有效表示，随着时间的流逝，捕获了关键的用户操作和意图。在此基础的基础上，我们介绍了ScreenLlm，这是一套为高级UI理解和动作预测而定制的多模式大语言模型（MLLM）。对开源和专有模型的广泛实验表明，ScreenLlm准确地模拟用户行为并预测操作。我们的工作为可扩展，健壮和智能的GUI代理奠定了基础，从而增强了在不同软件环境中的用户互动。

Title: Patients Speak, AI Listens: LLM-based Analysis of Online Reviews Uncovers Key Drivers for Urgent Care Satisfaction

• Authors: Xiaoran Xu, Zhaoqian Xue, Chi Zhang, Jhonatan Medri, Junjie Xiong, Jiayan Zhou, Jin Jin, Yongfeng Zhang, Siyuan Ma, Lingyao Li
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2503.20981
• Pdf URL: https://arxiv.org/pdf/2503.20981
• Copy Paste: [[2503.20981]] Patients Speak, AI Listens: LLM-based Analysis of Online Reviews Uncovers Key Drivers for Urgent Care Satisfaction(https://arxiv.org/abs/2503.20981)
• Keywords: language model, gpt, llm, prompt
• Abstract: Investigating the public experience of urgent care facilities is essential for promoting community healthcare development. Traditional survey methods often fall short due to limited scope, time, and spatial coverage. Crowdsourcing through online reviews or social media offers a valuable approach to gaining such insights. With recent advancements in large language models (LLMs), extracting nuanced perceptions from reviews has become feasible. This study collects Google Maps reviews across the DMV and Florida areas and conducts prompt engineering with the GPT model to analyze the aspect-based sentiment of urgent care. We first analyze the geospatial patterns of various aspects, including interpersonal factors, operational efficiency, technical quality, finances, and facilities. Next, we determine Census Block Group(CBG)-level characteristics underpinning differences in public perception, including population density, median income, GINI Index, rent-to-income ratio, household below poverty rate, no insurance rate, and unemployment rate. Our results show that interpersonal factors and operational efficiency emerge as the strongest determinants of patient satisfaction in urgent care, while technical quality, finances, and facilities show no significant independent effects when adjusted for in multivariate models. Among socioeconomic and demographic factors, only population density demonstrates a significant but modest association with patient ratings, while the remaining factors exhibit no significant correlations. Overall, this study highlights the potential of crowdsourcing to uncover the key factors that matter to residents and provide valuable insights for stakeholders to improve public satisfaction with urgent care.
• 摘要：调查紧急护理设施的公共经验对于促进社区医疗保健发展至关重要。由于范围有限，时间和空间覆盖范围，传统的调查方法通常缺乏。通过在线评论或社交媒体进行众包提供了一种获得此类见解的宝贵方法。随着大语言模型（LLM）的最新进展，从评论中提取细微的看法变得可行。这项研究收集了DMV和佛罗里达州的Google Maps评论，并通过GPT模型进行了及时的工程，以分析基于方面的紧急护理情感。我们首先分析各个方面的地理空间模式，包括人际交往因素，运营效率，技术质量，财务和设施。接下来，我们确定人口普查块组（CBG）级别的特征，包括公众看法差异，包括人口密度，中位收入，GINI指数，租金收入比率，低于贫困率，无保险费率和失业率的家庭。我们的结果表明，人际关系因素和运营效率是紧急护理中患者满意度的最强决定因素，而技术质量，财务和设施在多变量模型中进行调整时没有明显的独立效果。在社会经济和人口统计学因素中，只有人口密度表现出与患者评分的显着但适度的关联，而其余因素没有显着相关性。总体而言，这项研究强调了众包揭示对居民重要的关键因素的潜力，并为利益相关者提供了宝贵的见解，以提高公众对紧急护理的满意。

Title: Multi-head Reward Aggregation Guided by Entropy

• Authors: Xiaomin Li, Xupeng Chen, Jingxuan Fan, Eric Hanchen Jiang, Mingye Gao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20995
• Pdf URL: https://arxiv.org/pdf/2503.20995
• Copy Paste: [[2503.20995]] Multi-head Reward Aggregation Guided by Entropy(https://arxiv.org/abs/2503.20995)
• Keywords: language model, llm, prompt
• Abstract: Aligning large language models (LLMs) with safety guidelines typically involves reinforcement learning from human feedback (RLHF), relying on human-generated preference annotations. However, assigning consistent overall quality ratings is challenging, prompting recent research to shift towards detailed evaluations based on multiple specific safety criteria. This paper uncovers a consistent observation: safety rules characterized by high rating entropy are generally less reliable in identifying responses preferred by humans. Leveraging this finding, we introduce ENCORE, a straightforward entropy-guided approach that composes multi-head rewards by downweighting rules exhibiting high rating entropy. Theoretically, we demonstrate that rules with elevated entropy naturally receive minimal weighting in the Bradley-Terry optimization framework, justifying our entropy-based penalization. Through extensive experiments on RewardBench safety tasks, our method significantly surpasses several competitive baselines, including random weighting, uniform weighting, single-head Bradley-Terry models, and LLM-based judging methods. Our proposed approach is training-free, broadly applicable to various datasets, and maintains interpretability, offering a practical and effective solution for multi-attribute reward modeling.
• 摘要：将大型语言模型（LLM）与安全指南保持一致，通常涉及从人类反馈（RLHF）学习，依靠人类生成的偏好注释。但是，分配一致的总体质量评级是具有挑战性的，促使最近的研究基于多个特定的安全标准转向详细评估。本文揭示了一个一致的观察：以高评分熵为特征的安全规则通常在识别人类首选的反应方面的可靠性较差。利用这一发现，我们介绍了Encore，一种直接的熵引导的方法，该方法通过表现出高评分熵的下列规则来构成多头奖励。从理论上讲，我们证明，熵升高的规则自然会在布拉德利 - 特里·优化框架中获得最小的权重，从而证明了我们基于熵的惩罚是合理的。通过对奖励台式安全任务的广泛实验，我们的方法显着超过了几个竞争基线，包括随机加权，统一的加权，单头Bradley-Terry模型和基于LLM的判断方法。我们提出的方法是无培训的，广泛适用于各种数据集，并保持可解释性，为多属性奖励建模提供了实用有效的解决方案。

Title: Evaluating Large Language Models for Automated Clinical Abstraction in Pulmonary Embolism Registries: Performance Across Model Sizes, Versions, and Parameters

• Authors: Mahmoud Alwakeel, Emory Buck, Jonathan G. Martin, Imran Aslam, Sudarshan Rajagopal, Jian Pei, Mihai V. Podgoreanu, Christopher J. Lindsell, An-Kwok Ian Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21004
• Pdf URL: https://arxiv.org/pdf/2503.21004
• Copy Paste: [[2503.21004]] Evaluating Large Language Models for Automated Clinical Abstraction in Pulmonary Embolism Registries: Performance Across Model Sizes, Versions, and Parameters(https://arxiv.org/abs/2503.21004)
• Keywords: language model, llm
• Abstract: Pulmonary embolism (PE) is a leading cause of cardiovascular mortality, yet our understanding of optimal management remains limited due to heterogeneous and inaccessible radiology documentation. The PERT Consortium registry standardizes PE management data but depends on resource-intensive manual abstraction. Large language models (LLMs) offer a scalable alternative for automating concept extraction from computed tomography PE (CTPE) reports. This study evaluated the accuracy of LLMs in extracting PE-related concepts compared to a human-curated criterion standard. We retrospectively analyzed MIMIC-IV and Duke Health CTPE reports using multiple LLaMA models. Larger models (70B) outperformed smaller ones (8B), achieving kappa values of 0.98 (PE detection), 0.65-0.75 (PE location), 0.48-0.51 (right heart strain), and 0.65-0.70 (image artifacts). Moderate temperature tuning (0.2-0.5) improved accuracy, while excessive in-context examples reduced performance. A dual-model review framework achieved >80-90% precision. LLMs demonstrate strong potential for automating PE registry abstraction, minimizing manual workload while preserving accuracy.
• 摘要：肺栓塞（PE）是心血管死亡率的主要原因，但是由于异质和无法访问的放射学文档，我们对最佳管理的理解仍然有限。 PERT联盟注册表标准化PE管理数据，但取决于资源密集型手动抽象。大型语言模型（LLMS）为从计算机断层扫描PE（CTPE）报告中自动提取概念提供了可扩展的替代方案。这项研究评估了LLM与人类策略标准相比提取PE相关概念的准确性。我们回顾性地分析了使用多个Llama模型分析模仿和Duke Health CTPE报告。较大的型号（70B）的表现优于较小的模型（8b），可达到0.98（PE检测），0.65-0.75（PE位置），0.48-0.51（右心脏应变）和0.65-0.70（图像工件）。中等温度调整（0.2-0.5）提高了精度，而过多的内在示例降低了性能。双模型评论框架的精度> 80-90％。 LLMS表现出强大的PE注册表抽象的潜力，在保持精度的同时最大程度地减少了手动工作量。

Title: Can Large Language Models Predict Associations Among Human Attitudes?

• Authors: Ana Ma, Derek Powell
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21011
• Pdf URL: https://arxiv.org/pdf/2503.21011
• Copy Paste: [[2503.21011]] Can Large Language Models Predict Associations Among Human Attitudes?(https://arxiv.org/abs/2503.21011)
• Keywords: language model, gpt, llm
• Abstract: Prior work has shown that large language models (LLMs) can predict human attitudes based on other attitudes, but this work has largely focused on predictions from highly similar and interrelated attitudes. In contrast, human attitudes are often strongly associated even across disparate and dissimilar topics. Using a novel dataset of human responses toward diverse attitude statements, we found that a frontier language model (GPT-4o) was able to recreate the pairwise correlations among individual attitudes and to predict individuals' attitudes from one another. Crucially, in an advance over prior work, we tested GPT-4o's ability to predict in the absence of surface-similarity between attitudes, finding that while surface similarity improves prediction accuracy, the model was still highly-capable of generating meaningful social inferences between dissimilar attitudes. Altogether, our findings indicate that LLMs capture crucial aspects of the deeper, latent structure of human belief systems.
• 摘要：先前的工作表明，大型语言模型（LLM）可以根据其他态度来预测人类的态度，但是这项工作主要集中在高度相似和相互关联的态度的预测上。相比之下，即使在不同的和不同的话题中，人类的态度也常常是密切相关的。我们使用人类对各种态度陈述的反应的新数据集，我们发现边境语言模型（GPT-4O）能够重现个人态度之间的成对相关性，并彼此预测个人的态度。至关重要的是，在先前的工作中，我们测试了GPT-4O在态度之间没有表面相似性的情况下预测的能力，发现尽管表面相似性提高了预测准确性，但该模型仍然可以高度支持产生相似态度之间有意义的社会推论。总之，我们的发现表明，LLM捕获了人类信仰体系的更深层，潜在结构的关键方面。

Title: Shared Global and Local Geometry of Language Model Embeddings

• Authors: Andrew Lee, Melanie Weber, Fernanda Viégas, Martin Wattenberg
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21073
• Pdf URL: https://arxiv.org/pdf/2503.21073
• Copy Paste: [[2503.21073]] Shared Global and Local Geometry of Language Model Embeddings(https://arxiv.org/abs/2503.21073)
• Keywords: language model
• Abstract: Researchers have recently suggested that models share common representations. In this work, we find that the token embeddings of language models exhibit common geometric structure. First, we find ``global'' similarities: token embeddings often share similar relative orientations. Next, we characterize local geometry in two ways: (1) by using Locally Linear Embeddings, and (2) by defining a simple measure for the intrinsic dimension of each token embedding. Our intrinsic dimension measure demonstrates that token embeddings lie on a lower dimensional manifold. We qualitatively show that tokens with lower intrinsic dimensions often have semantically coherent clusters, while those with higher intrinsic dimensions do not. Both characterizations allow us to find similarities in the local geometry of token embeddings. Perhaps most surprisingly, we find that alignment in token embeddings persists through the hidden states of language models, allowing us to develop an application for interpretability. Namely, we empirically demonstrate that steering vectors from one language model can be transferred to another, despite the two models having different dimensions.
• 摘要：研究人员最近建议模型共同表示。在这项工作中，我们发现语言模型的令牌嵌入表现出常见的几何结构。首先，我们发现``全局''相似性：令牌嵌入通常具有相似的相对取向。接下来，我们通过两种方式表征局部几何形状：（1）使用局部线性嵌入，（2）通过为每个令牌嵌入的固有维度定义一个简单的度量。我们的固有维度表明，令牌嵌入在较低的维歧管上。我们从定性地表明，具有较低固有维度的令牌通常具有语义上的连贯簇，而固有尺寸较高的令牌则没有。这两种特征都使我们能够在令牌嵌入的局部几何形状中找到相似之处。也许最令人惊讶的是，我们发现令牌嵌入中的一致性通过语言模型的隐藏状态持续存在，从而使我们能够为可解释性开发应用程序。也就是说，我们从经验上证明，尽管两个模型具有不同的维度，但仍可以将转向向量转移到另一种语言模型。

Title: EQ-Negotiator: An Emotion-Reasoning LLM Agent in Credit Dialogues

• Authors: Yuhan Liu, Yunbo Long
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21080
• Pdf URL: https://arxiv.org/pdf/2503.21080
• Copy Paste: [[2503.21080]] EQ-Negotiator: An Emotion-Reasoning LLM Agent in Credit Dialogues(https://arxiv.org/abs/2503.21080)
• Keywords: language model, llm, chat, agent
• Abstract: While large language model (LLM)-based chatbots have been applied for effective engagement in credit dialogues, their capacity for dynamic emotional expression remains limited. Current agents primarily rely on passive empathy rather than affective reasoning. For instance, when faced with persistent client negativity, the agent should employ strategic emotional adaptation by expressing measured anger to discourage counterproductive behavior and guide the conversation toward resolution. This context-aware emotional modulation is essential for imitating the nuanced decision-making of human negotiators. This paper introduces an EQ-negotiator that combines emotion sensing from pre-trained language models (PLMs) with emotional reasoning based on Game Theory and Hidden Markov Models. It takes into account both the current and historical emotions of the client to better manage and address negative emotions during interactions. By fine-tuning pre-trained language models (PLMs) on public emotion datasets and validating them on the credit dialogue datasets, our approach enables LLM-based agents to effectively capture shifts in client emotions and dynamically adjust their response tone based on our emotion decision policies in real-world financial negotiations. This EQ-negotiator can also help credit agencies foster positive client relationships, enhancing satisfaction in credit services.
• 摘要：尽管已经应用了大型语言模型（LLM）的聊天机器人进行有效参与信用对话，但它们的动态情感表达能力仍然有限。当前的代理主要依靠被动移情而不是情感推理。例如，当面对持续的客户消极情绪时，代理人应通过表达愤怒来阻止适得其反的行为并指导对话解决方案，以采用战略情感适应。这种感知的情感调制对于模仿人类谈判者的细微损害决策至关重要。本文介绍了一个EQ-nogotiator，该EQ-线族可以将预先训练的语言模型（PLM）与基于游戏理论和隐藏马尔可夫模型的情感推理结合在一起。它考虑了客户的当前和历史情绪，以更好地管理和解决互动过程中的负面情绪。通过对公共情绪数据集进行预训练的预训练语言模型（PLM）并在信用对话数据集中验证它们，我们的方法使基于LLM的代理能够有效捕获客户情绪的转变，并基于我们的情感决策政策在现实世界中财务谈判中动态调整其响应态度。该EQ-Connotiator还可以帮助信贷机构建立积极的客户关系，从而提高信贷服务的满意度。

Title: ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging

• Authors: Haoming Xu, Shuxun Wang, Yanqiu Zhao, Yi Zhong, Ziyan Jiang, Ningyuan Zhao, Shumin Deng, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2503.21088
• Pdf URL: https://arxiv.org/pdf/2503.21088
• Copy Paste: [[2503.21088]] ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging(https://arxiv.org/abs/2503.21088)
• Keywords: language model
• Abstract: This paper presents the ZJUKLAB team's submission for SemEval-2025 Task 4: Unlearning Sensitive Content from Large Language Models. This task aims to selectively erase sensitive knowledge from large language models, avoiding both over-forgetting and under-forgetting issues. We propose an unlearning system that leverages Model Merging (specifically TIES-Merging), combining two specialized models into a more balanced unlearned model. Our system achieves competitive results, ranking second among 26 teams, with an online score of 0.944 for Task Aggregate and 0.487 for overall Aggregate. In this paper, we also conduct local experiments and perform a comprehensive analysis of the unlearning process, examining performance trajectories, loss dynamics, and weight perspectives, along with several supplementary experiments, to understand the effectiveness of our method. Furthermore, we analyze the shortcomings of our method and evaluation metrics, emphasizing that MIA scores and ROUGE-based metrics alone are insufficient to fully evaluate successful unlearning. Finally, we emphasize the need for more comprehensive evaluation methodologies and rethinking of unlearning objectives in future research. Code is available at this https URL.
• 摘要：本文介绍了Zjuklab团队的Semeval-2025任务4：来自大型语言模型的敏感内容。该任务旨在从大型语言模型中选择性地删除敏感知识，避免过度遗忘和遗忘问题。我们提出了一个利用模型合并（特别是领带合并）的学习系统，将两个专用模型组合到更平衡的未学习模型中。我们的系统取得了竞争成果，在26支球队中排名第二，任务汇总的在线得分为0.944，整体汇总分数为0.487。在本文中，我们还进行了本地实验，并对未学习过程进行了全面的分析，研究了性能轨迹，损失动态和体重观点以及几个补充实验，以了解我们方法的有效性。此外，我们分析了我们的方法和评估指标的缺点，并强调MIA分数和基于胭脂的指标不足以完全评估成功的未学习。最后，我们强调需要更全面的评估方法，并重新思考未来研究中未学习的目标。代码可在此HTTPS URL上找到。

Title: Leveraging Large Language Models for Risk Assessment in Hyperconnected Logistic Hub Network Deployment

• Authors: Yinzhu Quan, Yujia Xu, Guanlin Chen, Frederick Benaben, Benoit Montreuil
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21115
• Pdf URL: https://arxiv.org/pdf/2503.21115
• Copy Paste: [[2503.21115]] Leveraging Large Language Models for Risk Assessment in Hyperconnected Logistic Hub Network Deployment(https://arxiv.org/abs/2503.21115)
• Keywords: language model, llm, prompt
• Abstract: The growing emphasis on energy efficiency and environmental sustainability in global supply chains introduces new challenges in the deployment of hyperconnected logistic hub networks. In current volatile, uncertain, complex, and ambiguous (VUCA) environments, dynamic risk assessment becomes essential to ensure successful hub deployment. However, traditional methods often struggle to effectively capture and analyze unstructured information. In this paper, we design an Large Language Model (LLM)-driven risk assessment pipeline integrated with multiple analytical tools to evaluate logistic hub deployment. This framework enables LLMs to systematically identify potential risks by analyzing unstructured data, such as geopolitical instability, financial trends, historical storm events, traffic conditions, and emerging risks from news sources. These data are processed through a suite of analytical tools, which are automatically called by LLMs to support a structured and data-driven decision-making process for logistic hub selection. In addition, we design prompts that instruct LLMs to leverage these tools for assessing the feasibility of hub selection by evaluating various risk types and levels. Through risk-based similarity analysis, LLMs cluster logistic hubs with comparable risk profiles, enabling a structured approach to risk assessment. In conclusion, the framework incorporates scalability with long-term memory and enhances decision-making through explanation and interpretation, enabling comprehensive risk assessments for logistic hub deployment in hyperconnected supply chain networks.
• 摘要：对全球供应链中能源效率和环境可持续性的越来越强调引入了部署超连接的逻辑集线器网络的新挑战。在当前的挥发性，不确定，复杂和模棱两可的环境中，动态风险评估对于确保成功的中心部署至关重要。但是，传统方法通常难以有效捕获和分析非结构化信息。在本文中，我们设计了一个与多种分析工具集成的大型语言模型（LLM）驱动的风险评估管道，以评估Logistic Hub部署。该框架使LLMS能够通过分析非结构化数据，例如地缘政治不稳定，财务趋势，历史风暴事件，交通状况以及新闻来源的新兴风险来系统地识别潜在风险。这些数据是通过一套分析工具来处理的，LLM会自动调用这些数据，以支持逻辑集线器选择的结构化和数据驱动的决策过程。此外，我们设计提示LLMS通过评估各种风险类型和水平来利用这些工具来评估HUB选择的可行性。通过基于风险的相似性分析，LLMS群集逻辑中心具有可比的风险概况，从而实现了一种结构化的风险评估方法。总之，该框架将可伸缩性与长期记忆结合在一起，并通过解释和解释增强决策，从而为超连接供应链网络中的Logistic Hub部署提供了全面的风险评估。

Title: Collaborative Evolution: Multi-Round Learning Between Large and Small Language Models for Emergent Fake News Detection

• Authors: Ziyi Zhou, Xiaoming Zhang, Shenghan Tan, Litian Zhang, Chaozhuo Li
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2503.21127
• Pdf URL: https://arxiv.org/pdf/2503.21127
• Copy Paste: [[2503.21127]] Collaborative Evolution: Multi-Round Learning Between Large and Small Language Models for Emergent Fake News Detection(https://arxiv.org/abs/2503.21127)
• Keywords: language model, llm
• Abstract: The proliferation of fake news on social media platforms has exerted a substantial influence on society, leading to discernible impacts and deleterious consequences. Conventional deep learning methodologies employing small language models (SLMs) suffer from the necessity for extensive supervised training and the challenge of adapting to rapidly evolving circumstances. Large language models (LLMs), despite their robust zero-shot capabilities, have fallen short in effectively identifying fake news due to a lack of pertinent demonstrations and the dynamic nature of knowledge. In this paper, a novel framework Multi-Round Collaboration Detection (MRCD) is proposed to address these aforementioned limitations. The MRCD framework is capable of enjoying the merits from both LLMs and SLMs by integrating their generalization abilities and specialized functionalities, respectively. Our approach features a two-stage retrieval module that selects relevant and up-to-date demonstrations and knowledge, enhancing in-context learning for better detection of emerging news events. We further design a multi-round learning framework to ensure more reliable detection results. Our framework MRCD achieves SOTA results on two real-world datasets Pheme and Twitter16, with accuracy improvements of 7.4\% and 12.8\% compared to using only SLMs, which effectively addresses the limitations of current models and improves the detection of emergent fake news.
• 摘要：在社交媒体平台上的假新闻的扩散对社会产生了重大影响，从而产生了明显的影响和有害后果。采用小语言模型（SLM）的常规深度学习方法遭受了进行广泛监督培训的必要性以及适应快速发展的环境的挑战。大型语言模型（LLMS）尽管具有强大的零射击功能，但由于缺乏相关的示范和知识的动态性质，在有效地识别假新闻方面却缺乏。在本文中，提出了一个新型的框架多创协作检测（MRCD）来解决这些上述局限性。 MRCD框架能够分别整合其概括能力和专业功能，从而享受LLM和SLM的优点。我们的方法具有两个阶段的检索模块，该模块选择相关和最新的演示和知识，从而增强了内在学习，以更好地检测新闻新闻事件。我们进一步设计了一个多轮学习框架，以确保更可靠的检测结果。与仅使用SLM相比，我们的框架MRCD在两个现实世界数据集和Twitter16上实现了SOTA结果，精度提高了7.4 \％和12.8 \％，这有效地解决了当前模型的局限性并改善了出现的假新闻的检测。

Title: LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models

• Authors: Hengyuan Zhao, Ziqin Wang, Qixin Sun, Kaiyou Song, Yilin Li, Xiaolin Hu, Qingpei Guo, Si Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21227
• Pdf URL: https://arxiv.org/pdf/2503.21227
• Copy Paste: [[2503.21227]] LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models(https://arxiv.org/abs/2503.21227)
• Keywords: language model
• Abstract: Although applying Mixture of Experts to large language models for learning new tasks is widely regarded as an effective strategy for continuous learning, there still remain two major challenges: (1) As the number of tasks grows, simple parameter expansion strategies can lead to excessively large models. (2) Modifying the parameters of the existing router results in the erosion of previously acquired knowledge. In this paper, we present an innovative framework named LLaVA-CMoE, which is a continuous Mixture of Experts (MoE) architecture without any replay data. Specifically, we have developed a method called Probe-Guided Knowledge Extension (PGKE), which employs probe experts to assess whether additional knowledge is required for a specific layer. This approach enables the model to adaptively expand its network parameters based on task distribution, thereby significantly improving the efficiency of parameter expansion. Additionally, we introduce a hierarchical routing algorithm called Probabilistic Task Locator (PTL), where high-level routing captures inter-task information and low-level routing focuses on intra-task details, ensuring that new task experts do not interfere with existing ones. Our experiments shows that our efficient architecture has substantially improved model performance on the Coin benchmark while maintaining a reasonable parameter count.
• 摘要：尽管将专家的混合物应用于大型语言模型来学习新任务被广泛认为是连续学习的有效策略，但仍然存在两个主要挑战：（1）随着任务数量的增加，简单的参数扩展策略可能会导致过多的模型。 （2）修改现有路由器的参数会导致先前获得的知识的侵蚀。在本文中，我们提出了一个名为llava-cmoe的创新框架，该框架是专家（MOE）体系结构的连续混合物，而没有任何重播数据。具体而言，我们开发了一种称为探针引导的知识扩展（PGKE）的方法，该方法采用探测专家来评估特定层是否需要其他知识。这种方法使模型能够根据任务分布自适应扩展其网络参数，从而显着提高参数扩展的效率。此外，我们引入了一种称为Probabilistic Task定位器（PTL）的层次路由算法，其中高级路由捕获任务间信息和低级路由集中在任务内部的详细信息上，以确保新任务专家不会干扰现有的任务。我们的实验表明，我们的有效体系结构在保持合理的参数计数的同时，在硬币基准上的模型性能大大提高了。

Title: ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

• Authors: Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2503.21248
• Pdf URL: https://arxiv.org/pdf/2503.21248
• Copy Paste: [[2503.21248]] ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition(https://arxiv.org/abs/2503.21248)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. To address this gap, we introduce the first large-scale benchmark for evaluating LLMs with a near-sufficient set of sub-tasks of scientific discovery: inspiration retrieval, hypothesis composition, and hypothesis ranking. We develop an automated framework that extracts critical components - research questions, background surveys, inspirations, and hypotheses - from scientific papers across 12 disciplines, with expert validation confirming its accuracy. To prevent data contamination, we focus exclusively on papers published in 2024, ensuring minimal overlap with LLM pretraining data. Our evaluation reveals that LLMs perform well in retrieving inspirations, an out-of-distribution task, suggesting their ability to surface novel knowledge associations. This positions LLMs as "research hypothesis mines", capable of facilitating automated scientific discovery by generating innovative hypotheses at scale with minimal human intervention.
• 摘要：大型语言模型（LLM）表现出了协助科学研究的潜力，但由于缺乏专用的基准，他们发现高质量研究假设的能力仍然没有进行检查。为了解决这一差距，我们介绍了第一个大规模基准，用于评估LLM的科学发现子近任务集：灵感检索，假设组成和假设排名。我们开发了一个自动化的框架，该框架从12个学科的科学论文中提取了关键组件，研究问题，背景调查，灵感和假设，并具有专家验证确认其准确性。为了防止数据污染，我们专注于2024年发表的论文，以确保与LLM预处理数据的重叠最小。我们的评估表明，LLM在检索灵感，一项分布的任务方面表现出色，这表明他们表现出新颖的知识关联的能力。这将LLMS定位为“研究假设矿山”，能够通过最少的人类干预来大规模产生创新的假设来促进自动化的科学发现。

Title: Cultivating Game Sense for Yourself: Making VLMs Gaming Experts

• Authors: Wenxuan Lu, Jiangyang He, Zhanqiu Zhang, Yiwen Guo, Tianning Zang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21263
• Pdf URL: https://arxiv.org/pdf/2503.21263
• Copy Paste: [[2503.21263]] Cultivating Game Sense for Yourself: Making VLMs Gaming Experts(https://arxiv.org/abs/2503.21263)
• Keywords: language model, agent
• Abstract: Developing agents capable of fluid gameplay in first/third-person games without API access remains a critical challenge in Artificial General Intelligence (AGI). Recent efforts leverage Vision Language Models (VLMs) as direct controllers, frequently pausing the game to analyze screens and plan action through language reasoning. However, this inefficient paradigm fundamentally restricts agents to basic and non-fluent interactions: relying on isolated VLM reasoning for each action makes it impossible to handle tasks requiring high reactivity (e.g., FPS shooting) or dynamic adaptability (e.g., ACT combat). To handle this, we propose a paradigm shift in gameplay agent design: instead of directly controlling gameplay, VLM develops specialized execution modules tailored for tasks like shooting and combat. These modules handle real-time game interactions, elevating VLM to a high-level developer. Building upon this paradigm, we introduce GameSense, a gameplay agent framework where VLM develops task-specific game sense modules by observing task execution and leveraging vision tools and neural network training pipelines. These modules encapsulate action-feedback logic, ranging from direct action rules to neural network-based decisions. Experiments demonstrate that our framework is the first to achieve fluent gameplay in diverse genres, including ACT, FPS, and Flappy Bird, setting a new benchmark for game-playing agents.
• 摘要：在没有API访问的第一/第三人称游戏中，开发能够流体游戏玩法的代理仍然是人工通用情报（AGI）的关键挑战。最近的努力利用视觉语言模型（VLM）作为直接控制器，经常暂停游戏来分析屏幕并通过语言推理计划行动。但是，这种效率低下的范式从根本上限制了基本的基本和非浮力相互作用：依靠孤立的VLM推理来使每个动作都无法处理需要高反应性（例如FPS射击）或动态适应性的任务（例如，ACT战斗）。为了解决这个问题，我们提出了游戏节目设计的范式变化：而不是直接控制游戏玩法，而是开发了针对诸如射击和战斗之类的任务量身定制的专业执行模块。这些模块处理实时游戏互动，将VLM提升到高级开发人员。在此范式的基础上，我们介绍了Gamesense，这是一个游戏代理框架，VLM通过观察任务执行并利用视觉工具和神经网络培训管道来开发特定于任务的游戏感模块。这些模块封装了动作反馈逻辑，从直接动作规则到基于神经网络的决策。实验表明，我们的框架是第一个以各种流派（包括ACT，FPS和Flappy Bird）进行流利的游戏玩法的框架，为游戏玩法代理设定了新的基准。

Title: R-PRM: Reasoning-Driven Process Reward Modeling

• Authors: Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, Shujian Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21295
• Pdf URL: https://arxiv.org/pdf/2503.21295
• Copy Paste: [[2503.21295]] R-PRM: Reasoning-Driven Process Reward Modeling(https://arxiv.org/abs/2503.21295)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) inevitably make mistakes when performing step-by-step mathematical reasoning. Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. However, existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy, which is further exacerbated by the scarcity of annotated data. To address these issues, we propose Reasoning-Driven Process Reward Modeling (R-PRM). First, we leverage stronger LLMs to generate seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities and enabling comprehensive step-by-step evaluation. Second, we further enhance performance through preference optimization, without requiring additional annotated data. Third, we introduce inference-time scaling to fully harness the model's reasoning potential. Extensive experiments demonstrate R-PRM's effectiveness: on ProcessBench and PRMBench, it surpasses strong baselines by 11.9 and 8.5 points in F1 scores, respectively. When applied to guide mathematical reasoning, R-PRM achieves consistent accuracy improvements of over 8.5 points across six challenging datasets. Further analysis reveals that R-PRM exhibits more comprehensive evaluation and stronger generalization capabilities, thereby highlighting its significant potential.
• 摘要：大型语言模型（LLMS）在执行分步数学推理时不可避免地会犯错。通过评估每个推理步骤，流程奖励模型（PRM）已成为有前途的解决方案。但是，现有的PRM通常直接输出评估得分，从而限制了学习效率和评估准确性，这进一步加剧了注释数据的稀缺性。为了解决这些问题，我们建议以推理为导向的过程奖励建模（R-PRM）。首先，我们利用更强大的LLM从有限的注释中生成种子数据，有效地引导我们的模型的推理功能并实现全面的分步评估。其次，我们通过偏好优化进一步提高了性能，而无需其他带注释的数据。第三，我们介绍推理时间缩放，以充分利用模型的推理潜力。广泛的实验证明了R-PRM的有效性：在ProcessBench和Prmbench上，它在F1分数中分别超过了强大的基准，分别超过11.9和8.5分。当应用于指导数学推理时，R-PRM可在六个具有挑战性的数据集中实现超过8.5分的一致精度提高。进一步的分析表明，R-PRM表现出更全面的评估和更强的概括能力，从而强调了其巨大潜力。

Title: Fine-Tuning LLMs on Small Medical Datasets: Text Classification and Normalization Effectiveness on Cardiology reports and Discharge records

• Authors: Noah Losch, Lucas Plagwitz, Antonius Büscher, Julian Varghese
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21349
• Pdf URL: https://arxiv.org/pdf/2503.21349
• Copy Paste: [[2503.21349]] Fine-Tuning LLMs on Small Medical Datasets: Text Classification and Normalization Effectiveness on Cardiology reports and Discharge records(https://arxiv.org/abs/2503.21349)
• Keywords: language model, llm
• Abstract: We investigate the effectiveness of fine-tuning large language models (LLMs) on small medical datasets for text classification and named entity recognition tasks. Using a German cardiology report dataset and the i2b2 Smoking Challenge dataset, we demonstrate that fine-tuning small LLMs locally on limited training data can improve performance achieving comparable results to larger models. Our experiments show that fine-tuning improves performance on both tasks, with notable gains observed with as few as 200-300 training examples. Overall, the study highlights the potential of task-specific fine-tuning of LLMs for automating clinical workflows and efficiently extracting structured data from unstructured medical text.
• 摘要：我们研究了小型医学数据集上微调大语言模型（LLM）的有效性，以进行文本分类和命名实体识别任务。使用德国心脏病学报告数据集和I2B2吸烟挑战数据集，我们证明，在有限培训数据上进行微调小型LLM可以提高性能，从而实现与较大模型的可比结果。我们的实验表明，微调可以提高这两个任务的性能，而观察到的著名得益少于200-300个培训示例。总体而言，该研究强调了LLMS特定于任务特定的微调来自动化临床工作流程的潜力，并从非结构化的医学文本中有效提取结构化数据。

Title: From User Preferences to Optimization Constraints Using Large Language Models

• Authors: Manuela Sanguinetti, Alessandra Perniciano, Luca Zedda, Andrea Loddo, Cecilia Di Ruberto, Maurizio Atzori
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21360
• Pdf URL: https://arxiv.org/pdf/2503.21360
• Copy Paste: [[2503.21360]] From User Preferences to Optimization Constraints Using Large Language Models(https://arxiv.org/abs/2503.21360)
• Keywords: language model, llm
• Abstract: This work explores using Large Language Models (LLMs) to translate user preferences into energy optimization constraints for home appliances. We describe a task where natural language user utterances are converted into formal constraints for smart appliances, within the broader context of a renewable energy community (REC) and in the Italian scenario. We evaluate the effectiveness of various LLMs currently available for Italian in translating these preferences resorting to classical zero-shot, one-shot, and few-shot learning settings, using a pilot dataset of Italian user requests paired with corresponding formal constraint representation. Our contributions include establishing a baseline performance for this task, publicly releasing the dataset and code for further research, and providing insights on observed best practices and limitations of LLMs in this particular domain
• 摘要：这项工作使用大型语言模型（LLM）探讨将用户偏好转化为家庭电器的能源优化约束。我们描述了一项任务，在可再生能源社区（REC）和意大利方案中，自然语言用户话语被转换为智能电器的正式约束。我们使用意大利用户请求的试验数据集与相应的正式约束表示形式配对，评估了当前可用于意大利语将这些偏好转换为经典零拍，单杆和少量学习设置的各种LLM的有效性。我们的贡献包括为该任务建立基线绩效，公开发布数据集和代码以进行进一步研究，并提供有关该特定领域中LLM的最佳实践和局限性的见解。

Title: Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models

• Authors: Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21380
• Pdf URL: https://arxiv.org/pdf/2503.21380
• Copy Paste: [[2503.21380]] Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models(https://arxiv.org/abs/2503.21380)
• Keywords: language model, llm
• Abstract: In recent years, the rapid development of large reasoning models has resulted in the saturation of existing benchmarks for evaluating mathematical reasoning, highlighting the urgent need for more challenging and rigorous evaluation frameworks. To address this gap, we introduce OlymMATH, a novel Olympiad-level mathematical benchmark, designed to rigorously test the complex reasoning capabilities of LLMs. OlymMATH features 200 meticulously curated problems, each manually verified and available in parallel English and Chinese versions. The problems are systematically organized into two distinct difficulty tiers: (1) AIME-level problems (easy) that establish a baseline for mathematical reasoning assessment, and (2) significantly more challenging problems (hard) designed to push the boundaries of current state-of-the-art models. In our benchmark, these problems span four core mathematical fields, each including a verifiable numerical solution to enable objective, rule-based evaluation. Empirical results underscore the significant challenge presented by OlymMATH, with state-of-the-art models including DeepSeek-R1 and OpenAI's o3-mini demonstrating notably limited accuracy on the hard subset. Furthermore, the benchmark facilitates comprehensive bilingual assessment of mathematical reasoning abilities-a critical dimension that remains largely unaddressed in mainstream mathematical reasoning benchmarks. We release the OlymMATH benchmark at the STILL project: this https URL.
• 摘要：近年来，大型推理模型的快速发展导致现有基准饱和，以评估数学推理，从而强调了迫切需要更具挑战性和严格的评估框架。为了解决这一差距，我们介绍了奥林马斯（Olymmath），这是一种新型的奥林匹克级数学基准，旨在严格测试LLMS的复杂推理能力。奥运会具有200个精心策划的问题，每个问题都经过手动验证，并提供平行的英语和中文版本。这些问题被系统地组织为两个不同的困难层：（1）AIME级问题（简单）为数学推理评估建立基线，以及（2）旨在突破最新现有模型界限的更具挑战性的问题（硬）。在我们的基准测试中，这些问题跨越了四个核心数学字段，每个字段包括可验证的数值解决方案，以实现基于规则的评估。经验结果强调了奥运会带来的重大挑战，包括DeepSeek-R1和OpenAI的O3-Mini在内的最先进的模型表明，在硬式子集上的准确性显着有限。此外，基准有助于对数学推理能力的全面双语评估 - 在主流数学推理基准基准中仍然在很大程度上未受压制的临界维度。我们在静止项目中发布了奥运会基准：此HTTPS URL。

Title: Controlling Large Language Model with Latent Actions

• Authors: Chengxing Jia, Ziniu Li, Pengyuan Wang, Yi-Chen Li, Zhenyu Hou, Yuxiao Dong, Yang Yu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21383
• Pdf URL: https://arxiv.org/pdf/2503.21383
• Copy Paste: [[2503.21383]] Controlling Large Language Model with Latent Actions(https://arxiv.org/abs/2503.21383)
• Keywords: language model, llm, prompt, agent
• Abstract: Adapting Large Language Models (LLMs) to downstream tasks using Reinforcement Learning (RL) has proven to be an effective approach. However, LLMs do not inherently define the structure of an agent for RL training, particularly in terms of defining the action space. This paper studies learning a compact latent action space to enhance the controllability and exploration of RL for LLMs. We propose Controlling Large Language Models with Latent Actions (CoLA), a framework that integrates a latent action space into pre-trained LLMs. We apply CoLA to the Llama-3.1-8B model. Our experiments demonstrate that, compared to RL with token-level actions, CoLA's latent action enables greater semantic diversity in text generation. For enhancing downstream tasks, we show that CoLA with RL achieves a score of 42.4 on the math500 benchmark, surpassing the baseline score of 38.2, and reaches 68.2 when augmented with a Monte Carlo Tree Search variant. Furthermore, CoLA with RL consistently improves performance on agent-based tasks without degrading the pre-trained LLM's capabilities, unlike the baseline. Finally, CoLA reduces computation time by half in tasks involving enhanced thinking prompts for LLMs by RL. These results highlight CoLA's potential to advance RL-based adaptation of LLMs for downstream applications.
• 摘要：使用强化学习（RL）调整大型语言模型（LLM）已被证明是一种有效的方法。但是，LLM并未固有地定义用于RL训练的代理的结构，尤其是在定义动作空间方面。本文研究了一个紧凑的潜在作用空间，以增强LLM的RL的可控性和探索。我们建议控制具有潜在动作（COLA）的大型语言模型，该框架将潜在的动作空间整合到预训练的LLMS中。我们将可乐应用于Llama-3.1-8B模型。我们的实验表明，与rl相比，与令牌级别的动作相比，可乐的潜在作用可以使文本生成更大的语义多样性。为了增强下游任务，我们表明Cola在MATH500基准测试中获得42.4的成绩，超过38.2的基线得分，并在使用Monte Carlo Tree搜索变体增强时达到68.2。此外，与基线不同，带有RL的可乐始终提高基于代理的任务的性能，而不会降低预先训练的LLM的功能。最后，COLA在涉及RL的LLMS增强思维提示的任务中将计算时间缩短了一半。这些结果突出了可乐的潜力，可以推动基于RL的LLM适应下游应用。

Title: An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses

• Authors: Rohitash Chandra, Aryan Chaudhary, Yeshwanth Rayavarapu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21393
• Pdf URL: https://arxiv.org/pdf/2503.21393
• Copy Paste: [[2503.21393]] An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses(https://arxiv.org/abs/2503.21393)
• Keywords: language model, gpt, llm
• Abstract: Large Language models (LLMs) have been prominent for language translation, including low-resource languages. There has been limited study about the assessment of the quality of translations generated by LLMs, including Gemini, GPT and Google Translate. In this study, we address this limitation by using semantic and sentiment analysis of selected LLMs for Indian languages, including Sanskrit, Telugu and Hindi. We select prominent texts that have been well translated by experts and use LLMs to generate their translations to English, and then we provide a comparison with selected expert (human) translations. Our findings suggest that while LLMs have made significant progress in translation accuracy, challenges remain in preserving sentiment and semantic integrity, especially in figurative and philosophical contexts. The sentiment analysis revealed that GPT-4o and GPT-3.5 are better at preserving the sentiments for the Bhagavad Gita (Sanskrit-English) translations when compared to Google Translate. We observed a similar trend for the case of Tamas (Hindi-English) and Maha P (Telugu-English) translations. GPT-4o performs similarly to GPT-3.5 in the translation in terms of sentiments for the three languages. We found that LLMs are generally better at translation for capturing sentiments when compared to Google Translate.
• 摘要：大型语言模型（LLM）对于语言翻译（包括低资源语言）而言是突出的。关于LLMS生成的翻译质量的评估，包括Gemini，GPT和Google Translate的评估有限的研究。在这项研究中，我们通过使用梵语，泰卢固语和印地语在内的印度语言中选定的LLM的语义和情感分析来解决这一限制。我们选择了专家翻译的著名文本，并使用LLMS生成了英语翻译，然后我们与选定的专家（人类）翻译进行了比较。我们的发现表明，尽管LLM在翻译准确性方面取得了重大进展，但挑战仍在维护情感和语义完整性中，尤其是在象征性和哲学背景下。情感分析表明，与Google翻译相比，GPT-4O和GPT-3.5更好地保留了Bhagavad Gita（Sanskrit-English）翻译的情感。我们观察到Tamas（印地语）和Maha P（Telugu-English）翻译的类似趋势。 GPT-4O在翻译中与三种语言的情感相似。我们发现，与Google Translate相比，LLM通常在捕获情感的翻译方面更好。

Title: Large Language Model Agent: A Survey on Methodology, Applications and Challenges

• Authors: Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S. Yu, Ming Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21460
• Pdf URL: https://arxiv.org/pdf/2503.21460
• Copy Paste: [[2503.21460]] Large Language Model Agent: A Survey on Methodology, Applications and Challenges(https://arxiv.org/abs/2503.21460)
• Keywords: language model, llm, agent
• Abstract: The era of intelligent agents is upon us, driven by revolutionary advancements in large language models. Large Language Model (LLM) agents, with goal-driven behaviors and dynamic adaptation capabilities, potentially represent a critical pathway toward artificial general intelligence. This survey systematically deconstructs LLM agent systems through a methodology-centered taxonomy, linking architectural foundations, collaboration mechanisms, and evolutionary pathways. We unify fragmented research threads by revealing fundamental connections between agent design principles and their emergent behaviors in complex environments. Our work provides a unified architectural perspective, examining how agents are constructed, how they collaborate, and how they evolve over time, while also addressing evaluation methodologies, tool applications, practical challenges, and diverse application domains. By surveying the latest developments in this rapidly evolving field, we offer researchers a structured taxonomy for understanding LLM agents and identify promising directions for future research. The collection is available at this https URL.
• 摘要：智能代理商的时代来自我们，这是由大语言模型的革命进步驱动的。具有目标驱动行为和动态适应能力的大型语言模型（LLM）代理可能代表了人工通用智能的关键途径。该调查通过以方法论为中心的分类法，将建筑基础，协作机制和进化途径联系起来，系统地解构了LLM代理系统。我们通过揭示代理设计原理与其在复杂环境中的新兴行为之间的基本联系来统一零散的研究线程。我们的工作提供了统一的建筑观点，研究了代理的构造方式，如何协作以及它们如何随着时间的流逝而发展，同时还解决了评估方法，工具应用程序，实践挑战和不同的应用程序领域。通过调查这个快速发展的领域中的最新发展，我们为研究人员提供了一种结构化的分类法，以了解LLM代理并确定未来研究的有希望的方向。该集合可在此HTTPS URL上找到。

Title: Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detection

• Authors: Ryan Marinelli, Josef Pichlmeier, Tamas Bisztray
• Subjects: cs.CL, cs.AI, cs.PF
• Abstract URL: https://arxiv.org/abs/2503.21464
• Pdf URL: https://arxiv.org/pdf/2503.21464
• Copy Paste: [[2503.21464]] Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detection(https://arxiv.org/abs/2503.21464)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: In this work, we propose a metric called Number of Thoughts (NofT) to determine the difficulty of tasks pre-prompting and support Large Language Models (LLMs) in production contexts. By setting thresholds based on the number of thoughts, this metric can discern the difficulty of prompts and support more effective prompt routing. A 2% decrease in latency is achieved when routing prompts from the MathInstruct dataset through quantized, distilled versions of Deepseek with 1.7 billion, 7 billion, and 14 billion parameters. Moreover, this metric can be used to detect adversarial prompts used in prompt injection attacks with high efficacy. The Number of Thoughts can inform a classifier that achieves 95% accuracy in adversarial prompt detection. Our experiments ad datasets used are available on our GitHub page: this https URL.
• 摘要：在这项工作中，我们提出了一个称为“思想数量”（NOFT）的指标，以确定在生产环境中预先提出和支持大语言模型（LLM）的任务难度。通过根据思想数来设置阈值，该指标可以辨别提示的难度并支持更有效的提示路由。当通过量化的，蒸馏的版本的DeepSeek，17亿，70亿和140亿个参数从数学数据集进行路由提示时，延迟降低了2％。此外，该指标可用于检测具有高功效的快速注射攻击中使用的对抗提示。思想数量可以告知分类器，该分类器在对抗性及时检测中达到95％的准确性。我们所使用的AD数据集可在我们的GitHub页面上提供：此HTTPS URL。

Title: OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs

• Authors: John Murzaku, Owen Rambow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21480
• Pdf URL: https://arxiv.org/pdf/2503.21480
• Copy Paste: [[2503.21480]] OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs(https://arxiv.org/abs/2503.21480)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: The use of omni-LLMs (large language models that accept any modality as input), particularly for multimodal cognitive state tasks involving speech, is understudied. We present OmniVox, the first systematic evaluation of four omni-LLMs on the zero-shot emotion recognition task. We evaluate on two widely used multimodal emotion benchmarks: IEMOCAP and MELD, and find zero-shot omni-LLMs outperform or are competitive with fine-tuned audio models. Alongside our audio-only evaluation, we also evaluate omni-LLMs on text only and text and audio. We present acoustic prompting, an audio-specific prompting strategy for omni-LLMs which focuses on acoustic feature analysis, conversation context analysis, and step-by-step reasoning. We compare our acoustic prompting to minimal prompting and full chain-of-thought prompting techniques. We perform a context window analysis on IEMOCAP and MELD, and find that using context helps, especially on IEMOCAP. We conclude with an error analysis on the generated acoustic reasoning outputs from the omni-LLMs.
• 摘要：使用了Omni-llms（接受任何模式为输入的大语言模型），尤其是对于涉及语音的多模式认知状态任务的使用已被研究。我们介绍Omnivox，这是对零拍情感识别任务中四个Omni-llms的首次系统评估。我们评估了两个广泛使用的多模式情感基准：Iemocap和Meld，并找到零击的Omni-Lllms表现优于或与微调音频模型具有竞争力。除了我们的音频评估外，我们还仅在文本以及文本和音频上评估Omni-llms。我们提出了声学提示，这是一种针对Omni-llms的音频提示策略，重点是声学特征分析，对话上下文分析和逐步推理。我们将声音提示与最小的提示和完整的思想链接提示技术进行了比较。我们对Iemocap和MELD进行上下文窗口分析，并发现使用上下文有帮助，尤其是在Iemocap上。我们以对来自Omni-llms产生的声学推理输出的错误分析进行了错误分析。

Title: OpenHuEval: Evaluating Large Language Model on Hungarian Specifics

• Authors: Haote Yang, Xingjian Wei, Jiang Wu, Noémi Ligeti-Nagy, Jiaxing Sun, Yinfan Wang, Zijian Győző Yang, Junyuan Gao, Jingchao Wang, Bowen Jiang, Shasha Wang, Nanjun Yu, Zihao Zhang, Shixin Hong, Hongwei Liu, Wei Li, Songyang Zhang, Dahua Lin, Lijun Wu, Gábor Prószéky, Conghui He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21500
• Pdf URL: https://arxiv.org/pdf/2503.21500
• Copy Paste: [[2503.21500]] OpenHuEval: Evaluating Large Language Model on Hungarian Specifics(https://arxiv.org/abs/2503.21500)
• Keywords: language model, llm
• Abstract: We introduce OpenHuEval, the first benchmark for LLMs focusing on the Hungarian language and specifics. OpenHuEval is constructed from a vast collection of Hungarian-specific materials sourced from multiple origins. In the construction, we incorporated the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs' generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations. Ultimately, OpenHuEval encompasses eight Hungarian-specific dimensions, featuring five tasks and 3953 questions. Consequently, OpenHuEval provides the comprehensive, in-depth, and scientifically accurate assessment of LLM performance in the context of the Hungarian language and its specifics. We evaluated current mainstream LLMs, including both traditional LLMs and recently developed Large Reasoning Models. The results demonstrate the significant necessity for evaluation and model optimization tailored to the Hungarian language and specifics. We also established the framework for analyzing the thinking processes of LRMs with OpenHuEval, revealing intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example. We will release OpenHuEval at this https URL .
• 摘要：我们介绍了Openhueval，这是LLMS的第一个基准，专注于匈牙利语言和细节。 Openhueval是由来自多种起源的大量匈牙利特异性材料构建的。在构造中，我们纳入了评估LLM的最新设计原则，例如使用Internet中的真实用户查询，强调对LLMS的生成能力的评估，并采用LLM-As-As-Gudge来提高评估的多维性和准确性。最终，Openhueval涵盖了八个匈牙利特定的维度，其中包含五个任务和3953个问题。因此，Openhueval在匈牙利语言及其细节的背景下，提供了对LLM表现的全面，深入且科学准确的评估。我们评估了当前的主流LLM，包括传统的LLM和最近开发的大型推理模型。结果表明，对匈牙利语言和细节量身定制的评估和模型优化的重要性。我们还建立了一个框架，用于通过Openhueval分析LRM的思维过程，以非英语语言揭示了这些模型的内在模式和机制，而匈牙利语则是代表性的示例。我们将在此HTTPS URL上发布OpenHueval。

Title: Keyword-Oriented Multimodal Modeling for Euphemism Identification

• Authors: Yuxue Hu, Junsong Li, Meixuan Chen, Dongyu Su, Tongguan Wang, Ying Sha
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21504
• Pdf URL: https://arxiv.org/pdf/2503.21504
• Copy Paste: [[2503.21504]] Keyword-Oriented Multimodal Modeling for Euphemism Identification(https://arxiv.org/abs/2503.21504)
• Keywords: language model
• Abstract: Euphemism identification deciphers the true meaning of euphemisms, such as linking "weed" (euphemism) to "marijuana" (target keyword) in illicit texts, aiding content moderation and combating underground markets. While existing methods are primarily text-based, the rise of social media highlights the need for multimodal analysis, incorporating text, images, and audio. However, the lack of multimodal datasets for euphemisms limits further research. To address this, we regard euphemisms and their corresponding target keywords as keywords and first introduce a keyword-oriented multimodal corpus of euphemisms (KOM-Euph), involving three datasets (Drug, Weapon, and Sexuality), including text, images, and speech. We further propose a keyword-oriented multimodal euphemism identification method (KOM-EI), which uses cross-modal feature alignment and dynamic fusion modules to explicitly utilize the visual and audio features of the keywords for efficient euphemism identification. Extensive experiments demonstrate that KOM-EI outperforms state-of-the-art models and large language models, and show the importance of our multimodal datasets.
• 摘要：委婉的识别解释了委婉语的真正含义，例如将“杂草”（委婉语）与非法文本中的“大麻”（目标关键字）联系起来，有助于内容节制和打击地下市场。尽管现有方法主要基于文本，但社交媒体的兴起突出了对多模式分析的需求，并结合了文本，图像和音频。但是，缺乏委婉语的多模式数据集限制了进一步的研究。为了解决这个问题，我们将委婉语及其相应的目标关键字视为关键字，并首先引入了以关键字为导向的委婉语（KOM-euph），涉及三个数据集（药物，武器和性），包括文本，图像和语音。我们进一步提出了一种面向关键字的多模式委婉语识别方法（KOM-EI），该方法使用跨模式特征对齐和动态融合模块明确利用关键字的视觉和音频特征来有效地euphemismismiss识别。广泛的实验表明，KOM-EI的表现优于最先进的模型和大型语言模型，并显示了我们多模式数据集的重要性。

Title: Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving

• Authors: Yue Li, Meng Tian, Zhenyu Lin, Jiangtong Zhu, Dechang Zhu, Haiqiang Liu, Zining Wang, Yueyi Zhang, Zhiwei Xiong, Xinhai Zhao
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21505
• Pdf URL: https://arxiv.org/pdf/2503.21505
• Copy Paste: [[2503.21505]] Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving(https://arxiv.org/abs/2503.21505)
• Keywords: language model
• Abstract: Existing benchmarks for Vision-Language Model (VLM) on autonomous driving (AD) primarily assess interpretability through open-form visual question answering (QA) within coarse-grained tasks, which remain insufficient to assess capabilities in complex driving scenarios. To this end, we introduce $\textbf{VLADBench}$, a challenging and fine-grained dataset featuring close-form QAs that progress from static foundational knowledge and elements to advanced reasoning for dynamic on-road situations. The elaborate $\textbf{VLADBench}$ spans 5 key domains: Traffic Knowledge Understanding, General Element Recognition, Traffic Graph Generation, Target Attribute Comprehension, and Ego Decision-Making and Planning. These domains are further broken down into 11 secondary aspects and 29 tertiary tasks for a granular evaluation. A thorough assessment of general and domain-specific (DS) VLMs on this benchmark reveals both their strengths and critical limitations in AD contexts. To further exploit the cognitive and reasoning interactions among the 5 domains for AD understanding, we start from a small-scale VLM and train the DS models on individual domain datasets (collected from 1.4M DS QAs across public sources). The experimental results demonstrate that the proposed benchmark provides a crucial step toward a more comprehensive assessment of VLMs in AD, paving the way for the development of more cognitively sophisticated and reasoning-capable AD systems.
• 摘要：在自主驾驶（AD）上，视觉模型（VLM）的现有基准主要通过粗粒度任务中的启动形式的视觉问题答案（QA）评估可解释性，这些任务仍然不足以评估复杂驾驶场景中的能力。为此，我们介绍了$ \ textbf {vladbench} $，这是一个具有挑战性且细粒度的数据集，具有近距离质量质量的质量，从静态基础知识和元素到高级推理，以进行动态的道路上。详细的$ \ textbf {vladbench} $跨越5个关键域：交通知识理解，一般元素识别，流量图生成，目标属性理解以及自我决策和计划。这些领域进一步分为11个次要方面和29个三级任务，以进行颗粒状评估。对此基准的一般和领域特异性（DS）VLM的彻底评估揭示了它们在AD上下文中的优势和关键局限性。为了进一步利用5个域之间的认知和推理相互作用以了解广告的理解，我们从小型VLM开始，然后在单个域数据集中训练DS模型（从公共来源收集了14m ds QA）。实验结果表明，所提出的基准为对AD的VLM进行更全面的评估提供了至关重要的步骤，为开发更具认知能力和推理能力的AD系统铺平了道路。

Title: Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models

• Authors: Umer Butt, Stalin Veranasi, Günter Neumann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21530
• Pdf URL: https://arxiv.org/pdf/2503.21530
• Copy Paste: [[2503.21530]] Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models(https://arxiv.org/abs/2503.21530)
• Keywords: language model, gpt
• Abstract: As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. Transliteration between Urdu and its Romanized form, Roman Urdu, remains underexplored despite the widespread use of both scripts in South Asia. Prior work using RNNs on the Roman-Urdu-Parl dataset showed promising results but suffered from poor domain adaptability and limited evaluation. We propose a transformer-based approach using the m2m100 multilingual translation model, enhanced with masked language modeling (MLM) pretraining and fine-tuning on both Roman-Urdu-Parl and the domain-diverse Dakshina dataset. To address previous evaluation flaws, we introduce rigorous dataset splits and assess performance using BLEU, character-level BLEU, and CHRF. Our model achieves strong transliteration performance, with Char-BLEU scores of 96.37 for Urdu->Roman-Urdu and 97.44 for Roman-Urdu->Urdu. These results outperform both RNN baselines and GPT-4o Mini and demonstrate the effectiveness of multilingual transfer learning for low-resource transliteration tasks.
• 摘要：随着信息检索（IR）领域越来越认识到包容性的重要性，满足低资源语言的需求仍然是一个重大挑战。尽管在南亚都广泛使用了这两个脚本，但乌尔都语与罗马化形式之间的音译仍然没有得到反应。先前在罗马 - 乌尔都语-Parl数据集上使用RNN的工作表现出令人鼓舞的结果，但域的适应性差和评估有限。我们使用M2M100多语言翻译模型提出了一种基于变压器的方法，并通过蒙版语言建模（MLM）预处理和对Roman-urdu-Parl和Dokshina Dakshina数据集进行了微调。为了解决以前的评估缺陷，我们介绍了严格的数据集拆分，并使用BLEU，角色级BLEU和CHRF评估性能。我们的模型实现了强劲的音译性能，urdu-> Roman-urdu的Char-BlebleU得分为96.37，而Roman-urdu-> urdu的Char-Bleble分数为97.44。这些结果的表现优于RNN基准和GPT-4O MINI，并证明了多语言转移学习对低资源音译任务的有效性。

Title: SWI: Speaking with Intent in Large Language Models

• Authors: Yuwei Yin, EunJeong Hwang, Giuseppe Carenini
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21544
• Pdf URL: https://arxiv.org/pdf/2503.21544
• Copy Paste: [[2503.21544]] SWI: Speaking with Intent in Large Language Models(https://arxiv.org/abs/2503.21544)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: Intent, typically clearly formulated and planned, functions as a cognitive framework for reasoning and problem-solving. This paper introduces the concept of Speaking with Intent (SWI) in large language models (LLMs), where the explicitly generated intent encapsulates the model's underlying intention and provides high-level planning to guide subsequent analysis and communication. By emulating deliberate and purposeful thoughts in the human mind, SWI is hypothesized to enhance the reasoning capabilities and generation quality of LLMs. Extensive experiments on mathematical reasoning benchmarks consistently demonstrate the superiority of Speaking with Intent over Baseline (i.e., generation without explicit intent). Moreover, SWI outperforms answer-trigger prompting methods Chain-of-Thought and Plan-and-Solve and maintains competitive performance with the strong method ARR (Analyzing, Retrieving, and Reasoning). Additionally, the effectiveness and generalizability of SWI are solidified on reasoning-intensive question answering (QA) and text summarization benchmarks, where SWI brings consistent improvement to the Baseline generation. In text summarization, SWI-generated summaries exhibit greater accuracy, conciseness, and factual correctness, with fewer hallucinations. Furthermore, human evaluations verify the coherence, effectiveness, and interpretability of the intent produced by SWI. This proof-of-concept study creates a novel avenue for enhancing LLMs' reasoning abilities with cognitive notions.
• 摘要：意图，通常是清楚地制定和计划的，是推理和解决问题的认知框架。本文介绍了在大语言模型（LLMS）中以意图（SWI）讲话的概念，其中明确生成的意图封装了模型的基本意图，并提供了高级计划来指导后续的分析和交流。通过在人类思想中模仿有意和有目的的思想，SWI被假设以提高LLM的推理能力和产生质量。关于数学推理基准的广泛实验始终证明了意图高于基线的优越性（即没有明确意图的生成）。此外，SWI的表现优于回答触发触发方法，并通过强大的方法ARR（分析，检索和推理）来进行思考和计划链，并保持竞争性能。此外，SWI的有效性和概括性在推理密集的问题答案（QA）和文本摘要基准上得到了巩固，SWI为基线生成带来了一致的改进。在文本摘要中，SWI生成的摘要表现出更高的准确性，简洁性和事实正确性，并且幻觉更少。此外，人类评估验证了SWI产生的意图的连贯性，有效性和解释性。这项概念验证研究为增强LLMS具有认知概念的推理能力创造了一种新颖的途径。

Title: Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach

• Authors: Javier Coronado-Blázquez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21613
• Pdf URL: https://arxiv.org/pdf/2503.21613
• Copy Paste: [[2503.21613]] Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach(https://arxiv.org/abs/2503.21613)
• Keywords: language model, llm
• Abstract: We study the ability of large language models (LLMs) to generate comprehensive and accurate book summaries solely from their internal knowledge, without recourse to the original text. Employing a diverse set of books and multiple LLM architectures, we examine whether these models can synthesize meaningful narratives that align with established human interpretations. Evaluation is performed with a LLM-as-a-judge paradigm: each AI-generated summary is compared against a high-quality, human-written summary via a cross-model assessment, where all participating LLMs evaluate not only their own outputs but also those produced by others. This methodology enables the identification of potential biases, such as the proclivity for models to favor their own summarization style over others. In addition, alignment between the human-crafted and LLM-generated summaries is quantified using ROUGE and BERTScore metrics, assessing the depth of grammatical and semantic correspondence. The results reveal nuanced variations in content representation and stylistic preferences among the models, highlighting both strengths and limitations inherent in relying on internal knowledge for summarization tasks. These findings contribute to a deeper understanding of LLM internal encodings of factual information and the dynamics of cross-model evaluation, with implications for the development of more robust natural language generative systems.
• 摘要：我们研究大型语言模型（LLMS）的能力，可以仅根据其内部知识来生成全面，准确的书籍摘要，而无需求助于原始文本。采用各种各样的书籍和多种LLM架构，我们检查了这些模型是否可以综合有意义的叙述，这些叙述与已建立的人类解释相符。通过LLM-AS-A-Gudge范式进行评估：将每个AI生成的摘要与通过跨模型评估进行的高质量的，人为写的摘要进行比较，在此，所有参与的LLMS不仅评估其自己的输出，而且还评估了其他人的产出。这种方法可以识别潜在的偏见，例如模型倾向于自己的摘要风格而不是其他偏见。此外，使用Rouge和Bertscore指标对人类制作和LLM生成的摘要之间的对齐方式进行了量化，从而评估了语法和语义对应关系的深度。结果揭示了模型之间内容表示和风格偏好的细微差异，强调了依靠内部知识来进行摘要任务所固有的优势和局限性。这些发现有助于更深入地了解事实信息的LLM内部编码和跨模型评估的动态，这对开发更强大的自然语言生成系统的发展产生了影响。

Title: A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond

• Authors: Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian-Sheng Hua, Bowen Zhou, Yu Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21614
• Pdf URL: https://arxiv.org/pdf/2503.21614
• Copy Paste: [[2503.21614]] A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond(https://arxiv.org/abs/2503.21614)
• Keywords: chain-of-thought, agent
• Abstract: Recent Large Reasoning Models (LRMs), such as DeepSeek-R1 and OpenAI o1, have demonstrated strong performance gains by scaling up the length of Chain-of-Thought (CoT) reasoning during inference. However, a growing concern lies in their tendency to produce excessively long reasoning traces, which are often filled with redundant content (e.g., repeated definitions), over-analysis of simple problems, and superficial exploration of multiple reasoning paths for harder tasks. This inefficiency introduces significant challenges for training, inference, and real-world deployment (e.g., in agent-based systems), where token economy is critical. In this survey, we provide a comprehensive overview of recent efforts aimed at improving reasoning efficiency in LRMs, with a particular focus on the unique challenges that arise in this new paradigm. We identify common patterns of inefficiency, examine methods proposed across the LRM lifecycle, i.e., from pretraining to inference, and discuss promising future directions for research. To support ongoing development, we also maintain a real-time GitHub repository tracking recent progress in the field. We hope this survey serves as a foundation for further exploration and inspires innovation in this rapidly evolving area.
• 摘要：最近的大型推理模型（LRMS），例如DeepSeek-R1和OpenAI O1，通过在推理过程中扩大了思维链（COT）推理的长度来表现出强大的性能提高。但是，越来越多的关注在于它们倾向于产生过长的推理轨迹，这些轨迹通常充满了冗余内容（例如，重复定义），简单问题的过度分析以及对更艰难任务的多种推理路径的表面探索。这种效率低下对培训，推理和现实世界部署（例如，在基于代理的系统中）带来了重大挑战，而代币经济至关重要。在这项调查中，我们提供了旨在提高LRMS推理效率的最新努力的全面概述，特别着眼于这种新范式中出现的独特挑战。我们确定了效率低下的常见模式，检查整个LRM生命周期提出的方法，即从预处理到推理，并讨论有希望的未来研究方向。为了支持正在进行的开发，我们还维护了一个实时的GITHUB存储库跟踪该领域的最新进展。我们希望这项调查是进一步探索的基础，并激发了这个快速发展的地区的创新。

Title: COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing

• Authors: Rajvee Sheth, Himanshu Beniwal, Mayank Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21670
• Pdf URL: https://arxiv.org/pdf/2503.21670
• Copy Paste: [[2503.21670]] COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing(https://arxiv.org/abs/2503.21670)
• Keywords: llm
• Abstract: The rapid growth of digital communication has driven the widespread use of code-mixing, particularly Hindi-English, in multilingual communities. Existing datasets often focus on romanized text, have limited scope, or rely on synthetic data, which fails to capture realworld language nuances. Human annotations are crucial for assessing the naturalness and acceptability of code-mixed text. To address these challenges, We introduce COMI-LINGUA, the largest manually annotated dataset for code-mixed text, comprising 100,970 instances evaluated by three expert annotators in both Devanagari and Roman scripts. The dataset supports five fundamental NLP tasks: Language Identification, Matrix Language Identification, Part-of-Speech Tagging, Named Entity Recognition, and Translation. We evaluate LLMs on these tasks using COMILINGUA, revealing limitations in current multilingual modeling strategies and emphasizing the need for improved code-mixed text processing capabilities. COMI-LINGUA is publically availabe at: this https URL.
• 摘要：数字通信的快速增长推动了代码混合的广泛使用，尤其是印度英语社区。现有的数据集通常集中在罗马文本上，范围有限或依赖综合数据，而综合数据无法捕获现实世界的细微差别。人类注释对于评估代码混合文本的自然性和可接受性至关重要。为了应对这些挑战，我们介绍了Comi-Lingua，这是最大的手动注释数据集用于代码混合文本，其中包括Devanagari和Roman Scripts中三个专家注释者评估的100,970个实例。数据集支持五个基本的NLP任务：语言标识，矩阵语言标识，词性标记，词组标记，命名实体识别和翻译。我们使用COMILingua评估了这些任务的LLM，揭示了当前的多语言建模策略的局限性，并强调了改进的代码混合文本处理功能的需求。 comi-lingua公开可用，网址为：此HTTPS URL。

Title: How do language models learn facts? Dynamics, curricula and hallucinations

• Authors: Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, Soham De
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21676
• Pdf URL: https://arxiv.org/pdf/2503.21676
• Copy Paste: [[2503.21676]] How do language models learn facts? Dynamics, curricula and hallucinations(https://arxiv.org/abs/2503.21676)
• Keywords: language model, hallucination
• Abstract: Large language models accumulate vast knowledge during pre-training, yet the dynamics governing this acquisition remain poorly understood. This work investigates the learning dynamics of language models on a synthetic factual recall task, uncovering three key findings: First, language models learn in three phases, exhibiting a performance plateau before acquiring precise factual knowledge. Mechanistically, this plateau coincides with the formation of attention-based circuits that support recall. Second, the training data distribution significantly impacts learning dynamics, as imbalanced distributions lead to shorter plateaus. Finally, hallucinations emerge simultaneously with knowledge, and integrating new knowledge into the model through fine-tuning is challenging, as it quickly corrupts its existing parametric memories. Our results emphasize the importance of data distribution in knowledge acquisition and suggest novel data scheduling strategies to accelerate neural network training.
• 摘要：大型语言模型在预训练期间积累了广泛的知识，但是管理这一收购的动态仍然知之甚少。这项工作研究了语言模型在综合事实召回任务上的学习动力学，发现了三个关键发现：首先，语言模型分为三个阶段，在获得精确的事实知识之前展示了表演平稳。从机械上讲，该平台与支持回忆的基于注意的电路的形成相吻合。其次，训练数据分布显着影响学习动态，因为分布不平衡导致平台较短。最后，幻觉与知识同时出现，并通过微调将新知识整合到模型中是具有挑战性的，因为它迅速破坏了其现有的参数记忆。我们的结果强调了数据分布在知识获取中的重要性，并提出了新的数据调度策略以加速神经网络培训。

Title: JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community

• Authors: Yunze Xiao, Tingyu He, Lionel Z. Wang, Yiming Ma, Xingyu Song, Xiaohang Xu, Irene Li, Ka Chung Ng
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.21679
• Pdf URL: https://arxiv.org/pdf/2503.21679
• Copy Paste: [[2503.21679]] JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community(https://arxiv.org/abs/2503.21679)
• Keywords: language model, prompt
• Abstract: This paper introduces JiraiBench, the first bilingual benchmark for evaluating large language models' effectiveness in detecting self-destructive content across Chinese and Japanese social media communities. Focusing on the transnational "Jirai" (landmine) online subculture that encompasses multiple forms of self-destructive behaviors including drug overdose, eating disorders, and self-harm, we present a comprehensive evaluation framework incorporating both linguistic and cultural dimensions. Our dataset comprises 10,419 Chinese posts and 5,000 Japanese posts with multidimensional annotation along three behavioral categories, achieving substantial inter-annotator agreement. Experimental evaluations across four state-of-the-art models reveal significant performance variations based on instructional language, with Japanese prompts unexpectedly outperforming Chinese prompts when processing Chinese content. This emergent cross-cultural transfer suggests that cultural proximity can sometimes outweigh linguistic similarity in detection tasks. Cross-lingual transfer experiments with fine-tuned models further demonstrate the potential for knowledge transfer between these language systems without explicit target language training. These findings highlight the need for culturally-informed approaches to multilingual content moderation and provide empirical evidence for the importance of cultural context in developing more effective detection systems for vulnerable online communities.
• 摘要：本文介绍了Jiraibench，这是第一个双语基准，用于评估大型语言模型在检测中国和日本社交媒体社区的自我毁灭性内容方面的有效性。专注于跨国“吉列”（地雷）在线亚文化，其中包含多种形式的自我毁灭行为，包括药物过量，饮食失调和自我伤害，我们提出了一个综合语言和文化维度的全面评估框架。我们的数据集包含10,419个中国帖子和5,000个日本帖子，其中包括三个行为类别的多维注释，实现了实质性的通知者协议。跨四个最先进模型的实验评估揭示了基于教学语言的显着性能差异，日本提示在处理中文内容时意外表现出色的中国提示。这种新兴的跨文化转移表明，文化接近有时可能会超过检测任务的语言相似性。使用微调模型的跨语性转移实验进一步证明了在没有明确目标语言培训的情况下，这些语言系统之间的知识转移的潜力。这些发现凸显了对文化知识的多种语言内容适度方法的需求，并为文化背景在为脆弱的在线社区开发更有效的检测系统中的重要性提供了经验证据。

Title: As easy as PIE: understanding when pruning causes language models to disagree

• Authors: Pietro Tropeano, Maria Maistro, Tuukka Ruotsalo, Christina Lioma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21714
• Pdf URL: https://arxiv.org/pdf/2503.21714
• Copy Paste: [[2503.21714]] As easy as PIE: understanding when pruning causes language models to disagree(https://arxiv.org/abs/2503.21714)
• Keywords: language model
• Abstract: Language Model (LM) pruning compresses the model by removing weights, nodes, or other parts of its architecture. Typically, pruning focuses on the resulting efficiency gains at the cost of effectiveness. However, when looking at how individual data points are affected by pruning, it turns out that a particular subset of data points always bears most of the brunt (in terms of reduced accuracy) when pruning, but this effect goes unnoticed when reporting the mean accuracy of all data points. These data points are called PIEs and have been studied in image processing, but not in NLP. In a study of various NLP datasets, pruning methods, and levels of compression, we find that PIEs impact inference quality considerably, regardless of class frequency, and that BERT is more prone to this than BiLSTM. We also find that PIEs contain a high amount of data points that have the largest influence on how well the model generalises to unseen data. This means that when pruning, with seemingly moderate loss to accuracy across all data points, we in fact hurt tremendously those data points that matter the most. We trace what makes PIEs both hard and impactful to inference to their overall longer and more semantically complex text. These findings are novel and contribute to understanding how LMs are affected by pruning. The code is available at: this https URL
• 摘要：语言模型（LM）修剪通过删除权重，节点或其架构的其他部分来压缩模型。通常，修剪以有效性为代价，重点是产生的效率提高。但是，当查看单个数据点如何受到修剪的影响时，事实证明，特定的数据点在修剪时总是首当其冲（就精度降低而言），但是当报告所有数据点的平均准确性时，这种效果都不会引起注意。这些数据点称为派，已经在图像处理中进行了研究，但在NLP中不研究。在对各种NLP数据集，修剪方法和压缩水平的研究中，我们发现馅饼不管班级频率如何，都会显着影响推理质量，并且BERT比Bilstm更容易遇到这种质量。我们还发现，馅饼包含大量的数据点，这些数据点对模型对看不见的数据的概括程度具有最大的影响。这意味着，在修剪所有数据点的准确性似乎中等损失时，我们实际上会严重伤害最重要的数据点。我们追溯了使派既硬化又具有影响力的原因，以推断其整体更长，更复杂的文本。这些发现是新颖的，有助于理解LMS如何受到修剪的影响。代码可用：此HTTPS URL

Title: CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?

• Authors: Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, Benjamin Van Durme
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21717
• Pdf URL: https://arxiv.org/pdf/2503.21717
• Copy Paste: [[2503.21717]] CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?(https://arxiv.org/abs/2503.21717)
• Keywords: llm
• Abstract: A core part of scientific peer review involves providing expert critiques that directly assess the scientific claims a paper makes. While it is now possible to automatically generate plausible (if generic) reviews, ensuring that these reviews are sound and grounded in the papers' claims remains challenging. To facilitate LLM benchmarking on these challenges, we introduce CLAIMCHECK, an annotated dataset of NeurIPS 2023 and 2024 submissions and reviews mined from OpenReview. CLAIMCHECK is richly annotated by ML experts for weakness statements in the reviews and the paper claims that they dispute, as well as fine-grained labels of the validity, objectivity, and type of the identified weaknesses. We benchmark several LLMs on three claim-centric tasks supported by CLAIMCHECK, requiring models to (1) associate weaknesses with the claims they dispute, (2) predict fine-grained labels for weaknesses and rewrite the weaknesses to enhance their specificity, and (3) verify a paper's claims with grounded reasoning. Our experiments reveal that cutting-edge LLMs, while capable of predicting weakness labels in (2), continue to underperform relative to human experts on all other tasks.
• 摘要：科学同行评审的核心部分是提供直接评估论文提出的科学主张的专家批评。虽然现在可以自动生成合理的（如果是通用）评论，以确保这些评论是合理的，并在论文的主张中扎根仍然具有挑战性。为了促进LLM基准对这些挑战进行基准测试，我们介绍了SoipeCheck，这是一个带注释的Neurips 2023和2024年的注释数据集，并从OpenReview挖掘出来的评论。 ML专家对索赔的诉讼进行了丰富的注释。我们在SopearCheck支持的三个以索赔为中心的任务上基准了几个LLM，要求模型（1）将弱点与他们提出异议的索赔相关联，（2）预测弱点的细粒度标签，并重写弱点，以增强其特异性，以及（3）验证论文的索赔与基础推理。我们的实验表明，尖端的LLM虽然能够预测（2）中的弱点标签，但相对于所有其他任务的人类专家继续表现不佳。

Title: Outlier dimensions favor frequent tokens in language model

• Authors: Iuri Macocco, Nora Graichen, Gemma Boleda, Marco Baroni
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21718
• Pdf URL: https://arxiv.org/pdf/2503.21718
• Copy Paste: [[2503.21718]] Outlier dimensions favor frequent tokens in language model(https://arxiv.org/abs/2503.21718)
• Keywords: language model
• Abstract: We study last-layer outlier dimensions, this http URL that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern language models, and trace their function back to the heuristic of constantly predicting frequent words. We further show how a model can block this heuristic when it is not contextually appropriate, by assigning a counterbalancing weight mass to the remaining dimensions, and we investigate which model parameters boost outlier dimensions and when they arise during training. We conclude that outlier dimensions are a specialized mechanism discovered by many distinct models to implement a useful token prediction heuristic.
• 摘要：我们研究了最后的离群尺寸，该HTTP URL显示了大多数输入的极端激活。我们表明，在许多不同的现代语言模型中出现了离群尺寸，并将其功能追溯到不断预测频繁单词的启发式。我们进一步展示了模型在不适当的情况下如何通过将平衡重量质量分配给其余维度时可以阻止这种启发式，并且我们研究了哪些模型参数可以提高离群尺寸以及在训练过程中出现的何时出现。我们得出的结论是，离群维度是许多不同模型发现的专业机制，以实现有用的令牌预测启发式。

Title: Collab: Controlled Decoding using Mixture of Agents for LLM Alignment

• Authors: Souradip Chakraborty, Sujay Bhatt, Udari Madhushani Sehwag, Soumya Suvra Ghosal, Jiahao Qiu, Mengdi Wang, Dinesh Manocha, Furong Huang, Alec Koppel, Sumitra Ganesh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21720
• Pdf URL: https://arxiv.org/pdf/2503.21720
• Copy Paste: [[2503.21720]] Collab: Controlled Decoding using Mixture of Agents for LLM Alignment(https://arxiv.org/abs/2503.21720)
• Keywords: language model, gpt, llm, agent
• Abstract: Alignment of Large Language models (LLMs) is crucial for safe and trustworthy deployment in applications. Reinforcement learning from human feedback (RLHF) has emerged as an effective technique to align LLMs to human preferences and broader utilities, but it requires updating billions of model parameters, which is computationally expensive. Controlled Decoding, by contrast, provides a mechanism for aligning a model at inference time without retraining. However, single-agent decoding approaches often struggle to adapt to diverse tasks due to the complexity and variability inherent in these tasks. To strengthen the test-time performance w.r.t the target task, we propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies. Treating each prior policy as an agent in the spirit of mixture of agent collaboration, we develop a decoding method that allows for inference-time alignment through a token-level selection strategy among multiple agents. For each token, the most suitable LLM is dynamically chosen from a pool of models based on a long-term utility metric. This policy-switching mechanism ensures optimal model selection at each step, enabling efficient collaboration and alignment among LLMs during decoding. Theoretical analysis of our proposed algorithm establishes optimal performance with respect to the target task represented via a target reward for the given off-the-shelf models. We conduct comprehensive empirical evaluations with open-source aligned models on diverse tasks and preferences, which demonstrates the merits of this approach over single-agent decoding baselines. Notably, Collab surpasses the current SoTA decoding strategy, achieving an improvement of up to 1.56x in average reward and 71.89% in GPT-4 based win-tie rate.
• 摘要：大型语言模型（LLM）的一致性对于应用程序中的安全和值得信赖的部署至关重要。从人类反馈（RLHF）中学习的强化已成为一种有效的技术，可以使LLM与人类的偏好和更广泛的公用事业保持一致，但是它需要更新数十亿个模型参数，这在计算上昂贵。相比之下，受控解码提供了一种在不重新培训的情况下对模型对齐的机制。但是，由于这些任务中固有的复杂性和可变性，单一代理解码方法通常很难适应各种任务。为了加强测试时间性能W.R.T目标任务，我们提出了基于代理的解码策略来利用现有现成的LLM策略的混合。我们将每个先前的政策视为代理商协作精神的代理，我们开发了一种解码方法，可以通过多个代理之间的令牌选择策略进行推理时间对齐。对于每个令牌，最合适的LLM是基于长期实用性度量的模型池动态选择的。这种政策开关机制可确保每个步骤的最佳模型选择，从而在解码过程中有效地协作和对齐。对我们提出的算法的理论分析对通过给定现场模型的目标奖励表示的目标任务建立了最佳性能。我们对各种任务和偏好进行开源对齐模型进行全面的经验评估，这证明了这种方法比单格解码基准的优点。值得注意的是，合作超过了当前的SOTA解码策略，在平均奖励中提高了1.56倍，而基于GPT-4的Win-tie率的提高了71.89％。

Title: ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generation

• Authors: Zhicheng Lee, Shulin Cao, Jinxin Liu, Jiajie Zhang, Weichuan Liu, Xiaoyin Che, Lei Hou, Juanzi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21729
• Pdf URL: https://arxiv.org/pdf/2503.21729
• Copy Paste: [[2503.21729]] ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generation(https://arxiv.org/abs/2503.21729)
• Keywords: retrieval augmented generation, retrieval-augmented generation
• Abstract: Large Reasoning Models (LRMs) exhibit remarkable reasoning abilities but rely primarily on parametric knowledge, limiting factual accuracy. While recent works equip reinforcement learning (RL)-based LRMs with retrieval capabilities, they suffer from overthinking and lack robustness in reasoning, reducing their effectiveness in question answering (QA) tasks. To address this, we propose ReaRAG, a factuality-enhanced reasoning model that explores diverse queries without excessive iterations. Our solution includes a novel data construction framework with an upper bound on the reasoning chain length. Specifically, we first leverage an LRM to generate deliberate thinking, then select an action from a predefined action space (Search and Finish). For Search action, a query is executed against the RAG engine, where the result is returned as observation to guide reasoning steps later. This process iterates until a Finish action is chosen. Benefiting from ReaRAG's strong reasoning capabilities, our approach outperforms existing baselines on multi-hop QA. Further analysis highlights its strong reflective ability to recognize errors and refine its reasoning trajectory. Our study enhances LRMs' factuality while effectively integrating robust reasoning for Retrieval-Augmented Generation (RAG).
• 摘要：大型推理模型（LRMS）具有显着的推理能力，但主要依赖于参数知识，从而限制了事实准确性。尽管最近的作品配备了基于基于LRM的LRM的加强学习能力，但它们在推理方面遭受了过度思考和缺乏鲁棒性的困扰，从而降低了他们的有效性回答（QA）任务。为了解决这个问题，我们提出了Rearag，这是一种事实增强的推理模型，探讨了不同的查询而没有过多的迭代。我们的解决方案包括一个新型的数据构建框架，在推理链长度上具有上限。具体来说，我们首先利用LRM来产生故意思考，然后从预定义的动作空间（搜索和完成）中选择一个动作。对于搜索操作，针对抹布引擎执行查询，在该引擎中将结果返回以观察以引导推理步骤。此过程迭代直到选择完成操作。我们的方法受益于READAG的强大推理能力，优于多跳质量检查的现有基准。进一步的分析强调了其强大的反射能力，可以识别错误并完善其推理轨迹。我们的研究增强了LRMS的事实，同时有效地整合了检索型发电（RAG）的强大推理。

Title: Effective Skill Unlearning through Intervention and Abstention

• Authors: Yongce Li, Chung-En Sun, Tsui-Wei Weng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21730
• Pdf URL: https://arxiv.org/pdf/2503.21730
• Copy Paste: [[2503.21730]] Effective Skill Unlearning through Intervention and Abstention(https://arxiv.org/abs/2503.21730)
• Keywords: language model, llm
• Abstract: Large language Models (LLMs) have demonstrated remarkable skills across various domains. Understanding the mechanisms behind their abilities and implementing controls over them is becoming increasingly important for developing better models. In this paper, we focus on skill unlearning in LLMs, specifically unlearning a particular skill while retaining their overall capabilities. We introduce two lightweight, training-free machine skill unlearning techniques for LLMs. First, we observe that the pre-activation distribution of neurons in each Feed-Forward Layer (FFL) differs when the model demonstrates different skills. Additionally, we find that queries triggering the same skill cluster within the FFL key space and can be separated from other queries using a hypercube. Based on these observations, we propose two lightweight, training-free skill unlearning methods via \textit{intervention} and \textit{abstention} respectively: \texttt{Neuron Adjust} and \texttt{Key Space Detection}. We evaluate our methods on unlearning math-solving, Python-coding, and comprehension skills across seven different languages. The results demonstrate their strong unlearning capabilities for the designated skills. Specifically, \texttt{Key Space Detection} achieves over 80\% relative performance drop on the forgetting skill and less than 10\% relative performance drop on other skills and the model's general knowledge (MMLU) for most unlearning tasks. Our code is available at this https URL
• 摘要：大型语言模型（LLM）在各个领域都表现出了出色的技能。了解其能力背后的机制和对它们实施控制的机制对于开发更好的模型变得越来越重要。在本文中，我们专注于LLM中的技能，特别是在保留其整体功能的同时，专门学习特定技能。我们介绍了针对LLM的两种轻巧的，无训练的机器技能。首先，我们观察到，当模型表现出不同的技能时，每个馈送层（FFL）中神经元的前激活分布有所不同。此外，我们发现查询在FFL密钥空间内触发相同的技能群集，并可以使用HyperCube与其他查询分开。基于这些观察结果，我们分别通过\ textit {Issention}和\ textit {弃权}分别提出了两种轻巧的，无训练的技能学习方法：\ texttt {neuron awadion}和\ texttttt {键空间检测}。我们评估了跨七种不同语言的学习数学解决，python编码和理解能力的方法。结果证明了他们在指定技能方面的强大学习能力。具体而言，\ texttt {关键空间检测}在忘记技能上的相对性能下降到超过80 \％的相对性能下降，而对于大多数读书任务，其他技能的相对性能下降和模型的常识（MMLU）的相对性能下降。我们的代码可在此HTTPS URL上找到

Title: MemInsight: Autonomous Memory Augmentation for LLM Agents

• Authors: Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, Yassine Benajiba
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21760
• Pdf URL: https://arxiv.org/pdf/2503.21760
• Copy Paste: [[2503.21760]] MemInsight: Autonomous Memory Augmentation for LLM Agents(https://arxiv.org/abs/2503.21760)
• Keywords: language model, llm, agent
• Abstract: Large language model (LLM) agents have evolved to intelligently process information, make decisions, and interact with users or tools. A key capability is the integration of long-term memory capabilities, enabling these agents to draw upon historical interactions and knowledge. However, the growing memory size and need for semantic structuring pose significant challenges. In this work, we propose an autonomous memory augmentation approach, MemInsight, to enhance semantic data representation and retrieval mechanisms. By leveraging autonomous augmentation to historical interactions, LLM agents are shown to deliver more accurate and contextualized responses. We empirically validate the efficacy of our proposed approach in three task scenarios; conversational recommendation, question answering and event summarization. On the LLM-REDIAL dataset, MemInsight boosts persuasiveness of recommendations by up to 14%. Moreover, it outperforms a RAG baseline by 34% in recall for LoCoMo retrieval. Our empirical results show the potential of MemInsight to enhance the contextual performance of LLM agents across multiple tasks.
• 摘要：大型语言模型（LLM）代理已经演变为智能处理信息，做出决策并与用户或工具互动。关键能力是长期记忆能力的整合，使这些代理能够利用历史互动和知识。但是，记忆的规模不断增长，对语义结构的需求构成了重大挑战。在这项工作中，我们提出了一种自主记忆增强方法，即Meminsight，以增强语义数据表示和检索机制。通过利用自主增强对历史相互作用，LLM代理被证明可以提供更准确和上下文的响应。在三种任务方案中，我们从经验上验证了我们提议的方法的功效；会话建议，问答和事件摘要。在LLM-REDIAL数据集上，Meminsight的说服力提高了高达14％的建议。此外，它的召回率在召回率方面的表现要优于抹布基线34％。我们的经验结果表明，MESIntight的潜力可以增强LLM跨多个任务的LLM代理的上下文性能。