2025-05-07

Title: Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models

• Authors: Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, Xiaolong Li
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2505.02847
• Pdf URL: https://arxiv.org/pdf/2505.02847
• Copy Paste: [[2505.02847]] Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models(https://arxiv.org/abs/2505.02847)
• Keywords: language model, gpt, llm, agent
• Abstract: Assessing how well a large language model (LLM) understands human, rather than merely text, remains an open challenge. To bridge the gap, we introduce Sentient Agent as a Judge (SAGE), an automated evaluation framework that measures an LLM's higher-order social cognition. SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction, providing a more realistic evaluation of the tested model in multi-turn conversations. At every turn, the agent reasons about (i) how its emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a numerical emotion trajectory and interpretable inner thoughts. Experiments on 100 supportive-dialogue scenarios show that the final Sentient emotion score correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings and utterance-level empathy metrics, validating psychological fidelity. We also build a public Sentient Leaderboard covering 18 commercial and open-source models that uncovers substantial gaps (up to 4x) between frontier systems (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in conventional leaderboards (e.g., Arena). SAGE thus provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.
• 摘要：评估大型语言模型（LLM）对人类的理解，而不仅仅是文本，这仍然是一个开放的挑战。为了弥合差距，我们引入了有声学代理作为法官（SAGE），这是一个自动化评估框架，可衡量LLM的高阶社交认知。 Sage实例化了一种有知觉的代理，该代理在互动过程中模拟了类似人类的情感变化和内在思想，从而在多转交谈中对测试模型进行了更现实的评估。在任何情况下，代理的原因（i）情绪如何变化，（ii）感觉如何，以及（iii）应该如何回复，产生数值的情感轨迹和可解释的内在思想。对100个支持二元式方案的实验表明，最终的情感得分与Barrett-Lennard关系清单（BLRI）评分和话语级别的同理心度量密切相关，从而验证了心理忠诚度。我们还建立了一个公共知名的排行榜，涵盖了18种商业和开源模型，该模型在边境系统（GPT-4O-LATEST，GEMINI2.5-PRO）和早期的基线之间发现了巨大差距（最高4倍），并且在传统的排行榜中未反映的差距（例如，竞技场）。因此，Sage提供了一种原则性，可扩展性和可解释的工具，用于跟踪对真正善解人意和社会熟练的语言代理的进步。

Title: Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors

• Authors: Nicy Scaria, Silvester John Joseph Kennedy, Diksha Seth, Ananya Thakur, Deepak Subramani
• Subjects: cs.CL, cs.AI, cs.CY, cs.DB
• Abstract URL: https://arxiv.org/abs/2505.02850
• Pdf URL: https://arxiv.org/pdf/2505.02850
• Copy Paste: [[2505.02850]] Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors(https://arxiv.org/abs/2505.02850)
• Keywords: llm
• Abstract: Generating high-quality MCQs, especially those targeting diverse cognitive levels and incorporating common misconceptions into distractor design, is time-consuming and expertise-intensive, making manual creation impractical at scale. Current automated approaches typically generate questions at lower cognitive levels and fail to incorporate domain-specific misconceptions. This paper presents a hierarchical concept map-based framework that provides structured knowledge to guide LLMs in generating MCQs with distractors. We chose high-school physics as our test domain and began by developing a hierarchical concept map covering major Physics topics and their interconnections with an efficient database design. Next, through an automated pipeline, topic-relevant sections of these concept maps are retrieved to serve as a structured context for the LLM to generate questions and distractors that specifically target common misconceptions. Lastly, an automated validation is completed to ensure that the generated MCQs meet the requirements provided. We evaluate our framework against two baseline approaches: a base LLM and a RAG-based generation. We conducted expert evaluations and student assessments of the generated MCQs. Expert evaluation shows that our method significantly outperforms the baseline approaches, achieving a success rate of 75.20% in meeting all quality criteria compared to approximately 37% for both baseline methods. Student assessment data reveal that our concept map-driven approach achieved a significantly lower guess success rate of 28.05% compared to 37.10% for the baselines, indicating a more effective assessment of conceptual understanding. The results demonstrate that our concept map-based approach enables robust assessment across cognitive levels and instant identification of conceptual gaps, facilitating faster feedback loops and targeted interventions at scale.
• 摘要：产生高质量的MCQ，尤其是那些针对多种认知水平并将共同误解纳入干扰器设计的人，这是耗时且耗时的，使手动创建不切实际。当前的自动化方法通常会以较低的认知水平产生问题，并且无法纳入特定领域的误解。本文提出了一个基于层次概念图的框架，该框架提供结构化知识，以指导LLMS用干扰器生成MCQ。我们选择了高中物理学作为我们的测试领域，并首先开发了一个层次概念图，涵盖了主要物理主题及其与高效数据库设计的互连。接下来，通过自动管道，将这些概念图的与主题相关的部分被检索到LLM产生问题和干扰因素的结构性上下文，这些环境专门针对常见的误解。最后，完成自动验证，以确保生成的MCQ满足所提供的要求。我们根据两种基线方法评估框架：基本LLM和基于抹布的一代。我们对生成的MCQ进行了专家评估和学生评估。专家评估表明，我们的方法显着胜过基线方法，在满足所有质量标准的成功率为75.20％，而两种基线方法的成功率约为37％。学生评估数据表明，我们的概念地图驱动的方法的猜测成功率显着降低了28.05％，而基本线的猜测成功率为37.10％，这表明对概念理解的评估更有效。结果表明，我们的基于概念图的方法可以跨认知水平进行稳健的评估，并立即识别概念差距，促进更快的反馈回路和大规模的有针对性干预措施。

Title: 30DayGen: Leveraging LLMs to Create a Content Corpus for Habit Formation

• Authors: Franklin Zhang, Sonya Zhang, Alon Halevy
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2505.02851
• Pdf URL: https://arxiv.org/pdf/2505.02851
• Copy Paste: [[2505.02851]] 30DayGen: Leveraging LLMs to Create a Content Corpus for Habit Formation(https://arxiv.org/abs/2505.02851)
• Keywords: language model, llm
• Abstract: In this paper, we present 30 Day Me, a habit formation application that leverages Large Language Models (LLMs) to help users break down their goals into manageable, actionable steps and track their progress. Central to the app is the 30DAYGEN system, which generates 3,531 unique 30-day challenges sourced from over 15K webpages, and enables runtime search of challenge ideas aligned with user-defined goals. We showcase how LLMs can be harnessed to rapidly construct domain specific content corpora for behavioral and educational purposes, and propose a practical pipeline that incorporates effective LLM enhanced approaches for content generation and semantic deduplication.
• 摘要：在本文中，我们介绍了30天ME，这是一种利用大型语言模型（LLM）的习惯形成应用，以帮助用户将目标分解为可管理的，可行的步骤并跟踪其进度。该应用程序的核心是30DayGen系统，该系统产生了3,531个独特的30天挑战，这些挑战来自超过15K的网页，并启用了与用户定义的目标一致的挑战想法的运行时搜索。我们展示了如何利用LLM来用于行为和教育目的快速构建特定域的内容语料库，并提出了一条实用的管道，该管道结合了有效的LLM增强功能，以增强内容的生成和语义删除。

Title: Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets

• Authors: Masumi Morishige, Ryo Koshihara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.02854
• Pdf URL: https://arxiv.org/pdf/2505.02854
• Copy Paste: [[2505.02854]] Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets(https://arxiv.org/abs/2505.02854)
• Keywords: language model, gpt, llm, prompt
• Abstract: Reproducibility and reliability remain pressing challenges for generative AI systems whose behavior can drift with each model update or prompt revision. We introduce GPR-bench, a lightweight, extensible benchmark that operationalizes regression testing for general purpose use cases. GPR-bench couples an open, bilingual (English and Japanese) dataset covering eight task categories (e.g., text generation, code generation, and information retrieval) and 10 scenarios in each task categories (80 total test cases for each language) with an automated evaluation pipeline that employs "LLM-as-a-Judge" scoring of correctness and conciseness. Experiments across three recent model versions - gpt-4o-mini, o3-mini, and o4-mini - and two prompt configurations (default versus concise-writing instruction) reveal heterogeneous quality. Our results show that newer models generally improve correctness, but the differences are modest and not statistically significant, suggesting that GPR-bench may not be sufficiently challenging to differentiate between recent model versions. In contrast, the concise-writing instruction significantly enhances conciseness (+12.37 pp, Mann-Whitney U test: p < 0.001, effect size r = 0.2995) with minimal degradations on accuracy (-1.7 pp), demonstrating the effectiveness of prompt engineering. Released under the MIT License, GPR- bench lowers the barrier to initiating reproducibility monitoring and provides a foundation for community-driven extensions, while also raising important considerations about benchmark design for rapidly evolving language models.
• 摘要：可重复性和可靠性仍然是对生成AI系统的紧迫挑战，其行为可以随着每个模型更新或提示修订而漂移。我们介绍了GPR Bench，这是一种轻巧，可扩展的基准测试，可为通用用例操作回归测试。 GPR板凳夫妇夫妇涵盖了八个任务类别（例如，文本生成，代码生成和信息检索）的八个任务类别（例如，每个任务类别的10个场景（每种语言的80个总测试用例），具有自动化评估管道中的10个场景，这些管道采用了“ LLM-AS-AS-AS-A-A-Judge”，“ LLM-AS-A-A-Judge”得分。在最近的三个模型版本中进行的实验-GPT-4O-MINI，O3-MINI和O4-MINI-以及两种及时的配置（默认与简洁写入指令）揭示了异构质量。我们的结果表明，较新的模型通常会提高正确性，但是差异是适度的，并且没有统计学意义，这表明GPR基础台上可能没有足够的挑战以区分最近的模型版本。相反，简洁的写作指令显着提高了简洁性（+12.37 pp，Mann-Whitney U检验：P <0.001，效果大小r = 0.2995），精度的最小降解（-1.7 pp），证明了迅速工程的有效性。 GPR-Bench在MIT许可下发布的降低了启动可重复性监控的障碍，并为社区驱动的扩展提供了基础，同时还提出了有关快速发展的语言模型的基准设计的重要考虑因素。

Title: Towards High-Fidelity Synthetic Multi-platform Social Media Datasets via Large Language Models

• Authors: Henry Tari, Nojus Sereiva, Rishabh Kaushal, Thales Bertaglia, Adriana Iamnitchi
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2505.02858
• Pdf URL: https://arxiv.org/pdf/2505.02858
• Copy Paste: [[2505.02858]] Towards High-Fidelity Synthetic Multi-platform Social Media Datasets via Large Language Models(https://arxiv.org/abs/2505.02858)
• Keywords: language model, prompt
• Abstract: Social media datasets are essential for research on a variety of topics, such as disinformation, influence operations, hate speech detection, or influencer marketing practices. However, access to social media datasets is often constrained due to costs and platform restrictions. Acquiring datasets that span multiple platforms, which is crucial for understanding the digital ecosystem, is particularly challenging. This paper explores the potential of large language models to create lexically and semantically relevant social media datasets across multiple platforms, aiming to match the quality of real data. We propose multi-platform topic-based prompting and employ various language models to generate synthetic data from two real datasets, each consisting of posts from three different social media platforms. We assess the lexical and semantic properties of the synthetic data and compare them with those of the real data. Our empirical findings show that using large language models to generate synthetic multi-platform social media data is promising, different language models perform differently in terms of fidelity, and a post-processing approach might be needed for generating high-fidelity synthetic datasets for research. In addition to the empirical evaluation of three state of the art large language models, our contributions include new fidelity metrics specific to multi-platform social media datasets.
• 摘要：社交媒体数据集对于研究各种主题的研究至关重要，例如虚假信息，影响操作，仇恨言论检测或影响者的营销实践。但是，由于成本和平台限制，对社交媒体数据集的访问通常受到限制。获取跨越多个平台的数据集，这对于理解数字生态系统至关重要，这尤其具有挑战性。本文探讨了大语言模型在跨多个平台上创建词汇和语义相关的社交媒体数据集的潜力，旨在匹配真实数据的质量。我们提出了基于多平台主题的提示，并采用各种语言模型从两个真实数据集生成综合数据，每个数据都由来自三个不同社交媒体平台的帖子组成。我们评估合成数据的词汇和语义特性，并将其与真实数据的词汇和语义特性进行比较。我们的经验发现表明，使用大型语言模型生成合成的多平台社交媒体数据是有希望的，不同的语言模型在忠诚度方面的表现不同，并且可能需要采用后处理方法来生成高效率的合成数据集进行研究。除了对三种艺术状态的大型语言模型的经验评估外，我们的贡献还包括针对多平台社交媒体数据集的新的保真度指标。

Title: Enhancing ML Model Interpretability: Leveraging Fine-Tuned Large Language Models for Better Understanding of AI

• Authors: Jonas Bokstaller, Julia Altheimer, Julian Dormehl, Alina Buss, Jasper Wiltfang, Johannes Schneider, Maximilian Röglinger
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.02859
• Pdf URL: https://arxiv.org/pdf/2505.02859
• Copy Paste: [[2505.02859]] Enhancing ML Model Interpretability: Leveraging Fine-Tuned Large Language Models for Better Understanding of AI(https://arxiv.org/abs/2505.02859)
• Keywords: language model, llm, chat
• Abstract: Across various sectors applications of eXplainableAI (XAI) gained momentum as the increasing black-boxedness of prevailing Machine Learning (ML) models became apparent. In parallel, Large Language Models (LLMs) significantly developed in their abilities to understand human language and complex patterns. By combining both, this paper presents a novel reference architecture for the interpretation of XAI through an interactive chatbot powered by a fine-tuned LLM. We instantiate the reference architecture in the context of State-of-Health (SoH) prediction for batteries and validate its design in multiple evaluation and demonstration rounds. The evaluation indicates that the implemented prototype enhances the human interpretability of ML, especially for users with less experience with XAI.
• 摘要：随着盛行的机器学习（ML）模型的越来越多的黑色框，在各个部门的应用中都获得了动力。同时，大型语言模型（LLMS）在其能力中显着发展，以理解人类语言和复杂模式。通过将两者结合起来，本文通过通过微型LLM驱动的交互式聊天机器人来介绍一种新颖的参考体系结构，以解释XAI。我们在电池最先进（SOH）预测的背景下实例化参考体系结构，并在多次评估和示范回合中验证其设计。评估表明，实施的原型增强了ML的人类解释性，尤其是对于XAI经验较低的用户。

Title: Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs

• Authors: Haoming Yang, Ke Ma, Xiaojun Jia, Yingfei Sun, Qianqian Xu, Qingming Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.02862
• Pdf URL: https://arxiv.org/pdf/2505.02862
• Copy Paste: [[2505.02862]] Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs(https://arxiv.org/abs/2505.02862)
• Keywords: language model, llm, prompt
• Abstract: Despite the remarkable performance of Large Language Models (LLMs), they remain vulnerable to jailbreak attacks, which can compromise their safety mechanisms. Existing studies often rely on brute-force optimization or manual design, failing to uncover potential risks in real-world scenarios. To address this, we propose a novel jailbreak attack framework, ICRT, inspired by heuristics and biases in human cognition. Leveraging the simplicity effect, we employ cognitive decomposition to reduce the complexity of malicious prompts. Simultaneously, relevance bias is utilized to reorganize prompts, enhancing semantic alignment and inducing harmful outputs effectively. Furthermore, we introduce a ranking-based harmfulness evaluation metric that surpasses the traditional binary success-or-failure paradigm by employing ranking aggregation methods such as Elo, HodgeRank, and Rank Centrality to comprehensively quantify the harmfulness of generated content. Experimental results show that our approach consistently bypasses mainstream LLMs' safety mechanisms and generates high-risk content, providing insights into jailbreak attack risks and contributing to stronger defense strategies.
• 摘要：尽管大语言模型（LLM）表现出色，但它们仍然容易受到越狱攻击的影响，这可能会损害其安全机制。现有的研究通常依赖于蛮力优化或手动设计，在现实情况下未发现潜在的风险。为了解决这个问题，我们提出了一个新颖的越狱攻击框架ICRT，灵感来自人类认知中的启发式和偏见。利用简单效应，我们采用认知分解来减少恶意提示的复杂性。同时，相关性偏差可用于重组提示，增强语义对准并有效诱导有害输出。此外，我们引入了一种基于排名的有害性评估度量指标，该指标通过采用排名汇总方法（例如ELO，Hodgerank和等级中心地位）超过了传统的二进制成功或失败范式，以全面量化生成内容的有害性。实验结果表明，我们的方法始终绕过主流LLM的安全机制并产生高风险的内容，提供越狱攻击风险的见解，并为更强大的防御策略做出贡献。

Title: Accelerating Large Language Model Reasoning via Speculative Search

• Authors: Zhihai Wang, Jie Wang, Jilai Pan, Xilin Xia, Huiling Zhen, Mingxuan Yuan, Jianye Hao, Feng Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.02865
• Pdf URL: https://arxiv.org/pdf/2505.02865
• Copy Paste: [[2505.02865]] Accelerating Large Language Model Reasoning via Speculative Search(https://arxiv.org/abs/2505.02865)
• Keywords: language model, llm
• Abstract: Tree-search-based reasoning methods have significantly enhanced the reasoning capability of large language models (LLMs) by facilitating the exploration of multiple intermediate reasoning steps, i.e., thoughts. However, these methods suffer from substantial inference latency, as they have to generate numerous reasoning thoughts, severely limiting LLM applicability. To address this challenge, we propose a novel Speculative Search (SpecSearch) framework that significantly accelerates LLM reasoning by optimizing thought generation. Specifically, SpecSearch utilizes a small model to strategically collaborate with a large model at both thought and token levels, efficiently generating high-quality reasoning thoughts. The major pillar of SpecSearch is a novel quality-preserving rejection mechanism, which effectively filters out thoughts whose quality falls below that of the large model's outputs. Moreover, we show that SpecSearch preserves comparable reasoning quality to the large model. Experiments on both the Qwen and Llama models demonstrate that SpecSearch significantly outperforms state-of-the-art approaches, achieving up to 2.12$\times$ speedup with comparable reasoning quality.
• 摘要：基于树搜索的推理方法通过促进探索多个中间推理步骤，即思想，从而显着增强了大语言模型（LLMS）的推理能力。但是，这些方法具有大量的推断潜伏期，因为它们必须产生许多推理思想，从而严重限制了LLM的适用性。为了应对这一挑战，我们提出了一个新颖的投机搜索（SpecSearch）框架，该框架通过优化思想产生来显着加速LLM推理。具体而言，SpecSearch利用一个小型模型在思想和代币级别上与大型模型进行战略性合作，从而有效地产生了高质量的推理思想。 SpecSearch的主要支柱是一种新颖的质量保护拒绝机制，该机制有效地滤除了质量低于大型输出的思想。此外，我们表明SpecSearch保留了与大型模型相当的推理质量。 QWEN和LLAMA模型的实验表明，SpecSearch的表现明显优于最先进的方法，并具有可比的推理质量的2.12 $ \ times $速度。

Title: Decoding Open-Ended Information Seeking Goals from Eye Movements in Reading

• Authors: Cfir Avraham Hadar, Omer Shubi, Yoav Meiri, Yevgeni Berzak
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.02872
• Pdf URL: https://arxiv.org/pdf/2505.02872
• Copy Paste: [[2505.02872]] Decoding Open-Ended Information Seeking Goals from Eye Movements in Reading(https://arxiv.org/abs/2505.02872)
• Keywords: llm
• Abstract: When reading, we often have specific information that interests us in a text. For example, you might be reading this paper because you are curious about LLMs for eye movements in reading, the experimental design, or perhaps you only care about the question ``but does it work?''. More broadly, in daily life, people approach texts with any number of text-specific goals that guide their reading behavior. In this work, we ask, for the first time, whether open-ended reading goals can be automatically decoded from eye movements in reading. To address this question, we introduce goal classification and goal reconstruction tasks and evaluation frameworks, and use large-scale eye tracking for reading data in English with hundreds of text-specific information seeking tasks. We develop and compare several discriminative and generative multimodal LLMs that combine eye movements and text for goal classification and goal reconstruction. Our experiments show considerable success on both tasks, suggesting that LLMs can extract valuable information about the readers' text-specific goals from eye movements.
• 摘要：阅读时，我们经常有特定的信息使我们在文本中感兴趣。例如，您可能正在阅读本文，因为您对LLM对阅读，实验设计的眼睛动作感到好奇，或者您只关心这个问题``但是它起作用吗？''。更广泛地说，在日常生活中，人们使用任何数量的特定文本目标来指导其阅读行为。在这项工作中，我们首次询问是否可以从阅读中的眼动动作中自动解码开放式阅读目标。为了解决这个问题，我们介绍了目标分类和目标重建任务和评估框架，并使用大规模的注视跟踪来阅读英语数据，并使用数百个特定于文本的信息寻求任务。我们开发和比较了几种歧视性和生成性的多模式LLM，它们结合了眼动和文本，以进行目标分类和目标重建。我们的实验在这两个任务上都显示出很大的成功，这表明LLM可以从眼动中提取有关读者特定于文本目标的有价值信息。

Title: Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis

• Authors: Albérick Euraste Djiré, Abdoul Kader Kaboré, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03019
• Pdf URL: https://arxiv.org/pdf/2505.03019
• Copy Paste: [[2505.03019]] Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis(https://arxiv.org/abs/2505.03019)
• Keywords: language model, gpt, llm
• Abstract: While Large Language Models (LLMs) achieve remarkable performance through training on massive datasets, they can exhibit concerning behaviors such as verbatim reproduction of training data rather than true generalization. This memorization phenomenon raises significant concerns about data privacy, intellectual property rights, and the reliability of model evaluations. This paper introduces PEARL, a novel approach for detecting memorization in LLMs. PEARL assesses how sensitive an LLM's performance is to input perturbations, enabling memorization detection without requiring access to the model's internals. We investigate how input perturbations affect the consistency of outputs, enabling us to distinguish between true generalization and memorization. Our findings, following extensive experiments on the Pythia open model, provide a robust framework for identifying when the model simply regurgitates learned information. Applied on the GPT 4o models, the PEARL framework not only identified cases of memorization of classic texts from the Bible or common code from HumanEval but also demonstrated that it can provide supporting evidence that some data, such as from the New York Times news articles, were likely part of the training data of a given model.
• 摘要：尽管大型语言模型（LLMS）通过在大规模数据集上的培训中实现出色的性能，但它们可以展示有关诸如逐字化训练数据的行为，而不是真正的概括。这种记忆现象引起了人们对数据隐私，知识产权和模型评估的可靠性的重大关注。本文介绍了Pearl，这是一种用于检测LLMS中记忆的新方法。 PEARL评估了LLM对输入扰动的敏感性敏感，从而无需访问模型的内部设备，从而实现了记忆检测。我们研究了输入扰动如何影响输出的一致性，从而使我们能够区分真正的概括和记忆。我们的发现是在毕达斯（Pythia）开放模型的广泛实验之后，为识别何时简单地反省学习信息的框架提供了一个强大的框架。 Pearl框架应用于GPT 4O模型，不仅确定了《圣经》中经典文本或Humaneval的常见代码的记忆案例，而且还证明它可以提供支持证据，表明某些数据（例如《纽约时报》新闻文章中的某些数据）很可能是给定模型训练数据的一部分。

Title: UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Output

• Authors: Sicong Huang, Jincheng He, Shiyuan Huang, Karthik Raja Anandan, Arkajyoti Chakraborty, Ian Lane
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03030
• Pdf URL: https://arxiv.org/pdf/2505.03030
• Copy Paste: [[2505.03030]] UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Output(https://arxiv.org/abs/2505.03030)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Hallucinations pose a significant challenge for large language models when answering knowledge-intensive queries. As LLMs become more widely adopted, it is crucial not only to detect if hallucinations occur but also to pinpoint exactly where in the LLM output they occur. SemEval 2025 Task 3, Mu-SHROOM: Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes, is a recent effort in this direction. This paper describes the UCSC system submission to the shared Mu-SHROOM task. We introduce a framework that first retrieves relevant context, next identifies false content from the answer, and finally maps them back to spans in the LLM output. The process is further enhanced by automatically optimizing prompts. Our system achieves the highest overall performance, ranking #1 in average position across all languages. We release our code and experiment results.
• 摘要：回答知识密集的查询时，幻觉对大语言模型构成了重大挑战。随着LLM的采用更广泛，至关重要的是，要检测是否发生幻觉，而且要精确地指出在LLM输出中发生的位置。 Semeval 2025 Task 3，MU Shroum：关于幻觉和相关可观察到的过度错误的多语言共享任务，这是在这个方向上的最新努力。本文介绍了将UCSC系统提交到共享的MU Shroom任务中。我们介绍了一个首先检索相关上下文的框架，接下来从答案中标识错误的内容，最后将它们映射回LLM输出中的跨度。通过自动优化提示进一步增强该过程。我们的系统取得了最高的整体性能，在所有语言中平均位置排名第一。我们发布代码和实验结果。

Title: Teaching Models to Understand (but not Generate) High-risk Data

• Authors: Ryan Wang, Matthew Finlayson, Luca Soldaini, Swabha Swayamdipta, Robin Jia
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.03052
• Pdf URL: https://arxiv.org/pdf/2505.03052
• Copy Paste: [[2505.03052]] Teaching Models to Understand (but not Generate) High-risk Data(https://arxiv.org/abs/2505.03052)
• Keywords: language model
• Abstract: Language model developers typically filter out high-risk content -- such as toxic or copyrighted text -- from their pre-training data to prevent models from generating similar outputs. However, removing such data altogether limits models' ability to recognize and appropriately respond to harmful or sensitive content. In this paper, we introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm through which models learn to understand high-risk data without learning to generate it. Instead of uniformly applying the next-token prediction loss, SLUNG selectively avoids incentivizing the generation of high-risk tokens while ensuring they remain within the model's context window. As the model learns to predict low-risk tokens that follow high-risk ones, it is forced to understand the high-risk content. Through our experiments, we show that SLUNG consistently improves models' understanding of high-risk data (e.g., ability to recognize toxic content) without increasing its generation (e.g., toxicity of model responses). Overall, our SLUNG paradigm enables models to benefit from high-risk text that would otherwise be filtered out.
• 摘要：语言模型开发人员通常会从其预训练数据中滤除高风险内容（例如有毒或受版权保护的文本），以防止模型产生相似的输出。但是，删除此类数据完全限制了模型识别和适当响应有害或敏感内容的能力。在本文中，我们引入了选择性损失以理解但不产生（Slung），这是一种预训练范式，模型通过该范式学会理解高风险数据而无需学习生成它。 Slung并没有统一地应用下一步的预测损失，而是有选择地避免激励高风险令牌的产生，同时确保它们保留在模型的上下文窗口中。随着模型学会预测遵循高风险的低风险令牌，它被迫了解高风险的内容。通过我们的实验，我们表明Slung始终提高模型对高风险数据的理解（例如，识别有毒含量的能力）而无需增加其产生（例如，模型响应的毒性）。总体而言，我们的倾斜范式使模型能够从否则会过滤掉的高风险文本中受益。

Title: Developing A Framework to Support Human Evaluation of Bias in Generated Free Response Text

• Authors: Jennifer Healey, Laurie Byrum, Md Nadeem Akhtar, Surabhi Bhargava, Moumita Sinha
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03053
• Pdf URL: https://arxiv.org/pdf/2505.03053
• Copy Paste: [[2505.03053]] Developing A Framework to Support Human Evaluation of Bias in Generated Free Response Text(https://arxiv.org/abs/2505.03053)
• Keywords: llm, prompt
• Abstract: LLM evaluation is challenging even the case of base models. In real world deployments, evaluation is further complicated by the interplay of task specific prompts and experiential context. At scale, bias evaluation is often based on short context, fixed choice benchmarks that can be rapidly evaluated, however, these can lose validity when the LLMs' deployed context differs. Large scale human evaluation is often seen as too intractable and costly. Here we present our journey towards developing a semi-automated bias evaluation framework for free text responses that has human insights at its core. We discuss how we developed an operational definition of bias that helped us automate our pipeline and a methodology for classifying bias beyond multiple choice. We additionally comment on how human evaluation helped us uncover problematic templates in a bias benchmark.
• 摘要：LLM评估即使是基本模型的情况，也在挑战。在现实世界的部署中，由任务特定提示和体验环境的相互作用更加复杂。在大规模上，偏见评估通常基于短上下文，可以快速评估的固定选择基准测试基准，但是，当LLMS的部署上下文不同时，这些基准可能会失去有效性。大规模的人类评估通常被视为太棘手且昂贵。在这里，我们介绍了为自由文本响应开发半自动偏见评估框架的旅程，该框架具有人类的见解。我们讨论了如何开发偏见的操作定义，该定义有助于我们自动化管道和一种将偏见分类超出多项选择的方法。我们还评论了人类评估如何帮助我们在偏见基准中发现有问题的模板。

Title: Improving Model Alignment Through Collective Intelligence of Open-Source LLMS

• Authors: Junlin Wang, Roy Xie, Shang Zhu, Jue Wang, Ben Athiwaratkun, Bhuwan Dhingra, Shuaiwen Leon Song, Ce Zhang, James Zou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03059
• Pdf URL: https://arxiv.org/pdf/2505.03059
• Copy Paste: [[2505.03059]] Improving Model Alignment Through Collective Intelligence of Open-Source LLMS(https://arxiv.org/abs/2505.03059)
• Keywords: language model, gpt, llm, agent
• Abstract: Building helpful and harmless large language models (LLMs) requires effective model alignment approach based on human instructions and feedback, which necessitates high-quality human-labeled data. Constructing such datasets is often expensive and hard to scale, and may face potential limitations on diversity and generalization. To address these challenges, we introduce Mixture of Agents Alignment (MoAA), that leverages the collective strengths of various language models to provide high-quality data for model alignment. By employing MoAA, we enhance both supervised fine-tuning and preference optimization, leading to improved performance compared to using a single model alone to generate alignment data (e.g. using GPT-4o alone). Evaluation results show that our approach can improve win rate of LLaMA-3.1-8B-Instruct from 19.5 to 48.3 on Arena-Hard and from 22.33 to 57.23 on AlpacaEval2, highlighting a promising direction for model alignment through this new scalable and diverse synthetic data recipe. Furthermore, we demonstrate that MoAA enables a self-improvement pipeline, where models finetuned on MoA-generated data surpass their own initial capabilities, providing evidence that our approach can push the frontier of open-source LLMs without reliance on stronger external supervision. Data and code will be released.
• 摘要：建立有用且无害的大语言模型（LLM）需要基于人类的说明和反馈的有效模型对齐方式，这需要高质量的人类标记数据。构建此类数据集通常很昂贵且难以扩展，并且可能面临对多样性和概括的潜在局限性。为了应对这些挑战，我们介绍了代理对齐（MOAA）的混合物，以利用各种语言模型的集体优势来提供高质量的数据以进行模型对齐。通过使用MOAA，我们可以增强监督的微调和偏好优化，从而相比单独使用单个模型来生成对齐数据（例如，单独使用GPT-4O），从而提高了性能。评估结果表明，我们的方法可以在Alpacaeval2上从竞技场上的19.5到48.3提高Llama-3.1-8b教学的胜率，从22.33提高到22.33到57.23，突出了通过这项新的可扩展和多样的合成数据回收调整的模型对齐的有希望的指导。此外，我们证明了MOAA可以实现自我完善的管道，在MOA生成的数据上进行了固定的模型超过了他们自己的初始功能，提供了证据，表明我们的方法可以推动开放式LLM的前沿，而不会依赖更强大的外部监督。数据和代码将发布。

Title: Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback

• Authors: Shijing Zhu, Zhuang Chen, Guanqun Bi, Binghang Li, Yaxi Deng, Dazhen Wan, Libiao Peng, Xiyao Xiao, Rongsheng Zhang, Tangjie Lv, Zhipeng Hu, FangFang Li, Minlie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03293
• Pdf URL: https://arxiv.org/pdf/2505.03293
• Copy Paste: [[2505.03293]] Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback(https://arxiv.org/abs/2505.03293)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown promise in providing scalable mental health support, while evaluating their counseling capability remains crucial to ensure both efficacy and safety. Existing evaluations are limited by the static assessment that focuses on knowledge tests, the single perspective that centers on user experience, and the open-loop framework that lacks actionable feedback. To address these issues, we propose {\Psi}-Arena, an interactive framework for comprehensive assessment and optimization of LLM-based counselors, featuring three key characteristics: (1) Realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients, (2) Tripartite evaluation that integrates assessments from the client, counselor, and supervisor perspectives, and (3) Closed-loop optimization that iteratively improves LLM counselors using diagnostic feedback. Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives. Moreover, reflection-based optimization results in up to a 141% improvement in counseling performance. We hope PsychoArena provides a foundational resource for advancing reliable and human-aligned LLM applications in mental healthcare.
• 摘要：大型语言模型（LLMS）在提供可扩展的心理健康支持方面表现出了希望，同时评估其咨询能力对于确保疗效和安全性仍然至关重要。现有评估受到静态评估的限制，该评估的重点是知识测试，以用户体验为中心的单一观点以及缺乏可行反馈的开环框架。 To address these issues, we propose {\Psi}-Arena, an interactive framework for comprehensive assessment and optimization of LLM-based counselors, featuring three key characteristics: (1) Realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients, (2) Tripartite evaluation that integrates assessments from the client, counselor, and supervisor perspectives, and (3)闭环优化，可以使用诊断反馈来迭代地改善LLM辅导员。在不同的现实情况和评估观点中，八个最先进的LLM的实验显示出显着的性能差异。此外，基于反射的优化可将咨询绩效提高141％。我们希望PsychoArena为在心理保健中推进可靠和人类一致的LLM应用提供基本资源。

Title: Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation

• Authors: Junyu Ma, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03320
• Pdf URL: https://arxiv.org/pdf/2505.03320
• Copy Paste: [[2505.03320]] Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation(https://arxiv.org/abs/2505.03320)
• Keywords: long context, prompt, chain-of-thought
• Abstract: Mamba's theoretical infinite-context potential is limited in practice when sequences far exceed training lengths. This work explores unlocking Mamba's long-context memory ability by a simple-yet-effective method, Recall with Reasoning (RwR), by distilling chain-of-thought (CoT) summarization from a teacher model. Specifically, RwR prepends these summarization as CoT prompts during fine-tuning, teaching Mamba to actively recall and reason over long contexts. Experiments on LONGMEMEVAL and HELMET show RwR boosts Mamba's long-context performance against comparable Transformer/hybrid baselines under similar pretraining conditions, while preserving short-context capabilities, all without architectural changes.
• 摘要：当序列远远超过训练长度时，MAMBA的理论无限秘密潜力在实践中受到限制。这项工作通过一种简单的方法来探索Mamba的长篇小说记忆能力，通过将思想链（COT）摘要从教师模型中提炼出来。具体而言，RWR将这些摘要作为COT在微调过程中提示，教导Mamba积极回忆并在漫长的背景下进行推理。在Longmemeval和头盔上进行的实验显示，RWR在相似的预处理条件下对Mamba的长篇小写性能对可比的变压器/混合基线的长期性能提升，同时保留了短篇小说能力，所有这些都没有建筑变化。

Title: Lightweight Clinical Decision Support System using QLoRA-Fine-Tuned LLMs and Retrieval-Augmented Generation

• Authors: Mohammad Shoaib Ansari, Mohd Sohail Ali Khan, Shubham Revankar, Aditya Varma, Anil S. Mokhade
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03406
• Pdf URL: https://arxiv.org/pdf/2505.03406
• Copy Paste: [[2505.03406]] Lightweight Clinical Decision Support System using QLoRA-Fine-Tuned LLMs and Retrieval-Augmented Generation(https://arxiv.org/abs/2505.03406)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: This research paper investigates the application of Large Language Models (LLMs) in healthcare, specifically focusing on enhancing medical decision support through Retrieval-Augmented Generation (RAG) integrated with hospital-specific data and fine-tuning using Quantized Low-Rank Adaptation (QLoRA). The system utilizes Llama 3.2-3B-Instruct as its foundation model. By embedding and retrieving context-relevant healthcare information, the system significantly improves response accuracy. QLoRA facilitates notable parameter efficiency and memory optimization, preserving the integrity of medical information through specialized quantization techniques. Our research also shows that our model performs relatively well on various medical benchmarks, indicating that it can be used to make basic medical suggestions. This paper details the system's technical components, including its architecture, quantization methods, and key healthcare applications such as enhanced disease prediction from patient symptoms and medical history, treatment suggestions, and efficient summarization of complex medical reports. We touch on the ethical considerations-patient privacy, data security, and the need for rigorous clinical validation-as well as the practical challenges of integrating such systems into real-world healthcare workflows. Furthermore, the lightweight quantized weights ensure scalability and ease of deployment even in low-resource hospital environments. Finally, the paper concludes with an analysis of the broader impact of LLMs on healthcare and outlines future directions for LLMs in medical settings.
• 摘要：本研究论文研究了大型语言模型（LLM）在医疗保健中的应用，专门针对通过与医院特异性数据集成的检索增强发电（RAG）来增强医疗决策支持，并使用量化的低级适应性（Qlora）进行了微调。该系统利用Llama 3.2-3B教学作为其基础模型。通过嵌入和检索与上下文相关的医疗保健信息，该系统可显着提高响应准确性。 Qlora促进了显着的参数效率和内存优化，从而通过专门的量化技术来保存医疗信息的完整性。我们的研究还表明，我们的模型在各种医疗基准上的性能相对较好，表明它可以用于提出基本的医疗建议。本文详细介绍了该系统的技术组成部分，包括其架构，量化方法和关键医疗保健应用，例如从患者症状和病史提高疾病预测，治疗建议以及对复杂医疗报告的有效汇总。我们涉及适合患者隐私，数据安全以及对严格临床验证的需求，以及将这些系统集成到现实世界中的医疗保健工作流中的实际挑战。此外，轻巧的量化重量可以确保可扩展性和易于部署，即使在低资源医院环境中也是如此。最后，本文以分析LLM对医疗保健的更广泛影响的结论，并概述了医疗环境中LLM的未来方向。

Title: MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks

• Authors: Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, Farah E. Shamout
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2505.03427
• Pdf URL: https://arxiv.org/pdf/2505.03427
• Copy Paste: [[2505.03427]] MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks(https://arxiv.org/abs/2505.03427)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated significant promise for various applications in healthcare. However, their efficacy in the Arabic medical domain remains unexplored due to the lack of high-quality domain-specific datasets and benchmarks. This study introduces MedArabiQ, a novel benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and including multiple choice questions, fill-in-the-blank, and patient-doctor question answering. We first constructed the dataset using past medical exams and publicly available datasets. We then introduced different modifications to evaluate various LLM capabilities, including bias mitigation. We conducted an extensive evaluation with five state-of-the-art open-source and proprietary LLMs, including GPT-4o, Claude 3.5-Sonnet, and Gemini 1.5. Our findings highlight the need for the creation of new high-quality benchmarks that span different languages to ensure fair deployment and scalability of LLMs in healthcare. By establishing this benchmark and releasing the dataset, we provide a foundation for future research aimed at evaluating and enhancing the multilingual capabilities of LLMs for the equitable use of generative AI in healthcare.
• 摘要：大型语言模型（LLM）对医疗保健中的各种应用表现出了巨大的希望。但是，由于缺乏高质量的域特异性数据集和基准，它们在阿拉伯医学领域的功效仍未得到探索。这项研究介绍了Medarabiq，这是一个新型的基准数据集，该数据集由七个阿拉伯医学任务组成，涵盖了多个专业，包括多项选择问题，填空和患者doctoctor的问题回答。我们首先使用过去的体检和公开可用数据集构建了数据集。然后，我们引入了不同的修改，以评估各种LLM功能，包括缓解偏差。我们通过五个最先进的开源和专有LLM进行了广泛的评估，包括GPT-4O，Claude 3.5-Sonnet和Gemini 1.5。我们的发现凸显了建立新的高质量基准测试的需求，这些基准涵盖了不同的语言，以确保LLM在医疗保健中的公平部署和可扩展性。通过建立此基准并释放数据集，我们为未来的研究提供了基础，旨在评估和增强LLMS的多语言能力，以公平地使用生成AI在医疗保健中。

Title: An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation

• Authors: Matan Orbach, Ohad Eytan, Benjamin Sznajder, Ariel Gera, Odellia Boni, Yoav Kantor, Gal Bloch, Omri Levy, Hadas Abraham, Nitzan Barzilay, Eyal Shnarch, Michael E. Factor, Shila Ofek-Koifman, Paula Ta-Shma, Assaf Toledo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.03452
• Pdf URL: https://arxiv.org/pdf/2505.03452
• Copy Paste: [[2505.03452]] An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation(https://arxiv.org/abs/2505.03452)
• Keywords: retrieval augmented generation, retrieval-augmented generation
• Abstract: Finding the optimal Retrieval-Augmented Generation (RAG) configuration for a given use case can be complex and expensive. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To address this gap, we present a comprehensive study involving 5 HPO algorithms over 5 datasets from diverse domains, including a new one collected for this work on real-world product documentation. Our study explores the largest HPO search space considered to date, with two optimized evaluation metrics. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with iterative random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing models first is preferable to the prevalent practice of optimizing sequentially according to the RAG pipeline order.
• 摘要：找到给定用例的最佳检索效果生成（RAG）配置可能很复杂且昂贵。在这一挑战中，抹布超参数优化（HPO）的框架最近出现了，但其有效性并未受到严格的标准。为了解决这一差距，我们提出了一项全面的研究，其中涉及来自不同领域的5个数据集上的5种HPO算法，其中包括针对现实世界产品文档的这项工作收集的新算法。我们的研究探索了迄今为止考虑的最大的HPO搜索空间，并具有两个优化的评估指标。对结果的分析表明，可以有效地进行破布HPO，无论是贪婪还是迭代的随机搜索，并且可以显着提高所有数据集的抹布性能。对于贪婪的HPO方法，我们表明优化模型首先比根据RAG管道顺序进行顺序优化的普遍实践更可取。

Title: Uncertainty-Aware Large Language Models for Explainable Disease Diagnosis

• Authors: Shuang Zhou, Jiashuo Wang, Zidu Xu, Song Wang, David Brauer, Lindsay Welton, Jacob Cogan, Yuen-Hei Chung, Lei Tian, Zaifu Zhan, Yu Hou, Mingquan Lin, Genevieve B. Melton, Rui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03467
• Pdf URL: https://arxiv.org/pdf/2505.03467
• Copy Paste: [[2505.03467]] Uncertainty-Aware Large Language Models for Explainable Disease Diagnosis(https://arxiv.org/abs/2505.03467)
• Keywords: language model, llm
• Abstract: Explainable disease diagnosis, which leverages patient information (e.g., signs and symptoms) and computational models to generate probable diagnoses and reasonings, offers clear clinical values. However, when clinical notes encompass insufficient evidence for a definite diagnosis, such as the absence of definitive symptoms, diagnostic uncertainty usually arises, increasing the risk of misdiagnosis and adverse outcomes. Although explicitly identifying and explaining diagnostic uncertainties is essential for trustworthy diagnostic systems, it remains under-explored. To fill this gap, we introduce ConfiDx, an uncertainty-aware large language model (LLM) created by fine-tuning open-source LLMs with diagnostic criteria. We formalized the task and assembled richly annotated datasets that capture varying degrees of diagnostic ambiguity. Evaluating ConfiDx on real-world datasets demonstrated that it excelled in identifying diagnostic uncertainties, achieving superior diagnostic performance, and generating trustworthy explanations for diagnoses and uncertainties. To our knowledge, this is the first study to jointly address diagnostic uncertainty recognition and explanation, substantially enhancing the reliability of automatic diagnostic systems.
• 摘要：可解释的疾病诊断，该疾病诊断利用患者信息（例如体征和症状）和计算模型产生可能的诊断和推理，可提供明确的临床价值。但是，当临床注释不足以证明明确诊断的证据不足时，例如缺乏明确的症状，通常会出现诊断不确定性，从而增加了误诊和不良后果的风险。尽管明确识别和解释诊断不确定性对于值得信赖的诊断系统至关重要，但它仍然不足。为了填补这一空白，我们介绍了Concidx，这是一种不确定性的大语言模型（LLM），该模型（LLM）由用诊断标准进行微调开源LLM创建。我们正式化了任务，并组装了丰富的注释数据集，这些数据集捕获了不同程度的诊断歧义。在现实世界数据集上评估Confidx表明，它在识别诊断不确定性，达到卓越的诊断性能以及为诊断和不确定性中产生值得信赖的解释方面表现出色。据我们所知，这是第一个共同解决诊断不确定性识别和解释的研究，从而大大提高了自动诊断系统的可靠性。

Title: Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models

• Authors: Bin Yu, Hang Yuan, Yuliang Wei, Bailing Wang, Weizhen Qi, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03469
• Pdf URL: https://arxiv.org/pdf/2505.03469
• Copy Paste: [[2505.03469]] Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models(https://arxiv.org/abs/2505.03469)
• Keywords: language model, chain-of-thought
• Abstract: Recent advances in large language models have demonstrated that Supervised Fine-Tuning (SFT) with Chain-of-Thought (CoT) reasoning data distilled from large reasoning models (e.g., DeepSeek R1) can effectively transfer reasoning capabilities to non-reasoning models. However, models fine-tuned with this approach inherit the "overthinking" problem from teacher models, producing verbose and redundant reasoning chains during inference. To address this challenge, we propose \textbf{L}ong-\textbf{S}hort Chain-of-Thought \textbf{Mixture} \textbf{S}upervised \textbf{F}ine-\textbf{T}uning (\textbf{LS-Mixture SFT}), which combines long CoT reasoning dataset with their short counterparts obtained through structure-preserved rewriting. Our experiments demonstrate that models trained using the LS-Mixture SFT method, compared to those trained with direct SFT, achieved an average accuracy improvement of 2.3\% across various benchmarks while substantially reducing model response length by approximately 47.61\%. This work offers an approach to endow non-reasoning models with reasoning capabilities through supervised fine-tuning while avoiding the inherent overthinking problems inherited from teacher models, thereby enabling efficient reasoning in the fine-tuned models.
• 摘要：大型语言模型的最新进展表明，从大型推理模型（例如，DeepSeek R1）蒸馏出的经过三链（COT）推理数据的监督（SFT）可以有效地将推理能力传递给非调查模型。但是，使用这种方法进行微调的模型继承了教师模型中的“过度思考”问题，从而在推断过程中产生了冗长和冗余的推理链。为了应对这一挑战，我们提出\ textbf {l} ong- \ textbf {s} hort theark of thought \ textbf {混合物} \ textbf {s} upervised \ textbf {f} ine- \ \ textbf {f textbf {t} textbf {他们通过结构保存的重写获得的简短对应物。我们的实验表明，与经过直接SFT训练的模型相比，使用LS混合SFT方法训练的模型在各种基准测试中的平均准确度提高了2.3 \％，而基本上将模型响应长度降低了约47.61 \％。这项工作提供了一种方法，可以通过监督微调来赋予非争议模型具有推理能力，同时避免了从教师模型中继承的固有的过度思考问题，从而在微调模型中实现了有效的推理。

Title: Evaluation of LLMs on Long-tail Entity Linking in Historical Documents

• Authors: Marta Boscariol, Luana Bulla, Lia Draetta, Beatrice Fiumanò, Emanuele Lenzi, Leonardo Piano
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03473
• Pdf URL: https://arxiv.org/pdf/2505.03473
• Copy Paste: [[2505.03473]] Evaluation of LLMs on Long-tail Entity Linking in Historical Documents(https://arxiv.org/abs/2505.03473)
• Keywords: gpt, llm
• Abstract: Entity Linking (EL) plays a crucial role in Natural Language Processing (NLP) applications, enabling the disambiguation of entity mentions by linking them to their corresponding entries in a reference knowledge base (KB). Thanks to their deep contextual understanding capabilities, LLMs offer a new perspective to tackle EL, promising better results than traditional methods. Despite the impressive generalization capabilities of LLMs, linking less popular, long-tail entities remains challenging as these entities are often underrepresented in training data and knowledge bases. Furthermore, the long-tail EL task is an understudied problem, and limited studies address it with LLMs. In the present work, we assess the performance of two popular LLMs, GPT and LLama3, in a long-tail entity linking scenario. Using MHERCL v0.1, a manually annotated benchmark of sentences from domain-specific historical texts, we quantitatively compare the performance of LLMs in identifying and linking entities to their corresponding Wikidata entries against that of ReLiK, a state-of-the-art Entity Linking and Relation Extraction framework. Our preliminary experiments reveal that LLMs perform encouragingly well in long-tail EL, indicating that this technology can be a valuable adjunct in filling the gap between head and long-tail EL.
• 摘要：链接（EL）的实体在自然语言处理（NLP）应用中扮演着至关重要的角色，从而使实体的歧义通过将其与参考知识库（KB）中的相应条目联系起来来歧义。借助他们深刻的上下文理解能力，LLMS为解决EL提供了新的观点，比传统方法提供了更好的结果。尽管LLM具有令人印象深刻的概括能力，但将不太受欢迎的长尾实体连接起来仍然具有挑战性，因为这些实体在培训数据和知识库中通常不足。此外，长尾EL任务是一个研究的问题，有限的研究通过LLMS解决了问题。在目前的工作中，我们评估了两个链接场景的长尾实体中两个受欢迎的LLMS GPT和Llama3的性能。使用MHERCL V0.1，这是来自特定于领域的历史文本的句子的手动注释基准，我们定量地比较了LLMS在识别和将实体识别和将其相应的Wikidata条目与Relik相对的selik sear-of-at-Art实体链接和关系提取框架的性能进行比较。我们的初步实验表明，LLM在长尾EL中表现出色，这表明该技术在填补了头部和长尾EL之间的空白方面可能是有价值的辅助手段。

Title: Faster MoE LLM Inference for Extremely Large Models

• Authors: Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo Du, Mengjia Shen, Hai Zhao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.03531
• Pdf URL: https://arxiv.org/pdf/2505.03531
• Copy Paste: [[2505.03531]] Faster MoE LLM Inference for Extremely Large Models(https://arxiv.org/abs/2505.03531)
• Keywords: language model, llm
• Abstract: Sparse Mixture of Experts (MoE) large language models (LLMs) are gradually becoming the mainstream approach for ultra-large-scale models. Existing optimization efforts for MoE models have focused primarily on coarse-grained MoE architectures. With the emergence of DeepSeek Models, fine-grained MoE models are gaining popularity, yet research on them remains limited. Therefore, we want to discuss the efficiency dynamic under different service loads. Additionally, fine-grained models allow deployers to reduce the number of routed experts, both activated counts and total counts, raising the question of how this reduction affects the trade-off between MoE efficiency and performance. Our findings indicate that while deploying MoE models presents greater challenges, it also offers significant optimization opportunities. Reducing the number of activated experts can lead to substantial efficiency improvements in certain scenarios, with only minor performance degradation. Reducing the total number of experts provides limited efficiency gains but results in severe performance degradation. Our method can increase throughput by at least 10\% without any performance degradation. Overall, we conclude that MoE inference optimization remains an area with substantial potential for exploration and improvement.
• 摘要：专家（MOE）大语言模型（LLM）的稀疏混合物逐渐成为超大规模模型的主流方法。现有的MOE模型优化工作主要集中在粗粒的MOE架构上。随着DeepSeek模型的出现，细粒度的Moe模型正在越来越受欢迎，但是对它们的研究仍然有限。因此，我们想讨论不同服务负载下的效率动态。此外，细粒度的模型允许部署者减少激活计数和总数的路由专家数量，从而提出了这种减少如何影响MOE效率和性能之间的权衡问题的问题。我们的发现表明，在部署MOE模型提出了更大的挑战时，它还提供了重要的优化机会。减少活化专家的数量可以导致某些情况下的效率提高，并且只有较小的性能降级。减少专家总数可提供有限的效率提高，但导致严重的性能降解。我们的方法可以将吞吐量提高至少10 \％，而不会降解任何性能。总体而言，我们得出的结论是，MoE推理优化仍然是具有巨大探索和改进潜力的领域。

Title: Say It Another Way: A Framework for User-Grounded Paraphrasing

• Authors: Cléa Chataigner, Rebecca Ma, Prakhar Ganesh, Afaf Taïk, Elliot Creager, Golnoosh Farnadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03563
• Pdf URL: https://arxiv.org/pdf/2505.03563
• Copy Paste: [[2505.03563]] Say It Another Way: A Framework for User-Grounded Paraphrasing(https://arxiv.org/abs/2505.03563)
• Keywords: language model, llm, prompt
• Abstract: Small changes in how a prompt is worded can lead to meaningful differences in the behavior of large language models (LLMs), raising concerns about the stability and reliability of their evaluations. While prior work has explored simple formatting changes, these rarely capture the kinds of natural variation seen in real-world language use. We propose a controlled paraphrasing framework based on a taxonomy of minimal linguistic transformations to systematically generate natural prompt variations. Using the BBQ dataset, we validate our method with both human annotations and automated checks, then use it to study how LLMs respond to paraphrased prompts in stereotype evaluation tasks. Our analysis shows that even subtle prompt modifications can lead to substantial changes in model behavior. These results highlight the need for robust, paraphrase-aware evaluation protocols.
• 摘要：提示的方式的小变化可能导致大语言模型（LLM）行为的有意义差异，从而引起了人们对其评估的稳定性和可靠性的担忧。虽然先前的工作探索了简单的格式化变化，但这些工作很少捕获现实语言使用中看到的自然变化。我们提出了一个受控的释义框架，基于最小语言转换的分类法，以系统地产生自然及时变化。使用烧烤数据集，我们使用人类注释和自动检查验证我们的方法，然后使用它来研究LLM在刻板印象评估任务中如何响应释义的提示。我们的分析表明，即使是微妙的迅速修改也会导致模型行为发生重大变化。这些结果突出了需要强大的释义评估协议的需求。

Title: Towards conversational assistants for health applications: using ChatGPT to generate conversations about heart failure

• Authors: Anuja Tayal, Devika Salunke, Barbara Di Eugenio, Paula G Allen-Meares, Eulalia P Abril, Olga Garcia-Bedoya, Carolyn A Dickens, Andrew D. Boyd
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03675
• Pdf URL: https://arxiv.org/pdf/2505.03675
• Copy Paste: [[2505.03675]] Towards conversational assistants for health applications: using ChatGPT to generate conversations about heart failure(https://arxiv.org/abs/2505.03675)
• Keywords: gpt, prompt, chat
• Abstract: We explore the potential of ChatGPT (3.5-turbo and 4) to generate conversations focused on self-care strategies for African-American heart failure patients -- a domain with limited specialized datasets. To simulate patient-health educator dialogues, we employed four prompting strategies: domain, African American Vernacular English (AAVE), Social Determinants of Health (SDOH), and SDOH-informed reasoning. Conversations were generated across key self-care domains of food, exercise, and fluid intake, with varying turn lengths (5, 10, 15) and incorporated patient-specific SDOH attributes such as age, gender, neighborhood, and socioeconomic status. Our findings show that effective prompt design is essential. While incorporating SDOH and reasoning improves dialogue quality, ChatGPT still lacks the empathy and engagement needed for meaningful healthcare communication.
• 摘要：我们探讨了Chatgpt（3.5-Turbo和4）的潜力，以针对非裔美国人心力衰竭患者的自我护理策略进行对话，这是一个有限的专业数据集。为了模拟患者健康教育者的对话，我们采用了四种提示策略：领域，非裔美国人白话英语（AAVE），卫生社会决定因素（SDOH）和SDOH知名的推理。在食物，运动和液体摄入量的关键自我护理领域之间进行了对话，转弯长度不同（5、10、15），并结合了特定于患者的SDOH属性，例如年龄，性别，邻里和社会经济状况。我们的发现表明，有效的提示设计至关重要。在合并SDOH和推理的同时，Chatgpt仍然缺乏有意义的医疗沟通所需的同理心和参与。

Title: WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch

• Authors: Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, Hongsheng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.03733
• Pdf URL: https://arxiv.org/pdf/2505.03733
• Copy Paste: [[2505.03733]] WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch(https://arxiv.org/abs/2505.03733)
• Keywords: gpt, llm, agent
• Abstract: LLM-based agents have demonstrated great potential in generating and managing code within complex codebases. In this paper, we introduce WebGen-Bench, a novel benchmark designed to measure an LLM-based agent's ability to create multi-file website codebases from scratch. It contains diverse instructions for website generation, created through the combined efforts of human annotators and GPT-4o. These instructions span three major categories and thirteen minor categories, encompassing nearly all important types of web applications. To assess the quality of the generated websites, we use GPT-4o to generate test cases targeting each functionality described in the instructions, and then manually filter, adjust, and organize them to ensure accuracy, resulting in 647 test cases. Each test case specifies an operation to be performed on the website and the expected result after the operation. To automate testing and improve reproducibility, we employ a powerful web-navigation agent to execute tests on the generated websites and determine whether the observed responses align with the expected results. We evaluate three high-performance code-agent frameworks, this http URL, OpenHands, and Aider, using multiple proprietary and open-source LLMs as engines. The best-performing combination, this http URL powered by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting the challenging nature of our benchmark. Additionally, we construct WebGen-Instruct, a training set consisting of 6,667 website-generation instructions. Training Qwen2.5-Coder-32B-Instruct on this http URL trajectories generated from a subset of this training set achieves an accuracy of 38.2\%, surpassing the performance of the best proprietary model.
• 摘要：基于LLM的代理在复杂代码库中生成和管理代码方面具有巨大的潜力。在本文中，我们介绍了Webgen Bench，这是一种新颖的基准测试，旨在衡量基于LLM的代理商从头开始创建多文件网站代码库的能力。它包含通过人类注释者和GPT-4O的共同努力创建的网站生成的不同说明。这些说明涉及三个主要类别和13个小类别，其中包括几乎所有重要类型的Web应用程序类型。为了评估生成的网站的质量，我们使用GPT-4O来生成针对指令中描述的每个功能的测试用例，然后手动过滤，调整和组织它们以确保准确性，从而导致647个测试用例。每个测试用例指定要在网站上执行的操作以及操作后的预期结果。为了自动化测试并提高可重复性，我们采用强大的Web-Navigation代理在生成的网站上执行测试，并确定观察到的响应是否与预期结果保持一致。我们使用多个专有和开源的LLMS作为引擎评估了三个高性能代码代理框架，即开放式URL和Aider。最出色的组合是由DeepSeek-R1提供动力的HTTP URL，在测试用例上仅达到27.8％的准确性，突出了我们基准的挑战性质。此外，我们构建了WebGen-Instruct，这是一个由6,667个网站生成说明组成的培训集。在此HTTP URL轨迹上从该训练集产生的此HTTP URL轨迹上的训练QWEN2.5代码-32b-trinctuct的精度为38.2 \％，超过了最佳专有模型的性能。

Title: VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model

• Authors: Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, Haoyu Cao, Ke Li, Rongrong Ji, Xing Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.03739
• Pdf URL: https://arxiv.org/pdf/2505.03739
• Copy Paste: [[2505.03739]] VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model(https://arxiv.org/abs/2505.03739)
• Keywords: language model
• Abstract: With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.
• 摘要：随着对自然人类互动的日益增长的要求，基于语音的系统受到越来越多的关注，因为语音是每日交流的最常见形式之一。但是，现有的语音模型在流媒体过程中生成第一个音频令牌时仍具有很高的延迟，这为部署带来了重要的瓶颈。为了解决这个问题，我们提出了Vita-Audio，这是一种端到端的大型语音模型，具有快速的音频toxt代币产生。具体而言，我们引入了轻巧的多个跨模式令牌预测（MCTP）模块，该模块在单个模型前传中有效生成多个音频令牌，这不仅可以加速推理，而且还大大降低了流媒体场景中第一个音频的延迟。此外，还探索了四阶段的渐进式训练策略，以实现模型加速度，而言语质量的损失最小。据我们所知，Vita-Audio是第一个能够在第一次向前传球中产生音频输出的多模式大语言模型，从而实现了具有最小延迟的实时对话能力。 Vita-Audio是完全可重现的，仅在开源数据上接受培训。实验结果表明，我们的模型在7b参数量表上达到了3〜5倍的推理速度，但在多个基准测试基准上，自动语音识别（ASR），文本对语音（TTS）（TTS）（TTS）和口头答案（SQA）任务的推理速度显着优于具有相似模型大小的开源模型。