2025-04-28

Title: Optimism, Expectation, or Sarcasm? Multi-Class Hope Speech Detection in Spanish and English

• Authors: Sabur Butt, Fazlourrahman Balouchzahi, Ahmad Imam Amjad, Maaz Amjad, Hector G. Ceballos, Salud Maria Jimenez-Zafra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17974
• Pdf URL: https://arxiv.org/pdf/2504.17974
• Copy Paste: [[2504.17974]] Optimism, Expectation, or Sarcasm? Multi-Class Hope Speech Detection in Spanish and English(https://arxiv.org/abs/2504.17974)
• Keywords: language model, gpt, llm, prompt
• Abstract: Hope is a complex and underexplored emotional state that plays a significant role in education, mental health, and social interaction. Unlike basic emotions, hope manifests in nuanced forms ranging from grounded optimism to exaggerated wishfulness or sarcasm, making it difficult for Natural Language Processing systems to detect accurately. This study introduces PolyHope V2, a multilingual, fine-grained hope speech dataset comprising over 30,000 annotated tweets in English and Spanish. This resource distinguishes between four hope subtypes Generalized, Realistic, Unrealistic, and Sarcastic and enhances existing datasets by explicitly labeling sarcastic instances. We benchmark multiple pretrained transformer models and compare them with large language models (LLMs) such as GPT 4 and Llama 3 under zero-shot and few-shot regimes. Our findings show that fine-tuned transformers outperform prompt-based LLMs, especially in distinguishing nuanced hope categories and sarcasm. Through qualitative analysis and confusion matrices, we highlight systematic challenges in separating closely related hope subtypes. The dataset and results provide a robust foundation for future emotion recognition tasks that demand greater semantic and contextual sensitivity across languages.
• 摘要：希望是一种复杂且毫无创伤的情绪状态，在教育，心理健康和社会互动中起着重要作用。与基本的情绪不同，希望以细微差别的形式体现，从扎根的乐观到夸张的一厢情愿或讽刺，使自然语言处理系统难以准确检测。这项研究介绍了Polyhope V2，这是一种多语言，细颗粒的希望语音数据集，包含30,000多个带注释的英文和西班牙语的带注释的推文。该资源区分了四种希望子类型的广义，现实，不现实和讽刺，并通过明确标记讽刺实例来增强现有数据集。我们基准了多个预处理的变压器模型，并将它们与大型语言模型（LLMS）（例如GPT 4和Llama 3）进行比较，并在零射门和少数弹药方面进行了比较。我们的发现表明，微调的变压器优于基于及时的LLM，尤其是在区分细微的希望类别和讽刺方面。通过定性分析和混乱矩阵，我们重点介绍了分离密切相关的希望亚型的系统挑战。数据集和结果为未来的情感识别任务提供了稳健的基础，该任务需要在语言上更大的语义和上下文敏感性。

Title: Improving LLM Personas via Rationalization with Psychological Scaffolds

• Authors: Brihi Joshi, Xiang Ren, Swabha Swayamdipta, Rik Koncel-Kedziorski, Tim Paek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17993
• Pdf URL: https://arxiv.org/pdf/2504.17993
• Copy Paste: [[2504.17993]] Improving LLM Personas via Rationalization with Psychological Scaffolds(https://arxiv.org/abs/2504.17993)
• Keywords: language model, llm, prompt
• Abstract: Language models prompted with a user description or persona can predict a user's preferences and opinions, but existing approaches to building personas -- based solely on a user's demographic attributes and/or prior judgments -- fail to capture the underlying reasoning behind said user judgments. We introduce PB&J (Psychology of Behavior and Judgments), a framework that improves LLM personas by incorporating rationales of why a user might make specific judgments. These rationales are LLM-generated, and aim to reason about a user's behavior on the basis of their experiences, personality traits or beliefs. This is done using psychological scaffolds -- structured frameworks grounded in theories such as the Big 5 Personality Traits and Primal World Beliefs -- that help provide structure to the generated rationales. Experiments on public opinion and movie preference prediction tasks demonstrate that LLM personas augmented with PB&J rationales consistently outperform methods using only a user's demographics and/or judgments. Additionally, LLM personas constructed using scaffolds describing user beliefs perform competitively with those using human-written rationales.
• 摘要：使用用户描述或角色提示的语言模型可以预测用户的偏好和意见，但是现有的构建角色的方法（仅基于用户的人口统计属性和/或事先判断）未能捕获所述用户判断背后的基本推理。我们介绍了PB＆J（行为和判断心理学），该框架通过结合了为什么用户可以做出特定判断的理由来改善LLM角色。这些理由是LLM生成的，旨在根据用户的经验，个性特征或信念来推理用户的行为。这是使用心理脚手架（基于诸如5个巨大人格特征和原始世界信仰的理论的结构化框架）来完成的，这些框架有助于为产生的理由提供结构。关于公众舆论和电影偏好预测任务的实验表明，使用PB＆J理性的LLM角色始终仅使用用户的人口统计和/或判断来超越进出方法。此外，使用描述用户信念的脚手架构建的LLM角色与使用人写的理由的人竞争性能。

Title: RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models

• Authors: Bang An, Shiyue Zhang, Mark Dredze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.18041
• Pdf URL: https://arxiv.org/pdf/2504.18041
• Copy Paste: [[2504.18041]] RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models(https://arxiv.org/abs/2504.18041)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Efforts to ensure the safety of large language models (LLMs) include safety fine-tuning, evaluation, and red teaming. However, despite the widespread use of the Retrieval-Augmented Generation (RAG) framework, AI safety work focuses on standard LLMs, which means we know little about how RAG use cases change a model's safety profile. We conduct a detailed comparative analysis of RAG and non-RAG frameworks with eleven LLMs. We find that RAG can make models less safe and change their safety profile. We explore the causes of this change and find that even combinations of safe models with safe documents can cause unsafe generations. In addition, we evaluate some existing red teaming methods for RAG settings and show that they are less effective than when used for non-RAG settings. Our work highlights the need for safety research and red-teaming methods specifically tailored for RAG LLMs.
• 摘要：确保大语言模型（LLMS）安全的努力包括安全调查，评估和红色团队。但是，尽管广泛使用了检索功能的生成（RAG）框架，但AI安全工作集中在标准LLM上，这意味着我们对RAG用例如何改变模型的安全性概况一无所知。我们对具有11个LLM的抹布和非抹布框架进行了详细的比较分析。我们发现抹布可以使模型降低安全并改变其安全性。我们探讨了这种变化的原因，发现即使是安全模型与安全文件的组合也可能导致不安全的世代。此外，我们评估了一些用于抹布设置的现有红色小组方法，并表明它们的效率不如非抹布设置。我们的工作强调了针对RAG LLMS量身定制的安全研究和红色团队方法的需求。

Title: DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models

• Authors: Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, Xingwei Qu, Xiao Zhang, Yingshui Tan, Yanan Wu, Jihao Gu, Yangguang Li, Jianke Zhu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.18053
• Pdf URL: https://arxiv.org/pdf/2504.18053
• Copy Paste: [[2504.18053]] DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models(https://arxiv.org/abs/2504.18053)
• Keywords: language model, gpt, llm
• Abstract: Multimodal Large Language Models (MLLMs) pose unique safety challenges due to their integration of visual and textual data, thereby introducing new dimensions of potential attacks and complex risk combinations. In this paper, we begin with a detailed analysis aimed at disentangling risks through step-by-step reasoning within multimodal inputs. We find that systematic multimodal risk disentanglement substantially enhances the risk awareness of MLLMs. Via leveraging the strong discriminative abilities of multimodal risk disentanglement, we further introduce \textbf{DREAM} (\textit{\textbf{D}isentangling \textbf{R}isks to \textbf{E}nhance Safety \textbf{A}lignment in \textbf{M}LLMs}), a novel approach that enhances safety alignment in MLLMs through supervised fine-tuning and iterative Reinforcement Learning from AI Feedback (RLAIF). Experimental results show that DREAM significantly boosts safety during both inference and training phases without compromising performance on normal tasks (namely oversafety), achieving a 16.17\% improvement in the SIUO safe\&effective score compared to GPT-4V. The data and code are available at this https URL.
• 摘要：多模式大型语言模型（MLLM）由于将视觉和文本数据整合而构成了独特的安全挑战，从而引入了潜在攻击和复杂风险组合的新维度。在本文中，我们从详细的分析开始，该分析旨在通过多模式输入中的逐步推理来消除风险。我们发现，系统的多模式风险分离大大提高了MLLM的风险意识。通过利用多模式风险脱离的强大歧视能力，我们进一步介绍\ textbf {dream}（\ textit {\ textbf {\ textbf {d} isentangling \ textbf {r textbf {r} isks isk isk通过从AI反馈（RLAIF）中监督的微调和迭代增强学习来增强MLLM中安全对齐的方法。实验结果表明，梦想在推理和训练阶段都显着提高了安全性，而不会损害正常任务（即超安全性）的表现，与GPT-4V相比，SIUO安全的16.17 \％提高了。数据和代码可在此HTTPS URL上找到。

Title: Exploring Personality-Aware Interactions in Salesperson Dialogue Agents

• Authors: Sijia Cheng, Wen-Yu Chang, Yun-Nung Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.18058
• Pdf URL: https://arxiv.org/pdf/2504.18058
• Copy Paste: [[2504.18058]] Exploring Personality-Aware Interactions in Salesperson Dialogue Agents(https://arxiv.org/abs/2504.18058)
• Keywords: agent
• Abstract: The integration of dialogue agents into the sales domain requires a deep understanding of how these systems interact with users possessing diverse personas. This study explores the influence of user personas, defined using the Myers-Briggs Type Indicator (MBTI), on the interaction quality and performance of sales-oriented dialogue agents. Through large-scale testing and analysis, we assess the pre-trained agent's effectiveness, adaptability, and personalization capabilities across a wide range of MBTI-defined user types. Our findings reveal significant patterns in interaction dynamics, task completion rates, and dialogue naturalness, underscoring the future potential for dialogue agents to refine their strategies to better align with varying personality traits. This work not only provides actionable insights for building more adaptive and user-centric conversational systems in the sales domain but also contributes broadly to the field by releasing persona-defined user simulators. These simulators, unconstrained by domain, offer valuable tools for future research and demonstrate the potential for scaling personalized dialogue systems across diverse applications.
• 摘要：将对话代理集成到销售领域需要深入了解这些系统如何与具有多样性角色的用户相互作用。这项研究探讨了使用Myers-Briggs类型指标（MBTI）定义的用户角色的影响，对面向销售的对话代理的相互作用质量和性能。通过大规模的测试和分析，我们评估了预先训练的代理商的有效性，适应性和个性化功能，这些功能在广泛定义的用户类型中。我们的发现揭示了互动动力学，任务完成率和对话自然性的重要模式，从而强调了对话代理的未来潜力，以完善其策略，以更好地与不同的人格特征保持一致。这项工作不仅提供了可行的见解，可以在销售领域构建更多自适应和以用户为中心的对话系统，而且还通过发布角色定义的用户模拟器来对该领域做出广泛的贡献。这些模拟器不受域的影响，为将来的研究提供了有价值的工具，并证明了在不同应用程序中扩展个性化对话系统的潜力。

Title: PropRAG: Guiding Retrieval with Beam Search over Proposition Paths

• Authors: Jingjin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.18070
• Pdf URL: https://arxiv.org/pdf/2504.18070
• Copy Paste: [[2504.18070]] PropRAG: Guiding Retrieval with Beam Search over Proposition Paths(https://arxiv.org/abs/2504.18070)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) has become the standard non-parametric approach for equipping Large Language Models (LLMs) with up-to-date knowledge and mitigating catastrophic forgetting common in continual learning. However, standard RAG, relying on independent passage retrieval, fails to capture the interconnected nature of human memory crucial for complex reasoning (associativity) and contextual understanding (sense-making). While structured RAG methods like HippoRAG utilize knowledge graphs (KGs) built from triples, the inherent context loss limits fidelity. We introduce PropRAG, a framework leveraging contextually rich propositions and a novel beam search algorithm over proposition paths to explicitly discover multi-step reasoning chains. Crucially, PropRAG's online retrieval process operates entirely without invoking generative LLMs, relying instead on efficient graph traversal and pre-computed embeddings. This avoids online LLM inference costs and potential inconsistencies during evidence gathering. LLMs are used effectively offline for high-quality proposition extraction and post-retrieval for answer generation. PropRAG achieves state-of-the-art zero-shot Recall@5 results on PopQA (55.3%), 2Wiki (93.7%), HotpotQA (97.0%), and MuSiQue (77.3%), alongside top F1 scores (e.g., 52.4% on MuSiQue). By improving evidence retrieval through richer representation and explicit, LLM-free online path finding, PropRAG advances non-parametric continual learning.
• 摘要：检索增强生成（RAG）已成为为大型语言模型（LLM）配备最新知识的标准非参数方法，并减轻灾难性的遗忘在持续学习中常见。但是，依靠独立段落检索的标准抹布未能捕获人类记忆的相互联系性质，对于复杂的推理（联想）和上下文理解（理解性）至关重要。诸如Hipporag之类的结构化抹布方法利用了从三元组构建的知识图（kgs），但固有的上下文损失限制了忠诚度。我们介绍了主体，一个利用上下文丰富的命题的框架和一种新颖的光束搜索算法，对命题路径进行了明确发现多步性推理链。至关重要的是，Proprag的在线检索过程完全可以运行，而无需调用生成LLM，而是依靠有效的图形遍历和预计算的嵌入。这避免了在线LLM推理成本和在证据收集期间的潜在不一致之处。 LLM有效地使用了高质量提取的脱机和回答后的回答后。 Proprag在POPQA（55.3％），2Wiki（93.7％），HotPotQA（97.0％）和Musique（77.3％）（77.3％）上获得了最新的零射击@5结果，以及最高的F1分数（例如Myique上的52.4％）。通过通过更丰富的代表性和明确的，无LLM的在线路径查找来改善证据的检索，前列腺可以进步非参数持续学习。

Title: Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimization

• Authors: Wataru Kawakami, Keita Suzuki, Junichiro Iwasawa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18080
• Pdf URL: https://arxiv.org/pdf/2504.18080
• Copy Paste: [[2504.18080]] Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimization(https://arxiv.org/abs/2504.18080)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) show potential in medicine, yet clinical adoption is hindered by concerns over factual accuracy, language-specific limitations (e.g., Japanese), and critically, their reliability when required to generate reasoning explanations -- a prerequisite for trust. This paper introduces Preferred-MedLLM-Qwen-72B, a 72B-parameter model optimized for the Japanese medical domain to achieve both high accuracy and stable reasoning. We employ a two-stage fine-tuning process on the Qwen2.5-72B base model: first, Continued Pretraining (CPT) on a comprehensive Japanese medical corpus instills deep domain knowledge. Second, Reasoning Preference Optimization (RPO), a preference-based method, enhances the generation of reliable reasoning pathways while preserving high answer accuracy. Evaluations on the Japanese Medical Licensing Exam benchmark (IgakuQA) show Preferred-MedLLM-Qwen-72B achieves state-of-the-art performance (0.868 accuracy), surpassing strong proprietary models like GPT-4o (0.866). Crucially, unlike baseline or CPT-only models which exhibit significant accuracy degradation (up to 11.5\% and 3.8\% respectively on IgakuQA) when prompted for explanations, our model maintains its high accuracy (0.868) under such conditions. This highlights RPO's effectiveness in stabilizing reasoning generation. This work underscores the importance of optimizing for reliable explanations alongside accuracy. We release the Preferred-MedLLM-Qwen-72B model weights to foster research into trustworthy LLMs for specialized, high-stakes applications.
• 摘要：大型语言模型（LLMS）在医学方面具有潜力，但是临床采用受到事实准确性，特定于语言的限制（例如日语）的担忧，并且在产生推理解释的必要条件时，其可靠性 - 一种信任的先决条件。本文介绍了首选的Medllm-Qwen-72b，这是一种针对日本医疗领域优化的72B参数模型，以实现高精度和稳定的推理。我们在QWEN2.5-72B基本模型上采用了两阶段的微调过程：首先，在日本全面的医学语料库中持续预审进（CPT）灌输了深层的领域知识。其次，基于偏好的方法推理偏好优化（RPO）增强了可靠的推理途径的产生，同时保持高答案的准确性。对日本医学许可考试基准（IGAKUQA）的评估显示，首选的Medllm-Qwen-72b达到了最先进的性能（0.868精度），超过了GPT-4O（0.866）（0.866）的强大专有模型。至关重要的是，与仅基线或仅CPT模型表现出明显的准确性降解（在Igakuqa上分别为11.5 \％和3.8％）时，当提示解释时，我们的模型在这种情况下保持了高准确性（0.868）。这突出了RPO在稳定推理产生方面的有效性。这项工作强调了优化可靠解释以及准确性的重要性。我们发布了首选的Medllm-Qwen-72b模型权重，以促进针对专业高风险应用的值得信赖的LLM的研究。

Title: Random-Set Large Language Models

• Authors: Muhammad Mubashar, Shireen Kudukkil Manchingal, Fabio Cuzzolin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18085
• Pdf URL: https://arxiv.org/pdf/2504.18085
• Copy Paste: [[2504.18085]] Random-Set Large Language Models(https://arxiv.org/abs/2504.18085)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are known to produce very high-quality tests and responses to our queries. But how much can we trust this generated text? In this paper, we study the problem of uncertainty quantification in LLMs. We propose a novel Random-Set Large Language Model (RSLLM) approach which predicts finite random sets (belief functions) over the token space, rather than probability vectors as in classical LLMs. In order to allow so efficiently, we also present a methodology based on hierarchical clustering to extract and use a budget of "focal" subsets of tokens upon which the belief prediction is defined, rather than using all possible collections of tokens, making the method scalable yet effective. RS-LLMs encode the epistemic uncertainty induced in their generation process by the size and diversity of its training set via the size of the credal sets associated with the predicted belief functions. The proposed approach is evaluated on CoQA and OBQA datasets using Llama2-7b, Mistral-7b and Phi-2 models and is shown to outperform the standard model in both datasets in terms of correctness of answer while also showing potential in estimating the second level uncertainty in its predictions and providing the capability to detect when its hallucinating.
• 摘要：已知大型语言模型（LLM）会产生非常高质量的测试和对我们的查询的响应。但是，我们可以信任这个生成的文本多少？在本文中，我们研究了LLM中不确定性定量的问题。我们提出了一种新型的随机集合模型（RSLLM）方法，该方法可以预测令牌空间上有限的随机集（信念函数），而不是像经典LLM中那样的概率向量。为了如此有效地允许，我们还提出了一种基于层次聚类的方法，以提取和使用定义信念预测的“焦点”子集的预算，而不是使用所有可能的代币集合，从而使方法可扩展而有效。 RS-llms通过与预测的信念功能相关的信用集的大小来编码其生成过程中引起的认知不确定性。使用LLAMA2-7B，MISTRAL-7B和PHI-2模型对COQA和OBQA数据集进行了评估所提出的方法，并显示出在两个数据集中的答案正确性，同时还显示出预测的第二级不确定性，并在其预测中估算了其幻觉时的潜力。

Title: Application and Optimization of Large Models Based on Prompt Tuning for Fact-Check-Worthiness Estimation

• Authors: Yinglong Yu, Hao Shen, Zhengyi Lyu, Qi He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.18104
• Pdf URL: https://arxiv.org/pdf/2504.18104
• Copy Paste: [[2504.18104]] Application and Optimization of Large Models Based on Prompt Tuning for Fact-Check-Worthiness Estimation(https://arxiv.org/abs/2504.18104)
• Keywords: language model, gpt, prompt
• Abstract: In response to the growing problem of misinformation in the context of globalization and informatization, this paper proposes a classification method for fact-check-worthiness estimation based on prompt tuning. We construct a model for fact-check-worthiness estimation at the methodological level using prompt tuning. By applying designed prompt templates to large language models, we establish in-context learning and leverage prompt tuning technology to improve the accuracy of determining whether claims have fact-check-worthiness, particularly when dealing with limited or unlabeled data. Through extensive experiments on public datasets, we demonstrate that the proposed method surpasses or matches multiple baseline methods in the classification task of fact-check-worthiness estimation assessment, including classical pre-trained models such as BERT, as well as recent popular large models like GPT-3.5 and GPT-4. Experiments show that the prompt tuning-based method proposed in this study exhibits certain advantages in evaluation metrics such as F1 score and accuracy, thereby effectively validating its effectiveness and advancement in the task of fact-check-worthiness estimation.
• 摘要：为了应对在全球化和信息化的背景下越来越多的错误信息问题，本文提出了一种基于迅速调整的事实检查估算的分类方法。我们使用及时调整在方法论层面上构建了一个模型，以进行事实检查。通过将设计的及时模板应用于大型语言模型，我们建立了内在的学习学习并利用及时的调整技术来提高确定索赔是否具有事实检查的准确性，尤其是在处理有限或未标记的数据时。通过在公共数据集上进行的大量实验，我们证明了所提出的方法超过或匹配了事实检查的值得估算评估的分类任务中的多个基线方法，包括BERT等经典预培训模型，以及最近的流行大型模型，例如GPT-3.5和GPT-4。实验表明，本研究中提出的基于迅速的基于调整的方法在评估指标（例如F1得分和准确性）方面具有某些优势，从而有效验证了其在事实核对估算的任务中的有效性和进步。

Title: Comparative Study on the Discourse Meaning of Chinese and English Media in the Paris Olympics Based on LDA Topic Modeling Technology and LLM Prompt Engineering

• Authors: Yinglong Yu, Zhaopu Yao, Fang Yuan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18106
• Pdf URL: https://arxiv.org/pdf/2504.18106
• Copy Paste: [[2504.18106]] Comparative Study on the Discourse Meaning of Chinese and English Media in the Paris Olympics Based on LDA Topic Modeling Technology and LLM Prompt Engineering(https://arxiv.org/abs/2504.18106)
• Keywords: language model, llm, prompt
• Abstract: This study analyzes Chinese and English media reports on the Paris Olympics using topic modeling, Large Language Model (LLM) prompt engineering, and corpus phraseology methods to explore similarities and differences in discourse construction and attitudinal meanings. Common topics include the opening ceremony, athlete performance, and sponsorship brands. Chinese media focus on specific sports, sports spirit, doping controversies, and new technologies, while English media focus on female athletes, medal wins, and eligibility controversies. Chinese reports show more frequent prepositional co-occurrences and positive semantic prosody in describing the opening ceremony and sports spirit. English reports exhibit positive semantic prosody when covering female athletes but negative prosody in predicting opening ceremony reactions and discussing women's boxing controversies.
• 摘要：这项研究使用主题建模，大型语言模型（LLM）及时工程和语料库措辞方法分析了中国和英国媒体关于巴黎奥运会的报告，以探索话语构建和态度含义的相似性和差异。常见主题包括开幕式，运动员表演和赞助品牌。中国媒体专注于特定的运动，体育精神，兴奋剂争议和新技术，而英国媒体则关注女运动员，奖牌胜利和资格争议。中国的报告显示，在描述开幕式和体育精神时，介词更频繁地共同出现和积极的语义韵律。英语报告在涵盖女运动员时表现出积极的语义韵律，但在预测开幕式的反应和讨论女性拳击争议时，韵律是负面的。

Title: Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection

• Authors: Atharva Kulkarni, Yuan Zhang, Joel Ruben Antony Moniz, Xiou Ge, Bo-Hsiang Tseng, Dhivya Piraviperumal, Swabha Swayamdipta, Hong Yu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18114
• Pdf URL: https://arxiv.org/pdf/2504.18114
• Copy Paste: [[2504.18114]] Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection(https://arxiv.org/abs/2504.18114)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Hallucinations pose a significant obstacle to the reliability and widespread adoption of language models, yet their accurate measurement remains a persistent challenge. While many task- and domain-specific metrics have been proposed to assess faithfulness and factuality concerns, the robustness and generalization of these metrics are still untested. In this paper, we conduct a large-scale empirical evaluation of 6 diverse sets of hallucination detection metrics across 4 datasets, 37 language models from 5 families, and 5 decoding methods. Our extensive investigation reveals concerning gaps in current hallucination evaluation: metrics often fail to align with human judgments, take an overtly myopic view of the problem, and show inconsistent gains with parameter scaling. Encouragingly, LLM-based evaluation, particularly with GPT-4, yields the best overall results, and mode-seeking decoding methods seem to reduce hallucinations, especially in knowledge-grounded settings. These findings underscore the need for more robust metrics to understand and quantify hallucinations, and better strategies to mitigate them.
• 摘要：幻觉对语言模型的可靠性和广泛采用构成了重大障碍，但是它们的准确测量仍然是一个持续的挑战。尽管已经提出了许多任务和领域特定的指标来评估忠诚和事实问题，但仍未测试这些指标的稳健性和概括。在本文中，我们对4个数据集，5个家庭的37个语言模型和5种解码方法进行了大规模的经验评估。我们的广泛调查揭示了有关当前幻觉评估中差距的差距：指标通常无法与人类判断保持一致，对问题的近视看法，并与参数缩放显示出不一致的收益。令人鼓舞的是，基于LLM的评估，尤其是GPT-4的评估，可以产生最佳的总体结果，而寻求模式的解码方法似乎减少了幻觉，尤其是在知识接地的环境中。这些发现强调了需要更强大的指标来理解和量化幻觉的需求，并有更好的策略来减轻它们。

Title: Temporal Entailment Pretraining for Clinical Language Models over EHR Data

• Authors: Tatsunori Tanaka, Fi Zheng, Kai Sato, Zhifeng Li, Yuanyun Zhang, Shi Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18128
• Pdf URL: https://arxiv.org/pdf/2504.18128
• Copy Paste: [[2504.18128]] Temporal Entailment Pretraining for Clinical Language Models over EHR Data(https://arxiv.org/abs/2504.18128)
• Keywords: language model
• Abstract: Clinical language models have achieved strong performance on downstream tasks by pretraining on domain specific corpora such as discharge summaries and medical notes. However, most approaches treat the electronic health record as a static document, neglecting the temporally-evolving and causally entwined nature of patient trajectories. In this paper, we introduce a novel temporal entailment pretraining objective for language models in the clinical domain. Our method formulates EHR segments as temporally ordered sentence pairs and trains the model to determine whether a later state is entailed by, contradictory to, or neutral with respect to an earlier state. Through this temporally structured pretraining task, models learn to perform latent clinical reasoning over time, improving their ability to generalize across forecasting and diagnosis tasks. We pretrain on a large corpus derived from MIMIC IV and demonstrate state of the art results on temporal clinical QA, early warning prediction, and disease progression modeling.
• 摘要：临床语言模型通过在特定领域（例如放电摘要和医疗笔记）上仔细研究下游任务上的良好性能。但是，大多数方法将电子健康记录视为静态文件，忽略了患者轨迹的时间发展和因果关系的性质。在本文中，我们为临床领域中的语言模型介绍了一个新型的时间范围预处理目标。我们的方法将EHR段作为时间顺序排序的句子对，并训练模型以确定是否需要，与早期状态相矛盾或相互矛盾或中立。通过这项暂时结构化的预审前任务，模型学会随着时间的推移执行潜在的临床推理，从而提高了其在预测和诊断任务中概括的能力。我们在源自模拟IV的大型语料库上预处理，并在时间临床质量检查，预警预测和疾病进展模型上证明了最先进的结果。

Title: Aligning Language Models for Icelandic Legal Text Summarization

• Authors: Þórir Hrafn Harðarson, Hrafn Loftsson, Stefán Ólafsson
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18180
• Pdf URL: https://arxiv.org/pdf/2504.18180
• Copy Paste: [[2504.18180]] Aligning Language Models for Icelandic Legal Text Summarization(https://arxiv.org/abs/2504.18180)
• Keywords: language model
• Abstract: The integration of language models in the legal domain holds considerable promise for streamlining processes and improving efficiency in managing extensive workloads. However, the specialized terminology, nuanced language, and formal style of legal texts can present substantial challenges. This study examines whether preference-based training techniques, specifically Reinforcement Learning from Human Feedback and Direct Preference Optimization, can enhance models' performance in generating Icelandic legal summaries that align with domain-specific language standards and user preferences. We compare models fine-tuned with preference training to those using conventional supervised learning. Results indicate that preference training improves the legal accuracy of generated summaries over standard fine-tuning but does not significantly enhance the overall quality of Icelandic language usage. Discrepancies between automated metrics and human evaluations further underscore the importance of qualitative assessment in developing language models for the legal domain.
• 摘要：在法律领域中，语言模型的集成在简化流程和提高管理大量工作量的效率方面具有巨大的希望。但是，专业的术语，细微的语言和正式的法律文本风格可能会带来重大挑战。这项研究研究了基于偏好的培训技术，特别是从人类反馈和直接偏好优化中学习的强化学习是否可以增强模型在产生与特定领域的语言标准和用户偏好相符的冰岛法律摘要方面的性能。我们将微调的模型与偏好培训与使用常规监督学习的模型进行比较。结果表明，偏好培训提高了生成的摘要的法律准确性，而不是标准微调，但并不能显着提高冰岛语言使用的整体质量。自动指标与人类评估之间的差异进一步强调了定性评估在开发法律领域的语言模型中的重要性。

Title: Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish

• Authors: Shuxiang Du, Ana Guerberof Arenas, Antonio Toral, Kyo Gerrits, Josep Marco Borillo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18221
• Pdf URL: https://arxiv.org/pdf/2504.18221
• Copy Paste: [[2504.18221]] Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish(https://arxiv.org/abs/2504.18221)
• Keywords: gpt, prompt, chat
• Abstract: This study examines the variability of Chat-GPT machine translation (MT) outputs across six different configurations in four languages,with a focus on creativity in a literary text. We evaluate GPT translations in different text granularity levels, temperature settings and prompting strategies with a Creativity Score formula. We found that prompting ChatGPT with a minimal instruction yields the best creative translations, with "Translate the following text into [TG] creatively" at the temperature of 1.0 outperforming other configurations and DeepL in Spanish, Dutch, and Chinese. Nonetheless, ChatGPT consistently underperforms compared to human translation (HT).
• 摘要：这项研究研究了四种语言中六种不同配置的Chat-GPT机器翻译（MT）输出的可变性，重点是文学文本中的创造力。我们以不同的文本粒度水平，温度设置和以创造力得分公式提示策略来评估GPT翻译。我们发现，以最小的指示提示Chatgpt可以产生最佳的创意翻译，并以1.0的温度优于其他配置，并以西班牙语，荷兰语，荷兰语和中文的形式进行“将以下文本转换为[TG]”。尽管如此，与人类翻译（HT）相比，Chatgpt始终表现不佳。

Title: Efficient Single-Pass Training for Multi-Turn Reasoning

• Authors: Ritesh Goru, Shanay Mehta, Prateek Jain
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18246
• Pdf URL: https://arxiv.org/pdf/2504.18246
• Copy Paste: [[2504.18246]] Efficient Single-Pass Training for Multi-Turn Reasoning(https://arxiv.org/abs/2504.18246)
• Keywords: language model, llm
• Abstract: Training Large Language Models ( LLMs) to generate explicit reasoning before they produce an answer has been shown to improve their performance across various tasks such as mathematics and coding. However, fine-tuning LLMs on multi-turn reasoning datasets presents a unique challenge: LLMs must generate reasoning tokens that are excluded from subsequent inputs to the LLM. This discrepancy prevents us from processing an entire conversation in a single forward pass-an optimization readily available when we fine-tune on a multi-turn non-reasoning dataset. This paper proposes a novel approach that overcomes this limitation through response token duplication and a custom attention mask that enforces appropriate visibility constraints. Our approach significantly reduces the training time and allows efficient fine-tuning on multi-turn reasoning datasets.
• 摘要：培训大型语言模型（LLMS）在产生答案之前生成明确的推理，已证明可以改善其在数学和编码等各种任务中的性能。但是，多转变推理数据集上的微调LLM提出了一个独特的挑战：LLM必须生成推理令牌，这些代币被排除在后续输入到LLM的情况下。这种差异使我们无法在单个前向通行证中处理整个对话 - 当我们在多转折叠数据集中微调时，可以随时获得优化。本文提出了一种新颖的方法，该方法通过响应令牌复制和实施适当可见性限制的自定义注意力面罩克服了这种限制。我们的方法大大减少了训练时间，并允许对多转变推理数据集进行有效的微调。

Title: MAGI: Multi-Agent Guided Interview for Psychiatric Assessment

• Authors: Guanqun Bi, Zhuang Chen, Zhoufu Liu, Hongkai Wang, Xiyao Xiao, Yuqiang Xie, Wen Zhang, Yongkang Huang, Yuxuan Chen, Libiao Peng, Yi Feng, Minlie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18260
• Pdf URL: https://arxiv.org/pdf/2504.18260
• Copy Paste: [[2504.18260]] MAGI: Multi-Agent Guided Interview for Psychiatric Assessment(https://arxiv.org/abs/2504.18260)
• Keywords: language model, llm, agent
• Abstract: Automating structured clinical interviews could revolutionize mental healthcare accessibility, yet existing large language models (LLMs) approaches fail to align with psychiatric diagnostic protocols. We present MAGI, the first framework that transforms the gold-standard Mini International Neuropsychiatric Interview (MINI) into automatic computational workflows through coordinated multi-agent collaboration. MAGI dynamically navigates clinical logic via four specialized agents: 1) an interview tree guided navigation agent adhering to the MINI's branching structure, 2) an adaptive question agent blending diagnostic probing, explaining, and empathy, 3) a judgment agent validating whether the response from participants meet the node, and 4) a diagnosis Agent generating Psychometric Chain-of- Thought (PsyCoT) traces that explicitly map symptoms to clinical criteria. Experimental results on 1,002 real-world participants covering depression, generalized anxiety, social anxiety and suicide shows that MAGI advances LLM- assisted mental health assessment by combining clinical rigor, conversational adaptability, and explainable reasoning.
• 摘要：自动化结构化的临床访谈可能会彻底改变心理保健的可及性，但现有的大型语言模型（LLMS）方法无法与精神病诊断方案保持一致。我们提出了Magi，这是第一个将金色标准的迷你国际神经精神访谈（MINI）转变为通过协调的多代理协作而自动计算工作流程的框架。 MAGI通过四个专业剂动态地导航临床逻辑：1）访谈树引导剂依附于迷你分支结构，2）一种自适应问题，融合诊断，同情和移情的自适应问题，解释判断剂，3）判断剂验证参与者的回应是否会遇到诊断剂，以及4）诊断型临床（4）Psycot thoughtiment Interication-Psyccot（Psycc），Psyccot（Psycc） 标准。涵盖抑郁症，普遍焦虑，社交焦虑和自杀的1,002名现实世界参与者的实验结果表明，通过结合临床严格，对话适应性和可解释的推理，MAGI进步了LLM辅助心理健康评估。

Title: TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation

• Authors: Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai, Jingun Kwon, Hidetaka Kamigaito, Katsuhiko Hayashi, Manabu Okumura, Taro Watanabe
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.18269
• Pdf URL: https://arxiv.org/pdf/2504.18269
• Copy Paste: [[2504.18269]] TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation(https://arxiv.org/abs/2504.18269)
• Keywords: language model, llm, prompt
• Abstract: Generating images from prompts containing specific entities requires models to retain as much entity-specific knowledge as possible. However, fully memorizing such knowledge is impractical due to the vast number of entities and their continuous emergence. To address this, we propose Text-based Intelligent Generation with Entity prompt Refinement (TextTIGER), which augments knowledge on entities included in the prompts and then summarizes the augmented descriptions using Large Language Models (LLMs) to mitigate performance degradation from longer inputs. To evaluate our method, we introduce WiT-Cub (WiT with Captions and Uncomplicated Background-explanations), a dataset comprising captions, images, and an entity list. Experiments on four image generation models and five LLMs show that TextTIGER improves image generation performance in standard metrics (IS, FID, and CLIPScore) compared to caption-only prompts. Additionally, multiple annotators' evaluation confirms that the summarized descriptions are more informative, validating LLMs' ability to generate concise yet rich descriptions. These findings demonstrate that refining prompts with augmented and summarized entity-related descriptions enhances image generation capabilities. The code and dataset will be available upon acceptance.
• 摘要：从包含特定实体的提示中生成图像需要模型保留尽可能多的特定实体知识。但是，由于大量实体及其持续出现，完全记住这种知识是不切实际的。为了解决这个问题，我们提出了具有实体提示改进（TextTiger）的基于文本的智能生成，该生成增强了提示中包含的实体的知识，然后使用大语言模型（LLMS）汇总了增强描述，以减轻从较长输入中的绩效降低。为了评估我们的方法，我们介绍WIT-CUB（带有字幕和简单的背景解释），一个包含字幕，图像和实体列表的数据集。与仅有字幕提示相比，对四个图像生成模型和五个LLM的实验表明，TextTiger改善了标准指标（IS，FID和ClipsCore）的图像生成性能。此外，多个注释者的评估证实了汇总的描述更有信息，验证了LLMS生成简洁但丰富的描述的能力。这些发现表明，使用增强和汇总的实体相关描述提示提示增强了图像生成能力。该代码和数据集将在接受后可用。

Title: Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review

• Authors: Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Leon Witt, Muhammad Asif Ali, Yukai Miao, Dan Li, Qingsong Wei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.18346
• Pdf URL: https://arxiv.org/pdf/2504.18346
• Copy Paste: [[2504.18346]] Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review(https://arxiv.org/abs/2504.18346)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have been transformative across many domains. However, hallucination -- confidently outputting incorrect information -- remains one of the leading challenges for LLMs. This raises the question of how to accurately assess and quantify the uncertainty of LLMs. Extensive literature on traditional models has explored Uncertainty Quantification (UQ) to measure uncertainty and employed calibration techniques to address the misalignment between uncertainty and accuracy. While some of these methods have been adapted for LLMs, the literature lacks an in-depth analysis of their effectiveness and does not offer a comprehensive benchmark to enable insightful comparison among existing solutions. In this work, we fill this gap via a systematic survey of representative prior works on UQ and calibration for LLMs and introduce a rigorous benchmark. Using two widely used reliability datasets, we empirically evaluate six related methods, which justify the significant findings of our review. Finally, we provide outlooks for key future directions and outline open challenges. To the best of our knowledge, this survey is the first dedicated study to review the calibration methods and relevant metrics for LLMs.
• 摘要：大型语言模型（LLM）在许多领域都具有变革性。但是，幻觉 - 自信输出不正确的信息 - 仍然是LLM的主要挑战之一。这就提出了如何准确评估和量化LLM的不确定性的问题。关于传统模型的广泛文献探索了不确定性量化（UQ），以测量不确定性和使用的校准技术，以解决不确定性和准确性之间的错位。尽管其中一些方法已适用于LLM，但文献缺乏对其有效性的深入分析，并且没有提供全面的基准来实现现有解决方案之间的深入比较。在这项工作中，我们通过对LLMS的UQ和校准的代表性先前工作进行系统的调查来填补这一空白，并引入严格的基准。使用两个广泛使用的可靠性数据集，我们经验评估了六种相关方法，这证明了我们的审查的重要发现。最后，我们为未来的关键方向和概述开放挑战提供前景。据我们所知，这项调查是首次审查LLMS校准方法和相关指标的专门研究。

Title: Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant

• Authors: Lei Shen, Xiaoyu Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18373
• Pdf URL: https://arxiv.org/pdf/2504.18373
• Copy Paste: [[2504.18373]] Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant(https://arxiv.org/abs/2504.18373)
• Keywords: language model, llm, agent
• Abstract: In recent years, multi-agent frameworks powered by large language models (LLMs) have advanced rapidly. Despite this progress, there is still a notable absence of benchmark datasets specifically tailored to evaluate their performance. To bridge this gap, we introduce Auto-SLURP, a benchmark dataset aimed at evaluating LLM-based multi-agent frameworks in the context of intelligent personal assistants. Auto-SLURP extends the original SLURP dataset -- initially developed for natural language understanding tasks -- by relabeling the data and integrating simulated servers and external services. This enhancement enables a comprehensive end-to-end evaluation pipeline, covering language understanding, task execution, and response generation. Our experiments demonstrate that Auto-SLURP presents a significant challenge for current state-of-the-art frameworks, highlighting that truly reliable and intelligent multi-agent personal assistants remain a work in progress. The dataset and related code are available at this https URL.
• 摘要：近年来，由大语言模型（LLM）提供动力的多代理框架已迅速发展。尽管取得了这种进展，但仍然有明显的基准数据集专门为评估其性能而定。为了弥合这一差距，我们介绍了Auto-Slurp，这是一个基准数据集，旨在在智能个人助理的背景下评估基于LLM的多代理框架。通过重新标记数据并集成模拟的服务器和外部服务，自动弹出扩展了最初是为自然语言理解任务开发的原始SLURP数据集（最初是为了自然语言理解任务）。这种增强功能使全面的端到端评估管道涵盖了语言理解，任务执行和响应生成。我们的实验表明，自动弹出对当前最新框架提出了重大挑战，强调了真正可靠且聪明的多代理个人助理仍然在进行中。该数据集和相关代码可在此HTTPS URL上找到。

Title: Pushing the boundary on Natural Language Inference

• Authors: Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.18376
• Pdf URL: https://arxiv.org/pdf/2504.18376
• Copy Paste: [[2504.18376]] Pushing the boundary on Natural Language Inference(https://arxiv.org/abs/2504.18376)
• Keywords: language model, chain-of-thought
• Abstract: Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, and information retrieval. Despite its importance, current NLI systems heavily rely on supervised learning with datasets that often contain annotation artifacts and biases, limiting generalization and real-world applicability. In this work, we apply a reinforcement learning-based approach using Group Relative Policy Optimization (GRPO) for Chain-of-Thought (CoT) learning in NLI, eliminating the need for labeled rationales and enabling this type of training on more challenging datasets such as ANLI. We fine-tune 7B, 14B, and 32B language models using parameter-efficient techniques (LoRA and QLoRA), demonstrating strong performance across standard and adversarial NLI benchmarks. Our 32B AWQ-quantized model surpasses state-of-the-art results on 7 out of 11 adversarial sets$\unicode{x2013}$or on all of them considering our replication$\unicode{x2013}$within a 22GB memory footprint, showing that robust reasoning can be retained under aggressive quantization. This work provides a scalable and practical framework for building robust NLI systems without sacrificing inference quality.
• 摘要：自然语言推论（NLI）是自然语言理解的核心任务，并在事实检查，问题答案和信息检索中进行了应用。尽管它很重要，但当前的NLI系统在很大程度上依赖于经常包含注释工件和偏见的数据集的监督学习，从而限制了概括和现实世界的适用性。在这项工作中，我们使用小组相对政策优化（GRPO）应用了基于强化的学习方法，以在NLI中进行思考链（COT）学习，从而消除了对标签理由的需求，并在诸如ANLI等更具挑战性的数据集中实现了这种类型的培训。我们使用参数效率高效技术（Lora和Qlora）微调7B，14B和32B语言模型，在标准和对抗性NLI基准中表现出强劲的性能。我们的32B AWQ量化模型超过了最新的结果，在11个对抗套件中有7个$ \ unicode {x2013} $或所有这些都考虑我们的复制$ \ unicode {x2013} $在22GB内存足迹中，表明强大的推理可以在侵略性的情况下进行量化。这项工作提供了一个可扩展且实用的框架，用于构建强大的NLI系统而不牺牲推理质量。

Title: HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?

• Authors: Yusen Zhang, Wenliang Zheng, Aashrith Madasu, Peng Shi, Ryo Kamoi, Hao Zhou, Zhuoyang Zou, Shu Zhao, Sarkar Snigdha Sarathi Das, Vipul Gupta, Xiaoxin Lu, Nan Zhang, Ranran Haoran Zhang, Avitej Iyer, Renze Lou, Wenpeng Yin, Rui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18406
• Pdf URL: https://arxiv.org/pdf/2504.18406
• Copy Paste: [[2504.18406]] HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?(https://arxiv.org/abs/2504.18406)
• Keywords: language model, gpt
• Abstract: High-resolution image (HRI) understanding aims to process images with a large number of pixels, such as pathological images and agricultural aerial images, both of which can exceed 1 million pixels. Vision Large Language Models (VLMs) can allegedly handle HRIs, however, there is a lack of a comprehensive benchmark for VLMs to evaluate HRI understanding. To address this gap, we introduce HRScene, a novel unified benchmark for HRI understanding with rich scenes. HRScene incorporates 25 real-world datasets and 2 synthetic diagnostic datasets with resolutions ranging from 1,024 $\times$ 1,024 to 35,503 $\times$ 26,627. HRScene is collected and re-annotated by 10 graduate-level annotators, covering 25 scenarios, ranging from microscopic to radiology images, street views, long-range pictures, and telescope images. It includes HRIs of real-world objects, scanned documents, and composite multi-image. The two diagnostic evaluation datasets are synthesized by combining the target image with the gold answer and distracting images in different orders, assessing how well models utilize regions in HRI. We conduct extensive experiments involving 28 VLMs, including Gemini 2.0 Flash and GPT-4o. Experiments on HRScene show that current VLMs achieve an average accuracy of around 50% on real-world tasks, revealing significant gaps in HRI understanding. Results on synthetic datasets reveal that VLMs struggle to effectively utilize HRI regions, showing significant Regional Divergence and lost-in-middle, shedding light on future research.
• 摘要：高分辨率图像（HRI）的理解旨在用大量像素（例如病理图像和农业航空图像）处理图像，这两种图像都可以超过100万像素。 Vision大语模型（VLM）据称可以处理HRI，但是，VLMs缺乏评估HRI理解的全面基准。为了解决这一差距，我们介绍了HRScene，这是一种新颖的统一基准，用于HRI了解丰富的场景。 HRSCENE结合了25个现实世界数据集和2个合成诊断数据集，分辨率从1,024美元$ \ times $ 1,024到35,503 $ \ times $ 26,627。 HRSCENE由10个研究生级注释者收集和重新注册，涵盖25个场景，从微观到放射学图像，街道景观，远程图片和望远镜图像。它包括现实世界对象的HRI，扫描文档和复合多图像。两个诊断评估数据集通过将目标图像与金答案相结合并分散不同订单中的图像，从而评估模型在HRI中的使用率如何，从而综合了两个诊断评估数据集。我们进行了涉及28个VLM的广泛实验，包括Gemini 2.0 Flash和GPT-4O。 HRSCENE上的实验表明，当前VLM在现实世界中的平均准确性约为50％，这揭示了HRI理解的显着差距。合成数据集的结果表明，VLM难以有效利用HRI区域，显示出明显的区域差异和中间位置，从而阐明了未来的研究。

Title: Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers

• Authors: Jared Moore, Declan Grabb, William Agnew, Kevin Klyman, Stevie Chancellor, Desmond C. Ong, Nick Haber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18412
• Pdf URL: https://arxiv.org/pdf/2504.18412
• Copy Paste: [[2504.18412]] Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers(https://arxiv.org/abs/2504.18412)
• Keywords: language model, gpt, llm
• Abstract: Should a large language model (LLM) be used as a therapist? In this paper, we investigate the use of LLMs to *replace* mental health providers, a use case promoted in the tech startup and research space. We conduct a mapping review of therapy guides used by major medical institutions to identify crucial aspects of therapeutic relationships, such as the importance of a therapeutic alliance between therapist and client. We then assess the ability of LLMs to reproduce and adhere to these aspects of therapeutic relationships by conducting several experiments investigating the responses of current LLMs, such as `gpt-4o`. Contrary to best practices in the medical community, LLMs 1) express stigma toward those with mental health conditions and 2) respond inappropriately to certain common (and critical) conditions in naturalistic therapy settings -- e.g., LLMs encourage clients' delusional thinking, likely due to their sycophancy. This occurs even with larger and newer LLMs, indicating that current safety practices may not address these gaps. Furthermore, we note foundational and practical barriers to the adoption of LLMs as therapists, such as that a therapeutic alliance requires human characteristics (e.g., identity and stakes). For these reasons, we conclude that LLMs should not replace therapists, and we discuss alternative roles for LLMs in clinical therapy.
• 摘要：大型语言模型（LLM）是否应该用作治疗师？在本文中，我们调查了LLM的使用 *替换 *心理健康提供者，这是在技术初创公司和研究领域中促进的用例。我们对主要医疗机构使用的治疗指南进行了映射回顾，以确定治疗关系的关键方面，例如治疗师和客户之间治疗联盟的重要性。然后，我们通过进行多个研究当前LLM的响应（例如“ GPT-4O”）来评估LLM重现和遵守治疗关系的这些方面的能力。与医学界的最佳实践相反，LLMS 1）对患有心理健康状况的人表达污名，2）对自然主义治疗环境中某些常见（和关键）条件的反应不当 - 例如，LLMS鼓励客户的妄想思维，可能是由于其粘粘体。即使在更大和更新的LLM中也会发生这种情况，这表明当前的安全实践可能无法解决这些差距。此外，我们注意到采用LLM作为治疗师的基础和实际障碍，例如治疗联盟需要人类特征（例如身份和危险）。由于这些原因，我们得出结论，LLM不应取代治疗师，并且我们讨论了LLM在临床疗法中的替代作用。

Title: BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs

• Authors: Hongyu Wang, Shuming Ma, Furu Wei
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18415
• Pdf URL: https://arxiv.org/pdf/2504.18415
• Copy Paste: [[2504.18415]] BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs(https://arxiv.org/abs/2504.18415)
• Keywords: language model, llm
• Abstract: Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.
• 摘要：1位大语言模型（LLM）的有效部署受到激活异常值的阻碍，这使量化变得复杂到低宽度。我们介绍了Bitnet V2，这是一个新型框架，可实现1位LLM的天然4位激活量化。为了解决注意力和馈送网络激活方面的异常值，我们提出了H-Bitlinear，这是一个在激活量化之前应用在线Hadamard转换的模块。这种转换将尖锐的激活分布平滑为更类似高斯的形式，适用于低位表示。实验表明，比特网V2从头开始训练，其8位激活与BITNET B1.58性能相匹配。至关重要的是，在接受天然4位激活训练时，Bitnet V2可实现最小的性能降解，从而大大降低了批处理推理的记忆足迹和计算成本。

Title: PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts

• Authors: Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, Jingren Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18428
• Pdf URL: https://arxiv.org/pdf/2504.18428
• Copy Paste: [[2504.18428]] PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts(https://arxiv.org/abs/2504.18428)
• Keywords: llm
• Abstract: In this paper, we introduce PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation, making it a highly discriminative multilingual mathematical benchmark in the era of reasoning LLMs. We conduct a comprehensive evaluation for advanced LLMs and find that even Deepseek-R1-671B and Qwen-QwQ-32B, achieve only 43.4 and 41.8 benchmark scores, with less than 30% accuracy under the highest level. From a language perspective, our benchmark reveals several key challenges of LLMs in multilingual reasoning: (1) Reasoning performance varies widely across languages for current LLMs; (2) Input-output language consistency is low in reasoning LLMs and may be correlated with performance; (3) The thinking length differs significantly by language for current LLMs. Additionally, we demonstrate that controlling the output language in the instructions has the potential to affect reasoning performance, especially for some low-resource languages, suggesting a promising direction for improving multilingual capabilities in LLMs.
• 摘要：在本文中，我们介绍了Polymath，这是一种多语言的数学推理基准，涵盖18种语言和4种易于困难的难度水平。我们的基准确保了综合性，语言多样性和高质量的翻译，使其成为推理LLM时代的高度歧视性多语言数学基准。我们对高级LLM进行了全面的评估，发现即使DeepSeek-R1-671b和Qwen-QWQ-32B也仅获得43.4和41.8基准分数，在最高水平下精度少于30％。从语言的角度来看，我们的基准揭示了LLM在多语言推理中面临的几个关键挑战：（1）推理性能在当前LLM的语言中差异很大； （2）推理LLM的投入输出语言一致性较低，并且可能与性能相关； （3）思维长度因当前LLM的语言而显着不同。此外，我们证明，控制说明中的输出语言有可能影响推理性能，尤其是对于某些低资源语言，这表明了提高LLM中多语言能力的有希望的方向。

Title: Fast-Slow Thinking for Large Vision-Language Model Reasoning

• Authors: Wenyi Xiao, Leilei Gan, Weilong Dai, Wanggui He, Ziwei Huang, Haoyuan Li, Fangxun Shu, Zhelun Yu, Peng Zhang, Hao Jiang, Fei Wu
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.18458
• Pdf URL: https://arxiv.org/pdf/2504.18458
• Copy Paste: [[2504.18458]] Fast-Slow Thinking for Large Vision-Language Model Reasoning(https://arxiv.org/abs/2504.18458)
• Keywords: language model
• Abstract: Recent advances in large vision-language models (LVLMs) have revealed an \textit{overthinking} phenomenon, where models generate verbose reasoning across all tasks regardless of questions. To address this issue, we present \textbf{FAST}, a novel \textbf{Fa}st-\textbf{S}low \textbf{T}hinking framework that dynamically adapts reasoning depth based on question characteristics. Through empirical analysis, we establish the feasibility of fast-slow thinking in LVLMs by investigating how response length and data distribution affect performance. We develop FAST-GRPO with three components: model-based metrics for question characterization, an adaptive thinking reward mechanism, and difficulty-aware KL regularization. Experiments across seven reasoning benchmarks demonstrate that FAST achieves state-of-the-art accuracy with over 10\% relative improvement compared to the base model, while reducing token usage by 32.7-67.3\% compared to previous slow-thinking approaches, effectively balancing reasoning length and accuracy.
• 摘要：大型视觉模型（LVLM）的最新进展揭示了\ textit {过度思考}现象，其中模型在所有任务中都会在所有任务中产生冗长的推理。为了解决此问题，我们提出了\ textbf {fast}，这是一个新颖的\ textbf {fa} st- \ textbf {s} low \ textbf {t} hinking框架，该框架基于问题特征动态调整推理深度。通过经验分析，我们通过研究响应长度和数据分布如何影响性能来确定LVLM中快速思维的可行性。我们开发了具有三个组件的快速GRPO：基于模型的问题指标，用于问题表征，适应性思维奖励机制以及难以理解的KL正则化。七个推理基准的实验表明，与基本模型相比，快速实现了超过10 \％相对改进的最新精度，而与以前的慢速思维方法相比，将令牌使用量减少了32.7-67.3 \％，有效地平衡了推理的长度和准确性。

Title: Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions

• Authors: James D. Finch, Yasasvi Josyula, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18474
• Pdf URL: https://arxiv.org/pdf/2504.18474
• Copy Paste: [[2504.18474]] Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions(https://arxiv.org/abs/2504.18474)
• Keywords: language model, llm
• Abstract: In task-oriented dialogue (TOD) systems, Slot Schema Induction (SSI) is essential for automatically identifying key information slots from dialogue data without manual intervention. This paper presents a novel state-of-the-art (SoTA) approach that formulates SSI as a text generation task, where a language model incrementally constructs and refines a slot schema over a stream of dialogue data. To develop this approach, we present a fully automatic LLM-based TOD simulation method that creates data with high-quality state labels for novel task domains. Furthermore, we identify issues in SSI evaluation due to data leakage and poor metric alignment with human judgment. We resolve these by creating new evaluation data using our simulation method with human guidance and correction, as well as designing improved evaluation metrics. These contributions establish a foundation for future SSI research and advance the SoTA in dialogue understanding and system development.
• 摘要：在面向任务的对话（TOD）系统中，插槽模式诱导（SSI）对于自动从不手动干预的情况下从对话数据中自动识别关键信息插槽至关重要。本文提出了一种新颖的最先进的方法（SOTA）方法，该方法将SSI作为文本生成任务进行了提出，其中语言模型可以逐步构建和完善对话数据流的插槽模式。为了开发这种方法，我们提出了一种基于自动的LLM的TOD仿真方法，该方法具有用于新任务域的高质量状态标签的数据。此外，由于数据泄漏和人类判断力的度量差，我们确定了SSI评估中的问题。我们通过使用我们的指导和校正以及设计改进的评估指标来创建新的评估数据来解决这些问题。这些贡献为未来的SSI研究奠定了基础，并推进了对话理解和系统发展中的SOTA。

Title: Investigating Co-Constructive Behavior of Large Language Models in Explanation Dialogues

• Authors: Leandra Fichtel, Maximilian Spliethöver, Eyke Hüllermeier, Patricia Jimenez, Nils Klowait, Stefan Kopp, Axel-Cyrille Ngonga Ngomo, Amelie Robrecht, Ingrid Scharlau, Lutz Terfloth, Anna-Lisa Vollmer, Henning Wachsmuth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.18483
• Pdf URL: https://arxiv.org/pdf/2504.18483
• Copy Paste: [[2504.18483]] Investigating Co-Constructive Behavior of Large Language Models in Explanation Dialogues(https://arxiv.org/abs/2504.18483)
• Keywords: language model, llm
• Abstract: The ability to generate explanations that are understood by explainees is the quintessence of explainable artificial intelligence. Since understanding depends on the explainee's background and needs, recent research has focused on co-constructive explanation dialogues, where the explainer continuously monitors the explainee's understanding and adapts explanations dynamically. We investigate the ability of large language models (LLMs) to engage as explainers in co-constructive explanation dialogues. In particular, we present a user study in which explainees interact with LLMs, of which some have been instructed to explain a predefined topic co-constructively. We evaluate the explainees' understanding before and after the dialogue, as well as their perception of the LLMs' co-constructive behavior. Our results indicate that current LLMs show some co-constructive behaviors, such as asking verification questions, that foster the explainees' engagement and can improve understanding of a topic. However, their ability to effectively monitor the current understanding and scaffold the explanations accordingly remains limited.
• 摘要：解释者理解的解释能力是可解释的人工智能的精髓。由于理解取决于解释者的背景和需求，因此最近的研究集中在共同构建的解释对话上，其中解释器不断地监视了解释者的理解并动态调整说明。我们研究了大型语言模型（LLM）作为解释者参与共同构建解释对话的能力。特别是，我们提出了一项用户研究，其中解释性与LLM相互作用，其中一些人被指示在共同构建中解释一个预定义的主题。我们评估了对话之前和之后的解释者的理解，以及他们对LLMS共同构建行为的看法。我们的结果表明，当前的LLM显示了一些共同构建行为，例如提出验证问题，这些行为促进了解释者的参与度并可以改善对主题的理解。但是，他们有效监测当前理解和脚手架的能力相应地限制了解释。

Title: TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generation

• Authors: Gwen Yidou Weng, Benjie Wang, Guy Van den Broeck
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.18535
• Pdf URL: https://arxiv.org/pdf/2504.18535
• Copy Paste: [[2504.18535]] TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generation(https://arxiv.org/abs/2504.18535)
• Keywords: language model, llm
• Abstract: As large language models (LMs) advance, there is an increasing need to control their outputs to align with human values (e.g., detoxification) or desired attributes (e.g., personalization, topic). However, autoregressive models focus on next-token predictions and struggle with global properties that require looking ahead. Existing solutions either tune or post-train LMs for each new attribute - expensive and inflexible - or approximate the Expected Attribute Probability (EAP) of future sequences by sampling or training, which is slow and unreliable for rare attributes. We introduce TRACE (Tractable Probabilistic Reasoning for Adaptable Controllable gEneration), a novel framework that efficiently computes EAP and adapts to new attributes through tractable probabilistic reasoning and lightweight control. TRACE distills a Hidden Markov Model (HMM) from an LM and pairs it with a small classifier to estimate attribute probabilities, enabling exact EAP computation over the HMM's predicted futures. This EAP is then used to reweigh the LM's next-token probabilities for globally compliant continuations. Empirically, TRACE achieves state-of-the-art results in detoxification with only 10% decoding overhead, adapts to 76 low-resource personalized LLMs within seconds, and seamlessly extends to composite attributes.
• 摘要：随着大型语言模型（LMS）的发展，越来越需要控制其输出以与人类价值观（例如排毒）或所需属性（例如个性化，主题）保持一致。但是，自回归模型集中在下一步的预测上，并与需要展望未来的全球属性斗争。现有的解决方案对每个新属性进行调整或训练后的LMS（昂贵且不灵活），或通过采样或训练来近似未来序列的预期属性概率（EAP），这对于稀有属性而言是缓慢而不可靠的。我们介绍了痕迹（可自适应可控生成的可处理概率推理），这是一个新颖的框架，可通过可拖动的概率推理和轻量级控制有效地计算EAP并适应新属性。跟踪将隐藏的马尔可夫模型（HMM）从LM提炼，并将其与小分类器配对以估算属性概率，从而可以在HMM的预测期货上进行精确的EAP计算。然后，将使用此EAP来重新获得LM的下一步概率，以实现全球范围的连续性。从经验上讲，痕量实现最新的最新导致排毒，只有10％的开销，在几秒钟内适应76个低资源的个性化LLM，并无缝扩展到复合属性。