2025-04-25

Title: Tokenization Matters: Improving Zero-Shot NER for Indic Languages

• Authors: Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Amit Agarwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16977
• Pdf URL: https://arxiv.org/pdf/2504.16977
• Copy Paste: [[2504.16977]] Tokenization Matters: Improving Zero-Shot NER for Indic Languages(https://arxiv.org/abs/2504.16977)
• Keywords: language model
• Abstract: Tokenization is a critical component of Natural Language Processing (NLP), especially for low resource languages, where subword segmentation influences vocabulary structure and downstream task accuracy. Although Byte Pair Encoding (BPE) is a standard tokenization method in multilingual language models, its suitability for Named Entity Recognition (NER) in low resource Indic languages remains underexplored due to its limitations in handling morphological complexity. In this work, we systematically compare BPE, SentencePiece, and Character Level tokenization strategies using IndicBERT for NER tasks in low resource Indic languages like Assamese, Bengali, Marathi, and Odia, as well as extremely low resource Indic languages like Santali, Manipuri, and Sindhi. We assess both intrinsic linguistic properties tokenization efficiency, out of vocabulary (OOV) rates, and morphological preservation as well as extrinsic downstream performance, including fine tuning and zero shot cross lingual transfer. Our experiments show that SentencePiece is a consistently better performing approach than BPE for NER in low resource Indic Languages, particularly in zero shot cross lingual settings, as it better preserves entity consistency. While BPE provides the most compact tokenization form, it is not capable of generalization because it misclassifies or even fails to recognize entity labels when tested on unseen languages. In contrast, SentencePiece constitutes a better linguistic structural preservation model, benefiting extremely low resource and morphologically rich Indic languages, such as Santali and Manipuri, for superior entity recognition, as well as high generalization across scripts, such as Sindhi, written in Arabic. The results point to SentencePiece as the more effective tokenization strategy for NER within multilingual and low resource Indic NLP applications.
• 摘要：令牌化是自然语言处理（NLP）的关键组成部分，尤其是对于低资源语言，子词分割影响词汇结构和下游任务准确性。尽管字节对编码（BPE）是多语言语言模型中的标准令牌化方法，但由于其在处理形态复杂性方面的限制，其对命名实体识别（NER）的适用性仍未得到充满激发。在这项工作中，我们系统地比较了使用Adodert的BPE，句子和角色级别的令牌化策略，以低资源指示语言中的NER任务，例如Assamese，Bengali，Marathi和Odia，以及非常低的资源指示语言，例如Santali，Manipuri和Sindhi。我们评估了内在的语言特性令牌化效率，词汇效率（OOV）率，形态保存以及外部下游性能，包括微调和零射击交叉舌传递。我们的实验表明，低资源指示语言中的句子始终是比NER的BPE始终更好的方法，尤其是在零射击交叉语言设置中，因为它可以更好地保留实体的一致性。尽管BPE提供了最紧凑的令牌化形式，但它不能进行概括，因为在未见语言测试时，它错误分类甚至无法识别实体标签。相比之下，句子构成了一种更好的语言结构保护模型，从而使资源极低，形态上丰富的语言（例如Santali和Manipuri），用于上等实体识别，以及跨脚本（例如Sindhi）的高度概括，例如Sindhi，用阿拉伯语编写。结果指出，句子是多语言和低资源中NER的更有效的令牌化策略，指示NLP应用程序。

Title: Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation

• Authors: Luca Moroni, Giovanni Puccetti, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell'Orletta, Andrea Esuli, Roberto Navigli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17025
• Pdf URL: https://arxiv.org/pdf/2504.17025
• Copy Paste: [[2504.17025]] Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation(https://arxiv.org/abs/2504.17025)
• Keywords: language model, llm
• Abstract: The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs can handle other languages, due to language contamination or some degree of multilingual pretraining data, they are not optimized for non-English languages, leading to inefficient encoding (high token "fertility") and slower inference speed. In this work, we thoroughly compare a variety of vocabulary adaptation techniques for optimizing English LLMs for the Italian language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a novel method that leverages neural mapping for vocabulary substitution. SAVA achieves competitive performance across multiple downstream tasks, enhancing grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and reducing the number of parameters by 1 billion. We show that, following the adaptation of the vocabulary, these models can recover their performance with a relatively limited stage of continual training on the target language. Finally, we test the capabilities of the adapted models on various multi-choice and generative tasks.
• 摘要：审计的大语言模型（LLMS）的数量正在稳步增加，尽管大多数主要是为英语设计的。尽管最先进的LLM可以处理其他语言，但由于语言污染或某种程度的多语言预处理数据，但它们并未针对非英语语言进行优化，从而导致编码效率低下（高令牌“生育力”）和较慢的推理速度。在这项工作中，我们彻底比较了各种词汇适应技术，以优化意大利语的英语LLM，并提出语义一致性词汇适应（SAVA），这是一种利用神经映射的新方法来用于词汇替代。萨瓦（Sava）在多个下游任务中实现竞争性能，从而增强了基础对齐策略。我们适应两个LLM：Mistral-7b-V0.1，将令牌生育能力降低25 \％，Llama-3.1-8B，优化了词汇，并将参数数量减少了10亿。我们表明，在适应词汇后，这些模型可以通过对目标语言的持续培训阶段恢复其性能。最后，我们在各种多项选择和生成任务上测试了改编模型的功能。

Title: Do Words Reflect Beliefs? Evaluating Belief Depth in Large Language Models

• Authors: Shariar Kabir, Kevin Esterling, Yue Dong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17052
• Pdf URL: https://arxiv.org/pdf/2504.17052
• Copy Paste: [[2504.17052]] Do Words Reflect Beliefs? Evaluating Belief Depth in Large Language Models(https://arxiv.org/abs/2504.17052)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are increasingly shaping political discourse, yet their responses often display inconsistency when subjected to scrutiny. While prior research has primarily categorized LLM outputs as left- or right-leaning to assess their political stances, a critical question remains: Do these responses reflect genuine internal beliefs or merely surface-level alignment with training data? To address this, we propose a novel framework for evaluating belief depth by analyzing (1) argumentative consistency and (2) uncertainty quantification. We evaluate 12 LLMs on 19 economic policies from the Political Compass Test, challenging their belief stability with both supportive and opposing arguments. Our analysis reveals that LLMs exhibit topic-specific belief stability rather than a uniform ideological stance. Notably, up to 95% of left-leaning models' responses and 89% of right-leaning models' responses remain consistent under the challenge, enabling semantic entropy to achieve high accuracy (AUROC=0.78), effectively distinguishing between surface-level alignment from genuine belief. These findings call into question the assumption that LLMs maintain stable, human-like political ideologies, emphasizing the importance of conducting topic-specific reliability assessments for real-world applications.
• 摘要：大型语言模型（LLM）越来越多地塑造政治话语，但是在受到审查时，他们的反应通常会显示出不一致的情况。虽然先前的研究主要将LLM输出分为左倾角或右倾斜以评估其政治立场，但一个关键的问题仍然存在：这些反应是否反映了真正的内部信念，还是仅仅与培训数据相结合？为了解决这个问题，我们提出了一个新的框架，用于通过分析（1）论证一致性和（2）不确定性定量来评估信念深度。我们通过政治指南测试对19个经济政策进行了12个LLM，以支持性和反对论点挑战他们的信仰稳定。我们的分析表明，LLM表现出特定于主题的信念稳定性，而不是统一的意识形态立场。值得注意的是，在挑战下，多达95％的左倾模型的响应和89％的右倾模型响应保持一致，从而使语义熵具有高度的精度（AUROC = 0.78），从而有效地区分了表面水平的对准与真实信念。这些发现引起了人们对LLM保持稳定，类似人类的政治意识形态的假设的质疑，强调了对现实世界应用进行特定于主题的可靠性评估的重要性。

Title: Agree to Disagree? A Meta-Evaluation of LLM Misgendering

• Authors: Arjun Subramonian, Vagrant Gautam, Preethi Seshadri, Dietrich Klakow, Kai-Wei Chang, Yizhou Sun
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2504.17075
• Pdf URL: https://arxiv.org/pdf/2504.17075
• Copy Paste: [[2504.17075]] Agree to Disagree? A Meta-Evaluation of LLM Misgendering(https://arxiv.org/abs/2504.17075)
• Keywords: llm
• Abstract: Numerous methods have been proposed to measure LLM misgendering, including probability-based evaluations (e.g., automatically with templatic sentences) and generation-based evaluations (e.g., with automatic heuristics or human validation). However, it has gone unexamined whether these evaluation methods have convergent validity, that is, whether their results align. Therefore, we conduct a systematic meta-evaluation of these methods across three existing datasets for LLM misgendering. We propose a method to transform each dataset to enable parallel probability- and generation-based evaluation. Then, by automatically evaluating a suite of 6 models from 3 families, we find that these methods can disagree with each other at the instance, dataset, and model levels, conflicting on 20.2% of evaluation instances. Finally, with a human evaluation of 2400 LLM generations, we show that misgendering behaviour is complex and goes far beyond pronouns, which automatic evaluations are not currently designed to capture, suggesting essential disagreement with human evaluations. Based on our findings, we provide recommendations for future evaluations of LLM misgendering. Our results are also more widely relevant, as they call into question broader methodological conventions in LLM evaluation, which often assume that different evaluation methods agree.
• 摘要：已经提出了许多方法来衡量LLM错误的错误性，包括基于概率的评估（例如，使用灯泡句子自动）和基于生成的评估（例如，具有自动启发式方法或人类验证）。但是，这些评估方法是否具有收敛有效性，即它们的结果是否对齐，这已经尚未审查。因此，我们在三个现有数据集中对这些方法进行了系统的元评估，以实现LLM误解。我们提出了一种转换每个数据集以实现并行概率和基于生成的评估的方法。然后，通过自动评估来自3个家族的6个模型的套件，我们发现这些方法在此实例，数据集和模型级别上可以彼此不同意，在20.2％的评估实例上冲突。最后，通过对2400 LLM世代的人类评估，我们表明错误性行为的行为很复杂，并且远远超出了代词，而这些行为目前尚未旨在捕获自动评估，这表明与人类评估的基本分歧。根据我们的发现，我们为将来评估LLM误解的建议提供了建议。我们的结果也更加相关，因为他们在LLM评估中提出了更广泛的方法论惯例，这通常认为不同的评估方法一致。

Title: How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study

• Authors: Rendi Chevi, Kentaro Inui, Thamar Solorio, Alham Fikri Aji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17083
• Pdf URL: https://arxiv.org/pdf/2504.17083
• Copy Paste: [[2504.17083]] How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study(https://arxiv.org/abs/2504.17083)
• Keywords: llm, hallucination
• Abstract: What makes an interaction with the LLM more preferable for the user? While it is intuitive to assume that information accuracy in the LLM's responses would be one of the influential variables, recent studies have found that inaccurate LLM's responses could still be preferable when they are perceived to be more authoritative, certain, well-articulated, or simply verbose. These variables interestingly fall under the broader category of language style, implying that the style in the LLM's responses might meaningfully influence users' preferences. This hypothesized dynamic could have double-edged consequences: enhancing the overall user experience while simultaneously increasing their susceptibility to risks such as LLM's misinformation or hallucinations. In this short paper, we present our preliminary studies in exploring this subject. Through a series of exploratory and experimental user studies, we found that LLM's language style does indeed influence user's preferences, but how and which language styles influence the preference varied across different user populations, and more interestingly, moderated by the user's very own individual traits. As a preliminary work, the findings in our studies should be interpreted with caution, particularly given the limitations in our samples, which still need wider demographic diversity and larger sample sizes. Our future directions will first aim to address these limitations, which would enable a more comprehensive joint effect analysis between the language style, individual traits, and preferences, and further investigate the potential causal relationship between and beyond these variables.
• 摘要：是什么使与用户更可取的LLM互动？尽管直观地假设LLM响应中的信息准确性将是有影响力的变量之一，但最近的研究发现，当LLM的反应不准确时，当它们被认为是更具权威性，某些，良好的，良好的或简单的verbose时，它们的响应仍然是可取的。有趣的是，有趣的是，这些变量属于更广泛的语言样式类别，这意味着LLM响应中的样式可能会有意义地影响用户的偏好。这种假设的动态可能会带来双重的后果：增强整体用户体验，同时增加其对LLM的错误信息或幻觉等风险的敏感性。在这篇简短的论文中，我们介绍了探索这一主题的初步研究。通过一系列探索性和实验性用户研究，我们发现LLM的语言风格确实会影响用户的偏好，但是如何和哪些语言样式会影响不同的用户群体的偏好各不相同，更有趣的是，用户自己的个人特征更加主持。作为一项初步工作，我们的研究中的发现应谨慎解释，尤其是考虑到我们样本中的局限性，这些局限性仍然需要更广泛的人口统计学多样性和更大的样本量。我们未来的方向将首先旨在解决这些局限性，这将使语言风格，个体特征和偏好之间进行更全面的联合效应分析，并进一步研究这些变量之间和以后的潜在因果关系。

Title: Co-CoT: A Prompt-Based Framework for Collaborative Chain-of-Thought Reasoning

• Authors: Seunghyun Yoo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17091
• Pdf URL: https://arxiv.org/pdf/2504.17091
• Copy Paste: [[2504.17091]] Co-CoT: A Prompt-Based Framework for Collaborative Chain-of-Thought Reasoning(https://arxiv.org/abs/2504.17091)
• Keywords: prompt, chain-of-thought
• Abstract: Due to the proliferation of short-form content and the rapid adoption of AI, opportunities for deep, reflective thinking have significantly diminished, undermining users' critical thinking and reducing engagement with the reasoning behind AI-generated outputs. To address this issue, we propose an Interactive Chain-of-Thought (CoT) Framework that enhances human-centered explainability and responsible AI usage by making the model's inference process transparent, modular, and user-editable. The framework decomposes reasoning into clearly defined blocks that users can inspect, modify, and re-execute, encouraging active cognitive engagement rather than passive consumption. It further integrates a lightweight edit-adaptation mechanism inspired by preference learning, allowing the system to align with diverse cognitive styles and user intentions. Ethical transparency is ensured through explicit metadata disclosure, built-in bias checkpoint functionality, and privacy-preserving safeguards. This work outlines the design principles and architecture necessary to promote critical engagement, responsible interaction, and inclusive adaptation in AI systems aimed at addressing complex societal challenges.
• 摘要：由于短形式含量的扩散和AI的快速采用，深刻，反思性思维的机会大大减少，破坏了用户的批判性思维，并减少了与AI生成的产出背后的推理的参与。为了解决这个问题，我们提出了一个交互式链链（COT）框架，该框架通过使模型的推理过程透明，模块化和用户编辑来增强以人为本的解释性和负责任的AI使用。该框架将推理分解为明确定义的块，用户可以检查，修改和重新执行，鼓励主动认知参与，而不是被动消费。它进一步集成了受偏好学习启发的轻巧编辑适应机制，从而使系统能够与各种认知风格和用户意图保持一致。通过明确的元数据披露，内置偏见检查点功能和隐私保护保障来确保道德透明度。这项工作概述了旨在解决旨在应对复杂社会挑战的AI系统中的批判性参与，负责任的互动和包容性适应所必需的设计原理和体系结构。

Title: The Rise of Small Language Models in Healthcare: A Comprehensive Survey

• Authors: Muskan Garg, Shaina Raza, Shebuti Rayana, Xingyi Liu, Sunghwan Sohn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17119
• Pdf URL: https://arxiv.org/pdf/2504.17119
• Copy Paste: [[2504.17119]] The Rise of Small Language Models in Healthcare: A Comprehensive Survey(https://arxiv.org/abs/2504.17119)
• Keywords: language model, llm, prompt
• Abstract: Despite substantial progress in healthcare applications driven by large language models (LLMs), growing concerns around data privacy, and limited resources; the small language models (SLMs) offer a scalable and clinically viable solution for efficient performance in resource-constrained environments for next-generation healthcare informatics. Our comprehensive survey presents a taxonomic framework to identify and categorize them for healthcare professionals and informaticians. The timeline of healthcare SLM contributions establishes a foundational framework for analyzing models across three dimensions: NLP tasks, stakeholder roles, and the continuum of care. We present a taxonomic framework to identify the architectural foundations for building models from scratch; adapting SLMs to clinical precision through prompting, instruction fine-tuning, and reasoning; and accessibility and sustainability through compression techniques. Our primary objective is to offer a comprehensive survey for healthcare professionals, introducing recent innovations in model optimization and equipping them with curated resources to support future research and development in the field. Aiming to showcase the groundbreaking advancements in SLMs for healthcare, we present a comprehensive compilation of experimental results across widely studied NLP tasks in healthcare to highlight the transformative potential of SLMs in healthcare. The updated repository is available at Github
• 摘要：尽管大型语言模型（LLM）驱动的医疗保健应用程序取得了长足的进步，但对数据隐私的担忧日益加剧，并且资源有限；小型语言模型（SLM）为下一代医疗保健信息学提供了可扩展且临床上可行的解决方案。我们的综合调查提出了一个分类框架，以识别和将其分类为医疗保健专业人员和知情者。 Healthcare SLM贡献的时间表为分析三个维度的模型：NLP任务，利益相关者角色和护理连续体建立了一个基础框架。我们提出了一个分类框架，以确定从头开始建筑模型的建筑基础；通过提示，微调和推理将SLM适应临床精度；以及通过压缩技术的可及性和可持续性。我们的主要目标是为医疗保健专业人员提供全面的调查，引入模型优化方面的最新创新，并为他们提供精心策划的资源，以支持该领域的未来研究和发展。为了展示SLM在医疗保健方面的开创性进步，我们在医疗保健中广泛研究的NLP任务中进行了全面的实验结果汇编，以突出SLM在医疗保健中的变革潜力。更新的存储库可在GitHub上找到

Title: Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Control

• Authors: Hannah Cyberey, David Evans
• Subjects: cs.CL, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2504.17130
• Pdf URL: https://arxiv.org/pdf/2504.17130
• Copy Paste: [[2504.17130]] Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Control(https://arxiv.org/abs/2504.17130)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have transformed the way we access information. These models are often tuned to refuse to comply with requests that are considered harmful and to produce responses that better align with the preferences of those who control the models. To understand how this "censorship" works. We use representation engineering techniques to study open-weights safety-tuned models. We present a method for finding a refusal--compliance vector that detects and controls the level of censorship in model outputs. We also analyze recent reasoning LLMs, distilled from DeepSeek-R1, and uncover an additional dimension of censorship through "thought suppression". We show a similar approach can be used to find a vector that suppresses the model's reasoning process, allowing us to remove censorship by applying the negative multiples of this vector
• 摘要：大型语言模型（LLMS）已改变了我们访问信息的方式。这些模型通常被调整为拒绝遵守被认为有害的请求，并产生与控制模型的人的偏好更好的响应。了解这种“审查制度”是如何工作的。我们使用代表工程技术来研究开放式安全调整模型。我们提出了一种找到拒绝 - 合规矢量的方法，该矢量检测和控制模型输出中审查水平的水平。我们还分析了从DeepSeek-R1提炼的最新推理LLM，并通过“思想抑制”揭示了审查制度的额外维度。我们显示类似的方法可以用于找到抑制模型推理过程的向量，从而使我们可以通过应用此向量的负倍数来删除审查

Title: MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation

• Authors: Chanhee Park, Hyeonseok Moon, Chanjun Park, Heuiseok Lim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17137
• Pdf URL: https://arxiv.org/pdf/2504.17137
• Copy Paste: [[2504.17137]] MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation(https://arxiv.org/abs/2504.17137)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has gained prominence as an effective method for enhancing the generative capabilities of Large Language Models (LLMs) through the incorporation of external knowledge. However, the evaluation of RAG systems remains a challenge, due to the intricate interplay between retrieval and generation components. This limitation has resulted in a scarcity of benchmarks that facilitate a detailed, component-specific assessment. In this work, we present MIRAGE, a Question Answering dataset specifically designed for RAG evaluation. MIRAGE consists of 7,560 curated instances mapped to a retrieval pool of 37,800 entries, enabling an efficient and precise evaluation of both retrieval and generation tasks. We also introduce novel evaluation metrics aimed at measuring RAG adaptability, encompassing dimensions such as noise vulnerability, context acceptability, context insensitivity, and context misinterpretation. Through comprehensive experiments across various retriever-LLM configurations, we provide new insights into the optimal alignment of model pairs and the nuanced dynamics within RAG systems. The dataset and evaluation code are publicly available, allowing for seamless integration and customization in diverse research settings\footnote{The MIRAGE code and data are available at this https URL.
• 摘要：通过外部知识的结合，检索增强的生成（RAG）已成为一种有效的方法来增强大语言模型（LLMS）的生成能力的有效方法。但是，由于检索和发电组件之间的复杂相互作用，对抹布系统的评估仍然是一个挑战。这种局限性导致了基准的稀缺性，这些基准有助于详细的，特定于组成的评估。在这项工作中，我们提出了Mirage，这是一个问题，该问题回答了专门为抹布评估而设计的数据集。 Mirage由7,560个策划实例组成，这些实例映射到37,800个条目的检索池，从而对检索和发电任务进行了有效而精确的评估。我们还介绍了旨在衡量破布适应性的新颖评估指标，包括噪声脆弱性，上下文可接受性，上下文不敏感和上下文误解等维度。通过在各种Retriever-LLM配置上进行的全面实验，我们为模型对的最佳比对和抹布系统中细微的动态提供了新的见解。数据集和评估代码可公开可用，可以在不同的研究设置中进行无缝集成和自定义\脚注{Mirage代码和数据可在此HTTPS URL上获得。

Title: Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

• Authors: Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17192
• Pdf URL: https://arxiv.org/pdf/2504.17192
• Copy Paste: [[2504.17192]] Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning(https://arxiv.org/abs/2504.17192)
• Keywords: language model, llm, agent
• Abstract: Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Language Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into functional code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, specifically from the original paper authors, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins.
• 摘要：尽管机器学习研究的迅速增长，但相应的代码实施通常无法实现，这使研究人员重现结果并在先前的工作基础上进行劳动密集型。同时，最近的大型语言模型（LLMS）在了解科学文档并生成高质量代码方面表现出色。受此启发的启发，我们介绍了PaperCoder，这是一个多代理LLM框架，将机器学习论文转换为功能代码存储库。 PaperCoder在三个阶段进行操作：规划，它在其中构建高级路线图，使用图表设计系统体系结构，标识文件依赖项并生成配置文件；分析，侧重于解释特定于实施的细节；和生成，在其中产生模块化，依赖感知代码。此外，每个阶段都是通过旨在在整个管道中有效协作的一组专业代理来实例化的。然后，我们根据基于模型和人类评估的机器学习论文（特别是来自原始纸张作者）的机器学习论文生成代码实现的评估，以作者发行的存储库作为基础真相。我们的结果表明，PaperCoder在创建高质量，忠实的实施方面的有效性。此外，它始终显示出最近发布的PaperBench基准测试中的优势，超过了强大的基线。

Title: A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation

• Authors: Yangxinyu Xie, Bowen Jiang, Tanwi Mallick, Joshua David Bergerson, John K. Hutchison, Duane R. Verner, Jordan Branham, M. Ross Alexander, Robert B. Ross, Yan Feng, Leslie-Anne Levy, Weijie Su, Camillo J. Taylor
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17200
• Pdf URL: https://arxiv.org/pdf/2504.17200
• Copy Paste: [[2504.17200]] A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation(https://arxiv.org/abs/2504.17200)
• Keywords: language model, gpt, llm, retrieval-augmented generation, agent
• Abstract: Large language models (LLMs) are a transformational capability at the frontier of artificial intelligence and machine learning that can support decision-makers in addressing pressing societal challenges such as extreme natural hazard events. As generalized models, LLMs often struggle to provide context-specific information, particularly in areas requiring specialized knowledge. In this work we propose a retrieval-augmented generation (RAG)-based multi-agent LLM system to support analysis and decision-making in the context of natural hazards and extreme weather events. As a proof of concept, we present WildfireGPT, a specialized system focused on wildfire hazards. The architecture employs a user-centered, multi-agent design to deliver tailored risk insights across diverse stakeholder groups. By integrating natural hazard and extreme weather projection data, observational datasets, and scientific literature through an RAG framework, the system ensures both the accuracy and contextual relevance of the information it provides. Evaluation across ten expert-led case studies demonstrates that WildfireGPT significantly outperforms existing LLM-based solutions for decision support.
• 摘要：大型语言模型（LLM）是人工智能和机器学习边界的转型能力，可以支持决策者解决紧迫的社会挑战，例如极端自然危害事件。作为广义模型，LLMS通常很难提供特定于上下文的信息，尤其是在需要专业知识的领域。在这项工作中，我们提出了基于自然危害和极端天气事件的背景下的基于检索的增强生成（RAG）的多代理LLM系统。作为概念证明，我们提出了野火野外野火系统，该系统侧重于野火危害。该体系结构采用以用户为中心的多代理设计来提供跨不同利益相关者群体的量身定制风险见解。通过整合自然危害和极端天气投影数据，观察数据集和科学文献，该系统可确保其提供的信息的准确性和上下文相关性。跨十项专家案例研究的评估表明，Wildfiregpt显着优于现有的基于LLM的解决方案以获得决策支持。

Title: Does Knowledge Distillation Matter for Large Language Model based Bundle Generation?

• Authors: Kaidong Feng, Zhu Sun, Jie Yang, Hui Fang, Xinghua Qu, Wenyuan Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2504.17220
• Pdf URL: https://arxiv.org/pdf/2504.17220
• Copy Paste: [[2504.17220]] Does Knowledge Distillation Matter for Large Language Model based Bundle Generation?(https://arxiv.org/abs/2504.17220)
• Keywords: language model, llm
• Abstract: LLMs are increasingly explored for bundle generation, thanks to their reasoning capabilities and knowledge. However, deploying large-scale LLMs introduces significant efficiency challenges, primarily high computational costs during fine-tuning and inference due to their massive parameterization. Knowledge distillation (KD) offers a promising solution, transferring expertise from large teacher models to compact student models. This study systematically investigates knowledge distillation approaches for bundle generation, aiming to minimize computational demands while preserving performance. We explore three critical research questions: (1) how does the format of KD impact bundle generation performance? (2) to what extent does the quantity of distilled knowledge influence performance? and (3) how do different ways of utilizing the distilled knowledge affect performance? We propose a comprehensive KD framework that (i) progressively extracts knowledge (patterns, rules, deep thoughts); (ii) captures varying quantities of distilled knowledge through different strategies; and (iii) exploits complementary LLM adaptation techniques (in-context learning, supervised fine-tuning, combination) to leverage distilled knowledge in small student models for domain-specific adaptation and enhanced efficiency. Extensive experiments provide valuable insights into how knowledge format, quantity, and utilization methodologies collectively shape LLM-based bundle generation performance, exhibiting KD's significant potential for more efficient yet effective LLM-based bundle generation.
• 摘要：由于其推理能力和知识，LLM越来越多地探索捆绑生成。但是，部署大规模LLMS引入了重大效率挑战，由于其大规模参数化而在微调和推理期间主要是高计算成本。知识蒸馏（KD）提供了一个有希望的解决方案，将专业知识从大型教师模型转移到紧凑的学生模型。这项研究系统地研究了捆绑生成的知识蒸馏方法，旨在最大程度地减少计算需求，同时保持性能。我们探讨了三个关键的研究问题：（1）KD的格式如何影响捆绑发电的性能？ （2）蒸馏知识的数量在多大程度上影响性能？ （3）使用蒸馏知识的不同方式如何影响性能？我们提出了一个全面的KD框架，该框架（i）逐步提取知识（模式，规则，深思）； （ii）通过不同的策略捕获不同数量的蒸馏知识； （iii）利用互补的LLM适应技术（内在学习，监督微调，组合）来利用小型学生模型中的蒸馏知识来获得特定于领域的适应性和提高效率。广泛的实验提供了有关知识格式，数量和利用方法如何共同塑造基于LLM的捆绑生成性能的宝贵见解，从而表现出KD具有更有效但有效的基于LLM的捆绑捆绑生成的巨大潜力。

Title: Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialogues

• Authors: Jinfeng Zhou, Yuxuan Chen, Jianing Yin, Yongkang Huang, Yihan Shi, Xikun Zhang, Libiao Peng, Rongsheng Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, Minlie Huang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2504.17238
• Pdf URL: https://arxiv.org/pdf/2504.17238
• Copy Paste: [[2504.17238]] Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialogues(https://arxiv.org/abs/2504.17238)
• Keywords: llm
• Abstract: Cognitive Restructuring (CR) is a psychotherapeutic process aimed at identifying and restructuring an individual's negative thoughts, arising from mental health challenges, into more helpful and positive ones via multi-turn dialogues. Clinician shortage and stigma urge the development of human-LLM interactive psychotherapy for CR. Yet, existing efforts implement CR via simple text rewriting, fixed-pattern dialogues, or a one-shot CR workflow, failing to align with the psychotherapeutic process for effective CR. To address this gap, we propose CRDial, a novel framework for CR, which creates multi-turn dialogues with specifically designed identification and restructuring stages of negative thoughts, integrates sentence-level supportive conversation strategies, and adopts a multi-channel loop mechanism to enable iterative CR. With CRDial, we distill Crisp, a large-scale and high-quality bilingual dialogue dataset, from LLM. We then train Crispers, Crisp-based conversational LLMs for CR, at 7B and 14B scales. Extensive human studies show the superiority of Crispers in pointwise, pairwise, and intervention evaluations.
• 摘要：认知重组（CR）是一个心理治疗过程，旨在通过心理健康挑战引起和重组个人的负面思想，并通过多转向对话进行更有帮助和积极的对话。临床医生的短缺和污名敦促开发人类互动心理治疗CR。然而，现有的努力通过简单的文本重写，固定模式对话或一次性CR工作流程实现CR，无法与有效CR的心理治疗过程保持一致。为了解决这一差距，我们提出了Crdial，这是CR的新型框架，该框架通过专门设计的识别和重组阶段创建了多转的对话，整合了句子级的支持性对话策略，并采用了多通道环路机制来启用迭代性CR。通过Crdial，我们从LLM提炼了Crist Crisp，这是一个大规模和高质量的双语对话数据集。然后，我们在7B和14B尺度上训练CR CRISPERS，基于清晰的对话llms。广泛的人类研究表明，在角度，成对和干预评估中，Crispers的优越性。

Title: JurisCTC: Enhancing Legal Judgment Prediction via Cross-Domain Transfer and Contrastive Learning

• Authors: Zhaolu Kang, Hongtian Cai, Xiangyang Ji, Jinzhe Li, Nanfei Gu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17264
• Pdf URL: https://arxiv.org/pdf/2504.17264
• Copy Paste: [[2504.17264]] JurisCTC: Enhancing Legal Judgment Prediction via Cross-Domain Transfer and Contrastive Learning(https://arxiv.org/abs/2504.17264)
• Keywords: language model, llm
• Abstract: In recent years, Unsupervised Domain Adaptation (UDA) has gained significant attention in the field of Natural Language Processing (NLP) owing to its ability to enhance model generalization across diverse domains. However, its application for knowledge transfer between distinct legal domains remains largely unexplored. To address the challenges posed by lengthy and complex legal texts and the limited availability of large-scale annotated datasets, we propose JurisCTC, a novel model designed to improve the accuracy of Legal Judgment Prediction (LJP) tasks. Unlike existing approaches, JurisCTC facilitates effective knowledge transfer across various legal domains and employs contrastive learning to distinguish samples from different domains. Specifically, for the LJP task, we enable knowledge transfer between civil and criminal law domains. Compared to other models and specific large language models (LLMs), JurisCTC demonstrates notable advancements, achieving peak accuracies of 76.59% and 78.83%, respectively.
• 摘要：近年来，由于其能够增强跨不同领域的模型泛化的能力，因此无监督的领域适应性（UDA）在自然语言处理（NLP）领域受到了重大关注。但是，其在不同法律领域之间进行知识转移的应用仍未得到探索。为了解决冗长且复杂的法律文本带来的挑战以及大规模注释数据集的有限可用性，我们提出了尤其是一种新型模型，旨在提高法律判断预测（LJP）任务的准确性。与现有的方法不同，牙齿eTISCTC促进了跨各个法律领域的有效知识转移，并采用对比度学习以将样本与不同领域区分开。具体而言，对于LJP任务，我们可以在民事和刑法领域之间进行知识转移。与其他模型和特定的大语言模型（LLMS）相比，Juistctc证明了显着的进步，分别达到76.59％和78.83％的峰精度。

Title: CoheMark: A Novel Sentence-Level Watermark for Enhanced Text Quality

• Authors: Junyan Zhang, Shuliang Liu, Aiwei Liu, Yubo Gao, Jungang Li, Xiaojie Gu, Xuming Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17309
• Pdf URL: https://arxiv.org/pdf/2504.17309
• Copy Paste: [[2504.17309]] CoheMark: A Novel Sentence-Level Watermark for Enhanced Text Quality(https://arxiv.org/abs/2504.17309)
• Keywords: language model
• Abstract: Watermarking technology is a method used to trace the usage of content generated by large language models. Sentence-level watermarking aids in preserving the semantic integrity within individual sentences while maintaining greater robustness. However, many existing sentence-level watermarking techniques depend on arbitrary segmentation or generation processes to embed watermarks, which can limit the availability of appropriate sentences. This limitation, in turn, compromises the quality of the generated response. To address the challenge of balancing high text quality with robust watermark detection, we propose CoheMark, an advanced sentence-level watermarking technique that exploits the cohesive relationships between sentences for better logical fluency. The core methodology of CoheMark involves selecting sentences through trained fuzzy c-means clustering and applying specific next sentence selection criteria. Experimental evaluations demonstrate that CoheMark achieves strong watermark strength while exerting minimal impact on text quality.
• 摘要：水印技术是一种用于追踪大语言模型生成的内容的用法。句子级水印有助于保留单个句子中的语义完整性，同时保持更大的鲁棒性。但是，许多现有的句子级水印技术取决于嵌入水印的任意细分或发电过程，这可能会限制适当句子的可用性。反过来，这种限制损害了生成的响应的质量。为了解决平衡高文本质量与强大的水印检测的挑战，我们提出了Cohemark，这是一种先进的句子级水印技术，利用句子之间的凝聚力关系以获得更好的逻辑流利性。人群的核心方法涉及通过训练有素的模糊c均值聚类选择句子，并应用特定的下一个句子选择标准。实验评估表明，同事可以达到强大的水印强度，同时对文本质量产生最小的影响。

Title: FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation

• Authors: Yulia Otmakhova, Hung Thinh Truong, Rahmad Mahendra, Zenan Zhai, Rongxin Zhu, Daniel Beck, Jey Han Lau
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17311
• Pdf URL: https://arxiv.org/pdf/2504.17311
• Copy Paste: [[2504.17311]] FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation(https://arxiv.org/abs/2504.17311)
• Keywords: language model, llm
• Abstract: We present FLUKE (Framework for LingUistically-driven and tasK-agnostic robustness Evaluation), a task-agnostic framework for assessing model robustness through systematic minimal variations of test data. FLUKE introduces controlled variations across linguistic levels - from orthography to dialect and style varieties - and leverages large language models (LLMs) with human validation to generate modifications. We demonstrate FLUKE's utility by evaluating both fine-tuned models and LLMs across four diverse NLP tasks, and reveal that (1) the impact of linguistic variations is highly task-dependent, with some tests being critical for certain tasks but irrelevant for others; (2) while LLMs have better overall robustness compared to fine-tuned models, they still exhibit significant brittleness to certain linguistic variations; (3) all models show substantial vulnerability to negation modifications across most tasks. These findings highlight the importance of systematic robustness testing for understanding model behaviors.
• 摘要：我们提出Fluke（语言驱动和任务敏捷性鲁棒性评估的框架），这是一个任务不合时宜的框架，用于通过系统的测试数据进行系统的最小变化来评估模型鲁棒性。 Fluke引入了跨语言级别的受控变化 - 从拼字法到方言和样式品种 - 并利用人类验证的大语言模型（LLMS）来生成修改。我们通过评估四个不同的NLP任务中的微调模型和LLM来证明Fluke的实用性，并揭示（1）语言变化的影响高度依赖于任务，其中某些测试对于某些任务至关重要，但对其他任务无关； （2）尽管LLM与微调模型相比具有更好的总体鲁棒性，但它们仍然对某些语言变化表现出明显的脆性； （3）所有模型均显示出在大多数任务中对否定修改的实质性脆弱性。这些发现突出了系统鲁棒性测试对理解模型行为的重要性。

Title: Bridging Cognition and Emotion: Empathy-Driven Multimodal Misinformation Detection

• Authors: Zihan Wang, Lu Yuan, Zhengxuan Zhang, Qing Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17332
• Pdf URL: https://arxiv.org/pdf/2504.17332
• Copy Paste: [[2504.17332]] Bridging Cognition and Emotion: Empathy-Driven Multimodal Misinformation Detection(https://arxiv.org/abs/2504.17332)
• Keywords: language model, llm
• Abstract: In the digital era, social media has become a major conduit for information dissemination, yet it also facilitates the rapid spread of misinformation. Traditional misinformation detection methods primarily focus on surface-level features, overlooking the crucial roles of human empathy in the propagation process. To address this gap, we propose the Dual-Aspect Empathy Framework (DAE), which integrates cognitive and emotional empathy to analyze misinformation from both the creator and reader perspectives. By examining creators' cognitive strategies and emotional appeals, as well as simulating readers' cognitive judgments and emotional responses using Large Language Models (LLMs), DAE offers a more comprehensive and human-centric approach to misinformation detection. Moreover, we further introduce an empathy-aware filtering mechanism to enhance response authenticity and diversity. Experimental results on benchmark datasets demonstrate that DAE outperforms existing methods, providing a novel paradigm for multimodal misinformation detection.
• 摘要：在数字时代，社交媒体已成为信息传播的主要渠道，但它也促进了错误信息的迅速传播。传统的错误信息检测方法主要集中于表面级特征，忽视了人类同理心在传播过程中的关键作用。为了解决这一差距，我们提出了双重同情框架（DAE），该框架将认知和情感同理心整合在一起，以分析创建者和读者的观点的错误信息。通过检查创作者的认知策略和情感吸引力，并使用大语言模型（LLMS）模拟读者的认知判断和情感反应，DAE提供了一种更全面和以人为中心的方法来检测错误的方法。此外，我们进一步引入了一种移情感知的过滤机制，以增强响应的真实性和多样性。基准数据集的实验结果表明，DAE胜过现有方法，为多模式错误信息检测提供了一种新颖的范式。

Title: M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction

• Authors: Chengguang Gan, Sunbowen Lee, Zhixi Cai, Yanbin Wei, Lei Zheng, Yunhao Liang, Shiwen Ni, Tatsunori Mori
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.17353
• Pdf URL: https://arxiv.org/pdf/2504.17353
• Copy Paste: [[2504.17353]] M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction(https://arxiv.org/abs/2504.17353)
• Keywords: language model, prompt
• Abstract: Mutual Reinforcement Effect (MRE) is an emerging subfield at the intersection of information extraction and model interpretability. MRE aims to leverage the mutual understanding between tasks of different granularities, enhancing the performance of both coarse-grained and fine-grained tasks through joint modeling. While MRE has been explored and validated in the textual domain, its applicability to visual and multimodal domains remains unexplored. In this work, we extend MRE to the multimodal information extraction domain for the first time. Specifically, we introduce a new task: Multimodal Mutual Reinforcement Effect (M-MRE), and construct a corresponding dataset to support this task. To address the challenges posed by M-MRE, we further propose a Prompt Format Adapter (PFA) that is fully compatible with various Large Vision-Language Models (LVLMs). Experimental results demonstrate that MRE can also be observed in the M-MRE task, a multimodal text-image understanding scenario. This provides strong evidence that MRE facilitates mutual gains across three interrelated tasks, confirming its generalizability beyond the textual domain.
• 摘要：相互加固效应（MRE）是信息提取和模型可解释性交集的新兴子场。 MRE旨在利用不同粒度的任务之间的相互理解，从而通过关节建模来增强粗粒和细粒度任务的性能。虽然在文本域中探索和验证了MRE，但其对视觉和多模式域的适用性仍未得到探索。在这项工作中，我们首次将MRE扩展到多模式信息提取域。具体来说，我们介绍了一个新任务：多模式相互加固效果（M-MRE），并构建一个相应的数据集来支持此任务。为了应对M-MRE所带来的挑战，我们进一步提出了一个及时的格式适配器（PFA），该适配器与各种大型视觉模型（LVLMS）完全兼容。实验结果表明，在M-MRE任务中也可以观察到MRE，这是一种多模式的文本图像理解方案。这提供了有力的证据表明，MRE促进了三个相互关联的任务中的相互收益，从而证实了其概括性超出了文本领域。

Title: PatientDx: Merging Large Language Models for Protecting Data-Privacy in Healthcare

• Authors: Jose G. Moreno (IRIT-IRIS), Jesus Lovon (IRIT-IRIS), M'Rick Robin-Charlet (UT3), Christine Damase-Michel, Lynda Tamine (IRIT-IRIS)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17360
• Pdf URL: https://arxiv.org/pdf/2504.17360
• Copy Paste: [[2504.17360]] PatientDx: Merging Large Language Models for Protecting Data-Privacy in Healthcare(https://arxiv.org/abs/2504.17360)
• Keywords: language model, llm
• Abstract: Fine-tuning of Large Language Models (LLMs) has become the default practice for improving model performance on a given task. However, performance improvement comes at the cost of training on vast amounts of annotated data which could be sensitive leading to significant data privacy concerns. In particular, the healthcare domain is one of the most sensitive domains exposed to data privacy issues. In this paper, we present PatientDx, a framework of model merging that allows the design of effective LLMs for health-predictive tasks without requiring fine-tuning nor adaptation on patient data. Our proposal is based on recently proposed techniques known as merging of LLMs and aims to optimize a building block merging strategy. PatientDx uses a pivotal model adapted to numerical reasoning and tunes hyperparameters on examples based on a performance metric but without training of the LLM on these data. Experiments using the mortality tasks of the MIMIC-IV dataset show improvements up to 7% in terms of AUROC when compared to initial models. Additionally, we confirm that when compared to fine-tuned models, our proposal is less prone to data leak problems without hurting performance. Finally, we qualitatively show the capabilities of our proposal through a case study. Our best model is publicly available at this https URL Jgmorenof/mistral\_merged\_0\_4.
• 摘要：大型语言模型（LLM）的微调已成为改善给定任务上模型性能的默认做法。但是，绩效提高是以对大量注释数据进行培训的成本来提高的，这可能很敏感，从而导致重大数据隐私问题。特别是，医疗领域是暴露于数据隐私问题的最敏感的领域之一。在本文中，我们提出了DivateDX，即模型合并的框架，该框架允许设计有效的LLM用于健康预测任务，而无需对患者数据进行微调或适应。我们的建议基于最近提出的称为LLMS合并的技术，旨在优化基础合并策略。 DisteryDX使用适合数值推理的关键模型，并根据性能度量指标进行调音超参数调音，但在这些数据上未经LLM培训。与初始模型相比，使用MIMIC-IV数据集的死亡率任务的实验在AUROC方面显示出高达7％的改善。此外，我们确认与微调模型相比，我们的建议不容易出现数据泄漏问题而不会损害性能。最后，我们通过案例研究定性地展示了提案的能力。我们的最佳模型可在此HTTPS URL JGMORENOF/MISTRAL \ _MERGED \ _0 \ _4上公开获得。

Title: LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams

• Authors: Yongxuan Wu, Runyu Chen, Peiyu Liu, Hongjin Qian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17366
• Pdf URL: https://arxiv.org/pdf/2504.17366
• Copy Paste: [[2504.17366]] LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams(https://arxiv.org/abs/2504.17366)
• Keywords: language model, llm
• Abstract: Long-context understanding poses significant challenges in natural language processing, particularly for real-world dialogues characterized by speech-based elements, high redundancy, and uneven information density. Although large language models (LLMs) achieve impressive results on existing benchmarks, these datasets fail to reflect the complexities of such texts, limiting their applicability to practical scenarios. To bridge this gap, we construct the first spoken long-text dataset, derived from live streams, designed to reflect the redundancy-rich and conversational nature of real-world scenarios. We construct tasks in three categories: retrieval-dependent, reasoning-dependent, and hybrid. We then evaluate both popular LLMs and specialized methods to assess their ability to understand long-contexts in these tasks. Our results show that current methods exhibit strong task-specific preferences and perform poorly on highly redundant inputs, with no single method consistently outperforming others. We propose a new baseline that better handles redundancy in spoken text and achieves strong performance across tasks. Our findings highlight key limitations of current methods and suggest future directions for improving long-context understanding. Finally, our benchmark fills a gap in evaluating long-context spoken language understanding and provides a practical foundation for developing real-world e-commerce systems. The code and benchmark are available at this https URL.
• 摘要：长篇小说理解在自然语言处理中构成了重大挑战，尤其是对于以语音元素，高冗余性和不均匀信息密度为特征的现实对话。尽管大型语言模型（LLM）在现有基准上取得了令人印象深刻的结果，但这些数据集未能反映此类文本的复杂性，从而将其适用性限制在实际情况下。为了弥合这一差距，我们构建了第一个源自现场流的长篇文本数据集，旨在反映现实世界情景的冗余丰富和对话性质。我们分为三类构建任务：取回依赖性，依赖于推理和混合。然后，我们评估了流行的LLM和专业方法，以评估他们在这些任务中了解长篇小说的能力。我们的结果表明，当前的方法表现出强大的任务偏好，并且在高度冗余的输入上表现不佳，没有单一的方法始终超过其他方法。我们提出了一个新的基线，该基线可以更好地处理口语文本的冗余，并在整个任务中实现强大的绩效。我们的发现突出了当前方法的关键局限性，并提出了未来的方向，以改善长篇小说的理解。最后，我们的基准填补了评估长篇小说口语理解的空白，并为开发现实世界的电子商务系统提供了实践基础。该https URL可用代码和基准。

Title: PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona

• Authors: Jihyun Lee, Yejin Jeon, Seungyeon Seo, Gary Geunbae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17390
• Pdf URL: https://arxiv.org/pdf/2504.17390
• Copy Paste: [[2504.17390]] PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona(https://arxiv.org/abs/2504.17390)
• Keywords: hallucination, prompt
• Abstract: Task-Oriented Dialogue (TOD) systems are designed to fulfill user requests through natural language interactions, yet existing systems often produce generic, monotonic responses that lack individuality and fail to adapt to users' personal attributes. To address this, we introduce PicPersona-TOD, a novel dataset that incorporates user images as part of the persona, enabling personalized responses tailored to user-specific factors such as age or emotional context. This is facilitated by first impressions, dialogue policy-guided prompting, and the use of external knowledge to reduce hallucinations. Human evaluations confirm that our dataset enhances user experience, with personalized responses contributing to a more engaging interaction. Additionally, we introduce a new NLG model, Pictor, which not only personalizes responses, but also demonstrates robust performance across unseen domains this https URL.
• 摘要：以任务为导向的对话（TOD）系统旨在通过自然语言交互来满足用户请求，但是现有系统通常会产生通用的单调响应，这些响应缺乏个性，并且无法适应用户的个人属性。为了解决这个问题，我们介绍了Picpersona-TOD，这是一个新颖的数据集，将用户图像作为角色的一部分结合在一起，从而实现了针对特定于用户特定因素（例如年龄或情感上下文）的个性化响应。第一印象，对话政策引导的提示以及使用外部知识来减少幻觉的促进。人类评估证实，我们的数据集增强了用户体验，具有个性化的响应有助于更具吸引力的互动。此外，我们介绍了一种新的NLG模型Pictor，该模型不仅个性化了响应，而且还展示了在这个HTTPS URL的看不见的域中表现出色的性能。

Title: Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation

• Authors: Anna Lieb, Maneesh Arora, Eni Mustafaraj
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17445
• Pdf URL: https://arxiv.org/pdf/2504.17445
• Copy Paste: [[2504.17445]] Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation(https://arxiv.org/abs/2504.17445)
• Keywords: gpt, llm
• Abstract: Unsupervised machine learning techniques, such as topic modeling and clustering, are often used to identify latent patterns in unstructured text data in fields such as political science and sociology. These methods overcome common concerns about reproducibility and costliness involved in the labor-intensive process of human qualitative analysis. However, two major limitations of topic models are their interpretability and their practicality for answering targeted, domain-specific social science research questions. In this work, we investigate opportunities for using LLM-generated text augmentation to improve the usefulness of topic modeling output. We use a political science case study to evaluate our results in a domain-specific application, and find that topic modeling using GPT-4 augmentations creates highly interpretable categories that can be used to investigate domain-specific research questions with minimal human guidance.
• 摘要：无监督的机器学习技术（例如主题建模和聚类）通常用于识别政治学和社会学等领域中非结构化文本数据中的潜在模式。这些方法克服了人们对人类定性分析过程中涉及的可重复性和成本的普遍关注。但是，主题模型的两个主要局限性是它们的解释性和实用性用于回答针对性的，特定领域的社会科学研究问题。在这项工作中，我们调查了使用LLM生成的文本增强来提高主题建模输出的实用性的机会。我们使用一项政治学案例研究来评估我们在特定领域的应用中的结果，并发现使用GPT-4增强量的主题建模创建了高度可解释的类别，可用于以最少的人类指导来调查特定领域的研究问题。

Title: Unified Attacks to Large Language Model Watermarks: Spoofing and Scrubbing in Unauthorized Knowledge Distillation

• Authors: Xin Yi, Shunfan Zhengc, Linlin Wanga, Xiaoling Wang, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17480
• Pdf URL: https://arxiv.org/pdf/2504.17480
• Copy Paste: [[2504.17480]] Unified Attacks to Large Language Model Watermarks: Spoofing and Scrubbing in Unauthorized Knowledge Distillation(https://arxiv.org/abs/2504.17480)
• Keywords: language model, llm
• Abstract: Watermarking has emerged as a critical technique for combating misinformation and protecting intellectual property in large language models (LLMs). A recent discovery, termed watermark radioactivity, reveals that watermarks embedded in teacher models can be inherited by student models through knowledge distillation. On the positive side, this inheritance allows for the detection of unauthorized knowledge distillation by identifying watermark traces in student models. However, the robustness of watermarks against scrubbing attacks and their unforgeability in the face of spoofing attacks under unauthorized knowledge distillation remain largely unexplored. Existing watermark attack methods either assume access to model internals or fail to simultaneously support both scrubbing and spoofing attacks. In this work, we propose Contrastive Decoding-Guided Knowledge Distillation (CDG-KD), a unified framework that enables bidirectional attacks under unauthorized knowledge distillation. Our approach employs contrastive decoding to extract corrupted or amplified watermark texts via comparing outputs from the student model and weakly watermarked references, followed by bidirectional distillation to train new student models capable of watermark removal and watermark forgery, respectively. Extensive experiments show that CDG-KD effectively performs attacks while preserving the general performance of the distilled model. Our findings underscore critical need for developing watermarking schemes that are robust and unforgeable.
• 摘要：水印已成为打击错误信息和保护大语言模型（LLMS）中知识产权的关键技术。最近的发现，称为水印的放射性，表明嵌入在教师模型中的水印可以通过知识蒸馏来遗传。从积极的一面来看，这种继承可以通过识别学生模型中的水印痕迹来检测未经授权的知识蒸馏。然而，面对未经授权的知识蒸馏，水印不防止擦洗攻击及其在欺骗攻击的情况下，在很大程度上尚未探索。现有的水印攻击方法假设访问模型内​​部攻击，或者无法同时支持擦洗和欺骗攻击。在这项工作中，我们提出了对比度解码引导的知识蒸馏（CDG-KD），这是一个统一的框架，可以在未经授权的知识蒸馏中实现双向攻击。我们的方法采用对比度解码来通过比较学生模型的输出和弱水印参考的输出来提取损坏或放大的水印文本，然后进行双向蒸馏，以训练能够去除水印和水印的新学生模型，分别是水印和伪造的。广泛的实验表明，CDG-KD在保留蒸馏模型的一般性能的同时有效地执行了攻击。我们的发现强调了制定坚固且不可忍受的水印方案的关键需求。

Title: HalluLens: LLM Hallucination Benchmark

• Authors: Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, Pascale Fung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17550
• Pdf URL: https://arxiv.org/pdf/2504.17550
• Copy Paste: [[2504.17550]] HalluLens: LLM Hallucination Benchmark(https://arxiv.org/abs/2504.17550)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as "hallucination." These hallucinations undermine user trust and hinder the adoption of generative AI systems. Addressing hallucinations is essential for the advancement of LLMs. This paper introduces a comprehensive hallucination benchmark, incorporating both new extrinsic and existing intrinsic evaluation tasks, built upon clear taxonomy of hallucination. A major challenge in benchmarking hallucinations is the lack of a unified framework due to inconsistent definitions and categorizations. We disentangle LLM hallucination from "factuality," proposing a clear taxonomy that distinguishes between extrinsic and intrinsic hallucinations, to promote consistency and facilitate research. Extrinsic hallucinations, where the generated content is not consistent with the training data, are increasingly important as LLMs evolve. Our benchmark includes dynamic test set generation to mitigate data leakage and ensure robustness against such leakage. We also analyze existing benchmarks, highlighting their limitations and saturation. The work aims to: (1) establish a clear taxonomy of hallucinations, (2) introduce new extrinsic hallucination tasks, with data that can be dynamically regenerated to prevent saturation by leakage, (3) provide a comprehensive analysis of existing benchmarks, distinguishing them from factuality evaluations.
• 摘要：大型语言模型（LLMS）通常会产生偏离用户输入或培训数据的响应，这种现象称为“幻觉”。这些幻觉破坏了用户信任，并阻碍了生成AI系统的采用。解决幻觉对于LLM的发展至关重要。本文介绍了一个全面的幻觉基准，并结合了新的外部和现有的内在评估任务，并建立在清晰的幻觉分类法之上。基准幻觉的一个主要挑战是由于定义和分类不一致，缺乏统一的框架。我们将LLM幻觉与“事实”相关，提出了一种明确的分类法，该分类法区分外在和内在幻觉，以促进一致性并促进研究。随着LLMS的发展，生成的内容与培训数据不符，外在幻觉与培训数据不一致。我们的基准包括动态测试设置生成，以减轻数据泄漏并确保对这种泄漏的鲁棒性。我们还分析了现有的基准测试，突出了它们的局限性和饱和度。该作品的目的是：（1）建立清晰的幻觉分类法，（2）引入新的外部幻觉任务，可以通过动态重新生成以防止泄漏的数据，（3）对现有基准进行了全面分析，从而将其与事实评估区分开。

Title: When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars

• Authors: Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa, Kazusato Oko, Shoichiro Yamaguchi, Sosuke Kobayashi, Seiya Tokui, Kohei Hayashi, Daisuke Okanohara, Taiji Suzuki
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.17562
• Pdf URL: https://arxiv.org/pdf/2504.17562
• Copy Paste: [[2504.17562]] When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars(https://arxiv.org/abs/2504.17562)
• Keywords: language model, prompt
• Abstract: The ability to acquire latent semantics is one of the key properties that determines the performance of language models. One convenient approach to invoke this ability is to prepend metadata (e.g. URLs, domains, and styles) at the beginning of texts in the pre-training data, making it easier for the model to access latent semantics before observing the entire text. Previous studies have reported that this technique actually improves the performance of trained models in downstream tasks; however, this improvement has been observed only in specific downstream tasks, without consistent enhancement in average next-token prediction loss. To understand this phenomenon, we closely investigate how prepending metadata during pre-training affects model performance by examining its behavior using artificial data. Interestingly, we found that this approach produces both positive and negative effects on the downstream tasks. We demonstrate that the effectiveness of the approach depends on whether latent semantics can be inferred from the downstream task's prompt. Specifically, through investigations using data generated by probabilistic context-free grammars, we show that training with metadata helps improve model's performance when the given context is long enough to infer the latent semantics. In contrast, the technique negatively impacts performance when the context lacks the necessary information to make an accurate posterior inference.
• 摘要：获得潜在语义的能力是决定语言模型性能的关键属性之一。调用此功能的一种方便方法是在预训练数据中的文本开头预启动元数据（例如URL，域和样式），从而使模型更容易在观察整个文本之前访问潜在语义。先前的研究报告说，这项技术实际上改善了在下游任务中受过训练的模型的性能。但是，仅在特定的下游任务中观察到了这种改进，而没有一致的平均下一步预测损失。为了理解这种现象，我们仔细研究了预训练期间预先进行元数据如何通过使用人工数据检查其行为来影响模型性能。有趣的是，我们发现这种方法会对下游任务产生积极和负面影响。我们证明该方法的有效性取决于是否可以从下游任务的提示中推断出潜在语义。具体而言，通过使用概率无上下文语法生成的数据进行的调查，我们表明，当给定上下文足够长以推断潜在语义时，使用元数据训练有助于提高模型的性能。相反，当上下文缺乏必要的信息以进行准确的后推断时，该技术会对性能产生负面影响。

Title: DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training

• Authors: Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, Xiangang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17565
• Pdf URL: https://arxiv.org/pdf/2504.17565
• Copy Paste: [[2504.17565]] DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training(https://arxiv.org/abs/2504.17565)
• Keywords: language model, llm
• Abstract: Although large language models (LLMs) have recently achieved remarkable performance on various complex reasoning benchmarks, the academic community still lacks an in-depth understanding of base model training processes and data quality. To address this, we construct a large-scale, difficulty-graded reasoning dataset containing approximately 3.34 million unique queries of varying difficulty levels and about 40 million distilled responses generated by multiple models over several passes. Leveraging pass rate and Coefficient of Variation (CV), we precisely select the most valuable training data to enhance reasoning capability. Notably, we observe a training pattern shift, indicating that reasoning-focused training based on base models requires higher learning rates for effective training. Using this carefully selected data, we significantly improve the reasoning capabilities of the base model, achieving a pass rate of 79.2\% on the AIME2024 mathematical reasoning benchmark. This result surpasses most current distilled models and closely approaches state-of-the-art performance. We provide detailed descriptions of our data processing, difficulty assessment, and training methodology, and have publicly released all datasets and methods to promote rapid progress in open-source long-reasoning LLMs. The dataset is available at: this https URL
• 摘要：尽管大型语言模型（LLMS）最近在各种复杂的推理基准上取得了出色的表现，但学术界仍然缺乏对基本模型培训过程和数据质量的深入了解。为了解决这个问题，我们构建了一个大规模的，难以分级的推理数据集，其中包含大约334亿个不同难度水平的独特查询以及多个通过多个通过的多个模型产生的大约4000万个蒸馏响应。利用通过率和变异系数（CV），我们精确选择了最有价值的训练数据来增强推理能力。值得注意的是，我们观察到训练模式的转变，表明基于基本模型的以推理为中心的培训需要更高的学习率来进行有效的培训。使用这些精心选择的数据，我们显着提高了基本模型的推理能力，在AIME2024数学推理基准上实现了79.2 \％的通行率。该结果超过了大多数当前的蒸馏模型，并紧密探讨了最先进的性能。我们提供了有关数据处理，难度评估和培训方法的详细描述，并已公开发布了所有数据集和方法，以促进开源长期以来的LLMS快速进步。该数据集可用网址：此HTTPS URL

Title: Evaluating Grounded Reasoning by Code-Assisted Large Language Models for Mathematics

• Authors: Zena Al-Khalili, Nick Howell, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17665
• Pdf URL: https://arxiv.org/pdf/2504.17665
• Copy Paste: [[2504.17665]] Evaluating Grounded Reasoning by Code-Assisted Large Language Models for Mathematics(https://arxiv.org/abs/2504.17665)
• Keywords: language model, llm
• Abstract: Assisting LLMs with code generation improved their performance on mathematical reasoning tasks. However, the evaluation of code-assisted LLMs is generally restricted to execution correctness, lacking a rigorous evaluation of their generated programs. In this work, we bridge this gap by conducting an in-depth analysis of code-assisted LLMs' generated programs in response to math reasoning tasks. Our evaluation focuses on the extent to which LLMs ground their programs to math rules, and how that affects their end performance. For this purpose, we assess the generations of five different LLMs, on two different math datasets, both manually and automatically. Our results reveal that the distribution of grounding depends on LLMs' capabilities and the difficulty of math problems. Furthermore, mathematical grounding is more effective for closed-source models, while open-source models fail to employ math rules in their solutions correctly. On MATH500, the percentage of grounded programs decreased to half, while the ungrounded generations doubled in comparison to ASDiv grade-school problems. Our work highlights the need for in-depth evaluation beyond execution accuracy metrics, toward a better understanding of code-assisted LLMs' capabilities and limits in the math domain.
• 摘要：协助LLMS生成代码可以提高其在数学推理任务上的性能。但是，对代码辅助LLM的评估通常仅限于执行正确性，缺乏对其生成程序的严格评估。在这项工作中，我们通过对响应数学推理任务的代码辅助LLMS生成的程序进行深入分析来弥合这一差距。我们的评估重点是LLM将其计划纳入数学规则的程度，以及如何影响其最终绩效。为此，我们在两个不同的数学数据集上评估了五个不同的LLM的一代，无论是手动和自动的。我们的结果表明，接地的分布取决于LLMS的功能和数学问题的难度。此外，数学基础对于封闭源模型更有效，而开源模型无法正确地在其解决方案中使用数学规则。在Math500上，基础计划的百分比减少到一半，而与Asdiv级别的问题相比，未接地的几代人翻了一番。我们的工作强调了超出执行精度指标的深入评估的需求，以更好地理解代码辅助的LLMS在数学领域的功能和限制。

Title: Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction

• Authors: Yuanchang Ye, Weiyan Wen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.17671
• Pdf URL: https://arxiv.org/pdf/2504.17671
• Copy Paste: [[2504.17671]] Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction(https://arxiv.org/abs/2504.17671)
• Keywords: language model, hallucination
• Abstract: This study addresses the critical challenge of hallucination mitigation in Large Vision-Language Models (LVLMs) for Visual Question Answering (VQA) tasks through a Split Conformal Prediction (SCP) framework. While LVLMs excel in multi-modal reasoning, their outputs often exhibit hallucinated content with high confidence, posing risks in safety-critical applications. We propose a model-agnostic uncertainty quantification method that integrates dynamic threshold calibration and cross-modal consistency verification. By partitioning data into calibration and test sets, the framework computes nonconformity scores to construct prediction sets with statistical guarantees under user-defined risk levels ($\alpha$). Key innovations include: (1) rigorous control of \textbf{marginal coverage} to ensure empirical error rates remain strictly below $\alpha$; (2) dynamic adjustment of prediction set sizes inversely with $\alpha$, filtering low-confidence outputs; (3) elimination of prior distribution assumptions and retraining requirements. Evaluations on benchmarks (ScienceQA, MMMU) with eight LVLMs demonstrate that SCP enforces theoretical guarantees across all $\alpha$ values. The framework achieves stable performance across varying calibration-to-test split ratios, underscoring its robustness for real-world deployment in healthcare, autonomous systems, and other safety-sensitive domains. This work bridges the gap between theoretical reliability and practical applicability in multi-modal AI systems, offering a scalable solution for hallucination detection and uncertainty-aware decision-making.
• 摘要：这项研究解决了通过分裂的保形预测（SCP）框架来解决大型视觉语言模型（LVLM）在大型视觉语言模型（LVLM）中的关键挑战。尽管LVLM在多模式推理方面表现出色，但它们的输出经常表现出幻觉的内容，并具有很高的信心，对安全至关重要的应用产生了风险。我们提出了一种模型不确定性定量方法，该方法整合了动态阈值校准和跨模式一致性验证。通过将数据划分为校准和测试集，该框架计算不合格得分以构建在用户定义的风险水平（$ \ alpha $）下具有统计保证的预测集。关键创新包括：（1）严格控制\ textbf {边际覆盖范围}，以确保严格的经验错误率严格低于$ \ alpha $； （2）与$ \ alpha $相反的预测设置大小的动态调整，从而过滤低信心输出； （3）消除先前的分配假设和再培训要求。对基准（ScienceQA，MMMU）进行八个LVLM的评估表明，SCP在所有$ \ alpha $ valuep中都可以执行理论保证。该框架在不同的校准与测试拆分比率方面取得了稳定的性能，强调了其在医疗保健，自主系统和其他安全敏感域中现实部署的稳健性。这项工作弥合了多模式AI系统中理论可靠性与实际适用性之间的差距，为幻觉检测和不确定性觉醒的决策提供了可扩展的解决方案。

Title: Energy Considerations of Large Language Model Inference and Efficiency Optimizations

• Authors: Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, Emma Strubell
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.17674
• Pdf URL: https://arxiv.org/pdf/2504.17674
• Copy Paste: [[2504.17674]] Energy Considerations of Large Language Model Inference and Efficiency Optimizations(https://arxiv.org/abs/2504.17674)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) scale in size and adoption, their computational and environmental costs continue to rise. Prior benchmarking efforts have primarily focused on latency reduction in idealized settings, often overlooking the diverse real-world inference workloads that shape energy use. In this work, we systematically analyze the energy implications of common inference efficiency optimizations across diverse Natural Language Processing (NLP) and generative Artificial Intelligence (AI) workloads, including conversational AI and code generation. We introduce a modeling approach that approximates real-world LLM workflows through a binning strategy for input-output token distributions and batch size variations. Our empirical analysis spans software frameworks, decoding strategies, GPU architectures, online and offline serving settings, and model parallelism configurations. We show that the effectiveness of inference optimizations is highly sensitive to workload geometry, software stack, and hardware accelerators, demonstrating that naive energy estimates based on FLOPs or theoretical GPU utilization significantly underestimate real-world energy consumption. Our findings reveal that the proper application of relevant inference efficiency optimizations can reduce total energy use by up to 73% from unoptimized baselines. These insights provide a foundation for sustainable LLM deployment and inform energy-efficient design strategies for future AI infrastructure.
• 摘要：随着大型语言模型（LLMS）的规模和采用规模，其计算和环境成本继续上升。先前的基准测试工作主要集中在理想化的设置的延迟降低上，通常忽略了塑造能源使用的多样化的现实推理工作负载。在这项工作中，我们系统地分析了各种自然语言处理（NLP）和生成人工智能（AI）工作负载（包括对话式AI和代码生成）的共同推理效率优化的能量影响。我们介绍了一种建模方法，该方法通过用于输入输出令牌分布和批量尺寸变化的包装策略来近似现实世界LLM工作流程。我们的经验分析涵盖了软件框架，解码策略，GPU架构，在线和离线服务设置以及模型并行性配置。我们表明，推理优化的有效性对工作负载几何形状，软件堆栈和硬件加速器非常敏感，这表明基于拖船或理论GPU利用率的天真能量估计显着低估了现实世界中的能源消耗。我们的发现表明，相关推理效率优化的适当应用可以将总能源利用减少高达73％，而不受欢迎的基层的总能源使用。这些见解为可持续的LLM部署提供了基础，并为未来的AI基础设施提供了节能设计策略。

Title: Ensemble Bayesian Inference: Leveraging Small Language Models to Achieve LLM-level Accuracy in Profile Matching Tasks

• Authors: Haru-Tada Sato, Fuka Matsuzaki, Jun-ichiro Takahashi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17685
• Pdf URL: https://arxiv.org/pdf/2504.17685
• Copy Paste: [[2504.17685]] Ensemble Bayesian Inference: Leveraging Small Language Models to Achieve LLM-level Accuracy in Profile Matching Tasks(https://arxiv.org/abs/2504.17685)
• Keywords: language model, llm
• Abstract: This study explores the potential of small language model(SLM) ensembles to achieve accuracy comparable to proprietary large language models (LLMs). We propose Ensemble Bayesian Inference (EBI), a novel approach that applies Bayesian estimation to combine judgments from multiple SLMs, allowing them to exceed the performance limitations of individual models. Our experiments on diverse tasks(aptitude assessments and consumer profile analysis in both Japanese and English) demonstrate EBI's effectiveness. Notably, we analyze cases where incorporating models with negative Lift values into ensembles improves overall performance, and we examine the method's efficacy across different languages. These findings suggest new possibilities for constructing high-performance AI systems with limited computational resources and for effectively utilizing models with individually lower performance. Building on existing research on LLM performance evaluation, ensemble methods, and open-source LLM utilization, we discuss the novelty and significance of our approach.
• 摘要：这项研究探讨了小语言模型（SLM）合奏的潜力，以实现与专有大语模型（LLMS）相当的准确性。我们提出了合奏贝叶斯推断（EBI），这是一种应用贝叶斯估算的新方法，以结合多个SLM的判断，从而超过单个模型的性能限制。我们对各种任务（日语和英语中的能力评估和消费者概况分析）的实验证明了EBI的有效性。值得注意的是，我们分析了将带负升力值的模型纳入集成的案例，可以提高整体性能，并研究该方法在不同语言上的功效。这些发现提出了用于构建具有有限计算资源的高性能AI系统的新可能性，并有效地利用具有较低性能的模型。在现有关于LLM绩效评估，集合方法和开源LLM利用率的研究的基础上，我们讨论了方法的新颖性和意义。

Title: Multilingual Performance Biases of Large Language Models in Education

• Authors: Vansh Gupta, Sankalan Pal Chowdhury, Vilém Zouhar, Donya Rooein, Mrinmaya Sachan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.17720
• Pdf URL: https://arxiv.org/pdf/2504.17720
• Copy Paste: [[2504.17720]] Multilingual Performance Biases of Large Language Models in Education(https://arxiv.org/abs/2504.17720)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly being adopted in educational settings. These applications expand beyond English, though current LLMs remain primarily English-centric. In this work, we ascertain if their use in education settings in non-English languages is warranted. We evaluated the performance of popular LLMs on four educational tasks: identifying student misconceptions, providing targeted feedback, interactive tutoring, and grading translations in six languages (Hindi, Arabic, Farsi, Telugu, Ukrainian, Czech) in addition to English. We find that the performance on these tasks somewhat corresponds to the amount of language represented in training data, with lower-resource languages having poorer task performance. Although the models perform reasonably well in most languages, the frequent performance drop from English is significant. Thus, we recommend that practitioners first verify that the LLM works well in the target language for their educational task before deployment.
• 摘要：在教育环境中，大型语言模型（LLM）越来越多地被采用。尽管目前的LLM仍主要以英语为中心，但这些应用程序扩展了英语。在这项工作中，我们确定它们是否有必要在非英语语言中使用。我们评估了流行的LLM在四项教育任务上的表现：确定学生的误解，以六种语言（印地语，阿拉伯语，法尔西，泰卢固语，乌克兰，苏格兰，捷克语）提供针对性的反馈，交互式辅导和分级翻译。我们发现，这些任务的性能在某种程度上与培训数据中表示的语言数量相对应，而较低的资源语言的任务性能较差。尽管这些模型在大多数语言中都表现出色，但英语频繁的性能下降是重要的。因此，我们建议从业者首先验证LLM在部署前的目标语言中效果很好。

Title: Conversational Assistants to support Heart Failure Patients: comparing a Neurosymbolic Architecture with ChatGPT

• Authors: Anuja Tayal, Devika Salunke, Barbara Di Eugenio, Paula Allen-Meares, Eulalia Puig Abril, Olga Garcia, Carolyn Dickens, Andrew Boyd
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.17753
• Pdf URL: https://arxiv.org/pdf/2504.17753
• Copy Paste: [[2504.17753]] Conversational Assistants to support Heart Failure Patients: comparing a Neurosymbolic Architecture with ChatGPT(https://arxiv.org/abs/2504.17753)
• Keywords: language model, gpt, chat
• Abstract: Conversational assistants are becoming more and more popular, including in healthcare, partly because of the availability and capabilities of Large Language Models. There is a need for controlled, probing evaluations with real stakeholders which can highlight advantages and disadvantages of more traditional architectures and those based on generative AI. We present a within-group user study to compare two versions of a conversational assistant that allows heart failure patients to ask about salt content in food. One version of the system was developed in-house with a neurosymbolic architecture, and one is based on ChatGPT. The evaluation shows that the in-house system is more accurate, completes more tasks and is less verbose than the one based on ChatGPT; on the other hand, the one based on ChatGPT makes fewer speech errors and requires fewer clarifications to complete the task. Patients show no preference for one over the other.
• 摘要：会话助手变得越来越受欢迎，包括医疗保健，部分原因是大语言模型的可用性和能力。需要与真正的利益相关者进行控制的，探测评估，这些评估可以强调更传统的体系结构的优势和缺点，以及基于生成的AI的优势和缺点。我们提出了一项组内用户研究，以比较两个版本的会话助手，该版本允许心力衰竭患者询问食物中的盐含量。该系统的一个版本是在内部开发的，它具有神经构型架构，并且基于Chatgpt。评估表明，内部系统更准确，完成了更多的任务，并且比基于Chatgpt的任务更少。另一方面，基于chatgpt的一个造成的语音错误较少，并且需要更少的澄清来完成任务。患者不偏爱一个。

Title: The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

• Authors: Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.17768
• Pdf URL: https://arxiv.org/pdf/2504.17768
• Copy Paste: [[2504.17768]] The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs(https://arxiv.org/abs/2504.17768)
• Keywords: llm
• Abstract: Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its viability, its efficiency-accuracy trade-offs, and systematic scaling studies remain unexplored. To address this gap, we perform a careful comparison of training-free sparse attention methods at varying model scales, sequence lengths, and sparsity levels on a diverse collection of long-sequence tasks-including novel ones that rely on natural language while remaining controllable and easy to evaluate. Based on our experiments, we report a series of key findings: 1) an isoFLOPS analysis reveals that for very long sequences, larger and highly sparse models are preferable to smaller and dense ones. 2) The level of sparsity attainable while statistically guaranteeing accuracy preservation is higher during decoding than prefilling, and correlates with model size in the former. 3) There is no clear strategy that performs best across tasks and phases, with different units of sparsification or budget adaptivity needed for different scenarios. Even moderate sparsity levels often result in significant performance degradation on at least one task, highlighting that sparse attention is not a universal solution. 4) We introduce and validate novel scaling laws specifically tailored for sparse attention, providing evidence that our findings are likely to hold true beyond our range of experiments. Through these insights, we demonstrate that sparse attention is a key tool to enhance the capabilities of Transformer LLMs for processing longer sequences, but requires careful evaluation of trade-offs for performance-sensitive applications.
• 摘要：稀疏的注意力提供了一种有希望的策略，可以扩展变压器LLM中的长期文化功能，但其可行性，其效率 - 准确性权衡以及系统的扩展研究仍未得到探索。为了解决这一差距，我们在不同的模型量表，序列长度和稀疏水平上进行了仔细比较，在多种长期任务的集合中，包括依靠自然语言，同时保持可控且易于评估。根据我们的实验，我们报告了一系列关键发现：1）Isoflops分析表明，对于很长的序列，较大且高度稀疏的模型比较小且密集的模型更可取。 2）可达到的稀疏度可达到的水平，同时统计保证在解码过程中保留准确性的水平高于预填充，并且与前者的模型大小相关。 3）没有明确的策略可以在各个任务和阶段进行最佳的效果，而不同场景所需的稀疏或预算适应性不同。即使是中等的稀疏度，也通常会导致至少一项任务的绩效降解，从而强调了稀疏的注意力不是通用的解决方案。 4）我们介绍并验证专门针对稀疏注意的新颖缩放定律，提供了证据，表明我们的发现可能超出了我们的实验范围。通过这些见解，我们证明了稀疏的注意力是增强变压器LLM来处理更长序列功能的关键工具，但需要仔细评估对性能敏感应用程序的权衡。