2025-03-06

Title: ExpertGenQA: Open-ended QA generation in Specialized Domains

• Authors: Haz Sameen Shahgir, Chansong Lim, Jia Chen, Evangelos E. Papalexakis, Yue Dong
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.02948
• Pdf URL: https://arxiv.org/pdf/2503.02948
• Copy Paste: [[2503.02948]] ExpertGenQA: Open-ended QA generation in Specialized Domains(https://arxiv.org/abs/2503.02948)
• Keywords: llm
• Abstract: Generating high-quality question-answer pairs for specialized technical domains remains challenging, with existing approaches facing a tradeoff between leveraging expert examples and achieving topical diversity. We present ExpertGenQA, a protocol that combines few-shot learning with structured topic and style categorization to generate comprehensive domain-specific QA pairs. Using U.S. Federal Railroad Administration documents as a test bed, we demonstrate that ExpertGenQA achieves twice the efficiency of baseline few-shot approaches while maintaining $94.4\%$ topic coverage. Through systematic evaluation, we show that current LLM-based judges and reward models exhibit strong bias toward superficial writing styles rather than content quality. Our analysis using Bloom's Taxonomy reveals that ExpertGenQA better preserves the cognitive complexity distribution of expert-written questions compared to template-based approaches. When used to train retrieval models, our generated queries improve top-1 accuracy by $13.02\%$ over baseline performance, demonstrating their effectiveness for downstream applications in technical domains.
• 摘要：为专门的技术领域生成高质量的提问对，仍然具有挑战性，现有的方法在利用专家示例和实现主题多样性之间的权衡面临。我们提出了ExpertGenQA，该协议将几乎没有的学习与结构化主题和样式分类结合在一起，以生成全面的域特异性质量质量质量质量对。使用美国联邦铁路管理文件作为测试床，我们证明了ExpertgenQA在维持$ 94.4 \％$主题覆盖范围的同时，实现了两倍的基线方法。通过系统的评估，我们表明当前基于LLM的法官和奖励模型对肤浅的写作风格而不是内容质量表现出很大的偏见。我们使用Bloom的分类法分析表明，与基于模板的方法相比，专家Genqa更好地保留了专家编写问题的认知复杂性分布。当用于训练检索模型时，我们生成的查询将TOP-1的准确性提高了$ 13.02 \％$，而不是基线性能，证明了它们在技术域中下游应用的有效性。

Title: InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model

• Authors: Siqi Ouyang, Xi Xu, Lei Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.02969
• Pdf URL: https://arxiv.org/pdf/2503.02969
• Copy Paste: [[2503.02969]] InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model(https://arxiv.org/abs/2503.02969)
• Keywords: language model
• Abstract: Simultaneous translation of unbounded streaming speech remains a challenging problem due to the need for effectively processing the history speech context and past translations so that quality and latency, including computation overhead, can be balanced. Most prior works assume pre-segmented speech, limiting their real-world applicability. In this paper, we propose InfiniSST, a novel approach that formulates SST as a multi-turn dialogue task, enabling seamless translation of unbounded speech. We construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a key-value (KV) cache management strategy to facilitate efficient inference. Experiments on MuST-C En-Es, En-De, and En-Zh demonstrate that InfiniSST reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. Ablation studies further validate the contributions of our data construction and cache management strategy. We release the code at this https URL
• 摘要：由于需要有效地处理历史语音上下文和过去的翻译，因此无限流语言语音的同时翻译仍然是一个具有挑战性的问题，以便可以平衡质量和延迟（包括计算开销）。大多数先前的作品都采用预分段的语音，从而限制了其现实世界的适用性。在本文中，我们提出了一种新颖的方法，该方法将SST提出为多转化的对话任务，从而实现了无限语音的无缝翻译。我们在训练过程中构建了从必须使用的必需轨迹的翻译轨迹和稳健的细分，并制定了键值（KV）缓存管理策略，以促进有效的推理。对必需品，en-de和en-ZH的实验表明，与基准相比，Infinisst在保持相同的翻译质量的同时，将计算意识的潜伏期降低了0.5至1秒。消融研究进一步验证了我们的数据构建和缓存管理策略的贡献。我们在此HTTPS URL上发布代码

Title: Multilingual Relative Clause Attachment Ambiguity Resolution in Large Language Models

• Authors: So Young Lee, Russell Scheinberg, Amber Shore, Ameeta Agrawal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.02971
• Pdf URL: https://arxiv.org/pdf/2503.02971
• Copy Paste: [[2503.02971]] Multilingual Relative Clause Attachment Ambiguity Resolution in Large Language Models(https://arxiv.org/abs/2503.02971)
• Keywords: language model, llm
• Abstract: This study examines how large language models (LLMs) resolve relative clause (RC) attachment ambiguities and compares their performance to human sentence processing. Focusing on two linguistic factors, namely the length of RCs and the syntactic position of complex determiner phrases (DPs), we assess whether LLMs can achieve human-like interpretations amid the complexities of language. In this study, we evaluated several LLMs, including Claude, Gemini and Llama, in multiple languages: English, Spanish, French, German, Japanese, and Korean. While these models performed well in Indo-European languages (English, Spanish, French, and German), they encountered difficulties in Asian languages (Japanese and Korean), often defaulting to incorrect English translations. The findings underscore the variability in LLMs' handling of linguistic ambiguities and highlight the need for model improvements, particularly for non-European languages. This research informs future enhancements in LLM design to improve accuracy and human-like processing in diverse linguistic environments.
• 摘要：这项研究研究了大型语言模型（LLM）如何解决相对从句（RC）的依恋歧义，并将其绩效与人类句子处理进行比较。着眼于两个语言因素，即RC的长度和复杂的确定词短语（DPS）的句法位置，我们评估LLM在语言的复杂性中是否可以实现类似人类的解释。在这项研究中，我们用多种语言评估了几种LLM，包括克劳德，双子座和骆驼：英语，西班牙语，法语，德语，日语和韩语。尽管这些模型在印欧语（英语，西班牙语，法语和德语）中表现良好，但它们遇到了亚洲语言（日语和韩语）的困难，通常违约将英语翻译不正确。这些发现强调了LLMS对语言歧义的处理的可变性，并强调了对模型改进的需求，尤其是对于非欧洲语言。这项研究为LLM设计的未来增强提供了信息，以提高各种语言环境中的准确性和类似人类的处理。

Title: LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic Templatisation and Orthographic Obfuscation

• Authors: Jude Khouja, Karolina Korgul, Simi Hellsten, Lingyi Yang, Vlad Neacs, Harry Mayne, Ryan Kearns, Andrew Bean, Adam Mahdi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.02972
• Pdf URL: https://arxiv.org/pdf/2503.02972
• Copy Paste: [[2503.02972]] LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic Templatisation and Orthographic Obfuscation(https://arxiv.org/abs/2503.02972)
• Keywords: language model, llm
• Abstract: Effective evaluation of the reasoning capabilities of large language models (LLMs) are susceptible to overestimation due to data exposure of evaluation benchmarks. We introduce a framework for producing linguistic reasoning problems that reduces the effect of memorisation in model performance estimates and apply this framework to develop LINGOLY-TOO, a challenging evaluation benchmark for linguistic reasoning. By developing orthographic templates, we dynamically obfuscate the writing systems of real languages to generate numerous question variations. These variations preserve the reasoning steps required for each solution while reducing the likelihood of specific problem instances appearing in model training data. Our experiments demonstrate that frontier models, including OpenAI o1-preview and DeepSeem R1, struggle with advanced reasoning. Our analysis also shows that LLMs exhibit noticeable variance in accuracy across permutations of the same problem, and on average perform better on questions appearing in their original orthography. Our findings highlight the opaque nature of response generation in LLMs and provide evidence that prior data exposure contributes to overestimating the reasoning capabilities of frontier models.
• 摘要：有效评估大语言模型（LLM）的推理能力（LLMS）因评估基准的数据暴露而容易高估。我们介绍了一个框架，用于产生语言推理问题，从而减少了模型绩效估算中记忆的影响，并将此框架应用于开发Lingoly-Too，这是语言推理的挑战性评估基准。通过开发拼字模板，我们动态混淆了真实语言的写作系统，以产生许多问题变化。这些变化保留了每个解决方案所需的推理步骤，同时降低了模型培训数据中出现的特定问题实例的可能性。我们的实验表明，包括OpenAI O1-preview和DeepSeem R1在内的边境模型与先进的推理作斗争。我们的分析还表明，LLM在同一问题的排列中表现出明显的准确性差异，并且平均而言，在其原始拼字法中出现的问题上表现更好。我们的发现突出了LLM中响应产生的不透明性质，并提供了证据表明先前的数据暴露有助于高估边境模型的推理能力。

Title: Effectively Steer LLM To Follow Preference via Building Confident Directions

• Authors: Bingqing Song, Boran Han, Shuai Zhang, Hao Wang, Haoyang Fang, Bonan Min, Yuyang Wang, Mingyi Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.02989
• Pdf URL: https://arxiv.org/pdf/2503.02989
• Copy Paste: [[2503.02989]] Effectively Steer LLM To Follow Preference via Building Confident Directions(https://arxiv.org/abs/2503.02989)
• Keywords: gpt, llm, prompt
• Abstract: Having an LLM that aligns with human preferences is essential for accommodating individual needs, such as maintaining writing style or generating specific topics of interest. The majority of current alignment methods rely on fine-tuning or prompting, which can be either costly or difficult to control. Model steering algorithms, which modify the model output by constructing specific steering directions, are typically easy to implement and optimization-free. However, their capabilities are typically limited to steering the model into one of the two directions (i.e., bidirectional steering), and there has been no theoretical understanding to guarantee their performance. In this work, we propose a theoretical framework to understand and quantify the model steering methods. Inspired by the framework, we propose a confident direction steering method (CONFST) that steers LLMs via modifying their activations at inference time. More specifically, CONFST builds a confident direction that is closely aligned with users' preferences, and this direction is then added to the activations of the LLMs to effectively steer the model output. Our approach offers three key advantages over popular bidirectional model steering methods: 1) It is more powerful, since multiple (i.e. more than two) users' preferences can be aligned simultaneously; 2) It is simple to implement, since there is no need to determine which layer to add the steering vector to; 3) No explicit user instruction is required. We validate our method on GPT-2 XL (1.5B), Mistral (7B) and Gemma-it (9B) models for tasks that require shifting the output of LLMs across various topics and styles, achieving superior performance over competing methods.
• 摘要：拥有与人类偏好保持一致的LLM对于满足个人需求，例如保持写作风格或产生特定的兴趣主题。当前的大多数对齐方法都依赖于微调或提示，这可能是昂贵或难以控制的。通过构建特定的转向说明来修改模型输出的模型转向算法通常易于实现和优化。但是，它们的功能通常仅限于将模型转向两个方向之一（即双向转向），并且没有理论上的理解来保证其性能。在这项工作中，我们提出了一个理论框架，以了解和量化模型转向方法。受框架的启发，我们提出了一种自信的方向转向方法（confst），该方法通过在推理时修改其激活来引导LLMS。更具体地说，Conflst构建了一个自信方向，该方向与用户的喜好紧密相符，然后将此方向添加到LLMS的激活中，以有效地引导模型输出。我们的方法比流行的双向模型转向方法提供了三个关键优势：1）功能更强大，因为可以同时将多个用户的偏好（即两个以上）用户的喜好对准； 2）实现很容易，因为无需确定将转向向量添加到哪个层； 3）不需要明确的用户指令。我们在GPT-2 XL（1.5B），Mistral（7b）和Gemma-It（9B）模型上验证了我们的方法，以需要将LLM在各种主题和样式上转移的输出，从而实现优于竞争方法的卓越性能。

Title: Zero-Shot Multi-Label Classification of Bangla Documents: Large Decoders Vs. Classic Encoders

• Authors: Souvika Sarkar, Md. Najib Hasan, Santu Karmaker
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.02993
• Pdf URL: https://arxiv.org/pdf/2503.02993
• Copy Paste: [[2503.02993]] Zero-Shot Multi-Label Classification of Bangla Documents: Large Decoders Vs. Classic Encoders(https://arxiv.org/abs/2503.02993)
• Keywords: gpt, llm
• Abstract: Bangla, a language spoken by over 300 million native speakers and ranked as the sixth most spoken language worldwide, presents unique challenges in natural language processing (NLP) due to its complex morphological characteristics and limited resources. While recent Large Decoder Based models (LLMs), such as GPT, LLaMA, and DeepSeek, have demonstrated excellent performance across many NLP tasks, their effectiveness in Bangla remains largely unexplored. In this paper, we establish the first benchmark comparing decoder-based LLMs with classic encoder-based models for Zero-Shot Multi-Label Classification (Zero-Shot-MLC) task in Bangla. Our evaluation of 32 state-of-the-art models reveals that, existing so-called powerful encoders and decoders still struggle to achieve high accuracy on the Bangla Zero-Shot-MLC task, suggesting a need for more research and resources for Bangla NLP.
• 摘要：孟加拉语是以上3亿本人说的语言，是全球第六位口语最多的语言，由于其复杂的形态学特征和有限的资源，在自然语言处理（NLP）中提出了独特的挑战（NLP）。尽管最近基于大型解码器的模型（LLM），例如GPT，Llama和DeepSeek，在许多NLP任务中都表现出了出色的性能，但它们在孟加拉的有效性仍未得到探索。在本文中，我们建立了第一个基准测试，将基于解码器的LLM与基于经典编码器的模型进行了零击的多标签分类（零摄像-MLC）任务。我们对32种最先进模型的评估表明，现有的所谓强大编码器和解码器仍在努力实现Bangla Zero-Sho-Shot-MLC任务的高精度，这表明需要为Bangla NLP进行更多的研究和资源。

Title: One Model to Train them All: Hierarchical Self-Distillation for Enhanced Early Layer Embeddings

• Authors: Andrea Gurioli, Federico Pennino, João Monteiro, Maurizio Gabbrielli
• Subjects: cs.CL, cs.AI, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2503.03008
• Pdf URL: https://arxiv.org/pdf/2503.03008
• Copy Paste: [[2503.03008]] One Model to Train them All: Hierarchical Self-Distillation for Enhanced Early Layer Embeddings(https://arxiv.org/abs/2503.03008)
• Keywords: language model
• Abstract: Deploying language models often requires handling model size vs. performance trade-offs to satisfy downstream latency constraints while preserving the model's usefulness. Model distillation is commonly employed to reduce model size while maintaining acceptable performance. However, distillation can be inefficient since it involves multiple training steps. In this work, we introduce MODULARSTARENCODER, a modular multi-exit encoder with 1B parameters, useful for multiple tasks within the scope of code retrieval. MODULARSTARENCODER is trained with a novel self-distillation mechanism that significantly improves lower-layer representations-allowing different portions of the model to be used while still maintaining a good trade-off in terms of performance. Our architecture focuses on enhancing text-to-code and code-to-code search by systematically capturing syntactic and semantic structures across multiple levels of representation. Specific encoder layers are targeted as exit heads, allowing higher layers to guide earlier layers during training. This self-distillation effect improves intermediate representations, increasing retrieval recall at no extra training cost. In addition to the multi-exit scheme, our approach integrates a repository-level contextual loss that maximally utilizes the training context window, further enhancing the learned representations. We also release a new dataset constructed via code translation, seamlessly expanding traditional text-to-code benchmarks with code-to-code pairs across diverse programming languages. Experimental results highlight the benefits of self-distillation through multi-exit supervision.
• 摘要：部署语言模型通常需要处理模型大小与性能折衷，以满足下游延迟限制，同时保留模型的实用性。通常使用模型蒸馏来减少模型大小，同时保持可接受的性能。但是，由于蒸馏涉及多个训练步骤，因此蒸馏效率可能不佳。在这项工作中，我们介绍了ModularStarenCoder，这是一种带有1B参数的模块化多EXIT编码器，可用于代码检索范围内的多个任务。 ModullStarenCoder经过一种新型的自我验证机制的培训，该机制可显着改善较低的表示形式，从而允许使用模型的不同部分，同时仍然在绩效方面保持良好的权衡。我们的体系结构着重于通过系统地捕获多个表示级别的句法和语义结构来增强文本对编码和代码对代码搜索。特定的编码层作为出口头的靶向，使较高的层可以在训练过程中引导较早的层。这种自我验证效应改善了中间表示，以无额外的培训成本增加了检索召回。除了多次EXIT方案外，我们的方法还集成了存储库级的上下文损失，该损失最大程度地利用了训练上下文窗口，从而进一步增强了学习的表示形式。我们还发布了一个通过代码翻译构建的新数据集，无缝扩展传统的文本对代码基准，并在不同的编程语言上使用代码对代码对。实验结果突出了通过多exit监督自我验证的好处。

Title: SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMs

• Authors: Samir Abdaljalil, Filippo Pallucchini, Andrea Seveso, Hasan Kurban, Fabio Mercorio, Erchin Serpedin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03032
• Pdf URL: https://arxiv.org/pdf/2503.03032
• Copy Paste: [[2503.03032]] SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMs(https://arxiv.org/abs/2503.03032)
• Keywords: language model, llm, hallucination
• Abstract: Despite the state-of-the-art performance of Large Language Models (LLMs), these models often suffer from hallucinations, which can undermine their performance in critical applications. In this work, we propose SAFE, a novel method for detecting and mitigating hallucinations by leveraging Sparse Autoencoders (SAEs). While hallucination detection techniques and SAEs have been explored independently, their synergistic application in a comprehensive system, particularly for hallucination-aware query enrichment, has not been fully investigated. To validate the effectiveness of SAFE, we evaluate it on two models with available SAEs across three diverse cross-domain datasets designed to assess hallucination problems. Empirical results demonstrate that SAFE consistently improves query generation accuracy and mitigates hallucinations across all datasets, achieving accuracy improvements of up to 29.45%.
• 摘要：尽管大语言模型（LLM）的最新表现，但这些模型经常遭受幻觉的影响，这可能会破坏其在关键应用中的性能。在这项工作中，我们提出了一种安全的方法，一种新的方法，用于通过利用稀疏的自动编码器（SAE）来检测和减轻幻觉。尽管已经独立探索了幻觉检测技术和SAE，但它们在综合系统中的协同应用，尤其是对于幻觉感知的查询富集，尚未得到充分研究。为了验证安全的有效性，我们将其评估在两个模型上，该模型在三个不同的跨域数据集中有可用的SAE，旨在评估幻觉问题。经验结果表明，安全可以始终提高查询产生的准确性，并减轻所有数据集的幻觉，从而实现高达29.45％的准确性提高。

Title: SAGE: Steering and Refining Dialog Generation with State-Action Augmentation

• Authors: Yizhe Zhang, Navdeep Jaitly
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03040
• Pdf URL: https://arxiv.org/pdf/2503.03040
• Copy Paste: [[2503.03040]] SAGE: Steering and Refining Dialog Generation with State-Action Augmentation(https://arxiv.org/abs/2503.03040)
• Keywords: language model, llm, chat
• Abstract: Recent advances in large language models have demonstrated impressive capabilities in task-oriented applications, yet building emotionally intelligent chatbots that can engage in natural, strategic conversations remains a challenge. We present a novel approach called SAGE that uses latent variables to control long-horizon behavior in dialogue generation. At the core of our method is the State-Action Chain (SAC), which augments standard language model fine-tuning by introducing latent variables that encapsulate emotional states and conversational strategies between dialogue turns. During inference, these variables are generated before each response, enabling coarse-grained control over dialogue progression while maintaining natural interaction patterns. We also introduce a self-improvement pipeline that leverages dialogue tree search, LLM-based reward modeling, and targeted fine-tuning to optimize conversational trajectories. Our experimental results show that models trained with this approach demonstrate improved performance in emotional intelligence metrics while maintaining strong capabilities on LLM benchmarks. The discrete nature of our latent variables facilitates search-based strategies and provides a foundation for future applications of reinforcement learning to dialogue systems, where learning can occur at the state level rather than the token level.
• 摘要：大型语言模型的最新进展已在以任务为导向的应用程序中表现出令人印象深刻的能力，但是建立了可以进行自然战略对话的情感智能聊天机器人仍然是一个挑战。我们提出了一种名为Sage的新方法，该方法使用潜在变量来控制对话生成中的长途行为。我们方法的核心是国家行动链（SAC），它通过引入封装情绪状态和对话转弯之间的情绪状态和对话策略的潜在变量来增强标准语言模型。在推断期间，这些变量是在每个响应之前生成的，从而在维持自然相互作用模式的同时，可以对对话进展进行粗粒的控制。我们还介绍了一个自我完善的管道，该管道利用对话树搜索，基于LLM的奖励建模以及有针对性的微调来优化对话轨迹。我们的实验结果表明，使用这种方法训练的模型表明，情绪智力指标的性能提高，同时保持LLM基准测试的强大功能。我们潜在变量的离散性质促进了基于搜索的策略，并为对话系统的增强学习的未来应用奠定了基础，在该系统中可以在州一级进行学习，而不是代币层面。

Title: Semi-Supervised In-Context Learning: A Baseline Study

• Authors: Zhengyao Gu, Henry Peng Zou, Yankai Chen, Aiwei Liu, Weizhi Zhang, Philip S. Yu
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03062
• Pdf URL: https://arxiv.org/pdf/2503.03062
• Copy Paste: [[2503.03062]] Semi-Supervised In-Context Learning: A Baseline Study(https://arxiv.org/abs/2503.03062)
• Keywords: prompt
• Abstract: Most existing work in data selection for In-Context Learning (ICL) has focused on constructing demonstrations from ground truth annotations, with limited attention given to selecting reliable self-generated annotations. In this work, we propose a three-step semi-supervised ICL framework: annotation generation, demonstration selection, and semi-supervised inference. Our baseline, Naive-SemiICL, which prompts select high-confidence self-generated demonstrations for ICL prompting, outperforms a 16-shot baseline by an average of 9.94% across 16 datasets. We further introduce IterPSD, an annotation approach that refines pseudo-demonstrations iteratively, achieving up to 6.8% additional gains in classification tasks. Lastly, we reveal a scaling law for semi-supervised ICL, where models achieve optimal performance with over 1,000 demonstrations.
• 摘要：大多数现有的用于秘密学习数据选择（ICL）的工作都集中在从地面真理注释中构建演示，而对选择可靠的自我生成的注释的关注有限。在这项工作中，我们提出了一个三步半监督的ICL框架：注释生成，演示选择和半监督推理。我们的基线Naive-semiicl提示选择ICL提示的高信任自我生成的演示，在16个数据集中，平均比16次射击基线的表现平均高9.94％。我们进一步介绍了ITERPSD，这是一种注释方法，可以迭代地完善伪示例，在分类任务中获得多达6.8％的额外收益。最后，我们揭示了半监督ICL的缩放定律，其中模型以1,000多次示威来实现最佳性能。

Title: Improving LLM-as-a-Judge Inference with the Judgment Distribution

• Authors: Victor Wang, Michael J.Q. Zhang, Eunsol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03064
• Pdf URL: https://arxiv.org/pdf/2503.03064
• Copy Paste: [[2503.03064]] Improving LLM-as-a-Judge Inference with the Judgment Distribution(https://arxiv.org/abs/2503.03064)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Using language models to scalably approximate human preferences on text quality (LLM-as-a-judge) has become a standard practice applicable to many tasks. A judgment is often extracted from the judge's textual output alone, typically with greedy decoding. However, LLM judges naturally provide distributions over judgment tokens, inviting a breadth of inference methods for extracting fine-grained preferences. We find that taking the mean of the judgment distribution consistently outperforms taking the mode (i.e. greedy decoding) in all evaluation settings (i.e. pointwise, pairwise, and listwise). We further explore novel methods of deriving preferences from judgment distributions, and find that methods incorporating risk aversion often improve performance. Lastly, we analyze LLM-as-a-judge paired with chain-of-thought (CoT) prompting, showing that CoT can collapse the spread of the judgment distribution, often harming performance. Our findings suggest leveraging distributional output can improve LLM-as-a-judge, as opposed to using the text interface alone.
• 摘要：使用语言模型可缩减对文本质量的人类偏好（LLM-AS-A-Gudge）已成为适用于许多任务的标准实践。通常仅从法官的文本输出中提取判断，通常是贪婪的解码。但是，LLM法官自然会在判断令牌上提供分布，邀请推理方法广度以提取细粒度的偏好。我们发现，在所有评估设置（即尖，成对，成对和listwise）中，以判断分布的平均值始终优于采用模式（即贪婪解码）的表现。我们进一步探讨了从判断分布中得出偏好的新方法，并发现融合风险规定的方法通常会改善绩效。最后，我们分析了LLM-AS-A-A-Gudge与Thebough（COT）提示配对，表明COT可能会崩溃判断分布的传播，通常会损害绩效。我们的发现表明，利用分配输出可以改善llm-as-a-a-a-a-a gudge，而不是单独使用文本接口。

Title: Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generation

• Authors: Yurui Chang, Bochuan Cao, Lu Lin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03106
• Pdf URL: https://arxiv.org/pdf/2503.03106
• Copy Paste: [[2503.03106]] Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generation(https://arxiv.org/abs/2503.03106)
• Keywords: language model, hallucination
• Abstract: While large language models have demonstrated exceptional performance across a wide range of tasks, they remain susceptible to hallucinations -- generating plausible yet factually incorrect contents. Existing methods to mitigating such risk often rely on sampling multiple full-length generations, which introduces significant response latency and becomes ineffective when the model consistently produces hallucinated outputs with high confidence. To address these limitations, we introduce Monitoring Decoding (MD), a novel framework that dynamically monitors the generation process and selectively applies in-process interventions, focusing on revising crucial tokens responsible for hallucinations. Instead of waiting until completion of multiple full-length generations, we identify hallucination-prone tokens during generation using a monitor function, and further refine these tokens through a tree-based decoding strategy. This approach ensures an enhanced factual accuracy and coherence in the generated output while maintaining efficiency. Experimental results demonstrate that MD consistently outperforms self-consistency-based approaches in both effectiveness and efficiency, achieving higher factual accuracy while significantly reducing computational overhead.
• 摘要：尽管大型语言模型在广泛的任务中表现出了出色的表现，但它们仍然容易受到幻觉的影响 - 产生了合理但实际上不正确的内容。现有的减轻这种风险的方法通常依赖于对多个全长世代进行取样，这引入了显着的响应潜伏期，并且当模型始终以高信心产生幻觉输出时，变得无效。为了解决这些局限性，我们引入了监视解码（MD），这是一个动态监视生成过程并有选择地采用进程干预措施的新型框架，重点是修改负责幻觉的至关重要的令牌。我们没有等到完成多个全长世代的完成，而是使用监视器功能在生成过程中识别易幻觉的令牌，并通过基于树的解码策略进一步完善这些令牌。这种方法可确保在维持效率的同时，在生成的产出中提高了事实的准确性和连贯性。实验结果表明，MD在有效性和效率方面始终优于基于自符的方法，从而达到更高的事实准确性，同时显着降低了计算开销。

Title: The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models

• Authors: Zichao Li, Xueru Wen, Jie Lou, Yuqiu Ji, Yaojie Lu, Xianpei Han, Debing Zhang, Le Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03122
• Pdf URL: https://arxiv.org/pdf/2503.03122
• Copy Paste: [[2503.03122]] The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models(https://arxiv.org/abs/2503.03122)
• Keywords: language model, llm
• Abstract: Multimodal Reward Models (MM-RMs) are crucial for aligning Large Language Models (LLMs) with human preferences, particularly as LLMs increasingly interact with multimodal data. However, we find that MM-RMs trained on existing datasets often struggle to generalize to out-of-distribution data due to their reliance on unimodal spurious correlations, primarily text-only shortcuts within the training distribution, which prevents them from leveraging true multimodal reward functions. To address this, we introduce a Shortcut-aware MM-RM learning algorithm that mitigates this issue by dynamically reweighting training samples, shifting the distribution toward better multimodal understanding, and reducing dependence on unimodal spurious correlations. Our experiments demonstrate significant improvements in generalization, downstream task performance, and scalability, establishing a more robust framework for multimodal reward modeling.
• 摘要：多模式奖励模型（MM-RMS）对于将大语言模型（LLM）与人类偏好保持一致，尤其是当LLM越来越多地与多模式数据相互作用时。但是，我们发现，在现有数据集中训练的MM-RMS通常由于依赖单峰式伪造相关性，主要是培训分布中仅文本的快捷方式，因此通常难以推广到分布数据，这阻止了他们利用真正的多模式奖励功能。为了解决这个问题，我们介绍了一种快捷方式感知的MM-RM学习算法，该算法通过动态重新加权训练样本，将分布转移到更好的多模式理解并减少对单峰式伪造相关性的依赖性来减轻此问题。我们的实验表明，概括，下游任务性能和可伸缩性的显着改善，为多模式奖励建模建立了更强大的框架。

Title: DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models

• Authors: YiQiu Guo, Yuchen Yang, Zhe Chen, Pingjie Wang, Yusheng Liao, Ya Zhang, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03149
• Pdf URL: https://arxiv.org/pdf/2503.03149
• Copy Paste: [[2503.03149]] DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models(https://arxiv.org/abs/2503.03149)
• Keywords: language model, hallucination
• Abstract: The reliability of large language models remains a critical challenge, particularly due to their susceptibility to hallucinations and factual inaccuracies during text generation. Existing solutions either underutilize models' self-correction with preemptive strategies or use costly post-hoc verification. To further explore the potential of real-time self-verification and correction, we present Dynamic Self-Verify Decoding (DSVD), a novel decoding framework that enhances generation reliability through real-time hallucination detection and efficient error correction. DSVD integrates two key components: (1) parallel self-verification architecture for continuous quality assessment, (2) dynamic rollback mechanism for targeted error recovery. Extensive experiments across five benchmarks demonstrate DSVD's effectiveness, achieving significant improvement in truthfulness (Quesetion-Answering) and factual accuracy (FActScore). Results show the DSVD can be further incorporated with existing faithful decoding methods to achieve stronger performance. Our work establishes that real-time self-verification during generation offers a viable path toward more trustworthy language models without sacrificing practical deployability.
• 摘要：大型语言模型的可靠性仍然是一个关键的挑战，尤其是由于它们在文本生成过程中对幻觉和事实不准确的敏感性。现有的解决方案要么通过预先策略的自我纠正不足，要么使用昂贵的事后验证。为了进一步探索实时自我验证和校正的潜力，我们提出了动态自我验证解码（DSVD），这是一个新型的解码框架，可通过实时幻觉检测和有效的误差校正来增强产生可靠性。 DSVD集成了两个关键组成部分：（1）用于连续质量评估的平行自我验证体系结构，（2）针对目标误差恢复的动态回滚机制。跨五个基准的广泛实验表明了DSVD的有效性，可以显着改善真实性（征询问题）和事实准确性（FactScore）。结果表明，可以将DSVD与现有的忠实解码方法进一步合并，以实现更强的性能。我们的工作确定，一代人期间的实时自我验证为通往更可信赖的语言模型的可行途径而无需牺牲实际的可部署性。

Title: Structured Outputs Enable General-Purpose LLMs to be Medical Experts

• Authors: Guangfu Guo, Kai Zhang, Bryan Hoo, Yujun Cai, Xiaoqian Lu, Nanyun Peng, Yiwei Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03194
• Pdf URL: https://arxiv.org/pdf/2503.03194
• Copy Paste: [[2503.03194]] Structured Outputs Enable General-Purpose LLMs to be Medical Experts(https://arxiv.org/abs/2503.03194)
• Keywords: language model, llm, hallucination
• Abstract: Medical question-answering (QA) is a critical task for evaluating how effectively large language models (LLMs) encode clinical knowledge and assessing their potential applications in medicine. Despite showing promise on multiple-choice tests, LLMs frequently struggle with open-ended medical questions, producing responses with dangerous hallucinations or lacking comprehensive coverage of critical aspects. Existing approaches attempt to address these challenges through domain-specific fine-tuning, but this proves resource-intensive and difficult to scale across models. To improve the comprehensiveness and factuality of medical responses, we propose a novel approach utilizing structured medical reasoning. Our method guides LLMs through an seven-step cognitive process inspired by clinical diagnosis, enabling more accurate and complete answers without additional training. Experiments on the MedLFQA benchmark demonstrate that our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models. Notably, this improvement transfers to smaller models, highlighting the method's efficiency and scalability. Our code and datasets are available.
• 摘要：医疗问题救助（QA）是评估大型语言模型（LLM）如何编码临床知识并评估其在医学中的潜在应用的关键任务。尽管在多项选择测试上表现出了希望，但LLMS经常在开放式医疗问题上挣扎，产生危险幻觉或缺乏对关键方面的全面覆盖的回答。现有的方法试图通过特定领域的微调来应对这些挑战，但这证明了资源密集型且难以扩展在模型中。为了提高医学反应的全面性和事实，我们提出了一种利用结构化医学推理的新方法。我们的方法通过受临床诊断启发的七个步骤认知过程指导LLM，从而在没有其他培训的情况下可以更准确，完整的答案。 MEDLFQA基准测试的实验表明，我们的方法达到了85.8的最高事实得分，超过了微调模型。值得注意的是，这种改进转移到较小的模型，突出了该方法的效率和可扩展性。我们的代码和数据集可用。

Title: Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution

• Authors: Jizhao Zhu, Akang Shi, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03201
• Pdf URL: https://arxiv.org/pdf/2503.03201
• Copy Paste: [[2503.03201]] Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution(https://arxiv.org/abs/2503.03201)
• Keywords: language model, llm
• Abstract: In this paper, we aim to enhance the robustness of Universal Information Extraction (UIE) by introducing a new benchmark dataset, a comprehensive evaluation, and a feasible solution. Existing robust benchmark datasets have two key limitations: 1) They generate only a limited range of perturbations for a single Information Extraction (IE) task, which fails to evaluate the robustness of UIE models effectively; 2) They rely on small models or handcrafted rules to generate perturbations, often resulting in unnatural adversarial examples. Considering the powerful generation capabilities of Large Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE, called RUIE-Bench, which utilizes LLMs to generate more diverse and realistic perturbations across different IE tasks. Based on this dataset, we comprehensively evaluate existing UIE models and reveal that both LLM-based models and other models suffer from significant performance drops. To improve robustness and reduce training costs, we propose a data-augmentation solution that dynamically selects hard samples for iterative training based on the model's inference loss. Experimental results show that training with only \textbf{15\%} of the data leads to an average \textbf{7.5\%} relative performance improvement across three IE tasks.
• 摘要：在本文中，我们旨在通过引入新的基准数据集，全面评估和可行解决方案来增强通用信息提取（UIE）的鲁棒性。现有的强大基准数据集具有两个关键局限性：1）它们仅生成有限的单个信息提取（IE）任务的扰动范围，该任务无法有效地评估UIE模型的鲁棒性； 2）他们依靠小型模型或手工制作的规则来产生扰动，通常会导致不自然的对抗性示例。考虑到大型语言模型（LLMS）的强大生成能力，我们为Robust Uie（称为Ruie-Bench）引入了一个新的基准数据集，该数据集利用LLMS在不同的IE任务中生成更多样化和现实的扰动。基于此数据集，我们全面评估了现有的UIE模型，并揭示了基于LLM的模型和其他模型都有大量性能下降。为了提高鲁棒性并降低培训成本，我们提出了一种数据实践解决方案，该解决方案基于模型的推理损失，动态选择硬样品进行迭代培训。实验结果表明，仅对数据的\ textBf {15 \％}进行培训会导致平均\ textbf {7.5 \％}相对性能改进三个IE任务。

Title: MA-LoT: Multi-Agent Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving

• Authors: Ruida Wang, Rui Pan, Yuxin Li, Jipeng Zhang, Yizhen Jia, Shizhe Diao, Renjie Pi, Junjie Hu, Tong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03205
• Pdf URL: https://arxiv.org/pdf/2503.03205
• Copy Paste: [[2503.03205]] MA-LoT: Multi-Agent Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving(https://arxiv.org/abs/2503.03205)
• Keywords: language model, gpt, llm, chain-of-thought, agent
• Abstract: Solving mathematical problems using computer-verifiable languages like Lean has significantly impacted mathematical and computer science communities. State-of-the-art methods utilize single Large Language Models (LLMs) as agents or provers to either generate complete proof or perform tree searches. However, single-agent methods inherently lack a structured way to combine high-level reasoning in Natural Language (NL) with Formal Language (FL) verification feedback. To solve these issues, we propose MA-LoT: Multi-Agent Lean-based Long Chain-of-Thought framework, (to the best of our knowledge), the first multi-agent framework for Lean4 theorem proving that balance high-level NL reasoning and FL verification in Long CoT. Using this structured interaction, our approach enables deeper insights and long-term coherence in proof generation, with which past methods struggle. We do this by leveraging emergent formal reasoning ability in Long CoT using our novel LoT-Transfer Learning training-inference pipeline. Extensive experiments show that our framework achieves 54.51% accuracy rate on the Lean4 version of MiniF2F-Test dataset, largely outperforming GPT-4 (22.95%), single-agent tree search (InternLM-Step-Prover, 50.70%), and whole-proof generation (DeepSeek-Prover-v1.5, 48.36%) baselines. Furthermore, our findings highlight the potential of combining Long CoT with formal verification for a more insightful generation in a broader perspective.
• 摘要：使用计算机可验证的语言（如精益）解决数学问题已对数学和计算机科学社区产生重大影响。最先进的方法利用单个大语言模型（LLMS）作为代理或摊贩来生成完整的证明或执行树搜索。但是，单一代理方法固有地缺乏将自然语言（NL）与正式语言（FL）验证反馈相结合的结构化方法。为了解决这些问题，我们提出了MA-LOT：基于多代理的长期思考框架（据我们所知），这是第一个用于LEAN4定理的多代理框架，证明了长期COT中高级NL推理和FL验证的平衡。利用这种结构化的互动，我们的方法可以使过去方法与之奋斗的更深入的见解和长期连贯性。我们通过使用我们的新型Lot-Transfer学习训练推动管道来利用长床的新兴正式推理能力来做到这一点。广泛的实验表明，我们的框架在lean4版本的minif2f检验数据集上达到了54.51％的精度率，在很大程度上超过了GPT-4（22.95％），单格树搜索（Internlm-Step-prover，50.70％），50.70％），以及全耐用的生成（DeepSeek-Prover-prover-prover-verver-v1.5.5，48.3666666 66％）。此外，我们的发现突出了将长床与正式验证相结合的潜力，从更广泛的角度来看，从而更有见识。

Title: Targeted Distillation for Sentiment Analysis

• Authors: Yice Zhang, Guangyu Xie, Jingjie Lin, Jianzhu Bao, Qianlong Wang, Xi Zeng, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03225
• Pdf URL: https://arxiv.org/pdf/2503.03225
• Copy Paste: [[2503.03225]] Targeted Distillation for Sentiment Analysis(https://arxiv.org/abs/2503.03225)
• Keywords: language model, llm, prompt
• Abstract: This paper presents a compact model that achieves strong sentiment analysis capabilities through targeted distillation from advanced large language models (LLMs). Our methodology decouples the distillation target into two key components: sentiment-related knowledge and task alignment. To transfer these components, we propose a two-stage distillation framework. The first stage, knowledge-driven distillation (\textsc{KnowDist}), transfers sentiment-related knowledge to enhance fundamental sentiment analysis capabilities. The second stage, in-context learning distillation (\textsc{ICLDist}), transfers task-specific prompt-following abilities to optimize task alignment. For evaluation, we introduce \textsc{SentiBench}, a comprehensive sentiment analysis benchmark comprising 3 task categories across 12 datasets. Experiments on this benchmark demonstrate that our model effectively balances model size and performance, showing strong competitiveness compared to existing small-scale LLMs.
• 摘要：本文提出了一个紧凑的模型，该模型通过从高级大语模型（LLMS）的有针对性蒸馏来实现强大的情感分析功能。我们的方法将蒸馏目标分解为两个关键组成部分：与情感相关的知识和任务一致性。要转移这些组件，我们提出了一个两阶段的蒸馏框架。第一阶段是知识驱动的蒸馏（\ textsc {knowdist}），转移了与情感相关的知识，以增强基本情感分析能力。第二阶段，即文下面的学习蒸馏（\ textsc {icldist}），转移了特定于任务的及时关注能力以优化任务对齐。为了进行评估，我们介绍了\ textsc {sentibench}，这是一个全面的情感分析基准，其中包括12个数据集的3个任务类别。该基准测试的实验表明，与现有的小规模LLM相比，我们的模型有效地平衡了模型的大小和性能，显示出强大的竞争力。

Title: FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4

• Authors: Jiarui Yao, Ruida Wang, Tong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03238
• Pdf URL: https://arxiv.org/pdf/2503.03238
• Copy Paste: [[2503.03238]] FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4(https://arxiv.org/abs/2503.03238)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have displayed astonishing abilities in various tasks, especially in text generation, classification, question answering, etc. However, the reasoning ability of LLMs still faces many debates. The inherent ambiguity of Natural Language (NL) limits LLMs' ability to perform verifiable reasoning, making its answers lack coherence and trustworthy support. To tackle the above problems, we propose a novel framework named FANS: Formal ANswer Selection for Natural Language Math Reasoning Using Lean4. To the best of our knowledge, it is the first framework that utilizes Lean4 to enhance LLMs' NL math reasoning ability. In particular, given an NL math question and LLM-generated answers, FANS first translates it into Lean4 theorem statements. Then it tries to prove it using a Lean4 prover and verify it by Lean4. Finally, it uses the FL result to assist in answer selection. It enhances LLMs' NL math ability in providing a computer-verifiable solution for its correct answer and proposes an alternative method for answer selection beyond the reward model. Extensive experiments indicate the effectiveness of our framework. It can improve the accuracy rate of reward model enhanced LLMs in the MATH-500 dataset by at most 1.91% and AMC-23 by at most 8.33% on strong reward-model baselines. In some particular fields like number theory that Lean4 experts in, we can even select all correct solutions. The qualitative analysis also shows our framework can make NL results formally backed by Lean4 proofs. As a pioneering work in the corresponding field, we will open-source all our models and datasets to further boost the development of the field.
• 摘要：大型语言模型（LLM）在各种任务中表现出惊人的能力，尤其是在文本生成，分类，问题答案等中。但是，LLM的推理能力仍然面临许多辩论。自然语言（NL）的固有歧义限制了LLMS执行可验证推理的能力，使其答案缺乏连贯性和值得信赖的支持。为了解决上述问题，我们提出了一个名为“粉丝”的新颖框架：使用Lean 4进行自然语言数学推理的正式答案。据我们所知，这是第一个利用Lean4来增强LLMS NL数学推理能力的框架。特别是，鉴于NL数学问题和LLM生成的答案，粉丝首先将其转化为Lean4定理语句。然后，它试图使用Lean4供者证明它并通过Lean4进行验证。最后，它使用FL结果来帮助选择答案。它增强了LLMS的NL数学能力，以为其正确答案提供计算机验证的解决方案，并提出了一种替代奖励模型的答案选择方法。广泛的实验表明我们框架的有效性。它可以提高奖励模型在Math-500数据集中增强LLM的准确率最多的1.91％，而AMC-23的准确率最多可以在强奖励模型基线上最多增加8.33％。在某些特定领域，例如Lean4专家的数字理论，我们甚至可以选择所有正确的解决方案。定性分析还表明，我们的框架可以使NL结果正式支持Lean4证明。作为在相应领域的开创性工作，我们将打开所有模型和数据集，以进一步促进该领域的开发。

Title: Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions

• Authors: Yichong Zhao, Susumu Goto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03261
• Pdf URL: https://arxiv.org/pdf/2503.03261
• Copy Paste: [[2503.03261]] Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions(https://arxiv.org/abs/2503.03261)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) can perform various natural language processing (NLP) tasks through in-context learning without relying on supervised data. However, multiple previous studies have reported suboptimal performance of LLMs in biological text mining. By analyzing failure patterns in these evaluations, we identified three primary challenges for LLMs in biomedical corpora: (1) LLMs fail to learn implicit dataset-specific nuances from supervised data, (2) The common formatting requirements of discriminative tasks limit the reasoning capabilities of LLMs particularly for LLMs that lack test-time compute, and (3) LLMs struggle to adhere to annotation guidelines and match exact schemas, which hinders their ability to understand detailed annotation requirements which is essential in biomedical annotation workflow. To address these challenges, we experimented with prompt engineering techniques targeted to the above issues, and developed a pipeline that dynamically extracts instructions from annotation guidelines. Our findings show that frontier LLMs can approach or surpass the performance of state-of-the-art (SOTA) BERT-based models with minimal reliance on manually annotated data and without fine-tuning. Furthermore, we performed model distillation on a closed-source LLM, demonstrating that a BERT model trained exclusively on synthetic data annotated by LLMs can also achieve a practical performance. Based on these results, we explored the feasibility of partially replacing manual annotation with LLMs in production scenarios for biomedical text mining.
• 摘要：大型语言模型（LLM）可以通过内在学习执行各种自然语言处理（NLP）任务，而无需依赖监督数据。但是，先前的多项研究报告了LLM在生物文本挖掘中的次优性能。通过分析这些评估中的故障模式，我们确定了生物医学语料库中LLM的三个主要挑战：（1）LLMS无法从监督数据中学习隐式数据特定的细微差别，（2）（2）歧视任务的常见格式要求，歧视任务的常见格式要求限制了LLMS的限制，尤其是对llms的限制，尤其是对测试时间的范围和（3）LL的范围，并且（3）模式阻碍了他们了解生物医学注释工作流程至关重要的详细注释要求的能力。为了应对这些挑战，我们尝试了针对上述问题的及时工程技术，并开发了一种动态从注释指南中提取说明的管道。我们的发现表明，Frontier LLM可以接近或超越最先进的（SOTA）基于BERT的模型的性能，而对手动注释的数据的依赖最少，而无需进行微调。此外，我们在封闭源LLM上进行了模型蒸馏，表明仅在LLMS注释的合成数据上训练的BERT模型也可以实现实践性能。基于这些结果，我们探讨了在生物医学文本挖掘的生产场景中用LLM部分替换手动注释的可行性。

Title: LexGenie: Automated Generation of Structured Reports for European Court of Human Rights Case Law

• Authors: T.Y.S.S Santosh, Mahmoud Aly, Oana Ichim, Matthias Grabmair
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03266
• Pdf URL: https://arxiv.org/pdf/2503.03266
• Copy Paste: [[2503.03266]] LexGenie: Automated Generation of Structured Reports for European Court of Human Rights Case Law(https://arxiv.org/abs/2503.03266)
• Keywords: llm
• Abstract: Analyzing large volumes of case law to uncover evolving legal principles, across multiple cases, on a given topic is a demanding task for legal professionals. Structured topical reports provide an effective solution by summarizing key issues, principles, and judgments, enabling comprehensive legal analysis on a particular topic. While prior works have advanced query-based individual case summarization, none have extended to automatically generating multi-case structured reports. To address this, we introduce LexGenie, an automated LLM-based pipeline designed to create structured reports using the entire body of case law on user-specified topics within the European Court of Human Rights jurisdiction. LexGenie retrieves, clusters, and organizes relevant passages by topic to generate a structured outline and cohesive content for each section. Expert evaluation confirms LexGenie's utility in producing structured reports that enhance efficient, scalable legal analysis.
• 摘要：分析大量判例法以在多个案件中揭示不断发展的法律原则，这是法律专业人士的一项要求的任务。结构化的局部报告通过总结关键问题，原则和判断提供了有效的解决方案，从而对特定主题进行了全面的法律分析。虽然先前的工作已经具有高级基于查询的个人案例摘要，但没有一个扩展到自动生成多案例结构化报告。为了解决这个问题，我们介绍了莱克斯基（Lexgenie），这是一种基于LLM的自动化管道，旨在使用欧洲人权管辖权法院内的用户指定主题的整个案例法创建结构化报告。 Lexgenie检索，集群并按主题组织相关段落，以生成每个部分的结构化轮廓和凝聚力内容。专家评估证实了Lexgenie在生产有效，可扩展法律分析的结构化报告中的效用。

Title: SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection

• Authors: Yi-Fan Lu, Xian-Ling Mao, Tian Lan, Tong Zhang, Yu-Shi Zhu, Heyan Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03303
• Pdf URL: https://arxiv.org/pdf/2503.03303
• Copy Paste: [[2503.03303]] SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection(https://arxiv.org/abs/2503.03303)
• Keywords: language model, llm, agent
• Abstract: Automatic evaluation for Open Domain Event Detection (ODED) is a highly challenging task, because ODED is characterized by a vast diversity of un-constrained output labels from various domains. Nearly all existing evaluation methods for ODED usually first construct evaluation benchmarks with limited labels and domain coverage, and then evaluate ODED methods using metrics based on token-level label matching rules. However, this kind of evaluation framework faces two issues: (1) The limited evaluation benchmarks lack representatives of the real world, making it difficult to accurately reflect the performance of various ODED methods in real-world scenarios; (2) Evaluation metrics based on token-level matching rules fail to capture semantic similarity between predictions and golden labels. To address these two problems above, we propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection (SEOE) by constructing a more representative evaluation benchmark and introducing a semantic evaluation metric. Specifically, our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains, with a cost-effective supplementary annotation strategy to ensure the benchmark's representativeness. The strategy also allows for the supplement of new event types and domains in the future. Then, the proposed SEOE leverages large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels to enhance the reliability of the evaluation. Extensive experiments validate the representatives of the benchmark and the reliability of the semantic evaluation metric. Existing ODED methods are thoroughly evaluated, and the error patterns of predictions are analyzed, revealing several insightful findings.
• 摘要：对开放域事件检测（ODED）的自动评估是一项高度挑战的任务，因为ODED的特征是来自各个域中的大量无约束的输出标签。 ODED的几乎所有现有评估方法通常是第一个构建标签和域覆盖范围有限的首次构建评估基准，然后使用基于令牌级标签匹配规则的指标评估ODED方法。但是，这种评估框架面临两个问题：（1）有限的评估基准缺乏现实世界的代表，因此很难准确地反映现实世界中各种ODED方法的性能； （2）基于令牌级匹配规则的评估指标无法捕获预测和黄金标签之间的语义相似性。为了解决上面的这两个问题，我们通过构建更具代表性的评估基准并引入语义评估指标，提出了一个可扩展且可靠的语义级别评估框架（SEOE）。具体而言，我们提出的框架首先构建了可扩展的评估基准，该基准目前包括564种涵盖7个主要领域的事件类型，并采用具有成本效益的补充注释策略，以确保基准的代表性。该策略还允许将来补充新事件类型和域。然后，拟议的SEOE利用大语言模型（LLM）作为自动评估剂来计算语义F1分数，并结合了语义上相似标签的细粒度定义，以增强评估的可靠性。广泛的实验验证了基准的代表和语义评估指标的可靠性。对现有的ODED方法进行了详尽的评估，并分析了预测的误差模式，揭示了一些有见地的发现。

Title: The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation

• Authors: Jie He, Tao Wang, Deyi Xiong, Qun Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03308
• Pdf URL: https://arxiv.org/pdf/2503.03308
• Copy Paste: [[2503.03308]] The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation(https://arxiv.org/abs/2503.03308)
• Keywords: language model, gpt
• Abstract: Does neural machine translation yield translations that are congenial with common sense? In this paper, we present a test suite to evaluate the commonsense reasoning capability of neural machine translation. The test suite consists of three test sets, covering lexical and contextless/contextual syntactic ambiguity that requires commonsense knowledge to resolve. We manually create 1,200 triples, each of which contain a source sentence and two contrastive translations, involving 7 different common sense types. Language models pretrained on large-scale corpora, such as BERT, GPT-2, achieve a commonsense reasoning accuracy of lower than 72% on target translations of this test suite. We conduct extensive experiments on the test suite to evaluate commonsense reasoning in neural machine translation and investigate factors that have impact on this capability. Our experiments and analyses demonstrate that neural machine translation performs poorly on commonsense reasoning of the three ambiguity types in terms of both reasoning accuracy (60.1%) and reasoning consistency (31%). The built commonsense test suite is available at this https URL.
• 摘要：神经机器的翻译是否产生具有常识的友善的翻译？在本文中，我们提出了一个测试套件，以评估神经机器翻译的常识性推理能力。测试套件由三个测试组组成，涵盖了需要常识知识来解决的词汇和无上下文/上下文句法歧义。我们手动创建了1,200个三元组，每个三元词包含一个源句子和两个对比翻译，涉及7种不同的常识类型。在此测试套件的目标翻译上，在大规模语料库中鉴定的语言模型（例如BERT，GPT-2）的平均值推理精度低于72％。我们在测试套件上进行了广泛的实验，以评估神经机译中的常识性推理，并研究影响该能力的因素。我们的实验和分析表明，在推理准确性（60.1％）和推理一致性方面，神经机器的转换在三种歧义类型的常识性推理方面的表现较差（31％）。该HTTPS URL可用内置的常识测试套件。

Title: iNews: A Multimodal Dataset for Modeling Personalized Affective Responses to News

• Authors: Tiancheng Hu, Nigel Collier
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.03335
• Pdf URL: https://arxiv.org/pdf/2503.03335
• Copy Paste: [[2503.03335]] iNews: A Multimodal Dataset for Modeling Personalized Affective Responses to News(https://arxiv.org/abs/2503.03335)
• Keywords: llm
• Abstract: Current approaches to emotion detection often overlook the inherent subjectivity of affective experiences, instead relying on aggregated labels that mask individual variations in emotional responses. We introduce iNews, a novel large-scale dataset explicitly capturing subjective affective responses to news headlines. Our dataset comprises annotations from 291 demographically diverse UK participants across 2,899 multimodal Facebook news posts from major UK outlets, with an average of 5.18 annotators per sample. For each post, annotators provide multifaceted labels including valence, arousal, dominance, discrete emotions, content relevance judgments, sharing likelihood, and modality importance ratings (text, image, or both). Furthermore, we collect comprehensive annotator persona information covering demographics, personality, media trust, and consumption patterns, which explain 15.2% of annotation variance - higher than existing NLP datasets. Incorporating this information yields a 7% accuracy gain in zero-shot prediction and remains beneficial even with 32-shot. iNews will enhance research in LLM personalization, subjectivity, affective computing, and individual-level behavior simulation.
• 摘要：当前的情绪检测方法通常会忽略情感体验的固有主观性，而依靠掩盖情绪反应中个人变化的汇总标签。我们介绍了INEWS，这是一个新颖的大型数据集，明确捕获了对新闻头条的主观情感反应。我们的数据集包含来自2,899名英国主要渠道的2,899个多模式的Facebook新闻帖子的291个人口统计学不同的参与者的注释，每个样本平均有5.18个注释者。对于每篇文章，注释者都会提供多方面的标签，包括价，唤醒，优势，离散情绪，内容相关性判断，共享可能性和模态重要性等级（文本，图像或两者）。此外，我们收集涵盖人口统计学，个性，媒体信任和消费模式的全面注释人角色信息，这些信息解释了注释差异的15.2％ - 高于现有的NLP数据集。纳入此信息的零摄像预测中的准确性增长了7％，即使32次射击仍然有益。 INEWS将增强LLM个性化，主观性，情感计算和个人级别行为模拟的研究。

Title: EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity States

• Authors: Hainiu Xu, Siya Qi, Jiazheng Li, Yuxiang Zhou, Jinhua Du, Caroline Catmur, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03340
• Pdf URL: https://arxiv.org/pdf/2503.03340
• Copy Paste: [[2503.03340]] EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity States(https://arxiv.org/abs/2503.03340)
• Keywords: language model, llm
• Abstract: Theory-of-Mind (ToM), the ability to infer others' perceptions and mental states, is fundamental to human interaction but remains a challenging task for Large Language Models (LLMs). While existing ToM reasoning methods show promise with reasoning via perceptual perspective-taking, they often rely excessively on LLMs, reducing their efficiency and limiting their applicability to high-order ToM reasoning, which requires multi-hop reasoning about characters' beliefs. To address these issues, we present EnigmaToM, a novel neuro-symbolic framework that enhances ToM reasoning by integrating a Neural Knowledge Base of entity states (Enigma) for (1) a psychology-inspired iterative masking mechanism that facilitates accurate perspective-taking and (2) knowledge injection that elicits key entity information. Enigma generates structured representations of entity states, which construct spatial scene graphs -- leveraging spatial information as an inductive bias -- for belief tracking of various ToM orders and enhancing events with fine-grained entity state details. Experimental results on multiple benchmarks, including ToMi, HiToM, and FANToM, show that EnigmaToM significantly improves ToM reasoning across LLMs of varying sizes, particularly excelling in high-order reasoning scenarios.
• 摘要：理论（汤姆）是推断他人的看法和精神状态的能力，是人类互动的基础，但对于大型语言模型（LLMS）来说仍然是一项具有挑战性的任务。尽管现有的TOM推理方法通过感知观点提出推理表现出希望，但他们通常过度依赖LLM，降低其效率并将其适用性限制在高阶Tom推理上，这需要对角色信念进行多跳的推理。为了解决这些问题，我们提出了一种新型的神经符号框架，该框架是通过整合实体状态的神经知识基础（Enigma）来增强TOM推理的（（1）心理学启发性迭代的迭代掩盖机制，可促进准确的观点和（2）具有钥匙内信息信息的知识注入的知识。 Enigma生成了实体状态的结构化表示，这些状态构建空间场景图（利用空间信息作为电感偏见），以信念对各种TOM订单的信念跟踪，并使用细粒度实体状态细节来增强事件。包括Tomi，Hitom和Fantom在内的多个基准测试的实验结果表明，Enigmatom显着改善了跨不同尺寸的LLM的TOM推理，在高阶推理方案中尤其出色。

Title: The Serendipity of Claude AI: Case of the 13 Low-Resource National Languages of Mali

• Authors: Alou Dembele, Nouhoum Souleymane Coulibaly, Michael Leventhal (RobotsMali AI4D Lab, Bamako, Mali)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03380
• Pdf URL: https://arxiv.org/pdf/2503.03380
• Copy Paste: [[2503.03380]] The Serendipity of Claude AI: Case of the 13 Low-Resource National Languages of Mali(https://arxiv.org/abs/2503.03380)
• Keywords: llm
• Abstract: Recent advances in artificial intelligence (AI) and natural language processing (NLP) have improved the representation of underrepresented languages. However, most languages, including Mali's 13 official national languages, continue to be poorly supported or unsupported by automatic translation and generative AI. This situation appears to have slightly improved with certain recent LLM releases. The study evaluated Claude AI's translation performance on each of the 13 national languages of Mali. In addition to ChrF2 and BLEU scores, human evaluators assessed translation accuracy, contextual consistency, robustness to dialect variations, management of linguistic bias, adaptation to a limited corpus, and ease of understanding. The study found that Claude AI performs robustly for languages with very modest language resources and, while unable to produce understandable and coherent texts for Malian languages with minimal resources, still manages to produce results which demonstrate the ability to mimic some elements of the language.
• 摘要：人工智能（AI）和自然语言处理（NLP）的最新进展改善了代表性不足的语言。但是，包括马里的13种官方语言在内的大多数语言仍然受到自动翻译和生成AI的支持或不支持。由于最近的某些LLM版本，这种情况似乎有所改善。该研究评估了Claude AI对马里13种国家语言中每种语言的翻译表现。除了CHRF2和BLEU分数外，人类评估者还评估了翻译精度，情境一致性，对方言变化的稳健性，语言偏见的管理，适应有限的语料库以及易于理解。该研究发现，克劳德·AI（Claude AI）对具有非常适中的语言资源的语言表现出色，虽然无法生产具有最低资源的马里语语言的可理解且连贯的文本，但仍然设法产生结果，以表明能够模仿该语言的某些元素。

Title: When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits

• Authors: Jabez Magomere, Emanuele La Malfa, Manuel Tonneau, Ashkan Kazemi, Scott Hale
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03417
• Pdf URL: https://arxiv.org/pdf/2503.03417
• Copy Paste: [[2503.03417]] When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits(https://arxiv.org/abs/2503.03417)
• Keywords: llm
• Abstract: Online misinformation remains a critical challenge, and fact-checkers increasingly rely on embedding-based methods to retrieve relevant fact-checks. Yet, when debunked claims reappear in edited forms, the performance of these methods is unclear. In this work, we introduce a taxonomy of six common real-world misinformation edits and propose a perturbation framework that generates valid, natural claim variations. Our multi-stage retrieval evaluation reveals that standard embedding models struggle with user-introduced edits, while LLM-distilled embeddings offer improved robustness at a higher computational cost. Although a strong reranker helps mitigate some issues, it cannot fully compensate for first-stage retrieval gaps. Addressing these retrieval gaps, our train- and inference-time mitigation approaches enhance in-domain robustness by up to 17 percentage points and boost out-of-domain generalization by 10 percentage points over baseline models. Overall, our findings provide practical improvements to claim-matching systems, enabling more reliable fact-checking of evolving misinformation.
• 摘要：在线错误信息仍然是一个关键的挑战，事实检查者越来越依赖基于嵌入的方法来检索相关的事实检查。但是，当被揭穿的索赔重新出现时，这些方法的性能尚不清楚。在这项工作中，我们介绍了六个常见的现实世界错误信息编辑的分类法，并提出了一个产生有效自然主张变化的扰动框架。我们的多阶段检索评估表明，标准嵌入模型与用户引入的编辑斗争，而LLM-DISTISTLIDER EMBEDINGS则以较高的计算成本提供了改善的鲁棒性。尽管强大的重读者有助于减轻某些问题，但它无法完全弥补第一阶段的检索差距。在解决这些检索差距时，我们的火车和推理时间缓解方法可以提高内域的鲁棒性高达17个百分点，并使偏域概括提高了10个百分点，而不是基线模型。总体而言，我们的发现为索赔匹配系统提供了实际改进，从而使不断发展的错误信息进行了更可靠的事实检查。

Title: RASD: Retrieval-Augmented Speculative Decoding

• Authors: Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, Hao Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03434
• Pdf URL: https://arxiv.org/pdf/2503.03434
• Copy Paste: [[2503.03434]] RASD: Retrieval-Augmented Speculative Decoding(https://arxiv.org/abs/2503.03434)
• Keywords: language model, llm
• Abstract: Speculative decoding accelerates inference in large language models (LLMs) by generating draft tokens for target model verification. Current approaches for obtaining draft tokens rely on lightweight draft models or additional model structures to generate draft tokens and retrieve context from databases. Due to the draft model's small size and limited training data, model-based speculative decoding frequently becomes less effective in out-of-domain scenarios. Additionally, the time cost of the drafting phase results in a low upper limit on acceptance length during the verification step, limiting overall efficiency. This paper proposes RASD (Retrieval-Augmented Speculative Decoding), which adopts retrieval methods to enhance model-based speculative decoding. We introduce tree pruning and tree fusion to achieve this. Specifically, we develop a pruning method based on the draft model's probability distribution to construct the optimal retrieval tree. Second, we employ the longest prefix matching algorithm to merge the tree generated by the draft model with the retrieval tree, resulting in a unified tree for verification. Experimental results demonstrate that RASD achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA. Moreover, RASD exhibits strong scalability, seamlessly integrating with various speculative decoding approaches, including both generation-based and retrieval-based methods.
• 摘要：通过生成目标模型验证的草稿令牌，投机解码可以加速大型语言模型（LLM）的推断。当前获取令牌的方法依赖于轻巧的草稿模型或其他模型结构来生成草稿令牌并从数据库中检索上下文。由于模型草案的尺寸小和有限的培训数据，因此基于模型的投机解码经常在室外场景中效率降低。此外，起草阶段的时间成本导致验证步骤中接受度长度的上限最低，从而限制了总体效率。本文提出了RASD（检索提示的投机解码），该解码采用检索方法来增强基于模型的投机解码。我们介绍树木修剪和树融合以实现这一目标。具体来说，我们根据模型草案的概率分布开发一种修剪方法，以构建最佳检索树。其次，我们采用最长的前缀匹配算法将草稿模型生成的树合并与检索树，从而产生了一个统一的树进行验证。实验结果表明，RASD可以在诸如DOCQA，摘要，代码和内域QA等任务之间实现最新的推理加速度。此外，RASD具有强大的可扩展性，与各种投机解码方法无缝集成，包括基于生成的方法和基于基于生成的方法。

Title: Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penalties

• Authors: Eunkyung Choi, Young Jin Suh, Hun Park, Wonseok Hwang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03444
• Pdf URL: https://arxiv.org/pdf/2503.03444
• Copy Paste: [[2503.03444]] Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penalties(https://arxiv.org/abs/2503.03444)
• Keywords: language model, llm, agent
• Abstract: How capable are large language models (LLMs) in the domain of taxation? Although numerous studies have explored the legal domain in general, research dedicated to taxation remain scarce. Moreover, the datasets used in these studies are either simplified, failing to reflect the real-world complexities, or unavailable as open source. To address this gap, we introduce PLAT, a new benchmark designed to assess the ability of LLMs to predict the legitimacy of additional tax penalties. PLAT is constructed to evaluate LLMs' understanding of tax law, particularly in cases where resolving the issue requires more than just applying related statutes. Our experiments with six LLMs reveal that their baseline capabilities are limited, especially when dealing with conflicting issues that demand a comprehensive understanding. However, we found that enabling retrieval, self-reasoning, and discussion among multiple agents with specific role assignments, this limitation can be mitigated.
• 摘要：税收领域的大型语言模型（LLM）的能力如何？尽管许多研究探讨了一般的法律领域，但专门用于税收的研究仍然很少。此外，这些研究中使用的数据集是简化的，无法反映现实世界中的复杂性，或者无法作为开源。为了解决这一差距，我们引入了PLAT，这是一种新的基准测试，旨在评估LLMS预测额外税收合法性的能力。 PLAT的构建是为了评估LLM对税法的理解，特别是在解决问题的情况下，不仅需要应用相关法规。我们对六个LLM的实验表明，它们的基线能力是有限的，尤其是在处理需要全面理解的矛盾问题时。但是，我们发现，具有特定角色分配的多个代理人可以进行检索，自我调查和讨论，可以缓解这种限制。

Title: Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models

• Authors: Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03460
• Pdf URL: https://arxiv.org/pdf/2503.03460
• Copy Paste: [[2503.03460]] Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models(https://arxiv.org/abs/2503.03460)
• Keywords: language model, llm
• Abstract: Fine-tuning LLMs with first-order methods like back-propagation is computationally intensive. Zeroth-Order (ZO) optimisation, using function evaluations instead of gradients, reduces memory usage but suffers from slow convergence in high-dimensional models. As a result, ZO research in LLMs has mostly focused on classification, overlooking more complex generative tasks. In this paper, we introduce ZOPrO, a novel ZO algorithm designed for \textit{Preference Optimisation} in LLMs. We begin by analysing the interplay between policy and reward models during traditional (first-order) Preference Optimisation, uncovering patterns in their relative updates. Guided by these insights, we adapt Simultaneous Perturbation Stochastic Approximation (SPSA) with a targeted sampling strategy to accelerate convergence. Through experiments on summarisation, machine translation, and conversational assistants, we demonstrate that our method consistently enhances reward signals while achieving convergence times comparable to first-order methods. While it falls short of some state-of-the-art methods, our work is the first to apply Zeroth-Order methods to Preference Optimisation in LLMs, going beyond classification tasks and paving the way for a largely unexplored research direction. Code and visualisations are available at this https URL
• 摘要：具有后传播的一阶方法的微调LLM在计算上是强化的。使用功能评估而不是渐变的零阶（ZO）优化可减少内存使用情况，但在高维模型中会遭受缓慢的收敛性。结果，LLMS中的ZO研究主要集中在分类上，忽略了更复杂的生成任务。在本文中，我们介绍了Zopro，这是一种专为LLMS中\ textit {preferperion Optimization}设计的新型ZO算法。我们首先分析传统（一阶）偏好优化期间的策略和奖励模型之间的相互作用，并在其相对更新中发现模式。在这些见解的指导下，我们将同时扰动随机近似（SPSA）具有针对性的采样策略，以加速收敛。通过实验摘要，机器翻译和对话助手，我们证明了我们的方法一致增强奖励信号，同时达到与一阶方法相当的收敛时间。尽管它没有某些最先进的方法，但我们的工作是第一个将零级方法应用于LLM中的偏好优化的工作，而不是分类任务，并为在很大程度上未开发的研究方向铺平了道路。代码和可视化可在此HTTPS URL上获得

Title: Open-Source Large Language Models as Multilingual Crowdworkers: Synthesizing Open-Domain Dialogues in Several Languages With No Examples in Targets and No Machine Translation

• Authors: Ahmed Njifenjou, Virgile Sucal, Bassam Jabaian, Fabrice Lefèvre
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03462
• Pdf URL: https://arxiv.org/pdf/2503.03462
• Copy Paste: [[2503.03462]] Open-Source Large Language Models as Multilingual Crowdworkers: Synthesizing Open-Domain Dialogues in Several Languages With No Examples in Targets and No Machine Translation(https://arxiv.org/abs/2503.03462)
• Keywords: language model, llm, chat, agent
• Abstract: The prevailing paradigm in the domain of Open-Domain Dialogue agents predominantly focuses on the English language, encompassing both models and datasets. Furthermore, the financial and temporal investments required for crowdsourcing such datasets for finetuning are substantial, particularly when multiple languages are involved. Fortunately, advancements in Large Language Models (LLMs) have unveiled a plethora of possibilities across diverse tasks. Specifically, instruction-tuning has enabled LLMs to execute tasks based on natural language instructions, occasionally surpassing the performance of human crowdworkers. Additionally, these models possess the capability to function in various languages within a single thread. Consequently, to generate new samples in different languages, we propose leveraging these capabilities to replicate the data collection process. We introduce a pipeline for generating Open-Domain Dialogue data in multiple Target Languages using LLMs, with demonstrations provided in a unique Source Language. By eschewing explicit Machine Translation in this approach, we enhance the adherence to language-specific nuances. We apply this methodology to the PersonaChat dataset. To enhance the openness of generated dialogues and mimic real life scenarii, we added the notion of speech events corresponding to the type of conversation the speakers are involved in and also that of common ground which represents the premises of a conversation.
• 摘要：开放域对话代理领域的主要范式主要集中在英语上，涵盖了模型和数据集。此外，众包进行此类填充所需的财务和时间投资是实质性的，尤其是在涉及多种语言的情况下。幸运的是，大语言模型（LLM）的进步已经揭示了跨不同任务的大量可能性。具体而言，指令调整使LLM能够根据自然语言指令执行任务，偶尔超过人类人群的表现。此外，这些模型具有单个线程中各种语言的功能。因此，为了生成不同语言的新样本，我们建议利用这些功能复制数据收集过程。我们介绍了使用LLMs以多种目标语言生成开放域对话数据的管道，并以独特的源语言提供了演示。通过在这种方法中避开明确的机器翻译，我们可以增强对语言特定细微差别的依从性。我们将此方法应用于人为数据集。为了增强生成的对话和模仿现实生活中的场景的开放性，我们添加了与演讲者参与的对话类型相对应的语音事件的概念以及代表对话前提的共同基础。

Title: Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues

• Authors: Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03474
• Pdf URL: https://arxiv.org/pdf/2503.03474
• Copy Paste: [[2503.03474]] Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues(https://arxiv.org/abs/2503.03474)
• Keywords: language model
• Abstract: Research in linguistics shows that non-verbal cues, such as gestures, play a crucial role in spoken discourse. For example, speakers perform hand gestures to indicate topic shifts, helping listeners identify transitions in discourse. In this work, we investigate whether the joint modeling of gestures using human motion sequences and language can improve spoken discourse modeling in language models. To integrate gestures into language models, we first encode 3D human motion sequences into discrete gesture tokens using a VQ-VAE. These gesture token embeddings are then aligned with text embeddings through feature alignment, mapping them into the text embedding space. To evaluate the gesture-aligned language model on spoken discourse, we construct text infilling tasks targeting three key discourse cues grounded in linguistic research: discourse connectives, stance markers, and quantifiers. Results show that incorporating gestures enhances marker prediction accuracy across the three tasks, highlighting the complementary information that gestures can offer in modeling spoken discourse. We view this work as an initial step toward leveraging non-verbal cues to advance spoken language modeling in language models.
• 摘要：语言学研究表明，非语言提示（例如手势）在口语话语中起着至关重要的作用。例如，演讲者执行手势来指示主题转移，帮助听众确定话语中的过渡。在这项工作中，我们研究了使用人类运动序列和语言对手势的联合建模是否可以改善语言模型中的口语话语建模。为了将手势整合到语言模型中，我们首先使用VQ-VAE编码3D人体运动序列。然后，将这些手势令牌嵌入与文本嵌入通过特征对齐对齐，将它们映射到文本嵌入空间中。为了评估与语言话语的手势一致的语言模型，我们构建了针对以语言研究为基础的三个关键话语线索的文本填充任务：话语连接，立场标记和量词。结果表明，结合手势可以增强三个任务的标记预测准确性，从而突出了手势在对口语演讲中可以提供的互补信息。我们认为这项工作是利用非语言提示来推进语言模型中口语建模的第一步。

Title: CURVALID: Geometrically-guided Adversarial Prompt Detection

• Authors: Canaan Yung, Hanxun Huang, Sarah Monazam Erfani, Christopher Leckie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03502
• Pdf URL: https://arxiv.org/pdf/2503.03502
• Copy Paste: [[2503.03502]] CURVALID: Geometrically-guided Adversarial Prompt Detection(https://arxiv.org/abs/2503.03502)
• Keywords: language model, llm, prompt
• Abstract: Adversarial prompts capable of jailbreaking large language models (LLMs) and inducing undesirable behaviours pose a significant obstacle to their safe deployment. Current mitigation strategies rely on activating built-in defence mechanisms or fine-tuning the LLMs, but the fundamental distinctions between adversarial and benign prompts are yet to be understood. In this work, we introduce CurvaLID, a novel defense framework that efficiently detects adversarial prompts by leveraging their geometric properties. It is agnostic to the type of LLM, offering a unified detection framework across diverse adversarial prompts and LLM architectures. CurvaLID builds on the geometric analysis of text prompts to uncover their underlying differences. We theoretically extend the concept of curvature via the Whewell equation into an $n$-dimensional word embedding space, enabling us to quantify local geometric properties, including semantic shifts and curvature in the underlying manifolds. Additionally, we employ Local Intrinsic Dimensionality (LID) to capture geometric features of text prompts within adversarial subspaces. Our findings reveal that adversarial prompts differ fundamentally from benign prompts in terms of their geometric characteristics. Our results demonstrate that CurvaLID delivers superior detection and rejection of adversarial queries, paving the way for safer LLM deployment. The source code can be found at this https URL
• 摘要：对抗性提示能够越狱大语言模型（LLM）并引起不良行为构成了安全部署的重大障碍。当前的缓解策略依赖于激活内置的防御机制或微调LLM，但是对抗和良性提示之间的基本区别尚待理解。在这项工作中，我们引入了曲线，这是一个新型的防御框架，该框架通过利用其几何特性来有效地检测对抗性提示。这对LLM的类型不可知，在不同的对抗提示和LLM体系结构中提供了一个统一的检测框架。 Curvalid建立在文本提示的几何分析基础上，以发现其潜在的差异。从理论上讲，我们将曲率的概念扩展到了$ n $维单词嵌入空间中，从而使我们能够量化局部几何特性，包括基础歧管中的语义移动和曲率。此外，我们采用局部内在维度（LID）来捕获对抗子空间内文本提示的几何特征。我们的发现表明，对抗性提示在其几何特征方面与良性提示根本有所不同。我们的结果表明，曲线曲线提供了对抗性查询的卓越检测和拒绝，为更安全的LLM部署铺平了道路。可以在此HTTPS URL上找到源代码

Title: PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention

• Authors: Lida Chen, Dong Xu, Chenxin An, Xintao Wang, Yikai Zhang, Jiangjie Chen, Zujie Liang, Feng Wei, Jiaqing Liang, Yanghua Xiao, Wei Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03588
• Pdf URL: https://arxiv.org/pdf/2503.03588
• Copy Paste: [[2503.03588]] PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention(https://arxiv.org/abs/2503.03588)
• Keywords: language model, llm, long context
• Abstract: Large Language Models (LLMs) face efficiency bottlenecks due to the quadratic complexity of the attention mechanism when processing long contexts. Sparse attention methods offer a promising solution, but existing approaches often suffer from incomplete effective context and/or require complex implementation of pipeline. We present a comprehensive analysis of sparse attention for autoregressive LLMs from the respective of receptive field, recognize the suboptimal nature of existing methods for expanding the receptive field, and introduce PowerAttention, a novel sparse attention design that facilitates effective and complete context extension through the theoretical analysis. PowerAttention achieves exponential receptive field growth in $d$-layer LLMs, allowing each output token to attend to $2^d$ tokens, ensuring completeness and continuity of the receptive field. Experiments demonstrate that PowerAttention outperforms existing static sparse attention methods by $5\sim 40\%$, especially on tasks demanding long-range dependencies like Passkey Retrieval and RULER, while maintaining a comparable time complexity to sliding window attention. Efficiency evaluations further highlight PowerAttention's superior speedup in both prefilling and decoding phases compared with dynamic sparse attentions and full attention ($3.0\times$ faster on 128K context), making it a highly effective and user-friendly solution for processing long sequences in LLMs.
• 摘要：大型语言模型（LLMS）由于注意力机制在处理较长的上下文时的二次复杂性而面临效率瓶颈。稀疏注意方法提供了一种有希望的解决方案，但是现有的方法通常会遭受不完整的有效环境和/或需要复杂的管道实施。我们对各自接受场的自回旋LLM的稀疏注意力进行了全面分析，认识到现有方法的次优质，以扩展接受场，并引入PowerCountiention，这是一种新颖的稀疏注意设计，一种通过理论分析促进有效且完整的上下文扩展。 PowerAtteention在$ d $ layer llms中实现了指数级的接收场增长，使每个输出令牌都可以参加$ 2^d $令牌，从而确保了接受场的完整性和连续性。实验表明，PowerCtateention的表现优于现有的静态稀疏注意方法$ 5 \ sim 40 \％$，尤其是在要求远程依赖的任务上，例如Passkey检索和统治者，同时保持了与窗户关注相当的时间复杂性。效率评估进一步凸显了PowerCateention在预填充和解码阶段的卓越加速，与动态稀疏的注意力和完全关注（$ 3.0 \ times \ times $ 3.0 \ times $在128K上下文上），使其成为在LLMS中处理长序列的非常有效且用户友好的解决方案。

Title: English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance

• Authors: Karl Audun Borgersen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03592
• Pdf URL: https://arxiv.org/pdf/2503.03592
• Copy Paste: [[2503.03592]] English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance(https://arxiv.org/abs/2503.03592)
• Keywords: llm
• Abstract: For consumer usage of locally deployed LLMs, the GGUF format and k_quantization are invaluable tools for maintaining the performance of the original model while reducing it to sizes deployable with consumer-grade hardware. The number of bits dedicated to each weight from the original model is reduced based on how important they are thought to be during model inference. This importance is arrived at through the application of an 'importance matrix'-a relatively small text document meant to be representative of the LLM's standard use-cases. In the vast majority of quants available online, this document is primarily written in English. It was therefore an open question whether performance on English language tasks was preserved through the sacrifice of multilingual performance and whether it can be preserved with alternate importance matrices. This article investigates these hypotheses by quantizing Llama3.3 70B on importance matrices written in three languages (English, Norwegian, and Malayalam) and evaluating them on the MixEval dataset in both English and Norwegian. All experiments related to k_quantization yielded non-significant results (In all cases p > 0.237) indicating that current quantization practices do not disproportionately harm multilingual performance.
• 摘要：对于消费者使用本地部署的LLM，GGGUF格式和K_Quantization是维护原始模型性能的宝贵工具，同时将其降低到可以通过消费级硬件部署的尺寸。根据模型推断期间认为它们的重要性，减少了原始模型中每个重量的位数。通过应用“重要性矩阵” - 一个相对较小的文本文档来实现这一重要性，旨在代表LLM的标准用例。在绝大多数在线可用的Quants中，本文档主要用英语编写。因此，是否通过牺牲多语言表现来保留对英语任务的表现以及是否可以通过替代重要性矩阵保存，这是一个悬而未决的问题。本文通过量化用三种语言（英语，挪威语和马拉雅拉姆语）编写的重要性矩阵来量化Llama3.3 70B来研究这些假设，并在英语和挪威语中对Mixeval数据集进行评估。与K_Quantization相关的所有实验均未取得不重要的结果（在所有情况下p> 0.237），表明当前的量化实践不会损害多语言性能。

Title: Small but Mighty: Enhancing Time Series Forecasting with Lightweight LLMs

• Authors: Haoran Fan, Bin Li, Yixuan Weng, Shoujun Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03594
• Pdf URL: https://arxiv.org/pdf/2503.03594
• Copy Paste: [[2503.03594]] Small but Mighty: Enhancing Time Series Forecasting with Lightweight LLMs(https://arxiv.org/abs/2503.03594)
• Keywords: language model, llm, prompt
• Abstract: While LLMs have demonstrated remarkable potential in time series forecasting, their practical deployment remains constrained by excessive computational demands and memory footprints. Existing LLM-based approaches typically suffer from three critical limitations: Inefficient parameter utilization in handling numerical time series patterns; Modality misalignment between continuous temporal signals and discrete text embeddings; and Inflexibility for real-time expert knowledge integration. We present SMETimes, the first systematic investigation of sub-3B parameter SLMs for efficient and accurate time series forecasting. Our approach centers on three key innovations: A statistically-enhanced prompting mechanism that bridges numerical time series with textual semantics through descriptive statistical features; A adaptive fusion embedding architecture that aligns temporal patterns with language model token spaces through learnable parameters; And a dynamic mixture-of-experts framework enabled by SLMs' computational efficiency, adaptively combining base predictions with domain-specific models. Extensive evaluations across seven benchmark datasets demonstrate that our 3B-parameter SLM achieves state-of-the-art performance on five primary datasets while maintaining 3.8x faster training and 5.2x lower memory consumption compared to 7B-parameter LLM baselines. Notably, the proposed model exhibits better learning capabilities, achieving 12.3% lower MSE than conventional LLM. Ablation studies validate that our statistical prompting and cross-modal fusion modules respectively contribute 15.7% and 18.2% error reduction in long-horizon forecasting tasks. By redefining the efficiency-accuracy trade-off landscape, this work establishes SLMs as viable alternatives to resource-intensive LLMs for practical time series forecasting. Code and models are available at this https URL.
• 摘要：尽管LLM在时间序列预测中表现出巨大的潜力，但其实际部署仍受到过度计算需求和记忆足迹的限制。现有的基于LLM的方法通常遭受三个关键局限性：处理数值时间序列模式的参数利用率低下；连续的时间信号和离散文本嵌入之间的方式错位；和实时专家知识整合的僵化。我们提出Smetimes，这是对低3B参数SLM的首次系统研究，以进行有效，准确的时间序列预测。我们的方法集中在三个关键创新：一种具有统计增强的提示机制，该机制通过描述性统计特征将数字时间序列与文本语义融为一体；一种自适应融合嵌入架构，通过可学习的参数将时间模式与语言模型令牌空间保持一致；以及由SLMS的计算效率启用的Experts框架的动态混合物，将基本预测与域特异性模型自适应地结合在一起。对七个基准数据集进行了广泛的评估表明，与7B参数LLM Baseline相比，我们的3B参数SLM在五个主要数据集中实现了最先进的性能，同时保持3.8倍的训练和5.2倍的记忆消耗。值得注意的是，所提出的模型具有更好的学习能力，比传统LLM低12.3％。消融研究验证了我们的统计提示和跨模式融合模块分别贡献了15.7％和18.2％的长期预测任务误差降低。通过重新定义效率 - 准确性权衡现场，这项工作将SLMS确立为可行的替代品资源密集型LLMS实用时间序列的预测。代码和型号可在此HTTPS URL上找到。

Title: Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders

• Authors: Kristian Kuznetsov, Laida Kushnareva, Polina Druzhinina, Anton Razzhigaev, Anastasia Voznyuk, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03601
• Pdf URL: https://arxiv.org/pdf/2503.03601
• Copy Paste: [[2503.03601]] Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders(https://arxiv.org/abs/2503.03601)
• Keywords: language model, llm, prompt
• Abstract: Artificial Text Detection (ATD) is becoming increasingly important with the rise of advanced Large Language Models (LLMs). Despite numerous efforts, no single algorithm performs consistently well across different types of unseen text or guarantees effective generalization to new LLMs. Interpretability plays a crucial role in achieving this goal. In this study, we enhance ATD interpretability by using Sparse Autoencoders (SAE) to extract features from Gemma-2-2b residual stream. We identify both interpretable and efficient features, analyzing their semantics and relevance through domain- and model-specific statistics, a steering approach, and manual or LLM-based interpretation. Our methods offer valuable insights into how texts from various models differ from human-written content. We show that modern LLMs have a distinct writing style, especially in information-dense domains, even though they can produce human-like outputs with personalized prompts.
• 摘要：随着高级大语言模型（LLM）的兴起，人工文本检测（ATD）变得越来越重要。尽管做出了许多努力，但在不同类型的看不见的文本中，没有单一的算法始终如一地表现出色，或者保证对新LLM的有效概括。解释性在实现这一目标方面起着至关重要的作用。在这项研究中，我们通过使用稀疏的自动编码器（SAE）从GEMMA-2-2B残留流中提取特征来增强ATD的可解释性。我们通过域和特定于模型的统计数据，转向方法以及基于手动或基于LLM的解释来确定可解释和高效的功能，分析其语义和相关性。我们的方法提供了有关各种模型文本与人称内容内容的有价值的见解。我们表明，现代LLM具有独特的写作风格，尤其是在信息密集的域中，即使它们可以通过个性化提示产生类似人类的输出。

Title: Psy-Insight: Explainable Multi-turn Bilingual Dataset for Mental Health Counseling

• Authors: Keqi Chen, Zekai Sun, Yuhua Wen, Huijun Lian, Yingming Gao, Ya Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03607
• Pdf URL: https://arxiv.org/pdf/2503.03607
• Copy Paste: [[2503.03607]] Psy-Insight: Explainable Multi-turn Bilingual Dataset for Mental Health Counseling(https://arxiv.org/abs/2503.03607)
• Keywords: language model, llm
• Abstract: The in-context learning capabilities of large language models (LLMs) show great potential in mental health support. However, the lack of counseling datasets, particularly in Chinese corpora, restricts their application in this field. To address this, we constructed Psy-Insight, the first mental health-oriented explainable multi-task bilingual dataset. We collected face-to-face multi-turn counseling dialogues, which are annotated with multi-task labels and conversation process explanations. Our annotations include psychotherapy, emotion, strategy, and topic labels, as well as turn-level reasoning and session-level guidance. Psy-Insight is not only suitable for tasks such as label recognition but also meets the need for training LLMs to act as empathetic counselors through logical reasoning. Experiments show that training LLMs on Psy-Insight enables the models to not only mimic the conversation style but also understand the underlying strategies and reasoning of counseling.
• 摘要：大语言模型（LLMS）的内在学习能力在心理健康支持中表现出巨大的潜力。但是，缺乏咨询数据集，尤其是在中国语料库中，限制了其在这一领域的应用。为了解决这个问题，我们构建了PSY-Insight，这是第一个以心理健康为导向的可解释的多任务双语数据集。我们收集了面对面的多转向咨询对话，并用多任务标签和对话过程说明进行了注释。我们的注释包括心理治疗，情感，策略和主题标签，以及转向级别的推理和会议级指导。 PSY-Insigh不仅适合诸如标签识别之类的任务，而且还满足培训LLMS通过逻辑推理充当善解人意辅导员的必要性。实验表明，对PSY-Instright进行培训LLM，使模型不仅可以模仿对话风格，而且还可以理解咨询的基本策略和推理。

Title: Psy-Copilot: Visual Chain of Thought for Counseling

• Authors: Keqi Chen, Zekai Sun, Huijun Lian, Yingming Gao, Ya Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03645
• Pdf URL: https://arxiv.org/pdf/2503.03645
• Copy Paste: [[2503.03645]] Psy-Copilot: Visual Chain of Thought for Counseling(https://arxiv.org/abs/2503.03645)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are becoming increasingly popular in the field of psychological counseling. However, when human therapists work with LLMs in therapy sessions, it is hard to understand how the model gives the answers. To address this, we have constructed Psy-COT, a graph designed to visualize the thought processes of LLMs during therapy sessions. The Psy-COT graph presents semi-structured counseling conversations alongside step-by-step annotations that capture the reasoning and insights of therapists. Moreover, we have developed Psy-Copilot, which is a conversational AI assistant designed to assist human psychological therapists in their consultations. It can offer traceable psycho-information based on retrieval, including response candidates, similar dialogue sessions, related strategies, and visual traces of results. We have also built an interactive platform for AI-assisted counseling. It has an interface that displays the relevant parts of the retrieval sub-graph. The Psy-Copilot is designed not to replace psychotherapists but to foster collaboration between AI and human therapists, thereby promoting mental health development. Our code and demo are both open-sourced and available for use.
• 摘要：大型语言模型（LLM）在心理咨询领域变得越来越流行。但是，当人类治疗师在治疗课程中与LLMS合作时，很难理解该模型如何给出答案。为了解决这个问题，我们构建了PSY-COT，该图旨在可视化治疗过程中LLM的思维过程。 PSY-COT图提供了半结构化的咨询对话，并逐步注释捕捉治疗师的推理和见解。此外，我们开发了PSY-Copilot，这是一名对话式AI助理，旨在帮助人类的心理治疗师进行咨询。它可以基于检索提供可追溯的心理信息，包括响应候选者，类似的对话会议，相关策略和结果痕迹。我们还建立了一个交互式平台，用于AI辅助咨询。它具有一个界面，显示检索子图的相关部分。 Psy-Copilot的目的不是替代心理治疗师，而是为了促进AI和人类治疗师之间的合作，从而促进心理健康发展。我们的代码和演示都是开源的，可用于使用。

Title: Token-Level Privacy in Large Language Models

• Authors: Re'em Harel, Niv Gilboa, Yuval Pinter
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.03652
• Pdf URL: https://arxiv.org/pdf/2503.03652
• Copy Paste: [[2503.03652]] Token-Level Privacy in Large Language Models(https://arxiv.org/abs/2503.03652)
• Keywords: language model
• Abstract: The use of language models as remote services requires transmitting private information to external providers, raising significant privacy concerns. This process not only risks exposing sensitive data to untrusted service providers but also leaves it vulnerable to interception by eavesdroppers. Existing privacy-preserving methods for natural language processing (NLP) interactions primarily rely on semantic similarity, overlooking the role of contextual information. In this work, we introduce dchi-stencil, a novel token-level privacy-preserving mechanism that integrates contextual and semantic information while ensuring strong privacy guarantees under the dchi differential privacy framework, achieving 2epsilon-dchi-privacy. By incorporating both semantic and contextual nuances, dchi-stencil achieves a robust balance between privacy and utility. We evaluate dchi-stencil using state-of-the-art language models and diverse datasets, achieving comparable and even better trade-off between utility and privacy compared to existing methods. This work highlights the potential of dchi-stencil to set a new standard for privacy-preserving NLP in modern, high-risk applications.
• 摘要：将语言模型用作远程服务需要将私人信息传输到外部提供商，从而引发了重大的隐私问题。这个过程不仅有可能将敏感数据暴露于不受信任的服务提供商，而且使其容易受到窃听者拦截的攻击。自然语言处理（NLP）相互作用的现有隐私性方法主要依赖于语义相似性，从而忽略了上下文信息的作用。在这项工作中，我们介绍了DCHI模具，这是一种新颖的令牌级隐私机制，该机制将上下文和语义信息整合在一起，同时确保在DCHI差异隐私框架下确保强大的隐私范围，从而实现2epsilon-dchi-privacy。通过同时结合语义和上下文的细微差别，DCHI模具在隐私和实用程序之间取得了强大的平衡。我们使用最先进的语言模型和不同的数据集评估了DCHI模具，与现有方法相比，实用程序和隐私之间的权衡取舍了可比性甚至更好。这项工作突出了DCHI模具在现代高风险应用程序中为隐私保护NLP设定新标准的潜力。

Title: Improving Neutral Point of View Text Generation through Parameter-Efficient Reinforcement Learning and a Small-Scale High-Quality Dataset

• Authors: Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03654
• Pdf URL: https://arxiv.org/pdf/2503.03654
• Copy Paste: [[2503.03654]] Improving Neutral Point of View Text Generation through Parameter-Efficient Reinforcement Learning and a Small-Scale High-Quality Dataset(https://arxiv.org/abs/2503.03654)
• Keywords: language model, llm
• Abstract: This paper describes the construction of a dataset and the evaluation of training methods to improve generative large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e., to provide significantly more informative, diverse and impartial answers. The dataset, the SHQ-NPOV dataset, comprises 300 high-quality, human-written quadruplets: a query on a sensitive topic, an answer, an NPOV rating, and a set of links to source texts elaborating the various points of view. The first key contribution of this paper is a new methodology to create such datasets through iterative rounds of human peer-critique and annotator training, which we release alongside the dataset. The second key contribution is the identification of a highly effective training regime for parameter-efficient reinforcement learning (PE-RL) to improve NPOV generation. We compare and extensively evaluate PE-RL and multiple baselines-including LoRA finetuning (a strong baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\%\rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating good answers from the best answers ($60.25\%\rightarrow 85.21\%$ for presence of supportive details, $68.74\%\rightarrow 91.43\%$ for absence of oversimplification). A qualitative analysis corroborates this. Finally, our evaluation finds no statistical differences between results on topics that appear in the training dataset and those on separated evaluation topics, which provides strong evidence that our approach to training PE-RL exhibits very effective out of topic generalization.
• 摘要：本文介绍了数据集的构建以及培训方法的评估，以提高具有中性观点（NPOV）的敏感主题查询的能力（即），即提供更有意义的信息，多样性和公正的答案。数据集（SHQ-NPOV数据集）包括300个高质量的人为写的四分之一：关于敏感主题的查询，答案，NPOV等级以及一组链接，以详细说明各种视图的源文本。本文的第一个关键贡献是一种新的方法，可以通过迭代的人类同伴危机和注释培训来创建此类数据集，我们将与数据集一起发布。第二个关键贡献是鉴定用于参数有效增强学习（PE-RL）的高效训练制度以改善NPOV的产生。我们比较并广泛评估PE-RL和包括洛拉芬特（Lora Finetunting）（强大的基线），SFT和RLHF的多个基线。与最强的基线相比，PE-RL不仅可以提高NPOV质量（97.06美元\％\ rightarrow 99.08 \％$），而且在语言学家身上的分数要高得多。 91.43 \％$缺少过度简化）。定性分析证实了这一点。最后，我们的评估发现在培训数据集中出现的主题与分开评估主题的主题之间没有统计差异，这提供了有力的证据表明，我们的培训PE-RL方法表现出非常有效的主题概括。

Title: Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction

• Authors: Gustaw Opiełka, Hannes Rosenbusch, Claire E. Stevenson
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03666
• Pdf URL: https://arxiv.org/pdf/2503.03666
• Copy Paste: [[2503.03666]] Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction(https://arxiv.org/abs/2503.03666)
• Keywords: language model, llm
• Abstract: Analogical reasoning relies on conceptual abstractions, but it is unclear whether Large Language Models (LLMs) harbor such internal representations. We explore distilled representations from LLM activations and find that function vectors (FVs; Todd et al., 2024) - compact representations for in-context learning (ICL) tasks - are not invariant to simple input changes (e.g., open-ended vs. multiple-choice), suggesting they capture more than pure concepts. Using representational similarity analysis (RSA), we localize a small set of attention heads that encode invariant concept vectors (CVs) for verbal concepts like "antonym". These CVs function as feature detectors that operate independently of the final output - meaning that a model may form a correct internal representation yet still produce an incorrect output. Furthermore, CVs can be used to causally guide model behaviour. However, for more abstract concepts like "previous" and "next", we do not observe invariant linear representations, a finding we link to generalizability issues LLMs display within these domains.
• 摘要：类比推理取决于概念上的抽象，但尚不清楚大语言模型（LLMS）是否具有这种内部表示形式。我们从LLM激活中探索了蒸馏的表示，发现功能向量（FVS； Todd等，2024） - 对内部上下文学习（ICL）任务的紧凑表示 - 并不是简单输入变化（例如，开放式和多个选择）的不变性，这表明它们比纯粹的概念捕获更多的概念。使用代表性相似性分析（RSA），我们将一组少量的注意力头组定位，这些关注头编码不变的概念向量（CVS），以诸如“反义词”之类的口头概念。这些CVS充当特征检测器，独立于最终输出工作 - 这意味着模型可以形成正确的内部表示形式，但仍会产生不正确的输出。此外，CV可用于因果指导模型行为。但是，对于更抽象的概念，例如“上一个”和“ Next”，我们不观察到不变的线性表示，发现我们链接到这些域中LLMS显示的通用性问题。

Title: Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Models

• Authors: Bar Karov, Dor Zohar, Yam Marcovitz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03669
• Pdf URL: https://arxiv.org/pdf/2503.03669
• Copy Paste: [[2503.03669]] Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Models(https://arxiv.org/abs/2503.03669)
• Keywords: language model, llm, hallucination, chain-of-thought, agent
• Abstract: We present Attentive Reasoning Queries (ARQs), a novel structured reasoning approach that significantly improves instruction-following in Large Language Models through domain-specialized reasoning blueprints. While LLMs demonstrate remarkable capabilities across diverse tasks, they often fail to maintain adherence to complex, use-case-specific instructions during multi-turn conversations, presenting challenges for business-critical applications. ARQs address this limitation by guiding LLMs through systematic reasoning steps with targeted queries that reinstate critical instructions and facilitate intermediate reasoning throughout the completion process. In extensive testing within Parlant, our framework for reliable customer-facing agents in which ARQs were born out of necessity, they achieved a 90.2% success rate across 87 test scenarios, outperforming both Chain-of-Thought reasoning (86.1%) and direct response generation (81.5%). ARQs showed particular strength in addressing persistent failure modes like guideline re-application and hallucination prevention. Our analysis also revealed that ARQs can potentially be more computationally efficient than free-form reasoning when carefully designed. These findings demonstrate that structured reasoning approaches provide effective mechanisms for controlling how LLMs process information and make decisions in complex scenarios.
• 摘要：我们提出了细心的推理查询（ARQ），这是一种新型的结构化推理方法，可通过域特有的推理蓝图显着改善大语言模型中的指导跟踪。尽管LLMS在各种任务中都表现出了出色的功能，但它们通常无法在多转交谈中遵守复杂的，用例的指令，从而对业务至关重要的应用提出了挑战。 ARQ通过引导LLM通过有针对性查询的系统推理步骤来解决此限制，从而恢复关键指示并促进整个完成过程中的中间推理。在Parlant内的广泛测试中，我们针对ARQ出生的可靠客户面向客户的框架，他们在87个测试方案中取得了90.2％的成功率，表现优于基础推理（86.1％）和直接响应的链接（81.5％）。 ARQ在解决持续故障模式（例如重新施加和预防幻觉）方面表现出了特殊的力量。我们的分析还表明，在精心设计时，ARQ可能比自由形式推理更有效地计算效率。这些发现表明，结构化推理方法为控制LLM的处理信息和在复杂方案中做出决策提供了有效的机制。

Title: MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems

• Authors: Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, Jing Shao
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2503.03686
• Pdf URL: https://arxiv.org/pdf/2503.03686
• Copy Paste: [[2503.03686]] MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems(https://arxiv.org/abs/2503.03686)
• Keywords: gpt, llm, agent
• Abstract: LLM-based multi-agent systems (MAS) have shown significant potential in tackling diverse tasks. However, to design effective MAS, existing approaches heavily rely on manual configurations or multiple calls of advanced LLMs, resulting in inadaptability and high inference costs. In this paper, we simplify the process of building an MAS by reframing it as a generative language task, where the input is a user query and the output is a corresponding MAS. To address this novel task, we unify the representation of MAS as executable code and propose a consistency-oriented data construction pipeline to create a high-quality dataset comprising coherent and consistent query-MAS pairs. Using this dataset, we train MAS-GPT, an open-source medium-sized LLM that is capable of generating query-adaptive MAS within a single LLM inference. The generated MAS can be seamlessly applied to process user queries and deliver high-quality responses. Extensive experiments on 9 benchmarks and 5 LLMs show that the proposed MAS-GPT consistently outperforms 10+ baseline MAS methods on diverse settings, indicating MAS-GPT's high effectiveness, efficiency and strong generalization ability. Code will be available at this https URL.
• 摘要：基于LLM的多机构系统（MAS）在解决各种任务方面具有巨大潜力。但是，为了设计有效的MAS，现有的方法在很大程度上依赖于手动配置或高级LLM的多个调用，从而导致不适当和高推理成本。在本文中，我们简化了通过将MAS重新标记为生成语言任务的过程，其中输入是用户查询，而输出是相应的MAS。为了解决这一新颖的任务，我们将MAS的表示形式统一为可执行的代码，并提出了面向一致性的数据构建管道，以创建一个包括连贯且一致的查询MAS对的高质量数据集。使用此数据集，我们训练MAS-GPT，这是一种开源中型LLM，能够在单个LLM推理中生成查询自适应MAS。生成的MAS可以无缝应用于处理用户查询并提供高质量的响应。在9个基准和5个LLMS上进行的广泛实验表明，所提出的MAS-GPT始终在不同的环境上优于10+基线MAS方法，这表明MAS-GPT的高效率，效率和强大的通用能力。代码将在此HTTPS URL上可用。

Title: Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models

• Authors: Jiyue Jiang, Alfred Kar Yin Truong, Yanyu Chen, Qinghang Bao, Sheng Wang, Pengan Chen, Jiuming Wang, Lingpeng Kong, Yu Li, Chuan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03702
• Pdf URL: https://arxiv.org/pdf/2503.03702
• Copy Paste: [[2503.03702]] Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models(https://arxiv.org/abs/2503.03702)
• Keywords: language model, llm
• Abstract: High-quality data resources play a crucial role in learning large language models (LLMs), particularly for low-resource languages like Cantonese. Despite having more than 85 million native speakers, Cantonese is still considered a low-resource language in the field of natural language processing (NLP) due to factors such as the dominance of Mandarin, lack of cohesion within the Cantonese-speaking community, diversity in character encoding and input methods, and the tendency of overseas Cantonese speakers to prefer using English. In addition, rich colloquial vocabulary of Cantonese, English loanwords, and code-switching characteristics add to the complexity of corpus collection and processing. To address these challenges, we collect Cantonese texts from a variety of sources, including open source corpora, Hong Kong-specific forums, Wikipedia, and Common Crawl data. We conduct rigorous data processing through language filtering, quality filtering, content filtering, and de-duplication steps, successfully constructing a high-quality Cantonese corpus of over 2 billion tokens for training large language models. We further refined the model through supervised fine-tuning (SFT) on curated Cantonese tasks, enhancing its ability to handle specific applications. Upon completion of the training, the model achieves state-of-the-art (SOTA) performance on four Cantonese benchmarks. After training on our dataset, the model also exhibits improved performance on other mainstream language tasks.
• 摘要：高质量的数据资源在学习大语言模型（LLMS）中起着至关重要的作用，特别是对于像广东话这样的低资源语言。尽管有超过8500万母语的人，但由于因素，诸如普通话的优势，讲广东话社区内的凝聚力缺乏凝聚力，角色编码和输入方法的多样性以及海外广州扬声器的趋势以优先使用英语的趋势。此外，粤语，英语借用词和代码转换特征的丰富口语词汇增加了语料库收集和处理的复杂性。为了应对这些挑战，我们从各种来源收集广东话文本，包括开源语料库，香港特定论坛，维基百科和常见的爬网数据。我们通过语言过滤，质量过滤，内容过滤和删除步骤进行严格的数据处理，并成功地构建了超过20亿个代币的高质量的粤语语料库来培训大语言模型。我们通过对策划的粤语任务进行监督的微调（SFT）进一步完善了该模型，从而增强了其处理特定应用程序的能力。培训完成后，该模型在四个粤语基准上实现了最先进的表现（SOTA）。在我们的数据集进行了培训之后，该模型在其他主流语言任务上还表现出改善的性能。

Title: Effective LLM Knowledge Learning via Model Generalization

• Authors: Mingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu, Hengshuang Zhao, Jiaya Jia
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03705
• Pdf URL: https://arxiv.org/pdf/2503.03705
• Copy Paste: [[2503.03705]] Effective LLM Knowledge Learning via Model Generalization(https://arxiv.org/abs/2503.03705)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are trained on enormous documents that contain extensive world knowledge. However, it is still not well-understood how knowledge is acquired via autoregressive pre-training. This lack of understanding greatly hinders effective knowledge learning, especially for continued pretraining on up-to-date information, as this evolving information often lacks diverse repetitions like foundational knowledge. In this paper, we focus on understanding and improving LLM knowledge learning. We found and verified that knowledge learning for LLMs can be deemed as an implicit supervised task hidden in the autoregressive pre-training objective. Our findings suggest that knowledge learning for LLMs would benefit from methods designed to improve generalization ability for supervised tasks. Based on our analysis, we propose the formatting-based data augmentation to grow in-distribution samples, which does not present the risk of altering the facts embedded in documents as text paraphrasing. We also introduce sharpness-aware minimization as an effective optimization algorithm to better improve generalization. Moreover, our analysis and method can be readily extended to instruction tuning. Extensive experiment results validate our findings and demonstrate our methods' effectiveness in both continued pre-training and instruction tuning. This paper offers new perspectives and insights to interpret and design effective strategies for LLM knowledge learning.
• 摘要：大型语言模型（LLMS）经过培训，该文档包含广泛的世界知识。但是，仍然不太了解如何通过自回归预培训获得知识。缺乏理解极大地阻碍了有效的知识学习，尤其是对于继续预测的最新信息，因为这些不断发展的信息通常缺乏多种重复，例如基础知识。在本文中，我们专注于理解和改善LLM知识学习。我们发现并验证了LLM的知识学习可以被视为隐藏在自回归前训练目标中的隐性监督任务。我们的发现表明，LLM的知识学习将从旨在提高监督任务的概括能力的方法中受益。基于我们的分析，我们提出了基于格式的数据增强来生长分布样本，这并不具有改变文档中嵌入文档中嵌入的事实作为文本释义的风险。我们还将清晰度感知的最小化作为有效的优化算法，以更好地改善概括。此外，我们的分析和方法很容易扩展到指令调整。广泛的实验结果验证了我们的发现，并证明了我们的方法在持续的预训练和指导调整中的有效性。本文提供了解释和设计LLM知识学习有效策略的新观点和见解。

Title: Improving LLM Safety Alignment with Dual-Objective Optimization

• Authors: Xuandong Zhao, Will Cai, Tianneng Shi, David Huang, Licong Lin, Song Mei, Dawn Song
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03710
• Pdf URL: https://arxiv.org/pdf/2503.03710
• Copy Paste: [[2503.03710]] Improving LLM Safety Alignment with Dual-Objective Optimization(https://arxiv.org/abs/2503.03710)
• Keywords: language model, llm
• Abstract: Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization (DPO), a widely deployed alignment method, exhibits limitations in both experimental and theoretical contexts as its loss function proves suboptimal for refusal learning. Through gradient-based analysis, we identify these shortcomings and propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge. This approach significantly increases LLM robustness against a wide range of jailbreak attacks, including prefilling, suffix, and multi-turn attacks across both in-distribution and out-of-distribution scenarios. Furthermore, we introduce a method to emphasize critical refusal tokens by incorporating a reward-based token-level weighting mechanism for refusal learning, which further improves the robustness against adversarial exploits. Our research also suggests that robustness to jailbreak attacks is correlated with token distribution shifts in the training process and internal representations of refusal and harmful tokens, offering valuable directions for future research in LLM safety alignment. The code is available at this https URL
• 摘要：大型语言模型（LLM）的现有培训时间安全对准技术仍然容易受到越狱攻击的影响。直接偏好优化（DPO）是一种广泛部署的对齐方法，在实验和理论环境中都表现出局限性，因为其损失函数证明是拒绝学习的次优。通过基于梯度的分析，我们确定了这些缺点，并提出了改进的安全对准，将DPO目标分解为两个组成部分：（1）强大的拒绝训练，即使产生了部分不安全的世代，也鼓励拒绝拒绝，（2）有针对性的有害知识的靶向不学习。这种方法可大大提高LLM的鲁棒性，并在各种越狱攻击中，包括预填充，后缀和在分发和分发场景中的多转攻击。此外，我们引入了一种方法来强调关键的拒绝令牌，通过结合拒绝学习的基于奖励的令牌加权机制，这进一步提高了针对对抗性利用的鲁棒性。我们的研究还表明，越狱攻击的鲁棒性与训练过程中的令牌分配变化以及拒绝和有害令牌的内部表示相关，为LLM安全一致性方面的未来研究提供了宝贵的方向。该代码可在此HTTPS URL上找到

Title: Process-based Self-Rewarding Language Models

• Authors: Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, Yeyun Gong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03746
• Pdf URL: https://arxiv.org/pdf/2503.03746
• Copy Paste: [[2503.03746]] Process-based Self-Rewarding Language Models(https://arxiv.org/abs/2503.03746)
• Keywords: language model, llm
• Abstract: Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.
• 摘要：大型语言模型已经在各种下游任务中表现出出色的性能，并且已在多种情况下广泛应用。人类注销的偏好数据用于培训，以进一步提高LLMS的性能，这受到人类绩效上限的约束。因此，已经提出了自我奖励方法，其中LLM通过奖励自己的产出来生成培训数据。但是，现有的自我奖励范式在数学推理方案中无效，甚至可能导致性能下降。 In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm.我们的新范式通过基于迭代过程的自我奖励成功地增强了LLM在多个数学推理基准上的性能，这表明了自我奖励的巨大潜力以实现可能超过人类能力的LLM推理。