2025-02-14

Title: Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions

• Authors: Jingxin Xu, Guoshun Nan, Sheng Guan, Sicong Leng, Yilian Liu, Zixiao Wang, Yuyang Ma, Zhili Zhou, Yanzhao Hou, Xiaofeng Tao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08657
• Pdf URL: https://arxiv.org/pdf/2502.08657
• Copy Paste: [[2502.08657]] Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions(https://arxiv.org/abs/2502.08657)
• Keywords: language model, gpt, llm, chat, agent
• Abstract: Recent AI agents, such as ChatGPT and LLaMA, primarily rely on instruction tuning and reinforcement learning to calibrate the output of large language models (LLMs) with human intentions, ensuring the outputs are harmless and helpful. Existing methods heavily depend on the manual annotation of high-quality positive samples, while contending with issues such as noisy labels and minimal distinctions between preferred and dispreferred response data. However, readily available toxic samples with clear safety distinctions are often filtered out, removing valuable negative references that could aid LLMs in safety alignment. In response, we propose PT-ALIGN, a novel safety self-alignment approach that minimizes human supervision by automatically refining positive and toxic samples and performing fine-grained dual instruction tuning. Positive samples are harmless responses, while toxic samples deliberately contain extremely harmful content, serving as a new supervisory signals. Specifically, we utilize LLM itself to iteratively generate and refine training instances by only exploring fewer than 50 human annotations. We then employ two losses, i.e., maximum likelihood estimation (MLE) and fine-grained unlikelihood training (UT), to jointly learn to enhance the LLM's safety. The MLE loss encourages an LLM to maximize the generation of harmless content based on positive samples. Conversely, the fine-grained UT loss guides the LLM to minimize the output of harmful words based on negative samples at the token-level, thereby guiding the model to decouple safety from effectiveness, directing it toward safer fine-tuning objectives, and increasing the likelihood of generating helpful and reliable content. Experiments on 9 popular open-source LLMs demonstrate the effectiveness of our PT-ALIGN for safety alignment, while maintaining comparable levels of helpfulness and usefulness.
• 摘要：最近的AI代理（例如Chatgpt和Llama）主要依靠指令调整和强化学习，以人类意图校准大语言模型（LLMS）的产出，从而确保输出无害且有用。现有方法在很大程度上取决于高质量阳性样本的手动注释，同时与噪声标签和优先响应数据和分配响应数据之间的最小区分之类的问题竞争。但是，通常会滤除具有明确安全区分的有毒样品，从而消除了有助于LLMS安全对齐的有价值的负面参考。作为回应，我们提出了PT-Align，这是一种新型的安全自我对准方法，可以通过自动提炼正阳性和有毒样品并进行细粒的双重指导调整来最大程度地减少人类的监督。积极样本是无害的反应，而有毒样品故意包含非常有害的内容，作为新的监督信号。具体而言，我们仅通过探索少于50个人的注释来利用LLM本身来迭代生成和完善培训实例。然后，我们采取两种损失，即最大似然估计（MLE）和细粒度的不可能训练（UT），共同学会增强LLM的安全性。 MLE损失鼓励LLM根据阳性样本最大化无害含量的产生。相反，细颗粒的UT损失指导LLM最大程度地减少基于令牌级别的负样本的有害单词的输出产生有用和可靠的内容的可能性。在9个流行的开源LLMS上进行的实验证明了我们的PT-Align在安全对准方面的有效性，同时保持了可比的帮助和实用性水平。

Title: Semantic Role Labeling: A Systematical Survey

• Authors: Huiyao Chen, Meishan Zhang, Jing Li, Min Zhang, Lilja Øvrelid, Jan Hajič, Hao Fei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08660
• Pdf URL: https://arxiv.org/pdf/2502.08660
• Copy Paste: [[2502.08660]] Semantic Role Labeling: A Systematical Survey(https://arxiv.org/abs/2502.08660)
• Keywords: language model, llm
• Abstract: Semantic role labeling (SRL) is a central natural language processing (NLP) task aiming to understand the semantic roles within texts, facilitating a wide range of downstream applications. While SRL has garnered extensive and enduring research, there is currently a lack of a comprehensive survey that thoroughly organizes and synthesizes the field. This paper aims to review the entire research trajectory of the SRL community over the past two decades. We begin by providing a complete definition of SRL. To offer a comprehensive taxonomy, we categorize SRL methodologies into four key perspectives: model architectures, syntax feature modeling, application scenarios, and multi-modal extensions. Further, we discuss SRL benchmarks, evaluation metrics, and paradigm modeling approaches, while also exploring practical applications across various domains. Finally, we analyze future research directions in SRL, addressing the evolving role of SRL in the age of large language models (LLMs) and its potential impact on the broader NLP landscape. We maintain a public repository and consistently update related resources at: this https URL
• 摘要：语义角色标签（SRL）是一种中央自然语言处理（NLP）任务，旨在了解文本中的语义角色，从而促进了广泛的下游应用程序。尽管SRL获得了广泛而持久的研究，但目前缺乏全面的调查，可以彻底组织和综合该领域。本文旨在回顾过去二十年来SRL社区的整个研究轨迹。我们首先提供SRL的完整定义。为了提供全面的分类法，我们将SRL方法论分为四个关键角度：模型架构，语法功能建模，应用程序方案和多模式扩展。此外，我们讨论了SRL基准，评估指标和范式建模方法，同时还探索了各个领域的实际应用。最后，我们分析了SRL的未来研究方向，以解决SRL在大语言模型（LLMS）（LLMS）及其对更广泛的NLP景观的潜在影响的不断发展的作用。我们维护一个公共存储库，并始终如一地更新相关资源：此HTTPS URL

Title: Few-shot_LLM_Synthetic_Data_with_Distribution_Matching

• Authors: Jiyuan Ren, Zhaocheng Du, Zhihao Wen, Qinglin Jia, Sunhao Dai, Chuhan Wu, Zhenhua Dong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08661
• Pdf URL: https://arxiv.org/pdf/2502.08661
• Copy Paste: [[2502.08661]] Few-shot_LLM_Synthetic_Data_with_Distribution_Matching(https://arxiv.org/abs/2502.08661)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) advance, their ability to perform in-context learning and few-shot language generation has improved significantly. This has spurred using LLMs to produce high-quality synthetic data to enhance the performance of smaller models like online retrievers or weak LLMs. However, LLM-generated synthetic data often differs from the real data in key language attributes (e.g., styles, tones, content proportions, etc.). As a result, mixing these synthetic data directly with real data may distort the original data distribution, potentially hindering performance improvements. To solve this, we introduce SynAlign: a synthetic data generation and filtering framework based on key attribute distribution matching. Before generation, SynAlign employs an uncertainty tracker surrogated by the Gaussian Process model to iteratively select data clusters distinct from selected ones as demonstrations for new data synthesis, facilitating the efficient exploration diversity of the real data. Then, a latent attribute reasoning method is employed: the LLM summarizes linguistic attributes of demonstrations and then synthesizes new data based on them. This approach facilitates synthesizing diverse data with linguistic attributes that appear in real this http URL generation, the Maximum Mean Discrepancy is used as the objective function to learn the sampling weight of each synthetic data, ensuring distribution matching with the real data. Our experiments on multiple text prediction tasks show significant performance improvements. We also conducted an online A/B test on an online retriever to demonstrate SynAlign's effectiveness.
• 摘要：随着大型语言模型（LLM）的发展，他们执行秘密学习和几乎没有语言的语言的能力得到了显着提高。这刺激了使用LLMS生成高质量的合成数据，以增强较小模型（如在线检索器或弱LLMS）的性能。但是，LLM生成的合成数据通常与关键语言属性的真实数据（例如样式，音调，内容比例等）有所不同。结果，将这些综合数据与实际数据直接混合可能会扭曲原始数据分布，从而可能阻碍性能改善。为了解决这个问题，我们介绍了Synalign：基于关键属性分布匹配的合成数据生成和过滤框架。在生成之前，Synalign采用高斯流程模型替代的不确定性跟踪器，以迭代选择与所选数据的数据簇作为新数据合成的演示，从而促进了真实数据的有效勘探多样性。然后，采用了一种潜在属性推理方法：LLM总结了演示的语言属性，然后根据它们合成新数据。这种方法促进了与真实的HTTP URL生成中出现的语言属性合成的多种数据，将最大平均差异用作学习每个合成数据的采样权重的目标函数，从而确保分布与真实数据匹配。我们对多个文本预测任务的实验显示了重大的性能改进。我们还对在线检索器进行了在线A/B测试，以证明Synalign的有效性。

Title: RoToR: Towards More Reliable Responses for Order-Invariant Inputs

• Authors: Soyoung Yoon, Dongha Ahn, Youngwon Lee, Minkyu Jung, HyungJoo Jang, Seung-won Hwang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08662
• Pdf URL: https://arxiv.org/pdf/2502.08662
• Copy Paste: [[2502.08662]] RoToR: Towards More Reliable Responses for Order-Invariant Inputs(https://arxiv.org/abs/2502.08662)
• Keywords: language model
• Abstract: Mitigating positional bias of language models (LMs) for listwise inputs is a well-known and important problem (e.g., lost-in-the-middle). While zero-shot order-invariant LMs have been proposed to solve this issue, their success on practical listwise problems has been limited. In this work, as a first contribution, we identify and overcome two limitations to make zero-shot invariant LMs more practical: (1) training and inference distribution mismatch arising from modifying positional ID assignments to enforce invariance, and (2) failure to adapt to a mixture of order-invariant and sensitive inputs in practical listwise problems. To overcome, we propose (1) RoToR, a zero-shot invariant LM for genuinely order-invariant inputs with minimal modifications of positional IDs, and (2) Selective Routing, an adaptive framework that handles both order-invariant and order-sensitive inputs in listwise tasks. On the Lost in the middle (LitM), Knowledge Graph Question Answering (KGQA), and MMLU benchmarks, we show that RoToR with Selective Routing can effectively handle practical listwise input tasks in a zero-shot manner.
• 摘要：缓解语言模型（LMS）的位置偏差是列表输入是一个众所周知且重要的问题（例如，中间损失）。虽然已经提出了零射击订单不变的LMS来解决此问题，但它们在实际列表问题上的成功受到了限制。在这项工作中，作为第一个贡献，我们确定并克服了两个限制，使零射击不变的LMS更加实用：（1）培训和推理分布不匹配，而不是修改位置ID分配以执行不变性，以及（2）未能适应在实际列表问题中，有序不变和敏感输入的混合物。为了克服，我们提出了（1）转子，用于真正的订单不变的输入的零射击不变的LM，具有最小的位置ID修改，以及（2）选择性路由，一种自适应框架，一种处理订单不变和订单敏感性输入的自适应框架在ListWise任务中。在中间（LITM）的丢失，知识图答录（KGQA）和MMLU基准测试中，我们表明具有选择性路由的转子可以有效地以零拍的方式处理实际列表输入任务。

Title: Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis

• Authors: Emanuele Ricco, Lorenzo Cima, Roberto Di Pietro
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.08663
• Pdf URL: https://arxiv.org/pdf/2502.08663
• Copy Paste: [[2502.08663]] Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis(https://arxiv.org/abs/2502.08663)
• Keywords: llm, hallucination
• Abstract: Hallucinations are one of the major issues affecting LLMs, hindering their wide adoption in production systems. While current research solutions for detecting hallucinations are mainly based on heuristics, in this paper we introduce a mathematically sound methodology to reason about hallucination, and leverage it to build a tool to detect hallucinations. To the best of our knowledge, we are the first to show that hallucinated content has structural differences with respect to correct content. To prove this result, we resort to the Minkowski distances in the embedding space. Our findings demonstrate statistically significant differences in the embedding distance distributions, that are also scale free -- they qualitatively hold regardless of the distance norm used and the number of keywords, questions, or responses. We leverage these structural differences to develop a tool to detect hallucinated responses, achieving an accuracy of 66\% for a specific configuration of system parameters -- comparable with the best results in the field. In conclusion, the suggested methodology is promising and novel, possibly paving the way for further research in the domain, also along the directions highlighted in our future work.
• 摘要：幻觉是影响LLM的主要问题之一，阻碍了其在生产系统中的广泛采用。尽管目前用于检测幻觉的研究解决方案主要基于启发式方法，但在本文中，我们引入了一种数学上合理的方法来理解幻觉，并利用它来构建一种检测幻觉的工具。据我们所知，我们是第一个表明幻觉内容在正确内容方面存在结构性差异。为了证明这一结果，我们求助于嵌入空间中的Minkowski距离。我们的发现表明，嵌入距离分布的统计学显着差异，这些分布也是免费的 - 无论使用的距离规范和关键字，问题或响应的数量如何，它们都会定性地保持。我们利用这些结构差异来开发一种工具来检测幻觉响应，对于系统参数的特定配置，精度达到66 \％ - 与现场最佳结果相当。总之，建议的方法是有希望的和新颖的，可能为在域中进一步研究铺平了道路，也沿着我们未来的工作中强调的方向铺平了道路。

Title: Hallucination, Monofacts, and Miscalibration: An Empirical Investigation

• Authors: Muqing Miao, Michael Kearns
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08666
• Pdf URL: https://arxiv.org/pdf/2502.08666
• Copy Paste: [[2502.08666]] Hallucination, Monofacts, and Miscalibration: An Empirical Investigation(https://arxiv.org/abs/2502.08666)
• Keywords: llm, hallucination
• Abstract: Recent theoretical work by [Kalai and Vempala 2024] proves that a particular notion of hallucination rate in LLMs must be lower bounded by the training data monofact rate (related to the classical Good-Turing missing mass estimator) minus model miscalibration. Through systematic experiments with n-gram models and in-context learning with LLMs, we empirically investigate and validate this theory by examining how different underlying data distributions affect the monofact rate and a model's tendency to hallucinate. We then vary model miscalibration through controlled upweighting of training samples while holding monofact rates constant, allowing us to isolate miscalibration's reduction effect on hallucination. These findings suggest that both the distribution of fact frequencies in training data and the calibration-hallucination trade-off are inherent to probabilistic language generation. Our results also suggest that current practices of aggressive deduplication in training data may need to be reconsidered, as selective duplication could serve as a principled mechanism for reducing hallucination.
• 摘要：[Kalai和Vempala 2024]的最新理论工作证明，LLMS中特殊的幻觉率的特定概念必须由训练数据单声道速率（与经典的良好且缺失的质量估计器有关）减少，减去模型错误。通过使用N-Gram模型进行系统的实验和使用LLM的文化学习，我们通过检查不同的基础数据分布如何影响单声道速率以及模型幻觉的趋势，从经验研究和验证该理论。然后，我们通过控制训练样品的上升量，同时保持单稳定速率恒定，从而改变模型错误校准，从而使我们能够隔离误解对幻觉的减少效果。这些发现表明，训练数据中事实频率的分布和校准 - 呼吸折衷是概率产生的固有的。我们的结果还表明，可能需要重新考虑目前的培训数据中积极重复数据删除的实践，因为选择性重复可以用作减少幻觉的原则机制。

Title: Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models

• Authors: Tabinda Sarwar, Antonio Jose Jimeno Yepes, Lawrence Cavedon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08669
• Pdf URL: https://arxiv.org/pdf/2502.08669
• Copy Paste: [[2502.08669]] Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models(https://arxiv.org/abs/2502.08669)
• Keywords: language model, llm
• Abstract: Background: Data collected in controlled settings typically results in high-quality datasets. However, in real-world applications, the quality of data collection is often compromised. It is well established that the quality of a dataset significantly impacts the performance of machine learning models. Methods: A rudimentary error rate metric was developed to evaluate textual dataset quality at the token level. Mixtral Large Language Model (LLM) was used to quantify and correct errors in low quality datasets. The study analyzed two healthcare datasets: the high-quality MIMIC-III public hospital dataset and a lower-quality private dataset from Australian aged care homes. Errors were systematically introduced into MIMIC at varying rates, while the ACH dataset quality was improved using the LLM. Results: For the sampled 35,774 and 6,336 patients from the MIMIC and ACH datasets respectively, we used Mixtral to introduce errors in MIMIC and correct errors in ACH. Mixtral correctly detected errors in 63% of progress notes, with 17% containing a single token misclassified due to medical terminology. LLMs demonstrated potential for improving progress note quality by addressing various errors. Under varying error rates, feature representation performance was tolerant to lower error rates (<10%) but declined significantly at higher rates. Conclusions: The study revealed that models performed relatively well on datasets with lower error rates (<10%), but their performance declined significantly as error rates increased (>=10%). Therefore, it is crucial to evaluate the quality of a dataset before utilizing it for machine learning tasks. For datasets with higher error rates, implementing corrective measures is essential to ensure the reliability and effectiveness of machine learning models.
• 摘要：背景：在受控设置中收集的数据通常会导致高质量数据集。但是，在实际应用程序中，数据收集的质量通常会受到损害。众所周知，数据集的质量显着影响机器学习模型的性能。方法：开发了基本错误率指标，以评估令牌级别的文本数据集质量。混音大语言模型（LLM）用于量化和纠正低质量数据集中的错误。该研究分析了两个医疗保健数据集：高质量的模仿III公立医院数据集和澳大利亚老年护理院的较低质量的私人数据集。系统地以不同的速率将误差系统地引入模拟物中，而使用LLM则提高了ACH数据集质量。结果：对于来自模拟物和ACH数据集的35,774和6,336例采样的患者，我们使用Mixtral来引入ACH中的模拟和正确错误中的错误。混音在63％的进度票据中正确检测到了错误，其中17％包含由于医学术语而导致的单个令牌错误分类。 LLMS通过解决各种错误来提高进度注释质量的潜力。在不同的错误率下，特征表示性能耐受性能较低（<10％），但在较高速率下大大下降。结论：研究表明，模型在错误率较低的数据集上的性能相对较好（<10％），但随着错误率的增加，它们的性能大大下降（> = 10％）。因此，在将数据集用于机器学习任务之前评估数据集的质量至关重要。对于具有较高错误率的数据集，实施纠正措施对于确保机器学习模型的可靠性和有效性至关重要。

Title: Data Augmentation to Improve Large Language Models in Food Hazard and Product Detection

• Authors: Areeg Fahad Rasheed, M. Zarkoosh, Shimam Amer Chasib, Safa F. Abbas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08687
• Pdf URL: https://arxiv.org/pdf/2502.08687
• Copy Paste: [[2502.08687]] Data Augmentation to Improve Large Language Models in Food Hazard and Product Detection(https://arxiv.org/abs/2502.08687)
• Keywords: language model, gpt, chat
• Abstract: The primary objective of this study is to demonstrate the impact of data augmentation using ChatGPT-4o-mini on food hazard and product analysis. The augmented data is generated using ChatGPT-4o-mini and subsequently used to train two large language models: RoBERTa-base and Flan-T5-base. The models are evaluated on test sets. The results indicate that using augmented data helped improve model performance across key metrics, including recall, F1 score, precision, and accuracy, compared to using only the provided dataset. The full code, including model training and the augmented dataset, can be found in this repository: this https URL
• 摘要：这项研究的主要目的是证明使用Chatgpt-4O-Mini对食物危害和产品分析的数据扩展的影响。增强数据是使用Chatgpt-4O-Mini生成的，随后用于训练两个大型语言模型：Roberta-Base和Flan-T5-Base。在测试集上评估模型。结果表明，与仅使用提供的数据集相比，使用增强数据有助于提高关键指标的模型性能，包括召回，F1分数，精度和准确性。可以在此存储库中找到完整的代码，包括模型培训和增强数据集：此HTTPS URL

Title: IHEval: Evaluating Language Models on Following the Instruction Hierarchy

• Authors: Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08745
• Pdf URL: https://arxiv.org/pdf/2502.08745
• Copy Paste: [[2502.08745]] IHEval: Evaluating Language Models on Following the Instruction Hierarchy(https://arxiv.org/abs/2502.08745)
• Keywords: language model
• Abstract: The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing IHEval, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.
• 摘要：指令层次结构从系统消息到用户消息，对话历史记录和工具输出建立优先顺序，对于确保语言模型（LMS）中的一致和安全行为至关重要。尽管其重要性，但该主题受到了有限的关注，并且缺乏评估模型遵循指令层次结构的能力的全面基准。我们通过引入Iheval（一种新型的基准，包括九个任务中的3,538个例子）来弥合这一差距，涵盖了不同优先级或冲突的指令的情况。我们对流行LMS的评估强调了他们为认识指导重点的努力。与原始的指导跟随性能相比，所有评估的模型在面对冲突的指示时的性能下降。此外，最具竞争力的开源模型在解决此类冲突方面仅能达到48％的准确性。我们的结果强调了LMS未来开发中有针对性优化的需求。

Title: SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence

• Authors: Zhining Liu, Rana Ali Amjad, Ravinarayana Adkathimar, Tianxin Wei, Hanghang Tong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08767
• Pdf URL: https://arxiv.org/pdf/2502.08767
• Copy Paste: [[2502.08767]] SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence(https://arxiv.org/abs/2502.08767)
• Keywords: language model, prompt
• Abstract: Providing Language Models (LMs) with relevant evidence in the context (either via retrieval or user-provided) can significantly improve their ability to provide factually correct grounded responses. However, recent studies have found that LMs often struggle to fully comprehend and utilize key evidence from the context, especially when it contains noise and irrelevant information - an issue common in real-world scenarios. To address this, we propose SelfElicit, an inference-time approach that helps LMs focus on key contextual evidence through self-guided explicit highlighting. By leveraging the inherent evidence-finding capabilities of LMs using the attention scores of deeper layers, our method automatically identifies and emphasizes key evidence within the input context, facilitating more accurate and factually grounded responses without additional training or iterative prompting. We demonstrate that SelfElicit brings consistent and significant improvement on multiple evidence-based QA tasks for various LM families while maintaining computational efficiency. Our code and documentation are available at this https URL.
• 摘要：在上下文中（通过检索或提供用户提供的语言模型（LMS）提供相关证据）可以显着提高其提供实际正确的接地响应的能力。但是，最近的研究发现，LMS经常难以完全理解和利用背景下的关键证据，尤其是在包含噪声和无关的信息时，这是现实世界中常见的问题。为了解决这个问题，我们提出了自我选择，这是一种推理时间方法，可以通过自我引导的明确突出显示，以帮助LMS专注于关键上下文证据。通过使用更深层的注意力评分利用LMS的固有循证能力，我们的方法自动识别并强调输入环境中的关键证据，从而促进了更准确，实际上扎根的响应，而无需额外的培训或迭代提示。我们证明，自我选择为各个LM家族的多个基于证据的质量检查任务带来一致和显着改善，同时保持计算效率。我们的代码和文档可在此HTTPS URL上找到。

Title: Universal Model Routing for Efficient LLM Inference

• Authors: Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Zifeng Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, Sanjiv Kumar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08773
• Pdf URL: https://arxiv.org/pdf/2502.08773
• Copy Paste: [[2502.08773]] Universal Model Routing for Efficient LLM Inference(https://arxiv.org/abs/2502.08773)
• Keywords: language model, llm, prompt
• Abstract: Large language models' significant advances in capabilities are accompanied by significant increases in inference costs. Model routing is a simple technique for reducing inference cost, wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective strategies, relying on cluster-based routing and a learned cluster map respectively. We prove that these strategies are estimates of a theoretically optimal routing rule, and provide an excess risk bound to quantify their errors. Experiments on a range of public benchmarks show the effectiveness of the proposed strategies in routing amongst more than 30 unseen LLMs.
• 摘要：大型语言模型在功能方面的重大进展伴随着推理成本的显着增加。模型路由是一种简单的技术，用于降低推理成本，其中一个维护候选LLM的库，并学会将每个提示路由到最小的可行LLM。现有作品着重于学习固定LLM的路由器。在本文中，我们考虑了动态路由的问题，在测试时可以使用新的，以前未观察到的LLM。我们为此问题提出了一种新的方法，该方法依赖于将每个LLM表示为特征向量，该方法是根据一组代表提示的预测得出的。基于此，我们详细介绍了两种有效的策略，分别依靠基于群集的路由和一个学习的群集图。我们证明，这些策略是理论上最佳路由规则的估计，并提供了多余的风险来量化其错误。一系列公共基准的实验表明，拟议策略在30多个看不见的LLMS中的路线方面有效。

Title: Zero-Shot Belief: A Hard Problem for LLMs

• Authors: John Murzaku, Owen Rambow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08777
• Pdf URL: https://arxiv.org/pdf/2502.08777
• Copy Paste: [[2502.08777]] Zero-Shot Belief: A Hard Problem for LLMs(https://arxiv.org/abs/2502.08777)
• Keywords: llm
• Abstract: We present two LLM-based approaches to zero-shot source-and-target belief prediction on FactBank: a unified system that identifies events, sources, and belief labels in a single pass, and a hybrid approach that uses a fine-tuned DeBERTa tagger for event detection. We show that multiple open-sourced, closed-source, and reasoning-based LLMs struggle with the task. Using the hybrid approach, we achieve new state-of-the-art results on FactBank and offer a detailed error analysis. Our approach is then tested on the Italian belief corpus ModaFact.
• 摘要：我们提出了两种基于LLM的方法，用于对FactBank的零摄影和目标信念预测：一个统一的系统，该系统在单个传球中识别事件，来源和信念标签，以及一种使用微型Deberta Tagger的混合方法用于事件检测。我们表明，多个开源，封闭式和基于推理的LLM在这项任务上挣扎。使用混合方法，我们在FactBank上实现了新的最新结果，并提供了详细的错误分析。然后，我们的方法对意大利信仰语料库进行了测试。

Title: If Multi-Agent Debate is the Answer, What is the Question?

• Authors: Hangfan Zhang, Zhiyao Cui, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, Shuyue Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08788
• Pdf URL: https://arxiv.org/pdf/2502.08788
• Copy Paste: [[2502.08788]] If Multi-Agent Debate is the Answer, What is the Question?(https://arxiv.org/abs/2502.08788)
• Keywords: language model, llm, chain-of-thought, agent
• Abstract: Multi-agent debate (MAD) has emerged as a promising approach to enhance the factual accuracy and reasoning quality of large language models (LLMs) by engaging multiple agents in iterative discussions during inference. Despite its potential, we argue that current MAD research suffers from critical shortcomings in evaluation practices, including limited dataset overlap and inconsistent baselines, raising significant concerns about generalizability. Correspondingly, this paper presents a systematic evaluation of five representative MAD methods across nine benchmarks using four foundational models. Surprisingly, our findings reveal that MAD methods fail to reliably outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming additional inference-time computation. From our analysis, we found that model heterogeneity can significantly improve MAD frameworks. We propose Heter-MAD enabling a single LLM agent to access the output from heterogeneous foundation models, which boosts the performance of current MAD frameworks. Finally, we outline potential directions for advancing MAD, aiming to spark a broader conversation and inspire future work in this area.
• 摘要：多代理辩论（MAD）已成为一种有希望的方法，可以通过使多个代理参与推理期间的迭代讨论来提高大语模型（LLMS）的事实准确性和推理质量。尽管具有潜力，但我们认为当前的MAD研究遭受了评估实践的关键缺点，包括数据集有限的重叠和不一致的基线，引起了人们对概括性的重大关注。相应地，本文使用四个基础模型对九个基准的五种代表性MAD方法进行了系统评估。令人惊讶的是，我们的发现表明，即使消耗额外的推理时间计算，MAD方法也无法可靠地优于简单的单一基准基线，例如思考链和自耐心。从我们的分析中，我们发现模型异质性可以显着改善疯狂框架。我们提出了Heter-Mad，使单个LLM代理可以访问异质基础模型的输出，从而提高了当前的疯狂框架的性能。最后，我们概述了推进疯狂的潜在方向，旨在激发更广泛的对话并激发该领域的未来工作。

Title: A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks

• Authors: Karahan Sarıtaş, Kıvanç Tezören, Yavuz Durmazkeser
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2502.08796
• Pdf URL: https://arxiv.org/pdf/2502.08796
• Copy Paste: [[2502.08796]] A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks(https://arxiv.org/abs/2502.08796)
• Keywords: language model, llm, prompt
• Abstract: In recent years, evaluating the Theory of Mind (ToM) capabilities of large language models (LLMs) has received significant attention within the research community. As the field rapidly evolves, navigating the diverse approaches and methodologies has become increasingly complex. This systematic review synthesizes current efforts to assess LLMs' ability to perform ToM tasks, an essential aspect of human cognition involving the attribution of mental states to oneself and others. Despite notable advancements, the proficiency of LLMs in ToM remains a contentious issue. By categorizing benchmarks and tasks through a taxonomy rooted in cognitive science, this review critically examines evaluation techniques, prompting strategies, and the inherent limitations of LLMs in replicating human-like mental state reasoning. A recurring theme in the literature reveals that while LLMs demonstrate emerging competence in ToM tasks, significant gaps persist in their emulation of human cognitive abilities.
• 摘要：近年来，评估大语言模型（LLM）的思维理论（TOM）能力（LLMS）在研究界受到了极大的关注。随着领域的迅速发展，导航各种方法和方法论已经变得越来越复杂。这项系统的审查综合了当前评估LLMS执行TOM任务能力的努力，这是人类认知的一个基本方面，涉及将精神状态归因于自己和他人。尽管取得了显着的进步，但LLM在Tom中的熟练程度仍然是一个有争议的问题。通过通过植根于认知科学的分类法对基准和任务进行分类，该评论认真研究了评估技术，促使策略以及LLM在复制类似人类的精神状态推理方面的固有局限性。文献中的一个反复出现的主题表明，尽管LLMS在TOM任务中表现出了新兴的能力，但在仿真人类认知能力中的差距仍然存在。

Title: Lexical Manifold Reconfiguration in Large Language Models: A Novel Architectural Approach for Contextual Modulation

• Authors: Koinis Vassilis, Godfrey Milbourne, Harriet Featherstone, Xanthe Peverell, Yorick Bletchley, Zachary Montford
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08818
• Pdf URL: https://arxiv.org/pdf/2502.08818
• Copy Paste: [[2502.08818]] Lexical Manifold Reconfiguration in Large Language Models: A Novel Architectural Approach for Contextual Modulation(https://arxiv.org/abs/2502.08818)
• Keywords: language model
• Abstract: Contextual adaptation in token embeddings plays a central role in determining how well language models maintain coherence and retain semantic relationships over extended text sequences. Static embeddings often impose constraints on lexical flexibility, leading to suboptimal performance when faced with complex sentence structures or domain-specific terminology shifts. To address this limitation, a structured approach was developed for dynamically reconfiguring token embeddings through continuous geometric transformations, ensuring that representations evolved in response to evolving discourse structures. A manifold-based transformation mechanism was integrated to regulate lexical positioning, allowing embeddings to undergo controlled shifts while preserving linguistic relationships across varying textual contexts. Empirical evaluations demonstrated that embedding reconfiguration contributed to reductions in perplexity, improved lexical coherence, and enhanced sentence-level continuity, particularly in structured and domain-adaptive text generation tasks. Comparative analyses of embedding drift indicated that dynamically restructured representations maintained stronger contextual consistency, reducing misalignment in token dependencies while preserving fluency in language modeling outputs. Computational overhead assessments confirmed that while training complexity increased due to the iterative refinement of embeddings, inference remained efficient, ensuring practical feasibility for real-time generation. Evaluations across multiple datasets further demonstrated that dynamically modulated embeddings exhibited broader lexical diversity, reducing repetitive token patterns and enabling a more adaptable representation learning process.
• 摘要：令牌嵌入中的上下文适应在确定语言模型如何保持连贯性并在扩展文本序列上保持语义关系的核心作用。静态嵌入通常对词汇灵活性施加限制，当面对复杂的句子结构或特定领域的术语变化时，会导致次优性能。为了解决这一限制，开发了一种结构化方法，用于通过连续的几何变换动态重新配置令牌嵌入，从而确保表示表示响应不断发展的话语结构。集成了一种基于多种转化机制，以调节词汇定位，从而使嵌入可以进行受控的移位，同时在不同的文本上下文中保留语言关系。经验评估表明，嵌入重新配置有助于减少困惑，提高词汇连贯性和增强的句子级连续性，尤其是在结构化和域自适应的文本生成任务中。嵌入漂移的比较分析表明，动态重组的表示能保持更强的上下文一致性，从而减少了令牌依赖性的未对准，同时保持语言建模输出的流利度。计算间接费用评估证实，虽然训练复杂性由于嵌入的迭代性完善而增加，但推论仍然有效，可确保实时生成的实际可行性。多个数据集的评估进一步表明，动态调制的嵌入式表现出更广泛的词汇多样性，减少了重复的令牌模式，并启用了更适应性的表示表示过程。

Title: Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

• Authors: Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, Ehsaneddin Asgari
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.08826
• Pdf URL: https://arxiv.org/pdf/2502.08826
• Copy Paste: [[2502.08826]] Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation(https://arxiv.org/abs/2502.08826)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) struggle with hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information enhancing factual and updated grounding. Recent advances in multimodal learning have led to the development of Multimodal RAG, incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges to Multimodal RAG, distinguishing it from traditional unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, metrics, benchmarks, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We precisely review training strategies, robustness enhancements, and loss functions, while also exploring the diverse Multimodal RAG scenarios. Furthermore, we discuss open challenges and future research directions to support advancements in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. Resources are available at this https URL.
• 摘要：大型语言模型（LLM）由于依赖静态培训数据而与幻觉和过时的知识斗争。通过集成外部动态信息来增强事实和更新的基础，检索增强的生成（RAG）通过整合外部动态信息来减轻这些问题。多模式学习的最新进展导致了多模式抹布的发展，并结合了多种模式，例如文本，图像，音频和视频，以增强生成的输出。但是，跨模式的对齐和推理对多模式抹布引入了独特的挑战，将其与传统的单峰抹布区分开。这项调查提供了对多模式抹布系统的结构化和全面分析，涵盖了检索，融合，增强和一代中的数据集，指标，基准，评估，方法和创新。我们精确地审查了培训策略，鲁棒性增强和损失功能，同时还探索了多种多态的破布场景。此外，我们讨论了支持这个不断发展的领域进步的开放挑战和未来的研究方向。这项调查为开发更有效和可靠的AI系统的基础奠定了基础，这些系统有效地利用了多模式动态外部知识库。资源可在此HTTPS URL上找到。

Title: LLM-Enhanced Multiple Instance Learning for Joint Rumor and Stance Detection with Social Context Information

• Authors: Ruichao Yang, Jing Ma, Wei Gao, Hongzhan Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08888
• Pdf URL: https://arxiv.org/pdf/2502.08888
• Copy Paste: [[2502.08888]] LLM-Enhanced Multiple Instance Learning for Joint Rumor and Stance Detection with Social Context Information(https://arxiv.org/abs/2502.08888)
• Keywords: llm, prompt
• Abstract: The proliferation of misinformation, such as rumors on social media, has drawn significant attention, prompting various expressions of stance among users. Although rumor detection and stance detection are distinct tasks, they can complement each other. Rumors can be identified by cross-referencing stances in related posts, and stances are influenced by the nature of the rumor. However, existing stance detection methods often require post-level stance annotations, which are costly to obtain. We propose a novel LLM-enhanced MIL approach to jointly predict post stance and claim class labels, supervised solely by claim labels, using an undirected microblog propagation model. Our weakly supervised approach relies only on bag-level labels of claim veracity, aligning with multi-instance learning (MIL) principles. To achieve this, we transform the multi-class problem into multiple MIL-based binary classification problems. We then employ a discriminative attention layer to aggregate the outputs from these classifiers into finer-grained classes. Experiments conducted on three rumor datasets and two stance datasets demonstrate the effectiveness of our approach, highlighting strong connections between rumor veracity and expressed stances in responding posts. Our method shows promising performance in joint rumor and stance detection compared to the state-of-the-art methods.
• 摘要：错误信息的扩散，例如社交媒体上的谣言，引起了人们的重大关注，促使用户之间的各种表现表达。尽管谣言检测和立场检测是不同的任务，但它们可以相互补充。可以通过在相关职位的交叉引用立场来识别谣言，而立场则受谣言性质的影响。但是，现有的立场检测方法通常需要后水平的立场注释，这是昂贵的。我们提出了一种新型的LLM增强MIL方法，以使用无方向的微博传播模型共同预测仅由索赔标签监督的索赔类标签。我们弱监督的方法仅依赖于声称真实性的行李级标签，与多构想学习（MIL）原则保持一致。为了实现这一目标，我们将多级问题转换为多个基于MIL的二进制分类问题。然后，我们采用一个歧视性注意力层将这些分类器的输出汇总为细粒类别。在三个谣言数据集和两个立场数据集上进行的实验证明了我们方法的有效性，强调了谣言真实性与响应帖子表达的立场之间的牢固联系。与最先进的方法相比，我们的方法显示了联合谣言和立场检测的有希望的表现。

Title: Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communication

• Authors: Weicheng Ma, Hefan Zhang, Ivory Yang, Shiyu Ji, Joice Chen, Farnoosh Hashemi, Shubham Mohole, Ethan Gearey, Michael Macy, Saeed Hassanpour, Soroush Vosoughi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08896
• Pdf URL: https://arxiv.org/pdf/2502.08896
• Copy Paste: [[2502.08896]] Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communication(https://arxiv.org/abs/2502.08896)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown proficiency in generating persuasive dialogue, yet concerns about the fluency and sophistication of their outputs persist. This paper presents a multi-LLM communication framework designed to enhance the generation of persuasive data automatically. This framework facilitates the efficient production of high-quality, diverse linguistic content with minimal human oversight. Through extensive evaluations, we demonstrate that the generated data excels in naturalness, linguistic diversity, and the strategic use of persuasion, even in complex scenarios involving social taboos. The framework also proves adept at generalizing across novel contexts. Our results highlight the framework's potential to significantly advance research in both computational and social science domains concerning persuasive communication.
• 摘要：大型语言模型（LLM）表明，熟练地产生了有说服力的对话，但担心其产出的流利性和成熟程度持续存在。本文介绍了一个多LLM通信框架，旨在自动增强有说服力的数据的生成。该框架有助于有效地生产高质量的，多样化的语言内容，而人类的监督最少。通过广泛的评估，我们证明，即使在涉及社会禁忌的复杂情况下，生成的数据在自然性，语言多样性以及说服力的战略使用方面都表现出色。该框架还证明擅长于跨新颖环境概括。我们的结果突出了该框架在计算和社会科学领域有关有说服力沟通的研究的潜力。

Title: Can Uniform Meaning Representation Help GPT-4 Translate from Indigenous Languages?

• Authors: Shira Wein
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08900
• Pdf URL: https://arxiv.org/pdf/2502.08900
• Copy Paste: [[2502.08900]] Can Uniform Meaning Representation Help GPT-4 Translate from Indigenous Languages?(https://arxiv.org/abs/2502.08900)
• Keywords: gpt, prompt, chat
• Abstract: While ChatGPT and GPT-based models are able to effectively perform many tasks without additional fine-tuning, they struggle with related to extremely low-resource languages and indigenous languages. Uniform Meaning Representation (UMR), a semantic representation designed to capture the meaning of texts in many languages, is well-poised to be leveraged in the development of low-resource language technologies. In this work, we explore the downstream technical utility of UMR for low-resource languages by incorporating it into GPT-4 prompts. Specifically, we examine the ability of GPT-4 to perform translation from three indigenous languages (Navajo, Arápaho, and Kukama), with and without demonstrations, as well as with and without UMR annotations. Ultimately we find that in the majority of our test cases, integrating UMR into the prompt results in a statistically significant increase in performance, which is a promising indication of future applications of the UMR formalism.
• 摘要：尽管基于Chatgpt和基于GPT的模型能够有效执行许多任务，而无需进行其他微调，但它们与非常低的资源语言和土著语言有关。统一的含义表示（UMR）是一种旨在以多种语言捕获文本含义的语义表示，在低资源语言技术的开发中被充分利用。在这项工作中，我们通过将其纳入GPT-4提示中来探索UMR的下游技术实用程序。具体来说，我们检查了GPT-4从三种土著语言（纳瓦霍，阿拉帕霍和库卡马）进行翻译的能力，有无示威以及没有UMR注释。最终，我们发现，在大多数测试案例中，将UMR整合到及时的迅速中，导致统计学上的绩效提高，这是UMR形式主义未来应用的有希望的迹象。

Title: Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs

• Authors: Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, Vera Schmitt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08909
• Pdf URL: https://arxiv.org/pdf/2502.08909
• Copy Paste: [[2502.08909]] Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs(https://arxiv.org/abs/2502.08909)
• Keywords: language model, llm
• Abstract: Fact-checking is necessary to address the increasing volume of misinformation. Traditional fact-checking relies on manual analysis to verify claims, but it is slow and resource-intensive. This study establishes baseline comparisons for Automated Fact-Checking (AFC) using Large Language Models (LLMs) across multiple labeling schemes (binary, three-class, five-class) and extends traditional claim verification by incorporating analysis, verdict classification, and explanation in a structured setup to provide comprehensive justifications for real-world claims. We evaluate Llama-3 models of varying sizes (3B, 8B, 70B) on 17,856 claims collected from PolitiFact (2007-2024) using evidence retrieved via restricted web searches. We utilize TIGERScore as a reference-free evaluation metric to score the justifications. Our results show that larger LLMs consistently outperform smaller LLMs in classification accuracy and justification quality without fine-tuning. We find that smaller LLMs in a one-shot scenario provide comparable task performance to fine-tuned Small Language Models (SLMs) with large context sizes, while larger LLMs consistently surpass them. Evidence integration improves performance across all models, with larger LLMs benefiting most. Distinguishing between nuanced labels remains challenging, emphasizing the need for further exploration of labeling schemes and alignment with evidences. Our findings demonstrate the potential of retrieval-augmented AFC with LLMs.
• 摘要：对于解决错误信息的增加是必要的事实检查。传统的事实检查依赖于手动分析来验证主张，但它是缓慢且资源密集的。这项研究建立了在多个标签方案（二进制，三级，五级）中使用大语言模型（LLM）进行自动事实检查（AFC）的基线比较，并通过将分析，判决分类和解释中的解释来扩展传统的主张验证一个结构化的设置，可为现实世界索赔提供全面的理由。我们使用限制性的网络搜索获得了证据，评估了从Politifact（2007-2024）收集的17,856个索赔，评估从Politifact（2007-2024）收集的17,856个索赔的Llama-3模型（3b，8b，70b）。我们将TigersCore用作无参考评估指标来评分理由。我们的结果表明，较大的LLM在分类准确性和理由质量方面始终超过较小的LLM，而无需微调。我们发现，在一次性场景中，较小的LLM可提供与具有较大上下文大小的微调小语言模型（SLM）相当的任务性能，而较大的LLM始终超过它们。证据集成改善了所有模型的性能，较大的LLM受益最大。区分细微的标签仍然具有挑战性，强调需要进一步探索标签方案以及与证据的一致性。我们的发现证明了使用LLM的检索afc的潜力。

Title: InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU

• Authors: Heejun Lee, Geon Park, Jaduk Suh, Sung Ju Hwang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08910
• Pdf URL: https://arxiv.org/pdf/2502.08910
• Copy Paste: [[2502.08910]] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU(https://arxiv.org/abs/2502.08910)
• Keywords: language model, llm, long context
• Abstract: In modern large language models (LLMs), handling very long context lengths presents significant challenges as it causes slower inference speeds and increased memory costs. Additionally, most existing pre-trained LLMs fail to generalize beyond their original training sequence lengths. To enable efficient and practical long-context utilization, we introduce InfiniteHiP, a novel, and practical LLM inference framework that accelerates processing by dynamically eliminating irrelevant context tokens through a modular hierarchical token pruning algorithm. Our method also allows generalization to longer sequences by selectively applying various RoPE adjustment methods according to the internal attention patterns within LLMs. Furthermore, we offload the key-value cache to host memory during inference, significantly reducing GPU memory pressure. As a result, InfiniteHiP enables the processing of up to 3 million tokens on a single L40s 48GB GPU -- 3x larger -- without any permanent loss of context information. Our framework achieves an 18.95x speedup in attention decoding for a 1 million token context without requiring additional training. We implement our method in the SGLang framework and demonstrate its effectiveness and practicality through extensive evaluations.
• 摘要：在现代大型语言模型（LLMS）中，处理非常长的上下文长度会带来重大挑战，因为它会导致推理速度较慢并增加记忆成本。此外，大多数现有的预训练的LLM都无法推广其原始训练序列长度。为了实现高效且实用的长篇文化利用，我们引入了无限且实用的LLM推理框架，该框架通过动态消除无关的上下文标记通过模块化的层次标记固定算法来加速处理。我们的方法还允许通过根据LLM中的内部注意力模式选择性地应用各种绳索调整方法来概括更长的序列。此外，我们在推理过程中将密钥值缓存卸载到主机内存，从而大大降低了GPU内存压力。结果，无限型可以在单个L40S 48GB GPU上处理多达300万个令牌-3倍 - 而不会永久损失上下文信息。我们的框架在不需要额外的培训的情况下，对100万个令牌环境的注意力解码实现了18.95倍的速度。我们在SGLANG框架中实施我们的方法，并通过广泛的评估来证明其有效性和实用性。

Title: CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality

• Authors: Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav, Mihai Surdeanu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08923
• Pdf URL: https://arxiv.org/pdf/2502.08923
• Copy Paste: [[2502.08923]] CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality(https://arxiv.org/abs/2502.08923)
• Keywords: llm, chat
• Abstract: We introduce CopySpec, an innovative technique designed to tackle the inefficiencies LLMs face when generating responses that closely resemble previous outputs. CopySpec identifies repeated sequences in the model's chat history and speculates that the same tokens will follow, enabling seamless copying without compromising output quality or requiring additional GPU memory. To evaluate the effectiveness of our approach, we conducted experiments using five LLMs and five datasets: MT-Bench, CNN/DM, GSM-8K, HumanEval, and our newly created dataset, MT-Redundant. MT-Redundant, introduced in this paper, transforms the second turn of MT-Bench into a request for variations of the first turn's answer, simulating real-world scenarios where users request modifications to prior responses. Our results demonstrate significant speed-ups: up to 2.35x on CNN/DM, 3.08x on the second turn of select MT-Redundant categories, and 2.66x on the third turn of GSM-8K's self-correction tasks. Moreover, we show that CopySpec integrates seamlessly with speculative decoding, yielding an average 49% additional speed-up over speculative decoding for the second turn of MT-Redundant across all eight categories. While LLMs, even with speculative decoding, suffer from slower inference as context sizes grow, CopySpec leverages the expanded context to accelerate inference, making it faster as the context size increases. Our code and dataset are publicly available at this https URL.
• 摘要：我们介绍了CopySpec，这是一种创新的技术，旨在解决与以前的输出非常相似的响应时，旨在应对LLMS效率低下的LLM面临。 CopySpec在模型的聊天历史记录中标识重复的序列，并推测同样的令牌将随后进行，从而无缝复制而不会损害输出质量或需要其他GPU内存。为了评估我们的方法的有效性，我们使用五个LLM和五个数据集进行了实验：MT Bench，CNN/DM，GSM-8K，HumaneVal，以及我们新创建的数据集，MT-REDLOUNTT。本文介绍的MT冗余将MT基础的第二回合转换为第一个转弯答案变化的请求，从而模拟了现实世界中的情况，用户请求对先前响应请求修改。我们的结果表明，CNN/DM上最高2.35倍，选择的MT冗余类别的第二回合为3.08倍，而GSM-8K的自我纠正任务的第三圈则为2.66倍。此外，我们表明，CopySpec与投机解码无缝集成，在所有八个类别中，对于第二次MT冗余的投机解码，平均额外的加速度为49％。尽管LLMS即使进行投机解码，随着上下文大小的增长的推断，CopySpec的推断也较慢，但CopySpec利用了扩展的上下文来加速推断，随着上下文大小的增加，它的速度更快。我们的代码和数据集可在此HTTPS URL上公开获得。

Title: Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis

• Authors: Wenbo Zhang, Hengrui Cai, Wenyu Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08943
• Pdf URL: https://arxiv.org/pdf/2502.08943
• Copy Paste: [[2502.08943]] Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis(https://arxiv.org/abs/2502.08943)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated significant utilities in real-world applications, exhibiting impressive capabilities in natural language processing and understanding. Benchmark evaluations are crucial for assessing the capabilities of LLMs as they can provide a comprehensive assessment of their strengths and weaknesses. However, current evaluation methods often overlook the inherent randomness of LLMs by employing deterministic generation strategies or relying on a single random sample, resulting in unaccounted sampling variance and unreliable benchmark score estimates. In this paper, we propose a hierarchical statistical model that provides a more comprehensive representation of the benchmarking process by incorporating both benchmark characteristics and LLM randomness. We show that leveraging multiple generations improves the accuracy of estimating the benchmark score and reduces variance. We also introduce $\mathbb P\left(\text{correct}\right)$, a prompt-level difficulty score based on correct ratios, providing fine-grained insights into individual prompts. Additionally, we create a data map that visualizes difficulty and semantic prompts, enabling error detection and quality control in benchmark construction.
• 摘要：大型语言模型（LLMS）在现实世界应用中表现出重要的公用事业，在自然语言处理和理解中表现出令人印象深刻的功能。基准评估对于评估LLM的能力至关重要，因为它们可以对其优势和劣势进行全面评估。但是，当前的评估方法通常通过采用确定性生成策略或依靠单个随机样本来忽略LLM的固有随机性，从而导致不计算的采样方差和不可靠的基准分数得分估计。在本文中，我们提出了一个层次统计模型，该模型通过纳入基准特征和LLM随机性来对基准测试过程提供更全面的表示。我们表明，利用多代提高了估计基准分数并降低方差的准确性。我们还介绍了$ \ mathbb p \ left（\ text {preck {prive} \ right）$，这是一个基于正确比率的提示级别难度分数，从而为单个提示提供了细粒度的见解。此外，我们创建了一个数据图，该数据图可视化难度和语义提示，从而在基准结构中实现了错误检测和质量控制。

Title: The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding

• Authors: Mo Yu, Lemao Liu, Junjie Wu, Tsz Ting Chung, Shunchi Zhang, Jiangnan Li, Dit-Yan Yeung, Jie Zhou
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08946
• Pdf URL: https://arxiv.org/pdf/2502.08946
• Copy Paste: [[2502.08946]] The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding(https://arxiv.org/abs/2502.08946)
• Keywords: gpt, llm
• Abstract: In a systematic way, we investigate a widely asked question: Do LLMs really understand what they say?, which relates to the more familiar term Stochastic Parrot. To this end, we propose a summative assessment over a carefully designed physical concept understanding task, PhysiCo. Our task alleviates the memorization issue via the usage of grid-format inputs that abstractly describe physical phenomena. The grids represents varying levels of understanding, from the core phenomenon, application examples to analogies to other abstract patterns in the grid world. A comprehensive study on our task demonstrates: (1) state-of-the-art LLMs, including GPT-4o, o1 and Gemini 2.0 flash thinking, lag behind humans by ~40%; (2) the stochastic parrot phenomenon is present in LLMs, as they fail on our grid task but can describe and recognize the same concepts well in natural language; (3) our task challenges the LLMs due to intrinsic difficulties rather than the unfamiliar grid format, as in-context learning and fine-tuning on same formatted data added little to their performance.
• 摘要：我们以系统的方式研究了一个广泛提出的问题：LLMS是否真的了解他们说的话？这与更熟悉的术语随机鹦鹉有关。为此，我们提出了对理解的物理概念理解任务物理学的总结性评估。我们的任务通过使用网格形式的输入来减轻记忆问题，这些输入抽象地描述了物理现象。网格代表了不同水平的理解水平，从核心现象，应用程序示例到类比到网格世界中的其他抽象模式。对我们任务的全面研究表明：（1）最先进的LLM，包括GPT-4O，O1和Gemini 2.0 Flash Thinking，落后于人类落后于40％； （2）随机鹦鹉现象存在于LLM中，因为它们在我们的网格任务上失败，但可以很好地描述和认识到自然语言的相同概念； （3）由于内在的困难而不是陌生的网格格式，我们的任务挑战了LLMS，因为在同一格式的数据上进行了文化学习和微调，对其性能几乎没有增加。

Title: Structured Convergence in Large Language Model Representations via Hierarchical Latent Space Folding

• Authors: Fenella Harcourt, Naderdel Piero, Gilbert Sutherland, Daphne Holloway, Harriet Bracknell, Julian Ormsby
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08947
• Pdf URL: https://arxiv.org/pdf/2502.08947
• Copy Paste: [[2502.08947]] Structured Convergence in Large Language Model Representations via Hierarchical Latent Space Folding(https://arxiv.org/abs/2502.08947)
• Keywords: language model
• Abstract: Token representations in high-dimensional latent spaces often exhibit redundancy, limiting computational efficiency and reducing structural coherence across model layers. Hierarchical latent space folding introduces a structured transformation mechanism that enforces a multi-scale organization within learned embeddings, refining representational compactness while preserving essential contextual distinctions. The proposed approach incorporates dynamic folding operations that iteratively adjust token embeddings through structured transformations, influencing both short-range and long-range dependencies in sequential processing tasks. Empirical evaluation demonstrates a reduction in representational variance across layers, contributing to more stable perplexity distributions and enhancing predictive confidence in text generation. The structured redistribution of attention head utilization leads to more efficient allocation of computational resources, particularly in deeper layers, where hierarchical refinements improve contextual abstraction. Comparative analysis of activation sparsity patterns suggests that hierarchical adjustments selectively reinforce critical pathways while reducing computational overhead in non-essential regions of the model. Statistical assessments of token reordering frequencies reveal that hierarchical modifications introduce subtle shifts in sequential dependencies, improving contextual alignment while maintaining syntactic correctness. Computational trade-offs associated with hierarchical folding introduce marginal increases in training time per epoch, yet empirical findings indicate that inference efficiency benefits from the structured representation adjustments. The results highlight the impact of hierarchical latent space folding on optimizing model performance through improved representation structuring and computational efficiency.
• 摘要：高维潜在空间中的令牌表示通常表现出冗余，限制了计算效率并降低了模型层之间的结构相干性。分层潜在空间折叠引入了一种结构化的转换机制，该机制在学习的嵌入式中实施多尺度组织，在保留基本的上下文区分的同时，可以完善代表性的紧凑性。所提出的方法结合了动态折叠操作，通过结构化转换迭代地调整令牌嵌入，从而影响顺序处理任务中的短距离和远程依赖性。经验评估表明，跨层的代表性方差降低，导致更稳定的困惑分布并增强文本生成中的预测信心。注意力头利用的结构化重新分布会导致计算资源的更有效分配，尤其是在更深层次的层次中，层次改进可以改善上下文抽象。激活稀疏模式的比较分析表明，层次调整有选择地加强关键途径，同时减少模型非必需区域的计算开销。令牌重新排序频率的统计评估表明，层次修改会引入顺序依赖性的细微变化，改善上下文对齐，同时保持句法正确性。与分层折叠相关的计算权衡会引入每个时​​期训练时间的边际增加，但经验发现表明，推论效率受益于结构化表示的调整。结果突出了分层潜在空间折叠对通过提高表示形式结构和计算效率优化模型性能的影响。

Title: Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning

• Authors: Leon Nissen, Philipp Zagar, Vishnu Ravi, Aydin Zahedivash, Lara Marie Reimer, Stephan Jonas, Oliver Aalami, Paul Schmiedmayer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08954
• Pdf URL: https://arxiv.org/pdf/2502.08954
• Copy Paste: [[2502.08954]] Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning(https://arxiv.org/abs/2502.08954)
• Keywords: language model, llm
• Abstract: The deployment of Large Language Models (LLM) on mobile devices offers significant potential for medical applications, enhancing privacy, security, and cost-efficiency by eliminating reliance on cloud-based services and keeping sensitive health data local. However, the performance and accuracy of on-device LLMs in real-world medical contexts remain underexplored. In this study, we benchmark publicly available on-device LLMs using the AMEGA dataset, evaluating accuracy, computational efficiency, and thermal limitation across various mobile devices. Our results indicate that compact general-purpose models like Phi-3 Mini achieve a strong balance between speed and accuracy, while medically fine-tuned models such as Med42 and Aloe attain the highest accuracy. Notably, deploying LLMs on older devices remains feasible, with memory constraints posing a greater challenge than raw processing power. Our study underscores the potential of on-device LLMs for healthcare while emphasizing the need for more efficient inference and models tailored to real-world clinical reasoning.
• 摘要：移动设备上大型语言模型（LLM）的部署通过消除对基于云的服务的依赖并在本地保持敏感健康数据的依赖，从而为医疗应用程序提供了巨大的潜力，增强隐私，安全性和成本效益。但是，在现实世界中的医疗环境中，在设备LLMS的性能和准确性仍未得到充实。在这项研究中，我们使用AMEGA数据集对公开可用的设备LLM进行了基准，评估了各种移动设备的准确性，计算效率和热限制。我们的结果表明，诸如PHI-3 mini之类的紧凑型通用模型在速度和准确性之间达到了强劲的平衡，而医学微调模型（如Med42和芦荟）达到了最高的精度。值得注意的是，在较旧设备上部署LLMS仍然是可行的，并且内存约束比原始处理能力更大。我们的研究强调了evice llms在医疗保健方面的潜力，同时强调了更有效的推理和针对现实世界中临床推理量身定制的模型。

Title: Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning

• Authors: Hyundong Cho, Karishma Sharma, Nicolaas Jedema, Leonardo F. R. Ribeiro, Alessandro Moschitti, Ravi Krishnan, Jonathan May
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08972
• Pdf URL: https://arxiv.org/pdf/2502.08972
• Copy Paste: [[2502.08972]] Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning(https://arxiv.org/abs/2502.08972)
• Keywords: language model, llm, prompt
• Abstract: Language models are aligned to the collective voice of many, resulting in generic outputs that do not align with specific users' styles. In this work, we present Trial-Error-Explain In-Context Learning (TICL), a tuning-free method that personalizes language models for text generation tasks with fewer than 10 examples per user. TICL iteratively expands an in-context learning prompt via a trial-error-explain process, adding model-generated negative samples and explanations that provide fine-grained guidance towards a specific user's style. TICL achieves favorable win rates on pairwise comparisons with LLM-as-a-judge up to 91.5% against the previous state-of-the-art and outperforms competitive tuning-free baselines for personalized alignment tasks of writing emails, essays and news articles. Both lexical and qualitative analyses show that the negative samples and explanations enable language models to learn stylistic context more effectively and overcome the bias towards structural and formal phrases observed in their zero-shot outputs. By front-loading inference compute to create a user-specific in-context learning prompt that does not require extra generation steps at test time, TICL presents a novel yet simple approach for personalized alignment.
• 摘要：语言模型与许多人的集体语音保持一致，从而导致通用输出与特定用户的样式不符。在这项工作中，我们介绍了试用版 - 解释内部文化学习（TICL），这是一种无调的方法，为每位用户少于10个示例的文本生成任务的语言模型个性化。 TICL迭代地通过试用版解释过程扩展了秘密学习提示，并添加了模型生成的负面样本和解释，这些样本为特定用户的样式提供了细粒度的指导。 TICL在与以前的最先前的最先前的法官律师法官的成对比较中达到了有利的获胜率，并且在撰写电子邮件，文章和新闻文章的个性化对齐任务上，最高可达91.5％，并且优于竞争性的无调基准。词汇和定性分析都表明，负面样本和解释使语言模型能够更有效地学习风格上下文，并克服对在其零发出输出中观察到的结构和正式短语的偏见。通过前加载推理计算来创建一个特定于用户的内在学习提示，该提示在测试时不需要额外的生成步骤，TICL提出了一种新颖而简单的个性化对齐方式。

Title: Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech

• Authors: Jonathan Pofcher, Christopher M. Homan, Randall Sell, Ashiqur R. KhudaBukhsh
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09004
• Pdf URL: https://arxiv.org/pdf/2502.09004
• Copy Paste: [[2502.09004]] Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech(https://arxiv.org/abs/2502.09004)
• Keywords: language model, llm
• Abstract: This paper makes three contributions. First, via a substantial corpus of 1,419,047 comments posted on 3,161 YouTube news videos of major US cable news outlets, we analyze how users engage with LGBTQ+ news content. Our analyses focus both on positive and negative content. In particular, we construct a fine-grained hope speech classifier that detects positive (hope speech), negative, neutral, and irrelevant content. Second, in consultation with a public health expert specializing on LGBTQ+ health, we conduct an annotation study with a balanced and diverse political representation and release a dataset of 3,750 instances with fine-grained labels and detailed annotator demographic information. Finally, beyond providing a vital resource for the LGBTQ+ community, our annotation study and subsequent in-the-wild assessments reveal (1) strong association between rater political beliefs and how they rate content relevant to a marginalized community; (2) models trained on individual political beliefs exhibit considerable in-the-wild disagreement; and (3) zero-shot large language models (LLMs) align more with liberal raters.
• 摘要：本文做出了三项贡献。首先，通过在3,161个YouTube新闻视频中发布的1,419,047条评论的大量语料库，有关美国主要有线新闻媒体的新闻视频，我们分析了用户如何与LGBTQ+新闻内容互动。我们的分析侧重于正含量和负含量。特别是，我们构建了一个细颗粒的希望语音分类器，该分类器检测到积极的（希望言语），消极，中性和无关紧要的内容。其次，在与专门从事LGBTQ+ Health的公共卫生专家协商时，我们进行了一项注释研究，并具有平衡和多样化的政治代表制，并发布了3,750个实例的数据集，其中包含精细的标签和详细的注释人口统计信息。最后，除了为LGBTQ+社区提供重要资源外，我们的注释研究和随后的野外评估表明（1）评估者政治信念之间的密切关联以及它们如何评价与边缘化社区相关的内容； （2）接受过个人政治信仰的模型表现出相当大的野外分歧； （3）零击大语模型（LLMS）与自由主义者的评估者更加吻合。

Title: Diversity Enhances an LLM's Performance in RAG and Long-context Task

• Authors: Zhchao Wang, Bin Bi, Yanqi Luo, Sitaram Asur, Claire Na Cheng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09017
• Pdf URL: https://arxiv.org/pdf/2502.09017
• Copy Paste: [[2502.09017]] Diversity Enhances an LLM's Performance in RAG and Long-context Task(https://arxiv.org/abs/2502.09017)
• Keywords: language model, llm, long context, retrieval-augmented generation
• Abstract: The rapid advancements in large language models (LLMs) have highlighted the challenge of context window limitations, primarily due to the quadratic time complexity of the self-attention mechanism (\(O(N^2)\), where \(N\) denotes the context window length). This constraint impacts tasks such as retrieval-augmented generation (RAG) in question answering (Q\&A) and long context summarization. A common approach involves selecting content with the highest similarity to the query; however, this often leads to redundancy and the exclusion of diverse yet relevant information. Building on principles from Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), we integrate diversity into the content selection process. Our findings reveal that incorporating diversity substantially increases the recall of selecting relevant sentences or chunks before LLM-based Q\&A and summarization. These results highlight the importance of maintaining diversity in future LLM applications to further improve summarization and Q\&A outcomes.
• 摘要：大语言模型（LLMS）的快速进步突出了上下文窗口限制的挑战，这主要是由于自我注意机制的二次时间复杂性（\（o（o（n^2）\）），其中\（n \）表示上下文窗口长度）。这种限制会影响任务，例如回答（Q \＆a）和长上下文摘要之类的检索效果生成（RAG）。一种常见的方法涉及选择与查询最高相似性的内容；但是，这通常会导致冗余并排除多样化但相关的信息。在最大边缘相关性（MMR）和最远的点采样（FPS）的原则基础上，我们将多样性整合到内容选择过程中。我们的发现表明，在基于LLM的Q \＆A和Summarization和Summarization之前，合并多样性大大增加了选择相关句子或块的召回。这些结果强调了在未来的LLM应用程序中保持多样性以进一步改善摘要和Q \＆A结果的重要性。

Title: Typhoon T1: An Open Thai Reasoning Model

• Authors: Pittawat Taveekitworachai, Potsawee Manakul, Kasima Tharnpipitchai, Kunat Pipatanakul
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09042
• Pdf URL: https://arxiv.org/pdf/2502.09042
• Copy Paste: [[2502.09042]] Typhoon T1: An Open Thai Reasoning Model(https://arxiv.org/abs/2502.09042)
• Keywords: language model, llm
• Abstract: This paper introduces Typhoon T1, an open effort to develop an open Thai reasoning model. A reasoning model is a relatively new type of generative model built on top of large language models (LLMs). A reasoning model generates a long chain of thought before arriving at a final answer, an approach found to improve performance on complex tasks. However, details on developing such a model are limited, especially for reasoning models that can generate traces in a low-resource language. Typhoon T1 presents an open effort that dives into the details of developing a reasoning model in a more cost-effective way by leveraging supervised fine-tuning using open datasets, instead of reinforcement learning. This paper shares the details about synthetic data generation and training, as well as our dataset and model weights. Additionally, we provide insights gained from developing a reasoning model that generalizes across domains and is capable of generating reasoning traces in a low-resource language, using Thai as an example. We hope this open effort provides a foundation for further research in this field.
• 摘要：本文介绍了Typhoon T1，这是开放开放的泰语推理模型的开放努力。推理模型是建立在大型语言模型（LLMS）之上的一种相对较新的生成模型。推理模型在到达最终答案之前会产生漫长的思想链，这是一种提高复杂任务绩效的方法。但是，开发这种模型的详细信息是有限的，尤其是对于可以以低资源语言生成轨迹的推理模型。台风T1提出了开放的努力，该努力通过利用开放数据集（而不是强化学习）利用监督的微调来介绍以更具成本效益的方式开发推理模型的细节。本文分享了有关合成数据生成和培训以及我们的数据集和模型权重的详细信息。此外，我们还提供了开发跨域概括的推理模型而获得的见解，并能够以泰语为例以低资源语言生成推理痕迹。我们希望这种开放的努力为该领域的进一步研究提供了基础。

Title: An Open Recipe: Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging

• Authors: Kunat Pipatanakul, Pittawat Taveekitworachai, Potsawee Manakul, Kasima Tharnpipitchai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09056
• Pdf URL: https://arxiv.org/pdf/2502.09056
• Copy Paste: [[2502.09056]] An Open Recipe: Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging(https://arxiv.org/abs/2502.09056)
• Keywords: language model, llm
• Abstract: This paper investigates data selection and model merging methodologies aimed at incorporating advanced reasoning capabilities such as those of DeepSeek R1 into language-specific large language models (LLMs), with a particular focus on the Thai LLM. Our goal is to enhance the reasoning capabilities of language-specific LLMs while maintaining their target language abilities. DeepSeek R1 excels in reasoning but primarily benefits high-resource languages such as English and Chinese. However, low-resource languages remain underserved due to the dominance of English-centric training data and model optimizations, which limit performance in these languages. This limitation results in unreliable code-switching and diminished effectiveness on tasks in low-resource languages. Meanwhile, local and regional LLM initiatives have attempted to bridge this gap by developing language-specific LLMs that focus on improving local linguistic fidelity. We demonstrate that, with only publicly available datasets and a computational budget of $120, it is possible to enhance the reasoning capabilities of language-specific LLMs to match the level of DeepSeek R1, without compromising their performance on target language tasks.
• 摘要：本文研究了旨在将诸如DeepSeek R1之类的先进推理功能（例如DeepSeek R1）纳入特定语言大型语言模型（LLMS）的数据选择和模型合并方法，并特别关注泰语LLM。我们的目标是在保持其目标语言能力的同时增强特定语言LLM的推理能力。 DeepSeek R1在推理方面表现出色，但主要受益于高源语言，例如英语和中文。但是，由于以英语为中心的培训数据和模型优化，低资源语言仍在服务不足，从而限制了这些语言的性能。此限制导致不可靠的代码转换和对低资源语言任务的有效性降低。同时，本地和区域LLM计划试图通过开发针对改善本地语言忠诚的语言特异性LLM来弥合这一差距。我们证明，只有公开可用的数据集和120美元的计算预算，可以增强特定语言LLM的推理能力，以匹配DeepSeek R1的水平，而不会损害其在目标语言任务上的绩效。

Title: Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables

• Authors: Xuzhao Geng, Haozhao Wang, Jun Wang, Wei Liu, Ruixuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09073
• Pdf URL: https://arxiv.org/pdf/2502.09073
• Copy Paste: [[2502.09073]] Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables(https://arxiv.org/abs/2502.09073)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) is a key technique for leveraging external knowledge and reducing hallucinations in large language models (LLMs). However, RAG still struggles to fully prevent hallucinated responses. To address this, it is essential to identify samples prone to hallucination or guide LLMs toward correct responses, which experts then annotate to develop high-quality datasets for refining LLMs. However, the growing scarcity of such datasets makes their creation challenging. This paper proposes using the vast amount of conversations from widespread LLM usage to build these datasets, training LLMs to avoid hallucination-prone questions while accurately responding to manageable ones. Given the impracticality of expert-annotating all conversation records, the paper introduces AL4RAG, which uses active learning to select the most suitable conversation samples for annotation, optimizing performance within an annotation budget. Additionally, recognizing that traditional active learning methods are not fully compatible with RAG due to unsuitable distance metrics, we develop a novel sample distance measurement for RAG active learning. Extensive experiments show that our method consistently outperforms baselines across multiple metrics.
• 摘要：检索增强的生成（RAG）是利用外部知识并减少大语言模型（LLMS）的幻觉的关键技术。但是，破布仍在努力完全防止幻觉的反应。为了解决这个问题，必须识别容易幻觉或指导LLMS正确响应的样品，然后专家注释以开发用于完善LLM的高质量数据集。但是，此类数据集的稀缺性日益严重，使它们的创造具有挑战性。本文建议使用广泛的LLM使用中的大量对话来构建这些数据集，培训LLM，以避免容易发生幻觉问题，同时准确地响应可管理的问题。鉴于对所有对话记录的专家通知的不切实际性，本文介绍了AL4RAG，该论文使用主动学习来选择注释最合适的对话样本，从而在注释预算中优化性能。此外，由于不合适的距离指标，我们认识到传统的主动学习方法与RAG并不完全兼容，因此我们为RAG主动学习开发了一种新颖的样本距离测量。广泛的实验表明，我们的方法始终超过多个指标的基准。

Title: CoSER: Coordinating LLM-Based Persona Simulation of Established Roles

• Authors: Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Wei Wang, Yanghua Xiao, Shuchang Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09082
• Pdf URL: https://arxiv.org/pdf/2502.09082
• Copy Paste: [[2502.09082]] CoSER: Coordinating LLM-Based Persona Simulation of Established Roles(https://arxiv.org/abs/2502.09082)
• Keywords: language model, gpt, llm, agent
• Abstract: Role-playing language agents (RPLAs) have emerged as promising applications of large language models (LLMs). However, simulating established characters presents a challenging task for RPLAs, due to the lack of authentic character datasets and nuanced evaluation methods using such data. In this paper, we present CoSER, a collection of a high-quality dataset, open models, and an evaluation protocol towards effective RPLAs of established characters. The CoSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts. Drawing from acting methodology, we introduce given-circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially portray multiple characters in book scenes. Using our dataset, we develop CoSER 8B and CoSER 70B, i.e., advanced open role-playing LLMs built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. Moreover, CoSER 70B exhibits state-of-the-art performance surpassing or matching GPT-4o on our evaluation and three existing benchmarks, i.e., achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respectively.
• 摘要：角色扮演语言代理（RPLAS）已成为大型语言模型（LLMS）的有前途的应用。但是，由于缺乏真实的字符数据集和使用此类数据的细微评估方法，模拟已建立的字符为RPLA提供了具有挑战性的任务。在本文中，我们介绍了Coser，这是一个高质量数据集的集合，开放模型以及针对已建立字符的有效RPLA的评估协议。 COSER数据集涵盖了771本著名书籍中的17,966个字符。它提供了真实的对话，具有现实世界中的复杂性，以及各种数据类型，例如对话设置，角色经验和内部思想。从Acting方法论中汲取灵感，我们介绍了用于培训和评估角色扮演LLM的赋予circumstance Acting，其中LLM在书籍场景中依次描绘了多个字符。使用我们的数据集，我们开发了COSER 8B和COSER 70B，即在Llama-3.1型号上构建的高级开放角色扮演LLM。广泛的实验证明了COSER数据集对RPLA培训，评估和检索的价值。此外，COSER 70B在我们的评估中表现出最先进的性能超过或匹配GPT-4O，并在Incharacter和Lifechoice基准测试中分别达到75.80％和93.47％的准确性。

Title: The influence of visual and linguistic cues on ignorance inference in Vision-Language Models (VLMs)

• Authors: Ye-eun Cho, Yunho Maeng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09120
• Pdf URL: https://arxiv.org/pdf/2502.09120
• Copy Paste: [[2502.09120]] The influence of visual and linguistic cues on ignorance inference in Vision-Language Models (VLMs)(https://arxiv.org/abs/2502.09120)
• Keywords: language model, gpt
• Abstract: This study explored how Vision-Language Models (VLMs) process ignorance implicatures with visual and linguistic cues. Particularly, we focused on the effects of contexts (precise and approximate contexts) and modifier types (bare numerals, superlative, and comparative modifiers), which were considered pragmatic and semantic factors respectively. Methodologically, we conducted a truth-value judgment task in visually grounded settings using GPT-4o and Gemini 1.5 Pro. The results indicate that while both models exhibited sensitivity to linguistic cues (modifier), they failed to process ignorance implicatures with visual cues (context) as humans do. Specifically, the influence of context was weaker and inconsistent across models, indicating challenges in pragmatic reasoning for VLMs. On the other hand, superlative modifiers were more strongly associated with ignorance implicatures as compared to comparative modifiers, supporting the semantic view. These findings highlight the need for further advancements in VLMs to process language-vision information in a context-dependent way to achieve human-like pragmatic inference.
• 摘要：这项研究探讨了视觉模型（VLM）如何与视觉和语言提示有关的无知含义。特别是，我们专注于上下文（精确和近似上下文）和修饰符类型（裸机，最高级和比较修饰符）的影响，这些效果分别被认为是务实和语义因素。从方法上讲，我们使用GPT-4O和Gemini 1.5 Pro在视觉扎根的环境中执行了真实值判断任务。结果表明，尽管这两个模型对语言提示（修饰符）都表现出敏感性，但他们没有像人类那样处理视觉提示（上下文）的无知含义。具体而言，环境的影响在模型之间较弱且不一致，表明VLMS的务实推理挑战。另一方面，与比较修饰符相比，最高级修饰符与无知的暗示更加密切相关，从而支持语义观点。这些发现凸显了在VLM中进一步进步以以上下文相关的方式处理语言视觉信息以实现人类务实的务实推论。

Title: A Novel Dialect-Aware Framework for the Classification of Arabic Dialects and Emotions

• Authors: Nasser A Alsadhan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09128
• Pdf URL: https://arxiv.org/pdf/2502.09128
• Copy Paste: [[2502.09128]] A Novel Dialect-Aware Framework for the Classification of Arabic Dialects and Emotions(https://arxiv.org/abs/2502.09128)
• Keywords: chat
• Abstract: Arabic is one of the oldest languages still in use today. As a result, several Arabic-speaking regions have developed dialects that are unique to them. Dialect and emotion recognition have various uses in Arabic text analysis, such as determining an online customer's origin based on their comments. Furthermore, intelligent chatbots that are aware of a user's emotions can respond appropriately to the user. Current research in emotion detection in the Arabic language lacks awareness of how emotions are exhibited in different dialects, which motivates the work found in this study. This research addresses the problems of dialect and emotion classification in Arabic. Specifically, this is achieved by building a novel framework that can identify and predict Arabic dialects and emotions from a given text. The framework consists of three modules: A text-preprocessing module, a classification module, and a clustering module with the novel capability of building new dialect-aware emotion lexicons. The proposed framework generated a new emotional lexicon for different dialects. It achieved an accuracy of 88.9% in classifying Arabic dialects, which outperforms the state-of-the-art results by 6.45 percentage points. Furthermore, the framework achieved 89.1-79% accuracy in detecting emotions in the Egyptian and Gulf dialects, respectively.
• 摘要：阿拉伯语是当今仍在使用的最古老的语言之一。结果，一些讲阿拉伯语的地区已经开发了它们独特的方言。方言和情感识别在阿拉伯文本分析中有各种用途，例如根据他们的评论来确定在线客户的起源。此外，了解用户情绪的智能聊天机器人可以对用户做出适当的反应。目前在阿拉伯语中进行情感检测的研究缺乏对不同方言中情绪表现方式的认识，这激发了本研究中发现的作品。这项研究解决了阿拉伯语中方言和情感分类的问题。具体而言，这是通过建立一个可以识别和预测给定文本的阿拉伯语方言和情感的新颖框架来实现的。该框架由三个模块组成：一个文本预处理模块，一个分类模块和一个集群模块，具有构建新的方言感应情感词典的新颖能力。拟议的框架为不同的方言产生了一种新的情感词典。它在分类阿拉伯语方言的准确度中达到了88.9％，这表现优于最先进的结果6.45个百分点。此外，该框架在分别检测埃及和海湾方言中的情绪时达到了89.1-79％的精度。

Title: Improving TCM Question Answering through Tree-Organized Self-Reflective Retrieval with LLMs

• Authors: Chang Liu, Ying Chang, Jianmin Li, Yiqian Qu, Yu Li, Lingyong Cao, Shuyuan Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09156
• Pdf URL: https://arxiv.org/pdf/2502.09156
• Copy Paste: [[2502.09156]] Improving TCM Question Answering through Tree-Organized Self-Reflective Retrieval with LLMs(https://arxiv.org/abs/2502.09156)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Objectives: Large language models (LLMs) can harness medical knowledge for intelligent question answering (Q&A), promising support for auxiliary diagnosis and medical talent cultivation. However, there is a deficiency of highly efficient retrieval-augmented generation (RAG) frameworks within the domain of Traditional Chinese Medicine (TCM). Our purpose is to observe the effect of the Tree-Organized Self-Reflective Retrieval (TOSRR) framework on LLMs in TCM Q&A tasks. Materials and Methods: We introduce the novel approach of knowledge organization, constructing a tree structure knowledge base with hierarchy. At inference time, our self-reflection framework retrieves from this knowledge base, integrating information across chapters. Questions from the TCM Medical Licensing Examination (MLE) and the college Classics Course Exam (CCE) were randomly selected as benchmark datasets. Results: By coupling with GPT-4, the framework can improve the best performance on the TCM MLE benchmark by 19.85% in absolute accuracy, and improve recall accuracy from 27% to 38% on CCE datasets. In manual evaluation, the framework improves a total of 18.52 points across dimensions of safety, consistency, explainability, compliance, and coherence. Conclusion: The TOSRR framework can effectively improve LLM's capability in Q&A tasks of TCM.
• 摘要：目的：大型语言模型（LLM）可以利用医学知识来解决智能问题（问答），有望支持辅助诊断和医疗人才培养。但是，在传统中医（TCM）领域内，缺乏高效的检索生成（RAG）框架。我们的目的是观察TCM Q＆A任务中LLMS的树型自我反射检索（TOSRR）框架的影响。材料和方法：我们介绍了知识组织的新颖方法，并用层次结构构建树木结构知识基础。在推理时，我们的自我反思框架从这个知识库中检索，从章节中整合了信息。 TCM医学许可考试（MLE）和大学经典课程考试（CCE）的问题被随机选择为基准数据集。结果：通过与GPT-4结合，该框架可以使TCM MLE基准的最佳性能在绝对准确性上提高19.85％，并将召回精度从CCE数据集提高到38％。在手动评估中，该框架在安全性，一致性，解释性，合规性和连贯性方面总共提高了18.52点。结论：TOSRR框架可以有效地提高LLM在TCM的问答任务中的功能。

Title: RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation

• Authors: Changzhi Zhou, Xinyu Zhang, Dandan Song, Xiancai Chen, Wanli Gu, Huipeng Ma, Yuhang Tian, Mengdi Zhang, Linmei Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09183
• Pdf URL: https://arxiv.org/pdf/2502.09183
• Copy Paste: [[2502.09183]] RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation(https://arxiv.org/abs/2502.09183)
• Keywords: language model, llm
• Abstract: Code generation has attracted increasing attention with the rise of Large Language Models (LLMs). Many studies have developed powerful code LLMs by synthesizing code-related instruction data and applying supervised fine-tuning. However, these methods are limited by teacher model distillation and ignore the potential of iterative refinement by self-generated code. In this paper, we propose Adaptive Critique Refinement (ACR), which enables the model to refine itself by self-generated code and external critique, rather than directly imitating the code responses of the teacher model. Concretely, ACR includes a composite scoring system with LLM-as-a-Judge to evaluate the quality of code responses and a selective critique strategy with LLM-as-a-Critic to critique self-generated low-quality code responses. We develop the RefineCoder series by iteratively applying ACR, achieving continuous performance improvement on multiple code generation benchmarks. Compared to the baselines of the same size, our proposed RefineCoder series can achieve comparable or even superior performance using less data.
• 摘要：随着大语言模型（LLM）的兴起，代码生成引起了越来越多的关注。许多研究通过合成与代码相关的指令数据并应用监督的微调来开发强大的代码LLM。但是，这些方法受到教师模型蒸馏的限制，而忽略了通过自我生成的代码的迭代细化潜力。在本文中，我们提出了自适应批判性改进（ACR），这使该模型能够通过自我生成的代码和外部批评来完善自己，而不是直接模仿教师模型的代码响应。具体而言，ACR包括一个具有LLM-AS-A-A-Gudge的综合评分系统，以评估代码响应的质量和具有LLM-AS-A-A-Critic的选择性评论策略，以批评自我生成的低质量代码响应。我们通过迭代应用ACR来开发RefineCoder系列，从而在多个代码生成基准上实现持续的性能改进。与相同尺寸的基线相比，我们提出的炼油编码器系列可以使用较少的数据实现可比甚至卓越的性能。

Title: Matina: A Large-Scale 73B Token Persian Text Corpus

• Authors: Sara Bourbour Hosseinbeigi, Fatemeh Taherinezhad, Heshaam Faili, Hamed Baghbani, Fatemeh Nadi, Mostafa Amiri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09188
• Pdf URL: https://arxiv.org/pdf/2502.09188
• Copy Paste: [[2502.09188]] Matina: A Large-Scale 73B Token Persian Text Corpus(https://arxiv.org/abs/2502.09188)
• Keywords: language model, llm
• Abstract: Text corpora are essential for training models used in tasks like summarization, translation, and large language models (LLMs). While various efforts have been made to collect monolingual and multilingual datasets in many languages, Persian has often been underrepresented due to limited resources for data collection and preprocessing. Existing Persian datasets are typically small and lack content diversity, consisting mainly of weblogs and news articles. This shortage of high-quality, varied data has slowed the development of NLP models and open-source LLMs for Persian. Since model performance depends heavily on the quality of training data, we address this gap by introducing the Matina corpus, a new Persian dataset of 72.9B tokens, carefully preprocessed and deduplicated to ensure high data quality. We further assess its effectiveness by training and evaluating transformer-based models on key NLP tasks. Both the dataset and preprocessing codes are publicly available, enabling researchers to build on and improve this resource for future Persian NLP advancements.
• 摘要：文本语料库对于诸如摘要，翻译和大型语言模型（LLMS）等任务中使用的培训模型至关重要。尽管以多种语言收集单语和多语言数据集的各种努力，但由于数据收集和预处理资源有限，波斯人通常对波斯人的代表不足。现有的波斯数据集通常很小，并且缺乏内容多样性，主要由博客和新闻文章组成。高质量，各种数据的短缺减慢了波斯语NLP模型和开源LLM的开发。由于模型性能在很大程度上取决于培训数据的质量，因此我们通过引入Matina Corpus（一种新的波斯语数据集的72.9B代币，仔细预处理并进行了重复删除以确保高数据质量）来解决这一差距。我们通过培训和评估关键NLP任务上的基于变压器的模型进一步评估其有效性。数据集和预处理代码均可公开使用，使研究人员能够在未来的波斯NLP进步上依靠并改善此资源。

Title: Thinking beyond the anthropomorphic paradigm benefits LLM research

• Authors: Lujain Ibrahim, Myra Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09192
• Pdf URL: https://arxiv.org/pdf/2502.09192
• Copy Paste: [[2502.09192]] Thinking beyond the anthropomorphic paradigm benefits LLM research(https://arxiv.org/abs/2502.09192)
• Keywords: language model, llm
• Abstract: Anthropomorphism, or the attribution of human traits to technology, is an automatic and unconscious response that occurs even in those with advanced technical expertise. In this position paper, we analyze hundreds of thousands of computer science research articles from the past decade and present empirical evidence of the prevalence and growth of anthropomorphic terminology in research on large language models (LLMs). This terminology reflects deeper anthropomorphic conceptualizations which shape how we think about and conduct LLM research. We argue these conceptualizations may be limiting, and that challenging them opens up new pathways for understanding and improving LLMs beyond human analogies. To illustrate this, we identify and analyze five core anthropomorphic assumptions shaping prominent methodologies across the LLM development lifecycle, from the assumption that models must use natural language for reasoning tasks to the assumption that model capabilities should be evaluated through human-centric benchmarks. For each assumption, we demonstrate how non-anthropomorphic alternatives can open new directions for research and development.
• 摘要：拟人化或人类特征对技术的归因是一种自动和无意识的反应，即使在具有先进技术专长的人中也会发生。在该职位上，我们分析了过去十年来数十万计算机科学研究文章，并提供了拟人化术语在大语言模型（LLMS）中的流行和增长的经验证据。该术语反映了更深层次的拟人化概念化，从而影响了我们对LLM研究的看法和进行LLM研究。我们认为这些概念化可能是限制的，而挑战它们为理解和改善人类类似物以外的LLM的新途径开辟了新的途径。为了说明这一点，我们识别和分析了五个核心拟人化假设，从LLM开发生命周期中塑造了突出方法，从假设必须使用自然语言来推理任务到应通过以人为本的基准来评估模型能力的假设。对于每个假设，我们都证明了非拟人化替代方案如何为研发打开新的方向。

Title: SparQLe: Speech Queries to Text Translation Through LLMs

• Authors: Amirbek Djanibekov, Hanan Aldarmaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09284
• Pdf URL: https://arxiv.org/pdf/2502.09284
• Copy Paste: [[2502.09284]] SparQLe: Speech Queries to Text Translation Through LLMs(https://arxiv.org/abs/2502.09284)
• Keywords: language model, llm
• Abstract: With the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding. This study introduces a novel approach that leverages self-supervised speech representations in combination with instruction-tuned LLMs for speech-to-text translation. The proposed approach leverages a modality adapter to align extracted speech features with instruction-tuned LLMs using English-language data. Our experiments demonstrate that this method effectively preserves the semantic content of the input speech and serves as an effective bridge between self-supervised speech models and instruction-tuned LLMs, offering a promising solution for various speech understanding applications.
• 摘要：随着大语言模型（LLM）的不断增长的影响，对将语音表述与它们整合在一起的兴趣越来越多，以实现更多无缝的多模式处理和语音理解。这项研究介绍了一种新颖的方法，该方法通过教学调节的LLM结合语音到文本翻译来利用自我监督的语音表示形式。所提出的方法利用模式适配器使用英语数据将提取的语音特征与指令调整的LLMS对齐。我们的实验表明，该方法有效地保留了输入语音的语义内容，并作为自我监督的语音模型和教学调节的LLM之间的有效桥梁，为各种语音理解应用程序提供了有希望的解决方案。

Title: When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models

• Authors: Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09307
• Pdf URL: https://arxiv.org/pdf/2502.09307
• Copy Paste: [[2502.09307]] When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models(https://arxiv.org/abs/2502.09307)
• Keywords: language model, llm
• Abstract: Modern Large Language Models (LLMs) have shown human-like abilities in many language tasks, sparking interest in comparing LLMs' and humans' language processing. In this paper, we conduct a detailed comparison of the two on a sentence comprehension task using garden-path constructions, which are notoriously challenging for humans. Based on psycholinguistic research, we formulate hypotheses on why garden-path sentences are hard, and test these hypotheses on human participants and a large suite of LLMs using comprehension questions. Our findings reveal that both LLMs and humans struggle with specific syntactic complexities, with some models showing high correlation with human comprehension. To complement our findings, we test LLM comprehension of garden-path constructions with paraphrasing and text-to-image generation tasks, and find that the results mirror the sentence comprehension question results, further validating our findings on LLM understanding of these constructions.
• 摘要：现代大型语言模型（LLM）在许多语言任务中表现出类似人类的能力，引发了对比较LLM和人类语言处理的兴趣。在本文中，我们使用花园路径结构对两者进行了详细的比较，这对人类来说是充满挑战的。基于心理语言学研究，我们提出假设关于为什么园艺句子很难的假设，并使用理解问题对人类参与者和大型LLM的这些假设进行测试。我们的发现表明，LLM和人类都与特定的句法复杂性斗争，一些模型表现出与人类理解的高度相关性。为了补充我们的发现，我们通过释义和文本到图像生成任务来测试LLM对花园条件结构的理解，并发现结果反映了句子理解问题的结果，进一步验证了我们对LLM对这些结构的理解的发现。

Title: A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis

• Authors: Kentaro Imajo, Masanori Hirano, Shuji Suzuki, Hiroaki Mikami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09316
• Pdf URL: https://arxiv.org/pdf/2502.09316
• Copy Paste: [[2502.09316]] A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis(https://arxiv.org/abs/2502.09316)
• Keywords: language model, gpt, llm
• Abstract: Evaluating the open-ended text generation of large language models (LLMs) is challenging because of the lack of a clear ground truth and the high cost of human or LLM-based assessments. We propose a novel benchmark that evaluates LLMs using n-gram statistics and rules, without relying on human judgement or LLM-as-a-judge approaches. Using 50 question and reference answer sets, we introduce three new metrics based on n-grams and rules: Fluency, Truthfulness, and Helpfulness. Our benchmark strongly correlates with GPT-4o-based evaluations while requiring significantly fewer computational resources, demonstrating its effectiveness as a scalable alternative for assessing LLMs' open-ended generation capabilities.
• 摘要：由于缺乏明确的基础真理和人类或基于LLM的评估的高成本，评估大语言模型（LLM）的开放式文本生成（LLM）的挑战是具有挑战性的。我们提出了一种新颖的基准测试，该基准可以使用N-Gram统计数据和规则评估LLM，而无需依赖人类判断或法学判断方法。使用50个问题和参考答案集，我们基于n-gram和规则介绍了三个新的指标：流利，真实性和乐于助人。我们的基准测试与基于GPT-4O的评估密切相关，同时需要更少的计算资源，并证明其有效性是评估LLMS开放式生成能力的可扩展替代方法。

Title: Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs

• Authors: Itai Mondshine, Tzuf Paz-Argaman, Reut Tsarfaty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09331
• Pdf URL: https://arxiv.org/pdf/2502.09331
• Copy Paste: [[2502.09331]] Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs(https://arxiv.org/abs/2502.09331)
• Keywords: language model, llm, prompt
• Abstract: Despite advances in the multilingual capabilities of Large Language Models (LLMs) across diverse tasks, English remains the dominant language for LLM research and development. So, when working with a different language, this has led to the widespread practice of pre-translation, i.e., translating the task prompt into English before inference. Selective pre-translation, a more surgical approach, focuses on translating specific prompt components. However, its current use is sporagic and lacks a systematic research foundation. Consequently, the optimal pre-translation strategy for various multilingual settings and tasks remains unclear. In this work, we aim to uncover the optimal setup for pre-translation by systematically assessing its use. Specifically, we view the prompt as a modular entity, composed of four functional parts: instruction, context, examples, and output, either of which could be translated or not. We evaluate pre-translation strategies across 35 languages covering both low and high-resource languages, on various tasks including Question Answering (QA), Natural Language Inference (NLI), Named Entity Recognition (NER), and Abstractive Summarization. Our experiments show the impact of factors as similarity to English, translation quality and the size of pre-trained data, on the model performance with pre-translation. We suggest practical guidelines for choosing optimal strategies in various multilingual settings.
• 摘要：尽管大型语言模型（LLM）在各种任务中的多语言能力取得了进步，但英语仍然是LLM研发的主要语言。因此，在使用不同语言的工作时，这导致了预先译本的广泛实践，即在推断之前将任务提示转换为英语。选择性预译，一种更具外科手术的方法，重点是翻译特定的提示组件。但是，其目前的用途是孢子虫，缺乏系统的研究基金会。因此，对于各种多语言设置和任务的最佳翻译策略尚不清楚。在这项工作中，我们旨在通过系统地评估其使用来揭示用于预先翻译的最佳设置。具体来说，我们将提示视为一个模块化实体，由四个功能部分组成：指令，上下文，示例和输出，无论是否可以翻译。我们评估了涵盖低资源和高资源语言的35种语言的预先翻译策略，包括各种任务，包括问答（QA），自然语言推论（NLI），命名实体识别（NER）和抽象摘要。我们的实验表明，因素与英语相似，翻译质量和预训练数据的大小，对模型性能进行预译本的影响。我们建议在各种多语言环境中选择最佳策略的实用准则。

Title: Truth Knows No Language: Evaluating Truthfulness Beyond English

• Authors: Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria De Dios Flores, Rodrigo Agerri
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.09387
• Pdf URL: https://arxiv.org/pdf/2502.09387
• Copy Paste: [[2502.09387]] Truth Knows No Language: Evaluating Truthfulness Beyond English(https://arxiv.org/abs/2502.09387)
• Keywords: language model, llm
• Abstract: We introduce a professionally translated extension of the TruthfulQA benchmark designed to evaluate truthfulness in Basque, Catalan, Galician, and Spanish. Truthfulness evaluations of large language models (LLMs) have primarily been conducted in English. However, the ability of LLMs to maintain truthfulness across languages remains under-explored. Our study evaluates 12 state-of-the-art open LLMs, comparing base and instruction-tuned models using human evaluation, multiple-choice metrics, and LLM-as-a-Judge scoring. Our findings reveal that, while LLMs perform best in English and worst in Basque (the lowest-resourced language), overall truthfulness discrepancies across languages are smaller than anticipated. Furthermore, we show that LLM-as-a-Judge correlates more closely with human judgments than multiple-choice metrics, and that informativeness plays a critical role in truthfulness assessment. Our results also indicate that machine translation provides a viable approach for extending truthfulness benchmarks to additional languages, offering a scalable alternative to professional translation. Finally, we observe that universal knowledge questions are better handled across languages than context- and time-dependent ones, highlighting the need for truthfulness evaluations that account for cultural and temporal variability. Dataset and code are publicly available under open licenses.
• 摘要：我们介绍了旨在评估巴斯克，加泰罗尼亚，加利西亚语和西班牙语的真实性的真实基准的专业翻译扩展。大语言模型（LLM）的真实性评估主要是用英语进行的。但是，LLM在跨语言保持真实性的能力仍未得到探索。我们的研究评估了12个最先进的开放式LLM，并使用人类评估，多项选择指标和LLM-AS-A-A-A-A-Gudge评分比较了基础和指导调节模型。我们的发现表明，尽管LLM在巴斯克地区的英语和最差（资源最低的语言）中表现最佳，但语言之间的整体真实性差异比预期的要小。此外，我们表明，LLM-AS-A-A-Gudge与人类判断更加紧密地相关，而不是多选择指标，并且信息性在真实性评估中起着至关重要的作用。我们的结果还表明，机器翻译为将真实性基准扩展到其他语言提供了可行的方法，为专业翻译提供了可扩展的替代方法。最后，我们观察到，普遍的知识问题比上下文和时间依赖语言更好地处理了跨语言，这突出了对文化和时间变异性的真实性评估的需求。数据集和代码在公开许可下公开获得。

Title: SQuARE: Sequential Question Answering Reasoning Engine for Enhanced Chain-of-Thought in Large Language Models

• Authors: Daniel Fleischer, Moshe Berchansky, Gad Markovits, Moshe Wasserblat
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09390
• Pdf URL: https://arxiv.org/pdf/2502.09390
• Copy Paste: [[2502.09390]] SQuARE: Sequential Question Answering Reasoning Engine for Enhanced Chain-of-Thought in Large Language Models(https://arxiv.org/abs/2502.09390)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: In the rapidly evolving field of Natural Language Processing, Large Language Models (LLMs) are tasked with increasingly complex reasoning challenges. Traditional methods like chain-of-thought prompting have shown promise but often fall short in fully leveraging a model's reasoning capabilities. This paper introduces SQuARE (Sequential Question Answering Reasoning Engine), a novel prompting technique designed to improve reasoning through a self-interrogation paradigm. Building upon CoT frameworks, SQuARE prompts models to generate and resolve multiple auxiliary questions before tackling the main query, promoting a more thorough exploration of various aspects of a topic. Our expansive evaluations, conducted with Llama 3 and GPT-4o models across multiple question-answering datasets, demonstrate that SQuARE significantly surpasses traditional CoT prompts and existing rephrase-and-respond methods. By systematically decomposing queries, SQuARE advances LLM capabilities in reasoning tasks. The code is publicly available at this https URL.
• 摘要：在自然语言处理的快速发展领域中，大型语言模型（LLM）的任务是越来越复杂的推理挑战。诸如经过思想链条的提示之类的传统方法已经表现出了希望，但在完全利用模型的推理能力方面通常不足。本文介绍了Square（顺序回答推理引擎），这是一种新颖的提示技术，旨在通过自我插入范式来改善推理。 Square在COT框架的基础上，提示模型在解决主查询之前生成和解决多个辅助问题，从而更彻底地探索了主题的各个方面。我们在多个提问数据集中使用Llama 3和GPT-4O模型进行的广泛评估表明，Square显着超过了传统的COT提示和现有的rephrase and Respond方法。通过系统地分解查询，Square在推理任务中提高了LLM功能。该代码在此HTTPS URL上公开可用。

Title: Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?

• Authors: Takumi Goto, Yusuke Sakai, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09416
• Pdf URL: https://arxiv.org/pdf/2502.09416
• Copy Paste: [[2502.09416]] Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?(https://arxiv.org/abs/2502.09416)
• Keywords: gpt
• Abstract: One of the goals of automatic evaluation metrics in grammatical error correction (GEC) is to rank GEC systems such that it matches human preferences. However, current automatic evaluations are based on procedures that diverge from human evaluation. Specifically, human evaluation derives rankings by aggregating sentence-level relative evaluation results, e.g., pairwise comparisons, using a rating algorithm, whereas automatic evaluation averages sentence-level absolute scores to obtain corpus-level scores, which are then sorted to determine rankings. In this study, we propose an aggregation method for existing automatic evaluation metrics which aligns with human evaluation methods to bridge this gap. We conducted experiments using various metrics, including edit-based metrics, $n$-gram based metrics, and sentence-level metrics, and show that resolving the gap improves results for the most of metrics on the SEEDA benchmark. We also found that even BERT-based metrics sometimes outperform the metrics of GPT-4. We publish our unified implementation of the metrics and meta-evaluations.
• 摘要：语法误差校正（GEC）中自动评估指标的目标之一是对GEC系统进行排名，以使其与人类的偏好相匹配。但是，当前的自动评估是基于与人类评估不同的程序。具体而言，人类评估通过汇总句子级相对评估结果（例如，使用评级算法）来得出排名，而对自动评估的平均句子级绝对得分来获得语料库级别的得分，然后将其排序以确定排名。在这项研究中，我们提出了一种用于现有自动评估指标的聚合方法，该方法与人类评估方法一致以弥合这一差距。我们使用各种指标进行了实验，包括基于编辑的指标，基于$ n $的指标和句子级指标，并表明解决差距可以改善Seeda基准上的大多数指标的结果。我们还发现，即使是基于BERT的指标，有时甚至优于GPT-4的指标。我们发布了指标和元评估的统一实施。

Title: On multi-token prediction for efficient LLM inference

• Authors: Somesh Mehra, Javier Alonso Garcia, Lukas Mauch
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09419
• Pdf URL: https://arxiv.org/pdf/2502.09419
• Copy Paste: [[2502.09419]] On multi-token prediction for efficient LLM inference(https://arxiv.org/abs/2502.09419)
• Keywords: llm, prompt
• Abstract: We systematically investigate multi-token prediction (MTP) capabilities within LLMs pre-trained for next-token prediction (NTP). We first show that such models inherently possess MTP capabilities via numerical marginalization over intermediate token probabilities, though performance is data-dependent and improves with model scale. Furthermore, we explore the challenges of integrating MTP heads into frozen LLMs and find that their hidden layers are strongly specialized for NTP, making adaptation non-trivial. Finally, we show that while joint training of MTP heads with the backbone improves performance, it cannot fully overcome this barrier, prompting further research in this direction. Our findings provide a deeper understanding of MTP applied to pretrained LLMs, informing strategies for accelerating inference through parallel token prediction.
• 摘要：我们系统地研究了预先训练的下一个预测（NTP）的LLMS中的多型预测（MTP）功能。我们首先表明，这种模型固有地通过中间令牌概率通过数值边缘化具有MTP功能，尽管性能取决于数据依赖性并且随模型量表而改善。此外，我们探讨了将MTP头部整合到冷冻LLM中的挑战，并发现它们的隐藏层非常专门用于NTP，从而使适应性不足。最后，我们表明，尽管对骨干的MTP头部联合训练可以提高性能，但它无法完全克服这一障碍，从而促使进一步研究这一方向。我们的发现提供了对应用于验证的LLM的MTP的更深入的了解，从而为通过平行令牌预测提供了加速推理的策略。

Title: The Multilingual Mind : A Survey of Multilingual Reasoning in Language Models

• Authors: Akash Ghosh, Debayan Datta, Sriparna Saha, Chirag Agarwal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09457
• Pdf URL: https://arxiv.org/pdf/2502.09457
• Copy Paste: [[2502.09457]] The Multilingual Mind : A Survey of Multilingual Reasoning in Language Models(https://arxiv.org/abs/2502.09457)
• Keywords: language model
• Abstract: While reasoning and multilingual capabilities in Language Models (LMs) have achieved remarkable progress in recent years, their integration into a unified paradigm, multilingual reasoning, is at a nascent stage. Multilingual reasoning requires language models to handle logical reasoning across languages while addressing misalignment, biases, and challenges in low-resource settings. This survey provides the first in-depth review of multilingual reasoning in LMs. In this survey, we provide a systematic overview of existing methods that leverage LMs for multilingual reasoning, specifically outlining the challenges, motivations, and foundational aspects of applying language models to reason across diverse languages. We provide an overview of the standard data resources used for training multilingual reasoning in LMs and the evaluation benchmarks employed to assess their multilingual capabilities. Next, we analyze various state-of-the-art methods and their performance on these benchmarks. Finally, we explore future research opportunities to improve multilingual reasoning in LMs, focusing on enhancing their ability to handle diverse languages and complex reasoning tasks.
• 摘要：尽管近年来，语言模型（LMS）中的推理和多语言能力取得了显着的进步，但它们融入统一的范式，多语言推理却处于新生阶段。多语言推理需要语言模型来处理跨语言的逻辑推理，同时解决低资源设置中的未对准，偏见和挑战。这项调查提供了对LMS多语言推理的首次深入评论。在这项调查中，我们提供了一个系统的概述，该方法利用LMS用于多语言推理，特别是概述了将语言模型应用于跨不同语言推理的挑战，动机和基础方面。我们概述了用于培训LMS中多语言推理的标准数据资源以及用于评估其多语言能力的评估基准。接下来，我们分析各种最新方法及其在这些基准上的性能。最后，我们探索了未来的研究机会，以改善LMS中的多语言推理，重点是增强其处理各种语言和复杂推理任务的能力。

Title: Objective quantification of mood states using large language models

• Authors: Jakub Onysk, Quentin Huys
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09487
• Pdf URL: https://arxiv.org/pdf/2502.09487
• Copy Paste: [[2502.09487]] Objective quantification of mood states using large language models(https://arxiv.org/abs/2502.09487)
• Keywords: language model, llm, prompt
• Abstract: Emotional states influence human behaviour and cognition, leading to diverse thought trajectories. Similarly, Large Language Models (LLMs) showcase an excellent level of response consistency across wide-ranging contexts (prompts). We leverage these parallels to establish a framework for quantifying mental states. Our approach utilises self-report questionnaires that reliably assess these states due to their inherent sensitivity to patterns of co-occurring responses. Specifically, we recruited a large sample of participants (N=422) to investigate how well an LLM (Mistral-7B-OpenOrca) quantifies a heterogenous set of depressive mood states measured with participants' open-ended responses to a depression questionnaire. We show LLM responses to held-out multiple-choice questions, given participants' open-ended answers, correlate strongly (r: 0.52-0.84) with true questionnaire scores, demonstrating LLM's generalisation from mood representations. We explore a link between these representations and factor analysis. Using ridge regression, we find depression-related subspaces within LLM hidden states. We show these subspaces to be predictive of participants' "Depression" and "Somatic & Emotional Distress" factor scores, as well as suicidality severity. Overall, LLMs can provide quantitative measures of mental states. The reliability of these hinges upon how informative the questions we ask participants are. Used correctly, this approach could supplement mental state assessment in a variety of settings.
• 摘要：情绪状态会影响人类的行为和认知，从而导致各种思想轨迹。同样，大型语言模型（LLMS）在广泛的上下文（提示）上展示了出色的响应一致性。我们利用这些相似之处建立一个量化精神状态的框架。我们的方法利用自我报告问卷，这些问卷可可靠地评估这些状态，因为它们对同时发生反应的模式的固有敏感性。具体而言，我们招募了大量参与者（n = 422），以研究LLM（Mistral-7b-Openorca）如何量化一组异质的抑郁情绪状态，这些抑郁情绪状态是通过参与者对抑郁症问卷的开放式回答来衡量的。鉴于参与者的开放式答案，我们表明了对出局多项选择问题的回答，并与真正的问卷分数密切相关（R：0.52-0.84），这表明了LLM来自情绪表示的概括。我们探索这些表示形式与因子分析之间的联系。使用脊回归，我们发现LLM隐藏状态中与抑郁症相关的子空间。我们表明这些子空间可以预测参与者的“抑郁症”，“躯体和情绪困扰”因素得分以及自杀性严重程度。总体而言，LLM可以提供精神状态的定量度量。这些取决于我们问参与者的问题的信息。正确使用的是，这种方法可以在各种环境中补充精神状态评估。

Title: Improve LLM-based Automatic Essay Scoring with Linguistic Features

• Authors: Zhaoyi Joey Hou, Alejandro Ciuba, Xiang Lorraine Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09497
• Pdf URL: https://arxiv.org/pdf/2502.09497
• Copy Paste: [[2502.09497]] Improve LLM-based Automatic Essay Scoring with Linguistic Features(https://arxiv.org/abs/2502.09497)
• Keywords: language model, llm, prompt
• Abstract: Automatic Essay Scoring (AES) assigns scores to student essays, reducing the grading workload for instructors. Developing a scoring system capable of handling essays across diverse prompts is challenging due to the flexibility and diverse nature of the writing task. Existing methods typically fall into two categories: supervised feature-based approaches and large language model (LLM)-based methods. Supervised feature-based approaches often achieve higher performance but require resource-intensive training. In contrast, LLM-based methods are computationally efficient during inference but tend to suffer from lower performance. This paper combines these approaches by incorporating linguistic features into LLM-based scoring. Experimental results show that this hybrid method outperforms baseline models for both in-domain and out-of-domain writing prompts.
• 摘要：自动论文评分（AES）为学生论文分配了分数，从而减少了教师的评分工作量。由于写作任务的灵活性和多样性，开发能够在各种提示中处理论文的评分系统具有挑战性。现有方法通常分为两类：基于特征的方法和基于大型语言模型（LLM）的方法。基于特征的方法通常可以实现更高的性能，但需要进行资源密集型培训。相比之下，基于LLM的方法在推断过程中是计算上有效的，但倾向于遭受较低的性能。本文通过将语言特征纳入基于LLM的评分来结合这些方法。实验结果表明，这种混合方法的表现优于内域和室外写作提示的基线模型。

Title: Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages

• Authors: Shreyan Biswas, Alexander Erlei, Ujwal Gadiraju
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2502.09532
• Pdf URL: https://arxiv.org/pdf/2502.09532
• Copy Paste: [[2502.09532]] Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages(https://arxiv.org/abs/2502.09532)
• Keywords: language model, llm, agent
• Abstract: Recent advances in generative AI have precipitated a proliferation of novel writing assistants. These systems typically rely on multilingual large language models (LLMs), providing globalized workers the ability to revise or create diverse forms of content in different languages. However, there is substantial evidence indicating that the performance of multilingual LLMs varies between languages. Users who employ writing assistance for multiple languages are therefore susceptible to disparate output quality. Importantly, recent research has shown that people tend to generalize algorithmic errors across independent tasks, violating the behavioral axiom of choice independence. In this paper, we analyze whether user utilization of novel writing assistants in a charity advertisement writing task is affected by the AI's performance in a second language. Furthermore, we quantify the extent to which these patterns translate into the persuasiveness of generated charity advertisements, as well as the role of peoples' beliefs about LLM utilization in their donation choices. Our results provide evidence that writers who engage with an LLM-based writing assistant violate choice independence, as prior exposure to a Spanish LLM reduces subsequent utilization of an English LLM. While these patterns do not affect the aggregate persuasiveness of the generated advertisements, people's beliefs about the source of an advertisement (human versus AI) do. In particular, Spanish-speaking female participants who believed that they read an AI-generated advertisement strongly adjusted their donation behavior downwards. Furthermore, people are generally not able to adequately differentiate between human-generated and LLM-generated ads. Our work has important implications for the design, development, integration, and adoption of multilingual LLMs as assistive agents -- particularly in writing tasks.
• 摘要：生成AI的最新进展引起了新的写作助手的扩散。这些系统通常依靠多语言的大语言模型（LLM），为全球化的工人提供了以不同语言修改或创建各种形式的内容的能力。但是，有大量证据表明，多语言LLM的性能在语言之间有所不同。因此，使用多种语言提供书面帮助的用户容易受到不同的产出质量。重要的是，最近的研究表明，人们倾向于跨越独立任务的算法错误，违反了选择独立性的行为公理。在本文中，我们分析了用户在慈善广告写作任务中对新颖助手的利用是否受第二语言的AI表现的影响。此外，我们量化了这些模式转化为产生的慈善广告的说服力的程度，以及人们对LLM利用率的信念在其捐赠选择中的作用。我们的结果提供了证据表明，与基于LLM的写作助手互动的作家违反了选择独立性，因为事先接触西班牙LLM会减少随后的英语LLM利用。尽管这些模式不会影响生成的广告的总体说服力，但人们对广告来源（人类与AI）的信念。特别是，讲西班牙语的女性参与者认为自己阅读了AI生成的广告，强烈调整了她们的捐赠行为。此外，人们通常无法充分区分人类生成和LLM生成的广告。我们的工作对多语言LLM作为辅助代理的设计，开发，整合和采用具有重要意义，尤其是在编写任务方面。

Title: Zero-shot generation of synthetic neurosurgical data with large language models

• Authors: Austin A. Barr, Eddie Guo, Emre Sezgin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09566
• Pdf URL: https://arxiv.org/pdf/2502.09566
• Copy Paste: [[2502.09566]] Zero-shot generation of synthetic neurosurgical data with large language models(https://arxiv.org/abs/2502.09566)
• Keywords: language model, gpt, llm
• Abstract: Clinical data is fundamental to advance neurosurgical research, but access is often constrained by data availability, small sample sizes, privacy regulations, and resource-intensive preprocessing and de-identification procedures. Synthetic data offers a potential solution to challenges associated with accessing and using real-world data (RWD). This study aims to evaluate the capability of zero-shot generation of synthetic neurosurgical data with a large language model (LLM), GPT-4o, by benchmarking with the conditional tabular generative adversarial network (CTGAN). Synthetic datasets were compared to real-world neurosurgical data to assess fidelity (means, proportions, distributions, and bivariate correlations), utility (ML classifier performance on RWD), and privacy (duplication of records from RWD). The GPT-4o-generated datasets matched or exceeded CTGAN performance, despite no fine-tuning or access to RWD for pre-training. Datasets demonstrated high univariate and bivariate fidelity to RWD without directly exposing any real patient records, even at amplified sample size. Training an ML classifier on GPT-4o-generated data and testing on RWD for a binary prediction task showed an F1 score (0.706) with comparable performance to training on the CTGAN data (0.705) for predicting postoperative functional status deterioration. GPT-4o demonstrated a promising ability to generate high-fidelity synthetic neurosurgical data. These findings also indicate that data synthesized with GPT-4o can effectively augment clinical data with small sample sizes, and train ML models for prediction of neurosurgical outcomes. Further investigation is necessary to improve the preservation of distributional characteristics and boost classifier performance.
• 摘要：临床数据对于推进神经外科研究至关重要，但是访问通常受数据可用性，小样本量，隐私法规以及资源密集的预处理和去识别程序的限制。合成数据为与访问和使用现实世界数据（RWD）相关的挑战提供了潜在的解决方案。这项研究旨在通过用条件表格表的生成对抗网络（CTGAN）基准测试，通过大语言模型（LLM），GPT-4O评估合成神经外科数据的零发出能力。将合成数据集与现实世界的神经外科数据进行了比较，以评估保真度（均值，比例，分布和双变量相关性），实用程序（RWD上的ML分类器性能）和隐私（RWD的记录重复）。尽管没有微调或访问RWD进行预训练，但GPT-4O生成的数据集匹配或超过CTGAN性能。数据集证明了对RWD的高单变量和双变量保真度，而无需直接暴露任何实际患者记录，即使在放大样本量下也是如此。培训ML分类器对GPT-4O生成的数据和对RWD进行二进制预测任务的测试显示出F1分数（0.706），其性能与CTGAN数据（0.705）的培训相当，以预测术后功能状态的恶化。 GPT-4O证明了产生高保真合成神经外科数据的有希望的能力。这些发现还表明，与GPT-4O合成的数据可以有效地增加具有较小样本量的临床数据，并训练ML模型来预测神经外科结果。需要进一步研究，以改善分布特性和提高分类器性能的保存。

Title: MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing

• Authors: Vlad Andrei Negru, Robert Vacareanu, Camelia Lemnaru, Mihai Surdeanu, Rodica Potolea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09567
• Pdf URL: https://arxiv.org/pdf/2502.09567
• Copy Paste: [[2502.09567]] MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing(https://arxiv.org/abs/2502.09567)
• Keywords: language model
• Abstract: We introduce MorphNLI, a modular step-by-step approach to natural language inference (NLI). When classifying the premise-hypothesis pairs into {entailment, contradiction, neutral}, we use a language model to generate the necessary edits to incrementally transform (i.e., morph) the premise into the hypothesis. Then, using an off-the-shelf NLI model we track how the entailment progresses with these atomic changes, aggregating these intermediate labels into a final output. We demonstrate the advantages of our proposed method particularly in realistic cross-domain settings, where our method always outperforms strong baselines with improvements up to 12.6% (relative). Further, our proposed approach is explainable as the atomic edits can be used to understand the overall NLI label.
• 摘要：我们介绍了Morphnli，这是一种模块化的自然语言推理（NLI）的模块化方法。当将前提分类对分为{intailment，矛盾，中性}时，我们使用语言模型来生成必要的编辑以逐步转换（即变形）前提为假设。然后，使用现成的NLI模型，我们跟踪这些原子变化如何进行，将这些中间标签汇总为最终输出。我们证明了我们提出的方法的优势，尤其是在现实的跨域设置中，我们的方法始终优于强大的基准，改善高达12.6％（相对）。此外，我们提出的方法是可以解释的，因为原子编辑可用于了解整体NLI标签。

Title: Logical forms complement probability in understanding language model (and human) performance

• Authors: Yixuan Wang, Freda Shi
• Subjects: cs.CL, cs.LO
• Abstract URL: https://arxiv.org/abs/2502.09589
• Pdf URL: https://arxiv.org/pdf/2502.09589
• Copy Paste: [[2502.09589]] Logical forms complement probability in understanding language model (and human) performance(https://arxiv.org/abs/2502.09589)
• Keywords: language model, llm
• Abstract: With the increasing interest in using large language models (LLMs) for planning in natural language, understanding their behaviors becomes an important research question. This work conducts a systematic investigation of LLMs' ability to perform logical reasoning in natural language. We introduce a controlled dataset of hypothetical and disjunctive syllogisms in propositional and modal logic and use it as the testbed for understanding LLM performance. Our results lead to novel insights in predicting LLM behaviors: in addition to the probability of input (Gonen et al., 2023; McCoy et al., 2024), logical forms should be considered as orthogonal factors. In addition, we show similarities and differences between the logical reasoning performances of humans and LLMs by comparing LLM and human behavioral results.
• 摘要：随着对使用大型语言模型（LLM）以自然语言进行计划的兴趣越来越大，了解其行为成为一个重要的研究问题。这项工作对LLMS自然语言执行逻辑推理的能力进行了系统的研究。我们在命题和模态逻辑中介绍了一个假想和脱节的三段论的受控数据集，并将其用作理解LLM性能的测试床。我们的结果导致了预测LLM行为的新见解：除了输入的概率（Gonen等，2023; McCoy等，2024）外，还应将逻辑形式视为正交因素。此外，我们通过比较LLM和人类的行为结果来显示人类和LLM的逻辑推理性能之间的相似性和差异。

Title: SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models

• Authors: Yung-Sung Chuang, Benjamin Cohen-Wang, Shannon Zejiang Shen, Zhaofeng Wu, Hu Xu, Xi Victoria Lin, James Glass, Shang-Wen Li, Wen-tau Yih
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09604
• Pdf URL: https://arxiv.org/pdf/2502.09604
• Copy Paste: [[2502.09604]] SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models(https://arxiv.org/abs/2502.09604)
• Keywords: language model, llm
• Abstract: We introduce SelfCite, a novel self-supervised approach that aligns LLMs to generate high-quality, fine-grained, sentence-level citations for the statements in their generated responses. Instead of only relying on costly and labor-intensive annotations, SelfCite leverages a reward signal provided by the LLM itself through context ablation: If a citation is necessary, removing the cited text from the context should prevent the same response; if sufficient, retaining the cited text alone should preserve the same response. This reward can guide the inference-time best-of-N sampling strategy to improve citation quality significantly, as well as be used in preference optimization to directly fine-tune the models for generating better citations. The effectiveness of SelfCite is demonstrated by increasing citation F1 up to 5.3 points on the LongBench-Cite benchmark across five long-form question answering tasks.
• 摘要：我们介绍了一种新颖的自我监督方法，它使LLM对齐，以生成高质量的，细粒度的，句子级别的引用，以对其产生的响应中的陈述产生。自我引用不仅依靠昂贵和劳动密集型的注释，而是通过上下文消融来利用LLM本身提供的奖励信号：如果需要引用，则从上下文中删除引用的文本应防止相同的响应；如果足够的话，仅保留引用的文本应保留相同的响应。该奖励可以指导推理时间最佳抽样策略，以显着提高引用质量，并用于优化优化，以直接微调模型以产生更好的引用。通过将引文F1提高到五个长形式的回答任务上的长基型基准测试中，自我引用F1的有效性是通过将引用F1提高到5.3点的。

Title: Human-LLM Coevolution: Evidence from Academic Writing

• Authors: Mingmeng Geng, Roberto Trotta
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09606
• Pdf URL: https://arxiv.org/pdf/2502.09606
• Copy Paste: [[2502.09606]] Human-LLM Coevolution: Evidence from Academic Writing(https://arxiv.org/abs/2502.09606)
• Keywords: language model, gpt, llm, chat
• Abstract: With a statistical analysis of arXiv paper abstracts, we report a marked drop in the frequency of several words previously identified as overused by ChatGPT, such as "delve", starting soon after they were pointed out in early 2024. The frequency of certain other words favored by ChatGPT, such as "significant", has instead kept increasing. These phenomena suggest that some authors of academic papers have adapted their use of large language models (LLMs), for example, by selecting outputs or applying modifications to the LLM-generated content. Such coevolution and cooperation of humans and LLMs thus introduce additional challenges to the detection of machine-generated text in real-world scenarios. Estimating the impact of LLMs on academic writing by examining word frequency remains feasible, and more attention should be paid to words that were already frequently employed, including those that have decreased in frequency.
• 摘要：通过对Arxiv论文摘要的统计分析，我们报告了以前几个单词的频率显着下降，例如Chatgpt过度使用的频率，例如“ Delve”，在2024年初指出后不久就开始了。诸如“重要”之类的Chatgpt所青睐，而是不断增加。这些现象表明，一些学术论文的作者已经通过选择输出或对LLM生成的内容进行修改来调整他们对大语言模型（LLMS）的使用。因此，这种人类和LLM的协同进化和合作在现实情况下对机器生成的文本的检测引入了其他挑战。通过检查单词频率来估算LLM对学术写作的影响仍然可行，并且应更多地关注已经经常使用的单词，包括频率下降的单词。