2025-03-03

Title: Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration

• Authors: James Begin, Namit Agrawal, Eshan Singh, Yicheng Fu, Sean O'Brien, Vasu Sharma, Kevin Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20405
• Pdf URL: https://arxiv.org/pdf/2502.20405
• Copy Paste: [[2502.20405]] Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration(https://arxiv.org/abs/2502.20405)
• Keywords: language model, llm
• Abstract: LLMs have demonstrated remarkable proficiency in understanding tasks but continue to struggle with long-context comprehension, particularly with content located in the middle of extensive inputs. This limitation, known as the Lost-in-the-Middle (LITM) problem, hinders models from fully processing and utilizing information across lengthy contexts. To address this issue, we introduce pause-tuning, a technique that redistributes attention to enhance comprehension of long-context inputs. Our approach involves fine-tuning language models on datasets with artificially inserted pause tokens, which serve to segment the input into smaller, more manageable parts. We evaluate pause-tuning against alternative approaches using the Needle-in-a-Haystack benchmark, where models must retrieve information embedded within contexts of up to 128K tokens. Experimental results demonstrate significant performance gains, with the LLaMA 3.2 3B Instruct model and the LLaMA 3.1 8B Instruct model improving by 10.61% and 3.57% respectively on average, suggesting that pause-tuning successfully enhances attention redistribution and improves long-context retention. The code and data are available at this https URL.
• 摘要：LLM已表现出在理解任务方面表现出色的熟练程度，但继续在长期以来的理解中挣扎，尤其是在广泛投入中的内容中。这种限制被称为中间（LITM）问题，它阻碍了模型在漫长的环境中完全处理和利用信息。为了解决这个问题，我们引入了暂停调整，该技术将注意力重新分布以增强对长篇文本输入的理解。我们的方法涉及具有人工插入的暂停令牌的数据集上的微调语言模型，这些模型将输入分为较小，更易于管理的部分。我们使用针中的基准测试对替代方法评估暂停调节，其中模型必须检索嵌入在多达128K代币的上下文中的信息。实验结果表明，Llama 3.2 3B指导模型和Llama 3.1 8B指导模型平均提高了10.61％和3.57％，这表明暂停调查成功增强了注意力再分配并改善了长期文本保留。该代码和数据可在此HTTPS URL上找到。

Title: Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation

• Authors: Shaharukh Khan, Ayush Tarun, Ali Faraz, Palash Kamble, Vivek Dahiya, Praveen Pokala, Ashish Kulkarni, Chandra Khatri, Abhinav Ravi, Shubham Agarwal
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.20420
• Pdf URL: https://arxiv.org/pdf/2502.20420
• Copy Paste: [[2502.20420]] Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation(https://arxiv.org/abs/2502.20420)
• Keywords: llm
• Abstract: In this work, we provide the system description of our submission as part of the English to Lowres Multimodal Translation Task at the Workshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates Multilingual LLM and a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual token embeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all the three tracks (Image Captioning, Text only and Multimodal translation tasks) for Indic languages (ie. English translation to Hindi, Bengali and Malyalam) and achieved SOTA results for Hindi in all of them on the Challenge set while remaining competitive for the other languages in the shared task.
• 摘要：在这项工作中，我们在亚洲翻译研讨会（WAT2024）的研讨会上提供了作为英语的一部分提交的系统描述。我们介绍了Chitranuvad，这是一种多模型模型，可有效整合多语言LLM和一个视觉模块以进行多模式翻译。我们的方法使用VIT图像编码器将视觉表示形式提取为视觉令牌嵌入，通过适配器层将其投影到LLM空间并以自动回归方式生成翻译。我们参与了指示语言（即英文翻译到印地语，孟加拉语和马利亚拉姆）的所有三个曲目（图像字幕，仅文本和多模式翻译任务），并在挑战集中在挑战集中获得了印地语的SOTA结果，同时在共享任务中对其他语言保持竞争力。

Title: SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models

• Authors: Zicheng Cai, Yaohua Tang, Yutao Lai, Hua Wang, Zhi Chen, Hao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20422
• Pdf URL: https://arxiv.org/pdf/2502.20422
• Copy Paste: [[2502.20422]] SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models(https://arxiv.org/abs/2502.20422)
• Keywords: language model, llm, chain-of-thought
• Abstract: We introduce SEKI, a novel large language model (LLM)-based neural architecture search (NAS) method. Inspired by the chain-of-thought (CoT) paradigm in modern LLMs, SEKI operates in two key stages: self-evolution and knowledge distillation. In the self-evolution stage, LLMs initially lack sufficient reference examples, so we implement an iterative refinement mechanism that enhances architectures based on performance feedback. Over time, this process accumulates a repository of high-performance architectures. In the knowledge distillation stage, LLMs analyze common patterns among these architectures to generate new, optimized designs. Combining these two stages, SEKI greatly leverages the capacity of LLMs on NAS and without requiring any domain-specific data. Experimental results show that SEKI achieves state-of-the-art (SOTA) performance across various datasets and search spaces while requiring only 0.05 GPU-days, outperforming existing methods in both efficiency and accuracy. Furthermore, SEKI demonstrates strong generalization capabilities, achieving SOTA-competitive results across multiple tasks.
• 摘要：我们介绍了SEKI，这是一种新型的大型语言模型（LLM）的神经结构搜索（NAS）方法。受到现代LLMS中的经过思考链（COT）范式的启发，Seki在两个关键阶段运作：自我进化和知识蒸馏。在自我进化阶段，LLMS最初缺乏足够的参考示例，因此我们实施了一种迭代精致机制，该机制可以根据性能反馈来增强体系结构。随着时间的流逝，此过程积累了高性能体系结构的存储库。在知识蒸馏阶段，LLMS分析了这些体系结构之间的共同模式，以生成新的优化设计。结合了这两个阶段，SEKI极大地利用了LLM在NAS上的能力，而无需任何特定领域的数据。实验结果表明，SEKI在各个数据集和搜索空间上实现了最先进的（SOTA）性能，同时仅需要0.05 GPU-days，在效率和准确性方面都优于现有方法。此外，SEKI表现出强大的概括能力，在多个任务中实现了SOTA竞争的结果。

Title: Among Them: A game-based framework for assessing persuasion capabilities of LLMs

• Authors: Mateusz Idziejczak, Vasyl Korzavatykh, Mateusz Stawicki, Andrii Chmutov, Marcin Korcz, Iwo Błądek, Dariusz Brzezinski
• Subjects: cs.CL, cs.AI, cs.CY, cs.MA
• Abstract URL: https://arxiv.org/abs/2502.20426
• Pdf URL: https://arxiv.org/pdf/2502.20426
• Copy Paste: [[2502.20426]] Among Them: A game-based framework for assessing persuasion capabilities of LLMs(https://arxiv.org/abs/2502.20426)
• Keywords: language model, llm, agent
• Abstract: The proliferation of large language models (LLMs) and autonomous AI agents has raised concerns about their potential for automated persuasion and social influence. While existing research has explored isolated instances of LLM-based manipulation, systematic evaluations of persuasion capabilities across different models remain limited. In this paper, we present an Among Us-inspired game framework for assessing LLM deception skills in a controlled environment. The proposed framework makes it possible to compare LLM models by game statistics, as well as quantify in-game manipulation according to 25 persuasion strategies from social psychology and rhetoric. Experiments between 8 popular language models of different types and sizes demonstrate that all tested models exhibit persuasive capabilities, successfully employing 22 of the 25 anticipated techniques. We also find that larger models do not provide any persuasion advantage over smaller models and that longer model outputs are negatively correlated with the number of games won. Our study provides insights into the deception capabilities of LLMs, as well as tools and data for fostering future research on the topic.
• 摘要：大语言模型（LLM）和自主AI代理的扩散引起了人们对它们具有自动说服力和社会影响力的潜力的担忧。尽管现有研究探讨了基于LLM的操纵的孤立实例，但对不同模型的说服能力的系统评估仍然有限。在本文中，我们介绍了以美国为灵感的游戏框架，用于评估受控环境中的LLM欺骗技能。提出的框架使得可以按游戏统计数据比较LLM模型，并根据社会心理学和修辞学的25种说服策略来量化游戏中的操作。 8种流行类型和大小的流行语言模型之间的实验表明，所有经过测试的模型都表现出说服力，成功地采用了25种预期技术中的22种。我们还发现，较大的模型不能比较小的模型提供任何说服力的优势，并且更长的模型输出与赢得的游戏数量负相关。我们的研究提供了对LLMS欺骗能力的见解，以及用于促进该主题的未来研究的工具和数据。

Title: Shades of Zero: Distinguishing Impossibility from Inconceivability

• Authors: Jennifer Hu, Felix Sosa, Tomer Ullman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20469
• Pdf URL: https://arxiv.org/pdf/2502.20469
• Copy Paste: [[2502.20469]] Shades of Zero: Distinguishing Impossibility from Inconceivability(https://arxiv.org/abs/2502.20469)
• Keywords: language model
• Abstract: Some things are impossible, but some things may be even more impossible than impossible. Levitating a feather using one's mind is impossible in our world, but fits into our intuitive theories of possible worlds, whereas levitating a feather using the number five cannot be conceived in any possible world ("inconceivable"). While prior work has examined the distinction between improbable and impossible events, there has been little empirical research on inconceivability. Here, we investigate whether people maintain a distinction between impossibility and inconceivability, and how such distinctions might be made. We find that people can readily distinguish the impossible from the inconceivable, using categorization studies similar to those used to investigate the differences between impossible and improbable (Experiment 1). However, this distinction is not explained by people's subjective ratings of event likelihood, which are near zero and indistinguishable between impossible and inconceivable event descriptions (Experiment 2). Finally, we ask whether the probabilities assigned to event descriptions by statistical language models (LMs) can be used to separate modal categories, and whether these probabilities align with people's ratings (Experiment 3). We find high-level similarities between people and LMs: both distinguish among impossible and inconceivable event descriptions, and LM-derived string probabilities predict people's ratings of event likelihood across modal categories. Our findings suggest that fine-grained knowledge about exceedingly rare events (i.e., the impossible and inconceivable) may be learned via statistical learning over linguistic forms, yet leave open the question of whether people represent the distinction between impossible and inconceivable as a difference not of degree, but of kind.
• 摘要：有些事情是不可能的，但是有些事情可能比不可能更不可能。在我们的世界中，使用一个人的思想悬浮羽毛是不可能的，但是适合我们可能的世界的直觉理论，而在任何可能的世界中都无法想象使用第五的羽毛（“不可思议”）。尽管先前的工作已经检查了不可能的事件和不可能的事件之间的区别，但几乎没有关于不可思议性的实证研究。在这里，我们调查人们是否保持不可能和不可思议性之间的区别，以及如何做出这种区别。我们发现，使用类似于研究不可能和不可能之间的差异的分类研究，人们可以很容易地将不可能与不可想象的研究区分开来（实验1）。但是，这种区别并不能通过人们对事件可能性的主观评分来解释，事件可能性接近零且无法想象的事件描述（实验2）。最后，我们询问统计语言模型（LMS）分配给事件描述的概率是否可用于分开模态类别，以及这些概率是否与人们的评分一致（实验3）。我们发现人和LMS之间的高级相似性：在不可能的事件描述和不可想象的事件描述之间区分，LM衍生的字符串概率可以预测人们对模态类别中事件可能性的评分。我们的发现表明，可以通过对语言形式的统计学习来学习有关极少数事件（即不可能和不可思议）的细粒度知识（即，不可能的和不可思议的），但请留下一个问题，即人们是否代表了不可能的和不可想象的差异，而不是程度的差异，而是类型的差异。

Title: Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries

• Authors: Tianyi Lorena Yan, Robin Jia
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20475
• Pdf URL: https://arxiv.org/pdf/2502.20475
• Copy Paste: [[2502.20475]] Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries(https://arxiv.org/abs/2502.20475)
• Keywords: language model
• Abstract: To answer one-to-many factual queries (e.g., listing cities of a country), a language model (LM) must simultaneously recall knowledge and avoid repeating previous answers. How are these two subtasks implemented and integrated internally? Across multiple datasets and models, we identify a promote-then-suppress mechanism: the model first recalls all answers, and then suppresses previously generated ones. Specifically, LMs use both the subject and previous answer tokens to perform knowledge recall, with attention propagating subject information and MLPs promoting the answers. Then, attention attends to and suppresses previous answer tokens, while MLPs amplify the suppression signal. Our mechanism is corroborated by extensive experimental evidence: in addition to using early decoding and causal tracing, we analyze how components use different tokens by introducing both \emph{Token Lens}, which decodes aggregated attention updates from specified tokens, and a knockout method that analyzes changes in MLP outputs after removing attention to specified tokens. Overall, we provide new insights into how LMs' internal components interact with different input tokens to support complex factual recall. Code is available at this https URL.
• 摘要：要回答一对一的事实查询（例如，一个国家的列表城市），语言模型（LM）必须同时回忆知识并避免重复以前的答案。这两个子任务如何在内部实施和集成？在多个数据集和模型中，我们确定了一种促进式的抑制机制：该模型首先回忆所有答案，然后抑制先前生成的答案。具体而言，LMS使用主题和先前的答案令牌来执行知识回忆，并通过关注主题信息和MLP来促进答案。然后，注意并抑制先前的答案令牌，而MLP会放大抑制信号。 Our mechanism is corroborated by extensive experimental evidence: in addition to using early decoding and causal tracing, we analyze how components use different tokens by introducing both \emph{Token Lens}, which decodes aggregated attention updates from specified tokens, and a knockout method that analyzes changes in MLP outputs after removing attention to specified tokens.总体而言，我们提供了有关LMS内部组件如何与不同输入令牌相互作用以支持复杂事实召回的新见解。代码可在此HTTPS URL上找到。

Title: Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences

• Authors: Jun Hou, Lucy Lu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20478
• Pdf URL: https://arxiv.org/pdf/2502.20478
• Copy Paste: [[2502.20478]] Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences(https://arxiv.org/abs/2502.20478)
• Keywords: llm
• Abstract: Explainable AI (XAI) techniques are necessary to help clinicians make sense of AI predictions and integrate predictions into their decision-making workflow. In this work, we conduct a survey study to understand clinician preference among different XAI techniques when they are used to interpret model predictions over text-based EHR data. We implement four XAI techniques (LIME, Attention-based span highlights, exemplar patient retrieval, and free-text rationales generated by LLMs) on an outcome prediction model that uses ICU admission notes to predict a patient's likelihood of experiencing in-hospital mortality. Using these XAI implementations, we design and conduct a survey study of 32 practicing clinicians, collecting their feedback and preferences on the four techniques. We synthesize our findings into a set of recommendations describing when each of the XAI techniques may be more appropriate, their potential limitations, as well as recommendations for improvement.
• 摘要：可解释的AI（XAI）技术是必要的，以帮助临床医生理解AI预测并将预测整合到其决策工作流程中。在这项工作中，我们进行了一项调查研究，以了解不同XAI技术的临床医生偏好，以解释基于文本的EHR数据的模型预测。我们在结果预测模型上实施了四种XAI技术（石灰，基于注意力的跨度亮点，示例性患者检索以及LLMS产生的自由文本理由），该模型使用ICU录取注释来预测患者体验内医生死亡率的可能性。使用这些XAI实施，我们设计和进行了一项针对32位实践临床医生的调查研究，收集了他们对四种技术的反馈和偏好。我们将我们的发现综合为一组建议，描述了何时每种XAI技术更合适，它们的潜在局限性以及改进建议。

Title: Protecting multimodal large language models against misleading visualizations

• Authors: Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20503
• Pdf URL: https://arxiv.org/pdf/2502.20503
• Copy Paste: [[2502.20503]] Protecting multimodal large language models against misleading visualizations(https://arxiv.org/abs/2502.20503)
• Keywords: language model, llm
• Abstract: We assess the vulnerability of multimodal large language models to misleading visualizations - charts that distort the underlying data using techniques such as truncated or inverted axes, leading readers to draw inaccurate conclusions that may support misinformation or conspiracy theories. Our analysis shows that these distortions severely harm multimodal large language models, reducing their question-answering accuracy to the level of the random baseline. To mitigate this vulnerability, we introduce six inference-time methods to improve performance of MLLMs on misleading visualizations while preserving their accuracy on non-misleading ones. The most effective approach involves (1) extracting the underlying data table and (2) using a text-only large language model to answer questions based on the table. This method improves performance on misleading visualizations by 15.4 to 19.6 percentage points.
• 摘要：我们评估了多模式大语言模型误导可视化的脆弱性 - 图表使用诸如截短或倒轴等技术扭曲基础数据的图表，导致读者得出可能支持错误信息或阴谋理论的不准确结论。我们的分析表明，这些扭曲严重损害了多模式的大语言模型，将其提问的准确性降低到随机基线的水平。为了减轻这种脆弱性，我们介绍了六种推理时间方法，以提高MLLM在误导性可视化方面的性能，同时保留其在非误导性方面的准确性。最有效的方法涉及（1）提取基础数据表，以及（2）使用仅文本大型语言模型来回答基于表的问题。这种方法将误导性可视化的性能提高了15.4至19.6个百分点。

Title: A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs

• Authors: Julius Broomfield, Kartik Sharma, Srijan Kumar
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.20504
• Pdf URL: https://arxiv.org/pdf/2502.20504
• Copy Paste: [[2502.20504]] A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs(https://arxiv.org/abs/2502.20504)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have recently demonstrated remarkable advancements in embodying diverse personas, enhancing their effectiveness as conversational agents and virtual assistants. Consequently, LLMs have made significant strides in processing and integrating multimodal information. However, even though human personas can be expressed in both text and image, the extent to which the modality of a persona impacts the embodiment by the LLM remains largely unexplored. In this paper, we investigate how do different modalities influence the expressiveness of personas in multimodal LLMs. To this end, we create a novel modality-parallel dataset of 40 diverse personas varying in age, gender, occupation, and location. This consists of four modalities to equivalently represent a persona: image-only, text-only, a combination of image and small text, and typographical images, where text is visually stylized to convey persona-related attributes. We then create a systematic evaluation framework with 60 questions and corresponding metrics to assess how well LLMs embody each persona across its attributes and scenarios. Comprehensive experiments on $5$ multimodal LLMs show that personas represented by detailed text show more linguistic habits, while typographical images often show more consistency with the persona. Our results reveal that LLMs often overlook persona-specific details conveyed through images, highlighting underlying limitations and paving the way for future research to bridge this gap. We release the data and code at this https URL .
• 摘要：大型语言模型（LLMS）最近在体现各种角色，增强其作为对话代理和虚拟助手的有效性方面取得了显着进步。因此，LLM在处理和整合多模式信息方面已取得了重大进步。但是，即使可以在文本和图像中表达人类角色，角色的形式影响LLM的实施方案的程度仍然在很大程度上没有探索。在本文中，我们研究了不同模态在多模式LLM中的表现力如何影响。为此，我们创建了一个新颖的模式平行数据集，该数据集的年龄，性别，职业和位置各不相同。这包括四种方式，可以等效地代表角色：仅图像，仅文本，图像和小文本的组合以及印刷图像，在该图像上进行了视觉风格的文本以传达与角色相关的属性。然后，我们创建一个系统的评估框架，其中有60个问题和相应的指标，以评估LLM在其属性和场景中体现每个角色的表现如何。 $ 5 $多模式LLMS的综合实验表明，由详细文本代表的角色显示出更多的语言习惯，而印刷图像通常显示出与角色的一致性。我们的结果表明，LLM经常忽略通过图像传达的特定于人格特定的细节，突出了基本的局限性，并为将来的研究铺平了这一差距。我们在此HTTPS URL上发布数据和代码。

Title: TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning

• Authors: Soumyabrata Chaudhuri, Pranav Purkar, Ritwik Raghav, Shubhojit Mallick, Manish Gupta, Abhik Jana, Shreya Ghosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20508
• Pdf URL: https://arxiv.org/pdf/2502.20508
• Copy Paste: [[2502.20508]] TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning(https://arxiv.org/abs/2502.20508)
• Keywords: language model, llm, agent
• Abstract: Recent advancements in probing Large Language Models (LLMs) have explored their latent potential as personalized travel planning agents, yet existing benchmarks remain limited in real world applicability. Existing datasets, such as TravelPlanner and TravelPlanner+, suffer from semi synthetic data reliance, spatial inconsistencies, and a lack of key travel constraints, making them inadequate for practical itinerary generation. To address these gaps, we introduce TripCraft, a spatiotemporally coherent travel planning dataset that integrates real world constraints, including public transit schedules, event availability, diverse attraction categories, and user personas for enhanced personalization. To evaluate LLM generated plans beyond existing binary validation methods, we propose five continuous evaluation metrics, namely Temporal Meal Score, Temporal Attraction Score, Spatial Score, Ordering Score, and Persona Score which assess itinerary quality across multiple dimensions. Our parameter informed setting significantly enhances meal scheduling, improving the Temporal Meal Score from 61% to 80% in a 7 day scenario. TripCraft establishes a new benchmark for LLM driven personalized travel planning, offering a more realistic, constraint aware framework for itinerary generation. Dataset and Codebase will be made publicly available upon acceptance.
• 摘要：探索大语言模型（LLM）的最新进步探讨了其潜在的潜在潜力，因为个性化的旅行计划代理商，但现有基准在现实世界中的适用性仍然有限。现有的数据集，例如Travel Planner和TravelPlanner+，患有半合成数据的依赖，空间不一致以及缺乏关键的旅行限制，从而无法获得实用的行程生成。为了解决这些差距，我们介绍了TripCraft，这是一个时空连贯的旅行计划数据集，该数据集集成了现实世界的限制，包括公共交通计划，事件可用性，各种吸引力类别和用户角色，以增强个性化。为了评估LLM生成的计划以外的现有二进制验证方法，我们提出了五个连续的评估指标，即时间进餐得分，时间吸引分数，空间得分，订购得分和角色分数，以评估跨多个维度的行程质量。我们的参数通知设置可显着增强进餐时间表，在7天情景中将临时进餐评分从61％提高到80％。 TripCraft为LLM驱动的个性化旅行计划建立了新的基准，为行程生成提供了更现实，更约束的意识框架。接受后，数据集和代码库将在接受后公开可用。

Title: Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education

• Authors: Emily Ross, Yuval Kansal, Jake Renzella, Alexandra Vassar, Andrew Taylor
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.20527
• Pdf URL: https://arxiv.org/pdf/2502.20527
• Copy Paste: [[2502.20527]] Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education(https://arxiv.org/abs/2502.20527)
• Keywords: language model, gpt, llm, prompt, chat, agent
• Abstract: Large language models (LLMs) are increasingly being explored in higher education, yet their effectiveness as teaching agents remains underexamined. In this paper, we present the development of GuideLM, a fine-tuned LLM designed for programming education. GuideLM has been integrated into the Debugging C Compiler (DCC), an educational C compiler that leverages LLMs to generate pedagogically sound error explanations. Previously, DCC relied on off-the-shelf OpenAI models, which, while accurate, often over-assisted students by directly providing solutions despite contrary prompting. To address this, we employed supervised fine-tuning (SFT) on a dataset of 528 student-question/teacher-answer pairs, creating two models: GuideLM and GuideLM-mini, fine-tuned on ChatGPT-4o and 4o-mini, respectively. We conducted an expert analysis of 400 responses per model, comparing their pedagogical effectiveness against base OpenAI models. Our evaluation, grounded in constructivism and cognitive load theory, assessed factors such as conceptual scaffolding, clarity, and Socratic guidance. Results indicate that GuideLM and GuideLM-mini improve pedagogical performance, with an 8% increase in Socratic guidance and a 58% improvement in economy of words compared to GPT-4o. However, this refinement comes at the cost of a slight reduction in general accuracy. While further work is needed, our findings suggest that fine-tuning LLMs with targeted datasets is a promising approach for developing models better suited to educational contexts.
• 摘要：在高等教育中，大型语言模型（LLM）越来越多地探索，但由于教学代理人的效力仍然没有散发出来。在本文中，我们介绍了Guidelm的开发，Guidelm是一款专为编程教育而设计的精通LLM。 Guidelm已集成到调试C编译器（DCC）中，这是一种教育C编译器，利用LLMS生成教学上有声音错误的解释。此前，DCC依靠现成的OpenAI模型，尽管有相反的提示，但该模型通常通过直接提供解决方案而过度辅助学生。为了解决这个问题，我们在528个学生疑问/教师回答对的数据集上使用了监督的微调（SFT），创建了两种模型：Guidelm和Guidelm-Mini，分别在Chatgpt-4O和4O-Mini上进行了微调。我们对每个模型进行了400个响应的专家分析，将其教学效率与基本OpenAI模型进行了比较。我们的评估基于建构主义和认知负载理论，评估了概念脚手架，清晰度和苏格拉底指导等因素。结果表明，与GPT-4O相比，Guidelm和Guidelm-Mini提高了教学绩效，苏格拉底指南增长了8％，单词经济性提高了58％。但是，这种改进是以略有降低的一般准确性来实现的。尽管需要进一步的工作，但我们的发现表明，使用有针对性数据集的微调LLM是开发更适合教育环境的模型的有前途的方法。

Title: NANOGPT: A Query-Driven Large Language Model Retrieval-Augmented Generation System for Nanotechnology Research

• Authors: Achuth Chandrasekhar, Omid Barati Farimani, Olabode T. Ajenifujah, Janghoon Ock, Amir Barati Farimani
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20541
• Pdf URL: https://arxiv.org/pdf/2502.20541
• Copy Paste: [[2502.20541]] NANOGPT: A Query-Driven Large Language Model Retrieval-Augmented Generation System for Nanotechnology Research(https://arxiv.org/abs/2502.20541)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: This paper presents the development and application of a Large Language Model Retrieval-Augmented Generation (LLM-RAG) system tailored for nanotechnology research. The system leverages the capabilities of a sophisticated language model to serve as an intelligent research assistant, enhancing the efficiency and comprehensiveness of literature reviews in the nanotechnology domain. Central to this LLM-RAG system is its advanced query backend retrieval mechanism, which integrates data from multiple reputable sources. The system retrieves relevant literature by utilizing Google Scholar's advanced search, and scraping open-access papers from Elsevier, Springer Nature, and ACS Publications. This multifaceted approach ensures a broad and diverse collection of up-to-date scholarly articles and papers. The proposed system demonstrates significant potential in aiding researchers by providing a streamlined, accurate, and exhaustive literature retrieval process, thereby accelerating research advancements in nanotechnology. The effectiveness of the LLM-RAG system is validated through rigorous testing, illustrating its capability to significantly reduce the time and effort required for comprehensive literature reviews, while maintaining high accuracy, query relevance and outperforming standard, publicly available LLMS.
• 摘要：本文介绍了针对纳米技术研究量身定制的大型语言模型检索生成（LLM-rag）系统的开发和应用。该系统利用复杂的语言模型的能力来充当智能研究助理，增强了纳米技术领域文献评论的效率和全面性。该LLM-rag系统的核心是其高级查询后端检索机制，该机制集成了来自多个信誉量的数据。该系统通过利用Google Scholar的高级搜索并刮除来自Elsevier，Springer Nature和ACS出版物的开放式论文来检索相关文献。这种多方面的方法可确保广泛而多样的最新学术文章和论文集合。拟议的系统通过提供简化，准确和详尽的文献检索过程，在帮助研究人员方面具有巨大潜力，从而加速了纳米技术的研究进步。 LLM-rag系统的有效性通过严格的测试进行了验证，这说明了其能力大大减少全面文献综述所需的时间和精力，同时保持高准确性，查询相关性和胜过标准的公开，公开可用的LLMS。

Title: HuAMR: A Hungarian AMR Parser and Dataset

• Authors: Botond Barta, Endre Hamerlik, Milán Konor Nyist, Judit Ács
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20552
• Pdf URL: https://arxiv.org/pdf/2502.20552
• Copy Paste: [[2502.20552]] HuAMR: A Hungarian AMR Parser and Dataset(https://arxiv.org/abs/2502.20552)
• Keywords: language model
• Abstract: We present HuAMR, the first Abstract Meaning Representation (AMR) dataset and a suite of large language model-based AMR parsers for Hungarian, targeting the scarcity of semantic resources for non-English languages. To create HuAMR, we employed Llama-3.1-70B to automatically generate silver-standard AMR annotations, which we then refined manually to ensure quality. Building on this dataset, we investigate how different model architectures - mT5 Large and Llama-3.2-1B - and fine-tuning strategies affect AMR parsing performance. While incorporating silver-standard AMRs from Llama-3.1-70B into the training data of smaller models does not consistently boost overall scores, our results show that these techniques effectively enhance parsing accuracy on Hungarian news data (the domain of HuAMR). We evaluate our parsers using Smatch scores and confirm the potential of HuAMR and our parsers for advancing semantic parsing research.
• 摘要：我们介绍Huamr，这是第一个抽象含义表示（AMR）数据集和匈牙利语基于大型语言模型的大型AMR解析器，针对非英语语言的语义资源的稀缺性。为了创建HUAMR，我们采用Llama-3.1-70B自动生成银色标准AMR注释，然后手动进行完善以确保质量。在此数据集的基础上，我们研究了不同的模型体系结构-MT5大型和Llama-3.2-1b-以及微调策略如何影响AMR解析性能。在将来自Llama-3.1-70B的银色标准AMR纳入较小模型的训练数据中并不能始终提高整体分数，但我们的结果表明，这些技术有效提高了匈牙利新闻数据（Huamr的领域）的解析准确性。我们使用Smatch分数评估了我们的解析器，并确认了Huamr和我们的解析器的潜力来推进语义解析研究。

Title: Multi$^2$: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing

• Authors: Juntai Cao, Xiang Zhang, Raymond Li, Chuyuan Li, Shafiq Joty, Giuseppe Carenini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20592
• Pdf URL: https://arxiv.org/pdf/2502.20592
• Copy Paste: [[2502.20592]] Multi$^2$: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing(https://arxiv.org/abs/2502.20592)
• Keywords: language model, llm, prompt, agent
• Abstract: Recent advances in test-time scaling have shown promising results in improving Large Language Models (LLMs) performance through strategic computation allocation during inference. While this approach has demonstrated strong performance improvements in logical and mathematical reasoning tasks, its application to natural language generation (NLG), especially summarization, has yet to be explored. Multi-Document Summarization (MDS) is a challenging task that focuses on extracting and synthesizing useful information from multiple lengthy documents. Unlike reasoning tasks, MDS requires a more nuanced approach to prompt design and ensemble, as there is no "best" prompt to satisfy diverse summarization requirements. To address this, we propose a novel framework that leverages inference-time scaling for this task. Precisely, we take prompt ensemble approach by leveraging various prompt to first generate candidate summaries and then ensemble them with an aggregator to produce a refined summary. We also introduce two new evaluation metrics: Consistency-Aware Preference (CAP) score and LLM Atom-Content-Unit (ACU) score, to enhance LLM's contextual understanding while mitigating its positional bias. Extensive experiments demonstrate the effectiveness of our approach in improving summary quality while identifying and analyzing the scaling boundaries in summarization tasks.
• 摘要：测试时间缩放的最新进展显示出了通过推断期间的战略计算分配改善大语言模型（LLMS）绩效的有希望的结果。尽管这种方法表明逻辑和数学推理任务的绩效有了很大的提高，但其在自然语言生成（NLG）（尤其是摘要）中的应用尚未探讨。多文档摘要（MDS）是一项具有挑战性的任务，侧重于从多个冗长文档中提取和综合有用信息。与推理任务不同，MD需要采取更细微的方法来提示设计和合奏，因为没有“最佳”提示来满足各种摘要要求。为了解决这个问题，我们提出了一个新型框架，该框架利用推理时间扩展为此任务。确切地说，我们通过利用各种提示首先生成候选摘要，然后与聚合器合奏以产生精致的摘要来采取迅速的合奏方法。我们还介绍了两个新的评估指标：一致性感知偏好（CAP）分数和LLM Atom-Content-Unit（ACU）分数，以增强LLM的上下文理解，同时减轻其位置偏差。广泛的实验证明了我们的方法在提高摘要质量方面的有效性，同时识别和分析摘要任务中的缩放界限。

Title: Few-Shot, No Problem: Descriptive Continual Relation Extraction

• Authors: Nguyen Xuan Thanh, Anh Duc Le, Quyen Tran, Thanh-Thien Le, Linh Ngo Van, Thien Huu Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20596
• Pdf URL: https://arxiv.org/pdf/2502.20596
• Copy Paste: [[2502.20596]] Few-Shot, No Problem: Descriptive Continual Relation Extraction(https://arxiv.org/abs/2502.20596)
• Keywords: language model
• Abstract: Few-shot Continual Relation Extraction is a crucial challenge for enabling AI systems to identify and adapt to evolving relationships in dynamic real-world domains. Traditional memory-based approaches often overfit to limited samples, failing to reinforce old knowledge, with the scarcity of data in few-shot scenarios further exacerbating these issues by hindering effective data augmentation in the latent space. In this paper, we propose a novel retrieval-based solution, starting with a large language model to generate descriptions for each relation. From these descriptions, we introduce a bi-encoder retrieval training paradigm to enrich both sample and class representation learning. Leveraging these enhanced representations, we design a retrieval-based prediction method where each sample "retrieves" the best fitting relation via a reciprocal rank fusion score that integrates both relation description vectors and class prototypes. Extensive experiments on multiple datasets demonstrate that our method significantly advances the state-of-the-art by maintaining robust performance across sequential tasks, effectively addressing catastrophic forgetting.
• 摘要：对于使AI系统能够识别和适应动态现实世界中不断发展的关系，几乎没有持续的关系提取是一个至关重要的挑战。传统的基于内存的方法通常过于适应有限的样本，无法加强旧知识，而在几个方案中的数据稀缺，进一步加剧了这些问题，从而阻碍了潜在空间中的有效数据增强。在本文中，我们提出了一种新型基于检索的解决方案，从大型语言模型开始，以生成每个关系的描述。从这些描述中，我们引入了双重编码器检索训练范式，以丰富样本和班级表示学习。利用这些增强的表示形式，我们设计了一种基于检索的预测方法，其中每个样本通过相互等级融合得分“检索”最佳拟合关系，从而整合了关系描述向量和类原型。在多个数据集上进行的广泛实验表明，我们的方法通过在顺序任务中保持稳健的性能来显着提高最新性能，从而有效地解决了灾难性的遗忘。

Title: Leveraging Large Language Models for Building Interpretable Rule-Based Data-to-Text Systems

• Authors: Jędrzej Warczyński, Mateusz Lango, Ondrej Dusek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20609
• Pdf URL: https://arxiv.org/pdf/2502.20609
• Copy Paste: [[2502.20609]] Leveraging Large Language Models for Building Interpretable Rule-Based Data-to-Text Systems(https://arxiv.org/abs/2502.20609)
• Keywords: language model, llm, hallucination, prompt
• Abstract: We introduce a simple approach that uses a large language model (LLM) to automatically implement a fully interpretable rule-based data-to-text system in pure Python. Experimental evaluation on the WebNLG dataset showed that such a constructed system produces text of better quality (according to the BLEU and BLEURT metrics) than the same LLM prompted to directly produce outputs, and produces fewer hallucinations than a BART language model fine-tuned on the same data. Furthermore, at runtime, the approach generates text in a fraction of the processing time required by neural approaches, using only a single CPU
• 摘要：我们引入了一种简单的方法，该方法使用大型语言模型（LLM）自动在纯Python中实现完全可解释的基于规则的数据对文本系统。 WebNLG数据集的实验评估表明，这种构造的系统比同一LLM直接产生输出的质量（根据BLEU和BLEURT指标）更高的文本（根据BLEU和BLEURT指标），而与在同一数据上精细调整的BART语言相比，产生的幻觉更少。此外，在运行时，该方法仅使用单个CPU，在神经方法所需的处理时间的一小部分中生成文本

Title: Continuous Adversarial Text Representation Learning for Affective Recognition

• Authors: Seungah Son, Andrez Saurez, Dongsoo Har
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20613
• Pdf URL: https://arxiv.org/pdf/2502.20613
• Copy Paste: [[2502.20613]] Continuous Adversarial Text Representation Learning for Affective Recognition(https://arxiv.org/abs/2502.20613)
• Keywords: language model
• Abstract: While pre-trained language models excel at semantic understanding, they often struggle to capture nuanced affective information critical for affective recognition tasks. To address these limitations, we propose a novel framework for enhancing emotion-aware embeddings in transformer-based models. Our approach introduces a continuous valence-arousal labeling system to guide contrastive learning, which captures subtle and multi-dimensional emotional nuances more effectively. Furthermore, we employ a dynamic token perturbation mechanism, using gradient-based saliency to focus on sentiment-relevant tokens, improving model sensitivity to emotional cues. The experimental results demonstrate that the proposed framework outperforms existing methods, achieving up to 15.5% improvement in the emotion classification benchmark, highlighting the importance of employing continuous labels. This improvement demonstrates that the proposed framework is effective in affective representation learning and enables precise and contextually relevant emotional understanding.
• 摘要：尽管预训练的语言模型在语义理解方面表现出色，但他们经常努力捕获细微的情感信息对情感识别任务至关重要。为了解决这些局限性，我们提出了一个新颖的框架，以增强基于变压器模型中的情绪感知的嵌入。我们的方法引入了一个连续的价值标记系统来指导对比度学习，该系统更有效地捕捉了微妙和多维的情感细微差别。此外，我们采用动态令牌扰动机制，利用基于梯度的显着性来专注于情感与情感的代币，从而提高了模型对情感提示的敏感性。实验结果表明，所提出的框架的表现优于现有方法，在情绪分类基准中提高了15.5％，强调了采用连续标签的重要性。这种改进表明，所提出的框架在情感表示学习中有效，并实现了精确和上下文相关的情感理解。

Title: Rectifying Belief Space via Unlearning to Harness LLMs' Reasoning

• Authors: Ayana Niwa, Masahiro Kaneko, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20620
• Pdf URL: https://arxiv.org/pdf/2502.20620
• Copy Paste: [[2502.20620]] Rectifying Belief Space via Unlearning to Harness LLMs' Reasoning(https://arxiv.org/abs/2502.20620)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) can exhibit advanced reasoning yet still generate incorrect answers. We hypothesize that such errors frequently stem from spurious beliefs, propositions the model internally considers true but are incorrect. To address this, we propose a method to rectify the belief space by suppressing these spurious beliefs while simultaneously enhancing true ones, thereby enabling more reliable inferences. Our approach first identifies the beliefs that lead to incorrect or correct answers by prompting the model to generate textual explanations, using our Forward-Backward Beam Search (FBBS). We then apply unlearning to suppress the identified spurious beliefs and enhance the true ones, effectively rectifying the model's belief space. Empirical results on multiple QA datasets and LLMs show that our method corrects previously misanswered questions without harming overall model performance. Furthermore, our approach yields improved generalization on unseen data, suggesting that rectifying a model's belief space is a promising direction for mitigating errors and enhancing overall reliability.
• 摘要：大型语言模型（LLM）可以表现出高级推理，但仍会产生错误的答案。我们假设这种错误经常源于虚假的信念，命题在内部认为是真实的，但是不正确的。为了解决这个问题，我们提出了一种通过抑制这些虚假信念的同时增强真实信念的方法来纠正信仰空间的方法，从而实现了更可靠的推论。我们的方法首先通过提示模型，使用我们的前后光束搜索（FBB）来促使模型生成文本说明，从而确定导致错误或正确答案的信念。然后，我们采用不学习来抑制已确定的虚假信念并增强真实信念，从而有效地纠正了模型的信仰空间。多个质量检查数据集和LLM的经验结果表明，我们的方法纠正了以前误解的问题而不会损害整体模型性能。此外，我们的方法对看不见的数据有改善的概括，这表明纠正模型的信仰空间是减轻错误和增强整体可靠性的有希望的方向。

Title: LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation

• Authors: Haitao Li, Yifan Chen, Yiran Hu, Qingyao Ai, Junjie Chen, Xiaoyu Yang, Jianhui Yang, Yueyue Wu, Zeyang Liu, Yiqun Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.20640
• Pdf URL: https://arxiv.org/pdf/2502.20640
• Copy Paste: [[2502.20640]] LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation(https://arxiv.org/abs/2502.20640)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) has proven highly effective in improving large language models (LLMs) across various domains. However, there is no benchmark specifically designed to assess the effectiveness of RAG in the legal domain, which restricts progress in this area. To fill this gap, we propose LexRAG, the first benchmark to evaluate RAG systems for multi-turn legal consultations. LexRAG consists of 1,013 multi-turn dialogue samples and 17,228 candidate legal articles. Each sample is annotated by legal experts and consists of five rounds of progressive questioning. LexRAG includes two key tasks: (1) Conversational knowledge retrieval, requiring accurate retrieval of relevant legal articles based on multi-turn context. (2) Response generation, focusing on producing legally sound answers. To ensure reliable reproducibility, we develop LexiT, a legal RAG toolkit that provides a comprehensive implementation of RAG system components tailored for the legal domain. Additionally, we introduce an LLM-as-a-judge evaluation pipeline to enable detailed and effective assessment. Through experimental analysis of various LLMs and retrieval methods, we reveal the key limitations of existing RAG systems in handling legal consultation conversations. LexRAG establishes a new benchmark for the practical application of RAG systems in the legal domain, with its code and data available at this https URL.
• 摘要：被证明在改善各个领域的大型语言模型（LLM）方面已被证明是有效的。但是，没有专门设计的基准来评估抹布在法律领域的有效性，这限制了该领域的进展。为了填补这一空白，我们建议Lexrag，这是第一个评估抹布系统进行多转弯法律咨询的基准。 Lexrag由1,013个多转化对话样本和17,228个候选法律文章组成。每个样本都由法律专家注释，并由五轮进步的质疑组成。 Lexrag包括两个关键任务：（1）会话知识检索，需要基于多转移环境的相关法律文章的准确检索。 （2）响应生成，专注于产生合法的答案。为了确保可靠的可重复性，我们开发了Lexit，这是一种合法的抹布工具包，可全面实施为法律领域量身定制的抹布系统组件。此外，我们引入了LLM-AS-A-A-A-A-Gudge评估管道，以实现详细且有效的评估。通过对各种LLM和检索方法的实验分析，我们揭示了现有的抹布系统在处理法律咨询对话中的关键局限性。 Lexrag建立了一个新的基准，用于在法律域中使用RAG系统的实际应用，并在此HTTPS URL上获得其代码和数据。

Title: Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Models

• Authors: Colleen Gilhuly, Haleh Shahzad
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20647
• Pdf URL: https://arxiv.org/pdf/2502.20647
• Copy Paste: [[2502.20647]] Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Models(https://arxiv.org/abs/2502.20647)
• Keywords: language model, gpt, llm, prompt
• Abstract: Text summarizing is a critical Natural Language Processing (NLP) task with applications ranging from information retrieval to content generation. Large Language Models (LLMs) have shown remarkable promise in generating fluent abstractive summaries but they can produce hallucinated details not grounded in the source text. Regardless of the method of generating a summary, high quality automated evaluations remain an open area of investigation. This paper embarks on an exploration of text summarization with a diverse set of techniques, including TextRank, BART, Mistral-7B-Instruct, and OpenAI GPT-3.5-Turbo. The generated summaries are evaluated using traditional metrics such as the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) Score and Bidirectional Encoder Representations from Transformers (BERT) Score, as well as LLM-powered evaluation methods that directly assess a generated summary's consistency with the source text. We introduce a meta evaluation score which directly assesses the performance of the LLM evaluation system (prompt + model). We find that that all summarization models produce consistent summaries when tested on the XL-Sum dataset, exceeding the consistency of the reference summaries.
• 摘要：文本总结是一项关键的自然语言处理（NLP）任务，其应用程序从信息检索到内容生成。大型语言模型（LLM）在产生流利的抽象摘要方面表现出了巨大的希望，但它们可以产生幻觉细节，而不是基于源文本。无论产生摘要的方法如何，高质量的自动化评估仍然是开放的调查领域。本文将对文本摘要进行探索，其中包括Textrank，Bart，Mistral-7B-Instruct和Openai GPT-3.5-Turbo。使用传统指标进行评估生成的摘要，例如以召回的研究为目标评估（Rouge）得分（Rouge）得分和来自变形金刚（BERT）分数（BERT）分数的双向编码器表示，以及LLM驱动的评估方法，这些方法可以直接评估生成的摘要与源文本的一致性。我们介绍了一个直接评估LLM评估系统（提示 +模型）的性能的元评估评分。我们发现，在XL-SUM数据集上测试时，所有汇总模型都会产生一致的摘要，超过了参考摘要的一致性。

Title: Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository

• Authors: Radhika Kapoor, Sang T. Truong, Nick Haber, Maria Araceli Ruiz-Primo, Benjamin W. Domingue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20663
• Pdf URL: https://arxiv.org/pdf/2502.20663
• Copy Paste: [[2502.20663]] Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository(https://arxiv.org/abs/2502.20663)
• Keywords: llm
• Abstract: Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2017-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.52 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.
• 摘要：基于其文本内容的项目难度的预测引起了重大关注。在本文中，我们关注的问题是，当数据最初报告的项目p值（正确的响应百分比）时，恢复了基于IRT的难度的相关问题。我们使用来自纽约和德克萨斯州的美国标准化测试的阅读段落和学生数据的存储库来对该项目进行建模，以期为2017 - 23年的3 - 8年级。该存储库在（1）读取项目的语言特征，（2）段落的测试特征和（3）上下文特征上用元数据注释。与基线RMSE相比，具有所有这些功能的惩罚回归预测模型可以通过RMSE 0.52预测项目难度，而真实和预测难度之间的相关性为0.77。我们通过LLM（Modernbert，Bert和Llama）的嵌入来补充这些功能，这些嵌入方式略微改善了项目难度预测。当模型仅使用项目语言功能或LLM嵌入式时，预测性能相似，这表明可能只需其中一个特征类别。此项目难度预测模型可用于过滤和分类阅读项目，并将公开使用其他利益相关者使用。

Title: Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers

• Authors: Zixuan Gong, Jiaye Teng, Yong Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20681
• Pdf URL: https://arxiv.org/pdf/2502.20681
• Copy Paste: [[2502.20681]] Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers(https://arxiv.org/abs/2502.20681)
• Keywords: gpt
• Abstract: Transformers may exhibit two-stage training dynamics during the real-world training process. For instance, when training GPT-2 on the Counterfact dataset, the answers progress from syntactically incorrect to syntactically correct to semantically correct. However, existing theoretical analyses hardly account for this two-stage phenomenon. In this paper, we theoretically demonstrate how such two-stage training dynamics occur in transformers. Specifically, we analyze the dynamics of transformers using feature learning techniques under in-context learning regimes, based on a disentangled two-type feature structure. Such disentanglement of feature structure is general in practice, e.g., natural languages contain syntax and semantics, and proteins contain primary and secondary structures. To our best known, this is the first rigorous result regarding a two-stage optimization process in transformers. Additionally, a corollary indicates that such a two-stage process is closely related to the spectral properties of the attention weights, which accords well with empirical findings.
• 摘要：在现实世界训练过程中，变形金刚可以表现出两阶段的训练动力。例如，当对反事实数据集训练GPT-2时，答案从句法上不正确到语法上正确到语义上正确。但是，现有的理论分析几乎无法解释这一两阶段现象。在本文中，我们从理论上说明了这种两阶段的训练动态如何在变压器中发生。具体而言，我们基于分离的两种特征结构，使用特征学习技术在内部文化学习模式下使用特征学习技术来分析变压器的动力学。这种特征结构的分离在实践中通常是一般的，例如，自然语言包含语法和语义，蛋白质包含原始和次要结构。据我们最著名的是，这是针对变压器两阶段优化过程的第一个严格结果。另外，推论表明这种两个阶段的过程与注意力重量的光谱特性密切相关，这与经验发现非常相符。

Title: JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulation

• Authors: Yingbing Huang, Deming Chen, Abhishek K. Umrawal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20684
• Pdf URL: https://arxiv.org/pdf/2502.20684
• Copy Paste: [[2502.20684]] JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulation(https://arxiv.org/abs/2502.20684)
• Keywords: language model, gpt, llm
• Abstract: While large language models (LLMs) have made significant strides in generating coherent and contextually relevant text, they often function as opaque black boxes, trained on vast unlabeled datasets with statistical objectives, lacking an interpretable framework for responsible control. In this paper, we introduce JAM (Just A Move), a novel framework that interprets and controls text generation by integrating cause-effect analysis within the latent space of LLMs. Based on our observations, we uncover the inherent causality in LLM generation, which is critical for producing responsible and realistic outputs. Moreover, we explore latent vectors as fundamental components in LLM architectures, aiming to understand and manipulate them for more effective and efficient controllable text generation. We evaluate our framework using a range of tools, including the HHH criteria, toxicity reduction benchmarks, and GPT-4 alignment measures. Our results show that JAM achieves up to a 22% improvement over previous Controllable Text Generation (CTG) methods across multiple quantitative metrics and human-centric evaluations. Furthermore, JAM demonstrates greater computational efficiency compared to other CTG methods. These results highlight the effectiveness and efficiency of JAM for responsible and realistic text generation, paving the way for more interpretable and controllable models.
• 摘要：尽管大型语言模型（LLM）在生成连贯和上下文相关的文本方面取得了长足的进步，但它们通常充当不透明的黑匣子，在具有统计目标的庞大未标记的数据集中训练，缺乏可解释的负责控制框架。在本文中，我们介绍了JAM（只是一个举动），这是一个新颖的框架，该框架通过在LLM的潜在空间中集成因果分析来解释和控制文本生成。根据我们的观察，我们发现了LLM生成中固有的因果关系，这对于产生负责任和现实的产出至关重要。此外，我们将潜在向量作为LLM体系结构中的基本组成部分探索，旨在理解和操纵它们，以更有效，有效地可控文本生成。我们使用一系列工具来评估我们的框架，包括HHH标准，降低毒性基准和GPT-4对齐措施。我们的结果表明，在多个定量指标和以人为本的评估中，JAM比以前的可控文本生成（CTG）方法提高了22％。此外，与其他CTG方法相比，果酱表现出更高的计算效率。这些结果突出了果酱对负责任和现实的文本生成的有效性和效率，为更容易解释和可控制的模型铺平了道路。

Title: Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition

• Authors: Yifei Duan, Raphael Shang, Deng Liang, Yongqiang Cai
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20726
• Pdf URL: https://arxiv.org/pdf/2502.20726
• Copy Paste: [[2502.20726]] Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition(https://arxiv.org/abs/2502.20726)
• Keywords: language model
• Abstract: Language models can be viewed as functions that embed text into Euclidean space, where the quality of the embedding vectors directly determines model performance, training such neural networks involves various uncertainties. This paper focuses on improving the performance of pre-trained language models in zero-shot settings through a simple and easily implementable method. We propose a novel backward attention mechanism to enhance contextual information encoding. Evaluated on the Chinese Massive Text Embedding Benchmark (C-MTEB), our approach achieves significant improvements across multiple tasks, providing valuable insights for advancing zero-shot learning capabilities.
• 摘要：语言模型可以看作是将文本嵌入欧几里得空间中的功能，其中嵌入向量的质量直接决定了模型性能，训练此类神经网络涉及各种不确定性。本文着重于通过一种简单易于实现的方法在零拍设置中改善预训练的语言模型的性能。我们提出了一种新颖的落后注意机制，以增强上下文信息编码。根据中国大规模文本嵌入基准（C-MTEB）的评估，我们的方法在多个任务之间取得了重大改进，为促进零拍学习能力的有价值的见解提供了宝贵的见解。

Title: Teach-to-Reason with Scoring: Self-Explainable Rationale-Driven Multi-Trait Essay Scoring

• Authors: Heejin Do, Sangwon Ryu, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20748
• Pdf URL: https://arxiv.org/pdf/2502.20748
• Copy Paste: [[2502.20748]] Teach-to-Reason with Scoring: Self-Explainable Rationale-Driven Multi-Trait Essay Scoring(https://arxiv.org/abs/2502.20748)
• Keywords: language model, llm
• Abstract: Multi-trait automated essay scoring (AES) systems provide a fine-grained evaluation of an essay's diverse aspects. While they excel in scoring, prior systems fail to explain why specific trait scores are assigned. This lack of transparency leaves instructors and learners unconvinced of the AES outputs, hindering their practical use. To address this, we propose a self-explainable Rationale-Driven Multi-trait automated Essay scoring (RaDME) framework. RaDME leverages the reasoning capabilities of large language models (LLMs) by distilling them into a smaller yet effective scorer. This more manageable student model is optimized to sequentially generate a trait score followed by the corresponding rationale, thereby inherently learning to select a more justifiable score by considering the subsequent rationale during training. Our findings indicate that while LLMs underperform in direct AES tasks, they excel in rationale generation when provided with precise numerical scores. Thus, RaDME integrates the superior reasoning capacities of LLMs into the robust scoring accuracy of an optimized smaller model. Extensive experiments demonstrate that RaDME achieves both accurate and adequate reasoning while supporting high-quality multi-trait scoring, significantly enhancing the transparency of AES.
• 摘要：多特征自动论文评分（AES）系统提供了对论文各个方面的精细评估。尽管他们在评分方面表现出色，但先前的系统无法解释为什么分配特定特征分数。这种缺乏透明度使讲师和学习者不相信AES的输出，从而阻碍了他们的实际使用。为了解决这个问题，我们提出了一个可自我解释的理由驱动的多特征自动论文评分（RADME）框架。 Radme通过将其提炼成较小但有效的得分手来利用大语模型（LLM）的推理能力。优化了这种更易于管理的学生模型，以依次生成特征分数，然后是相应的基本原理，从而固有地学习通过考虑培训期间的后续理由来选择更合理的分数。我们的发现表明，当LLMS在直接AES任务中表现不佳时，它们在提供精确的数值分数时会出现理由生成。因此，RADME将LLM的优质推理能力集成到优化较小模型的强大评分精度中。广泛的实验表明，RADME在支持高质量的多特征评分的同时，可以实现准确和足够的推理，从而显着提高了AES的透明度。

Title: Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow

• Authors: Jiaqi Bai, Hongcheng Guo, Zhongyuan Peng, Jian Yang, Zhoujun Li, Mohan Li, Zhihong Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20750
• Pdf URL: https://arxiv.org/pdf/2502.20750
• Copy Paste: [[2502.20750]] Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow(https://arxiv.org/abs/2502.20750)
• Keywords: language model, llm, hallucination
• Abstract: Large vision-language models show tremendous potential in understanding visual information through human languages. However, they are prone to suffer from object hallucination, i.e., the generated image descriptions contain objects that do not exist in the image. In this paper, we reveal that object hallucination can be attributed to overconfidence in irrelevant visual features when soft visual tokens map to the LLM's word embedding space. Specifically, by figuring out the semantic similarity between visual tokens and LLM's word embedding, we observe that the smoothness of similarity distribution strongly correlates with the emergence of object hallucinations. To mitigate hallucinations, we propose using the Variational Information Bottleneck (VIB) to alleviate overconfidence by introducing stochastic noise, facilitating the constraining of irrelevant information. Furthermore, we propose an entropy-based noise-controlling strategy to enable the injected noise to be adaptively constrained regarding the smoothness of the similarity distribution. We adapt the proposed AdaVIB across distinct model architectures. Experimental results demonstrate that the proposed AdaVIB mitigates object hallucinations by effectively alleviating the overconfidence in irrelevant visual features, with consistent improvements on two object hallucination benchmarks.
• 摘要：大型视觉模型在通过人类语言理解视觉信息方面具有巨大的潜力。但是，它们容易患有对象幻觉，即生成的图像描述包含图像中不存在的对象。在本文中，我们揭示了对象幻觉可以归因于当柔软的视觉令牌映射到LLM的单词嵌入空间时，视觉特征的过度自信。具体而言，通过弄清视觉令牌和LLM单词嵌入之间的语义相似性，我们观察到相似性分布的平滑度与对象幻觉的出现密切相关。为了减轻幻觉，我们建议使用变分信息瓶颈（VIB）通过引入随机噪声来缓解过度自信，从而促进了无关信息的约束。此外，我们提出了一种基于熵的噪声控制策略，以使注入的噪声受到相似性分布的平滑度的适应性约束。我们在不同的模型体系结构上调整了拟议的Adavib。实验结果表明，提出的ADAVIB通过有效减轻无关的视觉特征的过度自信来减轻对象幻觉，并在两个对象幻觉基准上进行一致改进。

Title: The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents

• Authors: Yihong Tang, Kehai Chen, Xuefeng Bai, Zhengyu Niu, Bo Wang, Jie Liu, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20757
• Pdf URL: https://arxiv.org/pdf/2502.20757
• Copy Paste: [[2502.20757]] The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents(https://arxiv.org/abs/2502.20757)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have made remarkable advances in role-playing dialogue agents, demonstrating their utility in character simulations. However, it remains challenging for these agents to balance character portrayal utility with content safety because this essential character simulation often comes with the risk of generating unsafe content. To address this issue, we first conduct a systematic exploration of the safety-utility trade-off across multiple LLMs. Our analysis reveals that risk scenarios created by villain characters and user queries (referred to as risk coupling) contribute to this trade-off. Building on this, we propose a novel Adaptive Dynamic Multi-Preference (ADMP) method, which dynamically adjusts safety-utility preferences based on the degree of risk coupling and guides the model to generate responses biased toward utility or safety. We further introduce Coupling Margin Sampling (CMS) into coupling detection to enhance the model's ability to handle high-risk scenarios. Experimental results demonstrate that our approach improves safety metrics while maintaining utility.
• 摘要：大型语言模型（LLM）在角色扮演对话代理方面取得了显着进步，证明了它们在角色模拟中的效用。但是，对于这些代理商来说，平衡角色刻画实用程序与内容安全仍然是一项挑战，因为这种基本特征模拟通常会带来产生不安全内容的风险。为了解决这个问题，我们首先对多个LLM的安全性权衡进行系统探索。我们的分析表明，小人角色和用户查询创建的风险情景（称为风险耦合）会导致这种权衡。在此基础上，我们提出了一种新型的自适应动态多偏见（ADMP）方法，该方法根据风险耦合程度动态调整安全性偏好，并指导模型以产生对效用或安全性偏见的响应。我们进一步将耦合边距采样（CMS）引入耦合检测中，以增强模型处理高危场景的能力。实验结果表明，我们的方法在维持效用的同时改善了安全指标。

Title: Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective

• Authors: Yuko Nakagi, Keigo Tada, Sota Yoshino, Shinji Nishimoto, Yu Takagi
• Subjects: cs.CL, cs.AI, cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2502.20779
• Pdf URL: https://arxiv.org/pdf/2502.20779
• Copy Paste: [[2502.20779]] Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective(https://arxiv.org/abs/2502.20779)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often exhibit abrupt emergent behavior, whereby new abilities arise at certain points during their training. This phenomenon, commonly referred to as a ''phase transition'', remains poorly understood. In this study, we conduct an integrative analysis of such phase transitions by examining three interconnected perspectives: the similarity between LLMs and the human brain, the internal states of LLMs, and downstream task performance. We propose a novel interpretation for the learning dynamics of LLMs that vary in both training data and architecture, revealing that three phase transitions commonly emerge across these models during training: (1) alignment with the entire brain surges as LLMs begin adhering to task instructions Brain Alignment and Instruction Following, (2) unexpectedly, LLMs diverge from the brain during a period in which downstream task accuracy temporarily stagnates Brain Detachment and Stagnation, and (3) alignment with the brain reoccurs as LLMs become capable of solving the downstream tasks Brain Realignment and Consolidation. These findings illuminate the underlying mechanisms of phase transitions in LLMs, while opening new avenues for interdisciplinary research bridging AI and neuroscience.
• 摘要：大型语言模型（LLM）经常表现出突然的紧急行为，因此在训练过程中的某些时刻出现了新的能力。这种现象通常被称为“相位过渡”，仍然对此知之甚少。在这项研究中，我们通过检查三种相互联系的观点：LLMS与人脑之间的相似性，LLM的内部状态以及下游任务绩效，对此类相变的综合分析。我们对LLM的学习动力进行了一种新的解释，该解释在培训数据和体系结构中都不同，表明在培训期间，通常在这些模型中出现了三个相位过渡：（1）与LLM的整个大脑之间的一致性，因为LLM开始遵守任务指令大脑的对齐和指导，以下是脑部的脑误解，（2）在脑中的脑部脉络，（2）临时脑部的临时任务，该任务是在任务中的临时序列，并且在任务中的临时序列，并在任务中逐渐划分。停滞，（3）与脑重新占领的（3）能够解决下游任务的大脑重组和巩固。这些发现阐明了LLMS中相变的潜在机制，同时为跨学科研究开放了桥接AI和神经科学的新途径。

Title: GraphCheck: Multi-Path Fact-Checking with Entity-Relationship Graphs

• Authors: Hyewon Jeon, Jay-Yoon Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20785
• Pdf URL: https://arxiv.org/pdf/2502.20785
• Copy Paste: [[2502.20785]] GraphCheck: Multi-Path Fact-Checking with Entity-Relationship Graphs(https://arxiv.org/abs/2502.20785)
• Keywords: prompt
• Abstract: Automated fact-checking aims to assess the truthfulness of text based on relevant evidence, yet verifying complex claims requiring multi-hop reasoning remains a significant challenge. We propose GraphCheck, a novel framework that converts claims into entity-relationship graphs for comprehensive verification. By identifying relation between explicit entities and latent entities across multiple paths, GraphCheck enhances the adaptability and robustness of verification. Furthermore, we introduce DP-GraphCheck, a two-stage variant that improves performance by incorporating direct prompting as an initial filtering step. Experiments on the HOVER and EX-FEVER datasets show that our approach outperforms existing methods, particularly in multi-hop reasoning tasks. Furthermore, our two-stage framework generalizes well to other fact-checking pipelines, demonstrating its versatility.
• 摘要：自动化事实核对旨在根据相关证据评估文本的真实性，但验证需要多跳推理的复杂主张仍然是一个重大挑战。我们提出了GraphCheck，这是一个新颖的框架，将主张转换为实体关系图以进行全面验证。通过识别跨多个路径的显式实体与潜在实体之间的关系，GraphCheck可以增强验证的适应性和鲁棒性。此外，我们介绍了DP-Graphcheck，这是一种两阶段的变体，通过将直接提示作为初始滤波步骤结合来改善性能。悬停和前武器数据集的实验表明，我们的方法的表现优于现有方法，尤其是在多跳上推理任务中。此外，我们的两个阶段框架很好地概括了其他事实检查管道，证明了其多功能性。

Title: Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision

• Authors: Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, Sujian Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20790
• Pdf URL: https://arxiv.org/pdf/2502.20790
• Copy Paste: [[2502.20790]] Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision(https://arxiv.org/abs/2502.20790)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Recent advances in Large Language Models (LLMs) have highlighted the challenge of handling long-context tasks, where models need to reason over extensive input contexts to aggregate target information. While Chain-of-Thought (CoT) prompting has shown promise for multi-step reasoning, its effectiveness for long-context scenarios remains underexplored. Through systematic investigation across diverse tasks, we demonstrate that CoT's benefits generalize across most long-context scenarios and amplify with increasing context length. Motivated by this critical observation, we propose LongRePS, a process-supervised framework that teaches models to generate high-quality reasoning paths for enhanced long-context performance. Our framework incorporates a self-sampling mechanism to bootstrap reasoning paths and a novel quality assessment protocol specifically designed for long-context scenarios. Experimental results on various long-context benchmarks demonstrate the effectiveness of our approach, achieving significant improvements over outcome supervision baselines on both in-domain tasks (+13.6/+3.8 points for LLaMA/Qwen on MuSiQue) and cross-domain generalization (+9.3/+8.1 points on average across diverse QA tasks). Our code, data and trained models are made public to facilitate future research.
• 摘要：大型语言模型（LLMS）的最新进展突出了处理长篇小说任务的挑战，其中模型需要对广泛的输入上下文进行推理以汇总目标信息。虽然经过思考链（COT）提示表明了多步推理的希望，但其对长篇小说方案的有效性仍未得到充实。通过跨不同任务的系统调查，我们证明了COT的收益在大多数长篇小说方案中概括，并随上下文长度的增加而扩大。在这种批判性观察的促进下，我们提出了Longreps，这是一个程序监督的框架，该框架教授模型生成高质量的推理途径，以增强长篇小说性能。我们的框架将自我采样机制结合到引导性推理路径和专门为长篇文化场景设计的新型质量评估协议。对各种长篇文章基准测试的实验结果证明了我们方法的有效性，在域内任务（Musique on Musique上的Llama/Qwen+13.6/+3.8点）和跨域泛化（+9.3/+8.1分的平均QA任务）方面取得了显着改善（+13.6/+3.8点）。我们的代码，数据和受过训练的模型公开以促进未来的研究。

Title: Plan2Align: Predictive Planning Based Test-Time Preference Alignment in Paragraph-Level Machine Translation

• Authors: Kuang-Da Wang, Teng-Ruei Chen, Yu Heng Hung, Shuoyang Ding, Yueh-Hua Wu, Yu-Chiang Frank Wang, Chao-Han Huck Yang, Wen-Chih Peng, Ping-Chun Hsieh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20795
• Pdf URL: https://arxiv.org/pdf/2502.20795
• Copy Paste: [[2502.20795]] Plan2Align: Predictive Planning Based Test-Time Preference Alignment in Paragraph-Level Machine Translation(https://arxiv.org/abs/2502.20795)
• Keywords: language model, llm
• Abstract: Machine Translation (MT) has been predominantly designed for sentence-level translation using transformer-based architectures. While next-token prediction based Large Language Models (LLMs) demonstrate strong capabilities in long-text translation, non-extensive language models often suffer from omissions and semantic inconsistencies when processing paragraphs. Existing preference alignment methods improve sentence-level translation but fail to ensure coherence over extended contexts due to the myopic nature of next-token generation. We introduce Plan2Align, a test-time alignment framework that treats translation as a predictive planning problem, adapting Model Predictive Control to iteratively refine translation outputs. Experiments on WMT24 Discourse-Level Literary Translation show that Plan2Align significantly improves paragraph-level translation, achieving performance surpassing or on par with the existing training-time and test-time alignment methods on LLaMA-3.1 8B.
• 摘要：机器翻译（MT）的主要设计用于使用基于变压器的架构的句子级翻译。尽管下一步的基于预测的大语言模型（LLMS）在长篇文本翻译中表现出强大的功能，但在处理段落时，非扩展的语言模型通常会遭受遗漏和语义上的影响。现有的偏好一致性方法改善了句子级翻译，但由于下一代的近视性质，无法确保在扩展上下文上保持一致性。我们介绍了Plan2Align，这是一个测试时间对齐框架，将翻译视为预测计划问题，将模型预测控制适应迭代的改进翻译输出。 WMT24话语级文学翻译的实验表明，Plan2Align显着改善了段落级翻译，达到绩效超过或与现有的训练时间和测试时间对齐方式相同，在Llama-3.1 8B上。

Title: Learning to Substitute Components for Compositional Generalization

• Authors: Zhaoyi Li, Gangwei Jiang, Chenwang Wu, Ying Wei, Defu Lian, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20834
• Pdf URL: https://arxiv.org/pdf/2502.20834
• Copy Paste: [[2502.20834]] Learning to Substitute Components for Compositional Generalization(https://arxiv.org/abs/2502.20834)
• Keywords: language model, llm
• Abstract: Despite the rising prevalence of neural language models, recent empirical evidence suggests their deficiency in compositional generalization. One of the current de-facto solutions to this problem is compositional data augmentation, which aims to introduce additional compositional inductive bias. However, existing handcrafted augmentation strategies offer limited improvement when systematic generalization of neural language models requires multi-grained compositional bias (i.e., not limited to either lexical or structural biases alone) or when training sentences have an imbalanced difficulty distribution. To address these challenges, we first propose a novel compositional augmentation strategy called Component Substitution (CompSub), which enables multi-grained composition of substantial substructures across the entire training set. Furthermore, we introduce the Learning Component Substitution (LCS) framework. This framework empowers the learning of component substitution probabilities in CompSub in an end-to-end manner by maximizing the loss of neural language models, thereby prioritizing challenging compositions with elusive concepts and novel contexts. We extend the key ideas of CompSub and LCS to the recently emerging in-context learning scenarios of pre-trained large language models (LLMs), proposing the LCS-ICL algorithm to enhance the few-shot compositional generalization of state-of-the-art (SOTA) LLMs. Theoretically, we provide insights into why applying our algorithms to language models can improve compositional generalization performance. Empirically, our results on four standard compositional generalization benchmarks(SCAN, COGS, GeoQuery, and COGS-QL) demonstrate the superiority of CompSub, LCS, and LCS-ICL, with improvements of up to 66.5%, 10.3%, 1.4%, and 8.8%, respectively.
• 摘要：尽管神经语言模型的患病率上升，但最近的经验证据表明它们在组成概括方面的缺陷。当前解决此问题的事实上解决方案之一是组成数据增强，该数据旨在引入其他组成电感偏差。但是，当神经语言模型的系统概括需要多元透明的组成偏见（即，即不限于单独使用词汇或结构性偏见）或训练句子具有不平衡的难度分布时，现有的手工制作的增强策略可提供有限的改进。为了应对这些挑战，我们首先提出了一种称为组件替代（Compsub）的新颖组成促进策略，该策略能够在整个训练集中多透明大量的子结构组成。此外，我们介绍了学习组件替代（LCS）框架。该框架通过最大化神经语言模型的丧失，以端到端的方式学习组件替代概率，从而优先考虑具有难以捉摸的概念和新颖背景的具有挑战性的作品。我们将Compsub和LCS的关键思想扩展到了最近出现的预培训大语模型（LLMS）的新兴的内在学习方案，并提出了LCS-ICL算法，以增强最新的最新ART（SOTA）LLMS的少量组成概括。从理论上讲，我们提供了有关为什么将我们的算法应用于语言模型的洞察力可以改善组成概括性能。从经验上，我们对四个标准组成概括基准（扫描，齿轮，地球和齿轮）的结果证明了Compsub，LCS和LCS-ICL的优势，分别提高了66.5％，10.3％，1.4％，1.4％和8.8％。

Title: MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training

• Authors: Jonathan Drechsel, Anja Reusch, Steffen Herbold
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.20855
• Pdf URL: https://arxiv.org/pdf/2502.20855
• Copy Paste: [[2502.20855]] MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training(https://arxiv.org/abs/2502.20855)
• Keywords: language model
• Abstract: Mathematical formulas are a fundamental and widely used component in various scientific fields, serving as a universal language for expressing complex concepts and relationships. While state-of-the-art transformer models excel in processing and understanding natural language, they encounter challenges with mathematical notation, which involves a complex structure and diverse representations. This study focuses on the development of specialized training datasets to enhance the encoding of mathematical content. We introduce Math Mutator (MAMUT), a framework capable of generating equivalent and falsified versions of a given mathematical formula in LaTeX notation, effectively capturing the mathematical variety in notation of the same concept. Based on MAMUT, we have generated four large mathematical datasets containing diverse notation, which can be used to train language models with enhanced mathematical embeddings.
• 摘要：数学公式是各个科学领域的基本且广泛使用的组成部分，它是表达复杂概念和关系的通用语言。尽管最先进的变压器模型在处理和理解自然语言方面表现出色，但它们遇到了数学符号的挑战，涉及复杂的结构和各种表示。这项研究重点是开发专业培训数据集，以增强数学内容的编码。我们介绍了数学突变器（MAMUT），该框架能够在乳胶符号中生成给定数学公式的等效版本，有效地捕获了同一概念符号的数学变化。基于Mamut，我们生成了四个包含多种符号的大型数学数据集，这些数据集可用于使用增强的数学嵌入方式来训练语言模型。

Title: The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agents

• Authors: Yifan Duan, Yihong Tang, Xuefeng Bai, Kehai Chen, Juntao Li, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20859
• Pdf URL: https://arxiv.org/pdf/2502.20859
• Copy Paste: [[2502.20859]] The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agents(https://arxiv.org/abs/2502.20859)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) excel in both closed tasks (including problem-solving, and code generation) and open tasks (including creative writing), yet existing explanations for their capabilities lack connections to real-world human intelligence. To fill this gap, this paper systematically investigates LLM intelligence through the lens of ``human simulation'', addressing three core questions: (1) How do personality traits affect problem-solving in closed tasks? (2) How do traits shape creativity in open tasks? (3) How does single-agent performance influence multi-agent collaboration? By assigning Big Five personality traits to LLM agents and evaluating their performance in single- and multi-agent settings, we reveal that specific traits significantly influence reasoning accuracy (closed tasks) and creative output (open tasks). Furthermore, multi-agent systems exhibit collective intelligence distinct from individual capabilities, driven by distinguishing combinations of personalities. We demonstrate that LLMs inherently simulate human behavior through next-token prediction, mirroring human language, decision-making, and collaborative dynamics.
• 摘要：大型语言模型（LLMS）在两个封闭的任务（包括解决问题和代码生成）和开放任务（包括创意写作）中表现出色，但现有的对其能力的解释缺乏与现实世界人类智能的联系。为了填补这一空白，本文通过``人类仿真''的角度系统地研究了LLM智能，解决了三个核心问题：（1）人格特质如何影响封闭任务中的问题解决问题？ （2）特征如何在开放任务中塑造创造力？ （3）单格绩效如何影响多代理协作？通过为LLM代理商分配五个个性特征并评估其在单一和多代理设置中的性能，我们揭示了特定特征会显着影响推理准确性（封闭任务）和创意输出（开放任务）。此外，多代理系统表现出与个性组合驱动的不同能力不同的集体智能。我们证明，LLM固有地通过下一步的预测，反映人类语言，决策和协作动态来模拟人类行为。

Title: Do Language Models Understand Honorific Systems in Javanese?

• Authors: Mohammad Rifqi Farhansyah, Iwan Darmawan, Adryan Kusumawardhana, Genta Indra Winata, Alham Fikri Aji, Derry Tanti Wijaya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20864
• Pdf URL: https://arxiv.org/pdf/2502.20864
• Copy Paste: [[2502.20864]] Do Language Models Understand Honorific Systems in Javanese?(https://arxiv.org/abs/2502.20864)
• Keywords: language model
• Abstract: The Javanese language features a complex system of honorifics that vary according to the social status of the speaker, listener, and referent. Despite its cultural and linguistic significance, there has been limited progress in developing a comprehensive corpus to capture these variations for natural language processing (NLP) tasks. In this paper, we present Unggah-Ungguh, a carefully curated dataset designed to encapsulate the nuances of Unggah-Ungguh Basa, the Javanese speech etiquette framework that dictates the choice of words and phrases based on social hierarchy and context. Using Unggah-Ungguh, we assess the ability of language models (LMs) to process various levels of Javanese honorifics through classification and machine translation tasks. To further evaluate cross-lingual LMs, we conduct machine translation experiments between Javanese (at specific honorific levels) and Indonesian. Additionally, we explore whether LMs can generate contextually appropriate Javanese honorifics in conversation tasks, where the honorific usage should align with the social role and contextual cues. Our findings indicate that current LMs struggle with most honorific levels, exhibitinga bias toward certain honorific tiers.
• 摘要：爪哇语采用复杂的荣誉系统，该系统根据演讲者，听众和参考文献的社会地位而变化。尽管具有文化和语言意义，但在开发综合语料库以捕获这些自然语言处理（NLP）任务的差异方面的进展有限。在本文中，我们介绍了unggah-ungguh，这是一个经过精心策划的数据集，旨在封装unggah-ungguh basa的细微差别，unggah-ungguh basa是Javanese语音礼节框架，该框架决定了基于社交层次结构和上下文的单词和短语的选择。使用unggah-ungguh，我们评估语言模型（LMS）通过分类和机器翻译任务处理各种级别的Javanese荣誉的能力。为了进一步评估跨语义LMS，我们在爪哇人（特定的荣誉水平）和印度尼西亚人之间进行机器翻译实验。此外，我们探讨了LM是否可以在对话任务中产生上下文适当的爪哇荣誉，在这种任务中，荣誉用法应与社会角色和上下文提示保持一致。我们的发现表明，当前的LMS在大多数荣誉层面上挣扎，表现出对某些荣誉层的偏见。

Title: Better Benchmarking LLMs for Zero-Shot Dependency Parsing

• Authors: Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20866
• Pdf URL: https://arxiv.org/pdf/2502.20866
• Copy Paste: [[2502.20866]] Better Benchmarking LLMs for Zero-Shot Dependency Parsing(https://arxiv.org/abs/2502.20866)
• Keywords: llm
• Abstract: While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs.
• 摘要：尽管LLMS在零拍打任务中表现出色，但它们在语言挑战等语言挑战等方面的表现却较少。本文通过将其与无法访问输入句子的基线进行比较，包括在此上下文中未使用的基线，例如随机投影树或最佳线性布置，将其与无法访问输入句子的基线进行比较，从而研究了任务的最新开放式LLM。结果表明，大多数经过测试的LLM不能胜过最好的不知情基线，只有最新和最大的Llama版本使用的是大多数语言，并且仍然可以实现相当低的性能。因此，准确的零射击句法解析并不是开放的LLM即将进行的。

Title: ProBench: Benchmarking Large Language Models in Competitive Programming

• Authors: Lei Yang, Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20868
• Pdf URL: https://arxiv.org/pdf/2502.20868
• Copy Paste: [[2502.20868]] ProBench: Benchmarking Large Language Models in Competitive Programming(https://arxiv.org/abs/2502.20868)
• Keywords: language model, llm
• Abstract: With reasoning language models such as OpenAI-o3 and DeepSeek-R1 emerging, large language models (LLMs) have entered a new phase of development. However, existing benchmarks for coding evaluation are gradually inadequate to assess the capability of advanced LLMs in code reasoning. To bridge the gap for high-level code reasoning assessment, we propose ProBench to benchmark LLMs in competitive programming, drawing inspiration from the International Collegiate Programming Contest. ProBench collects a comprehensive set of competitive programming problems from Codeforces, Luogu, and Nowcoder platforms during the period from July to December 2024, obtaining real test results through online submissions to ensure the fairness and accuracy of the evaluation. We establish a unified problem attribute system, including difficulty grading and algorithm tagging. With carefully collected and annotated data in ProBench, we systematically assess 9 latest LLMs in competitive programming across multiple dimensions, including thought chain analysis, error type diagnosis, and reasoning depth evaluation. Experimental results show that QwQ-32B-Preview achieves the best score of 20.93 followed by DeepSeek-V3 with a score of 16.38, suggesting that models trained with specialized reasoning tasks significantly outperform general-purpose models (even larger than reasoning-oriented models) in programming. Further analysis also reveals key areas for programming capability enhancement, e.g., algorithm adaptability and reasoning sufficiency, providing important insights for the future development of reasoning models.
• 摘要：随着推理语言模型（例如OpenAI-O3和DeepSeek-R1），大型语言模型（LLMS）进入了一个新的开发阶段。但是，现有用于编码评估的基准逐渐不足以评估高级LLM在代码推理中的能力。为了弥合高级代码推理评估的差距，我们提议在竞争性编程中为LLMS进行基准LLM，从国际大学计划竞赛中汲取灵感。在2024年7月至1224年12月期间，Probench从CodeForces，Luogu和NowCoder平台中收集了一系列竞争性的编程问题，通过在线提交获得了实际的测试结果，以确保评估的公平性和准确性。我们建立了一个统一的问题属性系统，包括难以分级和算法标记。通过在探测器中进行了精心收集和注释的数据，我们系统地评估了跨多个维度的竞争编程中的9个最新LLM，包括思想链分析，错误类型诊断和推理深度评估。实验结果表明，QWQ-32B-preiveiew的最佳分数为20.93，其次是DeepSeek-V3，得分为16.38，这表明在编程中，经过专业推理任务训练的模型显着超过了通用通用模型（甚至比以推理为导向的模型更大）。进一步的分析还揭示了提高编程能力的关键领域，例如算法适应性和推理充足性，为推理模型的未来发展提供了重要的见解。

Title: Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions

• Authors: Matthias Orlikowski, Jiaxin Pei, Paul Röttger, Philipp Cimiano, David Jurgens, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20897
• Pdf URL: https://arxiv.org/pdf/2502.20897
• Copy Paste: [[2502.20897]] Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions(https://arxiv.org/abs/2502.20897)
• Keywords: language model, llm, prompt
• Abstract: People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also been used to label data, but recent work has shown that models perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemographic knowledge. Here, we ask whether LLMs can be trained to be accurate sociodemographic models of annotator variation. Using a curated dataset of five tasks with standardized sociodemographics, we show that models do improve in sociodemographic prompting when trained but that this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemographic patterns. Across all tasks, our results suggest that models learn little meaningful connection between sociodemographics and annotation, raising doubts about the current use of LLMs for simulating sociodemographic variation and behaviour.
• 摘要：人们在主观问题上的注释自然会有所不同，其中一些变化被认为是由于该人的社会人口统计学特征所致。 LLM也已用于标记数据，但最近的工作表明，在具有社会人口统计学属性的提示时，模型的性能很差，这表明固有的社会人口统计学知识有限。在这里，我们询问是否可以训练LLMS以成为注释器变化的准确社会人口统计学模型。使用具有标准化社会人口统计学的五个任务的策划数据集，我们表明模型在经过培训时的社会人口统计学提示确实有所改善，但是这种绩效增长很大程度上是由于模型学习注释特定于特定的行为而不是社会人口统计学模式。在所有任务中，我们的结果表明，模型在社会人口统计学和注释之间几乎没有有意义的联系，这引起了人们对当前使用LLMS模拟社会人口统计学变化和行为的疑问。

Title: A database to support the evaluation of gender biases in GPT-4o output

• Authors: Luise Mehner, Lena Alicija Philine Fiedler, Sabine Ammon, Dorothea Kolossa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20898
• Pdf URL: https://arxiv.org/pdf/2502.20898
• Copy Paste: [[2502.20898]] A database to support the evaluation of gender biases in GPT-4o output(https://arxiv.org/abs/2502.20898)
• Keywords: language model, gpt, llm
• Abstract: The widespread application of Large Language Models (LLMs) involves ethical risks for users and societies. A prominent ethical risk of LLMs is the generation of unfair language output that reinforces or exacerbates harm for members of disadvantaged social groups through gender biases (Weidinger et al., 2022; Bender et al., 2021; Kotek et al., 2023). Hence, the evaluation of the fairness of LLM outputs with respect to such biases is a topic of rising interest. To advance research in this field, promote discourse on suitable normative bases and evaluation methodologies, and enhance the reproducibility of related studies, we propose a novel approach to database construction. This approach enables the assessment of gender-related biases in LLM-generated language beyond merely evaluating their degree of neutralization.
• 摘要：大型语言模型（LLM）的广泛应用涉及用户和社会的道德风险。 LLMS的一个突出的道德风险是产生不公平的语言输出，通过性别偏见加强或加剧了对处境不利的社会群体成员的伤害（Weidinger等，2022； Bender等，2021； Kotekek等，Kotek等，2023）。因此，对LLM输出相对于此类偏见的公平性的评估是兴趣上升的话题。为了推进该领域的研究，请促进有关适当的规范基础和评估方法的论述，并增强相关研究的可重复性，我们提出了一种新颖的数据库构建方法。这种方法可以评估LLM生成语言中与性别相关的偏见，而不仅仅是评估其中和程度。

Title: Automated Evaluation of Meter and Rhyme in Russian Generative and Human-Authored Poetry

• Authors: Ilya Koziev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20931
• Pdf URL: https://arxiv.org/pdf/2502.20931
• Copy Paste: [[2502.20931]] Automated Evaluation of Meter and Rhyme in Russian Generative and Human-Authored Poetry(https://arxiv.org/abs/2502.20931)
• Keywords: language model
• Abstract: Generative poetry systems require effective tools for data engineering and automatic evaluation, particularly to assess how well a poem adheres to versification rules, such as the correct alternation of stressed and unstressed syllables and the presence of rhymes. In this work, we introduce the Russian Poetry Scansion Tool library designed for stress mark placement in Russian-language syllabo-tonic poetry, rhyme detection, and identification of defects of poeticness. Additionally, we release RIFMA -- a dataset of poem fragments spanning various genres and forms, annotated with stress marks. This dataset can be used to evaluate the capability of modern large language models to accurately place stress marks in poetic texts. The published resources provide valuable tools for researchers and practitioners in the field of creative generative AI, facilitating advancements in the development and evaluation of generative poetry systems.
• 摘要：生成诗系统需要有效的数据工程和自动评估工具，尤其是评估一首诗遵守的录音规则，例如压力和无压力音节的正确交替以及押韵的存在。在这项工作中，我们介绍了俄罗斯诗歌扫描工具库，旨在在俄罗斯语言课程诗歌诗歌，押韵检测和诗意缺陷的鉴定中供应压力标记。此外，我们释放了RIFMA-遍布各种流派和形式的诗片段的数据集，并用应力标记注释。该数据集可用于评估现代大型语言模型的能力，以准确地将压力标记在诗意文本中贴上。已发表的资源为创造性生成AI领域的研究人员和从业人员提供了宝贵的工具，从而促进了生成诗系统的发展和评估的进步。

Title: WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval

• Authors: Michael Dinzinger, Laura Caspari, Kanishka Ghosh Dastidar, Jelena Mitrović, Michael Granitzer
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.20936
• Pdf URL: https://arxiv.org/pdf/2502.20936
• Copy Paste: [[2502.20936]] WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval(https://arxiv.org/abs/2502.20936)
• Keywords: llm
• Abstract: We present WebFAQ, a large-scale collection of open-domain question answering datasets derived from FAQ-style this http URL annotations. In total, the data collection consists of 96 million natural question-answer (QA) pairs across 75 languages, including 47 million (49%) non-English samples. WebFAQ further serves as the foundation for 20 monolingual retrieval benchmarks with a total size of 11.2 million QA pairs (5.9 million non-English). These datasets are carefully curated through refined filtering and near-duplicate detection, yielding high-quality resources for training and evaluating multilingual dense retrieval models. To empirically confirm WebFAQ's efficacy, we use the collected QAs to fine-tune an in-domain pretrained XLM-RoBERTa model. Through this process of dataset-specific fine-tuning, the model achieves significant retrieval performance gains, which generalize - beyond WebFAQ - to other multilingual retrieval benchmarks evaluated in zero-shot setting. Last but not least, we utilize WebFAQ to construct a set of QA-aligned bilingual corpora spanning over 1000 language pairs using state-of-the-art bitext mining and automated LLM-assessed translation evaluation. Due to our advanced, automated method of bitext dataset generation, the resulting bilingual corpora demonstrate higher translation quality compared to similar datasets. WebFAQ and all associated resources are publicly available on GitHub and HuggingFace.
• 摘要：我们提出WebFAQ，这是一个大规模的开放域问题，答复了来自FAQ风格的此HTTP URL注释的数据集。总共数据收集包括75种语言的9600万天然提问（QA）对，其中包括4700万（49％）非英语样本。 WebFAQ进一步为20个单语检索基准的基础，总尺寸为1120万QA（590万非英语）。这些数据集通过精制的过滤和近乎填充的检测仔细策划，从而为训练和评估多语言密集检索模型提供了高质量的资源。为了从经验上确认WebFAQ的功效，我们使用收集的QA来微调验证的XLM-ROBERTA型号。通过这个数据集特定的微调过程，该模型实现了显着的检索性能提升，这些绩效增长超出了WebFAQ，可以将其概括为以零弹射设置评估的其他多语言检索基准。最后但并非最不重要的一点是，我们利用WebFAQ构建了一组与最新的Bitext挖掘和自动化LLM评估的翻译评估的QA一致的双语Corpora，跨越了1000多个语言对。由于我们对BiteXT数据集生成的高级，自动化的方法，与类似数据集相比，由此产生的双语Corpera表现出更高的翻译质量。 WebFAQ和所有相关资源都在GitHub和HuggingFace上公开可用。

Title: Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMs

• Authors: Weixiang Zhao, Yulin Hu, Yang Deng, Jiahe Guo, Xingyu Sui, Xinyang Han, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20968
• Pdf URL: https://arxiv.org/pdf/2502.20968
• Copy Paste: [[2502.20968]] Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMs(https://arxiv.org/abs/2502.20968)
• Keywords: language model, llm
• Abstract: Role-playing enables large language models (LLMs) to engage users in immersive and personalized interactions, but it also introduces significant safety risks. Existing role-play fine-tuning techniques improve role adaptability but may degrade safety performance, particularly for villainous characters. In this work, we conduct the first comprehensive assessment of role-play fine-tuning risks by training 95 role-specific LLMs using RoleBench. Our experiments reveal that role-play fine-tuning leads to a noticeable decline in safety performance, with safety risks varying based on character traits. To tackle this challenge, we propose Safety-Aware Role-Play Fine-Tuning (SaRFT), a novel method designed to balance role-playing capabilities and safety. Extensive experiments on LLaMA-3-8B-Instruct, Gemma-2-9B-it, and Qwen2.5-7B-Instruct demonstrate that SaRFT consistently outperforms state-of-the-art baselines under both LoRA and full-parameter fine-tuning settings. Our findings highlight the necessity of role-adaptive safety measures and provide insights into mitigating role-specific safety risks in role-playing LLMs.
• 摘要：角色扮演使大型语言模型（LLM）能够吸引用户进行沉浸式和个性化的互动，但也引入了重大的安全风险。现有的角色扮演微调技术可提高角色适应性，但可能会降低安全性能，尤其是针对邪恶的角色。在这项工作中，我们通过使用角色盒培训95个角色特定的LLM对角色扮演微调风险进行了首次全面评估。我们的实验表明，角色扮演微调导致安全性能的明显下降，安全风险根据特征性状而变化。为了应对这一挑战，我们提出了安全感知的角色扮演微调（SARFT），这是一种旨在平衡角色扮演能力和安全性的新颖方法。在Llama-3-8B教学，GEMMA-2-9B-IT和QWEN2.5-7B-INSTRUCT上进行的广泛实验表明，Sarft在Lora和Full-Paremeter Fine-Thuning设置下始终超过最先进的基线。我们的发现突出了作用自适应安全措施的必要性，并为减轻角色扮演LLM中的特定角色安全风险提供了见解。

Title: Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?

• Authors: Perla Al Almaoui, Pierrette Bouillon, Simon Hengchen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20973
• Pdf URL: https://arxiv.org/pdf/2502.20973
• Copy Paste: [[2502.20973]] Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?(https://arxiv.org/abs/2502.20973)
• Keywords: llm
• Abstract: In this era of rapid technological advancements, communication continues to evolve as new linguistic phenomena emerge. Among these is Arabizi, a hybrid form of Arabic that incorporates Latin characters and numbers to represent the spoken dialects of Arab communities. Arabizi is widely used on social media and allows people to communicate in an informal and dynamic way, but it poses significant challenges for machine translation due to its lack of formal structure and deeply embedded cultural nuances. This case study arises from a growing need to translate Arabizi for gisting purposes. It evaluates the capacity of different LLMs to decode and translate Arabizi, focusing on multiple Arabic dialects that have rarely been studied up until now. Using a combination of human evaluators and automatic metrics, this research project investigates the model's performance in translating Arabizi into both Modern Standard Arabic and English. Key questions explored include which dialects are translated most effectively and whether translations into English surpass those into Arabic.
• 摘要：在这个快速的技术进步时代，随着新的语言现象的出现，沟通不断发展。其中包括阿拉伯语，这是一种混合形式的阿拉伯语，其中包含拉丁字符和数字，以代表阿拉伯社区的口语方言。 Arabizi被广泛用于社交媒体，并允许人们以非正式和动态的方式进行交流，但是由于缺乏正式结构和深层嵌入的文化细微差别，它对机器翻译构成了重大挑战。该案例研究源于越来越需要将阿拉伯式翻译出来的目的。它评估了不同的LLM来解码和翻译阿拉伯式的能力，重点是到目前为止很少研究的多个阿拉伯方言。该研究项目结合了人类评估者和自动指标，研究了该模型在将阿拉伯语转化为现代标准阿拉伯语和英语方面的表现。探讨的关键问题包括最有效地翻译哪些方言，以及翻译成英文的文字是否超过了阿拉伯语。

Title: Set-Theoretic Compositionality of Sentence Embeddings

• Authors: Naman Bansal, Yash mahajan, Sanjeev Sinha, Santu Karmaker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20975
• Pdf URL: https://arxiv.org/pdf/2502.20975
• Copy Paste: [[2502.20975]] Set-Theoretic Compositionality of Sentence Embeddings(https://arxiv.org/abs/2502.20975)
• Keywords: language model, llm
• Abstract: Sentence encoders play a pivotal role in various NLP tasks; hence, an accurate evaluation of their compositional properties is paramount. However, existing evaluation methods predominantly focus on goal task-specific performance. This leaves a significant gap in understanding how well sentence embeddings demonstrate fundamental compositional properties in a task-independent context. Leveraging classical set theory, we address this gap by proposing six criteria based on three core "set-like" compositions/operations: \textit{TextOverlap}, \textit{TextDifference}, and \textit{TextUnion}. We systematically evaluate $7$ classical and $9$ Large Language Model (LLM)-based sentence encoders to assess their alignment with these criteria. Our findings show that SBERT consistently demonstrates set-like compositional properties, surpassing even the latest LLMs. Additionally, we introduce a new dataset of ~$192$K samples designed to facilitate future benchmarking efforts on set-like compositionality of sentence embeddings.
• 摘要：句子编码在各种NLP任务中起关键作用；因此，对其组成特性的准确评估至关重要。但是，现有的评估方法主要集中在特定于目标任务的性能上。这在理解句子嵌入如何在任务无关的环境中表现出基本的构图特性方面留下了很大的差距。利用经典集理论，我们通过基于三个核心“类似于设置的”组成/操作提出六个标准来解决这一差距：\ textit {textoverLap}，\ textit {textdifference}和\ textit {textunion}。我们系统地评估了$ 7 $经典和9美元的大语言模型（LLM）的句子编码器，以评估其与这些标准的一致性。我们的发现表明，Sbert始终展示了类似固定的成分特性，甚至超过了最新的LLM。此外，我们推出了一个〜$ 192 $ k样品的新数据集，旨在促进未来的基准测试措施，以实现嵌入句子的固定组成性。

Title: UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representation

• Authors: Thanet Markchom, Tong Wu, Liting Huang, Huizhi Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.20984
• Pdf URL: https://arxiv.org/pdf/2502.20984
• Copy Paste: [[2502.20984]] UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representation(https://arxiv.org/abs/2502.20984)
• Keywords: language model, llm
• Abstract: SemEval-2025 Task 1 focuses on ranking images based on their alignment with a given nominal compound that may carry idiomatic meaning in both English and Brazilian Portuguese. To address this challenge, this work uses generative large language models (LLMs) and multilingual CLIP models to enhance idiomatic compound representations. LLMs generate idiomatic meanings for potentially idiomatic compounds, enriching their semantic interpretation. These meanings are then encoded using multilingual CLIP models, serving as representations for image ranking. Contrastive learning and data augmentation techniques are applied to fine-tune these embeddings for improved performance. Experimental results show that multimodal representations extracted through this method outperformed those based solely on the original nominal compounds. The fine-tuning approach shows promising outcomes but is less effective than using embeddings without fine-tuning. The source code used in this paper is available at this https URL.
• 摘要：Semeval-2025任务1专注于基于其与给定标称化合物的对齐方式对图像进行排名，该化合物可能具有英语和巴西葡萄牙语的惯用含义。为了应对这一挑战，这项工作使用生成的大语言模型（LLM）和多语言剪辑模型来增强惯用化合物表示。 LLMS生成了潜在惯用化合物的惯用含义，从而丰富了其语义解释。然后，使用多语言剪辑模型对这些含义进行编码，并用作图像排名的表示形式。对比度学习和数据增强技术可用于微调这些嵌入，以提高性能。实验结果表明，通过这种方法提取的多模式表示法优于仅基于原始名义化合物的多模式表示。微调方法显示出令人鼓舞的结果，但不如使用嵌入而无需微调。本文中使用的源代码可在此HTTPS URL上获得。

Title: Capability Localization: Capabilities Can be Localized rather than Individual Knowledge

• Authors: Xiusheng Huang, Jiaxiang Liu, Yequan Wang, Jun Zhao, Kang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.20992
• Pdf URL: https://arxiv.org/pdf/2502.20992
• Copy Paste: [[2502.20992]] Capability Localization: Capabilities Can be Localized rather than Individual Knowledge(https://arxiv.org/abs/2502.20992)
• Keywords: language model
• Abstract: Large scale language models have achieved superior performance in tasks related to natural language processing, however, it is still unclear how model parameters affect performance improvement. Previous studies assumed that individual knowledge is stored in local parameters, and the storage form of individual knowledge is dispersed parameters, parameter layers, or parameter chains, which are not unified. We found through fidelity and reliability evaluation experiments that individual knowledge cannot be localized. Afterwards, we constructed a dataset for decoupling experiments and discovered the potential for localizing data commonalities. To further reveal this phenomenon, this paper proposes a Commonality Neuron Localization (CNL) method, which successfully locates commonality neurons and achieves a neuron overlap rate of 96.42% on the GSM8K dataset. Finally, we have demonstrated through cross data experiments that commonality neurons are a collection of capability neurons that possess the capability to enhance performance. Our code is available at this https URL.
• 摘要：大规模语言模型在与自然语言处理相关的任务中取得了卓越的性能，但是，尚不清楚模型参数如何影响性能改善。先前的研究假设单个知识存储在本地参数中，并且个人知识的存储形式是分散的参数，参数层或参数链，这些参数链未统一。我们通过忠诚度和可靠性评估实验发现，个人知识不能本地化。之后，我们构建了一个数据集用于解耦实验，并发现了本地化数据共同点的潜力。为了进一步揭示这一现象，本文提出了一种通用性神经元定位（CNL）方法，该方法成功地定位了通用性神经元，并在GSM8K数据集上达到了96.42％的神经元重叠率。最后，我们通过跨数据实验证明了共同性神经元是具有提高性能能力的能力神经元的集合。我们的代码可在此HTTPS URL上找到。

Title: PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues

• Authors: Fangxu Yu, Lai Jiang, Shenyi Huang, Zhen Wu, Xinyu Dai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.21017
• Pdf URL: https://arxiv.org/pdf/2502.21017
• Copy Paste: [[2502.21017]] PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues(https://arxiv.org/abs/2502.21017)
• Keywords: language model, llm
• Abstract: The ability to understand and predict the mental states of oneself and others, known as the Theory of Mind (ToM), is crucial for effective social interactions. Recent research has emerged to evaluate whether Large Language Models (LLMs) exhibit a form of ToM. Although recent studies have evaluated ToM in LLMs, existing benchmarks focus predominantly on physical perception with principles guided by the Sally-Anne test in synthetic stories and conversations, failing to capture the complex psychological activities of mental states in real-life social interactions. To mitigate this gap, we propose PersuasiveToM, a benchmark designed to evaluate the ToM abilities of LLMs in persuasive dialogues. Our framework introduces two categories of questions: (1) ToM Reasoning, assessing the capacity of LLMs to track evolving mental states (e.g., desire shifts in persuadees), and (2) ToM Application, evaluating whether LLMs can take advantage of inferred mental states to select effective persuasion strategies (e.g., emphasize rarity) and evaluate the effectiveness of persuasion strategies. Experiments across eight state-of-the-art LLMs reveal that while models excel on multiple questions, they struggle to answer questions that need tracking the dynamics and shifts of mental states and understanding the mental states in the whole dialogue comprehensively. Our aim with PersuasiveToM is to allow an effective evaluation of the ToM reasoning ability of LLMs with more focus on complex psychological activities. Our code is available at this https URL.
• 摘要：理解和预测自己和他人的心理状态的能力（称为心理理论（汤姆））对于有效的社会互动至关重要。最近的研究已经出现，以评估大语模型（LLMS）是否表现出汤姆的形式。尽管最近的研究对LLM中的TOM进行了评估，但现有的基准主要集中在身体感知上，其原则是由Sally-Anne测试在合成故事和对话中的指导下，未能捕获现实社交互动中精神状态的复杂心理活动。为了减轻这一差距，我们提出了Selpuasivetom，这是一种基准测试，旨在评估LLMS在有说服力的对话中的TOM能力。 Our framework introduces two categories of questions: (1) ToM Reasoning, assessing the capacity of LLMs to track evolving mental states (e.g., desire shifts in persuadees), and (2) ToM Application, evaluating whether LLMs can take advantage of inferred mental states to select effective persuasion strategies (e.g., emphasize rarity) and evaluate the effectiveness of persuasion strategies.在八个最先进的LLMS上进行的实验表明，尽管模型在多个问题上表现出色，但他们努力回答需要经过全面跟踪精神状态的动态和转变的问题，并在整个对话中全面了解精神状态。我们的说服力的目的是允许对LLM的TOM推理能力进行有效评估，而更加关注复杂的心理活动。我们的代码可在此HTTPS URL上找到。

Title: Beyond Words: A Latent Memory Approach to Internal Reasoning in LLMs

• Authors: José I. Orlicki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.21030
• Pdf URL: https://arxiv.org/pdf/2502.21030
• Copy Paste: [[2502.21030]] Beyond Words: A Latent Memory Approach to Internal Reasoning in LLMs(https://arxiv.org/abs/2502.21030)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: Recent advances in large language models (LLMs) have popularized the chain-of-thought (CoT) paradigm, in which models produce explicit reasoning steps in natural language. Although this approach improves interpretability and facilitates external auditing, it may not represent the most computationally efficient method for internal reasoning. In contrast, human cognition relies on implicit mental representations that recall past sensory and episodic information without requiring complete verbalization. In this paper, we propose a framework that integrates implicit mental representations into the internal reasoning processes of LLMs. Preliminary experiments indicate that incorporating an Implicit Memory Module (IMM) into a simple GPT model yields a reduction of between 35% and 57% in final training loss compared to a regular GPT baseline. The addition of an explicit interpretability channel (e.g., a chain-of-thought decoder) is straightforward to implement within this approach. We outline theoretical foundations, propose technical mechanisms to scale the memory module, and discuss how these ideas may lead to more efficient and robust reasoning, with optional future extensions for explicit auditability.
• 摘要：大型语言模型（LLMS）的最新进展已推广了对经营链（COT）范式的推广，其中模型以自然语言产生明确的推理步骤。尽管这种方法可提高可解释性并促进外部审计，但它可能并不代表内部推理的最有效方法。相比之下，人类的认知依赖于隐性心理表示，这些心理表征回忆起过去的感觉和情节信息而无需完全言语。在本文中，我们提出了一个将隐式心理表示形式集成到LLM的内部推理过程中的框架。初步实验表明，与常规的GPT基线相比，最终训练损失中，将隐式存储模块（IMM）纳入简单的GPT模型中的最终训练损失的降低在35％至57％之间。在这种方法中可以简单地实现明确的可解释性渠道（例如，经过思考的解码器）。我们概述了理论基础，提出了技术机制来扩展内存模块，并讨论了这些想法如何导致更有效和稳健的推理，并以未来的未来扩展进行明确的可审核性。

Title: CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation

• Authors: Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.21074
• Pdf URL: https://arxiv.org/pdf/2502.21074
• Copy Paste: [[2502.21074]] CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation(https://arxiv.org/abs/2502.21074)
• Keywords: language model, llm, chain-of-thought
• Abstract: Chain-of-Thought (CoT) enhances Large Language Models (LLMs) by enabling step-by-step reasoning in natural language. However, the language space may be suboptimal for reasoning. While implicit CoT methods attempt to enable reasoning without explicit CoT tokens, they have consistently lagged behind explicit CoT method in task performance. We propose CODI (Continuous Chain-of-Thought via Self-Distillation), a novel framework that distills CoT into a continuous space, where a shared model acts as both teacher and student, jointly learning explicit and implicit CoT while aligning their hidden activation on the token generating the final answer. CODI is the first implicit CoT method to match explicit CoT's performance on GSM8k while achieving 3.1x compression, surpassing the previous state-of-the-art by 28.2% in accuracy. Furthermore, CODI demonstrates scalability, robustness, and generalizability to more complex CoT datasets. Additionally, CODI retains interpretability by decoding its continuous thoughts, making its reasoning process transparent. Our findings establish implicit CoT as not only a more efficient but a powerful alternative to explicit CoT.
• 摘要：经过思考链（COT）通过以自然语言逐步推理来增强大语言模型（LLM）。但是，语言空间可能是推理的次优。虽然隐式COT方法试图在没有明确的COT代币的情况下启用推理，但它们一直落后于任务性能的显式COT方法。我们提出了Codi（通过自distillation的连续思维链），这是一个新颖的框架，将COT提炼成一个连续的空间，在该空间中，共享的模型既是教师又是学生，共同学习明确和隐含的COT，同时将其隐藏的激活对齐，以产生最终答案。 CODI是第一个与GSM8K上显式COT的性能相匹配的同时，同时达到3.1倍压缩，准确性超过了28.2％的先前最新ART。此外，Codi证明了对更复杂的COT数据集的可扩展性，鲁棒性和概括性。此外，CODI通过解码其连续思想来保留可解释性，从而使其推理过程透明。我们的发现将隐式婴儿床不仅是更有效的，而且是显式婴儿床的有力替代方案。

Title: PASemiQA: Plan-Assisted Agent for Question Answering on Semi-Structured Data with Text and Relational Information

• Authors: Hansi Yang, Qi Zhang, Wei Jiang, Jianguo Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.21087
• Pdf URL: https://arxiv.org/pdf/2502.21087
• Copy Paste: [[2502.21087]] PASemiQA: Plan-Assisted Agent for Question Answering on Semi-Structured Data with Text and Relational Information(https://arxiv.org/abs/2502.21087)
• Keywords: language model, llm, hallucination, retrieval-augmented generation, agent
• Abstract: Large language models (LLMs) have shown impressive abilities in answering questions across various domains, but they often encounter hallucination issues on questions that require professional and up-to-date knowledge. To address this limitation, retrieval-augmented generation (RAG) techniques have been proposed, which retrieve relevant information from external sources to inform their responses. However, existing RAG methods typically focus on a single type of external data, such as vectorized text database or knowledge graphs, and cannot well handle real-world questions on semi-structured data containing both text and relational information. To bridge this gap, we introduce PASemiQA, a novel approach that jointly leverages text and relational information in semi-structured data to answer questions. PASemiQA first generates a plan to identify relevant text and relational information to answer the question in semi-structured data, and then uses an LLM agent to traverse the semi-structured data and extract necessary information. Our empirical results demonstrate the effectiveness of PASemiQA across different semi-structured datasets from various domains, showcasing its potential to improve the accuracy and reliability of question answering systems on semi-structured data.
• 摘要：大型语言模型（LLM）在回答各个领域的问题方面表现出了令人印象深刻的能力，但是它们经常在需要专业和最新知识的问题上遇到幻觉问题。为了解决这一限制，已经提出了检索功能（RAG）技术，该技术从外部来源检索相关信息以告知其答复。但是，现有的抹布方法通常集中于单一类型的外部数据，例如矢量化文本数据库或知识图，并且不能很好地处理包含文本和关系信息的半结构化数据上的现实世界问题。为了弥合这一差距，我们介绍了Pasemiqa，这是一种新颖的方法，该方法共同利用半结构化数据中的文本和关系信息来回答问题。 Pasemiqa首先生成一个计划，以确定相关的文本和关系信息，以在半结构数据中回答该问题，然后使用LLM代理来遍历半结构化数据并提取必要的信息。我们的经验结果证明了Pasemiqa在来自各个领域的不同半结构数据集中的有效性，从而展示了其在半结构化数据上提高问题答案系统的准确性和可靠性的潜力。

Title: Generating patient cohorts from electronic health records using two-step retrieval-augmented text-to-SQL generation

• Authors: Angelo Ziletti, Leonardo D'Ambrosi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.21107
• Pdf URL: https://arxiv.org/pdf/2502.21107
• Copy Paste: [[2502.21107]] Generating patient cohorts from electronic health records using two-step retrieval-augmented text-to-SQL generation(https://arxiv.org/abs/2502.21107)
• Keywords: language model, retrieval augmented generation
• Abstract: Clinical cohort definition is crucial for patient recruitment and observational studies, yet translating inclusion/exclusion criteria into SQL queries remains challenging and manual. We present an automated system utilizing large language models that combines criteria parsing, two-level retrieval augmented generation with specialized knowledge bases, medical concept standardization, and SQL generation to retrieve patient cohorts with patient funnels. The system achieves 0.75 F1-score in cohort identification on EHR data, effectively capturing complex temporal and logical relationships. These results demonstrate the feasibility of automated cohort generation for epidemiological research.
• 摘要：临床队列的定义对于患者招募和观察性研究至关重要，但是将包含/排除标准转化为SQL查询仍然具有挑战性和手动。我们提出了一个使用大型语言模型的自动化系统，该系统结合了标准解析，两级检索增强产生，具有专门知识库，医学概念标准化和SQL生成，以使用患者渠道检索患者同类。该系统在EHR数据上的队列识别中达到0.75 F1得分，有效地捕获了复杂的时间和逻辑关系。这些结果证明了自动同伴生成流行病学研究的可行性。

Title: ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer

• Authors: Omer Goldman, Uri Shaham, Dan Malkin, Sivan Eiger, Avinatan Hassidim, Yossi Matias, Joshua Maynez, Adi Mayrav Gilady, Jason Riesa, Shruti Rijhwani, Laura Rimell, Idan Szpektor, Reut Tsarfaty, Matan Eyal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.21228
• Pdf URL: https://arxiv.org/pdf/2502.21228
• Copy Paste: [[2502.21228]] ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer(https://arxiv.org/abs/2502.21228)
• Keywords: language model, llm
• Abstract: To achieve equitable performance across languages, multilingual large language models (LLMs) must be able to abstract knowledge beyond the language in which it was acquired. However, the current literature lacks reliable ways to measure LLMs' capability of cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA (CBQA) dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. We detected information with uneven coverage across languages by controlling for presence and absence of Wikipedia articles in 12 languages. We generated knowledge-seeking questions in a source language, for which the answer appears in a relevant Wikipedia article and translated them to all other 11 languages, for which the respective Wikipedias lack equivalent articles. Assuming that Wikipedia reflects the prominent knowledge in the LLM's training data, to solve ECLeKTic's CBQA task the model is required to transfer knowledge between languages. Experimenting with 8 LLMs, we show that SOTA models struggle to effectively share knowledge across, languages even if they can predict the answer well for queries in the same language the knowledge was acquired in.
• 摘要：为了在跨语言中实现公平的表现，多语言大语模型（LLM）必须能够抽象超出其获取语言的知识。但是，目前的文献缺乏可靠的方法来衡量LLMS的跨语性知识转移能力。为此，我们提出了Eclektic，这是一种多语言闭合书质量质量图（CBQA）数据集，该数据集以简单的黑盒方式评估跨语性知识转移。我们通过控制和不存在12种语言的Wikipedia文章来检测跨语言覆盖不平的信息。我们用一种源语言产生了寻求知识的问题，答案在相关的Wikipedia文章中出现，并将其翻译成其他所有11种语言，因此各自的Wikipedias缺乏等效文章。假设Wikipedia反映了LLM培训数据中的重要知识，以解决Eclektic的CBQA任务，模型需要在语言之间传输知识。在尝试8个LLMS时，我们表明SOTA模型很难有效地跨越跨语言，即使他们可以用相同语言的相同语言来预测答案，从而获得了知识。

Title: Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs

• Authors: Xiaomin Li, Zhou Yu, Ziji Zhang, Yingying Zhuang, Swair Shah, Anurag Beniwal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.21239
• Pdf URL: https://arxiv.org/pdf/2502.21239
• Copy Paste: [[2502.21239]] Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs(https://arxiv.org/abs/2502.21239)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across diverse tasks by encoding vast amounts of factual knowledge. However, they are still prone to hallucinations, generating incorrect or misleading information, often accompanied by high uncertainty. Existing methods for hallucination detection primarily focus on quantifying internal uncertainty, which arises from missing or conflicting knowledge within the model. However, hallucinations can also stem from external uncertainty, where ambiguous user queries lead to multiple possible interpretations. In this work, we introduce Semantic Volume, a novel mathematical measure for quantifying both external and internal uncertainty in LLMs. Our approach perturbs queries and responses, embeds them in a semantic space, and computes the determinant of the Gram matrix of the embedding vectors, capturing their dispersion as a measure of uncertainty. Our framework provides a generalizable and unsupervised uncertainty detection method without requiring white-box access to LLMs. We conduct extensive experiments on both external and internal uncertainty detection, demonstrating that our Semantic Volume method consistently outperforms existing baselines in both tasks. Additionally, we provide theoretical insights linking our measure to differential entropy, unifying and extending previous sampling-based uncertainty measures such as the semantic entropy. Semantic Volume is shown to be a robust and interpretable approach to improving the reliability of LLMs by systematically detecting uncertainty in both user queries and model responses.
• 摘要：大型语言模型（LLM）通过编码大量的事实知识，在各种任务中表现出了出色的表现。但是，它们仍然容易出现幻觉，产生错误或误导性信息，通常伴随着高度不确定性。现有的幻觉检测方法主要集中于量化内部不确定性，这是由于模型中缺失或相互冲突的知识引起的。但是，幻觉也可能源于外部不确定性，在这种不确定性中，模棱两可的用户查询会导致多种可能的解释。在这项工作中，我们引入了语义量，这是一种量化LLM中外部和内部不确定性的新型数学措施。我们的方法将其查询和响应嵌入在语义空间中，并计算嵌入量的革兰氏矩阵的决定因素，从而捕获它们的分散体作为不确定性的度量。我们的框架提供了一种可普遍且无监督的不确定性检测方法，而无需白盒访问LLMS。我们对外部和内部不确定性检测进行了广泛的实验，这表明我们的语义体积方法在这两个任务中始终优于现有基准。此外，我们还提供理论见解，将我们的度量与差分熵联系起来，统一并扩展了以前的基于采样的不确定性度量，例如语义熵。语义量被证明是通过系统地检测用户查询和模型响应中的不确定性来提高LLM的可靠性的一种强大且可解释的方法。

Title: Token-level Ensembling of Models with Different Vocabularies

• Authors: Rachel Wicks, Kartik Ravisankar, Xinchen Yang, Philipp Koehn, Matt Post
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.21265
• Pdf URL: https://arxiv.org/pdf/2502.21265
• Copy Paste: [[2502.21265]] Token-level Ensembling of Models with Different Vocabularies(https://arxiv.org/abs/2502.21265)
• Keywords: llm
• Abstract: Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance. For ensembling in text generation, the next token's probability distribution is derived from a weighted sum of the distributions of each individual model. This requires the underlying models to share the same subword vocabulary, limiting the applicability of ensembling, since many open-sourced models have distinct vocabularies. In research settings, experimentation or upgrades to vocabularies may introduce multiple vocabulary sizes. This paper proposes an inference-time only algorithm that allows for ensembling models with different vocabularies, without the need to learn additional parameters or alter the underlying models. Instead, the algorithm ensures that tokens generated by the ensembled models \textit{agree} in their surface form. We apply this technique to combinations of traditional encoder-decoder models and decoder-only LLMs and evaluate on machine translation. In addition to expanding to model pairs that were previously incapable of token-level ensembling, our algorithm frequently improves translation performance over either model individually.
• 摘要：模型结合是一种结合两个或多个模型的预测分布的技术，通常会改善鲁棒性和性能。对于文本生成中的结合，下一个令牌的概率分布来自每个单独模型的分布的加权总和。这要求基础模型共享相同的子字词汇，从而限制了结合的适用性，因为许多开源模型都有不同的词汇。在研究环境中，词汇的实验或升级可能会引入多种词汇量。本文提出了一种仅推理时算法，该算法允许使用不同的词汇结合模型，而无需学习其他参数或更改基础模型。取而代之的是，该算法确保了由结合模型\ textit {clase}以其表面形式产生的令牌。我们将此技术应用于传统的编码器模型和仅解码器LLM的组合，并评估机器翻译。除了扩展到以前无法进行令牌级结合能力的模型对外，我们的算法还经常改善单独单独的翻译性能。

Title: Persuasion Should be Double-Blind: A Multi-Domain Dialogue Dataset With Faithfulness Based on Causal Theory of Mind

• Authors: Dingyi Zhang, Deyu Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.21297
• Pdf URL: https://arxiv.org/pdf/2502.21297
• Copy Paste: [[2502.21297]] Persuasion Should be Double-Blind: A Multi-Domain Dialogue Dataset With Faithfulness Based on Causal Theory of Mind(https://arxiv.org/abs/2502.21297)
• Keywords: prompt, agent
• Abstract: Persuasive dialogue plays a pivotal role in human communication, influencing various domains. Recent persuasive dialogue datasets often fail to align with real-world interpersonal interactions, leading to unfaithful representations. For instance, unrealistic scenarios may arise, such as when the persuadee explicitly instructs the persuader on which persuasion strategies to employ, with each of the persuadee's questions corresponding to a specific strategy for the persuader to follow. This issue can be attributed to a violation of the "Double Blind" condition, where critical information is fully shared between participants. In actual human interactions, however, key information such as the mental state of the persuadee and the persuasion strategies of the persuader is not directly accessible. The persuader must infer the persuadee's mental state using Theory of Mind capabilities and construct arguments that align with the persuadee's motivations. To address this gap, we introduce ToMMA, a novel multi-agent framework for dialogue generation that is guided by causal Theory of Mind. This framework ensures that information remains undisclosed between agents, preserving "double-blind" conditions, while causal ToM directs the persuader's reasoning, enhancing alignment with human-like persuasion dynamics. Consequently, we present CToMPersu, a multi-domain, multi-turn persuasive dialogue dataset that tackles both double-blind and logical coherence issues, demonstrating superior performance across multiple metrics and achieving better alignment with real human dialogues. Our dataset and prompts are available at this https URL .
• 摘要：有说服力的对话在人类交流中起着关键作用，影响了各个领域。最近的有说服力的对话数据集通常无法与现实世界中的人际关系互动保持一致，从而导致不忠的表示。例如，可能会出现不切实际的场景，例如，当说服的说服者指示说服策略所采用的说服者时，每个说服者的问题与说服者的特定策略相对应。这个问题可以归因于违反“双盲”条件的行为，在这种情况下，参与者之间的关键信息完全共享。然而，在实际的人类互动中，关键信息，例如说服力的精神状态和说服者的说服策略是无法直接访问的。说服者必须使用思想能力理论来推断说服的精神状态，并构建与说服力动机保持一致的论点。为了解决这一差距，我们介绍了Tomma，这是一个新型的对话生成框架，它以因果理论为指导。该框架确保了代理之间未公开的信息，并保留了“双盲”条件，而因果关系汤姆指导说服者的推理，从而增强了与类似人类的说服力动态的一致性。因此，我们提出了Ctompersu，这是一种多域，多转的说服力对话数据集，可以解决双盲和逻辑上的连贯性问题，表明了多个指标的卓越性能，并与真实的人类对话实现了更好的一致性。我们的数据集和提示可在此HTTPS URL上找到。

Title: FANformer: Improving Large Language Models Through Effective Periodicity Modeling

• Authors: Yihong Dong, Ge Li, Xue Jiang, Yongding Tao, Kechi Zhang, Hao Zhu, Huanyu Liu, Jiazheng Ding, Jia Li, Jinliang Deng, Hong Mei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.21309
• Pdf URL: https://arxiv.org/pdf/2502.21309
• Copy Paste: [[2502.21309]] FANformer: Improving Large Language Models Through Effective Periodicity Modeling(https://arxiv.org/abs/2502.21309)
• Keywords: language model, llm
• Abstract: Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which integrates Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. To further validate the effectiveness of FANformer, we pretrain a FANformer-1B on 1 trillion tokens. FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. The results position FANformer as an effective and promising architecture for advancing LLMs.
• 摘要：作为最重要的基本特征之一，周期性为促进人类学习范式内的结构化知识获取和系统认知过程奠定了基础。但是，变压器中周期性建模的潜在缺陷会影响基于大型语言模型（LLM）数据的学习效率和基本原理的建立。在本文中，我们证明，整合有效的周期性建模可以提高LLM的学习效率和性能。我们介绍了通过修改注意机制的特征投影过程，将傅立叶分析网络（FAN）集成到注意力机制中以实现有效的周期性建模。语言建模的广泛实验结果表明，在扩展模型大小和训练令牌时，粉丝形式的表现始终优于变形金刚，从而强调了其出色的学习效率。为了进一步验证粉丝形式的有效性，我们在1万亿个令牌上为粉丝粉1B预算了1B。与具有相似模型参数或训练令牌的开源LLM相比，FanFormer-1b在下游任务上表现出明显的改进。结果将粉丝形式定位为推进LLM的有效且有前途的架构。

Title: LLM Post-Training: A Deep Dive into Reasoning Large Language Models

• Authors: Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H.S. Torr, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.21321
• Pdf URL: https://arxiv.org/pdf/2502.21321
• Copy Paste: [[2502.21321]] LLM Post-Training: A Deep Dive into Reasoning Large Language Models(https://arxiv.org/abs/2502.21321)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Pretraining on vast web-scale data has laid the foundation for these models, yet the research community is now increasingly shifting focus toward post-training techniques to achieve further breakthroughs. While pretraining provides a broad linguistic foundation, post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations. Fine-tuning, reinforcement learning, and test-time scaling have emerged as critical strategies for optimizing LLMs performance, ensuring robustness, and improving adaptability across various real-world tasks. This survey provides a systematic exploration of post-training methodologies, analyzing their role in refining LLMs beyond pretraining, addressing key challenges such as catastrophic forgetting, reward hacking, and inference-time trade-offs. We highlight emerging directions in model alignment, scalable adaptation, and inference-time reasoning, and outline future research directions. We also provide a public repository to continually track developments in this fast-evolving field: this https URL.
• 摘要：大型语言模型（LLMS）改变了自然语言处理景观，并使多样化的应用栩栩如生。在大量的网络规模数据上进行了预处理为这些模型奠定了基础，但是研究界现在越来越多地转移到训练后技术方面，以实现进一步的突破。虽然预处理提供了广泛的语言基础，但培训后方法使LLMS能够完善其知识，提高推理，提高事实准确性，并更有效地与用户的意图和道德注意事项更有效地保持一致。微调，加强学习和测试时间缩放已成为优化LLMS性能，确保鲁棒性和改善各种现实世界任务的适应性的关键策略。这项调查提供了对训练后方法的系统探索，分析了它们在预处理以外的LLM中的作用，解决了诸如灾难性遗忘，奖励黑客和推理时间权衡等关键挑战。我们重点介绍了模型对齐，可扩展适应和推理时间推理的新兴方向，并概述了未来的研究方向。我们还提供了一个公共存储库来不断跟踪这个快速发展的领域的发展：此HTTPS URL。