2025-06-25

Title: MemeMind: A Large-Scale Multimodal Dataset with Chain-of-Thought Reasoning for Harmful Meme Detection

• Authors: Hexiang Gu, Qifan Yu, Saihui Hou, Zhiqin Fang, Huijia Wu, Zhaofeng He
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2506.18919
• Pdf URL: https://arxiv.org/pdf/2506.18919
• Copy Paste: [[2506.18919]] MemeMind: A Large-Scale Multimodal Dataset with Chain-of-Thought Reasoning for Harmful Meme Detection(https://arxiv.org/abs/2506.18919)
• Keywords: chain-of-thought
• Abstract: The rapid development of social media has intensified the spread of harmful content. Harmful memes, which integrate both images and text, pose significant challenges for automated detection due to their implicit semantics and complex multimodal interactions. Although existing research has made progress in detection accuracy and interpretability, the lack of a systematic, large-scale, diverse, and highly explainable dataset continues to hinder further advancement in this field. To address this gap, we introduce MemeMind, a novel dataset featuring scientifically rigorous standards, large scale, diversity, bilingual support (Chinese and English), and detailed Chain-of-Thought (CoT) annotations. MemeMind fills critical gaps in current datasets by offering comprehensive labeling and explicit reasoning traces, thereby providing a solid foundation for enhancing harmful meme detection. In addition, we propose an innovative detection framework, MemeGuard, which effectively integrates multimodal information with reasoning process modeling, significantly improving models' ability to understand and identify harmful memes. Extensive experiments conducted on the MemeMind dataset demonstrate that MemeGuard consistently outperforms existing state-of-the-art methods in harmful meme detection tasks.
• 摘要：社交媒体的快速发展加剧了有害内容的传播。有害模因既包含图像和文本），由于其隐式语义和复杂的多模式相互作用，对自动检测构成了重大挑战。尽管现有的研究在检测准确性和解释性方面取得了进展，但缺乏系统的，大规模，多样化和高度解释的数据集仍在继续阻碍该领域的进一步发展。为了解决这一差距，我们介绍了Mememind，这是一个新颖的数据集，具有科学严格的标准，大规模，多样性，双语支持（中文和英语），以及详细的经过经验链（COT）注释。 Mememind通过提供全面的标签和明确的推理轨迹来填补当前数据集中的关键空白，从而为增强有害模因检测提供了坚实的基础。此外，我们提出了一个创新的检测框架Memeguard，该框架有效地将多模式信息与推理过程建模相结合，从而显着提高了模型理解和识别有害模因的能力。在Mememind数据集上进行的广泛实验表明，在有害模因检测任务中，Memeguard始终优于现有的最新方法。

Title: Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge

• Authors: Sahil Kale, Vijaykant Nadadur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.18998
• Pdf URL: https://arxiv.org/pdf/2506.18998
• Copy Paste: [[2506.18998]] Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge(https://arxiv.org/abs/2506.18998)
• Keywords: llm
• Abstract: When artificial intelligence mistakes memorization for intelligence, it creates a dangerous mirage of reasoning. Existing studies treat memorization and self-knowledge deficits in LLMs as separate issues and do not recognize an intertwining link that degrades the trustworthiness of LLM responses. In our study, we utilize a novel framework to ascertain if LLMs genuinely learn reasoning patterns from training data or merely memorize them to assume competence across problems of similar complexity focused on STEM domains. Our analysis shows a noteworthy problem in generalization: LLMs draw confidence from memorized solutions to infer a higher self-knowledge about their reasoning ability, which manifests as an over 45% inconsistency in feasibility assessments when faced with self-validated, logically coherent task perturbations. This effect is most pronounced in science and medicine domains, which tend to have maximal standardized jargon and problems, further confirming our approach. Significant wavering within the self-knowledge of LLMs also shows flaws in current architectures and training patterns, highlighting the need for techniques that ensure a balanced, consistent stance on models' perceptions of their own knowledge for maximum AI explainability and trustworthiness. Our code and results are available publicly at this https URL.
• 摘要：当人工智能错误地将记忆误认为智力时，它会造成推理的危险幻影。现有研究将LLM中的记忆和自我知识缺陷视为单独的问题，并且不认识到降低LLM响应可信度的交织联系。在我们的研究中，我们利用一个新颖的框架来确定LLM是从训练数据中真正学习推理模式还是仅仅记住它们以在集中在STEM领域的类似复杂性问题上具有能力。我们的分析显示了一个值得注意的概括问题：LLMS从记忆的解决方案中获得信心，以推断出对其推理能力的更高自我知识，当面对自我验证的，逻辑上一致的任务扰动时，在可行性评估中表现出超过45％的不一致性。在科学和医学领域，这种效果最为明显，后者倾向于具有最大的标准术语和问题，进一步证实了我们的方法。 LLM的自我知识中的重要动力也显示出当前的架构和训练模式中的缺陷，强调了对技术的需求，以确保对模型对自己的知识的看法保持平衡，一致的立场，以实现最大的AI解释性和可信赖性。我们的代码和结果可在此HTTPS URL上公开获得。

Title: Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

• Authors: Brian Siyuan Zheng, Alisa Liu, Orevaoghene Ahia, Jonathan Hayase, Yejin Choi, Noah A. Smith
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19004
• Pdf URL: https://arxiv.org/pdf/2506.19004
• Copy Paste: [[2506.19004]] Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations(https://arxiv.org/abs/2506.19004)
• Keywords: language model
• Abstract: Modern tokenizers employ deterministic algorithms to map text into a single "canonical" token sequence, yet the same string can be encoded as many non-canonical tokenizations using the tokenizer vocabulary. In this work, we investigate the robustness of LMs to text encoded with non-canonical tokenizations entirely unseen during training. Surprisingly, when evaluated across 20 benchmarks, we find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization, and 90.8% with character-level tokenization. We see that overall stronger models tend to be more robust, and robustness diminishes as the tokenization departs farther from the canonical form. Motivated by these results, we then identify settings where non-canonical tokenization schemes can *improve* performance, finding that character-level segmentation improves string manipulation and code understanding tasks by up to +14%, and right-aligned digit grouping enhances large-number arithmetic by +33%. Finally, we investigate the source of this robustness, finding that it arises in the instruction-tuning phase. We show that while both base and post-trained models grasp the semantics of non-canonical tokenizations (perceiving them as containing misspellings), base models try to mimic the imagined mistakes and degenerate into nonsensical output, while post-trained models are committed to fluent responses. Overall, our findings suggest that models are less tied to their tokenizer than previously believed, and demonstrate the promise of intervening on tokenization at inference time to boost performance.
• 摘要：现代令牌使用确定性算法将文本映射到单个“规范”令牌序列中，但是可以使用令牌词汇将相同的字符串编码为许多非典型的标记。在这项工作中，我们调查了LMS对在训练过程中完全看不见的非规范引物编码的文本的鲁棒性。令人惊讶的是，当对20个基准进行评估时，我们发现指令调整的模型在随机采样的令牌化时保留了其原始性能的93.4％，而90.8％的模型则保留了字符级别的象征化的90.8％。我们看到，整体强大的模型往往更强大，并且随着令牌化与规范形式更远的形式，稳健性会降低。然后，在这些结果的启发下，我们确定了非规范令牌化方案可以 *提高性能的设置，发现字符级分割可改善字符串的操纵和代码理解任务，高达 +14％，而右对齐的数字组组件则增强了大量的数字算法，从而增强了大量的Arithmetic firs +33％。最后，我们研究了这种鲁棒性的来源，发现它是在指导调查阶段产生的。我们表明，尽管基本模型和训练后模型都掌握了非规范引导的语义（将其视为包含拼写错误），但基本模型试图模仿想象中的错误并退化为荒谬的输出，而后培训后的模型则致力于流利的响应。总体而言，我们的发现表明，与以前的模型相比，模型与以前所相信的要少，并证明了在推理时间介入以提高性能的介入的希望。

Title: Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective

• Authors: Weijie Xu, Yiwen Wang, Chi Xue, Xiangkun Hu, Xi Fang, Guimin Dong, Chandan K. Reddy
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2506.19028
• Pdf URL: https://arxiv.org/pdf/2506.19028
• Copy Paste: [[2506.19028]] Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective(https://arxiv.org/abs/2506.19028)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications. Existing evaluation methods often overlook biases in long-form responses and the intrinsic variability of LLM outputs. To address these challenges, we propose FiSCo(Fine-grained Semantic Computation), a novel statistical framework to evaluate group-level fairness in LLMs by detecting subtle semantic differences in long-form responses across demographic groups. Unlike prior work focusing on sentiment or token-level comparisons, FiSCo goes beyond surface-level analysis by operating at the claim level, leveraging entailment checks to assess the consistency of meaning across responses. We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities, enabling robust detection of subtle biases. We formalize a new group counterfactual fairness definition and validate FiSCo on both synthetic and human-annotated datasets spanning gender, race, and age. Experiments show that FiSco more reliably identifies nuanced biases while reducing the impact of stochastic LLM variability, outperforming various evaluation metrics.
• 摘要：大型语言模型（LLM）通常会以固有的偏见产生响应，从而破坏其在现实应用程序中的可靠性。现有的评估方法经常忽略长形响应中的偏见和LLM输出的内在变异性。为了应对这些挑战，我们提出了Fisco（细粒语义计算），这是一个新的统计框架，旨在通过检测人群跨人群群体长期响应的细微语义差异来评估LLMS中的群体级别公平性。与以前关注情感或令牌级比较的工作不同，Fisco通过在索赔级上运行，超越了表面级别的分析，利用需要检查来评估跨响应含义的一致性。我们将模型输出分解为语义上不同的主张，并应用统计假设测试以比较组间和组内相似性，从而可以强大的偏见检测。我们对跨性别，种族和年龄的合成数据集和人类宣传的数据集进行了正式的反事实公平定义，并验证了Fisco。实验表明，Fisco更可靠地确定了细微的偏见，同时降低了随机LLM变异性的影响，表现优于各种评估指标。

Title: Plan for Speed -- Dilated Scheduling for Masked Diffusion Language Models

• Authors: Omer Luxembourg, Haim Permuter, Eliya Nachmani
• Subjects: cs.CL, cs.AI, cs.IT, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2506.19037
• Pdf URL: https://arxiv.org/pdf/2506.19037
• Copy Paste: [[2506.19037]] Plan for Speed -- Dilated Scheduling for Masked Diffusion Language Models(https://arxiv.org/abs/2506.19037)
• Keywords: language model
• Abstract: Masked diffusion language models (MDLM) have shown strong promise for non-autoregressive text generation, yet existing samplers act as implicit planners, selecting tokens to unmask via denoiser confidence or entropy scores. Such heuristics falter under parallel unmasking - they ignore pairwise interactions between tokens and cannot account for dependencies when unmasking multiple positions at once, limiting their inference time to traditional auto-regressive (AR) models. We introduce the Dilated-scheduled Unmasking Strategy (DUS), an inference-only, planner-model-free method that requires no additional training. DUS leverages a first-order Markov assumption to partition sequence positions into dilation-based groups of non-adjacent tokens, enabling independent, parallel unmasking steps that respect local context that minimizes the joint entropy of each iteration step. Unlike semi-AR block approaches (e.g., LLADA and Dream) that still invoke the denoiser per block, DUS reduces the number of denoiser calls to O(log B) per generation block - yielding substantial speedup over the O(B) run time of state-of-the-art diffusion models, where B is the block size in the semi-AR inference process. In experiments on math (GSM8K) and code completion (Humaneval, MBPP) benchmarks - domains suited to non-ordinal generation - DUS improves scores over parallel confidence-based planner, without modifying the underlying denoiser. DUS offers a lightweight, budget-aware approach to efficient, high-quality text generation, paving the way to unlock the true capabilities of MDLMs.
• 摘要：蒙版扩散语言模型（MDLM）对非自动回归文本生成表现出很大的希望，但现有的采样器充当隐式计划者，选择令牌通过DeOiser Prosites或Entropy Scores进行揭露。这种启发式方法在并行揭露下步履蹒跚 - 它们忽略了令牌之间的成对相互作用，并且在一次揭示多个位置时无法解释依赖关系，从而将其推理时间限制为传统的自动回归（AR）模型。我们介绍了扩大的安排卸载策略（DUS），这是一种仅推理的，无计划的无需训练方法，不需要额外的培训。 DUS利用一阶马尔可夫假设将序列位置划分为基于扩张的非贴剂代币的组，从而实现独立的，平行的揭示步骤，这些步骤尊重局部上下文，从而最大程度地减少了每个迭代步骤的关节熵。与仍在每个块调用Denoisiser的半AR块方法（例如Llada和Dream）不同，DUS减少了DeOiser呼叫的数量，即每一代块O（log B）每一代块的呼叫数量 - 在O（b）最终的扩散模型的运行时间上产生大量加速，其中B是半级别量的块大小，其中B是半阶段的块大小。在数学（GSM8K）和代码完成（HUMANEVAL，MBPP）基准的实验中 - 适用于非界生成的域 - dus可以改善基于平行置信的计划者的分数，而无需修改基础的Denoiser。 DUS提供了一种轻巧，预算感知的方法，用于有效，高质量的文本生成，为解锁MDLM的真正功能铺平了道路。

Title: NLPnorth @ TalentCLEF 2025: Comparing Discriminative, Contrastive, and Prompt-Based Methods for Job Title and Skill Matching

• Authors: Mike Zhang, Rob van der Goot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19058
• Pdf URL: https://arxiv.org/pdf/2506.19058
• Copy Paste: [[2506.19058]] NLPnorth @ TalentCLEF 2025: Comparing Discriminative, Contrastive, and Prompt-Based Methods for Job Title and Skill Matching(https://arxiv.org/abs/2506.19058)
• Keywords: language model, prompt
• Abstract: Matching job titles is a highly relevant task in the computational job market domain, as it improves e.g., automatic candidate matching, career path prediction, and job market analysis. Furthermore, aligning job titles to job skills can be considered an extension to this task, with similar relevance for the same downstream tasks. In this report, we outline NLPnorth's submission to TalentCLEF 2025, which includes both of these tasks: Multilingual Job Title Matching, and Job Title-Based Skill Prediction. For both tasks we compare (fine-tuned) classification-based, (fine-tuned) contrastive-based, and prompting methods. We observe that for Task A, our prompting approach performs best with an average of 0.492 mean average precision (MAP) on test data, averaged over English, Spanish, and German. For Task B, we obtain an MAP of 0.290 on test data with our fine-tuned classification-based approach. Additionally, we made use of extra data by pulling all the language-specific titles and corresponding \emph{descriptions} from ESCO for each job and skill. Overall, we find that the largest multilingual language models perform best for both tasks. Per the provisional results and only counting the unique teams, the ranking on Task A is 5$^{\text{th}}$/20 and for Task B 3$^{\text{rd}}$/14.
• 摘要：匹配职位是计算工作市场领域中一项高度相关的任务，因为它改善了自动候选人匹配，职业路径预测和就业市场分析。此外，将职位与工作技能保持一致可以被认为是对此任务的扩展，与同一下游任务具有相似的相关性。在本报告中，我们概述了NLPNorth提交给Talentclef 2025的提交，其中包括这两个任务：多语言职位匹配和基于工作的技能预测。对于这两个任务，我们比较基于（微调的）基于（微调）基于对比度的（微调）和提示方法。我们观察到，对于任务A，我们的提示方法的表现最佳，平均在测试数据上平均平均平均精度（MAP），平均用英语，西班牙语和德语平均。对于任务B，我们使用基于微调的基于分类的方法在测试数据上获得了0.290的地图。此外，我们通过从ESCO中提取所有特定于语言的标题和相应的\ emph {Defictions}来利用额外的数据。总体而言，我们发现最大的多语言模型对这两个任务都表现最佳。根据临时结果，仅计算独特的团队，任务A上的排名为5 $^{\ text {th}} $/20，对于任务B 3 $^{\ text {rd}} $/14。

Title: MFTCXplain: A Multilingual Benchmark Dataset for Evaluating the Moral Reasoning of LLMs through Hate Speech Multi-hop Explanation

• Authors: Jackson Trager, Francielle Vargas, Diego Alves, Matteo Guida, Mikel K. Ngueajio, Ameeta Agrawal, Flor Plaza-del-Arco, Yalda Daryanai, Farzan Karimi-Malekabadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19073
• Pdf URL: https://arxiv.org/pdf/2506.19073
• Copy Paste: [[2506.19073]] MFTCXplain: A Multilingual Benchmark Dataset for Evaluating the Moral Reasoning of LLMs through Hate Speech Multi-hop Explanation(https://arxiv.org/abs/2506.19073)
• Keywords: language model, llm
• Abstract: Ensuring the moral reasoning capabilities of Large Language Models (LLMs) is a growing concern as these systems are used in socially sensitive tasks. Nevertheless, current evaluation benchmarks present two major shortcomings: a lack of annotations that justify moral classifications, which limits transparency and interpretability; and a predominant focus on English, which constrains the assessment of moral reasoning across diverse cultural settings. In this paper, we introduce MFTCXplain, a multilingual benchmark dataset for evaluating the moral reasoning of LLMs via hate speech multi-hop explanation using Moral Foundation Theory (MFT). The dataset comprises 3,000 tweets across Portuguese, Italian, Persian, and English, annotated with binary hate speech labels, moral categories, and text span-level rationales. Empirical results highlight a misalignment between LLM outputs and human annotations in moral reasoning tasks. While LLMs perform well in hate speech detection (F1 up to 0.836), their ability to predict moral sentiments is notably weak (F1 < 0.35). Furthermore, rationale alignment remains limited mainly in underrepresented languages. These findings show the limited capacity of current LLMs to internalize and reflect human moral reasoning.
• 摘要：确保大语言模型（LLM）的道德推理能力（LLM）越来越关注，因为这些系统被用于社会敏感的任务中。然而，当前的评估基准存在两个主要的缺点：缺乏证明道德分类合理的注释，这限制了透明度和解释性；并主要关注英语，这限制了各种文化环境中道德推理的评估。在本文中，我们介绍了MFTCXPLAIN，这是一种多语言基准数据集，用于使用仇恨言语多跳上解释，使用道德基础理论（MFT）评估LLM的道德推理。该数据集包含葡萄牙，意大利语，波斯语和英语的3,000条推文，并注释了二进制仇恨言语标签，道德类别和文本跨度级别的理由。经验结果突出了道德推理任务中LLM输出与人类注释之间的错位。尽管LLM在仇恨言语检测中表现良好（F1最高0.836），但他们预测道德情感的能力明显较弱（F1 <0.35）。此外，基本对准仍然有限，主要是代表性不足的语言。这些发现表明，当前LLM在内部化和反映人类道德推理的能力有限。

Title: Language Models Might Not Understand You: Evaluating Theory of Mind via Story Prompting

• Authors: Nathaniel Getachew, Abulhair Saparov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19089
• Pdf URL: https://arxiv.org/pdf/2506.19089
• Copy Paste: [[2506.19089]] Language Models Might Not Understand You: Evaluating Theory of Mind via Story Prompting(https://arxiv.org/abs/2506.19089)
• Keywords: language model, llm, prompt
• Abstract: We introduce $\texttt{StorySim}$, a programmable framework for synthetically generating stories to evaluate the theory of mind (ToM) and world modeling (WM) capabilities of large language models (LLMs). Unlike prior benchmarks that may suffer from contamination in pretraining data, $\texttt{StorySim}$ produces novel, compositional story prompts anchored by a highly controllable $\texttt{Storyboard}$, enabling precise manipulation of character perspectives and events. We use this framework to design first- and second-order ToM tasks alongside WM tasks that control for the ability to track and model mental states. Our experiments across a suite of state-of-the-art LLMs reveal that most models perform better on WM tasks than ToM tasks, and that models tend to perform better reasoning with humans compared to inanimate objects. Additionally, our framework enabled us to find evidence of heuristic behavior such as recency bias and an over-reliance on earlier events in the story. All code for generating data and evaluations is freely available.
• 摘要：我们介绍了$ \ texttt {storysim} $，这是一个可编程框架，用于合成生成故事以评估大语模型（LLMS）的思维理论（TOM）和世界建模（WM）功能。与可能在预处理数据中受到污染的先前基准测试不同，$ \ texttt {storysim} $产生了小说，构图故事提示由高度可控制的$ \ texttt {故事板} $锚定，实现了对角色观点和事件的精确操纵。我们使用此框架来设计一阶和二阶TOM任务以及控制跟踪和建模精神状态能力的WM任务。我们在一套最先进的LLMS套件中进行的实验表明，大多数模型在WM任务上的表现要比TOM任务更好，并且与无生命的对象相比，该模型倾向于对人类进行更好的推理。此外，我们的框架使我们能够找到启发式行为的证据，例如新闻偏见和对故事中早期事件的过度依赖。所有用于生成数据和评估的代码均可免费获得。

Title: Human-Aligned Faithfulness in Toxicity Explanations of LLMs

• Authors: Ramaravind K. Mothilal, Joanna Roy, Syed Ishtiaque Ahmed, Shion Guha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19113
• Pdf URL: https://arxiv.org/pdf/2506.19113
• Copy Paste: [[2506.19113]] Human-Aligned Faithfulness in Toxicity Explanations of LLMs(https://arxiv.org/abs/2506.19113)
• Keywords: llm, prompt
• Abstract: The discourse around toxicity and LLMs in NLP largely revolves around detection tasks. This work shifts the focus to evaluating LLMs' reasoning about toxicity -- from their explanations that justify a stance -- to enhance their trustworthiness in downstream tasks. Despite extensive research on explainability, it is not straightforward to adopt existing methods to evaluate free-form toxicity explanation due to their over-reliance on input text perturbations, among other challenges. To account for these, we propose a novel, theoretically-grounded multi-dimensional criterion, Human-Aligned Faithfulness (HAF), that measures the extent to which LLMs' free-form toxicity explanations align with those of a rational human under ideal conditions. We develop six metrics, based on uncertainty quantification, to comprehensively evaluate \haf of LLMs' toxicity explanations with no human involvement, and highlight how "non-ideal" the explanations are. We conduct several experiments on three Llama models (of size up to 70B) and an 8B Ministral model on five diverse toxicity datasets. Our results show that while LLMs generate plausible explanations to simple prompts, their reasoning about toxicity breaks down when prompted about the nuanced relations between the complete set of reasons, the individual reasons, and their toxicity stances, resulting in inconsistent and nonsensical responses. We open-source our code and LLM-generated explanations at this https URL.
• 摘要：NLP中有关毒性和LLM的论述在很大程度上围绕检测任务。这项工作将重点转移到评估LLMS关于毒性的推理（从他们证明立场合理的解释中）以增强其在下游任务中的信任度。尽管对解释性进行了广泛的研究，但由于对输入文本扰动的过度依赖以及其他挑战，采用现有方法来评估自由形式的毒性解释。为了解决这些问题，我们提出了一种新颖的，理论上的多维标准，人类一致的忠诚（HAF），该标准（HAF）衡量了LLMS在理想条件下LLMS的自由形式毒性的解释与合理人的自由形式的解释。我们根据不确定性量化开发了六个指标，以全面评估LLMS的毒性解释\ HAF而没有人类参与，并强调了这些解释的“非理想”。我们对三种千禧年模型（最高70B）进行了多个实验，并在五个不同的毒性数据集上进行了8B个官方模型。我们的结果表明，尽管LLMS对简单提示产生了合理的解释，但在提示有关完整原因，个人原因和它们的毒性立场之间的细微差别关系时，其有关毒性的推理会破裂，从而导致不一致和不敏感的反应。我们在此HTTPS URL上开放代码和LLM生成的解释。

Title: Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

• Authors: Christopher Toukmaji, Jeffrey Flanigan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19187
• Pdf URL: https://arxiv.org/pdf/2506.19187
• Copy Paste: [[2506.19187]] Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages(https://arxiv.org/abs/2506.19187)
• Keywords: llm, prompt
• Abstract: LLMs are typically trained in high-resource languages, and tasks in lower-resourced languages tend to underperform the higher-resource language counterparts for in-context learning. Despite the large body of work on prompting settings, it is still unclear how LLMs should be adapted cross-lingually specifically for in-context learning in the low-resource target languages. We perform a comprehensive study spanning five diverse target languages, three base LLMs, and seven downstream tasks spanning over 4,100 GPU training hours (9,900+ TFLOPs) across various adaptation techniques: few-shot prompting, translate-test, fine-tuning, embedding re-initialization, and instruction fine-tuning. Our results show that the few-shot prompting and translate-test settings tend to heavily outperform the gradient-based adaptation methods. To better understand this discrepancy, we design a novel metric, Valid Output Recall (VOR), and analyze model outputs to empirically attribute the degradation of these trained models to catastrophic forgetting. To the extent of our knowledge, this is the largest study done on in-context learning for low-resource languages with respect to train compute and number of adaptation techniques considered. We make all our datasets and trained models available for public use.
• 摘要：LLM通常接受使用高资源语言的培训，而资源较低的语言的任务往往不足以表现出较高的资源语言，以供文字学习。尽管在促使设置方面进行了大量工作，但仍不清楚如何在低资源目标语言中专门针对跨语言专门针对跨语言进行调整。我们进行了一项全面的研究，涵盖了五种不同的目标语言，三种基本LLM和七个下游任务，跨越了4,100个GPU培训时间（9,900多个TFLOPS），遍及各种适应技术：很少提示，转换测试，细化，微调，调查，嵌入重新定性和指导性微调。我们的结果表明，少数发动机的提示和翻译测试设置往往大大胜过基于梯度的适应方法。为了更好地理解这种差异，我们设计了一种新颖的指标，有效的输出召回（VOR），并分析模型输出以经验将这些训练的模型的退化归因于灾难性的遗忘。就我们的知识而言，这是关于训练计算和所考虑的适应性技术数量的低资源语言中文化学习的最大研究。我们将所有数据集和训练有素的模型可供公开使用。

Title: Augmenting Multi-Agent Communication with State Delta Trajectory

• Authors: Yichen Tang, Weihang Su, Yujia Zhou, Yiqun Liu, Min Zhang, Shaoping Ma, Qingyao Ai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19209
• Pdf URL: https://arxiv.org/pdf/2506.19209
• Copy Paste: [[2506.19209]] Augmenting Multi-Agent Communication with State Delta Trajectory(https://arxiv.org/abs/2506.19209)
• Keywords: language model, llm, agent
• Abstract: Multi-agent techniques such as role playing or multi-turn debates have been shown to be effective in improving the performance of large language models (LLMs) in downstream tasks. Despite their differences in workflows, existing LLM-based multi-agent systems mostly use natural language for agent communication. While this is appealing for its simplicity and interpretability, it also introduces inevitable information loss as one model must down sample its continuous state vectors to concrete tokens before transferring them to the other model. Such losses are particularly significant when the information to transfer is not simple facts, but reasoning logics or abstractive thoughts. To tackle this problem, we propose a new communication protocol that transfers both natural language tokens and token-wise state transition trajectory from one agent to another. Particularly, compared to the actual state value, we find that the sequence of state changes in LLMs after generating each token can better reflect the information hidden behind the inference process, so we propose a State Delta Encoding (SDE) method to represent state transition trajectories. The experimental results show that multi-agent systems with SDE achieve SOTA performance compared to other communication protocols, particularly in tasks that involve complex reasoning. This shows the potential of communication augmentation for LLM-based multi-agent systems.
• 摘要：诸如角色扮演或多转变辩论之类的多代理技术已被证明可以有效地改善下游任务中大型语言模型（LLM）的性能。尽管工作流程有所不同，但现有的基于LLM的多代理系统主要使用自然语言进行代理通信。尽管这对其简单性和可解释性具有吸引力，但它也引入了不可避免的信息丢失，因为一个模型必须降低其连续状态向量的样本，以便在将其转移到另一个模型之前，以将其转换为具体令牌。当转移的信息不是简单的事实，而是推理逻辑或抽象思想时，这种损失尤其重要。为了解决这个问题，我们提出了一种新的通信协议，该协议将自然语言令牌和象征状态过渡轨迹从一个代理转移到另一个代理。特别是，与实际状态值相比，我们发现生成每个令牌后LLMS中的状态变化序列可以更好地反映推理过程背后隐藏的信息，因此我们提出了一种状态Delta编码（SDE）方法来表示状态过渡轨迹。实验结果表明，与其他通信协议相比，具有SDE的多代理系统实现了SOTA性能，尤其是在涉及复杂推理的任务中。这显示了基于LLM的多代理系统的通信扩展的潜力。

Title: Personality Prediction from Life Stories using Language Models

• Authors: Rasiq Hussain, Jerry Ma, Rithik Khandelwal, Joshua Oltmanns, Mehak Gupta
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.19258
• Pdf URL: https://arxiv.org/pdf/2506.19258
• Copy Paste: [[2506.19258]] Personality Prediction from Life Stories using Language Models(https://arxiv.org/abs/2506.19258)
• Keywords: language model
• Abstract: Natural Language Processing (NLP) offers new avenues for personality assessment by leveraging rich, open-ended text, moving beyond traditional questionnaires. In this study, we address the challenge of modeling long narrative interview where each exceeds 2000 tokens so as to predict Five-Factor Model (FFM) personality traits. We propose a two-step approach: first, we extract contextual embeddings using sliding-window fine-tuning of pretrained language models; then, we apply Recurrent Neural Networks (RNNs) with attention mechanisms to integrate long-range dependencies and enhance interpretability. This hybrid method effectively bridges the strengths of pretrained transformers and sequence modeling to handle long-context data. Through ablation studies and comparisons with state-of-the-art long-context models such as LLaMA and Longformer, we demonstrate improvements in prediction accuracy, efficiency, and interpretability. Our results highlight the potential of combining language-based features with long-context modeling to advance personality assessment from life narratives.
• 摘要：自然语言处理（NLP）通过利用丰富的开放式文本超越传统问卷来提供新的人格评估途径。在这项研究中，我们应对建模长期叙事访谈的挑战，在该访谈中，每个访谈都超过2000个令牌，以预测五因素模型（FFM）人格特征。我们提出了一种两步的方法：首先，我们使用预审前的语言模型的滑动窗口来提取上下文嵌入；然后，我们将复发性神经网络（RNN）与注意机制相结合，以整合长期依赖性并增强可解释性。这种混合方法有效地桥接了经过预告片的变压器和序列建模的强度，以处理长篇文化数据。通过消融研究和与最先进的长篇文化模型（如骆驼和长形者）的比较，我们证明了预测准确性，效率和可解释性的提高。我们的结果突出了将基于语言的特征与长篇文化建模相结合以从生活叙事中推进人格评估的潜力。

Title: What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning

• Authors: Yuchang Zhu, Zhonghua zhen, Qunshu Lin, Haotong Wei, Xiaolong Sun, Zixuan Yu, Minghao Liu, Zibin Zheng, Liang Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.19262
• Pdf URL: https://arxiv.org/pdf/2506.19262
• Copy Paste: [[2506.19262]] What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning(https://arxiv.org/abs/2506.19262)
• Keywords: language model, llm
• Abstract: With the remarkable generative capabilities of large language models (LLMs), using LLM-generated data to train downstream models has emerged as a promising approach to mitigate data scarcity in specific domains and reduce time-consuming annotations. However, recent studies have highlighted a critical issue: iterative training on self-generated data results in model collapse, where model performance degrades over time. Despite extensive research on the implications of LLM-generated data, these works often neglect the importance of data diversity, a key factor in data quality. In this work, we aim to understand the implications of the diversity of LLM-generated data on downstream model performance. Specifically, we explore how varying levels of diversity in LLM-generated data affect downstream model performance. Additionally, we investigate the performance of models trained on data that mixes different proportions of LLM-generated data, which we refer to as synthetic data. Our experimental results show that, with minimal distribution shift, moderately diverse LLM-generated data can enhance model performance in scenarios with insufficient labeled data, whereas highly diverse generated data has a negative impact. We hope our empirical findings will offer valuable guidance for future studies on LLMs as data generators.
• 摘要：通过使用LLM生成的数据来训练下游模型的大型语言模型（LLMS）的显着生成能力已成为减轻特定域中数据稀缺并减少耗时注释的有前途的方法。但是，最近的研究强调了一个关键问题：自我生成数据的迭代培训导致模型崩溃，其中模型性能随着时间而变化。尽管对LLM生成数据的含义进行了广泛的研究，但这些作品经常忽略数据多样性的重要性，这是数据质量的关键因素。在这项工作中，我们旨在了解LLM生成数据对下游模型性能的多样性的含义。具体而言，我们探讨了LLM生成数据的多样性水平如何影响下游模型的性能。此外，我们研究了经过培训的模型的性能，该模型将不同比例的LLM生成数据混合在一起，我们称为合成数据。我们的实验结果表明，随着分布的最小变化，中等不同的LLM生成的数据可以在标记数据不足的情况下增强模型性能，而高度多样化的数据产生了负面影响。我们希望我们的经验发现将为以后的LLM作为数据生成器提供宝贵的指导。

Title: EmoStage: A Framework for Accurate Empathetic Response Generation via Perspective-Taking and Phase Recognition

• Authors: Zhiyang Qi, Keiko Takamizo, Mariko Ukiyo, Michimasa Inaba
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19279
• Pdf URL: https://arxiv.org/pdf/2506.19279
• Copy Paste: [[2506.19279]] EmoStage: A Framework for Accurate Empathetic Response Generation via Perspective-Taking and Phase Recognition(https://arxiv.org/abs/2506.19279)
• Keywords: language model, llm
• Abstract: The rising demand for mental health care has fueled interest in AI-driven counseling systems. While large language models (LLMs) offer significant potential, current approaches face challenges, including limited understanding of clients' psychological states and counseling stages, reliance on high-quality training data, and privacy concerns associated with commercial deployment. To address these issues, we propose EmoStage, a framework that enhances empathetic response generation by leveraging the inference capabilities of open-source LLMs without additional training data. Our framework introduces perspective-taking to infer clients' psychological states and support needs, enabling the generation of emotionally resonant responses. In addition, phase recognition is incorporated to ensure alignment with the counseling process and to prevent contextually inappropriate or inopportune responses. Experiments conducted in both Japanese and Chinese counseling settings demonstrate that EmoStage improves the quality of responses generated by base models and performs competitively with data-driven methods.
• 摘要：对心理保健的需求不断上升，激发了人们对AI驱动的咨询系统的兴趣。尽管大型语言模型（LLMS）具有巨大的潜力，但当前的方法面临挑战，包括对客户的心理状态和咨询阶段的了解有限，对高质量培训数据的依赖以及与商业部署有关的隐私问题。为了解决这些问题，我们提出了Emostage，该框架通过在没有其他培训数据的情况下利用开源LLM的推理能力来增强慈善响应的产生。我们的框架引入了观点，以推断客户的心理状态和支持需求，从而产生情感共鸣的反应。此外，还合并了阶段识别，以确保与咨询过程保持一致，并防止上下文不适当或不合时宜的响应。在日本和中国咨询环境中进行的实验表明，发射可以提高基本模型产生的响应质量，并通过数据驱动的方法进行竞争性。

Title: JCAPT: A Joint Modeling Approach for CAPT

• Authors: Tzu-Hsuan Yang, Yue-Yang He, Berlin Chen
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2506.19315
• Pdf URL: https://arxiv.org/pdf/2506.19315
• Copy Paste: [[2506.19315]] JCAPT: A Joint Modeling Approach for CAPT(https://arxiv.org/abs/2506.19315)
• Keywords: prompt
• Abstract: Effective pronunciation feedback is critical in second language (L2) learning, for which computer-assisted pronunciation training (CAPT) systems often encompass two key tasks: automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD). Recent work has shown that joint modeling of these two tasks can yield mutual benefits. Our unified framework leverages Mamba, a selective state space model (SSM), while integrating phonological features and think token strategies to jointly enhance interpretability and fine-grained temporal reasoning in APA and MDD. To our knowledge, this is the first study to combine phonological attribution, SSM-based modeling, and prompting in CAPT. A series of experiments conducted on the speechocean762 benchmark demonstrate that our model consistently outperforms prior methods, particularly on the MDD task.
• 摘要：有效的发音反馈在第二语言（L2）学习中至关重要，对于计算机辅助发音训练（Capt）系统通常涵盖了两个关键任务：自动发音评估（APA）和错误发音检测和诊断（MDD）。最近的工作表明，这两个任务的联合建模可以产生相互利益。我们的统一框架利用了选择性状态空间模型（SSM）Mamba，同时集成了语音特征并思考代币策略，以共同增强APA和MDD中的可解释性和细粒度的时间推理。据我们所知，这是第一项结合语音归因，基于SSM的建模和提示的研究。一系列在SecemoCean762基准上进行的实验表明，我们的模型始终优于先前的方法，尤其是在MDD任务上。

Title: Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation

• Authors: Jisu Shin, Juhyun Oh, Eunsu Kim, Hoyun Song, Alice Oh
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2506.19352
• Pdf URL: https://arxiv.org/pdf/2506.19352
• Copy Paste: [[2506.19352]] Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation(https://arxiv.org/abs/2506.19352)
• Keywords: language model, llm
• Abstract: Ensuring persona fidelity in large language models (LLMs) is essential for maintaining coherent and engaging human-AI interactions. However, LLMs often exhibit Out-of-Character (OOC) behavior, where generated responses deviate from an assigned persona, leading to inconsistencies that affect model reliability. Existing evaluation methods typically assign single scores to entire responses, struggling to capture subtle persona misalignment, particularly in long-form text generation. To address this limitation, we propose an atomic-level evaluation framework that quantifies persona fidelity at a finer granularity. Our three key metrics measure the degree of persona alignment and consistency within and across generations. Our approach enables a more precise and realistic assessment of persona fidelity by identifying subtle deviations that real users would encounter. Through our experiments, we demonstrate that our framework effectively detects persona inconsistencies that prior methods overlook. By analyzing persona fidelity across diverse tasks and personality types, we reveal how task structure and persona desirability influence model adaptability, highlighting challenges in maintaining consistent persona expression.
• 摘要：确保在大语言模型（LLM）中的角色忠诚度对于维持连贯且引人入胜的人类互动至关重要。但是，LLM经常表现出特征（OOC）的行为，而产生的响应偏离了分配的角色，从而导致影响模型可靠性的不一致之处。现有的评估方法通常将单个分数分配给整个响应，努力捕获微妙的角色未对准，尤其是在长篇文本生成中。为了解决这一限制，我们提出了一个原子级评估框架，该框架可以量化更精细的粒度。我们的三个关键指标衡量了世代和各个世代内的角色一致性和一致性的程度。我们的方法通过确定真实用户会遇到的微妙偏差来实现对角色忠诚度的更精确和现实的评估。通过我们的实验，我们证明了我们的框架有效地检测了先前方法忽略的角色不一致。通过分析各种任务和人格类型的角色忠诚，我们揭示了任务结构和性格可取性如何影响模型的适应性，强调了保持一致的角色表达方面的挑战。

Title: Measuring and Guiding Monosemanticity

• Authors: Ruben Härle, Felix Friedrich, Manuel Brack, Stephan Wäldchen, Björn Deiseroth, Patrick Schramowski, Kristian Kersting
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19382
• Pdf URL: https://arxiv.org/pdf/2506.19382
• Copy Paste: [[2506.19382]] Measuring and Guiding Monosemanticity(https://arxiv.org/abs/2506.19382)
• Keywords: language model, llm
• Abstract: There is growing interest in leveraging mechanistic interpretability and controllability to better understand and influence the internal dynamics of large language models (LLMs). However, current methods face fundamental challenges in reliably localizing and manipulating feature representations. Sparse Autoencoders (SAEs) have recently emerged as a promising direction for feature extraction at scale, yet they, too, are limited by incomplete feature isolation and unreliable monosemanticity. To systematically quantify these limitations, we introduce Feature Monosemanticity Score (FMS), a novel metric to quantify feature monosemanticity in latent representation. Building on these insights, we propose Guided Sparse Autoencoders (G-SAE), a method that conditions latent representations on labeled concepts during training. We demonstrate that reliable localization and disentanglement of target concepts within the latent space improve interpretability, detection of behavior, and control. Specifically, our evaluations on toxicity detection, writing style identification, and privacy attribute recognition show that G-SAE not only enhances monosemanticity but also enables more effective and fine-grained steering with less quality degradation. Our findings provide actionable guidelines for measuring and advancing mechanistic interpretability and control of LLMs.
• 摘要：利用机械性解释性和可控性，以更好地理解和影响大语言模型（LLMS）的内部动力学的兴趣越来越大。但是，当前的方法在可靠地本地化和操纵特征表示方面面临着根本挑战。稀疏的自动编码器（SAE）最近成为了大规模提取特征提取方向的方向，但是它们也受到不完整的特征隔离和不可靠的单色性的限制。为了系统地量化这些局限性，我们引入了特征单体性评分（FMS），这是一种量化潜在表示特征单体性的新型指标。在这些见解的基础上，我们提出了指导的稀疏自动编码器（G-SAE），这种方法可以在训练过程中对标记概念的潜在表示。我们证明，潜在空间内目标概念的可靠定位和分离可改善行为和控制的可解释性。具体而言，我们对毒性检测，写作样式识别和隐私属性识别的评估表明，G-SAE不仅增强了单体性，而且还可以使更有效和细粒度的转向质量较低。我们的发现提供了可行的指南，以衡量和推进LLM的机械性解释性和控制。

Title: Automated Detection of Pre-training Text in Black-box LLMs

• Authors: Ruihan Hu, Yu-Ming Shang, Jiankun Peng, Wei Luo, Yazhe Wang, Xi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19399
• Pdf URL: https://arxiv.org/pdf/2506.19399
• Copy Paste: [[2506.19399]] Automated Detection of Pre-training Text in Black-box LLMs(https://arxiv.org/abs/2506.19399)
• Keywords: language model, llm
• Abstract: Detecting whether a given text is a member of the pre-training data of Large Language Models (LLMs) is crucial for ensuring data privacy and copyright protection. Most existing methods rely on the LLM's hidden information (e.g., model parameters or token probabilities), making them ineffective in the black-box setting, where only input and output texts are accessible. Although some methods have been proposed for the black-box setting, they rely on massive manual efforts such as designing complicated questions or instructions. To address these issues, we propose VeilProbe, the first framework for automatically detecting LLMs' pre-training texts in a black-box setting without human intervention. VeilProbe utilizes a sequence-to-sequence mapping model to infer the latent mapping feature between the input text and the corresponding output suffix generated by the LLM. Then it performs the key token perturbations to obtain more distinguishable membership features. Additionally, considering real-world scenarios where the ground-truth training text samples are limited, a prototype-based membership classifier is introduced to alleviate the overfitting issue. Extensive evaluations on three widely used datasets demonstrate that our framework is effective and superior in the black-box setting.
• 摘要：检测给定文本是否是大语言模型（LLMS）的预训练数据的成员，对于确保数据隐私和版权保护至关重要。大多数现有方法依赖于LLM的隐藏信息（例如，模型参数或令牌概率），使它们在黑框设置中无效，在黑框设置中，只能访问输入和输出文本。尽管已经针对黑框设置提出了一些方法，但它们依靠大量的手动工作，例如设计复杂的问题或说明。为了解决这些问题，我们提出了Veilprobe，这是自动在没有人类干预的情况下在黑盒环境中自动检测LLM的预训练文本的第一个框架。 Veilprobe利用序列到序列映射模型来推断输入文本和LLM生成的相应输出后缀之间的潜在映射特征。然后，它执行关键令牌扰动以获得更明显的会员功能。此外，考虑到基础真相培训文本样本有限的现实情况，引入了基于原型的会员分类器来减轻过度拟合问题。对三个广泛使用的数据集进行了广泛的评估表明，我们的框架在黑框设置中是有效且优越的。

Title: Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study

• Authors: Yingji Zhang, Marco Valentino, Danilo S. Carvalho, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19418
• Pdf URL: https://arxiv.org/pdf/2506.19418
• Copy Paste: [[2506.19418]] Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study(https://arxiv.org/abs/2506.19418)
• Keywords: language model
• Abstract: Incorporating explicit reasoning rules within the latent space of language models (LMs) offers a promising pathway to enhance generalisation, interpretability, and controllability. While current Transformer-based language models have shown strong performance on Natural Language Inference (NLI) tasks, they often rely on memorisation rather than rule-based inference. This work investigates how reasoning rules can be explicitly embedded and memorised within the LMs through Language Variational Autoencoders (VAEs). We propose a complete pipeline for learning reasoning rules within Transformer-based language VAEs. This pipeline encompasses three rule-based reasoning tasks, a supporting theoretical framework, and a practical end-to-end architecture. The experiment illustrates the following findings: Disentangled reasoning: Under explicit signal supervision, reasoning rules - viewed as functional mappings - can be disentangled within the encoder's parametric space. This separation results in distinct clustering of rules in the output feature space. Prior knowledge injection: injecting reasoning information into the Query enables the model to more effectively retrieve the stored value Value from memory based on Key. This approach offers a simple method for integrating prior knowledge into decoder-only language models. Performance bottleneck: In mathematical reasoning tasks using Qwen2.5(0.5B), increasing sample count doesn't improve performance beyond a point. Moreover, ffn layers are better than attention layers at preserving the separation of reasoning rules in the model's parameters.
• 摘要：将明确的推理规则纳入语言模型的潜在空间（LMS）提供了一种有希望的途径，以增强概括，可解释性和可控性。尽管当前基于变压器的语言模型在自然语言推理（NLI）任务上表现出很强的表现，但它们通常依赖于记忆而不是基于规则的推论。这项工作调查了如何通过语言自动编码器（VAE）明确地将推理规则明确嵌入和记忆。我们建议在基于变压器的语言VAE中学习推理规则的完整管道。该管道包括三个基于规则的推理任务，一个支持的理论框架以及实用的端到端体系结构。该实验说明了以下发现：删除的推理：在明确的信号监督下，推理规则 - 被视为功能映射 - 可以在编码器的参数空间中删除。这种分离导致在输出特征空间中不同的规则聚类。先验知识注入：将推理信息注入查询中，该模型可以根据键从内存中更有效地检索存储的值。这种方法提供了一种简单的方法，可以将先验知识整合到仅解码器的语言模型中。绩效瓶颈：在使用QWEN2.5（0.5B）的数学推理任务中，增加样本计数并不能改善超出点的性能。此外，在保留模型参数中推理规则的分离时，FFN层比注意层要好。

Title: Can Large Language Models Capture Human Annotator Disagreements?

• Authors: Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Elliott Ash
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19467
• Pdf URL: https://arxiv.org/pdf/2506.19467
• Copy Paste: [[2506.19467]] Can Large Language Models Capture Human Annotator Disagreements?(https://arxiv.org/abs/2506.19467)
• Keywords: language model, llm
• Abstract: Human annotation variation (i.e., annotation disagreements) is common in NLP and often reflects important information such as task subjectivity and sample ambiguity. While Large Language Models (LLMs) are increasingly used for automatic annotation to reduce human effort, their evaluation often focuses on predicting the majority-voted "ground truth" labels. It is still unclear, however, whether these models also capture informative human annotation variation. Our work addresses this gap by extensively evaluating LLMs' ability to predict annotation disagreements without access to repeated human labels. Our results show that LLMs struggle with modeling disagreements, which can be overlooked by majority label-based evaluations. Notably, while RLVR-style (Reinforcement learning with verifiable rewards) reasoning generally boosts LLM performance, it degrades performance in disagreement prediction. Our findings highlight the critical need for evaluating and improving LLM annotators in disagreement modeling. Code and data at this https URL.
• 摘要：人类注释变化（即注释分歧）在NLP中很常见，并且通常反映了重要信息，例如任务主观性和样本歧义。尽管大型语言模型（LLM）越来越多地用于自动注释以减少人类的努力，但其评估通常集中于预测大多数投票的“地面真相”标签。然而，目前尚不清楚这些模型是否还捕获了人类注释变化。我们的工作通过广泛评估LLM预测注释分歧的能力而无需访问重复的人类标签的能力来解决这一差距。我们的结果表明，LLM在建模分歧上努力，这可能会因大多数基于标签的评估而忽略。值得注意的是，尽管RLVR风格（具有可验证奖励的增强学习）通常会提高LLM的性能，但它会降低分歧预测的性能。我们的发现突出了评估和改善分歧建模中LLM注释者的关键需​​求。此HTTPS URL上的代码和数据。

Title: MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages

• Authors: Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Liu, Haobin Lin, Bingni Zhang, Taifeng Wang, Mykola Pechenizkiy, Meng Fang, Yin Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19468
• Pdf URL: https://arxiv.org/pdf/2506.19468
• Copy Paste: [[2506.19468]] MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages(https://arxiv.org/abs/2506.19468)
• Keywords: language model, llm
• Abstract: Multilingual large language models (LLMs) are advancing rapidly, with new models frequently claiming support for an increasing number of languages. However, existing evaluation datasets are limited and lack cross-lingual alignment, leaving assessments of multilingual capabilities fragmented in both language and skill coverage. To address this, we introduce MuBench, a benchmark covering 61 languages and evaluating a broad range of capabilities. We evaluate several state-of-the-art multilingual LLMs and find notable gaps between claimed and actual language coverage, particularly a persistent performance disparity between English and low-resource languages. Leveraging MuBench's alignment, we propose Multilingual Consistency (MLC) as a complementary metric to accuracy for analyzing performance bottlenecks and guiding model improvement. Finally, we pretrain a suite of 1.2B-parameter models on English and Chinese with 500B tokens, varying language ratios and parallel data proportions to investigate cross-lingual transfer dynamics.
• 摘要：多语言大语言模型（LLM）正在迅速发展，新模型经常声称支持越来越多的语言。但是，现有的评估数据集是有限的，并且缺乏跨语性的一致性，从而使对语言和技能覆盖的多语言能力进行评估。为了解决这个问题，我们介绍了Mubench，这是一种涵盖61种语言并评估广泛功能的基准测试。我们评估了几种最先进的多语言LLM，并在声称和实际语言覆盖范围之间找到显着的差距，尤其是英语和低资源语言之间的持续性能差异。利用Mubench的对齐方式，我们提出多语言一致性（MLC），作为分析性能瓶颈和指导模型改进的准确性的补充度量。最后，我们在英语和中文上为1.2B参数模型提供了500b代币，不同语言比率和平行数据比例的套件，以研究跨语性转移动态。

Title: Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models

• Authors: Marcos Estecha-Garitagoitia, Chen Zhang, Mario Rodríguez-Cantelar, Luis Fernando D'Haro
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19483
• Pdf URL: https://arxiv.org/pdf/2506.19483
• Copy Paste: [[2506.19483]] Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models(https://arxiv.org/abs/2506.19483)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: This paper provides preliminary results on exploring the task of performing turn-level data augmentation for dialogue system based on different types of commonsense relationships, and the automatic evaluation of the generated synthetic turns. The proposed methodology takes advantage of the extended knowledge and zero-shot capabilities of pretrained Large Language Models (LLMs) to follow instructions, understand contextual information, and their commonsense reasoning capabilities. The approach draws inspiration from methodologies like Chain-of-Thought (CoT), applied more explicitly to the task of prompt-based generation for dialogue-based data augmentation conditioned on commonsense attributes, and the automatic evaluation of the generated dialogues. To assess the effectiveness of the proposed approach, first we extracted 200 randomly selected partial dialogues, from 5 different well-known dialogue datasets, and generate alternative responses conditioned on different event commonsense attributes. This novel dataset allows us to measure the proficiency of LLMs in generating contextually relevant commonsense knowledge, particularly up to 12 different specific ATOMIC [10] database relations. Secondly, we propose an evaluation framework to automatically detect the quality of the generated dataset inspired by the ACCENT [26] metric, which offers a nuanced approach to assess event commonsense. However, our method does not follow ACCENT's complex eventrelation tuple extraction process. Instead, we propose an instruction-based prompt for each commonsense attribute and use state-of-the-art LLMs to automatically detect the original attributes used when creating each augmented turn in the previous step. Preliminary results suggest that our approach effectively harnesses LLMs capabilities for commonsense reasoning and evaluation in dialogue systems.
• 摘要：本文提供了基于不同类型的常识关系以及对生成的合成转弯的自动评估对话系统执行对话系统的转向级数据增强任务的初步结果。所提出的方法利用了经过验证的大语言模型（LLMS）的扩展知识和零拍功能，以遵循说明，了解上下文信息及其常识性推理能力。该方法从诸如《思想链》（COT）之类的方法中汲取灵感，更明确地应用于基于对话的数据增强的基于对话的数据，以常识性属性为条件，以及对生成的对话的自动评估。为了评估所提出方法的有效性，首先，我们从5个不同的知名对话数据集中提取了200个随机选择的部分对话，并生成以不同事件常识性属性为条件的替代响应。这个新颖的数据集使我们能够衡量LLM在生成上下文相关的常识知识方面的熟练程度，尤其是多达12个不同的特定原子[10]数据库关系。其次，我们提出了一个评估框架，以自动检测受重音[26]指标启发的生成数据集的质量，该数据提供了一种细微的方法来评估事件常识。但是，我们的方法不遵循Accent复杂的事件元组提取过程。取而代之的是，我们为每个常识性属性提出了一个基于指令的提示，并使用最先进的LLMS自动检测到上一步中创建每个增强转弯时使用的原始属性。初步结果表明，我们的方法有效地利用了LLMS在对话系统中的常识性推理和评估功能。

Title: Dialogic Pedagogy for Large Language Models: Aligning Conversational AI with Proven Theories of Learning

• Authors: Russell Beale
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2506.19484
• Pdf URL: https://arxiv.org/pdf/2506.19484
• Copy Paste: [[2506.19484]] Dialogic Pedagogy for Large Language Models: Aligning Conversational AI with Proven Theories of Learning(https://arxiv.org/abs/2506.19484)
• Keywords: language model, llm, prompt, retrieval-augmented generation, agent
• Abstract: Large Language Models (LLMs) are rapidly transforming education by enabling rich conversational learning experiences. This article provides a comprehensive review of how LLM-based conversational agents are being used in higher education, with extensions to secondary and lifelong learning contexts. We synthesize existing literature on LLMs in education and theories of conversational and dialogic pedagogy - including Vygotsky's sociocultural learning (scaffolding and the Zone of Proximal Development), the Socratic method, and Laurillard's conversational framework - and examine how prompting strategies and retrieval-augmented generation (RAG) can align LLM behaviors with these pedagogical theories, and how it can support personalized, adaptive learning. We map educational theories to LLM capabilities, highlighting where LLM-driven dialogue supports established learning principles and where it challenges or falls short of traditional pedagogical assumptions. Notable gaps in applying prior theories to LLMs are identified, such as the models tendency to provide direct answers instead of fostering co-construction of knowledge, and the need to account for the constant availability and broad but non-human expertise of LLM tutors. In response, we propose practical strategies to better align LLM interactions with sound pedagogy - for example, designing prompts that encourage Socratic questioning, scaffolded guidance, and student reflection, as well as integrating retrieval mechanisms to ensure accuracy and contextual relevance. Our aim is to bridge the gap between educational theory and the emerging practice of AI-driven conversational learning, offering insights and tools for making LLM-based dialogues more educationally productive and theory-aligned.
• 摘要：大型语言模型（LLM）通过实现丰富的对话学习经验来迅速改变教育。本文对高等教育中如何使用基于LLM的对话代理的全面回顾，并扩展了中学和终身学习环境。 We synthesize existing literature on LLMs in education and theories of conversational and dialogic pedagogy - including Vygotsky's sociocultural learning (scaffolding and the Zone of Proximal Development), the Socratic method, and Laurillard's conversational framework - and examine how prompting strategies and retrieval-augmented generation (RAG) can align LLM behaviors with these pedagogical theories, and how it can support个性化的自适应学习。我们将教育理论映射到LLM的功能，突出显示了LLM驱动的对话支持确定的学习原理以及它挑战或涉及传统教学假设的地方。确定了将先验理论应用于LLM的显着差距，例如，提供直接答案而不是促进知识共同构建的模型趋势，以及需要考虑LLM Tutors的持续可用性以及广泛但非人类的专业知识的需求。作为回应，我们提出了实用策略，以更好地使LLM与声学教育学的相互作用保持一致 - 例如，设计提示，鼓励苏格拉底质询问，脚手架指导和学生反思，以及整合检索机制以确保准确性和上下文相关性。我们的目的是弥合教育理论与AI驱动的对话学习的新兴实践之间的差距，为使基于LLM的对话提供的见解和工具在教育上更具富有成效的效率和理论一致性。

Title: AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models

• Authors: Zeyu Li, Chuanfu Xiao, Yang Wang, Xiang Liu, Zhenheng Tang, Baotong Lu, Mao Yang, Xinyu Chen, Xiaowen Chu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19505
• Pdf URL: https://arxiv.org/pdf/2506.19505
• Copy Paste: [[2506.19505]] AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models(https://arxiv.org/abs/2506.19505)
• Keywords: language model, llm
• Abstract: Quantization has emerged as an effective and lightweight solution to reduce the memory footprint of the KV cache in Large Language Models (LLMs). Nevertheless, minimizing the performance degradation caused by ultra-low-bit KV cache quantization remains a significant challenge. We observe that quantizing the KV cache of different tokens has varying impacts on the quality of attention outputs. To systematically investigate this phenomenon, we perform forward error propagation analysis on attention and propose the Anchor Score (AnS) that quantifies the sensitivity of each token's KV cache to quantization-induced error. Our analysis reveals significant disparities in AnS across tokens, suggesting that preserving a small subset with full precision (FP16) of high-AnS tokens can greatly mitigate accuracy loss in aggressive quantization scenarios. Based on this insight, we introduce AnTKV, a novel framework that leverages Anchor Token-aware Vector Quantization to compress the KV cache. Furthermore, to support efficient deployment, we design and develop a triton kernel that is fully compatible with FlashAttention, enabling fast online Anchor Token selection. AnTKV enables LLaMA-3-8B to handle context lengths up to 840K tokens on a single 80GB A100 GPU, while achieving up to 3.5x higher decoding throughput compared to the FP16 baseline. Our experiment results demonstrate that AnTKV matches or outperforms prior works such as KIVI, SKVQ, KVQuant, and CQ under 4-bit settings. More importantly, AnTKV achieves significantly lower perplexity under ultra-low-bit quantization on Mistral-7B, with only 6.32 at 1-bit and 8.87 at 0.375-bit, compared to the FP16 baseline of 4.73.
• 摘要：量化已成为一种有效且轻巧的解决方案，以减少大语言模型（LLMS）中KV缓存的内存足迹。然而，最大程度地减少了由超低位KV缓存量化引起的性能降解仍然是一个重大挑战。我们观察到，量化不同令牌的KV缓存对注意力输出质量的影响有所不同。为了系统地研究这种现象，我们对注意力进行了正向错误传播分析，并提出了锚定评分（ANS），该分数（ANS）量化了每个令牌KV缓存对量化诱导的误差的敏感性。我们的分析揭示了整个代币的ANS的显着差异，这表明保留具有全精度（FP16）高度令牌的小子集可以大大减轻积极量化场景中的准确性损失。基于这种见解，我们引入了ANTKV，这是一个新型框架，利用锚点令牌量的矢量量化来压缩KV缓存。此外，为了支持有效的部署，我们设计和开发了与Flashertention完全兼容的Triton内核，从而可以快速在线锚定令牌选择。 ANTKV使Llama-3-8B可以在单个80GB A100 GPU上处理高达840K令牌的上下文长度，同时与FP16基线相比，分解吞吐量高达3.5倍。我们的实验结果表明，在4位设置下，ANTKV匹配或优于先前的作品，例如KIVI，SKVQ，KVQUANT和CQ。更重要的是，在Mistral-7b上的超低位量化下，ANTKV的困惑明显降低，而1位的6.32仅为6.32，而在0.375位的8.87比FP16基线为4.73。

Title: heiDS at ArchEHR-QA 2025: From Fixed-k to Query-dependent-k for Retrieval Augmented Generation

• Authors: Ashish Chouhan, Michael Gertz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19512
• Pdf URL: https://arxiv.org/pdf/2506.19512
• Copy Paste: [[2506.19512]] heiDS at ArchEHR-QA 2025: From Fixed-k to Query-dependent-k for Retrieval Augmented Generation(https://arxiv.org/abs/2506.19512)
• Keywords: retrieval augmented generation
• Abstract: This paper presents the approach of our team called heiDS for the ArchEHR-QA 2025 shared task. A pipeline using a retrieval augmented generation (RAG) framework is designed to generate answers that are attributed to clinical evidence from the electronic health records (EHRs) of patients in response to patient-specific questions. We explored various components of a RAG framework, focusing on ranked list truncation (RLT) retrieval strategies and attribution approaches. Instead of using a fixed top-k RLT retrieval strategy, we employ a query-dependent-k retrieval strategy, including the existing surprise and autocut methods and two new methods proposed in this work, autocut* and elbow. The experimental results show the benefits of our strategy in producing factual and relevant answers when compared to a fixed-$k$.
• 摘要：本文介绍了我们的团队称为Archehr-QA 2025共享任务的Heids的方法。使用检索增强产生（RAG）框架的管道旨在产生答案，这些答案归因于患者的电子健康记录（EHR）的临床证据，以应对患者特定的问题。我们探索了RAG框架的各种组件，重点介绍了排名的清单截断（RLT）检索策略和归因方法。我们没有使用固定的TOP-K RLT检索策略，而是采用了与查询有关的K检索策略，包括现有的惊喜和自动关闭方法以及这项工作中提出的两种新方法，Autocut*和Elbow。实验结果表明，与固定$ K $相比，我们策略在产生事实和相关答案方面的好处。

Title: Automatic Posology Structuration : What role for LLMs?

• Authors: Natalia Bobkova, Laura Zanella-Calzada, Anyes Tafoughalt, Raphaël Teboul, François Plesse, Félix Gaschi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19525
• Pdf URL: https://arxiv.org/pdf/2506.19525
• Copy Paste: [[2506.19525]] Automatic Posology Structuration : What role for LLMs?(https://arxiv.org/abs/2506.19525)
• Keywords: language model, llm, prompt
• Abstract: Automatically structuring posology instructions is essential for improving medication safety and enabling clinical decision support. In French prescriptions, these instructions are often ambiguous, irregular, or colloquial, limiting the effectiveness of classic ML pipelines. We explore the use of Large Language Models (LLMs) to convert free-text posologies into structured formats, comparing prompt-based methods and fine-tuning against a "pre-LLM" system based on Named Entity Recognition and Linking (NERL). Our results show that while prompting improves performance, only fine-tuned LLMs match the accuracy of the baseline. Through error analysis, we observe complementary strengths: NERL offers structural precision, while LLMs better handle semantic nuances. Based on this, we propose a hybrid pipeline that routes low-confidence cases from NERL (<0.8) to the LLM, selecting outputs based on confidence scores. This strategy achieves 91% structuration accuracy while minimizing latency and compute. Our results show that this hybrid approach improves structuration accuracy while limiting computational cost, offering a scalable solution for real-world clinical use.
• 摘要：自动构建知识的说明对于改善药物安全和实现临床决策支持至关重要。在法国处方中，这些说明通常是模棱两可，不规则或口语，从而限制了经典ML管道的有效性。我们探讨了大型语言模型（LLMS）将自由文本寄存器转换为结构化格式，比较基于及时的方法并与基于命名实体识别和链接（NERL）的“ pre-llm”系统进行微调。我们的结果表明，尽管提示提高性能，但只有微调的LLM与基线的准确性相匹配。通过错误分析，我们观察到互补的优势：NERL提供结构性精度，而LLM可以更好地处理语义细微差别。基于此，我们提出了一条混合管道，该管道将较低的信心案例从NERL（<0.8）伸向LLM，并根据置信度得分选择输出。该策略可实现91％的结构准确性，同时最大程度地减少延迟和计算。我们的结果表明，这种混合方法提高了结构的准确性，同时限制了计算成本，为现实世界中提供了可扩展的解决方案。

Title: KnowMap: Efficient Knowledge-Driven Task Adaptation for LLMs

• Authors: Kelin Fu, Kaigui Bian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19527
• Pdf URL: https://arxiv.org/pdf/2506.19527
• Copy Paste: [[2506.19527]] KnowMap: Efficient Knowledge-Driven Task Adaptation for LLMs(https://arxiv.org/abs/2506.19527)
• Keywords: language model, gpt, llm, agent
• Abstract: While Large Language Models (LLMs) possess significant capabilities in open-world agent tasks, they also face challenges in rapidly adapting to new, specialized tasks due to their reliance on static pre-trained knowledge. Traditional methods such as fine-tuning are often costly, data-intensive, and may lead to "catastrophic forgetting." Therefore, we present KnowMap, a novel approach that dynamically constructs a knowledge base from environmental and experiential data. KnowMap fine-tunes a small knowledge-embedding model to equip a larger LLM with valuable task-specific knowledge. Our experiments on the ScienceWorld benchmark demonstrate 17.71% improvement for the performance of gpt-4-turbo model. KnowMap not only provides an efficient and effective means for LLM task-adapting, but also highlights how integrating environmental and experiential knowledge can enhance LLMs' reasoning capabilities.
• 摘要：尽管大型语言模型（LLMS）在开放世界代理任务中具有重要的功能，但由于依赖静态预训练的知识，它们在迅速适应新的专业任务方面也面临挑战。传统方法（例如微调）通常是昂贵的，数据密集的，并且可能导致“灾难性遗忘”。因此，我们提出了Knowmap，这是一种新颖的方法，该方法从环境和体验数据中动态构建知识库。知道图片微型的小型知识装置模型，以配备较大的LLM，并具有有价值的特定任务知识。我们对科学世界基准测试的实验表明，GPT-4-Turbo模型的性能提高了17.71％。 Knowmap不仅为LLM任务适应功能提供了一种有效的手段，而且还强调了整合环境和体验知识如何增强LLMS的推理能力。

Title: ECCoT: A Framework for Enhancing Effective Cognition via Chain of Thought in Large Language Model

• Authors: Zhenke Duan, Jiqun Pan, Jiani Tu, Xiaoyi Wang, Yanqing Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19599
• Pdf URL: https://arxiv.org/pdf/2506.19599
• Copy Paste: [[2506.19599]] ECCoT: A Framework for Enhancing Effective Cognition via Chain of Thought in Large Language Model(https://arxiv.org/abs/2506.19599)
• Keywords: language model, llm, prompt
• Abstract: In the era of large-scale artificial intelligence, Large Language Models (LLMs) have made significant strides in natural language processing. However, they often lack transparency and generate unreliable outputs, raising concerns about their interpretability. To address this, the Chain of Thought (CoT) prompting method structures reasoning into step-by-step deductions. Yet, not all reasoning chains are valid, and errors can lead to unreliable conclusions. We propose ECCoT, an End-to-End Cognitive Chain of Thought Validation Framework, to evaluate and refine reasoning chains in LLMs. ECCoT integrates the Markov Random Field-Embedded Topic Model (MRF-ETM) for topic-aware CoT generation and Causal Sentence-BERT (CSBert) for causal reasoning alignment. By filtering ineffective chains using structured ordering statistics, ECCoT improves interpretability, reduces biases, and enhances the trustworthiness of LLM-based decision-making. Key contributions include the introduction of ECCoT, MRF-ETM for topic-driven CoT generation, and CSBert for causal reasoning enhancement. Code is released at: this https URL.
• 摘要：在大规模人工智能的时代，大型语言模型（LLM）在自然语言处理方面取得了长足的进步。但是，它们通常缺乏透明度并产生不可靠的产出，从而引起了对其解释性的担忧。为了解决这个问题，思想链（COT）提示方法将推理推理到分步扣除中。但是，并非所有的推理链都是有效的，错误可能会导致不可靠的结论。我们提出了ECCOT是一种思想验证框架的端到端认知链，以评估和完善LLM中的推理链。 ECCOT用于集成了Markov随机现场插入的主题模型（MRF-ETM），以进行主题感知的COT生成和因果关系 - 因果推理一致性。通过使用结构化秩序统计来过滤无效的链条，ECCOT可提高可解释性，减少偏见并增强基于LLM的决策的可信度。主要贡献包括引入ECCOT，主题驱动的COT生成的MRF-ETM和因果推理增强的CSBERT。代码在以下位置发布：此HTTPS URL。

Title: Correcting Hallucinations in News Summaries: Exploration of Self-Correcting LLM Methods with External Knowledge

• Authors: Juraj Vladika, Ihsan Soydemir, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19607
• Pdf URL: https://arxiv.org/pdf/2506.19607
• Copy Paste: [[2506.19607]] Correcting Hallucinations in News Summaries: Exploration of Self-Correcting LLM Methods with External Knowledge(https://arxiv.org/abs/2506.19607)
• Keywords: language model, llm, hallucination, prompt
• Abstract: While large language models (LLMs) have shown remarkable capabilities to generate coherent text, they suffer from the issue of hallucinations -- factually inaccurate statements. Among numerous approaches to tackle hallucinations, especially promising are the self-correcting methods. They leverage the multi-turn nature of LLMs to iteratively generate verification questions inquiring additional evidence, answer them with internal or external knowledge, and use that to refine the original response with the new corrections. These methods have been explored for encyclopedic generation, but less so for domains like news summarization. In this work, we investigate two state-of-the-art self-correcting systems by applying them to correct hallucinated summaries using evidence from three search engines. We analyze the results and provide insights into systems' performance, revealing interesting practical findings on the benefits of search engine snippets and few-shot prompts, as well as high alignment of G-Eval and human evaluation.
• 摘要：尽管大型语言模型（LLM）表现出了出色的能力来产生连贯的文本，但它们遭受了幻觉问题的困扰 - 实际上不准确的陈述。在解决幻觉的众多方法中，尤其是有前途的方法是自我校正的方法。他们利用LLM的多转弯性质迭代产生验证问题，以询问其他证据，以内部或外部知识回答，并使用它来完善新校正的原始响应。这些方法已被探索用于百科全书的生成，但对于新闻摘要等领域的却较少。在这项工作中，我们通过使用三种搜索引擎的证据来纠正幻觉的摘要来调查两个最先进的自我校正系统。我们分析结果并提供有关系统性能的见解，揭示了有关搜索引擎片段和少量提示的好处的有趣实际发现，以及G-eval和人类评估的高度对齐。

Title: Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager

• Authors: Lucie Galland, Catherine Pelachaud, Florian Pecune
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19652
• Pdf URL: https://arxiv.org/pdf/2506.19652
• Copy Paste: [[2506.19652]] Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager(https://arxiv.org/abs/2506.19652)
• Keywords: language model, llm
• Abstract: In this work, we propose a novel framework that integrates large language models (LLMs) with an RL-based dialogue manager for open-ended dialogue with a specific goal. By leveraging hierarchical reinforcement learning to model the structured phases of dialogue and employ meta-learning to enhance adaptability across diverse user profiles, our approach enhances adaptability and efficiency, enabling the system to learn from limited data, transition fluidly between dialogue phases, and personalize responses to heterogeneous patient needs. We apply our framework to Motivational Interviews, aiming to foster behavior change, and demonstrate that the proposed dialogue manager outperforms a state-of-the-art LLM baseline in terms of reward, showing a potential benefit of conditioning LLMs to create open-ended dialogue systems with specific goals.
• 摘要：在这项工作中，我们提出了一个新颖的框架，该框架将大型语言模型（LLM）与基于RL的对话经理进行了开放式对话，并具有特定的目标。通过利用层次强化学习来建模对话的结构化阶段并采用元学习来增强各种用户概况之间的适应性，我们的方法可以增强适应性和效率，使系统能够从有限的数据中学习，对话阶段之间的过渡以及对异性患者需求的个性化反应。我们将框架应用于励志访谈，旨在促进行为改变，并证明拟议的对话经理在奖励方面优于最先进的LLM基准，这表明了调节LLMS以创建具有特定目标的开放式对话系统的潜在好处。

Title: Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?

• Authors: Chuxuan Hu, Yuxuan Zhu, Antony Kellermann, Caleb Biddulph, Suppakit Waiwitlikhit, Jason Benn, Daniel Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19733
• Pdf URL: https://arxiv.org/pdf/2506.19733
• Copy Paste: [[2506.19733]] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?(https://arxiv.org/abs/2506.19733)
• Keywords: language model, llm
• Abstract: Reinforcement post training (RPT) has recently shown promise in improving the reasoning abilities of large language models (LLMs). However, it remains unclear how well these improvements generalize to new domains, as prior work evaluates RPT models on data from the same domains used for fine-tuning. To understand the generalizability of RPT, we conduct two studies. (1) Observational: We compare a wide range of open-weight RPT models against their corresponding base models across multiple domains, including both seen and unseen domains in their fine-tuning data. (2) Interventional: we fine-tune LLMs with RPT on single domains and evaluate their performance across multiple domains. Both studies converge on the same conclusion that, although RPT brings substantial gains on tasks similar to the fine-tuning data, the gains generalize inconsistently and can vanish on domains with different reasoning patterns.
• 摘要：强化后培训（RPT）最近表现出了提高大语模型（LLMS）的推理能力的希望。但是，尚不清楚这些改进如何推广到新领域，因为先前的工作评估了来自用于微调的相同域的数据的RPT模型。为了了解RPT的普遍性，我们进行了两项研究。 （1）观察：我们将广泛的开放式RPT模型与跨多个领域的相应基本模型进行比较，包括其微调数据中可见和看不见的域。 （2）介入：我们在单个域上对RPT进行微调，并评估其在多个领域的性能。这两项研究都取决于同样的结论，尽管RPT在与微调数据相似的任务上带来了可观的收益，但收益不一致，并且可能在具有不同推理模式的域上消失。

Title: Arabic Dialect Classification using RNNs, Transformers, and Large Language Models: A Comparative Analysis

• Authors: Omar A.Essameldin, Ali O.Elbeih, Wael H.Gomaa, Wael F.Elsersy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.19753
• Pdf URL: https://arxiv.org/pdf/2506.19753
• Copy Paste: [[2506.19753]] Arabic Dialect Classification using RNNs, Transformers, and Large Language Models: A Comparative Analysis(https://arxiv.org/abs/2506.19753)
• Keywords: language model, llm, prompt, chat
• Abstract: The Arabic language is among the most popular languages in the world with a huge variety of dialects spoken in 22 countries. In this study, we address the problem of classifying 18 Arabic dialects of the QADI dataset of Arabic tweets. RNN models, Transformer models, and large language models (LLMs) via prompt engineering are created and tested. Among these, MARBERTv2 performed best with 65% accuracy and 64% F1-score. Through the use of state-of-the-art preprocessing techniques and the latest NLP models, this paper identifies the most significant linguistic issues in Arabic dialect identification. The results corroborate applications like personalized chatbots that respond in users' dialects, social media monitoring, and greater accessibility for Arabic communities.
• 摘要：阿拉伯语是世界上最受欢迎的语言之一，在22个国家使用了各种各样的方言。在这项研究中，我们解决了对阿拉伯语推文QADI数据集的18个阿拉伯语方言进行分类的问题。通过及时工程创建和测试了RNN模型，变压器模型和大型语言模型（LLMS）。其中，MARBERTV2的精度为65％和64％的F1得分表现最佳。通过使用最新的预处理技术和最新的NLP模型，本文确定了阿拉伯方言识别中最重要的语言问题。结果证实了在用户方言，社交媒体监控以及对阿拉伯社区的更大可访问性中响应的个性化聊天机器人等应用程序。

Title: SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

• Authors: Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, Dongbin Zhao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.19767
• Pdf URL: https://arxiv.org/pdf/2506.19767
• Copy Paste: [[2506.19767]] SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning(https://arxiv.org/abs/2506.19767)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks.
• 摘要：大型语言模型（LLMS）在推理任务方面取得了显着进步，但是监督微调（SFT）和增强学习（RL）的最佳整合仍然是一个基本挑战。通过从基于熵的角度对令牌分布，学习动态和集成机制的全面分析，我们揭示了这些范式之间的关键差异：SFT诱导了对LLM策略分布的粗粒全局变化，而RL则可以进行精细的选择性优化，而熵可作为训练有效性的关键指标。在这些观察结果的基础上，我们提出了监督的加强微调（SRFT），这是一种单阶段方法，可以通过熵感知的加权机制统一两个微调范式。我们的方法同时使用SFT和RL，可以使用演示和自我探索的推出，而不是通过两个阶段的顺序方法直接优化LLM。广泛的实验表明，SRFT达到59.1％的平均准确性，在五个数学推理基准上优于零-RL方法，而在三个分布基准的基准上，SRFT的平均准确性优于9.0％。

Title: Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study

• Authors: Yuqi Zhu, Yi Zhong, Jintian Zhang, Ziheng Zhang, Shuofei Qiao, Yujie Luo, Lun Du, Da Zheng, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2506.19794
• Pdf URL: https://arxiv.org/pdf/2506.19794
• Copy Paste: [[2506.19794]] Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study(https://arxiv.org/abs/2506.19794)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) hold promise in automating data analysis tasks, yet open-source models face significant limitations in these kinds of reasoning-intensive scenarios. In this work, we investigate strategies to enhance the data analysis capabilities of open-source LLMs. By curating a seed dataset of diverse, realistic scenarios, we evaluate models across three dimensions: data understanding, code generation, and strategic planning. Our analysis reveals three key findings: (1) Strategic planning quality serves as the primary determinant of model performance; (2) Interaction design and task complexity significantly influence reasoning capabilities; (3) Data quality demonstrates a greater impact than diversity in achieving optimal performance. We leverage these insights to develop a data synthesis methodology, demonstrating significant improvements in open-source LLMs' analytical reasoning capabilities.
• 摘要：大型语言模型（LLMS）在自动化数据分析任务方面有希望，但开源模型在这些类型的强化场景中面临着重大限制。在这项工作中，我们研究了增强开源LLM的数据分析功能的策略。通过策划一个不同，现实的场景的种子数据集，我们可以评估三个维度的模型：数据理解，代码生成和战略计划。我们的分析揭示了三个关键发现：（1）战略规划质量是模型性能的主要决定因素； （2）互动设计和任务复杂性显着影响推理能力； （3）数据质量在实现最佳性能方面表现出的影响比多样性更大。我们利用这些见解来开发数据综合方法，证明了开源LLMS的分析推理能力的显着改善。

Title: MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration

• Authors: Yucheng Zhou, Lingran Song, Jianbing Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.19835
• Pdf URL: https://arxiv.org/pdf/2506.19835
• Copy Paste: [[2506.19835]] MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration(https://arxiv.org/abs/2506.19835)
• Keywords: language model, llm, agent
• Abstract: Recent advancements in medical Large Language Models (LLMs) have showcased their powerful reasoning and diagnostic capabilities. Despite their success, current unified multimodal medical LLMs face limitations in knowledge update costs, comprehensiveness, and flexibility. To address these challenges, we introduce the Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis (MAM). Inspired by our empirical findings highlighting the benefits of role assignment and diagnostic discernment in LLMs, MAM decomposes the medical diagnostic process into specialized roles: a General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director, each embodied by an LLM-based agent. This modular and collaborative framework enables efficient knowledge updates and leverages existing medical LLMs and knowledge bases. Extensive experimental evaluations conducted on a wide range of publicly accessible multimodal medical datasets, incorporating text, image, audio, and video modalities, demonstrate that MAM consistently surpasses the performance of modality-specific LLMs. Notably, MAM achieves significant performance improvements ranging from 18% to 365% compared to baseline models. Our code is released at this https URL.
• 摘要：医学大语言模型（LLM）的最新进步展示了其强大的推理和诊断能力。尽管取得了成功，但目前的统一多模式医学LLM面临知识更新成本，全面性和灵活性的限制。为了应对这些挑战，我们介绍了多模式医学诊断（MAM）的模块化多代理框架。受我们的经验发现的启发，强调了LLMS角色分配和诊断识别的好处，MAM将医学诊断过程分解为专业角色：全科医生，专业团队，放射科医生，医学助理和董事，每个人都由LLM基于LLM的代理体现。这个模块化和协作的框架可实现有效的知识更新，并利用现有的医学LLM和知识库。在各种公共可访问的多模式医学数据集上进行的广泛的实验评估，结合了文本，图像，音频和视频方式，表明MAM始终超过了特定于模态的LLM的性能。值得注意的是，与基线模型相比，MAM的绩效改善范围从18％到365％。我们的代码在此HTTPS URL上发布。