2025-06-27

Title: Towards Probabilistic Question Answering Over Tabular Data

• Authors: Chen Shen, Sajjadur Rahman, Estevam Hruschka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.20747
• Pdf URL: https://arxiv.org/pdf/2506.20747
• Copy Paste: [[2506.20747]] Towards Probabilistic Question Answering Over Tabular Data(https://arxiv.org/abs/2506.20747)
• Keywords: language model, llm
• Abstract: Current approaches for question answering (QA) over tabular data, such as NL2SQL systems, perform well for factual questions where answers are directly retrieved from tables. However, they fall short on probabilistic questions requiring reasoning under uncertainty. In this paper, we introduce a new benchmark LUCARIO and a framework for probabilistic QA over large tabular data. Our method induces Bayesian Networks from tables, translates natural language queries into probabilistic queries, and uses large language models (LLMs) to generate final answers. Empirical results demonstrate significant improvements over baselines, highlighting the benefits of hybrid symbolic-neural reasoning.
• 摘要：在表格数据（例如NL2SQL系统）上，当前的问答方法（QA）在直接从表中直接检索答案的事实问题。但是，他们缺乏概率问题，需要在不确定性下进行推理。在本文中，我们介绍了一个新的基准Lucario和一个在大型表格数据上的概率质量质量检查的框架。我们的方法从表中引起贝叶斯网络，将自然语言查询转化为概率查询，并使用大型语言模型（LLMS）来生成最终答案。经验结果表明，基准的显着改善，强调了混合象征性神经推理的益处。

Title: Multi-lingual Functional Evaluation for Large Language Models

• Authors: Victor Ojewale, Inioluwa Deborah Raji, Suresh Venkatasubramanian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.20793
• Pdf URL: https://arxiv.org/pdf/2506.20793
• Copy Paste: [[2506.20793]] Multi-lingual Functional Evaluation for Large Language Models(https://arxiv.org/abs/2506.20793)
• Keywords: language model
• Abstract: Multi-lingual competence in large language models is often evaluated via static data benchmarks such as Belebele, M-MMLU and M-GSM. However, these evaluations often fail to provide an adequate understanding of the practical performance and robustness of models across multi-lingual settings. In response, we create multi-lingual functional benchmarks -- Cross-Lingual Grade School Math Symbolic (CL-GSM Symbolic) and Cross-Lingual Instruction-Following Eval (CL-IFEval)-- by translating existing functional benchmark templates from English to five additional languages that span the range of resources available for NLP: French, Spanish, Hindi, Arabic and Yoruba. Our results reveal that some static multi-lingual benchmarks capture functional performance much more closely than others (i.e. across models, there is a 24%, 17% and 18% decrease in performance between M-GSM and CL-GSM Symbolic in English, French and Spanish respectively; similarly there's a 15 - 24% performance drop across languages between Belebele and CL-IFEval, and only a 0.5% to 3% performance drop between M-MMLU and CL-IFEval). Similarly, we find that model robustness across languages varies significantly, with certain languages (eg. Arabic, English) being the most consistently well performing across evaluation iterations.
• 摘要：通常通过BELEBELE，M-MMLU和M-GSM等静态数据基准来评估大语言模型中的多语言能力。但是，这些评估通常无法充分了解多语言环境中模型的实际性能和鲁棒性。作为回应，我们通过将现有的功能性基准模板从英语转换为跨越五个其他资源可用于NLP：法国，西班牙语，西班牙语，Hindi，ArabiC和yoruba的其他语言，创建了多种语言功能基准 - 跨语言小学数学符号（CL-GSM符号）和跨语性指令遵循评估（CL-IFERVAL）。我们的结果表明，某些静态多语言基准比其他静态多语言基准捕获的功能性能要比其他且分别分别为24％，17％和18％的M-GSM和CL-GSM在英语，法语，法语和西班牙语中符号符号；同样，在Bel-fife和Cl-fife Val和Cl-fife Val的语言中，M-GSM和CL-GSM的表现下降了，并且MM之间的性能下降了15-24％。 cl-fifeval）。同样，我们发现跨语言的模型鲁棒性差异很大，某些语言（例如阿拉伯语，英语）是在评估迭代中表现最佳的表现。

Title: The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas

• Authors: Chenglei Si, Tatsunori Hashimoto, Diyi Yang
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.20803
• Pdf URL: https://arxiv.org/pdf/2506.20803
• Copy Paste: [[2506.20803]] The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas(https://arxiv.org/abs/2506.20803)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown promise in accelerating the scientific research pipeline. A key capability for this process is the ability to generate novel research ideas, and prior studies have found settings in which LLM-generated research ideas were judged as more novel than human-expert ideas. However, a good idea should not simply appear to be novel, it should also result in better research after being executed. To test whether AI-generated ideas lead to better research outcomes, we conduct an execution study by recruiting 43 expert researchers to execute randomly-assigned ideas, either written by experts or generated by an LLM. Each expert spent over 100 hours implementing the idea and wrote a 4-page short paper to document the experiments. All the executed projects are then reviewed blindly by expert NLP researchers. Comparing the review scores of the same ideas before and after execution, the scores of the LLM-generated ideas decrease significantly more than expert-written ideas on all evaluation metrics (novelty, excitement, effectiveness, and overall; p < 0.05), closing the gap between LLM and human ideas observed at the ideation stage. When comparing the aggregated review scores from the execution study, we even observe that for many metrics there is a flip in rankings where human ideas score higher than LLM ideas. This ideation-execution gap highlights the limitations of current LLMs in generating truly effective research ideas and the challenge of evaluating research ideas in the absence of execution outcomes.
• 摘要：大型语言模型（LLM）在加速科学研究方面表现出了希望。该过程的关键能力是产生新颖的研究思想的能力，并且先前的研究发现了LLM生成的研究思想的设置比人类专家的思想更为新颖。但是，一个好主意不应该简单地看上去是新颖的，在执行后也应该可以更好地研究。为了测试AI生成的想法是否导致更好的研究结果，我们通过招募43位专家研究人员来执行由专家撰写或由LLM生成的随机分配的想法来进行执行研究。每个专家花费了100多个小时来实施这个想法，并写了一篇4页的短论文来记录实验。然后，NLP专家研究人员盲目审查所有执行的项目。比较执行前后相同想法的评论分数，LLM生成的想法的分数比所有评估指标（新颖，兴奋，有效性和整体; p <0.05; p <0.05）的专家写作的想法的降低明显高，缩小了LLM和人类思想之间在幻想阶段观察到的差距。在比较执行研究的汇总评论分数时，我们甚至观察到，对于许多指标，人类想法得分高于LLM的想法的排名有一个翻转。这种构想 - 执行差距突出了当前LLM在产生真正有效的研究思想以及在没有执行结果的情况下评估研究思想的挑战中的局限性。

Title: MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answering

• Authors: Chinmay Gondhalekar, Urjitkumar Patel, Fang-Chun Yeh
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2506.20821
• Pdf URL: https://arxiv.org/pdf/2506.20821
• Copy Paste: [[2506.20821]] MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answering(https://arxiv.org/abs/2506.20821)
• Keywords: language model, gpt, llm, chat, retrieval-augmented generation
• Abstract: Financial documents--such as 10-Ks, 10-Qs, and investor presentations--span hundreds of pages and combine diverse modalities, including dense narrative text, structured tables, and complex figures. Answering questions over such content often requires joint reasoning across modalities, which strains traditional large language models (LLMs) and retrieval-augmented generation (RAG) pipelines due to token limitations, layout loss, and fragmented cross-modal context. We introduce MultiFinRAG, a retrieval-augmented generation framework purpose-built for financial QA. MultiFinRAG first performs multimodal extraction by grouping table and figure images into batches and sending them to a lightweight, quantized open-source multimodal LLM, which produces both structured JSON outputs and concise textual summaries. These outputs, along with narrative text, are embedded and indexed with modality-aware similarity thresholds for precise retrieval. A tiered fallback strategy then dynamically escalates from text-only to text+table+image contexts when necessary, enabling cross-modal reasoning while reducing irrelevant context. Despite running on commodity hardware, MultiFinRAG achieves 19 percentage points higher accuracy than ChatGPT-4o (free-tier) on complex financial QA tasks involving text, tables, images, and combined multimodal reasoning.
• 摘要：财务文件（例如10-ks，10-Qs和投资者的演示文稿），包括数百页，并结合了各种方式，包括密集的叙事文本，结构化表和复杂的数字。回答有关此类内容的问题通常需要跨模态的联合推理，这会构成传统的大型语言模型（LLMS）和检索功能增强的生成（RAG）管道，这是由于令牌限制，布局损失和零碎的跨模式环境。我们介绍了MultiFinrag，这是一种为财务质量保证的检索生成框架。 MultiFinrag首先通过将表和图形图像分组为批处理并将其发送到轻巧的，量化的开源多模式LLM来执行多模式提取，从而产生结构化的JSON输出和简洁的文本摘要。这些输出以及叙事文本被嵌入并用模式感知的相似性阈值进行嵌入和索引，以进行精确检索。然后，一个分层的后备策略在必要时动态从仅文本+表+图像上下文升级，从而实现跨模式推理，同时减少无关紧要的上下文。尽管在商品硬件上运行，但在复杂的金融质量质量质量标准的任务上，多芬拉格在涉及文本，表，图像和组合多模式推理的复杂财务QA任务上的精度比ChatGpt-4O（自由层）高出19个百分点。

Title: Uncovering Hidden Violent Tendencies in LLMs: A Demographic Analysis via Behavioral Vignettes

• Authors: Quintin Myers, Yanjun Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20822
• Pdf URL: https://arxiv.org/pdf/2506.20822
• Copy Paste: [[2506.20822]] Uncovering Hidden Violent Tendencies in LLMs: A Demographic Analysis via Behavioral Vignettes(https://arxiv.org/abs/2506.20822)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are increasingly proposed for detecting and responding to violent content online, yet their ability to reason about morally ambiguous, real-world scenarios remains underexamined. We present the first study to evaluate LLMs using a validated social science instrument designed to measure human response to everyday conflict, namely the Violent Behavior Vignette Questionnaire (VBVQ). To assess potential bias, we introduce persona-based prompting that varies race, age, and geographic identity within the United States. Six LLMs developed across different geopolitical and organizational contexts are evaluated under a unified zero-shot setting. Our study reveals two key findings: (1) LLMs surface-level text generation often diverges from their internal preference for violent responses; (2) their violent tendencies vary across demographics, frequently contradicting established findings in criminology, social science, and psychology.
• 摘要：大型语言模型（LLM）越来越多地提出用于在线检测和响应暴力内容，但他们对道德上模棱两可的现实情况进行推理的能力仍然没有散发出来。我们提出了第一个使用经过验证的社会科学工具评估LLM的研究，旨在衡量人类对日常冲突的反应，即暴力行为小插图问卷（VBVQ）。为了评估潜在的偏见，我们介绍了基于角色的提示，即在美国境内各种种族，年龄和地理身份。在不同的地缘政治环境和组织环境中开发的六个LLM在统一的零照片设置下进行了评估。我们的研究揭示了两个关键发现：（1）LLMS表面水平的文本生成通常与他们对暴力反应的内部偏爱不同； （2）他们的暴力倾向在人口统计学方面有所不同，经常与犯罪学，社会科学和心理学中既定的发现相矛盾。

Title: Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine

• Authors: Sebastian Joseph, Lily Chen, Barry Wei, Michael Mackert, Iain J. Marshall, Paul Pu Liang, Ramez Kouzy, Byron C. Wallace, Junyi Jessy Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.20876
• Pdf URL: https://arxiv.org/pdf/2506.20876
• Copy Paste: [[2506.20876]] Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine(https://arxiv.org/abs/2506.20876)
• Keywords: agent
• Abstract: Technological progress has led to concrete advancements in tasks that were regarded as challenging, such as automatic fact-checking. Interest in adopting these systems for public health and medicine has grown due to the high-stakes nature of medical decisions and challenges in critically appraising a vast and diverse medical literature. Evidence-based medicine connects to every individual, and yet the nature of it is highly technical, rendering the medical literacy of majority users inadequate to sufficiently navigate the domain. Such problems with medical communication ripens the ground for end-to-end fact-checking agents: check a claim against current medical literature and return with an evidence-backed verdict. And yet, such systems remain largely unused. To understand this, we present the first study examining how clinical experts verify real claims from social media by synthesizing medical evidence. In searching for this upper-bound, we reveal fundamental challenges in end-to-end fact-checking when applied to medicine: Difficulties connecting claims in the wild to scientific evidence in the form of clinical trials; ambiguities in underspecified claims mixed with mismatched intentions; and inherently subjective veracity labels. We argue that fact-checking should be approached and evaluated as an interactive communication problem, rather than an end-to-end process.
• 摘要：技术进步导致了被视为具有挑战性的任务的具体进步，例如自动事实检查。由于医疗决策的高风险性质和挑战在严格评估广泛而多样化的医学文献中，因此对采用这些系统进行公共卫生和医学的兴趣不断增长。循证医学与每个人相关，但其性质是技术性高的，这使多数用户的医学素养不足以足够驾驶该领域。医学通信的此类问题使终端事实检查代理人的基础融入了基础：检查针对当前医学文献的主张，并通过证据表面的判决返回。然而，这样的系统在很大程度上仍然没有使用。为了理解这一点，我们介绍了第一项研究，研究了临床专家如何通过综合医学证据来验证社交媒体的真实主张。在搜索这一上限时，我们揭示了应用于医学的端到端事实检查中的基本挑战：将野外主张与科学证据联系起来的困难；指定的主张中的歧义与不匹配的意图混合在一起；并固有的主观真实性标签。我们认为，应将事实检查作为交互式交流问题，而不是端到端的过程。

Title: Optimising Language Models for Downstream Tasks: A Post-Training Perspective

• Authors: Zhengyan Shi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.20917
• Pdf URL: https://arxiv.org/pdf/2506.20917
• Copy Paste: [[2506.20917]] Optimising Language Models for Downstream Tasks: A Post-Training Perspective(https://arxiv.org/abs/2506.20917)
• Keywords: language model
• Abstract: Language models (LMs) have demonstrated remarkable capabilities in NLP, yet adapting them efficiently and robustly to specific tasks remains challenging. As their scale and complexity grow, fine-tuning LMs on labelled data often underutilizes available unlabelled data, leads to overfitting on small task-specific sets, and imposes significant computational costs. These limitations hamper their application to the open-ended landscape of real-world language tasks. This thesis proposes a series of methods to better adapt LMs to downstream applications. First, we explore strategies for extracting task-relevant knowledge from unlabelled data, introducing a novel continued pre-training technique that outperforms state-of-the-art semi-supervised approaches. Next, we present a parameter-efficient fine-tuning method that substantially reduces memory and compute costs while maintaining competitive performance. We also introduce improved supervised fine-tuning methods that enable LMs to better follow instructions, especially when labelled data is scarce, enhancing their performance across a range of NLP tasks, including open-ended generation. Finally, we develop new evaluation methods and benchmarks, such as multi-hop spatial reasoning tasks, to assess LM capabilities and adaptation more comprehensively. Through extensive empirical studies across diverse NLP tasks, our results demonstrate that these approaches substantially improve LM robustness, efficiency, and generalization, making them more adaptable to a broad range of applications. These advances mark a significant step towards more robust and efficient LMs, bringing us closer to the goal of artificial general intelligence.
• 摘要：语言模型（LMS）在NLP中表现出了显着的功能，但是将其有效，强大地适应特定任务仍然具有挑战性。随着规模和复杂性的增长，标记数据上的微调LM通常不足以使可用的未标记数据不足，导致特定于特定于任务的小型设置过度拟合，并施加了巨大的计算成本。这些限制阻碍了它们在现实世界语言任务的开放式景观中的应用。本文提出了一系列方法，以更好地适应LMS到下游应用。首先，我们探讨了从未标记的数据中提取与任务相关的知识的策略，引入了一种新颖的持续培训技术，该技术表现优于最先进的半监督方法。接下来，我们提出了一种参数有效的微调方法，该方法可在保持竞争性能的同时大大降低内存和计算成本。我们还引入了改进的监督微调方法，使LMS能够更好地遵循说明，尤其是当标记的数据稀缺时，在包括开放式一代（包括开放式一代）的一系列NLP任务中提高了其性能。最后，我们开发了新的评估方法和基准，例如多跳空间推理任务，以更全面地评估LM功能和适应性。通过跨不同NLP任务的广泛的经验研究，我们的结果表明，这些方法可以大大提高LM的鲁棒性，效率和概括，从而使它们更适合广泛的应用。这些进步标志着朝着更强大，更有效的LMS迈出的重要一步，使我们更接近人工通用情报的目标。

Title: FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

• Authors: Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, Thomas Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.20920
• Pdf URL: https://arxiv.org/pdf/2506.20920
• Copy Paste: [[2506.20920]] FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language(https://arxiv.org/abs/2506.20920)
• Keywords: language model, llm
• Abstract: Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-training datasets has seen substantial recent progress, training performant multilingual LLMs remains a challenge, in large part due to the inherent difficulty of tailoring filtering and deduplication pipelines to a large number of languages. In this work, we introduce a new pre-training dataset curation pipeline based on FineWeb that can be automatically adapted to support any language. We extensively ablate our pipeline design choices on a set of nine diverse languages, guided by a set of meaningful and informative evaluation tasks that were chosen through a novel selection process based on measurable criteria. Ultimately, we show that our pipeline can be used to create non-English corpora that produce more performant models than prior datasets. We additionally introduce a straightforward and principled approach to rebalance datasets that takes into consideration both duplication count and quality, providing an additional performance uplift. Finally, we scale our pipeline to over 1000 languages using almost 100 Common Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document) multilingual dataset which we release along with our pipeline, training, and evaluation codebases.
• 摘要：培训前最先进的大语言模型（LLMS）需要大量的干净和多样化的文本数据。虽然大型英语预训练数据集的开放开发已经取得了很大的进步，但培训表现的多语言LLMS仍然是一个挑战，在很大程度上是由于对大量语言的固定过滤和重复解说管道的固有困难。在这项工作中，我们引入了基于FineWeb的新的预训练数据集策展管道，可以自动调整以支持任何语言。我们在一组九种不同的语言上，广泛地消除了管道设计选择，并以一系列有意义且信息丰富的评估任务为指导，这些任务是通过基于可测量标准的新型选择过程选择的。最终，我们证明我们的管道可用于创建与先前数据集更具性能模型的非英语语料库。另外，我们还引入了一种简单明了的原则方法来重新平衡数据集，该方法既考虑重复数量又有质量，从而提供了额外的性能提升。最后，我们使用近100个常见的爬网快照来将管道扩展到1000多种语言，以生产FineWeb2，这是一种新的20 trabyte（50亿个文档）多语言数据集，我们将与管道，培训和评估准则一起发布。

Title: Can Gradient Descent Simulate Prompting?

• Authors: Eric Zhang, Leshem Choshen, Jacob Andreas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.20989
• Pdf URL: https://arxiv.org/pdf/2506.20989
• Copy Paste: [[2506.20989]] Can Gradient Descent Simulate Prompting?(https://arxiv.org/abs/2506.20989)
• Keywords: language model, prompt
• Abstract: There are two primary ways of incorporating new information into a language model (LM): changing its prompt or changing its parameters, e.g. via fine-tuning. Parameter updates incur no long-term storage cost for model changes. However, for many model updates, prompting is significantly more effective: prompted models can generalize robustly from single examples and draw logical inferences that do not occur under standard fine-tuning. Can models be modified so that fine-tuning does emulate prompting? This paper describes a method for meta-training LMs such that gradient updates emulate the effects of conditioning on new information. Our approach uses tools from gradient-based meta-learning but uses an LM's own prompted predictions as targets, eliminating the need for ground-truth labels. Subsequent gradient descent training recovers some (and occasionally all) of prompted model performance -- showing improvement on the ``reversal curse'' tasks, and answering questions about text passages after a single gradient update. These results suggest that, with appropriate initialization, gradient descent can be surprisingly expressive. Our results suggest new avenues for long-context modeling and offer insight into the generalization capabilities of gradient-based learning.
• 摘要：将新信息纳入语言模型（LM）有两种主要方法：更改其提示或更改其参数，例如通过微调。参数更新不会产生模型更改的长期存储成本。但是，对于许多模型更新，提示更加有效：提示的模型可以从单个示例中概括地概述，并绘制标准微调下未发生的逻辑推断。是否可以修改模型以使微调效仿提示？本文介绍了一种元训练LMS的方法，使梯度更新模仿条件对新信息的影响。我们的方法使用基于梯度的元学习的工具，但使用LM自己的促进预测作为目标，从而消除了对地面真实标签的需求。随后的梯度下降训练恢复了一些（甚至偶尔）提示的模型性能 - 显示了``反转诅咒''任务的改进，并在一次梯度更新后回答有关文本段落的问题。这些结果表明，有了适当的初始化，梯度下降可能令人惊讶地表达。我们的结果提出了用于长篇文化建模的新途径，并提供了有关基于梯度学习的概括能力的洞察力。

Title: SAC: A Framework for Measuring and Inducing Personality Traits in LLMs with Dynamic Intensity Control

• Authors: Adithya Chittem, Aishna Shrivastava, Sai Tarun Pendela, Jagat Sesh Challa, Dhruv Kumar
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2506.20993
• Pdf URL: https://arxiv.org/pdf/2506.20993
• Copy Paste: [[2506.20993]] SAC: A Framework for Measuring and Inducing Personality Traits in LLMs with Dynamic Intensity Control(https://arxiv.org/abs/2506.20993)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have gained significant traction across a wide range of fields in recent years. There is also a growing expectation for them to display human-like personalities during interactions. To meet this expectation, numerous studies have proposed methods for modelling LLM personalities through psychometric evaluations. However, most existing models face two major limitations: they rely on the Big Five (OCEAN) framework, which only provides coarse personality dimensions, and they lack mechanisms for controlling trait intensity. In this paper, we address this gap by extending the Machine Personality Inventory (MPI), which originally used the Big Five model, to incorporate the 16 Personality Factor (16PF) model, allowing expressive control over sixteen distinct traits. We also developed a structured framework known as Specific Attribute Control (SAC) for evaluating and dynamically inducing trait intensity in LLMs. Our method introduces adjective-based semantic anchoring to guide trait intensity expression and leverages behavioural questions across five intensity factors: \textit{Frequency}, \textit{Depth}, \textit{Threshold}, \textit{Effort}, and \textit{Willingness}. Through experimentation, we find that modelling intensity as a continuous spectrum yields substantially more consistent and controllable personality expression compared to binary trait toggling. Moreover, we observe that changes in target trait intensity systematically influence closely related traits in psychologically coherent directions, suggesting that LLMs internalize multi-dimensional personality structures rather than treating traits in isolation. Our work opens new pathways for controlled and nuanced human-machine interactions in domains such as healthcare, education, and interviewing processes, bringing us one step closer to truly human-like social machines.
• 摘要：近年来，大型语言模型（LLM）在广泛的领域中获得了巨大的吸引力。他们在互动期间表现出类似人的个性的期望也越来越期望。为了满足这一期望，许多研究提出了通过心理测量评估对LLM个性进行建模的方法。但是，大多数现有模型面临两个主要局限性：它们依靠五（海洋）框架，该框架仅提供粗糙的人格维度，并且缺乏控制特质强度的机制。在本文中，我们通过扩展机器人格库存（MPI）（最初使用五巨头模型）来解决这一差距，以结合16个人格因素（16pf）模型，从而在16个不同的特征中允许表达性控制。我们还开发了一个称为特定属性控制（SAC）的结构化框架，用于评估和动态诱导LLMS中的性状强度。我们的方法介绍了基于形容词的语义锚定，以指导性状强度表达并利用五个强度因子的行为问题：\ textit {facying {facying}，\ textIt {depth}，\ textit {threshold}，\ textit {lockit {locor}和\ textit {wildit {willdings}。通过实验，我们发现与二进制特质切换相比，将建模强度作为连续光谱的表达更加一致，可控的人格表达。此外，我们观察到，目标性状强度的变化系统地影响了心理一致的方向上紧密相关的性状，这表明LLMS内部化了多维人格结构，而不是孤立地处理特征。我们的工作为在医疗保健，教育和访谈过程等领域中受控和细微的人机相互作用开辟了新的途径，使我们更接近真正的人类般的社会机器。

Title: Large Language Models Acing Chartered Accountancy

• Authors: Jatin Gupta, Akhil Sharma, Saransh Singhania, Mohammad Adnan, Sakshi Deo, Ali Imam Abidi, Keshav Gupta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21031
• Pdf URL: https://arxiv.org/pdf/2506.21031
• Copy Paste: [[2506.21031]] Large Language Models Acing Chartered Accountancy(https://arxiv.org/abs/2506.21031)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Advanced intelligent systems, particularly Large Language Models (LLMs), are significantly reshaping financial practices through advancements in Natural Language Processing (NLP). However, the extent to which these models effectively capture and apply domain-specific financial knowledge remains uncertain. Addressing a critical gap in the expansive Indian financial context, this paper introduces CA-Ben, a Chartered Accountancy benchmark specifically designed to evaluate the financial, legal, and quantitative reasoning capabilities of LLMs. CA-Ben comprises structured question-answer datasets derived from the rigorous examinations conducted by the Institute of Chartered Accountants of India (ICAI), spanning foundational, intermediate, and advanced CA curriculum stages. Six prominent LLMs i.e. GPT 4o, LLAMA 3.3 70B, LLAMA 3.1 405B, MISTRAL Large, Claude 3.5 Sonnet, and Microsoft Phi 4 were evaluated using standardized protocols. Results indicate variations in performance, with Claude 3.5 Sonnet and GPT-4o outperforming others, especially in conceptual and legal reasoning. Notable challenges emerged in numerical computations and legal interpretations. The findings emphasize the strengths and limitations of current LLMs, suggesting future improvements through hybrid reasoning and retrieval-augmented generation methods, particularly for quantitative analysis and accurate legal interpretation.
• 摘要：先进的智能系统，尤其是大型语言模型（LLMS），通过自然语言处理（NLP）的进步大大重塑财务实践。但是，这些模型有效捕获和应用特定领域的财务知识的程度尚不确定。本文介绍了在广阔的印度财务环境中的关键差距，介绍了CA-BEN，这是一种特许会计基准，专门旨在评估LLMS的财务，法律和定量推理能力。 Ca-ben包括由印度特许会计师研究所（ICAI）进行的严格考试得出的结构化问题解答数据集，跨越基础，中级和高级CA课程阶段。六个突出的LLM，即GPT 4O，Llama 3.3 70b，Llama 3.1 405b，Mistral大，Claude 3.5 SONNET和Microsoft Phi 4使用标准化方案进行了评估。结果表明性能的变化，克劳德3.5十四行诗和GPT-4O的表现优于其他人，尤其是在概念和法律推理方面。在数值计算和法律解释中出现了显着的挑战。这些发现强调了当前LLM的优势和局限性，这表明通过混合推理和检索增强的生成方法提高了未来的改进，尤其是用于定量分析和准确的法律解释。

Title: MT2-CSD: A New Dataset and Multi-Semantic Knowledge Fusion Method for Conversational Stance Detection

• Authors: Fuqiang Niu, Genan Dai, Yisha Lu, Jiayu Liao, Xiang Li, Hu Huang, Bowen Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21053
• Pdf URL: https://arxiv.org/pdf/2506.21053
• Copy Paste: [[2506.21053]] MT2-CSD: A New Dataset and Multi-Semantic Knowledge Fusion Method for Conversational Stance Detection(https://arxiv.org/abs/2506.21053)
• Keywords: language model, llm
• Abstract: In the realm of contemporary social media, automatic stance detection is pivotal for opinion mining, as it synthesizes and examines user perspectives on contentious topics to uncover prevailing trends and sentiments. Traditional stance detection research often targets individual instances, thereby limiting its capacity to model multi-party discussions typical in real social media scenarios. This shortcoming largely stems from the scarcity of datasets that authentically capture the dynamics of social media interactions, hindering advancements in conversational stance detection. In this paper, we introduce MT2-CSD, a comprehensive dataset for multi-target, multi-turn conversational stance detection. To the best of our knowledge, MT2-CSD is the largest dataset available for this purpose, comprising 24,457 annotated instances and exhibiting the greatest conversational depth, thereby presenting new challenges for stance detection. To address these challenges, we propose the Large Language model enhanced Conversational Relational Attention Network (LLM-CRAN), which exploits the reasoning capabilities of LLMs to improve conversational understanding. We conduct extensive experiments to evaluate the efficacy of LLM-CRAN on the MT2-CSD dataset. The experimental results indicate that LLM-CRAN significantly outperforms strong baseline models in the task of conversational stance detection.
• 摘要：在当代社交媒体的领域中，自动立场检测对于挖掘而言至关重要，因为它综合并研究了用户对有争议主题的观点，以发现盛行的趋势和观点。传统的立场检测研究通常针对个别实例，从而限制了其在真实社交媒体场景中典型的多方讨论建模的能力。这一缺点很大程度上源于数据集的稀缺性，这些数据集可真正捕捉社交媒体互动的动态，从而阻碍了对话立场检测的进步。在本文中，我们介绍了MT2-CSD，这是一种用于多目标，多转向对话的立场检测的综合数据集。据我们所知，MT2-CSD是用于此目的的最大数据集，包括24,457个注释的实例，并展示了最大的对话深度，从而为立场检测带来了新的挑战。为了应对这些挑战，我们提出了大型语言模型增强了对话关系关注网络（LLM-CRAN），该网络利用了LLM的推理能力来提高对话的理解。我们进行了广泛的实验，以评估LLM-CRAN对MT2-CSD数据集的功效。实验结果表明，LLM-CRAN在对话姿势检测的任务中明显优于强大的基线模型。

Title: ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry

• Authors: Qinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi, Liangzhong Guo, Yuan Wang, Wei Wang, Yunshi Lan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21098
• Pdf URL: https://arxiv.org/pdf/2506.21098
• Copy Paste: [[2506.21098]] ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry(https://arxiv.org/abs/2506.21098)
• Keywords: retrieval-augmented generation
• Abstract: Community Question Answering (CQA) platforms can be deemed as important knowledge bases in community, but effectively leveraging historical interactions and domain knowledge in real-time remains a challenge. Existing methods often underutilize external knowledge, fail to incorporate dynamic historical QA context, or lack memory mechanisms suited for industrial deployment. We propose ComRAG, a retrieval-augmented generation framework for real-time industrial CQA that integrates static knowledge with dynamic historical QA pairs via a centroid-based memory mechanism designed for retrieval, generation, and efficient storage. Evaluated on three industrial CQA datasets, ComRAG consistently outperforms all baselines--achieving up to 25.9% improvement in vector similarity, reducing latency by 8.7% to 23.3%, and lowering chunk growth from 20.23% to 2.06% over iterations.
• 摘要：社区问题回答（CQA）平台可以被视为社区中的重要知识库，但是在实时利用历史互动和领域知识的有效利用仍然是一个挑战。现有的方法通常不足以外部知识，无法纳入动态的历史质量检查环境，或者缺乏适合工业部署的记忆机制。我们提出了Comrag，这是一个实时工业CQA的检索型生成框架，该框架通过基于质心的存储机制将静态知识与动态历史QA对整合在一起，该机制旨在检索，生成和有效的存储。对三个工业CQA数据集进行了评估，Comrag始终优于所有基准，其矢量相似性提高了25.9％，将延迟降低到8.7％至23.3％，并从20.23％的块增长到迭代率的2.23％降低到2.06％。

Title: Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

• Authors: Xiaoshuang Ji, Zhendong Zhao, Xiaojun Chen, Xin Zhao, Zeyao Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21119
• Pdf URL: https://arxiv.org/pdf/2506.21119
• Copy Paste: [[2506.21119]] Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models(https://arxiv.org/abs/2506.21119)
• Keywords: language model
• Abstract: Fine-tuning is a promising technique for leveraging Transformer-based language models in downstream tasks. As model sizes continue to grow, updating all model parameters becomes increasingly costly. Parameter-efficient fine-tuning methods effectively address this issue by selectively updating a small subset of parameters. However, fine-tuning and most existing parameter-efficient fine-tuning methods require updating the same number of parameters as the initial size, ignoring the unequal contribution across Transformer blocks and leading to extremely inefficient allocation of computing resources. In this paper, we propose Progtuning, the novel fine-tuning framework combined with progressive learning for Transformer-based language models. Specifically, Progtuning progressively reduces the number of updated transformer blocks based on the contribution. Remarkably, Progtuning optimizes resource allocation and reduces the number of updated parameters by approximately 25\%, while still maintaining competitive performance. And it also exhibits high adaptability with parameter-efficient fine-tuning methods, demonstrating excellent performance across various adaptation scenarios.
• 摘要：微调是一种在下游任务中利用基于变压器的语言模型的有前途的技术。随着模型大小的不断增长，更新所有模型参数的成本越来越高。参数有效的微调方法通过选择性更新一小部分参数来有效地解决此问题。但是，微调和大多数现有参数有效的微调方法需要更新与初始大小相同数量的参数，而忽略了跨变压器块的不相等贡献，而导致计算资源的极低分配。在本文中，我们提出了ProgTuning，这是一个新颖的微调框架，结合了基于变压器的语言模型的渐进学习。具体而言，基于贡献，逐步逐步减少了更新的变压器块的数量。值得注意的是，ProgTuning优化了资源分配，并将更新参数的数量减少约25 \％，同时仍保持竞争性能。它还使用参数有效的微调方法表现出较高的适应性，在各种适应方案中表现出卓越的性能。

Title: Compressed and Smooth Latent Space for Text Diffusion Modeling

• Authors: Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, Dmitry Vetrov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21170
• Pdf URL: https://arxiv.org/pdf/2506.21170
• Copy Paste: [[2506.21170]] Compressed and Smooth Latent Space for Text Diffusion Modeling(https://arxiv.org/abs/2506.21170)
• Keywords: language model
• Abstract: Autoregressive language models dominate modern text generation, yet their sequential nature introduces fundamental limitations: decoding is slow, and maintaining global coherence remains challenging. Diffusion models offer a promising alternative by enabling parallel generation and flexible control; however, their application to text generation is hindered by the high dimensionality of token-level representations. We introduce Cosmos, a novel approach to text generation that operates entirely in a compressed, smooth latent space tailored specifically for diffusion. This space is learned using an autoencoder trained simultaneously for token-level reconstruction and alignment with frozen activations from a pretrained language encoder, providing robust semantic grounding and enabling effective perturbation-based augmentations. Empirically, we demonstrate that text representations can be compressed by $8\times$ while maintaining generation quality comparable to token-level diffusion models. Furthermore, increasing the latent sequence length allows Cosmos to surpass both diffusion-based and autoregressive baselines. We evaluate Cosmos on four diverse generative tasks including story generation, question generation, summarization, and detoxification and compare it with various generative paradigms. Cosmos achieves comparable or superior generation quality while offering more than $2\times$ faster inference.
• 摘要：自回归语言模型主导了现代文本生成，但是它们的顺序性质引入了基本局限性：解码很慢，并且保持全球连贯性仍然具有挑战性。扩散模型通过实现平行生成和柔性控制提供了一种有希望的替代方法。但是，它们对文本生成的应用受到令牌级表示的高维度的阻碍。我们介绍了Cosmos，这是一种新颖的文本生成方法，完全在专门针对扩散的压缩，光滑的潜在空间中运行。使用同时训练的自动编码器，以对代币级别的重建和对齐方式进行训练，并与预审前语言编码器的冷冻激活相结合，从而提供了强大的语义接地，并实现了有效的基于扰动的增强。从经验上讲，我们证明文本表示形式可以被$ 8 \ times $压缩，同时保持与令牌级扩散模型相当的发电质量。此外，增加潜在序列长度使宇宙可以超过基于扩散的基准和自回归基准。我们评估宇宙的四种不同生成任务，包括故事产生，问题产生，摘要和排毒，并将其与各种生成范式进行比较。 Cosmos可以达到可比或优越的发电质量，同时提供超过2美元的$ \ times $ $。

Title: Prompt-Guided Turn-Taking Prediction

• Authors: Koji Inoue, Mikey Elmers, Yahui Fu, Zi Haur Pang, Divesh Lala, Keiko Ochi, Tatsuya Kawahara
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2506.21191
• Pdf URL: https://arxiv.org/pdf/2506.21191
• Copy Paste: [[2506.21191]] Prompt-Guided Turn-Taking Prediction(https://arxiv.org/abs/2506.21191)
• Keywords: language model, llm, prompt
• Abstract: Turn-taking prediction models are essential components in spoken dialogue systems and conversational robots. Recent approaches leverage transformer-based architectures to predict speech activity continuously and in real-time. In this study, we propose a novel model that enables turn-taking prediction to be dynamically controlled via textual prompts. This approach allows intuitive and explicit control through instructions such as "faster" or "calmer" adapting dynamically to conversational partners and contexts. The proposed model builds upon a transformer-based voice activity projection (VAP) model, incorporating textual prompt embeddings into both channel-wise transformers and a cross-channel transformer. We evaluated the feasibility of our approach using over 950 hours of human-human spoken dialogue data. Since textual prompt data for the proposed approach was not available in existing datasets, we utilized a large language model (LLM) to generate synthetic prompt sentences. Experimental results demonstrated that the proposed model improved prediction accuracy and effectively varied turn-taking timing behaviors according to the textual prompts.
• 摘要：转弯预测模型是口语对话系统和对话机器人中的重要组成部分。最近的方法利用基于变压器的架构进行实时和实时预测语音活动。在这项研究中，我们提出了一个新型模型，该模型可以通过文本提示动态控制转弯预测。这种方法允许通过“更快”或“平静”的说明进行直观和明确的控制，以动态地适应对话伙伴和上下文。所提出的模型建立在基于变压器的语音活动投影（VAP）模型上，将文本提示嵌入到频道的变压器和跨通道变压器中。我们使用超过950个小时的人类口语对话数据评估了方法的可行性。由于现有数据集中没有提出方法的文本提示数据，因此我们使用大型语言模型（LLM）来生成合成提示句子。实验结果表明，根据文本提示，提出的模型提高了预测准确性，并有效地改变了转弯时序行为。

Title: Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval

• Authors: Yongchan Chun, Minhyuk Kim, Dongjun Kim, Chanjun Park, Heuiseok Lim
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2506.21222
• Pdf URL: https://arxiv.org/pdf/2506.21222
• Copy Paste: [[2506.21222]] Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval(https://arxiv.org/abs/2506.21222)
• Keywords: language model, llm, prompt
• Abstract: Automatic Term Extraction (ATE) identifies domain-specific expressions that are crucial for downstream tasks such as machine translation and information retrieval. Although large language models (LLMs) have significantly advanced various NLP tasks, their potential for ATE has scarcely been examined. We propose a retrieval-based prompting strategy that, in the few-shot setting, selects demonstrations according to \emph{syntactic} rather than semantic similarity. This syntactic retrieval method is domain-agnostic and provides more reliable guidance for capturing term boundaries. We evaluate the approach in both in-domain and cross-domain settings, analyzing how lexical overlap between the query sentence and its retrieved examples affects performance. Experiments on three specialized ATE benchmarks show that syntactic retrieval improves F1-score. These findings highlight the importance of syntactic cues when adapting LLMs to terminology-extraction tasks.
• 摘要：自动术语提取（ATE）标识了针对下游任务（例如机器翻译和信息检索）至关重要的域特异性表达式。尽管大型语言模型（LLMS）已经显着提出了各种NLP任务，但几乎没有检查它们的饮食潜力。我们提出了一种基于检索的提示策略，在几个弹奏设置中，根据\ emph {stantactic}选择了演示，而不是语义相似性。这种句法检索方法是域 - 不可思议的，并为捕获期限范围提供了更可靠的指导。我们评估了内域和跨域设置中的方法，分析了查询句子及其检索的示例之间的词汇叠加如何影响性能。在三个专业ATE基准测试的实验表明，句法检索可以改善F1得分。这些发现突出了句法提示在将LLMS适应术语 - 算法任务时的重要性。

Title: Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents

• Authors: Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21252
• Pdf URL: https://arxiv.org/pdf/2506.21252
• Copy Paste: [[2506.21252]] Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents(https://arxiv.org/abs/2506.21252)
• Keywords: language model, llm, agent
• Abstract: As Multimodal Large Language Models (MLLMs) advance, multimodal agents show promise in real-world tasks like web navigation and embodied intelligence. However, due to limitations in a lack of external feedback, these agents struggle with self-correction and generalization. A promising approach is to use reward models as external feedback, but there is no clear on how to select reward models for agents. Thus, there is an urgent need to build a reward bench targeted at agents. To address these challenges, we propose Agent-RewardBench, a benchmark designed to evaluate reward modeling ability in MLLMs. The benchmark is characterized by three key features: (1) Multiple dimensions and real-world agent scenarios evaluation. It covers perception, planning, and safety with 7 scenarios; (2) Step-level reward evaluation. It allows for the assessment of agent capabilities at the individual steps of a task, providing a more granular view of performance during the planning process; and (3) Appropriately difficulty and high-quality. We carefully sample from 10 diverse models, difficulty control to maintain task challenges, and manual verification to ensure the integrity of the data. Experiments demonstrate that even state-of-the-art multimodal models show limited performance, highlighting the need for specialized training in agent reward modeling. Code is available at github.
• 摘要：随着多模式大语模型（MLLM）的发展，多模式代理在Web导航和具体智能等现实世界任务中显示出希望。但是，由于缺乏外部反馈的局限性，这些代理商在自我纠正和概括方面挣扎。一种有希望的方法是将奖励模型用作外部反馈，但尚不清楚如何为代理选择奖励模型。因此，迫切需要建立针对代理商的奖励席。为了应对这些挑战，我们提出了Agent-Rewardbench，这是一种旨在评估MLLM中奖励建模能力的基准。基准测试的特征是三个关键特征：（1）多个维度和现实世界代理方案评估。它涵盖了7种情况的感知，计划和安全； （2）步进奖励评估。它允许在任务的各个步骤中评估代理能力，从而在计划过程中对性能的更加详尽的看法； （3）适当的困难和高质量。我们仔细采样了10种不同模型，难以控制任务挑战的难度以及手动验证以确保数据的完整性。实验表明，即使是最先进的多模式模型也显示出有限的性能，强调了对代理奖励建模的专业培训的需求。代码可在GitHub上找到。

Title: Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems?

• Authors: Andrea McGlinchey, Peter J Barclay
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21274
• Pdf URL: https://arxiv.org/pdf/2506.21274
• Copy Paste: [[2506.21274]] Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems?(https://arxiv.org/abs/2506.21274)
• Keywords: language model, gpt, llm
• Abstract: Large language models can produce convincing "fake text" in domains such as academic writing, product reviews, and political news. Many approaches have been investigated for the detection of artificially generated text. While this may seem to presage an endless "arms race", we note that newer LLMs use ever more parameters, training data, and energy, while relatively simple classifiers demonstrate a good level of detection accuracy with modest resources. To approach the question of whether the models' ability to beat the detectors may therefore reach a plateau, we examine the ability of statistical classifiers to identify "fake text" in the style of classical detective fiction. Over a 0.5 version increase, we found that Gemini showed an increased ability to generate deceptive text, while GPT did not. This suggests that reliable detection of fake text may remain feasible even for ever-larger models, though new model architectures may improve their deceptiveness
• 摘要：大型语言模型可以在学术写作，产品评论和政治新闻等领域中产生令人信服的“假文本”。已经研究了许多方法以检测人为生成的文本。尽管这似乎可以预示着无尽的“军备竞赛”，但我们注意到较新的LLM使用更多的参数，培训数据和能量，而相对简单的分类器则使用适度的资源表现出良好的检测准确性。为了解决模型击败探测器的能力是否可以达到平稳的问题，我们研究了统计分类器以经典侦探小说风格识别“假文本”的能力。在增加0.5版本的情况下，我们发现双子座表现出产生欺骗性文本的能力，而GPT则没有。这表明，即使对于不断的模型，对假文本的可靠检测也可能仍然可行，尽管新的模型体系结构可能会提高其欺骗性

Title: Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning

• Authors: Xin Xu, Tianhao Chen, Fan Zhang, Wanlong Liu, Pengxiang Li, Ajay Kumar Jaiswal, Yuchen Yan, Jishan Hu, Yang Wang, Hao Chen, Shiwei Liu, Shizhe Diao, Can Yang, Lu Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21285
• Pdf URL: https://arxiv.org/pdf/2506.21285
• Copy Paste: [[2506.21285]] Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning(https://arxiv.org/abs/2506.21285)
• Keywords: language model, llm
• Abstract: While slow-thinking large language models (LLMs) exhibit reflection-like reasoning, commonly referred to as the "aha moment:, their ability to generate informative critiques and refine prior solutions remains limited. In this paper, we introduce Double-Checker, a principled framework designed to enhance the reasoning capabilities of slow-thinking LLMs by fostering explicit self-critique and iterative refinement of their previous solutions. By fine-tuning on our curated 1,730 self-critical instances, Double-Checker empowers long-CoT LLMs to iteratively critique and refine their outputs during inference until they evaluate their solutions as correct under self-generated critiques. We validate the efficacy of Double-Checker across a comprehensive suite of reasoning benchmarks, demonstrating that iterative self-critique significantly enhances the reasoning capabilities of long-CoT LLMs. Notably, our Double-Checker increases the pass@1 performance on challenging AIME benchmarks from 4.4% to 18.2% compared to the original long-CoT LLMs. These results highlight a promising direction for developing more trustworthy and effective LLMs capable of structured self-critique.
• 摘要：尽管缓慢思考大型语言模型（LLM）表现出反射的推理，通常称为“ aha arminm：它们产生信息的能力，他们的能力仍然有限。在本文中，我们在本文中介绍了双检查器，我们引入了旨在通过稳定的求解能力来提高稳定的求解能力的原则性框架。通过对我们精心策划的1,730个自我批评实例进行微调，双重检查器可以使长期以来的llms能够迭代的批评，并在推理过程中完善其输出，直到他们在自我生成的批评下评估其解决方案，我们对综合性的综合愿景进行了自我验证。长期以来，我们的双重检查者在挑战性AIME基准方面的通过@1的表现从4.4％提高到了18.2％，而这些结果与原始的Long-Cot LLM相比，突出了一个有前途的方向。

Title: Small Encoders Can Rival Large Decoders in Detecting Groundedness

• Authors: Istabrak Abbes, Gabriele Prato, Quentin Fournier, Fernando Rodriguez, Alaa Boukhary, Adam Elwood, Sarath Chandar
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.21288
• Pdf URL: https://arxiv.org/pdf/2506.21288
• Copy Paste: [[2506.21288]] Small Encoders Can Rival Large Decoders in Detecting Groundedness(https://arxiv.org/abs/2506.21288)
• Keywords: language model, gpt, llm
• Abstract: Augmenting large language models (LLMs) with external context significantly improves their performance in natural language processing (NLP) tasks. However, LLMs struggle to answer queries reliably when the provided context lacks information, often resorting to ungrounded speculation or internal knowledge. Groundedness - generating responses strictly supported by the context - is essential for ensuring factual consistency and trustworthiness. This study focuses on detecting whether a given query is grounded in a document provided in context before the costly answer generation by LLMs. Such a detection mechanism can significantly reduce both inference time and resource consumption. We show that lightweight, task specific encoder models such as RoBERTa and NomicBERT, fine-tuned on curated datasets, can achieve accuracy comparable to state-of-the-art LLMs, such as Llama3 8B and GPT4o, in groundedness detection while reducing inference latency by orders of magnitude. The code is available at : this https URL
• 摘要：增强具有外部环境的大型语言模型（LLMS）可显着提高其在自然语言处理（NLP）任务中的性能。但是，当提供的上下文缺乏信息时，LLM努力可靠地回答查询，通常会诉诸于未接地的猜测或内部知识。扎根 - 产生严格支持背景的反应 - 对于确保事实一致性和可信赖性至关重要。这项研究的重点是检测给定查询是否基于LLM昂贵的答案的上下文中提供的文档。这种检测机制可以显着减少推理时间和资源消耗。我们表明，在策划数据集上进行了微调的轻巧，特定于任务的编码模型，可以实现与最先进的LLM相当的准确性，例如Llama3 8B和GPT4O，在接地检测中以较小的阶数降低推理延迟。代码可用：此HTTPS URL

Title: Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models

• Authors: Bram Willemsen, Gabriel Skantze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21294
• Pdf URL: https://arxiv.org/pdf/2506.21294
• Copy Paste: [[2506.21294]] Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models(https://arxiv.org/abs/2506.21294)
• Keywords: language model, llm
• Abstract: In this paper, we explore the use of a text-only, autoregressive language modeling approach for the extraction of referring expressions from visually grounded dialogue. More specifically, the aim is to investigate the extent to which the linguistic context alone can inform the detection of mentions that have a (visually perceivable) referent in the visual context of the conversation. To this end, we adapt a pretrained large language model (LLM) to perform a relatively course-grained annotation of mention spans in unfolding conversations by demarcating mention span boundaries in text via next-token prediction. Our findings indicate that even when using a moderately sized LLM, relatively small datasets, and parameter-efficient fine-tuning, a text-only approach can be effective, highlighting the relative importance of the linguistic context for this task. Nevertheless, we argue that the task represents an inherently multimodal problem and discuss limitations fundamental to unimodal approaches.
• 摘要：在本文中，我们探讨了一种仅使用文本，自回归的语言建模方法来提取从视觉接地对话中提取表达式的使用。更具体地说，目的是研究单独语言环境可以在多大程度上告知在对话视觉上下文中具有（视觉可感知的）参考文献的提及。为此，我们适应了经过验证的大语言模型（LLM），以通过下一步的预测在文本中划分文本中的跨度界限，对跨越对话进行相对元素的注释。我们的发现表明，即使使用中等大小的LLM，相对较小的数据集和参数有效的微调，仅文本方法也可以有效，从而强调了语言上下文对本任务的相对重要性。然而，我们认为该任务代表了一个固有的多模式问题，并讨论了对单峰方法基本的局限性。

Title: Structuralist Approach to AI Literary Criticism: Leveraging Greimas Semiotic Square for Large Language Models

• Authors: Fangzhou Dong, Yifan Zeng, Yingpeng Sang, Hong Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21360
• Pdf URL: https://arxiv.org/pdf/2506.21360
• Copy Paste: [[2506.21360]] Structuralist Approach to AI Literary Criticism: Leveraging Greimas Semiotic Square for Large Language Models(https://arxiv.org/abs/2506.21360)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel in understanding and generating text but struggle with providing professional literary criticism for works with profound thoughts and complex narratives. This paper proposes GLASS (Greimas Literary Analysis via Semiotic Square), a structured analytical framework based on Greimas Semiotic Square (GSS), to enhance LLMs' ability to conduct in-depth literary analysis. GLASS facilitates the rapid dissection of narrative structures and deep meanings in narrative works. We propose the first dataset for GSS-based literary criticism, featuring detailed analyses of 48 works. Then we propose quantitative metrics for GSS-based literary criticism using the LLM-as-a-judge paradigm. Our framework's results, compared with expert criticism across multiple works and LLMs, show high performance. Finally, we applied GLASS to 39 classic works, producing original and high-quality analyses that address existing research gaps. This research provides an AI-based tool for literary research and education, offering insights into the cognitive mechanisms underlying literary engagement.
• 摘要：大型语言模型（LLM）在理解和生成文本方面表现出色，但努力为具有深刻思想和复杂叙事的作品提供专业的文学批评。本文提出了基于Greimas符号平方（GSS）的结构化分析框架（GSS）的玻璃（GREIMAS文学分析），以增强LLMS进行深入文学分析的能力。玻璃促进了叙事结构的快速解剖和叙事作品中的深层含义。我们提出了第一个用于基于GSS的文学批评的数据集，其中包含48幅作品的详细分析。然后，我们提出了使用LLM-AS-A-A-Gudge范式进行基于GSS的文学批评的定量指标。与多个作品和LLM的专家批评相比，我们的框架的结果显示出高性能。最后，我们将玻璃应用于39家经典作品，并产生了解决现有研究差距的原始和高质量分析。这项研究为文学研究和教育提供了一种基于AI的工具，为文学参与潜在的认知机制提供了见解。

Title: Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation

• Authors: Guanting Dong, Xiaoxi Li, Yuyao Zhang, Mengjie Deng
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2506.21384
• Pdf URL: https://arxiv.org/pdf/2506.21384
• Copy Paste: [[2506.21384]] Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation(https://arxiv.org/abs/2506.21384)
• Keywords: language model, llm, prompt, retrieval-augmented generation, chain-of-thought
• Abstract: Real-world live retrieval-augmented generation (RAG) systems face significant challenges when processing user queries that are often noisy, ambiguous, and contain multiple intents. While RAG enhances large language models (LLMs) with external knowledge, current systems typically struggle with such complex inputs, as they are often trained or evaluated on cleaner data. This paper introduces Omni-RAG, a novel framework designed to improve the robustness and effectiveness of RAG systems in live, open-domain settings. Omni-RAG employs LLM-assisted query understanding to preprocess user inputs through three key modules: (1) Deep Query Understanding and Decomposition, which utilizes LLMs with tailored prompts to denoise queries (e.g., correcting spelling errors) and decompose multi-intent queries into structured sub-queries; (2) Intent-Aware Knowledge Retrieval, which performs retrieval for each sub-query from a corpus (i.e., FineWeb using OpenSearch) and aggregates the results; and (3) Reranking and Generation, where a reranker (i.e., BGE) refines document selection before a final response is generated by an LLM (i.e., Falcon-10B) using a chain-of-thought prompt. Omni-RAG aims to bridge the gap between current RAG capabilities and the demands of real-world applications, such as those highlighted by the SIGIR 2025 LiveRAG Challenge, by robustly handling complex and noisy queries.
• 摘要：当处理通常嘈杂，模棱两可且包含多种意图的用户查询时，现实世界中的实时检索生成（RAG）系统在处理用户查询时面临重大挑战。尽管抹布可以增强具有外部知识的大型语言模型（LLM），但当前系统通常会在这种复杂的输入方面挣扎，因为它们经常在清洁数据上接受培训或评估。本文介绍了Omni-Rag，这是一个新颖的框架，旨在提高现场开放域设置中抹布系统的鲁棒性和有效性。 Omni-rag采用LLM辅助查询理解来通过三个关键模块进行预处理用户输入：（1）深度查询理解和分解，它利用带有量身定制提示的LLM来denoise查询（例如，纠正拼写错误），并将多Intintent Queries分解为结构性的Quereries to Cluctured subsececeries; （2）意图感知的知识检索，从语料库（即使用OpenSearch的FineWeb）为每个子问题执行检索并汇总结果； （3）重读和生成，其中reranker（即bge）在使用经过经过经过经过经过经验的提示的LLM（即Falcon-10b）生成最终响应之前先完善了文档选择。 Omni-Rag旨在弥合当前RAG功能与现实世界应用的需求之间的差距，例如Sigir 2025 Liverag Challenge强调的，通过强大的处理复合物和嘈杂的查询。

Title: Domain Knowledge-Enhanced LLMs for Fraud and Concept Drift Detection

• Authors: Ali Şenol, Garima Agrawal, Huan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21443
• Pdf URL: https://arxiv.org/pdf/2506.21443
• Copy Paste: [[2506.21443]] Domain Knowledge-Enhanced LLMs for Fraud and Concept Drift Detection(https://arxiv.org/abs/2506.21443)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Detecting deceptive conversations on dynamic platforms is increasingly difficult due to evolving language patterns and Concept Drift (CD)\-i.e., semantic or topical shifts that alter the context or intent of interactions over time. These shifts can obscure malicious intent or mimic normal dialogue, making accurate classification challenging. While Large Language Models (LLMs) show strong performance in natural language tasks, they often struggle with contextual ambiguity and hallucinations in risk\-sensitive scenarios. To address these challenges, we present a Domain Knowledge (DK)\-Enhanced LLM framework that integrates pretrained LLMs with structured, task\-specific insights to perform fraud and concept drift detection. The proposed architecture consists of three main components: (1) a DK\-LLM module to detect fake or deceptive conversations; (2) a drift detection unit (OCDD) to determine whether a semantic shift has occurred; and (3) a second DK\-LLM module to classify the drift as either benign or fraudulent. We first validate the value of domain knowledge using a fake review dataset and then apply our full framework to SEConvo, a multiturn dialogue dataset that includes various types of fraud and spam attacks. Results show that our system detects fake conversations with high accuracy and effectively classifies the nature of drift. Guided by structured prompts, the LLaMA\-based implementation achieves 98\% classification accuracy. Comparative studies against zero\-shot baselines demonstrate that incorporating domain knowledge and drift awareness significantly improves performance, interpretability, and robustness in high\-stakes NLP applications.
• 摘要：由于不断发展的语言模式和概念漂移（cd）\ - 即，语义或局部变化会改变随着时间的推移的上下文或意图，因此在动态平台上检测欺骗性对话越来越困难。这些转变可能会掩盖恶意意图或模仿正常对话，从而使准确的分类具有挑战性。尽管大型语言模型（LLMS）在自然语言任务中表现出强劲的表现，但它们通常在风险\敏感的情况下与上下文歧义和幻觉斗争。为了应对这些挑战，我们提出了一个域知识（DK）\ - 增强的LLM框架，该框架将经过验证的LLM与结构化的任务\特定的见解集成在一起，以执行欺诈和概念漂移检测。所提出的架构由三个主要组成部分组成：（1）DK \ -LLM模块检测假或欺骗性对话； （2）漂移检测单元（OCDD），以确定是否发生了语义移动； （3）第二个DK \ -llm模块将漂移分类为良性或欺诈。我们首先使用虚假评论数据集验证域知识的价值，然后将我们的完整框架应用于Seconvo，Seconvo是一个包括各种类型的欺诈和垃圾邮件攻击的多努力对话数据集。结果表明，我们的系统以高精度检测到虚假的对话，并有效地对漂移的性质进行了分类。在结构化提示的指导下，基于Llama \的实现实现了98 \％的分类精度。针对零\ shot基线的比较研究表明，融入域知识和漂移意识可显着提高高\ stakes NLP应用中的性能，可解释性和鲁棒性。

Title: Text2Cypher Across Languages: Evaluating Foundational Models Beyond English

• Authors: Makbule Gulcin Ozsoy, William Tai
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2506.21445
• Pdf URL: https://arxiv.org/pdf/2506.21445
• Copy Paste: [[2506.21445]] Text2Cypher Across Languages: Evaluating Foundational Models Beyond English(https://arxiv.org/abs/2506.21445)
• Keywords: language model, llm, prompt
• Abstract: Recent advances in large language models have enabled natural language interfaces that translate user questions into database queries, such as Text2SQL, Text2SPARQL, and Text2Cypher. While these interfaces enhance database accessibility, most research today focuses solely on English, with limited evaluation in other languages. This paper investigates the performance of foundational LLMs on the Text2Cypher task across multiple languages. We create and release a multilingual test set by translating English questions into Spanish and Turkish while preserving the original Cypher queries, enabling fair cross-lingual comparison. We evaluate multiple foundational models using standardized prompts and metrics. Our results show a consistent performance pattern: highest on English, then Spanish, and lowest on Turkish. We attribute this to differences in training data availability and linguistic characteristics. Additionally, we explore the impact of translating task prompts into Spanish and Turkish. Results show little to no change in evaluation metrics, suggesting prompt translation has minor impact. Our findings highlight the need for more inclusive evaluation and development in multilingual query generation. Future work includes schema localization and fine-tuning across diverse languages.
• 摘要：大型语言模型的最新进展使自然语言接口将用户问题转化为数据库查询，例如Text2SQL，Text2Sparql和Text2Cypher。尽管这些接口增强了数据库可访问性，但当今的大多数研究都只关注英语，而其他语言的评估有限。本文研究了基础LLM在多种语言上的Text2Cypher任务上的性能。我们通过将英语问题转换为西班牙语和土耳其语，同时保存原始的Cypher查询，从而创建和发布多种语言测试集，从而实现公平的跨语义比较。我们使用标准化的提示和指标评估多个基础模型。我们的结果显示出稳定的性能模式：在英语，西班牙语和土耳其最低的情况下最高。我们将其归因于培训数据可用性和语言特征的差异。此外，我们探讨了将任务提示转化为西班牙和土耳其语的影响。结果显示，评估指标几乎没有变化，这表明及时翻译的影响很小。我们的发现强调了在多语言查询一代中需要更具包容性评估和开发的需求。未来的工作包括跨不同语言的模式本地化和微调。

Title: Aligning Spoken Dialogue Models from User Interactions

• Authors: Anne Wu, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2506.21463
• Pdf URL: https://arxiv.org/pdf/2506.21463
• Copy Paste: [[2506.21463]] Aligning Spoken Dialogue Models from User Interactions(https://arxiv.org/abs/2506.21463)
• Keywords: language model
• Abstract: We propose a novel preference alignment framework for improving spoken dialogue models on real-time conversations from user interactions. Current preference learning methods primarily focus on text-based language models, and are not directly suited to the complexities of real-time speech interactions, with richer dynamics (e.g. interruption, interjection) and no explicit segmentation between speaker this http URL create a large-scale dataset of more than 150,000 preference pairs from raw multi-turn speech conversations, annotated with AI feedback, to cover preferences over both linguistic content and temporal context variations. We leverage offline alignment methods to finetune a full-duplex autoregressive speech-to-speech model. Extensive experiments demonstrate that feedback on generic conversations can be consistently effective in improving spoken dialogue models to produce more factual, safer and more contextually aligned interactions. We deploy the finetuned model and conduct holistic human evaluations to assess the impact beyond single-turn conversations. Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.
• 摘要：我们提出了一个新颖的偏好对齐框架，以改善用户交互中实时对话的口语对话模型。 Current preference learning methods primarily focus on text-based language models, and are not directly suited to the complexities of real-time speech interactions, with richer dynamics (e.g. interruption, interjection) and no explicit segmentation between speaker this http URL create a large-scale dataset of more than 150,000 preference pairs from raw multi-turn speech conversations, annotated with AI feedback, to cover preferences over both linguistic content and temporal context变化。我们利用离线对准方法来填补全双工自动回归语音到语音模型。广泛的实验表明，对通用对话的反馈可以始终有效地改善口语对话模型，从而产生更多的事实，更安全，更安全和更加上下文的相互作用。我们部署了固定模型，并进行整体人类评估，以评估超出单转交谈的影响。我们的发现阐明了各种动力学之间达到良好平衡的重要性，这对于自然的实时语音对话系统至关重要。

Title: TopK Language Models

• Authors: Ryosuke Takahashi, Tatsuro Inaba, Kentaro Inui, Benjamin Heinzerling
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21468
• Pdf URL: https://arxiv.org/pdf/2506.21468
• Copy Paste: [[2506.21468]] TopK Language Models(https://arxiv.org/abs/2506.21468)
• Keywords: language model
• Abstract: Sparse autoencoders (SAEs) have become an important tool for analyzing and interpreting the activation space of transformer-based language models (LMs). However, SAEs suffer several shortcomings that diminish their utility and internal validity. Since SAEs are trained post-hoc, it is unclear if the failure to discover a particular concept is a failure on the SAE's side or due to the underlying LM not representing this concept. This problem is exacerbated by training conditions and architecture choices affecting which features an SAE learns. When tracing how LMs learn concepts during training, the lack of feature stability also makes it difficult to compare SAEs features across different checkpoints. To address these limitations, we introduce a modification to the transformer architecture that incorporates a TopK activation function at chosen layers, making the model's hidden states equivalent to the latent features of a TopK SAE. This approach eliminates the need for post-hoc training while providing interpretability comparable to SAEs. The resulting TopK LMs offer a favorable trade-off between model size, computational efficiency, and interpretability. Despite this simple architectural change, TopK LMs maintain their original capabilities while providing robust interpretability benefits. Our experiments demonstrate that the sparse representations learned by TopK LMs enable successful steering through targeted neuron interventions and facilitate detailed analysis of neuron formation processes across checkpoints and layers. These features make TopK LMs stable and reliable tools for understanding how language models learn and represent concepts, which we believe will significantly advance future research on model interpretability and controllability.
• 摘要：稀疏自动编码器（SAE）已成为分析和解释基于变压器语言模型（LMS）的激活空间的重要工具。但是，SAE遭受了几个缺点，这些缺点降低了其效用和内部有效性。由于SAE是事后训练的，因此尚不清楚未能发现特定概念是SAE方面的失败，还是由于基础LM不代表这一概念的失败。培训条件和体系结构选择影响了SAE学习的内容，这使这个问题加剧了。在追踪LMS在训练过程中如何学习概念时，缺乏功能稳定性也使得很难比较不同检查站的SAES功能。为了解决这些限制，我们对变压器体系结构进行了修改，该架构在所选层上包含了TOPK激活功能，从而使模型的隐藏状态与Topk Sae的潜在特征等同。这种方法消除了对事后培训的需求，同时提供与SAE相当的可解释性。由此产生的TOPK LMS在模型大小，计算效率和解释性之间提供了有利的权衡。尽管发生了这种简单的建筑变化，但Topk LMS仍保持其原始功能，同时提供强大的可解释性优势。我们的实验表明，TOPK LMS所学到的稀疏表示能够通过靶向神经元干预成功转向，并促进了检查点和层之间对神经元形成过程的详细分析。这些功能使TOPK LMS稳定且可靠的工具可以理解语言模型如何学习和表示概念，我们认为这将大大推动对模型可解释性和可控性的未来研究。

Title: Bridging Offline and Online Reinforcement Learning for LLMs

• Authors: Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang, Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, Sainbayar Sukhbaatar, Ilia Kulikov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21495
• Pdf URL: https://arxiv.org/pdf/2506.21495
• Copy Paste: [[2506.21495]] Bridging Offline and Online Reinforcement Learning for LLMs(https://arxiv.org/abs/2506.21495)
• Keywords: language model, llm
• Abstract: We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types.
• 摘要：我们调查了在从离线到半联盟的大型语言模型中，增强学习方法的有效性，以完成可验证和不可验证的任务的完全在线制度。我们的实验涵盖了对可验证的数学以及不可验证的指导进行的培训，并对两者进行了一组基准评估。在这些设置中，我们广泛比较在线和半对线直接偏好优化和小组奖励策略优化目标，并且出人意料地发现这些变体之间的性能和收敛性相似，这些变体的表现强烈胜过离线方法。我们对训练动力学和超参数选择策略提供了详细的分析，以实现最佳结果。最后，我们表明，具有可验证和不可验证的奖励的多任务连接可在两种任务类型中的性能提高。

Title: Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments

• Authors: Jiashuo Wang, Kaitao Song, Chunpu Xu, Changhe Song, Yang Xiao, Dongsheng Li, Lili Qiu, Wenjie Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21497
• Pdf URL: https://arxiv.org/pdf/2506.21497
• Copy Paste: [[2506.21497]] Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments(https://arxiv.org/abs/2506.21497)
• Keywords: llm
• Abstract: Enhancing user engagement through interactions plays an essential role in socially-driven dialogues. While prior works have optimized models to reason over relevant knowledge or plan a dialogue act flow, the relationship between user engagement and knowledge or dialogue acts is subtle and does not guarantee user engagement in socially-driven dialogues. To this end, we enable interactive LLMs to learn user engagement by leveraging signals from the future development of conversations. Specifically, we adopt a more direct and relevant indicator of user engagement, i.e., the user's reaction related to dialogue intention after the interaction, as a reward to align interactive LLMs. To achieve this, we develop a user simulator to interact with target interactive LLMs and explore interactions between the user and the interactive LLM system via \textit{i$\times$MCTS} (\textit{M}onte \textit{C}arlo \textit{T}ree \textit{S}earch for \textit{i}nteraction). In this way, we collect a dataset containing pairs of higher and lower-quality experiences using \textit{i$\times$MCTS}, and align interactive LLMs for high-level user engagement by direct preference optimization (DPO) accordingly. Experiments conducted on two socially-driven dialogue scenarios (emotional support conversations and persuasion for good) demonstrate that our method effectively enhances user engagement in interactive LLMs.
• 摘要：通过互动增强用户参与在社会驱动的对话中起着至关重要的作用。虽然先前的作品已经优化了模型来推理相关知识或计划对话行为的流程，但用户参与与知识或对话行为之间的关系微妙，并且不能保证用户参与社会驱动的对话。为此，我们使Interactive LLM能够通过利用对话的未来发展来学习用户参与度。具体来说，我们采用了用户参与度更直接和相关的指标，即用户与互动后对话意图有关的反应，作为对Alignitive llms的奖励。为此，我们开发了一个用户模拟器，以与目标交互式LLM进行交互，并通过\ textIt {i $ \ times $ mcts}（\ textit {m} inte \ textit \ textit {c} arlo \ textit {arlo \ textit {t} ree \ textit {这样，我们收集了一个数据集，其中包含\ textit {i $ \ times $ mcts}的较高和低质量的体验，并相应地通过直接偏好优化（DPO）来使Interactive llms与高级用户参与度相适应。对两个社会驱动的对话方案进行的实验（情感支持对话和良好的说服力）表明，我们的方法有效地增强了用户在交互式LLM中的参与度。

Title: skLEP: A Slovak General Language Understanding Benchmark

• Authors: Marek Šuppa, Andrej Ridzik, Daniel Hládek, Tomáš Javůrek, Viktória Ondrejová, Kristína Sásiková, Martin Tamajka, Marián Šimko
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.21508
• Pdf URL: https://arxiv.org/pdf/2506.21508
• Copy Paste: [[2506.21508]] skLEP: A Slovak General Language Understanding Benchmark(https://arxiv.org/abs/2506.21508)
• Keywords: language model
• Abstract: In this work, we introduce skLEP, the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models. We have compiled skLEP to encompass nine diverse tasks that span token-level, sentence-pair, and document-level challenges, thereby offering a thorough assessment of model capabilities. To create this benchmark, we curated new, original datasets tailored for Slovak and meticulously translated established English NLU resources. Within this paper, we also present the first systematic and extensive evaluation of a wide array of Slovak-specific, multilingual, and English pre-trained language models using the skLEP tasks. Finally, we also release the complete benchmark data, an open-source toolkit facilitating both fine-tuning and evaluation of models, and a public leaderboard at this https URL in the hopes of fostering reproducibility and drive future research in Slovak NLU.
• 摘要：在这项工作中，我们介绍了Sklep，这是第一个专门设计用于评估斯洛伐克自然语言理解（NLU）模型的综合基准。我们已经编译了Sklep，以涵盖九种不同的任务，这些任务涵盖了令牌级别，句子对和文档级别的挑战，从而提供了对模型功能的彻底评估。为了创建这个基准，我们策划了针对斯洛伐克量身定制的新的原始数据集，并精心翻译成已建立的英语NLU资源。在本文中，我们还介绍了使用SKLEP任务的各种斯洛伐克特异性，多语言和英语预训练的语言模型的首次系统和广泛的评估。最后，我们还发布了完整的基准数据，这是一种开源工具包，可促进模型进行微调和评估，以及此HTTPS URL的公共排行榜，以期促进可重复性并推动Slovak NLU的未来研究。

Title: Potemkin Understanding in Large Language Models

• Authors: Marina Mancoridis, Bec Weeks, Keyon Vafa, Sendhil Mullainathan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.21521
• Pdf URL: https://arxiv.org/pdf/2506.21521
• Copy Paste: [[2506.21521]] Potemkin Understanding in Large Language Models(https://arxiv.org/abs/2506.21521)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are regularly evaluated using benchmark datasets. But what justifies making inferences about an LLM's capabilities based on its answers to a curated set of questions? This paper first introduces a formal framework to address this question. The key is to note that the benchmarks used to test LLMs -- such as AP exams -- are also those used to test people. However, this raises an implication: these benchmarks are only valid tests if LLMs misunderstand concepts in ways that mirror human misunderstandings. Otherwise, success on benchmarks only demonstrates potemkin understanding: the illusion of understanding driven by answers irreconcilable with how any human would interpret a concept. We present two procedures for quantifying the existence of potemkins: one using a specially designed benchmark in three domains, the other using a general procedure that provides a lower-bound on their prevalence. We find that potemkins are ubiquitous across models, tasks, and domains. We also find that these failures reflect not just incorrect understanding, but deeper internal incoherence in concept representations.
• 摘要：使用基准数据集定期评估大型语言模型（LLMS）。但是，根据其对一组策划的问题的回答，对LLM功能的推断有道理呢？本文首先介绍了一个正式框架来解决这个问题。关键是要注意，用于测试LLM的基准（例如AP考试）也是用于测试人员的基准。但是，这引起了一个含义：这些基准是在LLMS以反映人类误解的方式误解概念的情况下才是有效的测试。否则，基准上的成功只会证明Potemkin的理解：理解的幻觉是由答案驱动的与任何人如何解释概念的不可调节。我们提出了两个程序来量化波特姆金的存在：一个使用三个域中使用特殊设计的基准测试，另一个使用一般过程，该程序为其患病率提供较低的限制。我们发现Potemkins在模型，任务和域之间无处不在。我们还发现，这些故障不仅反映了不正确的理解，而且反映了概念表示的更深的内部不连贯性。

Title: "What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets

• Authors: Akshay Paruchuri, Maryam Aziz, Rohit Vartak, Ayman Ali, Best Uchehara, Xin Liu, Ishan Chatterjee, Monica Agrawal
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2506.21532
• Pdf URL: https://arxiv.org/pdf/2506.21532
• Copy Paste: [[2506.21532]] "What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets(https://arxiv.org/abs/2506.21532)
• Keywords: language model, llm, chat
• Abstract: People are increasingly seeking healthcare information from large language models (LLMs) via interactive chatbots, yet the nature and inherent risks of these conversations remain largely unexplored. In this paper, we filter large-scale conversational AI datasets to achieve HealthChat-11K, a curated dataset of 11K real-world conversations composed of 25K user messages. We use HealthChat-11K and a clinician-driven taxonomy for how users interact with LLMs when seeking healthcare information in order to systematically study user interactions across 21 distinct health specialties. Our analysis reveals insights into the nature of how and why users seek health information, such as common interactions, instances of incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy, underscoring the need for improvements in the healthcare support capabilities of LLMs deployed as conversational AI. Code and artifacts to retrieve our analyses and combine them into a curated dataset can be found here: this https URL
• 摘要：人们越来越多地通过互动聊天机器人从大语言模型（LLM）寻求医疗保健信息，但是这些对话的性质和固有风险仍然在很大程度上尚未开发。在本文中，我们过滤大规模的对话AI数据集，以实现HealthChat-11k，这是由25K用户消息组成的11K现实世界对话的策划数据集。我们使用HealthChat-11k和临床医生驱动的分类法，以便在寻求医疗保健信息时如何与LLM互动，以便系统地研究21种不同健康专业的用户互动。我们的分析揭示了对用户如何以及为什么寻求健康信息的本质的见解，例如共同的互动，不完整的环境，情感行为和互动（例如，主要问题）可以诱发无si虫，强调在LLMS部署为对话AI的LLMS医疗保健支持能力方面的提高需求。可以在此处找到代码和工件来检索我们的分析并将它们组合到策划数据集中：此HTTPS URL

Title: Data Efficacy for Language Model Training

• Authors: Yalun Dai, Yangyu Huang, Xin Zhang, Wenshan Wu, Chong Li, Wenhui Lu, Shijie Cao, Li Dong, Scarlett Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.21545
• Pdf URL: https://arxiv.org/pdf/2506.21545
• Copy Paste: [[2506.21545]] Data Efficacy for Language Model Training(https://arxiv.org/abs/2506.21545)
• Keywords: language model
• Abstract: Data is fundamental to the training of language models (LM). Recent research has been dedicated to data efficiency, which aims to maximize performance by selecting a minimal or optimal subset of training data. Techniques such as data filtering, sampling, and selection play a crucial role in this area. To complement it, we define Data Efficacy, which focuses on maximizing performance by optimizing the organization of training data and remains relatively underexplored. This work introduces a general paradigm, DELT, for considering data efficacy in LM training, which highlights the significance of training data organization. DELT comprises three components: Data Scoring, Data Selection, and Data Ordering. Among these components, we design Learnability-Quality Scoring (LQS), as a new instance of Data Scoring, which considers both the learnability and quality of each data sample from the gradient consistency perspective. We also devise Folding Ordering (FO), as a novel instance of Data Ordering, which addresses issues such as model forgetting and data distribution bias. Comprehensive experiments validate the data efficacy in LM training, which demonstrates the following: Firstly, various instances of the proposed DELT enhance LM performance to varying degrees without increasing the data scale and model size. Secondly, among these instances, the combination of our proposed LQS for data scoring and Folding for data ordering achieves the most significant improvement. Lastly, data efficacy can be achieved together with data efficiency by applying data selection. Therefore, we believe that data efficacy is a promising foundational area in LM training.
• 摘要：数据是语言模型培训（LM）的基础。最近的研究致力于数据效率，该研究旨在通过选择最小或最佳的培训数据子集来最大程度地提高性能。数据过滤，采样和选择等技术在该领域起着至关重要的作用。为了补充它，我们定义了数据功效，该数据功效通过优化培训数据的组织来最大程度地提高性能，并保持相对不受影响。这项工作介绍了一般范式Delt，用于考虑LM培训中的数据功效，这突出了培训数据组织的重要性。 Delt包括三个组件：数据评分，数据选择和数据排序。在这些组件中，我们将可学习性质量评分（LQS）设计为数据评分的新实例，从梯度一致性角度来看，它考虑了每个数据样本的可学习性和质量。我们还将折叠顺序（FO）设计为数据排序的新实例，该实例解决了模型遗忘和数据分配偏置等问题。全面的实验验证了LM培训中的数据功效，该实验证明了以下内容：首先，提出的delt的各种实例在不同程度上增强了LM性能，而不会增加数据量表和模型大小。其次，在这些情况下，我们提出的用于数据订购数据评分和折叠的LQ的组合可以实现最大的改进。最后，通过应用数据选择，可以与数据效率一起达到数据效率。因此，我们认为数据功效是LM培训中有前途的基础领域。