2025-08-28

Title: MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts

• Authors: Qing Wang, Xue Han, Jiahui Wang, Lehao Xing, Qian Hu, Lianlian Zhang, Chao Deng, Junlan Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19268
• Pdf URL: https://arxiv.org/pdf/2508.19268
• Copy Paste: [[2508.19268]] MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts(https://arxiv.org/abs/2508.19268)
• Keywords: language model, llm
• Abstract: Despite LLMs' excellent code creation capabilities, multilingual code generation remains extremely challenging. To address this, we intent to improve the multi-programming-lingual (MultiPL) performance of the base LLMs while retaining the most popular ones using restricted computational resources. We consider MultiPL to be a special case of multiple natural languages and propose a MultiPL extension of LLMs utilizing a hybrid mixture of experts (MoE), called MultiPL-MoE. Specifically, MultiPL-MoE combines two paired MoEs to optimize expert selection at both the token and segment levels. The token-level MoE is a standard upcycling MoE structure with a shared expert and a novel gate weight normalization approach that aids in the final fusion with the segment-level MoE. The segment-level MoE incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages: First, using a sliding window to partition the input token sequence into multiple segments; Then, adopting an expert-choice routing strategy that allows experts to select the top-k segments. The results of the experiment proved the effectiveness of MultiPL-MoE.
• 摘要：尽管LLMS出色的代码创建功能，但多语言代码生成仍然极具挑战性。为了解决这个问题，我们打算提高基本LLM的多编程 - 语言（乘法）性能，同时使用受限的计算资源保留最受欢迎的表现。我们认为乘以多种自然语言的特殊情况，并提出了使用专家（MOE）的混合混合物（称为Multipl-MoE）的LLMS乘以扩展。具体而言，乘以Moe结合了两个配对的MOE，以优化令牌和细分级别的专家选择。令牌级的MOE是一种标准的升级启动启动结构，具有共同的专家和一种新型的栅极重量归一化方法，有助于与细分级级的MOE进行最终融合。细分级的MOE结合了两个创新的设计，以更好地捕获编程语言的句法结构和上下文模式：首先，使用滑动窗口将输入令牌序列分配到多个段中；然后，采用专家选择的路由策略，允许专家选择顶级的细分市场。实验的结果证明了乘法的有效性。

Title: Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoT

• Authors: Rushitha Santhoshi Mamidala, Anshuman Chhabra, Ankur Mali
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19271
• Pdf URL: https://arxiv.org/pdf/2508.19271
• Copy Paste: [[2508.19271]] Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoT(https://arxiv.org/abs/2508.19271)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Prompt-based reasoning strategies such as Chain-of-Thought (CoT) and In-Context Learning (ICL) have become widely used for eliciting reasoning capabilities in large language models (LLMs). However, these methods rely on fragile, implicit mechanisms often yielding inconsistent outputs across seeds, formats, or minor prompt variations making them fundamentally unreliable for tasks requiring stable, interpretable reasoning. In contrast, automata-based neuro-symbolic frameworks like RetoMaton offer a more structured and trustworthy alternative by grounding retrieval in symbolic memory with deterministic transitions. In this work, we extend RetoMaton by replacing its global datastore with a local, task-adaptive Weighted Finite Automaton (WFA), constructed directly from external domain corpora. This local automaton structure promotes robust, context-aware retrieval while preserving symbolic traceability and low inference overhead. Unlike prompting, which entangles context and memory in opaque ways, our approach leverages the explicit structure of WFAs to provide verifiable and modular retrieval behavior, making it better suited for domain transfer and interoperability. We evaluate this local RetoMaton variant on two pretrained LLMs LLaMA-3.2-1B and Gemma-3-1B-PT across three reasoning tasks: TriviaQA (reading comprehension), GSM8K (multi-step math), and MMLU (domain knowledge). Compared to the base model and prompting-based methods, augmenting these setups with local RetoMaton consistently improves performance while enabling transparent and reproducible retrieval dynamics. Our results highlight a promising shift toward trustworthy, symbolic reasoning in modern LLMs via lightweight, automaton-guided memory.
• 摘要：基于迅速的推理策略，例如思考链（COT）和内部文化学习（ICL），已广泛用于在大语言模型（LLMS）中引发推理能力。但是，这些方法依赖于脆弱的隐式机制通常会在种子，格式或较小的及时变化之间产生不一致的产出，从而使其对需要稳定，可解释的推理的任务根本上不可靠。相比之下，基于自动机的神经符号框架（如retomaton）通过通过确定性转换中的象征性记忆中的检索来实现更具结构化和值得信赖的替代方案。在这项工作中，我们通过用直接从外部域Corpora构建的本地任务自动加权有限自动机（WFA）来替换其全局数据存储，从而扩展了Retomaton。这种局部自动机结构可促进健壮的上下文感知检索，同时保留象征性的可追溯性和低推理开销。与提示以不透明的方式纠缠上下文和内存的提示不同，我们的方法利用WFA的明确结构提供可验证和模块化的检索行为，使其更适合域传输和互操作性。我们在两个预处理的LLMS-3.2-1B和Gemma-3-1b-pt中评估了该局部retomaton变体，这些局部变体在三个推理任务中：Triviaqa（阅读理解），GSM8K（多步数学）和MMLU（域知识）。与基本模型和基于促进的方法相比，使用局部retomaton增强这些设置会始终提高性能，同时启用透明和可再现的检索动力学。我们的结果强调了通过轻巧，自动化的记忆在现代LLM中向值得信赖的，象征性推理的有前途的转变。

Title: RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits

• Authors: Kshitij Fadnis, Sara Rosenthal, Maeda Hanafi, Yannis Katsis, Marina Danilevsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19272
• Pdf URL: https://arxiv.org/pdf/2508.19272
• Copy Paste: [[2508.19272]] RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits(https://arxiv.org/abs/2508.19272)
• Keywords: language model, llm, chat, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) is an important aspect of conversing with Large Language Models (LLMs) when factually correct information is important. LLMs may provide answers that appear correct, but could contain hallucinated information. Thus, building benchmarks that can evaluate LLMs on multi-turn RAG conversations has become an increasingly important task. Simulating real-world conversations is vital for producing high quality evaluation benchmarks. We present RAGAPHENE, a chat-based annotation platform that enables annotators to simulate real-world conversations for benchmarking and evaluating LLMs. RAGAPHENE has been successfully used by approximately 40 annotators to build thousands of real-world conversations.
• 摘要：当事实正确的信息很重要时，检索增强生成（RAG）是与大语言模型（LLM）对话的重要方面。 LLM可以提供看起来正确的答案，但可能包含幻觉信息。因此，构建可以评估多转抹布对话中LLM的基准已成为越来越重要的任务。模拟现实世界对话对于产生高质量评估基准至关重要。我们提出了Ragaphene，这是一个基于聊天的注释平台，使注释者能够模拟现实世界中的对话，以进行基准测试和评估LLMS。大约40个注释者成功使用了Ragaphene来构建数千个现实世界的对话。

Title: Leveraging Language Models and Machine Learning in Verbal Autopsy Analysis

• Authors: Yue Chu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19274
• Pdf URL: https://arxiv.org/pdf/2508.19274
• Copy Paste: [[2508.19274]] Leveraging Language Models and Machine Learning in Verbal Autopsy Analysis(https://arxiv.org/abs/2508.19274)
• Keywords: language model
• Abstract: In countries without civil registration and vital statistics, verbal autopsy (VA) is a critical tool for estimating cause of death (COD) and inform policy priorities. In VA, interviewers ask proximal informants for details on the circumstances preceding a death, in the form of unstructured narratives and structured questions. Existing automated VA cause classification algorithms only use the questions and ignore the information in the narratives. In this thesis, we investigate how the VA narrative can be used for automated COD classification using pretrained language models (PLMs) and machine learning (ML) techniques. Using empirical data from South Africa, we demonstrate that with the narrative alone, transformer-based PLMs with task-specific fine-tuning outperform leading question-only algorithms at both the individual and population levels, particularly in identifying non-communicable diseases. We explore various multimodal fusion strategies combining narratives and questions in unified frameworks. Multimodal approaches further improve performance in COD classification, confirming that each modality has unique contributions and may capture valuable information that is not present in the other modality. We also characterize physician-perceived information sufficiency in VA. We describe variations in sufficiency levels by age and COD and demonstrate that classification accuracy is affected by sufficiency for both physicians and models. Overall, this thesis advances the growing body of knowledge at the intersection of natural language processing, epidemiology, and global health. It demonstrates the value of narrative in enhancing COD classification. Our findings underscore the need for more high-quality data from more diverse settings to use in training and fine-tuning PLM/ML methods, and offer valuable insights to guide the rethinking and redesign of the VA instrument and interview.
• 摘要：在没有民事注册和重要统计数据的国家中，口头尸检（VA）是估计死亡原因（COD）和为政策优先级提供信息的关键工具。在弗吉尼亚州，访调员以非结构化的叙述和结构化问题的形式向近端线人询问有关死亡之前情况的详细信息。现有的自动化VA导致分类算法仅使用问题，而忽略叙述中的信息。在本文中，我们研究了如何使用验证的语言模型（PLM）和机器学习（ML）技术将VA叙述用于自动化COD分类。使用来自南非的经验数据，我们证明，仅使用叙事，基于变压器的PLM具有特定于任务的微调优于领先的问题，仅在个人和人口水平上，尤其是在识别非跨性别疾病时。我们探索各种多模式融合策略，结合了统一框架中的叙述和问题。多模式方法进一步提高了COD分类的性能，证实每种模式都有独特的贡献，并且可能捕获其他模式中不存在的有价值的信息。我们还表征了VA中医师感知的信息充足性。我们描述了按年龄和COD划分的足够水平的变化，并证明了分类精度受医师和模型的充分性影响。总体而言，本论文在自然语言处理，流行病学和全球健康的交集中越来越多的知识体现。它证明了叙事在增强COD分类中的价值。我们的发现强调了从更多样化的环境中使用更多高质量数据的需求，以用于培训和微调PLM/ML方法，并提供有价值的见解，以指导VA工具和访谈的重新思考和重新设计。

Title: FLAIRR-TS -- Forecasting LLM-Agents with Iterative Refinement and Retrieval for Time Series

• Authors: Gunjan Jalori, Preetika Verma, Sercan Ö Arık
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19279
• Pdf URL: https://arxiv.org/pdf/2508.19279
• Copy Paste: [[2508.19279]] FLAIRR-TS -- Forecasting LLM-Agents with Iterative Refinement and Retrieval for Time Series(https://arxiv.org/abs/2508.19279)
• Keywords: llm, prompt, agent
• Abstract: Time series Forecasting with large languagemodels (LLMs) requires bridging numericalpatterns and natural language. Effective fore-casting on LLM often relies on extensive pre-processing and this http URL studiesshow that a frozen LLM can rival specializedforecasters when supplied with a carefully en-gineered natural-language prompt, but craft-ing such a prompt for each task is itself oner-ous and ad-hoc. We introduce FLAIRR-TS, atest-time prompt optimization framework thatutilizes an agentic system: a Forecaster-agentgenerates forecasts using an initial prompt,which is then refined by a refiner agent, in-formed by past outputs and retrieved this http URL adaptive prompting generalizes across do-mains using creative prompt templates andgenerates high-quality forecasts without inter-mediate code this http URL onbenchmark datasets show improved accuracyover static prompting and retrieval-augmentedbaselines, approaching the performance ofspecialized this http URL-TS providesa practical alternative to tuning, achievingstrong performance via its agentic approach toadaptive prompt refinement and retrieval.
• 摘要：时间序列预测大型语言模型（LLMS）需要桥接数字图案和自然语言。 LLM上有效的预先铸造通常依赖于广泛的预处理，而HTTP URL studiesshow当冷冻的LLM可以与精心设计的自然语言提示供应时，可以抗衡专用ForeCaster，但要努力的工艺，但是对于每个任务的提示本身就是Oner-os and-os and-os and-os and-os-os and-os and-os-os and-os-os and-os-os-os-os-os and-hoc。 We introduce FLAIRR-TS, atest-time prompt optimization framework thatutilizes an agentic system: a Forecaster-agentgenerates forecasts using an initial prompt,which is then refined by a refiner agent, in-formed by past outputs and retrieved this http URL adaptive prompting generalizes across do-mains using creative prompt templates andgenerates high-quality forecasts without inter-mediate code this http URL Onbenchmark数据集显示出改进的准确性静态提示和检索额定基础，以通过其Adadaptigation Adadaptive及时的精炼和检索来实施该http url-ts的性能，可为调整，实现的方法提供可用的替代方案。

Title: CORE: Lossless Compression for Retrieval-Augmented LLMs via Reinforcement Learning

• Authors: Ziqiang Cui, Yunpeng Weng, Xing Tang, Peiyang Liu, Shiwei Li, Bowei He, Jiamin Chen, Xiuqiang He, Chen Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19282
• Pdf URL: https://arxiv.org/pdf/2508.19282
• Copy Paste: [[2508.19282]] CORE: Lossless Compression for Retrieval-Augmented LLMs via Reinforcement Learning(https://arxiv.org/abs/2508.19282)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a promising approach to enhance the timeliness of knowledge and the factual accuracy of responses in Large Language Models (LLMs). However, the inclusion of excessive retrieved documents substantially increases the input length, leading to higher computational costs. Previous studies have attempted to compress retrieved documents into shorter texts before in-context integration, but such methods often compromise end-task performance. The lack of well-defined compression targets forces many approaches to rely on fixed heuristics, which cannot guarantee that the compressed content will effectively support the end task. To address these limitations, we propose CORE, a novel method designed to achieve lossless context compression for RAG. CORE employs reinforcement learning to optimize the compression process without relying on predefined compression labels. Specifically, it utilizes end-task performance as a reward signal and applies Generalized Reinforcement Learning Policy Optimization (GRPO) to train the compressor. This end-to-end training framework enables the compressor to generate summaries that maximize the accuracy of answers generated by the LLM. Extensive experiments on four datasets demonstrate the superiority of our approach. With a high compression ratio of 3\%, our method not only avoids performance degradation compared to prepending full documents across all datasets but also improves the average Exact Match (EM) score by 3.3 points. The code will be released soon.
• 摘要：在大语言模型（LLMS）中，检索提升的生成（RAG）已成为提高知识及时性和响应的事实准确性的一种有希望的方法。但是，包含过多的已检索文档大大增加了输入长度，从而导致了更高的计算成本。先前的研究试图将检索到的文档压缩到较短的文本中，但在内在集成之前，这种方法通常会损害最终任务的性能。缺乏定义明确的压缩目标迫使许多方法依靠固定的启发式方法，这不能保证压缩内容将有效地支持最终任务。为了解决这些局限性，我们提出了核心，这是一种旨在实现抹布无损上下文压缩的新方法。核心采用强化学习来优化压缩过程，而无需依赖预定义的压缩标签。具体而言，它利用终端性能作为奖励信号，并应用了广义的增强学习政策优化（GRPO）来训练压缩机。这个端到端的训练框架使压缩机能够生成摘要，从而最大程度地提高了LLM生成的答案的准确性。四个数据集上的广泛实验证明了我们方法的优势。高压比为3 \％，我们的方法不仅避免了性能降低，而且与所有数据集的完整文档相比，还可以将平均精确匹配（EM）得分提高3.3分。该代码将很快发布。

Title: Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domains

• Authors: Peiran Zhou, Junnan Zhu, Yichen Shen, Ruoxi Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19357
• Pdf URL: https://arxiv.org/pdf/2508.19357
• Copy Paste: [[2508.19357]] Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domains(https://arxiv.org/abs/2508.19357)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) excel in language tasks but are prone to hallucinations and outdated knowledge. Retrieval-Augmented Generation (RAG) mitigates these by grounding LLMs in external knowledge. However, in complex domains involving multiple, lengthy, or conflicting documents, traditional RAG suffers from information overload and inefficient synthesis, leading to inaccurate and untrustworthy answers. To address this, we propose CASC (Context-Adaptive Synthesis and Compression), a novel framework that intelligently processes retrieved contexts. CASC introduces a Context Analyzer & Synthesizer (CAS) module, powered by a fine-tuned smaller LLM, which performs key information extraction, cross-document consistency checking and conflict resolution, and question-oriented structured synthesis. This process transforms raw, scattered information into a highly condensed, structured, and semantically rich context, significantly reducing the token count and cognitive load for the final Reader LLM. We evaluate CASC on SciDocs-QA, a new challenging multi-document question answering dataset designed for complex scientific domains with inherent redundancies and conflicts. Our extensive experiments demonstrate that CASC consistently outperforms strong baselines.
• 摘要：大型语言模型（LLMS）在语言任务中表现出色，但容易幻觉和过时的知识。检索增强的生成（RAG）通过将LLM扎根于外部知识中来减轻这些作用。但是，在涉及多个，冗长或相互矛盾的文档的复杂领域中，传统的抹布遭受了信息过载和效率低下的合成，从而导致不准确和不可信的答案。为了解决这个问题，我们提出了CASC（上下文自适应合成和压缩），这是一个智能处理上下文的新型框架。 CASC引入了上下文分析仪和合成器（CAS）模块，该模块由微调的较小LLM提供动力，该模块执行关键信息提取，跨文档的一致性检查和冲突解决方案以及面向问题的结构化合成。该过程将原始的，分散的信息转化为高度凝结，结构化和语义上丰富的上下文，从而大大降低了最终读取器LLM的代币计数和认知负载。我们在SCIDOCS-QA上评估CASC，这是一个新的具有挑战性的多文章问题，答复了为具有固有裁员和冲突的复杂科学领域设计的数据集。我们的广泛实验表明，CASC始终优于强大的基准。

Title: Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction

• Authors: Fatemeh Haji, Mazal Bethany, Cho-Yu Jason Chiang, Anthony Rios, Peyman Najafirad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19359
• Pdf URL: https://arxiv.org/pdf/2508.19359
• Copy Paste: [[2508.19359]] Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction(https://arxiv.org/abs/2508.19359)
• Keywords: language model, llm, hallucination, agent
• Abstract: Event Extraction (EE) involves automatically identifying and extracting structured information about events from unstructured text, including triggers, event types, and arguments. Traditional discriminative models demonstrate high precision but often exhibit limited recall, particularly for nuanced or infrequent events. Conversely, generative approaches leveraging Large Language Models (LLMs) provide higher semantic flexibility and recall but suffer from hallucinations and inconsistent predictions. To address these challenges, we propose Agreement-based Reflective Inference System (ARIS), a hybrid approach combining a Self Mixture of Agents with a discriminative sequence tagger. ARIS explicitly leverages structured model consensus, confidence-based filtering, and an LLM reflective inference module to reliably resolve ambiguities and enhance overall event prediction quality. We further investigate decomposed instruction fine-tuning for enhanced LLM event extraction understanding. Experiments demonstrate our approach outperforms existing state-of-the-art event extraction methods across three benchmark datasets.
• 摘要：事件提取（EE）涉及从非结构化文本（包括触发器，事件类型和参数）中自动识别和提取有关事件的结构化信息。传统的判别模型表现出很高的精度，但经常表现出有限的回忆，尤其是对于细微或不经常事件的情况。相反，利用大语言模型（LLM）的生成方法提供了更高的语义灵活性和回忆，但却遭受了幻觉和预测不一致的影响。为了应对这些挑战，我们提出了基于协议的反思推理系统（ARIS），这是一种混合方法，将代理的自我混合物与判别序列标记器相结合。 ARIS明确利用结构化模型共识，基于置信度的过滤和LLM反射推理模块，以可靠地解决歧义并提高整体事件预测质量。我们进一步研究分解的指导微调，以增强LLM事件提取的理解。实验表明，我们的方法的表现优于三个基准数据集中现有的最新事件提取方法。

Title: LongReasonArena: A Long Reasoning Benchmark for Large Language Models

• Authors: Jiayu Ding, Shuming Ma, Lei Cui, Nanning Zheng, Furu Wei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19363
• Pdf URL: https://arxiv.org/pdf/2508.19363
• Copy Paste: [[2508.19363]] LongReasonArena: A Long Reasoning Benchmark for Large Language Models(https://arxiv.org/abs/2508.19363)
• Keywords: language model, llm
• Abstract: Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, Deepseek-R1 achieves only 7.5% accuracy on our task. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data is available at this https URL.
• 摘要：大型语言模型（LLMS）的现有长篇小说基准专注于评估对长投入的理解，同时忽略了对长推理能力的评估。为了解决这一差距，我们介绍了LongreasonArena，这是一种专门旨在评估LLM的长推理能力的基准。我们的任务需要模型来通过执行反映长期推理的关键方面（例如检索和回溯）的多步骤算法来解决问题。通过控制输入，可以任意扩展所需的推理长度，从而达到了最具挑战性的任务的100万个令牌。广泛的评估结果表明，Longreasonarena对开源和专有LLM都提出了重大挑战。例如，DeepSeek-R1在我们的任务上只能达到7.5％的精度。进一步的分析还表明，相对于预期的推理步骤数的对数，准确性表现出线性下降。我们的代码和数据可在此HTTPS URL上找到。

Title: Database Entity Recognition with Data Augmentation and Deep Learning

• Authors: Zikun Fu, Chen Yang, Kourosh Davoudi, Ken Q. Pu
• Subjects: cs.CL, cs.AI, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.19372
• Pdf URL: https://arxiv.org/pdf/2508.19372
• Copy Paste: [[2508.19372]] Database Entity Recognition with Data Augmentation and Deep Learning(https://arxiv.org/abs/2508.19372)
• Keywords: language model
• Abstract: This paper addresses the challenge of Database Entity Recognition (DB-ER) in Natural Language Queries (NLQ). We present several key contributions to advance this field: (1) a human-annotated benchmark for DB-ER task, derived from popular text-to-sql benchmarks, (2) a novel data augmentation procedure that leverages automatic annotation of NLQs based on the corresponding SQL queries which are available in popular text-to-SQL benchmarks, (3) a specialized language model based entity recognition model using T5 as a backbone and two down-stream DB-ER tasks: sequence tagging and token classification for fine-tuning of backend and performing DB-ER respectively. We compared our DB-ER tagger with two state-of-the-art NER taggers, and observed better performance in both precision and recall for our model. The ablation evaluation shows that data augmentation boosts precision and recall by over 10%, while fine-tuning of the T5 backbone boosts these metrics by 5-10%.
• 摘要：本文解决了自然语言查询（NLQ）中数据库实体识别（DB-ER）的挑战。我们为推进这一领域提出了几项关键贡献：（1）由人为宣传的DB-ER任务的基准，该基准从流行的文本到SQL基准测试得出，（2）一种新型的数据增强程序，该过程利用基于通俗的文本基于文本标记的SQL SQL查询来利用NLQ自动注释NLQ，并使用该通俗的SQL ASIDINE AS AS ASS AS ASSIDINE AS AS AS AS ASSITIDE AS AASINEDE AS AS AS AS AS AS ASSITION and（3）AS A AS A AS A AS A AS（3））（3））（3））两个下游DB-ER任务：分别用于对后端和执行DB-ER进行微调的序列标记和令牌分类。我们将我们的DB-ER标签器与两个最先进的标签器进行了比较，并观察到模型的精确度和回忆性能更好。消融评估表明，数据增强可提高精度并召回10％以上，而T5主链的微调将这些指标提高了5-10％。

Title: One Joke to Rule them All? On the (Im)possibility of Generalizing Humor

• Authors: Mor Turgeman, Chen Shani, Dafna Shahaf
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19402
• Pdf URL: https://arxiv.org/pdf/2508.19402
• Copy Paste: [[2508.19402]] One Joke to Rule them All? On the (Im)possibility of Generalizing Humor(https://arxiv.org/abs/2508.19402)
• Keywords: language model, llm
• Abstract: Humor is a broad and complex form of communication that remains challenging for machines. Despite its broadness, most existing research on computational humor traditionally focused on modeling a specific type of humor. In this work, we wish to understand whether competence on one or more specific humor tasks confers any ability to transfer to novel, unseen types; in other words, is this fragmentation inevitable? This question is especially timely as new humor types continuously emerge in online and social media contexts (e.g., memes, anti-humor, AI fails). If Large Language Models (LLMs) are to keep up with this evolving landscape, they must be able to generalize across humor types by capturing deeper, transferable mechanisms. To investigate this, we conduct a series of transfer learning experiments across four datasets, representing different humor tasks. We train LLMs under varied diversity settings (1-3 datasets in training, testing on a novel task). Experiments reveal that models are capable of some transfer, and can reach up to 75% accuracy on unseen datasets; training on diverse sources improves transferability (1.88-4.05%) with minimal-to-no drop in in-domain performance. Further analysis suggests relations between humor types, with Dad Jokes surprisingly emerging as the best enabler of transfer (but is difficult to transfer to). We release data and code.
• 摘要：幽默是一种广泛而复杂的沟通形式，对于机器仍然具有挑战性。尽管它广泛，但大多数关于计算幽默的研究传统上都集中在建模特定类型的幽默上。在这项工作中，我们希望了解一项或多项特定幽默任务的能力是否赋予转移到新颖，看不见的类型的能力。换句话说，这种分裂是不可避免的吗？这个问题尤其及时，因为在线和社交媒体环境中不断出现新的幽默类型（例如，模因，反幽默，AI失败）。如果大型语言模型（LLM）要跟上这一不断发展的景观，则它们必须能够通过捕获更深入，可转移的机制来概括跨幽默类型。为了调查这一点，我们在四个数据集中进行了一系列转移学习实验，代表了不同的幽默任务。我们在各种多样性设置下培训LLM（培训中的1-3个数据集，对新任务进行测试）。实验表明，模型能够进行某种传输，并且在看不见的数据集中可以达到75％的精度；对各种来源的培训可提高可转让性（1.88-4.05％），而内域性能下降最少。进一步的分析表明，幽默类型之间的关系，爸爸的笑话令人惊讶地成为转移的最佳推动者（但很难转移到）。我们发布数据和代码。

Title: A perishable ability? The future of writing in the face of generative artificial intelligence

• Authors: Evandro L. T. P. Cunha
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2508.19427
• Pdf URL: https://arxiv.org/pdf/2508.19427
• Copy Paste: [[2508.19427]] A perishable ability? The future of writing in the face of generative artificial intelligence(https://arxiv.org/abs/2508.19427)
• Keywords: language model
• Abstract: The 2020s have been witnessing a very significant advance in the development of generative artificial intelligence tools, including text generation systems based on large language models. These tools have been increasingly used to generate texts in the most diverse domains -- from technical texts to literary texts --, which might eventually lead to a lower volume of written text production by humans. This article discusses the possibility of a future in which human beings will have lost or significantly decreased their ability to write due to the outsourcing of this activity to machines. This possibility parallels the loss of the ability to write in other moments of human history, such as during the so-called Greek Dark Ages (approx. 1200 BCE - 800 BCE).
• 摘要：2020年代，人们一直在开发生成人工智能工具（包括基于大语言模型的文本生成系统）方面取得了很大的进步。这些工具越来越多地用于生成最多样化的领域（从技术文本到文学文本）中的文本，这最终可能导致人类的书面文本生产较低。本文讨论了人类由于将此活动外包给机器外包而丢失或大大降低其写作能力的未来的可能性。这种可能性与在人类历史的其他时刻（例如在所谓的希腊黑暗时代（公元前1200年至公元前1200年）中写作能力的丧失与丧失的能力相似。

Title: Heterogeneous LLM Methods for Ontology Learning (Few-Shot Prompting, Ensemble Typing, and Attention-Based Taxonomies)

• Authors: Aleksandra Beliaeva, Temurbek Rahmatullaev
• Subjects: cs.CL, cs.LO, cs.SC
• Abstract URL: https://arxiv.org/abs/2508.19428
• Pdf URL: https://arxiv.org/pdf/2508.19428
• Copy Paste: [[2508.19428]] Heterogeneous LLM Methods for Ontology Learning (Few-Shot Prompting, Ensemble Typing, and Attention-Based Taxonomies)(https://arxiv.org/abs/2508.19428)
• Keywords: llm, prompt, retrieval-augmented generation
• Abstract: We present a comprehensive system for addressing Tasks A, B, and C of the LLMs4OL 2025 challenge, which together span the full ontology construction pipeline: term extraction, typing, and taxonomy discovery. Our approach combines retrieval-augmented prompting, zero-shot classification, and attention-based graph modeling -- each tailored to the demands of the respective task. For Task A, we jointly extract domain-specific terms and their ontological types using a retrieval-augmented generation (RAG) pipeline. Training data was reformulated into a document to terms and types correspondence, while test-time inference leverages semantically similar training examples. This single-pass method requires no model finetuning and improves overall performance through lexical augmentation Task B, which involves assigning types to given terms, is handled via a dual strategy. In the few-shot setting (for domains with labeled training data), we reuse the RAG scheme with few-shot prompting. In the zero-shot setting (for previously unseen domains), we use a zero-shot classifier that combines cosine similarity scores from multiple embedding models using confidence-based weighting. In Task C, we model taxonomy discovery as graph inference. Using embeddings of type labels, we train a lightweight cross-attention layer to predict is-a relations by approximating a soft adjacency matrix. These modular, task-specific solutions enabled us to achieve top-ranking results in the official leaderboard across all three tasks. Taken together these strategies showcase the scalability, adaptability, and robustness of LLM-based architectures for ontology learning across heterogeneous domains. Code is available at: this https URL
• 摘要：我们提出了一个综合系统，用于解决LLMS4OL 2025挑战的任务A，B和C，该系统涵盖了完整的本体施工管道：术语提取，键入和分类学发现。我们的方法结合了检索提示的提示，零射击分类和基于注意力的图形建模 - 每种都根据各自任务的需求量身定制。对于任务A，我们使用检索功能（RAG）管道共同提取特定于领域的术语及其本体论类型。将培训数据重新计算为条款和类型对应的文档，而测试时间推理利用语义相似的培训示例。这种单通行方法不需要模型登录，并通过词汇增强任务B改善了整体性能B，涉及将类型分配给给定术语，是通过双重策略来处理的。在少数拍摄的设置（对于带有标记培训数据的域）中，我们在很少的弹药提示中重复使用了RAG方案。在零射击设置（对于以前看不见的域）中，我们使用零击分类器，该分类器使用基于置信的加权结合了来自多个嵌入模型的余弦相似性得分。在任务C中，我们将发现分类学发现为图推理。使用类型标签的嵌入方式，我们通过近似柔软的邻接矩阵来训练一个轻质的跨注意层来预测IS-A关系。这些模块化，特定于任务的解决方案使我们能够在所有三个任务中官方排行榜中获得顶级排行榜。结合这些策略，展示了基于LLM的架构在异构领域进行本体学习的可扩展性，适应性和鲁棒性。代码可用：此HTTPS URL

Title: Bridging Language Gaps: Enhancing Few-Shot Language Adaptation

• Authors: Philipp Borchert, Jochen De Weerdt, Marie-Francine Moens
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19464
• Pdf URL: https://arxiv.org/pdf/2508.19464
• Copy Paste: [[2508.19464]] Bridging Language Gaps: Enhancing Few-Shot Language Adaptation(https://arxiv.org/abs/2508.19464)
• Keywords: language model, prompt
• Abstract: The disparity in language resources poses a challenge in multilingual NLP, with high-resource languages benefiting from extensive data, while low-resource languages lack sufficient data for effective training. Our Contrastive Language Alignment with Prompting (CoLAP) method addresses this gap by integrating contrastive learning with cross-lingual representations, facilitating task-specific knowledge transfer from high-resource to lower-resource languages. The primary advantage of our approach is its data efficiency, enabling rapid adaptation to new languages and reducing the need for large labeled datasets. We conduct experiments with multilingual encoder-only and decoder-only language models on natural language understanding tasks, including natural language inference and relation extraction, evaluating performance across both high- and low-resource languages. Our results demonstrate that CoLAP outperforms few-shot cross-lingual transfer baselines and in-context learning, even with limited available data. This effectively narrows the cross-lingual performance gap, contributing to the development of more efficient multilingual NLP techniques.
• 摘要：语言资源的差异在多语言NLP中构成了挑战，高资源语言受益于广泛的数据，而低资源的语言则缺乏足够的数据来进行有效的培训。我们与提示（COLAP）方法的对比语言对齐方式通过将对比度学习与跨语性表示形式相结合，从而促进了特定于任务的知识转移从高资源到低资源语言，从而解决了这一差距。我们方法的主要优点是它的数据效率，可以快速适应新语言，并减少了对大型标签数据集的需求。我们对自然语言理解任务（包括自然语言推论和关系提取）进行多种语言的语言模型进行实验，并评估高资源语言和低资源语言的性能。我们的结果表明，即使可用的数据有限，COLAP的表现都超过了几乎没有传动的跨语义转移基线和文本学习。这有效地缩小了跨语性的性能差距，从而有助于开发更有效的多语言NLP技术。

Title: Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset

• Authors: Sumon Kanti Dey, Jeanne M. Powell, Azra Ismail, Jeanmarie Perrone, Abeed Sarker
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2508.19467
• Pdf URL: https://arxiv.org/pdf/2508.19467
• Copy Paste: [[2508.19467]] Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset(https://arxiv.org/abs/2508.19467)
• Keywords: language model, llm
• Abstract: Nonmedical opioid use is an urgent public health challenge, with far-reaching clinical and social consequences that are often underreported in traditional healthcare settings. Social media platforms, where individuals candidly share first-person experiences, offer a valuable yet underutilized source of insight into these impacts. In this study, we present a named entity recognition (NER) framework to extract two categories of self-reported consequences from social media narratives related to opioid use: ClinicalImpacts (e.g., withdrawal, depression) and SocialImpacts (e.g., job loss). To support this task, we introduce RedditImpacts 2.0, a high-quality dataset with refined annotation guidelines and a focus on first-person disclosures, addressing key limitations of prior work. We evaluate both fine-tuned encoder-based models and state-of-the-art large language models (LLMs) under zero- and few-shot in-context learning settings. Our fine-tuned DeBERTa-large model achieves a relaxed token-level F1 of 0.61 [95% CI: 0.43-0.62], consistently outperforming LLMs in precision, span accuracy, and adherence to task-specific guidelines. Furthermore, we show that strong NER performance can be achieved with substantially less labeled data, emphasizing the feasibility of deploying robust models in resource-limited settings. Our findings underscore the value of domain-specific fine-tuning for clinical NLP tasks and contribute to the responsible development of AI tools that may enhance addiction surveillance, improve interpretability, and support real-world healthcare decision-making. The best performing model, however, still significantly underperforms compared to inter-expert agreement (Cohen's kappa: 0.81), demonstrating that a gap persists between expert intelligence and current state-of-the-art NER/AI capabilities for tasks requiring deep domain knowledge.
• 摘要：非医疗阿片类药物使用是紧迫的公共卫生挑战，在传统医疗保健环境中通常会低估临床和社会后果。社交媒体平台，个人坦率地分享第一人称体验，它为这些影响提供了宝贵但未充分利用的见解来源。在这项研究中，我们提出了一个指定的实体识别框架（NER）框架，以从与阿片类药物使用相关的社交媒体叙事中提取两类自我报告的后果：临床影响（例如，戒断，抑郁症）和社会影响力（例如，失业）。为了支持此任务，我们介绍了Redditimpacts 2.0，这是一个具有精致注释指南的高质量数据集，并专注于第一人称披露，解决了先前工作的关键局限性。我们在零和几乎没有射击的内在学习设置下评估了基于微调的编码模型和最先进的大语言模型（LLMS）。我们的微调Deberta-Large模型达到了0.61 [95％CI：0.43-0.62]的放松令牌级别的F1，在精确度，跨度准确性和遵守特定于任务的准则方面始终超过LLMS。此外，我们表明，可以使用标签的数据较小的数据来实现强大的性能，这强调了在资源受限设置中部署强大模型的可行性。我们的发现强调了针对临床NLP任务的域特异性微调的价值，并有助于负责开发AI工具，这些工具可能会增强成瘾的监视，提高解释性并支持现实世界中的医疗保健决策。然而，与专家间协议（Cohen's Kappa：0.81）相比，最佳性能模型仍然显着表现，表明专家智能和当前最新的NER/AI/AI能力之间存在差距，以实现需要深层领域知识的任务。

Title: Automatic Question & Answer Generation Using Generative Large Language Model (LLM)

• Authors: Md. Alvee Ehsan, A.S.M Mehedi Hasan, Kefaya Benta Shahnoor, Syeda Sumaiya Tasneem
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19475
• Pdf URL: https://arxiv.org/pdf/2508.19475
• Copy Paste: [[2508.19475]] Automatic Question & Answer Generation Using Generative Large Language Model (LLM)(https://arxiv.org/abs/2508.19475)
• Keywords: language model, llm, prompt
• Abstract: \Abstract{In the realm of education, student evaluation holds equal significance as imparting knowledge. To be evaluated, students usually need to go through text-based academic assessment methods. Instructors need to make diverse sets of questions that need to be fair for all students to prove their adequacy over a particular topic. This can prove to be quite challenging as they may need to manually go through several different lecture materials. Our objective is to make this whole process much easier by implementing Automatic Question Answer Generation /(AQAG), using fine-tuned generative LLM. For tailoring the instructor's preferred question style (MCQ, conceptual, or factual questions), prompt Engineering (PE) is being utilized. In this research, we propose to leverage unsupervised learning methods in NLP, primarily focusing on the English language. This approach empowers the base Meta-Llama 2-7B model to integrate RACE dataset as training data for the fine-tuning process. Creating a customized model that will offer efficient solutions for educators, instructors, and individuals engaged in text-based evaluations. A reliable and efficient tool for generating questions and answers can free up valuable time and resources, thus streamlining their evaluation processes.}
• 摘要：\摘要{在教育领域，学生评估具有与传授知识相同的意义。待评估，学生通常需要采用基于文本的学术评估方法。讲师需要提出各种各样的问题，这些问题需要使所有学生证明他们对特定主题的适当性。由于他们可能需要手动浏览几种不同的演讲材料，因此这可能是很具有挑战性的。我们的目标是通过使用微型生成LLM实现自动质疑答案 /（AQAG），使整个过程变得更加容易。为了定制讲师的首选问题样式（MCQ，概念或事实问题），正在使用及时工程（PE）。在这项研究中，我们建议在NLP中利用无监督的学习方法，主要关注英语。这种方法赋予基本元路2-7B模型，将种族数据集集成为微调过程的培训数据。创建一个定制模型，该模型将为从事基于文本评估的教育者，讲师和个人提供有效的解决方案。一个可靠，有效的工具来产生问题和答案，可以释放宝贵的时间和资源，从而简化其评估过程。}

Title: Improving Low-Resource Translation with Dictionary-Guided Fine-Tuning and RL: A Spanish-to-Wayuunaiki Study

• Authors: Manuel Mosquera, Melissa Robles, Johan Rodriguez, Ruben Manrique
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19481
• Pdf URL: https://arxiv.org/pdf/2508.19481
• Copy Paste: [[2508.19481]] Improving Low-Resource Translation with Dictionary-Guided Fine-Tuning and RL: A Spanish-to-Wayuunaiki Study(https://arxiv.org/abs/2508.19481)
• Keywords: language model, llm
• Abstract: Low-resource machine translation remains a significant challenge for large language models (LLMs), which often lack exposure to these languages during pretraining and have limited parallel data for fine-tuning. We propose a novel approach that enhances translation for low-resource languages by integrating an external dictionary tool and training models end-to-end using reinforcement learning, in addition to supervised fine-tuning. Focusing on the Spanish-Wayuunaiki language pair, we frame translation as a tool-augmented decision-making problem in which the model can selectively consult a bilingual dictionary during generation. Our method combines supervised instruction tuning with Guided Reward Policy Optimization (GRPO), enabling the model to learn both when and how to use the tool effectively. BLEU similarity scores are used as rewards to guide this learning process. Preliminary results show that our tool-augmented models achieve up to +3.37 BLEU improvement over previous work, and a 18% relative gain compared to a supervised baseline without dictionary access, on the Spanish-Wayuunaiki test set from the AmericasNLP 2025 Shared Task. We also conduct ablation studies to assess the effects of model architecture and training strategy, comparing Qwen2.5-0.5B-Instruct with other models such as LLaMA and a prior NLLB-based system. These findings highlight the promise of combining LLMs with external tools and the role of reinforcement learning in improving translation quality in low-resource language settings.
• 摘要：对于大型语言模型（LLM），低资源机器翻译仍然是一个重大挑战，该模型通常在预处理过程中缺乏这些语言，并且并行数据以进行微调。我们提出了一种新颖的方法，该方法通过使用强化的微调来整合使用外部字典工具和端到端的培训模型，从而增强低资源语言的翻译。为了关注西班牙语的语言对，我们将翻译作为一种工具增强的决策问题，在该问题中，模型可以选择性地咨询双语词典。我们的方法将监督指令调整与有指导的奖励政策优化（GRPO）相结合，使模型能够有效地学习何时以及如何有效使用该工具。 BLEU相似性分数被用作指导这一学习过程的奖励。初步结果表明，我们的工具增强模型比以前的工作实现了+3.37 BLEU的改进，与没有词典访问的监督基线相比，相对增益为18％，在Americasnlp 2025 2025共享任务上的西班牙语 - 道路uunaiki测试集上。我们还进行了消融研究，以评估模型架构和培训策略的影响，将QWEN2.5-0.5B教学与其他模型（例如Llama和先前的基于NLLB的系统）进行比较。这些发现突出了将LLM与外部工具结合在一起的希望以及强化学习在改善低资源语言设置中翻译质量方面的作用。

Title: Rule Synergy Analysis using LLMs: State of the Art and Implications

• Authors: Bahar Bateni, Benjamin Pratt, Jim Whitehead
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19484
• Pdf URL: https://arxiv.org/pdf/2508.19484
• Copy Paste: [[2508.19484]] Rule Synergy Analysis using LLMs: State of the Art and Implications(https://arxiv.org/abs/2508.19484)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated strong performance across a variety of domains, including logical reasoning, mathematics, and more. In this paper, we investigate how well LLMs understand and reason about complex rule interactions in dynamic environments, such as card games. We introduce a dataset of card synergies from the game Slay the Spire, where pairs of cards are classified based on their positive, negative, or neutral interactions. Our evaluation shows that while LLMs excel at identifying non-synergistic pairs, they struggle with detecting positive and, particularly, negative synergies. We categorize common error types, including issues with timing, defining game states, and following game rules. Our findings suggest directions for future research to improve model performance in predicting the effect of rules and their interactions.
• 摘要：大型语言模型（LLMS）已在各种领域（包括逻辑推理，数学等）表现出强大的性能。在本文中，我们研究了LLM在动态环境（例如卡游戏）中有关复杂规则相互作用的理解和理由。我们从游戏杀死了尖峰的卡片协同作用数据集，其中成对的卡片是根据它们的正，负或中性相互作用进行分类的。我们的评估表明，尽管LLM擅长识别非相对性对，但他们在检测正面，尤其是负协同作用方面挣扎。我们对常见错误类型进行分类，包括定时问题，定义游戏状态以及以下游戏规则。我们的发现提出了未来研究的方向，以改善模型绩效，以预测规则的影响及其相互作用。

Title: Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding

• Authors: Bowen Sun, Yujun Cai, Ming-Hsuan Yang, Yiwei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19529
• Pdf URL: https://arxiv.org/pdf/2508.19529
• Copy Paste: [[2508.19529]] Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding(https://arxiv.org/abs/2508.19529)
• Keywords: language model
• Abstract: Discrete diffusion language models have shown strong potential for text generation, yet standard supervised fine-tuning (SFT) misaligns with their semi-autoregressive inference: training randomly masks tokens across the entire response, while inference generates fixed-size blocks sequentially. This mismatch introduces noisy prefixes and leaky suffixes, biasing gradients away from the desired blockwise likelihood. We propose Blockwise SFT, which partitions responses into fixed-size blocks, selects one active block per step for stochastic masking, freezes all preceding tokens, and fully hides future ones. Loss is computed only over the active block, directly mirroring the blockwise decoding process. Experiments on GSM8K, MATH, and MetaMathQA show consistent gains over classical SFT under equal compute or token budgets. Block size consistency studies and ablations confirm that improvements stem from faithful training-inference alignment rather than incidental masking effects. Our results highlight the importance of matching supervision granularity to the decoding procedure in diffusion-based language models.
• 摘要：离散的扩散语言模型显示出强大的文本生成潜力，但标准监督的微调（SFT）通过其半自动进程推断：训练在整个响应中随机掩盖了代币，而推理会依次生成固定尺寸的块。这种不匹配引入了嘈杂的前缀和后缀，使梯度偏离所需的块状可能性。我们提出了将响应分配到固定尺寸块中的块SFT，每个步骤选择一个活动块，以进行随机掩蔽，冻结所有之前的令牌，并完全掩盖未来的屏蔽。仅在活动块上计算损耗，直接反映了块解码过程。在GSM8K，数学和Metamathqa上进行的实验在相等的计算或代币预算下显示出对经典SFT的一致性。块大小的一致性研究和消融证实，改善源于忠实的训练推动对准而不是偶然的掩盖效应。我们的结果强调了将监督粒度与基于扩散的语言模型中解码过程相匹配的重要性。

Title: Alignment with Fill-In-the-Middle for Enhancing Code Generation

• Authors: Houxing Ren, Zimu Lu, Weikang Shi, Haotian Hou, Yunqiao Yang, Ke Wang, Aojun Zhou, Junting Pan, Mingjie Zhan, Hongsheng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19532
• Pdf URL: https://arxiv.org/pdf/2508.19532
• Copy Paste: [[2508.19532]] Alignment with Fill-In-the-Middle for Enhancing Code Generation(https://arxiv.org/abs/2508.19532)
• Keywords: language model, llm
• Abstract: The code generation capabilities of Large Language Models (LLMs) have advanced applications like tool invocation and problem-solving. However, improving performance in code-related tasks remains challenging due to limited training data that is verifiable with accurate test cases. While Direct Preference Optimization (DPO) has shown promise, existing methods for generating test cases still face limitations. In this paper, we propose a novel approach that splits code snippets into smaller, granular blocks, creating more diverse DPO pairs from the same test cases. Additionally, we introduce the Abstract Syntax Tree (AST) splitting and curriculum training method to enhance the DPO training. Our approach demonstrates significant improvements in code generation tasks, as validated by experiments on benchmark datasets such as HumanEval (+), MBPP (+), APPS, LiveCodeBench, and BigCodeBench. Code and data are available at this https URL.
• 摘要：大语言模型（LLM）的代码生成功能具有高级应用程序，例如工具调用和解决问题。但是，由于训练数据有限，可以通过准确的测试用例来证实，因此与代码相关任务的性能提高仍然具有挑战性。虽然直接偏好优化（DPO）已显示出希望，但现有的生成测试用例的方法仍面临限制。在本文中，我们提出了一种新颖的方法，该方法将代码片段拆分为较小的颗粒块，从相同的测试案例中产生更多样化的DPO对。此外，我们介绍了抽象的语法树（AST）分裂和课程培训方法，以增强DPO培训。我们的方法证明了代码生成任务的显着改进，这是通过基准数据集（例如HumaneVal（+），MBPP（+），Apps，LiveCodeBench和BigCodeBench等实验验证的。代码和数据可在此HTTPS URL上找到。

Title: Emotion Transfer with Enhanced Prototype for Unseen Emotion Recognition in Conversation

• Authors: Kun Peng, Cong Cao, Hao Peng, Guanlin Wu, Zhifeng Hao, Lei Jiang, Yanbing Liu, Philip S. Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19533
• Pdf URL: https://arxiv.org/pdf/2508.19533
• Copy Paste: [[2508.19533]] Emotion Transfer with Enhanced Prototype for Unseen Emotion Recognition in Conversation(https://arxiv.org/abs/2508.19533)
• Keywords: llm
• Abstract: Current Emotion Recognition in Conversation (ERC) research follows a closed-domain assumption. However, there is no clear consensus on emotion classification in psychology, which presents a challenge for models when it comes to recognizing previously unseen emotions in real-world applications. To bridge this gap, we introduce the Unseen Emotion Recognition in Conversation (UERC) task for the first time and propose ProEmoTrans, a solid prototype-based emotion transfer framework. This prototype-based approach shows promise but still faces key challenges: First, implicit expressions complicate emotion definition, which we address by proposing an LLM-enhanced description approach. Second, utterance encoding in long conversations is difficult, which we tackle with a proposed parameter-free mechanism for efficient encoding and overfitting prevention. Finally, the Markovian flow nature of emotions is hard to transfer, which we address with an improved Attention Viterbi Decoding (AVD) method to transfer seen emotion transitions to unseen emotions. Extensive experiments on three datasets show that our method serves as a strong baseline for preliminary exploration in this new area.
• 摘要：当前的对话中的情绪识别（ERC）研究遵循封闭域的假设。但是，心理学中的情绪分类尚无明确的共识，这在识别现实世界应用中以前看不见的情绪方面给模型带来了挑战。为了弥合这一差距，我们首次在对话（UERC）任务中介绍了看不见的情感识别，并提出了Proemotrans，这是一个基于原型的情感转移框架。这种基于原型的方法表现出希望，但仍然面临着关键挑战：首先，隐式表达使情绪定义复杂化，我们通过提出一种LLM增强描述方法来解决。其次，在长时间对话中编码的话语很困难，我们使用提出的无参数机制来对其进行有效编码和过度拟合预防。最后，马尔可维亚情绪的流动性质很难转移，我们通过改进的注意力viterbi解码（AVD）方法来解决这一问题，以将所见情感过渡转移到看不见的情绪。三个数据集的大量实验表明，我们的方法是该新领域初步探索的强大基线。

Title: Language Models Identify Ambiguities and Exploit Loopholes

• Authors: Jio Choi, Mohit Bansal, Elias Stengel-Eskin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19546
• Pdf URL: https://arxiv.org/pdf/2508.19546
• Copy Paste: [[2508.19546]] Language Models Identify Ambiguities and Exploit Loopholes(https://arxiv.org/abs/2508.19546)
• Keywords: language model, llm
• Abstract: Studying the responses of large language models (LLMs) to loopholes presents a two-fold opportunity. First, it affords us a lens through which to examine ambiguity and pragmatics in LLMs, since exploiting a loophole requires identifying ambiguity and performing sophisticated pragmatic reasoning. Second, loopholes pose an interesting and novel alignment problem where the model is presented with conflicting goals and can exploit ambiguities to its own advantage. To address these questions, we design scenarios where LLMs are given a goal and an ambiguous user instruction in conflict with the goal, with scenarios covering scalar implicature, structural ambiguities, and power dynamics. We then measure different models' abilities to exploit loopholes to satisfy their given goals as opposed to the goals of the user. We find that both closed-source and stronger open-source models can identify ambiguities and exploit their resulting loopholes, presenting a potential AI safety risk. Our analysis indicates that models which exploit loopholes explicitly identify and reason about both ambiguity and conflicting goals.
• 摘要：研究大语模型（LLM）对漏洞的反应带来了两个机会。首先，它为我们提供了一个镜头，可以通过LLM中的歧义和实用主义，因为利用漏洞需要识别歧义并执行复杂的务实推理。其次，漏洞构成了一个有趣而新颖的对齐问题，在该问题中，该模型具有相互矛盾的目标，并可以利用歧义为其自身的优势。为了解决这些问题，我们设计了在与目标冲突中为LLM的目标和模棱两可的用户指导的情况，场景涵盖了标量隐含，结构性歧义和权力动态。然后，我们测量不同模型的能力来利用漏洞以满足其给定的目标而不是用户的目标。我们发现，封闭源和更强的开源模型都可以识别歧义并利用其产生的漏洞，从而带来潜在的AI安全风险。我们的分析表明，利用漏洞的模型明确识别和理由歧义和冲突目标。

Title: Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts

• Authors: Jiaqi Deng, Yuho Lee, Nicole Hee-Yeon Kim, Hyangsuk Min, Taewon Yun, Minjeong Ban, Kim Yul, Hwanjun Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19578
• Pdf URL: https://arxiv.org/pdf/2508.19578
• Copy Paste: [[2508.19578]] Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts(https://arxiv.org/abs/2508.19578)
• Keywords: language model, llm
• Abstract: We introduce HAMLET, a holistic and automated framework for evaluating the long-context comprehension of large language models (LLMs). HAMLET structures source texts into a three-level key-fact hierarchy at root-, branch-, and leaf-levels, and employs query-focused summarization to evaluate how well models recall and faithfully represent information at each level. To validate the reliability of our fully automated pipeline, we conduct a systematic human study, showing that our automatic evaluation achieves over 90% agreement with expert human judgments, while reducing the cost by up to 25 times. HAMLET reveals that LLMs struggle with fine-grained comprehension, especially at the leaf level, and are sensitive to positional effects like the lost-in-the-middle. Analytical queries pose greater challenges than narrative ones, and consistent performance gaps emerge between open-source and proprietary models, as well as across model scales. Our code and dataset are publicly available at this https URL.
• 摘要：我们介绍了Hamlet，这是一个整体且自动化的框架，用于评估大型语言模型（LLMS）的长篇文章理解。小村庄将源文本源为根，分支和叶子级的三级钥匙范围层次结构，并采用以查询为中心的摘要来评估模型在每个级别上的回忆和忠实表示信息的很好。为了验证我们全自动管道的可靠性，我们进行了一项系统的人类研究，表明我们的自动评估与专家人类判断的一致性超过90％，同时将成本降低了25倍。哈姆雷特（Hamlet）透露，LLMS与细粒度的理解力，尤其是在叶子水平上挣扎，并且对诸如中间失落的位置效应敏感。分析查询比叙事质量提出的挑战更大，并且在开源模型和专有模型之间以及跨模型量表之间出现了一致的性能差距。我们的代码和数据集可在此HTTPS URL上公开获得。

Title: ArgCMV: An Argument Summarization Benchmark for the LLM-era

• Authors: Omkar Gurjar, Agam Goyal, Eshwar Chandrasekharan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19580
• Pdf URL: https://arxiv.org/pdf/2508.19580
• Copy Paste: [[2508.19580]] ArgCMV: An Argument Summarization Benchmark for the LLM-era(https://arxiv.org/abs/2508.19580)
• Keywords: language model, llm
• Abstract: Key point extraction is an important task in argument summarization which involves extracting high-level short summaries from arguments. Existing approaches for KP extraction have been mostly evaluated on the popular ArgKP21 dataset. In this paper, we highlight some of the major limitations of the ArgKP21 dataset and demonstrate the need for new benchmarks that are more representative of actual human conversations. Using SoTA large language models (LLMs), we curate a new argument key point extraction dataset called ArgCMV comprising of around 12K arguments from actual online human debates spread across over 3K topics. Our dataset exhibits higher complexity such as longer, co-referencing arguments, higher presence of subjective discourse units, and a larger range of topics over ArgKP21. We show that existing methods do not adapt well to ArgCMV and provide extensive benchmark results by experimenting with existing baselines and latest open source models. This work introduces a novel KP extraction dataset for long-context online discussions, setting the stage for the next generation of LLM-driven summarization research.
• 摘要：要点提取是参数摘要中的重要任务，它涉及从参数中提取高级简短摘要。现有的KP提取方法主要在流行的ARGKP21数据集上进行了评估。在本文中，我们重点介绍了ARGKP21数据集的一些主要局限性，并证明了对更代表实际人类对话的新基准的需求。我们使用SOTA大语言模型（LLMS），我们策划了一个新的参数键点提取数据集，称为ARGCMV，其中包括来自实际在线人类辩论的大约12K参数，分布在3K主题上。我们的数据集表现出更高的复杂性，例如更长的，共同引用的论点，更高的主观话语单元的存在以及与ARGKP21相比的主题范围更大。我们表明，现有方法不能很好地适应ARGCMV，并通过尝试现有基线和最新开源模型来提供广泛的基准结果。这项工作介绍了一个新颖的KP提取数据集，用于长篇文化在线讨论，为下一代LLM驱动的摘要研究奠定了基础。

Title: Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs

• Authors: Jun Bai, Minghao Tong, Yang Liu, Zixia Jia, Zilong Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19594
• Pdf URL: https://arxiv.org/pdf/2508.19594
• Copy Paste: [[2508.19594]] Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs(https://arxiv.org/abs/2508.19594)
• Keywords: language model, llm
• Abstract: Context faithfulness is essential for reliable reasoning in context-dependent scenarios. However, large language models often struggle to ground their outputs in the provided context, resulting in irrelevant responses. Inspired by the emergent expert specialization observed in mixture-of-experts architectures, this work investigates whether certain experts exhibit specialization in context utilization, offering a potential pathway toward targeted optimization for improved context faithfulness. To explore this, we propose Router Lens, a method that accurately identifies context-faithful experts. Our analysis reveals that these experts progressively amplify attention to relevant contextual information, thereby enhancing context grounding. Building on this insight, we introduce Context-faithful Expert Fine-Tuning (CEFT), a lightweight optimization approach that selectively fine-tunes context-faithful experts. Experiments across a wide range of benchmarks and models demonstrate that CEFT matches or surpasses the performance of full fine-tuning while being significantly more efficient.
• 摘要：上下文忠诚对于与上下文相关的场景中的可靠推理至关重要。但是，大型语言模型经常在提供的环境中努力将其产出扎根，从而产生无关的响应。受到专家专业专业的启发，这项工作调查了某些专家是否在上下文利用中表现出专业化，从而提供了针对性优化的潜在途径，以改善上下文忠诚。为了探索这一点，我们建议使用路由器镜头，该方法可以准确地识别上下文信仰专家。我们的分析表明，这些专家逐渐扩大了对相关上下文信息的关注，从而增强了背景基础。在此洞察力的基础上，我们介绍了上下文信仰专家微调（CEFT），这是一种轻巧的优化方法，有选择地微调上下文忠诚的专家。跨广泛的基准和模型进行的实验表明，CEFT匹配或超过了完整的微调性能，同时效率更高。

Title: LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation

• Authors: Yang Sun, Lixin Zou, Dan Luo, Zhiyong Xie, Long Zhang, Liming Dong, Yunwei Zhao, Xixun Lin, Yanxiong Lu, Chenliang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19614
• Pdf URL: https://arxiv.org/pdf/2508.19614
• Copy Paste: [[2508.19614]] LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation(https://arxiv.org/abs/2508.19614)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) incorporates external knowledge into large language models (LLMs), improving their adaptability to downstream tasks and enabling information updates. Surprisingly, recent empirical evidence demonstrates that injecting noise into retrieved relevant documents paradoxically facilitates exploitation of external knowledge and improves generation quality. Although counterintuitive and challenging to apply in practice, this phenomenon enables granular control and rigorous analysis of how LLMs integrate external knowledge. Therefore, in this paper, we intervene on noise injection and establish a layer-specific functional demarcation within the LLM: shallow layers specialize in local context modeling, intermediate layers focus on integrating long-range external factual knowledge, and deeper layers primarily rely on parametric internal knowledge. Building on this insight, we propose Layer Fused Decoding (LFD), a simple decoding strategy that directly combines representations from an intermediate layer with final-layer decoding outputs to fully exploit the external factual knowledge. To identify the optimal intermediate layer, we introduce an internal knowledge score (IKS) criterion that selects the layer with the lowest IKS value in the latter half of layers. Experimental results across multiple benchmarks demonstrate that LFD helps RAG systems more effectively surface retrieved context knowledge with minimal cost.
• 摘要：检索增强的生成（RAG）将外部知识纳入大语言模型（LLMS），改善了其对下游任务的适应性并启用信息更新。令人惊讶的是，最近的经验证据表明，将噪声注入矛盾的文件中，可以偶然地促进对外部知识的开采并提高发电质量。尽管在实践中适用违反直觉和挑战，但这种现象可以对LLMS整合外部知识的整合方式进行颗粒状的控制和严格分析。因此，在本文中，我们干预噪声注入，并在LLM中建立特定于层的功能分界：浅层层专门研究局部上下文建模，中间层专注于整合远程外部事实知识，而更深的层主要依赖于参数内部知识。在此洞察力的基础上，我们提出了层融合解码（LFD），这是一种简单的解码策略，它直接将中间层的表示形式与最终的解码输出相结合，以充分利用外部事实知识。为了确定最佳中间层，我们引入了一个内部知识分数（IKS）标准，该标准选择了后半层中IKS值最低的层。多个基准测试的实验结果表明，LFD帮助更有效地表面表面检索了上下文知识，而成本最低。

Title: A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detection

• Authors: Chong Tian, Qirong Ho, Xiuying Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19633
• Pdf URL: https://arxiv.org/pdf/2508.19633
• Copy Paste: [[2508.19633]] A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detection(https://arxiv.org/abs/2508.19633)
• Keywords: llm, prompt, agent
• Abstract: Rapid LLM advancements heighten fake news risks by enabling the automatic generation of increasingly sophisticated misinformation. Previous detection methods, including fine-tuned small models or LLM-based detectors, often struggle with its dynamically evolving nature. In this work, we propose a novel framework called the Symbolic Adversarial Learning Framework (SALF), which implements an adversarial training paradigm by an agent symbolic learning optimization process, rather than relying on numerical updates. SALF introduces a paradigm where the generation agent crafts deceptive narratives, and the detection agent uses structured debates to identify logical and factual flaws for detection, and they iteratively refine themselves through such adversarial interactions. Unlike traditional neural updates, we represent agents using agent symbolic learning, where learnable weights are defined by agent prompts, and simulate back-propagation and gradient descent by operating on natural language representations of weights, loss, and gradients. Experiments on two multilingual benchmark datasets demonstrate SALF's effectiveness, showing it generates sophisticated fake news that degrades state-of-the-art detection performance by up to 53.4% in Chinese and 34.2% in English on average. SALF also refines detectors, improving detection of refined content by up to 7.7%. We hope our work inspires further exploration into more robust, adaptable fake news detection systems.
• 摘要：快速LLM的进步通过实现越来越复杂的错误信息来提高假新闻的风险。先前的检测方法，包括微调的小型模型或基于LLM的检测器，通常在动态发展的性质上挣扎。在这项工作中，我们提出了一个名为“符号对手学习框架（SALF）”的新颖框架，该框架通过代理符号学习优化过程实现了对抗性培训范式，而不是依靠数值更新。萨尔夫（Salf）引入了一个范式，一代代理人在其中制作了欺骗性的叙事，检测剂使用结构化的辩论来识别逻辑和事实缺陷以进行检测，并且它们通过这种对抗性相互作用进行了迭代的改进。与传统的神经更新不同，我们使用代理符号学习来代表代理人，在座位提示中定义了可学习的权重，并通过在权重，损失和梯度的自然语言表示上进行模拟后传播和梯度下降。两个多语言基准数据集的实验证明了萨尔夫的有效性，这表明它产生了复杂的假新闻，使最新的检测性能降低了53.4％的中文，平均为34.2％的英语。萨尔夫还改善了检测器，将精制含量的检测提高了多达7.7％。我们希望我们的工作激发了进一步的探索，以进行更健壮，适应性的假新闻检测系统。

Title: Survey of Specialized Large Language Model

• Authors: Chenghan Yang, Ruiyu Zhao, Yang Liu, Ling Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19667
• Pdf URL: https://arxiv.org/pdf/2508.19667
• Copy Paste: [[2508.19667]] Survey of Specialized Large Language Model(https://arxiv.org/abs/2508.19667)
• Keywords: language model, llm, agent
• Abstract: The rapid evolution of specialized large language models (LLMs) has transitioned from simple domain adaptation to sophisticated native architectures, marking a paradigm shift in AI development. This survey systematically examines this progression across healthcare, finance, legal, and technical domains. Besides the wide use of specialized LLMs, technical breakthrough such as the emergence of domain-native designs beyond fine-tuning, growing emphasis on parameter efficiency through sparse computation and quantization, increasing integration of multimodal capabilities and so on are applied to recent LLM agent. Our analysis reveals how these innovations address fundamental limitations of general-purpose LLMs in professional applications, with specialized models consistently performance gains on domain-specific benchmarks. The survey further highlights the implications for E-Commerce field to fill gaps in the field.
• 摘要：专业大语言模型（LLM）的快速演变已经从简单的域适应到复杂的本机体系结构过渡，这标志着AI开发的范式转移。这项调查系统地研究了整个医疗保健，金融，法律和技术领域的进展。除了广泛使用专门的LLM外，技术突破，例如超出微调的领域本地设计的出现，通过稀疏计算和量化对参数效率的越来越重视，增加了多模式能力的整合，等等依次应用于最近的LLM代理。我们的分析揭示了这些创新如何解决专业应用中通用LLM的基本局限性，专业模型始终在特定领域的基准上获得性能提高。该调查进一步凸显了对电子商务领域填补该领域空白的影响。

Title: Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models

• Authors: Yilin Wang, Heng Wang, Yuyang Bai, Minnan Luo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19720
• Pdf URL: https://arxiv.org/pdf/2508.19720
• Copy Paste: [[2508.19720]] Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models(https://arxiv.org/abs/2508.19720)
• Keywords: language model, llm
• Abstract: In Large Language Models (LLMs) generation, there exist knowledge conflicts and scenarios where parametric knowledge contradicts knowledge provided in the context. Previous works studied tuning, decoding algorithms, or locating and editing context-aware neurons to adapt LLMs to be faithful to new contextual knowledge. However, they are usually inefficient or ineffective for large models, not workable for black-box models, or unable to continuously adjust LLMs' sensitivity to the knowledge provided in the context. To mitigate these problems, we propose CSKS (Continuously Steering Knowledge Sensitivity), a simple framework that can steer LLMs' sensitivity to contextual knowledge continuously at a lightweight cost. Specifically, we tune two small LMs (i.e. proxy models) and use the difference in their output distributions to shift the original distribution of an LLM without modifying the LLM weights. In the evaluation process, we not only design synthetic data and fine-grained metrics to measure models' sensitivity to contextual knowledge but also use a real conflict dataset to validate CSKS's practical efficacy. Extensive experiments demonstrate that our framework achieves continuous and precise control over LLMs' sensitivity to contextual knowledge, enabling both increased sensitivity and reduced sensitivity, thereby allowing LLMs to prioritize either contextual or parametric knowledge as needed flexibly. Our data and code are available at this https URL.
• 摘要：在大型语言模型（LLMS）的一代中，存在知识冲突和场景，其中参数知识与上下文中提供的知识相矛盾。以前的作品研究了调整，解码算法或定位和编辑上下文感知的神经元，以使LLMS忠于新的上下文知识。但是，对于大型模型而言，它们通常是低效或无效的，对于黑盒模型而言是不可行的，或者无法不断调整LLMS对上下文中提供的知识的敏感性。为了减轻这些问题，我们提出了CSK（不断转向知识敏感性），一个简单的框架可以以轻量级的成本连续地引导LLMS对上下文知识的敏感性。具体而言，我们调整两个小LM（即代理模型），并使用其输出分布的差异来移动LLM的原始分布而不修改LLM权重。在评估过程中，我们不仅设计合成数据和细粒度指标来衡量模型对上下文知识的敏感性，而且还使用真实的冲突数据集来验证CSK的实践效果。广泛的实验表明，我们的框架可以对LLMS对上下文知识的敏感性进行连续，精确的控制，从而提高灵敏度和降低灵敏度，从而使LLMS可以灵活地根据需要优先考虑上下文或参数知识。我们的数据和代码可在此HTTPS URL上找到。

Title: NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks

• Authors: Aritra Dutta, Swapnanil Mukherjee, Deepanway Ghosal, Somak Aditya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19724
• Pdf URL: https://arxiv.org/pdf/2508.19724
• Copy Paste: [[2508.19724]] NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks(https://arxiv.org/abs/2508.19724)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Commonsense visual-question answering often hinges on knowledge that is missing from the image or the question. Small vision-language models (sVLMs) such as ViLT, VisualBERT and FLAVA therefore lag behind their larger generative counterparts. To study the effect of careful commonsense knowledge integration on sVLMs, we present an end-to-end framework (NLKI) that (i) retrieves natural language facts, (ii) prompts an LLM to craft natural language explanations, and (iii) feeds both signals to sVLMs respectively across two commonsense VQA datasets (CRIC, AOKVQA) and a visual-entailment dataset (e-SNLI-VE). Facts retrieved using a fine-tuned ColBERTv2 and an object information-enriched prompt yield explanations that largely cut down hallucinations, while lifting the end-to-end answer accuracy by up to 7% (across 3 datasets), making FLAVA and other models in NLKI match or exceed medium-sized VLMs such as Qwen-2 VL-2B and SmolVLM-2.5B. As these benchmarks contain 10-25% label noise, additional finetuning using noise-robust losses (such as symmetric cross entropy and generalised cross entropy) adds another 2.5% in CRIC, and 5.5% in AOKVQA. Our findings expose when LLM-based commonsense knowledge beats retrieval from commonsense knowledge bases, how noise-aware training stabilises small models in the context of external knowledge augmentation, and why parameter-efficient commonsense reasoning is now within reach for 250M models.
• 摘要：常识性视觉问题的回答通常取决于图像或问题所缺少的知识。小型视觉模型（SVLM），例如Vilt，Visualbert和Flava，因此落后于其较大的生成性对应物。 To study the effect of careful commonsense knowledge integration on sVLMs, we present an end-to-end framework (NLKI) that (i) retrieves natural language facts, (ii) prompts an LLM to craft natural language explanations, and (iii) feeds both signals to sVLMs respectively across two commonsense VQA datasets (CRIC, AOKVQA) and a visual-entailment dataset （e-snli-ve）。 Facts retrieved using a fine-tuned ColBERTv2 and an object information-enriched prompt yield explanations that largely cut down hallucinations, while lifting the end-to-end answer accuracy by up to 7% (across 3 datasets), making FLAVA and other models in NLKI match or exceed medium-sized VLMs such as Qwen-2 VL-2B and SmolVLM-2.5B.由于这些基准含有10-25％的标签噪声，因此使用噪声损失（例如对称的交叉熵和广义横熵）的额外填充量增加了CRIC的2.5％，而AOKVQA则增加了5.5％。当基于LLM的常识知识从常识性知识基础中检索，噪音吸引力的训练如何在外部知识增强背景下稳定小型模型以及为何参数有效的常识性推理现在可以实现250m的模型时，我们的发现就会暴露出来。

Title: Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval

• Authors: Wenhao Li, Yuxin Zhang, Gen Luo, Haiyuan Wan, Ziyang Gong, Fei Chao, Rongrong Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19740
• Pdf URL: https://arxiv.org/pdf/2508.19740
• Copy Paste: [[2508.19740]] Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval(https://arxiv.org/abs/2508.19740)
• Keywords: language model, llm
• Abstract: Reducing the key-value (KV) cache burden in Large Language Models (LLMs) significantly accelerates inference. Dynamically selecting critical KV caches during decoding helps maintain performance. Existing methods use random linear hashing to identify important tokens, but this approach is inefficient due to the orthogonal distribution of queries and keys within two narrow cones in LLMs. We introduce Spotlight Attention, a novel method that employs non-linear hashing functions to optimize the embedding distribution of queries and keys, enhancing coding efficiency and robustness. We also developed a lightweight, stable training framework using a Bradley-Terry ranking-based loss, enabling optimization of the non-linear hashing module on GPUs with 16GB memory in 8 hours. Experimental results show that Spotlight Attention drastically improves retrieval precision while shortening the length of the hash code at least 5$\times$ compared to traditional linear hashing. Finally, we exploit the computational advantages of bitwise operations by implementing specialized CUDA kernels, achieving hashing retrieval for 512K tokens in under 100$\mu$s on a single A100 GPU, with end-to-end throughput up to 3$\times$ higher than vanilla decoding.
• 摘要：在大语言模型（LLMS）中减少键值（KV）缓存负担可显着加速推理。在解码过程中动态选择关键的KV缓存有助于保持性能。现有方法使用随机线性散列来识别重要令牌，但是由于LLMS中两个狭窄锥体内的查询和键的正交分布，这种方法效率低下。我们引入了Spotlight Guate，这是一种新型方法，它采用非线性散列功能来优化查询和键的嵌入分布，从而提高编码效率和鲁棒性。我们还使用基于Bradley-Terry排名的损失开发了轻巧，稳定的训练框架，从而在8小时内使用16GB内存来优化GPU上的非线性散列模块。实验结果表明，与传统的线性哈希相比，聚焦注意力的重点大大提高了检索精度，同时缩短了至少5 $ \ times $的长度。最后，我们通过实施专门的CUDA内核来利用位操作的计算优势，在单个A100 GPU上，在100美元以下的100美元以下的hashing检索中获得了512k代币的哈希检索，其端到端吞吐量高达3美元$ \ timper $ \ timper $ \ tim $ \ tim $ \倍。

Title: Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance

• Authors: Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19764
• Pdf URL: https://arxiv.org/pdf/2508.19764
• Copy Paste: [[2508.19764]] Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance(https://arxiv.org/abs/2508.19764)
• Keywords: language model, llm, prompt
• Abstract: Expert persona prompting -- assigning roles such as expert in math to language models -- is widely used for task improvement. However, prior work shows mixed results on its effectiveness, and does not consider when and why personas should improve performance. We analyze the literature on persona prompting for task improvement and distill three desiderata: 1) performance advantage of expert personas, 2) robustness to irrelevant persona attributes, and 3) fidelity to persona attributes. We then evaluate 9 state-of-the-art LLMs across 27 tasks with respect to these desiderata. We find that expert personas usually lead to positive or non-significant performance changes. Surprisingly, models are highly sensitive to irrelevant persona details, with performance drops of almost 30 percentage points. In terms of fidelity, we find that while higher education, specialization, and domain-relatedness can boost performance, their effects are often inconsistent or negligible across tasks. We propose mitigation strategies to improve robustness -- but find they only work for the largest, most capable models. Our findings underscore the need for more careful persona design and for evaluation schemes that reflect the intended effects of persona usage.
• 摘要：专家角色提示 - 将数学专家等角色分配给语言模型 - 被广泛用于改进任务。但是，先前的工作表明其有效性的结果不同，并且不考虑角色何时以及为什么要提高性能。我们分析了有关促使任务改进和提取三个desiderata的角色的文献：1）专家角色的性能优势，2）对无关性角色属性的稳健性，以及3）对角色属性的保真度。然后，我们在这些Desiderata的27个任务中评估了9个最先进的LLM。我们发现专家角色通常会导致积极或不重要的性能变化。令人惊讶的是，模型对无关紧要的角色细节高度敏感，性能下降了近30个百分点。在忠诚方面，我们发现，尽管高等教育，专业化和与领域有关的性能可以提高绩效，但它们的影响通常在整个任务中都不一致或可以忽略不计。我们提出了缓解策略来改善鲁棒性 - 但发现它们仅适用于最大，功能最强的模型。我们的发现强调了需要更仔细的角色设计和反映角色使用预期影响的评估方案的必要性。

Title: T2R-bench: A Benchmark for Generating Article-Level Reports from Real World Industrial Tables

• Authors: Jie Zhang, Changzai Pan, Kaiwen Wei, Sishi Xiong, Yu Zhao, Xiangyu Li, Jiaxin Peng, Xiaoyan Gu, Jian Yang, Wenhan Chang, Zhenhe Wu, Jiang Zhong, Shuangyong Song, Yongxiang Li, Xuelong Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19813
• Pdf URL: https://arxiv.org/pdf/2508.19813
• Copy Paste: [[2508.19813]] T2R-bench: A Benchmark for Generating Article-Level Reports from Real World Industrial Tables(https://arxiv.org/abs/2508.19813)
• Keywords: language model, llm
• Abstract: Extensive research has been conducted to explore the capabilities of large language models (LLMs) in table reasoning. However, the essential task of transforming tables information into reports remains a significant challenge for industrial applications. This task is plagued by two critical issues: 1) the complexity and diversity of tables lead to suboptimal reasoning outcomes; and 2) existing table benchmarks lack the capacity to adequately assess the practical application of this task. To fill this gap, we propose the table-to-report task and construct a bilingual benchmark named T2R-bench, where the key information flow from the tables to the reports for this task. The benchmark comprises 457 industrial tables, all derived from real-world scenarios and encompassing 19 industry domains as well as 4 types of industrial tables. Furthermore, we propose an evaluation criteria to fairly measure the quality of report generation. The experiments on 25 widely-used LLMs reveal that even state-of-the-art models like Deepseek-R1 only achieves performance with 62.71 overall score, indicating that LLMs still have room for improvement on T2R-bench. Source code and data will be available after acceptance.
• 摘要：已经进行了广泛的研究，以探索表格推理中大语言模型（LLM）的能力。但是，将表信息转换为报告的基本任务仍然是工业应用的重大挑战。这项任务受到两个关键问题的困扰：1）表的复杂性和多样性导致了次优推理结果； 2）现有的表基准缺乏充分评估此任务的实际应用的能力。为了填补这一空白，我们提出了表格报告任务，并构建一个名为T2R基础的双语基准，其中关键信息从表流到该任务的报告。该基准包括457个工业桌，所有这些都来自现实世界情景，并涵盖了19个行业领域以及4种工业表。此外，我们提出了一个评估标准，以公平地衡量报告生成的质量。在25个广泛使用的LLMS上进行的实验表明，即使是DeepSeek-R1（例如DeepSeek-R1）的最先进模型也只能以62.71的总分来实现性能，这表明LLMS在T2R板凳上仍然有改进的余地。源代码和数据将在接受后可用。

Title: Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning

• Authors: Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, Volker Tresp, Yunpu Ma
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2508.19828
• Pdf URL: https://arxiv.org/pdf/2508.19828
• Copy Paste: [[2508.19828]] Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning(https://arxiv.org/abs/2508.19828)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless, constrained by limited context windows that hinder long-horizon reasoning. Recent efforts to address this limitation often augment LLMs with an external memory bank, yet most existing pipelines are static and heuristic-driven, lacking any learned mechanism for deciding what to store, update, or retrieve. We present Memory-R1, a reinforcement learning (RL) framework that equips LLMs with the ability to actively manage and utilize external memory through two specialized agents: a Memory Manager that learns to perform structured memory operations {ADD, UPDATE, DELETE, NOOP}, and an Answer Agent that selects the most relevant entries and reasons over them to produce an answer. Both agents are fine-tuned with outcome-driven RL (PPO and GRPO), enabling adaptive memory management and use with minimal supervision. With as few as 152 question-answer pairs and a corresponding temporal memory bank for training, Memory-R1 outperforms the most competitive existing baseline and demonstrates strong generalization across diverse question types and LLM backbones. Beyond presenting an effective approach, this work provides insights into how RL can unlock more agentic, memory-aware behaviors in LLMs, pointing toward richer, more persistent reasoning systems.
• 摘要：大型语言模型（LLMS）在广泛的NLP任务中表现出了令人印象深刻的功能，但是它们在根本上仍然无状态，受到阻碍长期推理的有限上下文窗口的限制。最近解决此限制的最新努力通常会通过外部记忆库来增加LLM，但是大多数现有的管道都是静态和启发式驱动的，缺乏确定要存储，更新或检索的内容的任何学识机制。我们提出了Memory-R1，这是一种增强学习（RL）框架，它使LLMS具有通过两种专用代理来积极管理和利用外部内存的能力：一种学习的内存管理器，该内存管理器学会执行结构化的内存操作{添加，更新，删除，noop}，以及一个选择最相关的条目并选择答案的答案，以产生答案。两种代理都通过结果驱动的RL（PPO和GRPO）进行了微调，从而实现了自适应内存管理，并在最少的监督下使用。 Memory-R1仅有152个问答对以及用于培训的相应的时间内存库，比现有的基线最具竞争力，并且在不同的问题类型和LLM骨架上表现出强烈的概括。除了提出有效的方法外，这项工作还提供了有关RL如何在LLM中解锁更多代理，内存感知行为的见解，指出了更丰富，更持久的推理系统。

Title: Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis

• Authors: Anusha Kamath, Kanishk Singla, Rakesh Paul, Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.19831
• Pdf URL: https://arxiv.org/pdf/2508.19831
• Copy Paste: [[2508.19831]] Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis(https://arxiv.org/abs/2508.19831)
• Keywords: language model, llm, chat
• Abstract: Evaluating instruction-tuned Large Language Models (LLMs) in Hindi is challenging due to a lack of high-quality benchmarks, as direct translation of English datasets fails to capture crucial linguistic and cultural nuances. To address this, we introduce a suite of five Hindi LLM evaluation datasets: IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, and BFCL-Hi. These were created using a methodology that combines from-scratch human annotation with a translate-and-verify process. We leverage this suite to conduct an extensive benchmarking of open-source LLMs supporting Hindi, providing a detailed comparative analysis of their current capabilities. Our curation process also serves as a replicable methodology for developing benchmarks in other low-resource languages.
• 摘要：由于缺乏高质量的基准，评估印地语中的指导调整的大语言模型（LLM）是具有挑战性的，因为英语数据集的直接翻译未能捕获至关重要的语言和文化细微差别。为了解决这个问题，我们介绍了五个印地语LLM评估数据集的套件：ifeval-hi，mt-bench-hi，gsm8k-hi，chatrag-hi和bfcl-hi。这些是使用一种方法来创建的，该方法结合了从施法人类注释和翻译和验证过程。我们利用这套套件来对支持印地语的开源LLMS进行广泛的基准测试，从而对其当前功能进行了详细的比较分析。我们的策展过程也是一种可复制的方法，用于开发其他低资源语言的基准。

Title: Scalable and consistent few-shot classification of survey responses using text embeddings

• Authors: Jonas Timmann Mjaaland, Markus Fleten Kreutzer, Halvor Tyseng, Rebeckah K. Fussell, Gina Passante, N.G. Holmes, Anders Malthe-Sørenssen, Tor Ole B. Odden
• Subjects: cs.CL, physics.ed-ph
• Abstract URL: https://arxiv.org/abs/2508.19836
• Pdf URL: https://arxiv.org/pdf/2508.19836
• Copy Paste: [[2508.19836]] Scalable and consistent few-shot classification of survey responses using text embeddings(https://arxiv.org/abs/2508.19836)
• Keywords: language model
• Abstract: Qualitative analysis of open-ended survey responses is a commonly-used research method in the social sciences, but traditional coding approaches are often time-consuming and prone to inconsistency. Existing solutions from Natural Language Processing such as supervised classifiers, topic modeling techniques, and generative large language models have limited applicability in qualitative analysis, since they demand extensive labeled data, disrupt established qualitative workflows, and/or yield variable results. In this paper, we introduce a text embedding-based classification framework that requires only a handful of examples per category and fits well with standard qualitative workflows. When benchmarked against human analysis of a conceptual physics survey consisting of 2899 open-ended responses, our framework achieves a Cohen's Kappa ranging from 0.74 to 0.83 as compared to expert human coders in an exhaustive coding scheme. We further show how performance of this framework improves with fine-tuning of the text embedding model, and how the method can be used to audit previously-analyzed datasets. These findings demonstrate that text embedding-assisted coding can flexibly scale to thousands of responses without sacrificing interpretability, opening avenues for deductive qualitative analysis at scale.
• 摘要：对开放式调查回答的定性分析是社会科学中常用的研究方法，但是传统的编码方法通常是耗时的，易于不一致。从自然语言处理（例如监督分类器，主题建模技术和生成大语言模型）中的现有解决方案在定性分析中的适用性有限，因为它们要求广泛标记的数据，破坏确定的定性工作流以及/或产量可变结果。在本文中，我们介绍了一个基于文本的基于嵌入的分类框架，该框架仅需要每个类别的少数示例，并且非常适合标准定性工作流。当对人类对由2899个开放式响应组成的概念物理调查的分析进行基准分析时，与专家人类编码人员相比，我们的框架的Kappa的Kappa在0.74到0.83之间，在详尽的编码方案中。我们进一步展示了该框架的性能如何通过对文本嵌入模型进行微调以及如何使用该方法来审核先前分析的数据集进行改进。这些发现表明，嵌入辅助编码的文本可以灵活地扩展到数千个响应，而无需牺牲可解释性，为按大规模演绎定性分析开辟了途径。

Title: Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning

• Authors: Vanessa Toborek, Sebastian Müller, Tim Selbach, Tamás Horváth, Christian Bauckhage
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19873
• Pdf URL: https://arxiv.org/pdf/2508.19873
• Copy Paste: [[2508.19873]] Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning(https://arxiv.org/abs/2508.19873)
• Keywords: language model
• Abstract: Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training.
• 摘要：课程学习（CL）旨在通过将数据从“ Easy”到“硬”展示，但定义和衡量语言难度仍然是一个开放的挑战。我们研究了人类策划的简单语言是否可以作为CL的有效信号。使用来自简单的Wikipedia语料库的文章级标签，我们将基于标签的课程与依靠浅启发式方法的基于能力的策略进行了比较。我们使用Bert-Tiny模型进行的实验表明，仅添加简单数据并没有明显的好处。但是，通过课程进行构造（尤其是在首次引入时）始终会改善困惑，尤其是在简单的语言上。相比之下，基于能力的课程对随机排序没有一致的收益，这可能是因为它们无法有效地分开这两个类别。我们的结果表明，关于语言难度的人类直觉可以指导CL进行语言模型预训练。

Title: AI-Powered Detection of Inappropriate Language in Medical School Curricula

• Authors: Chiman Salavati, Shannon Song, Scott A. Hale, Roberto E. Montenegro, Shiri Dori-Hacohen, Fabricio Murai
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2508.19883
• Pdf URL: https://arxiv.org/pdf/2508.19883
• Copy Paste: [[2508.19883]] AI-Powered Detection of Inappropriate Language in Medical School Curricula(https://arxiv.org/abs/2508.19883)
• Keywords: language model, llm, prompt
• Abstract: The use of inappropriate language -- such as outdated, exclusionary, or non-patient-centered terms -- medical instructional materials can significantly influence clinical training, patient interactions, and health outcomes. Despite their reputability, many materials developed over past decades contain examples now considered inappropriate by current medical standards. Given the volume of curricular content, manually identifying instances of inappropriate use of language (IUL) and its subcategories for systematic review is prohibitively costly and impractical. To address this challenge, we conduct a first-in-class evaluation of small language models (SLMs) fine-tuned on labeled data and pre-trained LLMs with in-context learning on a dataset containing approximately 500 documents and over 12,000 pages. For SLMs, we consider: (1) a general IUL classifier, (2) subcategory-specific binary classifiers, (3) a multilabel classifier, and (4) a two-stage hierarchical pipeline for general IUL detection followed by multilabel classification. For LLMs, we consider variations of prompts that include subcategory definitions and/or shots. We found that both LLama-3 8B and 70B, even with carefully curated shots, are largely outperformed by SLMs. While the multilabel classifier performs best on annotated data, supplementing training with unflagged excerpts as negative examples boosts the specific classifiers' AUC by up to 25%, making them most effective models for mitigating harmful language in medical curricula.
• 摘要：使用不当语言（例如过时，排除或非患者术语）的使用医学教学材料会极大地影响临床培训，患者互动和健康结果。尽管具有信誉，但过去几十年来开发的许多材料都包含了现在按照当前医疗标准认为不合适的例子。考虑到课程内容的数量，手动识别语言不当使用（IUL）及其进行系统审查的子类别的实例非常昂贵且不切实际。为了应对这一挑战，我们对标记的数据和预先训练的LLM微调的小语言模型（SLM）进行了一流的评估，并在包含大约500个文档和超过12,000页的数据集上进行了内在学习。对于SLM，我们考虑：（1）一般IUL分类器，（2）特定于子类别的二进制分类器，（3）多标记分类器，以及（4）（4）两级层次层次管道，用于一般IUL检测，然后进行Multilabel分类。对于LLM，我们考虑包括子类别定义和/或拍摄的提示的变化。我们发现，即使经过精心策划的镜头，Llama-3 8b和70b都在很大程度上胜过SLM。尽管多标签分类器在注释的数据上表现最佳，但用无框的摘录补充培训作为负面示例，将特定分类器的AUC提高了25％，这使其成为减轻医疗课程中有害语言的最有效模型。

Title: Bangla-Bayanno: A 52K-Pair Bengali Visual Question Answering Dataset with LLM-Assisted Translation Refinement

• Authors: Mohammed Rakibul Hasan, Rafi Majid, Ahanaf Tahmid
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2508.19887
• Pdf URL: https://arxiv.org/pdf/2508.19887
• Copy Paste: [[2508.19887]] Bangla-Bayanno: A 52K-Pair Bengali Visual Question Answering Dataset with LLM-Assisted Translation Refinement(https://arxiv.org/abs/2508.19887)
• Keywords: llm
• Abstract: In this paper, we introduce Bangla-Bayanno, an open-ended Visual Question Answering (VQA) Dataset in Bangla, a widely used, low-resource language in multimodal AI research. The majority of existing datasets are either manually annotated with an emphasis on a specific domain, query type, or answer type or are constrained by niche answer formats. In order to mitigate human-induced errors and guarantee lucidity, we implemented a multilingual LLM-assisted translation refinement pipeline. This dataset overcomes the issues of low-quality translations from multilingual sources. The dataset comprises 52,650 question-answer pairs across 4750+ images. Questions are classified into three distinct answer types: nominal (short descriptive), quantitative (numeric), and polar (yes/no). Bangla-Bayanno provides the most comprehensive open-source, high-quality VQA benchmark in Bangla, aiming to advance research in low-resource multimodal learning and facilitate the development of more inclusive AI systems.
• 摘要：在本文中，我们介绍了孟加拉国的一个开放式视觉问题答案（VQA）数据集，这是一种多模式AI研究中广泛使用的低资源语言。现有的大多数数据集都是手动注释的，重点是特定域，查询类型或答案类型，或者受到利基答案格式的约束。为了减轻人类引起的错误并确保清醒性，我们实施了多种语言LLM辅助翻译简化管道。该数据集克服了来自多语言来源的低质量翻译问题。该数据集包含4750多个图像的52,650个问答对。问题分为三种不同的答案类型：名义（简短描述性），定量（数字）和极性（是/否）。 Bangla-Bayanno在孟加拉国提供了最全面的开源，高质量的VQA基准，旨在推进低资源多模式学习的研究，并促进更具包容性AI系统的开发。

Title: Logical Reasoning with Outcome Reward Models for Test-Time Scaling

• Authors: Ramya Keerthy Thatikonda, Wray Buntine, Ehsan Shareghi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19903
• Pdf URL: https://arxiv.org/pdf/2508.19903
• Copy Paste: [[2508.19903]] Logical Reasoning with Outcome Reward Models for Test-Time Scaling(https://arxiv.org/abs/2508.19903)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs' tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.
• 摘要：逻辑推理是评估大语言模型（LLMS）功能的关键基准，因为它反映了他们从给定前提中得出有效结论的能力。尽管测试时间缩放与专用结果或过程奖励模型的结合开放了新的途径，以增强复杂推理任务中的LLMS性能，但在演绎逻辑推理中，该空间却不足。我们为演绎推理提供了一组结果奖励模型（ORM）。为了训练ORMS，我们主要使用具有单个和多个样本的思想链（COT）生成数据。此外，我们提出了一种新颖的策略，以进一步扩大ORM训练数据集中涵盖的错误类型。特别是，我们提出了一种回声生成技术，该技术利用LLMS的趋势反映了提示中提取其他培训数据的错误假设，涵盖了以前未开发的错误类型。虽然标准的COT链可能包含推理器可能犯的错误，但回声策略故意将模型引导到不正确的推理方面。我们表明，在COT和ECHO增强的数据上训练的ORMS在四个不同的LLMS上的对开本，Justlogic和ProverQA数据集的性能提高了。

Title: Your AI Bosses Are Still Prejudiced: The Emergence of Stereotypes in LLM-Based Multi-Agent Systems

• Authors: Jingyu Guo, Yingying Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19919
• Pdf URL: https://arxiv.org/pdf/2508.19919
• Copy Paste: [[2508.19919]] Your AI Bosses Are Still Prejudiced: The Emergence of Stereotypes in LLM-Based Multi-Agent Systems(https://arxiv.org/abs/2508.19919)
• Keywords: llm, agent
• Abstract: While stereotypes are well-documented in human social interactions, AI systems are often presumed to be less susceptible to such biases. Previous studies have focused on biases inherited from training data, but whether stereotypes can emerge spontaneously in AI agent interactions merits further exploration. Through a novel experimental framework simulating workplace interactions with neutral initial conditions, we investigate the emergence and evolution of stereotypes in LLM-based multi-agent systems. Our findings reveal that (1) LLM-Based AI agents develop stereotype-driven biases in their interactions despite beginning without predefined biases; (2) stereotype effects intensify with increased interaction rounds and decision-making power, particularly after introducing hierarchical structures; (3) these systems exhibit group effects analogous to human social behavior, including halo effects, confirmation bias, and role congruity; and (4) these stereotype patterns manifest consistently across different LLM architectures. Through comprehensive quantitative analysis, these findings suggest that stereotype formation in AI systems may arise as an emergent property of multi-agent interactions, rather than merely from training data biases. Our work underscores the need for future research to explore the underlying mechanisms of this phenomenon and develop strategies to mitigate its ethical impacts.
• 摘要：虽然刻板印象在人类的社会互动中有充分的记录，但通常认为AI系统不太容易受到此类偏见的影响。先前的研究集中在训练数据中继承的偏见，但是刻板印象是否可以在AI代理相互作用中自发出现，这值得进一步探索。通过一个新的实验框架，模拟了与中性初始条件的工作场所相互作用，我们研究了基于LLM的多代理系统中刻板印象的出现和演变。我们的发现表明，（1）基于LLM的AI代理在没有预定义的偏见的情况下，在相互作用中发展了刻板印象驱动的偏见。 （2）刻板印象的影响会随着相互作用的增加和决策能力的增加而加剧，尤其是在引入层次结构之后； （3）这些系统表现出类似于人类社会行为的群体效应，包括光环效应，确认偏见和角色一致性； （4）这些刻板印象模式在不同的LLM体系结构中始终如一。通过全面的定量分析，这些发现表明，AI系统中的刻板印象形成可能是多代理相互作用的新兴特性，而不仅仅是培训数据偏见。我们的工作强调了未来研究的需求，以探索这种现象的基本机制，并制定策略来减轻其道德影响。

Title: HEAL: A Hypothesis-Based Preference-Aware Analysis Framework

• Authors: Yifu Huo, Chenglong Wang, Qiren Zhu, Shunjie Xing, Tong Xiao, Chunliang Zhang, Tongran Liu, Jinbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19922
• Pdf URL: https://arxiv.org/pdf/2508.19922
• Copy Paste: [[2508.19922]] HEAL: A Hypothesis-Based Preference-Aware Analysis Framework(https://arxiv.org/abs/2508.19922)
• Keywords: llm
• Abstract: Preference optimization methods like DPO have achieved remarkable performance in LLM alignment. However, the evaluation for these methods relies on a single response and overlooks other potential outputs, which could also be generated in real-world applications within this hypothetical space. To address this issue, this paper presents a \textbf{H}ypothesis-based Pr\textbf{E}ference-aware \textbf{A}na\textbf{L}ysis Framework (HEAL), a novel evaluation paradigm that formulates preference alignment as a re-ranking process within hypothesis spaces. The framework incorporates two complementary metrics: ranking accuracy for evaluating ordinal consistency and preference strength correlation for assessing continuous alignment. To facilitate this framework, we develop UniHypoBench, a unified hypothesis benchmark constructed from diverse instruction-response pairs. Through extensive experiments based on HEAL, with a particular focus on the intrinsic mechanisms of preference learning, we demonstrate that current preference learning methods can effectively capture preferences provided by proxy models while simultaneously suppressing negative samples. These findings contribute to preference learning research through two significant avenues. Theoretically, we introduce hypothesis space analysis as an innovative paradigm for understanding preference alignment. Practically, HEAL offers researchers robust diagnostic tools for refining preference optimization methods, while our empirical results identify promising directions for developing more advanced alignment algorithms capable of comprehensive preference capture.
• 摘要：诸如DPO之类的偏好优化方法在LLM对齐中取得了显着的性能。但是，对这些方法的评估取决于单个响应，并忽略了其他潜在输出，这也可以在该假设空间内的现实世界应用中生成。为了解决这个问题，本文介绍了基于pr \ textbf {e} ference-ware \ textbf \ textbf {a} na \ textbf {l textbf {l} ySIS框架（heal），这是一种新颖的评估范围，该范例是一种重新构建范围的新型评估范围，将其作为重新分类的过程。该框架结合了两个互补指标：评估序数一致性和偏好强度相关的排名准确性，以评估连续比对。为了促进这一框架，我们开发了Unihypobench，这是一种由不同的指令 - 响应对构建的统一假设基准。通过基于治愈的广泛实验，特别关注偏好学习的内在机制，我们证明了当前的偏好学习方法可以有效地捕获代理模型提供的偏好，同时抑制负面样本。这些发现通过两种重要途径有助于偏好学习研究。从理论上讲，我们将假设空间分析作为一种创新的范式，以理解偏好一致性。实际上，HEAL为研究人员提供了强大的诊断工具，用于提炼偏好优化方法，而我们的经验结果则确定了有希望的方向，用于开发能够全面偏好捕获的更先进的对齐算法。

Title: Dhati+: Fine-tuned Large Language Models for Arabic Subjectivity Evaluation

• Authors: Slimane Bellaouar, Attia Nehar, Soumia Souffi, Mounia Bouameur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19966
• Pdf URL: https://arxiv.org/pdf/2508.19966
• Copy Paste: [[2508.19966]] Dhati+: Fine-tuned Large Language Models for Arabic Subjectivity Evaluation(https://arxiv.org/abs/2508.19966)
• Keywords: language model, gpt
• Abstract: Despite its significance, Arabic, a linguistically rich and morphologically complex language, faces the challenge of being under-resourced. The scarcity of large annotated datasets hampers the development of accurate tools for subjectivity analysis in Arabic. Recent advances in deep learning and Transformers have proven highly effective for text classification in English and French. This paper proposes a new approach for subjectivity assessment in Arabic textual data. To address the dearth of specialized annotated datasets, we developed a comprehensive dataset, AraDhati+, by leveraging existing Arabic datasets and collections (ASTD, LABR, HARD, and SANAD). Subsequently, we fine-tuned state-of-the-art Arabic language models (XLM-RoBERTa, AraBERT, and ArabianGPT) on AraDhati+ for effective subjectivity classification. Furthermore, we experimented with an ensemble decision approach to harness the strengths of individual models. Our approach achieves a remarkable accuracy of 97.79\,\% for Arabic subjectivity classification. Results demonstrate the effectiveness of the proposed approach in addressing the challenges posed by limited resources in Arabic language processing.
• 摘要：尽管它具有重要意义，但阿拉伯语是一种语言上丰富且形态上复杂的语言，但面临资源不足的挑战。大型注释数据集的稀缺性阻碍了阿拉伯语中主观性分析的准确工具的开发。深度学习和变形金刚的最新进展已证明对英语和法语的文本分类非常有效。本文提出了一种新的阿拉伯文本数据主观评估方法。为了解决专门注释的数据集的缺乏，我们通过利用现有的阿拉伯数据集和收藏集（ASTD，LABR，HARD和SANAD）开发了一个全面的数据集Aradhati+。随后，我们在Aradhati+上微调了最先进的阿拉伯语模型（XLM-Roberta，Arabert和Arabiangpt），以进行有效的主观性分类。此外，我们尝试了一种合奏决策方法来利用单个模型的优势。我们的方法实现了阿拉伯主观性分类的97.79 \ \％的显着准确性。结果证明了拟议方法在解决阿拉伯语处理中资源有限的挑战方面的有效性。

Title: Diffusion Language Models Know the Answer Before Decoding

• Authors: Pengxiang Li, Yefan Zhou, Dilxat Muhtar, Lu Yin, Shilin Yan, Li Shen, Yi Liang, Soroush Vosoughi, Shiwei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.19982
• Pdf URL: https://arxiv.org/pdf/2508.19982
• Copy Paste: [[2508.19982]] Diffusion Language Models Know the Answer Before Decoding(https://arxiv.org/abs/2508.19982)
• Keywords: language model
• Abstract: Diffusion language models (DLMs) have recently emerged as an alternative to autoregressive approaches, offering parallel sequence generation and flexible token orders. However, their inference remains slower than that of autoregressive models, primarily due to the cost of bidirectional attention and the large number of refinement steps required for high quality outputs. In this work, we highlight and leverage an overlooked property of DLMs early answer convergence: in many cases, the correct answer can be internally identified by half steps before the final decoding step, both under semi-autoregressive and random remasking schedules. For example, on GSM8K and MMLU, up to 97% and 99% of instances, respectively, can be decoded correctly using only half of the refinement steps. Building on this observation, we introduce Prophet, a training-free fast decoding paradigm that enables early commit decoding. Specifically, Prophet dynamically decides whether to continue refinement or to go "all-in" (i.e., decode all remaining tokens in one step), using the confidence gap between the top-2 prediction candidates as the criterion. It integrates seamlessly into existing DLM implementations, incurs negligible overhead, and requires no additional training. Empirical evaluations of LLaDA-8B and Dream-7B across multiple tasks show that Prophet reduces the number of decoding steps by up to 3.4x while preserving high generation quality. These results recast DLM decoding as a problem of when to stop sampling, and demonstrate that early decode convergence provides a simple yet powerful mechanism for accelerating DLM inference, complementary to existing speedup techniques. Our code is publicly available at this https URL.
• 摘要：扩散语言模型（DLM）最近已成为自回旋方法的替代方法，提供并行序列生成和灵活的令牌订单。但是，它们的推论仍然比自回归模型的推论慢，这主要是由于双向关注的成本和高质量产出所需的大量精致步骤。在这项工作中，我们强调并利用DLMS早期答案融合的被忽视的属性：在许多情况下，可以在最后一个解码步骤之前通过半步骤在内部识别正确的答案，均在半自动进程和随机重新启动时间表下。例如，在GSM8K和MMLU上，只能使用一半的细化步骤正确解码高达97％和99％的实例。在这一观察结果的基础上，我们介绍了先知，这是一种无训练的快速解码范式，可以提前解码。具体而言，先知会动态决定是继续改进还是“全力以赴”（即，一步一步中解码所有剩余的令牌），使用TOP-2预测候选者之间的置信度差距作为标准。它无缝集成到现有的DLM实现中，构成可忽略的开销，并且不需要额外的培训。跨多个任务对LLADA-8B和DREAM-7B的经验评估表明，先知将解码步骤的数量降低了3.4倍，同时保留了高生成质量。这些结果将DLM解码重现为何时停止采样的问题，并证明了早期解码的收敛提供了一种简单而强大的机制，可加速DLM推理，互补，以补充现有的加速技术。我们的代码在此HTTPS URL上公开可用。

Title: AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios

• Authors: Lisa Alazraki, Lihu Chen, Ana Brassard, Joe Stacey, Hossein A. Rahmani, Marek Rei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.19988
• Pdf URL: https://arxiv.org/pdf/2508.19988
• Copy Paste: [[2508.19988]] AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios(https://arxiv.org/abs/2508.19988)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have achieved high accuracy on complex commonsense and mathematical problems that involve the composition of multiple reasoning steps. However, current compositional benchmarks testing these skills tend to focus on either commonsense or math reasoning, whereas LLM agents solving real-world tasks would require a combination of both. In this work, we introduce an Agentic Commonsense and Math benchmark (AgentCoMa), where each compositional task requires a commonsense reasoning step and a math reasoning step. We test it on 61 LLMs of different sizes, model families, and training strategies. We find that LLMs can usually solve both steps in isolation, yet their accuracy drops by ~30% on average when the two are combined. This is a substantially greater performance gap than the one we observe in prior compositional benchmarks that combine multiple steps of the same reasoning type. In contrast, non-expert human annotators can solve the compositional questions and the individual steps in AgentCoMa with similarly high accuracy. Furthermore, we conduct a series of interpretability studies to better understand the performance gap, examining neuron patterns, attention maps and membership inference. Our work underscores a substantial degree of model brittleness in the context of mixed-type compositional reasoning and offers a test bed for future improvement.
• 摘要：大型语言模型（LLMS）在复杂的常识和数学问题上取得了很高的准确性，涉及多个推理步骤的组成。但是，当前测试这些技能的构图基准倾向于集中于常识或数学推理，而LL​​M代理解决现实世界任务将需要两者的组合。在这项工作中，我们介绍了一个代理常识和数学基准（AgentComa），每个组成任务都需要一个常识性推理步骤和数学推理步骤。我们对61个不同尺寸，模型家庭和培训策略的LLM进行测试。我们发现LLM通常可以孤立地解决这两个步骤，但是当两者合并时，它们的准确性平均下降了约30％。这比我们在与相同推理类型的多个步骤相结合的先前组成基准测试中观察到的性能差距要大得多。相比之下，非专业人类注释者可以以类似的高精度来解决构图问题和代理的各个步骤。此外，我们进行了一系列可解释性研究，以更好地了解性能差距，研究神经元模式，注意力图和成员资格推断。在混合型组成推理的背景下，我们的工作强调了大量的模型脆性，并为未来的改进提供了测试床。

Title: MathBuddy: A Multimodal System for Affective Math Tutoring

• Authors: Debanjana Kar, Leopold Böss, Dacia Braca, Sebastian Maximilian Dennerlein, Nina Christine Hubig, Philipp Wintersberger, Yufang Hou
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2508.19993
• Pdf URL: https://arxiv.org/pdf/2508.19993
• Copy Paste: [[2508.19993]] MathBuddy: A Multimodal System for Affective Math Tutoring(https://arxiv.org/abs/2508.19993)
• Keywords: llm, prompt
• Abstract: The rapid adoption of LLM-based conversational systems is already transforming the landscape of educational technology. However, the current state-of-the-art learning models do not take into account the student's affective states. Multiple studies in educational psychology support the claim that positive or negative emotional states can impact a student's learning capabilities. To bridge this gap, we present MathBuddy, an emotionally aware LLM-powered Math Tutor, which dynamically models the student's emotions and maps them to relevant pedagogical strategies, making the tutor-student conversation a more empathetic one. The student's emotions are captured from the conversational text as well as from their facial expressions. The student's emotions are aggregated from both modalities to confidently prompt our LLM Tutor for an emotionally-aware response. We have effectively evaluated our model using automatic evaluation metrics across eight pedagogical dimensions and user studies. We report a massive 23 point performance gain using the win rate and a 3 point gain at an overall level using DAMR scores which strongly supports our hypothesis of improving LLM-based tutor's pedagogical abilities by modeling students' emotions.
• 摘要：基于LLM的对话系统的快速采用已经在改变教育技术的景观。但是，当前的最新学习模型没有考虑到学生的情感状态。教育心理学的多项研究支持以下主张，即积极或负面的情绪状态会影响学生的学习能力。为了弥合这一差距，我们介绍了Mathbuddy，这是一个具有情感意识的LLM驱动数学导师，该教师会动态地模拟学生的情绪并将其映射到相关的教学策略，从而使教师认可的对话变得更加同情。学生的情绪是从对话文本以及面部表情中捕获的。从两种方式中汇总了学生的情绪，可以自信地促使我们的LLM导师做出情感意识的反应。我们已经使用八个教学维度和用户研究的自动评估指标有效地评估了我们的模型。我们报告了使用赢率获得23分的23分性能增长，并使用DAMR分数在整个水平上获得3分，这强烈支持我们通过对学生的情绪进行建模来改善基于LLM的老师的教学能力的假设。

Title: Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks

• Authors: Sheng Liu, Qiang Sheng, Danding Wang, Yang Li, Guang Yang, Juan Cao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.20038
• Pdf URL: https://arxiv.org/pdf/2508.20038
• Copy Paste: [[2508.20038]] Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks(https://arxiv.org/abs/2508.20038)
• Keywords: language model, llm
• Abstract: Despite advances in improving large language model(LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.
• 摘要：尽管改进大型语言模型（LLM）拒绝回答恶意说明的进步，但广泛使用的LLM仍然容易受到越狱攻击的影响，在这些攻击者中，攻击者生成的指示与安全偏差不同。新的攻击表明LLMS无法识别看不见的恶意说明，突出了训练数据和现实世界攻击之间的关键分布不匹配，这迫使开发人员进入反应性修补周期。为了应对这一挑战，我们建议想象一下，一个合成框架，该框架利用嵌入空间分配分析来产生类似越狱的说明。这种方法有效地填补了真实的越狱模式与安全一致性公司之间的分布差距。想象一下，遵循一个迭代优化过程，该过程在遍历迭代中动态发展文本生成分布，从而通过合成的数据示例扩大了安全对齐数据分布的覆盖范围。基于通过Imagine增强的安全一致的语料库，我们的框架表明Qwen2.5，Llama3.1和Llama3.2的攻击成功率显着降低，而不会损害其效用。

Title: 11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis

• Authors: Chengzu Li, Wenshan Wu, Huanyu Zhang, Qingtao Li, Zeyu Gao, Yan Xia, José Hernández-Orallo, Ivan Vulić, Furu Wei
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.20068
• Pdf URL: https://arxiv.org/pdf/2508.20068
• Copy Paste: [[2508.20068]] 11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis(https://arxiv.org/abs/2508.20068)
• Keywords: language model, llm
• Abstract: For human cognitive process, spatial reasoning and perception are closely entangled, yet the nature of this interplay remains underexplored in the evaluation of multimodal large language models (MLLMs). While recent MLLM advancements show impressive performance on reasoning, their capacity for human-like spatial cognition remains an open question. In this work, we introduce a systematic evaluation framework to assess the spatial reasoning abilities of state-of-the-art MLLMs relative to human performance. Central to our work is 11Plus-Bench, a high-quality benchmark derived from realistic standardized spatial aptitude tests. 11Plus-Bench also features fine-grained expert annotations of both perceptual complexity and reasoning process, enabling detailed instance-level analysis of model behavior. Through extensive experiments across 14 MLLMs and human evaluation, we find that current MLLMs exhibit early signs of spatial cognition. Despite a large performance gap compared to humans, MLLMs' cognitive profiles resemble those of humans in that cognitive effort correlates strongly with reasoning-related complexity. However, instance-level performance in MLLMs remains largely random, whereas human correctness is highly predictable and shaped by abstract pattern complexity. These findings highlight both emerging capabilities and limitations in current MLLMs' spatial reasoning capabilities and provide actionable insights for advancing model design.
• 摘要：对于人类的认知过程，空间推理和感知紧密纠缠，但是在评估多模式大语言模型（MLLM）中，这种相互作用的性质仍然没有被忽视。尽管最近的MLLM进步在推理方面表现出了令人印象深刻的表现，但它们具有人类的空间认知能力仍然是一个悬而未决的问题。在这项工作中，我们引入了一个系统的评估框架，以评估最先进的MLLM相对于人类绩效的空间推理能力。我们工作的核心是11plus Bench，这是一种由现实的标准化空间能力测试得出的高质量基准测试。 11 Plus Bench还具有感知复杂性和推理过程的细粒专家注释，从而实现了模型行为的详细实例级分析。通过对14个MLLM和人类评估的广泛实验，我们发现当前的MLLM表现出空间认知的早期迹象。尽管与人类相比具有较大的性能差距，但MLLM的认知曲线类似于人类的认知曲线，因为认知工作与与推理相关的复杂性密切相关。但是，MLLM中实例级的性能在很大程度上是随机的，而人类的正确性是可以预测的，并且通过抽象模式的复杂性来塑造。这些发现突出了当前MLLM的空间推理能力的新兴功能和局限性，并为推进模型设计提供了可行的见解。