2025-08-06

Title: Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation

• Authors: Radhika Dua, Young Joon (Fred)Kwon, Siddhant Dogra, Daniel Freedman, Diana Ruan, Motaz Nashawaty, Danielle Rigau, Daniel Alexander Alber, Kang Zhang, Kyunghyun Cho, Eric Karl Oermann
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.02808
• Pdf URL: https://arxiv.org/pdf/2508.02808
• Copy Paste: [[2508.02808]] Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation(https://arxiv.org/abs/2508.02808)
• Keywords: language model, agent
• Abstract: Radiological imaging is central to diagnosis, treatment planning, and clinical decision-making. Vision-language foundation models have spurred interest in automated radiology report generation (RRG), but safe deployment requires reliable clinical evaluation of generated reports. Existing metrics often rely on surface-level similarity or behave as black boxes, lacking interpretability. We introduce ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation), an interpretable evaluation framework leveraging large language model agents and dynamic multiple-choice question answering (MCQA). Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other. Agreement on answers captures preservation and consistency of findings, serving as interpretable proxies for clinical precision and recall. By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment. Clinician studies show ICARE aligns significantly more with expert judgment than prior metrics. Perturbation analyses confirm sensitivity to clinical content and reproducibility, while model comparisons reveal interpretable error patterns.
• 摘要：放射学成像对于诊断，治疗计划和临床决策至关重要。视觉基础模型激发了对自动放射学报告生成（RRG）的兴趣，但是安全部署需要对生成报告的可靠临床评估。现有的指标通常依赖于表面级别的相似性或表现为黑匣子，缺乏解释性。我们介绍了ICARE（可解释和临床基础的基于代理的报告评估），这是一个可解释的评估框架，利用大型语言模型代理和动态多项选择问题答案（MCQA）。两个代理人，每个代理商都有基地真相或生成的报告，都会产生临床上有意义的问题并互相问。答案协议捕获了发现的保存和一致性，可作为临床精度和召回的可解释代理。通过将分数链接到问答对，ICARE可以实现透明和可解释的评估。临床医生的研究表明，与先前的指标相比，ICARE与专家判断的一致性要多得多。扰动分析证实了对临床含量和可重复性的敏感性，而模型比较揭示了可解释的误差模式。

Title: Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives

• Authors: Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.02853
• Pdf URL: https://arxiv.org/pdf/2508.02853
• Copy Paste: [[2508.02853]] Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives(https://arxiv.org/abs/2508.02853)
• Keywords: llm, prompt
• Abstract: We present an approach to modeling annotator disagreement in subjective NLP tasks through both architectural and data-centric innovations. Our model, DEM-MoE (Demographic-Aware Mixture of Experts), routes inputs to expert subnetworks based on annotator demographics, enabling it to better represent structured, group-level variation compared to prior models. DEM-MoE consistently performs competitively across demographic groups, and shows especially strong results on datasets with high annotator disagreement. To address sparse demographic coverage, we test whether LLM-generated synthetic annotations via zero-shot persona prompting can be used for data imputation. We show these synthetic judgments align moderately well with human annotations on our data and offer a scalable way to potentially enrich training data. We then propose and evaluate approaches for blending real and synthetic data using strategies tailored to dataset structure. We find that the optimal strategies depend on dataset structure. Together, these contributions improve the representation of diverse perspectives.
• 摘要：我们提出了一种通过建筑和以数据为中心的创新来建模主观NLP任务中注释者分歧的方法。我们的模型Dem-MoE（专家的人口统计学混合物），基于注释人口统计学的专家子网络输入，使其能够更好地表示与先前模型相比，可以更好地表示结构化的组级变化。 Dem-MoE始终在人口统计组中竞争性地表现，并且在具有高注释者分歧的数据集上表现出尤其强劲的结果。为了解决稀疏的人口统计覆盖范围，我们测试LLM生成的合成注释是否可以通过零拍摄角色提示来进行数据插补。我们表明，这些综合判断与我们数据的人类注释相吻合，并提供了一种可扩展的方法来富集培训数据。然后，我们使用针对数据集结构量身定制的策略提出和评估将实际和合成数据融合的方法。我们发现最佳策略取决于数据集结构。这些贡献共同提高了各种观点的代表。

Title: Highlight & Summarize: RAG without the jailbreaks

• Authors: Giovanni Cherubin, Andrew Paverd
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.02872
• Pdf URL: https://arxiv.org/pdf/2508.02872
• Copy Paste: [[2508.02872]] Highlight & Summarize: RAG without the jailbreaks(https://arxiv.org/abs/2508.02872)
• Keywords: language model, llm, prompt, chat, retrieval-augmented generation
• Abstract: Preventing jailbreaking and model hijacking of Large Language Models (LLMs) is an important yet challenging task. For example, when interacting with a chatbot, malicious users can input specially crafted prompts to cause the LLM to generate undesirable content or perform a completely different task from its intended purpose. Existing mitigations for such attacks typically rely on hardening the LLM's system prompt or using a content classifier trained to detect undesirable content or off-topic conversations. However, these probabilistic approaches are relatively easy to bypass due to the very large space of possible inputs and undesirable outputs. In this paper, we present and evaluate Highlight & Summarize (H&S), a new design pattern for retrieval-augmented generation (RAG) systems that prevents these attacks by design. The core idea is to perform the same task as a standard RAG pipeline (i.e., to provide natural language answers to questions, based on relevant sources) without ever revealing the user's question to the generative LLM. This is achieved by splitting the pipeline into two components: a highlighter, which takes the user's question and extracts relevant passages ("highlights") from the retrieved documents, and a summarizer, which takes the highlighted passages and summarizes them into a cohesive answer. We describe several possible instantiations of H&S and evaluate their generated responses in terms of correctness, relevance, and response quality. Surprisingly, when using an LLM-based highlighter, the majority of H&S responses are judged to be better than those of a standard RAG pipeline.
• 摘要：防止大语模型（LLM）的越狱和模型劫持是一项重要但具有挑战性的任务。例如，当与聊天机器人进行互动时，恶意用户可以输入专门制作的提示，以使LLM生成不良内容或与预期目的执行完全不同的任务。对此类攻击的现有缓解通常依赖于硬化LLM的系统提示或使用经过培训的内容分类器来检测不良内容或主题对话。但是，由于可能的输入和不良输出的很大空间，这些概率方法相对容易绕过。在本文中，我们介绍并评估了Emairlight＆Summarizize（H＆S），这是一种新的设计模式，用于检索功能增强的生成（RAG）系统，可通过设计来防止这些攻击。核心想法是执行与标准的抹布管道（即，基于相关来源的问题的自然语言答案）执行相同的任务，而无需向生成的LLM揭示用户的问题。这是通过将管道分成两个组件来实现的：一个荧光笔，它吸引了用户的问题并从检索到的文档中提取相关段落（“亮点”），以及一个摘要器，该摘要将突出显示的段落汇总为凝聚力的答案。我们描述了H＆S的几种可能的实例，并根据正确性，相关性和响应质量评估了它们产生的响应。令人惊讶的是，当使用基于LLM的荧光笔时，大多数H＆S响应被认为比标准RAG管道的响应更好。

Title: Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Models

• Authors: Wenjie Luo, Ruocheng Li, Shanshan Zhu, Julian Perry
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.02886
• Pdf URL: https://arxiv.org/pdf/2508.02886
• Copy Paste: [[2508.02886]] Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Models(https://arxiv.org/abs/2508.02886)
• Keywords: language model, llm
• Abstract: Despite significant advancements, current large language models (LLMs) and vision-language models (LVLMs) continue to struggle with complex, multi-step, cross-modal common sense reasoning tasks, often exhibiting a lack of "deliberative thinking." They tend to rely on superficial associations rather than deep, chained inference, particularly when integrating visual information with abstract concepts. To address this, we propose the Coherent Multimodal Reasoning Framework (CMRF), a novel approach that enhances LVLMs' common sense reasoning capabilities through an iterative, self-evaluating inference mechanism. CMRF mimics human problem-solving by decomposing complex queries, generating step-by-step inferences, and self-correcting errors. Our framework integrates three key modules: a Reasoning Decomposition Unit (RDU) for breaking down problems into sub-questions, a Contextual Inference Engine (CIE) for contextual inference, and a Coherence Assessment Module (CAM) for evaluating logical consistency and confidence. Coupled with an Adaptive Iterative Refinement strategy, CMRF systematically refines its reasoning paths. Built upon LLaVA-1.6-34B and trained on a novel Multimodal Daily Activity Reasoning (MDAR) dataset, CMRF achieves state-of-the-art performance among open-source LVLMs on challenging benchmarks like VCR, A-OKVQA, and DailyLife-MRC. It attains an average accuracy of 69.4%, surpassing the best open-source baseline by +2.4 percentage points, with particular strength in complex reasoning scenarios. Extensive ablation studies and human evaluations confirm the critical contributions of each module and the effectiveness of iterative refinement in fostering more coherent and accurate reasoning.
• 摘要：尽管取得了重大进步，但当前的大型语言模型（LLM）和视觉语言模型（LVLM）仍在与复杂的，多步骤的跨模式常识推理任务中挣扎，经常表现出缺乏“审议思维”。他们倾向于依靠表面的关联，而不是深层，被束缚的推论，尤其是在将视觉信息与抽象概念相结合时。为了解决这个问题，我们提出了连贯的多模式推理框架（CMRF），这是一种新颖的方法，通过迭代，自我评估的推理机制来增强LVLMS的常识推理能力。 CMRF通过分解复杂的查询，逐步推断和自我校正错误来模仿人类问题解决。我们的框架集成了三个关键模块：用于将问题分解为子问题的推理分解单元（RDU），上下文推理引擎（CIE）用于上下文推断，以及一个相干评估模块（CAM），以评估逻辑一致性和信心。加上自适应迭代精致策略，CMRF系统地完善了其推理路径。 CMRF建立在LLAVA-1.6-34B的基础上，并接受了新型的多式联运日常活动推理（MDAR）数据集的培训，在开源的LVLMS中实现了在挑战性的基准测试中，在VCR，A-OKVQA和Dailylife-Mrc中实现了最先进的性能。它的平均准确度为69.4％，超过最佳的开源基线+2.4个百分点，在复杂的推理方案中具有特殊的强度。广泛的消融研究和人类评估证实了每个模块的关键贡献以及迭代改进在促进更连贯和准确的推理方面的有效性。

Title: SLIM-LLMs: Modeling of Style-Sensory Language RelationshipsThrough Low-Dimensional Representations

• Authors: Osama Khalid, Sanvesh Srivastava, Padmini Srinivasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.02901
• Pdf URL: https://arxiv.org/pdf/2508.02901
• Copy Paste: [[2508.02901]] SLIM-LLMs: Modeling of Style-Sensory Language RelationshipsThrough Low-Dimensional Representations(https://arxiv.org/abs/2508.02901)
• Keywords: language model, llm
• Abstract: Sensorial language -- the language connected to our senses including vision, sound, touch, taste, smell, and interoception, plays a fundamental role in how we communicate experiences and perceptions. We explore the relationship between sensorial language and traditional stylistic features, like those measured by LIWC, using a novel Reduced-Rank Ridge Regression (R4) approach. We demonstrate that low-dimensional latent representations of LIWC features r = 24 effectively capture stylistic information for sensorial language prediction compared to the full feature set (r = 74). We introduce Stylometrically Lean Interpretable Models (SLIM-LLMs), which model non-linear relationships between these style dimensions. Evaluated across five genres, SLIM-LLMs with low-rank LIWC features match the performance of full-scale language models while reducing parameters by up to 80%.
• 摘要：感官语言 - 与我们的感官相关的语言，包括视觉，声音，触摸，口味，气味和互认为，在我们如何交流体验和看法方面起着基本作用。我们探讨了感官语言和传统风格特征之间的关系，例如LIWC使用新颖的降低山脊回归（R4）方法来衡量的特征。我们证明，与完整的功能集相比，LIWC特征的低维潜图r = 24有效地捕获了感官语言预测的风格信息（r = 74）。我们介绍了风格的精益解释模型（Slim-llms），该模型对这些样式维度之间的非线性关系进行了建模。经过五种类型的评估，具有低级别LIWC功能的Slim-LLM与全尺度语言模型的性能相匹配，同时将参数降低了80％。

Title: Can LLMs Generate High-Quality Task-Specific Conversations?

• Authors: Shengqi Li, Amarnath Gupta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.02931
• Pdf URL: https://arxiv.org/pdf/2508.02931
• Copy Paste: [[2508.02931]] Can LLMs Generate High-Quality Task-Specific Conversations?(https://arxiv.org/abs/2508.02931)
• Keywords: language model, llm
• Abstract: This paper introduces a parameterization framework for controlling conversation quality in large language models. We explore nine key parameters across six dimensions that enable precise specification of dialogue properties. Through experiments with state-of-the-art LLMs, we demonstrate that parameter-based control produces statistically significant differences in generated conversation properties. Our approach addresses challenges in conversation generation, including topic coherence, knowledge progression, character consistency, and control granularity. The framework provides a standardized method for conversation quality control with applications in education, therapy, customer service, and entertainment. Future work will focus on implementing additional parameters through architectural modifications and developing benchmark datasets for evaluation.
• 摘要：本文介绍了一个参数化框架，用于控制大语言模型中的对话质量。我们探讨了六个维度的九个关键参数，以实现对话属性的精确规范。通过对最先进的LLM的实验，我们证明了基于参数的控制在生成的对话属性上产生统计学上的显着差异。我们的方法解决了对话的挑战，包括主题连贯性，知识进步，性格一致性和控制粒度。该框架为对话质量控制提供了一种标准化的方法，并在教育，治疗，客户服务和娱乐中应用了应用。未来的工作将集中于通过架构修改和开发基准数据集来实施其他参数以进行评估。

Title: CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors

• Authors: Sri Durga Sai Sowmya Kadali, Evangelos E. Papalexakis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.02997
• Pdf URL: https://arxiv.org/pdf/2508.02997
• Copy Paste: [[2508.02997]] CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors(https://arxiv.org/abs/2508.02997)
• Keywords: language model, llm, prompt
• Abstract: The widespread use of Large Language Models (LLMs) in many applications marks a significant advance in research and practice. However, their complexity and hard-to-understand nature make them vulnerable to attacks, especially jailbreaks designed to produce harmful responses. To counter these threats, developing strong detection methods is essential for the safe and reliable use of LLMs. This paper studies this detection problem using the Contextual Co-occurrence Matrix, a structure recognized for its efficacy in data-scarce environments. We propose a novel method leveraging the latent space characteristics of Contextual Co-occurrence Matrices and Tensors for the effective identification of adversarial and jailbreak prompts. Our evaluations show that this approach achieves a notable F1 score of 0.83 using only 0.5% of labeled prompts, which is a 96.6% improvement over baselines. This result highlights the strength of our learned patterns, especially when labeled data is scarce. Our method is also significantly faster, speedup ranging from 2.3 to 128.4 times compared to the baseline models. To support future research and reproducibility, we have made our implementation publicly available.
• 摘要：在许多应用中，大型语言模型（LLM）的广泛使用标志着研究和实践的重大进步。但是，它们的复杂性和难以理解的性质使它们容易受到攻击，尤其是旨在产生有害反应的越狱。为了应对这些威胁，开发强大的检测方法对于安全可靠的LLMS是必不可少的。本文使用上下文共发生矩阵研究了此检测问题，该结构以其在数据筛选环境中的功效而认可。我们提出了一种新的方法，利用上下文共同出现矩阵和张量的潜在空间特征，以有效地识别对抗和越狱提示。我们的评估表明，这种方法仅使用标记的提示的0.5％获得了0.83的著名F1得分，比基准提高了96.6％。这个结果突出了我们学到的模式的强度，尤其是当标记的数据稀缺时。我们的方法的速度也更快，与基线模型相比，加速度的范围从2.3到128.4倍。为了支持未来的研究和可重复性，我们已公开实施。

Title: Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation

• Authors: Haoran Wang, Xiongxiao Xu, Baixiang Huang, Kai Shu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03098
• Pdf URL: https://arxiv.org/pdf/2508.03098
• Copy Paste: [[2508.03098]] Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation(https://arxiv.org/abs/2508.03098)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) enhances the factual accuracy of large language models (LLMs) by conditioning outputs on external knowledge sources. However, when retrieval involves private or sensitive data, RAG systems are susceptible to extraction attacks that can leak confidential information through generated responses. We propose Privacy-Aware Decoding (PAD), a lightweight, inference-time defense that adaptively injects calibrated Gaussian noise into token logits during generation. PAD integrates confidence-based screening to selectively protect high-risk tokens, efficient sensitivity estimation to minimize unnecessary noise, and context-aware noise calibration to balance privacy with generation quality. A \renyi Differential Privacy (RDP) accountant rigorously tracks cumulative privacy loss, enabling explicit per-response $(\varepsilon, \delta)$-DP guarantees for sensitive outputs. Unlike prior approaches requiring retraining or corpus-level filtering, PAD is model-agnostic and operates entirely at decoding time with minimal computational overhead. Experiments on three real-world datasets demonstrate that PAD substantially reduces private information leakage while preserving response utility, outperforming existing retrieval- and post-processing-based defenses. Our work takes an important step toward mitigating privacy risks in RAG via decoding strategies, paving the way for universal and scalable privacy solutions in sensitive domains. Our code is available: this https URL.
• 摘要：检索增强的生成（RAG）通过对外部知识源进行调节输出来提高大语言模型（LLMS）的事实准确性。但是，当检索涉及私人或敏感数据时，抹布系统易受提取攻击，这些攻击可以通过生成的响应泄露机密信息。我们建议使用隐私意识解码（PAD），这是一种轻巧的推理时间防御，可适应在世代相传地注入校准的高斯噪声中。 PAD集成了基于置信的筛选，以选择性地保护高风险令牌，有效的灵敏度估计以最大程度地减少不必要的噪声以及上下文感知的噪声校准，以平衡隐私与发电质量。 \ renyi差异隐私（RDP）会计师严格跟踪累积隐私损失，为敏感输出提供明确的人为响应$（\ varepsilon，\ delta）$ - DP保证。与需要重新训练或语料库级过滤的先验方法不同，PAD是模型不合时式，并且在最小的计算开销时完全在解码时间内运行。三个现实世界数据集的实验表明，PAD可以大大减少私人信息泄漏，同时保留响应实用程序，表现优于现有的检索和后处理后的防御。我们的工作通过解码策略迈出了重要的一步，以减轻抹布中的隐私风险，为敏感领域中的通用和可扩展的隐私解决方案铺平道路。我们的代码可用：此HTTPS URL。

Title: Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation

• Authors: Zizhong Li, Haopeng Zhang, Jiawei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03110
• Pdf URL: https://arxiv.org/pdf/2508.03110
• Copy Paste: [[2508.03110]] Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation(https://arxiv.org/abs/2508.03110)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: While large language models (LLMs) have achieved remarkable success in providing trustworthy responses for knowledge-intensive tasks, they still face critical limitations such as hallucinations and outdated knowledge. To address these issues, the retrieval-augmented generation (RAG) framework enhances LLMs with access to external knowledge via a retriever, enabling more accurate and real-time outputs about the latest events. However, this integration brings new security vulnerabilities: the risk that malicious content in the external database can be retrieved and used to manipulate model outputs. Although prior work has explored attacks on RAG systems, existing approaches either rely heavily on access to the retriever or fail to jointly consider both retrieval and generation stages, limiting their effectiveness, particularly in black-box scenarios. To overcome these limitations, we propose Token-level Precise Attack on the RAG (TPARAG), a novel framework that targets both white-box and black-box RAG systems. TPARAG leverages a lightweight white-box LLM as an attacker to generate and iteratively optimize malicious passages at the token level, ensuring both retrievability and high attack success in generation. Extensive experiments on open-domain QA datasets demonstrate that TPARAG consistently outperforms previous approaches in retrieval-stage and end-to-end attack effectiveness. These results further reveal critical vulnerabilities in RAG pipelines and offer new insights into improving their robustness.
• 摘要：尽管大型语言模型（LLM）在为知识密集型任务提供值得信赖的响应方面取得了巨大的成功，但它们仍然面临着诸如幻觉和过时的知识之类的关键局限性。为了解决这些问题，检索功能的生成（RAG）框架可以通过猎犬访问外部知识，从而增强了LLM，从而使有关最新事件的更准确和实时输出能够。但是，这种集成带来了新的安全漏洞：可以检索外部数据库中的恶意内容并用于操纵模型输出的风险。尽管先前的工作已经探索了对抹布系统的攻击，但现有方法要么很大程度上依赖于对猎犬的访问，要么无法共同考虑检索和发电阶段，从而限制了它们的有效性，尤其是在黑盒子的情况下。为了克服这些局限性，我们提出了对抹布（tparag）的令牌精确攻击，这是一个针对白盒和黑盒抹布系统的新型框架。 Tparag利用轻巧的白色盒子LLM作为攻击者生成并迭代地在代币级别优化恶意段落，从而确保了一代人的可填式性和高攻击成功。对开放域QA数据集进行的广泛实验表明，Tparag在检索阶段和端到端攻击效果方面始终超过先前的方法。这些结果进一步揭示了抹布管道中的关键脆弱性，并为改善其鲁棒性提供了新的见解。

Title: Long Story Generation via Knowledge Graph and Literary Theory

• Authors: Ge Shi, Kaiyu Huang, Guochen Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03137
• Pdf URL: https://arxiv.org/pdf/2508.03137
• Copy Paste: [[2508.03137]] Long Story Generation via Knowledge Graph and Literary Theory(https://arxiv.org/abs/2508.03137)
• Keywords: language model, llm, agent
• Abstract: The generation of a long story consisting of several thousand words is a sub-task in the field of long text generation~(LTG). Previous research has addressed this challenge through outline-based generation, which employs a multi-stage method for generating outlines into stories. However, this approach suffers from two common issues: almost inevitable theme drift caused by the loss of memory of previous outlines, and tedious plots with incoherent logic that are less appealing to human readers. In this paper, we propose the multi-agent Story Generator structure to improve the multi-stage method, using large language models~(LLMs) as the core components of agents. To avoid theme drift, we introduce a memory storage model comprising two components: a long-term memory storage that identifies the most important memories, thereby preventing theme drift; and a short-term memory storage that retains the latest outlines from each generation round. To incorporate engaging elements into the story, we design a story theme obstacle framework based on literary narratology theory that introduces uncertain factors and evaluation criteria to generate outline. This framework calculates the similarity of the former storyline and enhances the appeal of the story by building a knowledge graph and integrating new node content. Additionally, we establish a multi-agent interaction stage to simulate writer-reader interaction through dialogue and revise the story text according to feedback, to ensure it remains consistent and logical. Evaluations against previous methods demonstrate that our approach can generate higher-quality long stories.
• 摘要：长篇小说的一代由数千个单词组成，是长期文字一代（LTG）领域的子任务。先前的研究通过基于大纲的一代解决了这一挑战，该一代采用了一种多阶段的方法来将大纲纳入故事中。但是，这种方法遇到了两个常见的问题：几乎不可避免的主题漂移是由于以前的大纲的记忆而导致的，而乏味的情节不连贯，对人类读者的吸引力较小。在本文中，我们建议使用大型语言模型〜（LLM）作为代理的核心组成部分，提出多代理故事生成器结构，以改善多阶段方法。为了避免主题漂移，我们介绍了一个包括两个组件的内存存储模型：一个长期存储器存储，识别最重要的内存，从而防止主题漂移；以及一个短期内存存储，保留了每一轮的最新概述。为了将引人入胜的元素纳入故事中，我们设计了一个基于文学叙事学理论的故事主题障碍框架，该框架介绍了不确定的因素和评估标准以生成轮廓。该框架计算了以前的故事情节的相似性，并通过构建知识图和集成新节点内容来增强故事的吸引力。此外，我们建立了一个多代理互动阶段，以通过对话模拟作者阅读器的互动并根据反馈修改故事文本，以确保其保持一致和逻辑。针对以前的方法的评估表明，我们的方法可以产生更高质量的长篇小说。

Title: RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior

• Authors: Junyao Yang, Jianwei Wang, Huiping Zhuang, Cen Chen, Ziqian Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03140
• Pdf URL: https://arxiv.org/pdf/2508.03140
• Copy Paste: [[2508.03140]] RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior(https://arxiv.org/abs/2508.03140)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large Language Models (LLMs) with long chain-of-thought (CoT) capability, termed Reasoning Models, demonstrate superior intricate problem-solving abilities through multi-step long CoT reasoning. To create a dual-capability model with long CoT capability and domain-specific knowledge without substantial computational and data costs, model merging emerges as a highly resource-efficient method. However, significant challenges lie in merging domain-specific LLMs with long CoT ones since nowadays merging methods suffer from reasoning capability degradation, even gibberish output and output collapse. To overcome this, we introduce RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior, a novel merging framework designed to integrate domain-specific LLMs with long CoT capability, meanwhile maintaining model performance in the original domain. Treating reasoning model weights as foundational prior, our method utilizes a reasoning capability indicator to preserve core long CoT capability model weights while selectively merging essential domain-specific weights. We conducted extensive experiments on Qwen2.5-7B, Llama3.1-8B, and Qwen2.5-1.5B models in BioMedicine and Finance domains. Our results show that RCP-Merging successfully merges a reasoning model with domain-specific ones, improving domain task performance by 9.5% and 9.2% over state-of-the-art methods, without significantly harming the original long CoT reasoning capability.
• 摘要：具有长长的经过思考（COT）功能的大型语言模型（LLMS）称为推理模型，通过多步长的COT推理表明了卓越的复杂问题解决能力。为了创建具有长长的COT功能和特定于域的知识的双重能力模型，而没有实质性的计算和数据成本，模型合并作为一种资源高效的方法出现。然而，自如今合并方法以来，应有推理能力降解，甚至散发性产出和产出崩溃，构成了域特异性LLM与长COT的重大挑战。为了克服这一点，我们引入了RCP合并：通过将推理能力视为先验，将长期的经过思考模型与域特异性模型合并，这是一个新颖的合并框架，旨在将特定于域特异性的LLM与长COT能力集成在一起，同时在原始域中维持模型性能。我们的方法将推理模型权重视为基础先验，利用推理能力指标来保留核心长COT能力模型权重，同时选择性合并基本域特异性权重。我们在QWEN2.5-7B，LLAMA3.1-8B和QWEN2.5-1.5B模型上进行了广泛的实验。我们的结果表明，RCP合并成功地将推理模型与特定于域的推理模型合并，使域任务性能比最先进的方法提高了9.5％和9.2％，而不会显着损害原始的长COT推理能力。

Title: Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following

• Authors: Chenyang Wang, Liang Wen, Shousheng Jia, Xiangzheng Zhang, Liang Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.03178
• Pdf URL: https://arxiv.org/pdf/2508.03178
• Copy Paste: [[2508.03178]] Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following(https://arxiv.org/abs/2508.03178)
• Keywords: llm, prompt
• Abstract: While advancements in the reasoning abilities of LLMs have significantly enhanced their performance in solving mathematical problems, coding tasks, and general puzzles, their effectiveness in accurately adhering to instructions remains inconsistent, particularly with more complex directives. Our investigation identifies lazy reasoning during the thinking stage as the primary factor contributing to poor instruction adherence. To mitigate this issue, we propose a comprehensive framework designed to enable rigorous reasoning processes involving preview and self-checking, essential for satisfying strict instruction constraints. Specifically, we first generate instructions with complex constraints and apply a filtering process to obtain valid prompts, resulting in three distinct prompt datasets categorized as hard, easy, and pass. Then, we employ rejection sampling on the pass prompts to curate a small yet high-quality dataset, enabling a cold-start initialization of the model and facilitating its adaptation to effective reasoning patterns. Subsequently, we employ an entropy-preserving supervised fine-tuning (Entropy-SFT) strategy coupled with token-wise entropy-adaptive (TEA-RL) reinforcement learning guided by rule-based dense rewards. This approach encourages the model to transform its reasoning mechanism, ultimately fostering generalizable reasoning abilities that encompass preview and self-checking. Extensive experiments conducted on instruction-following benchmarks demonstrate remarkable performance improvements across various model scales. Notably, our Light-IF-32B model surpasses both larger open-source models such as DeepSeek-R1 and closed-source models like Doubao-1.6.
• 摘要：尽管LLM的推理能力的进步已经显着提高了它们在解决数学问题，编码任务和一般难题方面的性能，但它们在准确遵守指令方面的有效性仍然不一致，尤其是与更复杂的指令。我们的调查将思维阶段的懒惰推理确定为导致教学依从性不佳的主要因素。为了减轻此问题，我们提出了一个综合框架，旨在实现涉及预览和自我检查的严格推理过程，对于满足严格的指导约束至关重要。具体而言，我们首先生成具有复杂约束的指令，并应用过滤过程以获取有效的提示，从而导致三个不同的提示数据集被归类为硬，易于和通过。然后，我们在通行证上采用拒绝采样提示来策划一个小但高质量的数据集，从而使模型的冷启动初始化并促进了其对有效推理模式的适应性。随后，我们采用了熵保管的监督微调（熵-SFT）策略，并采用令牌的熵自适应（TEA-RL）加固学习，这些增强措施以基于规则的密集奖励为指导。这种方法鼓励该模型改变其推理机制，最终促进涵盖预览和自我检查的可概括推理能力。对遵循指令的基准进行的广泛实验表明，各种模型量表的性能得到了显着改进。值得注意的是，我们的Light-IF-32B模型超过了较大的开源模型，例如DeepSeek-R1和封闭源模型，例如Doubao-1.6。

Title: Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models

• Authors: Muhammed Saeed, Shaina Raza, Ashmal Vayani, Muhammad Abdul-Mageed, Ali Emami, Shady Shehata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03199
• Pdf URL: https://arxiv.org/pdf/2508.03199
• Copy Paste: [[2508.03199]] Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models(https://arxiv.org/abs/2508.03199)
• Keywords: prompt
• Abstract: Research on bias in Text-to-Image (T2I) models has primarily focused on demographic representation and stereotypical attributes, overlooking a fundamental question: how does grammatical gender influence visual representation across languages? We introduce a cross-linguistic benchmark examining words where grammatical gender contradicts stereotypical gender associations (e.g., ``une sentinelle'' - grammatically feminine in French but referring to the stereotypically masculine concept ``guard''). Our dataset spans five gendered languages (French, Spanish, German, Italian, Russian) and two gender-neutral control languages (English, Chinese), comprising 800 unique prompts that generated 28,800 images across three state-of-the-art T2I models. Our analysis reveals that grammatical gender dramatically influences image generation: masculine grammatical markers increase male representation to 73\% on average (compared to 22\% with gender-neutral English), while feminine grammatical markers increase female representation to 38\% (compared to 28\% in English). These effects vary systematically by language resource availability and model architecture, with high-resource languages showing stronger effects. Our findings establish that language structure itself, not just content, shapes AI-generated visual outputs, introducing a new dimension for understanding bias and fairness in multilingual, multimodal systems.
• 摘要：关于文本到图像（T2I）模型偏见的研究主要集中在人口统计学表示和刻板印象属性上，忽略了一个基本问题：语法性别如何影响跨语言的视觉表示？我们介绍了一个跨语言基准检查词，语法性别与刻板印象的性别关联相矛盾（例如，``'une sentinelle'' - 法语上语法上的女性化，但指刻板印象性男性概念概念``''''）。我们的数据集涵盖了五种性别语言（法语，西班牙语，德语，意大利语，俄语）和两种性别中立的控制语言（英语，中文），其中包括800个独特的提示，这些提示在三种最先进的T2I模型中产生了28,800张图像。我们的分析表明，语法性别会极大地影响图像的产生：男性语法标志物将男性的表现平均提高到73 \％（相比之下，性别中性英语为22 \％），而女性语法标志物将女性表示女性表示增加到38 \％（英语为28 \％）。这些效果因语言资源可用性和模型架构而系统地变化，高资源语言显示出更强的效果。我们的发现确定了语言结构本身，而不仅仅是内容，还塑造了AI生成的视觉输出，引入了一个新的维度，以理解多语言，多模式系统中的偏见和公平。

Title: Current State in Privacy-Preserving Text Preprocessing for Domain-Agnostic NLP

• Authors: Abhirup Sinha, Pritilata Saha, Tithi Saha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03204
• Pdf URL: https://arxiv.org/pdf/2508.03204
• Copy Paste: [[2508.03204]] Current State in Privacy-Preserving Text Preprocessing for Domain-Agnostic NLP(https://arxiv.org/abs/2508.03204)
• Keywords: language model
• Abstract: Privacy is a fundamental human right. Data privacy is protected by different regulations, such as GDPR. However, modern large language models require a huge amount of data to learn linguistic variations, and the data often contains private information. Research has shown that it is possible to extract private information from such language models. Thus, anonymizing such private and sensitive information is of utmost importance. While complete anonymization may not be possible, a number of different pre-processing approaches exist for masking or pseudonymizing private information in textual data. This report focuses on a few of such approaches for domain-agnostic NLP tasks.
• 摘要：隐私是人类的基本权利。数据隐私受到不同法规的保护，例如GDPR。但是，现代大型语言模型需要大量数据来学习语言变化，并且数据通常包含私人信息。研究表明，可以从此类语言模型中提取私人信息。因此，匿名化此类私人和敏感信息至关重要。虽然可能无法进行完整的匿名化，但在文本数据中存在掩盖或化名的私人信息，存在许多不同的预处理方法。该报告重点介绍了一些针对域 - 不合稳定任务的方法。

Title: Probing Syntax in Large Language Models: Successes and Remaining Challenges

• Authors: Pablo J. Diego-Simón, Emmanuel Chemla, Jean-Rémi King, Yair Lakretz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03211
• Pdf URL: https://arxiv.org/pdf/2508.03211
• Copy Paste: [[2508.03211]] Probing Syntax in Large Language Models: Successes and Remaining Challenges(https://arxiv.org/abs/2508.03211)
• Keywords: language model, llm
• Abstract: The syntactic structures of sentences can be readily read-out from the activations of large language models (LLMs). However, the ``structural probes'' that have been developed to reveal this phenomenon are typically evaluated on an indiscriminate set of sentences. Consequently, it remains unclear whether structural and/or statistical factors systematically affect these syntactic representations. To address this issue, we conduct an in-depth analysis of structural probes on three controlled benchmarks. Our results are three-fold. First, structural probes are biased by a superficial property: the closer two words are in a sentence, the more likely structural probes will consider them as syntactically linked. Second, structural probes are challenged by linguistic properties: they poorly represent deep syntactic structures, and get interfered by interacting nouns or ungrammatical verb forms. Third, structural probes do not appear to be affected by the predictability of individual words. Overall, this work sheds light on the current challenges faced by structural probes. Providing a benchmark made of controlled stimuli to better evaluate their performance.
• 摘要：句子的句法结构可以从大语模型（LLMS）的激活中很容易读取。但是，通常在一系列句子集上评估了为揭示这种现象而开发的``结构探针''。因此，尚不清楚结构性和/或统计因素是否会系统地影响这些句法表示。为了解决这个问题，我们对三个受控基准测试的结构探针进行了深入的分析。我们的结果是三倍。首先，结构探针被表面特性偏见：句子中越接近两个词，结构探针越可能将其视为句法链接。其次，结构探针受到语言特性的挑战：它们代表了深层的句法结构，并通过相互作用的名词或非语法动词形式干预。第三，结构探针似乎不受单个单词的可预测性影响。总体而言，这项工作阐明了结构探针所面临的当前挑战。提供由受控刺激制成的基准，以更好地评估其性能。

Title: CardiffNLP at CLEARS-2025: Prompting Large Language Models for Plain Language and Easy-to-Read Text Rewriting

• Authors: Mutaz Ayesh, Nicolás Gutiérrez-Rolón, Fernando Alva-Manchego
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03240
• Pdf URL: https://arxiv.org/pdf/2508.03240
• Copy Paste: [[2508.03240]] CardiffNLP at CLEARS-2025: Prompting Large Language Models for Plain Language and Easy-to-Read Text Rewriting(https://arxiv.org/abs/2508.03240)
• Keywords: language model, llm, prompt
• Abstract: This paper details the CardiffNLP team's contribution to the CLEARS shared task on Spanish text adaptation, hosted by IberLEF 2025. The shared task contained two subtasks and the team submitted to both. Our team took an LLM-prompting approach with different prompt variations. While we initially experimented with LLaMA-3.2, we adopted Gemma-3 for our final submission, and landed third place in Subtask 1 and second place in Subtask 2. We detail our numerous prompt variations, examples, and experimental results.
• 摘要：本文详细介绍了CardiffNLP团队对Iberlef 2025托管的西班牙文本改编的“清算共同任务”的贡献。共享任务包含两个子任务，并且团队提交给两者。我们的团队采用了LLM启动方法，并采用了不同的及时变化。虽然我们最初对Llama-3.2进行了实验，但我们采用了Gemma-3进行最终提交，并在子任务1中排名第三，在子任务2中排名第二。我们详细介绍了许多及时的及时变化，示例和实验结果。

Title: Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs

• Authors: Shintaro Sakai, Jisun An, Migyeong Kang, Haewoon Kwak
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2508.03247
• Pdf URL: https://arxiv.org/pdf/2508.03247
• Copy Paste: [[2508.03247]] Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs(https://arxiv.org/abs/2508.03247)
• Keywords: language model, llm, prompt
• Abstract: Prior clinical psychology research shows that Western individuals with depression tend to report psychological symptoms, while Eastern individuals report somatic ones. We test whether Large Language Models (LLMs), which are increasingly used in mental health, reproduce these cultural patterns by prompting them with Western or Eastern personas. Results show that LLMs largely fail to replicate the patterns when prompted in English, though prompting in major Eastern languages (i.e., Chinese, Japanese, and Hindi) improves alignment in several configurations. Our analysis pinpoints two key reasons for this failure: the models' low sensitivity to cultural personas and a strong, culturally invariant symptom hierarchy that overrides cultural cues. These findings reveal that while prompt language is important, current general-purpose LLMs lack the robust, culture-aware capabilities essential for safe and effective mental health applications.
• 摘要：先前的临床心理学研究表明，抑郁症的西方人倾向于报告心理症状，而东方人报告了躯体。我们测试了越来越多地用于心理健康中的大型语言模型（LLM）是否通过促使西方或东方角色来重现这些文化模式。结果表明，LLM在用英语提示时几乎无法复制模式，尽管以主要的东方语言提示（即中文，日语和印地语）可以改善几种配置的对齐方式。我们的分析指出了这一失败的两个关键原因：模型对文化角色的低敏感性以及强大的文化不变症状等级结构，覆盖了文化线索。这些发现表明，虽然迅速的语言很重要，但当前的通用LLM缺乏对安全有效的心理健康应用必不可少的鲁棒，文化意识的能力。

Title: RooseBERT: A New Deal For Political Language Modelling

• Authors: Deborah Dore, Elena Cabrio, Serena Villata
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03250
• Pdf URL: https://arxiv.org/pdf/2508.03250
• Copy Paste: [[2508.03250]] RooseBERT: A New Deal For Political Language Modelling(https://arxiv.org/abs/2508.03250)
• Keywords: language model
• Abstract: The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models. To address this issue, we introduce a novel pre-trained Language Model for political discourse language called RooseBERT. Pre-training a language model on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (8K debates, each composed of several sub-debates on different topics) in English. To evaluate its performances, we fine-tuned it on four downstream tasks related to political debate analysis, i.e., named entity recognition, sentiment analysis, argument component detection and classification, and argument relation prediction and classification. Our results demonstrate significant improvements over general-purpose Language Models on these four tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis. We release the RooseBERT language model for the research community.
• 摘要：越来越多的政治辩论和与政治有关的讨论要求定义新颖的计算方法，以自动分析此类内容，以减轻对公民进行政治审议的最终目标。但是，政治语言的特殊性以及这些辩论的论证形式（采用隐藏的交流策略和利用隐性论点）使这项任务变得非常具有挑战性，即使对于当前通用的通用预培训的语言模型也是如此。为了解决这个问题，我们介绍了一种新颖的培训训练的政治话语语言模型，称为罗斯伯特。在专业领域的语言模型预先培训提出了不同的技术和语言挑战，需要广泛的计算资源和大规模数据。罗斯伯特（Roosebert）接受了大规模的政治辩论和演讲语料库（8K辩论，每个辩论，每个辩论，每个由不同的主题组成）。为了评估其表现，我们对与政治辩论分析有关的四个下游任务进行了微调，即指定的实体识别，情感分析，论证成分检测和分类以及参数关系预测和分类。我们的结果表明，在这四个任务上，对通用语言模型进行了显着改善，强调了特定于领域的预训练如何在政治辩论分析中提高绩效。我们为研究界发布了罗斯伯特语言模型。

Title: Pay What LLM Wants: Can LLM Simulate Economics Experiment with 522 Real-human Persona?

• Authors: Junhyuk Choi, Hyeonchu Park, Haemin Lee, Hyebeen Shin, Hyun Joung Jin, Bugeun Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03262
• Pdf URL: https://arxiv.org/pdf/2508.03262
• Copy Paste: [[2508.03262]] Pay What LLM Wants: Can LLM Simulate Economics Experiment with 522 Real-human Persona?(https://arxiv.org/abs/2508.03262)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: Recent advances in Large Language Models (LLMs) have generated significant interest in their capacity to simulate human-like behaviors, yet most studies rely on fictional personas rather than actual human data. We address this limitation by evaluating LLMs' ability to predict individual economic decision-making using Pay-What-You-Want (PWYW) pricing experiments with real 522 human personas. Our study systematically compares three state-of-the-art multimodal LLMs using detailed persona information from 522 Korean participants in cultural consumption scenarios. We investigate whether LLMs can accurately replicate individual human choices and how persona injection methods affect prediction performance. Results reveal that while LLMs struggle with precise individual-level predictions, they demonstrate reasonable group-level behavioral tendencies. Also, we found that commonly adopted prompting techniques are not much better than naive prompting methods; reconstruction of personal narrative nor retrieval augmented generation have no significant gain against simple prompting method. We believe that these findings can provide the first comprehensive evaluation of LLMs' capabilities on simulating economic behavior using real human data, offering empirical guidance for persona-based simulation in computational social science.
• 摘要：大型语言模型（LLM）的最新进展对模拟人类行为的能力产生了重大兴趣，但是大多数研究都依赖虚构的角色，而不是实际的人类数据。我们通过评估LLMS使用付费款项（PWYW）的定价实验来预测个人经济决策的能力来解决这一限制。我们的研究系统地使用来自522个韩国参与者的文化消费方案中的522名参与者的详细角色信息来比较三个最先进的多模式LLM。我们研究LLM是否可以准确复制个人选择以及角色注射方法如何影响预测性能。结果表明，尽管LLM在精确的个人水平预测中挣扎，但它们表现出合理的群体级行为趋势。另外，我们发现通常采用的提示技术并不比天真的提示方法好得多。重建个人叙事或检索增强产生没有明显的收益，而对于简单的提示方法。我们认为，这些发现可以为LLMS使用真实人类数据模拟经济行为的能力提供首次全面评估，从而为基于角色的计算社会科学模拟提供经验指南。

Title: LECTOR: LLM-Enhanced Concept-based Test-Oriented Repetition for Adaptive Spaced Learning

• Authors: Jiahao Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03275
• Pdf URL: https://arxiv.org/pdf/2508.03275
• Copy Paste: [[2508.03275]] LECTOR: LLM-Enhanced Concept-based Test-Oriented Repetition for Adaptive Spaced Learning(https://arxiv.org/abs/2508.03275)
• Keywords: language model, llm
• Abstract: Spaced repetition systems are fundamental to efficient learning and memory retention, but existing algorithms often struggle with semantic interference and personalized adaptation. We present LECTOR (\textbf{L}LM-\textbf{E}nhanced \textbf{C}oncept-based \textbf{T}est-\textbf{O}riented \textbf{R}epetition), a novel adaptive scheduling algorithm specifically designed for test-oriented learning scenarios, particularly language examinations where success rate is paramount. LECTOR leverages large language models for semantic analysis while incorporating personalized learning profiles, addressing the critical challenge of semantic confusion in vocabulary learning by utilizing LLM-powered semantic similarity assessment and integrating it with established spaced repetition principles. Our comprehensive evaluation against six baseline algorithms (SSP-MMC, SM2, HLR, FSRS, ANKI, THRESHOLD) across 100 simulated learners over 100 days demonstrates significant improvements: LECTOR achieves a 90.2\% success rate compared to 88.4\% for the best baseline (SSP-MMC), representing a 2.0\% relative improvement. The algorithm shows particular strength in handling semantically similar concepts, reducing confusion-induced errors while maintaining computational efficiency. Our results establish LECTOR as a promising direction for intelligent tutoring systems and adaptive learning platforms.
• 摘要：间隔重复系统是有效学习和保留记忆力的基础，但是现有的算法通常会在语义干扰和个性化适应中挣扎。我们介绍LECTOR（\ textbf {lm- \ textbf {e} nhanced \ textbf {c}基于\ textbf {t} est- \ textbf {t} est- \ textbf {o} rentient \ textbf {o textbf {r textbf {r}尤其是针对新型的调整alggor algorthe alggor alggor testin，以专门设计的情况下设计。费率至关重要。 Lector利用大型语言模型进行语义分析，同时纳入了个性化的学习概况，通过利用LLM驱动的语义相似性评估，解决词汇学习中语义混乱的关键挑战，并将其与已确立的间隔重复原则相结合。我们对六种基线算法（SSP-MMC，SM2，HLR，FSR，ANKI，ANKI，Threshold）的全面评估在100天内进行了100天的模拟学习者：与最佳基线（SSP-MMC）相比，Loctor取得了90.2 \％的成功率。该算法在处理语义相似的概念时显示出特殊的强度，从而减少了混淆引起的错误，同时保持了计算效率。我们的结果确定了Lector，成为智能辅导系统和自适应学习平台的有希望的方向。

Title: Do language models accommodate their users? A study of linguistic convergence

• Authors: Terra Blevins, Susanne Schmalwieser, Benjamin Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03276
• Pdf URL: https://arxiv.org/pdf/2508.03276
• Copy Paste: [[2508.03276]] Do language models accommodate their users? A study of linguistic convergence(https://arxiv.org/abs/2508.03276)
• Keywords: language model, llm
• Abstract: While large language models (LLMs) are generally considered proficient in generating language, how similar their language usage is to that of humans remains understudied. In this paper, we test whether models exhibit linguistic convergence, a core pragmatic element of human language communication, asking: do models adapt, or converge, to the linguistic patterns of their user? To answer this, we systematically compare model completions of exisiting dialogues to the original human responses across sixteen language models, three dialogue corpora, and a variety of stylometric features. We find that models strongly converge to the conversation's style, often significantly overfitting relative to the human baseline. While convergence patterns are often feature-specific, we observe consistent shifts in convergence across modeling settings, with instruction-tuned and larger models converging less than their pretrained counterparts. Given the differences between human and model convergence patterns, we hypothesize that the underlying mechanisms for these behaviors are very different.
• 摘要：虽然通常认为大型语言模型（LLM）熟练地熟练了语言，但他们的语言使用与人类的使用程度仍然被忽略了。在本文中，我们测试模型是否表现出语言融合，这是人类语言交流的核心务实元素，问：模型是否适应或收敛于用户的语言模式？为了回答这个问题，我们将“向对话”的模型完成与16个语言模型，三个对话Corpora和各种风格特征的原始人类响应进行比较。我们发现，模型强烈融合了对话的风格，通常相对于人类基线而过于拟合。虽然收敛模式通常是特定于功能的，但我们观察到跨建模设置的收敛性持续变化，而指令调整和较大的模型的收敛性少于其预处理的对应物。鉴于人与模型收敛模式之间的差异，我们假设这些行为的基本机制是非常不同的。

Title: Investigating Gender Bias in LLM-Generated Stories via Psychological Stereotypes

• Authors: Shahed Masoudian, Gustavo Escobedo, Hannah Strauss, Markus Schedl
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03292
• Pdf URL: https://arxiv.org/pdf/2508.03292
• Copy Paste: [[2508.03292]] Investigating Gender Bias in LLM-Generated Stories via Psychological Stereotypes(https://arxiv.org/abs/2508.03292)
• Keywords: language model, llm, prompt
• Abstract: As Large Language Models (LLMs) are increasingly used across different applications, concerns about their potential to amplify gender biases in various tasks are rising. Prior research has often probed gender bias using explicit gender cues as counterfactual, or studied them in sentence completion and short question answering tasks. These formats might overlook more implicit forms of bias embedded in generative behavior of longer content. In this work, we investigate gender bias in LLMs using gender stereotypes studied in psychology (e.g., aggressiveness or gossiping) in an open-ended task of narrative generation. We introduce a novel dataset called StereoBias-Stories containing short stories either unconditioned or conditioned on (one, two, or six) random attributes from 25 psychological stereotypes and three task-related story endings. We analyze how the gender contribution in the overall story changes in response to these attributes and present three key findings: (1) While models, on average, are highly biased towards male in unconditioned prompts, conditioning on attributes independent from gender stereotypes mitigates this bias. (2) Combining multiple attributes associated with the same gender stereotype intensifies model behavior, with male ones amplifying bias and female ones alleviating it. (3) Model biases align with psychological ground-truth used for categorization, and alignment strength increases with model size. Together, these insights highlight the importance of psychology-grounded evaluation of LLMs.
• 摘要：由于大型语言模型（LLM）越来越多地在不同的应用程序中使用，因此对它们在各种任务中扩大性别偏见的潜力的担忧正在上升。先前的研究经常使用明确的性别线索作为反事实来探讨性别偏见，或者在句子完成和简短回答任务中进行了研究。这些格式可能会忽略更隐含的偏见形式，这些形式嵌入了较长内容的生成行为中。在这项工作中，我们使用在叙事产生的开放式任务中使用心理学（例如，侵略性或闲话）研究的性别刻板印象来调查LLMS中的性别偏见。我们介绍了一个新颖的数据集，名为“立体恐惧症”故事，其中包含无条件或以25种心理刻板印象和三个与任务相关的故事结尾的（一，二或六个）随机属性（一，二或六个）随机属性的短篇小说。我们分析了整体故事中的性别贡献如何响应这些属性，并提出了三个关键发现：（1）尽管模型平均在无条件的提示中对男性高度偏见，从而依靠与性别刻板印象相关的属性，从而减轻了这种偏见。 （2）结合与相同性别刻板印象相关的多个属性加剧了模型行为，而男性属性会放大偏见和女性减轻偏见。 （3）模型偏向与用于分类的心理基础真相一致，并且对齐强度随模型大小而增加。这些见解共同凸显了对LLM的心理学评估的重要性。

Title: NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty

• Authors: Leonidas Zotos, Ivo Pascal de Jong, Matias Valdenegro-Toro, Andreea Ioana Sburlea, Malvina Nissim, Hedderik van Rijn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03294
• Pdf URL: https://arxiv.org/pdf/2508.03294
• Copy Paste: [[2508.03294]] NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty(https://arxiv.org/abs/2508.03294)
• Keywords: language model, llm
• Abstract: Estimating the difficulty of exam questions is essential for developing good exams, but professors are not always good at this task. We compare various Large Language Model-based methods with three professors in their ability to estimate what percentage of students will give correct answers on True/False exam questions in the areas of Neural Networks and Machine Learning. Our results show that the professors have limited ability to distinguish between easy and difficult questions and that they are outperformed by directly asking Gemini 2.5 to solve this task. Yet, we obtained even better results using uncertainties of the LLMs solving the questions in a supervised learning setting, using only 42 training samples. We conclude that supervised learning using LLM uncertainty can help professors better estimate the difficulty of exam questions, improving the quality of assessment.
• 摘要：估计考试问题的难度对于发展良好考试至关重要，但是教授并不总是擅长这项任务。我们将各种基于语言模型的大型方法与三位教授进行比较，以估计在神经网络和机器学习领域的True/False考试问题中，哪些比例将给出了哪些比例的学生。我们的结果表明，教授区分简单和困难问题的能力有限，并且通过直接要求Gemini 2.5解决这项任务而表现出色。但是，我们使用LLM的不确定性在监督学习环境中仅使用42个培训样本来解决了更好的结果。我们得出的结论是，使用LLM不确定性的监督学习可以帮助教授更好地估计考试问题的困难，从而提高评估质量。

Title: Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling

• Authors: Anqi Li, Wenwei Jin, Jintao Tong, Pengda Qin, Weijia Li, Guo Lu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.03296
• Pdf URL: https://arxiv.org/pdf/2508.03296
• Copy Paste: [[2508.03296]] Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling(https://arxiv.org/abs/2508.03296)
• Keywords: prompt
• Abstract: Social platforms have revolutionized information sharing, but also accelerated the dissemination of harmful and policy-violating content. To ensure safety and compliance at scale, moderation systems must go beyond efficiency and offer accuracy and interpretability. However, current approaches largely rely on noisy, label-driven learning, lacking alignment with moderation rules and producing opaque decisions that hinder human review. Therefore, we propose Hierarchical Guard (Hi-Guard), a multimodal moderation framework that introduces a new policy-aligned decision paradigm. The term "Hierarchical" reflects two key aspects of our system design: (1) a hierarchical moderation pipeline, where a lightweight binary model first filters safe content and a stronger model handles fine-grained risk classification; and (2) a hierarchical taxonomy in the second stage, where the model performs path-based classification over a hierarchical taxonomy ranging from coarse to fine-grained levels. To ensure alignment with evolving moderation policies, Hi-Guard directly incorporates rule definitions into the model prompt. To further enhance structured prediction and reasoning, we introduce a multi-level soft-margin reward and optimize with Group Relative Policy Optimization (GRPO), penalizing semantically adjacent misclassifications and improving explanation quality. Extensive experiments and real-world deployment demonstrate that Hi-Guard achieves superior classification accuracy, generalization, and interpretability, paving the way toward scalable, transparent, and trustworthy content safety systems. Code is available at: this https URL.
• 摘要：社会平台彻底改变了信息共享，但也加速了对有害和政策竞争内容的传播。为了确保大规模的安全性和合规性，适度系统必须超越效率并提供准确性和解释性。但是，当前的方法在很大程度上依赖于嘈杂的，标签驱动的学习，缺乏与节制规则保持一致，并且产生了阻碍人类评论的不透明决策。因此，我们提出了分层卫队（Hi-Guard），这是一个多式联运框架，引入了新的政策对准决策范式。 “层次结构”一词反映了我们系统设计的两个关键方面：（1）层次结构调节管道，轻质二进制模型首先过滤安全的内容和更强大的模型处理精细颗粒的风险分类； （2）在第二阶段的层次分类法，该模型对层次分类法进行了基于路径的分类，从粗大到细粒度。为了确保与不断发展的节制策略保持一致，Hi-Guard将规则定义直接纳入模型提示。为了进一步增强结构化的预测和推理，我们引入了多级软性奖励奖励，并通过小组相对政策优化（GRPO）进行优化，对语义上相邻的错误分类进行惩罚并提高解释质量。广泛的实验和现实世界的部署表明，Hi-Guard可实现出色的分类准确性，概括性和解释性，为可扩展，透明和值得信赖的内容安全系统铺平了道路。代码可用：此HTTPS URL。

Title: CTTS: Collective Test-Time Scaling

• Authors: Zhende Song, Shengji Tang, Peng Ye, Jiayuan Fan, Tao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03333
• Pdf URL: https://arxiv.org/pdf/2508.03333
• Copy Paste: [[2508.03333]] CTTS: Collective Test-Time Scaling(https://arxiv.org/abs/2508.03333)
• Keywords: language model, llm, agent
• Abstract: Test-time scaling (TTS) has emerged as a promising research field for enhancing the effectiveness of large language models (LLMs) without extra training. However, most existing approaches, e.g., Best-of-N and Self-Consistency rely on a single agent interacting with a reward model (SA-SR), constrained by limited capabilities of a single test-time scaling (STTS) paradigm. On the other hand, recent works demonstrate that collective-agent methods can break through the upper bound of single-agent systems by orchestrating diverse models. Thus, in this paper, we take a first step towards exploring Collective Test-Time Scaling (CTTS). Consider the different interaction types of single and multiple models, we design three primary paradigms to investigate the optimal paradigm of CTTS: (1) single agent to multiple reward models (SA-MR); (2) multiple agents to single reward model (MA-SR); and (3) multiple agents to multiple reward models (MA-MR). Extensive experiments demonstrate that MA-MR consistently achieves the best performance. Based on this, we propose a novel framework named CTTS-MM that effectively leverages both multi-agent and multi-reward-model collaboration for enhanced inference. Specifically, for multi-agent collaboration, we propose an Agent Collaboration Search (ACS), which searches for the most effective combination of LLM agents from a large candidate pool; for multi-reward-model collaboration, we propose Mixture of Reword Models (MoR), which consists of a curated question pool and a Prior Reward model Ensemble Selection (PRES) to select the optimal combinations of reward models via Pair-wise Reward Ranking (PRR) metric. Experiments across seven mainstream benchmarks demonstrate that the proposed CTTS-MM consistently obtains superior performance. Code will be released at this https URL.
• 摘要：测试时间缩放（TTS）已成为一个有前途的研究领域，可以在没有额外培训的情况下增强大语模型（LLM）的有效性。但是，大多数现有的方法，例如，最佳N和自我一致性依赖于与奖励模型（SA-SR）相互作用的单个代理，受到单个测试时间缩放（STTS）范式的有限功能的约束。另一方面，最近的著作表明，集体代理方法可以通过策划各种模型来突破单代机系统的上限。因此，在本文中，我们迈出了探索集体测试时间缩放（CTT）的第一步。考虑单个模型和多个模型的不同相互作用类型，我们设计了三个主要范式来研究CTT的最佳范式：（1）单个代理到多个奖励模型（SA-MR）； （2）单个奖励模型（MA-SR）的多种代理； （3）多种奖励模型（MA-MR）的多种代理。广泛的实验表明，MA-MR始终达到最佳性能。基于此，我们提出了一个名为CTTS-MM的新型框架，该框架有效地利用了多代理和多回报模型协作来增强推理。具体来说，对于多代理协作，我们提出了一个代理协作搜索（ACS），该搜索搜索了大型候选池中最有效的LLM代理的组合；对于多回报模型的合作，我们提出了换句模型（MOR）的混合物，该模型由一个精选的问题库和先前的奖励模型集合选择（PRES）组成，以通过配对奖励排名（PRR）度量选择奖励模型的最佳组合。七个主流基准的实验表明，所提出的CTTS-MM始终获得较高的性能。代码将在此HTTPS URL上发布。

Title: Taggus: An Automated Pipeline for the Extraction of Characters' Social Networks from Portuguese Fiction Literature

• Authors: Tiago G Canário, Catarina Duarte, Flávio L. Pinheiro, João L.M. Pereira
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2508.03358
• Pdf URL: https://arxiv.org/pdf/2508.03358
• Copy Paste: [[2508.03358]] Taggus: An Automated Pipeline for the Extraction of Characters' Social Networks from Portuguese Fiction Literature(https://arxiv.org/abs/2508.03358)
• Keywords: language model, gpt, chat
• Abstract: Automatically identifying characters and their interactions from fiction books is, arguably, a complex task that requires pipelines that leverage multiple Natural Language Processing (NLP) methods, such as Named Entity Recognition (NER) and Part-of-speech (POS) tagging. However, these methods are not optimized for the task that leads to the construction of Social Networks of Characters. Indeed, the currently available methods tend to underperform, especially in less-represented languages, due to a lack of manually annotated data for training. Here, we propose a pipeline, which we call Taggus, to extract social networks from literary fiction works in Portuguese. Our results show that compared to readily available State-of-the-Art tools -- off-the-shelf NER tools and Large Language Models (ChatGPT) -- the resulting pipeline, which uses POS tagging and a combination of heuristics, achieves satisfying results with an average F1-Score of $94.1\%$ in the task of identifying characters and solving for co-reference and $75.9\%$ in interaction detection. These represent, respectively, an increase of $50.7\%$ and $22.3\%$ on results achieved by the readily available State-of-the-Art tools. Further steps to improve results are outlined, such as solutions for detecting relationships between characters. Limitations on the size and scope of our testing samples are acknowledged. The Taggus pipeline is publicly available to encourage development in this field for the Portuguese language.2
• 摘要：可以自动从小说书籍中识别字符及其相互作用是一项复杂的任务，它需要利用多种自然语言处理（NLP）方法的管道，例如命名实体识别（NER）和部分语音（POS）标记。但是，这些方法未针对导致角色的社交网络的任务进行优化。实际上，由于缺乏手动注释的培训数据，目前可用的方法往往表现不佳，尤其是在代表性较低的语言中。在这里，我们提出了一条称为Taggus的管道，以从葡萄牙语中的文学小说中提取社交网络。 Our results show that compared to readily available State-of-the-Art tools -- off-the-shelf NER tools and Large Language Models (ChatGPT) -- the resulting pipeline, which uses POS tagging and a combination of heuristics, achieves satisfying results with an average F1-Score of $94.1\%$ in the task of identifying characters and solving for co-reference and $75.9\%$ in interaction detection.这些分别代表了随时可用的最先进的工具所取得的结果，分别增加了$ 50.7 \％$ $和22.3 \％$。概述了改善结果的进一步步骤，例如用于检测字符之间关系的解决方案。确认了对我们测试样本的大小和范围的限制。 Taggus管道可公开使用，以鼓励在该领域开发葡萄牙语。2

Title: Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models

• Authors: Haotian Wu, Bo Xu, Yao Shu, Menglin Yang, Chengwei Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03363
• Pdf URL: https://arxiv.org/pdf/2508.03363
• Copy Paste: [[2508.03363]] Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models(https://arxiv.org/abs/2508.03363)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Reasoning large language models (RLLMs) have recently demonstrated remarkable capabilities through structured and multi-step reasoning. While prior research has primarily focused on improving their training and inference strategies, their potential for in-context learning (ICL) remains largely underexplored. To fill this gap, we propose Thinking with Nothinking Calibration (JointThinking), a new ICL paradigm that leverages the structured difference between two reasoning modes, i.e., Thinking and Nothinking, to improve reasoning accuracy. Specifically, our method prompts the model to generate two answers in parallel: one in Thinking mode and the other in Nothinking mode. A second round of Thinking is triggered only when the two initial responses are inconsistent, using a single prompt that incorporates the original question and both candidate answers. Since such disagreement occurs infrequently (e.g., only 6\% in GSM8K), our method performs just one round of reasoning in most cases, resulting in minimal latency overhead. Extensive experiments across multiple reasoning benchmarks demonstrate that JointThinking significantly outperforms few-shot chain-of-thought (CoT) and majority voting with improved answer robustness. Moreover, It achieves comparable in-distribution performance to training-based SOTA method, while substantially outperforming on out-of-distribution tasks. We further conduct a systematic analysis of the calibration mechanism, showing that leveraging different reasoning modes consistently lowers the error rate and highlights the value of structural thinking diversity. Additionally, we observe that the performance gap between actual and ideal reasoning narrows as model size increases in the second round of thinking, indicating the strong scalability of our approach. Finally, we discuss current limitations and outline promising directions for future ICL research in RLLMs.
• 摘要：推理大语言模型（RLLM）最近通过结构化和多步推理证明了出色的功能。虽然先前的研究主要集中于改善其培训和推理策略，但其潜在的文化学习潜力（ICL）仍然在很大程度上没有被淘汰。为了填补这一空白，我们提出了通过NotHink校准（联合思考）进行思考，这是一种新的ICL范式，利用了两种推理模式之间的结构性差异，即思考和不进行，以提高推理精度。具体而言，我们的方法促使模型并行生成两个答案：一个在思维模式下，另一个在NotHinking模式下。仅当两个初始响应不一致时，只有在包含原始问题和两个候选答案的单个提示中，才会触发第二轮思维。由于这种分歧很少发生（例如，GSM8K中只有6％），因此我们的方法在大多数情况下仅执行一轮推理，从而导致延迟最小。跨多个推理基准的广泛实验表明，联合思考显着优于几乎没有想法的链（COT）和多数投票，而回答鲁棒性的提高。此外，它与基于培训的SOTA方法相当地表现出可比的分布性能，同时在分布外任务上的表现基本上要优于分布性能。我们进一步对校准机制进行系统分析，表明利用不同的推理模式始终降低错误率并突出结构思维多样性的价值。此外，我们观察到，随着模型大小在第二轮思维的增加，实际和理想推理之间的性能差异变窄，这表明我们方法的可扩展性很强。最后，我们讨论了当前的局限性，并概述了RLLM中未来ICL研究的有希望的方向。

Title: LLMs Have a Heart of Stone: Demystifying the Soft Thinking Ability of Large Reasoning Models

• Authors: Junhong Wu, Jinliang Lu, Zixuan Ren, Ganqiang Hu, Zhi Wu, Dai Dai, Hua Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03440
• Pdf URL: https://arxiv.org/pdf/2508.03440
• Copy Paste: [[2508.03440]] LLMs Have a Heart of Stone: Demystifying the Soft Thinking Ability of Large Reasoning Models(https://arxiv.org/abs/2508.03440)
• Keywords: language model, llm
• Abstract: Human cognition naturally engages with abstract and fluid concepts, whereas existing reasoning models often rely on generating discrete tokens, potentially constraining their expressive capabilities. Recent advancements aim to address this limitation by enabling large language models (LLMs) to generate soft, abstract tokens, thus facilitating reasoning within a continuous concept space. This paper explores the `Soft Thinking' capabilities of various LLMs by examining the models' internal behavior using a suite of probing techniques. Contrary to the common belief that Soft Thinking enables the simultaneous exploration of diverse reasoning paths, our findings reveal that LLMs predominantly rely on the most influential component of the soft inputs during subsequent decoding steps. This reliance hinders the exploration of different reasoning paths and reduces vanilla Soft Thinking to a form of greedy decoding, obscuring the advantage of transmitting more information through Soft Tokens. To tackle this issue, we explore sampling strategies to introduce \emph{randomness}, employing methods such as Dirichlet resampling and the Gumbel-Softmax trick. Our experiments demonstrate that incorporating randomness can alleviate the limitations of vanilla approaches and unleash the potential of Soft Thinking. Notably, the Gumbel-Softmax trick provides adequate randomness with controlled smoothness, resulting in superior performance across eight reasoning benchmarks.
• 摘要：人类认知自然会参与抽象和流体概念，而现有的推理模型通常依赖于产生离散令牌，从而可能限制其表现力能力。最近的进步旨在通过使大型语言模型（LLM）能够生成柔软的抽象令牌，从而促进连续概念空间中的推理，从而解决这一限制。本文通过使用一套探测技术检查模型的内部行为来探讨各种LLM的“软思维”功能。与普遍的信念相反，即软思维能够同时探索各种推理路径，我们的发现表明，LLMS主要依赖于随后的解码步骤中软输入最有影响力的组成部分。这种依赖阻碍了对不同推理路径的探索，并将香草软思维减少到一种贪婪的解码形式，从而掩盖了通过软令牌传输更多信息的优势。为了解决此问题，我们探索采样策略，以使用Dirichlet重新采样和Gumbel-Softmax Trick等方法引入\ Emph {Randoments}。我们的实验表明，结合随机性可以减轻香草方法的局限性并释放出软思维的潜力。值得注意的是，Gumbel-Softmax技巧提供了充分的随机性，并具有控制的平滑度，从而在八个推理基准中提供了卓越的性能。

Title: Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings

• Authors: Rita González-Márquez, Philipp Berens, Dmitry Kobak
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.03453
• Pdf URL: https://arxiv.org/pdf/2508.03453
• Copy Paste: [[2508.03453]] Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings(https://arxiv.org/abs/2508.03453)
• Keywords: language model, retrieval-augmented generation
• Abstract: Text embeddings, i.e. vector representations of entire texts, play an important role in many NLP applications, such as retrieval-augmented generation, sentiment analysis, clustering, or visualizing collections of texts for data exploration. Currently, top-performing embedding models are derived from pre-trained language models via extensive supervised fine-tuning using curated text pairs. This contrasts with computer vision, where self-supervised training based on data augmentations has demonstrated remarkable success. Here we systematically compare the two most well-known augmentation strategies for positive pair generation in contrastive learning of text embeddings. We assess embedding quality on MTEB and additional in-domain evaluations and show that cropping augmentation strongly outperforms the dropout-based approach. We find that on out-of-domain data, the quality of resulting embeddings is below the supervised SOTA models, but for in-domain data, self-supervised fine-tuning produces high-quality text embeddings after very short fine-tuning, sometimes only marginally below the supervised SOTA. Finally, we show that representation quality increases towards the last transformer layers, which undergo the largest change during fine-tuning; and that fine-tuning only those last layers is sufficient to reach similar embedding quality.
• 摘要：文本嵌入，即整个文本的矢量表示，在许多NLP应用程序中都起着重要作用，例如检索效果的生成，情感分析，聚类或可视化文本集合以进行数据探索。当前，最佳表现的嵌入模型是通过使用策划的文本对的广泛监督微调来得出的。这与计算机视觉形成鲜明对比，在计算机的视野中，基于数据增强的自我监督培训表现出了显着的成功。在这里，我们从系统地比较了两种最著名的增强策略，用于对比对比文本嵌入的对比学习。我们评估了MTEB上的嵌入质量和其他内域评估，并表明裁剪的增强非常优于基于辍学的方法。我们发现，在室外数据中，所得嵌入的质量低于监督的SOTA模型，但是对于内域数据，自我监督的微调可产生高质量的文本嵌入在非常短的微调之后，有时仅在监督的SOTA以下略微。最后，我们表明表示质量质量会提高到最后的变压器层，在微调过程中发生了最大的变化。只有那些最后一层的微调足以达到类似的嵌入质量。

Title: CF-RAG: A Dataset and Method for Carbon Footprint QA Using Retrieval-Augmented Generation

• Authors: Kaiwen Zhao, Bharathan Balaji, Stephen Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03489
• Pdf URL: https://arxiv.org/pdf/2508.03489
• Copy Paste: [[2508.03489]] CF-RAG: A Dataset and Method for Carbon Footprint QA Using Retrieval-Augmented Generation(https://arxiv.org/abs/2508.03489)
• Keywords: gpt, llm, retrieval-augmented generation
• Abstract: Product sustainability reports provide valuable insights into the environmental impacts of a product and are often distributed in PDF format. These reports often include a combination of tables and text, which complicates their analysis. The lack of standardization and the variability in reporting formats further exacerbate the difficulty of extracting and interpreting relevant information from large volumes of documents. In this paper, we tackle the challenge of answering questions related to carbon footprints within sustainability reports available in PDF format. Unlike previous approaches, our focus is on addressing the difficulties posed by the unstructured and inconsistent nature of text extracted from PDF parsing. To facilitate this analysis, we introduce CarbonPDF-QA, an open-source dataset containing question-answer pairs for 1735 product report documents, along with human-annotated answers. Our analysis shows that GPT-4o struggles to answer questions with data inconsistencies. To address this limitation, we propose CarbonPDF, an LLM-based technique specifically designed to answer carbon footprint questions on such datasets. We develop CarbonPDF by fine-tuning Llama 3 with our training data. Our results show that our technique outperforms current state-of-the-art techniques, including question-answering (QA) systems finetuned on table and text data.
• 摘要：产品可持续性报告为产品的环境影响提供了宝贵的见解，并且通常以PDF格式分发。这些报告通常包括表和文本的组合，这使他们的分析变得复杂。缺乏标准化和报告格式的可变性进一步加剧了从大量文档中提取和解释相关信息的困难。在本文中，我们解决了回答与PDF格式可持续性报告中与碳足迹有关的问题的挑战。与以前的方法不同，我们的重点是解决从PDF解析中提取的文本的非结构化和不一致性质所带来的困难。为了促进此分析，我们介绍了CarbonPDF-QA，这是一个开源数据集，其中包含1735个产品报告文档的问答对以及人类宣传的答案。我们的分析表明，GPT-4O难以通过数据不一致回答问题。为了解决这一限制，我们提出了CarbonPDF，这是一种基于LLM的技术，专门旨在在此类数据集上回答碳足迹问题。我们通过培训数据通过微调乳拉3来开发CarbonPDF。我们的结果表明，我们的技术优于当前的最新技术，包括在桌面和文本数据上进行验证的问题（QA）系统。

Title: UPLME: Uncertainty-Aware Probabilistic Language Modelling for Robust Empathy Regression

• Authors: Md Rakibul Hasan, Md Zakir Hossain, Aneesh Krishna, Shafin Rahman, Tom Gedeon
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.03520
• Pdf URL: https://arxiv.org/pdf/2508.03520
• Copy Paste: [[2508.03520]] UPLME: Uncertainty-Aware Probabilistic Language Modelling for Robust Empathy Regression(https://arxiv.org/abs/2508.03520)
• Keywords: language model
• Abstract: Supervised learning for empathy regression is challenged by noisy self-reported empathy scores. While many algorithms have been proposed for learning with noisy labels in textual classification problems, the regression counterpart is relatively under-explored. We propose UPLME, an uncertainty-aware probabilistic language modelling framework to capture label noise in the regression setting of empathy detection. UPLME includes a probabilistic language model that predicts both empathy score and heteroscedastic uncertainty and is trained using Bayesian concepts with variational model ensembling. We further introduce two novel loss components: one penalises degenerate Uncertainty Quantification (UQ), and another enforces the similarity between the input pairs on which we predict empathy. UPLME provides state-of-the-art performance (Pearson Correlation Coefficient: $0.558\rightarrow0.580$ and $0.629\rightarrow0.634$) in terms of the performance reported in the literature in two public benchmarks, having label noise. Through synthetic label noise injection, we show that UPLME is effective in separating noisy and clean samples based on the predicted uncertainty. UPLME further outperform (Calibration error: $0.571\rightarrow0.376$) a recent variational model ensembling-based UQ method designed for regression problems.
• 摘要：嘈杂的自我报告的移情得分挑战了同理心回归的监督学习。尽管已经提出了许多用于在文本分类问题中使用嘈杂标签学习的算法，但回归对应物的探索相对较小。我们提出了UPLME，这是一种不确定性感知的概率语言建模框架，以在移情检测的回归设置中捕获标签噪声。 UPLME包括一个概率语言模型，该模型可以预测移情得分和异质性不确定性，并使用具有变化模型结合的贝叶斯概念进行了训练。我们进一步介绍了两个新的损失组成部分：一种惩罚退化的不确定性定量（UQ），另一个惩罚了我们预测同理心的输入对之间的相似性。 UPLME提供了最先进的性能（Pearson相关系数：$ 0.558 \ rightArrow0.580 $和$ 0.629 \ rightArrow0.634 $），以在两个公共基准的文献中报告的性能，具有标签噪音。通过合成标记噪声注入，我们表明UPLME可以根据预测的不确定性有效分离嘈杂和干净的样品。 UPLME进一步跑赢大盘（校准错误：$ 0.571 \ rightArrow0.376 $）最近基于变异模型的基于连接的UQ方法设计用于回归问题。

Title: FilBench: Can LLMs Understand and Generate Filipino?

• Authors: Lester James V. Miranda, Elyanah Aco, Conner Manuel, Jan Christian Blaise Cruz, Joseph Marvin Imperial
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03523
• Pdf URL: https://arxiv.org/pdf/2508.03523
• Copy Paste: [[2508.03523]] FilBench: Can LLMs Understand and Generate Filipino?(https://arxiv.org/abs/2508.03523)
• Keywords: gpt, llm
• Abstract: Despite the impressive performance of LLMs on English-based tasks, little is known about their capabilities in specific languages such as Filipino. In this work, we address this gap by introducing FilBench, a Filipino-centric benchmark designed to evaluate LLMs across a diverse set of tasks and capabilities in Filipino, Tagalog, and Cebuano. We carefully curate the tasks in FilBench to reflect the priorities and trends of NLP research in the Philippines such as Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation. By evaluating 27 state-of-the-art LLMs on FilBench, we find that several LLMs suffer from reading comprehension and translation capabilities. Our results indicate that FilBench is challenging, with the best model, GPT-4o, achieving only a score of 72.23%. Moreover, we also find that models trained specifically for Southeast Asian languages tend to underperform on FilBench, with the highest-performing model, SEA-LION v3 70B, achieving only a score of 61.07%. Our work demonstrates the value of curating language-specific LLM benchmarks to aid in driving progress on Filipino NLP and increasing the inclusion of Philippine languages in LLM development.
• 摘要：尽管LLM在基于英语的任务上的表现令人印象深刻，但对它们在菲律宾等特定语言中的能力知之甚少。在这项工作中，我们通过引入以菲律宾为中心的基准Filucbench来解决这一差距，旨在评估菲律宾，他加泰罗格和塞布诺的各种任务和能力的LLM。我们仔细策划了Filbench中的任务，以反映菲律宾NLP研究的优先事项和趋势，例如文化知识，经典NLP，阅读理解和世代。通过评估Filbench上的27个最先进的LLM，我们发现有几个LLM遭受了阅读理解和翻译能力的影响。我们的结果表明，Filbench具有最佳模型GPT-4O的挑战，仅获得72.23％的分数。此外，我们还发现，专门针对东南亚语言训练的模型往往在Filbench上表现不佳，其表现最高的模型是Sea-Lion V3 70B，仅获得61.07％的分数。我们的工作证明了策划特定语言的LLM基准的价值，以帮助推动菲律宾NLP的进度，并增加菲律宾语言在LLM开发中的纳入。

Title: Marito: Structuring and Building Open Multilingual Terminologies for South African NLP

• Authors: Vukosi Marivate, Isheanesu Dzingirai, Fiskani Banda, Richard Lastrucci, Thapelo Sindane, Keabetswe Madumo, Kayode Olaleye, Abiodun Modupe, Unarine Netshifhefhe, Herkulaas Combrink, Mohlatlego Nakeng, Matome Ledwaba
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03529
• Pdf URL: https://arxiv.org/pdf/2508.03529
• Copy Paste: [[2508.03529]] Marito: Structuring and Building Open Multilingual Terminologies for South African NLP(https://arxiv.org/abs/2508.03529)
• Keywords: language model, retrieval-augmented generation
• Abstract: The critical lack of structured terminological data for South Africa's official languages hampers progress in multilingual NLP, despite the existence of numerous government and academic terminology lists. These valuable assets remain fragmented and locked in non-machine-readable formats, rendering them unusable for computational research and development. \emph{Marito} addresses this challenge by systematically aggregating, cleaning, and standardising these scattered resources into open, interoperable datasets. We introduce the foundational \emph{Marito} dataset, released under the equitable, Africa-centered NOODL framework. To demonstrate its immediate utility, we integrate the terminology into a Retrieval-Augmented Generation (RAG) pipeline. Experiments show substantial improvements in the accuracy and domain-specific consistency of English-to-Tshivenda machine translation for large language models. \emph{Marito} provides a scalable foundation for developing robust and equitable NLP technologies, ensuring South Africa's rich linguistic diversity is represented in the digital age.
• 摘要：尽管存在许多政府和学术术语列表，但南非官方语言的结构性术语数据的严重缺乏使多语言NLP的进展阻碍。这些有价值的资产仍然以非光学可读格式分散和锁定，使它们无法使用计算研究和开发。 \ emph {marito}通过系统地汇总，清洁和标准化这些分散的资源将这些分散的资源分为开放的，可互操作的数据集来应对这一挑战。我们介绍了基础\ emph {marito}数据集，该数据集以公平的，以非洲为中心的NOODL框架发布。为了证明其直接的效用，我们将术语集成到检索型发电（RAG）管道中。实验表明，对于大型语言模型，英语到tshivenda机器翻译的准确性和特定于域的一致性方面有了很大的改善。 \ emph {Marito}为开发强大而公平的NLP技术提供了可扩展的基础，确保了南非在数字时代的丰富语言多样性。

Title: EmbedGrad: Gradient-Based Prompt Optimization in Embedding Space for Large Language Models

• Authors: Xiaoming Hou, Jiquan Zhang, Zibin Lin, DaCheng Tao, Shengli Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03533
• Pdf URL: https://arxiv.org/pdf/2508.03533
• Copy Paste: [[2508.03533]] EmbedGrad: Gradient-Based Prompt Optimization in Embedding Space for Large Language Models(https://arxiv.org/abs/2508.03533)
• Keywords: language model, prompt
• Abstract: Effectively adapting powerful pretrained foundation models to diverse tasks remains a key challenge in AI deployment. Current approaches primarily follow two paradigms:discrete optimization of text prompts through prompt engineering, or continuous adaptation via additional trainable parameters. Both exhibit limitations-discrete methods lack refinement precision while parameter-based techniques increase complexity and reduce interpretability. To address these constraints, we propose EmbedGrad, a novel framework that optimizes text prompt embeddings through gradient-based refinement. Our approach uniquely decouples training from deployment:during optimization,labeled examples guide precise embedding adjustments while preserving semantic meaning; during inference, only optimized embeddings integrate with user queries. This enables fine-grained calibration impossible in text space, such as enhancing the reasoning capability of prompts like please reason step by step. Comprehensive evaluations across mathematical reasoning, sentiment analysis, and causal judgment tasks demonstrate EmbedGrad's effectiveness:optimizing this reasoning prompt for Qwen2.5-Math-1.5B increased accuracy from 14.74\% to 58.96\% on mathematical problems. Consistent improvements were observed across model scales (0.5B-14B) and all tasks, with particularly significant gains for smaller models on complex problems like causal judgment. By bridging prompt engineering and parameter efficiency without architectural changes, our work establishes embedding refinement as a powerful new paradigm for task adaptation.
• 摘要：有效地将强大的审慎基础模型调整为各种任务仍然是AI部署的关键挑战。当前方法主要遵循两个范式：通过提示工程进行离散的文本提示，或通过其他可训练参数进行连续适应。两种均表现出局限性 - 二散方法缺乏改进精度，而基于参数的技术则提高了复杂性并降低了可解释性。为了解决这些约束，我们提出了嵌入式框架，这是一个新颖的框架，可通过基于梯度的改进来优化文本提示嵌入。我们的方法独特地将培训与部署培训：在优化，标记的示例中指南精确嵌入调整，同时保留语义含义；在推断期间，仅优化的嵌入与用户查询集成。这使得在文本空间中不可能进行细粒度的校准，例如增强提示的推理能力，例如请逐步推理。跨数学推理，情感分析和因果判断任务进行的全面评估证明了嵌入式的有效性：优化对QWEN2.5-MATH-1.5B的推理提示，从14.74 \％\％提高到58.96 \％\％。在模型量表（0.5B-14B）和所有任务中都观察到一致的改进，对于诸如因果判断之类的复杂问题，对于较小的模型而言，尤其显着提高。通过桥接工程和参数效率而没有架构更改，我们的工作将精炼嵌入到了任务适应的强大新范式中。

Title: Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations

• Authors: Peng Lai, Jianjie Zheng, Sijie Cheng, Yun Chen, Peng Li, Yang Liu, Guanhua Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03550
• Pdf URL: https://arxiv.org/pdf/2508.03550
• Copy Paste: [[2508.03550]] Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations(https://arxiv.org/abs/2508.03550)
• Keywords: language model, llm, prompt
• Abstract: The growing scale of evaluation tasks has led to the widespread adoption of automated evaluation using large language models, a paradigm known as "LLMas-a-judge." However, improving its alignment with human preferences without complex prompts or fine-tuning remains challenging. In this work, motivated by preliminary findings that middle-to-upper layers encode semantically and taskrelevant representations that are often more aligned with human judgments than the final layer, we propose LAGER, a lightweight and efficient framework for enhancing LLM-as-a-Judge alignment with human scoring, via internal representations. LAGER produces fine-grained judgment scores by aggregating cross-layer scoretoken logits and computing the expected score from a softmax-based distribution, with the LLM backbone kept frozen. LAGER fully leverages the complementary information across different layers, overcoming the limitations of relying solely on the final layer. We evaluate our method on the standard alignment benchmarks Flask, HelpSteer, and BIGGen using Spearman correlation, and find that LAGER achieves improvements of up to 7.5% over the best baseline across these benchmarks. Without reasoning steps, LAGER matches or outperforms reasoning-based methods. Experiments on downstream applications, such as data selection and emotional understanding, further show the effectiveness of our method.
• 摘要：评估任务的规模不断增长，导致使用大语言模型（一种称为“ llmas-a-aughde）的范式”广泛采用自动化评估。但是，在没有复杂的提示或微调的情况下，改善其与人类偏好的一致性仍然具有挑战性。在这项工作中，以初步发现的启发，即在语义上编码中层和任务相比，这些层通常比人类判断更加一致，我们建议通过内部表征来增强LLM-AS-AS-A-A-A-A-A-A-Judge与人类得分的一致性，以增强LLM-AS-AS-A-A-A-A-A-Judge一致性。 Lager通过汇总跨层的ScoreToken逻辑并计算基于软磁性的分布的预期得分，从而产生细粒度的判断分数，而LLM骨架保持冷冻。 Lager完全利用不同层的互补信息，从而克服了仅依赖最终层的局限性。我们使用Spearman相关性评估了标准对齐基准瓶，HelpSteer和BigGen的方法，并发现Lager的改善比这些基准的最佳基线高达7.5％。在没有推理步骤的情况下，啤酒匹配或优于基于推理的方法。关于数据选择和情感理解等下游应用程序的实验进一步显示了我们方法的有效性。

Title: Tackling Distribution Shift in LLM via KILO: Knowledge-Instructed Learning for Continual Adaptation

• Authors: Iing Muttakhiroh, Thomas Fevens
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2508.03571
• Pdf URL: https://arxiv.org/pdf/2508.03571
• Copy Paste: [[2508.03571]] Tackling Distribution Shift in LLM via KILO: Knowledge-Instructed Learning for Continual Adaptation(https://arxiv.org/abs/2508.03571)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) often suffer from performance degradation when faced with domain shifts, primarily due to catastrophic forgetting. In this work, we propose KILO (Knowledge-Instructed Learning for Continual Adaptation), a novel continual learning framework that integrates dynamic knowledge graphs with instruction tuning. By leveraging retrieved domain-specific knowledge as guidance during training, KILO enhances both adaptability to new domains and retention of previously acquired knowledge. We pretrain our model on WikiText-103 and evaluate sequential adaptation across four diverse target domains: BioASQ, SciQ, TweetEval, and MIND. Our experiments demonstrate that KILO consistently outperforms strong baselines, including continual fine-tuning, ERNIE 2.0, and CPT, in terms of backward transfer, forward transfer, F1 score, retention rate, and training efficiency. These results highlight the effectiveness of combining structured knowledge retrieval and instruction prompting to overcome domain shift challenges in continual learning scenarios.
• 摘要：面对领域变化时，大型语言模型（LLM）通常会遭受性能降解，这主要是由于灾难性的遗忘。在这项工作中，我们提出了Kilo（知识教学学习的持续适应），这是一个新颖的持续学习框架，将动态知识图与教学调整整合在一起。通过利用检索到特定领域的知识作为培训期间的指导，Kilo可以增强对新领域的适应性和先前获得的知识的保留。我们在Wikitext-103上预先介绍了模型，并评估了四个不同目标领域的顺序适应：Bioasq，Sciq，TweetEval和Mind。我们的实验表明，就向后转移，向前转移，F1分数，保留率和训练效率而言，Kilo始终胜过强大的基准，包括连续微调，Ernie 2.0和CPT。这些结果突出了结合结构化知识检索和指导的有效性，促使在连续学习场景中克服域转移挑战。

Title: Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?

• Authors: Wenxuan Shen, Mingjia Wang, Yaochen Wang, Dongping Chen, Junjie Yang, Yao Wan, Weiwei Lin
• Subjects: cs.CL, cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2508.03644
• Pdf URL: https://arxiv.org/pdf/2508.03644
• Copy Paste: [[2508.03644]] Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?(https://arxiv.org/abs/2508.03644)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems using Multimodal Large Language Models (MLLMs) show great promise for complex document understanding, yet their development is critically hampered by inadequate evaluation. Current benchmarks often focus on specific part of document RAG system and use synthetic data with incomplete ground truth and evidence labels, therefore failing to reflect real-world bottlenecks and challenges. To overcome these limitations, we introduce Double-Bench: a new large-scale, multilingual, and multimodal evaluation system that is able to produce fine-grained assessment to each component within document RAG systems. It comprises 3,276 documents (72,880 pages) and 5,168 single- and multi-hop queries across 6 languages and 4 document types with streamlined dynamic update support for potential data contamination issues. Queries are grounded in exhaustively scanned evidence pages and verified by human experts to ensure maximum quality and completeness. Our comprehensive experiments across 9 state-of-the-art embedding models, 4 MLLMs and 4 end-to-end document RAG frameworks demonstrate the gap between text and visual embedding models is narrowing, highlighting the need in building stronger document retrieval models. Our findings also reveal the over-confidence dilemma within current document RAG frameworks that tend to provide answer even without evidence support. We hope our fully open-source Double-Bench provide a rigorous foundation for future research in advanced document RAG systems. We plan to retrieve timely corpus and release new benchmarks on an annual basis.
• 摘要：使用多模式大语言模型（MLLM）检索提示的生成（RAG）系统对复杂的文档理解表现出了巨大的希望，但是由于评估不足，它们的发展受到严重阻碍。当前的基准通常关注文档抹布系统的特定部分，并使用具有不完整的地面真相和证据标签的合成数据，因此未能反映现实世界中的瓶颈和挑战。为了克服这些局限性，我们引入了双基础基础：一种新的大规模，多语言和多模式评估系统，能够为文档抹布系统中的每个组件进行细粒度评估。它包含3,276个文档（72,880页）和5,168个文档和5,168个单独和多跳的查询，跨6种语言和4种文档类型，具有简化的动态更新支持，以解决潜在的数据污染问题。查询基于详尽的扫描证据页面，并由人类专家进行验证，以确保最高质量和完整性。我们在9个最先进的嵌入模型，4个MLLM和4个端到端文档RAG框架上进行的全面实验证明了文本和视觉嵌入模型之间的差距正在缩小，这突出了建立更强文档检索模型的需求。我们的发现还揭示了当前文档框架内的过度信心困境，即使没有证据支持，也倾向于提供答案。我们希望我们的完全开源的双基础台基础为先进文档抹布系统的未来研究提供了严格的基础。我们计划每年及时检索及时的语料库并发布新的基准。

Title: Can Large Vision-Language Models Understand Multimodal Sarcasm?

• Authors: Xinyu Wang, Yue Zhang, Liqiang Jing
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2508.03654
• Pdf URL: https://arxiv.org/pdf/2508.03654
• Copy Paste: [[2508.03654]] Can Large Vision-Language Models Understand Multimodal Sarcasm?(https://arxiv.org/abs/2508.03654)
• Keywords: language model
• Abstract: Sarcasm is a complex linguistic phenomenon that involves a disparity between literal and intended meanings, making it challenging for sentiment analysis and other emotion-sensitive tasks. While traditional sarcasm detection methods primarily focus on text, recent approaches have incorporated multimodal information. However, the application of Large Visual Language Models (LVLMs) in Multimodal Sarcasm Analysis (MSA) remains underexplored. In this paper, we evaluate LVLMs in MSA tasks, specifically focusing on Multimodal Sarcasm Detection and Multimodal Sarcasm Explanation. Through comprehensive experiments, we identify key limitations, such as insufficient visual understanding and a lack of conceptual knowledge. To address these issues, we propose a training-free framework that integrates in-depth object extraction and external conceptual knowledge to improve the model's ability to interpret and explain sarcasm in multimodal contexts. The experimental results on multiple models show the effectiveness of our proposed framework. The code is available at this https URL.
• 摘要：讽刺是一种复杂的语言现象，涉及字面意义和预期含义之间的差异，这对于情感分析和其他对情感敏感的任务的挑战。尽管传统的讽刺检测方法主要集中在文本上，但最近的方法已包含多模式信息。但是，大型视觉语言模型（LVLM）在多模式讽刺分析（MSA）中的应用仍未得到充实。在本文中，我们评估了MSA任务中的LVLM，特别关注多模式讽刺检测和多模式讽刺解释。通过全面的实验，我们确定了关键的局限性，例如视觉理解不足和缺乏概念知识。为了解决这些问题，我们提出了一个无训练的框架，该框架集成了深入的对象提取和外部概念知识，以提高模型在多模式环境中解释和解释讽刺的能力。多个模型的实验结果显示了我们提出的框架的有效性。该代码可在此HTTPS URL上找到。

Title: CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction

• Authors: Zixuan Li, Binzong Geng, Jing Xiong, Yong He, Yuxuan Hu, Jian Chen, Dingwei Chen, Xiyu Chang, Liang Zhang, Linjian Mo, Chengming Li, Chuan Yuan, Zhenan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2508.03668
• Pdf URL: https://arxiv.org/pdf/2508.03668
• Copy Paste: [[2508.03668]] CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction(https://arxiv.org/abs/2508.03668)
• Keywords: language model
• Abstract: Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios.
• 摘要：点击率（CTR）预测是推荐系统中的核心任务，使用历史行为数据估算用户点击可能性。由于LMS强大的语义理解和上下文建模功能，将用户行为序列建模为文本以利用语言模型（LMS）进行此任务。但是，存在关键的结构差距：用户行为序列由通过语义上空的分离器连接的离散动作组成，与LM预训练中的连贯的自然语言不同。这种不匹配导致语义分散，其中LM注意力散布在无关的令牌上，而不是专注于有意义的行为边界和行为间的关系，从而降低了预测性能。为了解决这个问题，我们提出了$ \ textit {ctr-sink} $，这是一个新颖的框架，引入了为推荐方案量身定制的行为级别的关注点。受到关注点理论的启发，它构建了注意力集中的焦点，并通过外部信息动态调节注意力聚集。具体而言，我们在连续行为之间插入水槽令牌，并结合了特定于建议的信号，例如时间距离，以用作稳定的注意下沉。为了提高普遍性，我们设计了一种两阶段的训练策略，该策略明确指导LM注意下沉的令牌和一个注意下沉机制，该机制放大了互链依赖性以更好地捕获行为相关性。在一个工业数据集和两个开源数据集（Movielens，Kuairec）上进行的实验，以及可视化结果，验证了该方法在方案中的有效性。

Title: FairLangProc: A Python package for fairness in NLP

• Authors: Arturo Pérez-Peralta, Sandra Benítez-Peña, Rosa E. Lillo
• Subjects: cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2508.03677
• Pdf URL: https://arxiv.org/pdf/2508.03677
• Copy Paste: [[2508.03677]] FairLangProc: A Python package for fairness in NLP(https://arxiv.org/abs/2508.03677)
• Keywords: language model
• Abstract: The rise in usage of Large Language Models to near ubiquitousness in recent years has risen societal concern about their applications in decision-making contexts, such as organizational justice or healthcare. This, in turn, poses questions about the fairness of these models in critical settings, which leads to the developement of different procedures to address bias in Natural Language Processing. Although many datasets, metrics and algorithms have been proposed to measure and mitigate harmful prejudice in Natural Language Processing, their implementation is diverse and far from centralized. As a response, this paper presents FairLangProc, a comprehensive Python package providing a common implementation of some of the more recent advances in fairness in Natural Language Processing providing an interface compatible with the famous Hugging Face transformers library, aiming to encourage the widespread use and democratization of bias mitigation techniques. The implementation can be found on this https URL.
• 摘要：近年来，大型语言模型的使用量增加到几乎无处不在的，对他们在决策环境（例如组织正义或医疗保健等决策环境中的应用）的关注引起了人们的关注。反过来，这提出了有关这些模型在关键环境中的公平性的问题，这导致开发不同程序以解决自然语言处理中的偏见。尽管已经提出了许多数据集，指标和算法来衡量和减轻自然语言处理中的有害偏见，但它们的实施是多种多样的，而且远非集中式。作为回应，本文介绍了Fairlangproc，这是一个全面的Python软件包，为自然语言处理方面的一些最新进展提供了共同的实施，从而提供了与著名的拥抱面孔变形金刚库兼容的界面，旨在鼓励人们广泛使用和民主化偏见缓解技术。可以在此HTTPS URL上找到实现。

Title: More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation

• Authors: Yangtian Zi, Harshitha Menon, Arjun Guha
• Subjects: cs.CL, cs.LG, cs.PL
• Abstract URL: https://arxiv.org/abs/2508.03678
• Pdf URL: https://arxiv.org/pdf/2508.03678
• Copy Paste: [[2508.03678]] More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation(https://arxiv.org/abs/2508.03678)
• Keywords: language model, llm, prompt
• Abstract: State-of-the-art Large Language Models (LLMs) achieve high pass@1 on general benchmarks like HumanEval but underperform on specialized suites such as ParEval. Is this due to LLMs missing domain knowledge or insufficient prompt detail is given? To answer this, we introduce PartialOrderEval, which augments any code generation benchmark with a partial order of prompts from minimal to maximally detailed. Applying it to HumanEval and both serial and OpenMP subsets of ParEval, we measure how pass@1 scales with prompt specificity. Our experiments with Llama-3.x and Qwen2.5-Coder demonstrate varying degrees of prompt sensitivity across different tasks, and a qualitative analysis highlights explicit I/O specifications, edge-case handling, and stepwise breakdowns as the key drivers of prompt detail improvement.
• 摘要：最先进的大语言模型（LLMS）在像Humaneval这样的一般基准上获得了高通@1，但在Predized Suites（例如Pareval）上表现不佳。这是由于LLMS缺少域知识或不足的及时细节所致吗？为了回答这一点，我们介绍了PartialorDereVal，该eartordereval可以通过最小到至最大详细的提示来增强任何代码生成基准。将其应用于Pareval的人类事件以及串行和OpenMP子集，我们如何以迅速的特异性来测量@1级尺度。我们对Llama-3.x和Qwen2.5代码进行的实验表明，不同任务的迅速灵敏度不同，定性分析突出了明确的I/O规格，边缘处理和逐步分解，作为迅速细节改进的关键驱动因素。

Title: CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward

• Authors: Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F. Wong, Songyang Zhang, Kai Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2508.03686
• Pdf URL: https://arxiv.org/pdf/2508.03686
• Copy Paste: [[2508.03686]] CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward(https://arxiv.org/abs/2508.03686)
• Keywords: language model, llm, prompt
• Abstract: Answer verification is crucial not only for evaluating large language models (LLMs) by matching their unstructured outputs against standard answers, but also serves as the reward model to guide LLM optimization. Most evaluation frameworks rely on regularized matching or employ general LLMs for answer verification, which demands extensive, repetitive customization for regex rules or evaluation prompts. Two fundamental limitations persist in current methodologies: 1) the absence of comprehensive benchmarks that systematically evaluate verification capabilities across different LLMs; and 2) the nascent stage of verifier development, where existing approaches lack both the robustness to handle complex edge cases and the generalizability across different domains. In this work, we develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward. It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types, including multi-subproblems, formulas, and sequence answers, while effectively identifying abnormal/invalid responses. We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier. We anticipate that CompassVerifier and VerifierBench will facilitate answer verification, evaluation protocols, and reinforcement learning research. Code and dataset are available at this https URL.
• 摘要：答案验证不仅对于评估大语言模型（LLM）至关重要，通过将其非结构化输出与标准答案相匹配，而且是指导LLM优化的奖励模型。大多数评估框架都依赖于正规匹配或采用一般LLM进行答案验证，该验证需要广泛的，重复性的定制，以进行正则规则或评估提示。当前方法论中，有两个基本的局限性持续存在：1）缺乏系统地评估不同LLM的验证能力的全面基准； 2）验证者开发的新生阶段，现有方法既缺乏处理复杂边缘案例的鲁棒性，又缺乏跨不同领域的普遍性。在这项工作中，我们开发了Commassverifier，这是一种准确，可靠的轻质验证器模型，用于评估和结果奖励。它展示了跨越数学，知识和各种推理任务的多域功能，并具有处理各种答案类型的能力，包括多种副标题，公式和序列答案，同时有效地识别异常/无效的响应。我们介绍了VerifierBench基准测试，其中包括从多个数据源收集的模型输出，通过手动分析元元模式来增强，以增强同情者。我们预计Commanderifier和VerifierBench将有助于答案验证，评估协议和强化学习研究。代码和数据集可在此HTTPS URL上找到。