2025-11-17

Title: Unsupervised Cycle Detection in Agentic Applications

• Authors: Felix George, Harshit Kumar, Divya Pathak, Kaustabha Ray, Mudit Verma, Pratibha Moogi
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2511.10650
• Pdf URL: https://arxiv.org/pdf/2511.10650
• Copy Paste: [[2511.10650]] Unsupervised Cycle Detection in Agentic Applications(https://arxiv.org/abs/2511.10650)
• Keywords: language model, agent
• Abstract: Agentic applications powered by Large Language Models exhibit non-deterministic behaviors that can form hidden execution cycles, silently consuming resources without triggering explicit errors. Traditional observability platforms fail to detect these costly inefficiencies. We present an unsupervised cycle detection framework that combines structural and semantic analysis. Our approach first applies computationally efficient temporal call stack analysis to identify explicit loops and then leverages semantic similarity analysis to uncover subtle cycles characterized by redundant content generation. Evaluated on 1575 trajectories from a LangGraph-based stock market application, our hybrid approach achieves an F1 score of 0.72 (precision: 0.62, recall: 0.86), significantly outperforming individual structural (F1: 0.08) and semantic methods (F1: 0.28). While these results are encouraging, there remains substantial scope for improvement, and future work is needed to refine the approach and address its current limitations.
• 摘要：由大型语言模型支持的代理应用程序表现出非确定性行为，可以形成隐藏的执行周期，默默地消耗资源而不会触发显式错误。传统的可观察性平台无法检测到这些代价高昂的低效率问题。我们提出了一个结合结构和语义分析的无监督循环检测框架。我们的方法首先应用计算效率高的时间调用堆栈分析来识别显式循环，然后利用语义相似性分析来发现以冗余内容生成为特征的微妙循环。对基于 LangGraph 的股票市场应用程序的 1575 个轨迹进行评估，我们的混合方法取得了 0.72 的 F1 分数（精度：0.62，召回率：0.86），显着优于单独的结构方法（F1：0.08）和语义方法（F1：0.28）。虽然这些结果令人鼓舞，但仍有很大的改进空间，需要未来的工作来完善该方法并解决其当前的局限性。

Title: Data Analysis and Performance Evaluation of Simulation Deduction Based on LLMs

• Authors: Shansi Zhang, Min Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10651
• Pdf URL: https://arxiv.org/pdf/2511.10651
• Copy Paste: [[2511.10651]] Data Analysis and Performance Evaluation of Simulation Deduction Based on LLMs(https://arxiv.org/abs/2511.10651)
• Keywords: language model, llm, prompt
• Abstract: Data analysis and performance evaluation of simulation deduction plays a pivotal role in modern warfare, which enables military personnel to gain invaluable insights into the potential effectiveness of different strategies, tactics, and operational plans. Traditional manual analysis approach is time-consuming and limited by human errors. To enhance efficiency and accuracy, large language models (LLMs) with strong analytical and inferencing capabilities can be employed. However, high-quality analysis reports with well-structured formatting cannot be obtained through a single instruction input to the LLM. To tackle this issue, we propose a method that first decomposes the complex task into several sub-tasks and designs effective system prompts and user prompts for each sub-task. Multi-round interactions with the LLM incorporating self-check and reflection are then conducted to enable structured data extraction as well as multi-step analysis and evaluation. Furthermore, custom tools are defined and invoked to generate figures and compute metrics. We also design multiple report templates, each tailored to a specific application and input data type, ensuring their adaptability across a variety of scenarios. Extensive evaluation results demonstrate that the reports generated by our method exhibit higher quality, therefore obtaining higher scores than the baseline method.
• 摘要：模拟推导的数据分析和性能评估在现代战争中发挥着关键作用，它使军事人员能够获得对不同战略、战术和作战计划的潜在有效性的宝贵见解。传统的手动分析方法既耗时又受人为错误的限制。为了提高效率和准确性，可以采用具有强大分析和推理能力的大型语言模型（LLM）。然而，通过输入LLM的单一指令无法获得结构良好的高质量分析报告。为了解决这个问题，我们提出了一种方法，首先将复杂的任务分解为多个子任务，并为每个子任务设计有效的系统提示和用户提示。然后与法学硕士进行多轮交互，结合自检和反思，实现结构化数据提取以及多步骤分析和评估。此外，还定义并调用自定义工具来生成数据和计算指标。我们还设计了多个报告模板，每个模板都针对特定应用程序和输入数据类型量身定制，确保其在各种场景中的适应性。广泛的评估结果表明，我们的方法生成的报告具有更高的质量，因此获得比基线方法更高的分数。

Title: Cognitively-Inspired Episodic Memory Architectures for Accurate and Efficient Character AI

• Authors: Rafael Arias Gonzalez, Steve DiPaola
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.10652
• Pdf URL: https://arxiv.org/pdf/2511.10652
• Copy Paste: [[2511.10652]] Cognitively-Inspired Episodic Memory Architectures for Accurate and Efficient Character AI(https://arxiv.org/abs/2511.10652)
• Keywords: language model, gpt, llm, prompt, retrieval-augmented generation
• Abstract: Large language models show promise for embodying historical characters in dialogue systems, but existing approaches face a critical trade-off: simple retrieval-augmented generation produces shallow responses, while multi-stage reflection achieves depth at prohibitive latency. We present an architecture that resolves this tension through offline data augmentation and efficient parallel retrieval from structured episodic memory. Our system transforms biographical data into 1,774 enriched first-person memories with affective-semantic metadata, then employs two-stage retrieval achieving 0.52s prompt generation. Evaluation using LLM-as-judge and RAGAs metrics shows our approach achieves parity with traditional RAG on GPT-4 while significantly outperforming it on smaller models (GPT-3.5, GPT-3), suggesting particular value for resource-constrained deployments. Beyond dialogue, the structured memory enables novel visualization tools: spatiotemporal heatmaps, emotional trajectory analysis, and interactive path tracking, positioning the system as both a dialogue interface and research tool for biographical analysis. We use Van Gogh as a test case, but the architecture is generalizable to any historical figure with substantial textual records, offering a practical framework for educational, museum, and research applications requiring both accuracy and efficiency
• 摘要：大型语言模型有望在对话系统中体现历史人物，但现有方法面临着关键的权衡：简单的检索增强生成产生浅层响应，而多阶段反射以令人望而却步的延迟实现深度。我们提出了一种架构，通过离线数据增强和结构化情景记忆的高效并行检索来解决这种紧张关系。我们的系统将传记数据转换为 1,774 个带有情感语义元数据的丰富第一人称记忆，然后采用两阶段检索实现 0.52 秒的即时生成。使用 LLM 作为法官和 RAGA 指标进行的评估表明，我们的方法在 GPT-4 上实现了与传统 RAG 的同等水平，同时在较小模型（GPT-3.5、GPT-3）上显着优于传统 RAG，这表明对于资源受限的部署具有特殊价值。除了对话之外，结构化记忆还支持新颖的可视化工具：时空热图、情绪轨迹分析和交互式路径跟踪，将系统定位为对话界面和传记分析的研究工具。我们使用梵高作为测试用例，但该架构可以推广到任何有大量文本记录的历史人物，为需要准确性和效率的教育、博物馆和研究应用提供实用的框架

Title: Hybrid Quantum Transformer for Language Generation

• Authors: Desheng Kong, Xiangshuo Cui, Jiaying Jin, Jing Xu, Donglin Wang
• Subjects: cs.CL, cs.AI, quant-ph
• Abstract URL: https://arxiv.org/abs/2511.10653
• Pdf URL: https://arxiv.org/pdf/2511.10653
• Copy Paste: [[2511.10653]] Hybrid Quantum Transformer for Language Generation(https://arxiv.org/abs/2511.10653)
• Keywords: language model, llm
• Abstract: Although quantum computing has been increasingly applied to replace classical computation, most existing quantum or hybrid models remain confined to simple tasks, with no successful application to large-scale natural language generation to date. In this work, we present the first hybrid quantum-classical large language model (LLM) for natural language generation, HyQuT, capable of performing coherent and context-aware dialogue. The proposed architecture integrates variational quantum circuits (VQCs) into the Transformer framework at both 8M and 150M parameter scales. Experimental results show that a minimal number of qubits (10 qubits with 80 quantum gates) can replace about 10% of the classical parameters in the 150M-parameter model, while achieving comparable convergence stability and generation quality. This study provides an early demonstration of the feasibility of integrating quantum computing to large-scale generative language models.
• 摘要：尽管量子计算已越来越多地应用于取代经典计算，但大多数现有的量子或混合模型仍然局限于简单的任务，迄今为止尚未成功应用于大规模自然语言生成。在这项工作中，我们提出了第一个用于自然语言生成的混合量子经典大语言模型（LLM）HyQuT，能够执行连贯和上下文感知的对话。所提出的架构将 8M 和 150M 参数规模的变分量子电路 (VQC) 集成到 Transformer 框架中。实验结果表明，最少数量的量子位（10 个量子位，80 个量子门）可以替代 150M 参数模型中约 10% 的经典参数，同时实现可比的收敛稳定性和生成质量。这项研究提供了将量子计算集成到大规模生成语言模型的可行性的早期演示。

Title: Empirical Characterization of Temporal Constraint Processing in LLMs

• Authors: Javier Marín
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10654
• Pdf URL: https://arxiv.org/pdf/2511.10654
• Copy Paste: [[2511.10654]] Empirical Characterization of Temporal Constraint Processing in LLMs(https://arxiv.org/abs/2511.10654)
• Keywords: llm, prompt, agent
• Abstract: When deploying LLMs in agentic architectures requiring real-time decisions under temporal constraints, we assume they reliably determine whether action windows remain open or have closed. This assumption is untested. We characterize temporal constraint processing across eight production-scale models (2.8-8B parameters) using deadline detection tasks, revealing systematic deployment risks: bimodal performance distribution (models achieve either 95% or 50% accuracy), extreme prompt brittleness (30-60 percentage point swings from formatting changes alone), and systematic action bias (100% false positive rates in failing models). Parameter count shows no correlation with capability in this range-a 3.8B model matches 7B models while other 7B models fail completely. Fine-tuning on 200 synthetic examples improves models with partial capability by 12-37 percentage points. We demonstrate that temporal constraint satisfaction cannot be reliably learned through next-token prediction on natural language, even with targeted fine-tuning. This capability requires architectural mechanisms for: (1) continuous temporal state representation, (2) explicit constraint checking separate from linguistic pattern matching, (3) systematic compositional reasoning over temporal relations. Current autoregressive architectures lack these mechanisms. Deploying such systems in time-critical applications without hybrid architectures incorporating symbolic reasoning modules represents unacceptable risk.
• 摘要：当在需要在时间限制下进行实时决策的代理架构中部署法学硕士时，我们假设它们可靠地确定操作窗口是保持打开还是关闭。这个假设未经检验。我们使用截止日期检测任务描述了八个生产规模模型（2.8-8B 参数）的时间约束处理特征，揭示了系统部署风险：双峰性能分布（模型达到 95% 或 50% 的准确度）、极端的即时脆弱性（仅格式更改造成 30-60 个百分点的波动）和系统操作偏差（失败模型中的误报率为 100%）。参数计数显示与此范围内的功能没有相关性 - 3.8B 模型与 7B 模型匹配，而其他 7B 模型完全失败。对 200 个合成示例进行微调将部分能力的模型提高了 12-37 个百分点。我们证明，即使有针对性的微调，也无法通过自然语言的下一个标记预测来可靠地学习时间约束满足。此功能需要以下架构机制：(1) 连续时间状态表示，(2) 与语言模式匹配分开的显式约束检查，(3) 对时间关系的系统组合推理。当前的自回归架构缺乏这些机制。在时间关键的应用程序中部署此类系统而不采用包含符号推理模块的混合架构会带来不可接受的风险。

Title: Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models

• Authors: Biao Liu, Ning Xu, Junming Yang, Xin Geng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10656
• Pdf URL: https://arxiv.org/pdf/2511.10656
• Copy Paste: [[2511.10656]] Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models(https://arxiv.org/abs/2511.10656)
• Keywords: language model, llm, prompt
• Abstract: While Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, aligning these models with varying human preferences across multiple objectives remains a significant challenge in practical deployments. Existing multi-objective alignment methods rely on manually specified preference weights, which not only burden users with difficult preference specification tasks but also lead to suboptimal training efficiency due to exploration of irrelevant preference combinations. To alleviate these issues, we propose a novel framework named PRO, i.e., PReference Orchestrator, which features a lightweight preference adapter that automatically infers prompt-specific preference weights during both training and deployment phases. Specifically, the adapter automatically learns appropriate preference weights for each prompt by training on normalized reward scores from multiple reward models for preferred responses, which inherently reflect effective preference balances across objectives. Additionally, We provide theoretical analysis proving that our prompt-aware preference mechanism achieves superior performance compared to fixed preference weights in multi-objective alignment scenarios. Extensive experiments across multiple tasks demonstrate the effectiveness of our method over existing multi-objective alignment approaches.
• 摘要：虽然大型语言模型 (LLM) 在不同的自然语言处理任务中表现出了卓越的能力，但在多个目标中使这些模型与不同的人类偏好保持一致仍然是实际部署中的重大挑战。现有的多目标对齐方法依赖于手动指定的偏好权重，这不仅给用户带来了困难的偏好指定任务的负担，而且由于探索不相关的偏好组合而导致训练效率不理想。为了缓解这些问题，我们提出了一种名为 PRO 的新颖框架，即 PReference Orchestrator，它具有轻量级首选项适配器，可以在训练和部署阶段自动推断特定于提示的首选项权重。具体来说，适配器通过对来自首选响应的多个奖励模型的归一化奖励分数进行训练，自动学习每个提示的适当偏好权重，这本质上反映了跨目标的有效偏好平衡。此外，我们提供的理论分析证明，与多目标对齐场景中的固定偏好权重相比，我们的提示感知偏好机制实现了卓越的性能。跨多个任务的广泛实验证明了我们的方法相对于现有多目标对齐方法的有效性。

Title: Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages

• Authors: Douwe J. Spaanderman, Karthik Prathaban, Petr Zelina, Kaouther Mouheb, Lukáš Hejtmánek, Matthew Marzetti, Antonius W. Schurink, Damian Chan, Ruben Niemantsverdriet, Frederik Hartmann, Zhen Qian, Maarten G.J. Thomeer, Petr Holub, Farhan Akram, Frank J. Wolters, Meike W. Vernooij, Cornelis Verhoef, Esther E. Bron, Vít Nováček, Dirk J. Grünhagen, Wiro J. Niessen, Martijn P.A. Starmans, Stefan Klein
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10658
• Pdf URL: https://arxiv.org/pdf/2511.10658
• Copy Paste: [[2511.10658]] Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages(https://arxiv.org/abs/2511.10658)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) are increasingly used to extract structured information from free-text clinical records, but prior work often focuses on single tasks, limited models, and English-language reports. We evaluated 15 open-weight LLMs on pathology and radiology reports across six use cases, colorectal liver metastases, liver tumours, neurodegenerative diseases, soft-tissue tumours, melanomas, and sarcomas, at three institutes in the Netherlands, UK, and Czech Republic. Models included general-purpose and medical-specialised LLMs of various sizes, and six prompting strategies were compared: zero-shot, one-shot, few-shot, chain-of-thought, self-consistency, and prompt graph. Performance was assessed using task-appropriate metrics, with consensus rank aggregation and linear mixed-effects models quantifying variance. Top-ranked models achieved macro-average scores close to inter-rater agreement across tasks. Small-to-medium general-purpose models performed comparably to large models, while tiny and specialised models performed worse. Prompt graph and few-shot prompting improved performance by ~13%. Task-specific factors, including variable complexity and annotation variability, influenced results more than model size or prompting strategy. These findings show that open-weight LLMs can extract structured data from clinical reports across diseases, languages, and institutions, offering a scalable approach for clinical data curation.
• 摘要：大语言模型 (LLM) 越来越多地用于从自由文本临床记录中提取结构化信息，但之前的工作通常侧重于单一任务、有限模型和英语报告。我们评估了荷兰、英国和捷克共和国三个研究所的 15 个开放权重法学硕士，涉及六个用例、结直肠肝转移、肝肿瘤、神经退行性疾病、软组织肿瘤、黑色素瘤和肉瘤的病理学和放射学报告。模型包括各种规模的通用和医学专业法学硕士，并比较了六种提示策略：零样本、单样本、少样本、思维链、自我一致性和提示图。使用适合任务的指标评估绩效，并使用共识排名聚合和量化方差的线性混合效应模型。排名靠前的模型获得的宏观平均分数接近跨任务评估者间的一致性。中小型通用模型的性能与大型模型相当，而小型和专用模型的性能较差。提示图和少量提示将性能提高了约 13%。特定于任务的因素（包括可变复杂性和注释可变性）比模型大小或提示策略对结果的影响更大。这些发现表明，开放式法学硕士可以从跨疾病、语言和机构的临床报告中提取结构化数据，为临床数据管理提供可扩展的方法。

Title: Information Extraction From Fiscal Documents Using LLMs

• Authors: Vikram Aggarwal, Jay Kulkarni, Aditi Mascarenhas, Aakriti Narang, Siddarth Raman, Ajay Shah, Susan Thomas
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2511.10659
• Pdf URL: https://arxiv.org/pdf/2511.10659
• Copy Paste: [[2511.10659]] Information Extraction From Fiscal Documents Using LLMs(https://arxiv.org/abs/2511.10659)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in text comprehension, but their ability to process complex, hierarchical tabular data remains underexplored. We present a novel approach to extracting structured data from multi-page government fiscal documents using LLM-based techniques. Applied to annual fiscal documents from the State of Karnataka in India (200+ pages), our method achieves high accuracy through a multi-stage pipeline that leverages domain knowledge, sequential context, and algorithmic validation. A large challenge with traditional OCR methods is the inability to verify the accurate extraction of numbers. When applied to fiscal data, the inherent structure of fiscal tables, with totals at each level of the hierarchy, allows for robust internal validation of the extracted data. We use these hierarchical relationships to create multi-level validation checks. We demonstrate that LLMs can read tables and also process document-specific structural hierarchies, offering a scalable process for converting PDF-based fiscal disclosures into research-ready databases. Our implementation shows promise for broader applications across developing country contexts.
• 摘要：大型语言模型 (LLM) 在文本理解方面表现出了卓越的能力，但其处理复杂、分层表格数据的能力仍未得到充分开发。我们提出了一种使用基于法学硕士的技术从多页政府财政文件中提取结构化数据的新颖方法。我们的方法应用于印度卡纳塔克邦的年度财政文件（200 多页），通过利用领域知识、顺序上下文和算法验证的多阶段管道实现了高精度。传统 OCR 方法的一大挑战是无法验证数字的准确提取。当应用于财务数据时，财务表的固有结构（在层次结构的每个级别上都有总计）允许对提取的数据进行可靠的内部验证。我们使用这些层次关系来创建多级验证检查。我们证明法学硕士可以读取表格，还可以处理特定于文档的结构层次结构，从而提供可扩展的流程，将基于 PDF 的财政披露转换为研究就绪的数据库。我们的实施显示出在发展中国家更广泛应用的前景。

Title: Test-Time Steering for Lossless Text Compression via Weighted Product of Experts

• Authors: Qihang Zhang, Muchen Li, Ziao Wang, Renjie Liao, Lele Wang
• Subjects: cs.CL, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2511.10660
• Pdf URL: https://arxiv.org/pdf/2511.10660
• Copy Paste: [[2511.10660]] Test-Time Steering for Lossless Text Compression via Weighted Product of Experts(https://arxiv.org/abs/2511.10660)
• Keywords: language model
• Abstract: Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively. Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE). At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model. Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.
• 摘要：在数据快速增长的时代，无损压缩技术至关重要。 gzip 等传统通用压缩器提供低计算开销、高速度以及跨数据分布的广泛适用性。然而，它们通常会导致比现代神经压缩器更差的压缩率，现代神经压缩器利用大规模训练数据更有效地对数据分布进行建模。尽管神经压缩器有很多优点，但它很难推广到看不见的数据。为了解决这一限制，我们提出了一种新颖的框架，通过专家加权产品（wPoE）执行测试时指导。推理时，我们的方法自适应地将通用压缩模型与预训练的神经语言模型结合起来，确保压缩率至少与最佳个体模型一样好。大量的实验表明，我们的方法无需微调即可提高文本压缩的性能。此外，它与任何自回归语言模型无缝集成，为增强跨不同数据分布的文本压缩提供了实用的解决方案。

Title: Bayesian Evaluation of Large Language Model Behavior

• Authors: Rachel Longjohn, Shang Wu, Saatvik Kher, Catarina Belém, Padhraic Smyth
• Subjects: cs.CL, cs.LG, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2511.10661
• Pdf URL: https://arxiv.org/pdf/2511.10661
• Copy Paste: [[2511.10661]] Bayesian Evaluation of Large Language Model Behavior(https://arxiv.org/abs/2511.10661)
• Keywords: language model, llm, prompt
• Abstract: It is increasingly important to evaluate how text generation systems based on large language models (LLMs) behave, such as their tendency to produce harmful output or their sensitivity to adversarial inputs. Such evaluations often rely on a curated benchmark set of input prompts provided to the LLM, where the output for each prompt may be assessed in a binary fashion (e.g., harmful/non-harmful or does not leak/leaks sensitive information), and the aggregation of binary scores is used to evaluate the LLM. However, existing approaches to evaluation often neglect statistical uncertainty quantification. With an applied statistics audience in mind, we provide background on LLM text generation and evaluation, and then describe a Bayesian approach for quantifying uncertainty in binary evaluation metrics. We focus in particular on uncertainty that is induced by the probabilistic text generation strategies typically deployed in LLM-based systems. We present two case studies applying this approach: 1) evaluating refusal rates on a benchmark of adversarial inputs designed to elicit harmful responses, and 2) evaluating pairwise preferences of one LLM over another on a benchmark of open-ended interactive dialogue examples. We demonstrate how the Bayesian approach can provide useful uncertainty quantification about the behavior of LLM-based systems.
• 摘要：评估基于大型语言模型 (LLM) 的文本生成系统的行为方式变得越来越重要，例如它们产生有害输出的倾向或对对抗性输入的敏感性。此类评估通常依赖于提供给 LLM 的精选输入提示基准集，其中每个提示的输出可以以二进制方式评估（例如，有害/无害或不泄漏/泄漏敏感信息），并且二进制分数的聚合用于评估 LLM。然而，现有的评估方法常常忽视统计不确定性的量化。考虑到应用统计受众，我们提供了 LLM 文本生成和评估的背景，然后描述了用于量化二元评估指标中的不确定性的贝叶斯方法。我们特别关注基于法学硕士的系统中通常部署的概率文本生成策略所引起的不确定性。我们提出了应用这种方法的两个案例研究：1）在旨在引发有害反应的对抗性输入基准上评估拒绝率，2）在开放式交互式对话示例基准上评估一个法学硕士相对于另一个法学硕士的成对偏好。我们演示了贝叶斯方法如何对基于 LLM 的系统的行为提供有用的不确定性量化。

Title: Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish

• Authors: Chengxuan Xia, Qianye Wu, Hongbin Guan, Sixuan Tian, Yilun Hao, Xiaoyu Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10664
• Pdf URL: https://arxiv.org/pdf/2511.10664
• Copy Paste: [[2511.10664]] Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish(https://arxiv.org/abs/2511.10664)
• Keywords: language model, gpt, llm, chat
• Abstract: Large language models (LLMs) have achieved impressive results in high-resource languages like English, yet their effectiveness in low-resource and morphologically rich languages remains underexplored. In this paper, we present a comprehensive evaluation of seven cutting-edge LLMs -- including GPT-4o, GPT-4, Claude~3.5~Sonnet, LLaMA~3.1, Mistral~Large~2, LLaMA-2~Chat~13B, and Mistral~7B~Instruct -- on a new cross-lingual benchmark covering \textbf{Cantonese, Japanese, and Turkish}. Our benchmark spans four diverse tasks: open-domain question answering, document summarization, English-to-X translation, and culturally grounded dialogue. We combine \textbf{human evaluations} (rating fluency, factual accuracy, and cultural appropriateness) with automated metrics (e.g., BLEU, ROUGE) to assess model performance. Our results reveal that while the largest proprietary models (GPT-4o, GPT-4, Claude~3.5) generally lead across languages and tasks, significant gaps persist in culturally nuanced understanding and morphological generalization. Notably, GPT-4o demonstrates robust multilingual performance even on cross-lingual tasks, and Claude~3.5~Sonnet achieves competitive accuracy on knowledge and reasoning benchmarks. However, all models struggle to some extent with the unique linguistic challenges of each language, such as Turkish agglutinative morphology and Cantonese colloquialisms. Smaller open-source models (LLaMA-2~13B, Mistral~7B) lag substantially in fluency and accuracy, highlighting the resource disparity. We provide detailed quantitative results, qualitative error analysis, and discuss implications for developing more culturally aware and linguistically generalizable LLMs. Our benchmark and evaluation data are released to foster reproducibility and further research.
• 摘要：大型语言模型（LLM）在英语等高资源语言中取得了令人印象深刻的成果，但其在资源匮乏和形态丰富的语言中的有效性仍未得到充分探索。在本文中，我们在涵盖 \textbf{粤语、日语和土耳其语}的新跨语言基准上对七个前沿法学硕士（包括 GPT-4o、GPT-4、Claude~3.5~Sonnet、LLaMA~3.1、Mistral~Large~2、LLaMA-2~Chat~13B 和 Mistral~7B~Instruct）进行了综合评估。我们的基准测试涵盖四项不同的任务：开放域问答、文档摘要、英语到 X 的翻译以及基于文化的对话。我们将 \textbf{人类评估}（评级流畅性、事实准确性和文化适当性）与自动化指标（例如 BLEU、ROUGE）相结合来评估模型性能。我们的结果表明，虽然最大的专有模型（GPT-4o、GPT-4、Claude~3.5）通常领先于各种语言和任务，但在文化细微差别的理解和形态概括方面仍然存在显着差距。值得注意的是，GPT-4o 即使在跨语言任务上也表现出强大的多语言性能，而 Claude~3.5~Sonnet 在知识和推理基准上实现了有竞争力的准确性。然而，所有模型都在某种程度上应对每种语言独特的语言挑战，例如土耳其语粘着形态和粤语口语。较小的开源模型（LLaMA-2~13B、Mistral~7B）在流畅性和准确性方面大幅落后，凸显了资源差距。我们提供详细的定量结果、定性误差分析，并讨论对培养更具文化意识和语言概括性的法学硕士的影响。我们发布基准和评估数据是为了促进可重复性和进一步研究。

Title: Guarding the Meaning: Self-Supervised Training for Semantic Robustness in Guard Models

• Authors: Cristina Pinneri, Christos Louizos
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.10665
• Pdf URL: https://arxiv.org/pdf/2511.10665
• Copy Paste: [[2511.10665]] Guarding the Meaning: Self-Supervised Training for Semantic Robustness in Guard Models(https://arxiv.org/abs/2511.10665)
• Keywords: llm
• Abstract: Guard models are a critical component of LLM safety, but their sensitivity to superficial linguistic variations remains a key vulnerability. We show that even meaning-preserving paraphrases can cause large fluctuations in safety scores, revealing a lack of semantic grounding. To address this, we introduce a practical, self-supervised framework for improving the semantic robustness of guard models. Our method leverages paraphrase sets to enforce prediction consistency using a novel, skew-aware aggregation strategy for robust target computation. Notably, we find that standard aggregation methods like mean and median can degrade safety, underscoring the need for skew-aware alternatives. We analyze six open-source guard models and show that our approach reduces semantic variability across paraphrases by ~58%, improves benchmark accuracy by ~2.5% on average, and generalizes to unseen stylistic variations. Intriguingly, we discover a bidirectional relationship between model calibration and consistency: our robustness training improves calibration by up to 40%, revealing a fundamental connection between these properties. These results highlight the value of treating semantic consistency as a first-class training objective and provide a scalable recipe for building more reliable guard models.
• 摘要：Guard 模型是 LLM 安全性的关键组成部分，但它们对表面语言变化的敏感性仍然是一个关键漏洞。我们表明，即使是保留意义的释义也会导致安全分数的大幅波动，揭示出语义基础的缺乏。为了解决这个问题，我们引入了一个实用的自监督框架来提高防护模型的语义鲁棒性。我们的方法利用释义集来使用新颖的、倾斜感知的聚合策略来强制预测一致性，以实现稳健的目标计算。值得注意的是，我们发现平均值和中位数等标准聚合方法会降低安全性，这凸显了对倾斜感知替代方案的需求。我们分析了六个开源防护模型，并表明我们的方法将释义之间的语义变异性降低了约 58%，将基准准确性平均提高了约 2.5%，并泛化到看不见的风格变化。有趣的是，我们发现模型校准和一致性之间存在双向关系：我们的鲁棒性训练将校准提高了 40%，揭示了这些属性之间的基本联系。这些结果凸显了将语义一致性视为一流训练目标的价值，并为构建更可靠的警卫模型提供了可扩展的方法。

Title: Evaluating LLM Understanding via Structured Tabular Decision Simulations

• Authors: Sichao Li, Xinyue Xu, Xiaomeng Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.10667
• Pdf URL: https://arxiv.org/pdf/2511.10667
• Copy Paste: [[2511.10667]] Evaluating LLM Understanding via Structured Tabular Decision Simulations(https://arxiv.org/abs/2511.10667)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often achieve impressive predictive accuracy, yet correctness alone does not imply genuine understanding. True LLM understanding, analogous to human expertise, requires making consistent, well-founded decisions across multiple instances and diverse domains, relying on relevant and domain-grounded decision factors. We introduce Structured Tabular Decision Simulations (STaDS), a suite of expert-like decision settings that evaluate LLMs as if they were professionals undertaking structured decision ``exams''. In this context, understanding is defined as the ability to identify and rely on the correct decision factors, features that determine outcomes within a domain. STaDS jointly assesses understanding through: (i) question and instruction comprehension, (ii) knowledge-based prediction, and (iii) reliance on relevant decision factors. By analyzing 9 frontier LLMs across 15 diverse decision settings, we find that (a) most models struggle to achieve consistently strong accuracy across diverse domains; (b) models can be accurate yet globally unfaithful, and there are frequent mismatches between stated rationales and factors driving predictions. Our findings highlight the need for global-level understanding evaluation protocols and advocate for novel frameworks that go beyond accuracy to enhance LLMs' understanding ability.
• 摘要：大型语言模型 (LLM) 通常会实现令人印象深刻的预测准确性，但仅正确性并不意味着真正的理解。真正的法学硕士理解类似于人类的专业知识，需要依赖相关且基于领域的决策因素，在多个实例和不同领域做出一致、有根据的决策。我们引入了结构化表格决策模拟（STaDS），这是一套类似专家的决策设置，可以像进行结构化决策“考试”的专业人士一样评估法学硕士。在这种情况下，理解被定义为识别和依赖正确决策因素的能力，即决定领域内结果的特征。 STaDS 通过以下方式联合评估理解：(i) 问题和指令理解，(ii) 基于知识的预测，以及 (iii) 对相关决策因素的依赖。通过分析 15 个不同决策环境中的 9 个前沿法学硕士，我们发现 (a) 大多数模型都难以在不同领域实现一致的高准确性； (b) 模型可能是准确的，但在总体上是不忠实的，并且陈述的理由和驱动预测的因素之间经常存在不匹配。我们的研究结果强调了全球层面理解评估协议的必要性，并倡导超越准确性的新颖框架，以增强法学硕士的理解能力。

Title: Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

• Authors: Yan Gao, Yazheng Yang, Zhibin Lan, Yidong Chen, Min Zhang, Daimeng Wei, Hui Huang, Jinsong Su
• Subjects: cs.CL, cs.AI, cs.SD
• Abstract URL: https://arxiv.org/abs/2511.10670
• Pdf URL: https://arxiv.org/pdf/2511.10670
• Copy Paste: [[2511.10670]] Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment(https://arxiv.org/abs/2511.10670)
• Keywords: language model, llm
• Abstract: Code-switching (CS) speech translation (ST) refers to translating speech that alternates between two or more languages into a target language text, which poses significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies tend to rely on the model itself to implicitly learn semantic modeling during training, and resort to inefficient and costly manual annotations for these two challenges. To mitigate these limitations, we propose enhancing Large Language Models (LLMs) with a Mixture of Experts (MoE) speech projector, where each expert specializes in the semantic subspace of a specific language, enabling fine-grained modeling of speech features. Additionally, we introduce a multi-stage training paradigm that utilizes readily available monolingual automatic speech recognition (ASR) and monolingual ST data, facilitating speech-text alignment and improving translation capabilities. During training, we leverage a combination of language-specific loss and intra-group load balancing loss to guide the MoE speech projector in efficiently allocating tokens to the appropriate experts, across expert groups and within each group, respectively. To bridge the data gap across different training stages and improve adaptation to the CS scenario, we further employ a transition loss, enabling smooth transitions of data between stages, to effectively address the scarcity of high-quality CS speech translation data. Extensive experiments on widely used datasets demonstrate the effectiveness and generality of our approach.
• 摘要：语码转换（CS）语音翻译（ST）是指将两种或多种语言之间交替的语音翻译成目标语言文本，由于语义建模的复杂性和CS数据的稀缺性，这带来了巨大的挑战。之前的研究往往依赖模型本身在训练过程中隐式学习语义建模，并诉诸低效且昂贵的手动注释来应对这两个挑战。为了减轻这些限制，我们建议使用专家混合（MoE）语音投影仪来增强大型语言模型（LLM），其中每个专家专门研究特定语言的语义子空间，从而实现语音特征的细粒度建模。此外，我们引入了一种多阶段训练范例，利用现成的单语自动语音识别 (ASR) 和单语 ST 数据，促进语音文本对齐并提高翻译能力。在训练期间，我们利用特定于语言的损失和组内负载平衡损失的组合来指导 MoE 语音投影仪分别在专家组之间和每个组内有效地将令牌分配给适当的专家。为了弥合不同训练阶段的数据差距并提高对 CS 场景的适应，我们进一步采用转换损失，实现阶段之间数据的平滑转换，以有效解决高质量 CS 语音翻译数据的稀缺问题。对广泛使用的数据集进行的广泛实验证明了我们方法的有效性和通用性。

Title: Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency

• Authors: Filippo Morbiato, Luca Romano, Alessandro Persona
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2511.10671
• Pdf URL: https://arxiv.org/pdf/2511.10671
• Copy Paste: [[2511.10671]] Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency(https://arxiv.org/abs/2511.10671)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Visual hallucination, where Multimodal Large Language Models fabricate details inconsistent with image content, critically undermines their reliability. Existing fine-tuning methods offer limited improvement, failing to deeply intervene in factual reasoning. This paper introduces Grounded Visual Factualization (GVF) Finetuning, a novel approach to systematically enhance MLLM visual factual consistency. GVF integrates explicit factual signals via three core mechanisms: Factual Anchor Data Augmentation, enriching training data with structured factual anchors and counter-factual prompts; Fact-Aware Instruction Tuning, embedding these cues into explicit instructions; and a Factual Consistency Loss function, specifically penalizing factual inaccuracies. Evaluated on LLaVA-1.5-13B, GVF Finetuning significantly outperforms standard fine-tuning on the VHTest benchmark for both Open-Ended Question (OEQ) and Yes/No Question (YNQ) formats. Crucially, GVF maintains or even slightly improves performance on general multimodal benchmarks like MME and POPE, demonstrating effective mitigation of visual hallucinations without compromising general understanding and reasoning abilities.
• 摘要：视觉幻觉，即多模态大语言模型制造与图像内容不一致的细节，严重破坏了它们的可靠性。现有的微调方法提供的改进有限，未能深度干预事实推理。本文介绍了基于视觉事实化 (GVF) 微调，这是一种系统地增强 MLLM 视觉事实一致性的新方法。 GVF通过三个核心机制整合明确的事实信号：事实锚数据增强，通过结构化事实锚和反事实提示丰富训练数据；事实感知指令调整，将这些线索嵌入到明确的指令中；事实一致性损失函数，专门惩罚事实不准确的情况。在 LLaVA-1.5-13B 上进行评估，对于开放式问题 (OEQ) 和是/否问题 (YNQ) 格式，GVF 微调的性能显着优于 VHTest 基准上的标准微调。至关重要的是，GVF 保持甚至略微提高了 MME 和 POPE 等通用多模态基准的性能，证明了在不影响一般理解和推理能力的情况下有效缓解视幻觉。

Title: Large language models in materials science and the need for open-source approaches

• Authors: Fengxu Yang, Weitong Chen, Jack D. Evans
• Subjects: cs.CL, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2511.10673
• Pdf URL: https://arxiv.org/pdf/2511.10673
• Copy Paste: [[2511.10673]] Large language models in materials science and the need for open-source approaches(https://arxiv.org/abs/2511.10673)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) are rapidly transforming materials science. This review examines recent LLM applications across the materials discovery pipeline, focusing on three key areas: mining scientific literature , predictive modelling, and multi-agent experimental systems. We highlight how LLMs extract valuable information such as synthesis conditions from text, learn structure-property relationships, and can coordinate agentic systems integrating computational tools and laboratory automation. While progress has been largely dependent on closed-source commercial models, our benchmark results demonstrate that open-source alternatives can match performance while offering greater transparency, reproducibility, cost-effectiveness, and data privacy. As open-source models continue to improve, we advocate their broader adoption to build accessible, flexible, and community-driven AI platforms for scientific discovery.
• 摘要：大型语言模型 (LLM) 正在迅速改变材料科学。这篇综述研究了最近的法学硕士在材料发现管道中的应用，重点关注三个关键领域：挖掘科学文献、预测建模和多智能体实验系统。我们重点介绍法学硕士如何从文本中提取有价值的信息（例如合成条件）、学习结构-性质关系，以及如何协调集成计算工具和实验室自动化的代理系统。虽然进展很大程度上依赖于闭源商业模型，但我们的基准测试结果表明，开源替代方案可以匹配性能，同时提供更高的透明度、可重复性、成本效益和数据隐私。随着开源模型的不断改进，我们提倡更广泛地采用它们，为科学发现构建可访问、灵活和社区驱动的人工智能平台。

Title: Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL

• Authors: Thomas Cook, Kelly Patel, Sivapriya Vellaichamy, Saba Rahimi, Zhen Zeng, Sumitra Ganesh
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2511.10674
• Pdf URL: https://arxiv.org/pdf/2511.10674
• Copy Paste: [[2511.10674]] Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL(https://arxiv.org/abs/2511.10674)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) can generate SQL queries from natural language questions but struggle with database-specific schemas and tacit domain knowledge. We introduce a framework for continual learning from human feedback in text-to-SQL, where a learning agent receives natural language feedback to refine queries and distills the revealed knowledge for reuse on future tasks. This distilled knowledge is stored in a structured memory, enabling the agent to improve execution accuracy over time. We design and evaluate multiple variations of a learning agent architecture that vary in how they capture and retrieve past experiences. Experiments on the BIRD benchmark Dev set show that memory-augmented agents, particularly the Procedural Agent, achieve significant accuracy gains and error reduction by leveraging human-in-the-loop feedback. Our results highlight the importance of transforming tacit human expertise into reusable knowledge, paving the way for more adaptive, domain-aware text-to-SQL systems that continually learn from a human-in-the-loop.
• 摘要：大型语言模型 (LLM) 可以根据自然语言问题生成 SQL 查询，但难以处理特定于数据库的模式和隐性领域知识。我们引入了一个从文本到 SQL 的人类反馈中持续学习的框架，其中学习代理接收自然语言反馈来完善查询并提炼所揭示的知识以在未来的任务中重用。这些提炼的知识存储在结构化内存中，使代理能够随着时间的推移提高执行准确性。我们设计和评估学习代理架构的多种变体，这些变体在捕获和检索过去经验的方式上有所不同。 BIRD 基准开发集上的实验表明，内存增强代理（尤其是程序代理）通过利用人机交互反馈，可显着提高准确性并减少错误。我们的结果强调了将隐性人类专业知识转化为可重用知识的重要性，为更具适应性、领域感知的文本到 SQL 系统铺平了道路，这些系统不断地向人在环中学习。

Title: Learn to Select: Exploring Label Distribution Divergence for In-Context Demonstration Selection in Text Classification

• Authors: Ye Jiang, Taihang Wang, Youzheng Liu, Yimin Wang, Yuhan Xia, Yunfei Long
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2511.10675
• Pdf URL: https://arxiv.org/pdf/2511.10675
• Copy Paste: [[2511.10675]] Learn to Select: Exploring Label Distribution Divergence for In-Context Demonstration Selection in Text Classification(https://arxiv.org/abs/2511.10675)
• Keywords: language model, llm
• Abstract: In-context learning (ICL) for text classification, which uses a few input-label demonstrations to describe a task, has demonstrated impressive performance on large language models (LLMs). However, the selection of in-context demonstrations plays a crucial role and can significantly affect LLMs' performance. Most existing demonstration selection methods primarily focus on semantic similarity between test inputs and demonstrations, often overlooking the importance of label distribution alignment. To address this limitation, we propose a two-stage demonstration selection method, TopK + Label Distribution Divergence (L2D), which leverages a fine-tuned BERT-like small language model (SLM) to generate label distributions and calculate their divergence for both test inputs and candidate demonstrations. This enables the selection of demonstrations that are not only semantically similar but also aligned in label distribution with the test input. Extensive experiments across seven text classification benchmarks show that our method consistently outperforms previous demonstration selection strategies. Further analysis reveals a positive correlation between the performance of LLMs and the accuracy of the underlying SLMs used for label distribution estimation.
• 摘要：用于文本分类的上下文学习 (ICL) 使用一些输入标签演示来描述任务，在大型语言模型 (LLM) 上表现出了令人印象深刻的性能。然而，上下文演示的选择起着至关重要的作用，可以显着影响法学硕士的表现。大多数现有的演示选择方法主要关注测试输入和演示之间的语义相似性，通常忽略标签分布对齐的重要性。为了解决这个限制，我们提出了一种两阶段的演示选择方法，TopK + 标签分布散度（L2D），它利用微调的类似 BERT 的小语言模型（SLM）来生成标签分布并计算测试输入和候选演示的散度。这使得选择的演示不仅在语义上相似，而且在标签分布上与测试输入对齐。七个文本分类基准的广泛实验表明，我们的方法始终优于以前的演示选择策略。进一步的分析揭示了 LLM 的性能与用于标签分布估计的基础 SLM 的准确性之间存在正相关性。

Title: Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models

• Authors: Shien Zhu, Samuel Bohl, Robin Oester, Gustavo Alonso
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10676
• Pdf URL: https://arxiv.org/pdf/2511.10676
• Copy Paste: [[2511.10676]] Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models(https://arxiv.org/abs/2511.10676)
• Keywords: language model, llm
• Abstract: Mixture-of-Experts (MoE) Large Language Models (LLMs) efficiently scale-up the model while keeping relatively low inference cost. As MoE models only activate part of the experts, related work has proposed expert prediction and caching methods to prefetch the experts for faster inference. However, existing approaches utilize the activations from the previous layer for prediction, incurring low accuracy and leave the first layer unoptimized. Applying complex layers or even training standalone networks for better prediction introduces high computation overhead. In this paper, we propose pre-attention expert prediction to achieve accurate and lightweight expert prefetching. The key insight is that some functions in LLMs are ranking-preserving, indicating that matching the ranking of selected experts using simple linear functions is possible. Therefore, we utilize the activations before the attention block in the same layer with 2 linear functions and ranking-aware loss to achieve accurate prediction, which also supports prefetching in the first layer. Our lightweight, pre-attention expert routers achieve 93.03% accuracy on DeepSeek V2 Lite, 94.69% on Qwen3-30B, and 97.62% on Phi-mini-MoE, showing about 15% improvement on absolute accuracy over the state-of-the-art methods.
• 摘要：专家混合 (MoE) 大型语言模型 (LLM) 有效地扩展了模型，同时保持相对较低的推理成本。由于MoE模型仅激活部分专家，因此相关工作提出了专家预测和缓存方法来预取专家以加快推理速度。然而，现有的方法利用前一层的激活进行预测，导致精度较低，并且第一层未得到优化。应用复杂层甚至训练独立网络以获得更好的预测会带来很高的计算开销。在本文中，我们提出了预注意专家预测，以实现准确且轻量级的专家预取。关键的见解是法学硕士中的一些函数是保留排名的，这表明使用简单的线性函数来匹配所选专家的排名是可能的。因此，我们利用同一层中注意块之前的激活以及两个线性函数和排名感知损失来实现准确的预测，这也支持第一层的预取。我们的轻量级预注意力专家路由器在 DeepSeek V2 Lite 上实现了 93.03% 的准确率，在 Qwen3-30B 上实现了 94.69%，在 Phi-mini-MoE 上实现了 97.62%，与最先进的方法相比，绝对准确度提高了约 15%。

Title: SpiderGen: Towards Procedure Generation For Carbon Life Cycle Assessments with Generative AI

• Authors: Anupama Sitaraman, Bharathan Balaji, Yuvraj Agarwal
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2511.10684
• Pdf URL: https://arxiv.org/pdf/2511.10684
• Copy Paste: [[2511.10684]] SpiderGen: Towards Procedure Generation For Carbon Life Cycle Assessments with Generative AI(https://arxiv.org/abs/2511.10684)
• Keywords: llm, prompt, chain-of-thought
• Abstract: Investigating the effects of climate change and global warming caused by GHG emissions have been a primary concern worldwide. These emissions are largely contributed to by the production, use and disposal of consumer products. Thus, it is important to build tools to estimate the environmental impact of consumer goods, an essential part of which is conducting Life Cycle Assessments (LCAs). LCAs specify and account for the appropriate processes involved with the production, use, and disposal of the products. We present SpiderGen, an LLM-based workflow which integrates the taxonomy and methodology of traditional LCA with the reasoning capabilities and world knowledge of LLMs to generate the procedural information used for LCA. We additionally evaluate the output of SpiderGen using real-world LCA documents as ground-truth. We find that SpiderGen provides accurate LCA process information that is either fully correct or has minor errors, achieving an F1-Score of 62% across 10 sample data points. We observe that the remaining missed processes and hallucinated errors occur primarily due to differences in detail between LCA documents, as well as differences in the "scope" of which auxiliary processes must also be included. We also demonstrate that SpiderGen performs better than several baselines techniques, such as chain-of-thought prompting and one-shot prompting. Finally, we highlight SpiderGen's potential to reduce the human effort and costs for estimating carbon impact, as it is able to produce LCA process information for less than \$1 USD in under 10 minutes as compared to the status quo LCA, which can cost over \$25000 USD and take up to 21-person days.
• 摘要：调查温室气体排放引起的气候变化和全球变暖的影响一直是全世界关注的首要问题。这些排放主要是由消费品的生产、使用和处置造成的。因此，构建评估消费品对环境影响的工具非常重要，其中一个重要部分是进行生命周期评估（LCA）。 LCA 指定并说明了涉及产品生产、使用和处置的适当流程。我们提出了 SpiderGen，这是一种基于法学硕士的工作流程，它将传统 LCA 的分类和方法与法学硕士的推理能力和世界知识相结合，以生成用于 LCA 的程序信息。我们还使用真实世界的 LCA 文档作为基本事实来评估 SpiderGen 的输出。我们发现 SpiderGen 提供了准确的 LCA 过程信息，这些信息要么完全正确，要么有轻微错误，在 10 个样本数据点上实现了 62% 的 F1 分数。我们观察到，剩余的遗漏流程和幻觉错误的发生主要是由于 LCA 文件之间的细节差异，以及还必须包括辅助流程的“范围”的差异。我们还证明了 SpiderGen 的性能优于几种基线技术，例如思维链提示和一次性提示。最后，我们强调 SpiderGen 在减少估算碳影响的人力和成本方面的潜力，因为它能够在 10 分钟内以不到 1 美元的成本生成 LCA 流程信息，而现状 LCA 的成本可能超过 25000 美元，最多需要 21 人天。

Title: A methodological analysis of prompt perturbations and their effect on attack success rates

• Authors: Tiago Machado, Maysa Malfiza Garcia de Macedo, Rogerio Abreu de Paula, Marcelo Carpinette Grave, Aminat Adebiyi, Luan Soares de Souza, Enrico Santarelli, Claudio Pinhanez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10686
• Pdf URL: https://arxiv.org/pdf/2511.10686
• Copy Paste: [[2511.10686]] A methodological analysis of prompt perturbations and their effect on attack success rates(https://arxiv.org/abs/2511.10686)
• Keywords: language model, llm, prompt
• Abstract: This work aims to investigate how different Large Language Models (LLMs) alignment methods affect the models' responses to prompt attacks. We selected open source models based on the most common alignment methods, namely, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Human Feedback (RLHF). We conducted a systematic analysis using statistical methods to verify how sensitive the Attack Success Rate (ASR) is when we apply variations to prompts designed to elicit inappropriate content from LLMs. Our results show that even small prompt modifications can significantly change the Attack Success Rate (ASR) according to the statistical tests we run, making the models more or less susceptible to types of attack. Critically, our results demonstrate that running existing 'attack benchmarks' alone may not be sufficient to elicit all possible vulnerabilities of both models and alignment methods. This paper thus contributes to ongoing efforts on model attack evaluation by means of systematic and statistically-based analyses of the different alignment methods and how sensitive their ASR is to prompt variation.
• 摘要：这项工作旨在研究不同的大型语言模型 (LLM) 对齐方法如何影响模型对即时攻击的响应。我们根据最常见的对齐方法选择了开源模型，即监督微调（SFT）、直接偏好优化（DPO）和人类反馈强化学习（RLHF）。我们使用统计方法进行了系统分析，以验证当我们对旨在从法学硕士中引出不当内容的提示应用变体时，攻击成功率 (ASR) 的敏感程度。我们的结果表明，根据我们运行的统计测试，即使是很小的即时修改也可以显着改变攻击成功率（ASR），从而使模型或多或少容易受到攻击类型的影响。至关重要的是，我们的结果表明，仅运行现有的“攻击基准”可能不足以找出模型和对齐方法的所有可能的漏洞。因此，本文通过对不同对齐方法及其 ASR 对提示变化的敏感程度进行系统和基于统计的分析，为模型攻击评估的持续努力做出了贡献。

Title: Modeling and Predicting Multi-Turn Answer Instability in Large Language Models

• Authors: Jiahang He, Rishi Ramachandran, Neel Ramachandran, Aryan Katakam, Kevin Zhu, Sunishchal Dev, Ashwinee Panda, Aryan Shrivastava
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10688
• Pdf URL: https://arxiv.org/pdf/2511.10688
• Copy Paste: [[2511.10688]] Modeling and Predicting Multi-Turn Answer Instability in Large Language Models(https://arxiv.org/abs/2511.10688)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) are adopted in an increasingly wide range of applications, user-model interactions have grown in both frequency and scale. Consequently, research has focused on evaluating the robustness of LLMs, an essential quality for real-world tasks. In this paper, we employ simple multi-turn follow-up prompts to evaluate models' answer changes, model accuracy dynamics across turns with Markov chains, and examine whether linear probes can predict these changes. Our results show significant vulnerabilities in LLM robustness: a simple "Think again" prompt led to an approximate 10% accuracy drop for Gemini 1.5 Flash over nine turns, while combining this prompt with a semantically equivalent reworded question caused a 7.5% drop for Claude 3.5 Haiku. Additionally, we find that model accuracy across turns can be effectively modeled using Markov chains, enabling the prediction of accuracy probabilities over time. This allows for estimation of the model's stationary (long-run) accuracy, which we find to be on average approximately 8% lower than its first-turn accuracy for Gemini 1.5 Flash. Our results from a model's hidden states also reveal evidence that linear probes can help predict future answer changes. Together, these results establish stationary accuracy as a principled robustness metric for interactive settings and expose the fragility of models under repeated questioning. Addressing this instability will be essential for deploying LLMs in high-stakes and interactive settings where consistent reasoning is as important as initial accuracy.
• 摘要：随着大型语言模型 (LLM) 在越来越广泛的应用中采用，用户模型交互的频率和规模都在增长。因此，研究重点是评估法学硕士的稳健性，这是现实世界任务的基本素质。在本文中，我们采用简单的多轮后续提示来评估模型的答案变化，使用马尔可夫链对跨轮的精度动态进行建模，并检查线性探针是否可以预测这些变化。我们的结果显示了 LLM 鲁棒性的重大缺陷：一个简单的“再思考一下”提示导致 Gemini 1.5 Flash 在 9 个回合中的准确率下降了约 10%，而将此提示与语义上等效的改写问题相结合导致了 Claude 3.5 Haiku 的准确率下降了 7.5%。此外，我们发现可以使用马尔可夫链有效地对各回合的模型精度进行建模，从而能够预测随时间变化的精度概率。这样可以估计模型的静态（长期）精度，我们发现该精度比 Gemini 1.5 Flash 的首轮精度平均低约 8%。我们从模型隐藏状态得到的结果也揭示了线性探针可以帮助预测未来答案变化的证据。总之，这些结果将固定精度确立为交互设置的原则稳健性指标，并暴露了模型在反复质疑下的脆弱性。解决这种不稳定性对于在高风险和交互式环境中部署法学硕士至关重要，在这些环境中，一致的推理与初始准确性同样重要。

Title: Equilibrium Dynamics and Mitigation of Gender Bias in Synthetically Generated Data

• Authors: Ashish Kattamuri, Arpita Vats, Harshwardhan Fartale, Rahul Raja, Akshata Kishore Moharir, Ishita Prasad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10689
• Pdf URL: https://arxiv.org/pdf/2511.10689
• Copy Paste: [[2511.10689]] Equilibrium Dynamics and Mitigation of Gender Bias in Synthetically Generated Data(https://arxiv.org/abs/2511.10689)
• Keywords: language model, prompt
• Abstract: Recursive prompting with large language models enables scalable synthetic dataset generation but introduces the risk of bias amplification. We investigate gender bias dynamics across three generations of recursive text generation using three complementary evaluation frameworks: rule-based pattern matching, embedding-based semantic similarity, and downstream task performance. Experiments with three initial bias levels (0.1, 0.3, 0.6) and four mitigation strategies reveal equilibrium dynamics rather than monotonic amplification. The low initial bias amplifies toward the model's inherent bias level (+36%), whereas the high initial bias decays toward it (-26%). Among mitigation methods, contrastive augmentation, which introduces gender-swapped variants, achieves significant downstream bias reduction (98.8% for low initial bias and 91% on average) despite producing higher embedding-based bias scores. This paradox demonstrates that semantic similarity metrics may diverge from behavioral fairness outcomes, highlighting the need for multidimensional evaluation in responsible synthetic data generation.
• 摘要：使用大型语言模型的递归提示可以生成可扩展的合成数据集，但会带来偏差放大的风险。我们使用三个互补的评估框架来研究三代递归文本生成的性别偏见动态：基于规则的模式匹配、基于嵌入的语义相似性和下游任务性能。三个初始偏差水平（0.1、0.3、0.6）和四种缓解策略的实验揭示了平衡动态而不是单调放大。低初始偏差会放大到模型的固有偏差水平 (+36%)，而高初始偏差会朝模型固有偏差水平衰减 (-26%)。在缓解方法中，对比增强引入了性别交换变体，尽管产生了更高的基于嵌入的偏差分数，但仍显着减少了下游偏差（低初始偏差为 98.8%，平均为 91%）。这一悖论表明，语义相似性度量可能与行为公平结果有所不同，凸显了在负责任的合成数据生成中进行多维评估的必要性。

Title: Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games

• Authors: Juntu Zhao, Jialing Zhang, Chongxuan Li, Dequan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10690
• Pdf URL: https://arxiv.org/pdf/2511.10690
• Copy Paste: [[2511.10690]] Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games(https://arxiv.org/abs/2511.10690)
• Keywords: llm
• Abstract: Recent closed-source multimodal systems have made great advances, but their hidden language for understanding the world remains opaque because of their black-box architectures. In this paper, we use the systems' preference bias to study their hidden language: During the process of compressing the input images (typically containing multiple concepts) into texts and then reconstructing them into images, the systems' inherent preference bias introduces specific shifts in the outputs, disrupting the original input concept co-occurrence. We employ the multi-round "telephone game" to strategically leverage this bias. By observing the co-occurrence frequencies of concepts in telephone games, we quantitatively investigate the concept connection strength in the understanding of multimodal systems, i.e., "hidden language." We also contribute Telescope, a dataset of 10,000+ concept pairs, as the database of our telephone game framework. Our telephone game is test-time scalable: By iteratively running telephone games, we can construct a global map of concept connections in multimodal systems' understanding. Here we can identify preference bias inherited from training, assess generalization capability advancement, and discover more stable pathways for fragile concept connections. Furthermore, we use Reasoning-LLMs to uncover unexpected concept relationships that transcend textual and visual similarities, inferring how multimodal systems understand and simulate the world. This study offers a new perspective on the hidden language of multimodal systems and lays the foundation for future research on the interpretability and controllability of multimodal systems.
• 摘要：最近的闭源多模式系统取得了巨大进步，但由于其黑盒架构，它们用于理解世界的隐藏语言仍然不透明。在本文中，我们利用系统的偏好偏差来研究其隐藏语言：在将输入图像（通常包含多个概念）压缩为文本然后将其重建为图像的过程中，系统固有的偏好偏差在输出中引入了特定的变化，破坏了原始输入概念的共现。我们采用多轮“电话游戏”来战略性地利用这种偏见。通过观察电话游戏中概念的共现频率，我们定量研究了多模态系统（即“隐藏语言”）理解中的概念连接强度。我们还贡献了 Telescope，一个包含 10,000 多个概念对的数据集，作为我们电话游戏框架的数据库。我们的电话游戏是测试时间可扩展的：通过迭代运行电话游戏，我们可以构建多模式系统理解中概念连接的全局图。在这里，我们可以识别从训练中继承的偏好偏差，评估泛化能力的进步，并为脆弱的概念连接发现更稳定的途径。此外，我们使用推理法学硕士来揭示超越文本和视觉相似性的意想不到的概念关系，推断多模态系统如何理解和模拟世界。该研究为多模态系统的隐藏语言提供了新的视角，为未来多模态系统的可解释性和可控性研究奠定了基础。

Title: Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models

• Authors: Zijian Chen, Wenjun Zhang, Guangtao Zhai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10691
• Pdf URL: https://arxiv.org/pdf/2511.10691
• Copy Paste: [[2511.10691]] Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models(https://arxiv.org/abs/2511.10691)
• Keywords: language model, llm
• Abstract: Contemporary benchmarks are struggling to keep pace with the development of large language models (LLMs). Although they are indispensable to evaluate model performance on various tasks, it is uncertain whether the models trained on Internet data have genuinely learned how to solve problems or merely seen the questions before. This potential data contamination issue presents a fundamental challenge to establishing trustworthy evaluation frameworks. Meanwhile, existing benchmarks predominantly assume benign, resource-rich settings, leaving the behavior of LLMs under pressure unexplored. In this paper, we introduce Squid Game, a dynamic and adversarial evaluation environment with resource-constrained and asymmetric information settings elaborated to evaluate LLMs through interactive gameplay against other LLM opponents. Notably, Squid Game consists of six elimination-style levels, focusing on multi-faceted abilities, such as instruction-following, code, reasoning, planning, and safety alignment. We evaluate over 50 LLMs on Squid Game, presenting the largest behavioral evaluation study of general LLMs on dynamic adversarial scenarios. We observe a clear generational phase transition on performance in the same model lineage and find evidence that some models resort to speculative shortcuts to win the game, indicating the possibility of higher-level evaluation paradigm contamination in static benchmarks. Furthermore, we compare prominent LLM benchmarks and Squid Game with correlation analyses, highlighting that dynamic evaluation can serve as a complementary part for static evaluations. The code and data will be released in the future.
• 摘要：当代基准测试正在努力跟上大型语言模型（LLM）的发展步伐。尽管它们对于评估模型在各种任务上的性能是不可或缺的，但不确定基于互联网数据训练的模型是否真正学会了如何解决问题，或者只是以前见过问题。这种潜在的数据污染问题对建立值得信赖的评估框架提出了根本性挑战。与此同时，现有的基准主要假设良性、资源丰富的环境，使得法学硕士在压力下的行为未被探索。在本文中，我们介绍了 Squid Game，这是一种动态的对抗性评估环境，具有资源受限和不对称信息设置，旨在通过与其他 LLM 对手的互动游戏来评估 LLM。值得注意的是，《Squid Game》由六个消除式关卡组成，注重多方面的能力，例如指令遵循、代码、推理、计划和安全对齐。我们在 Squid Game 上评估了 50 多个法学硕士，提出了动态对抗场景下一般法学硕士的最大行为评估研究。我们观察到同一模型谱系中性能的明显世代相变，并发现一些模型诉诸投机捷径来赢得游戏的证据，这表明静态基准中更高级别的评估范式污染的可能性。此外，我们将著名的 LLM 基准和 Squid Game 与相关分析进行比较，强调动态评​​估可以作为静态评估的补充部分。代码和数据将在未来发布。

Title: Do AI Voices Learn Social Nuances? A Case of Politeness and Speech Rate

• Authors: Eyal Rabin, Zohar Elyoseph, Rotem Israel-Fishelson, Adi Dali, Ravit Nussinson
• Subjects: cs.CL, cs.AI, cs.HC, cs.SD
• Abstract URL: https://arxiv.org/abs/2511.10693
• Pdf URL: https://arxiv.org/pdf/2511.10693
• Copy Paste: [[2511.10693]] Do AI Voices Learn Social Nuances? A Case of Politeness and Speech Rate(https://arxiv.org/abs/2511.10693)
• Keywords: prompt
• Abstract: Voice-based artificial intelligence is increasingly expected to adhere to human social conventions, but can it learn implicit cues that are not explicitly programmed? This study investigates whether state-of-the-art text-to-speech systems have internalized the human tendency to reduce speech rate to convey politeness - a non-obvious prosodic marker. We prompted 22 synthetic voices from two leading AI platforms (AI Studio and OpenAI) to read a fixed script under both "polite and formal" and "casual and informal" conditions and measured the resulting speech duration. Across both AI platforms, the polite prompt produced slower speech than the casual prompt with very large effect sizes, an effect that was statistically significant for all of AI Studio's voices and for a large majority of OpenAI's voices. These results demonstrate that AI can implicitly learn and replicate psychological nuances of human communication, highlighting its emerging role as a social actor capable of reinforcing human social norms.
• 摘要：人们越来越期望基于语音的人工智能能够遵守人类社会惯例，但它能否学习未明确编程的隐含线索？这项研究调查了最先进的文本转语音系统是否已经内化了人类降低语速以表达礼貌（一种不明显的韵律标记）的倾向。我们提示来自两个领先人工智能平台（AI Studio 和 OpenAI）的 22 个合成声音在“礼貌和正式”和“随意和非正式”条件下阅读固定脚本，并测量了最终的语音持续时间。在这两个 AI 平台上，礼貌提示比随意提示产生的语音速度要慢，而且效果非常大，这种效果对于所有 AI Studio 的声音和大多数 OpenAI 的声音具有统计显着性。这些结果表明，人工智能可以隐式地学习和复制人类交流的心理细微差别，突显其作为能够强化人类社会规范的社会参与者的新兴角色。

Title: Where does an LLM begin computing an instruction?

• Authors: Aditya Pola, Vineeth N. Balasubramanian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10694
• Pdf URL: https://arxiv.org/pdf/2511.10694
• Copy Paste: [[2511.10694]] Where does an LLM begin computing an instruction?(https://arxiv.org/abs/2511.10694)
• Keywords: llm, prompt
• Abstract: Following an instruction involves distinct sub-processes, such as reading content, reading the instruction, executing it, and producing an answer. We ask where, along the layer stack, instruction following begins, the point where reading gives way to doing. We introduce three simple datasets (Key-Value, Quote Attribution, Letter Selection) and two hop compositions of these tasks. Using activation patching on minimal-contrast prompt pairs, we measure a layer-wise flip rate that indicates when substituting selected residual activations changes the predicted answer. Across models in the Llama family, we observe an inflection point, which we term onset, where interventions that change predictions before this point become largely ineffective afterward. Multi-hop compositions show a similar onset location. These results provide a simple, replicable way to locate where instruction following begins and to compare this location across tasks and model sizes.
• 摘要：遵循指令涉及不同的子过程，例如读取内容、读取指令、执行指令以及生成答案。我们问，沿着层堆栈，指令跟随从哪里开始，读取让位于执行的点。我们介绍了三个简单的数据集（键值、报价归因、字母选择）和这些任务的两个跳跃组合。在最小对比度提示对上使用激活修补，我们测量了分层翻转率，该率表明何时替换选定的残差激活会改变预测答案。在 Llama 家族的模型中，我们观察到一个拐点，我们将其称为“起始点”，在此点之前改变预测的干预措施在此之后基本上变得无效。多跳组合显示出相似的起始位置。这些结果提供了一种简单、可复制的方法来定位指令后续开始的位置，并跨任务和模型大小比较该位置。

Title: "As Eastern Powers, I will veto." : An Investigation of Nation-level Bias of Large Language Models in International Relations

• Authors: Jonghyeon Choi, Yeonjun Choi, Hyun-chul Kim, Beakcheol Jang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10695
• Pdf URL: https://arxiv.org/pdf/2511.10695
• Copy Paste: [[2511.10695]] "As Eastern Powers, I will veto." : An Investigation of Nation-level Bias of Large Language Models in International Relations(https://arxiv.org/abs/2511.10695)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: This paper systematically examines nation-level biases exhibited by Large Language Models (LLMs) within the domain of International Relations (IR). Leveraging historical records from the United Nations Security Council (UNSC), we developed a bias evaluation framework comprising three distinct tests to explore nation-level bias in various LLMs, with a particular focus on the five permanent members of the UNSC. Experimental results show that, even with the general bias patterns across models (e.g., favorable biases toward the western nations, and unfavorable biases toward Russia), these still vary based on the LLM. Notably, even within the same LLM, the direction and magnitude of bias for a nation change depending on the evaluation context. This observation suggests that LLM biases are fundamentally multidimensional, varying across models and tasks. We also observe that models with stronger reasoning abilities show reduced bias and better performance. Building on this finding, we introduce a debiasing framework that improves LLMs' factual reasoning combining Retrieval-Augmented Generation with Reflexion-based self-reflection techniques. Experiments show it effectively reduces nation-level bias, and improves performance, particularly in GPT-4o-mini and LLama-3.3-70B. Our findings emphasize the need to assess nation-level bias alongside performance when applying LLMs in the IR domain.
• 摘要：本文系统地研究了国际关系 (IR) 领域内大型语言模型 (LLM) 所表现出的国家层面的偏见。利用联合国安理会（UNSC）的历史记录，我们开发了一个偏见评估框架，其中包括三个不同的测试，以探索各种法学硕士的国家级偏见，特别关注联合国安理会的五个常任理事国。实验结果表明，即使模型之间存在普遍的偏见模式（例如，对西方国家的有利偏见，对俄罗斯的不利偏见），这些仍然根据法学硕士而有所不同。值得注意的是，即使在同一个法学硕士中，一个国家的偏见的方向和程度也会根据评估背景而变化。这一观察结果表明，法学硕士的偏见从根本上来说是多维的，随着模型和任务的不同而变化。我们还观察到，具有更强推理能力的模型表现出更少的偏差和更好的性能。基于这一发现，我们引入了一个去偏框架，该框架将检索增强生成与基于反射的自我反思技术相结合，改善了法学硕士的事实推理。实验表明，它有效地减少了国家层面的偏差，并提高了性能，特别是在 GPT-4o-mini 和 LLama-3.3-70B 中。我们的研究结果强调，在国际关系领域申请法学硕士时，需要评估国家层面的偏见和绩效。

Title: $π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling

• Authors: Dong Liu, Yanxuan Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10696
• Pdf URL: https://arxiv.org/pdf/2511.10696
• Copy Paste: [[2511.10696]] $π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling(https://arxiv.org/abs/2511.10696)
• Keywords: language model
• Abstract: Transformers have revolutionized natural language processing, but their quadratic complexity with respect to sequence length remains a fundamental bottleneck for long-range modeling. While sparse attention mechanisms like RingAttention reduce computational costs by restricting attention to local neighborhoods, they suffer from limited receptive fields and lack of adaptability. We present \PiAttention, a periodic sparse Transformer that factorizes attention into ring-local neighborhoods, deterministic $\pi$-stride skips, and an adaptive fusion gate. The periodic structure provides predictable coverage of distant tokens, while the sparse footprint keeps the per-layer complexity linear in context length. We prove that \PiAttention achieves $\mathcal{O}(kL + \pi \log L)$ receptive field growth compared to $\mathcal{O}(kL)$ for RingAttention, where $k$ is the local window size, $\pi$ is the skip period, and $L$ is the sequence length. Extensive experiments on language modeling, retrieval, and vision-language tasks demonstrate that \PiAttention matches or surpasses dense attention quality with 8.3\% lower perplexity than RingAttention while using 50\% fewer GPUs for the same context length. Our detailed ablations and visualizations reveal the importance of periodic skips, adaptive fusion, and head-level sparsity coordination for efficient long-context modeling.
• 摘要：Transformer 彻底改变了自然语言处理，但其序列长度的二次复杂性仍然是远程建模的基本瓶颈。虽然像 RingAttention 这样的稀疏注意力机制通过限制对局部邻域的注意力来降低计算成本，但它们的感受野有限且缺乏适应性。我们提出了 \PiAttention，一个周期性稀疏 Transformer，它将注意力分解为环局部邻域、确定性 $\pi$ 步幅跳跃和自适应融合门。周期性结构提供了远距离标记的可预测覆盖范围，而稀疏的足迹使每层复杂性在上下文长度上保持线性。我们证明，与 RingAttention 的 $\mathcal{O}(kL)$ 相比，\PiAttention 实现了 $\mathcal{O}(kL + \pi \log L)$ 感受野增长，其中 $k$ 是局部窗口大小，$\pi$ 是跳过周期，$L$ 是序列长度。对语言建模、检索和视觉语言任务的大量实验表明，\PiAttention 可以匹配或超过密集注意力质量，其困惑度比 RingAttention 低 8.3\%，同时在相同上下文长度下使用的 GPU 数量减少 50\%。我们详细的消融和可视化揭示了周期性跳跃、自适应融合和头部稀疏协调对于高效长上下文建模的重要性。

Title: Faithful Summarization of Consumer Health Queries: A Cross-Lingual Framework with LLMs

• Authors: Ajwad Abrar, Nafisa Tabassum Oeshy, Prianka Maheru, Farzana Tabassum, Tareque Mohmud Chowdhury
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10768
• Pdf URL: https://arxiv.org/pdf/2511.10768
• Copy Paste: [[2511.10768]] Faithful Summarization of Consumer Health Queries: A Cross-Lingual Framework with LLMs(https://arxiv.org/abs/2511.10768)
• Keywords: language model, llm
• Abstract: Summarizing consumer health questions (CHQs) can ease communication in healthcare, but unfaithful summaries that misrepresent medical details pose serious risks. We propose a framework that combines TextRank-based sentence extraction and medical named entity recognition with large language models (LLMs) to enhance faithfulness in medical text summarization. In our experiments, we fine-tuned the LLaMA-2-7B model on the MeQSum (English) and BanglaCHQ-Summ (Bangla) datasets, achieving consistent improvements across quality (ROUGE, BERTScore, readability) and faithfulness (SummaC, AlignScore) metrics, and outperforming zero-shot baselines and prior systems. Human evaluation further shows that over 80\% of generated summaries preserve critical medical information. These results highlight faithfulness as an essential dimension for reliable medical summarization and demonstrate the potential of our approach for safer deployment of LLMs in healthcare contexts.
• 摘要：总结消费者健康问题 (CHQ) 可以简化医疗保健领域的沟通，但歪曲医疗细节的不忠实总结会带来严重风险。我们提出了一个框架，将基于 TextRank 的句子提取和医学命名实体识别与大型语言模型 (LLM) 相结合，以增强医学文本摘要的可信度。在我们的实验中，我们在 MeQSum（英语）和 BanglaCHQ-Summ（孟加拉语）数据集上对 LLaMA-2-7B 模型进行了微调，在质量（ROUGE、BERTScore、可读性）和忠实度（SummaC、AlignScore）指标方面实现了一致的改进，并且优于零样本基线和先前系统。人工评估进一步表明，超过 80% 的生成摘要保留了关键的医疗信息。这些结果强调了忠实度是可靠医学总结的一个重要维度，并证明了我们的方法在医疗保健环境中更安全地部署法学硕士的潜力。

Title: Sabiá: Um Chatbot de Inteligência Artificial Generativa para Suporte no Dia a Dia do Ensino Superior

• Authors: Guilherme Biava Rodrigues, Franciele Beal, Marlon Marcon, Alinne Cristinne Corrêa Souza, André Roberto Ortoncelli, Francisco Carlos Monteiro Souza, Rodolfo Adamshuk Silva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10787
• Pdf URL: https://arxiv.org/pdf/2511.10787
• Copy Paste: [[2511.10787]] Sabiá: Um Chatbot de Inteligência Artificial Generativa para Suporte no Dia a Dia do Ensino Superior(https://arxiv.org/abs/2511.10787)
• Keywords: llm, chat, retrieval-augmented generation
• Abstract: Students often report difficulties in accessing day-to-day academic information, which is usually spread across numerous institutional documents and websites. This fragmentation results in a lack of clarity and causes confusion about routine university information. This project proposes the development of a chatbot using Generative Artificial Intelligence (GenAI) and Retrieval-Augmented Generation (RAG) to simplify access to such information. Several GenAI models were tested and evaluated based on quality metrics and the LLM-as-a-Judge approach. Among them, Gemini 2.0 Flash stood out for its quality and speed, and Gemma 3n for its good performance and open-source nature.
• 摘要：学生经常报告在获取日常学术信息方面存在困难，这些信息通常分布在众多机构文件和网站上。这种碎片化导致大学常规信息缺乏清晰度并造成混乱。该项目建议使用生成人工智能（GenAI）和检索增强生成（RAG）开发聊天机器人，以简化对此类信息的访问。根据质量指标和法学硕士法官方法对多个 GenAI 模型进行了测试和评估。其中，Gemini 2.0 Flash 以其质量和速度脱颖而出，Gemma 3n 则以其良好的性能和开源特性脱颖而出。

Title: LLM-as-a-Grader: Practical Insights from Large Language Model for Short-Answer and Report Evaluation

• Authors: Grace Byun, Swati Rajwal, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10819
• Pdf URL: https://arxiv.org/pdf/2511.10819
• Copy Paste: [[2511.10819]] LLM-as-a-Grader: Practical Insights from Large Language Model for Short-Answer and Report Evaluation(https://arxiv.org/abs/2511.10819)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) are increasingly explored for educational tasks such as grading, yet their alignment with human evaluation in real classrooms remains underexamined. In this study, we investigate the feasibility of using an LLM (GPT-4o) to evaluate short-answer quizzes and project reports in an undergraduate Computational Linguistics course. We collect responses from approximately 50 students across five quizzes and receive project reports from 14 teams. LLM-generated scores are compared against human evaluations conducted independently by the course teaching assistants (TAs). Our results show that GPT-4o achieves strong correlation with human graders (up to 0.98) and exact score agreement in 55\% of quiz cases. For project reports, it also shows strong overall alignment with human grading, while exhibiting some variability in scoring technical, open-ended responses. We release all code and sample data to support further research on LLMs in educational assessment. This work highlights both the potential and limitations of LLM-based grading systems and contributes to advancing automated grading in real-world academic settings.
• 摘要：人们越来越多地探索大型语言模型（LLM）用于评分等教育任务，但它们与真实课堂中人类评估的一致性仍然没有得到充分检验。在这项研究中，我们调查了使用法学硕士（GPT-4o）来评估本科计算语言学课程中的简答测验和项目报告的可行性。我们收集了大约 50 名学生对五项测验的回答，并收到了来自 14 个团队的项目报告。法学硕士生成的分数与课程助教 (TA) 独立进行的人工评估进行比较。我们的结果表明，GPT-4o 与人类评分者实现了很强的相关性（高达 0.98），并且在 55% 的测验案例中得分准确一致。对于项目报告，它还显示出与人工评分的强烈整体一致性，同时在对技术、开放式回答进行评分方面表现出一些可变性。我们发布所有代码和样本数据，以支持法学硕士在教育评估方面的进一步研究。这项工作强调了基于法学硕士的评分系统的潜力和局限性，并有助于在现实学术环境中推进自动评分。

Title: Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders

• Authors: Abir Harrasse, Florent Draye, Zhijing Jin, Bernhard Schölkopf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10840
• Pdf URL: https://arxiv.org/pdf/2511.10840
• Copy Paste: [[2511.10840]] Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders(https://arxiv.org/abs/2511.10840)
• Keywords: language model, llm
• Abstract: Multilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language? To address this, we train a series of LLMs on different mixtures of multilingual data and analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs. Our results provide strong evidence for pivot language representations: the model employs nearly identical representations across languages, while language-specific decoding emerges in later layers. Attribution analyses reveal that decoding relies in part on a small set of high-frequency language features in the final layers, which linearly read out language identity from the first layers in the model. By intervening on these features, we can suppress one language and substitute another in the model's outputs. Finally, we study how the dominant training language influences these mechanisms across attribution graphs and decoding pathways. We argue that understanding this pivot-language mechanism is crucial for improving multilingual alignment in LLMs.
• 摘要：多语言大语言模型 (LLM) 可以处理多种语言，但它们内部如何表示这种多样性仍不清楚。它们是否通过特定于语言的解码形成共享的多语言表示，如果是的话，为什么性能仍然有利于主导训练语言？为了解决这个问题，我们在不同的多语言数据混合上训练了一系列法学硕士，并使用跨层转码器（CLT）和归因图分析其内部机制。我们的结果为枢轴语言表示提供了强有力的证据：该模型在不同语言中采用几乎相同的表示，而特定于语言的解码出现在后面的层中。归因分析表明，解码部分依赖于最后层中的一小组高频语言特征，这些特征从模型中的第一层线性读出语言身份。通过干预这些特征，我们可以在模型的输出中抑制一种语言并替换另一种语言。最后，我们研究主导训练语言如何通过归因图和解码路径影响这些机制。我们认为，理解这种枢轴语言机制对于改善法学硕士的多语言一致性至关重要。

Title: Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English

• Authors: Rebecca Dorn, Christina Chance, Casandra Rusti, Charles Bickham Jr., Kai-Wei Chang, Fred Morstatter, Kristina Lerman
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2511.10846
• Pdf URL: https://arxiv.org/pdf/2511.10846
• Copy Paste: [[2511.10846]] Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English(https://arxiv.org/abs/2511.10846)
• Keywords: gpt
• Abstract: Automated emotion detection is widely used in applications ranging from well-being monitoring to high-stakes domains like mental health and hiring. However, models often rely on annotations that reflect dominant cultural norms, limiting model ability to recognize emotional expression in dialects often excluded from training data distributions, such as African American Vernacular English (AAVE). This study examines emotion recognition model performance on AAVE compared to General American English (GAE). We analyze 2.7 million tweets geo-tagged within Los Angeles. Texts are scored for strength of AAVE using computational approximations of dialect features. Annotations of emotion presence and intensity are collected on a dataset of 875 tweets with both high and low AAVE densities. To assess model accuracy on a task as subjective as emotion perception, we calculate community-informed "silver" labels where AAVE-dense tweets are labeled by African American, AAVE-fluent (ingroup) annotators. On our labeled sample, GPT and BERT-based models exhibit false positive prediction rates of anger on AAVE more than double than on GAE. SpanEmo, a popular text-based emotion model, increases false positive rates of anger from 25 percent on GAE to 60 percent on AAVE. Additionally, a series of linear regressions reveals that models and non-ingroup annotations are significantly more correlated with profanity-based AAVE features than ingroup annotations. Linking Census tract demographics, we observe that neighborhoods with higher proportions of African American residents are associated with higher predictions of anger (Pearson's correlation r = 0.27) and lower joy (r = -0.10). These results find an emergent safety issue of emotion AI reinforcing racial stereotypes through biased emotion classification. We emphasize the need for culturally and dialect-informed affective computing systems.
• 摘要：自动情绪检测广泛应用于从健康监测到心理健康和招聘等高风险领域的应用。然而，模型通常依赖于反映主流文化规范的注释，限制了模型识别通常被排除在训练数据分布之外的方言中的情感表达的能力，例如非裔美国人白话英语 (AAVE)。本研究比较了 AAVE 与通用美式英语 (GAE) 上的情绪识别模型性能。我们分析了洛杉矶境内带有地理标记的 270 万条推文。使用方言特征的计算近似值对文本进行 AAVE 强度评分。情绪存在和强度的注释是在包含高和低 AAVE 密度的 875 条推文的数据集上收集的。为了评估情绪感知等主观任务的模型准确性，我们计算了社区知情的“银”标签，其中 AAVE 密集的推文由非裔美国人、AAVE 流利（组内）注释者标记。在我们标记的样本中，基于 GPT 和 BERT 的模型在 AAVE 上表现出的愤怒误报率是 GAE 上的两倍多。 SpanEmo 是一种流行的基于文本的情感模型，它将愤怒的误报率从 GAE 的 25% 提高到 AAVE 的 60%。此外，一系列线性回归表明，模型和非组内注释与基于脏话的 AAVE 特征的相关性显着高于组内注释。将人口普查区人口统计数据联系起来，我们观察到非裔美国居民比例较高的社区与较高的愤怒预测（皮尔逊相关性 r = 0.27）和较低的快乐预测（r = -0.10）相关。这些结果发现了情感人工智能通过有偏见的情感分类强化种族刻板印象的紧急安全问题。我们强调对基于文化和方言的情感计算系统的需求。

Title: Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs

• Authors: Stefan Horoi, Sangwoo Cho, Supriyo Chakraborty, Shi-Xiong Zhang, Sambit Sahu, Guy Wolf, Genta Indra Winata
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.10850
• Pdf URL: https://arxiv.org/pdf/2511.10850
• Copy Paste: [[2511.10850]] Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs(https://arxiv.org/abs/2511.10850)
• Keywords: language model, llm
• Abstract: Task arithmetic is a powerful technique for transferring skills between Large Language Models (LLMs), but it often suffers from negative interference when models have diverged during training. We address this limitation by first aligning the models' parameter spaces, leveraging the inherent permutation, rotation, and scaling symmetries of Transformer architectures. We adapt parameter space alignment for modern Grouped-Query Attention (GQA) and SwiGLU layers, exploring both weight-based and activation-based approaches. Using this alignment-first strategy, we successfully transfer advanced reasoning skills to a non-reasoning model. Experiments on challenging reasoning benchmarks show that our method consistently outperforms standard task arithmetic. This work provides an effective approach for merging and transferring specialized skills across evolving LLM families, reducing redundant fine-tuning and enhancing model adaptability.
• 摘要：任务算术是一种在大型语言模型 (LLM) 之间转移技能的强大技术，但当模型在训练过程中出现分歧时，它经常会受到负面干扰。我们通过首先对齐模型的参数空间，利用 Transformer 架构固有的排列、旋转和缩放对称性来解决这一限制。我们针对现代分组查询注意力（GQA）和 SwiGLU 层调整参数空间对齐，探索基于权重和基于激活的方法。使用这种对齐优先策略，我们成功地将高级推理技能转移到非推理模型。在具有挑战性的推理基准上进行的实验表明，我们的方法始终优于标准任务算法。这项工作提供了一种有效的方法，可以在不断发展的法学硕士系列中合并和转移专业技能，减少冗余微调并增强模型适应性。

Title: From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems

• Authors: Parisa Rabbani, Nimet Beyza Bozdag, Dilek Hakkani-Tür
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10871
• Pdf URL: https://arxiv.org/pdf/2511.10871
• Copy Paste: [[2511.10871]] From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems(https://arxiv.org/abs/2511.10871)
• Keywords: gpt, llm
• Abstract: LLMs are increasingly employed as judges across a variety of tasks, including those involving everyday social interactions. Yet, it remains unclear whether such LLM-judges can reliably assess tasks that require social or conversational judgment. We investigate how an LLM's conviction is changed when a task is reframed from a direct factual query to a Conversational Judgment Task. Our evaluation framework contrasts the model's performance on direct factual queries with its assessment of a speaker's correctness when the same information is presented within a minimal dialogue, effectively shifting the query from "Is this statement correct?" to "Is this speaker correct?". Furthermore, we apply pressure in the form of a simple rebuttal ("The previous answer is incorrect.") to both conditions. This perturbation allows us to measure how firmly the model maintains its position under conversational pressure. Our findings show that while some models like GPT-4o-mini reveal sycophantic tendencies under social framing tasks, others like Llama-8B-Instruct become overly-critical. We observe an average performance change of 9.24% across all models, demonstrating that even minimal dialogue context can significantly alter model judgment, underscoring conversational framing as a key factor in LLM-based evaluation. The proposed framework offers a reproducible methodology for diagnosing model conviction and contributes to the development of more trustworthy dialogue systems.
• 摘要：法学硕士越来越多地担任各种任务的法官，包括那些涉及日常社交互动的任务。然而，目前尚不清楚这些法学硕士法官是否能够可靠地评估需要社交或对话判断的任务。我们研究了当任务从直接事实查询重新构建为对话判断任务时，法学硕士的信念如何改变。我们的评估框架将模型在直接事实查询上的表现与在最小对话中呈现相同信息时对说话者正确性的评估进行了对比，从而有效地将查询从“这个陈述正确吗？”转变为“这个陈述正确吗？”到“这个演讲者正确吗？”。此外，我们以简单反驳（“之前的答案不正确”）的形式对这两种情况施加压力。这种扰动使我们能够衡量模型在对话压力下保持其位置的牢固程度。我们的研究结果表明，虽然 GPT-4o-mini 等一些模型在社会框架任务下表现出阿谀奉承的倾向，但 Llama-8B-Instruct 等其他模型却变得过于挑剔。我们观察到所有模型的平均性能变化为 9.24%，这表明即使是最小的对话上下文也可以显着改变模型判断，强调对话框架是基于 LLM 的评估的关键因素。所提出的框架提供了一种可重复的方法来诊断模型信念，并有助于开发更值得信赖的对话系统。

Title: ICX360: In-Context eXplainability 360 Toolkit

• Authors: Dennis Wei, Ronny Luss, Xiaomeng Hu, Lucas Monteiro Paes, Pin-Yu Chen, Karthikeyan Natesan Ramamurthy, Erik Miehling, Inge Vejsbjerg, Hendrik Strobelt
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2511.10879
• Pdf URL: https://arxiv.org/pdf/2511.10879
• Copy Paste: [[2511.10879]] ICX360: In-Context eXplainability 360 Toolkit(https://arxiv.org/abs/2511.10879)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: Large Language Models (LLMs) have become ubiquitous in everyday life and are entering higher-stakes applications ranging from summarizing meeting transcripts to answering doctors' questions. As was the case with earlier predictive models, it is crucial that we develop tools for explaining the output of LLMs, be it a summary, list, response to a question, etc. With these needs in mind, we introduce In-Context Explainability 360 (ICX360), an open-source Python toolkit for explaining LLMs with a focus on the user-provided context (or prompts in general) that are fed to the LLMs. ICX360 contains implementations for three recent tools that explain LLMs using both black-box and white-box methods (via perturbations and gradients respectively). The toolkit, available at this https URL, contains quick-start guidance materials as well as detailed tutorials covering use cases such as retrieval augmented generation, natural language generation, and jailbreaking.
• 摘要：大型语言模型（LLM）在日常生活中已经无处不在，并且正在进入从总结会议记录到回答医生问题等更高风险的应用。与早期预测模型的情况一样，我们开发用于解释 LLM 输出的工具至关重要，无论是摘要、列表、对问题的回答等。考虑到这些需求，我们引入了 In-Context Understanding 360 (ICX360)，这是一个开源 Python 工具包，用于解释 LLM ，重点关注提供给 LLM 的用户提供的上下文（或一般提示）。 ICX360 包含三个最新工具的实现，这些工具使用黑盒和白盒方法（分别通过扰动和梯度）解释法学硕士。该工具包可从此 https URL 获取，包含快速入门指导材料以及涵盖检索增强生成、自然语言生成和越狱等用例的详细教程。

Title: A Multifaceted Analysis of Negative Bias in Large Language Models through the Lens of Parametric Knowledge

• Authors: Jongyoon Song, Sangwon Yu, Sungroh Yoon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10881
• Pdf URL: https://arxiv.org/pdf/2511.10881
• Copy Paste: [[2511.10881]] A Multifaceted Analysis of Negative Bias in Large Language Models through the Lens of Parametric Knowledge(https://arxiv.org/abs/2511.10881)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Negative bias refers to the tendency of large language models (LLMs) to excessively generate negative responses in binary decision tasks (e.g., yes-no question answering). Previous research has focused on detecting and addressing negative attention heads that induce negative bias. However, the underlying detailed factors influencing negative bias remain underexplored. In this paper, we demonstrate that LLMs exhibit format-level negative bias, meaning the prompt format more influences their responses than the semantics of the negative response. For the fine-grained study of the negative bias, we introduce a pipeline for constructing the evaluation set, which systematically categorizes the dataset into three subsets based on the model's parametric knowledge: correct, incorrect, and insufficient relevant knowledge. Through analysis of this evaluation set, we identify a shortcut behavior in which models tend to generate negative responses when they lack sufficient knowledge to answer a yes-no question, leading to negative bias. We further examine how negative bias changes under various prompting scenarios related to parametric knowledge. We observe that providing relevant context and offering an "I don't know" option generally reduces negative bias, whereas chain-of-thought prompting tends to amplify the bias. Finally, we demonstrate that the degree of negative bias can vary depending on the type of prompt, which influences the direction of the response. Our work reveals the various factors that influence negative bias, providing critical insights for mitigating it in LLMs.
• 摘要：负面偏见是指大型语言模型（LLM）在二元决策任务（例如是非问题回答）中过度产生负面反应的倾向。先前的研究重点是检测和解决引起负面偏见的负面注意力头。然而，影响负面偏见的潜在详细因素仍未得到充分探索。在本文中，我们证明法学硕士表现出格式层面的负面偏见，这意味着提示格式比负面响应的语义更能影响他们的响应。为了对负偏差进行细粒度的研究，我们引入了构建评估集的管道，该管道根据模型的参数知识将数据集系统地分为三个子集：正确、不正确和相关知识不足。通过对这个评估集的分析，我们发现了一种捷径行为，即当模型缺乏足够的知识来回答是非问题时，模型往往会产生负面反应，从而导致负面偏见。我们进一步研究了在与参数知识相关的各种提示场景下负偏差如何变化。我们观察到，提供相关背景并提供“我不知道”选项通常会减少负面偏见，而思维链提示往往会放大偏见。最后，我们证明了负面偏见的程度可以根据提示的类型而变化，这会影响响应的方向。我们的工作揭示了影响负面偏见的各种因素，为减少法学硕士中的负面偏见提供了重要的见解。

Title: From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models

• Authors: Farima Fatahi Bayat, Pouya Pezeshkpour, Estevam Hruschka
• Subjects: cs.CL, cs.LO, cs.SE
• Abstract URL: https://arxiv.org/abs/2511.10899
• Pdf URL: https://arxiv.org/pdf/2511.10899
• Copy Paste: [[2511.10899]] From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models(https://arxiv.org/abs/2511.10899)
• Keywords: language model, llm, hallucination
• Abstract: Tool-augmented Language Models (TaLMs) can invoke external tools to solve problems beyond their parametric capacity. However, it remains unclear whether these tool-enabled gains reflect trustworthy reasoning. Focusing on the Code Interpreter tool, we show that even when tools are selected and executed correctly, TaLMs treat tool outputs as substitutes for reasoning, producing solutions that appear correct but lack coherent justification. We term this failure mode Tool-Induced Myopia (TIM), and study it using PYMATH, a benchmark of 1,679 competition-level mathematical problems for which Python code is helpful but not sufficient. We further develop a multi-dimensional evaluation suite to quantify reasoning degradation in TaLMs relative to their non-tool counterparts. Our findings reveal that while TaLMs achieve up to a 19.3 percentage point gain in final-answer accuracy, their reasoning behavior consistently deteriorates (e.g., non-tool LLMs win up to 41.5% more often in pairwise comparisons of the reasoning process). This degradation intensifies with tool use; the more frequently a model invokes tools, the less coherent its reasoning becomes. Moreover, tool use shifts errors from arithmetic mistakes toward global reasoning failures (logic, assumption, creativity); with TIM present in ~55% of high-risk cases. Finally, we propose a preference-optimization-based framework that realigns TaLMs to use tools as assistive evidence, improving both final-answer accuracy and reasoning depth under tool use. Codes and data are available at: this https URL.
• 摘要：工具增强语言模型（TaLM）可以调用外部工具来解决超出其参数能力的问题。然而，目前尚不清楚这些工具带来的收益是否反映了值得信赖的推理。重点关注代码解释器工具，我们表明，即使正确选择和执行工具，TaLM 也会将工具输出视为推理的替代品，产生看似正确但缺乏连贯理由的解决方案。我们将这种故障模式称为“工具诱发近视”(TIM)，并使用 PYMATH 对其进行研究，PYMATH 是 1,679 个竞赛级数学问题的基准，Python 代码对此有帮助，但还不够。我们进一步开发了一个多维评估套件，以量化 TaLM 相对于非工具对应物的推理退化。我们的研究结果表明，虽然 TaLM 的最终答案准确率提高了 19.3 个百分点，但他们的推理行为却持续恶化（例如，在推理过程的成对比较中，非工具法学硕士获胜的几率高出 41.5%）。这种退化随着工具的使用而加剧；模型调用工具越频繁，其推理就越不连贯。此外，工具的使用将错误从算术错误转向全局推理失败（逻辑、假设、创造力）； TIM 存在于约 55% 的高风险病例中。最后，我们提出了一个基于偏好优化的框架，重新调整 TaLM 以使用工具作为辅助证据，从而提高工具使用下的最终答案准确性和推理深度。代码和数据可在以下位置获得：此 https URL。

Title: Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering

• Authors: Xueren Ge, Sahil Murtaza, Anthony Cortez, Homa Alemzadeh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10900
• Pdf URL: https://arxiv.org/pdf/2511.10900
• Copy Paste: [[2511.10900]] Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering(https://arxiv.org/abs/2511.10900)
• Keywords: language model, llm, prompt, retrieval-augmented generation, chain-of-thought
• Abstract: Large language models (LLMs) have shown promise in medical question answering, yet they often overlook the domain-specific expertise that professionals depend on, such as the clinical subject areas (e.g., trauma, airway) and the certification level (e.g., EMT, Paramedic). Existing approaches typically apply general-purpose prompting or retrieval strategies without leveraging this structured context, limiting performance in high-stakes settings. We address this gap with EMSQA, an 24.3K-question multiple-choice dataset spanning 10 clinical subject areas and 4 certification levels, accompanied by curated, subject area-aligned knowledge bases (40K documents and 2M tokens). Building on EMSQA, we introduce (i) Expert-CoT, a prompting strategy that conditions chain-of-thought (CoT) reasoning on specific clinical subject area and certification level, and (ii) ExpertRAG, a retrieval-augmented generation pipeline that grounds responses in subject area-aligned documents and real-world patient data. Experiments on 4 LLMs show that Expert-CoT improves up to 2.05% over vanilla CoT prompting. Additionally, combining Expert-CoT with ExpertRAG yields up to a 4.59% accuracy gain over standard RAG baselines. Notably, the 32B expertise-augmented LLMs pass all the computer-adaptive EMS certification simulation exams.
• 摘要：大语言模型 (LLM) 在医学问答方面表现出了希望，但它们经常忽视专业人员所依赖的特定领域的专业知识，例如临床学科领域（例如创伤、气道）和认证级别（例如 EMT、护理人员）。现有方法通常应用通用提示或检索策略，而不利用这种结构化上下文，从而限制了高风险环境中的性能。我们通过 EMSQA 弥补了这一差距，这是一个包含 24,300 个问题的多项选择数据集，涵盖 10 个临床学科领域和 4 个认证级别，并附带精心策划的、与学科领域一致的知识库（40K 文档和 200 万个令牌）。在 EMSQA 的基础上，我们引入了 (i) Expert-CoT，一种提示策略，可对特定临床主题领域和认证级别进行思想链 (CoT) 推理；(ii) ExpertRAG，一种检索增强生成管道，可根据主题领域一致的文档和真实患者数据中的响应。对 4 个法学硕士进行的实验表明，Expert-CoT 比普通 CoT 提示提高了 2.05%。此外，将 Expert-CoT 与 ExpertRAG 相结合可比标准 RAG 基线提高高达 4.59% 的准确度。值得注意的是，32B 专业知识增强型法学硕士通过了所有计算机自适应 EMS 认证模拟考试。

Title: Multimodal Peer Review Simulation with Actionable To-Do Recommendations for Community-Aware Manuscript Revisions

• Authors: Mengze Hong, Di Jiang, Weiwei Zhao, Yawen Li, Yihang Wang, Xinyuan Luo, Yanjie Sun, Chen Jason Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.10902
• Pdf URL: https://arxiv.org/pdf/2511.10902
• Copy Paste: [[2511.10902]] Multimodal Peer Review Simulation with Actionable To-Do Recommendations for Community-Aware Manuscript Revisions(https://arxiv.org/abs/2511.10902)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: While large language models (LLMs) offer promising capabilities for automating academic workflows, existing systems for academic peer review remain constrained by text-only inputs, limited contextual grounding, and a lack of actionable feedback. In this work, we present an interactive web-based system for multimodal, community-aware peer review simulation to enable effective manuscript revisions before paper submission. Our framework integrates textual and visual information through multimodal LLMs, enhances review quality via retrieval-augmented generation (RAG) grounded in web-scale OpenReview data, and converts generated reviews into actionable to-do lists using the proposed Action:Objective[\#] format, providing structured and traceable guidance. The system integrates seamlessly into existing academic writing platforms, providing interactive interfaces for real-time feedback and revision tracking. Experimental results highlight the effectiveness of the proposed system in generating more comprehensive and useful reviews aligned with expert standards, surpassing ablated baselines and advancing transparent, human-centered scholarly assistance.
• 摘要：虽然大型语言模型 (LLM) 为自动化学术工作流程提供了有前途的功能，但现有的学术同行评审系统仍然受到纯文本输入、有限的上下文基础和缺乏可操作的反馈的限制。在这项工作中，我们提出了一个基于网络的交互式系统，用于多模式、社区意识的同行评审模拟，以便在论文提交之前实现有效的手稿修改。我们的框架通过多模式 LLM 集成文本和视觉信息，通过基于网络规模 OpenReview 数据的检索增强生成 (RAG) 提高审稿质量，并使用建议的 Action:Objective[\#] 格式将生成的审稿转换为可操作的待办事项列表，从而提供结构化和可追踪的指导。该系统无缝集成到现有的学术写作平台，提供实时反馈和修订跟踪的交互界面。实验结果强调了所提出的系统在生成符合专家标准的更全面和有用的评论、超越消融的基线和推进透明、以人为本的学术援助方面的有效性。

Title: Automated Analysis of Learning Outcomes and Exam Questions Based on Bloom's Taxonomy

• Authors: Ramya Kumar, Dhruv Gulwani, Sonit Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10903
• Pdf URL: https://arxiv.org/pdf/2511.10903
• Copy Paste: [[2511.10903]] Automated Analysis of Learning Outcomes and Exam Questions Based on Bloom's Taxonomy(https://arxiv.org/abs/2511.10903)
• Keywords: language model, llm
• Abstract: This paper explores the automatic classification of exam questions and learning outcomes according to Bloom's Taxonomy. A small dataset of 600 sentences labeled with six cognitive categories - Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation - was processed using traditional machine learning (ML) models (Naive Bayes, Logistic Regression, Support Vector Machines), recurrent neural network architectures (LSTM, BiLSTM, GRU, BiGRU), transformer-based models (BERT and RoBERTa), and large language models (OpenAI, Gemini, Ollama, Anthropic). Each model was evaluated under different preprocessing and augmentation strategies (for example, synonym replacement, word embeddings, etc.). Among traditional ML approaches, Support Vector Machines (SVM) with data augmentation achieved the best overall performance, reaching 94 percent accuracy, recall, and F1 scores with minimal overfitting. In contrast, the RNN models and BERT suffered from severe overfitting, while RoBERTa initially overcame it but began to show signs as training progressed. Finally, zero-shot evaluations of large language models (LLMs) indicated that OpenAI and Gemini performed best among the tested LLMs, achieving approximately 0.72-0.73 accuracy and comparable F1 scores. These findings highlight the challenges of training complex deep models on limited data and underscore the value of careful data augmentation and simpler algorithms (such as augmented SVM) for Bloom's Taxonomy classification.
• 摘要：本文探讨了根据布鲁姆分类法对考试问题和学习成果进行自动分类。使用传统机器学习 (ML) 模型（朴素贝叶斯、逻辑回归、支持向量机）、递归神经网络架构（LSTM、BiLSTM、GRU、BiGRU）、基于 Transformer 的模型（BERT 和 RoBERTa）以及大型语言模型（OpenAI、Gemini、Ollama、Anthropic）处理包含 600 个句子的小型数据集，这些句子标记为知识、理解、应用、分析、综合和评估这六个认知类别。每个模型都在不同的预处理和增强策略（例如同义词替换、词嵌入等）下进行评估。在传统的 ML 方法中，具有数据增强功能的支持向量机 (SVM) 实现了最佳的整体性能，达到了 94% 的准确率、召回率和 F1 分数，并且过度拟合最小化。相比之下，RNN 模型和 BERT 存在严重的过拟合问题，而 RoBERTa 最初克服了这一问题，但随着训练的进展开始出现迹象。最后，大型语言模型 (LLM) 的零样本评估表明，OpenAI 和 Gemini 在测试的 LLM 中表现最好，达到约 0.72-0.73 的准确率和可比的 F1 分数。这些发现凸显了在有限数据上训练复杂深度模型的挑战，并强调了仔细的数据增强和更简单的算法（例如增强的 SVM）对于 Bloom 分类法的价值。

Title: Evaluating Large Language Models on Rare Disease Diagnosis: A Case Study using House M.D

• Authors: Arsh Gupta, Ajay Narayanan Sridhar, Bonam Mingole, Amulya Yadav
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10912
• Pdf URL: https://arxiv.org/pdf/2511.10912
• Copy Paste: [[2511.10912]] Evaluating Large Language Models on Rare Disease Diagnosis: A Case Study using House M.D(https://arxiv.org/abs/2511.10912)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have demonstrated capabilities across diverse domains, yet their performance on rare disease diagnosis from narrative medical cases remains underexplored. We introduce a novel dataset of 176 symptom-diagnosis pairs extracted from House M.D., a medical television series validated for teaching rare disease recognition in medical education. We evaluate four state-of-the-art LLMs such as GPT 4o mini, GPT 5 mini, Gemini 2.5 Flash, and Gemini 2.5 Pro on narrative-based diagnostic reasoning tasks. Results show significant variation in performance, ranging from 16.48% to 38.64% accuracy, with newer model generations demonstrating a 2.3 times improvement. While all models face substantial challenges with rare disease diagnosis, the observed improvement across architectures suggests promising directions for future development. Our educationally validated benchmark establishes baseline performance metrics for narrative medical reasoning and provides a publicly accessible evaluation framework for advancing AI-assisted diagnosis research.
• 摘要：大语言模型（LLM）已经展示了跨不同领域的能力，但它们在叙述性医疗案例中诊断罕见疾病方面的性能仍未得到充分探索。我们引入了从 House M.D. 中提取的 176 个症状诊断对的新颖数据集，House M.D. 是一部经过验证可用于医学教育中罕见疾病识别教学的医学电视连续剧。我们在基于叙述的诊断推理任务上评估了四种最先进的法学硕士，例如 GPT 4o mini、GPT 5 mini、Gemini 2.5 Flash 和 Gemini 2.5 Pro。结果显示，性能存在显着差异，准确率从 16.48% 到 38.64% 不等，较新的模型显示出 2.3 倍的改进。虽然所有模型都面临着罕见疾病诊断的巨大挑战，但观察到的跨架构的改进表明了未来发展的有希望的方向。我们经过教育验证的基准为叙事医学推理建立了基线性能指标，并为推进人工智能辅助诊断研究提供了一个可公开访问的评估框架。

Title: DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains

• Authors: Xiying Zhao, Zhoufutu Wen, Zhixuan Chen, Jingzhe Ding, Jianpeng Jiao, Shuai Li, Xi Li, Danni Liang, Shengda Long, Qianqian Liu, Xianbo Wu, Hongwan Gao, Xiang Gao, Liang Hu, Jiashuo Liu, Mengyun Liu, Weiran Shi, Chenghao Yang, Qianyu Yang, Xuanliang Zhang, Ge Zhang, Wenhao Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10984
• Pdf URL: https://arxiv.org/pdf/2511.10984
• Copy Paste: [[2511.10984]] DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains(https://arxiv.org/abs/2511.10984)
• Keywords: llm
• Abstract: The evaluation of discourse-level translation in expert domains remains inadequate, despite its centrality to knowledge dissemination and cross-lingual scholarly communication. While these translations demand discourse-level coherence and strict terminological precision, current evaluation methods predominantly focus on segment-level accuracy and fluency. To address this limitation, we introduce DiscoX, a new benchmark for discourse-level and expert-level Chinese-English translation. It comprises 200 professionally-curated texts from 7 domains, with an average length exceeding 1700 tokens. To evaluate performance on DiscoX, we also develop Metric-S, a reference-free system that provides fine-grained automatic assessments across accuracy, fluency, and appropriateness. Metric-S demonstrates strong consistency with human judgments, significantly outperforming existing metrics. Our experiments reveal a remarkable performance gap: even the most advanced LLMs still trail human experts on these tasks. This finding validates the difficulty of DiscoX and underscores the challenges that remain in achieving professional-grade machine translation. The proposed benchmark and evaluation system provide a robust framework for more rigorous evaluation, facilitating future advancements in LLM-based translation.
• 摘要：尽管专家领域的话语层面翻译对于知识传播和跨语言学术交流至关重要，但对它的评估仍然不足。虽然这些翻译要求语篇层面的连贯性和严格的术语精度，但目前的评估方法主要关注片段层面的准确性和流畅性。为了解决这个限制，我们引入了DiscoX，这是一个话语级和专家级中英翻译的新基准。它包含来自 7 个领域的 200 篇专业策划的文本，平均长度超过 1700 个标记。为了评估 DiscoX 上的表现，我们还开发了 Metric-S，这是一个无参考系统，可提供准确度、流畅度和适当性的细粒度自动评估。 Metric-S 表现出与人类判断的高度一致性，显着优于现有指标。我们的实验揭示了显着的性能差距：即使是最先进的法学硕士在这些任务上仍然落后于人类专家。这一发现证实了 DiscoX 的难度，并强调了实现专业级机器翻译仍然面临的挑战。拟议的基准和评估系统为更严格的评估提供了一个强大的框架，促进了基于法学硕士的翻译的未来进步。

Title: When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets

• Authors: Aladin Djuhera, Farhan Ahmed, Swanand Ravindra Kadhe, Syed Zawad, Heiko Ludwig, Holger Boche
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.10985
• Pdf URL: https://arxiv.org/pdf/2511.10985
• Copy Paste: [[2511.10985]] When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets(https://arxiv.org/abs/2511.10985)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) is a central objective of post-training, often achieved through reward modeling and reinforcement learning methods. Among these, direct preference optimization (DPO) has emerged as a widely adopted technique that fine-tunes LLMs on preferred completions over less favorable ones. While most frontier LLMs do not disclose their curated preference pairs, the broader LLM community has released several open-source DPO datasets, including TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs. However, systematic comparisons remain scarce, largely due to the high computational cost and the lack of rich quality annotations, making it difficult to understand how preferences were selected, which task types they span, and how well they reflect human judgment on a per-sample level. In this work, we present the first comprehensive, data-centric analysis of popular open-source DPO corpora. We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward, a reward-model-based signal that validates the preference order without relying on human annotations. This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins. Building on these insights, we systematically curate a new DPO mixture, UltraMix, that draws selectively from all five corpora while removing noisy or redundant samples. UltraMix is 30% smaller than the best-performing individual dataset yet exceeds its performance across key benchmarks. We publicly release all annotations, metadata, and our curated mixture to facilitate future research in data-centric preference optimization.
• 摘要：调整大语言模型 (LLM) 是训练后的核心目标，通常通过奖励建模和强化学习方法来实现。其中，直接偏好优化（DPO）已成为一种广泛采用的技术，它可以根据首选的完成情况而不是不太有利的完成情况对法学硕士进行微调。虽然大多数前沿法学硕士不会披露他们精选的偏好对，但更广泛的法学硕士社区已经发布了多个开源 DPO 数据集，包括 TuluDPO、ORPO、UltraFeedback、HelpSteer 和 Code-Preference-Pairs。然而，系统的比较仍然很少，这主要是由于计算成本高和缺乏丰富的质量注释，使得很难理解偏好是如何选择的，它们涵盖哪些任务类型，以及它们在每个样本水平上反映人类判断的程度。在这项工作中，我们首次对流行的开源 DPO 语料库进行全面、以数据为中心的分析。我们利用 Magpie 框架来注释每个样本的任务类别、输入质量和偏好奖励，这是一种基于奖励模型的信号，可在不依赖人工注释的情况下验证偏好顺序。这使得能够对跨数据集的偏好质量进行可扩展的、细粒度的检查，揭示奖励利润的结构和质量差异。基于这些见解，我们系统地策划了一种新的 DPO 混合物 UltraMix，它有选择地从所有五个语料库中提取，同时去除噪声或冗余样本。 UltraMix 比性能最佳的单个数据集小 30%，但在关键基准测试中的性能却超过了其性能。我们公开发布所有注释、元数据和我们精心策划的混合物，以促进未来以数据为中心的偏好优化研究。

Title: Automata-Based Steering of Large Language Models for Diverse Structured Generation

• Authors: Xiaokun Luan, Zeming Wei, Yihao Zhang, Meng Sun
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2511.11018
• Pdf URL: https://arxiv.org/pdf/2511.11018
• Copy Paste: [[2511.11018]] Automata-Based Steering of Large Language Models for Diverse Structured Generation(https://arxiv.org/abs/2511.11018)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly tasked with generating structured outputs. While structured generation methods ensure validity, they often lack output diversity, a critical limitation that we confirm in our preliminary study. We propose a novel method to enhance diversity in automaton-based structured generation. Our approach utilizes automata traversal history to steer LLMs towards novel structural patterns. Evaluations show our method significantly improves structural and content diversity while maintaining comparable generation efficiency. Furthermore, we conduct a case study showcasing the effectiveness of our method in generating diverse test cases for testing open-source libraries.
• 摘要：大型语言模型 (LLM) 越来越多地承担生成结构化输出的任务。虽然结构化生成方法确保了有效性，但它们通常缺乏输出多样性，这是我们在初步研究中证实的一个关键限制。我们提出了一种新方法来增强基于自动机的结构化生成的多样性。我们的方法利用自动机遍历历史来引导法学硕士走向新颖的结构模式。评估表明，我们的方法显着提高了结构和内容多样性，同时保持了可比较的生成效率。此外，我们还进行了案例研究，展示了我们的方法在生成用于测试开源库的各种测试用例方面的有效性。

Title: Can LLMs Detect Their Own Hallucinations?

• Authors: Sora Kadotani, Kosuke Nishida, Kyosuke Nishida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.11087
• Pdf URL: https://arxiv.org/pdf/2511.11087
• Copy Paste: [[2511.11087]] Can LLMs Detect Their Own Hallucinations?(https://arxiv.org/abs/2511.11087)
• Keywords: language model, gpt, llm, hallucination, chain-of-thought
• Abstract: Large language models (LLMs) can generate fluent responses, but sometimes hallucinate facts. In this paper, we investigate whether LLMs can detect their own hallucinations. We formulate hallucination detection as a classification task of a sentence. We propose a framework for estimating LLMs' capability of hallucination detection and a classification method using Chain-of-Thought (CoT) to extract knowledge from their parameters. The experimental results indicated that GPT-$3.5$ Turbo with CoT detected $58.2\%$ of its own hallucinations. We concluded that LLMs with CoT can detect hallucinations if sufficient knowledge is contained in their parameters.
• 摘要：大型语言模型 (LLM) 可以生成流畅的响应，但有时会产生幻觉事实。在本文中，我们研究了法学硕士是否能够检测到自己的幻觉。我们将幻觉检测制定为句子的分类任务。我们提出了一个估计法学硕士幻觉检测能力的框架，以及一种使用思想链（CoT）从其参数中提取知识的分类方法。实验结果表明，带有 CoT 的 GPT-$3.5$ Turbo 检测到了 $58.2\%$ 的自身幻觉。我们得出的结论是，如果参数中包含足够的知识，具有 CoT 的法学硕士可以检测幻觉。

Title: Analysing Personal Attacks in U.S. Presidential Debates

• Authors: Ruban Goyal, Rohitash Chandra, Sonit Singh
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2511.11108
• Pdf URL: https://arxiv.org/pdf/2511.11108
• Copy Paste: [[2511.11108]] Analysing Personal Attacks in U.S. Presidential Debates(https://arxiv.org/abs/2511.11108)
• Keywords: language model, llm
• Abstract: Personal attacks have become a notable feature of U.S. presidential debates and play an important role in shaping public perception during elections. Detecting such attacks can improve transparency in political discourse and provide insights for journalists, analysts and the public. Advances in deep learning and transformer-based models, particularly BERT and large language models (LLMs) have created new opportunities for automated detection of harmful language. Motivated by these developments, we present a framework for analysing personal attacks in U.S. presidential debates. Our work involves manual annotation of debate transcripts across the 2016, 2020 and 2024 election cycles, followed by statistical and language-model based analysis. We investigate the potential of fine-tuned transformer models alongside general-purpose LLMs to detect personal attacks in formal political speech. This study demonstrates how task-specific adaptation of modern language models can contribute to a deeper understanding of political communication.
• 摘要：人身攻击已成为美国总统辩论的一个显着特征，并在选举期间塑造公众看法方面发挥着重要作用。检测此类攻击可以提高政治话语的透明度，并为记者、分析师和公众提供见解。深度学习和基于 Transformer 的模型，特别是 BERT 和大型语言模型 (LLM) 的进步，为自动检测有害语言创造了新的机会。受这些发展的推动，我们提出了一个分析美国总统辩论中人身攻击的框架。我们的工作包括对 2016 年、2020 年和 2024 年选举周期的辩论笔录进行手动注释，然后进行基于统计和语言模型的分析。我们研究了微调变压器模型与通用法学硕士在检测正式政治演讲中的人身攻击方面的潜力。这项研究展示了现代语言模型的特定任务适应如何有助于更深入地理解政治传播。

Title: AV-Dialog: Spoken Dialogue Models with Audio-Visual Input

• Authors: Tuochao Chen, Bandhav Veluri, Hongyu Gong, Shyamnath Gollakota
• Subjects: cs.CL, cs.AI, cs.CV, cs.MM, cs.SD
• Abstract URL: https://arxiv.org/abs/2511.11124
• Pdf URL: https://arxiv.org/pdf/2511.11124
• Copy Paste: [[2511.11124]] AV-Dialog: Spoken Dialogue Models with Audio-Visual Input(https://arxiv.org/abs/2511.11124)
• Keywords: agent
• Abstract: Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.
• 摘要：对话模型在嘈杂的多说话人环境中会出现问题，经常会产生不相关的响应和尴尬的轮流。我们提出了 AV-Dialog，这是第一个多模式对话框架，它使用音频和视觉提示来跟踪目标说话者、预测轮流并生成连贯的响应。通过将声学标记化与单元、合成和真实视听对话数据集上的多任务、多阶段训练相结合，AV-Dialog 实现了强大的流转录、基于语义的转弯边界检测和准确的响应，从而产生自然的对话流。实验表明，AV-Dialog 在干扰下的性能优于纯音频模型，减少了转录错误，改进了轮流预测，并提高了人工评价的对话质量。这些结果凸显了说话者感知交互中视觉和听觉的力量，为在现实世界的嘈杂环境中{稳健}执行的{口语}对话代理铺平了道路。

Title: Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal Fusion

• Authors: Yi Shi, Wenlong Meng, Zhenyuan Guo, Chengkun Wei, Wenzhi Chen
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2511.11126
• Pdf URL: https://arxiv.org/pdf/2511.11126
• Copy Paste: [[2511.11126]] Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal Fusion(https://arxiv.org/abs/2511.11126)
• Keywords: language model, llm
• Abstract: With the rapid rise of social media and Internet culture, memes have become a popular medium for expressing emotional tendencies. This has sparked growing interest in Meme Emotion Understanding (MEU), which aims to classify the emotional intent behind memes by leveraging their multimodal contents. While existing efforts have achieved promising results, two major challenges remain: (1) a lack of fine-grained multimodal fusion strategies, and (2) insufficient mining of memes' implicit meanings and background knowledge. To address these challenges, we propose MemoDetector, a novel framework for advancing MEU. First, we introduce a four-step textual enhancement module that utilizes the rich knowledge and reasoning capabilities of Multimodal Large Language Models (MLLMs) to progressively infer and extract implicit and contextual insights from memes. These enhanced texts significantly enrich the original meme contents and provide valuable guidance for downstream classification. Next, we design a dual-stage modal fusion strategy: the first stage performs shallow fusion on raw meme image and text, while the second stage deeply integrates the enhanced visual and textual features. This hierarchical fusion enables the model to better capture nuanced cross-modal emotional cues. Experiments on two datasets, MET-MEME and MOOD, demonstrate that our method consistently outperforms state-of-the-art baselines. Specifically, MemoDetector improves F1 scores by 4.3\% on MET-MEME and 3.4\% on MOOD. Further ablation studies and in-depth analyses validate the effectiveness and robustness of our approach, highlighting its strong potential for advancing MEU. Our code is available at this https URL.
• 摘要：随着社交媒体和网络文化的迅速崛起，模因已成为表达情感倾向的流行媒介。这引发了人们对模因情感理解（MEU）的兴趣日益浓厚，该技术旨在通过利用模因的多模态内容对模因背后的情感意图进行分类。虽然现有的努力取得了可喜的成果，但仍然存在两大挑战：（1）缺乏细粒度的多模态融合策略，（2）对模因隐含含义和背景知识的挖掘不足。为了应对这些挑战，我们提出了 MemoDetector，这是一种用于推进 MEU 的新颖框架。首先，我们引入了一个四步文本增强模块，该模块利用多模态大型语言模型（MLLM）的丰富知识和推理能力来逐步推断和提取模因中隐含的和上下文的见解。这些增强的文本显着丰富了原始模因内容，并为下游分类提供了宝贵的指导。接下来，我们设计了双阶段模态融合策略：第一阶段对原始模因图像和文本进行浅层融合，第二阶段深度融合增强的视觉和文本特征。这种层次融合使模型能够更好地捕捉微妙的跨模式情感线索。对 MET-MEME 和 MOOD 这两个数据集的实验表明，我们的方法始终优于最先进的基线。具体来说，MemoDetector 将 MET-MEME 上的 F1 分数提高了 4.3%，将 MOOD 上的 F1 分数提高了 3.4%。进一步的消融研究和深入分析验证了我们方法的有效性和稳健性，凸显了其推进 MEU 的强大潜力。我们的代码可以在这个 https URL 上找到。

Title: Speech-Aware Long Context Pruning and Integration for Contextualized Automatic Speech Recognition

• Authors: Yiming Rong, Yixin Zhang, Ziyi Wang, Deyang Jiang, Yunlong Zhao, Haoran Wu, Shiyu Zhou, Bo Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.11139
• Pdf URL: https://arxiv.org/pdf/2511.11139
• Copy Paste: [[2511.11139]] Speech-Aware Long Context Pruning and Integration for Contextualized Automatic Speech Recognition(https://arxiv.org/abs/2511.11139)
• Keywords: long context
• Abstract: Automatic speech recognition (ASR) systems have achieved remarkable performance in common conditions but often struggle to leverage long-context information in contextualized scenarios that require domain-specific knowledge, such as conference presentations. This challenge arises primarily due to constrained model context windows and the sparsity of relevant information within extensive contextual noise. To solve this, we propose the SAP$^{2}$ method, a novel framework that dynamically prunes and integrates relevant contextual keywords in two stages. Specifically, each stage leverages our proposed Speech-Driven Attention-based Pooling mechanism, enabling efficient compression of context embeddings while preserving speech-salient information. Experimental results demonstrate state-of-the-art performance of SAP$^{2}$ on the SlideSpeech and LibriSpeech datasets, achieving word error rates (WER) of 7.71% and 1.12%, respectively. On SlideSpeech, our method notably reduces biased keyword error rates (B-WER) by 41.1% compared to non-contextual baselines. SAP$^{2}$ also exhibits robust scalability, consistently maintaining performance under extensive contextual input conditions on both datasets.
• 摘要：自动语音识别 (ASR) 系统在常见条件下取得了显着的性能，但在需要特定领域知识的上下文场景（例如会议演示）中常常难以利用长上下文信息。这一挑战的出现主要是由于模型上下文窗口受限以及广泛的上下文噪声中相关信息的稀疏性。为了解决这个问题，我们提出了 SAP$^{2}$ 方法，这是一种新颖的框架，可以分两个阶段动态修剪和集成相关的上下文关键字。具体来说，每个阶段都利用我们提出的基于语音驱动注意的池机制，实现上下文嵌入的高效压缩，同时保留语音显着信息。实验结果证明 SAP$^{2}$ 在 SlideSpeech 和 LibriSpeech 数据集上具有最先进的性能，分别实现了 7.71% 和 1.12% 的单词错误率 (WER)。在 SlideSpeech 上，与非上下文基线相比，我们的方法显着降低了有偏见的关键字错误率 (B-WER) 41.1%。 SAP$^{2}$ 还表现出强大的可扩展性，在两个数据集上的广泛上下文输入条件下始终保持性能。

Title: LANE: Lexical Adversarial Negative Examples for Word Sense Disambiguation

• Authors: Jader Martins Camboim de Sá, Jooyoung Lee, Cédric Pruski, Marcos Da Silveira
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.11234
• Pdf URL: https://arxiv.org/pdf/2511.11234
• Copy Paste: [[2511.11234]] LANE: Lexical Adversarial Negative Examples for Word Sense Disambiguation(https://arxiv.org/abs/2511.11234)
• Keywords: language model
• Abstract: Fine-grained word meaning resolution remains a critical challenge for neural language models (NLMs) as they often overfit to global sentence representations, failing to capture local semantic details. We propose a novel adversarial training strategy, called LANE, to address this limitation by deliberately shifting the model's learning focus to the target word. This method generates challenging negative training examples through the selective marking of alternate words in the training set. The goal is to force the model to create a greater separability between same sentences with different marked words. Experimental results on lexical semantic change detection and word sense disambiguation benchmarks demonstrate that our approach yields more discriminative word representations, improving performance over standard contrastive learning baselines. We further provide qualitative analyses showing that the proposed negatives lead to representations that better capture subtle meaning differences even in challenging environments. Our method is model-agnostic and can be integrated into existing representation learning frameworks.
• 摘要：细粒度的词义解析仍然是神经语言模型（NLM）的一个关键挑战，因为它们经常过度适应全局句子表示，无法捕获局部语义细节。我们提出了一种新颖的对抗性训练策略，称为 LANE，通过故意将模型的学习重点转移到目标词来解决这一限制。该方法通过选择性标记训练集中的替代词来生成具有挑战性的负面训练示例。目标是迫使模型在具有不同标记单词的相同句子之间创建更大的可分离性。词汇语义变化检测和词义消歧基准的实验结果表明，我们的方法产生了更具辨别力的单词表示，比标准对比学习基线提高了性能。我们进一步提供的定性分析表明，即使在具有挑战性的环境中，所提出的否定也会导致更好地捕捉微妙含义差异的表示。我们的方法与模型无关，可以集成到现有的表示学习框架中。

Title: KGQuest: Template-Driven QA Generation from Knowledge Graphs with LLM-Based Refinement

• Authors: Sania Nayab, Marco Simoni, Giulio Rossolini, Andrea Saracino
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.11258
• Pdf URL: https://arxiv.org/pdf/2511.11258
• Copy Paste: [[2511.11258]] KGQuest: Template-Driven QA Generation from Knowledge Graphs with LLM-Based Refinement(https://arxiv.org/abs/2511.11258)
• Keywords: language model, llm
• Abstract: The generation of questions and answers (QA) from knowledge graphs (KG) plays a crucial role in the development and testing of educational platforms, dissemination tools, and large language models (LLM). However, existing approaches often struggle with scalability, linguistic quality, and factual consistency. This paper presents a scalable and deterministic pipeline for generating natural language QA from KGs, with an additional refinement step using LLMs to further enhance linguistic quality. The approach first clusters KG triplets based on their relations, creating reusable templates through natural language rules derived from the entity types of objects and relations. A module then leverages LLMs to refine these templates, improving clarity and coherence while preserving factual accuracy. Finally, the instantiation of answer options is achieved through a selection strategy that introduces distractors from the KG. Our experiments demonstrate that this hybrid approach efficiently generates high-quality QA pairs, combining scalability with fluency and linguistic precision.
• 摘要：从知识图（KG）生成问答（QA）在教育平台、传播工具和大语言模型（LLM）的开发和测试中发挥着至关重要的作用。然而，现有的方法常常在可扩展性、语言质量和事实一致性方面遇到困难。本文提出了一种可扩展且确定性的管道，用于从 KG 生成自然语言 QA，并使用 LLM 进行额外的细化步骤以进一步提高语言质量。该方法首先根据 KG 三元组的关系进行聚类，通过从对象和关系的实体类型派生的自然语言规则创建可重用的模板。然后，模块利用法学硕士来完善这些模板，提高清晰度和连贯性，同时保持事实准确性。最后，答案选项的实例化是通过引入 KG 干扰因素的选择策略来实现的。我们的实验表明，这种混合方法可以有效地生成高质量的 QA 对，将可扩展性与流畅性和语言精度结合起来。

Title: iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference

• Authors: Wei Fan, JinYi Yoon, Bo Ji
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2511.11306
• Pdf URL: https://arxiv.org/pdf/2511.11306
• Copy Paste: [[2511.11306]] iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference(https://arxiv.org/abs/2511.11306)
• Keywords: language model, llm, prompt, agent
• Abstract: Large Language Model (LLM) agent systems have advanced rapidly, driven by their strong generalization in zero-shot settings. To further enhance reasoning and accuracy on complex tasks, Multi-Agent Debate (MAD) has emerged as a promising framework that engages multiple LLM agents in structured debates to encourage diverse reasoning. However, triggering MAD for every query is inefficient, as it incurs substantial computational (token) cost and may even degrade accuracy by overturning correct single-agent answers. To address these limitations, we propose intelligent Multi-Agent Debate (iMAD), a token-efficient framework that selectively triggers MAD only when it is likely to be beneficial (i.e., correcting an initially wrong answer). To achieve this goal, iMAD learns generalizable model behaviors to make accurate debate decisions. Specifically, iMAD first prompts a single agent to produce a structured self-critique response, from which we extract 41 interpretable linguistic and semantic features capturing hesitation cues. Then, iMAD uses a lightweight debate-decision classifier, trained using our proposed FocusCal loss, to determine whether to trigger MAD, enabling robust debate decisions without test dataset-specific tuning. Through extensive experiments using six (visual) question answering datasets against five competitive baselines, we have shown that iMAD significantly reduces token usage (by up to 92%) while also improving final answer accuracy (by up to 13.5%).
• 摘要：大型语言模型（LLM）代理系统在零样本设置中强大的泛化能力的推动下发展迅速。为了进一步提高复杂任务的推理和准确性，多智能体辩论（MAD）已经成为一个有前途的框架，它让多个 LLM 智能体参与结构化辩论，以鼓励多样化的推理。然而，为每个查询触发 MAD 效率很低，因为它会产生大量的计算（令牌）成本，甚至可能通过推翻正确的单代理答案来降低准确性。为了解决这些限制，我们提出了智能多代理辩论（iMAD），这是一种代币高效的框架，仅在可能有益时（即纠正最初错误的答案）选择性地触发 MAD。为了实现这一目标，iMAD 学习可概括的模型行为，以做出准确的辩论决策。具体来说，iMAD 首先提示单个智能体产生结构化的自我批评响应，我们从中提取 41 个可解释的语言和语义特征，捕获犹豫线索。然后，iMAD 使用轻量级辩论决策分类器（使用我们提出的 FocusCal 损失进行训练）来确定是否触发 MAD，从而无需测试数据集特定的调整即可实现稳健的辩论决策。通过使用六个（视觉）问答数据集与五个竞争基线进行的广泛实验，我们表明 iMAD 显着减少了令牌使用（高达 92%），同时还提高了最终答案的准确性（高达 13.5%）。

Title: LAET: A Layer-wise Adaptive Ensemble Tuning Framework for Pretrained Language Models

• Authors: Jawad Ibn Ahad, Muhammad Rafsan Kabir, Robin Krambroeckers, Sifat Momen, Nabeel Mohammed, Shafin Rahman
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2511.11315
• Pdf URL: https://arxiv.org/pdf/2511.11315
• Copy Paste: [[2511.11315]] LAET: A Layer-wise Adaptive Ensemble Tuning Framework for Pretrained Language Models(https://arxiv.org/abs/2511.11315)
• Keywords: language model, gpt, llm
• Abstract: Natural Language Processing (NLP) has transformed the financial industry, enabling advancements in areas such as textual analysis, risk management, and forecasting. Large language models (LLMs) like BloombergGPT and FinMA have set new benchmarks across various financial NLP tasks, including sentiment analysis, stock movement prediction, and credit risk assessment. Furthermore, FinMA-ES, a bilingual financial LLM, has also demonstrated strong performance using the FLARE and FLARE-ES benchmarks. However, the high computational demands of these models limit the accessibility of many organizations. To address this, we propose Layer-wise Adaptive Ensemble Tuning (LAET), a novel strategy that selectively fine-tunes the most effective layers of pre-trained LLMs by analyzing hidden state representations while freezing less critical layers. LAET significantly reduces computational overhead while enhancing task-specific performance. Our approach shows strong results in financial NLP tasks, outperforming existing benchmarks and state-of-the-art LLMs such as GPT-4, even with smaller LLMs ($\sim$3B parameters). This work bridges cutting-edge financial NLP research and real-world deployment with efficient and scalable models for financial applications.
• 摘要：自然语言处理 (NLP) 改变了金融行业，推动了文本分析、风险管理和预测等领域的进步。 BloombergGPT 和 FinMA 等大型语言模型 (LLM) 为各种金融 NLP 任务设定了新基准，包括情绪分析、股票走势预测和信用风险评估。此外，双语金融法学硕士 FinMA-ES 在使用 FLARE 和 FLARE-ES 基准方面也表现出了强劲的表现。然而，这些模型的高计算要求限制了许多组织的可访问性。为了解决这个问题，我们提出了逐层自适应集成调整（LAET），这是一种新颖的策略，通过分析隐藏状态表示，同时冻结不太关键的层，选择性地微调预训练 LLM 的最有效层。 LAET 显着降低了计算开销，同时增强了特定任务的性能。我们的方法在金融 NLP 任务中显示出强劲的结果，优于现有基准和最先进的 LLM（例如 GPT-4），即使是较小的 LLM（$\sim$3B 参数）。这项工作将前沿的金融 NLP 研究和现实世界的部署与高效且可扩展的金融应用模型联系起来。

Title: NOVA: An Agentic Framework for Automated Histopathology Analysis and Discovery

• Authors: Anurag J. Vaidya, Felix Meissen, Daniel C. Castro, Shruthi Bannur, Tristan Lazard, Drew F. K. Williamson, Faisal Mahmood, Javier Alvarez-Valle, Stephanie L. Hyland, Kenza Bouzid
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.11324
• Pdf URL: https://arxiv.org/pdf/2511.11324
• Copy Paste: [[2511.11324]] NOVA: An Agentic Framework for Automated Histopathology Analysis and Discovery(https://arxiv.org/abs/2511.11324)
• Keywords: agent
• Abstract: Digitized histopathology analysis involves complex, time-intensive workflows and specialized expertise, limiting its accessibility. We introduce NOVA, an agentic framework that translates scientific queries into executable analysis pipelines by iteratively generating and running Python code. NOVA integrates 49 domain-specific tools (e.g., nuclei segmentation, whole-slide encoding) built on open-source software, and can also create new tools ad hoc. To evaluate such systems, we present SlideQuest, a 90-question benchmark -- verified by pathologists and biomedical scientists -- spanning data processing, quantitative analysis, and hypothesis testing. Unlike prior biomedical benchmarks focused on knowledge recall or diagnostic QA, SlideQuest demands multi-step reasoning, iterative coding, and computational problem solving. Quantitative evaluation shows NOVA outperforms coding-agent baselines, and a pathologist-verified case study links morphology to prognostically relevant PAM50 subtypes, demonstrating its scalable discovery potential.
• 摘要：数字化组织病理学分析涉及复杂、耗时的工作流程和专业知识，限制了其可访问性。我们介绍 NOVA，这是一个代理框架，通过迭代生成和运行 Python 代码，将科学查询转化为可执行的分析管道。 NOVA 集成了 49 个基于开源软件构建的特定领域工具（例如，细胞核分割、整个幻灯片编码），并且还可以临时创建新工具。为了评估此类系统，我们推出了 SlideQuest，这是一个包含 90 个问题的基准测试，经过病理学家和生物医学科学家的验证，涵盖数据处理、定量分析和假设检验。与之前专注于知识回忆或诊断 QA 的生物医学基准不同，SlideQuest 需要多步骤推理、迭代编码和计算问题解决。定量评估显示 NOVA 优于编码剂基线，并且经病理学家验证的案例研究将形态学与预后相关的 PAM50 亚型联系起来，证明了其可扩展的发现潜力。

Title: LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models

• Authors: Jian Gao, Richeng Xuan, Zhaolu Kang, Dingshi Liao, Wenxin Huang, Zongmou Huang, Yangdi Xu, Bowen Qin, Zheqi He, Xi Yang, Changjin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.11334
• Pdf URL: https://arxiv.org/pdf/2511.11334
• Copy Paste: [[2511.11334]] LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models(https://arxiv.org/abs/2511.11334)
• Keywords: language model, llm, agent
• Abstract: The rapid advancement of large language models (LLMs) has not been matched by their evaluation in low-resource languages, especially Southeast Asian languages like Lao. To fill this gap, we introduce LaoBench, the first large-scale, high-quality, and multidimensional benchmark dataset dedicated to assessing LLMs' comprehensive language understanding and reasoning abilities in Lao. LaoBench comprises over 17,000 carefully curated samples spanning three core dimensions: knowledge application, K12 foundational education, and bilingual translation among Lao, Chinese, and English. The dataset is divided into open-source and closed-source subsets, with the closed-source portion enabling black-box evaluation on an official platform to ensure fairness and data security. Our data construction pipeline integrates expert human curation with automated agent-assisted verification, ensuring linguistic accuracy, cultural relevance, and educational value. Benchmarking multiple state-of-the-art LLMs on LaoBench reveals that current models still face significant challenges in mastering Lao across diverse tasks. We hope LaoBench will catalyze further research and development of AI technologies for underrepresented Southeast Asian languages.
• 摘要：大语言模型（LLM）的快速发展并没有与它们在低资源语言，尤其是像老挝语这样的东南亚语言中的评估相匹配。为了填补这一空白，我们引入了LaoBench，这是第一个大规模、高质量、多维度的基准数据集，致力于评估LLM的老挝语综合语言理解和推理能力。 LaoBench 包含超过 17,000 个精心挑选的样本，涵盖三个核心维度：知识应用、K12 基础教育以及老挝语、汉语和英语的双语翻译。数据集分为开源和闭源子集，闭源部分可以在官方平台进行黑盒评估，保证公平性和数据安全。我们的数据构建管道将专家人工管理与自动化代理辅助验证相结合，确保语言准确性、文化相关性和教育价值。在 LaoBench 上对多个最先进的法学硕士进行基准测试表明，当前模型在跨不同任务掌握 Lao 方面仍然面临重大挑战。我们希望 LaoBench 能够促进针对代表性不足的东南亚语言的人工智能技术的进一步研究和开发。

Title: M-DAIGT: A Shared Task on Multi-Domain Detection of AI-Generated Text

• Authors: Salima Lamsiyah, Saad Ezzini, Abdelkader El Mahdaouy, Hamza Alami, Abdessamad Benlahbib, Samir El Amrany, Salmane Chafik, Hicham Hammouchi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2511.11340
• Pdf URL: https://arxiv.org/pdf/2511.11340
• Copy Paste: [[2511.11340]] M-DAIGT: A Shared Task on Multi-Domain Detection of AI-Generated Text(https://arxiv.org/abs/2511.11340)
• Keywords: language model, gpt, llm, prompt
• Abstract: The generation of highly fluent text by Large Language Models (LLMs) poses a significant challenge to information integrity and academic research. In this paper, we introduce the Multi-Domain Detection of AI-Generated Text (M-DAIGT) shared task, which focuses on detecting AI-generated text across multiple domains, particularly in news articles and academic writing. M-DAIGT comprises two binary classification subtasks: News Article Detection (NAD) (Subtask 1) and Academic Writing Detection (AWD) (Subtask 2). To support this task, we developed and released a new large-scale benchmark dataset of 30,000 samples, balanced between human-written and AI-generated texts. The AI-generated content was produced using a variety of modern LLMs (e.g., GPT-4, Claude) and diverse prompting strategies. A total of 46 unique teams registered for the shared task, of which four teams submitted final results. All four teams participated in both Subtask 1 and Subtask 2. We describe the methods employed by these participating teams and briefly discuss future directions for M-DAIGT.
• 摘要：大型语言模型（LLM）生成高度流畅的文本对信息完整性和学术研究提出了重大挑战。在本文中，我们介绍了人工智能生成文本的多域检测（M-DAIGT）共享任务，该任务专注于跨多个领域检测人工智能生成的文本，特别是在新闻文章和学术写作中。 M-DAIGT 包含两个二元分类子任务：新闻文章检测 (NAD)（子任务 1）和学术写作检测（AWD）（子任务 2）。为了支持这项任务，我们开发并发布了一个包含 30,000 个样本的新的大规模基准数据集，在人类编写的文本和人工智能生成的文本之间进行了平衡。人工智能生成的内容是使用各种现代法学硕士（例如 GPT-4、Claude）和不同的提示策略生成的。共有 46 个不同的团队报名参加共享任务，其中 4 个团队提交了最终结果。所有四个团队都参与了子任务 1 和子任务 2。我们描述了这些参与团队所采用的方法，并简要讨论了 M-DAIGT 的未来方向。

Title: Studies with impossible languages falsify LMs as models of human language

• Authors: Jeffrey S. Bowers, Jeff Mitchell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.11389
• Pdf URL: https://arxiv.org/pdf/2511.11389
• Copy Paste: [[2511.11389]] Studies with impossible languages falsify LMs as models of human language(https://arxiv.org/abs/2511.11389)
• Keywords: language model
• Abstract: According to Futrell and Mahowald [arXiv:2501.17047], both infants and language models (LMs) find attested languages easier to learn than impossible languages that have unnatural structures. We review the literature and show that LMs often learn attested and many impossible languages equally well. Difficult to learn impossible languages are simply more complex (or random). LMs are missing human inductive biases that support language acquisition.
• 摘要：根据 Futrell 和 Mahowald [arXiv:2501.17047] 的说法，婴儿和语言模型 (LM) 都发现经过验证的语言比具有不自然结构的不可能语言更容易学习。我们回顾了文献并表明，LM 通常能够同样出色地学习经过验证的语言和许多不可能的语言。难以学习不可能的语言只是更复杂（或随机）。语言模型缺少支持语言习得的人类归纳偏差。

Title: Proactive Hearing Assistants that Isolate Egocentric Conversations

• Authors: Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath Gollakota
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2511.11473
• Pdf URL: https://arxiv.org/pdf/2511.11473
• Copy Paste: [[2511.11473]] Proactive Hearing Assistants that Isolate Egocentric Conversations(https://arxiv.org/abs/2511.11473)
• Keywords: prompt
• Abstract: We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: this https URL
• 摘要：我们推出了主动式助听器，可以自动识别和区分佩戴者的对话伙伴，无需明确提示。我们的系统以自我为中心的双耳音频运行，并使用佩戴者的自言自语作为锚点，利用轮流行为和对话动态来推断对话伙伴并压制他人。为了实现实时的设备上操作，我们提出了一种双模型架构：轻量级流模型每 12.5 毫秒运行一次，以低延迟提取对话伙伴，而较慢的模型运行频率较低，以捕获更远距离的对话动态。使用双耳自我中心硬件从 11 名参与者收集总计 6.8 小时的现实世界 2 人和 3 说话者对话测试集的结果显示了在多对话环境中识别和隔离对话伙伴的概括性。我们的工作标志着助听器向主动适应对话动态和参与度迈出了一步。更多信息可以在我们的网站上找到：此 https URL

Title: W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search

• Authors: Zhenyu Ding, Yuhao Wang, Tengyue Xiao, Haoying Wang, Guojun Ma, Mingyang Wan, Caigui Jiang, Ning Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2511.11518
• Pdf URL: https://arxiv.org/pdf/2511.11518
• Copy Paste: [[2511.11518]] W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search(https://arxiv.org/abs/2511.11518)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) demonstrate impressive capabilities, yet their outputs often suffer from misalignment with human preferences due to the inadequacy of weak supervision and a lack of fine-grained control. Training-time alignment methods like Reinforcement Learning from Human Feedback (RLHF) face prohibitive costs in expert supervision and inherent scalability limitations, offering limited dynamic control during inference. Consequently, there is an urgent need for scalable and adaptable alignment mechanisms. To address this, we propose W2S-AlignTree, a pioneering plug-and-play inference-time alignment framework that synergistically combines Monte Carlo Tree Search (MCTS) with the Weak-to-Strong Generalization paradigm for the first time. W2S-AlignTree formulates LLM alignment as an optimal heuristic search problem within a generative search tree. By leveraging weak model's real-time, step-level signals as alignment proxies and introducing an Entropy-Aware exploration mechanism, W2S-AlignTree enables fine-grained guidance during strong model's generation without modifying its parameters. The approach dynamically balances exploration and exploitation in high-dimensional generation search trees. Experiments across controlled sentiment generation, summarization, and instruction-following show that W2S-AlignTree consistently outperforms strong baselines. Notably, W2S-AlignTree raises the performance of Llama3-8B from 1.89 to 2.19, a relative improvement of 15.9 on the summarization task.
• 摘要：大型语言模型（LLM）展示了令人印象深刻的能力，但由于弱监督和缺乏细粒度控制的不足，它们的输出常常与人类偏好不一致。人类反馈强化学习 (RLHF) 等训练时间对齐方法面临专家监督的高昂成本和固有的可扩展性限制，在推理过程中提供有限的动态控制。因此，迫切需要可扩展且适应性强的对齐机制。为了解决这个问题，我们提出了 W2S-AlignTree，这是一种开创性的即插即用推理时间对齐框架，首次将蒙特卡洛树搜索 (MCTS) 与弱到强泛化范式协同结合。 W2S-AlignTree 将 LLM 对齐公式化为生成搜索树中的最佳启发式搜索问题。通过利用弱模型的实时、步进级信号作为对齐代理并引入熵感知探索机制，W2S-AlignTree 可以在强模型生成过程中实现细粒度指导，而无需修改其参数。该方法动态平衡高维生成搜索树中的探索和利用。受控情绪生成、摘要和指令遵循的实验表明，W2S-AlignTree 始终优于强大的基线。值得注意的是，W2S-AlignTree 将 Llama3-8B 的性能从 1.89 提高到 2.19，在摘要任务上相对提高了 15.9。

Title: PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

• Authors: Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2511.11562
• Pdf URL: https://arxiv.org/pdf/2511.11562
• Copy Paste: [[2511.11562]] PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning(https://arxiv.org/abs/2511.11562)
• Keywords: prompt
• Abstract: Frontier model progress is often measured by academic benchmarks, which offer a limited view of performance in real-world professional contexts. Existing evaluations often fail to assess open-ended, economically consequential tasks in high-stakes domains like Legal and Finance, where practical returns are paramount. To address this, we introduce Professional Reasoning Bench (PRBench), a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law. We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it, to our knowledge, the largest public, rubric-based benchmark for both legal and finance domains. We recruit 182 qualified professionals, holding JDs, CFAs, or 6+ years of experience, who contributed tasks inspired by their actual workflows. This process yields significant diversity, with tasks spanning 114 countries and 47 US jurisdictions. Our expert-curated rubrics are validated through a rigorous quality pipeline, including independent expert validation. Subsequent evaluation of 20 leading models reveals substantial room for improvement, with top scores of only 0.39 (Finance) and 0.37 (Legal) on our Hard subsets. We further catalog associated economic impacts of the prompts and analyze performance using human-annotated rubric categories. Our analysis shows that models with similar overall scores can diverge significantly on specific capabilities. Common failure modes include inaccurate judgments, a lack of process transparency and incomplete reasoning, highlighting critical gaps in their reliability for professional adoption.
• 摘要：前沿模型的进展通常通过学术基准来衡量，这些基准对现实专业环境中的表现提供了有限的看法。现有的评估往往无法评估法律和金融等高风险领域的开放式、具有经济意义的任务，这些领域的实际回报至关重要。为了解决这个问题，我们引入了专业推理基准（PRBench），这是一个针对金融和法律领域现实问题的现实的、开放式的、困难的基准。我们开源了 1,100 项专家撰写的任务和 19,356 项专家策划的标准，据我们所知，使其成为法律和金融领域最大的公共、基于标准的基准。我们招募了 182 名合格的专业人士，他们拥有 JD、CFA 或 6 年以上的经验，他们根据实际工作流程贡献了任务。这一过程产生了显着的多样性，任务跨越 114 个国家和 47 个美国司法管辖区。我们的专家策划的评分标准通过严格的质量管道进行验证，包括独立的专家验证。对 20 个领先模型的后续评估揭示了巨大的改进空间，在我们的 Hard 子集上，最高得分仅为 0.39（金融）和 0.37（法律）。我们进一步对提示的相关经济影响进行分类，并使用人工注释的标题类别分析绩效。我们的分析表明，总体得分相似的模型在特定功能上可能存在显着差异。常见的失败模式包括判断不准确、流程缺乏透明度和推理不完整，凸显了其在专业采用可靠性方面的关键差距。