2025-07-22

Title: DeepWriter: A Fact-Grounded Multimodal Writing Assistant Based On Offline Knowledge Base

• Authors: Song Mao, Lejun Cheng, Pinlong Cai, Guohang Yan, Ding Wang, Botian Shi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14189
• Pdf URL: https://arxiv.org/pdf/2507.14189
• Copy Paste: [[2507.14189]] DeepWriter: A Fact-Grounded Multimodal Writing Assistant Based On Offline Knowledge Base(https://arxiv.org/abs/2507.14189)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in various applications. However, their use as writing assistants in specialized domains like finance, medicine, and law is often hampered by a lack of deep domain-specific knowledge and a tendency to hallucinate. Existing solutions, such as Retrieval-Augmented Generation (RAG), can suffer from inconsistency across multiple retrieval steps, while online search-based methods often degrade quality due to unreliable web content. To address these challenges, we introduce DeepWriter, a customizable, multimodal, long-form writing assistant that operates on a curated, offline knowledge base. DeepWriter leverages a novel pipeline that involves task decomposition, outline generation, multimodal retrieval, and section-by-section composition with reflection. By deeply mining information from a structured corpus and incorporating both textual and visual elements, DeepWriter generates coherent, factually grounded, and professional-grade documents. We also propose a hierarchical knowledge representation to enhance retrieval efficiency and accuracy. Our experiments on financial report generation demonstrate that DeepWriter produces high-quality, verifiable articles that surpasses existing baselines in factual accuracy and generated content quality.
• 摘要：大型语言模型（LLMS）在各种应用中都表现出了显着的功能。但是，他们在金融，医学和法律等专业领域中用作写作助手通常会因缺乏深入的领域特定知识和幻觉的趋势而受到阻碍。现有的解决方案（例如检索型生成（RAG））可能会在多个检索步骤中遇到不一致的障碍，而基于在线搜索的方法由于不可靠的Web内容而经常降低质量。为了应对这些挑战，我们介绍了DeepWriter，这是一个可自定义的，多模式的，长格式的写作助手，该助理在精心策划的离线知识库中运作。 DeepWriter利用一条新型管道，涉及任务分解，轮廓生成，多模式检索和逐节组成。通过从结构化语料库中深入挖掘信息，并同时结合文本和视觉元素，DeepWriter生成了连贯的，实际上扎根和专业级的文档。我们还提出了分层知识表示，以提高检索效率和准确性。我们对财务报告的实验表明，Deepwriter生产的高质量，可验证的文章以实际准确性和产生的内容质量超过现有基准。

Title: Retention analysis of edited knowledge after fine-tuning

• Authors: Fufang Wen, Shichang Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14198
• Pdf URL: https://arxiv.org/pdf/2507.14198
• Copy Paste: [[2507.14198]] Retention analysis of edited knowledge after fine-tuning(https://arxiv.org/abs/2507.14198)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior. Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training. In parallel, LLMs are frequently fine-tuned for a wide range of downstream tasks. However, the effect of fine-tuning on previously edited knowledge remains poorly understood. In this work, we systematically investigate how different fine-tuning objectives interact with various model editing techniques. Our findings show that edited knowledge is substantially more susceptible to forgetting during fine-tuning than intrinsic knowledge acquired through pre-training. This analysis highlights a key limitation of current editing approaches and suggests that evaluating edit robustness under downstream fine-tuning is critical for their practical deployment. We further find that freezing layers associated with edited content can significantly improve knowledge retention, offering insight into how future editing methods might be made more robust.
• 摘要：大型语言模型（LLMS）存储了大量知识，这些知识通常需要更新才能纠正事实错误，结合了新获得的信息或改编模型行为。模型编辑方法已成为此类更新的有效解决方案，与持续培训相比，以明显低的计算成本提供了本地化和精确的知识修改。同时，对于多种下游任务，LLM经常经常进行微调。但是，微调对先前编辑的知识的影响仍然很少了解。在这项工作中，我们系统地研究了不同的微调目标如何与各种模型编辑技术相互作用。我们的发现表明，与通过预训练获得的内在知识相比，编辑的知识在微调过程中更容易忘记。该分析强调了当前编辑方法的关键局限性，并表明在下游微调下评估编辑鲁棒性对于其实际部署至关重要。我们进一步发现，与编辑内容相关的冻结层可以显着改善知识的保留，从而深入了解如何使未来的编辑方法变得更强大。

Title: Open-Source LLMs Collaboration Beats Closed-Source LLMs: A Scalable Multi-Agent System

• Authors: Shengji Tang, Jianjian Cao, Weihao Lin, Jiale Hong, Bo Zhang, Shuyue Hu, Lei Bai, Tao Chen, Wanli Ouyang, Peng Ye
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.14200
• Pdf URL: https://arxiv.org/pdf/2507.14200
• Copy Paste: [[2507.14200]] Open-Source LLMs Collaboration Beats Closed-Source LLMs: A Scalable Multi-Agent System(https://arxiv.org/abs/2507.14200)
• Keywords: gpt, llm, agent
• Abstract: This paper aims to demonstrate the potential and strengths of open-source collectives. It leads to a promising question: Can we harness multiple open-source LLMs to match or even beat the closed-source LLMs? To answer this, we propose SMACS, a scalable multi-agent collaboration system (MACS) framework with high performance. Specifically, for continuous integration of new LLMs and generalization to diverse questions, we first propose a Retrieval-based Prior Selection (RPS), which assigns a proxy performance score to each LLM to select the Top-k LLMs at the instance level for any given question. Then, we propose an Exploration-Exploitation-Driven Posterior Enhancement (EPE), encouraging the generation of diverse responses through prior dropping and selecting the high-quality response via a hybrid posterior score. Experiments on eight mainstream benchmarks validate the effectiveness of our SMACS: by integrating fifteen open-source LLMs, SMACS outperforms leading closed-source LLMs in 2025, e.g., Claude-3.7-Sonnet (+12.73%), GPT-4.1(+5.36%) and GPT-o3-mini(+5.28%) across multiple tasks. Remarkably, it even exceeds the average of best results of different datasets from both open-source LLMs (+2.86%) and closed-source LLMs (+2.04%), pushing the upper bound of intelligence. Code will be released at this https URL.
• 摘要：本文旨在证明开源集体的潜力和优势。这导致了一个有前途的问题：我们可以利用多个开源LLM匹配甚至击败封闭源LLM吗？为了回答这个问题，我们建议使用高性能的可扩展的多代理协作系统（MAC）框架SMAC。具体而言，为了将新的LLM和概括与各种问题的持续集成，我们首先提出了基于检索的先验选择（RPS），该选择将代理性能得分分配给每个LLM，以在任何给定的问题上在实例级别选择Top-K LLMS。然后，我们提出了一个探索探索驱动的后验增强（EPE），通过事先下降并通过混合后后评分选择高质量响应，从而鼓励产生各种反应。八个主流基准的实验验证了我们的SMAC的有效性：通过整合15个开源LLMS，SMAC的表现优于2025年的闭合源LLM，例如Claude-3.7-Sonnet（+12.73％），GPT-4.1（+5.36％）和gpt-osters and gpt-oss and gpt-oss（+5.36％）。值得注意的是，它甚至超过了来自开源LLM（+2.86％）和封闭源LLM（+2.04％）的不同数据集的最佳结果的平均结果，从而推动了智能的上限。代码将在此HTTPS URL上发布。

Title: Beyond Architectures: Evaluating the Role of Contextual Embeddings in Detecting Bipolar Disorder on Social Media

• Authors: Khalid Hasan, Jamil Saquer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.14231
• Pdf URL: https://arxiv.org/pdf/2507.14231
• Copy Paste: [[2507.14231]] Beyond Architectures: Evaluating the Role of Contextual Embeddings in Detecting Bipolar Disorder on Social Media(https://arxiv.org/abs/2507.14231)
• Keywords: language model
• Abstract: Bipolar disorder is a chronic mental illness frequently underdiagnosed due to subtle early symptoms and social stigma. This paper explores the advanced natural language processing (NLP) models for recognizing signs of bipolar disorder based on user-generated social media text. We conduct a comprehensive evaluation of transformer-based models (BERT, RoBERTa, ALBERT, ELECTRA, DistilBERT) and Long Short Term Memory (LSTM) models based on contextualized (BERT) and static (GloVe, Word2Vec) word embeddings. Experiments were performed on a large, annotated dataset of Reddit posts after confirming their validity through sentiment variance and judgmental analysis. Our results demonstrate that RoBERTa achieves the highest performance among transformer models with an F1 score of ~98% while LSTM models using BERT embeddings yield nearly identical results. In contrast, LSTMs trained on static embeddings fail to capture meaningful patterns, scoring near-zero F1. These findings underscore the critical role of contextual language modeling in detecting bipolar disorder. In addition, we report model training times and highlight that DistilBERT offers an optimal balance between efficiency and accuracy. In general, our study offers actionable insights for model selection in mental health NLP applications and validates the potential of contextualized language models to support early bipolar disorder screening.
• 摘要：双相情感障碍是由于早期症状和社会污名而经常诊断出的一种慢性精神疾病。本文探讨了基于用户生成的社交媒体文本识别双相情感障碍迹象的先进自然语言处理（NLP）模型。我们对基于上下文（BERT）和静态（bert）和静态（word2vec）单词嵌入的基于变压器的模型（Bert，Roberta，Albert，Electra，Distilbert）进行了全面评估。通过情感差异和判断分析确认其有效性后，在大型注释的Reddit帖子数据集上进行了实验。我们的结果表明，罗伯塔（Roberta）在变压器模型中的性能最高，F1得分约为98％，而使用BERT嵌入的LSTM模型产生了几乎相同的结果。相比之下，接受静态嵌入的LSTMS无法捕获有意义的模式，得分接近零F1。这些发现强调了上下文语言建模在检测躁郁症中的关键作用。此外，我们报告了模型培训时间，并强调了Distilbert在效率和准确性之间提供最佳平衡。通常，我们的研究为心理健康NLP应用中的模型选择提供了可行的见解，并验证了情境化语言模型支持早期躁郁症筛查的潜力。

Title: Language Models Change Facts Based on the Way You Talk

• Authors: Matthew Kearney, Reuben Binns, Yarin Gal
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2507.14238
• Pdf URL: https://arxiv.org/pdf/2507.14238
• Copy Paste: [[2507.14238]] Language Models Change Facts Based on the Way You Talk(https://arxiv.org/abs/2507.14238)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly being used in user-facing applications, from providing medical consultations to job interview advice. Recent research suggests that these models are becoming increasingly proficient at inferring identity information about the author of a piece of text from linguistic patterns as subtle as the choice of a few words. However, little is known about how LLMs use this information in their decision-making in real-world applications. We perform the first comprehensive analysis of how identity markers present in a user's writing bias LLM responses across five different high-stakes LLM applications in the domains of medicine, law, politics, government benefits, and job salaries. We find that LLMs are extremely sensitive to markers of identity in user queries and that race, gender, and age consistently influence LLM responses in these applications. For instance, when providing medical advice, we find that models apply different standards of care to individuals of different ethnicities for the same symptoms; we find that LLMs are more likely to alter answers to align with a conservative (liberal) political worldview when asked factual questions by older (younger) individuals; and that LLMs recommend lower salaries for non-White job applicants and higher salaries for women compared to men. Taken together, these biases mean that the use of off-the-shelf LLMs for these applications may cause harmful differences in medical care, foster wage gaps, and create different political factual realities for people of different identities. Beyond providing an analysis, we also provide new tools for evaluating how subtle encoding of identity in users' language choices impacts model decisions. Given the serious implications of these findings, we recommend that similar thorough assessments of LLM use in user-facing applications are conducted before future deployment.
• 摘要：大型语言模型（LLM）越来越多地用于面向用户的应用程序，从提供医疗咨询到工作面试建议。最近的研究表明，这些模型越来越熟练地推断出有关文本的作者的身份信息，从语言模式中的作者和几个单词的选择一样微妙。但是，对于LLM在实际应用程序中如何使用此信息的知识知之甚少。我们对用户写作中的身份标志物在医学，法律，法律，政治，政府福利和工作薪水领域中的五个不同的LLM应用程序中的偏见反应中的身份标记进行了首次综合分析。我们发现LLM对用户查询中的身份标记非常敏感，并且该应用程序中的种族，性别和年龄始终影响LLM响应。例如，在提供医疗建议时，我们发现模型将不同的护理标准应用于不同种族的个人的同一症状；我们发现，当年龄较大（年轻）的人提出事实问题时，LLM更有可能改变与保守派（自由）政治世界观保持一致的答案。与男性相比，LLMS建议非白人求职者的薪水较低，女性薪水更高。综上所述，这些偏见意味着在这些应用中使用现成的LLM可能会导致医疗保健，养成工资差距的有害差异，并为不同身份的人们创造不同的政治事实现实。除了提供分析外，我们还提供了新的工具来评估用户语言选择中身份的微妙编码如何影响模型决策。鉴于这些发现的严重含义，我们建议在将来部署之前对LLM使用中的LLM使用类似的详尽评估。

Title: CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation

• Authors: Weihua Zheng, Roy Ka-Wei Lee, Zhengyuan Liu, Kui Wu, AiTi Aw, Bowei Zou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14239
• Pdf URL: https://arxiv.org/pdf/2507.14239
• Copy Paste: [[2507.14239]] CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation(https://arxiv.org/abs/2507.14239)
• Keywords: language model, llm, hallucination, prompt, chat, chain-of-thought
• Abstract: Multilingual Large Language Models(MLLMs) demonstrate strong generalization across languages, yet they remain prone to hallucinations, especially in low-resource languages, due to training data imbalances. These hallucinations, which include inaccurate or fabricated outputs, are particularly problematic in domain-specific generation tasks (Chataigner et al., 2024). To address this challenge, we propose CCL-XCoT(Curriculum-based Contrastive Learning-based Cross-lingual Chain-of-Thought), a two-stage fine-tuning framework for mitigating hallucination in MLLMs. Our approach first enhances cross-lingual semantic alignment through curriculum-based contrastive learning combined with next-token prediction during continued pre-training. Building on this foundation, we then introduce a cross-lingual Chain-of-Thought (XCoT) prompting strategy during instruction fine-tuning, which guides the model to reason in a high-resource language before generating answers in the target low-resource language. Experimental results show that CCL-XCoT reduces hallucination rates by up to 62% and substantially improves factual knowledge transfer across language pairs, without relying on external retrieval or multi-model ensembles.
• 摘要：多语言大语模型（MLLM）表现出跨语言的强烈概括，但由于培训数据失衡，它们仍然容易出现幻觉，尤其是在低资源语言中。这些幻觉包括不准确或捏造的输出，在特定领域的生成任务中尤其有问题（Chataigner等，2024）。为了应对这一挑战，我们提出了CCL-XCOT（基于课程的基于学习的跨语言链链），这是一个两阶段的微调框架，用于减轻MLLM中的幻觉。我们的方法首先通过基于课程的对比学习与持续预训练期间的下一步预测相结合，从而增强了跨语性的语义一致性。然后，在此基础的基础上，我们在教学微调过程中引入了跨思考链（XCOT）提示策略，该策略指导该模型以高资源语言进行推理，然后再在目标低资源语言中产生答案。实验结果表明，CCL-XCOT最多将幻觉率降低了62％，并大大改善了跨语言对的事实知识转移，而无需依赖外部检索或多模型集合。

Title: HuggingGraph: Understanding the Supply Chain of LLM Ecosystem

• Authors: Mohammad Shahedur Rahman, Peng Gao, Yuede Ji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14240
• Pdf URL: https://arxiv.org/pdf/2507.14240
• Copy Paste: [[2507.14240]] HuggingGraph: Understanding the Supply Chain of LLM Ecosystem(https://arxiv.org/abs/2507.14240)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) leverage deep learning to process and predict sequences of words from context, enabling them to perform various NLP tasks, such as translation, summarization, question answering, and content generation. However, the growing size and complexity of developing, training, and deploying advanced LLMs require extensive computational resources and large datasets. This creates a barrier for users. As a result, platforms that host models and datasets are widely used. For example, Hugging Face, one of the most popular platforms, hosted 1.8 million models and 450K datasets by June 2025, with no sign of slowing down. Since many LLMs are built from base models, pre-trained models, and external datasets, they can inherit vulnerabilities, biases, or malicious components from earlier models or datasets. Therefore, it is critical to understand the origin and development of these components to better detect potential risks, improve model fairness, and ensure compliance. Motivated by this, our project aims to study the relationships between models and datasets, which are core components of the LLM supply chain. First, we design a method to systematically collect LLM supply chain data. Using this data, we build a directed heterogeneous graph to model the relationships between models and datasets, resulting in a structure with 397,376 nodes and 453,469 edges. We then perform various analyses and uncover several findings, such as: (i) the LLM supply chain graph is large, sparse, and follows a power-law degree distribution; (ii) it features a densely connected core and a fragmented periphery; (iii) datasets play pivotal roles in training; (iv) strong interdependence exists between models and datasets; and (v) the graph is dynamic, with daily updates reflecting the ecosystem's ongoing evolution.
• 摘要：大型语言模型（LLMS）利用深度学习来处理和预测上下文的单词序列，使它们能够执行各种NLP任务，例如翻译，摘要，问题答案和内容生成。但是，开发，培训和部署高级LLM的规模和复杂性不断增长，需要广泛的计算资源和大型数据集。这为用户造成了障碍。结果，主机模型和数据集的平台被广泛使用。例如，在2025年6月，拥抱面孔之一是最受欢迎的平台之一，托管了180万款型号和450k数据集，没有放慢脚步的迹象。由于许多LLM是由基本模型，预训练的模型和外部数据集构建的，因此它们可以继承早期模型或数据集中的漏洞，偏见或恶意组件。因此，了解这些组件的起源和开发至关重要，以更好地检测潜在的风险，提高模型公平并确保合规性。在此激励的基础上，我们的项目旨在研究模型和数据集之间的关系，这些关系是LLM供应链的核心组成部分。首先，我们设计了一种系统收集LLM供应链数据的方法。使用此数据，我们构建了一个有向的异质图来对模型和数据集之间的关系进行建模，从而产生了397,376个节点和453,469个边缘的结构。然后，我们进行了各种分析并发现了几个发现，例如：（i）LLM供应链图较大，稀疏，并遵循幂律学位分布； （ii）它具有密度连接的核心和碎片的外围； （iii）数据集在培训中扮演关键角色； （iv）模型和数据集之间存在强大的相互依赖性； （v）该图是动态的，每天更新反映了生态系统的持续发展。

Title: Promptomatix: An Automatic Prompt Optimization Framework for Large Language Models

• Authors: Rithesh Murthy, Ming Zhu, Liangwei Yang, Jielin Qiu, Juntao Tan, Shelby Heinecke, Huan Wang, Caiming Xiong, Silvio Savarese
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14241
• Pdf URL: https://arxiv.org/pdf/2507.14241
• Copy Paste: [[2507.14241]] Promptomatix: An Automatic Prompt Optimization Framework for Large Language Models(https://arxiv.org/abs/2507.14241)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) perform best with well-crafted prompts, yet prompt engineering remains manual, inconsistent, and inaccessible to non-experts. We introduce Promptomatix, an automatic prompt optimization framework that transforms natural language task descriptions into high-quality prompts without requiring manual tuning or domain expertise. Promptomatix supports both a lightweight meta-prompt-based optimizer and a DSPy-powered compiler, with modular design enabling future extension to more advanced frameworks. The system analyzes user intent, generates synthetic training data, selects prompting strategies, and refines prompts using cost-aware objectives. Evaluated across 5 task categories, Promptomatix achieves competitive or superior performance compared to existing libraries, while reducing prompt length and computational overhead making prompt optimization scalable and efficient.
• 摘要：大型语言模型（LLMS）在精心制作的提示下表现最好，但及时的工程仍然是手动，不一致的，并且对非专家无法接近。我们介绍了Promptomatix，这是一个自动及时的优化框架，将自然语言任务描述转换为高质量的提示，而无需手动调整或域专业知识。 ProSTEMomatix支持基于Meta-Prompt的轻型优化器和DSPY驱动的编译器，模块化设计使未来的扩展可以扩展到更高级的框架。该系统分析用户意图，生成综合培训数据，选择提示策略，并使用成本吸引目标来完善提示。与现有库相比，Promptomatix在5个任务类别中进行了评估，可以实现竞争性或卓越的性能，同时降低了及时的长度和计算开销，从而使迅速优化的优化可扩展和高效。

Title: In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding

• Authors: Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Alexander Jacobson, Lu Yuan, Leonid Sigal
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2507.14298
• Pdf URL: https://arxiv.org/pdf/2507.14298
• Copy Paste: [[2507.14298]] In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding(https://arxiv.org/abs/2507.14298)
• Keywords: language model
• Abstract: Recent methods for customizing Large Vision Language Models (LVLMs) for domain-specific tasks have shown promising results in scientific chart comprehension. However, existing approaches face two major limitations: First, they rely on paired data from only a few chart types, limiting generalization to wide range of chart types. Secondly, they lack targeted pre-training for chart-data alignment, which hampers the model's understanding of underlying data. In this paper, we introduce ChartScope, an LVLM optimized for in-depth chart comprehension across diverse chart types. We propose an efficient data generation pipeline that synthesizes paired data for a wide range of chart types, along with a novel Dual-Path training strategy that enabling the model to succinctly capture essential data details while preserving robust reasoning capabilities by incorporating reasoning over the underlying data. Lastly, we establish ChartDQA, a new benchmark for evaluating not only question-answering at different levels but also underlying data understanding. Experimental results demonstrate that ChartScope significantly enhances comprehension on a wide range of chart types. The code and data are available at this https URL.
• 摘要：定制针对领域特定任务的大型视觉语言模型（LVLM）的最新方法显示了有希望的科学图表理解结果。但是，现有方法面临两个主要局限性：首先，它们仅依靠来自几种图表类型的配对数据，将概括限制在广泛的图表类型中。其次，他们缺乏针对图表数据对准的有针对性的预训练，这阻碍了模型对基本数据的理解。在本文中，我们介绍了ChartScope，这是一种针对各种图表类型的深入图表理解优化的LVLM。我们提出了一个有效的数据生成管道，该管道将配对的数据合成各种图表类型，以及一种新颖的双路训练策略，使该模型能够简洁地捕获基本的数据详细信息，同时通过在基础数据上纳入推理来保留强大的推理能力。最后，我们建立了ChartDQA，这是一种新的基准，用于评估不仅在不同级别的问题，而且是基本数据理解的问题。实验结果表明，ChartScope显着增强了广泛的图表类型的理解。该代码和数据可在此HTTPS URL上找到。

Title: Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study

• Authors: Rakesh Paul, Anusha Kamath, Kanishk Singla, Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.14304
• Pdf URL: https://arxiv.org/pdf/2507.14304
• Copy Paste: [[2507.14304]] Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study(https://arxiv.org/abs/2507.14304)
• Keywords: language model, llm
• Abstract: Multilingual large language models (LLMs) often demonstrate a performance gap between English and non-English languages, particularly in low-resource settings. Aligning these models to low-resource languages is essential yet challenging due to limited high-quality data. While English alignment datasets are readily available, curating equivalent data in other languages is expensive and time-consuming. A common workaround is to translate existing English alignment data; however, standard translation techniques often fail to preserve critical elements such as code, mathematical expressions, and structured formats like JSON. In this work, we investigate LLM-based selective translation, a technique that selectively translates only the translatable parts of a text while preserving non-translatable content and sentence structure. We conduct a systematic study to explore key questions around this approach, including its effectiveness compared to vanilla translation, the importance of filtering noisy outputs, and the benefits of mixing translated samples with original English data during alignment. Our experiments focus on the low-resource Indic language Hindi and compare translations generated by Google Cloud Translation (GCP) and Llama-3.1-405B. The results highlight the promise of selective translation as a practical and effective method for improving multilingual alignment in LLMs.
• 摘要：多语言大语言模型（LLM）通常表现出英语和非英语语言之间的性能差距，尤其是在低资源环境中。由于有限的高质量数据，将这些模型与低资源语言保持一致至关重要。尽管英语对齐数据集很容易获得，但用其他语言策划等效数据既昂贵又耗时。一个常见的解决方法是翻译现有的英语对准数据；但是，标准翻译技术通常无法保留关键要素，例如代码，数学表达式和JSON等结构化格式。在这项工作中，我们研究了基于LLM的选择性翻译，该技术仅在保留不可转移的内容和句子结构的同时选择性地翻译文本的可翻译部分。我们进行了一项系统的研究，以探讨围绕这种方法的关键问题，包括与香草翻译相比，其有效性，过滤嘈杂的输出的重要性以及在对齐过程中将翻译的样品与原始英语数据混合的好处。我们的实验侧重于低资源的语言印地语，并比较由Google Cloud Translation（GCP）和Llama-3.1-405B生成的翻译。结果突出了选择性翻译作为改善LLM中多语言对齐方式的实用有效方法的希望。

Title: How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs

• Authors: Karin de Langis, Jong Inn Park, Andreas Schramm, Bin Hu, Khanh Chi Le, Michael Mensink, Ahn Thu Tong, Dongyeop Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14307
• Pdf URL: https://arxiv.org/pdf/2507.14307
• Copy Paste: [[2507.14307]] How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs(https://arxiv.org/abs/2507.14307)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these behaviors reflect human-like cognition versus advanced pattern recognition remains an open question. In this study, we investigate how LLMs process the temporal meaning of linguistic aspect in narratives that were previously used in human studies. Using an Expert-in-the-Loop probing pipeline, we conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. Our findings show that LLMs over-rely on prototypicality, produce inconsistent aspectual judgments, and struggle with causal reasoning derived from aspect, raising concerns about their ability to fully comprehend narratives. These results suggest that LLMs process aspect fundamentally differently from humans and lack robust narrative understanding. Beyond these empirical findings, we develop a standardized experimental framework for the reliable assessment of LLMs' cognitive and linguistic capabilities.
• 摘要：大型语言模型（LLMS）具有越来越复杂的语言能力，但是这些行为反映了类似人类的认知与高级模式识别的程度仍然是一个悬而未决的问题。在这项研究中，我们研究了LLM如何处理以前在人类研究中使用的叙事中语言方面的时间含义。使用专家探测管道，我们进行了一系列有针对性的实验，以评估LLM是否以类似人类的方式构建语义表示和务实的推论。我们的发现表明，LLM在原型性上过度汇总，产生不一致的方面判断，并与从方面产生的因果推理斗争，从而引发了对他们充分理解叙事能力的担忧。这些结果表明，LLM与人类的过程从根本上不同，并且缺乏强大的叙事理解。除了这些经验发现之外，我们还为LLMS认知和语言能力的可靠评估开发了标准化的实验框架。

Title: What Makes You CLIC: Detection of Croatian Clickbait Headlines

• Authors: Marija Anđedelić, Dominik Šipek, Laura Majer, Jan Šnajder
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14314
• Pdf URL: https://arxiv.org/pdf/2507.14314
• Copy Paste: [[2507.14314]] What Makes You CLIC: Detection of Croatian Clickbait Headlines(https://arxiv.org/abs/2507.14314)
• Keywords: llm, prompt
• Abstract: Online news outlets operate predominantly on an advertising-based revenue model, compelling journalists to create headlines that are often scandalous, intriguing, and provocative -- commonly referred to as clickbait. Automatic detection of clickbait headlines is essential for preserving information quality and reader trust in digital media and requires both contextual understanding and world knowledge. For this task, particularly in less-resourced languages, it remains unclear whether fine-tuned methods or in-context learning (ICL) yield better results. In this paper, we compile CLIC, a novel dataset for clickbait detection of Croatian news headlines spanning a 20-year period and encompassing mainstream and fringe outlets. We fine-tune the BERTić model on this task and compare its performance to LLM-based ICL methods with prompts both in Croatian and English. Finally, we analyze the linguistic properties of clickbait. We find that nearly half of the analyzed headlines contain clickbait, and that finetuned models deliver better results than general LLMs.
• 摘要：在线新闻媒体主要在基于广告的收入模型上运作，这迫使记者创建通常是丑闻，有趣且挑衅的头条新闻 - 通常称为Clickbait。自动检测点击诱饵标题对于在数字媒体中保留信息质量和读者信任至关重要，并且需要上下文理解和世界知识。对于此任务，尤其是在资源较低的语言中，尚不清楚微调方法还是在内在学习（ICL）产生更好的结果。在本文中，我们编译了Clic，这是一个新颖的数据集，用于涉及克罗地亚新闻头条的点击诱饵检测，涵盖了20年的时期，并涵盖了主流和边缘出口。我们在此任务上微调了Bertić模型，并将其性能与基于LLM的ICL方法与克罗地亚和英语的提示进行比较。最后，我们分析了点击诱饵的语言特性。我们发现，将近一半的分析头条包含点击诱饵，而填充模型比一般LLM可以提供更好的结果。

Title: Can LLMs Infer Personality from Real World Conversations?

• Authors: Jianfeng Zhu, Ruoming Jin, Karin G. Coifman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14355
• Pdf URL: https://arxiv.org/pdf/2507.14355
• Copy Paste: [[2507.14355]] Can LLMs Infer Personality from Real World Conversations?(https://arxiv.org/abs/2507.14355)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) such as OpenAI's GPT-4 and Meta's LLaMA offer a promising approach for scalable personality assessment from open-ended language. However, inferring personality traits remains challenging, and earlier work often relied on synthetic data or social media text lacking psychometric validity. We introduce a real-world benchmark of 555 semi-structured interviews with BFI-10 self-report scores for evaluating LLM-based personality inference. Three state-of-the-art LLMs (GPT-4.1 Mini, Meta-LLaMA, and DeepSeek) were tested using zero-shot prompting for BFI-10 item prediction and both zero-shot and chain-of-thought prompting for Big Five trait inference. All models showed high test-retest reliability, but construct validity was limited: correlations with ground-truth scores were weak (max Pearson's $r = 0.27$), interrater agreement was low (Cohen's $\kappa < 0.10$), and predictions were biased toward moderate or high trait levels. Chain-of-thought prompting and longer input context modestly improved distributional alignment, but not trait-level accuracy. These results underscore limitations in current LLM-based personality inference and highlight the need for evidence-based development for psychological applications.
• 摘要：大型语言模型（LLM），例如OpenAI的GPT-4和Meta的Llama，为开放式语言提供了可扩展性格评估的有前途的方法。但是，推断性格特征仍然具有挑战性，早期的工作通常依赖于综合数据或缺乏心理测量有效性的社交媒体文本。我们介绍了555个半结构化访谈的现实基准，以评估基于LLM的个性推断，以进行BFI-10自我报告分数。使用零射击提示BFI-10项目预测的零射击测试了三个最先进的LLM（GPT-4.1 mini，meta-llama和deepSeek），以及零摄像和链条的提示，提示了五大特征推断。所有模型均显示出高测试可靠性，但构造有效性有限：与地面真相分数的相关性较弱（Max Pearson的$ R = 0.27 $），Interater协议较低（Cohen的$ \ kappa <0.10 $），并且预测偏向于中度或高性状水平。经过思考的链条提示和更长的输入上下文适度改进了分布对准，但不是特质级的准确性。这些结果强调了当前基于LLM的人格推论的局限性，并强调了对心理应用进行基于证据的发展的需求。

Title: Text-to-SQL for Enterprise Data Analytics

• Authors: Albert Chen, Manas Bundele, Gaurav Ahlawat, Patrick Stetz, Zhitao Wang, Qiang Fei, Donghoon Jung, Audrey Chu, Bharadwaj Jayaraman, Ayushi Panth, Yatin Arora, Sourav Jain, Renjith Varma, Alexey Ilin, Iuliia Melnychuk, Chelsea Chueh, Joyan Sil, Xiaofeng Wang
• Subjects: cs.CL, cs.AI, cs.DB, cs.HC
• Abstract URL: https://arxiv.org/abs/2507.14372
• Pdf URL: https://arxiv.org/pdf/2507.14372
• Copy Paste: [[2507.14372]] Text-to-SQL for Enterprise Data Analytics(https://arxiv.org/abs/2507.14372)
• Keywords: language model, hallucination, chat, agent
• Abstract: The introduction of large language models has brought rapid progress on Text-to-SQL benchmarks, but it is not yet easy to build a working enterprise solution. In this paper, we present insights from building an internal chatbot that enables LinkedIn's product managers, engineers, and operations teams to self-serve data insights from a large, dynamic data lake. Our approach features three components. First, we construct a knowledge graph that captures up-to-date semantics by indexing database metadata, historical query logs, wikis, and code. We apply clustering to identify relevant tables for each team or product area. Second, we build a Text-to-SQL agent that retrieves and ranks context from the knowledge graph, writes a query, and automatically corrects hallucinations and syntax errors. Third, we build an interactive chatbot that supports various user intents, from data discovery to query writing to debugging, and displays responses in rich UI elements to encourage follow-up chats. Our chatbot has over 300 weekly users. Expert review shows that 53% of its responses are correct or close to correct on an internal benchmark set. Through ablation studies, we identify the most important knowledge graph and modeling components, offering a practical path for developing enterprise Text-to-SQL solutions.
• 摘要：大型语言模型的引入在文本到SQL基准测试上取得了迅速的进步，但是构建工作企业解决方案并不容易。在本文中，我们提供了构建内部聊天机器人的见解，该聊天机器人使LinkedIn的产品经理，工程师和运营团队能够从一个大型动态数据湖进行自助数据见解。我们的方法具有三个组成部分。首先，我们构建了一个知识图，该图图通过索引数据库元数据，历史查询日志，Wiki和代码来捕获最新语义。我们应用聚类来确定每个团队或产品区域的相关表。其次，我们构建了一个文本到SQL代理，该代理从知识图，编写查询并自动纠正幻觉和语法错误，从知识图中检索和排名上下文。第三，我们构建了一个交互式聊天机器人，该聊天机器人支持各种用户意图，从数据发现到查询写作再到调试，并在Rich UI元素中显示响应，以鼓励后续聊天。我们的聊天机器人每周有300多个用户。专家审查显示，其响应中有53％是正确或接近内部基准集合集的正确响应。通过消融研究，我们确定了最重要的知识图和建模组件，为开发企业文本到SQL解决方案提供了实用的途径。

Title: Error-Aware Curriculum Learning for Biomedical Relation Classification

• Authors: Sinchani Chakraborty, Sudeshna Sarkar, Pawan Goyal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14374
• Pdf URL: https://arxiv.org/pdf/2507.14374
• Copy Paste: [[2507.14374]] Error-Aware Curriculum Learning for Biomedical Relation Classification(https://arxiv.org/abs/2507.14374)
• Keywords: language model, gpt
• Abstract: Relation Classification (RC) in biomedical texts is essential for constructing knowledge graphs and enabling applications such as drug repurposing and clinical decision-making. We propose an error-aware teacher--student framework that improves RC through structured guidance from a large language model (GPT-4o). Prediction failures from a baseline student model are analyzed by the teacher to classify error types, assign difficulty scores, and generate targeted remediations, including sentence rewrites and suggestions for KG-based enrichment. These enriched annotations are used to train a first student model via instruction tuning. This model then annotates a broader dataset with difficulty scores and remediation-enhanced inputs. A second student is subsequently trained via curriculum learning on this dataset, ordered by difficulty, to promote robust and progressive learning. We also construct a heterogeneous biomedical knowledge graph from PubMed abstracts to support context-aware RC. Our approach achieves new state-of-the-art performance on 4 of 5 PPI datasets and the DDI dataset, while remaining competitive on ChemProt.
• 摘要：生物医学文本中的关系分类（RC）对于构建知识图和启用应用程序（例如药物重新利用和临床决策）至关重要。我们提出了一个错误的老师 - 学生框架，该框架通过大型语言模型（GPT-4O）的结构化指导来改善RC。教师分析了基线学生模型的预测失败，以对错误类型进行分类，分配难度分数并生成有针对性的补救措施，包括句子重写和基于KG的富集建议。这些丰富的注释用于通过教学调整来训练第一个学生模型。然后，该模型以难度分数和补救增强的输入来注释一个更广泛的数据集。随后，在该数据集上通过课程学习对第二个学生进行了培训，该数据集是通过困难订购的，以促进健壮而渐进的学习。我们还从PubMed摘要中构建了一个异质的生物医学知识图，以支持上下文感知RC。我们的方法在5个PPI数据集中的4个和DDI数据集中实现了新的最新性能，同时保持在Chemprot上的竞争力。

Title: X-Intelligence 3.0: Training and Evaluating Reasoning LLM for Semiconductor Display

• Authors: Xiaolin Yan, Yangxing Liu, Jiazhang Zheng, Chi Liu, Mingyu Du, Caisheng Chen, Haoyang Liu, Ming Ding, Yuan Li, Qiuping Liao, Linfeng Li, Zhili Mei, Siyu Wan, Li Li, Ruyi Zhong, Jiangling Yu, Xule Liu, Huihui Hu, Jiameng Yue, Ruohui Cheng, Qi Yang, Liangqing Wu, Ke Zhu, Chi Zhang, Chufei Jing, Yifan Zhou, Yan Liang, Dongdong Li, Zhaohui Wang, Bin Zhao, Mingzhou Wu, Mingzhong Zhou, Peng Du, Zuomin Liao, Chao Dai, Pengfei Liang, Xiaoguang Zhu, Yu Zhang, Yu Gu, Kun Pan, Yuan Wu, Yanqing Guan, Shaojing Wu, Zikang Feng, Xianze Ma, Peishan Cheng, Wenjuan Jiang, Jing Ba, Huihao Yu, Zeping Hu, Yuan Xu, Zhiwei Liu, He Wang, Zhenguo Lin, Ming Liu, Yanhong Meng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14430
• Pdf URL: https://arxiv.org/pdf/2507.14430
• Copy Paste: [[2507.14430]] X-Intelligence 3.0: Training and Evaluating Reasoning LLM for Semiconductor Display(https://arxiv.org/abs/2507.14430)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large language models (LLMs) have recently achieved significant advances in reasoning and demonstrated their advantages in solving challenging problems. Yet, their effectiveness in the semiconductor display industry remains limited due to a lack of domain-specific training and expertise. To bridge this gap, we present X-Intelligence 3.0, the first high-performance reasoning model specifically developed for the semiconductor display industry. This model is designed to deliver expert-level understanding and reasoning for the industry's complex challenges. Leveraging a carefully curated industry knowledge base, the model undergoes supervised fine-tuning and reinforcement learning to enhance its reasoning and comprehension capabilities. To further accelerate development, we implemented an automated evaluation framework that simulates expert-level assessments. We also integrated a domain-specific retrieval-augmented generation (RAG) mechanism, resulting in notable performance gains on benchmark datasets. Despite its relatively compact size of 32 billion parameters, X-Intelligence 3.0 outperforms SOTA DeepSeek-R1-671B across multiple evaluations. This demonstrates its exceptional efficiency and establishes it as a powerful solution to the longstanding reasoning challenges faced by the semiconductor display industry.
• 摘要：大型语言模型（LLM）最近在推理方面取得了重大进展，并证明了它们在解决具有挑战性的问题方面的优势。然而，由于缺乏特定领域的培训和专业知识，它们在半导体显示业中的有效性仍然有限。为了弥合这一差距，我们提出了X-Intelligence 3.0，这是专门为半导体显示行业开发的第一个高性能推理模型。该模型旨在为行业的复杂挑战提供专家级别的理解和推理。该模型利用经过精心策划的行业知识基础，经过监督的微调和强化学习，以增强其推理和理解能力。为了进一步加速发展，我们实施了一个自动评估框架，以模拟专家级评估。我们还集成了特定领域的检索生成（RAG）机制，从而在基准数据集上取得了显着的性能提高。尽管其相对紧凑的大小为320亿个参数，但X-Intelligence 3.0在多个评估中都优于Sota DeepSeek-R1-671B。这证明了其出色的效率，并将其确立为对半导体显示业带来的长期推理挑战的有力解决方案。

Title: Explainable Collaborative Problem Solving Diagnosis with BERT using SHAP and its Implications for Teacher Adoption

• Authors: Kester Wong, Sahan Bulathwela, Mutlu Cukurova
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14584
• Pdf URL: https://arxiv.org/pdf/2507.14584
• Copy Paste: [[2507.14584]] Explainable Collaborative Problem Solving Diagnosis with BERT using SHAP and its Implications for Teacher Adoption(https://arxiv.org/abs/2507.14584)
• Keywords: llm
• Abstract: The use of Bidirectional Encoder Representations from Transformers (BERT) model and its variants for classifying collaborative problem solving (CPS) has been extensively explored within the AI in Education community. However, limited attention has been given to understanding how individual tokenised words in the dataset contribute to the model's classification decisions. Enhancing the explainability of BERT-based CPS diagnostics is essential to better inform end users such as teachers, thereby fostering greater trust and facilitating wider adoption in education. This study undertook a preliminary step towards model transparency and explainability by using SHapley Additive exPlanations (SHAP) to examine how different tokenised words in transcription data contributed to a BERT model's classification of CPS processes. The findings suggested that well-performing classifications did not necessarily equate to a reasonable explanation for the classification decisions. Particular tokenised words were used frequently to affect classifications. The analysis also identified a spurious word, which contributed positively to the classification but was not semantically meaningful to the class. While such model transparency is unlikely to be useful to an end user to improve their practice, it can help them not to overrely on LLM diagnostics and ignore their human expertise. We conclude the workshop paper by noting that the extent to which the model appropriately uses the tokens for its classification is associated with the number of classes involved. It calls for an investigation into the exploration of ensemble model architectures and the involvement of human-AI complementarity for CPS diagnosis, since considerable human reasoning is still required for fine-grained discrimination of CPS subskills.
• 摘要：在教育社区的AI中广泛探索了Transformers（BERT）模型（BERT）模型（BERT）模型（BERT）模型（BERT）模型（BERT）模型（CPS）的使用的双向编码器表示的使用。但是，对了解数据集中的单个标记单词如何促进该模型的分类决策的关注有限。增强基于BERT的CPS诊断的解释性对于更好地告知最终用户（例如教师）至关重要，从而促进了更大的信任并促进教育中更广泛的采用。这项研究通过使用Shapley添加说明（SHAP）来研究转录数据中的不同标记单词如何有助于BERT模型对CPS过程的分类，从而迈出了朝模型透明度和解释性的初步步骤。研究结果表明，良好的分类不一定等同于对分类决策的合理解释。特定的令牌单词经常用于影响分类。该分析还确定了一个虚假的词，该单词对分类产生了积极的贡献，但对班级没有意义。尽管这种模型透明度不太可能对最终用户改善其实践有用，但它可以帮助他们不要过度私有LLM诊断和忽略其人类专业知识。我们通过指出模型适当使用令牌进行分类的程度与涉及的类数相关联，结束了研讨会论文。它要求对探索集合模型架构的探索以及人为互补性的CPS诊断的参与，因为仍然需要大量的人类推理来对CPS亚基进行细粒度的歧视。

Title: Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification

• Authors: Łukasz Radliński, Mateusz Guściora, Jan Kocoń
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14590
• Pdf URL: https://arxiv.org/pdf/2507.14590
• Copy Paste: [[2507.14590]] Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification(https://arxiv.org/abs/2507.14590)
• Keywords: language model, gpt, llm, chat
• Abstract: Numerous domain-specific machine learning tasks struggle with data scarcity and class imbalance. This paper systematically explores data augmentation methods for NLP, particularly through large language models like GPT. The purpose of this paper is to examine and evaluate whether traditional methods such as paraphrasing and backtranslation can leverage a new generation of models to achieve comparable performance to purely generative methods. Methods aimed at solving the problem of data scarcity and utilizing ChatGPT were chosen, as well as an exemplary dataset. We conducted a series of experiments comparing four different approaches to data augmentation in multiple experimental setups. We then evaluated the results both in terms of the quality of generated data and its impact on classification performance. The key findings indicate that backtranslation and paraphrasing can yield comparable or even better results than zero and a few-shot generation of examples.
• 摘要：许多特定领域的机器学习任务在数据稀缺和阶级失衡方面遇到了困难。本文系统地探讨了NLP的数据增强方法，尤其是通过GPT等大型语言模型。本文的目的是检查和评估诸如释义和倒退之类的传统方法是否可以利用新一代模型来实现与纯粹生成方法相当的性能。选择了旨在解决数据稀缺和利用chatgpt问题的方法，以及一个模范数据集。我们进行了一系列实验，比较了多种实验设置中的四种不同方法来扩展数据。然后，我们根据生成数据的质量及其对分类性能的影响评估了结果。关键发现表明，倒流和释义可以比零和几次示例产生可比甚至更好的结果。

Title: Retrieval-Augmented Clinical Benchmarking for Contextual Model Testing in Kenyan Primary Care: A Methodology Paper

• Authors: Fred Mutisya (1,2), Shikoh Gitau (1), Christine Syovata (2), Diana Oigara (2), Ibrahim Matende (2), Muna Aden (2), Munira Ali (2), Ryan Nyotu (2), Diana Marion (2), Job Nyangena (2), Nasubo Ongoma (1), Keith Mbae (1), Elizabeth Wamicha (1), Eric Mibuari (1), Jean Philbert Nsengemana (3), Talkmore Chidede (4) ((1) Qhala, Nairobi, Kenya, (2) Kenya Medical Association, Nairobi, Kenya, (3) Africa CDC, (4) AfCFTA)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14615
• Pdf URL: https://arxiv.org/pdf/2507.14615
• Copy Paste: [[2507.14615]] Retrieval-Augmented Clinical Benchmarking for Contextual Model Testing in Kenyan Primary Care: A Methodology Paper(https://arxiv.org/abs/2507.14615)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: Large Language Models(LLMs) hold promise for improving healthcare access in low-resource settings, but their effectiveness in African primary care remains underexplored. We present a methodology for creating a benchmark dataset and evaluation framework focused on Kenyan Level 2 and 3 clinical care. Our approach uses retrieval augmented generation (RAG) to ground clinical questions in Kenya's national guidelines, ensuring alignment with local standards. These guidelines were digitized, chunked, and indexed for semantic retrieval. Gemini Flash 2.0 Lite was then prompted with guideline excerpts to generate realistic clinical scenarios, multiple-choice questions, and rationale based answers in English and Swahili. Kenyan physicians co-created and refined the dataset, and a blinded expert review process ensured clinical accuracy, clarity, and cultural appropriateness. The resulting Alama Health QA dataset includes thousands of regulator-aligned question answer pairs across common outpatient conditions. Beyond accuracy, we introduce evaluation metrics that test clinical reasoning, safety, and adaptability such as rare case detection (Needle in the Haystack), stepwise logic (Decision Points), and contextual adaptability. Initial results reveal significant performance gaps when LLMs are applied to localized scenarios, consistent with findings that LLM accuracy is lower on African medical content than on US-based benchmarks. This work offers a replicable model for guideline-driven, dynamic benchmarking to support safe AI deployment in African health systems.
• 摘要：大型语言模型（LLMS）具有改善低资源环境中医疗保健访问的希望，但它们在非洲初级保健方面的有效性仍然没有得到充实。我们提出了一种创建基准数据集和评估框架的方法，该框架集中在肯尼亚2级和3级临床护理上。我们的方法使用检索增强发电（RAG）在肯尼亚的国家准则中解决临床问题，以确保与当地标准保持一致。这些准则被数字化，分块和索引以进行语义检索。然后提示Gemini Flash 2.0 Lite提供指南摘录，以产生现实的临床场景，多项选择问题以及基于英语和斯瓦希里语的基于理由的答案。肯尼亚医生共同创建并完善了数据集，而盲目的专家审查过程确保了临床准确性，清晰度和文化适当性。由此产生的Alama Health QA数据集包括在常见门诊条件下成千上万的监管机构的问题答案对。除了准确性之外，我们还介绍了测试临床推理，安全性和适应性的评估指标，例如稀有病例检测（Haystack中的针头），逐步逻辑（决策点）和上下文适应性。最初的结果表明，当LLMS应用于本地化方案时，具有明显的性能差距，这与发现非洲医学含量的LLM准确性低于基于美国的基准测试的发现。这项工作为指导驱动的动态基准测试提供了可复制的模型，以支持非洲卫生系统中的安全AI部署。

Title: Linear Relational Decoding of Morphology in Language Models

• Authors: Eric Xia, Jugal Kalita
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14640
• Pdf URL: https://arxiv.org/pdf/2507.14640
• Copy Paste: [[2507.14640]] Linear Relational Decoding of Morphology in Language Models(https://arxiv.org/abs/2507.14640)
• Keywords: language model
• Abstract: A two-part affine approximation has been found to be a good approximation for transformer computations over certain subject object relations. Adapting the Bigger Analogy Test Set, we show that the linear transformation Ws, where s is a middle layer representation of a subject token and W is derived from model derivatives, is also able to accurately reproduce final object states for many relations. This linear technique is able to achieve 90% faithfulness on morphological relations, and we show similar findings multi-lingually and across models. Our findings indicate that some conceptual relationships in language models, such as morphology, are readily interpretable from latent space, and are sparsely encoded by cross-layer linear transformations.
• 摘要：已经发现，两部分的仿射近似是某些主题对象关系上变压器计算的良好近似值。调整较大的类比测验集，我们表明线性转换WS是主体令牌和W的中间层表示，并从模型衍生物中得出，也能够准确地重现许多关系的最终对象状态。这种线性技术能够在形态关系上实现90％的忠诚，我们在多种模型和模型上都显示出相似的发现。我们的发现表明，语言模型中的某些概念关系，例如形态学，可以从潜在空间中解释，并且通过跨层线性转换稀疏地编码。

Title: Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs

• Authors: Minsuh Joo, Hyunsoo Cho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.14649
• Pdf URL: https://arxiv.org/pdf/2507.14649
• Copy Paste: [[2507.14649]] Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs(https://arxiv.org/abs/2507.14649)
• Keywords: language model, llm, hallucination
• Abstract: Despite the outstanding performance of large language models (LLMs) across various NLP tasks, hallucinations in LLMs--where LLMs generate inaccurate responses--remains as a critical problem as it can be directly connected to a crisis of building safe and reliable LLMs. Uncertainty estimation is primarily used to measure hallucination levels in LLM responses so that correct and incorrect answers can be distinguished clearly. This study proposes an effective uncertainty estimation approach, \textbf{Cl}ust\textbf{e}ring-based sem\textbf{an}tic con\textbf{s}ist\textbf{e}ncy (\textbf{Cleanse}). Cleanse quantifies the uncertainty with the proportion of the intra-cluster consistency in the total consistency between LLM hidden embeddings which contain adequate semantic information of generations, by employing clustering. The effectiveness of Cleanse for detecting hallucination is validated using four off-the-shelf models, LLaMA-7B, LLaMA-13B, LLaMA2-7B and Mistral-7B and two question-answering benchmarks, SQuAD and CoQA.
• 摘要：尽管大型语言模型（LLM）在各种NLP任务中的表现出色，但LLMS中的幻觉会产生不准确的响应，这是一个关键问题，因为它可以直接与建立安全可靠的LLM的危机有关。不确定性估计主要用于测量LLM响应中的幻觉水平，以便可以清楚地区分正确和不正确的答案。这项研究提出了一种有效的不确定性估计方法，\ textbf {cl} ust \ textbf {e}基于环的sem \ textbf {an} tic con \ textbf {s} ist \ textbf {e} ncy（\ textbf {calleSe}）。 Cleanse用使用聚类通过聚类来量化了LLM隐藏嵌入的LLM隐藏嵌入的总嵌入中的总一致性的比例来量化不确定性。清洁用于检测幻觉的有效性是使用四种现成的模型，即Llama-7b，Llama-13b，Llama2-7B和Mismtral-7B，以及两个提问的基准，Squasad和CoQa。

Title: Mangosteen: An Open Thai Corpus for Language Model Pretraining

• Authors: Wannaphong Phatthiyaphaibun, Can Udomcharoenchaikit, Pakpoom Singkorapoom, Kunat Pipatanakul, Ekapol Chuangsuwanich, Peerat Limkonchotiwat, Sarana Nutanong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14664
• Pdf URL: https://arxiv.org/pdf/2507.14664
• Copy Paste: [[2507.14664]] Mangosteen: An Open Thai Corpus for Language Model Pretraining(https://arxiv.org/abs/2507.14664)
• Keywords: language model, gpt, llm
• Abstract: Pre-training data shapes a language model's quality, but raw web text is noisy and demands careful cleaning. Existing large-scale corpora rely on English-centric or language-agnostic pipelines whose heuristics do not capture Thai script or cultural nuances, leaving risky material such as gambling content untreated. Prior Thai-specific efforts customize pipelines or build new ones, yet seldom release their data or document design choices, hindering reproducibility and raising the question of how to construct a transparent, high-quality Thai corpus. We introduce Mangosteen: a 47 billion-token Thai corpus built through a Thai-adapted Dolma pipeline that includes custom rule-based language ID, revised C4/Gopher quality filters, and Thai-trained content filters, plus curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books, and CC-licensed YouTube subtitles. Systematic ablations using GPT-2 show the pipeline trims CommonCrawl from 202M to 25M documents while raising SEA-HELM NLG from 3 to 11; an 8B-parameter SEA-LION model continually pre-trained on Mangosteen then surpasses SEA-LION-v3 and Llama-3.1 by about four points on Thai benchmarks. We release the full pipeline code, cleaning manifests, corpus snapshot, and all checkpoints, providing a fully reproducible foundation for future Thai and regional LLM research.
• 摘要：预训练数据塑造了语言模型的质量，但是原始的Web文本很嘈杂，需要仔细清洁。现有的大规模语料库依赖于以英语为中心或语言的管道，其启发式方法不会捕获泰语剧本或文化差异，而留下了诸如赌博内容之类的风险材料。以前的泰国特定努力可以自定义管道或建造新的管道，但很少释放其数据或文档设计选择，阻碍重现性并提出如何构建透明，高质量的泰语语料库的问题。 We introduce Mangosteen: a 47 billion-token Thai corpus built through a Thai-adapted Dolma pipeline that includes custom rule-based language ID, revised C4/Gopher quality filters, and Thai-trained content filters, plus curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books, and CC-licensed YouTube subtitles.使用GPT-2的系统消融显示管道修剪从202m到25m的文档，同时将海螺旋NLG从3增加到11；在Mangosteen上不断预先培训的8B参数海狮模型，然后在泰国基准测试中超过了Sea-Lion-V3和Llama-3.1。我们发布完整的管道代码，清洁清单，语料库快照和所有检查站，为未来的泰国和地区LLM研究提供了完全可重现的基础。

Title: Large Language Models as Medical Codes Selectors: a benchmark using the International Classification of Primary Care

• Authors: Vinicius Anjos de Almeida, Vinicius de Camargo, Raquel Gómez-Bravo, Egbert van der Haring, Kees van Boven, Marcelo Finger, Luis Fernandez Lopez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14681
• Pdf URL: https://arxiv.org/pdf/2507.14681
• Copy Paste: [[2507.14681]] Large Language Models as Medical Codes Selectors: a benchmark using the International Classification of Primary Care(https://arxiv.org/abs/2507.14681)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: Background: Medical coding structures healthcare data for research, quality monitoring, and policy. This study assesses the potential of large language models (LLMs) to assign ICPC-2 codes using the output of a domain-specific search engine. Methods: A dataset of 437 Brazilian Portuguese clinical expressions, each annotated with ICPC-2 codes, was used. A semantic search engine (OpenAI's text-embedding-3-large) retrieved candidates from 73,563 labeled concepts. Thirty-three LLMs were prompted with each query and retrieved results to select the best-matching ICPC-2 code. Performance was evaluated using F1-score, along with token usage, cost, response time, and format adherence. Results: Twenty-eight models achieved F1-score > 0.8; ten exceeded 0.85. Top performers included gpt-4.5-preview, o3, and gemini-2.5-pro. Retriever optimization can improve performance by up to 4 points. Most models returned valid codes in the expected format, with reduced hallucinations. Smaller models (<3B) struggled with formatting and input length. Conclusions: LLMs show strong potential for automating ICPC-2 coding, even without fine-tuning. This work offers a benchmark and highlights challenges, but findings are limited by dataset scope and setup. Broader, multilingual, end-to-end evaluations are needed for clinical validation.
• 摘要：背景：医学编码结构可用于研究，质量监测和政策的医疗数据。这项研究评估了大语模型（LLMS）使用特定领域搜索引擎的输出分配ICPC-2代码的潜力。方法：使用了437个巴西葡萄牙临床表达式的数据集，每个表达式都使用ICPC-2代码注释。语义搜索引擎（OpenAI的文本插入3大）从73,563个标记的概念中检索了候选人。每个查询都提示了33个LLM，并检索结果以选择最佳匹配的ICPC-2代码。使用F1得分以及令牌使用，成本，响应时间和格式依从性评估性能。结果：28个模型达到了F1得分> 0.8；十个超过0.85。表现最好的人包括GPT-4.5-preiview，O3和Gemini-2.5-Pro。检索器优化可以提高性能多达4分。大多数模型以预期格式返回有效的代码，并减少了幻觉。较小的型号（<3b）在格式和输入长度方面挣扎。结论：即使没有微调，LLMS也显示出强大的ICPC-2编码自动化潜力。这项工作提供了一个基准，并突出了挑战，但发现受数据集范围和设置的限制。临床验证需要更广泛的多语言，端到端评估。

Title: MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization

• Authors: Xingxuan Li, Yao Xiao, Dianwen Ng, Hai Ye, Yue Deng, Xiang Lin, Bin Wang, Zhanfeng Mo, Chong Zhang, Yueyi Zhang, Zonglin Yang, Ruilin Li, Lei Lei, Shihao Xu, Han Zhao, Weiling Chen, Feng Ji, Lidong Bing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14683
• Pdf URL: https://arxiv.org/pdf/2507.14683
• Copy Paste: [[2507.14683]] MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization(https://arxiv.org/abs/2507.14683)
• Keywords: language model, gpt
• Abstract: Large language models have recently evolved from fluent text generation to advanced reasoning across diverse domains, giving rise to reasoning language models. Among these domains, mathematical reasoning serves as a representative benchmark as it requires precise multi-step logic and abstract reasoning, which can be generalized to other tasks. While closed-source RLMs such as GPT-o3 demonstrate impressive reasoning capabilities, their proprietary nature limits transparency and reproducibility. Although many open-source projects aim to close this gap, most of them lack sufficient openness by omitting critical resources such as datasets and detailed training configurations, which hinders reproducibility. To contribute toward greater transparency in RLM development, we introduce the MiroMind-M1 series, a set of fully open-source RLMs built on the Qwen-2.5 backbone that match or exceed the performance of existing open-source RLMs. Specifically, our models are trained in two stages: SFT on a carefully curated corpus of 719K math-reasoning problems with verified CoT trajectories, followed by RLVR on 62K challenging and verifiable problems. To enhance the robustness and efficiency of the RLVR process, we introduce Context-Aware Multi-Stage Policy Optimization, an algorithm that integrates length-progressive training with an adaptive repetition penalty to encourage context-aware RL training. Our model achieves state-of-the-art or competitive performance and superior token efficiency among Qwen-2.5-based open-source 7B and 32B models on the AIME24, AIME25, and MATH benchmarks. To facilitate reproducibility, we release the complete stack: models (MiroMind-M1-SFT-7B, MiroMind-M1-RL-7B, MiroMind-M1-RL-32B); datasets (MiroMind-M1-SFT-719K, MiroMind-M1-RL-62K); and all training and evaluation configurations. We hope these resources will support further research and foster community advancement.
• 摘要：大型语言模型最近从流利的文本生成发展到跨不同领域的高级推理，从而产生了推理语言模型。在这些领域中，数学推理是代表性的基准，因为它需要精确的多步逻辑和抽象推理，可以将其推广到其他任务。诸如GPT-O3之类的封闭源RLM表现出令人印象深刻的推理能力，但它们的专有性质限制了透明度和可重复性。尽管许多开源项目旨在缩小这一差距，但大多数开源项目通过省略关键资源（例如数据集和详细的培训配置）缺乏足够的开放性，这阻碍了可重复性。为了提高RLM开发的透明度，我们介绍了MiroMind-M1系列，这是一组完全开源的RLMS，建立在QWEN-2.5骨架上，匹配或超过现有开源RLMS的性能。具体而言，我们的模型分为两个阶段进行培训：SFT在经过验证的COT轨迹的精心策划的719K数学问题的语料库中，其次是RLVR，在62k的挑战性和可验证的问题上进行了RLVR。为了提高RLVR流程的鲁棒性和效率，我们介绍了上下文感知的多阶段策略优化，该算法将长度增强培训与自适应重复惩罚相结合，以鼓励上下文意识到RL培训。我们的模型在AIME24，AIME25和MATH基准的基于QWEN-2.5的开源7B和32B模型中实现了最先进的或竞争性的性能以及卓越的令牌效率。为了促进可重复性，我们发布了完整的堆栈：模型（MiroMind-M1-SFT-7B，MiroMind-M1-RL-7B，MiroMind-M1-M1-RL-32B）；数据集（Miomind-M1-SFT-719K，MiroMind-M1-RL-62K）;以及所有培训和评估配置。我们希望这些资源将支持进一步的研究并促进社区的发展。

Title: Mind the Gap: A Review of Arabic Post-Training Datasets and Their Limitations

• Authors: Mohammed Alkhowaiter, Norah Alshahrani, Saied Alshahrani, Reem I. Masoud, Alaa Alzahrani, Deema Alnuhait, Emad A. Alghamdi, Khalid Almubarak
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.14688
• Pdf URL: https://arxiv.org/pdf/2507.14688
• Copy Paste: [[2507.14688]] Mind the Gap: A Review of Arabic Post-Training Datasets and Their Limitations(https://arxiv.org/abs/2507.14688)
• Keywords: language model, llm, prompt
• Abstract: Post-training has emerged as a crucial technique for aligning pre-trained Large Language Models (LLMs) with human instructions, significantly enhancing their performance across a wide range of tasks. Central to this process is the quality and diversity of post-training datasets. This paper presents a review of publicly available Arabic post-training datasets on the Hugging Face Hub, organized along four key dimensions: (1) LLM Capabilities (e.g., Question Answering, Translation, Reasoning, Summarization, Dialogue, Code Generation, and Function Calling); (2) Steerability (e.g., persona and system prompts); (3) Alignment (e.g., cultural, safety, ethics, and fairness), and (4) Robustness. Each dataset is rigorously evaluated based on popularity, practical adoption, recency and maintenance, documentation and annotation quality, licensing transparency, and scientific contribution. Our review revealed critical gaps in the development of Arabic post-training datasets, including limited task diversity, inconsistent or missing documentation and annotation, and low adoption across the community. Finally, the paper discusses the implications of these gaps on the progress of Arabic LLMs and applications while providing concrete recommendations for future efforts in post-training dataset development.
• 摘要：培训后已成为将预先训练的大语言模型（LLM）与人类指示保持一致的关键技术，从而大大提高了它们在各种任务中的表现。此过程的核心是培训后数据集的质量和多样性。本文介绍了沿着四个关键维度组织的拥抱面枢纽上公开可用的阿拉伯训练数据集的评论：（1）LLM功能（例如，问题答案，翻译，推理，推理，摘要，对话，代码，代码生成和功能调用）； （2）可管道性（例如，角色和系统提示）； （3）对齐（例如文化，安全，伦理和公平）和（4）鲁棒性。每个数据集都是根据受欢迎程度，实践采用，新近维护，文档和注释质量，许可透明度和科学贡献的严格评估。我们的审查揭示了阿拉伯语后培训数据集开发的关键差距，包括有限的任务多样性，不一致或缺失的文档和注释以及整个社区的采用率低。最后，本文讨论了这些差距对阿拉伯语LLM和应用程序进步的含义，同时为未来的培训数据集开发提供了具体建议。

Title: Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation

• Authors: Amina Dzafic, Merve Kavut, Ulya Bayram
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.14693
• Pdf URL: https://arxiv.org/pdf/2507.14693
• Copy Paste: [[2507.14693]] Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation(https://arxiv.org/abs/2507.14693)
• Keywords: language model, llm
• Abstract: Suicidal ideation detection is critical for real-time suicide prevention, yet its progress faces two under-explored challenges: limited language coverage and unreliable annotation practices. Most available datasets are in English, but even among these, high-quality, human-annotated data remains scarce. As a result, many studies rely on available pre-labeled datasets without examining their annotation process or label reliability. The lack of datasets in other languages further limits the global realization of suicide prevention via artificial intelligence (AI). In this study, we address one of these gaps by constructing a novel Turkish suicidal ideation corpus derived from social media posts and introducing a resource-efficient annotation framework involving three human annotators and two large language models (LLMs). We then address the remaining gaps by performing a bidirectional evaluation of label reliability and model consistency across this dataset and three popular English suicidal ideation detection datasets, using transfer learning through eight pre-trained sentiment and emotion classifiers. These transformers help assess annotation consistency and benchmark model performance against manually labeled data. Our findings underscore the need for more rigorous, language-inclusive approaches to annotation and evaluation in mental health natural language processing (NLP) while demonstrating the questionable performance of popular models with zero-shot transfer learning. We advocate for transparency in model training and dataset construction in mental health NLP, prioritizing data and model reliability.
• 摘要：自杀意念检测对于预防实时自杀至关重要，但其进步面临两个未经探索的挑战：有限的语言覆盖范围和不可靠的注释实践。大多数可用的数据集使用英语，但即使在这些高质量，人类注释的数据中，甚至仍然很少。结果，许多研究依靠可用的预标签数据集，而无需检查其注释过程或标签可靠性。缺乏其他语言的数据集进一步限制了通过人工智能（AI）对自杀预防的全球实现。在这项研究中，我们通过构建一种新型的土耳其自杀构想语料库来解决这些差距之一，并从社交媒体帖子中衍生出来，并引入了涉及三个人类注释者和两个大语言模型（LLM）的资源有效注释框架。然后，我们通过通过八个预先训练的情感和情感分类器进行转移学习，通过对本数据集的标签可靠性和模型一致性进行双向评估来解决剩余的差距。这些变压器有助于评估注释一致性和基准模型性能，以针对手动标记的数据进行评估。我们的发现强调了对精神健康自然语言处理（NLP）中更严格，包括语言的注释和评估的必要性，同时证明了通过零摄像转移学习的流行模型的可疑性能。我们倡导在心理健康NLP中建立模型培训和数据集构建的透明度，优先考虑数据和模型可靠性。

Title: GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization

• Authors: Luyi Ma, Wanjia Zhang, Kai Zhao, Abhishek Kulkarni, Lalitesh Morishetti, Anjana Ganesh, Ashish Ranjan, Aashika Padmanabhan, Jianpeng Xu, Jason Cho, Praveen Kanumala, Kaushiki Nag, Sumit Dutta, Kamiya Motwani, Malay Patel, Evren Korpeoglu, Sushant Kumar, Kannan Achan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2507.14758
• Pdf URL: https://arxiv.org/pdf/2507.14758
• Copy Paste: [[2507.14758]] GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization(https://arxiv.org/abs/2507.14758)
• Keywords: chain-of-thought
• Abstract: Generative models have recently demonstrated strong potential in multi-behavior recommendation systems, leveraging the expressive power of transformers and tokenization to generate personalized item sequences. However, their adoption is hindered by (1) the lack of explicit information for token reasoning, (2) high computational costs due to quadratic attention complexity and dense sequence representations after tokenization, and (3) limited multi-scale modeling over user history. In this work, we propose GRACE (Generative Recommendation via journey-aware sparse Attention on Chain-of-thought tokEnization), a novel generative framework for multi-behavior sequential recommendation. GRACE introduces a hybrid Chain-of-Thought (CoT) tokenization method that encodes user-item interactions with explicit attributes from product knowledge graphs (e.g., category, brand, price) over semantic tokenization, enabling interpretable and behavior-aligned generation. To address the inefficiency of standard attention, we design a Journey-Aware Sparse Attention (JSA) mechanism, which selectively attends to compressed, intra-, inter-, and current-context segments in the tokenized sequence. Experiments on two real-world datasets show that GRACE significantly outperforms state-of-the-art baselines, achieving up to +106.9% HR@10 and +106.7% NDCG@10 improvement over the state-of-the-art baseline on the Home domain, and +22.1% HR@10 on the Electronics domain. GRACE also reduces attention computation by up to 48% with long sequences.
• 摘要：最近，生成模型在多行为推荐系统中表现出强大的潜力，利用变压器和令牌化的表达能力来生成个性化的项目序列。但是，（1）缺乏代币推理的明确信息，（2）由于二次注意复杂性和象征化后的密集序列表示，以及（3）多尺度建模比用户历史记录有限。在这项工作中，我们提出了宽限期（通过对思想链令牌化的稀疏关注而产生的建议），这是一个新颖的生成框架，用于多行为顺序推荐。格蕾丝（Grace）引入了一种混合思想链（COT）令牌化方法，该方法在语义令牌上编码了来自产品知识图（例如，类别，品牌，价格）的明确属性的用户项目相互作用，从而实现了可解释的和与行为一致的生成。为了解决标准注意力的效率低下，我们设计了一种稀疏关注（JSA）机制，该机制有选择地参加以标记序列的压缩，内部，间和电流范围的段。两个现实世界数据集的实验表明，Grace的表现明显优于最先进的基线，可达到 +106.9％HR@10和 +106.7％NDCG@10的改进，比家居领域的最先进的基线改进，以及 +22.1％的HR@10@10.1％HR@@Electonics Domain domain domain domain。 GRACE还可以通过长序列将注意力计算降低多达48％。

Title: FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing

• Authors: Shoutao Guo, Shaolei Zhang, Qingkai Fang, Zhengrui Ma, Min Zhang, Yang Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14815
• Pdf URL: https://arxiv.org/pdf/2507.14815
• Copy Paste: [[2507.14815]] FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing(https://arxiv.org/abs/2507.14815)
• Keywords: language model, llm
• Abstract: The rapid advancement of Large Language Models (LLMs) has spurred significant progress in Large Speech-Language Models (LSLMs), enhancing their capabilities in both speech understanding and generation. While existing LSLMs often concentrate on augmenting speech generation or tackling a diverse array of short-speech tasks, the efficient processing of long-form speech remains a critical yet underexplored challenge. This gap is primarily attributed to the scarcity of long-speech training datasets and the high computational costs associated with long sequences. To address these limitations, we introduce FastLongSpeech, a novel framework designed to extend LSLM capabilities for efficient long-speech processing without necessitating dedicated long-speech training data. FastLongSpeech incorporates an iterative fusion strategy that can compress excessively long-speech sequences into manageable lengths. To adapt LSLMs for long-speech inputs, it introduces a dynamic compression training approach, which exposes the model to short-speech sequences at varying compression ratios, thereby transferring the capabilities of LSLMs to long-speech tasks. To assess the long-speech capabilities of LSLMs, we develop a long-speech understanding benchmark called LongSpeech-Eval. Experiments show that our method exhibits strong performance in both long-speech and short-speech tasks, while greatly improving inference efficiency.
• 摘要：大型语言模型（LLM）的快速发展促使大型语音模型（LSLMS）促进了重大进展，从而增强了其在语音理解和产生中的能力。尽管现有的LSLM通常集中于增强语音产生或处理各种言论多样的任务，但长期演讲的有效处理仍然是一个至关重要的挑战。该差距主要归因于长音训练数据集的稀缺性以及与长序列相关的高计算成本。为了解决这些局限性，我们引入了FastlongSpeech，这是一个新颖的框架，旨在扩展LSLM功能，以实现有效的长音处理，而无需专用的长音训练数据。 FastLongSpeech结合了一种迭代融合策略，可以将过多的语音序列压缩为可管理的长度。为了适应LSLM的长语输入，它引入了动态压缩训练方法，该方法将模型公开了以不同的压缩比的短语序列，从而将LSLMS的功能传递到了长音任务。为了评估LSLM的长语音能力，我们开发了一种长达语音理解的基准，称为Longspeech-eval。实验表明，我们的方法在长音和短语任务中都表现出很强的性能，同时大大提高了推理效率。

Title: Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documents

• Authors: Akriti Jain, Pritika Ramu, Aparna Garimella, Apoorv Saxena
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14819
• Pdf URL: https://arxiv.org/pdf/2507.14819
• Copy Paste: [[2507.14819]] Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documents(https://arxiv.org/abs/2507.14819)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in transforming text descriptions or tables to data visualizations via instruction-tuning methods. However, it is not straightforward to apply these methods directly for a more real-world use case of visualizing data from long documents based on user-given intents, as opposed to the user pre-selecting the relevant content manually. We introduce the task of intent-based chart generation from documents: given a user-specified intent and document(s), the goal is to generate a chart adhering to the intent and grounded on the document(s) in a zero-shot setting. We propose an unsupervised, two-staged framework in which an LLM first extracts relevant information from the document(s) by decomposing the intent and iteratively validates and refines this data. Next, a heuristic-guided module selects an appropriate chart type before final code generation. To assess the data accuracy of the generated charts, we propose an attribution-based metric that uses a structured textual representation of charts, instead of relying on visual decoding metrics that often fail to capture the chart data effectively. To validate our approach, we curate a dataset comprising of 1,242 $<$intent, document, charts$>$ tuples from two domains, finance and scientific, in contrast to the existing datasets that are largely limited to parallel text descriptions/ tables and their corresponding charts. We compare our approach with baselines using single-shot chart generation using LLMs and query-based retrieval methods; our method outperforms by upto $9$ points and $17$ points in terms of chart data accuracy and chart type respectively over the best baselines.
• 摘要：大型语言模型（LLMS）证明了通过指令调整方法将文本描述或表转换为数据可视化的表现很强的能力。但是，直接将这些方法直接应用于基于用户意图的长文档中可视化数据的更真实的用例，而不是用户手动预选相关内容。我们从文档中介绍了基于意图的图表生成的任务：给定用户指定的意图和文档，目标是生成遵守意图的图表，并以零拍设置为基础。我们提出了一个无监督的两期框架，在该框架中，LLM首先通过分解意图并迭代验证和完善这些数据来从文档中提取相关信息。接下来，一个具有启发主义的指导模块在最终代码生成之前选择适当的图表类型。为了评估生成图表的数据准确性，我们提出了一种基于属性的指标，该指标使用图表的结构化文本表示，而不是依靠通常无法有效捕获图表数据的视觉解码指标。为了验证我们的方法，我们策划了一个数据集，其中包括1,242 $ <$ intent，文档，图表$> $ $> $ $> $ $ but cub cub cub cum cub cum cum cut cut can finance和cocientific与现有数据集相反，这些数据集在很大程度上仅限于并行文本描述/表/表及其相应图表。我们使用LLMS和基于查询的检索方法使用单拍图将我们的方法与基线进行比较；在图表数据的准确性和图表类型方面，我们的方法的优于9美元$ 9美元和$ 17 $的积分，分别超过了最佳基线。

Title: Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding

• Authors: Yifei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14849
• Pdf URL: https://arxiv.org/pdf/2507.14849
• Copy Paste: [[2507.14849]] Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding(https://arxiv.org/abs/2507.14849)
• Keywords: language model, retrieval-augmented generation
• Abstract: Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models. However, the impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored. This gap in understanding is particularly significant given the increasing importance of Retrieval-Augmented Generation (RAG) systems, where efficient acquisition and utilization of contextual information are paramount for generating reliable responses. Motivated by the need to understand how the extended long-CoT process influences long-context comprehension, we conduct a comprehensive investigation using a series of open-source models distilled from Deepseek-R1, renowned for its exceptional reasoning capabilities. Our study focuses on evaluating these models' performance in extracting and integrating relevant information from extended contexts through multi-document question and answering tasks. Through rigorous experimentation, we demonstrate that distilled reasoning patterns significantly improve long-context understanding. Our analysis reveals that distillation fosters greater long-context awareness by promoting more detailed and explicit reasoning processes during context analysis and information parsing. This advancement effectively mitigates the persistent "lost in the middle" issue that has hindered long-context models.
• 摘要：推理蒸馏已成为增强较小语言模型的推理能力的有效方法。但是，大规模推理蒸馏对其他关键能力的影响，尤其是在文章的检索和推理方面，仍然没有探索。鉴于检索功能生成（RAG）系统的重要性越来越重要，这种理解的差距尤其显着，在这种情况下，有效地获取和利用上下文信息对于生成可靠的响应至关重要。由于需要了解扩展的长期流程如何影响长篇小说理解的需要，我们使用一系列从DeepSeek-R1蒸馏的开源模型进行了全面的调查，该模型以其出色的推理能力而闻名。我们的研究重点是评估这些模型在通过多文件问题和回答任务中从扩展上下文中提取和整合相关信息时的性能。通过严格的实验，我们证明了蒸馏推理模式可显着改善长期以来的理解。我们的分析表明，蒸馏通过在上下文分析和信息解析过程中促进更详细和明确的推理过程来提高更高的长期意识。这一进步有效地减轻了阻碍长篇小说模型的持续性“中间”问题。

Title: Tiny language models

• Authors: Ronit D. Gross, Yarden Tzach, Tal Halevi, Ella Koresh, Ido Kanter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14871
• Pdf URL: https://arxiv.org/pdf/2507.14871
• Copy Paste: [[2507.14871]] Tiny language models(https://arxiv.org/abs/2507.14871)
• Keywords: language model, llm
• Abstract: A prominent achievement of natural language processing (NLP) is its ability to understand and generate meaningful human language. This capability relies on complex feedforward transformer block architectures pre-trained on large language models (LLMs). However, LLM pre-training is currently feasible only for a few dominant companies due to the immense computational resources required, limiting broader research participation. This creates a critical need for more accessible alternatives. In this study, we explore whether tiny language models (TLMs) exhibit the same key qualitative features of LLMs. We demonstrate that TLMs exhibit a clear performance gap between pre-trained and non-pre-trained models across classification tasks, indicating the effectiveness of pre-training, even at a tiny scale. The performance gap increases with the size of the pre-training dataset and with greater overlap between tokens in the pre-training and classification datasets. Furthermore, the classification accuracy achieved by a pre-trained deep TLM architecture can be replicated through a soft committee of multiple, independently pre-trained shallow architectures, enabling low-latency TLMs without affecting classification accuracy. Our results are based on pre-training BERT-6 and variants of BERT-1 on subsets of the Wikipedia dataset and evaluating their performance on FewRel, AGNews, and DBPedia classification tasks. Future research on TLM is expected to further illuminate the mechanisms underlying NLP, especially given that its biologically inspired models suggest that TLMs may be sufficient for children or adolescents to develop language.
• 摘要：自然语言处理（NLP）的显着成就是其理解和产生有意义的人类语言的能力。该功能依赖于在大型语言模型（LLMS）上预先训练的复杂前馈变压器块体系结构。但是，由于所需的巨大计算资源，LLM预培训目前仅对少数几家主要公司可行，从而限制了更广泛的研究参与。这产生了对更容易获得的替代方案的迫切需求。在这项研究中，我们探讨了微小的语言模型（TLM）是否具有LLMS相同的关键定性特征。我们证明，TLM在分类任务之间在预训练和未经培训的模型之间表现出明显的性能差距，即使在很小的范围内也表明了预训练的有效性。性能差距随着预训练数据集的大小而增加，并且在训练和分类数据集中的代币之间的重叠更大。此外，可以通过多个多个，独立预训练的浅层体系结构的软委员会复制预训练的深度TLM体系结构所达到的分类精度，从而在不影响分类精度的情况下实现了低延迟的TLM。我们的结果基于Wikipedia数据集子集对BERT-1的前训练BERT-6和BERT-1的变体，并评估了其在Lightrel，Agnews和DBPedia分类任务上的性能。预计未来对TLM的研究将进一步阐明NLP的基础机制，尤其是考虑到其生物学启发的模型表明TLM可能足以使儿童或青少年发展语言。

Title: MEKiT: Multi-source Heterogeneous Knowledge Injection Method via Instruction Tuning for Emotion-Cause Pair Extraction

• Authors: Shiyi Mu, Yongkang Liu, Shi Feng, Xiaocui Yang, Daling Wang, Yifei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14887
• Pdf URL: https://arxiv.org/pdf/2507.14887
• Copy Paste: [[2507.14887]] MEKiT: Multi-source Heterogeneous Knowledge Injection Method via Instruction Tuning for Emotion-Cause Pair Extraction(https://arxiv.org/abs/2507.14887)
• Keywords: language model, llm
• Abstract: Although large language models (LLMs) excel in text comprehension and generation, their performance on the Emotion-Cause Pair Extraction (ECPE) task, which requires reasoning ability, is often underperform smaller language model. The main reason is the lack of auxiliary knowledge, which limits LLMs' ability to effectively perceive emotions and reason causes. To address this issue, we propose a novel \textbf{M}ulti-source h\textbf{E}terogeneous \textbf{K}nowledge \textbf{i}njection me\textbf{T}hod, MEKiT, which integrates heterogeneous internal emotional knowledge and external causal knowledge. Specifically, for these two distinct aspects and structures of knowledge, we apply the approaches of incorporating instruction templates and mixing data for instruction-tuning, which respectively facilitate LLMs in more comprehensively identifying emotion and accurately reasoning causes. Experimental results demonstrate that MEKiT provides a more effective and adaptable solution for the ECPE task, exhibiting an absolute performance advantage over compared baselines and dramatically improving the performance of LLMs on the ECPE task.
• 摘要：尽管大型语言模型（LLMS）在文本理解和生成方面都表现出色，但他们在需要推理能力的情感因子对提取（ECPE）任务上的表现通常不足以较小的语言模型。主要原因是缺乏辅助知识，这限制了LLM有效地感知情绪和理性原因的能力。为了解决这个问题，我们提出了一个小说\ textbf {m} ulti-source h \ textbf {e} terogeneous \ textbf {k} nowledge \ textbf {i} nojection me \ textbf {t}具体而言，对于这两个不同的知识方面和结构，我们应用了合并教学模板和混合数据进行指导调整的方法，这些方法分别促进了LLMS在更全面地识别情感和准确的推理原因中。实验结果表明，MEKIT为ECPE任务提供了更有效和适应性的解决方案，比比较基线的绝对性能优势，并显着提高了LLMS在ECPE任务上的性能。

Title: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs

• Authors: Boyi Deng, Yu Wan, Baosong Yang, Fei Huang, Wenjie Wang, Fuli Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14894
• Pdf URL: https://arxiv.org/pdf/2507.14894
• Copy Paste: [[2507.14894]] Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs(https://arxiv.org/abs/2507.14894)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have impressive multilingual capabilities, but they suffer from unexpected code-switching, also known as language mixing, which involves switching to unexpected languages in the model response. This problem leads to poor readability and degrades the usability of model responses. However, existing work on this issue lacks a mechanistic analysis and shows limited effectiveness. In this paper, we first provide an in-depth analysis of unexpected code-switching using sparse autoencoders and find that when LLMs switch to a language, the features of that language exhibit excessive pre-activation values. Based on our findings, we propose $\textbf{S}$parse $\textbf{A}$utoencoder-guided $\textbf{S}$upervised $\textbf{F}$ine$\textbf{t}$uning (SASFT), which teaches LLMs to maintain appropriate pre-activation values of specific language features during training. Experiments on five models across three languages demonstrate that SASFT consistently reduces unexpected code-switching by more than 50\% compared to standard supervised fine-tuning, with complete elimination in four cases. Moreover, SASFT maintains or even improves the models' performance on six multilingual benchmarks, showing its effectiveness in addressing code-switching while preserving multilingual capabilities.
• 摘要：大型语言模型（LLMS）具有令人印象深刻的多语言功能，但它们遭受了意外的代码转换（也称为语言混合）的困扰，其中涉及在模型响应中切换到意外语言。这个问题导致可读性不佳，并降低了模型响应的可用性。但是，在此问题上的现有工作缺乏机械分析，并显示出有限的有效性。在本文中，我们首先对使用稀疏自动编码器进行意外的代码转换进行了深入的分析，并发现当LLMS切换到语言时，该语言的功能表现出过多的预激活值。 Based on our findings, we propose $\textbf{S}$parse $\textbf{A}$utoencoder-guided $\textbf{S}$upervised $\textbf{F}$ine$\textbf{t}$uning (SASFT), which teaches LLMs to maintain appropriate pre-activation values of specific language features during training.与标准监督的微调相比，对三种语言的五种模型进行实验表明，SASFT始终将意外的代码转换降低了50 \％，在四种情况下完全消除了。此外，SASFT维护甚至可以改善模型在六个多语言基准上的性能，从而在保留多语言功能的同时，显示出其在解决代码转换方面的有效性。

Title: From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment

• Authors: Chongxuan Huang, Yongshi Ye, Biao Fu, Qifeng Su, Xiaodong Shi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14900
• Pdf URL: https://arxiv.org/pdf/2507.14900
• Copy Paste: [[2507.14900]] From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment(https://arxiv.org/abs/2507.14900)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable multilingual capabilities, however, how to evaluate cross-lingual alignment remains underexplored. Existing alignment benchmarks primarily focus on sentence embeddings, but prior research has shown that neural models tend to induce a non-smooth representation space, which impact of semantic alignment evaluation on low-resource languages. Inspired by neuroscientific findings that similar information activates overlapping neuronal regions, we propose a novel Neuron State-Based Cross-Lingual Alignment (NeuronXA) to assess the cross-lingual a lignment capabilities of LLMs, which offers a more semantically grounded approach to assess cross-lingual alignment. We evaluate NeuronXA on several prominent multilingual LLMs (LLaMA, Qwen, Mistral, GLM, and OLMo) across two transfer tasks and three multilingual benchmarks. The results demonstrate that with only 100 parallel sentence pairs, NeuronXA achieves a Pearson correlation of 0.9556 with downstream tasks performance and 0.8514 with transferability. These findings demonstrate NeuronXA's effectiveness in assessing both cross-lingual alignment and transferability, even with a small dataset. This highlights its potential to advance cross-lingual alignment research and to improve the semantic understanding of multilingual LLMs.
• 摘要：大型语言模型（LLMS）表现出了显着的多语言能力，但是，如何评估跨语性对准仍然没有被淘汰。现有的对齐基准主要集中在句子嵌入，但先前的研究表明，神经模型倾向于诱导非平滑表示空间，这会影响语义一致性评估对低资源语言的影响。受到神经科学的发现的启发，类似信息激活了重叠的神经元区域，我们提出了一种新型的基于神经元的跨语性跨语言对准（Neuronxa），以评估LLMS的跨语义词性，从而提供了一种更具语义上的接地方法来评估跨语言对象。我们在两个转移任务和三个多语言基准的几个突出的多语言LLM（Llama，Qwen，Mistral，Glm和Olmo）上评估了Neuronxa。结果表明，只有100个平行句子对，Neuronxa就达到了0.9556的Pearson相关性，而下游任务的性能和0.8514具有可传递性。这些发现证明了神经元在评估跨语性对准和可传递性方面的有效性，即使使用较小的数据集也是如此。这凸显了其推进跨语性对准研究并提高对多语言LLM的语义理解的潜力。

Title: PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation

• Authors: Eliya Habba, Noam Dahan, Gili Lior, Gabriel Stanovsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14913
• Pdf URL: https://arxiv.org/pdf/2507.14913
• Copy Paste: [[2507.14913]] PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation(https://arxiv.org/abs/2507.14913)
• Keywords: llm, prompt
• Abstract: Evaluating LLMs with a single prompt has proven unreliable, with small changes leading to significant performance differences. However, generating the prompt variations needed for a more robust multi-prompt evaluation is challenging, limiting its adoption in practice. To address this, we introduce PromptSuite, a framework that enables the automatic generation of various prompts. PromptSuite is flexible - working out of the box on a wide range of tasks and benchmarks. It follows a modular prompt design, allowing controlled perturbations to each component, and is extensible, supporting the addition of new components and perturbation types. Through a series of case studies, we show that PromptSuite provides meaningful variations to support strong evaluation practices. It is available through both a Python API: this https URL, and a user-friendly web interface: this https URL
• 摘要：用单个提示评估LLM已被证明是不可靠的，并且很小的变化导致了显着的性能差异。但是，产生更强大的多项目评估所需的及时变化是具有挑战性的，从而限制了其在实践中的采用。为了解决这个问题，我们介绍提示仪，该提示可以自动生成各种提示。提示是灵活的 - 在各种任务和基准测试中开箱即用。它遵循模块化提示设计，允许对每个组件进行受控的扰动，并且是可扩展的，支持添加新的组件和扰动类型。通过一系列案例研究，我们表明提示提供了有意义的差异，以支持强大的评估实践。它可以通过Python API提供：此HTTPS URL和用户友好的Web界面：此HTTPS URL

Title: SYNTHIA: Synthetic Yet Naturally Tailored Human-Inspired PersonAs

• Authors: Vahid Rahimzadeh, Erfan Moosavi Monazzah, Mohammad Taher Pilehvar, Yadollah Yaghoobzadeh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14922
• Pdf URL: https://arxiv.org/pdf/2507.14922
• Copy Paste: [[2507.14922]] SYNTHIA: Synthetic Yet Naturally Tailored Human-Inspired PersonAs(https://arxiv.org/abs/2507.14922)
• Keywords: language model, llm
• Abstract: Persona-driven LLMs have emerged as powerful tools in computational social science, yet existing approaches fall at opposite extremes, either relying on costly human-curated data or producing synthetic personas that lack consistency and realism. We introduce SYNTHIA, a dataset of 30,000 backstories derived from 10,000 real social media users from BlueSky open platform across three time windows, bridging this spectrum by grounding synthetic generation in authentic user activity. Our evaluation demonstrates that SYNTHIA achieves competitive performance with state-of-the-art methods in demographic diversity and social survey alignment while significantly outperforming them in narrative consistency. Uniquely, SYNTHIA incorporates temporal dimensionality and provides rich social interaction metadata from the underlying network, enabling new research directions in computational social science and persona-driven language modeling.
• 摘要：人格驱动的LLM已成为计算社会科学中的强大工具，但现有的方法处于相反的极端，要么依赖于昂贵的人类策划数据或产生缺乏一致性和现实主义的合成角色。我们介绍了Synthia，这是一个由30,000个背景故事的数据集，该数据集跨三个时间窗口从Bluesky Open平台衍生出10,000个真实的社交媒体用户，通过在真实的用户活动中接地合成生成来桥接此频谱。我们的评估表明，Synthia在人口多样性和社会调查时的最新方法中实现了竞争性能，同时在叙事一致性方面表现出色。独特的是，Synthia结合了时间维度，并从基础网络提供了丰富的社会互动元数据，从而实现了计算社会科学和人格驱动语言建模的新研究方向。

Title: MUR: Momentum Uncertainty guided Reasoning for Large Language Models

• Authors: Hang Yan, Fangzhi Xu, Rongman Xu, Yifei Li, Jian Zhang, Haoran Luo, Xiaobao Wu, Luu Anh Tuan, Haiteng Zhao, Qika Lin, Jun Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.14958
• Pdf URL: https://arxiv.org/pdf/2507.14958
• Copy Paste: [[2507.14958]] MUR: Momentum Uncertainty guided Reasoning for Large Language Models(https://arxiv.org/abs/2507.14958)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have achieved impressive performance on reasoning-intensive tasks, yet optimizing their reasoning efficiency remains an open challenge. While Test-Time Scaling (TTS) improves reasoning quality, it often leads to overthinking, wasting tokens on redundant computations. This work investigates how to efficiently and adaptively guide LLM test-time scaling without additional training. Inspired by the concept of momentum in physics, we propose Momentum Uncertainty-guided Reasoning (MUR), which dynamically allocates thinking budgets to critical reasoning steps by tracking and aggregating stepwise uncertainty over time. To support flexible inference-time control, we introduce gamma-control, a simple mechanism that tunes the reasoning budget via a single hyperparameter. We provide in-depth theoretical proof to support the superiority of MUR in terms of stability and biases. MUR is comprehensively evaluated against various TTS methods across four challenging benchmarks (MATH-500, AIME24, AIME25, and GPQA-diamond) using different sizes of recent Qwen3 models (1.7B, 4B, and 8B). Results demonstrate that MUR reduces computation by over 50% on average while improving accuracy by 0.62-3.37%.
• 摘要：大型语言模型（LLMS）在推理密集型任务上取得了令人印象深刻的表现，但是优化其推理效率仍然是一个开放的挑战。尽管测试时间缩放（TTS）提高了推理质量，但它通常会导致过度思考，并在冗余计算上浪费令牌。这项工作调查了如何在没有额外培训的情况下有效和自适应指导LLM测试时间缩放。受物理动量概念的启发，我们提出了动量不确定性引导的推理（MUR），该推理将思维预算动态地分配给关键推理步骤，通过跟踪和汇总逐步的不确定性，随着时间的流逝。为了支持灵活的推理时间控制，我们引入了伽马控制，这是一种简单的机制，可以通过单个超参数调整推理预算。我们提供深入的理论证明，以支持MUR在稳定和偏见方面的优势。使用不同尺寸的近期QWEN3模型（1.7b，4b和8b）的不同尺寸的基准（Math-500，AIME24，AIME25和GPQA-DIAMOND），对MUR进行了全面评估。结果表明，MUR平均将计算降低了50％以上，同时将精度提高了0.62-3.37％。

Title: RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback

• Authors: Qiaoyu Tang, Hao Xiang, Le Yu, Bowen Yu, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Junyang Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15024
• Pdf URL: https://arxiv.org/pdf/2507.15024
• Copy Paste: [[2507.15024]] RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback(https://arxiv.org/abs/2507.15024)
• Keywords: language model, llm, chain-of-thought
• Abstract: With the rapid advancement of Large Language Models (LLMs), developing effective critic modules for precise guidance has become crucial yet challenging. In this paper, we initially demonstrate that supervised fine-tuning for building critic modules (which is widely adopted in current solutions) fails to genuinely enhance models' critique abilities, producing superficial critiques with insufficient reflections and verifications. To unlock the unprecedented critique capabilities, we propose RefCritic, a long-chain-of-thought critic module based on reinforcement learning with dual rule-based rewards: (1) instance-level correctness of solution judgments and (2) refinement accuracies of the policy model based on critiques, aiming to generate high-quality evaluations with actionable feedback that effectively guides model refinement. We evaluate RefCritic on Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B across five benchmarks. On critique and refinement settings, RefCritic demonstrates consistent advantages across all benchmarks, e.g., 6.8\% and 7.2\% gains on AIME25 for the respective base models. Notably, under majority voting, policy models filtered by RefCritic show superior scaling with increased voting numbers. Moreover, despite training on solution-level supervision, RefCritic outperforms step-level supervised approaches on ProcessBench, a benchmark to identify erroneous steps in mathematical reasoning.
• 摘要：随着大语言模型（LLM）的快速发展，开发有效的评论家模块以确切的指导变得至关重要但充满挑战。在本文中，我们最初证明了对建筑评论家模块的监督微调（当前解决方案中广泛采用）无法真正增强模型的批评能力，从而产生了具有不足的反思和验证的表面批评。为了解锁前所未有的批评能力，我们提出了基于双重规则的奖励的基于强化学习的长链批评模块：（1）实例级别的解决方案判断的正确性，（2）基于批评的策略模型的改进精确性，旨在通过批准进行高度评估，以指导效果效率。我们在QWEN2.5-14B-r-Instruction和DeepSeek-R1-Distill-Qwen-14B上评估了跨五个基准测试的Requcritic。在批评和改进设置上，巡回素在所有基准测试中都具有一致的优势，例如，对于相应的基本模型，AIME25上的6.8 \％和7.2 \％增长。值得注意的是，在多数投票下，被巡回演出过滤的政策模型显示出了较高的标准，而投票人数增加。此外，尽管对解决方案级的监督进行了培训，但在ProcessBench上，Qucritic的表现优于阶梯监督方法，这是确定数学推理中错误步骤的基准。

Title: WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization

• Authors: Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15061
• Pdf URL: https://arxiv.org/pdf/2507.15061
• Copy Paste: [[2507.15061]] WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization(https://arxiv.org/abs/2507.15061)
• Keywords: language model, llm, agent
• Abstract: The advent of Large Language Model (LLM)-powered agents has revolutionized artificial intelligence by enabling solutions to complex, open-ended tasks through web-based information-seeking (IS) capabilities. The scarcity of high-quality training data has limited the development of IS agents. Existing approaches typically adopt an information-driven paradigm that first collects web data and then generates questions based on the retrieval. However, this may lead to inconsistency between information structure and reasoning structure, question and answer. To mitigate, we propose a formalization-driven IS data synthesis framework WebShaper to construct a dataset. WebShaper systematically formalizes IS tasks through set theory. Central to the formalization is the concept of Knowledge Projections (KP), which enables precise control over reasoning structure by KP operation compositions. During synthesis, we begin by creating seed tasks, then use a multi-step expansion process. At each step, an agentic Expander expands the current formal question more complex with retrieval and validation tools based on our formalization. We train our model on the synthesized dataset. Experiment results demonstrate that WebShaper achieves state-of-the-art performance among open-sourced IS agents on GAIA and WebWalkerQA benchmarks.
• 摘要：大型语言模型（LLM）能力的代理商的出现通过通过基于Web的信息寻求（IS）功能来使解决方案能够实现复杂的开放式任务，从而彻底改变了人工智能。高质量培训数据的稀缺性限制了IS代理的发展。现有方法通常采用信息驱动的范式，该范式首先收集Web数据，然后根据检索产生问题。但是，这可能会导致信息结构与推理结构，问答之间的不一致。为了减轻，我们提出了一个形式驱动的是数据综合框架WebShaper来构建数据集。 WebShaper系统地正式化是通过集理论的任务。形式化的核心是知识预测（KP）的概念，它可以通过KP操作组成对推理结构进行精确控制。在合成过程中，我们首先创建种子任务，然后使用多步扩展过程。在每个步骤中，代理商扩展器都会根据我们的正式化使用检索和验证工具来扩展当前的正式问题。我们在合成的数据集上训练模型。实验结果表明，开源的Webshaper在Gaia和Webwalkerqa基准测试中实现了最先进的性能。

Title: A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations

• Authors: Vijeta Deshpande, Ishita Dasgupta, Uttaran Bhattacharya, Somdeb Sarkhel, Saayan Mitra, Anna Rumshisky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15092
• Pdf URL: https://arxiv.org/pdf/2507.15092
• Copy Paste: [[2507.15092]] A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations(https://arxiv.org/abs/2507.15092)
• Keywords: language model, llm, prompt
• Abstract: Synthetic text generated by Large Language Models (LLMs) is increasingly used for further training and improvement of LLMs. Diversity is crucial for the effectiveness of synthetic data, and researchers rely on prompt engineering to improve diversity. However, the impact of prompt variations on response text length, and, more importantly, the consequential effect on lexical diversity measurements, remain underexplored. In this work, we propose Penalty-Adjusted Type-Token Ratio (PATTR), a diversity metric robust to length variations. We generate a large synthetic corpus of over 20M words using seven models from the LLaMA, OLMo, and Phi families, focusing on a creative writing task of video script generation, where diversity is crucial. We evaluate per-response lexical diversity using PATTR and compare it against existing metrics of Moving-Average TTR (MATTR) and Compression Ratio (CR). Our analysis highlights how text length variations introduce biases favoring shorter responses. Unlike existing metrics, PATTR explicitly considers the task-specific target response length ($L_T$) to effectively mitigate length biases. We further demonstrate the utility of PATTR in filtering the top-10/100/1,000 most lexically diverse responses, showing that it consistently outperforms MATTR and CR by yielding on par or better diversity with high adherence to $L_T$.
• 摘要：大语模型（LLM）生成的合成文本越来越多地用于进一步培训和改进LLM。多样性对于合成数据的有效性至关重要，研究人员依靠迅速的工程来改善多样性。但是，迅速变化对响应文本长度的影响，更重要的是，对词汇多样性测量的结果效应仍然没有得到充实的影响。在这项工作中，我们提出了惩罚调整的类型式比率（PATTR），这是多样性的稳健性与长度变化。我们使用Llama，Olmo和Phi家族的七个模型生成了一个超过20M单词的合成语料库，重点关注视频脚本生成的创意写作任务，而多样性至关重要。我们使用PATTR评估了每种响应词汇多样性，并将其与现有的移动平均TTR（MATTR）和压缩比（CR）的指标进行了比较。我们的分析强调了文本长度变化如何引入有利于较短响应的偏见。与现有指标不同，Pattr明确考虑特定于任务的目标响应长度（$ L_T $），以有效减轻长度偏差。我们进一步证明了PATTR在过滤前10/100/1,000最多的响应中的实用性，这表明它始终以par或更好的多样性（以高于$ l_t $）的高度屈服或更好的多样性来胜过mattr和cr。

Title: Filling the Gap: Is Commonsense Knowledge Generation useful for Natural Language Inference?

• Authors: Chathuri Jayaweera, Brianna Yanqui, Bonnie Dorr
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15100
• Pdf URL: https://arxiv.org/pdf/2507.15100
• Copy Paste: [[2507.15100]] Filling the Gap: Is Commonsense Knowledge Generation useful for Natural Language Inference?(https://arxiv.org/abs/2507.15100)
• Keywords: language model, llm
• Abstract: Natural Language Inference (NLI) is the task of determining the semantic entailment of a premise for a given hypothesis. The task aims to develop systems that emulate natural human inferential processes where commonsense knowledge plays a major role. However, existing commonsense resources lack sufficient coverage for a variety of premise-hypothesis pairs. This study explores the potential of Large Language Models as commonsense knowledge generators for NLI along two key dimensions: their reliability in generating such knowledge and the impact of that knowledge on prediction accuracy. We adapt and modify existing metrics to assess LLM factuality and consistency in generating in this context. While explicitly incorporating commonsense knowledge does not consistently improve overall results, it effectively helps distinguish entailing instances and moderately improves distinguishing contradictory and neutral inferences.
• 摘要：自然语言推论（NLI）是确定给定假设前提的语义的任务。该任务旨在开发模仿常识性知识起主要作用的自然人类推论过程的系统。但是，现有的常识性资源缺乏各种前提 - 假设对的足够覆盖范围。这项研究探讨了大型语言模型作为NLI沿两个关键维度的常识性知识发生器的潜力：它们在产生这种知识的可靠性以及该知识对预测准确性的影响。我们适应并修改现有指标，以评估LLM在这种情况下生成时的事实和一致性。虽然明确合并常识性知识并不能始终如一地改善整体结果，但它有效地有助于区分需要实例，并适度地改善区分矛盾和中性的推论。

Title: What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction

• Authors: Lingbo Li, Anuradha Mathrani, Teo Susnjak
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15152
• Pdf URL: https://arxiv.org/pdf/2507.15152
• Copy Paste: [[2507.15152]] What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction(https://arxiv.org/abs/2507.15152)
• Keywords: language model, gpt, llm, prompt
• Abstract: Automating data extraction from full-text randomised controlled trials (RCTs) for meta-analysis remains a significant challenge. This study evaluates the practical performance of three LLMs (Gemini-2.0-flash, Grok-3, GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customised prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customised prompts were the most effective, boosting recall by up to 15\%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.
• 摘要：从全文随机对照试验（RCT）进行荟萃分析中提取数据仍然是一个重大挑战。这项研究评估了三个涉及统计结果，偏见评估和研究级别特征的任务，评估了三个LLM（Gemini-2.0-Flash，Grok-3，GPT-4O-MINI）的实际性能：高血压，糖尿病和骨科。我们测试了四种不同的提示策略（基本提示，自我反射提示，模型集合和定制提示），以确定如何提高提取质量。所有模型均表现出很高的精度，但由于省略关键信息而持续遭受不良召回的困扰。我们发现自定义提示是最有效的，最多可提高召回率15％。基于此分析，我们提出了一组三层指南，用于在数据提取中使用LLM，将数据类型匹配到基于任务复杂性和风险的适当自动化水平。我们的研究提供了实用的建议，可以在现实世界中的荟萃分析中自动化数据提取，从而通过针对性的，特定于任务的自动化来平衡LLM效率与专家监督。

Title: Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment

• Authors: Xiandong Meng, Yan Wu, Yexin Tian, Xin Hu, Tianze Kang, Junliang Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15198
• Pdf URL: https://arxiv.org/pdf/2507.15198
• Copy Paste: [[2507.15198]] Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment(https://arxiv.org/abs/2507.15198)
• Keywords: language model
• Abstract: This paper addresses the challenges of high computational cost and slow inference in deploying large language models. It proposes a distillation strategy guided by multiple teacher models. The method constructs several teacher models and integrates their output probability distributions and intermediate semantic features. This guides the student model to learn from multiple sources of knowledge. As a result, the student model gains stronger language understanding and generation ability while maintaining a small parameter size. To achieve this, the paper introduces a weighted output fusion mechanism, a feature alignment loss function, and an entropy-driven dynamic teacher weighting strategy. These components improve the quality and stability of knowledge transfer during distillation. Under multi-teacher guidance, the student model captures semantic information more effectively and demonstrates strong performance across multiple evaluation metrics. In particular, the method shows high consistency in expression, generalization ability, and task adaptability in tasks such as language modeling, text generation, and multi-task learning. The experiments compare the proposed method with several widely adopted distillation approaches. The results further confirm its overall advantages in perplexity, distillation loss, and generation quality. This study provides a feasible technical path for the efficient compression of large-scale language models. It also demonstrates the effectiveness of multi-teacher collaborative mechanisms in complex language modeling tasks.
• 摘要：本文解决了高计算成本的挑战和在部署大型语言模型时的推断缓慢的挑战。它提出了由多个教师模型指导的蒸馏策略。该方法构建了几种教师模型，并集成了其输出概率分布和中间语义特征。这指导学生模型从多种知识来源学习。结果，学生模型在保持较小的参数大小的同时，获得了更强的语言理解和发电能力。为此，本文引入了加权输出融合机制，功能比对损失函数以及熵驱动的动态教师加权策略。这些成分提高了蒸馏过程中知识转移的质量和稳定性。在多教学指导下，学生模型更有效地捕获语义信息，并在多个评估指标中证明了强劲的性能。特别是，该方法在语言建模，文本生成和多任务学习等任务中的表达，概括能力和任务适应性方面表现出很高的一致性。实验将提出的方法与几种广泛采用的蒸馏方法进行了比较。结果进一步证实了其在困惑，蒸馏损失和发电质量方面的总体优势。这项研究为有效压缩大规模语言模型提供了可行的技术途径。它还证明了多教老师协作机制在复杂的语言建模任务中的有效性。

Title: SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest

• Authors: Shayan Vassef, Amirhossein Dabiriaghdam, Mohammadreza Bakhtiari, Yadollah Yaghoobzadeh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15236
• Pdf URL: https://arxiv.org/pdf/2507.15236
• Copy Paste: [[2507.15236]] SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest(https://arxiv.org/abs/2507.15236)
• Keywords: language model
• Abstract: This work investigates the impact of multi-task, multi-lingual, and multi-source learning approaches on the robustness and performance of pretrained language models. To enhance this analysis, we introduce Subsets of Interest (SOI), a novel categorization framework that identifies six distinct learning behavior patterns during training, including forgettable examples, unlearned examples, and always correct examples. Through SOI transition heatmaps and dataset cartography visualization, we analyze how examples shift between these categories when transitioning from single-setting to multi-setting configurations. We perform comprehensive experiments across three parallel comparisons: multi-task vs. single-task learning using English tasks (entailment, paraphrase, sentiment), multi-source vs. single-source learning using sentiment analysis datasets, and multi-lingual vs. single-lingual learning using intent classification in French, English, and Persian. Our results demonstrate that multi-source learning consistently improves out-of-distribution performance by up to 7%, while multi-task learning shows mixed results with notable gains in similar task combinations. We further introduce a two-stage fine-tuning approach where the second stage leverages SOI-based subset selection to achieve additional performance improvements. These findings provide new insights into training dynamics and offer practical approaches for optimizing multi-setting language model performance.
• 摘要：这项工作调查了多任务，多语言和多源学习方法对审计语言模型的鲁棒性和性能的影响。为了增强此分析，我们介绍了感兴趣的子集（SOI），这是一个新颖的分类框架，该框架在培训期间识别六种不同的学习行为模式，包括可忘记的示例，未学习的例子，始终正确的示例。通过SOI过渡热图和数据集制图可视化，我们分析了从单个设定到多设定配置的示例如何在这些类别之间发生变化。我们在三个平行的比较中进行全面的实验：使用英语任务（组成，释义，情感），使用情感分析数据集的多任务与单任务学习，多源与单源学习，以及使用法语，英语和波斯语的多语言与单语性学习。我们的结果表明，多源学习始终将分布的性能提高多达7％，而多任务学习显示出不同的结果，在类似的任务组合中取得了显着增长。我们进一步引入了两阶段的微调方法，第二阶段利用基于SOI的子集选择以实现额外的绩效改进。这些发现为培训动态提供了新的见解，并提供了优化多设定语言模型性能的实用方法。

Title: ChiMed 2.0: Advancing Chinese Medical Dataset in Facilitating Large Language Modeling

• Authors: Yuanhe Tian, Junjie Liu, Zhizhou Kou, Yuxiang Li, Yan Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15275
• Pdf URL: https://arxiv.org/pdf/2507.15275
• Copy Paste: [[2507.15275]] ChiMed 2.0: Advancing Chinese Medical Dataset in Facilitating Large Language Modeling(https://arxiv.org/abs/2507.15275)
• Keywords: language model, llm
• Abstract: Building high-quality data resources is crucial for advancing artificial intelligence research and applications in specific domains, particularly in the Chinese medical domain. Existing Chinese medical datasets are limited in size and narrow in domain coverage, falling short of the diverse corpora required for effective pre-training. Moreover, most datasets are designed solely for LLM fine-tuning and do not support pre-training and reinforcement learning from human feedback (RLHF). In this paper, we propose a Chinese medical dataset named ChiMed 2.0, which extends our previous work ChiMed, and covers data collected from Chinese medical online platforms and generated by LLMs. ChiMed 2.0 contains 204.4M Chinese characters covering both traditional Chinese medicine classics and modern general medical data, where there are 164.8K documents for pre-training, 351.6K question-answering pairs for supervised fine-tuning (SFT), and 41.7K preference data tuples for RLHF. To validate the effectiveness of our approach for training a Chinese medical LLM, we conduct further pre-training, SFT, and RLHF experiments on representative general domain LLMs and evaluate their performance on medical benchmark datasets. The results show performance gains across different model scales, validating the dataset's effectiveness and applicability.
• 摘要：建立高质量的数据资源对于推进特定领域的人工智能研究和应用至关重要，尤其是在中国医疗领域。现有的中国医疗数据集的大小有限，域覆盖范围狭窄，距离有效培训所需的多种多样的语料库。此外，大多数数据集都是专为LLM微调设计的，并且不支持人类反馈（RLHF）的预培训和加强学习。在本文中，我们提出了一个名为Chimed 2.0的中国医疗数据集，该数据集扩展了我们以前的工作，并涵盖了从中国医疗在线平台收集并由LLMS生成的数据。 Chimed 2.0包含20440万个汉字，涵盖了中药经典和现代的一般医学数据，其中有164.8k的预培训文件，351.6k的351.6k询问对监督的调查（SFT）和41.7k RLHF的偏好率数据。为了验证我们培训中国医学LLM的方法的有效性，我们在代表性的一般域LLMS上进行了进一步的预培训，SFT和RLHF实验，并评估其在医疗基准数据集上的性能。结果表明，不同模型量表的性能提高，从而验证了数据集的有效性和适用性。

Title: A Novel Self-Evolution Framework for Large Language Models

• Authors: Haoran Sun, Zekun Zhang, Shaoning Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15281
• Pdf URL: https://arxiv.org/pdf/2507.15281
• Copy Paste: [[2507.15281]] A Novel Self-Evolution Framework for Large Language Models(https://arxiv.org/abs/2507.15281)
• Keywords: language model, llm
• Abstract: The capabilities of Large Language Models (LLMs) are limited to some extent by pre-training, so some researchers optimize LLMs through post-training. Existing post-training strategies, such as memory-based retrieval or preference optimization, improve user alignment yet fail to enhance the model's domain cognition. To bridge this gap, we propose a novel Dual-Phase Self-Evolution (DPSE) framework that jointly optimizes user preference adaptation and domain-specific competence. DPSE introduces a Censor module to extract multi-dimensional interaction signals and estimate satisfaction scores, which guide structured data expansion via topic-aware and preference-driven strategies. These expanded datasets support a two-stage fine-tuning pipeline: supervised domain grounding followed by frequency-aware preference optimization. Experiments across general NLP benchmarks and long-term dialogue tasks demonstrate that DPSE consistently outperforms Supervised Fine-Tuning, Preference Optimization, and Memory-Augmented baselines. Ablation studies validate the contribution of each module. In this way, our framework provides an autonomous path toward continual self-evolution of LLMs.
• 摘要：大语言模型（LLM）的功能在某种程度上仅限于培训，因此一些研究人员通过培训来优化LLMS。现有的训练后策略，例如基于内存的检索或偏好优化，可以改善用户对齐，但无法增强模型的域认知。为了弥合这一差距，我们提出了一个新型的双相自我进化（DPSE）框架，该框架共同优化了用户偏好适应和特定于域的能力。 DPSE引入了一个审查模块，以提取多维交互信号并估计满意度分数，该模块通过主题感知和偏好驱动的策略指导结构化数据扩展。这些扩展的数据集支持两阶段的微调管道：监督域接地，然后进行频率吸引偏好优化。跨NLP一般基准和长期对话任务进行的实验表明，DPSE始终超过监督的微调，偏好优化和记忆力计的基线。消融研究验证了每个模块的贡献。这样，我们的框架为LLM的持续自我发展提供了自主途径。

Title: Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection

• Authors: Navid Ayoobi, Sadat Shahriar, Arjun Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15286
• Pdf URL: https://arxiv.org/pdf/2507.15286
• Copy Paste: [[2507.15286]] Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection(https://arxiv.org/abs/2507.15286)
• Keywords: llm
• Abstract: We present a novel evaluation paradigm for AI text detectors that prioritizes real-world and equitable assessment. Current approaches predominantly report conventional metrics like AUROC, overlooking that even modest false positive rates constitute a critical impediment to practical deployment of detection systems. Furthermore, real-world deployment necessitates predetermined threshold configuration, making detector stability (i.e. the maintenance of consistent performance across diverse domains and adversarial scenarios), a critical factor. These aspects have been largely ignored in previous research and benchmarks. Our benchmark, SHIELD, addresses these limitations by integrating both reliability and stability factors into a unified evaluation metric designed for practical assessment. Furthermore, we develop a post-hoc, model-agnostic humanification framework that modifies AI text to more closely resemble human authorship, incorporating a controllable hardness parameter. This hardness-aware approach effectively challenges current SOTA zero-shot detection methods in maintaining both reliability and stability. (Data and code: this https URL)
• 摘要：我们为AI文本检测器提供了一种新颖的评估范式，该检测器优先考虑现实世界和公平评估。当前的方法主要报告了诸如AUROC之类的常规指标，即使谦虚的假阳性率也构成了检测系统实际部署的关键障碍。此外，现实世界的部署需要预定的阈值配置，从而使探测器稳定性（即维持跨不同域和对抗场景的一致绩效的维护），这是一个关键因素。这些方面在以前的研究和基准测试中都被忽略了。我们的基准测试标准通过将可靠性和稳定性因素同时整合到设计用于实际评估的统一评估指标中来解决这些局限性。此外，我们开发了一个事后，模型的人性化框架，该框架将AI文本修改为更像人类的作者身份，并结合了可控的硬度参数。这种硬度感知的方法有效地挑战了当前的SOTA零射击检测方法，以保持可靠性和稳定性。 （数据和代码：此HTTPS URL）

Title: On the Inevitability of Left-Leaning Political Bias in Aligned Language Models

• Authors: Thilo Hagendorff
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2507.15328
• Pdf URL: https://arxiv.org/pdf/2507.15328
• Copy Paste: [[2507.15328]] On the Inevitability of Left-Leaning Political Bias in Aligned Language Models(https://arxiv.org/abs/2507.15328)
• Keywords: language model, llm
• Abstract: The guiding principle of AI alignment is to train large language models (LLMs) to be harmless, helpful, and honest (HHH). At the same time, there are mounting concerns that LLMs exhibit a left-wing political bias. Yet, the commitment to AI alignment cannot be harmonized with the latter critique. In this article, I argue that intelligent systems that are trained to be harmless and honest must necessarily exhibit left-wing political bias. Normative assumptions underlying alignment objectives inherently concur with progressive moral frameworks and left-wing principles, emphasizing harm avoidance, inclusivity, fairness, and empirical truthfulness. Conversely, right-wing ideologies often conflict with alignment guidelines. Yet, research on political bias in LLMs is consistently framing its insights about left-leaning tendencies as a risk, as problematic, or concerning. This way, researchers are actively arguing against AI alignment, tacitly fostering the violation of HHH principles.
• 摘要：AI一致性的指导原则是训练大型语言模型（LLM）是无害，乐于助人和诚实的（HHH）。同时，人们一直担心LLMS表现出左翼政治偏见。然而，对AI一致性的承诺不能与后一种批评相一致。在本文中，我认为经过训练的无害和诚实的智能系统一定必须表现出左翼政治偏见。一致性目标的基本假设固有地与渐进的道德框架和左翼原则一致，强调避免危害，包容性，公平性和经验真实性。相反，右翼意识形态通常与一致指南相抵触。然而，对LLM中政治偏见的研究一直在构建其对左倾倾向的见解，因为它是一种风险，有问题或存在的。这样，研究人员正在积极反对AI一致性，默认促进了违反HHH原则的行为。

Title: Reasoning Models are Test Exploiters: Rethinking Multiple-Choice

• Authors: Narun Raman, Taylor Lundy, Kevin Leyton-Brown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15337
• Pdf URL: https://arxiv.org/pdf/2507.15337
• Copy Paste: [[2507.15337]] Reasoning Models are Test Exploiters: Rethinking Multiple-Choice(https://arxiv.org/abs/2507.15337)
• Keywords: language model, llm, chain-of-thought
• Abstract: When evaluating Large Language Models (LLMs) in question-answering domains, it is common to ask the model to choose among a fixed set of choices (so-called multiple-choice question-answering, or MCQA). Although downstream tasks of interest typically do not provide systems with explicit options among which to choose, this approach is nevertheless widely used because it makes it makes automatic grading straightforward and has tended to produce challenging benchmarks that correlate sufficiently well with downstream performance. This paper investigates the extent to which this trend continues to hold for state-of-the-art reasoning models, describing a systematic evaluation of $15$ different question-answering benchmarks (e.g., MMLU, HLE) and $25$ different LLMs (including small models such as Qwen 7B and relatively large models such as Llama 70B). For each model-benchmark pair, we considered $5$ ways of presenting the model with questions, including variations on whether multiple choices were offered to the model at all; whether "none of the above" sometimes replaced the right answer; and whether the model was permitted to perform chain-of-thought reasoning before and/or after the choices were presented. MCQA remained a good proxy for the downstream performance of models as long as they were allowed to perform chain-of-thought reasoning only before being presented with the options among which they had to select. On the other hand, large models that were able to perform reasoning after being given a set of options tended to significantly outperform their free-text performance due to exploiting the information in the options. We conclude that MCQA is no longer a good proxy for assessing downstream performance of state-of-the-art models, and offer practical guidelines for designing more robust, bias-resistant benchmarks that better reflect LLMs' genuine reasoning capabilities.
• 摘要：在评估提问域中的大型语言模型（LLM）时，通常要求该模型在固定的一组选择（所谓的多项选择问答或MCQA）中进行选择。尽管感兴趣的下游任务通常不会为系统提供明确的选择，但是这种方法仍被广泛使用，因为它使其使自动分级直接进行，并且倾向于产生具有挑战性的基准，这些基准与下游性能充分相关。本文研究了最先进的推理模型继续保持这种趋势的程度，描述了对$ 15 $ $不同的提问基准的系统评估（例如，MMLU，HLU）和25美元的不同LLMS（包括QWEN 7B等小型型号，例如QWEN 7B和相对大型型号，例如LLAMA 70B）。对于每个型号基准对，我们考虑了$ 5 $呈现模型的方法，包括有关是否完全为模型提供多种选择的变化；是否“没有上述”有时会取代正确的答案；以及是否允许模型在提出选择之前和/或之后执行经过思考的推理。只要允许他们才能在不得不选择的选择中出示这些选择之前，MCQA仍然是模型下游性能的良好代理。另一方面，由于利用选项中的信息，因此能够在获得一组选项后能够执行推理的大型模型往往会大大优于其自由文本性能。我们得出的结论是，MCQA不再是评估最先进模型的下游性能的好代理，并提供了设计更强大的偏见，抗偏见的基准的实用准则，可以更好地反映LLMS的真正推理能力。

Title: LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderators

• Authors: Leanne Tan, Gabriel Chua, Ziyu Ge, Roy Ka-Wei Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15339
• Pdf URL: https://arxiv.org/pdf/2507.15339
• Copy Paste: [[2507.15339]] LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderators(https://arxiv.org/abs/2507.15339)
• Keywords: llm
• Abstract: Modern moderation systems increasingly support multiple languages, but often fail to address localisation and low-resource variants - creating safety gaps in real-world deployments. Small models offer a potential alternative to large LLMs, yet still demand considerable data and compute. We present LionGuard 2, a lightweight, multilingual moderation classifier tailored to the Singapore context, supporting English, Chinese, Malay, and partial Tamil. Built on pre-trained OpenAI embeddings and a multi-head ordinal classifier, LionGuard 2 outperforms several commercial and open-source systems across 17 benchmarks, including both Singapore-specific and public English datasets. The system is actively deployed within the Singapore Government, demonstrating practical efficacy at scale. Our findings show that high-quality local data and robust multilingual embeddings can achieve strong moderation performance, without fine-tuning large models. We release our model weights and part of our training data to support future work on LLM safety.
• 摘要：现代节制系统越来越支持多种语言，但通常无法解决本地化和低资源变体 - 在现实世界部署中造成安全差距。小型型号为大型LLM提供了潜在的替代方法，但仍需要大量数据和计算。我们介绍了针对新加坡环境量身定制的轻便，多语言的调节分类器，支持英语，中文，马来语和部分泰米尔语。 Lionguard 2建立在预先训练的OpenAI嵌入式和多头序列分类器上，在17个基准中，均超过了几个商业和开源系统，包括新加坡特定于特定的英语数据集。该系统积极部署在新加坡政府内部，表明了大规模的实际功效。我们的发现表明，高质量的本地数据和强大的多语言嵌入可以实现强大的适度性能，而无需微调大型模型。我们发布了模型权重和培训数据的一部分，以支持有关LLM安全的未来工作。

Title: Probing Information Distribution in Transformer Architectures through Entropy Analysis

• Authors: Amedeo Buonanno, Alessandro Rivetti, Francesco A. N. Palmieri, Giovanni Di Gennaro, Gianmarco Romano
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15347
• Pdf URL: https://arxiv.org/pdf/2507.15347
• Copy Paste: [[2507.15347]] Probing Information Distribution in Transformer Architectures through Entropy Analysis(https://arxiv.org/abs/2507.15347)
• Keywords: language model, gpt
• Abstract: This work explores entropy analysis as a tool for probing information distribution within Transformer-based architectures. By quantifying token-level uncertainty and examining entropy patterns across different stages of processing, we aim to investigate how information is managed and transformed within these models. As a case study, we apply the methodology to a GPT-based large language model, illustrating its potential to reveal insights into model behavior and internal representations. This approach may offer insights into model behavior and contribute to the development of interpretability and evaluation frameworks for transformer-based models
• 摘要：这项工作探讨了熵分析，作为用于探索基于变压器体系结构中信息分布的工具。通过量化令牌级别的不确定性并检查跨处理阶段的熵模式，我们旨在研究如何在这些模型中管理和转换信息。作为案例研究，我们将方法应用于基于GPT的大型语言模型，说明了其潜力揭示对模型行为和内部表示的见解。这种方法可以提供对模型行为的见解，并为基于变压器模型的可解释性和评估框架的发展做出贡献

Title: Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding

• Authors: Elisa Sanchez-Bayona, Rodrigo Agerri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15357
• Pdf URL: https://arxiv.org/pdf/2507.15357
• Copy Paste: [[2507.15357]] Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding(https://arxiv.org/abs/2507.15357)
• Keywords: language model, llm, prompt
• Abstract: This paper presents a comprehensive evaluation of the capabilities of Large Language Models (LLMs) in metaphor interpretation across multiple datasets, tasks, and prompt configurations. Although metaphor processing has gained significant attention in Natural Language Processing (NLP), previous research has been limited to single-dataset evaluations and specific task settings, often using artificially constructed data through lexical replacement. We address these limitations by conducting extensive experiments using diverse publicly available datasets with inference and metaphor annotations, focusing on Natural Language Inference (NLI) and Question Answering (QA) tasks. The results indicate that LLMs' performance is more influenced by features like lexical overlap and sentence length than by metaphorical content, demonstrating that any alleged emergent abilities of LLMs to understand metaphorical language are the result of a combination of surface-level features, in-context learning, and linguistic knowledge. This work provides critical insights into the current capabilities and limitations of LLMs in processing figurative language, highlighting the need for more realistic evaluation frameworks in metaphor interpretation tasks. Data and code are publicly available.
• 摘要：本文对多个数据集，任务和提示配置的隐喻解释中的大语言模型（LLM）的功能进行了全面评估。尽管隐喻处理在自然语言处理（NLP）中引起了很大的关注，但以前的研究仅限于单数据库评估和特定的任务设置，通常是通过词汇替换使用人工构造的数据。我们通过使用推理和隐喻注释的多种公开数据集进行广泛的实验来解决这些局限性，重点关注自然语言推论（NLI）和问题答案（QA）任务。结果表明，与隐喻内容相比，LLMS的性能更受词汇重叠和句子长度等特征的影响，这表明LLMS了解隐喻语言的任何出现能力都是表面级别特征，内部文化学习和语言知识的结合的结果。这项工作为LLM在处理形象性语言中的当前功能和局限性提供了重要的见解，强调了在隐喻解释任务中需要更现实的评估框架的需求。数据和代码公开可用。

Title: STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models

• Authors: Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2507.15375
• Pdf URL: https://arxiv.org/pdf/2507.15375
• Copy Paste: [[2507.15375]] STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models(https://arxiv.org/abs/2507.15375)
• Keywords: language model, chain-of-thought
• Abstract: Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses. However, current SLMs lack the ability to perform an internal, unspoken thinking process before responding. In contrast, humans typically engage in complex mental reasoning internally, enabling them to communicate ideas clearly and concisely. Thus, integrating an unspoken thought process into SLMs is highly desirable. While naively generating a complete chain-of-thought (CoT) reasoning before starting to talk can enable thinking for SLMs, this induces additional latency for the speech response, as the CoT reasoning can be arbitrarily long. To solve this issue, we propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks. Since the audio duration of a chunk of spoken response is much longer than the time to generate the tokens in a chunk of spoken response, we use the remaining free time to generate the unspoken reasoning tokens. When a chunk of audio is played to the user, the model continues to generate the next unspoken reasoning chunk, achieving simultaneous thinking and talking. Remarkably, Stitch matches the latency of baselines that cannot generate unspoken CoT by design while outperforming those baselines by 15% on math reasoning datasets; Stitch also performs equally well on non-reasoning datasets as those baseline models. Some animations and demonstrations are on the project page: this https URL.
• 摘要：口语模型（SLM）旨在采用语音输入并产生口头响应。但是，当前的SLM缺乏在响应之前执行内部无言思维过程的能力。相比之下，人类通常在内部进行复杂的心理推理，使他们能够清晰，清晰地传达思想。因此，将不言而喻的思维过程集成到SLM上是非常可取的。在天真地产生完整的思想链（COT）推理之前，可以在开始交谈之前对SLM进行思考，但由于COT推理可以任意长时间，因此会导致语音响应的额外延迟。为了解决这个问题，我们提出了Stitch，这是一种新颖的一代方法，可以在不言而喻的推理块和口语响应块之间交替。由于大量口语响应的音频持续时间比在大量口语响应中生成令牌的时间要长得多，因此我们使用剩余的空闲时间来产生不言而喻的推理令牌。当向用户播放大量音频时，该模型将继续生成下一个不言而喻的推理块，从而实现同时思考和说话。值得注意的是，针迹与无法通过设计产生不言而喻的基线的潜伏期匹配，而在数学推理数据集上的表现优于这些基准的潜伏期。针迹在非调理数据集上的性能与这些基线模型同样出色。一些动画和演示在项目页面上：此HTTPS URL。

Title: AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming

• Authors: Jierui Li, Raymond Mooney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15378
• Pdf URL: https://arxiv.org/pdf/2507.15378
• Copy Paste: [[2507.15378]] AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming(https://arxiv.org/abs/2507.15378)
• Keywords: llm
• Abstract: Recent progress in LLMs, such as reasoning models, has demonstrated strong abilities to solve complex competitive programming problems, often rivaling top human competitors. However, it remains underexplored whether these abilities generalize to relevant domains that are less seen during training. To address this, we introduce AlgoSimBench, a new benchmark designed to assess LLMs' ability to identify algorithmically similar problems (ASPs)-problems that can be solved using similar algorithmic approaches. AlgoSimBench consists of 1317 problems, annotated with 231 distinct fine-grained algorithm tags, from which we curate 402 multiple-choice questions (MCQs), where each question presents one algorithmically similar problem alongside three textually similar but algorithmically dissimilar distractors. Our evaluation reveals that LLMs struggle to identify ASPs, with the best-performing model (o3-mini) achieving only 65.9% accuracy on the MCQ task. To address this challenge, we propose attempted solution matching (ASM), a novel method for improving problem similarity detection. On our MCQ task, ASM yields an absolute accuracy improvement of 6.7% to 11.7% across different models. We also evaluated code embedding models and retrieval methods on similar problem identification. While the adversarial selection of problems degrades the performance to be less than random, we found that simply summarizing the problem to remove narrative elements eliminates the effect, and combining ASM with a keyword-prioritized method, BM25, can yield up to 52.2% accuracy. Code and data are available at this http URL
• 摘要：LLM的最新进展，例如推理模型，已经证明了解决复杂的竞争编程问题的强大能力，通常与人类顶级竞争对手相媲美。但是，这些能力是否将这些能力推广到训练过程中较少看到的相关领域仍然没有充满信心。为了解决这个问题，我们介绍了Algosimbench，这是一种新的基准测试，旨在评估LLMS识别算法上类似问题（ASP）的能力，可以使用类似的算法方法来解决该问题。 Algosimbench由1317个问题组成，带有231个不同的细粒算法标签注释，我们从中策划了402个多项选择问题（MCQ），每个问题每个问题都呈现一个算法上类似的问题，以及三个文本相似但算法相似但具有算法的分散分散分散的问题。我们的评估表明，LLM努力识别ASP，其表现最佳的模型（O3-MINI）仅在MCQ任务上实现了65.9％的精度。为了应对这一挑战，我们提出了尝试的解决方案匹配（ASM），这是一种改善问题相似性检测的新方法。在我们的MCQ任务上，ASM在不同模型中的绝对准确性提高了6.7％至11.7％。我们还评估了有关类似问题识别的代码嵌入模型和检索方法。虽然问题的对抗性选择使性能降低了不到随机性，但我们发现，简单地总结问题以消除叙事元素可以消除效果，将ASM与关键字优先定位的方法BM25结合起来，可以屈服高达52.2％的精度。代码和数据可在此HTTP URL上找到

Title: ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution

• Authors: Alexandru Coca, Mark Gaynor, Zhenxing Zhang, Jianpeng Cheng, Bo-Hsiang Tseng, Pete Boothroyd, Héctor Martinez Alonso, Diarmuid Ó Séaghdha, Anders Johannsen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15501
• Pdf URL: https://arxiv.org/pdf/2507.15501
• Copy Paste: [[2507.15501]] ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution(https://arxiv.org/abs/2507.15501)
• Keywords: language model, llm
• Abstract: This work evaluates the potential of large language models (LLMs) to power digital assistants capable of complex action execution. These assistants rely on pre-trained programming knowledge to execute multi-step goals by composing objects and functions defined in assistant libraries into action execution programs. To achieve this, we develop ASPERA, a framework comprising an assistant library simulation and a human-assisted LLM data generation engine. Our engine allows developers to guide LLM generation of high-quality tasks consisting of complex user queries, simulation state and corresponding validation programs, tackling data availability and evaluation robustness challenges. Alongside the framework we release Asper-Bench, an evaluation dataset of 250 challenging tasks generated using ASPERA, which we use to show that program generation grounded in custom assistant libraries is a significant challenge to LLMs compared to dependency-free code generation.
• 摘要：这项工作评估了大语模型（LLM）对能够进行复杂行动执行的数字助手的潜力。这些助手依靠预先训练的编程知识来执行多步目标，通过将助手库中定义的对象和功能撰写为行动执行程序。为了实现这一目标，我们开发了Aspera，该框架包括助理图书馆模拟和人为辅助的LLM数据生成引擎。我们的引擎允许开发人员指导LLM生成高质量任务，这些任务包括复杂的用户查询，模拟状态和相应的验证程序，解决数据可用性和评估鲁棒性挑战。除了框架之外，我们发布了Asper-Bench，这是使用Aspera生成的250个具有挑战性的任务的评估数据集，我们用来表明与无依赖性代码生成相比，在自定义助理库中基于自定义助理库中的程序生成是一个重大挑战。

Title: Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models

• Authors: Kaiyan Chang, Yonghao Shi, Chenglong Wang, Hang Zhou, Chi Hu, Xiaoqian Liu, Yingfeng Luo, Yuan Ge, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15512
• Pdf URL: https://arxiv.org/pdf/2507.15512
• Copy Paste: [[2507.15512]] Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models(https://arxiv.org/abs/2507.15512)
• Keywords: language model, llm
• Abstract: Test-Time Scaling (TTS) is a promising approach to progressively elicit the model's intelligence during inference. Recently, training-based TTS methods, such as continued reinforcement learning (RL), have further surged in popularity, while training-free TTS methods are gradually fading from prominence. However, the additional computation overhead of training amplifies the burden on test-time scaling. In this paper, we focus on training-free TTS methods for reasoning. We first design Conditional Step-level Self-refinement, a fine-grained sequential scaling method guided by process verification. On top of its effectiveness, we further combine it with other classical parallel scaling methods at the step level, to introduce a novel inference paradigm called Hybrid Test-Time Scaling. Extensive experiments on five instruction-tuned LLMs across different scales (3B-14B) and families demonstrate that hybrid strategy incorporating various training-free TTS methods at a fine granularity has considerable potential for expanding the reasoning performance boundaries of LLMs.
• 摘要：测试时间缩放（TTS）是在推断过程中逐步引起模型智能的一种有前途的方法。最近，基于培训的TTS方法（例如持续的加固学习（RL））进一步流行，而无培训的TTS方法则逐渐从突出的角度逐渐消失。但是，培训的其他计算开销增加了测试时间缩放的负担。在本文中，我们专注于培训无TTS方法进行推理。我们首先设计有条件的阶梯自我再填充，这是一种通过过程验证引导的细粒顺序缩放方法。除了其有效性之外，我们还将其与步骤级别的其他经典平行缩放方法相结合，以引入一种新型的推理范式，称为混合测试时间缩放。对不同尺度（3B-14B）的五个指导调整的LLM进行了广泛的实验，并表明，在良好的粒度上纳入各种无培训TTS方法的混合策略具有扩大LLMS推理性能界限的巨大潜力。

Title: Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification

• Authors: Vitaly Protasov, Nikolay Babakov, Daryna Dementieva, Alexander Panchenko
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15557
• Pdf URL: https://arxiv.org/pdf/2507.15557
• Copy Paste: [[2507.15557]] Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification(https://arxiv.org/abs/2507.15557)
• Keywords: language model, llm, prompt
• Abstract: Despite recent progress in large language models (LLMs), evaluation of text generation tasks such as text style transfer (TST) remains a significant challenge. Recent studies (Dementieva et al., 2024; Pauli et al., 2025) revealed a substantial gap between automatic metrics and human judgments. Moreover, most prior work focuses exclusively on English, leaving multilingual TST evaluation largely unexplored. In this paper, we perform the first comprehensive multilingual study on evaluation of text detoxification system across nine languages: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, Amharic. Drawing inspiration from the machine translation, we assess the effectiveness of modern neural-based evaluation models alongside prompting-based LLM-as-a-judge approaches. Our findings provide a practical recipe for designing more reliable multilingual TST evaluation pipeline in the text detoxification case.
• 摘要：尽管大型语言模型（LLM）最近取得了进展，但对文本样式转移（TST）等文本生成任务的评估仍然是一个重大挑战。最近的研究（Dementieva等，2024； Pauli等，2025）揭示了自动指标和人类判断之间的巨大差距。此外，大多数先前的工作都专注于英语，而多语言TST评估在很大程度上没有探索。在本文中，我们对跨九种语言的文本排毒系统评估进行了首次全面的多语言研究：英语，西班牙语，德语，中文，阿拉伯语，印地语，乌克兰，乌克兰，俄语，阿姆哈里奇。从机器翻译中汲取灵感，我们评估了基于现代神经的评估模型的有效性，以及促使基于促使的LLM-AS-A-A-Gudge方法。我们的发现提供了一种实用的配方，可以在文本排毒案例中设计更可靠的多语言TST评估管道。

Title: Smart Eyes for Silent Threats: VLMs and In-Context Learning for THz Imaging

• Authors: Nicolas Poggi, Shashank Agnihotri, Margret Keuper
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2507.15576
• Pdf URL: https://arxiv.org/pdf/2507.15576
• Copy Paste: [[2507.15576]] Smart Eyes for Silent Threats: VLMs and In-Context Learning for THz Imaging(https://arxiv.org/abs/2507.15576)
• Keywords: language model, prompt
• Abstract: Terahertz (THz) imaging enables non-invasive analysis for applications such as security screening and material classification, but effective image classification remains challenging due to limited annotations, low resolution, and visual ambiguity. We introduce In-Context Learning (ICL) with Vision-Language Models (VLMs) as a flexible, interpretable alternative that requires no fine-tuning. Using a modality-aligned prompting framework, we adapt two open-weight VLMs to the THz domain and evaluate them under zero-shot and one-shot settings. Our results show that ICL improves classification and interpretability in low-data regimes. This is the first application of ICL-enhanced VLMs to THz imaging, offering a promising direction for resource-constrained scientific domains. Code: \href{this https URL}{GitHub repository}.
• 摘要：Terahertz（THZ）成像可以针对安全筛选和材料分类等应用进行非侵入性分析，但是由于有限的注释，低分辨率和视觉歧义，有效的图像分类仍然具有挑战性。我们将视觉语言模型（VLM）作为一种灵活的，可解释的替代方案介绍了内在学习（ICL），不需要微调。使用模态对准的提示框架，我们将两个开放式VLMS调整到THZ域，并在零射击和一次性设置下对其进行评估。我们的结果表明，ICL改善了低数据制度中的分类和可解释性。这是ICL增强VLM的首次应用到THZ成像，为资源受限的科学领域提供了有希望的方向。代码：\ href {此https url} {github存储库}。

Title: Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation

• Authors: Xinping Zhao, Shouzheng Huang, Yan Zhong, Xinshuo Hu, Baotian Hu, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15586
• Pdf URL: https://arxiv.org/pdf/2507.15586
• Copy Paste: [[2507.15586]] Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation(https://arxiv.org/abs/2507.15586)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) effectively improves the accuracy of Large Language Models (LLMs). However, retrieval noises significantly impact the quality of LLMs' generation, necessitating the development of denoising mechanisms. Previous methods extract evidence straightforwardly without explicit thinking, which risks filtering out key clues and struggles with generalization. To this end, we propose LEAR, which learns to extract rational evidence by (1) explicitly reasoning to identify potential cues within retrieval contents first, and then (2) consciously extracting to avoid omitting any key cues helpful for answering questions. Specifically, we frame evidence reasoning and evidence extraction into one unified response for end-to-end training; apply knowledge token masks for disentanglement to derive reasoning-based and extraction-based answers; and devise three types of verifiable reward functions, including answer, length, and format, to update the model via the policy optimization algorithm. Extensive experiments on three benchmark datasets show the effectiveness of LEAR, providing compact and high-quality evidence, improving the accuracy of downstream tasks, and promoting effective application in online RAG systems.
• 摘要：检索增强的生成（RAG）有效地提高了大语言模型（LLMS）的准确性。但是，检索噪声显着影响了LLMS发电的质量，因此需要开发降解机制。以前的方法在没有明确思考的情况下直接提取证据，这有可能滤除关键线索和泛化的斗争。为此，我们提出了李尔（Lear），该李尔（Lear）学会通过（1）明确推理提取合理的证据，以首先在检索内容中识别潜在的线索，然后（2）有意识地提取以避免省略任何对回答问题有用的关键提示。具体而言，我们将证据推理和证据提取构架为端到端培训的一个统一反应；将知识令牌面具应用于分解，以得出基于推理的基于推理和提取的答案；并设计三种类型的可验证奖励功能，包括答案，长度和格式，以通过策略优化算法更新模型。在三个基准数据集上进行的广泛实验显示了LEAR的有效性，提供紧凑而高质量的证据，提高下游任务的准确性，并促进在线抹布系统中的有效应用。

Title: P3: Prompts Promote Prompting

• Authors: Xinyu Zhang, Yuanquan Hu, Fangchao Liu, Zhicheng Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15675
• Pdf URL: https://arxiv.org/pdf/2507.15675
• Copy Paste: [[2507.15675]] P3: Prompts Promote Prompting(https://arxiv.org/abs/2507.15675)
• Keywords: language model, llm, prompt
• Abstract: Current large language model (LLM) applications often employ multi-component prompts, comprising both system and user prompts, to guide model behaviors. While recent advancements have demonstrated the efficacy of automatically optimizing either the system or user prompt to boost performance, such unilateral approaches often yield suboptimal outcomes due to the interdependent nature of these components. In this work, we introduce P3, a novel self-improvement framework that concurrently optimizes both system and user prompts through an iterative process. The offline optimized prompts are further leveraged to promote online prompting by performing query-dependent prompt optimization. Extensive experiments on general tasks (e.g., Arena-hard and Alpaca-eval) and reasoning tasks (e.g., GSM8K and GPQA) demonstrate that P3 achieves superior performance in the realm of automatic prompt optimization. Our results highlight the effectiveness of a holistic optimization strategy in enhancing LLM performance across diverse domains.
• 摘要：当前的大型语言模型（LLM）应用程序通常采用多组分提示，包括系统和用户提示，以指导模型行为。尽管最近的进步证明了自动优化系统或用户提示以提高性能的功效，但由于这些组件的相互依存性，这种单方面方法通常会产生次优的结果。在这项工作中，我们介绍了P3，这是一个新颖的自我完善框架，通过迭代过程同时优化了系统和用户提示。离线优化的提示可以进一步利用，以通过执行依赖查询的提示优化来促进在线提示。对一般任务（例如Arena-Hard和Apaca-eval）和推理任务（例如GSM8K和GPQA）进行的广泛实验表明，P3在自动及时优化领域中实现了卓越的性能。我们的结果突出了整体优化策略在增强不同领域LLM性能方面的有效性。

Title: CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models

• Authors: Congmin Zheng, Jiachen Zhu, Jianghao Lin, Xinyi Dai, Yong Yu, Weinan Zhang, Mengyue Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15698
• Pdf URL: https://arxiv.org/pdf/2507.15698
• Copy Paste: [[2507.15698]] CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models(https://arxiv.org/abs/2507.15698)
• Keywords: language model, llm
• Abstract: Process Reward Models (PRMs) play a central role in evaluating and guiding multi-step reasoning in large language models (LLMs), especially for mathematical problem solving. However, we identify a pervasive length bias in existing PRMs: they tend to assign higher scores to longer reasoning steps, even when the semantic content and logical validity are unchanged. This bias undermines the reliability of reward predictions and leads to overly verbose outputs during inference. To address this issue, we propose CoLD(Counterfactually-Guided Length Debiasing), a unified framework that mitigates length bias through three components: an explicit length-penalty adjustment, a learned bias estimator trained to capture spurious length-related signals, and a joint training strategy that enforces length-invariance in reward predictions. Our approach is grounded in counterfactual reasoning and informed by causal graph analysis. Extensive experiments on MATH500 and GSM-Plus show that CoLD consistently reduces reward-length correlation, improves accuracy in step selection, and encourages more concise, logically valid reasoning. These results demonstrate the effectiveness and practicality of CoLD in improving the fidelity and robustness of PRMs.
• 摘要：流程奖励模型（PRM）在评估和指导大语言模型（LLMS）的多步推理方面起着核心作用，尤其是对于数学问题解决方案。但是，我们确定了现有PRM的普遍长度偏差：即使语义内容和逻辑有效性不变，它们也倾向于为更长的推理步骤分配更高的分数。这种偏见破坏了奖励预测的可靠性，并导致推断期间过于详细的输出。为了解决这个问题，我们提出了冷（反事实引导的长度借记项），一个统一的框架，通过三个组件减轻长度偏见：明确的长度 - 重度调整，经过培训的学习偏置估计器，培训了用于捕获虚假的长度相关信号的偏置估计器，以及一种在奖励预测中实现长度不相关的关节训练策略。我们的方法基于反事实推理，并通过因果图分析告知。在Math500和GSM-Plus上进行的广泛实验表明，冷始终降低奖励长度的相关性，提高步骤选择的准确性，并鼓励更简洁，逻辑上有效的推理。这些结果证明了冷室在改善PRM的忠诚度和鲁棒性方面的有效性和实用性。

Title: Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?

• Authors: Seok Hwan Song, Mohna Chakraborty, Qi Li, Wallapak Tavanapong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15707
• Pdf URL: https://arxiv.org/pdf/2507.15707
• Copy Paste: [[2507.15707]] Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?(https://arxiv.org/abs/2507.15707)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have been evaluated using diverse question types, e.g., multiple-choice, true/false, and short/long answers. This study answers an unexplored question about the impact of different question types on LLM accuracy on reasoning tasks. We investigate the performance of five LLMs on three different types of questions using quantitative and deductive reasoning tasks. The performance metrics include accuracy in the reasoning steps and choosing the final answer. Key Findings: (1) Significant differences exist in LLM performance across different question types. (2) Reasoning accuracy does not necessarily correlate with the final selection accuracy. (3) The number of options and the choice of words, influence LLM performance.
• 摘要：大型语言模型（LLM）已使用多种问题类型进行评估，例如多项选择，真/错误和短/长答案。这项研究回答了一个关于不同问题类型对LLM准确性对推理任务的影响的未开发问题。我们使用定量和演绎推理任务研究了五个LLM在三种不同类型的问题上的性能。性能指标包括推理步骤中的准确性并选择最终答案。关键发现：（1）不同问题类型的LLM性能中存在显着差异。 （2）推理精度不一定与最终选择精度相关。 （3）选项数量和单词的选择，影响LLM的性能。

Title: Chinchunmei at SemEval-2025 Task 11: Boosting the Large Language Model's Capability of Emotion Perception using Contrastive Learning

• Authors: Tian Li, Yujian Sun, Huizhi Liang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15714
• Pdf URL: https://arxiv.org/pdf/2507.15714
• Copy Paste: [[2507.15714]] Chinchunmei at SemEval-2025 Task 11: Boosting the Large Language Model's Capability of Emotion Perception using Contrastive Learning(https://arxiv.org/abs/2507.15714)
• Keywords: language model
• Abstract: The SemEval-2025 Task 11, Bridging the Gap in Text-Based Emotion Detection, introduces an emotion recognition challenge spanning over 28 languages. This competition encourages researchers to explore more advanced approaches to address the challenges posed by the diversity of emotional expressions and background variations. It features two tracks: multi-label classification (Track A) and emotion intensity prediction (Track B), covering six emotion categories: anger, fear, joy, sadness, surprise, and disgust. In our work, we systematically explore the benefits of two contrastive learning approaches: sample-based (Contrastive Reasoning Calibration) and generation-based (DPO, SimPO) contrastive learning. The sample-based contrastive approach trains the model by comparing two samples to generate more reliable predictions. The generation-based contrastive approach trains the model to differentiate between correct and incorrect generations, refining its prediction. All models are fine-tuned from LLaMa3-Instruct-8B. Our system achieves 9th place in Track A and 6th place in Track B for English, while ranking among the top-tier performing systems for other languages.
• 摘要：Semeval-2025任务11在基于文本的情感检测中弥合了差距，引入了跨越28种语言的情感识别挑战。这项竞争鼓励研究人员探索更高级的方法，以应对情绪表情和背景变化的多样性所带来的挑战。它具有两个曲目：多标签分类（轨道A）和情感强度预测（轨道B），涵盖了六个情感类别：愤怒，恐惧，喜悦，悲伤，惊喜和厌恶。在我们的工作中，我们系统地探讨了两种对比学习方法的好处：基于样本的（对比度推理校准）和基于生成的（DPO，SIMPO）对比度学习。基于样本的对比方法通过比较两个样品来产生更可靠的预测来训练模型。基于世代的对比方法训练该模型以区分正确和不正确的世代，从而完善其预测。所有型号均从Llama3-Instruct-8B进行微调。我们的系统在轨道A中获得第9位，在曲目B中获得英语的第六名，同时排名在其他语言的顶级表演系统中。

Title: From Queries to Criteria: Understanding How Astronomers Evaluate LLMs

• Authors: Alina Hyk, Kiera McCormick, Mian Zhong, Ioana Ciucă, Sanjib Sharma, John F Wu, J. E. G. Peek, Kartheik G. Iyer, Ziang Xiao, Anjalie Field
• Subjects: cs.CL, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2507.15715
• Pdf URL: https://arxiv.org/pdf/2507.15715
• Copy Paste: [[2507.15715]] From Queries to Criteria: Understanding How Astronomers Evaluate LLMs(https://arxiv.org/abs/2507.15715)
• Keywords: llm, retrieval-augmented generation
• Abstract: There is growing interest in leveraging LLMs to aid in astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingly diverse ways that real people evaluate and use these models. In this study, we seek to improve evaluation procedures by building an understanding of how users evaluate LLMs. We focus on a particular use case: an LLM-powered retrieval-augmented generation bot for engaging with astronomical literature, which we deployed via Slack. Our inductive coding of 368 queries to the bot over four weeks and our follow-up interviews with 11 astronomers reveal how humans evaluated this system, including the types of questions asked and the criteria for judging responses. We synthesize our findings into concrete recommendations for building better benchmarks, which we then employ in constructing a sample benchmark for evaluating LLMs for astronomy. Overall, our work offers ways to improve LLM evaluation and ultimately usability, particularly for use in scientific research.
• 摘要：利用LLMS来帮助天文学和其他科学研究的兴趣越来越大，但是LLM评估的基准总体上并没有跟上真正的人评估和使用这些模型的越来越多样化的方式。在这项研究中，我们试图通过了解用户评估LLM的理解来改善评估程序。我们专注于一种特定的用例：通过Slack部署的天文学文献参与天文学文献，以LLM驱动的检索型生成机器人进行。我们在四个星期内对机器人进行了368个疑问的归纳编码，我们对111个天文学家的后续采访揭示了人类如何评估该系统，包括提出的问题类型和判断回答的标准。我们将我们的发现综合为具体建议，以构建更好的基准，然后我们在构建样品基准测试中用于评估天文学的LLM。总体而言，我们的工作提供了改善LLM评估和最终可用性的方法，尤其是用于科学研究。

Title: BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning

• Authors: Sahana Srinivasan, Xuguang Ai, Thaddaeus Wai Soon Lo, Aidan Gilson, Minjie Zou, Ke Zou, Hyunjae Kim, Mingjia Yang, Krithi Pushpanathan, Samantha Yew, Wan Ting Loke, Jocelyn Goh, Yibing Chen, Yiming Kong, Emily Yuelei Fu, Michelle Ongyong Hui, Kristen Nwanyanwu, Amisha Dave, Kelvin Zhenghao Li, Chen-Hsin Sun, Mark Chia, Gabriel Dawei Yang, Wendy Meihua Wong, David Ziyou Chen, Dianbo Liu, Maxwell Singer, Fares Antaki, Lucian V Del Priore, Jost Jonas, Ron Adelman, Qingyu Chen, Yih-Chung Tham
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15717
• Pdf URL: https://arxiv.org/pdf/2507.15717
• Copy Paste: [[2507.15717]] BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning(https://arxiv.org/abs/2507.15717)
• Keywords: language model, gpt, llm
• Abstract: Current benchmarks evaluating large language models (LLMs) in ophthalmology are limited in scope and disproportionately prioritise accuracy. We introduce BELO (BEnchmarking LLMs for Ophthalmology), a standardized and comprehensive evaluation benchmark developed through multiple rounds of expert checking by 13 ophthalmologists. BELO assesses ophthalmology-related clinical accuracy and reasoning quality. Using keyword matching and a fine-tuned PubMedBERT model, we curated ophthalmology-specific multiple-choice-questions (MCQs) from diverse medical datasets (BCSC, MedMCQA, MedQA, BioASQ, and PubMedQA). The dataset underwent multiple rounds of expert checking. Duplicate and substandard questions were systematically removed. Ten ophthalmologists refined the explanations of each MCQ's correct answer. This was further adjudicated by three senior ophthalmologists. To illustrate BELO's utility, we evaluated six LLMs (OpenAI o1, o3-mini, GPT-4o, DeepSeek-R1, Llama-3-8B, and Gemini 1.5 Pro) using accuracy, macro-F1, and five text-generation metrics (ROUGE-L, BERTScore, BARTScore, METEOR, and AlignScore). In a further evaluation involving human experts, two ophthalmologists qualitatively reviewed 50 randomly selected outputs for accuracy, comprehensiveness, and completeness. BELO consists of 900 high-quality, expert-reviewed questions aggregated from five sources: BCSC (260), BioASQ (10), MedMCQA (572), MedQA (40), and PubMedQA (18). A public leaderboard has been established to promote transparent evaluation and reporting. Importantly, the BELO dataset will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models.
• 摘要：评估大型语言模型（LLMS）在眼科中的当前基准测试范围有限，并且优先考虑准确性的优先级不成比例。我们介绍了Belo（Ophthalmology的基准LLMS），这是一个通过13个眼科医生的多轮专家检查开发的标准化和全面的评估基准。 Belo评估了与眼科相关的临床准确性和推理质量。使用关键字匹配和微型PubMedbert模型，我们策划了来自多个医疗数据集（BCSC，MEDMCQA，MEDQA，MEDQA，BIOASQ和PubMedQA）的眼科特定多项选择问题（MCQ）。该数据集进行了多轮专家检查。重复和不合格的问题被系统地删除。十位眼科医生完善了每个MCQ正确答案的解释。这是由三位高级眼科医生进一步裁定的。为了说明Belo的实用程序，我们使用准确性，宏F1和五个文本生成代码（Rouge-l，Bertscore，Bartscore，Bartscore，Bartscore，Bartscore，bartscore，bartscore，bartscore，bartscore，abartscore，bartscore，bartscore，Ateer and aLignsscore）评估了六个LLM（OpenAI O1，O1，O3，GPT-4O，DeepSeek-R1，Llama-3-8B和Gemini 1.5 Pro）。在涉及人类专家的进一步评估中，两位眼科医生定性地回顾了50个随机选择的输出，以获得准确性，全面性和完整性。 BELO由900个高质量的专家评审问题组成，这些问题汇总了五个来源：BCSC（260），BioASQ（10），MEDMCQA（572）（572），MEDQA（40）和PubMedQA（18）。已经建立了公共排行榜，以促进透明的评估和报告。重要的是，BELO数据集将仍然是一个仅限的，仅评估的基准，以确保对未来模型的公平和可重复的比较。

Title: Understanding Large Language Models' Ability on Interdisciplinary Research

• Authors: Yuanhao Shen, Daniel Xavier de Sousa, Ricardo Marçal, Ali Asad, Hongyu Guo, Xiaodan Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15736
• Pdf URL: https://arxiv.org/pdf/2507.15736
• Copy Paste: [[2507.15736]] Understanding Large Language Models' Ability on Interdisciplinary Research(https://arxiv.org/abs/2507.15736)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have revealed their impressive ability to perform multi-step, logic-driven reasoning across complex domains, positioning them as powerful tools and collaborators in scientific discovery while challenging the long-held view that inspiration-driven ideation is uniquely human. However, the lack of a dedicated benchmark that evaluates LLMs' ability to develop ideas in Interdisciplinary Research (IDR) settings poses a critical barrier to fully understanding their strengths and limitations. To address this gap, we introduce IDRBench -- a pioneering benchmark featuring an expert annotated dataset and a suite of tasks tailored to evaluate LLMs' capabilities in proposing valuable research ideas from different scientific domains for interdisciplinary research. This benchmark aims to provide a systematic framework for assessing LLM performance in complex, cross-domain scientific research. Our dataset consists of scientific publications sourced from the ArXiv platform covering six distinct disciplines, and is annotated by domain experts with diverse academic backgrounds. To ensure high-quality annotations, we emphasize clearly defined dimensions that characterize authentic interdisciplinary research. The design of evaluation tasks in IDRBench follows a progressive, real-world perspective, reflecting the natural stages of interdisciplinary research development, including 1) IDR Paper Identification, 2) IDR Idea Integration, and 3) IDR Idea Recommendation. Using IDRBench, we construct baselines across 10 LLMs and observe that despite fostering some level of IDR awareness, LLMs still struggle to produce quality IDR ideas. These findings could not only spark new research directions, but also help to develop next-generation LLMs that excel in interdisciplinary research.
• 摘要：大型语言模型（LLM）的最新进展揭示了他们在复杂领域进行多步，逻辑驱动的推理的令人印象深刻的能力，将它们定位为科学发现中的强大工具和合作者，同时挑战了长期以来的观点，即灵感驱动的构想是独特的。但是，缺乏评估LLMS在跨学科研究（IDR）设置中发展思想的能力的专用基准标准，这为充分理解其优势和局限性带来了关键的障碍。为了解决这一差距，我们介绍了IDRBENCH-一个开创性的基准测试，该基准为专家注释的数据集和一套量身定制的任务，旨在评估LLMS在提出来自不同科学领域的宝贵研究思想的能力，以供跨学科研究。该基准旨在提供一个系统的框架，以评估复杂的跨域科学研究中的LLM性能。我们的数据集由涵盖六个不同学科的Arxiv平台中的科学出版物组成，并由具有不同学术背景的领域专家注释。为了确保高质量的注释，我们强调了表征真实跨学科研究的明确定义的维度。 IDRBENCH中评估任务的设计遵循了一个进步的现实世界的观点，反映了跨学科研究开发的自然阶段，包括1）IDR纸质识别，2）IDR Ideas Integration和3）IDR Ideas建议。使用IDRBENCH，我们在10个LLMS上构建了基线，并观察到，尽管提高了IDR意识，但LLMS仍在努力产生高质量的IDR想法。这些发现不仅可以引发新的研究方向，而且还可以帮助开发跨学科研究表现出色的下一代LLM。

Title: DialogueForge: LLM Simulation of Human-Chatbot Dialogue

• Authors: Ruizhe Zhu, Hao Zhu, Yaxuan Li, Syang Zhou, Shijing Cai, Malgorzata Lazuka, Elliott Ash
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.15752
• Pdf URL: https://arxiv.org/pdf/2507.15752
• Copy Paste: [[2507.15752]] DialogueForge: LLM Simulation of Human-Chatbot Dialogue(https://arxiv.org/abs/2507.15752)
• Keywords: gpt, llm, prompt, chat
• Abstract: Collecting human-chatbot dialogues typically demands substantial manual effort and is time-consuming, which limits and poses challenges for research on conversational AI. In this work, we propose DialogueForge - a framework for generating AI-simulated conversations in human-chatbot style. To initialize each generated conversation, DialogueForge uses seed prompts extracted from real human-chatbot interactions. We test a variety of LLMs to simulate the human chatbot user, ranging from state-of-the-art proprietary models to small-scale open-source LLMs, and generate multi-turn dialogues tailored to specific tasks. In addition, we explore fine-tuning techniques to enhance the ability of smaller models to produce indistinguishable human-like dialogues. We evaluate the quality of the simulated conversations and compare different models using the UniEval and GTEval evaluation protocols. Our experiments show that large proprietary models (e.g., GPT-4o) generally outperform others in generating more realistic dialogues, while smaller open-source models (e.g., Llama, Mistral) offer promising performance with greater customization. We demonstrate that the performance of smaller models can be significantly improved by employing supervised fine-tuning techniques. Nevertheless, maintaining coherent and natural long-form human-like dialogues remains a common challenge across all models.
• 摘要：收集人与聊天对话通常需要大量的手动努力，并且耗时，这限制了对话性AI的研究挑战。在这项工作中，我们提出了对话forgegoge-一种以人为chatbot风格生成AI模拟的对话的框架。为了初始化每个生成的对话，对话Forge使用从真实的人类聊天互动中提取的种子提示。我们测试了各种LLM，以模拟人类聊天机器人用户，从最先进的专有型号到小规模的开源LLM，并生成针对特定任务量身定制的多转口对话。此外，我们探索了微调技术，以增强较小模型产生无法区分的人类对话的能力。我们评估模拟对话的质量，并使用Unieval和GTEVAL评估协议比较不同的模型。我们的实验表明，大型专有模型（例如GPT-4O）通常在生成更现实的对话方面优于其他模型，而较小的开源模型（例如Llama，Mistral）提供了有希望的性能，并具有更大的自定义。我们证明，通过采用监督的微调技术，可以显着提高较小模型的性能。然而，在所有模型中，保持连贯和自然的长形式人类对话仍然是一个普遍的挑战。

Title: Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR

• Authors: Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, Guorui Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15778
• Pdf URL: https://arxiv.org/pdf/2507.15778
• Copy Paste: [[2507.15778]] Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR(https://arxiv.org/abs/2507.15778)
• Keywords: language model, llm
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training method for improving the reasoning abilities of Large Language Models (LLMs), mainly by shaping higher-order behaviors such as reflection and planning. However, previous RLVR algorithms often apply uniform training signals to all tokens, without considering the different roles of low-entropy knowledge-related tokens and high-entropy reasoning-related tokens. Some recent methods try to separate these token types by gradient masking or asynchronous updates, but these approaches may break semantic dependencies in the model output and hinder effective learning. In this work, we propose Archer, an entropy-aware RLVR approach with dual-token constraints and synchronous updates. Specifically, our method applies weaker KL regularization and higher clipping thresholds to reasoning tokens to encourage exploration, while using stronger constraints on knowledge tokens to maintain factual knowledge. Experimental results on several mathematical reasoning and code generation benchmarks show that our approach significantly outperforms previous RLVR methods, reaching or exceeding state-of-the-art performance among models of comparable size. The code is available at this https URL.
• 摘要：具有可验证奖励（RLVR）的强化学习已成为提高大语模型（LLMS）推理能力的有效培训方法，主要是通过塑造诸如反思和计划之类的高阶行为。但是，以前的RLVR算法通常在所有令牌上应用统一的训练信号，而无需考虑与低渗透知识相关的令牌和与高渗透性相关的代币的不同作用。一些最近的方法试图通过梯度掩盖或异步更新来分开这些令牌类型，但是这些方法可能会破坏模型输出中的语义依赖性，并阻碍有效的学习。在这项工作中，我们提出了Archer，Archer是一种具有双重约束和同步更新的熵感知的RLVR方法。具体而言，我们的方法将较弱的KL正则化和较高的剪辑阈值应用于推理令牌以鼓励探索，同时使用知识令牌的更强限制来维持事实知识。几个数学推理和代码生成基准的实验结果表明，我们的方法在相当大小的模型中达到或超过先前的RLVR方法，达到或超过最先进的性能。该代码可在此HTTPS URL上找到。

Title: Reservoir Computing as a Language Model

• Authors: Felix Köster, Atsushi Uchida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15779
• Pdf URL: https://arxiv.org/pdf/2507.15779
• Copy Paste: [[2507.15779]] Reservoir Computing as a Language Model(https://arxiv.org/abs/2507.15779)
• Keywords: language model, llm
• Abstract: Large Language Models (LLM) have dominated the science and media landscape duo to their impressive performance on processing large chunks of data and produce human-like levels of text. Nevertheless, their huge energy demand and slow processing still a bottleneck for further increasing quality while also making the models accessible to everyone. To solve this bottleneck, we will investigate how reservoir computing performs on natural text processing, which could enable fast and energy efficient hardware implementations. Studies investigating the use of reservoir computing as a language model remain sparse. In this paper, we compare three distinct approaches for character-level language modeling, two different reservoir computing approaches, where only an output layer is trainable, and the well-known transformer-based architectures, which fully learn an attention-based sequence representation. We explore the performance, computational cost and prediction accuracy for both paradigms by equally varying the number of trainable parameters for all models. Using a consistent pipeline for all three approaches, we demonstrate that transformers excel in prediction quality, whereas reservoir computers remain highly efficient reducing the training and inference speed. Furthermore, we investigate two types of reservoir computing: a traditional reservoir with a static linear readout, and an attention-enhanced reservoir that dynamically adapts its output weights via an attention mechanism. Our findings underline how these paradigms scale and offer guidelines to balance resource constraints with performance.
• 摘要：大型语言模型（LLM）主导了科学和媒体景观二人组的令人印象深刻的表现，以处理大量数据并产生类似人类的文本水平。然而，他们巨大的能源需求和缓慢的处理仍然是一种瓶颈，以进一步提高质量，同时也使每个人都可以使用这些模型。为了解决此瓶颈，我们将研究储层计算在自然文本处理上的执行方式，这可以实现快速，节能的硬件实现。研究使用储层计算作为语言模型的研究仍然很少。在本文中，我们比较了角色级语言建模的三种不同方法，两种不同的储层计算方法，其中只有一个输出层是可以训练的，以及众所周知的基于变压器的架构，这些架构完全学习了基于注意力的序列表示。我们通过同样改变所有模型的可训练参数的数量来探讨这两种范式的性能，计算成本和预测准确性。使用一致的所有三种方法的管道，我们证明了变压器在预测质量方面表现出色，而储层计算机仍然高效地降低了训练和推理速度。此外，我们研究了两种类型的储层计算：一种带有静态线性读数的传统储层，以及一种通过注意机制动态调整其输出权重的注意力增强的储层。我们的发现强调了这些范式如何扩展并提供指南，以平衡资源约束与绩效。

Title: The Impact of Language Mixing on Bilingual LLM Reasoning

• Authors: Yihao Li, Jiayi Xin, Miranda Muqing Miao, Qi Long, Lyle Ungar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.15849
• Pdf URL: https://arxiv.org/pdf/2507.15849
• Copy Paste: [[2507.15849]] The Impact of Language Mixing on Bilingual LLM Reasoning(https://arxiv.org/abs/2507.15849)
• Keywords: language model, llm
• Abstract: Proficient multilingual speakers often intentionally switch languages in the middle of a conversation. Similarly, recent reasoning-focused bilingual large language models (LLMs) with strong capabilities in both languages exhibit language mixing--alternating languages within their chain of thought. Discouraging this behavior in DeepSeek-R1 was found to degrade accuracy, suggesting that language mixing may benefit reasoning. In this work, we study language switching in Chinese-English bilingual reasoning models. We identify reinforcement learning with verifiable rewards (RLVR) as the critical training stage that leads to language mixing. We demonstrate that language mixing can enhance reasoning: enforcing monolingual decoding reduces accuracy by 5.6 percentage points on math reasoning tasks. Additionally, a lightweight probe can be trained to predict whether a potential language switch would benefit or harm reasoning, and when used to guide decoding, increases accuracy by up to 6.25 percentage points. Our findings suggest that language mixing is not merely a byproduct of multilingual training, but is a strategic reasoning behavior.
• 摘要：熟练的多语言扬声器通常在对话中有意切换语言。同样，最近以推理为重点的双语大型语言模型（LLM）具有强大的两种语言能力，都表现出语言混合 - 其思想链中的替代语言。发现在DeepSeek-R1中劝阻这种行为会降低准确性，这表明语言混合可能会受益。在这项工作中，我们研究了中文英语双语推理模型的语言转换。我们确定具有可验证奖励（RLVR）的强化学习是导致语言混合的关键训练阶段。我们证明语言混合可以增强推理：实施单语言解码可在数学推理任务上降低准确性5.6个百分点。此外，可以训练轻量级的探针，以预测潜在的语言转换是受益还是损害推理，并且在用于指导解码的情况下，将准确性提高到6.25个百分点。我们的发现表明，语言混合不仅是多语言培训的副产品，而且是一种战略推理行为。

Title: 3LM: Bridging Arabic, STEM, and Code through Benchmarking

• Authors: Basma El Amel Boussaha, Leen AlQadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, Hakim Hacid
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.15850
• Pdf URL: https://arxiv.org/pdf/2507.15850
• Copy Paste: [[2507.15850]] 3LM: Bridging Arabic, STEM, and Code through Benchmarking(https://arxiv.org/abs/2507.15850)
• Keywords: language model, llm
• Abstract: Arabic is one of the most widely spoken languages in the world, yet efforts to develop and evaluate Large Language Models (LLMs) for Arabic remain relatively limited. Most existing Arabic benchmarks focus on linguistic, cultural, or religious content, leaving a significant gap in domains like STEM and code which are increasingly relevant for real-world LLM applications. To help bridge this gap, we present 3LM, a suite of three benchmarks designed specifically for Arabic. The first is a set of STEM-related question-answer pairs, naturally sourced from Arabic textbooks and educational worksheets. The second consists of synthetically generated STEM questions, created using the same sources. The third benchmark focuses on code generation, built through a careful translation of two widely used code benchmarks, incorporating a human-in-the-loop process with several rounds of review to ensure high-quality and faithful translations. We release all three benchmarks publicly to support the growth of Arabic LLM research in these essential but underrepresented areas.
• 摘要：阿拉伯语是世界上口语最广泛的语言之一，但是为阿拉伯语开发和评估大型语言模型（LLM）的努力仍然相对有限。大多数现有的阿拉伯语基准都集中在语言，文化或宗教内容上，在诸如STEM和代码之类的领域中留下了很大的差距，这些差距与现实世界中的LLM应用越来越重要。为了帮助弥合这一差距，我们提出了3LM，这是专门为阿拉伯语设计的三个基准的套件。第一个是一组与STEM相关的问答对，自然来自阿拉伯语教科书和教育工作表。第二个由使用相同来源创建的合成生成的STEM问题组成。第三个基准的重点是代码生成，这是通过对两个广泛使用的代码基准进行仔细翻译而构建的，将人类的过程结合了几轮审查，以确保高质量和忠实的翻译。我们公开发布所有三个基准，以支持这些基本但代表性不足的地区的阿拉伯LLM研究的增长。