2025-03-19

Title: Ensemble Learning for Large Language Models in Text and Code Generation: A Survey

• Authors: Mari Ashiga, Wei Jie, Fan Wu, Vardan Voskanyan, Fateme Dinmohammadi, Paul Brookes, Jingzhi Gong, Zheng Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.13505
• Pdf URL: https://arxiv.org/pdf/2503.13505
• Copy Paste: [[2503.13505]] Ensemble Learning for Large Language Models in Text and Code Generation: A Survey(https://arxiv.org/abs/2503.13505)
• Keywords: language model, gpt, llm
• Abstract: Generative pretrained transformers (GPT) are the common large language models (LLMs) used for generating text from natural language inputs. However, the fixed properties of language parameters in individual LLMs can lead to inconsistencies in the generated outputs. This limitation also restricts the models' ability to represent diverse language patterns due to inherent biases. Moreover, many powerful LLMs are closed-source. This prevents organizations from integrating their data into these systems, raising concerns about data privacy and limiting industry applications. Inspired by the successful application of LLM ensemble models in text generation, recent literature has also investigated their potential in code generation. This article reviews these emerging LLM ensemble approaches. Our goal is to enhance readers' understanding of existing techniques and encourage further research and practical implementation, aiming to expand the real-world applications of LLM ensemble models in both text and code generation. We categorize these approaches into seven main methods: weight merging, knowledge fusion, mixture of experts, reward ensemble, output ensemble, routing, and cascading. From this list, we focus on four methods and models that show strong performance and potential for broader applications. We analyze their modeling steps, training methods, and output features to provide a clear understanding of their capabilities. Our findings highlight the benefits of LLM ensemble techniques. These include better representation of diversity, improved output quality, and greater flexibility in applications. This information offers valuable insights for selecting models for various real-world tasks involving text and code generation, and potentially applying methods to multimodal LLMs.
• 摘要：生成预审慎的变压器（GPT）是用于从自然语言输入中生成文本的常见大型语言模型（LLMS）。但是，单个LLMS中语言参数的固定属性可能导致生成的输出中的不一致。该限制还限制了模型由于固有的偏见而代表各种语言模式的能力。此外，许多强大的LLM都是封闭的。这样可以防止组织将其数据集成到这些系统中，从而引起人们对数据隐私和限制行业应用程序的担忧。受LLM集成模型在文本生成中的成功应用的启发，最近的文献还研究了它们在代码生成中的潜力。本文回顾了这些新兴的LLM合奏方法。我们的目标是增强读者对现有技术的理解，并鼓励进一步的研究和实际实施，旨在扩大文本和代码生成中LLM集成模型的现实应用。我们将这些方法分为七种主要方法：重量合并，知识融合，专家的混合，奖励合奏，输出合奏，路由和级联。从此列表中，我们专注于四种方法和模型，这些方法和模型表现出强大的性能和更广泛的应用的潜力。我们分析了他们的建模步骤，培训方法和输出功能，以清楚地了解其功能。我们的发现突出了LLM合奏技术的好处。这些包括更好地表示多样性，提高的产出质量以及应用程序的灵活性。该信息为选择涉及文本和代码生成的各种现实世界任务的模型提供了宝贵的见解，并可能将方法应用于多模式LLMS。

Title: NeurIPS 2023 LLM Efficiency Fine-tuning Competition

• Authors: Mark Saroufim, Yotam Perlitz, Leshem Choshen, Luca Antiga, Greg Bowyer, Christian Puhrsch, Driss Guessous, Supriya Rao, Geeta Chauhan, Ashvini Kumar, Jindal Pawan Kumar, Rajpoot Ankur Parikh, Joe Isaacson, Weiwei Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13507
• Pdf URL: https://arxiv.org/pdf/2503.13507
• Copy Paste: [[2503.13507]] NeurIPS 2023 LLM Efficiency Fine-tuning Competition(https://arxiv.org/abs/2503.13507)
• Keywords: language model, llm
• Abstract: Our analysis of the NeurIPS 2023 large language model (LLM) fine-tuning competition revealed the following trend: top-performing models exhibit significant overfitting on benchmark datasets, mirroring the broader issue of benchmark overfitting on popular leaderboards and that data curation is essential in order to get a high performing LLM. The competition, which consisted of two stages - an open evaluation stage with publicly available tasks and a closed evaluation stage with unseen tasks - allowed us to assess the generalizability of fine-tuned LLMs. Our results highlight the limitations of current benchmark-based evaluation schemes for generative models and demonstrate the need for more robust evaluation methods. Notably, the winning submissions utilized standard open-source libraries and focused primarily on data curation. To facilitate further research and promote reproducibility, we release all competition entries, Docker files, and evaluation infrastructure, providing a valuable resource for the community to explore fine-tuning, overfitting, and reproducibility in LLMs.
• 摘要：我们对神经2023大语言模型（LLM）微调竞赛的分析揭示了以下趋势：表现最好的模型在基准数据集中表现出明显的过度拟合，反映出在受欢迎的排行榜上更广泛的基准过度问题，并且该数据策划对于获得高表现LLM是必不可少的。该竞赛由两个阶段组成 - 一个开放评估阶段，具有公开可用的任务，并具有看不见的任务的封闭式评估阶段 - 使我们能够评估微调LLMS的普遍性。我们的结果突出了当前基于基准的评估方案对生成模型的局限性，并证明了需要更强大的评估方法。值得注意的是，获奖提交的提交利用了标准的开源库，主要集中在数据策划上。为了促进进一步的研究和促进可重复性，我们发布了所有竞争条目，Docker文件和评估基础架构，为社区提供了探索LLM中微调，过度拟合和可重复性的宝贵资源。

Title: It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education

• Authors: Shrutika Singh, Anton Alyakin, Daniel Alexander Alber, Jaden Stryker, Ai Phuong S Tong, Karl Sangwon, Nicolas Goff, Mathew de la Paz, Miguel Hernandez-Rovira, Ki Yun Park, Eric Claude Leuthardt, Eric Karl Oermann
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.13508
• Pdf URL: https://arxiv.org/pdf/2503.13508
• Copy Paste: [[2503.13508]] It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education(https://arxiv.org/abs/2503.13508)
• Keywords: language model, gpt, llm
• Abstract: The performance of Large Language Models (LLMs) on multiple-choice question (MCQ) benchmarks is frequently cited as proof of their medical capabilities. We hypothesized that LLM performance on medical MCQs may in part be illusory and driven by factors beyond medical content knowledge and reasoning capabilities. To assess this, we created a novel benchmark of free-response questions with paired MCQs (FreeMedQA). Using this benchmark, we evaluated three state-of-the-art LLMs (GPT-4o, GPT-3.5, and LLama-3-70B-instruct) and found an average absolute deterioration of 39.43% in performance on free-response questions relative to multiple-choice (p = 1.3 * 10-5) which was greater than the human performance decline of 22.29%. To isolate the role of the MCQ format on performance, we performed a masking study, iteratively masking out parts of the question stem. At 100% masking, the average LLM multiple-choice performance was 6.70% greater than random chance (p = 0.002) with one LLM (GPT-4o) obtaining an accuracy of 37.34%. Notably, for all LLMs the free-response performance was near zero. Our results highlight the shortcomings in medical MCQ benchmarks for overestimating the capabilities of LLMs in medicine, and, broadly, the potential for improving both human and machine assessments using LLM-evaluated free-response questions.
• 摘要：大型语言模型（LLM）在多项选择问题（MCQ）基准测试中的性能经常被认为是其医疗功能的证明。我们假设LLM在医疗MCQ上的表现可能部分是虚幻的，并且是由医学内容知识和推理能力以外的因素驱动的。为了评估这一点，我们通过配对MCQ（FreemedQA）创建了一个新颖的自由响应问题的基准。使用此基准，我们评估了三个最先进的LLM（GPT-4O，GPT-3.5和Llama-3-70b-Instruct），并发现相对于多个选择的自由回答问题的平均绝对恶化为39.43％，而自由回应问题的平均绝对恶化（p = 1.3 * 10-5）（P = 1.3 * 10-5），这比人类绩效降低了22.29％的降低。为了隔离MCQ格式在性能中的作用，我们进行了一项掩盖研究，迭代地掩盖了问题的一部分。在100％掩盖时，平均LLM多项选择性性能比随机机会（p = 0.002）高6.70％（p = 0.002），一个LLM（GPT-4O）的准确性为37.34％。值得注意的是，对于所有LLM，自由响应性能接近零。我们的结果强调了医学MCQ基准的缺点，即高估了LLM在医学中的能力，并且从广义上讲，使用LLM评估的自由回答问题来改善人类和机器评估的潜力。

Title: Prompt Sentiment: The Catalyst for LLM Change

• Authors: Vishal Gandhi, Sagar Gandhi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13510
• Pdf URL: https://arxiv.org/pdf/2503.13510
• Copy Paste: [[2503.13510]] Prompt Sentiment: The Catalyst for LLM Change(https://arxiv.org/abs/2503.13510)
• Keywords: language model, gpt, llm, prompt
• Abstract: The rise of large language models (LLMs) has revolutionized natural language processing (NLP), yet the influence of prompt sentiment, a latent affective characteristic of input text, remains underexplored. This study systematically examines how sentiment variations in prompts affect LLM-generated outputs in terms of coherence, factuality, and bias. Leveraging both lexicon-based and transformer-based sentiment analysis methods, we categorize prompts and evaluate responses from five leading LLMs: Claude, DeepSeek, GPT-4, Gemini, and LLaMA. Our analysis spans six AI-driven applications, including content generation, conversational AI, legal and financial analysis, healthcare AI, creative writing, and technical documentation. By transforming prompts, we assess their impact on output quality. Our findings reveal that prompt sentiment significantly influences model responses, with negative prompts often reducing factual accuracy and amplifying bias, while positive prompts tend to increase verbosity and sentiment propagation. These results highlight the importance of sentiment-aware prompt engineering for ensuring fair and reliable AI-generated content.
• 摘要：大型语言模型（LLM）的兴起已经彻底改变了自然语言处理（NLP），但是迅速情感的影响是输入文本的潜在情感特征，仍然没有被逐渐解散。这项研究系统地研究了提示中的情感变化如何影响LLM生成的产出，从而在相干，事实和偏见方面影响。利用基于词典和基于变压器的情感分析方法，我们对提示进行了分类并评估五个领先的LLM的响应：Claude，DeepSeek，GPT-4，Gemini和Llama。我们的分析涵盖了六个AI驱动的应用程序，包括内容生成，对话AI，法律和财务分析，医疗保健AI，创意写作和技术文档。通过转换提示，我们评估它们对产出质量的影响。我们的发现表明，迅速情绪会显着影响模型的反应，负面提示通常会降低事实准确性和放大偏见，而正提示则倾向于增加详细的和情感的传播。这些结果强调了情感意识及时工程的重要性，以确保公平可靠的AI生成的内容。

Title: RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integration

• Authors: Hong Qing Yu (University of Derby), Frank McQuade (Bloc Digital)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13514
• Pdf URL: https://arxiv.org/pdf/2503.13514
• Copy Paste: [[2503.13514]] RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integration(https://arxiv.org/abs/2503.13514)
• Keywords: language model, gpt, llm, hallucination, retrieval-augmented generation, agent
• Abstract: This paper presents RAG-KG-IL, a novel multi-agent hybrid framework designed to enhance the reasoning capabilities of Large Language Models (LLMs) by integrating Retrieval-Augmented Generation (RAG) and Knowledge Graphs (KGs) with an Incremental Learning (IL) approach. Despite recent advancements, LLMs still face significant challenges in reasoning with structured data, handling dynamic knowledge evolution, and mitigating hallucinations, particularly in mission-critical domains. Our proposed RAG-KG-IL framework addresses these limitations by employing a multi-agent architecture that enables continuous knowledge updates, integrates structured knowledge, and incorporates autonomous agents for enhanced explainability and reasoning. The framework utilizes RAG to ensure the generated responses are grounded in verifiable information, while KGs provide structured domain knowledge for improved consistency and depth of understanding. The Incremental Learning approach allows for dynamic updates to the knowledge base without full retraining, significantly reducing computational overhead and improving the model's adaptability. We evaluate the framework using real-world case studies involving health-related queries, comparing it to state-of-the-art models like GPT-4o and a RAG-only baseline. Experimental results demonstrate that our approach significantly reduces hallucination rates and improves answer completeness and reasoning accuracy. The results underscore the potential of combining RAG, KGs, and multi-agent systems to create intelligent, adaptable systems capable of real-time knowledge integration and reasoning in complex domains.
• 摘要：本文介绍了RAG-KG-IL，这是一种新型的多代理混合框架，旨在通过将检索功能（RAG）和知识图（kgs）与增量学习（IL）方法整合到大语模型（LLMS）的推理能力（LLMS）。尽管有最近的进步，但LLMS在结构化数据，处理动态知识演变以及缓解幻觉的推理方面仍然面临重大挑战，尤其是在关键任务领域。我们提出的RAG-KG-IL框架通过采用一个多代理体系结构来解决这些局限性，该架构可以实现持续的知识更新，整合结构化知识，并结合了自主的代理，以增强解释性和推理。该框架利用抹布来确保生成的响应基于可验证的信息，而KG则提供结构化的域知识，以提高一致性和理解深度。增量学习方法允许在不完全重新培训的情况下对知识库进行动态更新，从而大大降低计算开销并改善模型的适应性。我们使用涉及与健康相关的查询的现实案例研究评估了框架，将其与GPT-4O和仅使用抹布的基线等最新模型进行了比较。实验结果表明，我们的方法大大降低了幻觉率，并提高了答案的完整性和推理准确性。结果强调了组合抹布，千克和多代理系统的潜力，以创建能够在复杂域中实时知识集成和推理的智能，适应性的系统。

Title: CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning

• Authors: Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Norgaard, Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, Yasaman Bahri, Victor V. Albert, Pranesh Srinivasan, Haining Pan, Philippe Faist, Brian Rohr, Michael J. Statt, Dan Morris, Drew Purves, Elise Kleeman, Ruth Alcantara, Matthew Abraham, Muqthar Mohammad, Ean Phing VanLee, Chenfei Jiang, Elizabeth Dorfman, Eun-Ah Kim, Michael P Brenner, Viren Jain, Sameera Ponda, Subhashini Venugopalan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13517
• Pdf URL: https://arxiv.org/pdf/2503.13517
• Copy Paste: [[2503.13517]] CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning(https://arxiv.org/abs/2503.13517)
• Keywords: language model, gpt, llm, long context
• Abstract: Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Evaluation code and data are in this https URL
• 摘要：解决问题的科学问题涉及在应用专家知识的同时综合信息。我们介绍了Curie，这是一种科学的长篇文化理解，推理和信息提取基准，以衡量大语模型（LLM）在科学问题解决和协助科学家进行现实工作流程中的潜力。该基准介绍了十项具有挑战性的任务，共有580个问题，解决方案对由六个学科的专家组成 - 材料科学，凝结物理学，量子计算，地理空间分析，生物多样性和蛋白质 - 涵盖了科学领域的实验性工作和理论工作。我们在Curie的任务上评估了一系列封闭式和开放的LLM，这些任务需要域专业知识，对长篇文章信息的理解以及多步推理。尽管Gemini Flash 2.0和Claude-3在跨域中表现出一致的高度理解，但流行的GPT-4O和Command-R+在蛋白质测序任务上急剧失败。在32％的最佳性能中，所有型号都有很大的改进空间。我们希望从居里获得的见解可以指导科学中LLM的未来发展。评估代码和数据在此HTTPS URL中

Title: Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commerce

• Authors: Jingying Zeng, Zhenwei Dai, Hui Liu, Samarth Varshney, Zhiji Liu, Chen Luo, Zhen Li, Qi He, Xianfeng Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13518
• Pdf URL: https://arxiv.org/pdf/2503.13518
• Copy Paste: [[2503.13518]] Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commerce(https://arxiv.org/abs/2503.13518)
• Keywords: llm, prompt
• Abstract: Prompting LLMs offers an efficient way to guide output generation without explicit model training. In the e-commerce domain, prompting-based applications are widely used for tasks such as query understanding, recommender systems, and customer support. However, adapting LLMs to different tasks often requires extensive prompt engineering by domain experts, along with frequent updates to align with evolving business needs. Additionally, crafting fully unbiased natural language prompts remains a challenge for humans. To address these challenges, we propose a novel framework, Examples as the Prompt (EaP) which leverages labeled data to enhance prompts. Specifically, EaP automatically selects the most representative examples to maximize the few-shot capability of LLMs. It is efficient due to its unsupervised example selection and adaptive to potential data distribution shifts. We validate EaP on four real-world production use cases, demonstrating that it achieves comparable or even superior performance comparing to hand-crafted prompts designed by domain experts. Additionally, we introduce EaP_lite, which entirely replaces the natural language components of prompts with labeled examples. EaP_lite improves LLM inference speed by up to 70% without compromising performance. Latest online A/B test shows that using EaP and EaP_lite for data labeling can bring significant composite revenue gain by 0.06%.
• 摘要：提示LLMS提供了一种有效的方法来指导产量生成，而无需明确的模型培训。在电子商务领域中，基于促进的应用程序广泛用于诸如查询了解，推荐系统和客户支持之类的任务。但是，将LLM适应不同的任务通常需要域专家的广泛及时工程，以及频繁的更新以与不断发展的业务需求保持一致。此外，制定完全公正的自然语言提示仍然是人类的挑战。为了应对这些挑战，我们提出了一个新颖的框架，例如提示（EAP），该提示（EAP）利用标记数据来增强提示。具体而言，EAP会自动选择最具代表性的示例，以最大程度地提高LLM的少量功能。由于其无监督的示例选择并适应潜在的数据分布变化，因此这是有效的。我们验证了四个现实世界中生产用例的EAP，表明它与域专家设计的手工制作的提示相比，它可以达到可比甚至出色的性能。此外，我们介绍了EAP_LITE，它完全用标记的示例替换了提示的自然语言组成部分。 EAP_LITE将LLM推理速度提高了高达70％，而不会损害性能。最新的在线A/B测试表明，使用EAP和EAP_LITE进行数据标记可以带来大量的综合收入增长0.06％。

Title: Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results

• Authors: Peter Fettke, Constantin Houy
• Subjects: cs.CL, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2503.13520
• Pdf URL: https://arxiv.org/pdf/2503.13520
• Copy Paste: [[2503.13520]] Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results(https://arxiv.org/abs/2503.13520)
• Keywords: language model, llm
• Abstract: Large language models (LLM) have revolutionized the processing of natural language. Although first benchmarks of the process modeling abilities of LLM are promising, it is currently under debate to what extent an LLM can generate good process models. In this contribution, we argue that the evaluation of the process modeling abilities of LLM is far from being trivial. Hence, available evaluation results must be taken carefully. For example, even in a simple scenario, not only the quality of a model should be taken into account, but also the costs and time needed for generation. Thus, an LLM does not generate one optimal solution, but a set of Pareto-optimal variants. Moreover, there are several further challenges which have to be taken into account, e.g. conceptualization of quality, validation of results, generalizability, and data leakage. We discuss these challenges in detail and discuss future experiments to tackle these challenges scientifically.
• 摘要：大型语言模型（LLM）彻底改变了自然语言的处理。尽管LLM的过程建模能力的第一个基准是有希望的，但目前正在争论LLM在多大程度上可以生成良好的过程模型。在这项贡献中，我们认为对LLM的过程建模能力的评估远非微不足道。因此，必须仔细采取可用的评估结果。例如，即使在简单的情况下，不仅应考虑模型的质量，而且还应考虑生成所需的成本和时间。因此，LLM不会生成一个最佳解决方案，而是一组帕累托最佳变体。此外，还必须考虑到一些其他挑战，例如质量概念化，结果验证，可推广性和数据泄漏。我们详细讨论了这些挑战，并讨论了未来的实验，以科学地应对这些挑战。

Title: Agent-Enhanced Large Language Models for Researching Political Institutions

• Authors: Joseph R. Loffredo, Suyeol Yun
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.13524
• Pdf URL: https://arxiv.org/pdf/2503.13524
• Copy Paste: [[2503.13524]] Agent-Enhanced Large Language Models for Researching Political Institutions(https://arxiv.org/abs/2503.13524)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: The applications of Large Language Models (LLMs) in political science are rapidly expanding. This paper demonstrates how LLMs, when augmented with predefined functions and specialized tools, can serve as dynamic agents capable of streamlining tasks such as data collection, preprocessing, and analysis. Central to this approach is agentic retrieval-augmented generation (Agentic RAG), which equips LLMs with action-calling capabilities for interaction with external knowledge bases. Beyond information retrieval, LLM agents may incorporate modular tools for tasks like document summarization, transcript coding, qualitative variable classification, and statistical modeling. To demonstrate the potential of this approach, we introduce CongressRA, an LLM agent designed to support scholars studying the U.S. Congress. Through this example, we highlight how LLM agents can reduce the costs of replicating, testing, and extending empirical research using the domain-specific data that drives the study of political institutions.
• 摘要：大语模型（LLM）在政治学中的应用正在迅速扩展。本文演示了LLM在使用预定义的功能和专业工具增强时如何用作能够简化数据收集，预处理和分析等任务的动态代理。这种方法的核心是代理检索增强的生成（Agentic rag），它使LLMS具有与外部知识基础相互作用的动作呼叫功能。除了信息检索之外，LLM代理还可以合并用于文档摘要，成绩单编码，定性可变分类和统计建模等任务的模块化工具。为了证明这种方法的潜力，我们介绍了旨在支持研究美国国会的学者Congressra。通过此示例，我们强调了LLM代理如何使用驱动政治机构研究的特定领域数据来降低复制，测试和扩展实证研究的成本。

Title: Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models

• Authors: Teng Wang, Zhangyi Jiang, Zhenqi He, Wenhan Yang, Yanan Zheng, Zeyu Li, Zifan He, Shenyang Tong, Hailei Gong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13551
• Pdf URL: https://arxiv.org/pdf/2503.13551
• Copy Paste: [[2503.13551]] Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models(https://arxiv.org/abs/2503.13551)
• Keywords: language model, llm
• Abstract: Recent studies show that Large Language Models (LLMs) achieve strong reasoning capabilities through supervised fine-tuning or reinforcement learning. However, a key approach, the Process Reward Model (PRM), suffers from reward hacking, making it unreliable in identifying the best intermediate steps. In this paper, we propose a novel reward model approach, Hierarchical Reward Model (HRM), which evaluates both individual and consecutive reasoning steps from fine-grained and coarse-grained level. HRM performs better in assessing reasoning coherence and self-reflection, particularly when the previous reasoning step is incorrect. Furthermore, to address the inefficiency of autonomous generating PRM training data via Monte Carlo Tree Search (MCTS), we introduce a lightweight and effective data augmentation strategy called Hierarchical Node Compression (HNC) based on node merging (combining two consecutive reasoning steps into one step) in the tree structure. This approach diversifies MCTS results for HRM with negligible computational overhead, enhancing label robustness by introducing noise. Empirical results on the PRM800K dataset demonstrate that HRM, in conjunction with HNC, achieves superior stability and reliability in evaluation compared to PRM. Furthermore, cross-domain evaluations on MATH500 and GSM8K confirm HRM's superior generalization and robustness across diverse reasoning tasks. The code for all experiments will be released at https: //github.com/tengwang0318/hierarchial_reward_model.
• 摘要：最近的研究表明，大型语言模型（LLMS）通过监督的微调或加强学习来实现强大的推理能力。但是，一种关键方法，即过程奖励模型（PRM），遭受了奖励黑客攻击，使其无法确定最佳的中间步骤。在本文中，我们提出了一种新型的奖励模型方法，即分层奖励模型（HRM），该模型评估了从细粒度和粗粒水平的个人和连续推理步骤。 HRM在评估推理连贯性和自我反射方面的表现更好，尤其是在上一步不正确的情况下。此外，为了通过Monte Carlo Tree搜索（MCT）解决自主生成PRM培训数据的效率低下，我们引入了基于节点合并（将两个连续的推理步骤组合为一个连续的推理步骤）中的轻巧有效的数据增强策略，称为层次结构节点压缩（HNC）。这种方法通过可忽略不计的计算开销来使HRM的MCTS结果多样化，从而通过引入噪声来增强标签的鲁棒性。 PRM800K数据集的经验结果表明，与PRM相比，HRM与HNC结合使用，在评估中实现了卓越的稳定性和可靠性。此外，对MATH500和GSM8K的跨域评估证实了HRM在各种推理任务中的出色概括和鲁棒性。所有实验的代码将在https：//github.com/tengwang0318/hierarchial_reward_model上发布。

Title: MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAG

• Authors: Pingyu Wu, Daiheng Gao, Jing Tang, Huimin Chen, Wenbo Zhou, Weiming Zhang, Nenghai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13563
• Pdf URL: https://arxiv.org/pdf/2503.13563
• Copy Paste: [[2503.13563]] MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAG(https://arxiv.org/abs/2503.13563)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by using external knowledge, but it struggles with precise entity information retrieval. In this paper, we proposed MES-RAG framework, which enhances entity-specific query handling and provides accurate, secure, and consistent responses. MES-RAG introduces proactive security measures that ensure system integrity by applying protections prior to data access. Additionally, the system supports real-time multi-modal outputs, including text, images, audio, and video, seamlessly integrating into existing RAG architectures. Experimental results demonstrate that MES-RAG significantly improves both accuracy and recall, highlighting its effectiveness in advancing the security and utility of question-answering, increasing accuracy to 0.83 (+0.25) on targeted task. Our code and data are available at this https URL.
• 摘要：检索增强的生成（RAG）通过使用外部知识来改善大语言模型（LLM），但它会在精确的实体信息检索中挣扎。在本文中，我们提出了Mes-rag框架，该框架可以增强特定于实体的查询处理，并提供准确，安全和一致的响应。 Mes-rag引入了主动的安全措施，通过在数据访问之前应用保护来确保系统完整性。此外，该系统支持实时多模式输出，包括文本，图像，音频和视频，无缝集成到现有的抹布架构中。实验结果表明，MES-rag显着提高了准确性和召回率，突出了其在提高问题驱动器的安全性和实用性方面的有效性，将准确性提高到目标任务的0.83（+0.25）。我们的代码和数据可在此HTTPS URL上找到。

Title: ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts

• Authors: Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, Alexander Heinecke
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.13565
• Pdf URL: https://arxiv.org/pdf/2503.13565
• Copy Paste: [[2503.13565]] ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts(https://arxiv.org/abs/2503.13565)
• Keywords: llm
• Abstract: Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing any accuracy over the 16-bit model inference. In a typical SD setup, the idea is to use a full-precision, small, fast model as "draft" to generate the next few tokens and use the "target" large model to verify the draft-generated tokens. The efficacy of this method heavily relies on the acceptance ratio of the draft-generated tokens and the relative token throughput of the draft versus the target model. Nevertheless, an efficient SD pipeline requires pre-training and aligning the draft model to the target model, making it impractical for LLM inference in a plug-and-play fashion. In this work, we propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4. In practice, our plug-and-play solution gives speedups up to 2x over the BF16 baseline. Then we pursue an opportunity for further acceleration: the MXFP4 draft token generation itself can be accelerated via speculative decoding by using yet another smaller draft. We call our method ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts since it recursively applies speculation for accelerating the draft-token generation. Combining Multi-Level Speculative Decoding with MXFP4 Quantized Drafts we outperform state-of-the-art speculative decoding, yielding speedups up to 2.72x over the BF16 baseline.
• 摘要：投机解码（SD）已成为加速LLM推断的方法，而无需牺牲16位模型推断的任何精度。在典型的SD设置中，这个想法是使用完整的，小的，快速的模型作为“草稿”来生成接下来的几个令牌，并使用“ Target”大型模型来验证草稿生成的令牌。这种方法的功效在很大程度上取决于草稿生成的令牌的接受率以及草稿与目标模型的相对令牌吞吐量。然而，有效的SD管道需要预先培训并将草案模型与目标模型保持一致，这使得对LLM推断以插件方式不切实际。在这项工作中，我们建议以插件方式使用MXFP4型号作为草稿，因为MXFP4仅重量量化（WOQ）仅将BF16目标模型权重与MXFP4进行直接铸造。在实践中，我们的插件解决方案在BF16基线上可提供高达2倍的加速度。然后，我们寻求进一步加速的机会：通过使用另一个较小的草稿，可以通过投机解码来加速MXFP4的标记生成本身。我们称我们的方法ML-SpecQD：使用量化的草稿进行多级投机解码，因为它递归地采用了推测来加速草稿一代。将多级投机解码与MXFP4量化的草稿相结合，我们胜过最先进的投机解码，在BF16基线上产生高达2.72倍的速度。

Title: Pensez: Less Data, Better Reasoning -- Rethinking French LLM

• Authors: Huy Hoang Ha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13661
• Pdf URL: https://arxiv.org/pdf/2503.13661
• Copy Paste: [[2503.13661]] Pensez: Less Data, Better Reasoning -- Rethinking French LLM(https://arxiv.org/abs/2503.13661)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, achieving strong performance in specialized domains like mathematical reasoning and non-English languages often requires extensive training on massive datasets. This paper investigates a contrasting approach: strategic fine-tuning on a small, high-quality, bilingual (English-French) dataset to enhance both the reasoning capabilities and French language proficiency of a large language model. Rather than relying on scale, we explore the hypothesis that targeted data curation and optimized training can achieve competitive, or even superior, performance. We demonstrate, through targeted supervised fine-tuning (SFT) on only 2,000 carefully selected samples, significant improvements in mathematical reasoning. Specifically, Pensez 7B exhibits an increase in accuracy of the base model up to 20% on the AIME25 and a 12% increase on a French MATH level 5 benchmark. These results challenge the prevailing assumption that massive datasets are aprerequisite for strong reasoning performance in LLMs, highlighting the potential of strategic data curation and optimized fine-tuning for enhancing both specialized skills and multilingual capabilities. Our findings have implications for the efficient development of high-performing, multilingual LLMs, especially in resource-constrained scenarios.
• 摘要：大型语言模型（LLM）在各种自然语言处理任务中表现出了显着的功能。但是，在数学推理和非英语语言等专业领域中实现强大的性能通常需要在大规模数据集上进行广泛的培训。本文研究了一种对比方法：对小型，高质量的双语（英语）数据集进行战略性微调，以增强大语言模型的推理能力和法语能力。我们不依赖规模，而是探讨了针对的数据策展和优化培训可以实现竞争性甚至优越的表现的假设。我们通过仅针对2,000个精心选择的样本进行了有针对性的监督微调（SFT）来证明，数学推理的显着改善。具体而言，Pensez 7b的基础模型的准确性提高了AIME25的20％，而法国数学5级基准的精度则提高了12％。这些结果挑战了主要的假设，即大规模数据集是LLM中强大推理性能的基本条件，突出了战略数据策展的潜力，并优化了对增强专业技能和多语言功能的微调。我们的发现对高性能，多语言LLM的有效发展具有影响，尤其是在资源约束的情况下。

Title: Feature Extraction and Analysis for GPT-Generated Text

• Authors: A. Selvioğlu, V. Adanova, M. Atagoziev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13687
• Pdf URL: https://arxiv.org/pdf/2503.13687
• Copy Paste: [[2503.13687]] Feature Extraction and Analysis for GPT-Generated Text(https://arxiv.org/abs/2503.13687)
• Keywords: language model, gpt
• Abstract: With the rise of advanced natural language models like GPT, distinguishing between human-written and GPT-generated text has become increasingly challenging and crucial across various domains, including academia. The long-standing issue of plagiarism has grown more pressing, now compounded by concerns about the authenticity of information, as it is not always clear whether the presented facts are genuine or fabricated. In this paper, we present a comprehensive study of feature extraction and analysis for differentiating between human-written and GPT-generated text. By applying machine learning classifiers to these extracted features, we evaluate the significance of each feature in detection. Our results demonstrate that human and GPT-generated texts exhibit distinct writing styles, which can be effectively captured by our features. Given sufficiently long text, the two can be differentiated with high accuracy.
• 摘要：随着GPT等先进的自然语言模型的兴起，在包括学术界在内的各个领域中，人体写入和GPT生成的文本之间的越来越具有挑战性和至关重要。长期以来的窃问题变得越来越紧迫，现在对信息的真实性的担忧更加复杂，因为并不总是清楚提出的事实是真实的还是捏造的。在本文中，我们介绍了一项针对特征提取和分析的全面研究，以区分人文和GPT生成的文本。通过将机器学习分类器应用于这些提取的特征，我们评估了每个功能在检测中的重要性。我们的结果表明，人类和GPT生成的文本表现出独特的写作风格，这可以被我们的特征有效地捕获。给定足够长的文本，两者可以以很高的精度进行区分。

Title: Atyaephyra at SemEval-2025 Task 4: Low-Rank NPO

• Authors: Jan Bronec (1), Jindřich Helcl (1) ((1) Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics)
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.13690
• Pdf URL: https://arxiv.org/pdf/2503.13690
• Copy Paste: [[2503.13690]] Atyaephyra at SemEval-2025 Task 4: Low-Rank NPO(https://arxiv.org/abs/2503.13690)
• Keywords: llm
• Abstract: We present a submission to the SemEval 2025 shared task on unlearning sensitive content from LLMs. Our approach employs negative preference optimization using low-rank adaptation. We show that we can utilize this combination to cheaply compute additional regularization terms, which help with unlearning stabilization. The results of our approach significantly exceed the shared task baselines.
• 摘要：我们向Semeval 2025共享任务提交了关于从LLMS学习敏感内容的共享任务。我们的方法采用低排名适应性的负偏好优化。我们表明，我们可以利用这种组合来廉价计算其他正则化术语，这有助于稳定。我们方法的结果大大超过了共享的任务基准。

Title: CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings

• Authors: Daniil Orel, Dilshod Azizov, Preslav Nakov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13733
• Pdf URL: https://arxiv.org/pdf/2503.13733
• Copy Paste: [[2503.13733]] CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings(https://arxiv.org/abs/2503.13733)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have revolutionized code generation, automating programming with remarkable efficiency. However, these advancements challenge programming skills, ethics, and assessment integrity, making the detection of LLM-generated code essential for maintaining accountability and standards. While, there has been some research on this problem, it generally lacks domain coverage and robustness, and only covers a small number of programming languages. To this end, we propose a framework capable of distinguishing between human- and LLM-written code across multiple programming languages, code generators, and domains. We use a large-scale dataset from renowned platforms and LLM-based code generators, alongside applying rigorous data quality checks, feature engineering, and comparative analysis using evaluation of traditional machine learning models, pre-trained language models (PLMs), and LLMs for code detection. We perform an evaluation on out-of-domain scenarios, such as detecting the authorship and hybrid authorship of generated code and generalizing to unseen models, domains, and programming languages. Moreover, our extensive experiments show that our framework effectively distinguishes human- from LLM-written code and sets a new benchmark for this task.
• 摘要：大型语言模型（LLM）彻底改变了代码生成，并以显着的效率自动化编程。但是，这些进步挑战了编程技能，道德和评估完整性，这使得发现LLM生成的代码对于维持问责制和标准至关重要。虽然对这个问题进行了一些研究，但它通常缺乏域覆盖范围和稳健性，仅涵盖了少量编程语言。为此，我们提出了一个能够在多种编程语言，代码生成器和域之间区分人类和LLM编写的代码的框架。我们使用来自著名平台和基于LLM的代码生成器的大规模数据集，以及使用传统的机器学习模型，预训练的语言模型（PLMS）和LLMS的评估进行严格的数据质量检查，功能工程和比较分析以进行代码检测。我们对室外场景进行评估，例如检测生成的代码的作者身份和混合作者身份，并概括了看不见的模型，域和编程语言。此外，我们的广泛实验表明，我们的框架有效地将人类与LLM写的代码区分开，并为此任务设定了新的基准。

Title: AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications

• Authors: Haiying Shen, Tanmoy Sen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13737
• Pdf URL: https://arxiv.org/pdf/2503.13737
• Copy Paste: [[2503.13737]] AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications(https://arxiv.org/abs/2503.13737)
• Keywords: language model, llm, prompt
• Abstract: In this paper, we consider a mixed-prompt scenario for a large language model (LLM) inference serving system that supports diverse applications with both short prompts and long prompts and heterogeneous SLOs for iteration time. To improve throughput when handling long prompts, previous research introduces a chunking method, but has not addressed heterogeneous SLOs. To address the limitation, we propose AccelGen, a high-throughput LLM inference serving system with heterogeneous SLO guarantees for diverse applications. AccelGen introduces four core components: (1) SLO-guaranteed dynamic chunking, which dynamically adjusts chunk sizes to maximize GPU compute utilization while meeting iteration-level SLOs; (2) Iteration-level SLO-based task prioritization, which prioritizes tight-SLO requests and batches requests with similar SLOs; (3) Multi-resource-aware batching, which selects queued requests to maximize the utilizations of both GPU compute resource and key-value cache (KVC). Trace-driven real experiments demonstrate that AccelGen achieves 1.42-11.21X higher throughput, 1.43-13.71X higher goodput, 37-90% higher SLO attainment, and 1.61-12.22X lower response latency compared to the state-of-the-art approaches. It achieves performance near the Oracle, which optimally maximizes goodput.
• 摘要：在本文中，我们考虑了大型语言模型（LLM）推理服务系统的混合奖励方案，该系统支持各种应用程序，并在短时间和长提示和长时间和异质性SLO中进行迭代时间。为了改善吞吐量在处理长提示时，先前的研究引入了块状方法，但尚未解决异质的SLO。为了解决限制，我们提出了Accelgen，Accelgen是一种高通量LLM推理服务系统，具有异质性SLO保证各种应用。 Accelgen引入了四个核心组件：（1）SLO保证的动态块，该块动态调节块大小，以最大程度地提高GPU计算的利用率，同时满足迭代级别的slos； （2）基于迭代级SLO的任务优先级，该任务优先级，优先考虑使用类似SLO的紧密SLO请求和批次请求； （3）多资源感知批处理，它选择了排队的请求，以最大程度地利用GPU计算资源和键值缓存（KVC）的利用。痕量驱动的实际实验表明，与先进方法相比，Accelgen的吞吐量高1.42-11.21倍，高1.43-13.71x高1.43-13.71倍，SLO成立率高37-90％，与先进方法相比，Accelgen的吞吐量提高了1.42-11.21倍。它在Oracle附近实现了性能，从而最大程度地提高了Goodput。

Title: Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

• Authors: Haiying Shen, Tanmoy Sen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13773
• Pdf URL: https://arxiv.org/pdf/2503.13773
• Copy Paste: [[2503.13773]] Mitigating KV Cache Competition to Enhance User Experience in LLM Inference(https://arxiv.org/abs/2503.13773)
• Keywords: language model, llm
• Abstract: In Large Language Model (LLM) serving, the KV-cache (KVC) bottleneck causes high tail Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT), impairing user experience, particularly in time-sensitive applications. However, satisfying both TTFT and TBT service-level objectives (SLOs) is challenging. To address this, we propose a system, named CacheOPT for mitigating KV Cache competition, based on key insights from our measurements, incorporating novel components. First, it estimates a request's output length, bounding the deviation with a high specified probability, adjusted based on the request arrival rate. Second, it allocates the estimated KVC demand to a request, and reuses other requests' allocated KVC to avoid preemptions while reducing waiting time. Third, it proactively allocates KVC before instead of at the time a request exhausts its allocation and reserves KVC globally to prevent preemptions. Fourth, it chooses a request that has long TBT SLO, long job remaining time and short preemption time to preempt. Fifth, it selects the shortest-latency strategy between swapping and recomputation for preemptions. Experiments show that CacheOPT achieves up to 3.29$\times$ and 2.83$\times$ lower tail TBT and tail TTFT, 47\% and 53\% higher TTFT and TBT SLO attainments, and supports up to 1.58$\times$ higher request arrival rate than the state-of-the-art methods.
• 摘要：在大型语言模型（LLM）服务中，KV-CACHE（KVC）瓶颈会导致高尾巴的高尾巴时间（TTFT）和TIME-BET-TOKENS（TBT）（TBT），尤其是在时间敏感的应用程序中。但是，满足TTFT和TBT服务级目标（SLO）都具有挑战性。为了解决这个问题，我们提出了一个基于我们测量的关键见解，并结合了新颖的组件，提出了一个名为Cacheopt来减轻KV缓存竞争的系统。首先，它估计请求的输出长度，并根据请求到达率调整了偏差的偏差。其次，它将估计的KVC需求分配给请求，并重用其他请求分配的KVC，以避免在减少等待时间的同时抢占。第三，它主动地主动分配KVC，而不是当时请求耗尽其分配，并在全球范围内保留KVC以防止抢先。第四，它选择了一个较长的TBT SLO的请求，剩下的时间很长，抢占率很短。第五，它选择了抢先的交换和重新签约之间的最短延迟策略。实验表明，Cacheopt达到3.29 $ \ times $和2.83 $ \ times $较低的尾巴和尾巴TTFT，47 \％\％和53 \％的TTFT和TBT SLO达到53 \％，并支持高达1.58 $ \ tims $的$ \ tim $ \ \ \％。

Title: Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations

• Authors: Rui Yang, Jiayi Tong, Haoyuan Wang, Hui Huang, Ziyang Hu, Peiyu Li, Nan Liu, Christopher J. Lindsell, Michael J. Pencina, Yong Chen, Chuan Hong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13857
• Pdf URL: https://arxiv.org/pdf/2503.13857
• Copy Paste: [[2503.13857]] Enabling Inclusive Systematic Reviews: Incorporating Preprint Articles with Large Language Model-Driven Evaluations(https://arxiv.org/abs/2503.13857)
• Keywords: language model, llm
• Abstract: Background. Systematic reviews in comparative effectiveness research require timely evidence synthesis. Preprints accelerate knowledge dissemination but vary in quality, posing challenges for systematic reviews. Methods. We propose AutoConfidence (automated confidence assessment), an advanced framework for predicting preprint publication, which reduces reliance on manual curation and expands the range of predictors, including three key advancements: (1) automated data extraction using natural language processing techniques, (2) semantic embeddings of titles and abstracts, and (3) large language model (LLM)-driven evaluation scores. Additionally, we employed two prediction models: a random forest classifier for binary outcome and a survival cure model that predicts both binary outcome and publication risk over time. Results. The random forest classifier achieved AUROC 0.692 with LLM-driven scores, improving to 0.733 with semantic embeddings and 0.747 with article usage metrics. The survival cure model reached AUROC 0.716 with LLM-driven scores, improving to 0.731 with semantic embeddings. For publication risk prediction, it achieved a concordance index of 0.658, increasing to 0.667 with semantic embeddings. Conclusion. Our study advances the framework for preprint publication prediction through automated data extraction and multiple feature integration. By combining semantic embeddings with LLM-driven evaluations, AudoConfidence enhances predictive performance while reducing manual annotation burden. The framework has the potential to facilitate systematic incorporation of preprint articles in evidence-based medicine, supporting researchers in more effective evaluation and utilization of preprint resources.
• 摘要：背景。比较有效性研究中的系统评价需要及时的证据综合。预印象会加速知识传播，但质量各不相同，对系统评价构成了挑战。方法。我们提出自我信仰（自动置信度评估），这是一个预测预印本出版物的高级框架，可降低对手动策划的依赖，并扩大预测因子的范围，包括三个关键进步：（1）使用自然语言处理技术自动化数据提取，（2）标题和摘要的语义嵌入，以及（3）大型语言（3）大型语言（3）大型录音。此外，我们采用了两个预测模型：一个用于二元结果的随机森林分类器和一种生存的治疗模型，可预测随着时间的推移二进制结果和出版风险。结果。随机的森林分类器以LLM驱动的分数达到了AUROC 0.692，具有语义嵌入的0.733，并具有0.747，具有文章使用指标。以LLM驱动的分数为0.716的生存疗法模型，具有语义嵌入的0.731。为了进行出版风险预测，它的一致性指数为0.658，随着语义嵌入而增至0.667。结论。我们的研究通过自动数据提取和多个功能集成来推动预印本发布预测的框架。通过将语义嵌入与LLM驱动的评估相结合，AudoConconcentives可以增强预测性能，同时减轻手动注释负担。该框架有可能促进在循证医学中系统地纳入预印本文章，从而支持研究人员更有效地评估和利用预印资源。

Title: ConSCompF: Consistency-focused Similarity Comparison Framework for Generative Large Language Models

• Authors: Alexey Karev, Dong Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13923
• Pdf URL: https://arxiv.org/pdf/2503.13923
• Copy Paste: [[2503.13923]] ConSCompF: Consistency-focused Similarity Comparison Framework for Generative Large Language Models(https://arxiv.org/abs/2503.13923)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large language models (LLMs) have been one of the most important discoveries in machine learning in recent years. LLM-based artificial intelligence (AI) assistants, such as ChatGPT, have consistently attracted the attention from researchers, investors, and the general public, driving the rapid growth of this industry. With the frequent introduction of new LLMs to the market, it becomes increasingly difficult to differentiate between them, creating a demand for new LLM comparison methods. In this research, the Consistency-focused Similarity Comparison Framework (ConSCompF) for generative large language models is proposed. It compares texts generated by two LLMs and produces a similarity score, indicating the overall degree of similarity between their responses. The main advantage of this framework is that it can operate on a small number of unlabeled data, such as chatbot instruction prompts, and does not require LLM developers to disclose any information about their product. To evaluate the efficacy of ConSCompF, two experiments aimed at identifying similarities between multiple LLMs are conducted. Additionally, these experiments examine the correlation between the similarity scores generated by ConSCompF and the differences in the outputs produced by other benchmarking techniques, such as ROUGE-L. Finally, a series of few-shot LLM comparison experiments is conducted to evaluate the performance of ConSCompF in a few-shot LLM comparison scenario. The proposed framework can be used for calculating similarity matrices of multiple LLMs, which can be effectively visualized using principal component analysis (PCA). The ConSCompF output may provide useful insights into data that might have been used during LLM training and help detect possible investment fraud attempts.
• 摘要：近年来，大型语言模型（LLM）一直是机器学习中最重要的发现之一。基于LLM的人工智能（AI）助理（例如ChatGpt）一直吸引研究人员，投资者和公众的关注，推动了这一行业的快速发展。经常向市场推出新的LLM，越来越难以区分它们，从而产生了对新LLM比较方法的需求。在这项研究中，提出了针对生成大语言模型的以一致性相似性比较框架（CONCOMPF）。它比较了由两个LLM产生的文本并产生相似性评分，表明其响应之间的总体相似度。该框架的主要优点是它可以在少数未标记的数据（例如聊天机器人说明提示）上运行，并且不需要LLM开发人员披露有关其产品的任何信息。为了评估CONCOMPF的疗效，两个实验旨在确定多个LLM之间的相似性。此外，这些实验检查了CONCOMPF产生的相似性分数与其他基准测试技术（例如Rouge-l）产生的输出差异之间的相关性。最后，进行了一系列少数LLM比较实验，以评估COSCOMPF的表现，以在几个shot LLM比较方案中。所提出的框架可用于计算多个LLM的相似性矩阵，可以使用主成分分析（PCA）有效地可视化。 CONCOMPF输出可能会提供有用的见解，以了解可能在LLM培训期间使用的数据，并有助于检测可能的投资欺诈尝试。

Title: Navigating Rifts in Human-LLM Grounding: Study and Benchmark

• Authors: Omar Shaikh, Hussein Mozannar, Gagan Bansal, Adam Fourney, Eric Horvitz
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2503.13975
• Pdf URL: https://arxiv.org/pdf/2503.13975
• Copy Paste: [[2503.13975]] Navigating Rifts in Human-LLM Grounding: Study and Benchmark(https://arxiv.org/abs/2503.13975)
• Keywords: language model, llm, prompt, chat
• Abstract: Language models excel at following instructions but often struggle with the collaborative aspects of conversation that humans naturally employ. This limitation in grounding -- the process by which conversation participants establish mutual understanding -- can lead to outcomes ranging from frustrated users to serious consequences in high-stakes scenarios. To systematically study grounding challenges in human-LLM interactions, we analyze logs from three human-assistant datasets: WildChat, MultiWOZ, and Bing Chat. We develop a taxonomy of grounding acts and build models to annotate and forecast grounding behavior. Our findings reveal significant differences in human-human and human-LLM grounding: LLMs were three times less likely to initiate clarification and sixteen times less likely to provide follow-up requests than humans. Additionally, early grounding failures predicted later interaction breakdowns. Building on these insights, we introduce RIFTS: a benchmark derived from publicly available LLM interaction data containing situations where LLMs fail to initiate grounding. We note that current frontier models perform poorly on RIFTS, highlighting the need to reconsider how we train and prompt LLMs for human interaction. To this end, we develop a preliminary intervention that mitigates grounding failures.
• 摘要：语言模型在以下说明上表现出色，但经常在人类自然使用的对话的协作方面挣扎。接地的局限性（对话参与者建立相互理解的过程）可能会导致结果，从沮丧的用户到高风险情景的严重后果。为了系统地研究人类相互作用中的基础挑战，我们分析了三个人类辅助数据集的日志：Wildchat，Multiwoz和Bing Chat。我们开发了基础行为的分类法，并建立了注释和预测基础行为的模型。我们的发现表明，人类和人类基础的基础存在显着差异：LLM的澄清可能性少三倍，提供后续请求的可能性少于人类的16倍。此外，早期的接地故障预测了以后的相互作用故障。在这些见解的基础上，我们介绍了Rifts：一个基准，该基准来自包含LLM无法启动接地的情况的公开可用的LLM交互数据。我们注意到，当前的边界模型在裂谷上的表现较差，强调了重新考虑我们如何训练并提示LLM进行人类互动的需要。为此，我们开发了一种初步的干预措施，以减轻接地失败。

Title: Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks

• Authors: Mykyta Syromiatnikov, Victoria Ruvinskaya, Nataliia Komleva
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.13988
• Pdf URL: https://arxiv.org/pdf/2503.13988
• Copy Paste: [[2503.13988]] Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks(https://arxiv.org/abs/2503.13988)
• Keywords: language model, gpt, chain-of-thought
• Abstract: Leading large language models have demonstrated impressive capabilities in reasoning-intensive tasks, such as standardized educational testing. However, they often require extensive training in low-resource settings with inaccessible infrastructure. Small or compact models, though more efficient, frequently lack sufficient support for underrepresented languages, leaving a performance gap in critical domains. This work explores the potential of parameter-efficient fine-tuning of compact open-weight language models to handle reasoning-intensive tasks in the underrepresented Ukrainian language, building on the findings of the ZNO-Eval benchmark. Parameter-efficient fine-tuning of LLaMA 3.1 (8 billion parameters), LLaMA 3.2 (3 billion parameters), and Gemma 2 (9 billion parameters) models on chain-of-thought solutions resulted in a modest test score improvement of up to 17.4% on complex matching tasks and 1.6% overall compared to tuning on answer letters alone, offering enhanced interpretability and robustness. In addition, the proposed tuning method with joint task topic and step-by-step solution generation outperforms standard chain-of-thought tuning in matching tasks and provides a 5.4% gain over the best LLaMA 3.2 model due to guiding the model to recall and apply domain-relevant information. Contrasting obtained results with zero-shot evaluations of leading open-weight and proprietary models such as Qwen, DeepSeek R1, OpenAI o1 and o3, Gemini, and Claude, highlight that fine-tuning LLaMA and Gemma models with 2,032 step-by-step solutions and 20 to 50 million trainable parameters on a single A100 GPU lets them outperform GPT-4o mini, Mistral Large, and larger open-weight models. This research also evaluates how merging the quantized adapter with the base model influences the generation quality. Source code and tuned models are available at this https URL.
• 摘要：领先的大型语言模型在推理密集型任务（例如标准化的教育测试）中表现出了令人印象深刻的能力。但是，他们通常需要在无法访问的基础设施的低资源环境中进行广泛的培训。小型或紧凑的模型虽然更有效，但通常缺乏对代表性不足的语言的足够支持，在关键领域中留下了性能差距。这项工作探讨了紧凑型开放式语言模型的参数有效微调的潜力，以在Zno-eval基准的发现建立基于代表性不足的乌克兰语言中的推理密集型任务。遍历的参数有效的遍历3.1（80亿参数），3.2（30亿参数）和Gemma 2（90亿个参数）模型在经过实施链解决方案上，可在复杂的匹配任务上和单独的互补性增强性，可在复杂的匹配任务上提高高达17.4％的测试得分，高达17.4％。此外，提出的使用联合任务主题和逐步解决方案生成的调整方法在匹配任务中优于标准链链调整，并且由于指导该模型来召回和应用与域相关的信息，因此比最佳Llama 3.2模型提供了5.4％的增益。通过对领先的开放权力和专有模型的零拍摄评估（例如Qwen，DeepSeek R1，OpenAi O1和O3，Gemini和Claude）的对比鲜明的结果，突显了该微调的Llama和Gemma模型，具有2,032个逐步解决方案，以及20至50亿个可训练的参数，并将其超级训练有素的参数用于单个单位a100 gpus-lits-lits-lits imrimi mis lits-lits themer lits imry 4大型和更大的开放权重型号。这项研究还评估了如何将量化适配器与基本模型合并的如何影响发电质量。源代码和调谐模型可在此HTTPS URL上找到。

Title: The KoLMogorov Test: Compression by Code Generation

• Authors: Ori Yoran, Kunhao Zheng, Fabian Gloeckle, Jonas Gehring, Gabriel Synnaeve, Taco Cohen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.13992
• Pdf URL: https://arxiv.org/pdf/2503.13992
• Copy Paste: [[2503.13992]] The KoLMogorov Test: Compression by Code Generation(https://arxiv.org/abs/2503.13992)
• Keywords: gpt, llm
• Abstract: Compression is at the heart of intelligence. A theoretically optimal way to compress any sequence of data is to find the shortest program that outputs that sequence and then halts. However, such 'Kolmogorov compression' is uncomputable, and code generating LLMs struggle to approximate this theoretical ideal, as it requires reasoning, planning and search capabilities beyond those of current models. In this work, we introduce the KoLMogorov-Test (KT), a compression-as-intelligence test for code generating LLMs. In KT a model is presented with a sequence of data at inference time, and asked to generate the shortest program that produces the sequence. We identify several benefits of KT for both evaluation and training: an essentially infinite number of problem instances of varying difficulty is readily available, strong baselines already exist, the evaluation metric (compression) cannot be gamed, and pretraining data contamination is highly unlikely. To evaluate current models, we use audio, text, and DNA data, as well as sequences produced by random synthetic programs. Current flagship models perform poorly - both GPT4-o and Llama-3.1-405B struggle on our natural and synthetic sequences. On our synthetic distribution, we are able to train code generation models with lower compression rates than previous approaches. Moreover, we show that gains on synthetic data generalize poorly to real data, suggesting that new innovations are necessary for additional gains on KT.
• 摘要：压缩是智力的核心。理论上压缩任何数据序列的理论最佳方法是找到输出该序列然后停止的最短程序。但是，这种“ kolmogorov压缩”是不可兼容的，而生成LLM的代码则难以估算这一理论理想，因为它需要超出当前模型的推理，计划和搜索功能。在这项工作中，我们介绍了Kolmogorov-Test（KT），这是针对代码生成LLM的压缩测试。在KT中，一个模型在推理时有一系列数据，并要求生成产生序列的最短程序。我们确定了KT在评估和培训方面的几个好处：本质上很容易获得各种难度的问题实例，已经存在强大的基线，评估指标（压缩）不能被盖上，并且预处理的数据污染极不可能。为了评估当前模型，我们使用音频，文本和DNA数据以及随机合成程序产生的序列。当前的旗舰模型的性能很差 - GPT4-O和Llama-3.1-405B在我们的自然和合成序列上挣扎。在我们的合成分布上，我们能够训练与以前的方法相比，压缩率较低的代码生成模型。此外，我们表明，合成数据的收益对真实数据的推广不佳，这表明在KT上额外收益是必需的。

Title: Synthetic Data Generation Using Large Language Models: Advances in Text and Code

• Authors: Mihai Nadas, Laura Diosan, Andreea Tomescu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14023
• Pdf URL: https://arxiv.org/pdf/2503.14023
• Copy Paste: [[2503.14023]] Synthetic Data Generation Using Large Language Models: Advances in Text and Code(https://arxiv.org/abs/2503.14023)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have unlocked new possibilities for generating synthetic training data in both natural language and code. By producing artificial but task-relevant examples, these models can significantly augment or even replace real-world datasets, especially when labeled data is scarce or sensitive. This paper surveys recent advances in using LLMs to create synthetic text and code, emphasizing prompt-based generation, retrieval-augmented pipelines, and iterative self-refinement. We show how these methods enrich low-resource tasks such as classification and question answering, as well as code-centric applications such as instruction tuning, code translation, and bug repair, by enabling automated verification of functional correctness. Alongside potential benefits like cost-effectiveness, broad coverage, and controllable diversity, we address challenges such as factual inaccuracies in generated text, lack of stylistic realism, and the risk of bias amplification. Proposed mitigations include filtering and weighting outputs and reinforcement learning with execution feedback for code. We conclude with open research directions like automated prompt engineering, cross-modal data synthesis, and robust evaluation frameworks, highlighting the importance of LLM-generated synthetic data in advancing AI while emphasizing ethical and quality safeguards.
• 摘要：大型语言模型（LLMS）已解锁了以自然语言和代码生成合成训练数据的新可能性。通过产生人工的但与任务相关的示例，这些模型可以显着增加甚至替代现实世界中的数据集，尤其是当标记的数据稀缺或敏感时。本文调查了使用LLM创建合成文本和代码的最新进展，强调基于迅速的一代，检索式管道和迭代自我进行。我们通过启用自动化功能正确性的验证，展示了这些方法如何丰富低资源任务，例如分类和问题答案，以及以代码为中心的应用程序，例如指令调整，代码翻译和错误修复。除了具有成本效益，广泛的覆盖范围和可控的多样性之类的潜在收益外，我们还应对诸如生成文本中的事实不准确，缺乏风格现实主义以及偏见放大的风险等挑战。提出的缓解包括过滤和加权输出和加强辅助学习，并使用代码执行反馈。我们以开放研究方向为结论，例如自动化及时工程，跨模式数据综合和强大的评估框架，强调了LLM生成的合成数据在推进AI时的重要性，同时强调道德和优质的保障措施。

Title: CARE: A QLoRA-Fine Tuned Multi-Domain Chatbot With Fast Learning On Minimal Hardware

• Authors: Ankit Dutta, Nabarup Ghosh, Ankush Chatterjee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.14136
• Pdf URL: https://arxiv.org/pdf/2503.14136
• Copy Paste: [[2503.14136]] CARE: A QLoRA-Fine Tuned Multi-Domain Chatbot With Fast Learning On Minimal Hardware(https://arxiv.org/abs/2503.14136)
• Keywords: language model, chat
• Abstract: Large Language models have demonstrated excellent domain-specific question-answering capabilities when finetuned with a particular dataset of that specific domain. However, fine-tuning the models requires a significant amount of training time and a considerable amount of hardware. In this work, we propose CARE (Customer Assistance and Response Engine), a lightweight model made by fine-tuning Phi3.5-mini on very minimal hardware and data, designed to handle queries primarily across three domains: telecommunications support, medical support, and banking support. For telecommunications and banking, the chatbot addresses issues and problems faced by customers regularly in the above-mentioned domains. In the medical domain, CARE provides preliminary support by offering basic diagnoses and medical suggestions that a user might take before consulting a healthcare professional. Since CARE is built on Phi3.5-mini, it can be used even on mobile devices, increasing its usability. Our research also shows that CARE performs relatively well on various medical benchmarks, indicating that it can be used to make basic medical suggestions.
• 摘要：大型语言模型在使用该特定域的特定数据集中进行了填充时，表现出了出色的特定领域提问功能。但是，对模型进行微调需要大量的培训时间和大量的硬件。在这项工作中，我们提出了护理（客户援助和响应引擎），这是一种通过微调PHI3.5-MINI制作的轻量级模型，该模型在非常最小的硬件和数据上，旨在处理主要跨三个领域的查询：电信支持，医疗支持和银行支持。对于电信和银行业务，聊天机器人解决了上述域中定期客户面临的问题和问题。在医疗领域，CARE通过提供用户在咨询医疗保健专业人员之前可能会采取的基本诊断和医疗建议来提供初步支持。由于护理是建立在PHI3.5-MINI上的，因此即使在移动设备上也可以使用它，从而提高了其可用性。我们的研究还表明，护理在各种医疗基准上的表现相对较好，表明它可以用于提出基本的医疗建议。

Title: Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approach

• Authors: Christian Poelitz, Nick McKenna
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14167
• Pdf URL: https://arxiv.org/pdf/2503.14167
• Copy Paste: [[2503.14167]] Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approach(https://arxiv.org/abs/2503.14167)
• Keywords: llm
• Abstract: Real dialogues with AI assistants for solving data-centric tasks often follow dynamic, unpredictable paths due to imperfect information provided by the user or in the data, which must be caught and handled. Developing datasets which capture such user-AI interactions is difficult and time-consuming. In this work, we develop a novel framework for synthetically generating controlled, multi-turn conversations between a user and AI assistant for the task of table-based question answering, which can be generated from an existing dataset with fully specified table QA examples for any target domain. Each conversation aims to solve a table-based reasoning question through collaborative effort, modeling one of two real-world scenarios: (1) an AI-initiated clarification, or (2) a user-initiated correction. Critically, we employ a strong teacher LLM to verify the correctness of our synthetic conversations, ensuring high quality. We demonstrate synthetic datasets generated from TAT-QA and WikiTableQuestions as benchmarks of frontier LLMs. We find that even larger models struggle to effectively issuing clarification questions and accurately integrate user feedback for corrections.
• 摘要：与AI助理解决以数据为中心的任务的真实对话通常遵循动态，不可预测的途径，因为用户或数据中提供的不完善信息，必须捕获和处理数据。开发捕获此类用户交互的数据集很困难且耗时。在这项工作中，我们开发了一个新颖的框架，用于在用户和AI助手之间合成基于表的问题答案的任务之间的多转交流，可以从现有数据集中生成，并具有完整指定的表QA示例的任何目标域。每个对话旨在通过协作努力来解决一个基于表的推理问题，建模两个现实世界中的一个方案之一：（1）AI引发的澄清，或（2）用户启动的校正。至关重要的是，我们采用强大的老师LLM来验证合成对话的正确性，以确保高质量。我们演示了由Tat-QA生成的合成数据集，并作为Frontier LLMS的基准生成的合成数据集。我们发现，甚至更大的模型也很难有效地发出澄清问题，并准确整合用户反馈以进行更正。

Title: Towards Harmless Multimodal Assistants with Blind Preference Optimization

• Authors: Yongqi Li, Lu Yang, Jian Wang, Runyang You, Wenjie Li, Liqiang Nie
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.14189
• Pdf URL: https://arxiv.org/pdf/2503.14189
• Copy Paste: [[2503.14189]] Towards Harmless Multimodal Assistants with Blind Preference Optimization(https://arxiv.org/abs/2503.14189)
• Keywords: language model, llm
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. Given the extensive applications of MLLMs, the associated safety issues have become increasingly critical. Due to the effectiveness of preference optimization in aligning MLLMs with human preferences, there is an urgent need for safety-related preference data for MLLMs. To address this, we construct the MMSafe-PO preference dataset towards harmless multimodal assistants, featuring multimodal instructions, the conversational format, and ranked paired responses from human feedback. We also identify two insightful observations: modality co-defense and modality cheating, which illustrate that MLLMs possess a certain level of inherent defense while still presenting unique safety challenges. Based on these observations, we propose the Blind Preference Optimization (BPO) approach. Comprehensive experiments on three benchmarks show that BPO effectively enhances the safety capabilities of MLLMs. Notably, BPO significantly improves the safety rate of the base MLLM by 45.0%, outperforming the DPO approach. Additionally, applying BPO to the MMSafe-PO dataset greatly reduces the base MLLM's unsafe rate on other safety benchmarks (14.5% on MM-SafetyBench and 82.9% on HarmEval, demonstrating the effectiveness and robustness of both the dataset and the approach. We release code and data at this https URL.
• 摘要：多模式的大语言模型（MLLM）在多模式理解，推理和互动方面表现出了令人印象深刻的能力。鉴于MLLM的广泛应用，相关的安全问题已经变得越来越危急。由于偏好优化在对齐MLLM与人类偏好方面的有效性，因此迫切需要对MLLM的安全相关偏好数据。为了解决这个问题，我们将MMSAFE-PO偏好数据集构建为无害的多模式助手，该数据集具有多模式指令，对话格式和人类反馈中排名的配对响应。我们还确定了两个有见地的观察：形态的共同防御和模式作弊，这表明MLLM具有一定程度的固有防御，同时仍提出独特的安全挑战。基于这些观察结果，我们提出了盲人偏好优化（BPO）方法。三个基准测试的全面实验表明，BPO有效提高了MLLM的安全能力。值得注意的是，BPO显着提高了基本MLLM的安全率45.0％，表现优于DPO方法。此外，将BPO应用于MMSAFE-PO数据集，大大降低了MLLM对其他安全基准的不安全率（MM-SafetyBench的14.5％，Harmeval的82.9％，表明数据集和方法的有效性和可靠性。

Title: JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System

• Authors: Weihang Su, Baoqing Yue, Qingyao Ai, Yiran Hu, Jiaqi Li, Changyue Wang, Kaiyuan Zhang, Yueyue Wu, Yiqun Liu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.14258
• Pdf URL: https://arxiv.org/pdf/2503.14258
• Copy Paste: [[2503.14258]] JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System(https://arxiv.org/abs/2503.14258)
• Keywords: llm, retrieval-augmented generation
• Abstract: This paper introduces JuDGE (Judgment Document Generation Evaluation), a novel benchmark for evaluating the performance of judgment document generation in the Chinese legal system. We define the task as generating a complete legal judgment document from the given factual description of the case. To facilitate this benchmark, we construct a comprehensive dataset consisting of factual descriptions from real legal cases, paired with their corresponding full judgment documents, which serve as the ground truth for evaluating the quality of generated documents. This dataset is further augmented by two external legal corpora that provide additional legal knowledge for the task: one comprising statutes and regulations, and the other consisting of a large collection of past judgment documents. In collaboration with legal professionals, we establish a comprehensive automated evaluation framework to assess the quality of generated judgment documents across various dimensions. We evaluate various baseline approaches, including few-shot in-context learning, fine-tuning, and a multi-source retrieval-augmented generation (RAG) approach, using both general and legal-domain LLMs. The experimental results demonstrate that, while RAG approaches can effectively improve performance in this task, there is still substantial room for further improvement. All the codes and datasets are available at: this https URL.
• 摘要：本文介绍了法官（判断文件生成评估），这是评估中国法律制度中判断文件生成的绩效的新基准。我们将任务定义为从给定的事实描述中生成完整的法律判决文件。为了促进此基准，我们构建了一个全面的数据集，该数据集由实际法律案件的事实描述组成，并与相应的完整判断文件配对，这是评估生成文档质量的基础真相。两个外部法律语料库进一步增强了该数据集，这些文件为任务提供了额外的法律知识：一个包括法规和法规，另一个由大量过去的判决文件组成。与法律专业人员合作，我们建立了一个全面的自动化评估框架，以评估各个方面的判断文件的质量。我们使用通用和法律域LLMS评估了各种基线方法，包括几乎没有射击的内在学习，微调和多源检索型生成一代（RAG）方法。实验结果表明，尽管RAG方法可以有效地改善此任务的性能，但仍有大量进一步改进的空间。所有代码和数据集均可在以下网址提供：此HTTPS URL。

Title: DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal

• Authors: Vaibhav Aggarwal, Ojasv Kamal, Abhinav Japesh, Zhijing Jin, Bernhard Schölkopf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14269
• Pdf URL: https://arxiv.org/pdf/2503.14269
• Copy Paste: [[2503.14269]] DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal(https://arxiv.org/abs/2503.14269)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have revolutionized various domains, including natural language processing, data analysis, and software development, by enabling automation. In software engineering, LLM-powered coding agents have garnered significant attention due to their potential to automate complex development tasks, assist in debugging, and enhance productivity. However, existing approaches often struggle with sub-optimal decision-making, requiring either extensive manual intervention or inefficient compute scaling strategies. To improve coding agent performance, we present Dynamic Action Re-Sampling (DARS), a novel inference time compute scaling approach for coding agents, that is faster and more effective at recovering from sub-optimal decisions compared to baselines. While traditional agents either follow linear trajectories or rely on random sampling for scaling compute, our approach DARS works by branching out a trajectory at certain key decision points by taking an alternative action given the history of the trajectory and execution feedback of the previous attempt from that point. We evaluate our approach on SWE-Bench Lite benchmark, demonstrating that this scaling strategy achieves a pass@k score of 55% with Claude 3.5 Sonnet V2. Our framework achieves a pass@1 rate of 47%, outperforming state-of-the-art (SOTA) open-source frameworks.
• 摘要：大型语言模型（LLM）通过启用自动化，彻底改变了各个领域，包括自然语言处理，数据分析和软件开发。在软件工程中，LLM驱动的编码代理由于其具有自动化复杂开发任务的潜力，协助调试和提高生产率而引起了极大的关注。但是，现有的方法通常在次优决策方面遇到困难，需要大量的手动干预或效率低下的计算策略。为了提高编码剂性能，我们提出了动态动作重新采样（DARS），这是一种新颖的推理时间计算量表方法，用于编码剂，与基线相比，它可以更快，更有效地从次级决策中恢复。尽管传统代理遵循线性轨迹，要么依靠随机抽样进行缩放计算，但我们的方法通过在某些关键决策点上分支轨迹来进行轨迹，鉴于从那时起先前尝试的轨迹和执行反馈的历史，通过采取替代行动。我们在SWE-Bench Lite基准测试上评估了我们的方法，这表明这种缩放策略在Claude 3.5 SONNET V2的情况下达到了55％的传球@k得分。我们的框架实现了47％的1次通过，优于最先进的（SOTA）开源框架。

Title: Good/Evil Reputation Judgment of Celebrities by LLMs via Retrieval Augmented Generation

• Authors: Rikuto Tsuchida, Hibiki Yokoyama, Takehito Utsuro
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14382
• Pdf URL: https://arxiv.org/pdf/2503.14382
• Copy Paste: [[2503.14382]] Good/Evil Reputation Judgment of Celebrities by LLMs via Retrieval Augmented Generation(https://arxiv.org/abs/2503.14382)
• Keywords: language model, gpt, llm, chat, retrieval augmented generation
• Abstract: The purpose of this paper is to examine whether large language models (LLMs) can understand what is good and evil with respect to judging good/evil reputation of celebrities. Specifically, we first apply a large language model (namely, ChatGPT) to the task of collecting sentences that mention the target celebrity from articles about celebrities on Web pages. Next, the collected sentences are categorized based on their contents by ChatGPT, where ChatGPT assigns a category name to each of those categories. Those assigned category names are referred to as "aspects" of each celebrity. Then, by applying the framework of retrieval augmented generation (RAG), we show that the large language model is quite effective in the task of judging good/evil reputation of aspects and descriptions of each celebrity. Finally, also in terms of proving the advantages of the proposed method over existing services incorporating RAG functions, we show that the proposed method of judging good/evil of aspects/descriptions of each celebrity significantly outperform an existing service incorporating RAG functions.
• 摘要：本文的目的是检查大型语言模型（LLMS）是否可以理解关于判断名人的善良/邪恶声誉的善与恶。具体来说，我们首先将大型语言模型（即chatgpt）应用于收集句子的任务，这些句子从网页上的名人文章中提到了目标名人。接下来，收集的句子是根据chatgpt的内容分类的，其中chatgpt将类别名称分配给每个类别。那些分配的类别名称称为每个名人的“方面”。然后，通过应用检索增强一代（RAG）的框架，我们表明，大语言模型在判断每个名人方面的良好/邪恶声誉方面非常有效。最后，在证明拟议方法比包含抹布功能的现有服务的优势方面，我们表明，判断每个名人方面/描述的良好/邪恶方法的拟议方法显着超过了现有的现有服务，其中包含了抹布功能。

Title: How much do LLMs learn from negative examples?

• Authors: Shadi Hamdan, Deniz Yuret
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14391
• Pdf URL: https://arxiv.org/pdf/2503.14391
• Copy Paste: [[2503.14391]] How much do LLMs learn from negative examples?(https://arxiv.org/abs/2503.14391)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) undergo a three-phase training process: unsupervised pre-training, supervised fine-tuning (SFT), and learning from human feedback (RLHF/DPO). Notably, it is during the final phase that these models are exposed to negative examples -- incorrect, rejected, or suboptimal responses to queries. This paper delves into the role of negative examples in the training of LLMs, using a likelihood-ratio (Likra) model on multiple-choice question answering benchmarks to precisely manage the influence and the volume of negative examples. Our findings reveal three key insights: (1) During a critical phase in training, Likra with negative examples demonstrates a significantly larger improvement per training example compared to SFT using only positive examples. This leads to a sharp jump in the learning curve for Likra unlike the smooth and gradual improvement of SFT; (2) negative examples that are plausible but incorrect (near-misses) exert a greater influence; and (3) while training with positive examples fails to significantly decrease the likelihood of plausible but incorrect answers, training with negative examples more accurately identifies them. These results indicate a potentially significant role for negative examples in improving accuracy and reducing hallucinations for LLMs.
• 摘要：大型语言模型（LLMS）经历了三相训练过程：无监督的预训练，监督的微调（SFT）和从人类反馈中学习（RLHF/DPO）。值得注意的是，在最后阶段，这些模型暴露于负面示例 - 对查询的不正确，拒绝或次优响应。本文在多项选择的问题上，使用了一个可能的比例（LIKRA）模型来回答基准测试基准，以精确管理负面示例的影响和数量。我们的发现揭示了三个关键的见解：（1）在训练的关键阶段，likra的负面例子表明，与仅使用正面示例相比，与SFT相比，每个训练示例的改进明显更大。与SFT的平稳和逐步改善不同，这导致了Likra的学习曲线的急剧发展。 （2）合理但不正确（接近罪）的负面例子具有更大的影响； （3）尽管有积极示例的训练并没有显着降低合理但不正确的答案的可能性，但更准确地识别出负面例子的训练。这些结果表明，负面例子在提高LLM的准确性和降低幻觉方面具有潜在的重要作用。

Title: From "Hallucination" to "Suture": Insights from Language Philosophy to Enhance Large Language Models

• Authors: Qiantong Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14392
• Pdf URL: https://arxiv.org/pdf/2503.14392
• Copy Paste: [[2503.14392]] From "Hallucination" to "Suture": Insights from Language Philosophy to Enhance Large Language Models(https://arxiv.org/abs/2503.14392)
• Keywords: language model, llm, hallucination
• Abstract: This paper explores hallucination phenomena in large language models (LLMs) through the lens of language philosophy and psychoanalysis. By incorporating Lacan's concepts of the "chain of signifiers" and "suture points," we propose the Anchor-RAG framework as a novel approach to mitigate hallucinations. In contrast to the predominant reliance on trial-and-error experiments, constant adjustments of mathematical formulas, or resource-intensive methods that emphasize quantity over quality, our approach returns to the fundamental principles of linguistics to analyze the root causes of hallucinations in LLMs. Drawing from robust theoretical foundations, we derive algorithms and models that are not only effective in reducing hallucinations but also enhance LLM performance and improve output quality. This paper seeks to establish a comprehensive theoretical framework for understanding hallucinations in LLMs and aims to challenge the prevalent "guess-and-test" approach and rat race mentality in the field. We aspire to pave the way for a new era of interpretable LLMs, offering deeper insights into the inner workings of language-based AI systems.
• 摘要：本文通过语言哲学和精神分析的角度探讨了大语言模型（LLM）中的幻觉现象。通过将Lacan的“指示符链”和“缝合点”的概念结合在一起，我们提出了锚固窗格框架，作为减轻幻觉的新方法。与对反复试验实验的主要依赖，数学公式的持续调整或强调质量量高于质量的资源密集型方法的主要依赖相反，我们的方法回到了语言学的基本原理，以分析LLMS幻觉的根本原因。我们从可靠的理论基础中得出了算法和模型，这些算法和模型不仅可以有效地减少幻觉，还可以提高LLM性能并提高产量质量。本文旨在建立一个全面的理论框架，以理解LLM中的幻觉，并旨在挑战该领域中普遍的“猜测”方法和老鼠种族心态。我们渴望为可解释的LLM的新时代铺平道路，从而更深入地了解基于语言的AI系统的内部运作。

Title: Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Models

• Authors: Siwei Zhang, Yun Xiong, Yateng Tang, Xi Chen, Zian Jia, Zehao Gu, Jiarong Xu, Jiawei Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.14411
• Pdf URL: https://arxiv.org/pdf/2503.14411
• Copy Paste: [[2503.14411]] Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Models(https://arxiv.org/abs/2503.14411)
• Keywords: language model, llm
• Abstract: Temporal graph neural networks (TGNNs) have shown remarkable performance in temporal graph modeling. However, real-world temporal graphs often possess rich textual information, giving rise to temporal text-attributed graphs (TTAGs). Such combination of dynamic text semantics and evolving graph structures introduces heightened complexity. Existing TGNNs embed texts statically and rely heavily on encoding mechanisms that biasedly prioritize structural information, overlooking the temporal evolution of text semantics and the essential interplay between semantics and structures for synergistic reinforcement. To tackle these issues, we present \textbf{Cross}, a novel framework that seamlessly extends existing TGNNs for TTAG modeling. The key idea is to employ the advanced large language models (LLMs) to extract the dynamic semantics in text space and then generate expressive representations unifying both semantics and structures. Specifically, we propose a Temporal Semantics Extractor in the {Cross} framework, which empowers the LLM to offer the temporal semantic understanding of node's evolving contexts of textual neighborhoods, facilitating semantic dynamics. Subsequently, we introduce the Semantic-structural Co-encoder, which collaborates with the above Extractor for synthesizing illuminating representations by jointly considering both semantic and structural information while encouraging their mutual reinforcement. Extensive experimental results on four public datasets and one practical industrial dataset demonstrate {Cross}'s significant effectiveness and robustness.
• 摘要：时间图神经网络（TGNN）在时间图建模中表现出色。但是，现实世界中的时间图通常具有丰富的文本信息，从而产生了时间文本属性图（TTAG）。动态文本语义和不断发展的图形结构的这种组合引入了增强的复杂性。现有的TGNN在静态上嵌入文本，并严重依赖编码偏见优先级结构信息的机制，忽略了文本语义的时间演变以及语义与协同增强的结构之间的基本相互作用。为了解决这些问题，我们提出了\ textbf {Cross}，这是一个新颖的框架，无缝扩展现有的TGNN用于TTAG建模。关键思想是使用先进的大型语言模型（LLM）来提取文本空间中的动态语义，然后生成统一语义和结构的表达性表示。具体来说，我们在{Cross}框架中提出了一个时间语义提取器，该框架赋予了LLM的能力，以提供对Node不断发展的文本邻域不断发展的上下文的时间语义理解，从而促进语义动力学。随后，我们介绍了语义结构的共编码器，该共同编码器与上述提取器合作，通过共同考虑语义和结构信息，同时鼓励其相互强化，以合成照明表示。在四个公共数据集和一个实用的工业数据集上进行了广泛的实验结果，证明了{Cross}的重要性和鲁棒性。

Title: PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play

• Authors: Wei Fang, Yang Zhang, Kaizhi Qian, James Glass, Yada Zhu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.14432
• Pdf URL: https://arxiv.org/pdf/2503.14432
• Copy Paste: [[2503.14432]] PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play(https://arxiv.org/abs/2503.14432)
• Keywords: language model, llm, prompt, agent
• Abstract: Large language models (LLMs) are increasingly integrated with specialized external tools, yet many tasks demand zero-shot tool usage with minimal or noisy documentation. Existing solutions rely on manual rewriting or labeled data for validation, making them inapplicable in true zero-shot settings. To address these challenges, we propose PLAY2PROMPT, an automated framework that systematically "plays" with each tool to explore its input-output behaviors. Through this iterative trial-and-error process, PLAY2PROMPT refines tool documentation and generates usage examples without any labeled data. These examples not only guide LLM inference but also serve as validation to further enhance tool utilization. Extensive experiments on real-world tasks demonstrate that PLAY2PROMPT significantly improves zero-shot tool performance across both open and closed models, offering a scalable and effective solution for domain-specific tool integration.
• 摘要：大型语言模型（LLMS）越来越多地与专门的外部工具集成在一起，但是许多任务都需要使用最少或嘈杂的文档使用零拍的工具。现有的解决方案依靠手动重写或标记的数据进行验证，从而使其在真实的零照片设置中不适用。为了应对这些挑战，我们提出了Play2-Prompt，这是一个自动化的框架，该框架使用每个工具系统地“播放”以探索其输入输出行为。通过此迭代试验过程，Play2-Prompt完善了工具文档，并生成了没有任何标记数据的示例。这些示例不仅指导LLM推断，而且还可以作为进一步增强工具利用率的验证。对现实世界任务的广泛实验表明，Play2Prompt显着改善了开放型和封闭模型的零击工具性能，为域特异性工具集成提供了可扩展有效的解决方案。

Title: RWKV-7 "Goose" with Expressive Dynamic State Evolution

• Authors: Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, Christian Zhou-Zheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.14456
• Pdf URL: https://arxiv.org/pdf/2503.14456
• Copy Paste: [[2503.14456]] RWKV-7 "Goose" with Expressive Dynamic State Evolution(https://arxiv.org/abs/2503.14456)
• Keywords: language model
• Abstract: We present RWKV-7 "Goose", a new sequence modeling architecture, along with pre-trained language models that establish a new state-of-the-art in downstream performance at the 3 billion parameter scale on multilingual tasks, and match current SoTA English language performance despite being trained on dramatically fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only constant memory usage and constant inference time per token. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to $\mathsf{TC}^0$. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at this https URL, and our training and inference code at this https URL all under the Apache 2.0 License.
• 摘要：我们提出了RWKV-7“ Goose”，这是一种新的序列建模体系结构，以及预训练的语言模型，以30亿个多语言任务的参数量表建立了新的下游性能的最先进，尽管与其他前3B型号相比，对当前的SOTA英语表现进行了匹配，尽管经过了较少的代价。然而，RWKV-7模型仅需要恒定的内存使用和每个令牌的恒定推理时间。 RWKV-7引入了新的概括为三角洲规则的概述，并具有矢量值的门控和内在的学习率，以及放松的价值替代规则。我们表明，RWKV-7可以执行状态跟踪并识别所有常规语言，同时保留培训的并行性。这超过了标准复杂性猜想下变压器的功能，这些功能仅限于$ \ m rathsf {tc}^0 $。为了演示RWKV-7的语言建模能力，我们还提供了一个扩展的开源源为3.1万亿代币多语言语料库，并在此数据集上训练四种RWKV-7型号的范围从10.9亿到29亿参数不等。为了促进开放性，再现和采用，我们在此HTTPS URL上发布了模型和数据集组件列表，以及我们在此HTTPS URL的培训和推理代码都在Apache 2.0许可下。

Title: Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations

• Authors: Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14477
• Pdf URL: https://arxiv.org/pdf/2503.14477
• Copy Paste: [[2503.14477]] Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations(https://arxiv.org/abs/2503.14477)
• Keywords: llm, hallucination
• Abstract: LLMs often adopt an assertive language style also when making false claims. Such ``overconfident hallucinations'' mislead users and erode trust. Achieving the ability to express in language the actual degree of uncertainty around a claim is therefore of great importance. We find that ``verbal uncertainty'' is governed by a single linear feature in the representation space of LLMs, and show that this has only moderate correlation with the actual ``semantic uncertainty'' of the model. We apply this insight and show that (1) the mismatch between semantic and verbal uncertainty is a better predictor of hallucinations than semantic uncertainty alone and (2) we can intervene on verbal uncertainty at inference time and reduce hallucinations on short-form answers, achieving an average relative reduction of 32%.
• 摘要：在提出虚假主张时，LLMS通常也会采用自信的语言风格。这样的``过度自信幻觉''误导用户并侵蚀了信任。因此，实现语言表达索赔的实际不确定性程度至关重要。我们发现``言语不确定性''由LLMS表示空间中的单个线性特征控制，并表明这仅与模型的实际``语义不确定性''具有中等相关性。我们应用了这种见解，并表明（1）语义不确定性之间的不匹配是幻觉的更好预测指标，而不是单独的语义不确定性，（2）我们可以在推理时间干预言语不确定性，并减少短形式答案的幻觉，从而达到平均相对降低32％。

Title: Temporal Consistency for LLM Reasoning Process Error Identification

• Authors: Jiacheng Guo, Yue Wu, Jiahao Qiu, Kaixuan Huang, Xinzhe Juan, Ling Yang, Mengdi Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.14495
• Pdf URL: https://arxiv.org/pdf/2503.14495
• Copy Paste: [[2503.14495]] Temporal Consistency for LLM Reasoning Process Error Identification(https://arxiv.org/abs/2503.14495)
• Keywords: gpt, llm
• Abstract: Verification is crucial for effective mathematical reasoning. We present a new temporal consistency method where verifiers iteratively refine their judgments based on the previous assessment. Unlike one-round verification or multi-model debate approaches, our method leverages consistency in a sequence of self-reflection actions to improve verification accuracy. Empirical evaluations across diverse mathematical process error identification benchmarks (Mathcheck, ProcessBench, and PRM800K) show consistent performance improvements over baseline methods. When applied to the recent DeepSeek R1 distilled models, our method demonstrates strong performance, enabling 7B/8B distilled models to outperform all 70B/72B models and GPT-4o on ProcessBench. Notably, the distilled 14B model with our method achieves performance comparable to Deepseek-R1. Our codes are available at this https URL
• 摘要：验证对于有效的数学推理至关重要。我们提出了一种新的时间一致性方法，在该方法中，验证者根据先前的评估迭代地完善其判断。与单轮验证或多模型辩论方法不同，我们的方法以一系列自我反射动作的序列利用一致性，以提高验证精度。跨基线方法对不同数学过程错误识别基准（Mathcheck，ProcessBench和PRM800K）的经验评估表现出一致的性能改进。当应用于最近的DeepSeek R1蒸馏模型时，我们的方法显示出强大的性能，使7b/8b蒸馏模型在ProcessBench上的表现优于所有70B/72B型号和GPT-4O。值得注意的是，带有我们方法的蒸馏14B模型可实现与DeepSeek-R1相当的性能。我们的代码可在此HTTPS URL上找到