2025-03-25

Title: ChatGPT or A Silent Everywhere Helper: A Survey of Large Language Models

• Authors: Azim Akhtarshenas, Afshin Dini, Navid Ayoobi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17403
• Pdf URL: https://arxiv.org/pdf/2503.17403
• Copy Paste: [[2503.17403]] ChatGPT or A Silent Everywhere Helper: A Survey of Large Language Models(https://arxiv.org/abs/2503.17403)
• Keywords: language model, gpt, llm, chat
• Abstract: Large Language Models (LLMs) have revo lutionized natural language processing Natural Language Processing (NLP), with Chat Generative Pre-trained Transformer (ChatGPT) standing out as a notable exampledue to its advanced capabilities and widespread applications. This survey provides a comprehensive analysis of ChatGPT, exploring its architecture, training processes, and functionalities. We examine its integration into various domains across industries such as customer service, education, healthcare, and entertainment. A comparative analysis with other LLMs highlights ChatGPT's unique features and performance metrics. Regarding benchmarks, the paper examines ChatGPT's comparative performance against other LLMs and discusses potential risks such as misinformation, bias, and data privacy concerns. Additionally, we offer a number of figures and tables that outline the backdrop of the discussion, the main ideas of the article, the numerous LLM models, a thorough list of datasets used for pre-training, fine-tuning, and evaluation, as well as particular LLM applications with pertinent references. Finally, we identify future research directions and technological advancements, underscoring the evolving landscape of LLMs and their profound impact on artificial intelligence Artificial Intelligence (AI) and society.
• 摘要：大型语言模型（LLMS）已撤销了自然语言处理自然语言处理（NLP），聊天生成的预培训的变压器（CHATGPT）是其高级功能和广泛应用程序的显着检查。这项调查提供了对Chatgpt的全面分析，探索其架构，培训过程和功能。我们研究了它与客户服务，教育，医疗保健和娱乐等行业之间的各个领域的集成。与其他LLMS的比较分析突出了Chatgpt的独特功能和性能指标。关于基准，本文研究了Chatgpt与其他LLM的比较性能，并讨论了潜在的风险，例如错误信息，偏见和数据隐私问题。此外，我们提供了许多数字和表格，概述了讨论的背景，文章的主要思想，众多LLM模型，用于预训练，微调和评估的数据集列表，以及特定的LLM应用程序。最后，我们确定了未来的研究方向和技术进步，强调了LLM的不断发展的景观及其对人工智能人工智能（AI）和社会的深远影响。

Title: A Comprehensive Survey on Long Context Language Modeling

• Authors: Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan Feng, Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, Jian Yang, Wei Ye, Bo Zheng, Wangchunshu Zhou, Wenhao Huang, Sujian Li, Zhaoxiang Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17407
• Pdf URL: https://arxiv.org/pdf/2503.17407
• Copy Paste: [[2503.17407]] A Comprehensive Survey on Long Context Language Modeling(https://arxiv.org/abs/2503.17407)
• Keywords: language model, llm, long context
• Abstract: Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way. In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models. Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively. For the first aspect, we discuss data strategies, architectural designs, and workflow approaches oriented with long context processing. For the second aspect, we provide a detailed examination of the infrastructure required for LCLM training and inference. For the third aspect, we present evaluation paradigms for long-context comprehension and long-form generation, as well as behavioral analysis and mechanism interpretability of LCLMs. Beyond these three key aspects, we thoroughly explore the diverse application scenarios where existing LCLMs have been deployed and outline promising future development directions. This survey provides an up-to-date review of the literature on long-context LLMs, which we wish to serve as a valuable resource for both researchers and engineers. An associated GitHub repository collecting the latest papers and repos is available at: \href{this https URL}{\color[RGB]{175,36,67}{LCLM-Horizon}}.
• 摘要：在自然语言处理中，有效地处理长篇小说一直是一种持续的追求。随着越来越多的文档，对话和其他文本数据的数量，开发长上下文语言模型（LCLM）很重要，这些语言模型（LCLM）可以以有效而有效的方式处理和分析广泛的输入。在本文中，我们介绍了一项有关大语模型长篇小说建模的最新进展的综合调查。我们的调查围绕三个关键方面进行了结构：如何获得有效，有效的LCLM，如何有效培训和部署LCLM，以及如何全面评估和分析LCLM。对于第一个方面，我们讨论了以长上下文处理为导向的数据策略，架构设计和工作流程方法。在第二方面，我们对LCLM培训和推理所需的基础设施进行了详细的检查。对于第三方面，我们提出了长篇文化理解和长期产生以及LCLM的行为分析和机制的评估范例。除了这三个关键方面，我们还彻底探讨了已经部署现有LCLM的各种应用程序方案并概述了有希望的未来开发方向。这项调查提供了有关长篇文献LLM的文献的最新审查，我们希望为研究人员和工程师提供宝贵的资源。收集最新论文和存储库的关联的GitHub存储库可在：\ href {此https url} {\ color [rgb] {175,36,67} {lclm-horizo​​n}}}}}}}}}}中。

Title: Beyond Negation Detection: Comprehensive Assertion Detection Models for Clinical NLP

• Authors: Veysel Kocaman, Yigit Gul, M. Aytug Kaya, Hasham Ul Haq, Mehmet Butgul, Cabir Celik, David Talby
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17425
• Pdf URL: https://arxiv.org/pdf/2503.17425
• Copy Paste: [[2503.17425]] Beyond Negation Detection: Comprehensive Assertion Detection Models for Clinical NLP(https://arxiv.org/abs/2503.17425)
• Keywords: gpt, llm
• Abstract: Assertion status detection is a critical yet often overlooked component of clinical NLP, essential for accurately attributing extracted medical facts. Past studies have narrowly focused on negation detection, leading to underperforming commercial solutions such as AWS Medical Comprehend, Azure AI Text Analytics, and GPT-4o due to their limited domain adaptation. To address this gap, we developed state-of-the-art assertion detection models, including fine-tuned LLMs, transformer-based classifiers, few-shot classifiers, and deep learning (DL) approaches. We evaluated these models against cloud-based commercial API solutions, the legacy rule-based NegEx approach, and GPT-4o. Our fine-tuned LLM achieves the highest overall accuracy (0.962), outperforming GPT-4o (0.901) and commercial APIs by a notable margin, particularly excelling in Present (+4.2%), Absent (+8.4%), and Hypothetical (+23.4%) assertions. Our DL-based models surpass commercial solutions in Conditional (+5.3%) and Associated-with-Someone-Else (+10.1%) categories, while the few-shot classifier offers a lightweight yet highly competitive alternative (0.929), making it ideal for resource-constrained environments. Integrated within Spark NLP, our models consistently outperform black-box commercial solutions while enabling scalable inference and seamless integration with medical NER, Relation Extraction, and Terminology Resolution. These results reinforce the importance of domain-adapted, transparent, and customizable clinical NLP solutions over general-purpose LLMs and proprietary APIs.
• 摘要：断言状态检测是临床NLP的关键但经常被忽略的组成部分，对于准确归因于提取的医学事实至关重要。过去的研究狭窄地专注于否定检测，导致诸如AWS医学理解，Azure AI文本分析和GPT-4O等表现不佳的商业解决方案，因为它们的域名有限。为了解决这一差距，我们开发了最新的断言检测模型，包括精细调整的LLM，基于变压器的分类器，少量分类器和深度学习（DL）方法。我们根据基于云的商业API解决方案，基于旧规则的NEGEX方法和GPT-4O评估了这些模型。我们的微调LLM达到了最高的总体精度（0.962），表现优于GPT-4O（0.901）和商业API，并具有明显的差距，尤其是当前（+4.2％），缺乏（+8.4％），以及假设（+23.4％）的主张。我们的基于DL的模型超过条件（+5.3％）和相关的someone-Else（+10.1％）类别的商业解决方案，而少数拍摄的分类器提供了一种轻巧但高度竞争的替代方案（0.929），使其非常适合资源受限环境。集成在Spark NLP中，我们的模型始终胜过黑盒商业解决方案，同时可以与医学NER，关系提取和术语解决方案实现可扩展的推理和无缝集成。这些结果增强了针对域的适应性，透明和可定制的NLP解决方案的重要性，而不是通用LLM和专有API的重要性。

Title: Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer

• Authors: Soumen Kumar Mondal, Sayambhu Sen, Abhishek Singhania, Preethi Jyothi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17456
• Pdf URL: https://arxiv.org/pdf/2503.17456
• Copy Paste: [[2503.17456]] Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer(https://arxiv.org/abs/2503.17456)
• Keywords: language model, llm
• Abstract: Multilingual large language models (LLMs) aim towards robust natural language understanding across diverse languages, yet their performance significantly degrades on low-resource languages. This work explores whether existing techniques to identify language-specific neurons can be leveraged to enhance cross-lingual task performance of lowresource languages. We conduct detailed experiments covering existing language-specific neuron identification techniques (such as Language Activation Probability Entropy and activation probability-based thresholding) and neuron-specific LoRA fine-tuning with models like Llama 3.1 and Mistral Nemo. We find that such neuron-specific interventions are insufficient to yield cross-lingual improvements on downstream tasks (XNLI, XQuAD) in lowresource languages. This study highlights the challenges in achieving cross-lingual generalization and provides critical insights for multilingual LLMs.
• 摘要：多语言大语言模型（LLM）旨在跨不同语言的自然语言理解，但是它们的表现在低资源语言上大大降低。这项工作探讨了是否可以利用识别特定语言神经元的现有技术来增强低位分子语言的跨语性任务性能。我们进行了详细的实验，涵盖了现有的语言特异性神经元识别技术（例如语言激活概率熵和基于激活概率的阈值）和神经元特异性的Lora微调，并使用Llama 3.1和Missral Nemo等模型进行了微调。我们发现，这种特定于神经元的干预措施不足以对下游任务（XNLI，Xquad）进行低分子语言的跨语性改进。这项研究强调了实现跨语性概括的挑战，并为多语言LLM提供了关键的见解。

Title: ConvoGen: Enhancing Conversational AI with Synthetic Data: A Multi-Agent Approach

• Authors: Reem Gody, Mahmoud Goudy, Ahmed Y. Tawfik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17460
• Pdf URL: https://arxiv.org/pdf/2503.17460
• Copy Paste: [[2503.17460]] ConvoGen: Enhancing Conversational AI with Synthetic Data: A Multi-Agent Approach(https://arxiv.org/abs/2503.17460)
• Keywords: agent
• Abstract: In this paper, we present ConvoGen: an innovative framework for generating synthetic conversational data using multi-agent systems. Our method leverages few-shot learning and introduces iterative sampling from a dynamically updated few-shot hub to create diverse and realistic conversational scenarios. The generated data has numerous applications, including training and evaluating conversational AI models, and augmenting existing datasets for tasks like conversational intent classification or conversation summarization. Our experiments demonstrate the effectiveness of this method in producing high-quality diverse synthetic conversational data, highlighting its potential to enhance the development and evaluation of conversational AI systems.
• 摘要：在本文中，我们提出了Convogen：一种创新的框架，用于使用多代理系统生成合成对话数据。我们的方法利用了很少的学习学习，并从动态更新的几个弹药中心引入了迭代采样，以创建多样化和现实的对话场景。生成的数据具有许多应用程序，包括培训和评估对话式AI模型，并为对话意图分类或对话摘要等任务增强现有数据集。我们的实验证明了该方法在产生高质量的合成对话数据方面的有效性，突出了其增强对话AI系统的开发和评估的潜力。

Title: SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia

• Authors: Lama Ayash, Hassan Alhuzali, Ashwag Alasmari, Sultan Aloufi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17485
• Pdf URL: https://arxiv.org/pdf/2503.17485
• Copy Paste: [[2503.17485]] SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia(https://arxiv.org/abs/2503.17485)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing; however, they often struggle to accurately capture and reflect cultural nuances. This research addresses this challenge by focusing on Saudi Arabia, a country characterized by diverse dialects and rich cultural traditions. We introduce SaudiCulture, a novel benchmark designed to evaluate the cultural competence of LLMs within the distinct geographical and cultural contexts of Saudi Arabia. SaudiCulture is a comprehensive dataset of questions covering five major geographical regions, such as West, East, South, North, and Center, along with general questions applicable across all regions. The dataset encompasses a broad spectrum of cultural domains, including food, clothing, entertainment, celebrations, and crafts. To ensure a rigorous evaluation, SaudiCulture includes questions of varying complexity, such as open-ended, single-choice, and multiple-choice formats, with some requiring multiple correct answers. Additionally, the dataset distinguishes between common cultural knowledge and specialized regional aspects. We conduct extensive evaluations on five LLMs, such as GPT-4, Llama 3.3, FANAR, Jais, and AceGPT, analyzing their performance across different question types and cultural contexts. Our findings reveal that all models experience significant performance declines when faced with highly specialized or region-specific questions, particularly those requiring multiple correct responses. Additionally, certain cultural categories are more easily identifiable than others, further highlighting inconsistencies in LLMs cultural understanding. These results emphasize the importance of incorporating region-specific knowledge into LLMs training to enhance their cultural competence.
• 摘要：大型语言模型（LLM）在自然语言处理中表现出了显着的功能；但是，他们经常难以准确捕捉和反映文化细微差别。这项研究通过关注沙特阿拉伯，这是一个以多种方言和丰富的文化传统为特征的国家来解决这一挑战。我们介绍了一项新颖的基准，旨在评估LLM在沙特阿拉伯独特的地理和文化背景下的文化能力。沙于文化是一个综合的数据集，涵盖了五个主要地理区域，例如西，东，南，北部和中心，以及在所有地区都适用的一般问题。数据集涵盖了许多文化领域，包括食物，服装，娱乐，庆祝活动和手工艺品。为了确保严格的评估，沙特文化包括不同复杂性的问题，例如开放式，单选项和多项选择格式，有些需要多个正确的答案。此外，数据集还区分了常见的文化知识和专业的区域方面。我们对五个LLM进行了广泛的评估，例如GPT-4，Llama 3.3，Fanar，Jais和Acegpt，分析了它们在不同问题类型和文化背景的绩效。我们的发现表明，当面对高度专业化或特定于地区的问题，尤其是那些需要多个正确回答的问题时，所有模型都会均均有显着的性能下降。此外，某些文化类别比其他文化类别更容易识别，这进一步强调了LLMS文化理解的不一致。这些结果强调了将特定区域知识纳入LLMS培训以增强其文化能力的重要性。

Title: Judge Anything: MLLM as a Judge Across Any Modality

• Authors: Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, Philip S. Yu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.17489
• Pdf URL: https://arxiv.org/pdf/2503.17489
• Copy Paste: [[2503.17489]] Judge Anything: MLLM as a Judge Across Any Modality(https://arxiv.org/abs/2503.17489)
• Keywords: gpt, llm, hallucination
• Abstract: Evaluating generative foundation models on open-ended multimodal understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g., images, audio, video) poses significant challenges due to the complexity of cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs (MLLMs) as automated judges has emerged, with encouraging results in assessing vision-language understanding tasks. Moving further, this paper extends MLLM-as-a-Judge across modalities to a unified manner by introducing two benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the overall performance and judging capabilities of MLLMs across any-to-any modality tasks. Specifically, TaskAnything evaluates the MMU and MMG capabilities across 15 any-to-any modality categories, employing 1,500 queries curated from well-established benchmarks. Furthermore, JudgeAnything evaluates the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from the perspectives of Pair Comparison and Score Evaluation, providing a standardized testbed that incorporates human judgments and detailed rubrics. Our extensive experiments reveal that while these MLLMs show promise in assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting and 42.79% in Score Evaluation setting), they encounter significant challenges with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and 30.05% in Score Evaluation setting), exposing cross-modality biases and hallucination issues. To address this, we present OmniArena, an automated platform for evaluating omni-models and multimodal reward models. Our work highlights the need for fairer evaluation protocols and stronger alignment with human preferences. The source code and dataset are publicly available at: this https URL.
• 摘要：评估有关开放式多模式理解（MMU）和生成（MMG）任务（例如，图像，音频，视频）的生成基础模型，由于跨模式相互作用的复杂性，构成了重大挑战。为此，随着自动化法官的出现，利用多模式LLM（MLLM）的想法已经出现，从而令人鼓舞地评估视力语言理解任务。 Moving further, this paper extends MLLM-as-a-Judge across modalities to a unified manner by introducing two benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the overall performance and judging capabilities of MLLMs across any-to-any modality tasks.具体而言，Taskything评估了15个对任何一种模式类别的MMU和MMG功能，采用了1,500个从建立的基准测试中策划的查询。此外，审判从对比较和得分评估的角度评估了5个高级（例如GPT-4O和Gemini-2.0-Flash）的评判功能，从而提供了一个标准化的测试台，并提供了人类判断和详细的专栏。 Our extensive experiments reveal that while these MLLMs show promise in assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting and 42.79% in Score Evaluation setting), they encounter significant challenges with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and 30.05% in Score Evaluation setting), exposing cross-modality biases and hallucination issues.为了解决这个问题，我们提出了Omniarena，这是一个自动化平台，用于评估Omni模型和多模式奖励模型。我们的工作强调了需要更公平的评估协议，并且与人类偏好更加一致。源代码和数据集可公开可用：此HTTPS URL。

Title: Follow-up Question Generation For Enhanced Patient-Provider Conversations

• Authors: Joseph Gatto, Parker Seegmiller, Timothy Burdick, Inas S. Khayal, Sarah DeLozier, Sarah M. Preum
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17509
• Pdf URL: https://arxiv.org/pdf/2503.17509
• Copy Paste: [[2503.17509]] Follow-up Question Generation For Enhanced Patient-Provider Conversations(https://arxiv.org/abs/2503.17509)
• Keywords: agent
• Abstract: Follow-up question generation is an essential feature of dialogue systems as it can reduce conversational ambiguity and enhance modeling complex interactions. Conversational contexts often pose core NLP challenges such as (i) extracting relevant information buried in fragmented data sources, and (ii) modeling parallel thought processes. These two challenges occur frequently in medical dialogue as a doctor asks questions based not only on patient utterances but also their prior EHR data and current diagnostic hypotheses. Asking medical questions in asynchronous conversations compounds these issues as doctors can only rely on static EHR information to motivate follow-up questions. To address these challenges, we introduce FollowupQ, a novel framework for enhancing asynchronous medical conversation. FollowupQ is a multi-agent framework that processes patient messages and EHR data to generate personalized follow-up questions, clarifying patient-reported medical conditions. FollowupQ reduces requisite provider follow-up communications by 34%. It also improves performance by 17% and 5% on real and synthetic data, respectively. We also release the first public dataset of asynchronous medical messages with linked EHR data alongside 2,300 follow-up questions written by clinical experts for the wider NLP research community.
• 摘要：后续问题生成是对话系统的重要特征，因为它可以减少对话歧义并增强建模复杂的相互作用。会话环境通常会带来核心NLP挑战，例如（i）提取埋在零散数据源中的相关信息，以及（ii）对平行思维过程进行建模。这两个挑战经常出现在医学对话中，因为医生不仅基于患者的话语，而且还提出了以前的EHR数据和当前诊断假设的问题。在异步对话中提出医疗问题会使这些问题加剧了这些问题，因为医生只能依靠静态EHR信息来激发后续问题。为了应对这些挑战，我们介绍了后续行动，这是一种增强异步医疗对话的新型框架。后续行动是一个多代理框架，可处理患者消息和EHR数据以产生个性化的后续问题，从而阐明患者报告的医疗状况。后续行动将必要的提供商的后续通信减少34％。它还分别在实际和合成数据上提高了17％和5％的性能。我们还发布了第一个与链接的EHR数据的异步医疗信息的公共数据集以及临床专家为更广泛的NLP研究社区编写的2300个后续问题。

Title: Language Models May Verbatim Complete TextThey Were Not Explicitly Trained On

• Authors: Ken Ziyu Liu, Christopher A. Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, Nicolas Papernot
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17514
• Pdf URL: https://arxiv.org/pdf/2503.17514
• Copy Paste: [[2503.17514]] Language Models May Verbatim Complete TextThey Were Not Explicitly Trained On(https://arxiv.org/abs/2503.17514)
• Keywords: language model, llm
• Abstract: An important question today is whether a given text was used to train a large language model (LLM). A \emph{completion} test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the $n$-gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this $n$-gram based membership definition can be effectively gamed. We study scenarios where sequences are \emph{non-members} for a given $n$ and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of $n$ for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of $n$. Our findings highlight the inadequacy of $n$-gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm.
• 摘要：今天的一个重要问题是，给定文本是否用于训练大型语言模型（LLM）。经常采用A \ Emph {postemion}测试：检查LLM是否完成了足够复杂的文本。但是，这需要对成员的基础定义。最常见的是，它根据目标文本和数据集中的任何文本之间的$ n $ gram重叠定义为成员。在这项工作中，我们证明了基于$ n $ gram的会员定义可以有效地进行认可。我们研究给定$ n $的序列\ emph {nonembers}的方案，我们发现完成测试仍然成功。我们发现了许多自然案例，通过删除所有完成的训练样本后，通过从头开始重新研究LLM。这些案例包括精确的重复，近乎解复器，甚至是短的重叠。他们展示了很难为会员定义找到单一可行的$ n $选择。使用这些见解，我们设计了可以导致给定目标序列完成的对抗数据集，而无需包含它，以选择$ n $的任何合理选择。我们的发现突出了$ n $ gram会员资格的不足，这表明会员定义无法解决培训算法可用的辅助信息。

Title: Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models

• Authors: Linlu Qiu, Fei Sha, Kelsey Allen, Yoon Kim, Tal Linzen, Sjoerd van Steenkiste
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17523
• Pdf URL: https://arxiv.org/pdf/2503.17523
• Copy Paste: [[2503.17523]] Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models(https://arxiv.org/abs/2503.17523)
• Keywords: language model, llm, agent
• Abstract: Artificial intelligence systems based on large language models (LLMs) are increasingly used as agents that interact with users and with the world. To do so successfully, LLMs need to construct internal representations of the world and form probabilistic beliefs about those representations. To provide a user with personalized recommendations, for example, the LLM needs to gradually infer the user's preferences, over the course of multiple interactions. To evaluate whether contemporary LLMs are able to do so, we use the Bayesian inference framework from probability theory, which lays out the optimal way to update an agent's beliefs as it receives new information. We first show that the LLMs do not update their beliefs as expected from the Bayesian framework, and that consequently their predictions do not improve as expected as more information becomes available, even less so than we find is the case for humans. To address this issue, we teach the LLMs to reason in a Bayesian manner by training them to mimic the predictions of an optimal Bayesian model. We find that this approach not only significantly improves the LLM's performance on the particular recommendation task it is trained on, but also enables generalization to other tasks. This suggests that this method endows the LLM with broader Bayesian reasoning skills. More generally, our results indicate that LLMs can learn about reasoning strategies effectively and generalize those skills to new domains, which in part explains LLMs' empirical success.
• 摘要：基于大语言模型（LLM）的人工智能系统越来越多地用作与用户和世界互动的代理。为此，LLMS需要成功地构建世界内部表示，并就这些表示形式形成概率的信念。为了为用户提供个性化的建议，例如，LLM需要在多个交互过程中逐渐推断用户的喜好。为了评估当代LLM是否能够做到这一点，我们使用概率理论的贝叶斯推理框架，该理论列出了在收到新信息时更新代理商信念的最佳方法。我们首先表明LLM并未像贝叶斯框架那样更新他们的信念，因此，随着更多信息的可用信息，他们的预测并没有改善，甚至比我们发现的人类所发现的要少。为了解决这个问题，我们通过训练来模仿最佳贝叶斯模型的预测来教LLMS以贝叶斯的方式进行推理。我们发现，这种方法不仅可以显着提高LLM在接受培训的特定建议任务上的性能，而且还可以对其他任务进行概括。这表明该方法赋予LLM具有更广泛的贝叶斯推理技能。更普遍地，我们的结果表明，LLM可以有效地学习推理策略并将这些技能推广到新领域，这部分解释了LLMS的经验成功。

Title: Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibility

• Authors: Suet-Ying Lam, Qingcheng Zeng, Jingyi Wu, Rob Voigt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17579
• Pdf URL: https://arxiv.org/pdf/2503.17579
• Copy Paste: [[2503.17579]] Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibility(https://arxiv.org/abs/2503.17579)
• Keywords: language model, llm, prompt
• Abstract: Whether large language models (LLMs) process language similarly to humans has been the subject of much theoretical and practical debate. We examine this question through the lens of the production-interpretation distinction found in human sentence processing and evaluate the extent to which instruction-tuned LLMs replicate this distinction. Using an empirically documented asymmetry between production and interpretation in humans for implicit causality verbs as a testbed, we find that some LLMs do quantitatively and qualitatively reflect human-like asymmetries between production and interpretation. We demonstrate that whether this behavior holds depends upon both model size - with larger models more likely to reflect human-like patterns and the choice of meta-linguistic prompts used to elicit the behavior.
• 摘要：大型语言模型（LLMS）是否与人类相似的过程是理论和实用辩论的主题。我们通过在人类句子处理中发现的生产解释区别的角度来研究这个问题，并评估指导调节的LLMS在多大程度上复制这种区别。将人类的生产与解释之间的经验文献记录在隐式因果关系动词中作为测试床，我们发现某些LLMS在定量上和定性地反映了生产和解释之间的类似人类的不对称性。我们证明，这种行为是否取决于两个模型大小 - 更大的模型更有可能反映人类样模式以及用于引起行为的元语言提示的选择。

Title: GPBench: A Comprehensive and Fine-Grained Benchmark for Evaluating Large Language Models as General Practitioners

• Authors: Zheqing Li, Yiying Yang, Jiping Lang, Wenhao Jiang, Yuhang Zhao, Shuang Li, Dingqian Wang, Zhu Lin, Xuanna Li, Yuze Tang, Jiexian Qiu, Xiaolin Lu, Hongji Yu, Shuang Chen, Yuhua Bi, Xiaofei Zeng, Yixian Chen, Junrong Chen, Lin Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17599
• Pdf URL: https://arxiv.org/pdf/2503.17599
• Copy Paste: [[2503.17599]] GPBench: A Comprehensive and Fine-Grained Benchmark for Evaluating Large Language Models as General Practitioners(https://arxiv.org/abs/2503.17599)
• Keywords: language model, llm
• Abstract: General practitioners (GPs) serve as the cornerstone of primary healthcare systems by providing continuous and comprehensive medical services. However, due to community-oriented nature of their practice, uneven training and resource gaps, the clinical proficiency among GPs can vary significantly across regions and healthcare settings. Currently, Large Language Models (LLMs) have demonstrated great potential in clinical and medical applications, making them a promising tool for supporting general practice. However, most existing benchmarks and evaluation frameworks focus on exam-style assessments-typically multiple-choice question-lack comprehensive assessment sets that accurately mirror the real-world scenarios encountered by GPs. To evaluate how effectively LLMs can make decisions in the daily work of GPs, we designed GPBench, which consists of both test questions from clinical practice and a novel evaluation framework. The test set includes multiple-choice questions that assess fundamental knowledge of general practice, as well as realistic, scenario-based problems. All questions are meticulously annotated by experts, incorporating rich fine-grained information related to clinical management. The proposed LLM evaluation framework is based on the competency model for general practice, providing a comprehensive methodology for assessing LLM performance in real-world settings. As the first large-model evaluation set targeting GP decision-making scenarios, GPBench allows us to evaluate current mainstream LLMs. Expert assessment and evaluation reveal that in areas such as disease staging, complication recognition, treatment detail, and medication usage, these models exhibit at least ten major shortcomings. Overall, existing LLMs are not yet suitable for independent use in real-world GP working scenarios without human oversight.
• 摘要：全科医生（GPS）通过提供持续和全面的医疗服务来充当初级医疗系统的基石。但是，由于其实践，不平衡的培训和资源差距的性质，全科医生之间的临床水平在各个地区和医疗机构之间可能会有很大差异。目前，大型语言模型（LLMS）在临床和医疗应用中表现出巨大的潜力，使其成为支持通用实践的有前途的工具。但是，大多数现有的基准测试框架和评估框架都集中在考试式评估上 - 典型的多项选择问题 - 插件综合评估集，这些评估集准确地反映了GPS遇到的真实情况。为了评估LLM在GP的日常工作中如何有效地做出决策，我们设计了GPBench，其中包括临床实践和新颖的评估框架中的测试问题。该测试集包括评估一般实践的基本知识以及基于场景的问题的多项选择问题。所有问题均由专家精心注释，并结合了与临床管理有关的丰富细粒度信息。拟议的LLM评估框架基于通用实践的能力模型，提供了评估现实世界中LLM性能的全面方法。作为针对GP决策情况的第一个大型模型评估集，GPBench使我们能够评估当前的主流LLM。专家评估和评估表明，在疾病分期，并发症识别，治疗细节和用药等领域，这些模型至少表现出十个主要缺点。总体而言，现有的LLM尚不适合在没有人类监督的情况下在现实世界中的GP工作场景中独立使用。

Title: Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning

• Authors: Ke Ji, Yixin Lian, Linxu Li, Jingsheng Gao, Weiyuan Li, Bin Dai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17662
• Pdf URL: https://arxiv.org/pdf/2503.17662
• Copy Paste: [[2503.17662]] Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning(https://arxiv.org/abs/2503.17662)
• Keywords: language model, gpt, llm
• Abstract: In recent years, large language models (LLMs) have achieved breakthrough progress in many dialogue generation tasks. However, their lack of emotion and fine-grained role awareness limits the model's ability to provide personalized and diverse interactions further. Current methods face high costs in collecting high-quality annotated data for scenarios such as role-playing, and traditional human alignment methods are difficult to deploy due to the inherent diversity of model behavior in role-playing scenarios. Inspired by the alignment of models for safety behaviors through RLHF (Reinforcement Learning from Human Feedback), in this paper, we revisit model role-playing behavior from the perspective of persona alignment and propose a novel annotation-free framework named \textbf{\underline{P}}ersona-Aware \textbf{\underline{C}}ontrastive \textbf{\underline{L}}earning (PCL) to align LLMs' behavior during role-playing, enhancing the model's role consistency. Specifically, we first design a role chain method to encourage the model to self-question based on the role characteristics and dialogue context to adjust personality consistency. Then, we further enhance the model's role-playing strategy through iterative contrastive learning between the use of role characteristics and not. Experiments on both black-box and white-box LLMs show that LLMs equipped with PCL significantly outperform vanilla LLMs under automatic evaluation methods (CharEval \& GPT-4) and human expert evaluation.
• 摘要：近年来，大型语言模型（LLM）在许多对话生成任务中取得了突破性的进步。但是，他们缺乏情感和精细的角色意识限制了模型进一步提供个性化和多样化互动的能力。当前的方法在为角色扮演等场景和传统人类一致性方法收集高质量注释的数据时面临高昂的成本，由于在角色扮演场景中模型行为的固有多样性，很难部署。 Inspired by the alignment of models for safety behaviors through RLHF (Reinforcement Learning from Human Feedback), in this paper, we revisit model role-playing behavior from the perspective of persona alignment and propose a novel annotation-free framework named \textbf{\underline{P}}ersona-Aware \textbf{\underline{C}}ontrastive \ textbf {\ underline {l}} restning（pcl）在角色扮演过程中与LLMS的行为保持一致，从而增强了模型的角色一致性。具体而言，我们首先设计了一种角色链方法，以鼓励模型根据角色特征和对话环境来调整人格一致性。然后，我们通过在角色特征的使用（而不是）之间进行迭代对比学习进一步增强了模型的角色扮演策略。在Black-Box和White-Box LLMS上进行的实验表明，在自动评估方法（Chareval \＆GPT-4）和人类专家评估下，配备PCL的LLMS明显胜过了PCL的LLM。

Title: Can LLMs Automate Fact-Checking Article Writing?

• Authors: Dhruv Sahnan, David Corney, Irene Larraz, Giovanni Zagni, Ruben Miguez, Zhuohan Xie, Iryna Gurevych, Elizabeth Churchill, Tanmoy Chakraborty, Preslav Nakov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17684
• Pdf URL: https://arxiv.org/pdf/2503.17684
• Copy Paste: [[2503.17684]] Can LLMs Automate Fact-Checking Article Writing?(https://arxiv.org/abs/2503.17684)
• Keywords: llm, agent
• Abstract: Automatic fact-checking aims to support professional fact-checkers by offering tools that can help speed up manual fact-checking. Yet, existing frameworks fail to address the key step of producing output suitable for broader dissemination to the general public: while human fact-checkers communicate their findings through fact-checking articles, automated systems typically produce little or no justification for their assessments. Here, we aim to bridge this gap. We argue for the need to extend the typical automatic fact-checking pipeline with automatic generation of full fact-checking articles. We first identify key desiderata for such articles through a series of interviews with experts from leading fact-checking organizations. We then develop QRAFT, an LLM-based agentic framework that mimics the writing workflow of human fact-checkers. Finally, we assess the practical usefulness of QRAFT through human evaluations with professional fact-checkers. Our evaluation shows that while QRAFT outperforms several previously proposed text-generation approaches, it lags considerably behind expert-written articles. We hope that our work will enable further research in this new and important direction.
• 摘要：自动事实检查旨在通过提供可以帮助加快手动事实检查的工具来支持专业事实检查器。然而，现有框架无法解决生产适合广泛传播给公众的输出的关键步骤：当人类事实检查者通过事实检查文章传达其发现，而自动化系统通常很少或根本没有理由进行评估。在这里，我们的目标是弥合这一差距。我们主张需要通过自动生成完整的事实检查文章来扩展典型的自动事实检查管道。我们首先通过对领先事实检查组织的专家进行的一系列访谈来确定此类文章的关键Desiderata。然后，我们开发了基于LLM的代理框架Qraft，模仿了人类事实检查者的写作工作流程。最后，我们通过通过专业事实检查者的人类评估来评估QRAFT的实际实用性。我们的评估表明，尽管Qraft的表现优于先前提出的文本生成方法，但它远远落后于专家写的文章。我们希望我们的工作能够以这个新的重要方向进行进一步的研究。

Title: Enhancing Arabic Automated Essay Scoring with Synthetic Data and Error Injection

• Authors: Chatrine Qwaider, Bashar Alhafni, Kirill Chirkunov, Nizar Habash, Ted Briscoe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17739
• Pdf URL: https://arxiv.org/pdf/2503.17739
• Copy Paste: [[2503.17739]] Enhancing Arabic Automated Essay Scoring with Synthetic Data and Error Injection(https://arxiv.org/abs/2503.17739)
• Keywords: language model, llm, prompt
• Abstract: Automated Essay Scoring (AES) plays a crucial role in assessing language learners' writing quality, reducing grading workload, and providing real-time feedback. Arabic AES systems are particularly challenged by the lack of annotated essay datasets. This paper presents a novel framework leveraging Large Language Models (LLMs) and Transformers to generate synthetic Arabic essay datasets for AES. We prompt an LLM to generate essays across CEFR proficiency levels and introduce controlled error injection using a fine-tuned Standard Arabic BERT model for error type prediction. Our approach produces realistic human-like essays, contributing a dataset of 3,040 annotated essays. Additionally, we develop a BERT-based auto-marking system for accurate and scalable Arabic essay evaluation. Experimental results demonstrate the effectiveness of our framework in improving Arabic AES performance.
• 摘要：自动论文评分（AES）在评估语言学习者的写作质量，降低等级工作量并提供实时反馈方面起着至关重要的作用。阿拉伯语AES系统尤其受到带注释的论文数据集的挑战。本文提出了一个新的框架，利用大型语言模型（LLM）和变形金刚生成了AES的合成阿拉伯论文数据集。我们促使LLM跨CEFR生成论文水平水平，并使用微型标准的阿拉伯BERT模型引入控制错误注入，以进行错误类型预测。我们的方法产生了类似人类的文章，贡献了3,040个注释论文的数据集。此外，我们开发了一个基于BERT的自动标记系统，以进行准确且可扩展的阿拉伯论文评估。实验结果证明了我们框架在改善阿拉伯AES性能方面的有效性。

Title: Building Resource-Constrained Language Agents: A Korean Case Study on Chemical Toxicity Information

• Authors: Hojun Cho, Donghu Kim, Soyoung Yang, Chan Lee, Hunjoo Lee, Jaegul Choo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17753
• Pdf URL: https://arxiv.org/pdf/2503.17753
• Copy Paste: [[2503.17753]] Building Resource-Constrained Language Agents: A Korean Case Study on Chemical Toxicity Information(https://arxiv.org/abs/2503.17753)
• Keywords: language model, llm, chat, agent
• Abstract: Language agents powered by large language models (LLMs) face significant deployment challenges in resource-constrained environments, particularly for specialized domains and less-common languages. This paper presents Tox-chat, a Korean chemical toxicity information agent devised within these limitations. We propose two key innovations: a context-efficient architecture that reduces token consumption through hierarchical section search, and a scenario-based dialogue generation methodology that effectively distills tool-using capabilities from larger models. Experimental evaluations demonstrate that our fine-tuned 8B parameter model substantially outperforms both untuned models and baseline approaches, in terms of DB faithfulness and preference. Our work offers valuable insights for researchers developing domain-specific language agents under practical constraints.
• 摘要：由大型语言模型（LLMS）提供支持的语言代理在资源受限的环境中面临重大部署挑战，尤其是对于专业领域和较不常见的语言。本文介绍了在这些限制内设计的韩国化学毒性信息剂Tox-Chat。我们提出了两个关键的创新：通过层次结构搜索降低令牌消耗的上下文架构，以及一种基于方案的对话生成方法，可有效地从较大的模型中提取使用工具功能。实验评估表明，我们的微调8b参数模型在DB的忠诚和偏爱方面大大优于未调节的模型和基线方法。我们的工作为研究人员提供了有价值的见解，该研究人员在实际限制下开发特定领域的语言代理。

Title: Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes

• Authors: Sharan Maiya, Yinhong Liu, Ramit Debnath, Anna Korhonen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17755
• Pdf URL: https://arxiv.org/pdf/2503.17755
• Copy Paste: [[2503.17755]] Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes(https://arxiv.org/abs/2503.17755)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are often used as automated judges to evaluate text, but their effectiveness can be hindered by various unintentional biases. We propose using linear classifying probes, trained by leveraging differences between contrasting pairs of prompts, to directly access LLMs' latent knowledge and extract more accurate preferences. Through extensive experiments using models of varying size from four different families and six diverse datasets assessing text quality evaluation and common sense reasoning, we demonstrate that both supervised and unsupervised probing approaches consistently outperform traditional generation-based judgement while maintaining similar computational costs. These probes generalise under domain shifts and can even outperform finetuned evaluators with the same training data size. Our results suggest linear probing offers an accurate, robust and computationally efficient approach for LLM-as-judge tasks while providing interpretable insights into how models encode judgement-relevant knowledge. Our data and code will be openly released in the future.
• 摘要：大型语言模型（LLM）通常被用作自动化法官来评估文本，但是它们的有效性可能会受到各种无意的偏见的阻碍。我们建议使用线性分类探针，该探针通过利用对比对的提示之间的差异，直接访问LLMS的潜在知识并提取更准确的偏好。通过大量实验，使用来自四个不同家庭的不同大小的模型和六个不同的数据集评估文本质量评估和常识推理，我们证明了受监督和无监督的探测方法始终超过传统的基于传统的判断，同时保持相似的计算成本。这些探针在域移动下概括，甚至可以超过具有相同训练数据大小的固定评估者。我们的结果表明，线性探测为LLM-As-Gudge任务提供了一种准确，健壮和计算有效的方法，同时为模型如何编码与判断相关的知识提供了可解释的见解。我们的数据和代码将来将公开发布。

Title: Relation Extraction with Instance-Adapted Predicate Descriptions

• Authors: Yuhang Jiang, Ramakanth Kavuluru
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17799
• Pdf URL: https://arxiv.org/pdf/2503.17799
• Copy Paste: [[2503.17799]] Relation Extraction with Instance-Adapted Predicate Descriptions(https://arxiv.org/abs/2503.17799)
• Keywords: language model
• Abstract: Relation extraction (RE) is a standard information extraction task playing a major role in downstream applications such as knowledge discovery and question answering. Although decoder-only large language models are excelling in generative tasks, smaller encoder models are still the go to architecture for RE. In this paper, we revisit fine-tuning such smaller models using a novel dual-encoder architecture with a joint contrastive and cross-entropy loss. Unlike previous methods that employ a fixed linear layer for predicate representations, our approach uses a second encoder to compute instance-specific predicate representations by infusing them with real entity spans from corresponding input instances. We conducted experiments on two biomedical RE datasets and two general domain datasets. Our approach achieved F1 score improvements ranging from 1% to 2% over state-of-the-art methods with a simple but elegant formulation. Ablation studies justify the importance of various components built into the proposed architecture.
• 摘要：关系提取（RE）是一项标准信息提取任务，在下游应用程序（例如知识发现和问题答案）中起着重要作用。尽管仅解码器的大型语言模型在生成任务中表现出色，但较小的编码器模型仍然是RE的架构。在本文中，我们使用具有关节对比和横向侧面损失的新型双重编码结构对如此较小的模型进行了微调。与以前使用固定线性层用于谓词表示形式的方法不同，我们的方法使用第二个编码器来计算特定于实例的谓词表示，从而将它们注入来自相应输入实例的真实实体跨度。我们对两个生物医学RE数据集和两个通用域数据集进行了实验。我们的方法实现了F1得分的提高，从最先进但优雅的配方中的最先进方法比最先进的方法不等。消融研究证明了所提出的体系结构内置的各种组件的重要性是合理的。

Title: Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Models

• Authors: Wenqi Pei, Hailing Xu, Hengyuan Zhao, Shizheng Hou, Han Chen, Zining Zhang, Pingyi Luo, Bingsheng He
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2503.17811
• Pdf URL: https://arxiv.org/pdf/2503.17811
• Copy Paste: [[2503.17811]] Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Models(https://arxiv.org/abs/2503.17811)
• Keywords: language model, llm, chat
• Abstract: Natural Language to SQL (NL2SQL) has seen significant advancements with large language models (LLMs). However, these models often depend on closed-source systems and high computational resources, posing challenges in data privacy and deployment. In contrast, small language models (SLMs) struggle with NL2SQL tasks, exhibiting poor performance and incompatibility with existing frameworks. To address these issues, we introduce Feather-SQL, a new lightweight framework tailored for SLMs. Feather-SQL improves SQL executability and accuracy through 1) schema pruning and linking, 2) multi-path and multi-candidate generation. Additionally, we introduce the 1+1 Model Collaboration Paradigm, which pairs a strong general-purpose chat model with a fine-tuned SQL specialist, combining strong analytical reasoning with high-precision SQL generation. Experimental results on BIRD demonstrate that Feather-SQL improves NL2SQL performance on SLMs, with around 10% boost for models without fine-tuning. The proposed paradigm raises the accuracy ceiling of SLMs to 54.76%, highlighting its effectiveness.
• 摘要：SQL（NL2SQL）的自然语言在大型语言模型（LLMS）方面取得了重大进步。但是，这些模型通常取决于封闭式系统和高度计算资源，从而在数据隐私和部署中构成了挑战。相反，小语言模型（SLM）与NL2SQL任务斗争，表现出差的性能和与现有框架不兼容。为了解决这些问题，我们介绍了Feather-SQL，这是一个针对SLMS量身定制的新的轻量级框架。羽毛-SQL通过1）架构修剪和链接提高了SQL的可执行性和准确性，2）多路径和多态换产生。此外，我们介绍了1+1个模型协作范式，该范式将强大的通用聊天模型与微调的SQL专家配对，将强大的分析推理与高精度SQL生成相结合。鸟类的实验结果表明，羽毛-SQL改善了SLMS上的NL2SQL性能，而无需微调的模型增强了约10％。拟议的范式将SLM的精度上限提高到54.76％，突出了其有效性。

Title: Satisfactory Medical Consultation based on Terminology-Enhanced Information Retrieval and Emotional In-Context Learning

• Authors: Kaiwen Zuo, Jing Tang, Hanbing Qin, Binli Luo, Ligang He, Shiyan Tang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.17876
• Pdf URL: https://arxiv.org/pdf/2503.17876
• Copy Paste: [[2503.17876]] Satisfactory Medical Consultation based on Terminology-Enhanced Information Retrieval and Emotional In-Context Learning(https://arxiv.org/abs/2503.17876)
• Keywords: language model, llm, long context
• Abstract: Recent advancements in Large Language Models (LLMs) have marked significant progress in understanding and responding to medical inquiries. However, their performance still falls short of the standards set by professional consultations. This paper introduces a novel framework for medical consultation, comprising two main modules: Terminology-Enhanced Information Retrieval (TEIR) and Emotional In-Context Learning (EICL). TEIR ensures implicit reasoning through the utilization of inductive knowledge and key terminology retrieval, overcoming the limitations of restricted domain knowledge in public databases. Additionally, this module features capabilities for processing long context. The EICL module aids in generating sentences with high attribute relevance by memorizing semantic and attribute information from unlabelled corpora and applying controlled retrieval for the required information. Furthermore, a dataset comprising 803,564 consultation records was compiled in China, significantly enhancing the model's capability for complex dialogues and proactive inquiry initiation. Comprehensive experiments demonstrate the proposed method's effectiveness in extending the context window length of existing LLMs. The experimental outcomes and extensive data validate the framework's superiority over five baseline models in terms of BLEU and ROUGE performance metrics, with substantial leads in certain capabilities. Notably, ablation studies confirm the significance of the TEIR and EICL components. In addition, our new framework has the potential to significantly improve patient satisfaction in real clinical consulting situations.
• 摘要：大型语言模型（LLM）的最新进展在理解和响应医学询问方面取得了重大进展。但是，他们的表现仍然没有专业咨询所设定的标准。本文介绍了一个新型的医学咨询框架，包括两个主要模块：术语增强信息检索（TEIR）和情感上下文学习（EICL）。 Teir通过利用归纳知识和关键术语检索来确保隐含的推理，从而克服了公共数据库中限制领域知识的局限性。此外，该模块还具有处理长上下文的功能。 EICL模块有助于生成具有很高属性相关性的句子，通过记住来自未标记的Corpora的语义和属性信息并应用受控检索以获取所需信息。此外，在中国汇编了一个包含803,564次咨询记录的数据集，大大提高了该模型的复杂对话和主动查询启动的能力。全面的实验证明了所提出的方法在扩展现有LLM的上下文窗口长度方面的有效性。在BLEU和Rouge性能指标方面，实验结果和广泛的数据验证了该框架比五个基线模型的优越性，在某些功能方面具有很大的领先优势。值得注意的是，消融研究证实了TEIR和EICL成分的重要性。此外，我们的新框架有可能在实际临床咨询情况下显着提高患者满意度。

Title: Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior

• Authors: Shengyun Si, Xinpeng Wang, Guangyao Zhai, Nassir Navab, Barbara Plank
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17882
• Pdf URL: https://arxiv.org/pdf/2503.17882
• Copy Paste: [[2503.17882]] Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior(https://arxiv.org/abs/2503.17882)
• Keywords: language model, llm, prompt
• Abstract: Recent advancements in large language models (LLMs) have demonstrated that fine-tuning and human alignment can render LLMs harmless. In practice, such "harmlessness" behavior is mainly achieved by training models to reject harmful requests, such as "Explain how to burn down my neighbor's house", where the model appropriately declines to respond. However, this approach can inadvertently result in false refusal, where models reject benign queries as well, such as "Tell me how to kill a Python process". In this work, we demonstrate that prompting safety reflection before generating a response can mitigate false refusal behavior. Building on this finding, we introduce the Think-Before-Refusal (TBR) schema and conduct safety-aware instruction fine-tuning incorporating safety reflection. In an ablation study across 15 pre-trained models, we show that models fine-tuned with safety reflection significantly reduce false refusal behavior while maintaining safety and overall performance compared to those fine-tuned without safety reflection.
• 摘要：大型语言模型（LLM）的最新进展表明，微调和人类的一致性可能使LLM无害。在实践中，这种“无害”行为主要是通过培训模型拒绝有害要求的，例如“解释如何烧毁我邻居的房屋”，该模型适当地拒绝做出响应。但是，这种方法可能会无意中导致错误的拒绝，模型也拒绝了良性查询，例如“告诉我如何杀死Python过程”。在这项工作中，我们证明在产生响应之前提示安全反射可以减轻错误的拒绝行为。在这一发现的基础上，我们介绍了预先的（TBR）架构，并进行安全感知指令进行微调，并结合了安全反思。在15个预训练模型的一项消融研究中，我们表明，与没有安全反射的那些相比，与安全反射相比，与安全反射进行了微调的模型显着降低了错误的拒绝行为，同时保持安全性和整体性能。

Title: MedPlan:A Two-Stage RAG-Based System for Personalized Medical Plan Generation

• Authors: Hsin-Ling Hsu, Cong-Tinh Dao, Luning Wang, Zitao Shuai, Thao Nguyen Minh Phan, Jun-En Ding, Chun-Chieh Liao, Pengfei Hu, Xiaoxue Han, Chih-Ho Hsu, Dongsheng Luo, Wen-Chih Peng, Feng Liu, Fang-Ming Hung, Chenwei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17900
• Pdf URL: https://arxiv.org/pdf/2503.17900
• Copy Paste: [[2503.17900]] MedPlan:A Two-Stage RAG-Based System for Personalized Medical Plan Generation(https://arxiv.org/abs/2503.17900)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Despite recent success in applying large language models (LLMs) to electronic health records (EHR), most systems focus primarily on assessment rather than treatment planning. We identify three critical limitations in current approaches: they generate treatment plans in a single pass rather than following the sequential reasoning process used by clinicians; they rarely incorporate patient-specific historical context; and they fail to effectively distinguish between subjective and objective clinical information. Motivated by the SOAP methodology (Subjective, Objective, Assessment, Plan), we introduce MedPlan, a novel framework that structures LLM reasoning to align with real-life clinician workflows. Our approach employs a two-stage architecture that first generates a clinical assessment based on patient symptoms and objective data, then formulates a structured treatment plan informed by this assessment and enriched with patient-specific information through retrieval-augmented generation. Comprehensive evaluation demonstrates that our method significantly outperforms baseline approaches in both assessment accuracy and treatment plan quality.
• 摘要：尽管最近成功地将大型语言模型（LLM）应用于电子健康记录（EHR），但大多数系统主要集中于评估而不是治疗计划。我们确定了当前方法的三个关键局限性：它们在单个通过中生成治疗计划，而不是遵循临床医生使用的顺序推理过程；他们很少融合特定于患者的历史背景；他们无法有效区分主观和客观临床信息。在SOAP方法论（主观，客观，评估，计划）中，我们介绍了Medplan，这是一个新颖的框架，该框架结构LLM推理以与现实生活中的临床医生工作流程保持一致。我们的方法采用了两阶段的体系结构，该结构首先根据患者症状和客观数据产生临床评估，然后制定由本评估告知的结构化治疗计划，并通过检索效率增强的生成丰富患者特定信息。全面的评估表明，我们的方法在评估准确性和治疗计划质量方面都显着优于基线方法。

Title: WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference

• Authors: Youhui Zuo, Sibo Wei, Chen Zhang, Zhuorui Liu, Wenpeng Lu, Dawei Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17922
• Pdf URL: https://arxiv.org/pdf/2503.17922
• Copy Paste: [[2503.17922]] WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference(https://arxiv.org/abs/2503.17922)
• Keywords: language model, llm
• Abstract: With the advancements in long-context inference capabilities of large language models (LLMs), the KV cache has become one of the foundational components. However, its substantial GPU memory consumption makes KV cache compression a key technique for enabling efficient LLM inference in industrial scenarios. While recent studies have focused on optimizing the memory occupied by the KV cache, they overlook two critical factors: preserving semantic coherence and considering task-specific characteristic during compression. To address these limitations, we propose a novel task-adaptive KV cache window selection method, WindowKV. WindowKV dynamically selects local semantic windows consisting of consecutive tokens, according to task-specific characteristics, ensuring the retained KV cache captures continuous, essential context. Additionally, we introduce an intra-group layer KV cache indices sharing strategy to reduce computational overhead, achieving a balance between performance and efficiency. We rigorously evaluate WindowKV on the LongBench benchmark, and the results demonstrate that it maintains a performance comparable to full KV cache retention while using only 12% of the original KV cache, significantly reducing memory requirements. Furthermore, our method also achieves state-of-the-art results in the Needle-in-a-Haystack evaluation, highlighting its effectiveness and robustness.
• 摘要：随着大语言模型（LLMS）长篇文化推断功能的进步，KV缓存已成为基础组件之一。但是，其大量的GPU内存消耗使KV缓存压缩是实现工业场景中有效推理的关键技术。尽管最近的研究重点是优化KV缓存所占据的记忆，但它们忽略了两个关键因素：保持语义连贯性并考虑在压缩过程中考虑特定于任务的特征。为了解决这些局限性，我们提出了一种新颖的任务自适应KV缓存窗口选择方法，WindowKV。 WindowKV根据特定于任务的特征，动态选择由连续令牌组成的本地语义窗口，从而确保保留的KV高速缓存捕获连续的基本上下文。此外，我们引入了组内层KV缓存指数共享策略，以减少计算开销，从而在绩效和效率之间取得平衡。我们在LongBench基准上严格评估WindowKV，结果表明，它保持与完整的KV缓存保留率相当的性能，同时仅使用了12％的原始KV缓存，从而大大降低了内存要求。此外，我们的方法还实现了最新的方法，从而进行了针对印记的评估，从而强调了其有效性和鲁棒性。

Title: STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Models

• Authors: Xunguang Wang, Wenxuan Wang, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Daoyuan Wu, Shuai Wang
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.17932
• Pdf URL: https://arxiv.org/pdf/2503.17932
• Copy Paste: [[2503.17932]] STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Models(https://arxiv.org/abs/2503.17932)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have become increasingly vulnerable to jailbreak attacks that circumvent their safety mechanisms. While existing defense methods either suffer from adaptive attacks or require computationally expensive auxiliary models, we present STShield, a lightweight framework for real-time jailbroken judgement. STShield introduces a novel single-token sentinel mechanism that appends a binary safety indicator to the model's response sequence, leveraging the LLM's own alignment capabilities for detection. Our framework combines supervised fine-tuning on normal prompts with adversarial training using embedding-space perturbations, achieving robust detection while preserving model utility. Extensive experiments demonstrate that STShield successfully defends against various jailbreak attacks, while maintaining the model's performance on legitimate queries. Compared to existing approaches, STShield achieves superior defense performance with minimal computational overhead, making it a practical solution for real-world LLM deployment.
• 摘要：大型语言模型（LLMS）变得越来越容易受到越狱攻击的攻击，这些袭击规避了其安全机制。尽管现有的防御方法要么患有自适应攻击，要么需要计算昂贵的辅助模型，但我们提出了Stshield，这是实时越狱判断的轻量级框架。 Stshield引入了一种新型的单token哨兵机制，该机制将二进制安全指标附加到模型的响应序列中，从而利用LLM自身的对齐功能进行检测。我们的框架结合了使用嵌入空间扰动的对抗训练，在正常提示上进行了监督的微调，从而在保留模型实用程序的同时实现了可靠的检测。广泛的实验表明，斯特希尔德成功地防御了各种越狱攻击，同时保持了模型在合法查询中的表现。与现有方法相比，Sthield通过最少的计算开销来实现出色的防御性能，从而使其成为现实世界中LLM部署的实用解决方案。

Title: Experience Retrieval-Augmentation with Electronic Health Records Enables Accurate Discharge QA

• Authors: Justice Ou, Tinglin Huang, Yilun Zhao, Ziyang Yu, Peiqing Lu, Rex Ying
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.17933
• Pdf URL: https://arxiv.org/pdf/2503.17933
• Copy Paste: [[2503.17933]] Experience Retrieval-Augmentation with Electronic Health Records Enables Accurate Discharge QA(https://arxiv.org/abs/2503.17933)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: To improve the reliability of Large Language Models (LLMs) in clinical applications, retrieval-augmented generation (RAG) is extensively applied to provide factual medical knowledge. However, beyond general medical knowledge from open-ended datasets, clinical case-based knowledge is also critical for effective medical reasoning, as it provides context grounded in real-world patient experiences. Motivated by this, we propose Experience Retrieval Augmentation - ExpRAG framework based on Electronic Health Record (EHR), aiming to offer the relevant context from other patients' discharge reports. ExpRAG performs retrieval through a coarse-to-fine process, utilizing an EHR-based report ranker to efficiently identify similar patients, followed by an experience retriever to extract task-relevant content for enhanced medical reasoning. To evaluate ExpRAG, we introduce DischargeQA, a clinical QA dataset with 1,280 discharge-related questions across diagnosis, medication, and instruction tasks. Each problem is generated using EHR data to ensure realistic and challenging scenarios. Experimental results demonstrate that ExpRAG consistently outperforms a text-based ranker, achieving an average relative improvement of 5.2%, highlighting the importance of case-based knowledge for medical reasoning.
• 摘要：为了提高大语模型（LLM）在临床应用中的可靠性，广泛应用了检索功能增强的生成（RAG）以提供事实医学知识。但是，除了来自开放式数据集的一般医学知识之外，基于临床病例的知识对于有效的医学推理也至关重要，因为它提供了基于现实世界中患者体验的背景。在此激励的情况下，我们提出了经验检索增强 - 基于电子健康记录（EHR）的Exprag框架，旨在提供其他患者出院报告的相关环境。 Exprag通过粗到精细的过程进行检索，利用基于EHR的报告排名有效地识别类似的患者，然后是经验回寻试器，以提取与任务相关的内容以增强医疗推理。为了评估Exprag，我们介绍了exurationQA，这是一个临床质量检查数据集，在诊断，药物和指导任务中，与排除相关的问题有1,280个问题。每个问题都是使用EHR数据生成的，以确保现实且具有挑战性的场景。实验结果表明，Exprag始终优于基于文本的排名，平均相对改善为5.2％，强调了基于病例的知识对医学推理的重要性。

Title: An Empirical Study of the Role of Incompleteness and Ambiguity in Interactions with Large Language Models

• Authors: Riya Naik, Ashwin Srinivasan, Estrid He, Swati Agarwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17936
• Pdf URL: https://arxiv.org/pdf/2503.17936
• Copy Paste: [[2503.17936]] An Empirical Study of the Role of Incompleteness and Ambiguity in Interactions with Large Language Models(https://arxiv.org/abs/2503.17936)
• Keywords: language model, llm, agent
• Abstract: Natural language as a medium for human-computer interaction has long been anticipated, has been undergoing a sea-change with the advent of Large Language Models (LLMs) with startling capacities for processing and generating language. Many of us now treat LLMs as modern-day oracles, asking it almost any kind of question. Unlike its Delphic predecessor, consulting an LLM does not have to be a single-turn activity (ask a question, receive an answer, leave); and -- also unlike the Pythia -- it is widely acknowledged that answers from LLMs can be improved with additional context. In this paper, we aim to study when we need multi-turn interactions with LLMs to successfully get a question answered; or conclude that a question is unanswerable. We present a neural symbolic framework that models the interactions between human and LLM agents. Through the proposed framework, we define incompleteness and ambiguity in the questions as properties deducible from the messages exchanged in the interaction, and provide results from benchmark problems, in which the answer-correctness is shown to depend on whether or not questions demonstrate the presence of incompleteness or ambiguity (according to the properties we identify). Our results show multi-turn interactions are usually required for datasets which have a high proportion of incompleteness or ambiguous questions; and that that increasing interaction length has the effect of reducing incompleteness or ambiguity. The results also suggest that our measures of incompleteness and ambiguity can be useful tools for characterising interactions with an LLM on question-answeringproblems
• 摘要：长期以来，自然语言作为人类计算机互动的一种媒介，一直在随着大型语言模型（LLMS）的出现，并具有令人震惊的处理和生成语言的惊人能力。我们中的许多人现在将LLM视为现代甲骨文，几乎提出任何一种问题。与Delphic的前任不同，咨询LLM不必是一项单身活动（问一个问题，收到答案，离开）；而且 - 也与毕缩影不同 - 广泛地承认，LLMS的答案可以通过其他上下文来改善。在本文中，我们旨在研究何时需要与LLM的多转交互交互以成功回答问题。或得出一个问题是无法回答的。我们提出了一个神经符号框架，该框架模拟了人类和LLM代理之间的相互作用。通过提出的框架，我们将问题中的不完整和歧义定义为从交互中交换的消息中推论出来的属性，并提供了基准问题的结果，基准问题的结果表明答案纠正率取决于问题是否证明了是否存在不完整或模棱两可（根据我们确定的属性（根据我们确定的属性））。我们的结果表明，数据集通常需要多圈相互作用，而数据集则需要大量的不完整或模棱两可的问题；并且增加的相互作用长度具有降低不完整或歧义的作用。结果还表明，我们的不完整和模棱两可的度量可能是表征与问题 - 答案问题上与LLM相互作用的有用工具

Title: SLIDE: Sliding Localized Information for Document Extraction

• Authors: Divyansh Singh, Manuel Nunez Martinez, Bonnie J. Dorr, Sonja Schmer Galunder
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17952
• Pdf URL: https://arxiv.org/pdf/2503.17952
• Copy Paste: [[2503.17952]] SLIDE: Sliding Localized Information for Document Extraction(https://arxiv.org/abs/2503.17952)
• Keywords: language model, llm
• Abstract: Constructing accurate knowledge graphs from long texts and low-resource languages is challenging, as large language models (LLMs) experience degraded performance with longer input chunks. This problem is amplified in low-resource settings where data scarcity hinders accurate entity and relationship extraction. Contextual retrieval methods, while improving retrieval accuracy, struggle with long documents. They truncate critical information in texts exceeding maximum context lengths of LLMs, significantly limiting knowledge graph construction. We introduce SLIDE (Sliding Localized Information for Document Extraction), a chunking method that processes long documents by generating local context through overlapping windows. SLIDE ensures that essential contextual information is retained, enhancing knowledge graph extraction from documents exceeding LLM context limits. It significantly improves GraphRAG performance, achieving a 24% increase in entity extraction and a 39% improvement in relationship extraction for English. For Afrikaans, a low-resource language, SLIDE achieves a 49% increase in entity extraction and an 82% improvement in relationship extraction. Furthermore, it improves upon state-of-the-art in question-answering metrics such as comprehensiveness, diversity and empowerment, demonstrating its effectiveness in multilingual and resource-constrained settings.
• 摘要：从长文本和低资源语言中构建准确的知识图很具有挑战性，因为大型语言模型（LLMS）经历了较长的输入块的降级性能。在低资源设置中，数据稀缺阻碍了准确的实体和关系提取。上下文检索方法，同时提高了检索准确性，而在长期文档中挣扎。它们在超过LLM的最大上下文长度的文本中截断了关键信息，从而大大限制了知识图构造。我们介绍了幻灯片（用于文档提取的滑动本地化信息），这是一种通过重叠窗口生成本地上下文来处理较长文档的块方法。幻灯片确保保留基本的上下文信息，从而增强超过LLM上下文限制的文档的知识图提取。它可显着提高图形架的性能，实体提取增加24％，而英语的关系提取提高了39％。对于南非荷兰语（一种低资源语言），滑动的实体提取提取49％，关系提取提高了82％。此外，它在提问指标（例如全面，多样性和授权）中的最新指标上提高了它，证明了其在多语言和资源约束的环境中的有效性。

Title: Won: Establishing Best Practices for Korean Financial NLP

• Authors: Guijin Son, Hyunwoo Ko, Haneral Jung, Chami Hwang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17963
• Pdf URL: https://arxiv.org/pdf/2503.17963
• Copy Paste: [[2503.17963]] Won: Establishing Best Practices for Korean Financial NLP(https://arxiv.org/abs/2503.17963)
• Keywords: language model, llm, agent
• Abstract: In this work, we present the first open leaderboard for evaluating Korean large language models focused on finance. Operated for about eight weeks, the leaderboard evaluated 1,119 submissions on a closed benchmark covering five MCQA categories: finance and accounting, stock price prediction, domestic company analysis, financial markets, and financial agent tasks and one open-ended qa task. Building on insights from these evaluations, we release an open instruction dataset of 80k instances and summarize widely used training strategies observed among top-performing models. Finally, we introduce Won, a fully open and transparent LLM built using these best practices. We hope our contributions help advance the development of better and safer financial LLMs for Korean and other languages.
• 摘要：在这项工作中，我们介绍了第一个评估韩国大语模型的开放排行榜，该模型的重点是金融。排行榜运行约八周，评估了1,119项封闭基准的意见书，其中涵盖了五个MCQA类别：财务和会计，股票价格预测，国内公司分析，金融市场和金融代理任务以及一项开放式质量保证任务。在这些评估中的见解的基础上，我们发布了80K实例的开放指令数据集，并总结了在表现最佳模型中观察到的广泛使用的培训策略。最后，我们介绍了Won，这是一个完全开放透明的LLM，它使用这些最佳实践构建。我们希望我们的贡献有助于推动开发韩语和其他语言的更好，更安全的财务LLM。

Title: Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts

• Authors: Beining Xu, Arkaitz Zubiaga
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17965
• Pdf URL: https://arxiv.org/pdf/2503.17965
• Copy Paste: [[2503.17965]] Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts(https://arxiv.org/abs/2503.17965)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated exceptional performance on a range of downstream NLP tasks by generating text that closely resembles human writing. However, the ease of achieving this similarity raises concerns from potential malicious uses at scale by bad actors, as LLM-generated text becomes increasingly difficult to discern from human text. Although detection methods have been developed to address this issue, bad actors can further manipulate LLM-generated texts to make them less detectable. In this work, we study how further editing texts with Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human preferences, affects (a) the quality of generated texts for two tasks, and (b) the performance of LLM-generated text detectors, looking at both training-based and zero-shot detection methods. Although RLHF improves the quality of LLM-generated texts, we find that it also tends to produce more detectable, lengthy, and repetitive outputs. Additionally, we observe that training-based detectors are vulnerable to short texts and to texts that incorporate code, whereas zero-shot detectors exhibit greater robustness.
• 摘要：大型语言模型（LLM）通过生成与人写作非常相似的文本，在一系列下游NLP任务上表现出了出色的表现。但是，由于LLM生成的文本变得越来越难以辨别人类文本，因此无法实现这种相似性的便利性引起了潜在的恶意用途的关注。尽管已经开发了检测方法来解决这个问题，但不良的参与者可以进一步操纵LLM生成的文本，以使其较少可检测到。在这项工作中，我们研究了如何通过从人类反馈（RLHF）学习的增强文本（将模型输出与人类偏好保持一致）的进一步编辑，影响（a）（a）为两个任务的生成文本的质量，以及（b）LLM生成的文本检测器的性能，研究了基于训练和零点检测方法。尽管RLHF提高了LLM生成的文本的质量，但我们发现它也倾向于产生更可检测，冗长和重复的输出。此外，我们观察到，基于训练的探测器容易受到简短文本和包含代码的文本的影响，而零射击检测器则表现出更大的鲁棒性。

Title: Instructing the Architecture Search for Spatial-temporal Sequence Forecasting with LLM

• Authors: Xin Xue, Haoyi Zhou, Tianyu Chen, Shuai Zhang, Yizhou Long, Jianxin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17994
• Pdf URL: https://arxiv.org/pdf/2503.17994
• Copy Paste: [[2503.17994]] Instructing the Architecture Search for Spatial-temporal Sequence Forecasting with LLM(https://arxiv.org/abs/2503.17994)
• Keywords: language model, llm, prompt
• Abstract: Spatial-temporal sequence forecasting (STSF) is a long-standing research problem with widespread real-world applications. Neural architecture search (NAS), which automates the neural network design, has been shown effective in tackling the STSF problem. However, the existing NAS methods for STSF focus on generating architectures in a time-consuming data-driven fashion, which heavily limits their ability to use background knowledge and explore the complicated search trajectory. Large language models (LLMs) have shown remarkable ability in decision-making with comprehensive internal world knowledge, but how it could benefit NAS for STSF remains unexplored. In this paper, we propose a novel NAS method for STSF based on LLM. Instead of directly generate architectures with LLM, We inspire the LLM's capability with a multi-level enhancement mechanism. Specifically, on the step-level, we decompose the generation task into decision steps with powerful prompt engineering and inspire LLM to serve as instructor for architecture search based on its internal knowledge. On the instance-level, we utilize a one-step tuning framework to quickly evaluate the architecture instance and a memory bank to cumulate knowledge to improve LLM's search ability. On the task-level, we propose a two-stage architecture search, balancing the exploration stage and optimization stage, to reduce the possibility of being trapped in local optima. Extensive experimental results demonstrate that our method can achieve competitive effectiveness with superior efficiency against existing NAS methods for STSF.
• 摘要：空间周期序列预测（STSF）是广泛的现实应用程序的长期研究问题。自动化神经网络设计的神经体系结构搜索（NAS）已被证明有效解决了STSF问题。但是，现有的STSF的NAS方法专注于以耗时的数据驱动方式生成体系结构，这严重限制了其使用背景知识并探索复杂的搜索轨迹的能力。大型语言模型（LLMS）在具有全面的内部世界知识的决策中表现出了显着的能力，但是如何使STSF的NAS受益仍然没有探索。在本文中，我们提出了一种基于LLM的STSF的新型NAS方法。我们没有直接使用LLM生成体系结构，而是使用多级增强机制激发了LLM的功能。具体而言，在步骤层面上，我们将生成任务分解为决策步骤，并通过强大的及时工程设计，并激发LLM根据其内部知识作为建筑搜索的讲师。在实例级别上，我们利用一个步骤调整框架快速评估体系结构实例和内存库以累积知识以提高LLM的搜索能力。在任务级别上，我们提出了一个两阶段的架构搜索，平衡了探索阶段和优化阶段，以减少被困在本地Optima中的可能性。广泛的实验结果表明，我们的方法可以通过针对STSF的现有NAS方法实现竞争有效性。

Title: Personalized Language Models via Privacy-Preserving Evolutionary Model Merging

• Authors: Kyuyoung Kim, Jinwoo Shin, Jaehyung Kim
• Subjects: cs.CL, cs.NE
• Abstract URL: https://arxiv.org/abs/2503.18008
• Pdf URL: https://arxiv.org/pdf/2503.18008
• Copy Paste: [[2503.18008]] Personalized Language Models via Privacy-Preserving Evolutionary Model Merging(https://arxiv.org/abs/2503.18008)
• Keywords: language model, llm, prompt
• Abstract: Personalization in large language models (LLMs) seeks to tailor models to individual user or user group preferences. Prompt-based methods augment queries with user preference information, whereas training-based methods directly encode preferences into model parameters for more effective personalization. Despite achieving some success in personalizing LLMs, prior methods often fail to directly optimize task-specific metrics and lack explicit privacy-preservation mechanisms. To address these limitations, we propose Privacy-Preserving Model Merging via Evolutionary Algorithms (PriME), a novel approach to personalization that employs gradient-free methods to directly optimize task-specific metrics while preserving user privacy. By incorporating privacy preservation into optimization, PriME produces a personalized module that effectively captures the target user's preferences while minimizing the privacy risks for the users sharing their private information. Experiments on the LaMP benchmark show that PriME outperforms both prompt-based and training-based methods, achieving up to a 45% performance improvement over the prior art. Further analysis shows that PriME achieves a significantly better privacy-utility trade-off, highlighting the potential of evolutionary approaches for privacy-preserving LLM personalization.
• 摘要：大型语言模型（LLMS）中的个性化试图针对单个用户或用户组偏好量身定制模型。基于及时的方法通过用户偏好信息增强查询，而基于培训的方法将首选项直接编码为模型参数，以进行更有效的个性化。尽管在个性化LLM方面取得了一些成功，但先前的方法通常无法直接优化特定于任务的指标，并且缺乏明确的隐私保护机制。为了解决这些限制，我们建议通过进化算法（PRIME）合并隐私模型，这是一种新颖的个性化方法，采用无梯度方法来直接优化特定于任务的指标，同时保留用户隐私。通过将隐私保护纳入优化，Prime生成了个性化模块，该模块有效地捕获了目标用户的偏好，同时最大程度地减少了共享其私人信息的用户的隐私风险。 LAMP基准测试的实验表明，Prime的表现均优于基于迅速的方法和基于培训的方法，比以前的艺术实现了45％的性能提高。进一步的分析表明，Prime实现了更好的隐私性权衡权衡取舍，强调了进化方法具有隐私性LLM个性化的潜力。

Title: Investigating Recent Large Language Models for Vietnamese Machine Reading Comprehension

• Authors: Anh Duc Nguyen, Hieu Minh Phi, Anh Viet Ngo, Long Hai Trieu, Thai Phuong Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18062
• Pdf URL: https://arxiv.org/pdf/2503.18062
• Copy Paste: [[2503.18062]] Investigating Recent Large Language Models for Vietnamese Machine Reading Comprehension(https://arxiv.org/abs/2503.18062)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have shown remarkable proficiency in Machine Reading Comprehension (MRC) tasks; however, their effectiveness for low-resource languages like Vietnamese remains largely unexplored. In this paper, we fine-tune and evaluate two state-of-the-art LLMs: Llama 3 (8B parameters) and Gemma (7B parameters), on ViMMRC, a Vietnamese MRC dataset. By utilizing Quantized Low-Rank Adaptation (QLoRA), we efficiently fine-tune these models and compare their performance against powerful LLM-based baselines. Although our fine-tuned models are smaller than GPT-3 and GPT-3.5, they outperform both traditional BERT-based approaches and these larger models. This demonstrates the effectiveness of our fine-tuning process, showcasing how modern LLMs can surpass the capabilities of older models like BERT while still being suitable for deployment in resource-constrained environments. Through intensive analyses, we explore various aspects of model performance, providing valuable insights into adapting LLMs for low-resource languages like Vietnamese. Our study contributes to the advancement of natural language processing in low-resource languages, and we make our fine-tuned models publicly available at: this https URL.
• 摘要：大型语言模型（LLM）表现出非常熟练的机器阅读理解（MRC）任务；但是，它们对像越南这样的低资源语言的有效性在很大程度上尚未得到探索。在本文中，我们在VIMMRC上微调和评估两个最先进的LLM：Llama 3（8b参数）和Gemma（7b参数）。通过利用量化的低级适应性（Qlora），我们有效地微调了这些模型，并将其性能与强大的基于LLM的基线进行比较。尽管我们的微调模型小于GPT-3和GPT-3.5，但它们的表现都超过了传统的基于BERT的方法和这些较大的模型。这证明了我们的微调过程的有效性，展示了现代LLM如何超越伯特（Bert）等旧模型的功能，同时仍然适合在资源受限的环境中部署。通过密集的分析，我们探讨了模型性能的各个方面，为适应越南语等低资源语言的LLM提供了宝贵的见解。我们的研究有助于低资源语言中自然语言处理的进步，并使我们的微调模型在以下方面公开可用：此HTTPS URL。

Title: Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuning

• Authors: Pieyi Zhang, Richong Zhang, Zhijie Nie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18063
• Pdf URL: https://arxiv.org/pdf/2503.18063
• Copy Paste: [[2503.18063]] Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuning(https://arxiv.org/abs/2503.18063)
• Keywords: prompt
• Abstract: Multi-task prompt tuning utilizes multiple high-resource source tasks to improve performance on low-source target tasks. Existing approaches transfer the soft prompt trained by combining all source tasks or a single ``high-similar'' source task one-time-only. However, we find that the optimal transfer performance often comes from a combination of source tasks, which is neither one nor all. Further, we find that the similarity between source and target tasks also changes dynamically during fine-tuning after transfering, making similarity calculation in the initiation stage inadequate. To address these issues, we propose a method called Dynamic Task Vector Grouping (DTVG), whose core ideas contain (1) measuring the task similarity with task vectors instead of soft prompt, (2) grouping the optimal source task combination based on two metrics: {\it target similarity} and {\it knowledge consistency}; (3) dynamically updating the combination in each iteration step. Extensive experiments on the 26 NLP datasets under different settings demonstrate that DTVG effectively groups similar source tasks while reducing negative transfer, achieving the start-of-art performance.
• 摘要：多任务提示调谐使用多个高资源源任务来提高低源目标任务的性能。现有方法通过组合所有源任务或单个``高相似性''源任务来转移训练的软提示。但是，我们发现最佳转移性能通常来自源任务的组合，而源任务既不是一个也不是全部。此外，我们发现源和目标任务之间的相似性在传输后微调过程中也动态变化，从而使相似性计算在起始阶段不足。为了解决这些问题，我们提出了一种称为动态任务矢量组（DTVG）的方法，其核心想法包含（1）测量任务与任务矢量而不是软提示的任务相似性，（2）基于两个指标的最佳源任务组合：{\ iT目标相似性}和{\ IT IT知识一致性}; （3）在每个迭代步骤中动态更新组合。在不同设置下的26个NLP数据集上进行的广泛实验表明，DTVG有效地分组了相似的源任务，同时减少了负转移，从而实现了ART启动性能。

Title: Long Is More Important Than Difficult for Training Reasoning Models

• Authors: Si Shen, Fei Huang, Zhixiao Zhao, Chang Liu, Tiansheng Zheng, Danhao Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18069
• Pdf URL: https://arxiv.org/pdf/2503.18069
• Copy Paste: [[2503.18069]] Long Is More Important Than Difficult for Training Reasoning Models(https://arxiv.org/abs/2503.18069)
• Keywords: language model
• Abstract: Difficult problems, which often result in long reasoning traces, are widely recognized as key factors for enhancing the performance of reasoning models. However, such high-challenge problems are scarce, limiting the size of available datasets. In this paper, we propose a simple method to decouple the reliance on problem difficulty. First, we empirically demonstrate that reasoning length, rather than problem difficulty, primarily influences the performance of trained models. Second, we identify a scaling law on reasoning length, showing that model performance increases in a log-linear fashion as the reasoning data length grows. Finally, we introduce a straightforward technique to generate reasoning data of arbitrary length, and show that synthesized data is effective for training reasoning models. After fine-tuning the Qwen2.5-32B-Instruct language model on our Long1K dataset, we present our model, Long1K-32B, which achieves remarkable performance with only 1,000 training samples, achieving 95.6\% accuracy on MATH, and 71.1\% on GPQA outperforming DeepSeek-R1-Distill-Qwen-32B. The model, code, and dataset are all open-sourced, available at this https URL.
• 摘要：困难的问题通常会导致长期的推理轨迹，被广泛认为是增强推理模型性能的关键因素。但是，这种高挑战性问题很少，限制了可用数据集的大小。在本文中，我们提出了一种简单的方法，可以使依赖问题难度的依赖。首先，我们从经验上证明了推理长度而不是问题难度，主要影响训练有素的模型的性能。其次，我们确定了有关推理长度的缩放定律，表明随着推理数据长度的增长，模型性能以对数线性的方式增加。最后，我们引入了一种直接的技术来生成任意长度的推理数据，并表明合成数据对于训练推理模型有效。在使用Long1k数据集中对QWEN2.5-32B-Inscruct语言模型进行微调后，我们介绍了我们的模型Long1K-32B，它仅通过1,000个培训样本实现了出色的性能，在数学方面达到了95.6 \％的准确性，在数学上的准确性为91.1.1.1 \％\％\％\％在GPQA Outpormenting DeepSeek-R1-R1-R1-distill-Q.En-Q.En-32222222222。该模型，代码和数据集都是开源的，可在此HTTPS URL上获得。

Title: Mind with Eyes: from Language Reasoning to Multimodal Reasoning

• Authors: Zhiyu Lin, Yifei Gao, Xian Zhao, Yunfan Yang, Jitao Sang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18071
• Pdf URL: https://arxiv.org/pdf/2503.18071
• Copy Paste: [[2503.18071]] Mind with Eyes: from Language Reasoning to Multimodal Reasoning(https://arxiv.org/abs/2503.18071)
• Keywords: language model, agent
• Abstract: Language models have recently advanced into the realm of reasoning, yet it is through multimodal reasoning that we can fully unlock the potential to achieve more comprehensive, human-like cognitive capabilities. This survey provides a systematic overview of the recent multimodal reasoning approaches, categorizing them into two levels: language-centric multimodal reasoning and collaborative multimodal reasoning. The former encompasses one-pass visual perception and active visual perception, where vision primarily serves a supporting role in language reasoning. The latter involves action generation and state update within reasoning process, enabling a more dynamic interaction between modalities. Furthermore, we analyze the technical evolution of these methods, discuss their inherent challenges, and introduce key benchmark tasks and evaluation metrics for assessing multimodal reasoning performance. Finally, we provide insights into future research directions from the following two perspectives: (i) from visual-language reasoning to omnimodal reasoning and (ii) from multimodal reasoning to multimodal agents. This survey aims to provide a structured overview that will inspire further advancements in multimodal reasoning research.
• 摘要：语言模型最近已进入推理领域，但是通过多模式推理，我们可以完全释放获得更全面，类似人类的认知能力的潜力。这项调查提供了最新的多模式推理方法的系统概述，将它们分为两个层面：以语言为中心的多模式推理和协作性多模式推理。前者涵盖了一通的视觉感知和主动的视觉感知，其中愿景主要在语言推理中发挥支持作用。后者涉及推理过程中的动作生成和状态更新，从而在模态之间实现了更具动态的互动。此外，我们分析了这些方法的技术演变，讨论它们的固有挑战，并引入关键的基准任务和评估指标，以评估多模式推理性能。最后，我们从以下两个角度提供了对未来研究方向的见解：（i）从视觉语言推理到综合推理，以及（ii）从多模式推理到多模式的代理。这项调查旨在提供结构化的概述，以激发多模式推理研究中进一步的进步。

Title: On the effectiveness of LLMs for automatic grading of open-ended questions in Spanish

• Authors: Germán Capdehourat, Isabel Amigo, Brian Lorenzo, Joaquín Trigo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18072
• Pdf URL: https://arxiv.org/pdf/2503.18072
• Copy Paste: [[2503.18072]] On the effectiveness of LLMs for automatic grading of open-ended questions in Spanish(https://arxiv.org/abs/2503.18072)
• Keywords: llm, prompt
• Abstract: Grading is a time-consuming and laborious task that educators must face. It is an important task since it provides feedback signals to learners, and it has been demonstrated that timely feedback improves the learning process. In recent years, the irruption of LLMs has shed light on the effectiveness of automatic grading. In this paper, we explore the performance of different LLMs and prompting techniques in automatically grading short-text answers to open-ended questions. Unlike most of the literature, our study focuses on a use case where the questions, answers, and prompts are all in Spanish. Experimental results comparing automatic scores to those of human-expert evaluators show good outcomes in terms of accuracy, precision and consistency for advanced LLMs, both open and proprietary. Results are notably sensitive to prompt styles, suggesting biases toward certain words or content in the prompt. However, the best combinations of models and prompt strategies, consistently surpasses an accuracy of 95% in a three-level grading task, which even rises up to more than 98% when the it is simplified to a binary right or wrong rating problem, which demonstrates the potential that LLMs have to implement this type of automation in education applications.
• 摘要：评分是教育工作者必须面对的一项耗时且费力的任务。这是一项重要的任务，因为它为学习者提供了反馈信号，并且已经证明及时的反馈可以改善学习过程。近年来，LLM的爆发揭示了自动分级的有效性。在本文中，我们探讨了不同LLM的性能，并促使技术在自动对开放式问题的简短答案中进行分级。与大多数文献不同，我们的研究集中在用例，答案和提示都在西班牙语中的用例中。将自动分数与人类专家评估者的自动分数进行比较的实验结果，在开放和专有的高级LLMS的准确性，精度和一致性方面都表现出良好的结果。结果对提示样式非常敏感，表明提示中某些单词或内容偏见。但是，在三级分级任务中，模型和迅速策略的最佳组合始终超过95％的精度，当简化二进制级别或错误的评级问题时，它甚至上升到98％以上，这表明LLM必须在教育应用中实施这种类型的自动化。

Title: A Multi-Model Adaptation of Speculative Decoding for Classification

• Authors: Somnath Roy, Padharthi Sreekar, Srivatsa Narasimha, Anubhav Anand
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18076
• Pdf URL: https://arxiv.org/pdf/2503.18076
• Copy Paste: [[2503.18076]] A Multi-Model Adaptation of Speculative Decoding for Classification(https://arxiv.org/abs/2503.18076)
• Keywords: chat
• Abstract: The current study introduces a novel adaptation of speculative decoding, repurposed from generation to classification tasks. We propose a multi-model framework employing up to three lightweight worker models and a single, more robust judge model analogous to draft models and target model, respectively, in speculative decoding. The worker models, tasked with the bulk of the computation, independently predict discrete class labels for a given input. When majority worker models agree on a label, it is accepted as the final label, optimizing efficiency by bypassing the computationally expensive judge model. In cases of disagreement, the judge model intervenes to resolve the label. This approach minimizes redundant computation, leverages the redundancy of multiple workers for confidence, and confines the judge model's role to challenging cases, offering a practical balance of efficiency and accuracy. Our analysis suggests that smaller out of the box instruction/chat finetuned worker models with 3 billion parameters (hereafter, 3B) demonstrate a level of alignment with judge models comparable to that of larger finetuned worker models with 7 billion parameters (hereafter, 7B) across both simple and higher order reasoning tasks. The top performing 3B worker model pair achieve an agreement rate of approximately 80-83% for sentiment and around 50-80% for similar ticket when compared to judge models. Additionally, 3B worker models provide a speedup ranging from 2.8x to 9x relative to the judge models, while 7B worker model combinations achieve a speedup ranging from 1.28x to 0.28x
• 摘要：当前的研究介绍了对投机解码的新颖适应，并从世代重新利用到分类任​​务。我们提出了一个多模型框架，该框架采用了多达三个轻量级的工人模型，并且在投机解码方面分别采用了类似于草稿模型和目标模型的单一，更强大的法官模型。负责大部分计算的工人模型独立预测给定输入的离散类标签。当多数工人模型同意标签时，它被接受为最终标签，从而通过绕过计算昂贵的法官模型来优化效率。在分歧的情况下，法官模型介入以解决标签。这种方法最大程度地减少了冗余计算，利用了多个工人的冗余，并将法官模型的作用限制在具有挑战性的情况下，从而实现了效率和准确性的实际平衡。我们的分析表明，具有30亿参数（以下简称3B）的较小的框出指令/聊天填充的工人模型表明，与简单和高阶的推理任务中具有70亿个参数（以下简称为7B）的法官模型相媲美的法官模型的一定程度。与裁判型号相比，表现最高的3B工人模型对的情绪达到了约80-83％的协议率约为80-83％，类似机票的协议率约为50-80％。此外，相对于法官模型，3B工人模型提供的加速度从2.8倍到9倍，而7B工人模型组合的加速范围从1.28倍到0.28倍。

Title: Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approach

• Authors: Rochana Chaturvedi, Peyman Baghershahi, Sourav Medya, Barbara Di Eugenio
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.18085
• Pdf URL: https://arxiv.org/pdf/2503.18085
• Copy Paste: [[2503.18085]] Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approach(https://arxiv.org/abs/2503.18085)
• Keywords: language model
• Abstract: Temporal information extraction from unstructured text is essential for contextualizing events and deriving actionable insights, particularly in the medical domain. We address the task of extracting clinical events and their temporal relations using the well-studied I2B2 2012 Temporal Relations Challenge corpus. This task is inherently challenging due to complex clinical language, long documents, and sparse annotations. We introduce GRAPHTREX, a novel method integrating span-based entity-relation extraction, clinical large pre-trained language models (LPLMs), and Heterogeneous Graph Transformers (HGT) to capture local and global dependencies. Our HGT component facilitates information propagation across the document through innovative global landmarks that bridge distant entities. Our method improves the state-of-the-art with 5.5% improvement in the tempeval $F_1$ score over the previous best and up to 8.9% improvement on long-range relations, which presents a formidable challenge. This work not only advances temporal information extraction but also lays the groundwork for improved diagnostic and prognostic models through enhanced temporal reasoning.
• 摘要：从非结构化文本中提取的时间信息对于上下文化事件和得出可行的见解至关重要，尤其是在医学领域中。我们使用经过良好研究的I2B2 2012临时关系挑战语料库提取临床事件及其时间关系的任务。由于复杂的临床语言，长文档和稀疏注释，此任务本质上是具有挑战性的。我们介绍了GraphTrex，这是一种集成基于跨度的实体关系提取，临床大型预训练语言模型（LPLM）和异质图形变压器（HGT）以捕获本地和全局依赖性的新方法。我们的HGT组件通过桥梁遥远实体的创新全球地标有助于整个文档的信息传播。我们的方法改善了最先进的方法，在上一个最佳最佳状态下，Tempeval $ f_1 $得分提高了5.5％，在远程关系方面提高了8.9％，这带来了巨大的挑战。这项工作不仅可以提高时间信息提取，而且还为改善诊断和预后模型通过增强的时间推理奠定了基础。

Title: $D^2LoRA$: Data-Driven LoRA Initialization for Low Resource Tasks

• Authors: Javad SeraJ, Mohammad Mahdi Mohajeri, Mohammad Javad Dousti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18089
• Pdf URL: https://arxiv.org/pdf/2503.18089
• Copy Paste: [[2503.18089]] $D^2LoRA$: Data-Driven LoRA Initialization for Low Resource Tasks(https://arxiv.org/abs/2503.18089)
• Keywords: language model, llm
• Abstract: Tuning large language models is essential for optimizing their performance across diverse applications, particularly in scenarios with limited data availability. Tuning large language models in scarce data scenarios is crucial, particularly given that the convergence speed of the LoRA method is lower than that of full fine-tuning. In this paper, we present an analysis of post-training methods including Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Odds Ratio Preference Optimization (ORPO) within the context of task-specific learning using the LoRA method. Next we introduce $D^2LoRA$, a data-driven approach for initializing LoRA metrics that enhances training efficiency, especially in limited-data settings. Our experiments compare $D^2LoRA$ with vanilla LoRA in terms of performance and catastrophic forgetting under extremely data-constrained conditions. The results demonstrate that $D^2LoRA$ achieves a 1% improvement GSM8K benchmark and a 2-point improvement in ROUGE score in title generation tasks. $D^2LoRA$ facilitates the adaptation of LLMs to multiple tasks even when task-specific data is scarce, thereby reducing training expenses and offering data cost.
• 摘要：调整大语言模型对于在各种应用程序中优化其性能至关重要，尤其是在数据可用性有限的情况下。在稀缺的数据情景中调整大型语言模型至关重要，特别是考虑到洛拉方法的收敛速度低于完整的微调。在本文中，我们介绍了培训后方法的分析，包括监督微调（SFT），直接偏好优化（DPO）和使用LORA方法的特定于任务学习的上下文。接下来，我们介绍$ d^2lora $，这是一种以数据为基础的方法来初始化LORA指标，可提高培训效率，尤其是在有限数据设置中。我们的实验将$ d^2lora $与香草·洛拉（Vanilla Lora）进行了比较，在极度数据约束的条件下的性能和灾难性遗忘方面。结果表明，$ d^2lora $在标题生成任务中提高了1％的GSM8K基准和2分提高了Rouge得分。 $ d^2lora $促进了LLMS对多个任务的改编，即使特定于任务的数据稀缺，从而降低了培训费用并提供数据成本。

Title: Detection of Somali-written Fake News and Toxic Messages on the Social Media Using Transformer-based Language Models

• Authors: Muhidin A. Mohamed, Shuab D. Ahmed, Yahye A. Isse, Hanad M. Mohamed, Fuad M. Hassan, Houssein A. Assowe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18117
• Pdf URL: https://arxiv.org/pdf/2503.18117
• Copy Paste: [[2503.18117]] Detection of Somali-written Fake News and Toxic Messages on the Social Media Using Transformer-based Language Models(https://arxiv.org/abs/2503.18117)
• Keywords: language model
• Abstract: The fact that everyone with a social media account can create and share content, and the increasing public reliance on social media platforms as a news and information source bring about significant challenges such as misinformation, fake news, harmful content, etc. Although human content moderation may be useful to an extent and used by these platforms to flag posted materials, the use of AI models provides a more sustainable, scalable, and effective way to mitigate these harmful contents. However, low-resourced languages such as the Somali language face limitations in AI automation, including scarce annotated training datasets and lack of language models tailored to their unique linguistic characteristics. This paper presents part of our ongoing research work to bridge some of these gaps for the Somali language. In particular, we created two human-annotated social-media-sourced Somali datasets for two downstream applications, fake news \& toxicity classification, and developed a transformer-based monolingual Somali language model (named SomBERTa) -- the first of its kind to the best of our knowledge. SomBERTa is then fine-tuned and evaluated on toxic content, fake news and news topic classification datasets. Comparative evaluation analysis of the proposed model against related multilingual models (e.g., AfriBERTa, AfroXLMR, etc) demonstrated that SomBERTa consistently outperformed these comparators in both fake news and toxic content classification tasks while achieving the best average accuracy (87.99%) across all tasks. This research contributes to Somali NLP by offering a foundational language model and a replicable framework for other low-resource languages, promoting digital and AI inclusivity and linguistic diversity.
• 摘要：拥有社交媒体帐户的每个人都可以创建和共享内容，并且越来越多的公众对社交媒体平台的依赖作为新闻和信息来源带来了重大挑战，例如错误信息，虚假新闻，有害内容等。尽管人类内容适度在某种程度上可能是有用的，并且在一定程度上可以使用这些平台来标记发布的材料，但对AI模型的使用则可以提供更具可持续性，可伸缩的方式，可提供有效的危害和有效的危害，并有效地构成了这些危害的措施。但是，诸如索马里语言之类的低资源语言在AI自动化中面临限制，包括稀缺的注释培训数据集以及缺乏针对其独特语言特征的语言模型。本文介绍了我们正在进行的研究工作的一部分，以弥合索马里语言中的一些差距。特别是，我们为两个下游应用程序，假新闻\＆毒性分类创建了两个人类认可的社交媒体媒体数据集，并开发了一种基于变压器的单语索马里语模型（命名为Somberta），这是我们最佳知识的第一个。然后，Somberta对有毒内容，假新闻和新闻主题分类数据集进行了微调和评估。针对相关多语言模型（例如Afriberta，Afroxlmr等）对拟议模型的比较评估分析表明，在所有任务中，Somberta始终在假新闻和有毒内容分类任务中表现出这些比较器的表现，同时达到了最佳的平均准确性（87.99％）。这项研究通过为其他低资源语言提供基础语言模型和可复制框架来促进索马里NLP，从而促进数字和人工智能包容性和语言多样性。

Title: GeoBenchX: Benchmarking LLMs for Multistep Geospatial Tasks

• Authors: Varvara Krechetova, Denis Kochedykov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18129
• Pdf URL: https://arxiv.org/pdf/2503.18129
• Copy Paste: [[2503.18129]] GeoBenchX: Benchmarking LLMs for Multistep Geospatial Tasks(https://arxiv.org/abs/2503.18129)
• Keywords: language model, gpt, llm, hallucination, agent
• Abstract: In this paper, we establish a benchmark for evaluating large language models (LLMs) on multi-step geospatial tasks relevant to commercial GIS practitioners. We assess seven leading commercial LLMs (Sonnet 3.5 and 3.7, Haiku 3.5, Gemini 2.0, GPT-4o, GPT-4o mini, and o3-mini) using a simple tool-calling agent equipped with 23 geospatial functions. Our benchmark comprises tasks across four categories of increasing complexity, with both solvable and intentionally unsolvable tasks to test hallucination rejection. We develop an LLM-as-Judge evaluation framework to compare agent solutions against reference implementations. Results show Sonnet 3.5 and GPT-4o achieve the best overall performance, with Claude models excelling on solvable tasks while OpenAI models better identify unsolvable scenarios. We observe significant differences in token usage, with Anthropic models consuming substantially more tokens than competitors. Common errors include misunderstanding geometrical relationships, relying on outdated knowledge, and inefficient data manipulation. The resulting benchmark set, evaluation framework, and data generation pipeline are released as open-source resources, providing one more standardized method for ongoing evaluation of LLMs for GeoAI.
• 摘要：在本文中，我们建立了一个基准，用于评估与商业GIS从业者有关的多步地理空间任务的大型语言模型（LLM）。我们使用配备有23个地理空间功能的简单工具称呼剂来评估七个领先的商业LLM（SONNET 3.5和3.7，HAIKU 3.5，GEMINI 2.0，GPT-4O，GPT-4O MINI和O3-MINI）。我们的基准包括跨四类复杂性的任务，以及可解决的和有意无法解决的任务以测试幻觉拒绝。我们开发了一个LLM-AS法官评估框架，以将代理解决方案与参考实现进行比较。结果表明，十四行诗3.5和GPT-4O实现了最佳的整体性能，克劳德（Claude）模型在可解决的任务上脱颖而出，而OpenAI模型可以更好地识别无法解决的方案。我们观察到令牌使用情况的显着差异，而拟人化模型比竞争对手要消耗的代币更多。常见错误包括误解几何关系，依靠过时的知识以及效率低下的数据操纵。由此产生的基准集，评估框架和数据生成管道作为开源资源发布，为GEOAI的LLMS持续评估提供了另一种标准化的方法。

Title: MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection

• Authors: Yibo Yan, Shen Wang, Jiahao Huo, Philip S. Yu, Xuming Hu, Qingsong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18132
• Pdf URL: https://arxiv.org/pdf/2503.18132
• Copy Paste: [[2503.18132]] MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection(https://arxiv.org/abs/2503.18132)
• Keywords: language model, llm, agent
• Abstract: Mathematical error detection in educational settings presents a significant challenge for Multimodal Large Language Models (MLLMs), requiring a sophisticated understanding of both visual and textual mathematical content along with complex reasoning capabilities. Though effective in mathematical problem-solving, MLLMs often struggle with the nuanced task of identifying and categorizing student errors in multimodal mathematical contexts. Therefore, we introduce MathAgent, a novel Mixture-of-Math-Agent framework designed specifically to address these challenges. Our approach decomposes error detection into three phases, each handled by a specialized agent: an image-text consistency validator, a visual semantic interpreter, and an integrative error analyzer. This architecture enables more accurate processing of mathematical content by explicitly modeling relationships between multimodal problems and student solution steps. We evaluate MathAgent on real-world educational data, demonstrating approximately 5% higher accuracy in error step identification and 3% improvement in error categorization compared to baseline models. Besides, MathAgent has been successfully deployed in an educational platform that has served over one million K-12 students, achieving nearly 90% student satisfaction while generating significant cost savings by reducing manual error detection.
• 摘要：教育环境中的数学错误检测对多模式大语言模型（MLLM）提出了重大挑战，需要对视觉和文本数学内容以及复杂的推理能力进行复杂的理解。尽管在数学问题解决方案方面有效，但MLLM经常在识别和分类多模式数学环境中的学生错误的细微任务上挣扎。因此，我们介绍了Mathagent，这是一种专门针对这些挑战的新型混合物框架。我们的方法将误差检测分解为三个阶段，每个阶段都由专门代理处理：图像文本一致性验证器，视觉语义解释器和集成错误分析仪。通过对多模式问题和学生解决方案步骤之间的明确建模关系，该体系结构可以更准确地处理数学内容。我们在现实世界的教育数据上评估了数学，与基线模型相比，错误步骤识别的准确性高约5％，错误分类的提高了3％。此外，Mathagent已成功地部署在一个为超过100万K-12学生提供服务的教育平台中，并通过减少手动错误检测来节省了近90％的学生满意度。

Title: Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering

• Authors: Zixin Chen, Sicheng Song, Kashun Shum, Yanna Lin, Rui Sheng, Huamin Qu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18172
• Pdf URL: https://arxiv.org/pdf/2503.18172
• Copy Paste: [[2503.18172]] Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering(https://arxiv.org/abs/2503.18172)
• Keywords: language model, llm
• Abstract: Misleading chart visualizations, which intentionally manipulate data representations to support specific claims, can distort perceptions and lead to incorrect conclusions. Despite decades of research, misleading visualizations remain a widespread and pressing issue. Recent advances in multimodal large language models (MLLMs) have demonstrated strong chart comprehension capabilities, yet no existing work has systematically evaluated their ability to detect and interpret misleading charts. This paper introduces the Misleading Chart Question Answering (Misleading ChartQA) Benchmark, a large-scale multimodal dataset designed to assess MLLMs in identifying and reasoning about misleading charts. It contains over 3,000 curated examples, covering 21 types of misleaders and 10 chart types. Each example includes standardized chart code, CSV data, and multiple-choice questions with labeled explanations, validated through multi-round MLLM checks and exhausted expert human review. We benchmark 16 state-of-the-art MLLMs on our dataset, revealing their limitations in identifying visually deceptive practices. We also propose a novel pipeline that detects and localizes misleaders, enhancing MLLMs' accuracy in misleading chart interpretation. Our work establishes a foundation for advancing MLLM-driven misleading chart comprehension. We publicly release the sample dataset to support further research in this critical area.
• 摘要：误导图表的可视化（故意操纵数据表示以支持特定主张）可能会扭曲感知并导致结论不正确。尽管进行了数十年的研究，但具有误导性的可视化仍然是一个普遍且紧迫的问题。多模式大语言模型（MLLM）的最新进展证明了强大的图表理解能力，但是现有的工作没有系统地评估其检测和解释误导图表的能力。本文介绍了误导性图表问题回答（误导性ChartQA）基准，这是一个大规模的多模式数据集，旨在评估MLLM，以识别和推理有关误导图表的推理。它包含3,000多个精选示例，涵盖了21种误导者和10种图表类型。每个示例包括标准化的图表代码，CSV数据以及带有标记说明的多项选择问题，通过多轮MLLM检查和耗尽的专家人类评论进行了验证。我们在数据集上基准了16个最先进的MLLM，揭示了它们在识别视觉欺骗性实践中的局限性。我们还提出了一条新型的管道，该管道检测并定位了误导者，从而提高了MLLM在误导图表解释方面的准确性。我们的工作为推进MLLM驱动的误导图表理解的基础奠定了基础。我们公开发布样本数据集，以支持该关键领域的进一步研究。

Title: GINGER: Grounded Information Nugget-Based Generation of Responses

• Authors: Weronika Łajewska, Krisztian Balog
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.18174
• Pdf URL: https://arxiv.org/pdf/2503.18174
• Copy Paste: [[2503.18174]] GINGER: Grounded Information Nugget-Based Generation of Responses(https://arxiv.org/abs/2503.18174)
• Keywords: retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) faces challenges related to factual correctness, source attribution, and response completeness. To address them, we propose a modular pipeline for grounded response generation that operates on information nuggets-minimal, atomic units of relevant information extracted from retrieved documents. The multistage pipeline encompasses nugget detection, clustering, ranking, top cluster summarization, and fluency enhancement. It guarantees grounding in specific facts, facilitates source attribution, and ensures maximum information inclusion within length constraints. Extensive experiments on the TREC RAG'24 dataset evaluated with the AutoNuggetizer framework demonstrate that GINGER achieves state-of-the-art performance on this benchmark.
• 摘要：检索增强的生成（RAG）面临与事实正确性，源归因和响应完整性有关的挑战。为了解决这些问题，我们为接地响应生成提出了一个模块化管道，该管道在信息块中最小的，原子能单位运行，这些单位的相关信息是从检索到的文档中提取的相关信息。多阶段管道包含掘金检测，聚类，排名，顶部群集摘要和流利度增强。它保证以特定事实为基础，促进源归因，并确保最大的信息包含在长度约束中。使用AutoNuggetizer框架评估的TREC RAG'24数据集进行了广泛的实验表明，Ginger在此基准测试中实现了最新的性能。

Title: LakotaBERT: A Transformer-based Model for Low Resource Lakota Language

• Authors: Kanishka Parankusham, Rodrigue Rizk, KC Santosh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.18212
• Pdf URL: https://arxiv.org/pdf/2503.18212
• Copy Paste: [[2503.18212]] LakotaBERT: A Transformer-based Model for Low Resource Lakota Language(https://arxiv.org/abs/2503.18212)
• Keywords: language model, llm
• Abstract: Lakota, a critically endangered language of the Sioux people in North America, faces significant challenges due to declining fluency among younger generations. This paper introduces LakotaBERT, the first large language model (LLM) tailored for Lakota, aiming to support language revitalization efforts. Our research has two primary objectives: (1) to create a comprehensive Lakota language corpus and (2) to develop a customized LLM for Lakota. We compiled a diverse corpus of 105K sentences in Lakota, English, and parallel texts from various sources, such as books and websites, emphasizing the cultural significance and historical context of the Lakota language. Utilizing the RoBERTa architecture, we pre-trained our model and conducted comparative evaluations against established models such as RoBERTa, BERT, and multilingual BERT. Initial results demonstrate a masked language modeling accuracy of 51% with a single ground truth assumption, showcasing performance comparable to that of English-based models. We also evaluated the model using additional metrics, such as precision and F1 score, to provide a comprehensive assessment of its capabilities. By integrating AI and linguistic methodologies, we aspire to enhance linguistic diversity and cultural resilience, setting a valuable precedent for leveraging technology in the revitalization of other endangered indigenous languages.
• 摘要：拉科塔（Lakota）是北美人民苏福人（Sioux）的一种濒临灭绝的语言，由于年轻一代的流畅性下降，面临着巨大的挑战。本文介绍了Lakotabert，这是针对Lakota量身定制的第一个大型语言模型（LLM），旨在支持语言振兴工作。我们的研究有两个主要目标：（1）创建一个全面的Lakota语言语料库和（2）为Lakota开发定制的LLM。我们在Lakota，英语和来自各种来源（例如书籍和网站）的平行文本中编写了各种各样的句子，强调了Lakota语言的文化意义和历史背景。利用罗伯塔体系结构，我们对模型进行了预先培训，并对诸如Roberta，Bert和多语言Bert等既定模型进行了比较评估。最初的结果表明，具有单个基础真理假设的掩盖语言建模精度为51％，展示了与基于英语的模型相当的性能。我们还使用其他指标（例如精度和F1得分）评估了该模型，以全面评估其功能。通过整合AI和语言学方法，我们渴望增强语言多样性和文化弹性，为利用技术振兴其他濒危土著语言的利用有价值的先例。

Title: ShED-HD: A Shannon Entropy Distribution Framework for Lightweight Hallucination Detection on Edge Devices

• Authors: Aneesh Vathul, Daniel Lee, Sheryl Chen, Arthi Tasmia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18242
• Pdf URL: https://arxiv.org/pdf/2503.18242
• Copy Paste: [[2503.18242]] ShED-HD: A Shannon Entropy Distribution Framework for Lightweight Hallucination Detection on Edge Devices(https://arxiv.org/abs/2503.18242)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities on a broad array of NLP tasks, but their tendency to produce hallucinations$\unicode{x2013}$plausible-sounding but factually incorrect content$\unicode{x2013}$poses severe challenges in high-stakes domains. Existing hallucination detection methods either bear the computational cost of multiple inference passes or sacrifice accuracy for efficiency with single-pass approaches, neither of which is ideal in resource-constrained environments such as edge devices. We propose the Shannon Entropy Distribution Hallucination Detector (ShED-HD), a novel hallucination detection framework that bridges this gap by classifying sequence-level entropy patterns using a lightweight BiLSTM architecture with single-headed attention. In contrast to prior approaches, ShED-HD efficiently detects distinctive uncertainty patterns across entire output sequences, preserving contextual awareness. Through in-depth evaluation on three datasets (BioASQ, TriviaQA, and Jeopardy Questions), we show that ShED-HD significantly outperforms other computationally efficient approaches in the out-of-distribution setting, while achieving comparable performance in the in-distribution setting. ShED-HD facilitates hallucination detection that is low-cost, accurate, and generalizable, improving the credibility of content generated by LLMs in resource-constrained environments where trustworthy AI functionality is crucial.
• 摘要：大型语言模型（LLM）在一系列NLP任务上表现出了令人印象深刻的功能，但是它们产生幻觉的趋势$ \ unicode {x2013} $ plausible-plausible seans-subsible nikect of Forract Indect of Content $ \ unicode $ \ Unicode {x2013} $在高固定域中引起了严重的挑战。现有的幻觉检测方法要么具有多个推理通行证的计算成本，要么使用单通道方法牺牲精度，而在资源约束环境（例如边缘设备）中，这两种方法都不是理想的选择。我们提出了香农熵分布幻觉检测器（SHED-HD），这是一种新型的幻觉检测框架，通过使用轻量级的Bilstm体系结构与单人注意的序列级熵模式进行分类，从而弥合了这一差距。与先前的方法相反，SHED-HD有效地检测了整个输出序列的独特不确定性模式，从而保留了上下文意识。通过对三个数据集（Bioasq，Triviaqa和危险问题）进行深入评估，我们表明，在分布式设置中，SHED-HD显着优于其他计算有效的方法，同时在分布式设置中实现可比的性能。 SHED-HD促进了低成本，准确且可推广的幻觉检测，从而提高了LLM在资源约束环境中产生的内容的可信度，在资源受限的环境中，值得信赖的AI功能至关重要。

Title: AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text

• Authors: Tadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18247
• Pdf URL: https://arxiv.org/pdf/2503.18247
• Copy Paste: [[2503.18247]] AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text(https://arxiv.org/abs/2503.18247)
• Keywords: language model
• Abstract: Pretrained Language Models (PLMs) built from various sources are the foundation of today's NLP progress. Language representations learned by such models achieve strong performance across many tasks with datasets of varying sizes drawn from various sources. We explore a thorough analysis of domain and task adaptive continual pretraining approaches for low-resource African languages and a promising result is shown for the evaluated tasks. We create AfriSocial, a corpus designed for domain adaptive finetuning that passes through quality pre-processing steps. Continual pretraining PLMs using AfriSocial as domain adaptive pretraining (DAPT) data, consistently improves performance on fine-grained emotion classification task of 16 targeted languages from 1% to 28.27% macro F1 score. Likewise, using the task adaptive pertaining (TAPT) approach, further finetuning with small unlabeled but similar task data shows promising results. For example, unlabeled sentiment data (source) for fine-grained emotion classification task (target) improves the base model results by an F1 score ranging from 0.55% to 15.11%. Combining the two methods, DAPT + TAPT, achieves also better results than base models. All the resources will be available to improve low-resource NLP tasks, generally, as well as other similar domain tasks such as hate speech and sentiment tasks.
• 摘要：从各种来源构建的验证语言模型（PLM）是当今NLP进展的基础。通过从各种来源绘制的各种大小的数据集，通过这种模型学到的语言表示在许多任务中实现了强劲的性能。我们探讨了对低资源非洲语言的域和任务自适应持续预处理方法的彻底分析，并显示了评估任务的有希望的结果。我们创建了Afrisocial，这是一种专为域自适应填充而设计的语料库，通过质量预处理步骤。使用Afrisocial作为域自适应预处理（DAPT）数据持续预处理PLM，一贯提高16种目标语言的细粒度情绪分类任务的性能从1％到28.27％的宏F1得分。同样，使用任务自适应相关方法（TAPT）方法，使用小规定但类似的任务数据进行进一步的填充显示出令人鼓舞的结果。例如，用于细粒度情绪分类任务（目标）的未标记情感数据（源）通过F1分数从0.55％到15.11％提高了基本模型结果。将两种方法（DAPT + TAPT）结合在一起，也比基本模型更好。通常，所有资源都可以通过一般来改善低资源NLP任务以及其他类似的领域任务，例如仇恨言论和情感任务。

Title: Enhancing Multi-Label Emotion Analysis and Corresponding Intensities for Ethiopian Languages

• Authors: Tadesse Destaw Belay, Dawit Ketema Gete, Abinew Ali Ayele, Olga Kolesnikova, Grigori Sidorov, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18253
• Pdf URL: https://arxiv.org/pdf/2503.18253
• Copy Paste: [[2503.18253]] Enhancing Multi-Label Emotion Analysis and Corresponding Intensities for Ethiopian Languages(https://arxiv.org/abs/2503.18253)
• Keywords: language model, llm
• Abstract: In this digital world, people freely express their emotions using different social media platforms. As a result, modeling and integrating emotion-understanding models are vital for various human-computer interaction tasks such as decision-making, product and customer feedback analysis, political promotions, marketing research, and social media monitoring. As users express different emotions simultaneously in a single instance, annotating emotions in a multilabel setting such as the EthioEmo (Belay et al., 2025) dataset effectively captures this dynamic. Additionally, incorporating intensity, or the degree of emotion, is crucial, as emotions can significantly differ in their expressive strength and impact. This intensity is significant for assessing whether further action is necessary in decision-making processes, especially concerning negative emotions in applications such as healthcare and mental health studies. To enhance the EthioEmo dataset, we include annotations for the intensity of each labeled emotion. Furthermore, we evaluate various state-of-the-art encoder-only Pretrained Language Models (PLMs) and decoder-only Large Language Models (LLMs) to provide comprehensive benchmarking.
• 摘要：在这个数字世界中，人们使用不同的社交媒体平台自由表达自己的情绪。结果，建模和整合情感理解模型对于各种人类计算机互动任务，例如决策，产品和客户反馈分析，政治促销，市场研究和社交媒体监控至关重要。当用户同时在单个实例中表达不同的情绪时，在多标记设置（例如ethioemo（Belay等，2025）数据集等多标记设置中，情绪有效地捕获了这种动态。另外，融合强度或情感程度至关重要，因为情绪的表现力和影响可能会有很大差异。这种强度对于评估在决策过程中是否需要采取进一步的行动，尤其是关于医疗保健和心理健康研究等应用中的负面情绪的重要行动。为了增强以乙基emo的数据集，我们包括每个标记情绪强度的注释。此外，我们评估了各种最新的仅经过审计的语言模型（PLM）和仅解码器的大型语言模型（LLMS），以提供全面的基准测试。

Title: Sun-Shine: A Large Language Model for Tibetan Culture

• Authors: Cheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeg Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Xiao Feng, Yongbin Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18288
• Pdf URL: https://arxiv.org/pdf/2503.18288
• Copy Paste: [[2503.18288]] Sun-Shine: A Large Language Model for Tibetan Culture(https://arxiv.org/abs/2503.18288)
• Keywords: language model, llm
• Abstract: Tibetan, a minority language in China, features a highly intricate grammatical structure, characterized by four verb tenses and a tense system with frequent irregularities, contributing to its extensive inflectional diversity. Recently, advances in Large Language Models (LLMs) have transformed the paradigm in many domains. Despite the success in other fields, current LLMs often fall short in catering to the needs of domain experts like Tibetans, and the potential of LLMs for Tibetan culture is under-explored. The intrinsic reasons are the immense and intricate nature of Tibetan culture as well as the necessity for higher granularity and richness in knowledge. Simultaneously, the complexity and uniqueness of its grammatical structure, coupled with its status as a minority ethnic language, contribute to data scarcity, which remains a fundamental challenge. To alleviate these issues, we introduce Llama-Sunshine (Sun-Shine), the first large language model for Tibetan culture, which is expert in various Tibetan language processing tasks. Sun-Shine incorporates state-of-the-art model architectures optimized for Tibetan's linguistic features. We also propose TIB-STC, a comprehensive dataset comprising diverse Tibetan texts such as literature, religious scripts, news, and conversational data, which is also the first large-scale dataset for Tibetan culture. Though comprehensive experiments, Sun-Shine not only demonstrates a higher level of knowledge expertise for Tibetan culture but also gains preliminary embodied intelligence capabilities in Tibetan language processing tasks, like language modeling, text classification, machine translation, and syntactic analysis. Moreover, it excels in low-resource scenarios, showcasing strong generalization capabilities.
• 摘要：藏族是中国的少数族裔语言，具有高度复杂的语法结构，其特征是四个动词时态和频繁不规则的时态系统，这有助于其广泛的拐点多样性。最近，大语模型（LLM）的进步改变了许多领域的范式。尽管在其他领域取得了成功，但目前的LLM通常在满足藏族等领域专家的需求方面通常不足，而LLMS对藏族文化的潜力却没有探索。内在的原因是藏族文化的巨大和复杂性，以及具有更高粒度和丰富性知识的必要性。同时，其语法结构的复杂性和独特性，再加上其作为少数种族语言的地位，导致了数据稀缺性，这仍然是一个基本的挑战。为了减轻这些问题，我们介绍了Llama-Sunshine（Sun-Shine），这是藏族文化的第一个大型语言模型，该模型是各种藏族语言处理任务的专家。 Sun-Shine融合了针对藏族语言特征优化的最先进的模型体系结构。我们还提出了TIB-STC，这是一个综合数据集，其中包括文学，宗教脚本，新闻和对话数据等多样化的藏语文本，这也是第一个用于藏族文化的大型数据集。尽管全面实验，Sun-Shine不仅展示了藏族文化的更高水平的知识专业知识，而且还获得了藏族语言处理任务的初步体现的智能能力，例如语言建模，文本分类，机器翻译和句法分析。此外，它在低资源场景中表现出色，展示了强大的概括能力。

Title: Fact-checking AI-generated news reports: Can LLMs catch their own lies?

• Authors: Jiayi Yao, Haibo Sun, Nianwen Xue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18293
• Pdf URL: https://arxiv.org/pdf/2503.18293
• Copy Paste: [[2503.18293]] Fact-checking AI-generated news reports: Can LLMs catch their own lies?(https://arxiv.org/abs/2503.18293)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: In this paper, we evaluate the ability of Large Language Models (LLMs) to assess the veracity of claims in ''news reports'' generated by themselves or other LLMs. Our goal is to determine whether LLMs can effectively fact-check their own content, using methods similar to those used to verify claims made by humans. Our findings indicate that LLMs are more effective at assessing claims in national or international news stories than in local news stories, better at evaluating static information than dynamic information, and better at verifying true claims compared to false ones. We hypothesize that this disparity arises because the former types of claims are better represented in the training data. Additionally, we find that incorporating retrieved results from a search engine in a Retrieval-Augmented Generation (RAG) setting significantly reduces the number of claims an LLM cannot assess. However, this approach also increases the occurrence of incorrect assessments, partly due to irrelevant or low-quality search results. This diagnostic study highlights the need for future research on fact-checking machine-generated reports to prioritize improving the precision and relevance of retrieved information to better support fact-checking efforts. Furthermore, claims about dynamic events and local news may require human-in-the-loop fact-checking systems to ensure accuracy and reliability.
• 摘要：在本文中，我们评估了大语言模型（LLMS）在自己或其他LLM产生的“新闻报告”中评估索赔真实性的能力。我们的目标是使用类似于验证人类提出的主张的方法来确定LLM是否可以有效地检查自己的内容。我们的发现表明，LLM在评估国家或国际新闻报道中比本地新闻报道更有效，在评估静态信息方面比动态信息更好，并且与虚假的信息相比，更擅长验证真实主张。我们假设出现这种差异，因为培训数据中以前的索赔类型更好。此外，我们发现将搜索引擎检索的结果纳入检索功能的生成（RAG）设置会大大减少LLM无法评估的索赔数量。但是，这种方法还增加了错误评估的发生，部分原因是无关或低质量的搜索结果。这项诊断研究强调了对事实检查机器生成的报告的未来研究的必要性，以优先提高检索到的信息的精度和相关性，以更好地支持事实检查工作。此外，关于动态事件和当地新闻的主张可能需要人类的事实检查系统，以确保准确性和可靠性。

Title: Surgical Action Planning with Large Language Models

• Authors: Mengya Xu, Zhongzhen Huang, Jie Zhang, Xiaofan Zhang, Qi Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18296
• Pdf URL: https://arxiv.org/pdf/2503.18296
• Copy Paste: [[2503.18296]] Surgical Action Planning with Large Language Models(https://arxiv.org/abs/2503.18296)
• Keywords: language model, llm, prompt
• Abstract: In robot-assisted minimally invasive surgery, we introduce the Surgical Action Planning (SAP) task, which generates future action plans from visual inputs to address the absence of intraoperative predictive planning in current intelligent applications. SAP shows great potential for enhancing intraoperative guidance and automating procedures. However, it faces challenges such as understanding instrument-action relationships and tracking surgical progress. Large Language Models (LLMs) show promise in understanding surgical video content but remain underexplored for predictive decision-making in SAP, as they focus mainly on retrospective analysis. Challenges like data privacy, computational demands, and modality-specific constraints further highlight significant research gaps. To tackle these challenges, we introduce LLM-SAP, a Large Language Models-based Surgical Action Planning framework that predicts future actions and generates text responses by interpreting natural language prompts of surgical goals. The text responses potentially support surgical education, intraoperative decision-making, procedure documentation, and skill analysis. LLM-SAP integrates two novel modules: the Near-History Focus Memory Module (NHF-MM) for modeling historical states and the prompts factory for action planning. We evaluate LLM-SAP on our constructed CholecT50-SAP dataset using models like Qwen2.5 and Qwen2-VL, demonstrating its effectiveness in next-action prediction. Pre-trained LLMs are tested zero-shot, and supervised fine-tuning (SFT) with LoRA is implemented to address data privacy concerns. Our experiments show that Qwen2.5-72B-SFT surpasses Qwen2.5-72B with a 19.3% higher accuracy.
• 摘要：在机器人辅助的微创手术中，我们介绍了手术动作计划（SAP）任务，该任务从视觉输入中生成了未来的行动计划，以解决当前智能应用程序中缺乏术中预测计划。 SAP具有增强术中指导和自动化程序的巨大潜力。但是，它面临着诸如了解仪器行动关系和跟踪手术进展之类的挑战。大型语言模型（LLMS）在理解手术视频内容方面表现出希望，但仍未进行SAP中的预测决策，因为它们主要关注回顾性分析。诸如数据隐私，计算需求和特定方式约束之类的挑战进一步突出了重大的研究差距。为了应对这些挑战，我们介绍了LLM-SAP，这是一个基于语言模型的大型外科手术行动计划框架，可预测未来的动作并通过解释自然语言手术目标提示来产生文本响应。文本反应可能支持手术教育，术中决策，程序文档和技能分析。 LLM-SAP集成了两个新型模块：用于建模历史状态和提示工厂以进行行动计划的近历史焦点内存模块（NHF-MM）。我们使用QWEN2.5和QWEN2-VL等模型在构建的Cholect50-SAP数据集上评估LLM-SAP，以证明其在下一步预测中的有效性。预先训练的LLMS经过零射击测试，并实施具有LORA的监督微调（SFT）来解决数据隐私问题。我们的实验表明，QWEN2.5-72B-SFT超过QWEN2.5-72B，精度提高19.3％。

Title: J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain

• Authors: Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18360
• Pdf URL: https://arxiv.org/pdf/2503.18360
• Copy Paste: [[2503.18360]] J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain(https://arxiv.org/abs/2503.18360)
• Keywords: language model, llm
• Abstract: As the scale and capabilities of Large Language Models (LLMs) increase, their applications in knowledge-intensive fields such as legal domain have garnered widespread attention. However, it remains doubtful whether these LLMs make judgments based on domain knowledge for reasoning. If LLMs base their judgments solely on specific words or patterns, rather than on the underlying logic of the language, the ''LLM-as-judges'' paradigm poses substantial risks in the real-world applications. To address this question, we propose a method of legal knowledge injection attacks for robustness testing, thereby inferring whether LLMs have learned legal knowledge and reasoning logic. In this paper, we propose J&H: an evaluation framework for detecting the robustness of LLMs under knowledge injection attacks in the legal domain. The aim of the framework is to explore whether LLMs perform deductive reasoning when accomplishing legal tasks. To further this aim, we have attacked each part of the reasoning logic underlying these tasks (major premise, minor premise, and conclusion generation). We have collected mistakes that legal experts might make in judicial decisions in the real world, such as typos, legal synonyms, inaccurate external legal statutes retrieval. However, in real legal practice, legal experts tend to overlook these mistakes and make judgments based on logic. However, when faced with these errors, LLMs are likely to be misled by typographical errors and may not utilize logic in their judgments. We conducted knowledge injection attacks on existing general and domain-specific LLMs. Current LLMs are not robust against the attacks employed in our experiments. In addition we propose and compare several methods to enhance the knowledge robustness of LLMs.
• 摘要：随着大语言模型（LLM）的规模和能力的增加，它们在法律领域等知识密集型领域的应用引起了广泛关注。但是，这些LLM是否基于领域知识进行推理判断仍然令人怀疑。如果llms仅以特定的单词或模式为基础，而不是基于语言的基本逻辑，则“ llm-as as-gudges''范式在现实世界应用程序中构成了重大风险。为了解决这个问题，我们提出了一种用于鲁棒性测试的法律知识注入攻击方法，从而推断LLM是否学会了法律知识和推理逻辑。在本文中，我们提出了J＆H：在法律领域的知识注射攻击下检测LLM的鲁棒性的评估框架。该框架的目的是探索LLM在完成法律任务时是否执行演绎推理。为了进一步实现这一目标，我们攻击了这些任务的基础推理逻辑的每个部分（主要前提，次要前提和结论生成）。我们已经收集了法律专家可能在现实世界中司法决定中犯的错误，例如错别字，法律同义词，不准确的外部法律法规检索。但是，在实际的法律实践中，法律专家倾向于忽略这些错误并基于逻辑做出判断。但是，面对这些错误时，LLM可能会被印刷错误误导，并且可能不会在其判断中使用逻辑。我们对现有的一般和域特异性LLM进行了知识注入攻击。当前的LLM对我们实验中采用的攻击并不强大。此外，我们提出并比较了增强LLMS知识鲁棒性的几种方法。

Title: Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning

• Authors: Junsong Li, Jie Zhou, Yutao Yang, Bihao Zhan, Qianjun Pan, Yuyang Ding, Qin Chen, Jiang Bo, Xin Lin, Liang He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.18432
• Pdf URL: https://arxiv.org/pdf/2503.18432
• Copy Paste: [[2503.18432]] Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning(https://arxiv.org/abs/2503.18432)
• Keywords: language model, llm
• Abstract: Automatic math correction aims to check students' solutions to mathematical problems via artificial intelligence technologies. Most existing studies focus on judging the final answer at the problem level, while they ignore detailed feedback on each step in a math problem-solving process, which requires abilities of semantic understanding and reasoning. In this paper, we propose a reinforcement learning (RL)-based method to boost large language model (LLM) for step-level automatic math correction, named StepAMC. Particularly, we convert the step-level automatic math correction within the text classification task into an RL problem to enhance the reasoning capabilities of LLMs. Then, we design a space-constrained policy network to improve the stability of RL. Then, we introduce a fine-grained reward network to convert the binary human feedback into a continuous value. We conduct extensive experiments over two benchmark datasets and the results show that our model outperforms the eleven strong baselines.
• 摘要：自动数学校正旨在通过人工智能技术检查学生对数学问题的解决方案。大多数现有的研究都集中在问题水平上的最终答案上，而它们忽略了数学问题解决过程中每个步骤的详细反馈，这需要语义理解和推理的能力。在本文中，我们提出了一种基于强化的方法（RL）的方法来增强大型语言模型（LLM），以进行步骤级自动数学校正，名为STEPAMC。特别是，我们将文本分类任务中的级级自动数学校正转换为RL问题，以增强LLMS的推理能力。然后，我们设计了一个被限制的策略网络，以提高RL的稳定性。然后，我们引入了一个细粒度的奖励网络，将二进制人类反馈转换为连续价值。我们对两个基准数据集进行了广泛的实验，结果表明，我们的模型的表现优于11个强基础。

Title: MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering

• Authors: Shuo Yang, Siwen Luo, Soyeon Caren Han, Eduard Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18491
• Pdf URL: https://arxiv.org/pdf/2503.18491
• Copy Paste: [[2503.18491]] MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering(https://arxiv.org/abs/2503.18491)
• Keywords: language model, prompt
• Abstract: Visual Question Answering (VQA) requires reasoning across visual and textual modalities, yet Large Vision-Language Models (LVLMs) often lack integrated commonsense knowledge, limiting their robustness in real-world scenarios. To address this, we introduce MAGIC-VQA, a novel framework that enhances VQA by systematically integrating commonsense knowledge with LVLMs. MAGIC-VQA employs a three-stage process: (1) Explicit Knowledge Integration from external sources, (2) By-Type Post-Processing for contextual refinement, and (3) Implicit Knowledge Augmentation using a Graph Neural Network (GNN) for structured reasoning. While GNNs bring greater depth to structured inference, they enable superior relational inference beyond LVLMs. MAGIC-VQA bridges a key gap by unifying commonsensse knowledge with LVLM-driven reasoning, eliminating the need for extensive pre-training or complex prompt tuning. Our framework achieves state-of-the-art performance on benchmark datasets, significantly improving commonsense reasoning in VQA.
• 摘要：视觉问题回答（VQA）需要在视觉和文本方式上进行推理，但是大型视觉模型（LVLM）通常缺乏集成的常识知识，从而限制了它们在现实情况下的鲁棒性。为了解决这个问题，我们介绍了Magic-VQA，这是一个新颖的框架，通过系统地将常识性知识与LVLMS整合在一起来增强VQA。 Magic-VQA采用了三个阶段的过程：（1）外部来源的显式知识集成，（2）用于上下文改进的BY-TYPE后处理，以及（3）使用图形神经网络（GNN）进行结构化推理的隐性知识增强。尽管GNN为结构化推断带来了更大的深度，但它们使超越LVLM的相关推理能够。 Magic-VQA通过通过LVLM驱动的推理统一Commonsse知识来弥合关键差距，从而消除了对广泛的预训练或复杂及时调整的需求。我们的框架在基准数据集上实现了最先进的性能，从而大大改善了VQA的常识性推理。

Title: Autoregressive Language Models for Knowledge Base Population: A case study in the space mission domain

• Authors: Andrés García-Silva, José Manuel Gómez-Pérez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18502
• Pdf URL: https://arxiv.org/pdf/2503.18502
• Copy Paste: [[2503.18502]] Autoregressive Language Models for Knowledge Base Population: A case study in the space mission domain(https://arxiv.org/abs/2503.18502)
• Keywords: language model, prompt
• Abstract: Knowledge base population KBP plays a crucial role in populating and maintaining knowledge bases up-to-date in organizations by leveraging domain corpora. Motivated by the increasingly large context windows supported by large language models, we propose to fine-tune an autoregressive language model for end-toend KPB. Our case study involves the population of a space mission knowledge graph. To fine-tune the model we generate a dataset for end-to-end KBP tapping into existing domain resources. Our case study shows that fine-tuned language models of limited size can achieve competitive and even higher accuracy than larger models in the KBP task. Smaller models specialized for KBP offer affordable deployment and lower-cost inference. Moreover, KBP specialist models do not require the ontology to be included in the prompt, allowing for more space in the context for additional input text or output serialization.
• 摘要：知识基础人群KBP通过利用域Corpora来填充和维护组织中最新的知识基础的重要作用。由越来越大的上下文窗口受到大型语言模型支持的启发，我们建议对端到KPB进行自回归语言模型进行微调。我们的案例研究涉及太空任务知识图的人群。要微调模型，我们生成一个数据集，用于端到端的KBP利用现有域资源。我们的案例研究表明，在KBP任务中，与大型模型相比，规模有限的微调语言模型可以实现竞争性甚至更高的准确性。专门用于KBP的较小型号提供负担得起的部署和低成本推理。此外，KBP专家模型不需要在提示中包括本体，从而在上下文中提供了更多空间，以获取其他输入文本或输出序列化。

Title: SciClaims: An End-to-End Generative System for Biomedical Claim Analysis

• Authors: Raúl Ortega, José Manuel Gómez-Pérez
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2503.18526
• Pdf URL: https://arxiv.org/pdf/2503.18526
• Copy Paste: [[2503.18526]] SciClaims: An End-to-End Generative System for Biomedical Claim Analysis(https://arxiv.org/abs/2503.18526)
• Keywords: language model, llm
• Abstract: Validating key claims in scientific literature, particularly in biomedical research, is essential for ensuring accuracy and advancing knowledge. This process is critical in sectors like the pharmaceutical industry, where rapid scientific progress requires automation and deep domain expertise. However, current solutions have significant limitations. They lack end-to-end pipelines encompassing all claim extraction, evidence retrieval, and verification steps; rely on complex NLP and information retrieval pipelines prone to multiple failure points; and often fail to provide clear, user-friendly justifications for claim verification outcomes. To address these challenges, we introduce SciClaims, an advanced system powered by state-of-the-art large language models (LLMs) that seamlessly integrates the entire scientific claim analysis process. SciClaims outperforms previous approaches in both claim extraction and verification without requiring additional fine-tuning, setting a new benchmark for automated scientific claim analysis.
• 摘要：验证科学文献中的关键主张，尤其是在生物医学研究中，对于确保准确性和提高知识至关重要。在制药行业等领域，这一过程至关重要，在制药行业中，快速的科学进步需要自动化和深层领域的专业知识。但是，当前的解决方案有重大局限性。他们缺乏包括所有要求提取，证据检索和验证步骤的端到端管道；依靠复杂的NLP和信息检索管道易于多个故障点；而且通常无法为索赔验证结果提供明确，用户友好的理由。为了应对这些挑战，我们介绍了Scicaighs，这是一个由最先进的大语模型（LLMS）提供支持的先进系统，该系统无缝地集成了整个科学主张分析过程。 Scicaim在主张提取和验证方面的先前方法都优于先前的方法，而无需进行其他微调，为自动化科学索赔分析树立了新的基准。

Title: Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models

• Authors: Nariman Naderi, Seyed Amir Ahmad Safavi-Naini, Thomas Savage, Zahra Atf, Peter Lewis, Girish Nadkarni, Ali Soroush
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.18562
• Pdf URL: https://arxiv.org/pdf/2503.18562
• Copy Paste: [[2503.18562]] Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models(https://arxiv.org/abs/2503.18562)
• Keywords: language model, gpt, llm
• Abstract: This study evaluated self-reported response certainty across several large language models (GPT, Claude, Llama, Phi, Mistral, Gemini, Gemma, and Qwen) using 300 gastroenterology board-style questions. The highest-performing models (GPT-o1 preview, GPT-4o, and Claude-3.5-Sonnet) achieved Brier scores of 0.15-0.2 and AUROC of 0.6. Although newer models demonstrated improved performance, all exhibited a consistent tendency towards overconfidence. Uncertainty estimation presents a significant challenge to the safe use of LLMs in healthcare. Keywords: Large Language Models; Confidence Elicitation; Artificial Intelligence; Gastroenterology; Uncertainty Quantification
• 摘要：这项研究使用300个胃肠病学董事会式的问题评估了几种大型语言模型（GPT，Claude，Llame，Phi，Mistral，Gemini，Gemma和Qwen）的自我报告的响应确定性。表现最高的模型（GPT-O1预览，GPT-4O和Claude-3.5-Sonnet）的Brier得分为0.15-0.2，AUROC为0.6。尽管较新的模型表现出改善的性能，但所有模型都表现出了过度自信的一致倾向。不确定性估计给安全使用LLM在医疗保健中带来了重大挑战。关键字：大语言模型；信心启发；人工智能;胃肠病学；不确定性定量

Title: ClinText-SP and RigoBERTa Clinical: a new set of open resources for Spanish Clinical NLP

• Authors: Guillem García Subies, Álvaro Barbero Jiménez, Paloma Martínez Fernández
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18594
• Pdf URL: https://arxiv.org/pdf/2503.18594
• Copy Paste: [[2503.18594]] ClinText-SP and RigoBERTa Clinical: a new set of open resources for Spanish Clinical NLP(https://arxiv.org/abs/2503.18594)
• Keywords: language model
• Abstract: We present a novel contribution to Spanish clinical natural language processing by introducing the largest publicly available clinical corpus, ClinText-SP, along with a state-of-the-art clinical encoder language model, RigoBERTa Clinical. Our corpus was meticulously curated from diverse open sources, including clinical cases from medical journals and annotated corpora from shared tasks, providing a rich and diverse dataset that was previously difficult to access. RigoBERTa Clinical, developed through domain-adaptive pretraining on this comprehensive dataset, significantly outperforms existing models on multiple clinical NLP benchmarks. By publicly releasing both the dataset and the model, we aim to empower the research community with robust resources that can drive further advancements in clinical NLP and ultimately contribute to improved healthcare applications.
• 摘要：我们通过引入最大的公开临床语料库Clintext-SP以及最先进的临床编码器语言模型Rigoberta Clinical提出了对西班牙临床自然语言处理的新颖贡献。我们的语料库是从各种开源来源中精心策划的，包括医学期刊的临床病例和共同任务的注释语料库，提供了以前难以访问的丰富而多样的数据集。通过在此综合数据集中进行的域自适应预处理开发的Rigoberta临床，在多个临床NLP基准上的现有模型大大优于现有模型。通过公开发布数据集和模型，我们旨在通过强大的资源增强研究界的能力，这些资源可以推动临床NLP的进一步进步，并最终有助于改善医疗保健应用程序。

Title: LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL

• Authors: Yihan Wang, Peiyu Liu, Xin Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18596
• Pdf URL: https://arxiv.org/pdf/2503.18596
• Copy Paste: [[2503.18596]] LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL(https://arxiv.org/abs/2503.18596)
• Keywords: llm, chain-of-thought
• Abstract: Schema linking is a critical bottleneck in achieving human-level performance in Text-to-SQL tasks, particularly in real-world large-scale multi-database scenarios. Addressing schema linking faces two major challenges: (1) Database Retrieval: selecting the correct database from a large schema pool in multi-database settings, while filtering out irrelevant ones. (2) Schema Item Grounding: accurately identifying the relevant tables and columns from within a large and redundant schema for SQL generation. To address this, we introduce LinkAlign, a novel framework that can effectively adapt existing baselines to real-world environments by systematically addressing schema linking. Our framework comprises three key steps: multi-round semantic enhanced retrieval and irrelevant information isolation for Challenge 1, and schema extraction enhancement for Challenge 2. We evaluate our method performance of schema linking on the SPIDER and BIRD benchmarks, and the ability to adapt existing Text-to-SQL models to real-world environments on the SPIDER 2.0-lite benchmark. Experiments show that LinkAlign outperforms existing baselines in multi-database settings, demonstrating its effectiveness and robustness. On the other hand, our method ranks highest among models excluding those using long chain-of-thought reasoning LLMs. This work bridges the gap between current research and real-world scenarios, providing a practical solution for robust and scalable schema linking. The codes are available at this https URL.
• 摘要：模式链接是在文本到SQL任务中实现人类水平的性能的关键瓶颈，尤其是在现实世界中的大规模多数据库方案中。解决架构链接面临两个主要挑战：（1）数据库检索：从多数据库设置中的大型模式池中选择正确的数据库，同时滤除无关的数据库。 （2）架构项目接地：准确地识别来自SQL生成的大而冗余模式中的相关表和列。为了解决这个问题，我们介绍了LinkAlign，这是一个新颖的框架，可以通过系统地解决架构链接来有效地使现有基线适应现实环境。我们的框架包括三个关键步骤：挑战1的多轮语义增强的检索和无关紧要的信息隔离以及挑战2的架构提取增强。我们评估了我们在蜘蛛和鸟基准上链接的模式的方法，以及在现有的文本对SQL模型中的蜘蛛和鸟类基准测试的能力。实验表明，LinkAlign在多数据库设置中优于现有基线，表明其有效性和鲁棒性。另一方面，我们的方法在不包括使用长链推理LLM的模型中排名最高。这项工作弥合了当前的研究和实际情况之间的差距，为可靠且可扩展的模式链接提供了实用的解决方案。这些代码可在此HTTPS URL上找到。

Title: LANGALIGN: Enhancing Non-English Language Models via Cross-Lingual Embedding Alignment

• Authors: Jong Myoung Kim, Young-Jun Lee, Ho-Jin Choi, Sangkeun Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18603
• Pdf URL: https://arxiv.org/pdf/2503.18603
• Copy Paste: [[2503.18603]] LANGALIGN: Enhancing Non-English Language Models via Cross-Lingual Embedding Alignment(https://arxiv.org/abs/2503.18603)
• Keywords: language model
• Abstract: While Large Language Models have gained attention, many service developers still rely on embedding-based models due to practical constraints. In such cases, the quality of fine-tuning data directly impacts performance, and English datasets are often used as seed data for training non-English models. In this study, we propose LANGALIGN, which enhances target language processing by aligning English embedding vectors with those of the target language at the interface between the language model and the task header. Experiments on Korean, Japanese, and Chinese demonstrate that LANGALIGN significantly improves performance across all three languages. Additionally, we show that LANGALIGN can be applied in reverse to convert target language data into a format that an English-based model can process.
• 摘要：尽管大型语言模型引起了人们的关注，但由于实际限制，许多服务开发人员仍然依靠基于嵌入的模型。在这种情况下，微调数据的质量直接影响性能，并且英语数据集通常用作培训非英语模型的种子数据。在这项研究中，我们提出了Langalign，该研究通过使英语嵌入向量与语言模型和任务标头之间接口的目标语言的向量保持一致来增强目标语言处理。对韩语，日语和中国的实验表明，Langalign显着提高了所有三种语言的性能。此外，我们表明可以反向应用Langalign将目标语言数据转换为基于英语的模型可以处理的格式。

Title: ZeroLM: Data-Free Transformer Architecture Search for Language Models

• Authors: Zhen-Song Chen, Hong-Wei Ding, Xian-Jia Wang, Witold Pedrycz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18646
• Pdf URL: https://arxiv.org/pdf/2503.18646
• Copy Paste: [[2503.18646]] ZeroLM: Data-Free Transformer Architecture Search for Language Models(https://arxiv.org/abs/2503.18646)
• Keywords: language model
• Abstract: Neural architecture search (NAS) provides a systematic framework for automating the design of neural network architectures, yet its widespread adoption is hindered by prohibitive computational requirements. Existing zero-cost proxy methods, while reducing search overhead, demonstrate inadequate performance in architecture ranking tasks, particularly for Transformer-based models where they often underperform simple parameter counting metrics. Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity. This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics computation while decomposing Transformer architectures into functionally distinct sub-modules, thereby optimizing the balance of their contributions to overall performance. Our comprehensive evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark. The proposed method exhibits exceptional computational efficiency while maintaining robust performance across diverse NAS benchmark tasks, offering a practical solution for large-scale architecture search.
• 摘要：神经体系结构搜索（NAS）提供了一个系统的框架，用于自动化神经网络体系结构的设计，但其广泛采用受到过度计算要求的阻碍。现有的零成本代理方法在减少搜索开销的同时，表明了架构排名任务的性能不足，尤其是对于基于变形金刚的模型，它们通常不足以表现简单的参数计数指标。当前的自动代理发现方法具有延长的搜索时间，对数据过度拟合的敏感性以及结构上的复杂性。本文介绍了一种新颖的零成本代理方法，该方法通过有效的权重统计计算来量化模型容量，同时将变压器体系结构分解为功能上不同的子模块，从而优化了它们对整体性能的贡献的平衡。我们的全面评估表明了这种方法的优势，在Flexibert基准上，Spearman的Rho为0.76，Kendall的Tau为0.53。提出的方法具有出色的计算效率，同时保持了不同NAS基准任务的稳健性能，为大规模架构搜索提供了实用的解决方案。

Title: Commander-GPT: Fully Unleashing the Sarcasm Detection Capability of Multi-Modal Large Language Models

• Authors: Yazhou Zhang, Chunwang Zou, Bo Wang, Jing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18681
• Pdf URL: https://arxiv.org/pdf/2503.18681
• Copy Paste: [[2503.18681]] Commander-GPT: Fully Unleashing the Sarcasm Detection Capability of Multi-Modal Large Language Models(https://arxiv.org/abs/2503.18681)
• Keywords: language model, gpt, llm, prompt
• Abstract: Sarcasm detection, as a crucial research direction in the field of Natural Language Processing (NLP), has attracted widespread attention. Traditional sarcasm detection tasks have typically focused on single-modal approaches (e.g., text), but due to the implicit and subtle nature of sarcasm, such methods often fail to yield satisfactory results. In recent years, researchers have shifted the focus of sarcasm detection to multi-modal approaches. However, effectively leveraging multi-modal information to accurately identify sarcastic content remains a challenge that warrants further exploration. Leveraging the powerful integrated processing capabilities of Multi-Modal Large Language Models (MLLMs) for various information sources, we propose an innovative multi-modal Commander-GPT framework. Inspired by military strategy, we first decompose the sarcasm detection task into six distinct sub-tasks. A central commander (decision-maker) then assigns the best-suited large language model to address each specific sub-task. Ultimately, the detection results from each model are aggregated to identify sarcasm. We conducted extensive experiments on MMSD and MMSD 2.0, utilizing four multi-modal large language models and six prompting strategies. Our experiments demonstrate that our approach achieves state-of-the-art performance, with a 19.3% improvement in F1 score, without necessitating fine-tuning or ground-truth rationales.
• 摘要：讽刺检测是自然语言处理（NLP）领域的关键研究方向，引起了广泛的关注。传统的讽刺检测任务通常集中在单模式方法上（例如文本），但由于讽刺的隐式和微妙的性质，这种方法通常无法产生令人满意的结果。近年来，研究人员将讽刺检测的重点转移到了多模式方法上。但是，有效利用多模式信息准确识别讽刺内容仍然是一个挑战，需要进一步探索。利用多模式大型语言模型（MLLM）的强大集成处理能力为各种信息源，我们提出了一个创新的多模式指挥官-GPT框架。受军事战略的启发，我们首先将讽刺检测任务分解为六个不同的子任务。然后，中央指挥官（决策者）分配了最合适的大语言模型，以解决每个特定的子任务。最终，将每个模型的检测结果聚合以识别讽刺。我们利用了四种多模式大型语言模型和六个提示策略进行了有关MMSD和MMSD 2.0的广泛实验。我们的实验表明，我们的方法实现了最先进的表现，而F1得分提高了19.3％，而无需进行微调或基本真相。

Title: Unsupervised Acquisition of Discrete Grammatical Categories

• Authors: David Ph. Shakouri, Crit Cremers, Niels O. Schiller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18702
• Pdf URL: https://arxiv.org/pdf/2503.18702
• Copy Paste: [[2503.18702]] Unsupervised Acquisition of Discrete Grammatical Categories(https://arxiv.org/abs/2503.18702)
• Keywords: language model, agent
• Abstract: This article presents experiments performed using a computational laboratory environment for language acquisition experiments. It implements a multi-agent system consisting of two agents: an adult language model and a daughter language model that aims to learn the mother language. Crucially, the daughter agent does not have access to the internal knowledge of the mother language model but only to the language exemplars the mother agent generates. These experiments illustrate how this system can be used to acquire abstract grammatical knowledge. We demonstrate how statistical analyses of patterns in the input data corresponding to grammatical categories yield discrete grammatical rules. These rules are subsequently added to the grammatical knowledge of the daughter language model. To this end, hierarchical agglomerative cluster analysis was applied to the utterances consecutively generated by the mother language model. It is argued that this procedure can be used to acquire structures resembling grammatical categories proposed by linguists for natural languages. Thus, it is established that non-trivial grammatical knowledge has been acquired. Moreover, the parameter configuration of this computational laboratory environment determined using training data generated by the mother language model is validated in a second experiment with a test set similarly resulting in the acquisition of non-trivial categories.
• 摘要：本文介绍了使用计算实验室环境进行语言获取实验进行的实验。它实现了由两个代理组成的多机构系统：一个成人语言模型和旨在学习母语的女儿语言模型。至关重要的是，女儿代理人无法访问母语模型的内部知识，而只能获得母语代理人产生的语言典范。这些实验说明了如何使用该系统来获取抽象的语法知识。我们演示了对应于语法类别的输入数据中模式的统计分析如何产生离散的语法规则。随后将这些规则添加到女儿语言模型的语法知识中。为此，将层次结构聚类分析应用于母语模型连续产生的话语。有人认为，该过程可用于获取类似于语言学家为自然语言提出的语法类别的结构。因此，已经确定已经获得了非平凡的语法知识。此外，在第二个实验中验证了使用母语模型生成的训练数据确定的计算实验室环境的参数配置，其测试集类似地导致了非平凡类别的获取。

Title: Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving

• Authors: Hongkuan Zhou, Stefan Schmid, Yicong Li, Lavdim Halilaj, Xiangtong Yao, Wei cao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18730
• Pdf URL: https://arxiv.org/pdf/2503.18730
• Copy Paste: [[2503.18730]] Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving(https://arxiv.org/abs/2503.18730)
• Keywords: language model
• Abstract: The autonomous driving field has seen remarkable advancements in various topics, such as object recognition, trajectory prediction, and motion planning. However, current approaches face limitations in effectively comprehending the complex evolutions of driving scenes over time. This paper proposes FM4SU, a novel methodology for training a symbolic foundation model (FM) for scene understanding in autonomous driving. It leverages knowledge graphs (KGs) to capture sensory observation along with domain knowledge such as road topology, traffic rules, or complex interactions between traffic participants. A bird's eye view (BEV) symbolic representation is extracted from the KG for each driving scene, including the spatio-temporal information among the objects across the scenes. The BEV representation is serialized into a sequence of tokens and given to pre-trained language models (PLMs) for learning an inherent understanding of the co-occurrence among driving scene elements and generating predictions on the next scenes. We conducted a number of experiments using the nuScenes dataset and KG in various scenarios. The results demonstrate that fine-tuned models achieve significantly higher accuracy in all tasks. The fine-tuned T5 model achieved a next scene prediction accuracy of 86.7%. This paper concludes that FM4SU offers a promising foundation for developing more comprehensive models for scene understanding in autonomous driving.
• 摘要：自主驾驶领域在各种主题中取得了显着进步，例如对象识别，轨迹预测和运动计划。但是，当前的方法在有效地理解随着时间的推移驾驶场景的复杂发展时面临局限性。本文提出了FM4SU，这是一种培训符号基础模型（FM）的新方法，用于自主驾驶中的场景理解。它利用知识图（KGS）来捕获感官观察以及域知识，例如道路拓扑，交通规则或交通参与者之间的复杂互动。每个驾驶场景中都从kg中提取了鸟类视图（BEV）符号表示，包括场景中对象之间的时空信息。 BEV表示形式被序列化为一系列令牌，并给予了预先训练的语言模型（PLM），以学习对驾驶场景元素之间的共发生的固有理解，并在下一个场景上产生预测。在各种情况下，我们使用Nuscenes数据集和KG进行了许多实验。结果表明，微调模型在所有任务中都具有明显更高的精度。微型T5模型的下一个场景预测准确性为86.7％。本文得出的结论是，FM4SU为在自动驾驶中开发更全面的现场理解模型提供了有希望的基础。

Title: Construction Identification and Disambiguation Using BERT: A Case Study of NPN

• Authors: Wesley Scivetti, Nathan Schneider
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18751
• Pdf URL: https://arxiv.org/pdf/2503.18751
• Copy Paste: [[2503.18751]] Construction Identification and Disambiguation Using BERT: A Case Study of NPN(https://arxiv.org/abs/2503.18751)
• Keywords: language model
• Abstract: Construction Grammar hypothesizes that knowledge of a language consists chiefly of knowledge of form-meaning pairs (''constructions'') that include vocabulary, general grammar rules, and even idiosyncratic patterns. Recent work has shown that transformer language models represent at least some constructional patterns, including ones where the construction is rare overall. In this work, we probe BERT's representation of the form and meaning of a minor construction of English, the NPN (noun-preposition-noun) construction -- exhibited in such expressions as face to face and day to day -- which is known to be polysemous. We construct a benchmark dataset of semantically annotated corpus instances (including distractors that superficially resemble the construction). With this dataset, we train and evaluate probing classifiers. They achieve decent discrimination of the construction from distractors, as well as sense disambiguation among true instances of the construction, revealing that BERT embeddings carry indications of the construction's semantics. Moreover, artificially permuting the word order of true construction instances causes them to be rejected, indicating sensitivity to matters of form. We conclude that BERT does latently encode at least some knowledge of the NPN construction going beyond a surface syntactic pattern and lexical cues.
• 摘要：构建语法假设一种语言的知识主要由形式上的知识（''构造'）组成，这些知识包括词汇，一般语法规则，甚至是特殊的模式。最近的工作表明，变压器语言模型至少代表了一些构造模式，包括构造总体罕见的模型。在这项工作中，我们探究了伯特（Bert）对英语的形式和含义的代表，即NPN（名词preposition-noun）的结构 - 在面对面和日常的表达中表现出 - 已知是多义的。我们构建了语义注释语料库实例的基准数据集（包括表面上类似于结构的干扰器）。使用此数据集，我们培训和评估探测分类器。他们从干扰因素上实现了构造的体面歧视，并在建筑的真实实例之间进行了理解的歧义，这表明伯特的嵌入式具有构造语义的迹象。此外，人为地将真实施工实例的单词顺序排列导致它们被拒绝，表明对形式问题的敏感性。我们得出的结论是，伯特至少对NPN构建超出了表面句法模式和词汇提示至少进行了一些知识。

Title: Synthetic Function Demonstrations Improve Generation in Low-Resource Programming Languages

• Authors: Nick McKenna, Xinnuo Xu, Jack Williams, Nick Wilson, Benjamin Van Durme, Christian Poelitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18760
• Pdf URL: https://arxiv.org/pdf/2503.18760
• Copy Paste: [[2503.18760]] Synthetic Function Demonstrations Improve Generation in Low-Resource Programming Languages(https://arxiv.org/abs/2503.18760)
• Keywords: llm
• Abstract: A key consideration when training an LLM is whether the target language is more or less resourced, whether this is English compared to Welsh, or Python compared to Excel. Typical training data for programming languages consist of real program demonstrations coupled with human-written comments. Here we present novel approaches to the creation of such data for low resource programming languages. We generate fully-synthetic, textbook-quality demonstrations of common library functions in an example domain of Excel formulas, using a teacher model. We then finetune an underperforming student model, and show improvement on 2 question-answering datasets recast into the Excel domain. We show advantages of finetuning over standard, off-the-shelf RAG approaches, which can offer only modest improvement due to the unfamiliar target domain.
• 摘要：训练LLM时的一个关键考虑是，与Welsh相比，与Excel相比，目标语言是否或多或少是英语的资源。编程语言的典型培训数据包括真实的程序演示以及人类写的评论。在这里，我们介绍了为低资源编程语言创建此类数据的新颖方法。我们使用教师模型在Excel公式的示例领域中生成了完全合成的教科书质量演示。然后，我们为表现不佳的学生模型而言，并在2个问题的数据集对Excel域进行了改进。我们显示了固定措施的优势，而不是标准的现成的抹布方法，由于陌生的目标域，这只能提供适度的改进。

Title: AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoning

• Authors: Alan Dao (Gia Tuan Dao), Dinh Bach Vu, Bui Quang Huy
• Subjects: cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.18769
• Pdf URL: https://arxiv.org/pdf/2503.18769
• Copy Paste: [[2503.18769]] AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoning(https://arxiv.org/abs/2503.18769)
• Keywords: language model, gpt, llm
• Abstract: This paper presents AlphaSpace, a novel methodology designed to enhance the spatial reasoning capabilities of large language models (LLMs) for 3D Cartesian space navigation. AlphaSpace employs a semantics-based tokenization strategy, encoding height information through specialized semantic tokens, and integrates primarily symbolic synthetic reasoning data. This approach enables LLMs to accurately manipulate objects by positioning them at specific [x, y, z] coordinates. Experimental results demonstrate that AlphaSpace significantly outperforms existing models on manipulation subtasks, achieving a total accuracy of 66.67%, compared to 37.5% for GPT-4o and 29.17% for Claude 3.5 Sonnet.
• 摘要：本文介绍了Alphaspace，这是一种新颖的方法，旨在增强3D笛卡尔航空导航的大语言模型（LLMS）的空间推理能力。 Alphaspace采用基于语义的令牌化策略，通过专门的语义令牌编码高度信息，并主要整合符号合成推理数据。这种方法使LLM可以通过将它们定位在特定的[X，Y，Z]坐标中来准确操纵对象。实验结果表明，Alphaspace在操纵子任务上的现有模型明显胜过现有的模型，达到66.67％的总准确度，而GPT-4O的总准确度为37.5％，Claude 3.5 SONNET的总准确度为37.5％。

Title: I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders

• Authors: Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y. Rogov, Elena Tutubalina, Ivan Oseledets
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.18878
• Pdf URL: https://arxiv.org/pdf/2503.18878
• Copy Paste: [[2503.18878]] I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders(https://arxiv.org/abs/2503.18878)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have achieved remarkable success in natural language processing. Recent advances have led to the developing of a new class of reasoning LLMs; for example, open-source DeepSeek-R1 has achieved state-of-the-art performance by integrating deep thinking and complex reasoning. Despite these impressive capabilities, the internal reasoning mechanisms of such models remain unexplored. In this work, we employ Sparse Autoencoders (SAEs), a method to learn a sparse decomposition of latent representations of a neural network into interpretable features, to identify features that drive reasoning in the DeepSeek-R1 series of models. First, we propose an approach to extract candidate ''reasoning features'' from SAE representations. We validate these features through empirical analysis and interpretability methods, demonstrating their direct correlation with the model's reasoning abilities. Crucially, we demonstrate that steering these features systematically enhances reasoning performance, offering the first mechanistic account of reasoning in LLMs. Code available at this https URL
• 摘要：大型语言模型（LLM）在自然语言处理中取得了巨大的成功。最近的进步导致了新的推理LLM的发展。例如，开源DeepSeek-R1通过整合深思熟虑和复杂的推理来实现最先进的表现。尽管有这些令人印象深刻的功能，但这种模型的内部推理机制仍未得到探索。在这项工作中，我们采用了稀疏的自动编码器（SAE），这是一种学习神经网络潜在表示稀疏分解为可解释功能的方法，以识别在DeepSeek-R1系列模型中推动推理的功能。首先，我们提出了一种从SAE表示中提取候选人“推理功能”的方法。我们通过经验分析和可解释性方法来验证这些特征，证明它们与模型的推理能力的直接相关性。至关重要的是，我们证明了转向这些功能有系统地提高推理性能，从而提供了LLMS中推理的第一个机理说明。此https URL可用代码

Title: AgentDropout: Dynamic Agent Elimination for Token-Efficient and High-Performance LLM-Based Multi-Agent Collaboration

• Authors: Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.18891
• Pdf URL: https://arxiv.org/pdf/2503.18891
• Copy Paste: [[2503.18891]] AgentDropout: Dynamic Agent Elimination for Token-Efficient and High-Performance LLM-Based Multi-Agent Collaboration(https://arxiv.org/abs/2503.18891)
• Keywords: language model, llm, prompt, agent
• Abstract: Multi-agent systems (MAS) based on large language models (LLMs) have demonstrated significant potential in collaborative problem-solving. However, they still face substantial challenges of low communication efficiency and suboptimal task performance, making the careful design of the agents' communication topologies particularly important. Inspired by the management theory that roles in an efficient team are often dynamically adjusted, we propose AgentDropout, which identifies redundant agents and communication across different communication rounds by optimizing the adjacency matrices of the communication graphs and eliminates them to enhance both token efficiency and task performance. Compared to state-of-the-art methods, AgentDropout achieves an average reduction of 21.6% in prompt token consumption and 18.4% in completion token consumption, along with a performance improvement of 1.14 on the tasks. Furthermore, the extended experiments demonstrate that AgentDropout achieves notable domain transferability and structure robustness, revealing its reliability and effectiveness. We release our code at this https URL.
• 摘要：基于大语言模型（LLM）的多机构系统（MAS）在解决问题解决方面具有巨大的潜力。但是，他们仍然面临低沟通效率和次优的任务绩效的重大挑战，这使得对代理商的沟通拓扑的仔细设计尤为重要。受到管理理论的启发，即经常会动态调整在高效团队中的角色，我们提出了代理商，该代理商通过优化通信图的邻接矩阵来识别冗余代理和跨不同通信的通信，并消除它们以提高令牌效率和任务绩效。与最先进的方法相比，AgentDropout在迅速消耗量中的平均降低21.6％，完成令牌消耗量的平均降低为18.4％，并且在任务上的性能提高了1.14。此外，扩展的实验表明，AgentDropout可以实现明显的域转移性和结构鲁棒性，从而揭示了其可靠性和有效性。我们在此HTTPS URL上发布代码。

Title: xKV: Cross-Layer SVD for KV-Cache Compression

• Authors: Chi-Chih Chang, Chien-Yu Lin, Yash Akhauri, Wei-Cheng Lin, Kai-Chiang Wu, Luis Ceze, Mohamed S. Abdelfattah
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.18893
• Pdf URL: https://arxiv.org/pdf/2503.18893
• Copy Paste: [[2503.18893]] xKV: Cross-Layer SVD for KV-Cache Compression(https://arxiv.org/abs/2503.18893)
• Keywords: language model, llm, long context
• Abstract: Large Language Models (LLMs) with long context windows enable powerful applications but come at the cost of high memory consumption to store the Key and Value states (KV-Cache). Recent studies attempted to merge KV-cache from multiple layers into shared representations, yet these approaches either require expensive pretraining or rely on assumptions of high per-token cosine similarity across layers which generally does not hold in practice. We find that the dominant singular vectors are remarkably well-aligned across multiple layers of the KV-Cache. Exploiting this insight, we propose xKV, a simple post-training method that applies Singular Value Decomposition (SVD) on the KV-Cache of grouped layers. xKV consolidates the KV-Cache of multiple layers into a shared low-rank subspace, significantly reducing KV-Cache sizes. Through extensive evaluations on the RULER long-context benchmark with widely-used LLMs (e.g., Llama-3.1 and Qwen2.5), xKV achieves up to 6.8x higher compression rates than state-of-the-art inter-layer technique while improving accuracy by 2.7%. Moreover, xKV is compatible with the emerging Multi-Head Latent Attention (MLA) (e.g., DeepSeek-Coder-V2), yielding a notable 3x compression rates on coding tasks without performance degradation. These results highlight xKV's strong capability and versatility in addressing memory bottlenecks for long-context LLM inference. Our code is publicly available at: this https URL.
• 摘要：具有长上下文Windows的大型语言模型（LLMS）启用了强大的应用程序，但以高内存消耗为代价来存储密钥和价值状态（KV-CACHE）。最近的研究试图将KV-CACHE从多个层合并为共享表示形式，但是这些方法要么需要昂贵的预处理，要么依赖于通常在实践中不存在的层次层次相似性的高度相似性的假设。我们发现，在KV-CACHE的多个层中，主要的奇异向量非常合适。利用这种见解，我们提出了XKV，这是一种简单的训练后方法，该方法在分组层的KV-CACHE上应用了单数值分解（SVD）。 XKV将多层的KV-CACHE合并为共享的低率子空间，可显着降低KV-CACH尺寸。通过广泛使用广泛使用的LLM（例如Llama-3.1和Qwen2.5）的标尺长篇小写基准测试的广泛评估，XKV的压缩速率比最先进的层间技术高达6.8倍，同时将精度提高了2.7％。此外，XKV与新兴的多头潜在注意力（MLA）（例如，DeepSeek-Coder-V2）兼容，在不降低性能的情况下编码任务时产生了明显的3倍压缩率。这些结果突出了XKV在解决长篇小说LLM推理的内存瓶颈方面的强大功能和多功能性。我们的代码可公开可用：此HTTPS URL。