2025-05-05

Title: FinBERT-QA: Financial Question Answering with pre-trained BERT Language Models

• Authors: Bithiah Yuan
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00725
• Pdf URL: https://arxiv.org/pdf/2505.00725
• Copy Paste: [[2505.00725]] FinBERT-QA: Financial Question Answering with pre-trained BERT Language Models(https://arxiv.org/abs/2505.00725)
• Keywords: language model
• Abstract: Motivated by the emerging demand in the financial industry for the automatic analysis of unstructured and structured data at scale, Question Answering (QA) systems can provide lucrative and competitive advantages to companies by facilitating the decision making of financial advisers. Consequently, we propose a novel financial QA system using the transformer-based pre-trained BERT language model to address the limitations of data scarcity and language specificity in the financial domain. Our system focuses on financial non-factoid answer selection, which retrieves a set of passage-level texts and selects the most relevant as the answer. To increase efficiency, we formulate the answer selection task as a re-ranking problem, in which our system consists of an Answer Retriever using BM25, a simple information retrieval approach, to first return a list of candidate answers, and an Answer Re-ranker built with variants of pre-trained BERT language models to re-rank and select the most relevant answers. We investigate various learning, further pre-training, and fine-tuning approaches for BERT. Our experiments suggest that FinBERT-QA, a model built from applying the Transfer and Adapt further fine-tuning and pointwise learning approach, is the most effective, improving the state-of-the-art results of task 2 of the FiQA dataset by 16% on MRR, 17% on NDCG, and 21% on Precision@1.
• 摘要：由于金融业在大规模对非结构化和结构化数据进行自动分析的新兴需求的激励，问答系统（QA）系统可以通过促进财务顾问的决策来为公司提供有利可图的和竞争优势。因此，我们使用基于变压器的预训练的BERT语言模型提出了一种新颖的金融质量检查系统，以解决金融领域中数据稀缺和语言特异性的局限性。我们的系统专注于财务非事实答案选择，该选择检索一组段落级文本，并选择最相关的答案。为了提高效率，我们将答案选择任务作为重新排列问题制定，其中我们的系统由使用BM25（一种简单的信息检索方法）组成，以首先返回候选答案的列表，并将答案重新列入一个带有预先培训的BERT语言模型的变体来重新划分并选择最相关的答案。我们研究了BERT的各种学习，进一步的预训练和微调方法。我们的实验表明，Finbert-QA是通过应用转移并适应进一步的微调和指尖学习方法而构建的模型，是最有效的，将FIQA数据集的任务2的最新结果提高了16％的MRR，NDCG对NDCG的17％，而Precision@1则提高了21％。

Title: A Survey on Large Language Model based Human-Agent Systems

• Authors: Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Yankai Chen, Chunyu Miao, Hoang Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Langzhou He, Yangning Li, Yuwei Cao, Dongyuan Li, Renhe Jiang, Philip S. Yu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00753
• Pdf URL: https://arxiv.org/pdf/2505.00753
• Copy Paste: [[2505.00753]] A Survey on Large Language Model based Human-Agent Systems(https://arxiv.org/abs/2505.00753)
• Keywords: language model, llm, hallucination, agent
• Abstract: Recent advances in large language models (LLMs) have sparked growing interest in building fully autonomous agents. However, fully autonomous LLM-based agents still face significant challenges, including limited reliability due to hallucinations, difficulty in handling complex tasks, and substantial safety and ethical risks, all of which limit their feasibility and trustworthiness in real-world applications. To overcome these limitations, LLM-based human-agent systems (LLM-HAS) incorporate human-provided information, feedback, or control into the agent system to enhance system performance, reliability and safety. This paper provides the first comprehensive and structured survey of LLM-HAS. It clarifies fundamental concepts, systematically presents core components shaping these systems, including environment & profiling, human feedback, interaction types, orchestration and communication, explores emerging applications, and discusses unique challenges and opportunities. By consolidating current knowledge and offering a structured overview, we aim to foster further research and innovation in this rapidly evolving interdisciplinary field. Paper lists and resources are available at this https URL.
• 摘要：大型语言模型（LLM）的最新进展激发了人们对建立完全自主的代理商的兴趣。但是，完全自主的LLM代理仍然面临重大挑战，包括由于幻觉而导致的可靠性有限，难以处理复杂的任务以及实质性的安全性和道德风险，所有这些都限制了他们在现实应用程序中的可行性和可信度。为了克服这些局限性，基于LLM的人类代理系统（LLM-HAS）将人提供的信息，反馈或控制纳入代理系统中，以增强系统性能，可靠性和安全性。本文提供了LLM-HAS的首次全面且结构化的调查。它阐明了基本概念，系统地呈现了构成这些系统的核心组件，包括环境和分析，人类反馈，互动类型，编排和交流，探索新兴应用，并讨论独特的挑战和机会。通过巩固当前的知识并提供结构化的概述，我们旨在在这个快速发展的跨学科领域中促进进一步的研究和创新。纸张清单和资源可在此HTTPS URL上找到。

Title: Reasoning Capabilities and Invariability of Large Language Models

• Authors: Alessandro Raganato, Rafael Peñaloza, Marco Viviani, Gabriella Pasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00776
• Pdf URL: https://arxiv.org/pdf/2505.00776
• Copy Paste: [[2505.00776]] Reasoning Capabilities and Invariability of Large Language Models(https://arxiv.org/abs/2505.00776)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities in manipulating natural language across multiple applications, but their ability to handle simple reasoning tasks is often questioned. In this work, we aim to provide a comprehensive analysis of LLMs' reasoning competence, specifically focusing on their prompt dependency. In particular, we introduce a new benchmark dataset with a series of simple reasoning questions demanding shallow logical reasoning. Aligned with cognitive psychology standards, the questions are confined to a basic domain revolving around geometric figures, ensuring that responses are independent of any pre-existing intuition about the world and rely solely on deduction. An empirical analysis involving zero-shot and few-shot prompting across 24 LLMs of different sizes reveals that, while LLMs with over 70 billion parameters perform better in the zero-shot setting, there is still a large room for improvement. An additional test with chain-of-thought prompting over 22 LLMs shows that this additional prompt can aid or damage the performance of models, depending on whether the rationale is required before or after the answer.
• 摘要：大型语言模型（LLM）在多种应用程序中操纵自然语言方面表现出了显着的功能，但是他们处理简单推理任务的能力经常受到质疑。在这项工作中，我们旨在对LLMS的推理能力进行全面分析，特别是专注于他们的及时依赖性。特别是，我们介绍了一个新的基准数据集，其中包含一系列简单的推理问题，要求浅逻辑推理。这些问题与认知心理学标准保持一致，仅限于围绕几何形象的基本领域，确保响应独立于对世界的任何预先存在的直觉，而仅依靠推论。一项涉及零射击和几乎没有射击的经验分析在24个不同尺寸的LLM上均表明，尽管有超过700亿个参数的LLM在零射击设置中的表现更好，但仍有大量改进的空间。一项额外的测试促使经过22个LLM超过22个LLM，这表明该额外的提示可以帮助或损害模型的性能，具体取决于答案之前还是之后是否需要基本原理。

Title: Knowledge-augmented Pre-trained Language Models for Biomedical Relation Extraction

• Authors: Mario Sänger, Ulf Leser
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00814
• Pdf URL: https://arxiv.org/pdf/2505.00814
• Copy Paste: [[2505.00814]] Knowledge-augmented Pre-trained Language Models for Biomedical Relation Extraction(https://arxiv.org/abs/2505.00814)
• Keywords: language model
• Abstract: Automatic relationship extraction (RE) from biomedical literature is critical for managing the vast amount of scientific knowledge produced each year. In recent years, utilizing pre-trained language models (PLMs) has become the prevalent approach in RE. Several studies report improved performance when incorporating additional context information while fine-tuning PLMs for RE. However, variations in the PLMs applied, the databases used for augmentation, hyper-parameter optimization, and evaluation methods complicate direct comparisons between studies and raise questions about the generalizability of these findings. Our study addresses this research gap by evaluating PLMs enhanced with contextual information on five datasets spanning four relation scenarios within a consistent evaluation framework. We evaluate three baseline PLMs and first conduct extensive hyperparameter optimization. After selecting the top-performing model, we enhance it with additional data, including textual entity descriptions, relational information from knowledge graphs, and molecular structure encodings. Our findings illustrate the importance of i) the choice of the underlying language model and ii) a comprehensive hyperparameter optimization for achieving strong extraction performance. Although inclusion of context information yield only minor overall improvements, an ablation study reveals substantial benefits for smaller PLMs when such external data was included during fine-tuning.
• 摘要：生物医学文献的自动关系提取（RE）对于管理每年生产的大量科学知识至关重要。近年来，利用预训练的语言模型（PLM）已成为RE的普遍方法。几项研究报告说，在纳入其他上下文信息时，在为RE进行微调PLM时提高了性能。但是，应用PLM的变化​​，用于增强，超参数优化和评估方法的数据库使研究之间的直接比较变得复杂，并提出了有关这些发现的普遍性的问题。我们的研究通过评估PLM在跨越一致的评估框架内的五个数据集上的上下文信息来评估PLM来解决这一研究差距。我们评估了三个基线PLM，并首先进行了广泛的高参数优化。选择最佳模型后，我们使用其他数据来对其进行增强，包括文本实体描述，知识图中的关系信息和分子结构编码。我们的发现说明了i）基础语言模型的选择和ii）综合的超参数优化，以实现强大的提取性能。尽管包含上下文信息仅会产生较小的总体改进，但消融研究揭示了在微调过程中包含此类外部数据时，较小的PLM的可观好处。

Title: Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing

• Authors: Timur Jaganov, John Blake, Julián Villegas, Nicholas Carr
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00931
• Pdf URL: https://arxiv.org/pdf/2505.00931
• Copy Paste: [[2505.00931]] Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing(https://arxiv.org/abs/2505.00931)
• Keywords: language model, gpt, llm, chat
• Abstract: This study investigates the potential for Large Language Models (LLMs) to scale-up Dynamic Assessment (DA). To facilitate such an investigation, we first developed DynaWrite-a modular, microservices-based grammatical tutoring application which supports multiple LLMs to generate dynamic feedback to learners of English. Initial testing of 21 LLMs, revealed GPT-4o and neural chat to have the most potential to scale-up DA in the language learning classroom. Further testing of these two candidates found both models performed similarly in their ability to accurately identify grammatical errors in user sentences. However, GPT-4o consistently outperformed neural chat in the quality of its DA by generating clear, consistent, and progressively explicit hints. Real-time responsiveness and system stability were also confirmed through detailed performance testing, with GPT-4o exhibiting sufficient speed and stability. This study shows that LLMs can be used to scale-up dynamic assessment and thus enable dynamic assessment to be delivered to larger groups than possible in traditional teacher-learner settings.
• 摘要：这项研究调查了大语模型（LLMS）扩展动态评估（DA）的潜力。为了促进这种调查，我们首先开发了Dynawrite-A基于微服务的语法辅导应用程序，该应用支持多个LLM，以对英语学习者产生动态反馈。 21 LLM的初步测试揭示了GPT-4O和神经聊天，具有在语言学习教室中扩大DA的最大潜力。对这两个候选人的进一步测试发现，这两种模型在准确识别用户句子中的语法错误的能力方面都相似。但是，GPT-4O通过产生清晰，一致和逐步明确的提示，在其DA的质量方面始终超过其DA质量的神经聊天。还通过详细的性能测试确认了实时响应能力和系统稳定性，GPT-4O表现出足够的速度和稳定性。这项研究表明，LLM可用于扩展动态评估，从而使动态评估能够交付给传统教师学习者中的大组。

Title: Llama-Nemotron: Efficient Reasoning Models

• Authors: Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00949
• Pdf URL: https://arxiv.org/pdf/2505.00949
• Copy Paste: [[2505.00949]] Llama-Nemotron: Efficient Reasoning Models(https://arxiv.org/abs/2505.00949)
• Keywords: chat
• Abstract: We introduce the Llama-Nemotron series of models, an open family of heterogeneous reasoning models that deliver exceptional reasoning capabilities, inference efficiency, and an open license for enterprise use. The family comes in three sizes -- Nano (8B), Super (49B), and Ultra (253B) -- and performs competitively with state-of-the-art reasoning models such as DeepSeek-R1 while offering superior inference throughput and memory efficiency. In this report, we discuss the training procedure for these models, which entails using neural architecture search from Llama 3 models for accelerated inference, knowledge distillation, and continued pretraining, followed by a reasoning-focused post-training stage consisting of two main parts: supervised fine-tuning and large scale reinforcement learning. Llama-Nemotron models are the first open-source models to support a dynamic reasoning toggle, allowing users to switch between standard chat and reasoning modes during inference. To further support open research and facilitate model development, we provide the following resources: 1. We release the Llama-Nemotron reasoning models -- LN-Nano, LN-Super, and LN-Ultra -- under the commercially permissive NVIDIA Open Model License Agreement. 2. We release the complete post-training dataset: Llama-Nemotron-Post-Training-Dataset. 3. We also release our training codebases: NeMo, NeMo-Aligner, and Megatron-LM.
• 摘要：我们介绍了Llama-Nemotron系列模型，这是一个开放的异构推理模型家族，可提供出色的推理能力，推理效率和企业使用的开放许可。这个家庭有三种尺寸 - 纳米（8b），Super（49b）和Ultra（253b） - 并与最先进的推理模型（例如DeepSeek-R1）一起竞争，同时提供了出色的推理吞吐量和记忆效率。在本报告中，我们讨论了这些模型的培训程序，这些模型需要使用Llama 3模型的神经体系结构搜索进行加速推理，知识蒸馏和持续预处理，然后进行以推理为中心的训练后阶段，由两个主要部分组成：受监管的精细调整和大规模增强学习。 Llama-Nemotron模型是支持动态推理切换的第一个开源模型，从而使用户可以在推理过程中在标准聊天和推理模式之间进行切换。为了进一步支持开放研究并促进模型开发，我们提供以下资源：1。我们发布了商业上允许的Nvidia Open Model许可协议。 2。我们发布完整的训练后数据集：Llama-Nemotron-Post-Training-Dataset。 3。我们还发布了培训代码库：Nemo，Nemo-Aligner和Megatron-LM。

Title: Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models

• Authors: Xuhui Jiang, Shengjie Ma, Chengjin Xu, Cehao Yang, Liyu Zhang, Jian Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00979
• Pdf URL: https://arxiv.org/pdf/2505.00979
• Copy Paste: [[2505.00979]] Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models(https://arxiv.org/abs/2505.00979)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large Language Models (LLMs) have achieved remarkable success but remain data-inefficient, especially when learning from small, specialized corpora with limited and proprietary data. Existing synthetic data generation methods for continue pre-training focus on intra-document content and overlook cross-document knowledge associations, limiting content diversity and depth. We propose Synthetic-on-Graph (SoG), a synthetic data generation framework that incorporates cross-document knowledge associations for efficient corpus expansion. SoG constructs a context graph by extracting entities and concepts from the original corpus, representing cross-document associations, and employing a graph walk strategy for knowledge-associated sampling. This enhances synthetic data diversity and coherence, enabling models to learn complex knowledge structures and handle rare knowledge. To further improve synthetic data quality, we integrate Chain-of-Thought (CoT) and Contrastive Clarifying (CC) synthetic, enhancing reasoning processes and discriminative power. Experiments show that SoG outperforms the state-of-the-art (SOTA) method in a multi-hop document Q&A dataset while performing comparably to the SOTA method on the reading comprehension task datasets, which also underscores the better generalization capability of SoG. Our work advances synthetic data generation and provides practical solutions for efficient knowledge acquisition in LLMs, especially in domains with limited data availability.
• 摘要：大型语言模型（LLM）取得了显着的成功，但仍保持数据智能，尤其是在从具有有限和专有数据的小型专业语料库中学习时。现有的合成数据生成方法继续预训练专注于文档内内容和忽略跨文档知识关联，限制了内容多样性和深度。我们提出了合成式（SOG）的合成数据生成框架（SOG），该框架结合了跨文档知识关联以进行有效的语料库扩展。 SOG通过从原始语料库中提取实体和概念来构建上下文图，代表跨文档关联，并采用图形步行策略进行知识相关采样。这增强了综合数据的多样性和连贯性，使模型能够学习复杂的知识结构并处理罕见知识。为了进一步提高合成数据质量，我们整合了经过思考链（COT）和对比度澄清（CC）合成，增强了推理过程和判别能力。实验表明，SOG在多跳文档的问答数据集中优于最先进的方法（SOTA）方法，同时在阅读理解任务数据集中对SOTA方法进行了相当的作用，这也强调了SOG的更好的概括能力。我们的工作促进了合成数据生成，并提供了实用的解决方案，以在LLMS中，尤其是在数据可用性有限的域中获得有效的知识获取。

Title: Position: Enough of Scaling LLMs! Lets Focus on Downscaling

• Authors: Ayan Sengupta, Yash Goel, Tanmoy Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00985
• Pdf URL: https://arxiv.org/pdf/2505.00985
• Copy Paste: [[2505.00985]] Position: Enough of Scaling LLMs! Lets Focus on Downscaling(https://arxiv.org/abs/2505.00985)
• Keywords: language model, llm
• Abstract: We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.
• 摘要：我们挑战了对神经缩放法的主要关注，并倡导在大型语言模型（LLMS）开发中朝着缩减范围的范式转变。尽管扩展法律通过增加模型和数据集大小为改进性能提高了批判性见解，但我们强调了这种方法的显着局限性，尤其是在计算效率低下，环境影响和部署限制方面。为了应对这些挑战，我们为降级LLM提出了一个整体框架，该框架试图保持绩效，同时大大减少资源需求。本文概述了从传统规模范式过渡的实用策略，并提倡采用更可持续，高效和易于使用的LLM开发方法。

Title: VTS-LLM: Domain-Adaptive LLM Agent for Enhancing Awareness in Vessel Traffic Services through Natural Language

• Authors: Sijin Sun, Liangbin Zhao, Ming Deng, Xiuju Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00989
• Pdf URL: https://arxiv.org/pdf/2505.00989
• Copy Paste: [[2505.00989]] VTS-LLM: Domain-Adaptive LLM Agent for Enhancing Awareness in Vessel Traffic Services through Natural Language(https://arxiv.org/abs/2505.00989)
• Keywords: llm, agent
• Abstract: Vessel Traffic Services (VTS) are essential for maritime safety and regulatory compliance through real-time traffic management. However, with increasing traffic complexity and the prevalence of heterogeneous, multimodal data, existing VTS systems face limitations in spatiotemporal reasoning and intuitive human interaction. In this work, we propose VTS-LLM Agent, the first domain-adaptive large LLM agent tailored for interactive decision support in VTS operations. We formalize risk-prone vessel identification as a knowledge-augmented Text-to-SQL task, combining structured vessel databases with external maritime knowledge. To support this, we construct a curated benchmark dataset consisting of a custom schema, domain-specific corpus, and a query-SQL test set in multiple linguistic styles. Our framework incorporates NER-based relational reasoning, agent-based domain knowledge injection, semantic algebra intermediate representation, and query rethink mechanisms to enhance domain grounding and context-aware understanding. Experimental results show that VTS-LLM outperforms both general-purpose and SQL-focused baselines under command-style, operational-style, and formal natural language queries, respectively. Moreover, our analysis provides the first empirical evidence that linguistic style variation introduces systematic performance challenges in Text-to-SQL modeling. This work lays the foundation for natural language interfaces in vessel traffic services and opens new opportunities for proactive, LLM-driven maritime real-time traffic management.
• 摘要：通过实时交通管理，船舶交通服务（VTS）对于海上安全和监管合规性至关重要。但是，随着交通复杂性的增加和异质性多模式数据的普遍性，现有的VTS系统面临时空推理和直观人类互动的局限性。在这项工作中，我们提出了VTS-LLM代理，这是针对VTS操作中的交互式决策支持量身定制的第一个域自适应大型LLM代理。我们将易风险的船只识别形式化为知识增强的文本到SQL任务，将结构化的船只数据库与外部海上知识相结合。为了支持这一点，我们构建了一个由自定义模式，域特异性语料库和多种语言样式的查询SQL测试集组成的策划的基准数据集。我们的框架结合了基于NER的关系推理，基于代理的域知识注入，语义代数中间表示以及查询重新思考机制，以增强域接地和背景感知的理解。实验结果表明，VTS-LLM在命令式，操作式和正式的自然语言查询下的通用和以SQL为中心的基线的表现分别胜过。此外，我们的分析提供了第一个经验证据，即语言风格的变化在文本到SQL建模中引入了系统的性能挑战。这项工作奠定了船只交通服务中自然语言界面的基础，并为主动，LLM驱动的海上实时交通管理开辟了新的机会。

Title: Value Portrait: Understanding Values of LLMs with Human-aligned Benchmark

• Authors: Jongwook Han, Dongmin Choi, Woojung Song, Eun-Ju Lee, Yohan Jo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.01015
• Pdf URL: https://arxiv.org/pdf/2505.01015
• Copy Paste: [[2505.01015]] Value Portrait: Understanding Values of LLMs with Human-aligned Benchmark(https://arxiv.org/abs/2505.01015)
• Keywords: language model, llm
• Abstract: The importance of benchmarks for assessing the values of language models has been pronounced due to the growing need of more authentic, human-aligned responses. However, existing benchmarks rely on human or machine annotations that are vulnerable to value-related biases. Furthermore, the tested scenarios often diverge from real-world contexts in which models are commonly used to generate text and express values. To address these issues, we propose the Value Portrait benchmark, a reliable framework for evaluating LLMs' value orientations with two key characteristics. First, the benchmark consists of items that capture real-life user-LLM interactions, enhancing the relevance of assessment results to real-world LLM usage and thus ecological validity. Second, each item is rated by human subjects based on its similarity to their own thoughts, and correlations between these ratings and the subjects' actual value scores are derived. This psychometrically validated approach ensures that items strongly correlated with specific values serve as reliable items for assessing those values. Through evaluating 27 LLMs with our benchmark, we find that these models prioritize Benevolence, Security, and Self-Direction values while placing less emphasis on Tradition, Power, and Achievement values. Also, our analysis reveals biases in how LLMs perceive various demographic groups, deviating from real human data.
• 摘要：由于需求日益增长，人为安排的响应的需求日益增长，基准对评估语言模型值的重要性已被宣布。但是，现有的基准依赖于容易受到价值相关偏见的人类或机器注释。此外，经过测试的方案通常与现实世界中的上下文不同，在这些环境中，模型通常用于生成文本和表达值。为了解决这些问题，我们提出了价值肖像基准，这是一个可靠的框架，用于评估具有两个关键特征的LLMS的价值取向。首先，基准由捕获现实生活中用户-LLM交互的项目组成，增强了评估结果与现实世界LLM使用情况的相关性，从而增强了生态有效性。其次，每个项目都由人类受试者根据其与自己的思想的相似性进行评分，并且这些评分与受试者的实际价值分数之间的相关性。经过心理验证的方法可确保与特定值密切相关的项目作为评估这些值的可靠项目。通过通过我们的基准评估27个LLM，我们发现这些模型优先考虑仁慈，安全和自我指导价值观，同时更少强调传统，权力和成就价值。此外，我们的分析揭示了LLM对各种人口组的感知，与真实人类数据偏离的偏见。

Title: Do We Need a Detailed Rubric for Automated Essay Scoring using Large Language Models?

• Authors: Lui Yoshida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.01035
• Pdf URL: https://arxiv.org/pdf/2505.01035
• Copy Paste: [[2505.01035]] Do We Need a Detailed Rubric for Automated Essay Scoring using Large Language Models?(https://arxiv.org/abs/2505.01035)
• Keywords: language model, gpt, llm
• Abstract: This study investigates the necessity and impact of a detailed rubric in automated essay scoring (AES) using large language models (LLMs). While using rubrics are standard in LLM-based AES, creating detailed rubrics requires substantial ef-fort and increases token usage. We examined how different levels of rubric detail affect scoring accuracy across multiple LLMs using the TOEFL11 dataset. Our experiments compared three conditions: a full rubric, a simplified rubric, and no rubric, using four different LLMs (Claude 3.5 Haiku, Gemini 1.5 Flash, GPT-4o-mini, and Llama 3 70B Instruct). Results showed that three out of four models maintained similar scoring accuracy with the simplified rubric compared to the detailed one, while significantly reducing token usage. However, one model (Gemini 1.5 Flash) showed decreased performance with more detailed rubrics. The findings suggest that simplified rubrics may be sufficient for most LLM-based AES applications, offering a more efficient alternative without compromis-ing scoring accuracy. However, model-specific evaluation remains crucial as per-formance patterns vary across different LLMs.
• 摘要：这项研究调查了使用大语言模型（LLMS）在自动论文评分（AES）中详细标题的必要性和影响。虽然使用标准是基于LLM的AES中的标准配置，但是创建详细的专栏需要大量的EF验证并增加令牌使用情况。我们检查了使用TOEFL11数据集的不同级别的标题细节如何影响多个LLM的评分精度。我们的实验比较了三个条件：使用四种不同的LLM（Claude 3.5 Haiku，Gemini 1.5 Flash，GPT-4O-Mini和Llama 3 70B指示），使用了四个不同的LLM（Claude 3.5 Haiku，Gemini 1.5 Haiku，Gemini 1.5 Haiku，Gemini 1.5 Haiku，Gemini 1.5）。结果表明，与详细的四个模型相比，四个模型中的三个与简化的标语保持相似的评分精度，同时大大降低了令牌使用情况。但是，一种模型（Gemini 1.5闪光灯）显示出更详细的专栏表现降低。研究结果表明，简化的专栏可能足以满足大多数基于LLM的AES应用程序，提供了更有效的替代方案而无需折衷评分精度。但是，由于各种LLM的每个形式模式，模型特定的评估仍然至关重要。

Title: MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learning

• Authors: Murtadha Ahmed, Wenbo, Liu yunfeng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.01110
• Pdf URL: https://arxiv.org/pdf/2505.01110
• Copy Paste: [[2505.01110]] MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learning(https://arxiv.org/abs/2505.01110)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in In-Context Learning (ICL). However, the fixed position length constraints in pre-trained models limit the number of demonstration examples. Recent efforts to extend context suffer from attention dispersion as the number of demonstrations increases. In this paper, we introduce Mitigating Attention Dispersion in large-scale ICL (MateICL) that enables LLMs to maintain effective self-attention as the context size grows. We first split the context into multiple windows, each filled to the model's context capacity, which are processed separately. Then, we introduce an additional layer to recalibrate the attention weights, prioritizing the query tokens as the number of demonstrations increases. Our empirical results show that MateICL can effectively leverage larger contexts to improve ICL performance. Compared to retrieval-based baselines, MateICL consistently achieves better performance without requiring an externally trained retrieval model. Despite recent advances in inference strategies (e.g., 32k token contexts), our results demonstrate that MateICL remains beneficial in computationally resource-constrained settings. The code is publicly available at this https URL.
• 摘要：大型语言模型（LLMS）在文化学习（ICL）中表现出了显着的功能。但是，预训练模型中的固定位置长度约束限制了演示示例的数量。随着示威数量的增加，最近延长情境的努力遭受了注意力分散。在本文中，我们引入了大规模ICL（Mateicl）中的减轻注意力分散，使LLMS能够随着上下文大小的增长而保持有效的自我注意力。我们首先将上下文分为多个窗口，每个窗口都填充到模型的上下文容量，并分别处理。然后，我们引入了一个额外的层来重新校准注意力重量，随着演示次数的增加，优先考虑查询令牌。我们的经验结果表明，Mateicl可以有效利用较大的环境来提高ICL性能。与基于检索的基线相比，Mateicl始终在不需要外部训练的检索模型的情况下实现更好的性能。尽管最近的推理策略进展（例如32K令牌上下文），但我们的结果表明，Mateicl在计算资源约束的设置中仍然有益。该代码在此HTTPS URL上公开可用。

Title: On the Limitations of Steering in Language Model Alignment

• Authors: Chebrolu Niranjan, Kokil Jaidka, Gerard Christopher Yeo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.01162
• Pdf URL: https://arxiv.org/pdf/2505.01162
• Copy Paste: [[2505.01162]] On the Limitations of Steering in Language Model Alignment(https://arxiv.org/abs/2505.01162)
• Keywords: language model, llm, prompt
• Abstract: Steering vectors are a promising approach to aligning language model behavior at inference time. In this paper, we propose a framework to assess the limitations of steering vectors as alignment mechanisms. Using a framework of transformer hook interventions and antonym-based function vectors, we evaluate the role of prompt structure and context complexity in steering effectiveness. Our findings indicate that steering vectors are promising for specific alignment tasks, such as value alignment, but may not provide a robust foundation for general-purpose alignment in LLMs, particularly in complex scenarios. We establish a methodological foundation for future investigations into steering capabilities of reasoning models.
• 摘要：转向向量是一种在推理时间对齐语言模型行为的有前途的方法。在本文中，我们提出了一个框架，以评估转向向量作为对准机制的局限性。使用变压器挂钩干预和基于反义词的功能向量的框架，我们评估了及时结构和上下文复杂性在转向有效性中的作用。我们的发现表明，转向向量对于特定的一致性任务（例如价值对齐）有希望，但可能无法为LLMS中的通用对齐提供强大的基础，尤其是在复杂的情况下。我们为未来对推理模型的转向能力进行研究的方法学基础。

Title: Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods

• Authors: Mahdi Dhaini, Ege Erdogan, Nils Feldhus, Gjergji Kasneci
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.01198
• Pdf URL: https://arxiv.org/pdf/2505.01198
• Copy Paste: [[2505.01198]] Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods(https://arxiv.org/abs/2505.01198)
• Keywords: language model
• Abstract: While research on applications and evaluations of explanation methods continues to expand, fairness of the explanation methods concerning disparities in their performance across subgroups remains an often overlooked aspect. In this paper, we address this gap by showing that, across three tasks and five language models, widely used post-hoc feature attribution methods exhibit significant gender disparity with respect to their faithfulness, robustness, and complexity. These disparities persist even when the models are pre-trained or fine-tuned on particularly unbiased datasets, indicating that the disparities we observe are not merely consequences of biased training data. Our results highlight the importance of addressing disparities in explanations when developing and applying explainability methods, as these can lead to biased outcomes against certain subgroups, with particularly critical implications in high-stakes contexts. Furthermore, our findings underscore the importance of incorporating the fairness of explanations, alongside overall model fairness and explainability, as a requirement in regulatory frameworks.
• 摘要：虽然对应用程序的研究和解释方法的评估不断扩大，但有关其跨亚组差异的解释方法的公平性仍然是一个经常被忽视的方面。在本文中，我们通过表明在三个任务和五个语言模型中，广泛使用的事后特征归因方法在其忠诚，鲁棒性和复杂性方面表现出很大的性别差异。即使在模型进行了预先训练或在特别无偏的数据集上进行了预训练，这些差异仍然存在，这表明我们观察到的差异不仅是偏见培训数据的后果。我们的结果强调了在开发和应用解释性方法时解决解释中差异的重要性，因为这些方法可能导致对某些亚组的偏见结果，在高风险环境中具有特别关键的含义。此外，我们的发现强调了将解释的公平性纳入整体模型公平性和解释性的重要性，这是监管框架的要求。

Title: EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models

• Authors: Mahdi Dhaini, Kafaite Zahra Hussain, Efstratios Zaradoukas, Gjergji Kasneci
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.01238
• Pdf URL: https://arxiv.org/pdf/2505.01238
• Copy Paste: [[2505.01238]] EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models(https://arxiv.org/abs/2505.01238)
• Keywords: llm
• Abstract: As Natural Language Processing (NLP) models continue to evolve and become integral to high-stakes applications, ensuring their interpretability remains a critical challenge. Given the growing variety of explainability methods and diverse stakeholder requirements, frameworks that help stakeholders select appropriate explanations tailored to their specific use cases are increasingly important. To address this need, we introduce EvalxNLP, a Python framework for benchmarking state-of-the-art feature attribution methods for transformer-based NLP models. EvalxNLP integrates eight widely recognized explainability techniques from the Explainable AI (XAI) literature, enabling users to generate and evaluate explanations based on key properties such as faithfulness, plausibility, and complexity. Our framework also provides interactive, LLM-based textual explanations, facilitating user understanding of the generated explanations and evaluation outcomes. Human evaluation results indicate high user satisfaction with EvalxNLP, suggesting it is a promising framework for benchmarking explanation methods across diverse user groups. By offering a user-friendly and extensible platform, EvalxNLP aims at democratizing explainability tools and supporting the systematic comparison and advancement of XAI techniques in NLP.
• 摘要：随着自然语言处理（NLP）模型继续发展并成为高风险应用程序不可或缺的一部分，确保其可解释性仍然是一个关键的挑战。鉴于越来越多的解释性方法和各种利益相关者的要求，帮助利益相关者选择适合其特定用例的适当解释的框架越来越重要。为了满足这一需求，我们介绍了evalxnlp，这是一个为基于变压器NLP模型的最新功能归因方法基准测试的Python框架。 evalxnlp从可解释的AI（XAI）文献中集成了八种公认的解释性技术，使用户能够基于忠实，合理性和复杂性等关键属性生成和评估解释。我们的框架还提供了基于互动的，基于LLM的文本说明，从而促进了用户对生成的解释和评估结果的理解。人类评估结果表明对eRDXNLP的用户满意度很高，这表明它是基准对不同用户组的解释方法进行基准测试的有希望的框架。通过提供一个用户友好且可扩展的平台，ExtXNLP旨在使解释性工具民主化，并支持NLP中XAI技术的系统比较和进步。

Title: Anti-adversarial Learning: Desensitizing Prompts for Large Language Models

• Authors: Xuan Li, Zhe Yin, Xiaodong Gu, Beijun Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.01273
• Pdf URL: https://arxiv.org/pdf/2505.01273
• Copy Paste: [[2505.01273]] Anti-adversarial Learning: Desensitizing Prompts for Large Language Models(https://arxiv.org/abs/2505.01273)
• Keywords: language model, llm, prompt
• Abstract: With the widespread use of LLMs, preserving privacy in user prompts has become crucial, as prompts risk exposing privacy and sensitive data to the cloud LLMs. Traditional techniques like homomorphic encryption, secure multi-party computation, and federated learning face challenges due to heavy computational costs and user participation requirements, limiting their applicability in LLM scenarios. In this paper, we propose PromptObfus, a novel method for desensitizing LLM prompts. The core idea of PromptObfus is "anti-adversarial" learning, which perturbs privacy words in the prompt to obscure sensitive information while retaining the stability of model predictions. Specifically, PromptObfus frames prompt desensitization as a masked language modeling task, replacing privacy-sensitive terms with a [MASK] token. A desensitization model is trained to generate candidate replacements for each masked position. These candidates are subsequently selected based on gradient feedback from a surrogate model, ensuring minimal disruption to the task output. We demonstrate the effectiveness of our approach on three NLP tasks. Results show that PromptObfus effectively prevents privacy inference from remote LLMs while preserving task performance.
• 摘要：随着LLM的广泛使用，在用户提示中保留隐私已变得至关重要，因为提示将隐私和敏感数据暴露于Cloud LLMS的风险。传统技术，例如同态加密，安全的多方计算以及联合学习，由于沉重的计算成本和用户参与要求，面临挑战，从而限制了其在LLM场景中的适用性。在本文中，我们提出了PromptObfus，这是一种使LLM提示脱敏的新方法。 PromptObfus的核心思想是“反对”学习，它在提示中掩盖敏感信息的同时保留模型预测的稳定性。具体而言，PromptObfus框架将脱敏作为掩盖语言建模任务，用[bask]令牌代替对隐私敏感的术语。训练了脱敏模型，以生成每个蒙版位置的候选替代品。随后根据替代模型的梯度反馈选择了这些候选者，从而确保对任务输出的破坏最小。我们证明了方法对三个NLP任务的有效性。结果表明，PictherObfus有效地阻止了远程LLM的隐私推论，同时保留任务性能。

Title: Helping Big Language Models Protect Themselves: An Enhanced Filtering and Summarization System

• Authors: Sheikh Samit Muhaimin, Spyridon Mastorakis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.01315
• Pdf URL: https://arxiv.org/pdf/2505.01315
• Copy Paste: [[2505.01315]] Helping Big Language Models Protect Themselves: An Enhanced Filtering and Summarization System(https://arxiv.org/abs/2505.01315)
• Keywords: language model, llm, prompt
• Abstract: The recent growth in the use of Large Language Models has made them vulnerable to sophisticated adversarial assaults, manipulative prompts, and encoded malicious inputs. Existing countermeasures frequently necessitate retraining models, which is computationally costly and impracticable for deployment. Without the need for retraining or fine-tuning, this study presents a unique defense paradigm that allows LLMs to recognize, filter, and defend against adversarial or malicious inputs on their own. There are two main parts to the suggested framework: (1) A prompt filtering module that uses sophisticated Natural Language Processing (NLP) techniques, including zero-shot classification, keyword analysis, and encoded content detection (e.g. base64, hexadecimal, URL encoding), to detect, decode, and classify harmful inputs; and (2) A summarization module that processes and summarizes adversarial research literature to give the LLM context-aware defense knowledge. This approach strengthens LLMs' resistance to adversarial exploitation by fusing text extraction, summarization, and harmful prompt analysis. According to experimental results, this integrated technique has a 98.71% success rate in identifying harmful patterns, manipulative language structures, and encoded prompts. By employing a modest amount of adversarial research literature as context, the methodology also allows the model to react correctly to harmful inputs with a larger percentage of jailbreak resistance and refusal rate. While maintaining the quality of LLM responses, the framework dramatically increases LLM's resistance to hostile misuse, demonstrating its efficacy as a quick and easy substitute for time-consuming, retraining-based defenses.
• 摘要：大型语言模型的最新增长使它们容易受到复杂的对抗性攻击，操纵提示和编码恶意输入的影响。现有的对策经常需要重新测试模型，这在计算上是昂贵且不切实际的部署。本研究无需再进行一次或微调，提供了独特的防御范式，使LLM可以自己识别，过滤和防御对抗性或恶意输入。建议的框架有两个主要部分：（1）使用复杂的自然语言处理（NLP）技术的及时过滤模块，包括零摄像机分类，关键字分析和编码内容检测（例如Base64，Hexadecimal，url，URL，url编码），以检测，解码，分解，分类和分类和危害危害损害插入； （2）处理和总结对抗性研究文献以赋予LLM背景感知的防御知识的摘要模块。这种方法通过融合文本提取，摘要和有害的及时分析来增强LLMS对对抗性剥削的抵抗。根据实验结果，该集成技术在识别有害模式，操纵语言结构和编码提示方面具有98.71％的成功率。通过使用适度的对抗性研究文献作为背景，该方法还允许该模型对具有越来越比例的越狱抵抗和拒绝率的有害输入做出正确的反应。该框架在维持LLM响应的质量的同时，大大提高了LLM对敌对滥用的抵抗力，这表明了其有效性是快速简便的替代时间，以替代耗时，基于重试的防御力。

Title: TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References

• Authors: Svenja Kenneweg, Jörg Deigmöller, Philipp Cimiano, Julian Eggert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.01325
• Pdf URL: https://arxiv.org/pdf/2505.01325
• Copy Paste: [[2505.01325]] TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References(https://arxiv.org/abs/2505.01325)
• Keywords: llm
• Abstract: Understanding and resolving temporal references is essential in Natural Language Understanding as we often refer to the past or future in daily communication. Although existing benchmarks address a system's ability to reason about and resolve temporal references, systematic evaluation of specific temporal references remains limited. Towards closing this gap, we introduce TRAVELER, a novel synthetic benchmark dataset that follows a Question Answering paradigm and consists of questions involving temporal references with the corresponding correct answers. TRAVELER assesses models' abilities to resolve explicit, implicit relative to speech time, and vague temporal references. Beyond investigating the performance of state-of-the-art LLMs depending on the type of temporal reference, our benchmark also allows evaluation of performance in relation to the length of the set of events. For the category of vague temporal references, ground-truth answers were established via human surveys on Prolific, following a procedure similar to the one from Kenneweg et al. To demonstrate the benchmark's applicability, we evaluate four state-of-the-art LLMs using a question-answering task encompassing 3,300 questions. Our findings show that while the benchmarked LLMs can answer questions over event sets with a handful of events and explicit temporal references successfully, performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the vague question category exhibits the lowest performance across all models. The benchmark is publicly available at: this https URL
• 摘要：理解和解决时间参考对于自然语言理解至关重要，因为我们经常指的是日常交流的过去或将来。尽管现有基准测试了系统推理和解决时间参考的能力，但对特定时间参考的系统评估仍然有限。为了缩小这一差距，我们介绍了旅行者，这是一个新颖的合成基准数据集，遵循一个问题回答范式的问题，并包括涉及时间参考的问题，并带有相应的正确答案。旅行者评估模型能够解决明确的，与语音时间的隐式和含糊的时间参考的能力。除了根据时间参考的类型研究最新的LLMS的性能外，我们的基准还允许评估与事件集长度相关的性能。对于模糊的时间参考类别，通过与Kenneweg等人类似的程序进行了人类调查，通过人类的调查建立了基础真相答案。为了证明基准的适用性，我们使用包含3,300个问题的提问任务评估了四个最先进的LLMS。我们的发现表明，尽管基准的LLM可以通过少数事件和明确的时间参考成功回答事件集的问题，但绩效显然会随着事件设置的长度和时间参考而变得较少明确。值得注意的是，模糊的问题类别在所有模型中表现出最低的性能。该基准标准可公开可用：此HTTPS URL