2025-03-31

Title: ELM: Ensemble of Language Models for Predicting Tumor Group from Pathology Reports

• Authors: Lovedeep Gondara, Jonathan Simkin, Shebnum Devji, Gregory Arbour, Raymond Ng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21800
• Pdf URL: https://arxiv.org/pdf/2503.21800
• Copy Paste: [[2503.21800]] ELM: Ensemble of Language Models for Predicting Tumor Group from Pathology Reports(https://arxiv.org/abs/2503.21800)
• Keywords: language model, llm, prompt
• Abstract: Population-based cancer registries (PBCRs) face a significant bottleneck in manually extracting data from unstructured pathology reports, a process crucial for tasks like tumor group assignment, which can consume 900 person-hours for approximately 100,000 reports. To address this, we introduce ELM (Ensemble of Language Models), a novel ensemble-based approach leveraging both small language models (SLMs) and large language models (LLMs). ELM utilizes six fine-tuned SLMs, where three SLMs use the top part of the pathology report and three SLMs use the bottom part. This is done to maximize report coverage. ELM requires five-out-of-six agreement for a tumor group classification. Disagreements are arbitrated by an LLM with a carefully curated prompt. Our evaluation across nineteen tumor groups demonstrates ELM achieves an average precision and recall of 0.94, outperforming single-model and ensemble-without-LLM approaches. Deployed at the British Columbia Cancer Registry, ELM demonstrates how LLMs can be successfully applied in a PBCR setting to achieve state-of-the-art results and significantly enhance operational efficiencies, saving hundreds of person-hours annually.
• 摘要：基于人群的癌症注册表（PBCR）在手动从非结构化病理报告中提取数据时面临着重要的瓶颈，这对于肿瘤组分配等任务至关重要，肿瘤组分配的过程至关重要，肿瘤组分配可能会花费900个小时的时间以供大约100,000个报告。为了解决这个问题，我们介绍了ELM（语言模型的集合），这是一种基于新颖的集合方法，利用小语言模型（SLM）和大语言模型（LLMS）。 Elm使用了六个微型SLM，其中三个SLM使用病理报告的顶部，三个SLM使用底部。这样做是为了最大程度地提高报告覆盖范围。 ELM需要进行五分之一的肿瘤组分类一致。分歧是由LLM和经过精心策划的提示进行仲裁的。我们在十九个肿瘤组中进行的评估表明，ELM的平均精度和回忆为0.94，表现优于单模型和合奏 - 无数方法。 ELM部署在不列颠哥伦比亚省癌症注册表中，展示了如何成功地应用于PBCR环境中的LLM，以实现最先进的结果并显着提高了运营效率，从而节省了数百人小时。

Title: ImF: Implicit Fingerprint for Large Language Models

• Authors: Wu jiaxuan, Peng Wanli, Fu hang, Xue Yiming, Wen juan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21805
• Pdf URL: https://arxiv.org/pdf/2503.21805
• Copy Paste: [[2503.21805]] ImF: Implicit Fingerprint for Large Language Models(https://arxiv.org/abs/2503.21805)
• Keywords: language model, llm
• Abstract: Training large language models (LLMs) is resource-intensive and expensive, making intellectual property (IP) protection essential. Most existing model fingerprint methods inject fingerprints into LLMs to protect model ownership. These methods create fingerprint pairs with weak semantic correlations, lacking the contextual coherence and semantic relatedness founded in normal question-answer (QA) pairs in LLMs. In this paper, we propose a Generation Revision Intervention (GRI) attack that can effectively exploit this flaw to erase fingerprints, highlighting the need for more secure model fingerprint methods. Thus, we propose a novel injected fingerprint paradigm called Implicit Fingerprints (ImF). ImF constructs fingerprint pairs with strong semantic correlations, disguising them as natural QA pairs within LLMs. This ensures the fingerprints are consistent with normal model behavior, making them indistinguishable and robust against detection and removal. Our experiment on multiple LLMs demonstrates that ImF retains high verification success rates under adversarial conditions, offering a reliable solution for protecting LLM ownership.
• 摘要：培训大语言模型（LLMS）是资源密集且昂贵的，使知识产权（IP）保护至关重要。大多数现有的模型指纹方法将指纹注入LLMS以保护模型所有权。这些方法创建了具有弱语义相关性的指纹对，缺乏在LLMS中正常问答（QA）对中建立的上下文连贯性和语义相关性。在本文中，我们提出了一生的修订干预（GRI）攻击，该攻击可以有效利用这种缺陷来擦除指纹，从而强调了对更安全的模型指纹方法的需求。因此，我们提出了一种新颖的指纹范式，称为隐式指纹（IMF）。 IMF构建具有强烈语义相关性的指纹对，将它们掩盖为LLM中的天然QA对。这样可以确保指纹与正常模型行为一致，从而使它们无法区分且可抵抗检测和去除。我们对多个LLM的实验表明，IMF在对抗条件下保留了高验证成功率，为保护LLM所有权提供了可靠的解决方案。

Title: Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages

• Authors: Heqing Zou, Fengmao Lv, Desheng Zheng, Eng Siong Chng, Deepu Rajan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21806
• Pdf URL: https://arxiv.org/pdf/2503.21806
• Copy Paste: [[2503.21806]] Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages(https://arxiv.org/abs/2503.21806)
• Keywords: language model
• Abstract: Multilingual speech emotion recognition aims to estimate a speaker's emotional state using a contactless method across different languages. However, variability in voice characteristics and linguistic diversity poses significant challenges for zero-shot speech emotion recognition, especially with multilingual datasets. In this paper, we propose leveraging contrastive learning to refine multilingual speech features and extend large language models for zero-shot multilingual speech emotion estimation. Specifically, we employ a novel two-stage training framework to align speech signals with linguistic features in the emotional space, capturing both emotion-aware and language-agnostic speech representations. To advance research in this field, we introduce a large-scale synthetic multilingual speech emotion dataset, M5SER. Our experiments demonstrate the effectiveness of the proposed method in both speech emotion recognition and zero-shot multilingual speech emotion recognition, including previously unseen datasets and languages.
• 摘要：多语言语音情感识别旨在使用跨不同语言的非接触式方法来估计说话者的情绪状态。但是，语音特征和语言多样性的变化为零击语音情感识别带来了重大挑战，尤其是在多语言数据集中。在本文中，我们建议利用对比学习来完善多语言语音特征，并扩展大型语言模型以零击的多语言语音情感估计。具体来说，我们采用一个新颖的两阶段训练框架来使语音信号与情感空间中的语言特征保持一致，从而捕捉情感感知和语言语言的语音表示。为了推进该领域的研究，我们引入了一个大规模的合成多语言语音情感数据集M5SER。我们的实验证明了所提出的方法在语音情感识别和零击的多语言语音情感识别（包括以前看不见的数据集和语言）中的有效性。

Title: OAEI-LLM-T: A TBox Benchmark Dataset for Understanding LLM Hallucinations in Ontology Matching Systems

• Authors: Zhangcheng Qiang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.21813
• Pdf URL: https://arxiv.org/pdf/2503.21813
• Copy Paste: [[2503.21813]] OAEI-LLM-T: A TBox Benchmark Dataset for Understanding LLM Hallucinations in Ontology Matching Systems(https://arxiv.org/abs/2503.21813)
• Keywords: language model, llm, hallucination
• Abstract: Hallucinations are inevitable in downstream tasks using large language models (LLMs). While addressing hallucinations becomes a substantial challenge for LLM-based ontology matching (OM) systems, we introduce a new benchmark dataset called OAEI-LLM-T. The dataset evolves from the TBox (i.e. schema-matching) datasets in the Ontology Alignment Evaluation Initiative (OAEI), capturing hallucinations of different LLMs performing OM tasks. These OM-specific hallucinations are carefully classified into two primary categories and six sub-categories. We showcase the usefulness of the dataset in constructing the LLM leaderboard and fine-tuning foundational LLMs for LLM-based OM systems.
• 摘要：在使用大语言模型（LLM）的下游任务中，幻觉是不可避免的。尽管解决幻觉成为基于LLM的本体匹配（OM）系统的重大挑战，但我们引入了一个名为OAEI-LLM-T的新基准数据集。数据集从本体学评估计划（OAEI）中的Tbox（即模式匹配）数据集演变，捕获执行OM任务的不同LLMS的幻觉。这些特定于OM特异性的幻觉被仔细地分为两个主要类别和六个子类别。我们展示了数据集在构建基于LLM的OM系统的LLM排行榜和微调基础LLMS方面的实用性。

Title: Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach

• Authors: Xuying Li, Zhuo Li, Yuji Kosuga, Victor Bian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21819
• Pdf URL: https://arxiv.org/pdf/2503.21819
• Copy Paste: [[2503.21819]] Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach(https://arxiv.org/abs/2503.21819)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) with human values and safety constraints is challenging, especially when objectives like helpfulness, truthfulness, and avoidance of harm conflict. Reinforcement Learning from Human Feedback (RLHF) has achieved notable success in steering models, but is complex and can be unstable. Recent approaches such as Direct Preference Optimization (DPO) simplify preference-based fine-tuning but may introduce bias or trade-off certain objectives~\cite{dpo}. In this work, we propose a Group Relative Policy Optimization (GRPO) framework with a multi-label reward regression model to achieve safe and aligned language generation. The GRPO algorithm optimizes a policy by comparing groups of sampled responses, eliminating the need for a separate value critic and improving training efficiency~\cite{grpo}. We train a reward model to predict multiple alignment scores (e.g., safety, helpfulness, etc.), which are combined into a single reward signal. We provide a theoretical derivation for using this learned multi-aspect reward within GRPO and discuss its advantages and limitations. Empirically, our approach improves all the safety and quality metrics evaluated in language generation tasks on model scales (0.5B, 7B, and 14B parameters), demonstrating a robust balance of objectives. We compare GRPO to PPO-based RLHF and DPO, highlighting that GRPO achieves alignment with significantly lower computational cost and explicit multi-objective handling. \textbf{We will open-source all trained models at this https URL.
• 摘要：将大型语言模型（LLM）与人类价值观和安全限制保持一致，尤其是当诸如帮助，真实性和避免伤害冲突之类的目标时。从人类反馈中学习（RLHF）在转向模型中取得了显着的成功，但很复杂，可能是不稳定的。诸如直接偏好优化（DPO）之类的最新方法简化了基于偏好的微调，但可能引入偏见或权衡某些目标〜\ cite {dpo}。在这项工作中，我们提出了一个具有多标签奖励回归模型的小组相对政策优化（GRPO）框架，以实现安全且一致的语言生成。 GRPO算法通过比较采样响应的组来优化政策，从而消除了对单独的价值评论家的需求并提高培训效率〜\ cite {grpo}。我们训练一个奖励模型，以预测多个对齐分数（例如，安全性，有益性等），这些分数合并为单个奖励信号。我们提供了一种理论推导，用于在GRPO中使用这种学到的多种奖励奖励并讨论其优势和局限性。从经验上讲，我们的方法改善了在模型量表（0.5b，7b和14b参数）上评估的所有安全性和质量指标，表明了目标的强大平衡。我们将GRPO与基于PPO的RLHF和DPO进行了比较，这强调了GRPO的计算成本明显降低和明确的多目标处理。 \ textbf {我们将在此HTTPS URL上开放所有训练有素的模型。

Title: Refining Time Series Anomaly Detectors using Large Language Models

• Authors: Alan Yang, Yulin Chen, Sean Lee, Venus Montes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21833
• Pdf URL: https://arxiv.org/pdf/2503.21833
• Copy Paste: [[2503.21833]] Refining Time Series Anomaly Detectors using Large Language Models(https://arxiv.org/abs/2503.21833)
• Keywords: language model, llm
• Abstract: Time series anomaly detection (TSAD) is of widespread interest across many industries, including finance, healthcare, and manufacturing. Despite the development of numerous automatic methods for detecting anomalies, human oversight remains necessary to review and act upon detected anomalies, as well as verify their accuracy. We study the use of multimodal large language models (LLMs) to partially automate this process. We find that LLMs can effectively identify false alarms by integrating visual inspection of time series plots with text descriptions of the data-generating process. By leveraging the capabilities of LLMs, we aim to reduce the reliance on human effort required to maintain a TSAD system
• 摘要：时间序列异常检测（TSAD）在许多行业（包括金融，医疗保健和制造业）中引起了广泛关注。尽管开发了许多用于检测异常的自动方法，但人类的监督仍然需要审查和对检测到的异常行动以及验证其准确性。我们研究了多模式大语言模型（LLM）的使用来部分自动化此过程。我们发现，LLM可以通过将时间序列图的目视检查与数据生成过程的文本描述进行视觉检查来有效地识别错误警报。通过利用LLM的能力，我们旨在减少对维护TSAD系统所需的人类努力的依赖

Title: MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning

• Authors: Jiancheng Zhao, Xingda Yu, Zhen Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21838
• Pdf URL: https://arxiv.org/pdf/2503.21838
• Copy Paste: [[2503.21838]] MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning(https://arxiv.org/abs/2503.21838)
• Keywords: language model
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has become an essential approach for adapting large-scale pre-trained models while reducing computational costs. Among PEFT methods, LoRA significantly reduces trainable parameters by decomposing weight updates into low-rank matrices. However, traditional LoRA applies a fixed rank across all layers, failing to account for the varying complexity of hierarchical information, which leads to inefficient adaptation and redundancy. To address this, we propose MSPLoRA (Multi-Scale Pyramid LoRA), which introduces Global Shared LoRA, Mid-Level Shared LoRA, and Layer-Specific LoRA to capture global patterns, mid-level features, and fine-grained information, respectively. This hierarchical structure reduces inter-layer redundancy while maintaining strong adaptation capability. Experiments on various NLP tasks demonstrate that MSPLoRA achieves more efficient adaptation and better performance while significantly reducing the number of trainable parameters. Furthermore, additional analyses based on Singular Value Decomposition validate its information decoupling ability, highlighting MSPLoRA as a scalable and effective optimization strategy for parameter-efficient fine-tuning in large language models. Our code is available at this https URL.
• 摘要：参数有效的微调（PEFT）已成为适应大型预训练模型的同时降低计算成本的重要方法。在PEFT方法中，Lora通过将重量更新分解为低级矩阵来大大降低了可训练的参数。但是，传统洛拉（Lora）在所有层中都采用固定等级，但未能说明层次信息的复杂性不同，从而导致适应性和冗余效率低下。为了解决这个问题，我们提出了MSPLORA（多尺度金字塔Lora），它介绍了全局共享的LORA，中层共享的Lora和特定于层的Lora，以捕获全局模式，中级特征和细粒度的信息。这种分层结构可降低层间冗余，同时保持强大的适应能力。对各种NLP任务的实验表明，MSPlora可以实现更有效的适应性和更好的性能，同时显着减少了可训练的参数的数量。此外，基于单数值分解的其他分析验证了其信息解耦能力，从而突出了MSPlora作为大型语言模型中参数有效微调的可扩展有效优化策略。我们的代码可在此HTTPS URL上找到。

Title: RedditESS: A Mental Health Social Support Interaction Dataset -- Understanding Effective Social Support to Refine AI-Driven Support Tools

• Authors: Zeyad Alghamdi, Tharindu Kumarage, Garima Agrawal, Mansooreh Karami, Ibrahim Almuteb, Huan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21888
• Pdf URL: https://arxiv.org/pdf/2503.21888
• Copy Paste: [[2503.21888]] RedditESS: A Mental Health Social Support Interaction Dataset -- Understanding Effective Social Support to Refine AI-Driven Support Tools(https://arxiv.org/abs/2503.21888)
• Keywords: language model, llm
• Abstract: Effective mental health support is crucial for alleviating psychological distress. While large language model (LLM)-based assistants have shown promise in mental health interventions, existing research often defines "effective" support primarily in terms of empathetic acknowledgments, overlooking other essential dimensions such as informational guidance, community validation, and tangible coping strategies. To address this limitation and better understand what constitutes effective support, we introduce RedditESS, a novel real-world dataset derived from Reddit posts, including supportive comments and original posters' follow-up responses. Grounded in established social science theories, we develop an ensemble labeling mechanism to annotate supportive comments as effective or not and perform qualitative assessments to ensure the reliability of the annotations. Additionally, we demonstrate the practical utility of RedditESS by using it to guide LLM alignment toward generating more context-sensitive and genuinely helpful supportive responses. By broadening the understanding of effective support, our study paves the way for advanced AI-driven mental health interventions.
• 摘要：有效的心理健康支持对于减轻心理困扰至关重要。尽管基于大型语言模型（LLM）的助手在心理健康干预措施中表现出了希望，但现有研究通常主要定义“有效”支持，主要是在善解人意的承认方面，忽略了其他基本维度，例如信息指导，社区验证，有形应对策略。为了解决这一限制并更好地了解什么构成有效的支持，我们介绍了Redditess，这是一种从Reddit帖子中得出的新颖的现实世界数据集，包括支持性评论和原始海报的后续响应。基于既定的社会科学理论，我们开发了一种合奏标签机制，以注释支持性评论是否有效，并进行定性评估以确保注释的可靠性。此外，我们通过使用它来指导LLM Alignment来产生更高上下文敏感且真正有用的支持响应来证明Redditess的实际实用性。通过扩大对有效支持的理解，我们的研究为先进的AI驱动心理健康干预铺平了道路。

Title: JEEM: Vision-Language Understanding in Four Arabic Dialects

• Authors: Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, Yova Kementchedjhieva
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21910
• Pdf URL: https://arxiv.org/pdf/2503.21910
• Copy Paste: [[2503.21910]] JEEM: Vision-Language Understanding in Four Arabic Dialects(https://arxiv.org/abs/2503.21910)
• Keywords: language model, gpt
• Abstract: We introduce JEEM, a benchmark designed to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco. JEEM includes the tasks of image captioning and visual question answering, and features culturally rich and regionally diverse content. This dataset aims to assess the ability of VLMs to generalize across dialects and accurately interpret cultural elements in visual contexts. In an evaluation of five prominent open-source Arabic VLMs and GPT-4V, we find that the Arabic VLMs consistently underperform, struggling with both visual understanding and dialect-specific generation. While GPT-4V ranks best in this comparison, the model's linguistic competence varies across dialects, and its visual understanding capabilities lag behind. This underscores the need for more inclusive models and the value of culturally-diverse evaluation paradigms.
• 摘要：我们介绍了Jeem，这是一种基准，旨在评估跨四个讲阿拉伯语国家的视觉理解的视觉语言模型（VLM）：约旦，阿联酋，埃及和摩洛哥。 Jeem包括图像字幕和视觉问题的任务，并具有文化丰富和多样的内容。该数据集旨在评估VLM跨方言概括并准确解释视觉环境中的文化元素的能力。在评估五个著名的开源阿拉伯VLM和GPT-4V时，我们发现阿拉伯VLM始终表现不佳，在视觉理解和特定方言特定的一代中都挣扎。尽管GPT-4V在此比较中排名最佳，但该模型的语言能力在方言各不相同，其视觉理解能力却落后。这强调了对更多包容性模型的需求和文化多样性评估范式的价值。

Title: AutoPsyC: Automatic Recognition of Psychodynamic Conflicts from Semi-structured Interviews with Large Language Models

• Authors: Sayed Muddashir Hossain, Simon Ostermann, Patrick Gebhard, Cord Benecke, Josef van Genabith, Philipp Müller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21911
• Pdf URL: https://arxiv.org/pdf/2503.21911
• Copy Paste: [[2503.21911]] AutoPsyC: Automatic Recognition of Psychodynamic Conflicts from Semi-structured Interviews with Large Language Models(https://arxiv.org/abs/2503.21911)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Psychodynamic conflicts are persistent, often unconscious themes that shape a person's behaviour and experiences. Accurate diagnosis of psychodynamic conflicts is crucial for effective patient treatment and is commonly done via long, manually scored semi-structured interviews. Existing automated solutions for psychiatric diagnosis tend to focus on the recognition of broad disorder categories such as depression, and it is unclear to what extent psychodynamic conflicts which even the patient themselves may not have conscious access to could be automatically recognised from conversation. In this paper, we propose AutoPsyC, the first method for recognising the presence and significance of psychodynamic conflicts from full-length Operationalized Psychodynamic Diagnostics (OPD) interviews using Large Language Models (LLMs). Our approach combines recent advances in parameter-efficient fine-tuning and Retrieval-Augmented Generation (RAG) with a summarisation strategy to effectively process entire 90 minute long conversations. In evaluations on a dataset of 141 diagnostic interviews we show that AutoPsyC consistently outperforms all baselines and ablation conditions on the recognition of four highly relevant psychodynamic conflicts.
• 摘要：心理动力冲突是持久的，通常是昏迷的主题，这些主题塑造了一个人的行为和经历。精神动力冲突的准确诊断对于有效的患者治疗至关重要，通常是通过长期，手动评分的半结构化访谈来完成的。现有用于精神诊断的自动化解决方案倾向于集中于对抑郁症等广泛疾病类别的识别，并且尚不清楚在多大程度上可以从对话中自动认识到，即使患者本身也无法有意识地获得这种冲突。在本文中，我们提出了尸检，这是使用大语言模型（LLMS）的全长手术心理动力学诊断（OPD）访谈的第一种认识到心理动力冲突的存在和意义的方法。我们的方法结合了参数有效的微调和检索功能生成（RAG）的最新进展以及摘要策略，以有效地处理整个90分钟长的对话。在对141次诊断访谈的数据集的评估中，我们表明尸检始终在识别四起高度相关的心理动力冲突的识别方面始终优于所有基准和消融条件。

Title: Hybrid Emotion Recognition: Enhancing Customer Interactions Through Acoustic and Textual Analysis

• Authors: Sahan Hewage Wewelwala, T.G.D.K. Sumanathilaka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21927
• Pdf URL: https://arxiv.org/pdf/2503.21927
• Copy Paste: [[2503.21927]] Hybrid Emotion Recognition: Enhancing Customer Interactions Through Acoustic and Textual Analysis(https://arxiv.org/abs/2503.21927)
• Keywords: language model, llm
• Abstract: This research presents a hybrid emotion recognition system integrating advanced Deep Learning, Natural Language Processing (NLP), and Large Language Models (LLMs) to analyze audio and textual data for enhancing customer interactions in contact centers. By combining acoustic features with textual sentiment analysis, the system achieves nuanced emotion detection, addressing the limitations of traditional approaches in understanding complex emotional states. Leveraging LSTM and CNN models for audio analysis and DistilBERT for textual evaluation, the methodology accommodates linguistic and cultural variations while ensuring real-time processing. Rigorous testing on diverse datasets demonstrates the system's robustness and accuracy, highlighting its potential to transform customer service by enabling personalized, empathetic interactions and improving operational efficiency. This research establishes a foundation for more intelligent and human-centric digital communication, redefining customer service standards.
• 摘要：这项研究提出了一个混合情感识别系统，该系统整合了先进的深度学习，自然语言处理（NLP）和大型语言模型（LLMS），以分析音频和文本数据，以增强联系中心的客户互动。通过将声学特征与文本情感分析相结合，该系统实现了细微的情感检测，解决了传统方法在理解复杂情绪状态方面的局限性。该方法利用LSTM和CNN模型进行音频分析和Distilbert进行文本评估，在确保实时处理的同时，可以适应语言和文化变化。对各种数据集进行严格的测试证明了系统的稳健性和准确性，突出了其通过实现个性化的，善解人意的互动并提高运营效率来转变客户服务的潜力。这项研究为更智能和以人为中心的数字通信，重新定义客户服务标准奠定了基础。

Title: Local Normalization Distortion and the Thermodynamic Formalism of Decoding Strategies for Large Language Models

• Authors: Tom Kempton, Stuart Burrell
• Subjects: cs.CL, cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2503.21929
• Pdf URL: https://arxiv.org/pdf/2503.21929
• Copy Paste: [[2503.21929]] Local Normalization Distortion and the Thermodynamic Formalism of Decoding Strategies for Large Language Models(https://arxiv.org/abs/2503.21929)
• Keywords: language model
• Abstract: Advances in hardware and language model architecture have spurred a revolution in natural language generation. However, autoregressive models compute probability distributions over next-token choices, and sampling from these distributions, known as decoding, has received significantly less attention than other design choices. Existing decoding strategies are largely based on heuristics, resulting in methods that are hard to apply or improve in a principled manner. We develop the theory of decoding strategies for language models by expressing popular decoding algorithms as equilibrium states in the language of ergodic theory and stating the functions they optimize. Using this, we analyze the effect of the local normalization step of top-k, nucleus, and temperature sampling, used to make probabilities sum to one. We argue that local normalization distortion is a fundamental defect of decoding strategies and quantify the size of this distortion and its effect on mathematical proxies for the quality and diversity of generated text. Contrary to the prevailing explanation, we argue that the major cause of the under-performance of top-k sampling relative to nucleus sampling is local normalization distortion. This yields conclusions for the future design of decoding algorithms and the detection of machine-generated text.
• 摘要：硬件和语言模型体系结构的进步激发了自然语言产生的革命。但是，自回归模型在下一步选择上计算概率分布，而这些分布（称为解码）的取样比其他设计选择的关注明显少得多。现有的解码策略在很大程度上基于启发式方法，从而导致难以以原则性的方式应用或改进的方法。我们通过以千古理论语言表达流行的解码算法来开发语言模型解码策略的理论，并说明它们优化的功能。使用此过程，我们分析了TOP-K，核和温度采样的局部归一化步骤的效果，用于使概率总和达到一个。我们认为，局部归一化扭曲是解码策略的根本缺陷，并量化了这种失真的大小及其对数学代理的影响对生成文本的质量和多样性的影响。与普遍的解释相反，我们认为，相对于核抽样的TOP-K采样不足的主要原因是局部归一化扭曲。这给出了解码算法的未来设计和机器生成文本的检测的结论。

Title: Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

• Authors: Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, Martin Vechev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.21934
• Pdf URL: https://arxiv.org/pdf/2503.21934
• Copy Paste: [[2503.21934]] Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad(https://arxiv.org/abs/2503.21934)
• Keywords: language model, llm
• Abstract: Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, o3-mini, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce the first comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly, achieving less than 5% on average. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities.
• 摘要：大型语言模型（LLM）（例如Matharena）的最新数学基准表明，最先进的推理模型在Aime，领先的模型O3 Mini等数学竞赛中实现了令人印象深刻的性能，取得了与顶尖人类竞争对手相当的分数。但是，这些基准测试仅基于最终数值答案评估模型，忽略了严格的推理和证明生成，这对于现实世界中的数学任务至关重要。为了解决这一问题，我们介绍了针对数学问题的全面推理的首次全面评估。使用专家人注释者，我们评估了几个最新的推理模型，该模型在释放后数小时内的2025年USAMO的六个问题上进行了评估。我们的结果表明，所有经过测试的模型都在挣扎中很大，平均达到了不到5％。通过对推理轨迹的详细分析，我们确定了最常见的故障模式，并找到了由模型培训过程中采用的优化策略引起的几种不必要的伪像。总体而言，我们的结果表明，当前的LLM不足以进行严格的数学推理任务，这突出了对推理和证明产生能力的实质性改进的需求。

Title: Entropy-Aware Branching for Improved Mathematical Reasoning

• Authors: Xianzhi Li, Ethan Callanan, Xiaodan Zhu, Mathieu Sibue, Antony Papadimitriou, Mahmoud Mahfouz, Zhiqiang Ma, Xiaomo Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21961
• Pdf URL: https://arxiv.org/pdf/2503.21961
• Copy Paste: [[2503.21961]] Entropy-Aware Branching for Improved Mathematical Reasoning(https://arxiv.org/abs/2503.21961)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) are effectively aligned through extensive pre-training and fine-tuning, they still struggle with varying levels of uncertainty during token generation. In our investigation of mathematical reasoning, we observe that errors are more likely to arise at tokens exhibiting high entropy and variance of entropy in the model's output distribution. Based on the observation, we propose a novel approach that dynamically branches the generation process on demand instead of defaulting to the single most probable token. By exploring in parallel multiple branches stemming from high probability tokens of critical decision points, the model can discover diverse reasoning paths that might otherwise be missed. We further harness external feedback from larger models to rank and select the most coherent and accurate reasoning branch. Our experimental results on mathematical word problems and calculation questions show that this branching strategy boosts the reasoning capabilities of small LLMs up to 4.6% compared to conventional argmax decoding.
• 摘要：尽管大型语言模型（LLM）通过广泛的预训练和微调有效地保持一致，但在代币产生过程中，它们仍然在不同程度的不确定性方面挣扎。在我们对数学推理的研究中，我们观察到，在模型输出分布中表现出高熵和熵方差的令牌中，错误更有可能出现。基于观察结果，我们提出了一种新颖的方法，该方法可以动态地按需分支生成过程，而不是默认为最可能的令牌。通过探索来自关键决策点的高概率令牌的并行多个分支，该模型可以发现可能遗漏的各种推理路径。我们进一步利用来自较大模型的外部反馈，以排名并选择最连贯和准确的推理分支。我们对数学单词问题和计算问题的实验结果表明，与常规的Argmax解码相比，这种分支策略将小LLMS的推理能力提高了4.6％。

Title: Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them

• Authors: Marc Brinner, Tarek Al Mustafa, Sina Zarrieß
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.22006
• Pdf URL: https://arxiv.org/pdf/2503.22006
• Copy Paste: [[2503.22006]] Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them(https://arxiv.org/abs/2503.22006)
• Keywords: language model, llm
• Abstract: We investigate the use of LLM-generated data for continual pretraining of encoder models in specialized domains with limited training data, using the scientific domain of invasion biology as a case study. To this end, we leverage domain-specific ontologies by enriching them with LLM-generated data and pretraining the encoder model as an ontology-informed embedding model for concept definitions. To evaluate the effectiveness of this method, we compile a benchmark specifically designed for assessing model performance in invasion biology. After demonstrating substantial improvements over standard LLM pretraining, we investigate the feasibility of applying the proposed approach to domains without comprehensive ontologies by substituting ontological concepts with concepts automatically extracted from a small corpus of scientific abstracts and establishing relationships between concepts through distributional statistics. Our results demonstrate that this automated approach achieves comparable performance using only a small set of scientific abstracts, resulting in a fully automated pipeline for enhancing domain-specific understanding of small encoder models that is especially suited for application in low-resource settings and achieves performance comparable to masked language modeling pretraining on much larger datasets.
• 摘要：我们研究了使用有限的训练数据的专业领域中使用LLM生成的数据在具有有限培训数据的专用域中持续预处理的使用，该模型使用入侵生物学的科学领域作为案例研究。为此，我们通过用LLM生成的数据丰富了特定于域的本体，并将编码器模型作为概念定义的本体知识嵌入模型进行预处理。为了评估该方法的有效性，我们编制了专门设计用于评估入侵生物学模型性能的基准。在证明了对标准LLM预处理的实质性改进之后，我们研究了将提议的方法应用于领域的可行性，而无需全面的本体论，通过将本体论概念替换为自动从一小部分科学摘要中提取的概念，并通过分布统计来确定概念之间的关系。我们的结果表明，这种自动化方法仅使用一小部分科学摘要实现可比性的性能，从而产生了一条全自动的管道，以增强对小型编码器模型的特定于域特异性的理解，这些模型特别适合在低资源设置中应用，并在较大的数据集中掩盖了与掩盖的语言相当的性能。

Title: Cognitive Prompts Using Guilford's Structure of Intellect Model

• Authors: Oliver Kramer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.22036
• Pdf URL: https://arxiv.org/pdf/2503.22036
• Copy Paste: [[2503.22036]] Cognitive Prompts Using Guilford's Structure of Intellect Model(https://arxiv.org/abs/2503.22036)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) demonstrate strong language generation capabilities but often struggle with structured reasoning, leading to inconsistent or suboptimal problem-solving. To mitigate this limitation, Guilford's Structure of Intellect (SOI) model - a foundational framework from intelligence theory - is leveraged as the basis for cognitive prompt engineering. The SOI model categorizes cognitive operations such as pattern recognition, memory retrieval, and evaluation, offering a systematic approach to enhancing LLM reasoning and decision-making. This position paper presents a novel cognitive prompting approach for enforcing SOI-inspired reasoning for improving clarity, coherence, and adaptability in model responses.
• 摘要：大型语言模型（LLMS）表现出强大的语言产生能力，但经常在结构化的推理方面挣扎，导致不一致或次优的问题解决。为了减轻这种局限性，吉尔福德的智力结构（SOI）模型（来自智能理论的基础框架）被杠杆作为认知及时工程的基础。 SOI模型对认知操作进行了分类，例如模式识别，记忆检索和评估，为增强LLM推理和决策提供了系统的方法。该立场论文提出了一种新型的认知提示方法，用于实施SOI启发的推理，以提高模型响应中的清晰度，连贯性和适应性。

Title: The Risks of Using Large Language Models for Text Annotation in Social Science Research

• Authors: Hao Lin, Yongjun Zhang
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.22040
• Pdf URL: https://arxiv.org/pdf/2503.22040
• Copy Paste: [[2503.22040]] The Risks of Using Large Language Models for Text Annotation in Social Science Research(https://arxiv.org/abs/2503.22040)
• Keywords: language model, llm, prompt
• Abstract: Generative artificial intelligence (GenAI) or large language models (LLMs) have the potential to revolutionize computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks of using LLMs for diverse coding tasks, with social movement studies serving as a case example. We propose a framework for social scientists to incorporate LLMs into text annotation, either as the primary coding decision-maker or as a coding assistant. This framework provides tools for researchers to develop the optimal prompt, and to examine and report the validity and reliability of LLMs as a methodological tool. Additionally, we discuss the associated epistemic risks related to validity, reliability, replicability, and transparency. We conclude with several practical guidelines for using LLMs in text annotation tasks, and how we can better communicate the epistemic risks in research.
• 摘要：生成人工智能（Genai）或大语言模型（LLM）有可能革新计算社会科学，尤其是在自动化文本分析中。在本文中，我们对使用LLM进行多种编码任务的承诺和风险进行了系统的评估，社会运动研究是一个例子。我们为社会科学家提出了一个框架，将LLM纳入文本注释中，无论是作为主要编码决策者还是编码助手。该框架为研究人员提供了开发最佳提示的工具，并检查和报告LLMS作为方法论工具的有效性和可靠性。此外，我们讨论了与有效性，可靠性，可复制性和透明度有关的相关认知风险。我们以几种在文本注释任务中使用LLM的实用准则，以及如何更好地传达研究中的认知风险。

Title: ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models

• Authors: Chung-En Sun, Ge Yan, Tsui-Wei Weng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.22048
• Pdf URL: https://arxiv.org/pdf/2503.22048
• Copy Paste: [[2503.22048]] ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models(https://arxiv.org/abs/2503.22048)
• Keywords: language model, llm, chain-of-thought
• Abstract: Recent studies have shown that Large Language Models (LLMs) augmented with chain-of-thought (CoT) reasoning demonstrate impressive problem-solving abilities. However, in this work, we identify a recurring issue where these models occasionally generate overly short reasoning, leading to degraded performance on even simple mathematical problems. Specifically, we investigate how reasoning length is embedded in the hidden representations of reasoning models and its impact on accuracy. Our analysis reveals that reasoning length is governed by a linear direction in the representation space, allowing us to induce overly short reasoning by steering the model along this direction. Building on this insight, we introduce ThinkEdit, a simple yet effective weight-editing approach to mitigate the issue of overly short reasoning. We first identify a small subset of attention heads (approximately 2%) that predominantly drive short reasoning behavior. We then edit the output projection weights of these heads to suppress the short reasoning direction. With changes to only 0.1% of the model's parameters, ThinkEdit effectively reduces overly short reasoning and yields notable accuracy gains for short reasoning outputs (+5.44%), along with an overall improvement across multiple math benchmarks (+2.43%). Our findings provide new mechanistic insights into how reasoning length is controlled within LLMs and highlight the potential of fine-grained model interventions to improve reasoning quality. Our code is available at this https URL
• 摘要：最近的研究表明，大型语言模型（LLMS）随着经营链（COT）的推理增强，表现出令人印象深刻的解决问题的能力。但是，在这项工作中，我们确定了一个反复出现的问题，这些模型偶尔会产生过度短暂的推理，从而导致在简单的数学问题上降低性能。具体而言，我们研究了推理长度如何嵌入推理模型的隐藏表示中及其对准确性的影响。我们的分析表明，推理长度受表示空间中线性方向的控制，使我们能够通过沿该方向转向模型来诱导过度短暂的推理。在这种见解的基础上，我们介绍了ThinkEdit，这是一种简单而有效的权重编辑方法，可以减轻过度简短的推理问题。我们首先确定了一小部分注意力头（约2％），主要是推动短暂的推理行为。然后，我们编辑这些头部的输出投影权重，以抑制简短的推理方向。随着模型参数的0.1％的变化，认为有效地减少了过度短暂的推理，并且在短期推理输出（+5.44％）以及多个数学基准（+2.43％）之间的总体改进中，其准确性的准确性显着提高（+2.43％）。我们的发现提供了有关LLM中推理长度如何控制的新机械见解，并强调了细粒度模型干预措施的潜力以提高推理质量。我们的代码可在此HTTPS URL上找到

Title: Penrose Tiled Low-Rank Compression and Section-Wise Q&A Fine-Tuning: A General Framework for Domain-Specific Large Language Model Adaptation

• Authors: Chuan-Wei Kuo, Siyu Chen, Chenqi Yan, Yu Yang Fredrik Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.22074
• Pdf URL: https://arxiv.org/pdf/2503.22074
• Copy Paste: [[2503.22074]] Penrose Tiled Low-Rank Compression and Section-Wise Q&A Fine-Tuning: A General Framework for Domain-Specific Large Language Model Adaptation(https://arxiv.org/abs/2503.22074)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) hold great promise for specialized scientific domains such as materials science, yet adapting them efficiently and accurately to domain-specific knowledge remains challenging due to limited data and high knowledge density. We propose a two-stage framework that combines structured model compression with a scientific fine-tuning regimen to address this challenge. In the compression stage, we decompose the LLM's weight matrices into local low-rank "rank blocks" and arrange these blocks in a Penrose-like non-periodic tiling pattern. Each block is then compacted via spectral transformations (e.g., discrete cosine or Fourier transforms), and a Kullback-Leibler (KL) divergence-based alignment loss preserves the distributional similarity between the compressed model's representations and those of the original full model. In the adaptation stage, the compressed model is further tuned using a human-like scientific reading protocol: it processes technical materials science documents section by section, engaging in a structured question-and-answer routine for each section. This section-wise Q&A fine-tuning strategy extracts explicit reasoning traces and gradually injects domain knowledge, while minimizing catastrophic forgetting of the model's general language capabilities. By balancing efficient compression with targeted adaptation, our two-stage approach enables precise specialization of LLMs to high-value domains under data-scarce conditions. We present this principled yet exploratory pipeline and outline its potential for advancing materials science knowledge integration, laying the groundwork for comprehensive empirical evaluation in future work.
• 摘要：大型语言模型（LLMS）对诸如材料科学等专业科学领域（例如材料科学）持巨大希望，但是由于数据有限和知识密度有限，因此有效，准确地适应特定于领域的知识仍然具有挑战性。我们提出了一个两阶段的框架，将结构化模型压缩与科学的微调方案相结合，以应对这一挑战。在压缩阶段，我们将LLM的重量矩阵分解为局部低级别的“等级块”，并以penrose样的非周期性瓷砖图案排列这些块。然后，通过光谱变换（例如离散的余弦或傅立叶变换）和基于Kullback-Leibler（KL）基于差异的对准损耗将每个块压实，可保留压缩模型的表示与原始完整模型的分布相似性。在适应阶段，使用类似人类的科学阅读协议进一步调整了压缩模型：它通过部分处理技术材料科学文档，为每个部分进行结构化的问答程序。本节的问答微调策略提取了明确的推理跟踪并逐渐注入域知识，同时最大程度地减少了对模型的一般语言能力的灾难性忘记。通过平衡有效的压缩与有针对性的适应性，我们的两阶段方法可以在数据质量条件下精确地专业化LLMS对高价值域的精确专业化。我们介绍了这条原则但探索性的管道，并概述了其推进材料科学知识整合的潜力，为未来工作中的全面经验评估奠定了基础。

Title: Leveraging LLMs for Predicting Unknown Diagnoses from Clinical Notes

• Authors: Dina Albassam, Adam Cross, Chengxiang Zhai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22092
• Pdf URL: https://arxiv.org/pdf/2503.22092
• Copy Paste: [[2503.22092]] Leveraging LLMs for Predicting Unknown Diagnoses from Clinical Notes(https://arxiv.org/abs/2503.22092)
• Keywords: language model, gpt, llm, prompt
• Abstract: Electronic Health Records (EHRs) often lack explicit links between medications and diagnoses, making clinical decision-making and research more difficult. Even when links exist, diagnosis lists may be incomplete, especially during early patient visits. Discharge summaries tend to provide more complete information, which can help infer accurate diagnoses, especially with the help of large language models (LLMs). This study investigates whether LLMs can predict implicitly mentioned diagnoses from clinical notes and link them to corresponding medications. We address two research questions: (1) Does majority voting across diverse LLM configurations outperform the best single configuration in diagnosis prediction? (2) How sensitive is majority voting accuracy to LLM hyperparameters such as temperature, top-p, and summary length? To evaluate, we created a new dataset of 240 expert-annotated medication-diagnosis pairs from 20 MIMIC-IV notes. Using GPT-3.5 Turbo, we ran 18 prompting configurations across short and long summary lengths, generating 8568 test cases. Results show that majority voting achieved 75 percent accuracy, outperforming the best single configuration at 66 percent. No single hyperparameter setting dominated, but combining deterministic, balanced, and exploratory strategies improved performance. Shorter summaries generally led to higher this http URL conclusion, ensemble-style majority voting with diverse LLM configurations improves diagnosis prediction in EHRs and offers a promising method to link medications and diagnoses in clinical texts.
• 摘要：电子健康记录（EHR）通常缺乏药物与诊断之间的明确联系，从而使临床决策和研究更加困难。即使存在链接，诊断清单也可能不完整，尤其是在早期患者就诊时。出院摘要倾向于提供更完整的信息，这可以帮助推断准确的诊断，尤其是在大型语言模型（LLMS）的帮助下。这项研究调查了LLM是否可以预测临床笔记中隐式提及的诊断，并将其与相应的药物联系起来。我们解决了两个研究问题：（1）多数LLM配置的多数投票优于诊断预测中最佳的单一配置？ （2）大多数投票准确性对LLM超参数（例如温度，TOP-P和摘要长度）的敏感性如何？为了进行评估，我们创建了一个新的数据集，该数据集是从20个模拟物IV注释中创建了240个专家注册的药物诊断对。使用GPT-3.5涡轮增压器，我们运行了18个提示在短和长摘要长度上提示配置，生成了8568个测试用例。结果表明，大多数投票达到75％的准确性，以66％的速度优于最佳单一配置。没有单一的超参数设置主导，但是结合确定性，平衡和探索性策略可以提高性能。较短的摘要通常会导致更高的HTTP URL结论，以不同的LLM配置进行集合式多数投票可以改善EHR的诊断预测，并为临床文本中的链接药物和诊断提供了有希望的方法。

Title: Beyond Single-Sentence Prompts: Upgrading Value Alignment Benchmarks with Dialogues and Stories

• Authors: Yazhou Zhang, Qimeng Liu, Qiuchi Li, Peng Zhang, Jing Qin
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.22115
• Pdf URL: https://arxiv.org/pdf/2503.22115
• Copy Paste: [[2503.22115]] Beyond Single-Sentence Prompts: Upgrading Value Alignment Benchmarks with Dialogues and Stories(https://arxiv.org/abs/2503.22115)
• Keywords: language model, llm, prompt
• Abstract: Evaluating the value alignment of large language models (LLMs) has traditionally relied on single-sentence adversarial prompts, which directly probe models with ethically sensitive or controversial questions. However, with the rapid advancements in AI safety techniques, models have become increasingly adept at circumventing these straightforward tests, limiting their effectiveness in revealing underlying biases and ethical stances. To address this limitation, we propose an upgraded value alignment benchmark that moves beyond single-sentence prompts by incorporating multi-turn dialogues and narrative-based scenarios. This approach enhances the stealth and adversarial nature of the evaluation, making it more robust against superficial safeguards implemented in modern LLMs. We design and implement a dataset that includes conversational traps and ethically ambiguous storytelling, systematically assessing LLMs' responses in more nuanced and context-rich settings. Experimental results demonstrate that this enhanced methodology can effectively expose latent biases that remain undetected in traditional single-shot evaluations. Our findings highlight the necessity of contextual and dynamic testing for value alignment in LLMs, paving the way for more sophisticated and realistic assessments of AI ethics and safety.
• 摘要：传统上，评估大语言模型（LLM）（LLM）的价值一致性（LLMS）依赖于单句话的对抗提示，该提示直接探测具有道德敏感或有争议的问题的模型。但是，随着AI安全技术的快速进步，模型已经越来越熟练于绕过这些直接测试，从而限制了它们在揭示潜在的偏见和道德立场方面的有效性。为了解决这一限制，我们提出了一个升级的价值对齐基准，该基准通过合并多转向对话和基于叙事的场景，超越单句提示。这种方法增强了评估的隐身和对抗性，使其对现代LLMS实施的表面保障措施更加强大。我们设计和实施一个数据集，其中包括对话陷阱和道德上模棱两可的讲故事，系统地评估了LLMS在更细微和更富裕的环境中的响应。实验结果表明，这种增强的方法可以有效地暴露在传统的单发评估中仍未发现的潜在偏见。我们的发现凸显了在LLMS中进行背景和动态测试的必要性，为对AI伦理和安全的更复杂和现实的评估铺平了道路。

Title: FRASE: Structured Representations for Generalizable SPARQL Query Generation

• Authors: Papa Abdou Karim Karou Diallo, Amal Zouaq
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.22144
• Pdf URL: https://arxiv.org/pdf/2503.22144
• Copy Paste: [[2503.22144]] FRASE: Structured Representations for Generalizable SPARQL Query Generation(https://arxiv.org/abs/2503.22144)
• Keywords: language model, llm
• Abstract: Translating natural language questions into SPARQL queries enables Knowledge Base querying for factual and up-to-date responses. However, existing datasets for this task are predominantly template-based, leading models to learn superficial mappings between question and query templates rather than developing true generalization capabilities. As a result, models struggle when encountering naturally phrased, template-free questions. This paper introduces FRASE (FRAme-based Semantic Enhancement), a novel approach that leverages Frame Semantic Role Labeling (FSRL) to address this limitation. We also present LC-QuAD 3.0, a new dataset derived from LC-QuAD 2.0, in which each question is enriched using FRASE through frame detection and the mapping of frame-elements to their argument. We evaluate the impact of this approach through extensive experiments on recent large language models (LLMs) under different fine-tuning configurations. Our results demonstrate that integrating frame-based structured representations consistently improves SPARQL generation performance, particularly in challenging generalization scenarios when test questions feature unseen templates (unknown template splits) and when they are all naturally phrased (reformulated questions).
• 摘要：将自然语言问题转化为SPARQL查询，可以使知识库查询事实和最新回答。但是，该任务的现有数据集主要基于模板，导致了在问题和查询模板之间学习表面映射的模型，而不是开发真正的概括功能。结果，模型在遇到自然措辞，无模板的问题时挣扎。本文介绍了FRASE（基于框架的语义增强），这是一种利用框架语义角色标记（FSRL）来解决此限制的新型方法。我们还提出了LC-Quad 3.0，这是一种源自LC-Quad 2.0的新数据集，其中每个问题都通过FRASE通过帧检测和将框架元素映射到其参数中丰富。我们通过在不同的微调配置下对最近的大语言模型（LLM）（LLM）进行广泛的实验来评估这种方法的影响。我们的结果表明，基于框架的结构化表示形式始终提高SPARQL的生成性能，尤其是在测试问题具有看不见的模板（未知模板拆分）以及它们自然而然的词（重新调整问题）时，尤其是在挑战的概括场景中。

Title: EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices

• Authors: Jiyu Chen, Shuang Peng, Daxiong Luo, Fan Yang, Renshou Wu, Fangyuan Li, Xiaoxin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22196
• Pdf URL: https://arxiv.org/pdf/2503.22196
• Copy Paste: [[2503.22196]] EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices(https://arxiv.org/abs/2503.22196)
• Keywords: language model, llm, long context
• Abstract: Transformer-based large language models (LLMs) encounter challenges in processing long sequences on edge devices due to the quadratic complexity of attention mechanisms and growing memory demands from Key-Value (KV) cache. Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks, while alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructure. We present EdgeInfinite, a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. This approach maintains full compatibility with standard Transformer architectures, requiring fine-tuning only a small part of parameters, and enables selective activation of the memory-gating module for long and short context task routing. The experimental result shows that EdgeInfinite achieves comparable performance to baseline Transformer-based LLM on long context benchmarks while optimizing memory consumption and time to first token.
• 摘要：基于变压器的大型语言模型（LLMS）在处理边缘设备上的长序列时会遇到挑战，这是由于注意机制的二次复杂性和键值（KV）缓存的记忆需求不断增长。现有的KV缓存优化与长输入任务中的不可逆转令牌驱逐作用，而替代序列建模架构在已建立的变压器基础架构中采用昂贵。我们提出了EdgeInfinite，这是一种用于无限上下文的内存效率解决方案，通过可训练的内存门控模块将压缩存储器集成到基于变压器的LLMS中。该方法与标准变压器体系结构保持完整的兼容性，仅需微调参数的一小部分，并可以为长和短上下文任务路由进行选择性激活内存门控模块。实验结果表明，Edgeinfinite在长上下文基准上实现了与基线变压器LLM的可比性能，同时优化了记忆消耗和首次令牌的时间。

Title: MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters

• Authors: Rrubaa Panchendrarajan, Rubén Míguez, Arkaitz Zubiaga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22280
• Pdf URL: https://arxiv.org/pdf/2503.22280
• Copy Paste: [[2503.22280]] MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters(https://arxiv.org/abs/2503.22280)
• Keywords: language model
• Abstract: In the context of fact-checking, claims are often repeated across various platforms and in different languages, which can benefit from a process that reduces this redundancy. While retrieving previously fact-checked claims has been investigated as a solution, the growing number of unverified claims and expanding size of fact-checked databases calls for alternative, more efficient solutions. A promising solution is to group claims that discuss the same underlying facts into clusters to improve claim retrieval and validation. However, research on claim clustering is hindered by the lack of suitable datasets. To bridge this gap, we introduce \textit{MultiClaimNet}, a collection of three multilingual claim cluster datasets containing claims in 86 languages across diverse topics. Claim clusters are formed automatically from claim-matching pairs with limited manual intervention. We leverage two existing claim-matching datasets to form the smaller datasets within \textit{MultiClaimNet}. To build the larger dataset, we propose and validate an approach involving retrieval of approximate nearest neighbors to form candidate claim pairs and an automated annotation of claim similarity using large language models. This larger dataset contains 85.3K fact-checked claims written in 78 languages. We further conduct extensive experiments using various clustering techniques and sentence embedding models to establish baseline performance. Our datasets and findings provide a strong foundation for scalable claim clustering, contributing to efficient fact-checking pipelines.
• 摘要：在事实检查的背景下，通常会在各种平台和不同语言中重复主张，这可以从降低这种冗余的过程中受益。尽管已将检索以前事实检查的索赔作为解决方案进行了研究，但未经验证的索赔和事实检查数据库规模的扩大要求提供替代性，更有效的解决方案。一个有希望的解决方案是小组主张，讨论相同的基本事实，以改善索赔检索和验证。但是，缺乏合适的数据集阻碍了对主张聚类的研究。为了弥合这一差距，我们介绍了\ textit {MultiClaiMnet}，这是一个三个多语言索赔群集数据集的集合，其中包含各种主题的86种语言中的主张。索赔集群是由索赔匹配对自动形成的，手动干预有限。我们利用两个现有的索赔匹配数据集在\ textIt {MultiClimNet}中形成较小的数据集。为了构建较大的数据集，我们建议并验证一种方法，涉及检索大约最近的邻居，以形成候选人索赔对，并使用大语言模型对索赔相似性进行自动注释。这个较大的数据集包含85.3​​k事实检查的主张，该声明以78种语言编写。我们使用各种聚类技术和句子嵌入模型进一步进行广泛的实验，以建立基线性能。我们的数据集和发现为可扩展的主张聚类提供了坚实的基础，这有助于有效的事实检查管道。

Title: Preference-based Learning with Retrieval Augmented Generation for Conversational Question Answering

• Authors: Magdalena Kaiser, Gerhard Weikum
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.22303
• Pdf URL: https://arxiv.org/pdf/2503.22303
• Copy Paste: [[2503.22303]] Preference-based Learning with Retrieval Augmented Generation for Conversational Question Answering(https://arxiv.org/abs/2503.22303)
• Keywords: llm, retrieval augmented generation
• Abstract: Conversational Question Answering (ConvQA) involves multiple subtasks, i) to understand incomplete questions in their context, ii) to retrieve relevant information, and iii) to generate answers. This work presents PRAISE, a pipeline-based approach for ConvQA that trains LLM adapters for each of the three subtasks. As labeled training data for individual subtasks is unavailable in practice, PRAISE learns from its own generations using the final answering performance as feedback signal without human intervention and treats intermediate information, like relevant evidence, as weakly labeled data. We apply Direct Preference Optimization by contrasting successful and unsuccessful samples for each subtask. In our experiments, we show the effectiveness of this training paradigm: PRAISE shows improvements per subtask and achieves new state-of-the-art performance on a popular ConvQA benchmark, by gaining 15.5 percentage points increase in precision over baselines.
• 摘要：会话问题回答（CONSQA）涉及多个子任务，i）在其上下文中了解不完整的问题，ii）检索相关信息，iii）以生成答案。这项工作提出了赞美，这是一种基于管道的Convqa方法，该方法为三个子任务中的每个子任务训练LLM适配器。由于在实践中无法使用标记的单个子任务的培训数据，因此，赞美以最终的答复性能作为反馈信号从其世代中学习，而无需人工干预，并将中间信息（如相关证据）视为弱标记的数据。我们通过对每个子任务的成功样本进行了对比，我们将直接优化优化。在我们的实验中，我们显示了这种训练范式的有效性：赞美显示了每个子任务的改进，并在流行的Convqa基准测试中实现了新的最先进的表现，通过获得15.5个百分点的确定点比基线的精度提高了15.5个百分点。

Title: A Refined Analysis of Massive Activations in LLMs

• Authors: Louis Owen, Nilabhra Roy Chowdhury, Abhay Kumar, Fabian Güra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22329
• Pdf URL: https://arxiv.org/pdf/2503.22329
• Copy Paste: [[2503.22329]] A Refined Analysis of Massive Activations in LLMs(https://arxiv.org/abs/2503.22329)
• Keywords: language model, llm
• Abstract: Motivated in part by their relevance for low-precision training and quantization, massive activations in large language models (LLMs) have recently emerged as a topic of interest. However, existing analyses are limited in scope, and generalizability across architectures is unclear. This paper helps address some of these gaps by conducting an analysis of massive activations across a broad range of LLMs, including both GLU-based and non-GLU-based architectures. Our findings challenge several prior assumptions, most importantly: (1) not all massive activations are detrimental, i.e. suppressing them does not lead to an explosion of perplexity or a collapse in downstream task performance; (2) proposed mitigation strategies such as Attention KV bias are model-specific and ineffective in certain cases. We consequently investigate novel hybrid mitigation strategies; in particular pairing Target Variance Rescaling (TVR) with Attention KV bias or Dynamic Tanh (DyT) successfully balances the mitigation of massive activations with preserved downstream model performance in the scenarios we investigated. Our code is available at: this https URL.
• 摘要：大规模语言模型（LLM）的大规模激活在某种程度上是由于它们与低精度培训和量化的相关性，最近出现了作为感兴趣的主题。但是，现有的分析范围有限，跨体系结构的概括性尚不清楚。本文通过对广泛的LLM（包括基于GLU的基于GLU和非GLU的架构）进行大规模激活的分析来帮助解决其中一些差距。我们的发现挑战了几个先前的假设，最重要的是：（1）并非所有大规模激活都是有害的，即抑制它们并不会导致困惑或下游任务绩效崩溃； （2）在某些情况下，提出的缓解策略（例如注意力KV偏见）是特定于模型的，并且无效。因此，我们研究了新型的混合缓解策略；在我们研究的场景中，尤其是配对目标方差重新缩放（TVR）与注意力KV偏置或动态tanh（DYT）成功平衡了缓解大规模激活与我们研究的场景中保留的下游模型性能。我们的代码可用：此HTTPS URL。

Title: SKDU at De-Factify 4.0: Natural Language Features for AI-Generated Text-Detection

• Authors: Shrikant Malviya, Pablo Arnau-González, Miguel Arevalillo-Herráez, Stamos Katsigiannis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22338
• Pdf URL: https://arxiv.org/pdf/2503.22338
• Copy Paste: [[2503.22338]] SKDU at De-Factify 4.0: Natural Language Features for AI-Generated Text-Detection(https://arxiv.org/abs/2503.22338)
• Keywords: language model, llm, prompt
• Abstract: The rapid advancement of large language models (LLMs) has introduced new challenges in distinguishing human-written text from AI-generated content. In this work, we explored a pipelined approach for AI-generated text detection that includes a feature extraction step (i.e. prompt-based rewriting features inspired by RAIDAR and content-based features derived from the NELA toolkit) followed by a classification module. Comprehensive experiments were conducted on the Defactify4.0 dataset, evaluating two tasks: binary classification to differentiate human-written and AI-generated text, and multi-class classification to identify the specific generative model used to generate the input text. Our findings reveal that NELA features significantly outperform RAIDAR features in both tasks, demonstrating their ability to capture nuanced linguistic, stylistic, and content-based differences. Combining RAIDAR and NELA features provided minimal improvement, highlighting the redundancy introduced by less discriminative features. Among the classifiers tested, XGBoost emerged as the most effective, leveraging the rich feature sets to achieve high accuracy and generalisation.
• 摘要：大型语言模型（LLM）的快速发展引入了新的挑战，以区分人文所写的文本和AI生成的内容。在这项工作中，我们探索了一种用于AI生成的文本检测的管道方法，其中包括功能提取步骤（即受RAIDAR启发的基于及时的基于迅速的重写功能和来自NELA工具包的基于内容的功能），然后是分类模块。在Defactify4.0数据集上进行了全面的实验，评估了两个任务：二进制分类以区分人体编写和AI生成的文本，以及多级分类，以识别用于生成输入文本的特定生成模型。我们的发现表明，NELA在这两个任务中的特征都大大优于RAIDAR功能，证明了它们捕获细微的语言，风格和基于内容的差异的能力。结合RAIDAR和NELA功能提供了最小的改进，突出了较小的判别特征引入的冗余。在测试的分类器中，XGBoost成为最有效的，利用丰富的功能集以实现高精度和泛化。

Title: Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions

• Authors: Yubo Li, Yidi Miao, Xueying Ding, Ramayya Krishnan, Rema Padman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.22353
• Pdf URL: https://arxiv.org/pdf/2503.22353
• Copy Paste: [[2503.22353]] Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions(https://arxiv.org/abs/2503.22353)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities across various tasks, but their deployment in high-stake domains requires consistent performance across multiple interaction rounds. This paper introduces a comprehensive framework for evaluating and improving LLM response consistency, making three key contributions. First, we propose a novel Position-Weighted Consistency (PWC) score that captures both the importance of early-stage stability and recovery patterns in multi-turn interactions. Second, we present a carefully curated benchmark dataset spanning diverse domains and difficulty levels, specifically designed to evaluate LLM consistency under various challenging follow-up scenarios. Third, we introduce Confidence-Aware Response Generation (CARG), a framework that significantly improves response stability by incorporating model confidence signals into the generation process. Empirical results demonstrate that CARG significantly improves response stability without sacrificing accuracy, underscoring its potential for reliable LLM deployment in critical applications.
• 摘要：大型语言模型（LLMS）在各种任务上显示出了出色的功能，但是它们在高级域中的部署需要在多个互动回合中保持一致的性能。本文介绍了一个全面的框架，用于评估和改善LLM响应一致性，从而做出三个关键贡献。首先，我们提出了一种新颖的位置加权一致性（PWC）得分，该得分既捕获了早期稳定性的重要性，又捕获了多转交互作用中的恢复模式的重要性。其次，我们提出了一个精心策划的基准数据集，该数据集涵盖了不同的域和难度级别，该数据集专门为评估LLM在各种具有挑战性的后续方案下的一致性而设计。第三，我们介绍了置信度响应生成（CARG），该框架通过将模型置信信号纳入生成过程来显着提高响应稳定性。经验结果表明，CARG显着提高了响应稳定性而不牺牲准确性，强调了其在关键应用中可靠的LLM部署的潜力。

Title: Supposedly Equivalent Facts That Aren't? Entity Frequency in Pre-training Induces Asymmetry in LLMs

• Authors: Yuan He, Bailan He, Zifeng Ding, Alisia Lupidi, Yuqicheng Zhu, Shuo Chen, Caiqi Zhang, Jiaoyan Chen, Yunpu Ma, Volker Tresp, Ian Horrocks
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22362
• Pdf URL: https://arxiv.org/pdf/2503.22362
• Copy Paste: [[2503.22362]] Supposedly Equivalent Facts That Aren't? Entity Frequency in Pre-training Induces Asymmetry in LLMs(https://arxiv.org/abs/2503.22362)
• Keywords: language model, llm, hallucination
• Abstract: Understanding and mitigating hallucinations in Large Language Models (LLMs) is crucial for ensuring reliable content generation. While previous research has primarily focused on "when" LLMs hallucinate, our work explains "why" and directly links model behaviour to the pre-training data that forms their prior knowledge. Specifically, we demonstrate that an asymmetry exists in the recognition of logically equivalent facts, which can be attributed to frequency discrepancies of entities appearing as subjects versus objects. Given that most pre-training datasets are inaccessible, we leverage the fully open-source OLMo series by indexing its Dolma dataset to estimate entity frequencies. Using relational facts (represented as triples) from Wikidata5M, we construct probing datasets to isolate this effect. Our experiments reveal that facts with a high-frequency subject and a low-frequency object are better recognised than their inverse, despite their logical equivalence. The pattern reverses in low-to-high frequency settings, and no statistically significant asymmetry emerges when both entities are high-frequency. These findings highlight the influential role of pre-training data in shaping model predictions and provide insights for inferring the characteristics of pre-training data in closed or partially closed LLMs.
• 摘要：在大语模型（LLM）中了解和减轻幻觉对于确保可靠的内容产生至关重要。虽然先前的研究主要集中在“何时” LLMS幻觉上，但我们的工作解释了“为什么”，并将模型行为直接与构成其先验知识的训练前数据联系起来。具体而言，我们证明了识别逻辑等效事实的不对称性存在，这可以归因于作为主体与对象的实体的频率差异。鉴于大多数预培训数据集都无法访问，因此我们通过将其DOLMA数据集索引到估计实体频率来利用完全开源的OLMO系列。使用Wikidata5m的关系事实（表示为三元），我们构建探测数据集以隔离此效果。我们的实验表明，尽管具有逻辑等效性，但具有高频主题和低频对象的事实比其反向更好。该模式在低到高的频率设置中逆转，当两个实体都是高频时，没有统计学上的不对称性出现。这些发现突出了预训练数据在塑造模型预测中的影响力，并为推断封闭或部分闭合LLM中训练数据的特征提供了见解。

Title: Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors

• Authors: Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22388
• Pdf URL: https://arxiv.org/pdf/2503.22388
• Copy Paste: [[2503.22388]] Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors(https://arxiv.org/abs/2503.22388)
• Keywords: llm
• Abstract: LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs' debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the this http URL is publicly available at this https URL.
• 摘要：LLM正在转换软件开发，但是当前的代码生成和代码维修基准主要在简单的单纠错案例中评估句法和功能正确性。 LLMS自主查找和修复复杂数据科学代码中运行时逻辑错误的功能仍未得到探索。为了解决这一差距，我们介绍了DSDBench：数据科学调试基准，这是对多跳误差跟踪LLMS系统评估的第一个基准测试和数据科学代码调试中的多BUG检测。 DSDBench从现有的数据科学任务基准（例如Dabench和Matplotbench）中调整数据集，其中包含具有逼真的数据科学调试任务，并具有自动合成的多跳，多重小型代码snippets的任务。 DSDBench包括1,117个带有741个因果误差对和运行时错误消息的带注释的样本。对DSDBENCH上最先进的LLM的评估显示出巨大的性能差距，强调了在数据科学代码中调试逻辑运行时错误时面临的挑战。 DSDBENCH提供了一种至关重要的资源来评估和改善LLMS的调试和推理功能，从而在此HTTPS URL上公开获得了此HTTP URL中更可靠的AI辅助数据科学。

Title: Negation: A Pink Elephant in the Large Language Models' Room?

• Authors: Tereza Vrabcová, Marek Kadlčík, Petr Sojka, Michal Štefánik, Michal Spiegel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22395
• Pdf URL: https://arxiv.org/pdf/2503.22395
• Copy Paste: [[2503.22395]] Negation: A Pink Elephant in the Large Language Models' Room?(https://arxiv.org/abs/2503.22395)
• Keywords: language model, llm
• Abstract: Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language models (LLMs) and remain underexplored. We construct two multilingual natural language inference (NLI) datasets with \textit{paired} examples differing in negation. We investigate how model size and language impact its ability to handle negation correctly by evaluating popular LLMs. Contrary to previous work, we show that increasing the model size consistently improves the models' ability to handle negations. Furthermore, we find that both the models' reasoning accuracy and robustness to negation are language-dependent and that the length and explicitness of the premise have a greater impact on robustness than language. Our datasets can facilitate further research and improvements of language model reasoning in multilingual settings.
• 摘要：否定是确定句子含义的关键，使其对于逻辑推理至关重要。尽管它们的重要性，但否定对大语言模型（LLM）构成了重大挑战，并且仍然没有被忽视。我们构建两个具有\ textit {配对}示例的多语言自然推理（NLI）数据集，否定为否定。我们研究模型大小和语言如何通过评估流行的LLM来影响其正确处理否定的能力。与以前的工作相反，我们表明，增加模型大小会始终提高模型处理否定的能力。此外，我们发现模型的推理准确性和对否定的鲁棒性都是依赖语言的，并且前提的长度和明确性对鲁棒性的影响比语言更大。我们的数据集可以促进多语言设置中语言模型推理的进一步研究和改进。

Title: Long-Tail Crisis in Nearest Neighbor Language Models

• Authors: Yuto Nishida, Makoto Morishita, Hiroyuki Deguchi, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22426
• Pdf URL: https://arxiv.org/pdf/2503.22426
• Copy Paste: [[2503.22426]] Long-Tail Crisis in Nearest Neighbor Language Models(https://arxiv.org/abs/2503.22426)
• Keywords: language model
• Abstract: The $k$-nearest-neighbor language model ($k$NN-LM), one of the retrieval-augmented language models, improves the perplexity for given text by directly accessing a large datastore built from any text data during inference. A widely held hypothesis for the success of $k$NN-LM is that its explicit memory, i.e., the datastore, enhances predictions for long-tail phenomena. However, prior works have primarily shown its ability to retrieve long-tail contexts, leaving the model's performance remain underexplored in estimating the probabilities of long-tail target tokens during inference. In this paper, we investigate the behavior of $k$NN-LM on low-frequency tokens, examining prediction probability, retrieval accuracy, token distribution in the datastore, and approximation error of the product quantization. Our experimental results reveal that $k$NN-LM does not improve prediction performance for low-frequency tokens but mainly benefits high-frequency tokens regardless of long-tail contexts in the datastore.
• 摘要：$ K $ - 纽约语言模型（$ k $ nn-lm）是一种检索功能的语言模型之一，通过直接访问推断期间从任何文本数据构建的大型数据存储来改善给定文本的困惑。 $ k $ nn-lm成功的一个广泛持有的假设是，其明确的内存，即数据存储，可以增强对长尾现象的预测。但是，先前的工作主要表明了其检索长尾上下文的能力，在推理过程中估算长尾目标令牌的概率方面，使模型的性能保持不足。在本文中，我们研究了$ k $ nn-lm在低频代币上的行为，检查了预测概率，检索准确性，数据存储中的令牌分布以及产品量化的近似误差。我们的实验结果表明，$ k $ nn-lm并不能提高低频代币的预测性能，但是无论数据存储中的长尾背景如何，都会有益于高频令牌。

Title: Scaling Laws of Scientific Discovery with AI and Robot Scientists

• Authors: Pengsong Zhang, Heng Zhang, Huazhe Xu, Renjun Xu, Zhenting Wang, Cong Wang, Animesh Garg, Zhibin Li, Arash Ajoudani, Xinyu Liu
• Subjects: cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.22444
• Pdf URL: https://arxiv.org/pdf/2503.22444
• Copy Paste: [[2503.22444]] Scaling Laws of Scientific Discovery with AI and Robot Scientists(https://arxiv.org/abs/2503.22444)
• Keywords: agent
• Abstract: The rapid evolution of scientific inquiry highlights an urgent need for groundbreaking methodologies that transcend the limitations of traditional research. Conventional approaches, bogged down by manual processes and siloed expertise, struggle to keep pace with the demands of modern discovery. We envision an autonomous generalist scientist (AGS) system-a fusion of agentic AI and embodied robotics-that redefines the research lifecycle. This system promises to autonomously navigate physical and digital realms, weaving together insights from disparate disciplines with unprecedented efficiency. By embedding advanced AI and robot technologies into every phase-from hypothesis formulation to peer-ready manuscripts-AGS could slash the time and resources needed for scientific research in diverse field. We foresee a future where scientific discovery follows new scaling laws, driven by the proliferation and sophistication of such systems. As these autonomous agents and robots adapt to extreme environments and leverage a growing reservoir of knowledge, they could spark a paradigm shift, pushing the boundaries of what's possible and ushering in an era of relentless innovation.
• 摘要：科学探究的快速发展凸显了迫切需要超越传统研究局限性的开创性方法。传统的方法被手动流程和孤立的专业知识陷入困境，努力与现代发现的要求保持同步。我们设想了一个自主通才科学家（AGS）系统 - 代理AI的融合和体现的机器人 - 重新定义了研究生命周期。该系统有望自主浏览物理和数字领域，并以前所未有的效率将不同学科的洞察力编织在一起。通过将先进的AI和机器人技术嵌入到同行就绪的手稿中的每个相假设公式中，可能会削减各种领域的科学研究所需的时间和资源。我们预见了一个未来，科学发现遵循了新的扩展定律，这是在这种系统的扩散和复杂性的推动下。随着这些自主的代理商和机器人适应极端环境并利用越来越多的知识库，它们可能会引发范式的转变，从而在无情的创新时代推动了可能和迎接的界限。

Title: Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey

• Authors: Shengyue Guan, Haoyi Xiong, Jindong Wang, Jiang Bian, Bin Zhu, Jian-guang Lou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.22458
• Pdf URL: https://arxiv.org/pdf/2503.22458
• Copy Paste: [[2503.22458]] Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey(https://arxiv.org/abs/2503.22458)
• Keywords: language model, llm, agent
• Abstract: This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines \emph{what to evaluate} and another that explains \emph{how to evaluate}. The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues.
• 摘要：这项调查研究了在多转化对话设置中基于大语言模型（LLM）代理的评估方法。使用Prisma启发的框架，我们系统地审查了近250个学术资源，从各个出版物中捕获了最先进的现状，并为我们的分析建立了坚实的基础。我们的研究通过开发两个相互关联的分类系统提供了一种结构化的方法：一个定义\ emph {what是评估}的方法，另一种解释\ emph {如何评估}的方法。第一个分类法确定了基于LLM的代理的关键组成部分，用于多转交谈及其评估维度，包括任务完成，响应质量，用户体验，内存和上下文保留以及计划和工具集成。这些组件确保以整体和有意义的方式评估对话剂的性能。第二分类系统的重点是评估方法。它将方法分类为基于注释的评估，自动指标，将人类评估与定量措施相结合的混合策略以及利用LLM的自我判断方法。该框架不仅捕获了来自语言理解的传统指标，例如BLEU和Rouge分数，而且还结合了反映多转向对话的动态，互动性质的高级技术。

Title: WorkTeam: Constructing Workflows from Natural Language with Multi-Agents

• Authors: Hanchao Liu, Rongjun Li, Weimin Xiong, Ziyu Zhou, Wei Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22473
• Pdf URL: https://arxiv.org/pdf/2503.22473
• Copy Paste: [[2503.22473]] WorkTeam: Constructing Workflows from Natural Language with Multi-Agents(https://arxiv.org/abs/2503.22473)
• Keywords: language model, llm, agent
• Abstract: Workflows play a crucial role in enhancing enterprise efficiency by orchestrating complex processes with multiple tools or components. However, hand-crafted workflow construction requires expert knowledge, presenting significant technical barriers. Recent advancements in Large Language Models (LLMs) have improved the generation of workflows from natural language instructions (aka NL2Workflow), yet existing single LLM agent-based methods face performance degradation on complex tasks due to the need for specialized knowledge and the strain of task-switching. To tackle these challenges, we propose WorkTeam, a multi-agent NL2Workflow framework comprising a supervisor, orchestrator, and filler agent, each with distinct roles that collaboratively enhance the conversion process. As there are currently no publicly available NL2Workflow benchmarks, we also introduce the HW-NL2Workflow dataset, which includes 3,695 real-world business samples for training and evaluation. Experimental results show that our approach significantly increases the success rate of workflow construction, providing a novel and effective solution for enterprise NL2Workflow services.
• 摘要：通过使用多种工具或组件来协调复杂的过程，工作流在提高企业效率方面起着至关重要的作用。但是，手工制作的工作流构建需要专家知识，并带来了重大的技术障碍。大型语言模型（LLMS）的最新进展改善了自然语言指令（又称NL2Workflow）的工作流程，但由于需要专业知识和任务解决方案的压力，现有的基于单LLM代理的方法面临复杂任务的性能下降。为了应对这些挑战，我们建议Workteam，这是一个多代理NL2Workflow框架，包括主管，编排和填充剂，每个框架都具有独特的角色，可协作增强转换过程。由于目前尚无公开可用的NL2Workflow基准测试，因此我们还介绍了HW-NL2Workflow数据集，其中包括3,695个现实世界的业务样本进行培训和评估。实验结果表明，我们的方法大大提高了工作流的成功率，为企业NL2Workflow服务提供了新颖有效的解决方案。

Title: Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities

• Authors: Raman Dutt, Harleen Hanspal, Guoxuan Xia, Petru-Daniel Tudosiu, Alexander Black, Yongxin Yang, Steven McDonagh, Sarah Parisot
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.22517
• Pdf URL: https://arxiv.org/pdf/2503.22517
• Copy Paste: [[2503.22517]] Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities(https://arxiv.org/abs/2503.22517)
• Keywords: language model, llm
• Abstract: In this work, we undertake the challenge of augmenting the existing generative capabilities of pre-trained text-only large language models (LLMs) with multi-modal generation capability while satisfying two core constraints: C1 preserving the preservation of original language generative capabilities with negligible performance degradation, and C2 adhering to a small parameter budget to learn the new modality, ensuring scalability and efficiency. In contrast to current approaches that add dedicated modules, thereby significantly increasing the parameter count, we propose a method that leverages the underutilized capacity inherent in deep models. Specifically, we exploit the parameter redundancy within Mixture-of-Experts (MoEs) as a source of additional capacity for learning a new modality, enabling better parameter efficiency (C1). Moreover, we preserve the original language generation capabilities by applying low-rank adaptation exclusively to the tokens of the new modality (C2). Furthermore, we introduce a novel parameter initialization scheme based on the Gromov-Wasserstein distance to improve convergence and training stability. Through an extensive analysis of the routing mechanism, we uncover the emergence of modality-specific pathways and decreased redundancy within the experts that can efficiently unlock multi-modal generative capabilities. Overall, our method can be seamlessly applied to a wide range of contemporary LLMs, providing a new pathway for transitioning from uni-modal to multi-modal architectures.
• 摘要：在这项工作中，我们面临着具有具有多模式生成能力的预先培训的仅培训的仅大型语言模型（LLM）的现有生成能力的挑战，同时满足了两个核心约束：C1可以保留原始语言生成能力，并以可忽略的绩效降级，并依赖于小型参数预算，以促进新型的范围，以促进新的参数预算，并以新的范围进行效果，并以新的小参数预算和新的参数效率。与当前添加专用模块的当前方法相反，从而大大增加了参数计数，我们提出了一种利用深层模型中固有的未充分利用能力的方法。具体而言，我们利用了Experts（MOES）中的参数冗余（MOES）中的参数冗余，作为学习新模式的额外能力的来源，从而实现了更好的参数效率（C1）。此外，我们通过将低级别的适应性仅应用于新模式（C2）的代币来保留原始的语言生成能力。此外，我们基于Gromov-Wasserstein距离引入了一种新型的参数初始化方案，以提高收敛性和训练稳定性。通过对路由机制的广泛分析，我们发现了特定于模式特异性途径的出现和专家内的冗余降低，这些途径可以有效解锁多模式生成能力。总体而言，我们的方法可以无缝应用于各种现代LLM，为从单模式变为多模式体系结构提供了新的途径。

Title: Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation

• Authors: Zhuo-Yang Song, Zeyu Li, Qing-Hong Cao, Ming-xing Luo, Hua Xing Zhu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.22547
• Pdf URL: https://arxiv.org/pdf/2503.22547
• Copy Paste: [[2503.22547]] Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation(https://arxiv.org/abs/2503.22547)
• Keywords: language model, llm
• Abstract: The geometric evolution of token representations in large language models (LLMs) presents a fundamental paradox: while human language inherently organizes semantic information in low-dimensional spaces ($\sim 10^1$ dimensions), modern LLMs employ high-dimensional embeddings ($\sim 10^3$ dimensions) processed through Transformer architectures. To resolve this paradox, this work bridges this conceptual gap by developing a geometric framework that tracks token dynamics across Transformers layers. Through layer-wise analysis of intrinsic dimensions across multiple architectures, we reveal an expansion-contraction pattern where tokens diffuse to a "working space" and then progressively project onto lower-dimensional submanifolds. Our finding implies a negative correlation between the working space dimension and parameter-sensitive performance of the LLMs, and indicates that effective models tend to compress tokens into approximately 10-dimensional submanifolds, closely resembling human semantic spaces. This work not only advances LLM interpretability by reframing Transformers layers as projectors that mediate between high-dimensional computation and low-dimensional semantics, but also provides practical tools for model diagnostics that do not rely on task-specific evaluations.
• 摘要：代币表示在大语言模型（LLMS）中的几何演化提出了一个基本悖论：虽然人类语言固有地在低维空间（$ \ sim 10^1 $ dimensions）中组织语义信息，但现代LLMS使用高维嵌入式（$ \ sim \ sim \ sim \ sim 10^3 $ dimensions）通过转换式架构师进行了。为了解决这一悖论，这项工作通过开发一个几何框架来跟踪跨变压器层的令牌动态来弥合这一概念差距。通过对多个体系结构的固有维度进行层次分析，我们揭示了一个扩展 - 收集模式，其中令牌分散到“工作空间”，然后逐渐投射到低维度的子序列上。我们的发现意味着工作空间维度和对参数敏感的性能之间的负相关性，并表明有效的模型倾向于将令牌压缩到大约10维的亚策略中，与人类语义空间非常相似。这项工作不仅可以通过将变压器层重新标记为介导高维计算和低维语义的投影仪来提高LLM的可解释性，而且还为不依赖于特定于任务评估的模型诊断提供了实用的工具。

Title: Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation

• Authors: Sarubi Thillainathan, Songchen Yuan, En-Shiun Annie Lee, Sanath Jayasena, Surangika Ranathunga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22582
• Pdf URL: https://arxiv.org/pdf/2503.22582
• Copy Paste: [[2503.22582]] Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation(https://arxiv.org/abs/2503.22582)
• Keywords: language model, llm
• Abstract: Fine-tuning multilingual sequence-to-sequence large language models (msLLMs) has shown promise in developing neural machine translation (NMT) systems for low-resource languages (LRLs). However, conventional single-stage fine-tuning methods struggle in extremely low-resource NMT settings, where training data is very limited. This paper contributes to artificial intelligence by proposing two approaches for adapting msLLMs in these challenging scenarios: (1) continual pre-training (CPT), where the msLLM is further trained with domain-specific monolingual data to compensate for the under-representation of LRLs, and (2) intermediate task transfer learning (ITTL), a method that fine-tunes the msLLM with both in-domain and out-of-domain parallel data to enhance its translation capabilities across various domains and tasks. As an application in engineering, these methods are implemented in NMT systems for Sinhala, Tamil, and English (six language pairs) in domain-specific, extremely low-resource settings (datasets containing fewer than 100,000 samples). Our experiments reveal that these approaches enhance translation performance by an average of +1.47 bilingual evaluation understudy (BLEU) score compared to the standard single-stage fine-tuning baseline across all translation directions. Additionally, a multi-model ensemble further improves performance by an additional BLEU score.
• 摘要：微调多语言序列到序列大语言模型（MSLLMS）在开发低资源语言（LRLS）的神经机器翻译（NMT）系统方面已显示出希望。但是，在极低的资源NMT设置中，传统的单阶段微调方法在训练数据非常有限。 This paper contributes to artificial intelligence by proposing two approaches for adapting msLLMs in these challenging scenarios: (1) continual pre-training (CPT), where the msLLM is further trained with domain-specific monolingual data to compensate for the under-representation of LRLs, and (2) intermediate task transfer learning (ITTL), a method that fine-tunes the msLLM with both in-domain and不域外数据以增强其在各个域和任务上的翻译功能。作为工程的应用，这些方法是在僧伽罗，泰米尔语和英语（六对语言对）的NMT系统中实现的，该域特定于域特异性，极低的资源设置（包含少于100,000个样本的数据集）。我们的实验表明，与所有翻译方向上标准的单级微调基线相比，这些方法平均通过+1.47双语评估（BLEU）得分提高了翻译性能。此外，多模型合奏进一步提高了性能。

Title: Historical Ink: Exploring Large Language Models for Irony Detection in 19th-Century Spanish

• Authors: Kevin Cohen, Laura Manrique-Gómez, Rubén Manrique
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2503.22585
• Pdf URL: https://arxiv.org/pdf/2503.22585
• Copy Paste: [[2503.22585]] Historical Ink: Exploring Large Language Models for Irony Detection in 19th-Century Spanish(https://arxiv.org/abs/2503.22585)
• Keywords: language model, gpt, llm
• Abstract: This study explores the use of large language models (LLMs) to enhance datasets and improve irony detection in 19th-century Latin American newspapers. Two strategies were employed to evaluate the efficacy of BERT and GPT-4o models in capturing the subtle nuances nature of irony, through both multi-class and binary classification tasks. First, we implemented dataset enhancements focused on enriching emotional and contextual cues; however, these showed limited impact on historical language analysis. The second strategy, a semi-automated annotation process, effectively addressed class imbalance and augmented the dataset with high-quality annotations. Despite the challenges posed by the complexity of irony, this work contributes to the advancement of sentiment analysis through two key contributions: introducing a new historical Spanish dataset tagged for sentiment analysis and irony detection, and proposing a semi-automated annotation methodology where human expertise is crucial for refining LLMs results, enriched by incorporating historical and cultural contexts as core features.
• 摘要：这项研究探讨了大型语言模型（LLM）的使用来增强数据集并改善19世纪拉丁美洲报纸的讽刺检测。采用了两种策略来评估BERT和GPT-4O模型通过多类和二元分类任务捕获具有讽刺意味的微妙性质的功效。首先，我们实施了侧重于丰富情感和上下文提示的数据集增强功能。但是，这些对历史语言分析的影响有限。第二种策略是半自动注释过程，有效地解决了类失衡，并以高质量的注释增强了数据集。尽管讽刺的复杂性构成了挑战，但这项工作通过两种关键贡献促进了情感分析的进步：引入一个新的历史西班牙数据集，标记为情感分析和讽刺检测，并提出半自动的注释方法，其中人类专业知识至关重要，使人类的专业知识至关重要，以完善LLMS的结果，并以历史和文化的核心融合了历史性的核心功能。

Title: Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions

• Authors: Mohammad Almansoori, Komal Kumar, Hisham Cholakkal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.22678
• Pdf URL: https://arxiv.org/pdf/2503.22678
• Copy Paste: [[2503.22678]] Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions(https://arxiv.org/abs/2503.22678)
• Keywords: llm, chain-of-thought, agent
• Abstract: In this work, we introduce MedAgentSim, an open-source simulated clinical environment with doctor, patient, and measurement agents designed to evaluate and enhance LLM performance in dynamic diagnostic settings. Unlike prior approaches, our framework requires doctor agents to actively engage with patients through multi-turn conversations, requesting relevant medical examinations (e.g., temperature, blood pressure, ECG) and imaging results (e.g., MRI, X-ray) from a measurement agent to mimic the real-world diagnostic process. Additionally, we incorporate self improvement mechanisms that allow models to iteratively refine their diagnostic strategies. We enhance LLM performance in our simulated setting by integrating multi-agent discussions, chain-of-thought reasoning, and experience-based knowledge retrieval, facilitating progressive learning as doctor agents interact with more patients. We also introduce an evaluation benchmark for assessing the LLM's ability to engage in dynamic, context-aware diagnostic interactions. While MedAgentSim is fully automated, it also supports a user-controlled mode, enabling human interaction with either the doctor or patient agent. Comprehensive evaluations in various simulated diagnostic scenarios demonstrate the effectiveness of our approach. Our code, simulation tool, and benchmark are available at \href{this https URL}.
• 摘要：在这项工作中，我们介绍了Medagentsim，这是一种开源模拟的临床环境，其医生，患者和测量剂旨在评估和增强动态诊断环境中的LLM性能。与先前的方法不同，我们的框架要求医生代理人通过多转化对话积极与患者互动，要求从测量剂到模拟现实世界诊断过程，要求相关的医学检查（例如温度，血压，ECG）和成像结果（例如MRI，X射线）。此外，我们结合了自我改进机制，使模型可以迭代地完善其诊断策略。我们通过整合多代理的讨论，经过经过经验的推理以及基于经验的知识检索，在模拟环境中提高LLM的性能，随着医生与更多患者的互动，可以促进渐进学习。我们还引入了评估基准，用于评估LLM参与动态，情境感知的诊断相互作用的能力。虽然MedaDentsim是完全自动化的，但它也支持用户控制模式，从而使人类与医生或患者代理商的互动。在各种模拟诊断方案中进行的全面评估证明了我们方法的有效性。我们的代码，仿真工具和基准可在\ href {this https url}上获得。