2025-04-24

Title: FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking

• Authors: Jabez Magomere, Elena Kochkina, Samuel Mensah, Simerjot Kaur, Charese H. Smiley
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.16188
• Pdf URL: https://arxiv.org/pdf/2504.16188
• Copy Paste: [[2504.16188]] FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking(https://arxiv.org/abs/2504.16188)
• Keywords: language model, llm
• Abstract: We introduce FinNLI, a benchmark dataset for Financial Natural Language Inference (FinNLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. Our dataset framework ensures diverse premise-hypothesis pairs while minimizing spurious correlations. FinNLI comprises 21,304 pairs, including a high-quality test set of 3,304 instances annotated by finance experts. Evaluations show that domain shift significantly degrades general-domain NLI performance. The highest Macro F1 scores for pre-trained (PLMs) and large language models (LLMs) baselines are 74.57% and 78.62%, respectively, highlighting the dataset's difficulty. Surprisingly, instruction-tuned financial LLMs perform poorly, suggesting limited generalizability. FinNLI exposes weaknesses in current LLMs for financial reasoning, indicating room for improvement.
• 摘要：我们介绍了Finnli，这是一款用于财务自然语言推断（FINNLI）的基准数据集，涉及SEC申请，年度报告和收益呼叫成绩单等各种财务文本。我们的数据集框架可确保多样化的前提 - 假设对，同时最大程度地减少虚假相关性。 Finnli包括21,304对，其中包括由财务专家注释的3,304个实例的高质量测试。评估表明，域移动显着降低了通用域NLI性能。预训练（PLM）和大语言模型（LLMS）基准的最高宏F1得分分别为74.57％和78.62％，突出了数据集的困难。令人惊讶的是，指导调整的财务LLM的表现不佳，表明可推广性有限。 Finnli在当前的LLM中暴露了财务推理的弱点，这表明改善了空间。

Title: The Paradox of Poetic Intent in Back-Translation: Evaluating the Quality of Large Language Models in Chinese Translation

• Authors: Li Weigang, Pedro Carvalho Brom
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16286
• Pdf URL: https://arxiv.org/pdf/2504.16286
• Copy Paste: [[2504.16286]] The Paradox of Poetic Intent in Back-Translation: Evaluating the Quality of Large Language Models in Chinese Translation(https://arxiv.org/abs/2504.16286)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancement of large language models (LLMs) has reshaped the landscape of machine translation, yet challenges persist in preserving poetic intent, cultural heritage, and handling specialized terminology in Chinese-English translation. This study constructs a diverse corpus encompassing Chinese scientific terminology, historical translation paradoxes, and literary metaphors. Utilizing a back-translation and Friedman test-based evaluation system (BT-Fried), we evaluate BLEU, CHRF, TER, and semantic similarity metrics across six major LLMs (e.g., GPT-4.5, DeepSeek V3) and three traditional translation tools. Key findings include: (1) Scientific abstracts often benefit from back-translation, while traditional tools outperform LLMs in linguistically distinct texts; (2) LLMs struggle with cultural and literary retention, exemplifying the "paradox of poetic intent"; (3) Some models exhibit "verbatim back-translation", reflecting emergent memory behavior; (4) A novel BLEU variant using Jieba segmentation and n-gram weighting is proposed. The study contributes to the empirical evaluation of Chinese NLP performance and advances understanding of cultural fidelity in AI-mediated translation.
• 摘要：大型语言模型（LLM）的快速发展已重塑了机器翻译的景观，但在维护诗意意图，文化遗产和处理专业术语方面的挑战仍然存在于中文英语翻译中。这项研究构建了一个多样化的语料库，其中包括中国科学术语，历史翻译悖论和文学隐喻。利用反向翻译和基于弗里德曼测试的评估系统（BT-Fried），我们评估了六个主要LLM（例如GPT-4.5，DeepSeek V3）和三种传统翻译工具的BLEU，CHRF，TER和语义相似性指标。主要发现包括：（1）科学摘要通常受益于反翻译，而传统工具在语言上不同的文本中的表现都优于LLM； （2）LLM与文化和文学保留斗争，体现了“诗意意图的悖论”； （3）一些模型表现出“逐字化的背面翻译”，反映了新兴的记忆行为； （4）提出了使用Jieba分割和N-Gram加权的新型BLEU变体。该研究有助于对中国NLP表现的经验评估，并进步对AI介导的翻译中文化忠诚度的理解。

Title: Capturing Symmetry and Antisymmetry in Language Models through Symmetry-Aware Training Objectives

• Authors: Zhangdie Yuan, Andreas Vlachos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16312
• Pdf URL: https://arxiv.org/pdf/2504.16312
• Copy Paste: [[2504.16312]] Capturing Symmetry and Antisymmetry in Language Models through Symmetry-Aware Training Objectives(https://arxiv.org/abs/2504.16312)
• Keywords: language model, llm
• Abstract: Capturing symmetric (e.g., country borders another country) and antisymmetric (e.g., parent_of) relations is crucial for a variety of applications. This paper tackles this challenge by introducing a novel Wikidata-derived natural language inference dataset designed to evaluate large language models (LLMs). Our findings reveal that LLMs perform comparably to random chance on this benchmark, highlighting a gap in relational understanding. To address this, we explore encoder retraining via contrastive learning with k-nearest neighbors. The retrained encoder matches the performance of fine-tuned classification heads while offering additional benefits, including greater efficiency in few-shot learning and improved mitigation of catastrophic forgetting.
• 摘要：捕获对称性（例如，国家与另一个国家接壤）和反对称（例如，parent_of）关系对于各种应用至关重要。本文通过引入一种旨在评估大型语言模型（LLM）的新型Wikidata衍生的自然语言推理数据集来应对这一挑战。我们的发现表明，LLMS在此基准测试中的随机机会相当，突出了关系理解的差距。为了解决这个问题，我们通过与k-nearest邻居进行对比学习来探索编码器再培训。再培训编码器与微调分类头的性能相匹配，同时提供了额外的好处，包括在几次学习中提高效率以及改善灾难性遗忘的缓解。

Title: Transformer-Based Extraction of Statutory Definitions from the U.S. Code

• Authors: Arpana Hosabettu (Google), Harsh Shah (Cornell University)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16353
• Pdf URL: https://arxiv.org/pdf/2504.16353
• Copy Paste: [[2504.16353]] Transformer-Based Extraction of Statutory Definitions from the U.S. Code(https://arxiv.org/abs/2504.16353)
• Keywords: language model
• Abstract: Automatic extraction of definitions from legal texts is critical for enhancing the comprehension and clarity of complex legal corpora such as the United States Code (U.S.C.). We present an advanced NLP system leveraging transformer-based architectures to automatically extract defined terms, their definitions, and their scope from the U.S.C. We address the challenges of automatically identifying legal definitions, extracting defined terms, and determining their scope within this complex corpus of over 200,000 pages of federal statutory law. Building upon previous feature-based machine learning methods, our updated model employs domain-specific transformers (Legal-BERT) fine-tuned specifically for statutory texts, significantly improving extraction accuracy. Our work implements a multi-stage pipeline that combines document structure analysis with state-of-the-art language models to process legal text from the XML version of the U.S. Code. Each paragraph is first classified using a fine-tuned legal domain BERT model to determine if it contains a definition. Our system then aggregates related paragraphs into coherent definitional units and applies a combination of attention mechanisms and rule-based patterns to extract defined terms and their jurisdictional scope. The definition extraction system is evaluated on multiple titles of the U.S. Code containing thousands of definitions, demonstrating significant improvements over previous approaches. Our best model achieves 96.8% precision and 98.9% recall (98.2% F1-score), substantially outperforming traditional machine learning classifiers. This work contributes to improving accessibility and understanding of legal information while establishing a foundation for downstream legal reasoning tasks.
• 摘要：从法律文本中自动提取定义对于增强复杂法律语料库（例如《美国法典》（U.S.C.））的理解和清晰度至关重要。我们提出了一个高级NLP系统利用基于变压器的架构，以自动提取定义的术语，其定义以及它们的范围。我们应对自动识别法律定义，提取定义术语并确定其范围的挑战，并在这一复杂的200,000多页的联邦法定法中确定其范围。在以前的基于功能的机器学习方法的基础上，我们的更新模型采用特定领域的变压器（Legal-Bert）精心调整，专门针对法定文本，可显着提高提取精度。我们的工作实现了多个阶段的管道，该管道将文档结构分析与最先进的语言模型相结合，以从美国代码的XML版本中处理法律文本。首先，使用微型法律域BERT模型对每个段落进行分类，以确定其是否包含定义。然后，我们的系统将相关的段落汇总到连贯的定义单元中，并将注意机制和基于规则的模式的组合用于提取定义的术语及其管辖权范围。定义提取系统对包含数千个定义的美国代码的多个标题进行了评估，这表明对以前的方法有了显着改进。我们的最佳模型可实现96.8％的精度和98.9％的召回率（98.2％的F1得分），其表现大大优于传统的机器学习分类器。这项工作有助于改善对法律信息的可访问性和理解，同时为下游法律推理任务建立基础。

Title: Text-to-TrajVis: Enabling Trajectory Data Visualizations from Natural Language Questions

• Authors: Tian Bai, Huiyan Ying, Kailong Suo, Junqiu Wei, Tao Fan, Yuanfeng Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16358
• Pdf URL: https://arxiv.org/pdf/2504.16358
• Copy Paste: [[2504.16358]] Text-to-TrajVis: Enabling Trajectory Data Visualizations from Natural Language Questions(https://arxiv.org/abs/2504.16358)
• Keywords: language model, gpt, llm
• Abstract: This paper introduces the Text-to-TrajVis task, which aims to transform natural language questions into trajectory data visualizations, facilitating the development of natural language interfaces for trajectory visualization systems. As this is a novel task, there is currently no relevant dataset available in the community. To address this gap, we first devised a new visualization language called Trajectory Visualization Language (TVL) to facilitate querying trajectory data and generating visualizations. Building on this foundation, we further proposed a dataset construction method that integrates Large Language Models (LLMs) with human efforts to create high-quality data. Specifically, we first generate TVLs using a comprehensive and systematic process, and then label each TVL with corresponding natural language questions using LLMs. This process results in the creation of the first large-scale Text-to-TrajVis dataset, named TrajVL, which contains 18,140 (question, TVL) pairs. Based on this dataset, we systematically evaluated the performance of multiple LLMs (GPT, Qwen, Llama, etc.) on this task. The experimental results demonstrate that this task is both feasible and highly challenging and merits further exploration within the research community.
• 摘要：本文介绍了文本到Trajvis任务，该任务旨在将自然语言问题转换为轨迹数据可视化，从而促进了轨迹可视化系统的自然语言界面的开发。由于这是一项新颖的任务，因此社区中目前尚无相关数据集。为了解决这一差距，我们首先设计了一种称为轨迹可视化语言（TVL）的新可视化语言，以促进查询轨迹数据并生成可视化。在这个基础的基础上，我们进一步提出了一种数据集构造方法，该方法将大型语言模型（LLMS）与人类创建高质量数据的努力集成在一起。具体来说，我们首先使用全面和系统的过程生成TVL，然后使用LLMS使用相应的自然语言问题标记每个TVL。此过程导致创建了第一个大规模的文本到trajvis数据集，该数据集名为Trajvl，其中包含18,140（问题，TVL）对。基于此数据集，我们系统地评估了此任务上多个LLM（GPT，QWEN，LLAMA等）的性能。实验结果表明，这项任务既可行又极具挑战性，因此在研究界内进一步探索了这项任务。

Title: SplitReason: Learning To Offload Reasoning

• Authors: Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F. AbouElhamayed, Yueying Li, Mohamed S. Abdelfattah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16379
• Pdf URL: https://arxiv.org/pdf/2504.16379
• Copy Paste: [[2504.16379]] SplitReason: Learning To Offload Reasoning(https://arxiv.org/abs/2504.16379)
• Keywords: language model, llm, chain-of-thought
• Abstract: Reasoning in large language models (LLMs) tends to produce substantially longer token generation sequences than simpler language modeling tasks. This extended generation length reflects the multi-step, compositional nature of reasoning and is often correlated with higher solution accuracy. From an efficiency perspective, longer token generation exacerbates the inherently sequential and memory-bound decoding phase of LLMs. However, not all parts of this expensive reasoning process are equally difficult to generate. We leverage this observation by offloading only the most challenging parts of the reasoning process to a larger, more capable model, while performing most of the generation with a smaller, more efficient model; furthermore, we teach the smaller model to identify these difficult segments and independently trigger offloading when needed. To enable this behavior, we annotate difficult segments across 18k reasoning traces from the OpenR1-Math-220k chain-of-thought (CoT) dataset. We then apply supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to a 1.5B-parameter reasoning model, training it to learn to offload the most challenging parts of its own reasoning process to a larger model. This approach improves AIME24 reasoning accuracy by 24% and 28.3% while offloading 1.35% and 5% of the generated tokens respectively. We open-source our SplitReason model, data, code and logs.
• 摘要：与简单的语言建模任务相比，大语言模型（LLM）中的推理往往产生的代币生成序列更长。这种扩展的生成长度反映了推理的多步骤的组成性质，并且通常与较高的溶液精度相关。从效率的角度来看，较长的令牌生成加剧了LLMS固有的顺序和内存解码阶段。但是，并非这种昂贵的推理过程的所有部分都难以生成。我们通过仅将推理过程中最具挑战性的部分卸载到更大，功能更强大的模型中来利用这一观察结果，同时以较小，更高效的模型执行大多数一代。此外，我们教较小的模型来识别这些困难的细分市场，并在需要时独立触发卸载。为了启用这种行为，我们注释了OpenR1-Math-220k Theark（COT）数据集的18K推理痕迹的困难段。然后，我们将监督的微调（SFT）和加强学习微调（RLFT）应用于1.5B参数推理模型，训练它以学习将其自身推理过程中最具挑战性的部分卸载到更大的模型中。这种方法将AIME24推理精度提高了24％和28.3％，同时分别卸载生成的令牌的1.35％和5％。我们开源我们的分季模型，数据，代码和日志。

Title: ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs

• Authors: Fahmida Liza Piya, Rahmatollah Beheshti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16394
• Pdf URL: https://arxiv.org/pdf/2504.16394
• Copy Paste: [[2504.16394]] ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs(https://arxiv.org/abs/2504.16394)
• Keywords: llm
• Abstract: Unstructured clinical data can serve as a unique and rich source of information that can meaningfully inform clinical practice. Extracting the most pertinent context from such data is critical for exploiting its true potential toward optimal and timely decision-making in patient care. While prior research has explored various methods for clinical text summarization, most prior studies either process all input tokens uniformly or rely on heuristic-based filters, which can overlook nuanced clinical cues and fail to prioritize information critical for decision-making. In this study, we propose Contextual, a novel framework that integrates a Context-Preserving Token Filtering method with a Domain-Specific Knowledge Graph (KG) for contextual augmentation. By preserving context-specific important tokens and enriching them with structured knowledge, ConTextual improves both linguistic coherence and clinical fidelity. Our extensive empirical evaluations on two public benchmark datasets demonstrate that ConTextual consistently outperforms other baselines. Our proposed approach highlights the complementary role of token-level filtering and structured retrieval in enhancing both linguistic and clinical integrity, as well as offering a scalable solution for improving precision in clinical text generation.
• 摘要：非结构化的临床数据可以作为一种独特而丰富的信息来源，可以有意义地为临床实践提供信息。从此类数据中提取最相关的环境对于利用其在患者护理中最佳和及时决策的真正潜力至关重要。虽然先前的研究探索了临床文本摘要的各种方法，但大多数先前的研究要么统一地处理所有输入令牌，要么依赖基于启发式的过滤器，这些过滤器可以忽略细微的临床提示，并且无法确定对决策至关重要的信息。在这项研究中，我们提出了上下文，这是一个新颖的框架，该框架将上下文的代币过滤方法与特定领域的知识图（KG）集成在一起，以进行上下文增强。通过保留特定于上下文的重要代币并用结构化知识丰富它们，上下文可以提高语言连贯性和临床保真度。我们对两个公共基准数据集的广泛经验评估表明，上下文始终优于其他基准。我们提出的方法强调了令牌级过滤和结构化检索在增强语言和临床完整性方面的互补作用，并提供了可扩展的解决方案，以提高临床文本生成中的精度。

Title: Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillation

• Authors: Jiahao Yuan, Xingzhe Sun, Xing Yu, Jingwen Wang, Dehui Du, Zhiqing Cui, Zixiang Di
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16408
• Pdf URL: https://arxiv.org/pdf/2504.16408
• Copy Paste: [[2504.16408]] Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillation(https://arxiv.org/abs/2504.16408)
• Keywords: gpt, llm, prompt, agent
• Abstract: The XLLM@ACL2025 Shared Task-III formulates a low-resource structural reasoning task that challenges LLMs to generate interpretable, step-by-step rationales with minimal labeled data. We present Less is More, the third-place winning approach in the XLLM@ACL2025 Shared Task-III, which focuses on structured reasoning from only 24 labeled examples. Our approach leverages a multi-agent framework with reverse-prompt induction, retrieval-augmented reasoning synthesis via GPT-4o, and dual-stage reward-guided filtering to distill high-quality supervision across three subtasks: question parsing, CoT parsing, and step-level verification. All modules are fine-tuned from Meta-Llama-3-8B-Instruct under a unified LoRA+ setup. By combining structure validation with reward filtering across few-shot and zero-shot prompts, our pipeline consistently improves structure reasoning quality. These results underscore the value of controllable data distillation in enhancing structured inference under low-resource constraints. Our code is available at this https URL.
• 摘要：XLLM@ACL2025共享任务-III制定了一个低资源的结构推理任务，该任务挑战LLMS以使用最小的标记数据生成可解释的，逐步的理由。我们更少提出更多是XLLM@ACL2025共享任务-III中的第三名获胜方法，该方法仅从24个标记的示例中介绍了结构化推理。我们的方法利用了一个多代理框架，具有反向预付的诱导，通过GPT-4O检索 - 调查的推理合成以及双阶段奖励引导的过滤，以提炼三个子任务：问题分析，cot解析，解析和级别验证验证。所有模块均在统一的Lora+设置下从Meta-llama-3-8b教学中进行微调。通过将结构验证与在几次射击和零拍的提示中进行奖励过滤结合，我们的管道始终提高结构推理质量。这些结果强调了可控数据蒸馏在在低资源约束下增强结构化推断方面的价值。我们的代码可在此HTTPS URL上找到。

Title: Out-of-the-Box Conditional Text Embeddings from Large Language Models

• Authors: Kosuke Yamada, Peinan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16411
• Pdf URL: https://arxiv.org/pdf/2504.16411
• Copy Paste: [[2504.16411]] Out-of-the-Box Conditional Text Embeddings from Large Language Models(https://arxiv.org/abs/2504.16411)
• Keywords: language model, prompt
• Abstract: Conditional text embedding is a proposed representation that captures the shift in perspective on texts when conditioned on a specific aspect. Previous methods have relied on extensive training data for fine-tuning models, leading to challenges in terms of labor and resource costs. We propose PonTE, a novel unsupervised conditional text embedding method that leverages a causal large language model and a conditional prompt. Through experiments on conditional semantic text similarity and text clustering, we demonstrate that PonTE can generate useful conditional text embeddings and achieve performance comparable to supervised methods without fine-tuning. We also show the interpretability of text embeddings with PonTE by analyzing word generation following prompts and embedding visualization.
• 摘要：有条件的文本嵌入是一种提出的表示，在特定方面的条件下，可以捕获文本的透视图。以前的方法依靠广泛的培训数据来进行微调模型，从而在劳动和资源成本方面面临挑战。我们提出了一种新颖的无监督条件文本嵌入方法Ponte，它利用了因果关系大型语言模型和有条件的提示。通过对条件语义文本相似性和文本聚类的实验，我们证明了Ponte可以生成有用的条件文本嵌入，并实现与无需细调的监督方法相媲美的性能。我们还通过在提示后分析单词生成并嵌入可视化来显示带有Ponte文本嵌入的解释性。

Title: Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study

• Authors: Mohammad Khodadad, Ali Shiraee Kasmaee, Mahdi Astaraki, Nicholas Sherck, Hamidreza Mahyar, Soheila Samiee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16414
• Pdf URL: https://arxiv.org/pdf/2504.16414
• Copy Paste: [[2504.16414]] Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study(https://arxiv.org/abs/2504.16414)
• Keywords: language model, llm
• Abstract: In this study, we introduced a new benchmark consisting of a curated dataset and a defined evaluation process to assess the compositional reasoning capabilities of large language models within the chemistry domain. We designed and validated a fully automated pipeline, verified by subject matter experts, to facilitate this task. Our approach integrates OpenAI reasoning models with named entity recognition (NER) systems to extract chemical entities from recent literature, which are then augmented with external knowledge bases to form a comprehensive knowledge graph. By generating multi-hop questions across these graphs, we assess LLM performance in both context-augmented and non-context augmented settings. Our experiments reveal that even state-of-the-art models face significant challenges in multi-hop compositional reasoning. The results reflect the importance of augmenting LLMs with document retrieval, which can have a substantial impact on improving their performance. However, even perfect retrieval accuracy with full context does not eliminate reasoning errors, underscoring the complexity of compositional reasoning. This work not only benchmarks and highlights the limitations of current LLMs but also presents a novel data generation pipeline capable of producing challenging reasoning datasets across various domains. Overall, this research advances our understanding of reasoning in computational linguistics.
• 摘要：在这项研究中，我们介绍了一个新的基准测试，该基准包括一个策划的数据集和定义的评估过程，以评估化学领域中大语言模型的组成推理能力。我们设计并验证了由主题专家验证的全自动管道，以促进这项任务。我们的方法将OpenAI推理模型与命名实体识别（NER）系统集成在一起，以从最近的文献中提取化学实体，然后将其与外部知识基础增强，以形成全面的知识图。通过在这些图表上产生多跳的问题，我们可以在上下文启动和非上下文增强设置中评估LLM性能。我们的实验表明，即使是最先进的模型，在多跳组成推理中都面临重大挑战。结果反映了通过文件检索增强LLM的重要性，这可能会对提高其性能产生重大影响。但是，即使是完整上下文的完美检索准确性也不会消除推理错误，从而强调了构图推理的复杂性。这项工作不仅基准并强调了当前LLM的局限性，而且还提出了一条新型的数据生成管道，能够在各个领域生成具有挑战性的推理数据集。总体而言，这项研究促进了我们对计算语言学推理的理解。

Title: Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark

• Authors: Hanlei Zhang, Zhuohang Li, Yeshuang Zhu, Hua Xu, Peiwu Wang, Jinchao Zhang, Jie Zhou, Haige Zhu
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.16427
• Pdf URL: https://arxiv.org/pdf/2504.16427
• Copy Paste: [[2504.16427]] Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark(https://arxiv.org/abs/2504.16427)
• Keywords: language model, llm
• Abstract: Multimodal language analysis is a rapidly evolving field that leverages multiple modalities to enhance the understanding of high-level semantics underlying human conversational utterances. Despite its significance, little research has investigated the capability of multimodal large language models (MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA comprises over 61K multimodal utterances drawn from both staged and real-world scenarios, covering six core dimensions of multimodal semantics: intent, emotion, dialogue act, sentiment, speaking style, and communication behavior. We evaluate eight mainstream branches of LLMs and MLLMs using three methods: zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive experiments reveal that even fine-tuned models achieve only about 60%~70% accuracy, underscoring the limitations of current MLLMs in understanding complex human language. We believe that MMLA will serve as a solid foundation for exploring the potential of large language models in multimodal language analysis and provide valuable resources to advance this field. The datasets and code are open-sourced at this https URL.
• 摘要：多模式语言分析是一个快速发展的领域，它利用多种方式来增强对人类对话话语基础的高级语义的理解。尽管具有重要意义，但很少研究研究了多模式大语言模型（MLLM）的能力，以理解认知水平的语义。在本文中，我们介绍了MMLA，这是一种专门旨在解决此差距的综合基准。 MMLA包括从上演和现实世界中绘制的61k多模式的言论，涵盖了多模式语义的六个核心维度：意图，情感，对话行为，情感，讲话风格和沟通行为。我们使用三种方法评估了LLM和MLLM的八个主流分支：零射击推理，监督微调和指令调整。广泛的实验表明，即使微调模型也仅达到60％〜70％的精度，强调了当前MLLM在理解复杂人类语言中的局限性。我们认为，MMLA将是探索多模式分析中大语言模型的潜力并提供宝贵资源来推进这一领域的稳固基础。数据集和代码在此HTTPS URL上开源。

Title: EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records

• Authors: Shuguang Zhao, Qiangzhong Feng, Zhiyang He, Peipei Sun, Yingying Wang, Xiaodong Tao, Xiaoliang Lu, Mei Cheng, Xinyue Wu, Yanyan Wang, Wei Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16448
• Pdf URL: https://arxiv.org/pdf/2504.16448
• Copy Paste: [[2504.16448]] EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records(https://arxiv.org/abs/2504.16448)
• Keywords: language model, prompt
• Abstract: Medical consultation dialogues contain critical clinical information, yet their unstructured nature hinders effective utilization in diagnosis and treatment. Traditional methods, relying on rule-based or shallow machine learning techniques, struggle to capture deep and implicit semantics. Recently, large pre-trained language models and Low-Rank Adaptation (LoRA), a lightweight fine-tuning method, have shown promise for structured information extraction. We propose EMRModel, a novel approach that integrates LoRA-based fine-tuning with code-style prompt design, aiming to efficiently convert medical consultation dialogues into structured electronic medical records (EMRs). Additionally, we construct a high-quality, realistically grounded dataset of medical consultation dialogues with detailed annotations. Furthermore, we introduce a fine-grained evaluation benchmark for medical consultation information extraction and provide a systematic evaluation methodology, advancing the optimization of medical natural language processing (NLP) models. Experimental results show EMRModel achieves an F1 score of 88.1%, improving by49.5% over standard pre-trained models. Compared to traditional LoRA fine-tuning methods, our model shows superior performance, highlighting its effectiveness in structured medical record extraction tasks.
• 摘要：医疗咨询对话包含关键的临床信息，但它们的非结构化性质阻碍了在诊断和治疗中的有效利用。依靠基于规则或浅的机器学习技术的传统方法难以捕获深层和隐性的语义。最近，一种轻巧的微调方法，大型的预训练的语言模型和低级适应性（LORA）已显示出对结构化信息提取的有望。我们提出了EMRMODEL，这是一种新颖的方法，将基于Lora的微调与代码风格的及时设计集成在一起，旨在有效地将医疗咨询对话转换为结构化的电子医疗记录（EMRS）。此外，我们构建了具有详细注释的医学咨询对话的高质量，实际扎根的数据集。此外，我们引入了用于医学咨询信息提取的细粒度评估基准，并提供了系统的评估方法，并推进了医学自然语言处理（NLP）模型的优化。实验结果表明，EMRMODEL的F1得分为88.1％，比标准预训练模型提高了49.5％。与传统的洛拉微调方法相比，我们的模型显示出卓越的性能，突出了其在结构化病历提取任务中的有效性。

Title: QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining

• Authors: Fengze Liu, Weidong Zhou, Binbin Liu, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Xiaohuan Zhou, Taifeng Wang, Yong Cao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16511
• Pdf URL: https://arxiv.org/pdf/2504.16511
• Copy Paste: [[2504.16511]] QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining(https://arxiv.org/abs/2504.16511)
• Keywords: language model, llm
• Abstract: Quality and diversity are two critical metrics for the training data of large language models (LLMs), positively impacting performance. Existing studies often optimize these metrics separately, typically by first applying quality filtering and then adjusting data proportions. However, these approaches overlook the inherent trade-off between quality and diversity, necessitating their joint consideration. Given a fixed training quota, it is essential to evaluate both the quality of each data point and its complementary effect on the overall dataset. In this paper, we introduce a unified data selection framework called QuaDMix, which automatically optimizes the data distribution for LLM pretraining while balancing both quality and diversity. Specifically, we first propose multiple criteria to measure data quality and employ domain classification to distinguish data points, thereby measuring overall diversity. QuaDMix then employs a unified parameterized data sampling function that determines the sampling probability of each data point based on these quality and diversity related labels. To accelerate the search for the optimal parameters involved in the QuaDMix framework, we conduct simulated experiments on smaller models and use LightGBM for parameters searching, inspired by the RegMix method. Our experiments across diverse models and datasets demonstrate that QuaDMix achieves an average performance improvement of 7.2% across multiple benchmarks. These results outperform the independent strategies for quality and diversity, highlighting the necessity and ability to balance data quality and diversity.
• 摘要：质量和多样性是大型语言模型（LLM）培训数据的两个关键指标，从而对性能产生积极影响。现有研究通常通过首先应用质量过滤然后调整数据比例来分别优化这些指标。但是，这些方法忽略了质量和多样性之间固有的权衡，因此需要共同考虑。鉴于固定的培训配额，必须评估每个数据点的质量及其对整体数据集的互补影响。在本文中，我们介绍了一个名为QuadMix的统一数据选择框架，该框架自动优化了LLM预处理的数据分布，同时平衡质量和多样性。具体而言，我们首先提出多个标准来衡量数据质量并采用域分类以区分数据点，从而衡量整体多样性。然后，quadmix采用统一的参数化数据采样函数，该功能根据这些质量和多样性与多样性标签确定每个数据点的采样概率。为了加速搜索QuadMix框架中涉及的最佳参数，我们在较小的型号上进行了模拟实验，并使用LightGBM进行参数搜索，灵感来自Regmix方法。我们跨不同模型和数据集进行的实验表明，QuadMix在多个基准测试中的平均绩效提高了7.2％。这些结果的表现优于质量和多样性的独立策略，强调了平衡数据质量和多样性的必要性和能力。

Title: PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression

• Authors: Lizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, Shiguang NI
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16574
• Pdf URL: https://arxiv.org/pdf/2504.16574
• Copy Paste: [[2504.16574]] PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression(https://arxiv.org/abs/2504.16574)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have achieved remarkable progress, demonstrating unprecedented capabilities across various natural language processing tasks. However, the high costs associated with such exceptional performance limit the widespread adoption of LLMs, highlighting the need for prompt compression. Existing prompt compression methods primarily rely on heuristic truncation or abstractive summarization techniques, which fundamentally overlook the intrinsic mechanisms of LLMs and lack a systematic evaluation of token importance for generation. In this work, we introduce Prompt Importance Sampling (PIS), a novel compression framework that dynamically compresses prompts by sampling important tokens based on the analysis of attention scores of hidden states. PIS employs a dual-level compression mechanism: 1) at the token level, we quantify saliency using LLM-native attention scores and implement adaptive compression through a lightweight 9-layer reinforcement learning (RL) network; 2) at the semantic level, we propose a Russian roulette sampling strategy for sentence-level importance sampling. Comprehensive evaluations across multiple domain benchmarks demonstrate that our method achieves state-of-the-art compression performance. Notably, our framework serendipitously enhances reasoning efficiency through optimized context structuring. This work advances prompt engineering by offering both theoretical grounding and practical efficiency in context management for LLMs.
• 摘要：大型语言模型（LLMS）取得了显着的进步，证明了各种自然语言处理任务中前所未有的功能。但是，与此类出色绩效相关的高成本限制了LLM的广泛采用，强调了迅速压缩的需求。现有的及时压缩方法主要依赖于启发式截断或抽象性摘要技术，从根本上讲，这些技术从根本上忽略了LLM的内在机制，并且缺乏对代币对生成重要性的系统评估。在这项工作中，我们引入了迅速的重要性采样（PIS），这是一个新型的压缩框架，该框架通过基于对隐藏状态的注意力评分的分析来对重要标记进行动态压缩提示。 PI采用双重级压缩机制：1）在令牌级别，我们使用LLM本地注意分数来量化显着性，并通过轻量级的9层增强器学习（RL）网络实现适应性压缩； 2）在语义层面上，我们提出了俄罗斯轮盘采样策略，以实现句子级别的重要性采样。跨多个领域基准的全面评估表明，我们的方法实现了最新的压缩性能。值得注意的是，我们的框架偶然地通过优化的上下文结构来提高推理效率。这项工作通过在LLM的上下文管理中提供理论基础和实践效率来促进工程的发展。

Title: Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: A Pilot Study

• Authors: Andy Li, Wei Zhou, Rashina Hoda, Chris Bain, Peter Poon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16601
• Pdf URL: https://arxiv.org/pdf/2504.16601
• Copy Paste: [[2504.16601]] Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: A Pilot Study(https://arxiv.org/abs/2504.16601)
• Keywords: language model, llm
• Abstract: This study evaluates how well large language models (LLMs) and traditional machine translation (MT) tools translate medical consultation summaries from English into Arabic, Chinese, and Vietnamese. It assesses both patient, friendly and clinician, focused texts using standard automated metrics. Results showed that traditional MT tools generally performed better, especially for complex texts, while LLMs showed promise, particularly in Vietnamese and Chinese, when translating simpler summaries. Arabic translations improved with complexity due to the language's morphology. Overall, while LLMs offer contextual flexibility, they remain inconsistent, and current evaluation metrics fail to capture clinical relevance. The study highlights the need for domain-specific training, improved evaluation methods, and human oversight in medical translation.
• 摘要：这项研究评估了大型语言模型（LLM）和传统的机器翻译（MT）工具如何将医疗咨询摘要从英语转化为阿拉伯语，中文和越南语。它使用标准自动化指标评估了患者，友好和临床医生的重点文本。结果表明，传统的MT工具通常表现更好，尤其是对于复杂的文本，而LLM在翻译更简单的摘要时表现出希望，尤其是在越南语和中文中。由于语言的形态，阿拉伯语翻译的复杂性得到了改善。总体而言，尽管LLM提供了上下文灵活性，但它们仍然不一致，当前的评估指标无法捕获临床相关性。该研究强调了对特定领域的培训，改进的评估方法以及医学翻译中的人类监督的需求。

Title: Debunking with Dialogue? Exploring AI-Generated Counterspeech to Challenge Conspiracy Theories

• Authors: Mareike Lisker, Christina Gottschalk, Helena Mihaljević
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2504.16604
• Pdf URL: https://arxiv.org/pdf/2504.16604
• Copy Paste: [[2504.16604]] Debunking with Dialogue? Exploring AI-Generated Counterspeech to Challenge Conspiracy Theories(https://arxiv.org/abs/2504.16604)
• Keywords: language model, gpt, llm, prompt
• Abstract: Counterspeech is a key strategy against harmful online content, but scaling expert-driven efforts is challenging. Large Language Models (LLMs) present a potential solution, though their use in countering conspiracy theories is under-researched. Unlike for hate speech, no datasets exist that pair conspiracy theory comments with expert-crafted counterspeech. We address this gap by evaluating the ability of GPT-4o, Llama 3, and Mistral to effectively apply counterspeech strategies derived from psychological research provided through structured prompts. Our results show that the models often generate generic, repetitive, or superficial results. Additionally, they over-acknowledge fear and frequently hallucinate facts, sources, or figures, making their prompt-based use in practical applications problematic.
• 摘要：CounterSpeech是反对有害在线内容的关键策略，但扩大专家驱动的努力是具有挑战性的。大型语言模型（LLMS）提出了潜在的解决方案，尽管他们在反对阴谋论中的使用量不足。与仇恨言论不同，没有任何数据集将阴谋论与专家制作的反语评论配对。我们通过评估GPT-4O，Llama 3和Mistral的能力来解决这一差距，以有效地应用通过结构化提示提供的心理学研究得出的反语策略。我们的结果表明，这些模型通常会产生通用，重复或表面的结果。此外，他们过分地恐惧，经常幻觉的事实，来源或数字，使其在实际应用中及时使用问题。

Title: TIFIN India at SemEval-2025: Harnessing Translation to Overcome Multilingual IR Challenges in Fact-Checked Claim Retrieval

• Authors: Prasanna Devadiga, Arya Suneesh, Pawan Kumar Rajpoot, Bharatdeep Hazarika, Aditya U Baliga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16627
• Pdf URL: https://arxiv.org/pdf/2504.16627
• Copy Paste: [[2504.16627]] TIFIN India at SemEval-2025: Harnessing Translation to Overcome Multilingual IR Challenges in Fact-Checked Claim Retrieval(https://arxiv.org/abs/2504.16627)
• Keywords: llm
• Abstract: We address the challenge of retrieving previously fact-checked claims in monolingual and crosslingual settings - a critical task given the global prevalence of disinformation. Our approach follows a two-stage strategy: a reliable baseline retrieval system using a fine-tuned embedding model and an LLM-based reranker. Our key contribution is demonstrating how LLM-based translation can overcome the hurdles of multilingual information retrieval. Additionally, we focus on ensuring that the bulk of the pipeline can be replicated on a consumer GPU. Our final integrated system achieved a success@10 score of 0.938 and 0.81025 on the monolingual and crosslingual test sets, respectively.
• 摘要：我们应对在单语和跨语言环境中检索以前事实检查的主张的挑战 - 鉴于全球虚假信息的普遍性，这是一项至关重要的任务。我们的方法遵循两阶段的策略：使用微型嵌入模型和基于LLM的Reranker的可靠基线检索系统。我们的主要贡献是证明基于LLM的翻译如何克服多语言信息检索的障碍。此外，我们专注于确保可以将大部分管道复制到消费者GPU上。我们的最终集成系统分别在单语和跨语言测试集上取得了10分的成功@10分数和0.81025。

Title: A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics

• Authors: Luisa Shimabucoro, Ahmet Ustun, Marzieh Fadaee, Sebastian Ruder
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16677
• Pdf URL: https://arxiv.org/pdf/2504.16677
• Copy Paste: [[2504.16677]] A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics(https://arxiv.org/abs/2504.16677)
• Keywords: language model
• Abstract: In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a clear understanding of the dynamics that enable cross-lingual transfer remains elusive. This study examines cross-lingual transfer (CLT) dynamics in realistic post-training settings. We study two model families of up to 35B parameters in size trained on carefully controlled mixtures of multilingual data on three generative tasks with varying levels of complexity (summarization, instruction following, and mathematical reasoning) in both single-task and multi-task instruction tuning settings. Overall, we find that the dynamics of cross-lingual transfer and multilingual performance cannot be explained by isolated variables, varying depending on the combination of post-training settings. Finally, we identify the conditions that lead to effective cross-lingual transfer in practice.
• 摘要：为了使大型语言模型在全球范围内都有用，它们经过微调以遵循有关多语言数据的说明。尽管这种训练后的无处不在，但对使跨语言转移的动态有清晰的理解仍然难以捉摸。这项研究检查了现实的训练后设置中的跨语性转移（CLT）动力学。我们研究了两种模型族，其中有35B参数的尺寸，对三种生成任务的多语言混合物进行了训练，该任务具有不同的复杂性（摘要，指导下和数学推理）的不同级别，并在单任务和多任务指令调谐设置中进行了调整。总体而言，我们发现跨语性转移和多语言性能的动力学无法通过隔离变量来解释，这取决于训练后设置的组合。最后，我们确定导致实践中有效跨语性转移的条件。

Title: HEMA : A Hippocampus-Inspired Extended Memory Architecture for Long-Context AI Conversations

• Authors: Kwangseob Ahn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16754
• Pdf URL: https://arxiv.org/pdf/2504.16754
• Copy Paste: [[2504.16754]] HEMA : A Hippocampus-Inspired Extended Memory Architecture for Long-Context AI Conversations(https://arxiv.org/abs/2504.16754)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) struggle with maintaining coherence in extended conversations spanning hundreds of turns, despite performing well within their context windows. This paper introduces HEMA (Hippocampus-Inspired Extended Memory Architecture), a dual-memory system inspired by human cognitive processes. HEMA combines Compact Memory - a continuously updated one-sentence summary preserving global narrative coherence, and Vector Memory - an episodic store of chunk embeddings queried via cosine similarity. When integrated with a 6B-parameter transformer, HEMA maintains coherent dialogues beyond 300 turns while keeping prompt length under 3,500 tokens. Experimental results show substantial improvements: factual recall accuracy increases from 41% to 87%, and human-rated coherence improves from 2.7 to 4.3 on a 5-point scale. With 10K indexed chunks, Vector Memory achieves P@5 >= 0.80 and R@50 >= 0.74, doubling the area under the precision-recall curve compared to summarization-only approaches. Ablation studies reveal two key insights: semantic forgetting through age-weighted pruning reduces retrieval latency by 34% with minimal recall loss, and a two-level summary hierarchy prevents cascade errors in ultra-long conversations exceeding 1,000 turns. HEMA demonstrates that combining verbatim recall with semantic continuity provides a practical solution for privacy-aware conversational AI capable of month-long dialogues without model retraining.
• 摘要：大型语言模型（LLM）努力在跨越数百个回合的扩展对话中保持连贯性，尽管在其上下文窗口中表现良好。本文介绍了HEMA（海马风格的扩展内存体系结构），这是一个受人类认知过程启发的双记忆系统。 Hema结合了紧凑的内存 - 连续更新的一句摘要保存全局叙事连贯性和矢量记忆 - 通过余弦相似性查询的块嵌入的情节存储。当与6B参数变压器集成时，Hema保持了300圈以上的连贯对话，同时将及时长度保持在3500个令牌以下。实验结果显示出实质性的改善：事实召回准确性从41％提高到87％，而人等级的连贯性在5分制中从2.7提高到4.3。使用10K索引块，向量内存达到P@5> = 0.80和R@50> = 0.74，与仅摘要方法相比，Precision-Recall曲线下的面积增加了一倍。消融研究揭示了两个关键的见解：语义忘记通过年龄加权的修剪使检索潜伏期减少了34％，而召回损失最少，而两级摘要层次结构可阻止超长对话中的级联错误超过1,000圈。 Hema表明，将逐字记忆与语义连续性相结合为隐私感知的对话AI提供了一个实用的解决方案，可以在没有模型重新训练的情况下进行长达一个月的对话。

Title: How Effective are Generative Large Language Models in Performing Requirements Classification?

• Authors: Waad Alhoshan, Alessio Ferrari, Liping Zhao
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2504.16768
• Pdf URL: https://arxiv.org/pdf/2504.16768
• Copy Paste: [[2504.16768]] How Effective are Generative Large Language Models in Performing Requirements Classification?(https://arxiv.org/abs/2504.16768)
• Keywords: language model, llm, prompt
• Abstract: In recent years, transformer-based large language models (LLMs) have revolutionised natural language processing (NLP), with generative models opening new possibilities for tasks that require context-aware text generation. Requirements engineering (RE) has also seen a surge in the experimentation of LLMs for different tasks, including trace-link detection, regulatory compliance, and others. Requirements classification is a common task in RE. While non-generative LLMs like BERT have been successfully applied to this task, there has been limited exploration of generative LLMs. This gap raises an important question: how well can generative LLMs, which produce context-aware outputs, perform in requirements classification? In this study, we explore the effectiveness of three generative LLMs-Bloom, Gemma, and Llama-in performing both binary and multi-class requirements classification. We design an extensive experimental study involving over 400 experiments across three widely used datasets (PROMISE NFR, Functional-Quality, and SecReq). Our study concludes that while factors like prompt design and LLM architecture are universally important, others-such as dataset variations-have a more situational impact, depending on the complexity of the classification task. This insight can guide future model development and deployment strategies, focusing on optimising prompt structures and aligning model architectures with task-specific needs for improved performance.
• 摘要：近年来，基于变压器的大型语言模型（LLMS）彻底改变了自然语言处理（NLP），生成模型为需要上下文感知文本生成的任务打开了新的可能性。需求工程（RE）也看到了针对不同任务的LLM的实验激增，包括跟踪链接检测，法规合规性等。需求分类是RE中的常见任务。尽管像BERT这样的非生成LLM已成功地应用于此任务，但对生成LLM的探索有限。这个差距提出了一个重要的问题：生成的LLM可以在需求分类中产生上下文感知的输出？在这项研究中，我们探讨了执行二进制和多类需求分类的三个生成LLMS-blom，Gemma和Llama-In的有效性。我们设计了一项广泛的实验研究，涉及三个广泛使用的数据集（Promist NFR，功能性质量和Secreq）的400多个实验。我们的研究得出的结论是，尽管及时设计和LLM架构等因素普遍重要，但其他因素也像数据集的变化一样具有更大的情境影响，具体取决于分类任务的复杂性。这种见解可以指导未来的模型开发和部署策略，专注于优化及时的结构，并将模型架构与特定于任务的需求保持一致，以提高性能。

Title: MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores

• Authors: Fengwei Zhou, Jiafei Song, Wenjin Jason Li, Gengjian Xue, Zhikang Zhao, Yichao Lu, Bailin Na
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.16786
• Pdf URL: https://arxiv.org/pdf/2504.16786
• Copy Paste: [[2504.16786]] MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores(https://arxiv.org/abs/2504.16786)
• Keywords: language model
• Abstract: Recent advances in large language models have significantly improved their ability to process long-context input, but practical applications are challenged by increased inference time and resource consumption, particularly in resource-constrained environments. To address these challenges, we propose MOOSComp, a token-classification-based long-context compression method that enhances the performance of a BERT-based compressor by mitigating the over-smoothing problem and incorporating outlier scores. In the training phase, we add an inter-class cosine similarity loss term to penalize excessively similar token representations, thereby improving the token classification accuracy. During the compression phase, we introduce outlier scores to preserve rare but critical tokens that are prone to be discarded in task-agnostic compression. These scores are integrated with the classifier's output, making the compressor more generalizable to various tasks. Superior performance is achieved at various compression ratios on long-context understanding and reasoning benchmarks. Moreover, our method obtains a speedup of 3.3x at a 4x compression ratio on a resource-constrained mobile device.
• 摘要：大型语言模型的最新进展已大大提高了其处理长篇小说输入的能力，但是实际应用会受到推理时间和资源消耗的增加挑战，尤其是在资源受限的环境中。为了应对这些挑战，我们提出了MOOSCOMP，这是一种基于令牌分类的长篇小写压缩方法，该方法通过缓解过度光滑的问题并结合了离群值分数来增强基于BERT的压缩机的性能。在训练阶段，我们添加了一个类间余弦相似性损失项，以惩罚过度相似的令牌表示形式，从而提高令牌分类精度。在压缩阶段，我们引入了离群得分，以保留稀有但关键的令牌，这些令牌容易被丢弃在任务不合时宜的压缩中。这些分数与分类器的输出集成在一起，从而使压缩机更具概括性到各种任务。在长篇小说理解和推理基准的情况下，以各种压缩比以各种压缩比实现了卓越的性能。此外，我们的方法在资源约束的移动设备上以4倍的压缩比以3.3倍的速度获得了加速度。

Title: Credible plan-driven RAG method for Multi-hop Question Answering

• Authors: Ningning Zhang, Chi Zhang, Zhizhong Tan, Xingxing Yang, Weiping Deng, Wenyong Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16787
• Pdf URL: https://arxiv.org/pdf/2504.16787
• Copy Paste: [[2504.16787]] Credible plan-driven RAG method for Multi-hop Question Answering(https://arxiv.org/abs/2504.16787)
• Keywords: retrieval-augmented generation
• Abstract: Multi-hop question answering (QA) presents a considerable challenge for Retrieval-Augmented Generation (RAG), requiring the structured decomposition of complex queries into logical reasoning paths and the generation of dependable intermediate results. However, deviations in reasoning paths or errors in intermediate results, which are common in current RAG methods, may propagate and accumulate throughout the reasoning process, diminishing the accuracy of the answer to complex queries. To address this challenge, we propose the Plan-then-Act-and-Review (PAR RAG) framework, which is organized into three key stages: planning, act, and review, and aims to offer an interpretable and incremental reasoning paradigm for accurate and reliable multi-hop question answering by mitigating error this http URL RAG initially applies a top-down problem decomposition strategy, formulating a comprehensive plan that integrates multiple executable steps from a holistic viewpoint. This approach avoids the pitfalls of local optima common in traditional RAG methods, ensuring the accuracy of the entire reasoning path. Subsequently, PAR RAG incorporates a plan execution mechanism based on multi-granularity verification. By utilizing both coarse-grained similarity information and fine-grained relevant data, the framework thoroughly checks and adjusts intermediate results, ensuring process accuracy while effectively managing error propagation and amplification. Experimental results on multi-hop QA datasets demonstrate that the PAR RAG framework substantially outperforms existing state-of-the-art methods in key metrics, including EM and F1 scores.
• 摘要：多跳问题回答（QA）给检索增强生成（RAG）带来了巨大的挑战，需要将复杂查询的结构化分解为逻辑推理路径，并产生可靠的中间结果。但是，在当前的抹布方法中常见的中间结果中的推理路径或错误的偏差可能会在整个推理过程中传播和积累，从而降低了答案对复杂查询的准确性。为了应对这一挑战，我们提出了计划 - 行动和审查（PAR RAG）框架，该框架分为三个关键阶段：规划，行动和审查，并旨在提供可解释的，可解释的和增量的推理范式，以准确且可靠的可靠的多跳跃问题通过使多个策略构成较高的策略来解决此策略，从而使多个策略构成较高的策略。从整体角度来看可执行步骤。这种方法避免了传统的抹布方法中常见的局部优点的陷阱，从而确保了整个推理路径的准确性。随后，PAR RAG结合了基于多粒性验证的计划执行机制。通过利用粗粒度的相似性信息和细粒度相关数据，该框架可以彻底检查并调整中间结果，确保过程准确性，同时有效地管理错误传播和放大。多跳QA数据集的实验结果表明，PAR RAG框架在包括EM和F1分数在内的关键指标中的现有最新方法大大优于现有的最新方法。

Title: LLM-assisted Graph-RAG Information Extraction from IFC Data

• Authors: Sima Iranmanesh, Hadeel Saadany, Edlira Vakaj
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16813
• Pdf URL: https://arxiv.org/pdf/2504.16813
• Copy Paste: [[2504.16813]] LLM-assisted Graph-RAG Information Extraction from IFC Data(https://arxiv.org/abs/2504.16813)
• Keywords: gpt, llm, retrieval-augmented generation
• Abstract: IFC data has become the general building information standard for collaborative work in the construction industry. However, IFC data can be very complicated because it allows for multiple ways to represent the same product information. In this research, we utilise the capabilities of LLMs to parse the IFC data with Graph Retrieval-Augmented Generation (Graph-RAG) technique to retrieve building object properties and their relations. We will show that, despite limitations due to the complex hierarchy of the IFC data, the Graph-RAG parsing enhances generative LLMs like GPT-4o with graph-based knowledge, enabling natural language query-response retrieval without the need for a complex pipeline.
• 摘要：IFC数据已成为建筑行业协作工作的一般建筑信息标准。但是，IFC数据可能非常复杂，因为它允许多种方式表示相同的产品信息。在这项研究中，我们利用LLM的功能通过图检索效果（图形窗格）技术来解析IFC数据，以检索建筑物对象属性及其关系。我们将表明，尽管由于IFC数据的复杂层次结构而造成的局限性，但图形rag解析增强了具有基于图的知识的GPT-4O等生成LLM，从而无需复杂的管道就可以自然语言查询响应检索。

Title: GreenMind: A Next-Generation Vietnamese Large Language Model for Structured and Logical Reasoning

• Authors: Luu Quy Tung, Hoang Quoc Viet, Vo Trong Thu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16832
• Pdf URL: https://arxiv.org/pdf/2504.16832
• Copy Paste: [[2504.16832]] GreenMind: A Next-Generation Vietnamese Large Language Model for Structured and Logical Reasoning(https://arxiv.org/abs/2504.16832)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Chain-of-Thought (CoT) is a robust approach for tackling LLM tasks that require intermediate reasoning steps prior to generating a final answer. In this paper, we present GreenMind-Medium-14B-R1, the Vietnamese reasoning model inspired by the finetuning strategy based on Group Relative Policy Optimization. We also leverage a high-quality Vietnamese synthesized reasoning dataset and design two reward functions to tackle the main limitations of this technique: (i) language mixing, where we explicitly detect the presence of biased language characters during the process of sampling tokens, and (ii) we leverage Sentence Transformer-based models to ensure that the generated reasoning content maintains factual correctness and does not distort the final output. Experimental results on the Vietnamese dataset from the VLSP 2023 Challenge demonstrate that our model outperforms prior works and enhances linguistic consistency in its responses. Furthermore, we extend our evaluation to SeaExam-a multilingual multiple-choice dataset, showing the effectiveness of our reasoning method compared to few-shot prompting techniques.
• 摘要：经过思考链（COT）是解决LLM任务的强大方法，在产生最终答案之前，需要中间推理步骤。在本文中，我们提出了GreenMind-Medium-14b-R1，这是受基于群体相对政策优化的填充策略启发的越南推理模型。我们还利用高质量的越南人合成的推理数据集并设计了两个奖励功能来应对该技术的主要局限性：（i）语言混合，我们明确地检测出偏见的语言字符在采样代币过程中的存在，以及（ii）我们利用基于句子变形金器的模型来确保未经事实的成分和最终的成果，并确保了正确的事实，并确保了最终的成果。 VLSP 2023挑战的越南数据集的实验结果表明，我们的模型表现优于先前的作品，并提高了语言的响应中的语言一致性。此外，我们将评估扩展到Seexam-a多语言多项选择数据集，显示了我们推理方法的有效性与少量发动机提示技术相比。

Title: Monte Carlo Planning with Large Language Model for Text-Based Game Agents

• Authors: Zijing Shi, Meng Fang, Ling Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16855
• Pdf URL: https://arxiv.org/pdf/2504.16855
• Copy Paste: [[2504.16855]] Monte Carlo Planning with Large Language Model for Text-Based Game Agents(https://arxiv.org/abs/2504.16855)
• Keywords: language model, llm, agent
• Abstract: Text-based games provide valuable environments for language-based autonomous agents. However, planning-then-learning paradigms, such as those combining Monte Carlo Tree Search (MCTS) and reinforcement learning (RL), are notably time-consuming due to extensive iterations. Additionally, these algorithms perform uncertainty-driven exploration but lack language understanding and reasoning abilities. In this paper, we introduce the Monte Carlo planning with Dynamic Memory-guided Large language model (MC-DML) algorithm. MC-DML leverages the language understanding and reasoning capabilities of Large Language Models (LLMs) alongside the exploratory advantages of tree search algorithms. Specifically, we enhance LLMs with in-trial and cross-trial memory mechanisms, enabling them to learn from past experiences and dynamically adjust action evaluations during planning. We conduct experiments on a series of text-based games from the Jericho benchmark. Our results demonstrate that the MC-DML algorithm significantly enhances performance across various games at the initial planning phase, outperforming strong contemporary methods that require multiple iterations. This demonstrates the effectiveness of our algorithm, paving the way for more efficient language-grounded planning in complex environments.
• 摘要：基于文本的游戏为基于语言的自主代理提供了宝贵的环境。但是，计划学习范例，例如结合蒙特卡洛树搜索（MCT）和增强学习（RL）的范例，由于广泛的迭代，尤其是耗时的。此外，这些算法执行不确定性驱动的探索，但缺乏语言理解和推理能力。在本文中，我们介绍了具有动态记忆引导的大型语言模型（MC-DML）算法的蒙特卡洛计划。 MC-DML利用大型语言模型（LLM）的语言理解和推理能力以及树木搜索算法的探索性优势。具体而言，我们通过审判和盘问记忆机制增强了LLM，使他们能够从过去的经验中学习并在计划期间动态调整动作评估。我们对耶利哥基准测试的一系列基于文本的游戏进行实验。我们的结果表明，在初始计划阶段，MC-DML算法显着提高了各种游戏的性能，超过了需要多次迭代的强大当代方法。这证明了我们的算法的有效性，为在复杂环境中更有效的语言计划铺平了道路。

Title: Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification

• Authors: Alexander Shvets
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16856
• Pdf URL: https://arxiv.org/pdf/2504.16856
• Copy Paste: [[2504.16856]] Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification(https://arxiv.org/abs/2504.16856)
• Keywords: language model, gpt, llm
• Abstract: Most datasets for sentiment analysis lack context in which an opinion was expressed, often crucial for emotion understanding, and are mainly limited by a few emotion categories. Foundation large language models (LLMs) like GPT-4 suffer from over-predicting emotions and are too resource-intensive. We design an LLM-based data synthesis pipeline and leverage a large model, Mistral-7b, for the generation of training examples for more accessible, lightweight BERT-type encoder models. We focus on enlarging the semantic diversity of examples and propose grounding the generation into a corpus of narratives to produce non-repetitive story-character-centered utterances with unique contexts over 28 emotion classes. By running 700K inferences in 450 GPU hours, we contribute with the dataset of 100K contextual and also 300K context-less examples to cover both scenarios. We use it for fine-tuning pre-trained encoders, which results in several Emo Pillars models. We show that Emo Pillars models are highly adaptive to new domains when tuned to specific tasks such as GoEmotions, ISEAR, IEMOCAP, and EmoContext, reaching the SOTA performance on the first three. We also validate our dataset, conducting statistical analysis and human evaluation, and confirm the success of our measures in utterance diversification (although less for the neutral class) and context personalization, while pointing out the need for improved handling of out-of-taxonomy labels within the pipeline.
• 摘要：大多数用于情感分析的数据集都缺乏表达意见的上下文，通常对于情绪理解至关重要，并且主要受到一些情感类别的限制。基金会大型语言模型（LLMS）等GPT-4（LLM）遭受了过度预测的情绪，并且资源密集度过多。我们设计了一个基于LLM的数据合成管道，并利用大型模型Mismtral-7B来生成培训示例，以实现更容易访问的轻质Bert型编码器模型。我们专注于扩大示例的语义多样性，并提出将一代人扎根于叙事语料库中，以产生以28个情感阶级的独特背景来产生以故事为字象的非重复性故事。通过在450 GPU小时内运行700K推断，我们使用100K上下文和300K上下文示例的数据集进行了贡献，以涵盖这两种情况。我们将其用于微调预训练的编码器，这导致了几种EMO支柱模型。我们表明，在调整诸如GoEmotions，Isear，Iemocap和Emocontext之类的特定任务时，Emo Pillars模型对新域具有高度适应性，并在前三个方面达到了SOTA性能。我们还验证了我们的数据集，进行统计分析和人类评估，并确认我们的措施多样化（尽管中性阶层的较少）和上下文个性化的成功，同时指出了需要改善管道中及外截法标签的需求。

Title: Planning with Diffusion Models for Target-Oriented Dialogue Systems

• Authors: Hanwen Du, Bo Peng, Xia Ning
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16858
• Pdf URL: https://arxiv.org/pdf/2504.16858
• Copy Paste: [[2504.16858]] Planning with Diffusion Models for Target-Oriented Dialogue Systems(https://arxiv.org/abs/2504.16858)
• Keywords: language model, llm
• Abstract: Target-Oriented Dialogue (TOD) remains a significant challenge in the LLM era, where strategic dialogue planning is crucial for directing conversations toward specific targets. However, existing dialogue planning methods generate dialogue plans in a step-by-step sequential manner, and may suffer from compounding errors and myopic actions. To address these limitations, we introduce a novel dialogue planning framework, DiffTOD, which leverages diffusion models to enable non-sequential dialogue planning. DiffTOD formulates dialogue planning as a trajectory generation problem with conditional guidance, and leverages a diffusion language model to estimate the likelihood of the dialogue trajectory. To optimize the dialogue action strategies, DiffTOD introduces three tailored guidance mechanisms for different target types, offering flexible guidance towards diverse TOD targets at test time. Extensive experiments across three diverse TOD settings show that DiffTOD can effectively perform non-myopic lookahead exploration and optimize action strategies over a long horizon through non-sequential dialogue planning, and demonstrates strong flexibility across complex and diverse dialogue scenarios. Our code and data are accessible through this https URL.
• 摘要：在LLM时代，面向目标的对话（TOD）仍然是一个重大挑战，在LLM时代，战略对话计划对于将对话引导到特定目标至关重要。但是，现有的对话计划方法以分步的顺序生成对话计划，并可能遭受更复杂的错误和近视动作。为了解决这些局限性，我们介绍了一个新颖的对话计划框架，即difftod，该框架利用扩散模型来实现非序列对话计划。 DIFFTOD将对话规划作为有条件指导的轨迹生成问题，并利用扩散语言模型来估计对话轨迹的可能性。为了优化对话行动策略，DiFFTOD引入了针对不同目标类型的三种量身定制的指导机制，在测试时为各种TOD目标提供了灵活的指导。在三个不同的TOD环境中进行的广泛实验表明，通过非序列对话计划，DIFFTOD可以有效地执行非侧面的LookAhead探索，并在长时间的地平线上优化行动策略，并在复杂和多样化的对话方案中表现出强大的灵活性。我们的代码和数据可通过此HTTPS URL访问。

Title: Do Large Language Models know who did what to whom?

• Authors: Joseph M. Denning, Xiaohan (Hannah)Guo, Bryor Snefjella, Idan A. Blank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16884
• Pdf URL: https://arxiv.org/pdf/2504.16884
• Copy Paste: [[2504.16884]] Do Large Language Models know who did what to whom?(https://arxiv.org/abs/2504.16884)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) are commonly criticized for not understanding language. However, many critiques focus on cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring who did what to whom (thematic roles) in a sentence. Does the central training objective of LLMs-word prediction-result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LLMs. In contrast to human similarity judgments, in LLMs the overall representational similarity of sentence pairs reflected syntactic similarity but not whether their agent and patient assignments were identical vs. reversed. Furthermore, we found little evidence that thematic role information was available in any subset of hidden units. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LLMs can extract thematic roles but, relative to humans, this information influences their representations more weakly.
• 摘要：大型语言模型（LLM）通常因不了解语言而受到批评。但是，许多批评的重点是人类与语言处理不同的认知能力。在这里，我们研究了一种与语言紧密相关的理解：推断谁在句子中扮演谁（主题角色）。 LLMS字预测的中心培训目标是否在扮演主题角色的句子表示中？在两个实验中，我们表征了四个LLM中的句子表示。与人类的相似性判断相反，在llms中，句子对的总体代表性相似性反映了句法相似性，而不是其代理和患者分配是否相同。此外，我们几乎没有证据表明在隐藏单元的任何子集中都可以使用主题角色信息。但是，一些关注的头脑强烈地抓住了主题角色，独立于语法。因此，LLM可以提取主题角色，但相对于人类，该信息对其表示形式更为微弱。

Title: Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text

• Authors: Shifali Agrahari, Sanasam Ranbir Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16913
• Pdf URL: https://arxiv.org/pdf/2504.16913
• Copy Paste: [[2504.16913]] Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text(https://arxiv.org/abs/2504.16913)
• Keywords: language model, llm, chain-of-thought
• Abstract: In recent years, the detection of AI-generated text has become a critical area of research due to concerns about academic integrity, misinformation, and ethical AI deployment. This paper presents COT Fine-tuned, a novel framework for detecting AI-generated text and identifying the specific language model. responsible for generating the text. We propose a dual-task approach, where Task A involves classifying text as AI-generated or human-written, and Task B identifies the specific LLM behind the text. The key innovation of our method lies in the use of Chain-of-Thought reasoning, which enables the model to generate explanations for its predictions, enhancing transparency and interpretability. Our experiments demonstrate that COT Fine-tuned achieves high accuracy in both tasks, with strong performance in LLM identification and human-AI classification. We also show that the CoT reasoning process contributes significantly to the models effectiveness and interpretability.
• 摘要：近年来，由于对学术完整性，错误信息和道德AI部署的担忧，对AI生成的文本的检测已成为研究的关键领域。本文介绍了COT微调，这是一个新颖的框架，用于检测AI生成的文本并确定特定的语言模型。负责生成文本。我们提出了一种双任务方法，其中任务A涉及将文本分类为AI生成或人工编写，并且任务B标识文本背后的特定LLM。我们方法的关键创新在于使用思想链推理，这使该模型能够为其预测产生解释，从而提高透明度和解释性。我们的实验表明，COT微调在这两个任务中都具有很高的精度，并且在LLM识别和人类AI分类方面具有良好的性能。我们还表明，COT推理过程对模型的有效性和解释性产生了重大贡献。

Title: OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents

• Authors: Raghav Thind, Youran Sun, Ling Liang, Haizhao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.16918
• Pdf URL: https://arxiv.org/pdf/2504.16918
• Copy Paste: [[2504.16918]] OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents(https://arxiv.org/abs/2504.16918)
• Keywords: llm, agent
• Abstract: Optimization plays a vital role in scientific research and practical applications, but formulating a concrete optimization problem described in natural language into a mathematical form and selecting a suitable solver to solve the problem requires substantial domain expertise. We introduce \textbf{OptimAI}, a framework for solving \underline{Optim}ization problems described in natural language by leveraging LLM-powered \underline{AI} agents, achieving superior performance over current state-of-the-art methods. Our framework is built upon four key roles: (1) a \emph{formulator} that translates natural language problem descriptions into precise mathematical formulations; (2) a \emph{planner} that constructs a high-level solution strategy prior to execution; and (3) a \emph{coder} and a \emph{code critic} capable of interacting with the environment and reflecting on outcomes to refine future actions. Ablation studies confirm that all roles are essential; removing the planner or code critic results in $5.8\times$ and $3.1\times$ drops in productivity, respectively. Furthermore, we introduce UCB-based debug scheduling to dynamically switch between alternative plans, yielding an additional $3.3\times$ productivity gain. Our design emphasizes multi-agent collaboration, allowing us to conveniently explore the synergistic effect of combining diverse models within a unified system. Our approach attains 88.1\% accuracy on the NLP4LP dataset and 71.2\% on the Optibench (non-linear w/o table) subset, reducing error rates by 58\% and 50\% respectively over prior best results.
• 摘要：优化在科学研究和实际应用中起着至关重要的作用，但是将自然语言中描述的具体优化问题提出为数学形式，并选择合适的求解器来解决该问题需要实质性的领域专业知识。我们介绍了\ textbf {Optimai}，这是一个解决\下划线{optim}的框架{optim} ization问题，它通过利用llm驱动的\下划线{ai}代理来实现优于当前最新方法的卓越性能。我们的框架建立在四个关键角色上：（1）将自然语言问题描述转化为精确的数学表述； （2）在执行之前构建高级解决方案策略的A \ emph {planner}； （3）A \ Emph {Coder}和A \ Emph {Code Critic}能够与环境进行交互并反思以完善未来动作的结果。消融研究证实，所有角色都是必不可少的。删除规划师或代码评论家的结果分别为$ 5.8 \ times $和$ 3.1 \ times $的生产率。此外，我们介绍了基于UCB的调试计划以在替代计划之间动态切换，从而产生了$ 3.3 \ times $生产率增长。我们的设计强调了多代理协作，使我们可以方便地探索将各种模型组合在统一系统中的协同效果。我们的方法在NLP4LP数据集上达到88.1 \％的准确性，而OptiBench（非线性w/o表）子集的71.2 \％，将错误率分别降低了58 \％\％和50 \％，而不是先前的最佳结果。

Title: IberBench: LLM Evaluation on Iberian Languages

• Authors: José Ángel González, Ian Borrego Obrador, Álvaro Romo Herrero, Areg Mikael Sarvazyan, Mara Chinea-Ríos, Angelo Basile, Marc Franco-Salvador
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.16921
• Pdf URL: https://arxiv.org/pdf/2504.16921
• Copy Paste: [[2504.16921]] IberBench: LLM Evaluation on Iberian Languages(https://arxiv.org/abs/2504.16921)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) remain difficult to evaluate comprehensively, particularly for languages other than English, where high-quality data is often limited. Existing benchmarks and leaderboards are predominantly English-centric, with only a few addressing other languages. These benchmarks fall short in several key areas: they overlook the diversity of language varieties, prioritize fundamental Natural Language Processing (NLP) capabilities over tasks of industrial relevance, and are static. With these aspects in mind, we present IberBench, a comprehensive and extensible benchmark designed to assess LLM performance on both fundamental and industry-relevant NLP tasks, in languages spoken across the Iberian Peninsula and Ibero-America. IberBench integrates 101 datasets from evaluation campaigns and recent benchmarks, covering 22 task categories such as sentiment and emotion analysis, toxicity detection, and summarization. The benchmark addresses key limitations in current evaluation practices, such as the lack of linguistic diversity and static evaluation setups by enabling continual updates and community-driven model and dataset submissions moderated by a committee of experts. We evaluate 23 LLMs ranging from 100 million to 14 billion parameters and provide empirical insights into their strengths and limitations. Our findings indicate that (i) LLMs perform worse on industry-relevant tasks than in fundamental ones, (ii) performance is on average lower for Galician and Basque, (iii) some tasks show results close to random, and (iv) in other tasks LLMs perform above random but below shared task systems. IberBench offers open-source implementations for the entire evaluation pipeline, including dataset normalization and hosting, incremental evaluation of LLMs, and a publicly accessible leaderboard.
• 摘要：大型语言模型（LLM）仍然很难全面评估，特别是对于英语以外的语言，高质量数据通常受到限制。现有的基准和排行榜主要以英语为中心，只有少数语言解决其他语言。这些基准在几个关键领域都缺乏：它们忽略了语言品种的多样性，优先考虑基本自然语言处理（NLP）功能，而不是工业相关任务，并且是静态的。考虑到这些方面，我们提出了Iberbench，这是一种全面且可扩展的基准测试，旨在评估伊比利亚半岛和伊比利亚裔美国人的语言，旨在评估基本和与行业相关的NLP任务的LLM性能。 Iberbench从评估活动和最近的基准中集成了101个数据集，涵盖了22个任务类别，例如情感和情感分析，毒性检测和摘要。基准测试了当前评估实践中的关键局限性，例如缺乏语言多样性和静态评估设置，通过启用不断的更新以及由专家委员会主持的社区驱动模型和数据集提交。我们评估了23个LLM，范围从1亿到140亿参数，并为其优势和局限性提供了经验见解。我们的发现表明，（i）LLM在与行业相关的任务上的表现要比基本任务差，（ii）（ii）加利西亚和巴斯克式的绩效平均表现较低，（iii）某些任务显示了接近随机的结果，而在其他任务中，llms llms llms llms llms llms llms均表现出在随机的随机性上，但在共享任务系统之下。 Iberbench为整个评估管道提供了开源实现，包括数据集归一化和托管，对LLM的增量评估以及公开访问的排行榜。