2025-03-14

Title: Probabilistic Reasoning with LLMs for k-anonymity Estimation

• Authors: Jonathan Zheng, Sauvik Das, Alan Ritter, Wei Xu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.09674
• Pdf URL: https://arxiv.org/pdf/2503.09674
• Copy Paste: [[2503.09674]] Probabilistic Reasoning with LLMs for k-anonymity Estimation(https://arxiv.org/abs/2503.09674)
• Keywords: gpt, llm, retrieval-augmented generation, chain-of-thought
• Abstract: Probabilistic reasoning is a key aspect of both human and artificial intelligence that allows for handling uncertainty and ambiguity in decision-making. In this paper, we introduce a novel numerical reasoning task under uncertainty, focusing on estimating the k-anonymity of user-generated documents containing privacy-sensitive information. We propose BRANCH, which uses LLMs to factorize a joint probability distribution to estimate the k-value-the size of the population matching the given information-by modeling individual pieces of textual information as random variables. The probability of each factor occurring within a population is estimated using standalone LLMs or retrieval-augmented generation systems, and these probabilities are combined into a final k-value. Our experiments show that this method successfully estimates the correct k-value 67% of the time, an 11% increase compared to GPT-4o chain-of-thought reasoning. Additionally, we leverage LLM uncertainty to develop prediction intervals for k-anonymity, which include the correct value in nearly 92% of cases.
• 摘要：概率推理是人工智能的关键方面，它允许处理决策中的不确定性和歧义。在本文中，我们在不确定性下介绍了一项新颖的数值推理任务，重点是估计包含对隐私信息的用户生成的文档的K匿名性。我们提出了分支，该分支使用LLMS分解了联合概率分布，以估计k值的大小 - 与给定信息匹配的人口的大小，将单个文本信息的单个文本信息作为随机变量匹配。使用独立的llms或检索功能增强的生成系统估算每个因素发生在人群中的可能性，并且这些概率合并为最终的K值。我们的实验表明，该方法在67％的时间内成功估计了正确的K值，与GPT-4O链链的推理相比增加了11％。此外，我们利用LLM的不确定性来开发K-匿名性的预测间隔，其中包括近92％的病例中的正确值。

Title: Have LLMs Made Active Learning Obsolete? Surveying the NLP Community

• Authors: Julia Romberg, Christopher Schröder, Julius Gonsior, Katrin Tomanek, Fredrik Olsson
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.09701
• Pdf URL: https://arxiv.org/pdf/2503.09701
• Copy Paste: [[2503.09701]] Have LLMs Made Active Learning Obsolete? Surveying the NLP Community(https://arxiv.org/abs/2503.09701)
• Keywords: language model, llm
• Abstract: Supervised learning relies on annotated data, which is expensive to obtain. A longstanding strategy to reduce annotation costs is active learning, an iterative process, in which a human annotates only data instances deemed informative by a model. Large language models (LLMs) have pushed the effectiveness of active learning, but have also improved methods such as few- or zero-shot learning, and text synthesis - thereby introducing potential alternatives. This raises the question: has active learning become obsolete? To answer this fully, we must look beyond literature to practical experiences. We conduct an online survey in the NLP community to collect previously intangible insights on the perceived relevance of data annotation, particularly focusing on active learning, including best practices, obstacles and expected future developments. Our findings show that annotated data remains a key factor, and active learning continues to be relevant. While the majority of active learning users find it effective, a comparison with a community survey from over a decade ago reveals persistent challenges: setup complexity, estimation of cost reduction, and tooling. We publish an anonymized version of the collected dataset
• 摘要：监督学习取决于注释的数据，这很昂贵。降低注释成本的长期策略是积极学习，这是一个迭代过程，其中人类仅通过模型被视为信息丰富的数据实例。大型语言模型（LLMS）推动了积极学习的有效性，但也改进了诸如少或零学习和文本合成之类的方法，从而引入了潜在的替代方法。这就提出了一个问题：积极学习是否已过时？为了充分回答这一点，我们必须超越文学作用。我们在NLP社区进行了一项在线调查，以收集有关数据注释相关性的先前无形的见解，尤其是专注于积极学习，包括最佳实践，障碍和预期的未来发展。我们的发现表明，注释数据仍然是一个关键因素，并且积极学习仍然是相关的。尽管大多数活跃的学习用户都认为它有效，但与十多年前的社区调查进行了比较揭示了持续的挑战：设置复杂性，降低成本和工具的估计。我们发布了收集的数据集的匿名版本

Title: Review GIDE -- Restaurant Review Gastrointestinal Illness Detection and Extraction with Large Language Models

• Authors: Timothy Laurence, Joshua Harris, Leo Loman, Amy Douglas, Yung-Wai Chan, Luke Hounsome, Lesley Larkin, Michael Borowitz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.09743
• Pdf URL: https://arxiv.org/pdf/2503.09743
• Copy Paste: [[2503.09743]] Review GIDE -- Restaurant Review Gastrointestinal Illness Detection and Extraction with Large Language Models(https://arxiv.org/abs/2503.09743)
• Keywords: language model, llm, prompt
• Abstract: Foodborne gastrointestinal (GI) illness is a common cause of ill health in the UK. However, many cases do not interact with the healthcare system, posing significant challenges for traditional surveillance methods. The growth of publicly available online restaurant reviews and advancements in large language models (LLMs) present potential opportunities to extend disease surveillance by identifying public reports of GI illness. In this study, we introduce a novel annotation schema, developed with experts in GI illness, applied to the Yelp Open Dataset of reviews. Our annotations extend beyond binary disease detection, to include detailed extraction of information on symptoms and foods. We evaluate the performance of open-weight LLMs across these three tasks: GI illness detection, symptom extraction, and food extraction. We compare this performance to RoBERTa-based classification models fine-tuned specifically for these tasks. Our results show that using prompt-based approaches, LLMs achieve micro-F1 scores of over 90% for all three of our tasks. Using prompting alone, we achieve micro-F1 scores that exceed those of smaller fine-tuned models. We further demonstrate the robustness of LLMs in GI illness detection across three bias-focused experiments. Our results suggest that publicly available review text and LLMs offer substantial potential for public health surveillance of GI illness by enabling highly effective extraction of key information. While LLMs appear to exhibit minimal bias in processing, the inherent limitations of restaurant review data highlight the need for cautious interpretation of results.
• 摘要：食源性胃肠道（GI）疾病是英国健康不良的常见原因。但是，许多情况不与医疗保健系统互动，对传统监视方法提出了重大挑战。大型语言模型（LLMS）中公开可用的在线餐厅评论和进步的增长，通过识别胃肠道疾病的公开报告，提出了扩展疾病监测的潜在机会。在这项研究中，我们介绍了一种新颖的注释模式，该模式是由GI疾病专家开发的，应用于Yelp开放的评论数据集。我们的注释超出了二元疾病检测，包括详细提取有关症状和食物的信息。我们在这三个任务中评估了开放重量LLM的性能：胃肠道疾病检测，症状提取和食物提取。我们将这种性能与基于罗伯塔的分类模型进行了比较，专门针对这些任务进行了微调。我们的结果表明，使用基于及时的方法，LLMS在我们所有三个任务中都能达到90％以上的Micro-F1分数。仅使用提示，我们获得了超过较小微调模型的Micro-F1分数。我们进一步证明了在三个以偏见为中心的实验中，LLM在GI疾病检测中的鲁棒性。我们的结果表明，公开可用的审查文本和LLMS通过实现高效提取关键信息来实现对GI疾病的公共卫生监测的巨大潜力。尽管LLM似乎在处理中表现出最小的偏见，但餐厅评论数据的固有局限性强调了对结果的谨慎解释的必要性。

Title: Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment

• Authors: Luyang Fang, Ehsan Latif, Haoran Lu, Yifan Zhou, Ping Ma, Xiaoming Zhai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.09774
• Pdf URL: https://arxiv.org/pdf/2503.09774
• Copy Paste: [[2503.09774]] Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment(https://arxiv.org/abs/2503.09774)
• Keywords: gpt
• Abstract: Automatic scoring of student responses enhances efficiency in education, but deploying a separate neural network for each task increases storage demands, maintenance efforts, and redundant computations. To address these challenges, this paper introduces the Gromov-Wasserstein Scoring Model Merging (GW-SMM) method, which merges models based on feature distribution similarities measured via the Gromov-Wasserstein distance. Our approach begins by extracting features from student responses using individual models, capturing both item-specific context and unique learned representations. The Gromov-Wasserstein distance then quantifies the similarity between these feature distributions, identifying the most compatible models for merging. Models exhibiting the smallest pairwise distances, typically in pairs or trios, are merged by combining only the shared layers preceding the classification head. This strategy results in a unified feature extractor while preserving separate classification heads for item-specific scoring. We validated our approach against human expert knowledge and a GPT-o1-based merging method. GW-SMM consistently outperformed both, achieving a higher micro F1 score, macro F1 score, exact match accuracy, and per-label accuracy. The improvements in micro F1 and per-label accuracy were statistically significant compared to GPT-o1-based merging (p=0.04, p=0.01). Additionally, GW-SMM reduced storage requirements by half without compromising much accuracy, demonstrating its computational efficiency alongside reliable scoring performance.
• 摘要：学生反应的自动评分提高了教育效率，但是为每个任务部署一个单独的神经网络会增加存储需求，维护工作和冗余计算。为了应对这些挑战，本文介绍了Gromov-Wasserstein评分模型合并（GW-SMM）方法，该方法基于通过Gromov-Wasserstein距离测得的特征分布相似性合并模型。我们的方法首先是使用单个模型从学生响应中提取功能，捕获特定于项目的上下文和独特的学习表示。然后，Gromov-Wasserstein距离量化了这些特征分布之间的相似性，从而确定了最兼容的合并模型。通过仅组合分类头之前的共享层，可以合并表现出最小成对距离的模型，通常是成对或三重奏。此策略会导致统一的特征提取器，同时保留单独的分类头进行特定于项目的评分。我们验证了我们违反人类专家知识和基于GPT-O1的合并方法的方法。 GW-SMM始终胜过两者，获得了更高的微型F1分数，宏F1得分，精确匹配的精度和每个标签的精度。与基于GPT-O1的合并相比，微F1和每标签精度的改善在统计学上具有显着意义（P = 0.04，P = 0.01）。此外，GW-SMM将存储需求减少了一半，而不会损害太多准确性，从而证明了其计算效率以及可靠的评分性能。

Title: Constrained Language Generation with Discrete Diffusion Models

• Authors: Michael Cardei, Jacob K Christopher, Thomas Hartvigsen, Brian R. Bartoldson, Bhavya Kailkhura, Ferdinando Fioretto
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.09790
• Pdf URL: https://arxiv.org/pdf/2503.09790
• Copy Paste: [[2503.09790]] Constrained Language Generation with Discrete Diffusion Models(https://arxiv.org/abs/2503.09790)
• Keywords: llm
• Abstract: Constraints are critical in text generation as LLM outputs are often unreliable when it comes to ensuring generated outputs adhere to user defined instruction or general safety guidelines. To address this gap, we present Constrained Discrete Diffusion (CDD), a novel method for enforcing constraints on natural language by integrating discrete diffusion models with differentiable optimization. Unlike conventional text generators, which often rely on post-hoc filtering or model retraining for controllable generation, we propose imposing constraints directly into the discrete diffusion sampling process. We illustrate how this technique can be applied to satisfy a variety of natural language constraints, including (i) toxicity mitigation by preventing harmful content from emerging, (ii) character and sequence level lexical constraints, and (iii) novel molecule sequence generation with specific property adherence. Experimental results show that our constraint-aware procedure achieves high fidelity in meeting these requirements while preserving fluency and semantic coherence, outperforming auto-regressive and existing discrete diffusion approaches.
• 摘要：在文本生成中，约束至关重要，因为在确保生成的输出遵守用户定义的指令或一般安全指南时，LLM输出通常是不可靠的。为了解决这一差距，我们提出了受限制的离散扩散（CDD），这是一种通过将离散扩散模型与可区分优化整合到自然语言上的新方法。与常规的文本生成器不同，通常依赖于事后过滤或模型重新进行可控生成，我们将施加约束直接在离散扩散采样过程中。我们说明了如何应用该技术来满足各种自然语言约束，包括（i）通过防止出现的有害含量来减轻毒性，（ii）特征和（iii）具有特定特性依从性的新型分子序列产生（iii）新型分子序列。实验结果表明，我们的约束过程在满足这些要求方面达到了高保真度，同时保持流利性和语义连贯性，表现优于自动回归和现有的离散扩散方法。

Title: Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval

• Authors: Yuwei Zhang, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.09819
• Pdf URL: https://arxiv.org/pdf/2503.09819
• Copy Paste: [[2503.09819]] Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval(https://arxiv.org/abs/2503.09819)
• Keywords: language model, llm, long context, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) often exhibit substantially shorter effective context lengths than their claimed capacities, especially when handling complex reasoning tasks that require integrating information from multiple parts of a long context and performing multi-step reasoning. Although Chain-of-Thought (CoT) prompting has shown promise in reducing task complexity, our empirical analysis reveals that it does not fully resolve this limitation. Through controlled experiments, we identify poor recall of implicit facts as the primary cause of failure, which significantly hampers reasoning performance. Interestingly, we observe that the internal attention weights from the generated CoT tokens can effectively ground implicit facts, even when these facts are not explicitly recalled. Building on this insight, we propose a novel training-free algorithm, Attrieval, which leverages attention weights to retrieve relevant facts from the long context and incorporates them into the reasoning process. Additionally, we find that selecting context tokens from CoT tokens further improves performance. Our results demonstrate that Attrieval enhances long-context reasoning capability notably on both synthetic and real-world QA datasets with various models.
• 摘要：大型语言模型（LLMS）通常比其声称的能力更短的有效上下文长度，尤其是在处理需要从长篇小说的多个部分集成信息并执行多步推理的复杂推理任务时。尽管经过思考链（COT）提示已显示出在降低任务复杂性方面有希望的，但我们的经验分析表明，它无法完全解决此限制。通过受控的实验，我们确定对隐性事实的不良回忆是失败的主要原因，这显着妨碍了推理性能。有趣的是，我们观察到，即使没有明确召回这些事实，也可以有效地理解产生的COT令牌的内部注意力权重有效地理解隐性事实。在这种见解的基础上，我们提出了一种新颖的无培训算法，即归纳，该算法利用注意力的权重从漫长的上下文中检索相关事实，并将其纳入推理过程中。此外，我们发现从COT令牌中选择上下文令牌进一步提高了性能。我们的结果表明，具有各种模型的属性在合成和现实世界中的QA数据集上尤其增强了长篇下说推理能力。

Title: Generative AI for Named Entity Recognition in Low-Resource Language Nepali

• Authors: Sameer Neupane (University of Memphis), Jeevan Chapagain (University of Memphis), Nobal B. Niraula (Nowa Lab), Diwa Koirala (Nowa Lab)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09822
• Pdf URL: https://arxiv.org/pdf/2503.09822
• Copy Paste: [[2503.09822]] Generative AI for Named Entity Recognition in Low-Resource Language Nepali(https://arxiv.org/abs/2503.09822)
• Keywords: language model, llm, prompt
• Abstract: Generative Artificial Intelligence (GenAI), particularly Large Language Models (LLMs), has significantly advanced Natural Language Processing (NLP) tasks, such as Named Entity Recognition (NER), which involves identifying entities like person, location, and organization names in text. LLMs are especially promising for low-resource languages due to their ability to learn from limited data. However, the performance of GenAI models for Nepali, a low-resource language, has not been thoroughly evaluated. This paper investigates the application of state-of-the-art LLMs for Nepali NER, conducting experiments with various prompting techniques to assess their effectiveness. Our results provide insights into the challenges and opportunities of using LLMs for NER in low-resource settings and offer valuable contributions to the advancement of NLP research in languages like Nepali.
• 摘要：生成人工智能（Genai），尤其是大型语言模型（LLMS），具有明显高级的自然语言处理（NLP）任务，例如命名实体识别（NER），涉及识别诸如文本中的人，位置和组织名称之类的实体。对于低资源语言，LLM尤其有希望，因为它们能够从有限的数据中学习。但是，尚未对尼泊尔语的吉奈模型（一种低资源语言）的性能进行彻底评估。本文调查了最先进的LLM在尼泊尔的应用，并使用各种提示技术进行了实验以评估其有效性。我们的结果提供了对在低资源环境中使用LLM为NER使用LLM的挑战和机遇的见解，并为NLP研究的发展提供了宝贵的贡献。

Title: Who Are You Behind the Screen? Implicit MBTI and Gender Detection Using Artificial Intelligence

• Authors: Kourosh Shahnazari, Seyed Moein Ayyoubzadeh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09853
• Pdf URL: https://arxiv.org/pdf/2503.09853
• Copy Paste: [[2503.09853]] Who Are You Behind the Screen? Implicit MBTI and Gender Detection Using Artificial Intelligence(https://arxiv.org/abs/2503.09853)
• Keywords: language model
• Abstract: In personalized technology and psychological research, precisely detecting demographic features and personality traits from digital interactions becomes ever more important. This work investigates implicit categorization, inferring personality and gender variables directly from linguistic patterns in Telegram conversation data, while conventional personality prediction techniques mostly depend on explicitly self-reported labels. We refine a Transformer-based language model (RoBERTa) to capture complex linguistic cues indicative of personality traits and gender differences using a dataset comprising 138,866 messages from 1,602 users annotated with MBTI types and 195,016 messages from 2,598 users annotated with gender. Confidence levels help to greatly raise model accuracy to 86.16\%, hence proving RoBERTa's capacity to consistently identify implicit personality types from conversational text data. Our results highlight the usefulness of Transformer topologies for implicit personality and gender classification, hence stressing their efficiency and stressing important trade-offs between accuracy and coverage in realistic conversational environments. With regard to gender classification, the model obtained an accuracy of 74.4\%, therefore capturing gender-specific language patterns. Personality dimension analysis showed that people with introverted and intuitive preferences are especially more active in text-based interactions. This study emphasizes practical issues in balancing accuracy and data coverage as Transformer-based models show their efficiency in implicit personality and gender prediction tasks from conversational texts.
• 摘要：在个性化技术和心理学研究中，精确检测数字互动的人口特征和人格特征变得越来越重要。这项工作调查了隐式分类，直接从电报对话数据中的语言模式中推断性格和性别变量，而传统的人格预测技术主要取决于明确自我报告的标签。我们完善了基于变压器的语言模型（Roberta），以捕获复杂的语言提示，该模型使用包含1,602个用户注释的1,602个用户的138,866条消息的数据集和195,016条来自2,598个用户的消息，该数据集中有138,866条消息，来自2,598个用户。置信度水平有助于将模型准确性大大提高到86.16 \％，因此证明了罗伯塔能够从对话文本数据中始终如一地识别隐性人格类型的能力。我们的结果突出了变压器拓扑对于隐式性格和性别分类的有用性，因此强调了它们的效率，并强调了在现实的对话环境中的准确性和覆盖范围之间的重要权衡。关于性别分类，该模型的准确性为74.4 \％，因此捕获了特定于性别的语言模式。人格维度分析表明，具有内向和直观偏好的人在基于文本的互动中特别活跃。这项研究强调了平衡准确性和数据覆盖范围的实际问题，因为基于变压器的模型表明了它们在对话文本中的隐性人格和性别预测任务方面的效率。

Title: What's In Your Field? Mapping Scientific Research with Knowledge Graphs and Large Language Models

• Authors: Abhipsha Das, Nicholas Lourie, Siavash Golkar, Mariel Pettee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.09894
• Pdf URL: https://arxiv.org/pdf/2503.09894
• Copy Paste: [[2503.09894]] What's In Your Field? Mapping Scientific Research with Knowledge Graphs and Large Language Models(https://arxiv.org/abs/2503.09894)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: The scientific literature's exponential growth makes it increasingly challenging to navigate and synthesize knowledge across disciplines. Large language models (LLMs) are powerful tools for understanding scientific text, but they fail to capture detailed relationships across large bodies of work. Unstructured approaches, like retrieval augmented generation, can sift through such corpora to recall relevant facts; however, when millions of facts influence the answer, unstructured approaches become cost prohibitive. Structured representations offer a natural complement -- enabling systematic analysis across the whole corpus. Recent work enhances LLMs with unstructured or semistructured representations of scientific concepts; to complement this, we try extracting structured representations using LLMs. By combining LLMs' semantic understanding with a schema of scientific concepts, we prototype a system that answers precise questions about the literature as a whole. Our schema applies across scientific fields and we extract concepts from it using only 20 manually annotated abstracts. To demonstrate the system, we extract concepts from 30,000 papers on arXiv spanning astrophysics, fluid dynamics, and evolutionary biology. The resulting database highlights emerging trends and, by visualizing the knowledge graph, offers new ways to explore the ever-growing landscape of scientific knowledge. Demo: abby101/surveyor-0 on HF Spaces. Code: this https URL.
• 摘要：科学文献的指数增长使得跨学科的知识变得越来越具有挑战性。大型语言模型（LLMS）是理解科学文本的强大工具，但它们无法捕获大型工作的详细关系。非结构化的方法，例如检索增强产生，可以通过此类语料库进行筛选以回顾相关事实。但是，当数百万事实影响答案时，非结构化的方法变得越来越高。结构化表示提供了自然的补充 - 在整个语料库中实现系统分析。最近的工作通过科学概念的非结构化或半结构化表示增强了LLM。为了补充这一点，我们尝试使用LLM提取结构化表示。通过将LLMS的语义理解与科学概念的模式相结合，我们原型制作了一个系统，该系统回答了整个文献的精确问题。我们的模式适用于跨科学领域，我们仅使用20个手动注释的摘要从中提取概念。为了展示系统，我们从30,000篇有关植入天体物理学，流体动力学和进化生物学的论文中提取概念。由此产生的数据库突出了新兴趋势，并通过可视化知识图，提供了探索不断增长的科学知识景观的新方法。演示：HF空间上的Abby101/Surveyor-0。代码：此HTTPS URL。

Title: Information Density Principle for MLLM Benchmarks

• Authors: Chunyi Li, Xiaozhe Li, Zicheng Zhang, Yuan Tian, Ziheng Jia, Xiaohong Liu, Xiongkuo Min, Jia Wang, Haodong Duan, Kai Chen, Guangtao Zhai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10079
• Pdf URL: https://arxiv.org/pdf/2503.10079
• Copy Paste: [[2503.10079]] Information Density Principle for MLLM Benchmarks(https://arxiv.org/abs/2503.10079)
• Keywords: language model, llm
• Abstract: With the emergence of Multimodal Large Language Models (MLLMs), hundreds of benchmarks have been developed to ensure the reliability of MLLMs in downstream tasks. However, the evaluation mechanism itself may not be reliable. For developers of MLLMs, questions remain about which benchmark to use and whether the test results meet their requirements. Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs. We characterize it from four key dimensions: (1) Fallacy, (2) Difficulty, (3) Redundancy, (4) Diversity. Through a comprehensive analysis of more than 10,000 samples, we measured the information density of 19 MLLM benchmarks. Experiments show that using the latest benchmarks in testing can provide more insight compared to previous ones, but there is still room for improvement in their information density. We hope this principle can promote the development and application of future MLLM benchmarks. Project page: this https URL
• 摘要：随着多模式大语言模型（MLLM）的出现，已经开发了数百个基准，以确保MLLM在下游任务中的可靠性。但是，评估机制本身可能并不可靠。对于MLLM的开发人员，有关使用哪种基准以及测试结果是否满足其要求的问题。因此，我们提出了信息密度的关键原则，该原则研究了基准可以为MLLM的发展提供多少洞察力。我们从四个关键维度来表征它：（1）谬误，（2）难度，（3）冗余，（4）多样性。通过对10,000多个样本的全面分析，我们测量了19 mllm基准的信息密度。实验表明，与以前的测试相比，在测试中使用最新的基准可以提供更多的见解，但是仍然可以提高其信息密度。我们希望这一原则可以促进未来MLLM基准的发展和应用。项目页面：此HTTPS URL

Title: Why Does Your CoT Prompt (Not) Work? Theoretical Analysis of Prompt Space Complexity, its Interaction with Answer Space During CoT Reasoning with LLMs: A Recurrent Perspective

• Authors: Xiang Zhang, Juntai Cao, Jiaqi Wei, Chenyu You, Dujian Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10084
• Pdf URL: https://arxiv.org/pdf/2503.10084
• Copy Paste: [[2503.10084]] Why Does Your CoT Prompt (Not) Work? Theoretical Analysis of Prompt Space Complexity, its Interaction with Answer Space During CoT Reasoning with LLMs: A Recurrent Perspective(https://arxiv.org/abs/2503.10084)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Despite the remarkable successes of Large Language Models (LLMs), their fundamental Transformer architecture possesses inherent theoretical limitations that restrict their capability to handle reasoning tasks with increasing computational complexity. Chain-of-Thought (CoT) prompting has emerged as a practical solution, supported by several theoretical studies. However, current CoT-based methods (including ToT, GoT, etc.) generally adopt a "one-prompt-fits-all" strategy, using fixed templates (e.g., "think step by step") across diverse reasoning tasks. This method forces models to navigate an extremely complex prompt space to identify effective reasoning paths. The current prompt designing research are also heavily relying on trial-and-error rather than theoretically informed guidance. In this paper, we provide a rigorous theoretical analysis of the complexity and interplay between two crucial spaces: the prompt space (the space of potential prompt structures) and the answer space (the space of reasoning solutions generated by LLMs) in CoT reasoning. We demonstrate how reliance on a single universal prompt (e.g. think step by step) can negatively impact the theoretical computability of LLMs, illustrating that prompt complexity directly influences the structure and effectiveness of the navigation in answer space. Our analysis highlights that sometimes human supervision is critical for efficiently navigating the prompt space. We theoretically and empirically show that task-specific prompting significantly outperforms unsupervised prompt generation, emphasizing the necessity of thoughtful human guidance in CoT prompting.
• 摘要：尽管大语言模型（LLMS）取得了显着的成功，但其基本变压器架构具有固有的理论局限性，这些局限性限制了其能够以增加计算复杂性来处理推理任务的能力。经过多个理论研究支持的实用解决方案已经出现了一项思维链（COT）的提示。但是，当前基于COT的方法（包括TOT，GOT等）通常使用固定的模板（例如，“思考逐步思考”）跨不同的推理任务采用“单次合适”策略。该方法迫使模型浏览一个极其复杂的提示空间，以识别有效的推理路径。当前的及时设计研究也很大程度上依赖于反复试验，而不是理论上知情的指导。在本文中，我们对两个关键空间之间的复杂性和相互作用进行了严格的理论分析：及时空间（潜在及时结构的空间）和在COT推理中产生的答案空间（LLMS产生的推理解决方案的空间）。我们证明了对单个通用提示的依赖（例如，逐步思考）如何对LLM的理论计算产生负面影响，这说明了提示复杂性直接影响答案空间中导航的结构和有效性。我们的分析强调，有时人类的监督对于有效地导航及时空间至关重要。从理论上讲，我们从经验上表明，特定于任务的提示明显优于无监督的及时生成，强调了在COT提示中进行周到的人类指导的必要性。

Title: Representation-based Reward Modeling for Efficient Safety Alignment of Large Language Model

• Authors: Qiyuan Deng, Xuefeng Bai, Kehai Chen, Yaowei Wang, Liqiang Nie, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10093
• Pdf URL: https://arxiv.org/pdf/2503.10093
• Copy Paste: [[2503.10093]] Representation-based Reward Modeling for Efficient Safety Alignment of Large Language Model(https://arxiv.org/abs/2503.10093)
• Keywords: language model, llm
• Abstract: Reinforcement Learning (RL) algorithms for safety alignment of Large Language Models (LLMs), such as Direct Preference Optimization (DPO), encounter the challenge of distribution shift. Current approaches typically address this issue through online sampling from the target policy, which requires significant computational resources. In this paper, we hypothesize that during off-policy training, while the ranking order of output generated by policy changes, their overall distribution remains relatively stable. This stability allows the transformation of the sampling process from the target policy into a re-ranking of preference data. Building on this hypothesis, We propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preferences reordering. Extensive experimental results and theoretical analysis demonstrate that the proposed method effectively addresses the distribution shift issue, remarkably enhancing the safety performance while reducing about 300x computational overheads.
• 摘要：大型语言模型（LLMS）安全对齐（例如直接偏好优化（DPO））的强化学习（RL）算法遇到了分配转移的挑战。当前方法通常通过目标策略的在线抽样解决此问题，这需要大量的计算资源。在本文中，我们假设在政策培训期间，而政策变化产生的排名顺序，其整体分布仍然相对稳定。这种稳定性允许将采样过程从目标策略转换为优先数据的重新排序。在此假设的基础上，我们提出了一个新框架，该框架利用该模型的内在安全判断能力提取奖励信号，然后将其用于计算重新排序的偏好的标签信心。广泛的实验结果和理论分析表明，所提出的方法有效地解决了分布移位问题，从而显着提高了安全性能，同时降低了约300倍的计算开销。

Title: Cognitive-Mental-LLM: Leveraging Reasoning in Large Language Models for Mental Health Prediction via Online Text

• Authors: Avinash Patil, Amardeep Kour Gedhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10095
• Pdf URL: https://arxiv.org/pdf/2503.10095
• Copy Paste: [[2503.10095]] Cognitive-Mental-LLM: Leveraging Reasoning in Large Language Models for Mental Health Prediction via Online Text(https://arxiv.org/abs/2503.10095)
• Keywords: language model, llm, prompt, chain-of-thought, tree-of-thought
• Abstract: Large Language Models (LLMs) have demonstrated potential in predicting mental health outcomes from online text, yet traditional classification methods often lack interpretability and robustness. This study evaluates structured reasoning techniques-Chain-of-Thought (CoT), Self-Consistency (SC-CoT), and Tree-of-Thought (ToT)-to improve classification accuracy across multiple mental health datasets sourced from Reddit. We analyze reasoning-driven prompting strategies, including Zero-shot CoT and Few-shot CoT, using key performance metrics such as Balanced Accuracy, F1 score, and Sensitivity/Specificity. Our findings indicate that reasoning-enhanced techniques improve classification performance over direct prediction, particularly in complex cases. Compared to baselines such as Zero Shot non-CoT Prompting, and fine-tuned pre-trained transformers such as BERT and Mental-RoBerta, and fine-tuned Open Source LLMs such as Mental Alpaca and Mental-Flan-T5, reasoning-driven LLMs yield notable gains on datasets like Dreaddit (+0.52\% over M-LLM, +0.82\% over BERT) and SDCNL (+4.67\% over M-LLM, +2.17\% over BERT). However, performance declines in Depression Severity, and CSSRS predictions suggest dataset-specific limitations, likely due to our using a more extensive test set. Among prompting strategies, Few-shot CoT consistently outperforms others, reinforcing the effectiveness of reasoning-driven LLMs. Nonetheless, dataset variability highlights challenges in model reliability and interpretability. This study provides a comprehensive benchmark of reasoning-based LLM techniques for mental health text classification. It offers insights into their potential for scalable clinical applications while identifying key challenges for future improvements.
• 摘要：大型语言模型（LLM）表现出了预测在线文本中心理健康结果的潜力，但是传统的分类方法通常缺乏可解释性和鲁棒性。这项研究评估了结构化的推理技术 - 思想链（COT），自矛盾（SC-COT）和经营树（TOT），以提高来自Reddit的多个心理健康数据集的分类准确性。我们使用诸如平衡精度，F1得分和灵敏度/特异性等关键性能指标（包括零射床和几乎没有射击的婴儿床）分析了推理驱动的提示策略，包括零射cot和几乎没有射击的婴儿床。我们的发现表明，推理增强技术改善了直接预测的分类性能，尤其是在复杂的情况下。与诸如零拍摄的非局部发动机和诸如伯特和心理 - 罗伯塔等精细训练的预训练的变压器以及精心调整的开源llms（例如心理羊驼和心理 - flan-t5）相比，推理驱动的LLM会产生可观的增长，例如dreaddit（+0.52 fly bermmmm-llmm， +fer）， （+4.67 \％超过m-llm， +2.17 \％\％\％）。但是，抑郁严重程度的性能下降，CSSRS预测表明数据集特定的限制可能是由于我们使用了更广泛的测试集。在提示策略中，很少有射击COT始终优于其他cot，从而增强了推理驱动的LLM的有效性。但是，数据集的可变性突出了模型可靠性和解释性的挑战。这项研究为基于推理的LLM技术进行了精神健康文本分类的全面基准。它为他们提供可扩展的临床应用的潜力提供了见解，同时确定了未来改进的关键挑战。

Title: Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding

• Authors: Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai, Emad Barsoum
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10135
• Pdf URL: https://arxiv.org/pdf/2503.10135
• Copy Paste: [[2503.10135]] Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding(https://arxiv.org/abs/2503.10135)
• Keywords: language model, llm
• Abstract: Speculative decoding (SPD) aims to accelerate the auto-regressive token generation process of a target Large Language Model (LLM). Some approaches employ a draft model with multiple heads to predict a sequence of future tokens, where each head handles a token in the sequence. The target LLM verifies the predicted sequence and accepts aligned tokens, enabling efficient multi-token generation. However, existing methods assume that all tokens within a sequence are equally important, employing identical head structures and relying on a single-generation paradigm, either serial or parallel. To this end, we theoretically demonstrate that initial tokens in the draft sequence are more important than later ones. Building on this insight, we propose Gumiho, a hybrid model combining serial and parallel heads. Specifically, given the critical importance of early tokens, we employ a sophisticated Transformer architecture for the early draft heads in a serial configuration to improve accuracy. For later tokens, we utilize multiple lightweight MLP heads operating in parallel to enhance efficiency. By allocating more advanced model structures and longer running times to the early heads, Gumiho achieves improved overall performance. The experimental results demonstrate that our method outperforms existing approaches, fully validating its effectiveness.
• 摘要：投机解码（SPD）旨在加速目标大语言模型（LLM）的自动回归令牌生成过程。某些方法采用多个头部模型的草稿模型来预测未来令牌的顺序，每个头部都在序列中处理一个令牌。目标LLM验证预测的序列并接受对齐的令牌，从而有效地多togen oferation。但是，现有方法假定序列中的所有令牌都同样重要，它采用相同的头结构并依赖单一代范式（序列或平行）。为此，我们从理论上证明了草稿序列中的初始令牌比以后的标记更为重要。在这个见解的基础上，我们提出了Gumiho，这是一种结合了串行和平行头的混合模型。具体而言，鉴于早期令牌的重要性，我们采用复杂的变压器体系结构来为早期的绘制头部串行配置以提高准确性。对于以后的令牌，我们使用多个轻巧的MLP头，并平行运行以提高效率。通过将更先进的模型结构和更长的运行时间分配给早期的头部，Gumiho可以提高整体性能。实验结果表明，我们的方法表现优于现有方法，从而充分验证其有效性。

Title: Retrieval-Augmented Generation with Hierarchical Knowledge

• Authors: Haoyu Huang, Yongfeng Huang, Junjie Yang, Zhenyu Pan, Yongqiang Chen, Kaili Ma, Hongzhi Chen, James Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10150
• Pdf URL: https://arxiv.org/pdf/2503.10150
• Copy Paste: [[2503.10150]] Retrieval-Augmented Generation with Hierarchical Knowledge(https://arxiv.org/abs/2503.10150)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Graph-based Retrieval-Augmented Generation (RAG) methods have significantly enhanced the performance of large language models (LLMs) in domain-specific tasks. However, existing RAG methods do not adequately utilize the naturally inherent hierarchical knowledge in human cognition, which limits the capabilities of RAG systems. In this paper, we introduce a new RAG approach, called HiRAG, which utilizes hierarchical knowledge to enhance the semantic understanding and structure capturing capabilities of RAG systems in the indexing and retrieval processes. Our extensive experiments demonstrate that HiRAG achieves significant performance improvements over the state-of-the-art baseline methods. The code of our proposed method is available at \href{this https URL}{this https URL}.
• 摘要：基于图的检索演示生成（RAG）方法显着提高了特定于域特定任务中大语言模型（LLM）的性能。但是，现有的抹布方法不能充分利用人类认知中自然固有的层次知识，这限制了抹布系统的能力。在本文中，我们介绍了一种新的抹布方法，称为Hirag，该方法利用层次知识来增强索引和检索过程中抹布系统的语义理解和结构捕获功能。我们的广泛实验表明，Hirag在最新的基线方法方面取得了重大的性能提高。我们提出的方法的代码可在\ href {此https url} {this https url}上获得。

Title: "Well, Keep Thinking": Enhancing LLM Reasoning with Adaptive Injection Decoding

• Authors: Hyunbin Jin, Je Won Yeom, Seunghyun Bae, Taesup Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10167
• Pdf URL: https://arxiv.org/pdf/2503.10167
• Copy Paste: [[2503.10167]] "Well, Keep Thinking": Enhancing LLM Reasoning with Adaptive Injection Decoding(https://arxiv.org/abs/2503.10167)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) exhibit strong reasoning abilities, often attributed to few-shot or zero-shot chain-of-thought (CoT) prompting. While effective, these methods require labor-intensive prompt engineering, raising the question of whether reasoning can be induced without reliance on explicit prompts. In this work, we unlock the reasoning capabilities of LLMs without explicit prompting. Inspired by zero-shot CoT and CoT-decoding, we propose a novel decoding strategy that systematically nudges LLMs to continue reasoning, thereby preventing immature reasoning processes. Specifically, we monitor the model's generation and inject a designated phrase whenever it is likely to conclude its response prematurely, before completing the reasoning process. Our experimental evaluations on diverse reasoning benchmarks demonstrate that our proposed strategy substantially improves LLM reasoning capabilities, highlighting the potential of decoding-based interventions as an alternative to traditional prompting techniques.
• 摘要：大型语言模型（LLMS）具有强大的推理能力，通常归因于很少或零投入的链链（COT）提示。尽管有效，这些方法需要劳动密集型的及时工程，并提出了是否可以在不依赖明确提示的情况下引起推理的问题。在这项工作中，我们在没有明确提示的情况下解锁了LLM的推理功能。受零拍子和COT编码的启发，我们提出了一种新颖的解码策略，该策略会系统地推动LLMS继续推理，从而防止不成熟的推理过程。具体而言，我们在完成推理过程之前可能会过早地结论其响应，并在该模型的生成中监视该模型的生成并注入指定的短语。我们对各种推理基准的实验评估表明，我们提出的策略基本上提高了LLM推理能力，强调了基于解码的干预措施作为传统提示技术的替代方案的潜力。

Title: PRISM: Preference Refinement via Implicit Scene Modeling for 3D Vision-Language Preference-Based Reinforcement Learning

• Authors: Yirong Sun, Yanjun Chen
• Subjects: cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.10177
• Pdf URL: https://arxiv.org/pdf/2503.10177
• Copy Paste: [[2503.10177]] PRISM: Preference Refinement via Implicit Scene Modeling for 3D Vision-Language Preference-Based Reinforcement Learning(https://arxiv.org/abs/2503.10177)
• Keywords: language model, llm, chain-of-thought
• Abstract: We propose PRISM, a novel framework designed to overcome the limitations of 2D-based Preference-Based Reinforcement Learning (PBRL) by unifying 3D point cloud modeling and future-aware preference refinement. At its core, PRISM adopts a 3D Point Cloud-Language Model (3D-PC-LLM) to mitigate occlusion and viewpoint biases, ensuring more stable and spatially consistent preference signals. Additionally, PRISM leverages Chain-of-Thought (CoT) reasoning to incorporate long-horizon considerations, thereby preventing the short-sighted feedback often seen in static preference comparisons. In contrast to conventional PBRL techniques, this integration of 3D perception and future-oriented reasoning leads to significant gains in preference agreement rates, faster policy convergence, and robust generalization across unseen robotic environments. Our empirical results, spanning tasks such as robotic manipulation and autonomous navigation, highlight PRISM's potential for real-world applications where precise spatial understanding and reliable long-term decision-making are critical. By bridging 3D geometric awareness with CoT-driven preference modeling, PRISM establishes a comprehensive foundation for scalable, human-aligned reinforcement learning.
• 摘要：我们提出了Prism，这是一个新颖的框架，旨在通过统一3D点云建模和未来感知的偏好优化来克服基于2D的偏好增强学习（PBRL）的局限性。棱镜以此为核心采用3D点云模型（3D-PC-LLM）来减轻遮挡和观点偏见，从而确保更稳定且在空间上一致的偏好信号。此外，棱镜利用了经过思考链（COT）的推理，以纳入长马的注意事项，从而防止在静态偏好比较中经常看到的短视反馈。与常规的PBRL技术相反，3D感知和面向未来的推理的这种整合导致优先协议率，更快的策略收敛速度以及在未看到的机器人环境中的强大概括。我们的经验结果涵盖了机器人操纵和自动导航等任务，突显了Prism在精确的空间理解和可靠的长期决策至关重要的现实世界中的潜力。通过将3D几何意识与COT驱动的偏好建模桥接，Prism为可扩展的人类一致的强化学习建立了全面的基础。

Title: Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation

• Authors: Henglyu Liu, Andong Chen, Kehai Chen, Xuefeng Bai, Meizhi Zhong, Yuan Qiu, Min Zhang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2503.10211
• Pdf URL: https://arxiv.org/pdf/2503.10211
• Copy Paste: [[2503.10211]] Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation(https://arxiv.org/abs/2503.10211)
• Keywords: language model, llm
• Abstract: Recent advancement of large language models (LLMs) has led to significant breakthroughs across various tasks, laying the foundation for the development of LLM-based speech translation systems. Existing methods primarily focus on aligning inputs and outputs across modalities while overlooking deeper semantic alignment within model representations. To address this limitation, we propose an Adaptive Inner Speech-Text Alignment (AI-STA) method to bridge the modality gap by explicitly aligning speech and text representations at selected layers within LLMs. To achieve this, we leverage the optimal transport (OT) theory to quantify fine-grained representation discrepancies between speech and text. Furthermore, we utilize the cross-modal retrieval technique to identify the layers that are best suited for alignment and perform joint training on these layers. Experimental results on speech translation (ST) tasks demonstrate that AI-STA significantly improves the translation performance of large speech-text models (LSMs), outperforming previous state-of-the-art approaches. Our findings highlight the importance of inner-layer speech-text alignment in LLMs and provide new insights into enhancing cross-modal learning.
• 摘要：大型语言模型（LLM）的最新进步导致了各种任务的重大突破，为开发基于LLM的语音翻译系统奠定了基础。现有方法主要集中在对齐方式和输出跨模态的同时，同时忽略模型表示中更深层次的语义对齐。为了解决此限制，我们提出了一种自适应的内部语音文本对齐（AI-STA）方法，以通过在LLMS内的选定层上明确对准语音和文本表示来弥合模式差距。为了实现这一目标，我们利用最佳运输理论（OT）理论来量化语音和文本之间的细粒度表示差异。此外，我们利用跨模式检索技术来识别最适合对齐的层，并在这些层上进行联合培训。语音翻译（ST）任务的实验结果表明，AI-STA显着提高了大型语音文本模型（LSM）的翻译性能，表现优于先前的最新方法。我们的发现突出了内层语音文本在LLM中的重要性，并为增强跨模式学习提供了新的见解。

Title: R.U.Psycho? Robust Unified Psychometric Testing of Language Models

• Authors: Julian Schelb, Orr Borin, David Garcia, Andreas Spitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10229
• Pdf URL: https://arxiv.org/pdf/2503.10229
• Copy Paste: [[2503.10229]] R.U.Psycho? Robust Unified Psychometric Testing of Language Models(https://arxiv.org/abs/2503.10229)
• Keywords: language model, prompt
• Abstract: Generative language models are increasingly being subjected to psychometric questionnaires intended for human testing, in efforts to establish their traits, as benchmarks for alignment, or to simulate participants in social science experiments. While this growing body of work sheds light on the likeness of model responses to those of humans, concerns are warranted regarding the rigour and reproducibility with which these experiments may be conducted. Instabilities in model outputs, sensitivity to prompt design, parameter settings, and a large number of available model versions increase documentation requirements. Consequently, generalization of findings is often complex and reproducibility is far from guaranteed. In this paper, we present this http URL, a framework for designing and running robust and reproducible psychometric experiments on generative language models that requires limited coding expertise. We demonstrate the capability of our framework on a variety of psychometric questionnaires, which lend support to prior findings in the literature. this http URL is available as a Python package at this https URL.
• 摘要：生成语言模型越来越多地受到用于人体测试的心理测量问卷，以建立自己的特征，作为对齐的基准或模拟社会科学实验的参与者。尽管这种日益增长的工作阐明了模型对人类的响应的相似性，但对可以进行这些实验的严格性和可重复性的担忧是有必要的。模型输出中的不稳定性，对及时设计的敏感性，参数设置以及大量可用模型版本增加了文档要求。因此，发现的概括通常是复杂的，并且可重复性远非保证。在本文中，我们介绍了此HTTP URL，这是在需要有限的编码专业知识的生成语言模型上设计和运行可重复的心理测量实验的框架。我们证明了我们在各种心理测量问卷上的框架的能力，这些问卷为文献中的先前发现提供了支持。此HTTP URL可作为python包装在此HTTPS URL上使用。

Title: MinorBench: A hand-built benchmark for content-based risks for children

• Authors: Shaun Khoo, Gabriel Chua, Rachel Shong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10242
• Pdf URL: https://arxiv.org/pdf/2503.10242
• Copy Paste: [[2503.10242]] MinorBench: A hand-built benchmark for content-based risks for children(https://arxiv.org/abs/2503.10242)
• Keywords: language model, llm, prompt, chat
• Abstract: Large Language Models (LLMs) are rapidly entering children's lives - through parent-driven adoption, schools, and peer networks - yet current AI ethics and safety research do not adequately address content-related risks specific to minors. In this paper, we highlight these gaps with a real-world case study of an LLM-based chatbot deployed in a middle school setting, revealing how students used and sometimes misused the system. Building on these findings, we propose a new taxonomy of content-based risks for minors and introduce MinorBench, an open-source benchmark designed to evaluate LLMs on their ability to refuse unsafe or inappropriate queries from children. We evaluate six prominent LLMs under different system prompts, demonstrating substantial variability in their child-safety compliance. Our results inform practical steps for more robust, child-focused safety mechanisms and underscore the urgency of tailoring AI systems to safeguard young users.
• 摘要：大型语言模型（LLMS）正在迅速进入儿童的生活 - 通过父母驱动的收养，学校和同伴网络 - 但是当前的AI伦理和安全研究并不能充分解决未成年人特定的与内容相关的风险。在本文中，我们通过在中学环境中部署的基于LLM的聊天机器人进行的真实案例研究来强调这些差距，从而揭示了学生的使用方式和有时滥用系统。在这些发现的基础上，我们提出了针对未成年人的基于内容的风险的新分类法，并引入了Minerce，这是一种开源基准测试，旨在评估LLM的能力，以拒绝儿童的不安全或不适当的查询。我们在不同的系统提示下评估了六个突出的LLM，表明其儿童安全依从性的差异很大。我们的结果为实用的步骤提供了更大的稳健，以儿童为中心的安全机制，并强调了为保护年轻用户量身定制AI系统的紧迫性。

Title: An Expanded Massive Multilingual Dataset for High-Performance Language Technologies

• Authors: Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, and Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, and Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, and Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, and Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, and Amanda Myntti, Dayyán O'Brien, Stephan Oepen, Proyag Pal, Jousia Piha, and Sampo Pyysalo, Gema Ramírez-Sánchez, David Samuel, Pavel Stepachev, and Jörg Tiedemann, Dušan Variš, Tereza Vojtěchová, Jaume Zaragoza-Bernabeu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10267
• Pdf URL: https://arxiv.org/pdf/2503.10267
• Copy Paste: [[2503.10267]] An Expanded Massive Multilingual Dataset for High-Performance Language Technologies(https://arxiv.org/abs/2503.10267)
• Keywords: language model
• Abstract: Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.
• 摘要：培训最先进的大语言模型需要大量的干净和多样化的文本数据。但是，构建合适的多语言数据集仍然是一个挑战。在这项工作中，我们介绍了HPLT V2，这是一系列高质量的多语言单语言和平行语料库。数据的单语部分包含涵盖193种语言的8T令牌，而并行数据包含380m句子对，涵盖了51种语言。我们记录整个数据管道并发布代码以复制它。我们对数据的质量和特征进行了广泛的分析。最后，我们评估了在HPLT V2上训练的语言模型和机器翻译系统的性能，以证明其价值。

Title: Proceedings of the ISCA/ITG Workshop on Diversity in Large Speech and Language Models

• Authors: Sebastian Möller, Pia Knoeferle, Britta Schulte, Nils Feldhus
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10298
• Pdf URL: https://arxiv.org/pdf/2503.10298
• Copy Paste: [[2503.10298]] Proceedings of the ISCA/ITG Workshop on Diversity in Large Speech and Language Models(https://arxiv.org/abs/2503.10298)
• Keywords: language model, llm, chat
• Abstract: Machine learning techniques have conquered many different tasks in speech and natural language processing, such as speech recognition, information extraction, text and speech generation, and human machine interaction using natural language or speech (chatbots). Modern techniques typically rely on large models for representing general knowledge of one or several languages (Large Language Models, LLMs), or for representing speech and general audio characteristics. These models have been trained with large amounts of speech and language data, typically including web content. When humans interact with such technologies, the effectiveness of the interaction will be influenced by how far humans make use of the same type of language the models have been trained on or, in other words, if the models are able to generalize to the language used by humans when interacting with the technology. This may lead to some gradual forms of adaptation in human speech and language production, and users who do not adapt may be excluded from efficient use of such technologies. On top of this, as commercial model development follows market needs, under-represented languages and dialects/sociolects may decrease in terms of priorities. Furthermore, for many lesser spoken languages the necessary data is not available, which will worsen a digital divide in speech and language technology usage. The workshop sets out to discuss this problem based on scientific contributions from the perspective of computer science and linguistics (including computational linguistics and NLP).
• 摘要：机器学习技术已经征服了语音和自然语言处理中的许多不同任务，例如语音识别，信息提取，文本和语音生成以及使用自然语言或语音（聊天机器人）的人类机器互动。现代技术通常依靠大型模型来表示一种或几种语言（大语言模型，LLM）或代表语音和一般音频特征的通用知识。这些模型已经接受了大量语音和语言数据的培训，通常包括Web内容。当人类与此类技术互动时，互动的有效性将受到人类使用相同类型的语言的影响，或者换句话说，如果模型能够推广到与该技术互动时使用的语言，则对模型进行了培训。这可能会导致人类言语和语言生产中一些逐渐适应的形式，而不适应的用户可能会被排除在有效使用此类技术之外。最重要的是，随着商业模型开发遵循市场需求，代表性不足的语言和方言/社会文法可能会在优先级中减少。此外，对于许多较少的口语语言，不可用的数据将使语音和语言技术使用的数字鸿沟恶化。该研讨会从计算机科学和语言学的角度（包括计算语言学和NLP）开始根据科学贡献进行讨论这个问题。

Title: KV-Distill: Nearly Lossless Learnable Context Compression for LLMs

• Authors: Vivek Chari, Guanghui Qin, Benjamin Van Durme
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10337
• Pdf URL: https://arxiv.org/pdf/2503.10337
• Copy Paste: [[2503.10337]] KV-Distill: Nearly Lossless Learnable Context Compression for LLMs(https://arxiv.org/abs/2503.10337)
• Keywords: llm, long context
• Abstract: Sequence-to-sequence tasks often benefit from long contexts, but the quadratic complexity of self-attention in standard Transformers renders this non-trivial. During generation, temporary representations -stored in the so-called KV cache-account for a large portion of GPU memory usage and scale linearly with context length. We introduce KV-Distill, a Transformer compression framework that distills long context KV caches into significantly shorter representations in a question-independent fashion. KV-Distill can be trained as a parameter-efficient adaptor for pretrained models, and enables the compression of arbitrary spans of a context while preserving pre-trained model capabilities. We treat a compressed-uncompressed cache as a student-teacher pairing and apply a KL-type divergence to match the generated outputs. KV-Distill outperforms other compression techniques in worst-case extractive tasks and approaches uncompressed performance in long context question answering and summarization, and it can be fine-tuned on domain-specific contexts to reduce lengths by up to 99% while preserving downstream performance. We demonstrate the generalizability of KV-Distill across various model sizes and architectures.
• 摘要：序列到序列任务通常会受益于较长的上下文，但是标准变压器中自我注意的二次复杂性使这种非平凡。在生成期间，临时表示形式（存储在所谓的KV高速cache-Account中，用于大部分GPU存储器使用情况，并与上下文长度线性地缩放。我们介绍了KV-Distill，这是一个变压器压缩框架，以与问题无关的方式将较长的上下文kv scaches提炼成明显较短的表示形式。可以将KV-Distill作为验证模型的参数效率适配器训练，并可以在保留预训练的模型功能的同时压缩上下文的任意跨度。我们将压缩未压制的缓存视为学生教师配对，并应用KL型散布以匹配生成的输出。 KV-Distill在最严重的提取任务中优于其他压缩技术，并在长上下文的问题答案和摘要中使用未压缩的性能，并且可以在特定领域的上下文中进行微调，以减少多达99％的长度，同时保持下游绩效。我们证明了KV-Distill在各种模型和体系结构上的普遍性。

Title: New Trends for Modern Machine Translation with Large Reasoning Models

• Authors: Sinuo Liu, Chenyang Lyu, Minghao Wu, Longyue Wang, Weihua Luo, Kaifu Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10351
• Pdf URL: https://arxiv.org/pdf/2503.10351
• Copy Paste: [[2503.10351]] New Trends for Modern Machine Translation with Large Reasoning Models(https://arxiv.org/abs/2503.10351)
• Keywords: llm, chain-of-thought, agent
• Abstract: Recent advances in Large Reasoning Models (LRMs), particularly those leveraging Chain-of-Thought reasoning (CoT), have opened brand new possibility for Machine Translation (MT). This position paper argues that LRMs substantially transformed traditional neural MT as well as LLMs-based MT paradigms by reframing translation as a dynamic reasoning task that requires contextual, cultural, and linguistic understanding and reasoning. We identify three foundational shifts: 1) contextual coherence, where LRMs resolve ambiguities and preserve discourse structure through explicit reasoning over cross-sentence and complex context or even lack of context; 2) cultural intentionality, enabling models to adapt outputs by inferring speaker intent, audience expectations, and socio-linguistic norms; 3) self-reflection, LRMs can perform self-reflection during the inference time to correct the potential errors in translation especially extremely noisy cases, showing better robustness compared to simply mapping X->Y translation. We explore various scenarios in translation including stylized translation, document-level translation and multimodal translation by showcasing empirical examples that demonstrate the superiority of LRMs in translation. We also identify several interesting phenomenons for LRMs for MT including auto-pivot translation as well as the critical challenges such as over-localisation in translation and inference efficiency. In conclusion, we think that LRMs redefine translation systems not merely as text converters but as multilingual cognitive agents capable of reasoning about meaning beyond the text. This paradigm shift reminds us to think of problems in translation beyond traditional translation scenarios in a much broader context with LRMs - what we can achieve on top of it.
• 摘要：大型推理模型（LRMS）的最新进展，尤其是那些利用经过思考推理的那些（COT），已经为机器翻译（MT）打开了全新的可能性。该立场论文认为，LRMS通过将翻译翻译为一项动态推理任务，实质上改变了传统的神经MT以及基于LLMS的MT范式，需要上下文，文化和语言理解和推理。我们确定了三个基本转变：1）上下文连贯性，LRMS通过在跨句子和复杂的背景上的明确推理甚至缺乏背景来解决歧义并保护话语结构； 2）文化意图，使模型能够通过推断说话者意图，观众期望和社会语言规范来调整产出； 3）自我反射，LRM可以在推理时间内执行自我反射，以纠正翻译中的潜在误差，尤其是极度嘈杂的情况，与简单地绘制X-> y翻译相比，表现出更好的鲁棒性。通过展示经验示例，我们探讨了翻译中的各种场景，包括风格化翻译，文档级翻译和多模式翻译，这些示例证明了LRMS在翻译中的优越性。我们还为MT的LRMS确定了几种有趣的现象，包括自动转换，以及关键挑战，例如翻译和推理效率的过度定位。总之，我们认为LRMS重新定义了翻译系统不仅是文本转换器，而且还可以作为能够推理文本超出文本意义的多语言认知剂。这种范式的转变提醒我们在更广泛的环境中，通过LRMS在更广泛的环境中思考翻译中的问题 - 我们可以在其上实现的目标。

Title: Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmark

• Authors: Viktor Moskvoretskii, Alina Lobanova, Ekaterina Neminova, Chris Biemann, Alexander Panchenko, Irina Nikishina
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.10357
• Pdf URL: https://arxiv.org/pdf/2503.10357
• Copy Paste: [[2503.10357]] Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmark(https://arxiv.org/abs/2503.10357)
• Keywords: gpt, llm
• Abstract: This paper explores the feasibility of using text-to-image models in a zero-shot setup to generate images for taxonomy concepts. While text-based methods for taxonomy enrichment are well-established, the potential of the visual dimension remains unexplored. To address this, we propose a comprehensive benchmark for Taxonomy Image Generation that assesses models' abilities to understand taxonomy concepts and generate relevant, high-quality images. The benchmark includes common-sense and randomly sampled WordNet concepts, alongside the LLM generated predictions. The 12 models are evaluated using 9 novel taxonomy-related text-to-image metrics and human feedback. Moreover, we pioneer the use of pairwise evaluation with GPT-4 feedback for image generation. Experimental results show that the ranking of models differs significantly from standard T2I tasks. Playground-v2 and FLUX consistently outperform across metrics and subsets and the retrieval-based approach performs poorly. These findings highlight the potential for automating the curation of structured data resources.
• 摘要：本文探讨了在零拍设置中使用文本对图像模型来生成分类法概念的图像的可行性。虽然基于文本的分类富集方法是良好的，但视觉维度的潜力仍未开发。为了解决这个问题，我们为分类图像生成的全面基准提出了一个全面的基准，该基准评估了模型理解分类法概念并产生相关高质量图像的能力。基准包括常识性和随机采样网络概念，以及LLM生成的预测。使用9种新型分类学相关的文本对象指标和人类反馈评估了12个模型。此外，我们开创了将成对评估与GPT-4反馈的使用进行图像生成。实验结果表明，模型的排名与标准T2I任务明显不同。 Playground-V2和Flux在指标和子集上始终胜过且基于检索的方法的表现不佳。这些发现突出了自动化结构化数据资源策划的潜力。

Title: G-Boost: Boosting Private SLMs with General LLMs

• Authors: Yijiang Fan, Yuren Mao, Longbin Lai, Ying Zhang, Zhengping Qian, Yunjun Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10367
• Pdf URL: https://arxiv.org/pdf/2503.10367
• Copy Paste: [[2503.10367]] G-Boost: Boosting Private SLMs with General LLMs(https://arxiv.org/abs/2503.10367)
• Keywords: language model, llm
• Abstract: Due to the limited computational resources, most Large Language Models (LLMs) developers can only fine-tune Small Language Models (SLMs) on their own data. These private SLMs typically have limited effectiveness. To boost the performance of private SLMs, this paper proposes to ask general LLMs for help. The general LLMs can be APIs or larger LLMs whose inference cost the developers can afford. Specifically, we propose the G-Boost framework where a private SLM adaptively performs collaborative inference with a general LLM under the guide of process reward. Experiments demonstrate that our framework can significantly boost the performance of private SLMs.
• 摘要：由于计算资源有限，大多数大型语言模型（LLMS）开发人员只能在自己的数据上微调小语言模型（SLM）。这些私人SLM通常的有效性有限。为了提高私人SLM的性能，本文提议向一般LLM寻求帮助。一般的LLM可以是API或更大的LLM，其推理成本可以负担得起。具体而言，我们提出了G-Boost框架，其中私人SLM在过程奖励指南下与一般LLM的一般LLM进行了协作。实验表明，我们的框架可以显着提高私人SLM的性能。

Title: VisTai: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

• Authors: Zhi Rui Tam, Ya-Ting Pai, Yen-Wei Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10427
• Pdf URL: https://arxiv.org/pdf/2503.10427
• Copy Paste: [[2503.10427]] VisTai: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan(https://arxiv.org/abs/2503.10427)
• Keywords: language model
• Abstract: In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTai-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTai-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.
• 摘要：在本文中，我们为传统汉语中的视觉语言模型（VLM）提出了一个全面的评估基准。我们的评估套件（同类产品中的第一个）包含两个互补的组成部分：（1）Vistai-MCQ，这是来自21个学术主题的手动策划考试的多项选择问题的集合，旨在测试VLMS的广泛知识和推理能力； （2）Vistai-Dialogue，一个开放的对话基准，包括131个图像问题对，手动创建，以评估VLMS在台湾文化背景下的自由形式对话生成的能力。这些基准介绍了评估领域中的一个关键差距，在该领域中，现有基准主要集中在英语或简化的中文上，忽略了在台湾和香港等地区使用的传统中国人的独特语言和文化方面。我们的分析揭示了各种VLM的绩效差异，并突出了处理传统中国视觉内容时面临的特定挑战。

Title: DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

• Authors: Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, Kaidi Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10452
• Pdf URL: https://arxiv.org/pdf/2503.10452
• Copy Paste: [[2503.10452]] DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation(https://arxiv.org/abs/2503.10452)
• Keywords: language model, llm
• Abstract: The rapid advancement of large language models (LLMs) has significantly improved their performance in code generation tasks. However, existing code benchmarks remain static, consisting of fixed datasets with predefined problems. This makes them vulnerable to memorization during training, where LLMs recall specific test cases instead of generalizing to new problems, leading to data contamination and unreliable evaluation results. To address these issues, we introduce DynaCode, a dynamic, complexity-aware benchmark that overcomes the limitations of static datasets. DynaCode evaluates LLMs systematically using a complexity-aware metric, incorporating both code complexity and call-graph structures. DynaCode achieves large-scale diversity, generating up to 189 million unique nested code problems across four distinct levels of code complexity, referred to as units, and 16 types of call graphs. Results on 12 latest LLMs show an average performance drop of 16.8% to 45.7% compared to MBPP+, a static code generation benchmark, with performance progressively decreasing as complexity increases. This demonstrates DynaCode's ability to effectively differentiate LLMs. Additionally, by leveraging call graphs, we gain insights into LLM behavior, particularly their preference for handling subfunction interactions within nested code.
• 摘要：大型语言模型（LLM）的快速发展已大大提高了其在代码生成任务中的性能。但是，现有的代码基准仍然是静态的，由固定的数据集组成，这些数据集具有预定义的问题。这使他们容易在培训期间记忆，其中LLM会回忆特定的测试案例，而不是概括为新问题，从而导致数据污染和不可靠的评估结果。为了解决这些问题，我们介绍了DynaCode，这是一种动态，复杂性感知的基准，它克服了静态数据集的局限性。 DynAcode使用复杂性的度量来系统地评估LLMS，并结合了代码复杂性和呼叫图形结构。 Dynacode实现了大规模的多样性，在四个不同级别的代码复杂性（称为单位）和16种呼叫图中产生了多达1.89亿个独特的嵌套代码问题。与静态代码生成基准MBPP+相比，12个最新LLM的结果表明，平均性能下降了16.8％至45.7％，随着复杂性的增加，性能逐渐降低。这证明了Dynacode有效区分LLM的能力。此外，通过利用呼叫图，我们可以深入了解LLM行为，尤其是它们偏爱嵌套代码中的负函数相互作用。

Title: Statistical Analysis of Sentence Structures through ASCII, Lexical Alignment and PCA

• Authors: Abhijeet Sahdev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10470
• Pdf URL: https://arxiv.org/pdf/2503.10470
• Copy Paste: [[2503.10470]] Statistical Analysis of Sentence Structures through ASCII, Lexical Alignment and PCA(https://arxiv.org/abs/2503.10470)
• Keywords: llm
• Abstract: While utilizing syntactic tools such as parts-of-speech (POS) tagging has helped us understand sentence structures and their distribution across diverse corpora, it is quite complex and poses a challenge in natural language processing (NLP). This study focuses on understanding sentence structure balance - usages of nouns, verbs, determiners, etc - harmoniously without relying on such tools. It proposes a novel statistical method that uses American Standard Code for Information Interchange (ASCII) codes to represent text of 11 text corpora from various sources and their lexical category alignment after using their compressed versions through PCA, and analyzes the results through histograms and normality tests such as Shapiro-Wilk and Anderson-Darling Tests. By focusing on ASCII codes, this approach simplifies text processing, although not replacing any syntactic tools but complementing them by offering it as a resource-efficient tool for assessing text balance. The story generated by Grok shows near normality indicating balanced sentence structures in LLM outputs, whereas 4 out of the remaining 10 pass the normality tests. Further research could explore potential applications in text quality evaluation and style analysis with syntactic integration for more broader tasks.
• 摘要：虽然利用句法工具，例如言论部分（POS）标记，帮助我们了解了句子结构及其在各种语料库中的分布，但它非常复杂，并且在自然语言处理（NLP）中构成了挑战。这项研究的重点是理解句子结构平衡 - 名词，动词，确定词等的用法 - 不依赖此类工具。它提出了一种新颖的统计方法，该方法在通过PCA使用压缩版本后，使用美国标准代码进行信息互换（ASCII）代码来代表来自各种来源及其词汇类别对齐的11个文本语料库的文本，并通过直方图和正常测试（例如Shapiro-Wilk和Shapiro-Wilk和Anderson-darling Tests）分析结果。通过关注ASCII代码，这种方法简化了文本处理，尽管没有替换任何句法工具，而是通过将其作为评估文本平衡的资源有效工具来补充它们。格罗克（Grok）产生的故事显示几乎正态性，表明LLM输出中的句子结构平衡，而其余10个通过了正态性测试。进一步的研究可以通过句法集成在文本质量评估和样式分析中探索潜在的应用，以实现更广泛的任务。

Title: World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning

• Authors: Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, Xipeng Qiu
• Subjects: cs.CL, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.10480
• Pdf URL: https://arxiv.org/pdf/2503.10480
• Copy Paste: [[2503.10480]] World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning(https://arxiv.org/abs/2503.10480)
• Keywords: language model, gpt
• Abstract: Recent advances in large vision-language models (LVLMs) have shown promise for embodied task planning, yet they struggle with fundamental challenges like dependency constraints and efficiency. Existing approaches either solely optimize action selection or leverage world models during inference, overlooking the benefits of learning to model the world as a way to enhance planning capabilities. We propose Dual Preference Optimization (D$^2$PO), a new learning framework that jointly optimizes state prediction and action selection through preference learning, enabling LVLMs to understand environment dynamics for better planning. To automatically collect trajectories and stepwise preference data without human annotation, we introduce a tree search mechanism for extensive exploration via trial-and-error. Extensive experiments on VoTa-Bench demonstrate that our D$^2$PO-based method significantly outperforms existing methods and GPT-4o when applied to Qwen2-VL (7B), LLaVA-1.6 (7B), and LLaMA-3.2 (11B), achieving superior task success rates with more efficient execution paths.
• 摘要：大型视力语言模型（LVLM）的最新进展显示出了体现任务计划的希望，但他们在依赖性约束和效率等基本挑战中挣扎。现有方法可以仅优化行动选择，或者在推理期间利用世界模型，忽视学习建模世界的好处，以增强计划能力。我们提出了双重偏好优化（D $^2 $ PO），这是一个新的学习框架，可以通过偏好学习共同优化状态预测和行动选择，使LVLMS能够了解环境动态以进行更好的计划。为了在没有人类注释的情况下自动收集轨迹和逐步偏好数据，我们引入了一种通过反复试验进行广泛探索的树搜索机制。对投票台的广泛实验表明，当应用于QWEN2-VL（7B），LLAVA-1.6（7B）和LLAMA-3.2（11B）时，我们的D $^2 $ PO方法明显优于现有方法和GPT-4O，并以更有效的执行路径获得了较高的任务成功率。

Title: LLMs in Disease Diagnosis: A Comparative Study of DeepSeek-R1 and O3 Mini Across Chronic Health Conditions

• Authors: Gaurav Kumar Gupta, Pranal Pande
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10486
• Pdf URL: https://arxiv.org/pdf/2503.10486
• Copy Paste: [[2503.10486]] LLMs in Disease Diagnosis: A Comparative Study of DeepSeek-R1 and O3 Mini Across Chronic Health Conditions(https://arxiv.org/abs/2503.10486)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are revolutionizing medical diagnostics by enhancing both disease classification and clinical decision-making. In this study, we evaluate the performance of two LLM- based diagnostic tools, DeepSeek R1 and O3 Mini, using a structured dataset of symptoms and diagnoses. We assessed their predictive accuracy at both the disease and category levels, as well as the reliability of their confidence scores. DeepSeek R1 achieved a disease-level accuracy of 76% and an overall accuracy of 82%, outperforming O3 Mini, which attained 72% and 75% respectively. Notably, DeepSeek R1 demonstrated exceptional performance in Mental Health, Neurological Disorders, and Oncology, where it reached 100% accuracy, while O3 Mini excelled in Autoimmune Disease classification with 100% accuracy. Both models, however, struggled with Respiratory Disease classification, recording accuracies of only 40% for DeepSeek R1 and 20% for O3 Mini. Additionally, the analysis of confidence scores revealed that DeepSeek R1 provided high-confidence predictions in 92% of cases, compared to 68% for O3 Mini. Ethical considerations regarding bias, model interpretability, and data privacy are also discussed to ensure the responsible integration of LLMs into clinical practice. Overall, our findings offer valuable insights into the strengths and limitations of LLM-based diagnostic systems and provide a roadmap for future enhancements in AI-driven healthcare.
• 摘要：大型语言模型（LLM）正在通过增强疾病分类和临床决策来彻底改变医学诊断。在这项研究中，我们使用症状和诊断的结构化数据集评估了两种基于LLM的诊断工具，DeepSeek R1和O3 Mini的性能。我们评估了他们在疾病和类别水平上的预测准确性，以及其置信度得分的可靠性。 DeepSeek R1的疾病水平准确度为76％，总体准确度为82％，表现优于O3 Mini，分别达到72％和75％。值得注意的是，DeepSeek R1在心理健康，神经系统疾病和肿瘤学方面表现出了出色的表现，在那里它达到了100％的准确性，而O3 Mini在自身免疫性疾病分类方面表现出色，精度为100％。然而，这两种模型都在呼吸道疾病分类方面挣扎，DeepSeek R1仅记录40％的精度，而O3 Mini仅记录了20％。此外，对置信度评分的分析表明，DeepSeek R1在92％的病例中提供了高信心的预测，而O3 Mini则提供了68％。还讨论了有关偏见，模型可解释性和数据隐私性的道德考虑因素，以确保LLM的负责任整合到临床实践中。总体而言，我们的发现为基于LLM的诊断系统的优势和局限性提供了宝贵的见解，并为AI驱动的医疗保健的未来增强提供了路线图。

Title: Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents

• Authors: Hanxu Hu, Jannis Vamvas, Rico Sennrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10494
• Pdf URL: https://arxiv.org/pdf/2503.10494
• Copy Paste: [[2503.10494]] Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents(https://arxiv.org/abs/2503.10494)
• Keywords: language model, llm
• Abstract: LLMs have paved the way for truly simple document-level machine translation, but challenges such as omission errors remain. In this paper, we study a simple method for handling document-level machine translation, by leveraging previous contexts in a multi-turn conversational manner. Specifically, by decomposing documents into segments and iteratively translating them while maintaining previous turns, this method ensures coherent translations without additional training, and can fully re-use the KV cache of previous turns thus minimizing computational overhead. We further propose a `source-primed' method that first provides the whole source document before multi-turn translation. We empirically show this multi-turn method outperforms both translating entire documents in a single turn and translating each segment independently according to multiple automatic metrics in representative LLMs, establishing a strong baseline for document-level translation using LLMs.
• 摘要：LLM为真正简单的文档级机器翻译铺平了道路，但是仍然存在诸如遗漏错误之类的挑战。在本文中，我们通过以多转交谈方式利用以前的上下文来研究一种处理文档级机器翻译的简单方法。具体而言，通过将文档分解为细分市场并在保持前一回合的同时迭代地翻译它们，该方法可确保连贯的翻译而无需额外的培训，并且可以完全重复使用以前的转弯的KV缓存，从而最大程度地减少了计算开销。我们进一步提出了一种“源代码”方法，该方法首先在多转移翻译之前提供整个源文档。我们从经验上表明，这种多转弯方法的表现优于单个转弯的整个文档，又要根据代表性LLM中的多个自动指标独立转换整个文档，建立了使用LLMS的文档级翻译的强基线。

Title: MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation

• Authors: Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Yun Xing, Junjue Wang, Huitao Li, Xin Li, Kunyu Yu, Nan Liu, Qingyu Chen, Douglas Teodoro, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10497
• Pdf URL: https://arxiv.org/pdf/2503.10497
• Copy Paste: [[2503.10497]] MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation(https://arxiv.org/abs/2503.10497)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Traditional benchmarks struggle to evaluate increasingly sophisticated language models in multilingual and culturally diverse contexts. To address this gap, we introduce MMLU-ProX, a comprehensive multilingual benchmark covering 13 typologically diverse languages with approximately 11,829 questions per language. Building on the challenging reasoning-focused design of MMLU-Pro, our framework employs a semi-automatic translation process: translations generated by state-of-the-art large language models (LLMs) are rigorously evaluated by expert annotators to ensure conceptual accuracy, terminological consistency, and cultural relevance. We comprehensively evaluate 25 state-of-the-art LLMs using 5-shot chain-of-thought (CoT) and zero-shot prompting strategies, analyzing their performance across linguistic and cultural boundaries. Our experiments reveal consistent performance degradation from high-resource languages to lower-resource ones, with the best models achieving over 70% accuracy on English but dropping to around 40% for languages like Swahili, highlighting persistent gaps in multilingual capabilities despite recent advances. MMLU-ProX is an ongoing project; we are expanding our benchmark by incorporating additional languages and evaluating more language models to provide a more comprehensive assessment of multilingual capabilities.
• 摘要：传统的基准努力在多语言和文化不同的环境中评估日益复杂的语言模型。为了解决这一差距，我们介绍了MMLU-Prox，这是一种全面的多语言基准，涵盖13种类型上的语言，每个语言约为11,829个问题。我们的框架以最具挑战性的推理为中心的MMLU-PRO设计采用了半自动翻译过程：由最先进的大语言模型（LLMS）生成的翻译经过专家注释者的严格评估，以确保概念准确性，术语一致性和文化相关性。我们使用5次思维链（COT）和零射击策略对25种最先进的LLM进行了全面评估，从而分析了它们在语言和文化边界的绩效。我们的实验揭示了从高资源语言到低资源语言的一致性降解，最佳模型在英语上达到了70％以上的精度，但对于斯瓦希里语（例如斯瓦希里语）的语言下降到40％左右，尽管最近进步了，但在多种方面的差距中却突出了多种语言的持续差距。 mmlu-prox是一个正在进行的项目；我们通过合并其他语言并评估更多语言模型来扩展基准，以提供对多语言功能的更全面评估。

Title: Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set

• Authors: Florian Eichin, Yang Janet Liu, Barbara Plank, Michael A. Hedderich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10515
• Pdf URL: https://arxiv.org/pdf/2503.10515
• Copy Paste: [[2503.10515]] Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set(https://arxiv.org/abs/2503.10515)
• Keywords: language model, llm
• Abstract: Discourse understanding is essential for many NLP tasks, yet most existing work remains constrained by framework-dependent discourse representations. This work investigates whether large language models (LLMs) capture discourse knowledge that generalizes across languages and frameworks. We address this question along two dimensions: (1) developing a unified discourse relation label set to facilitate cross-lingual and cross-framework discourse analysis, and (2) probing LLMs to assess whether they encode generalizable discourse abstractions. Using multilingual discourse relation classification as a testbed, we examine a comprehensive set of 23 LLMs of varying sizes and multilingual capabilities. Our results show that LLMs, especially those with multilingual training corpora, can generalize discourse information across languages and frameworks. Further layer-wise analyses reveal that language generalization at the discourse level is most salient in the intermediate layers. Lastly, our error analysis provides an account of challenging relation classes.
• 摘要：话语理解对于许多NLP任务都是必不可少的，但是大多数现有的工作仍受到依赖框架的话语表示的限制。这项工作调查了大型语言模型（LLMS）是否捕获了跨语言和框架概括的话语知识。我们沿两个维度解决了这个问题：（1）开发一个统一的话语关系标签，以促进跨语性和跨框架的话语分析，以及（2）探测LLMS以评估它们是否编码可通用的话语抽象。使用多语言话语关系分类作为测试床，我们研究了一组23个不同尺寸和多语言功能的LLM。我们的结果表明，LLM，尤其是那些具有多语言培训语料库的LLM，可以概括跨语言和框架的话语信息。进一步的层次分析表明，在话语层面上的语言概括在中间层中最为突出。最后，我们的错误分析提供了充满挑战的关系类别的说明。

Title: Compositional Subspace Representation Fine-tuning for Adaptive Large Language Models

• Authors: Andy Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10617
• Pdf URL: https://arxiv.org/pdf/2503.10617
• Copy Paste: [[2503.10617]] Compositional Subspace Representation Fine-tuning for Adaptive Large Language Models(https://arxiv.org/abs/2503.10617)
• Keywords: language model, gpt
• Abstract: Adapting large language models to multiple tasks can cause cross-skill interference, where improvements for one skill degrade another. While methods such as LoRA impose orthogonality constraints at the weight level, they do not fully address interference in hidden-state representations. We propose Compositional Subspace Representation Fine-tuning (CS-ReFT), a novel representation-based approach that learns multiple orthonormal subspace transformations, each specializing in a distinct skill, and composes them via a lightweight router. By isolating these subspace edits in the hidden state, rather than weight matrices, CS-ReFT prevents cross-task conflicts more effectively. On the AlpacaEval benchmark, applying CS-ReFT to Llama-2-7B achieves a 93.94% win rate, surpassing GPT-3.5 Turbo (86.30%) while requiring only 0.0098% of model parameters. These findings show that specialized representation edits, composed via a simple router, significantly enhance multi-task instruction following with minimal overhead.
• 摘要：将大型语言模型调整为多个任务可能会导致跨技能干扰，其中一种技能的改进会使另一种技能降低。尽管诸如Lora之类的方法在重量级别施加正交性约束，但它们并不能完全解决隐藏状态表示的干扰。我们提出了组成子空间表示微调（CS-REFT），这是一种基于新颖的表示的方法，可以学习多个正统的子空间变换，每种都专门具有独特的技能，并通过轻量级路由器组成。通过在隐藏状态下隔离这些子空间编辑，而不是重量矩阵，CS reft可以更有效地防止交叉任务冲突。在Alpacaeval基准测试中，对Llama-2-7B应用CS-REFT的胜利率达到93.94％，超过了GPT-3.5 Turbo（86.30％），而仅需要0.0098％的模型参数。这些发现表明，通过简单的路由器组成的专门表示编辑，可以显着增强随后的多任务指令，并以最小的开销。

Title: From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM

• Authors: Kshitij Ambilduke, Ben Peters, Sonal Sannigrahi, Anil Keshwani, Tsz Kin Lam, Bruno Martins, Marcely Zanon Boito, André F.T. Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10620
• Pdf URL: https://arxiv.org/pdf/2503.10620
• Copy Paste: [[2503.10620]] From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM(https://arxiv.org/abs/2503.10620)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown remarkable performance and generalization capabilities across multiple languages and tasks, making them very attractive targets for multi-modality integration (e.g., images or speech). In this work, we extend an existing LLM to the speech modality via speech discretization and continued pre-training. In particular, we are interested in multilingual LLMs, such as TOWER, as their pre-training setting allows us to treat discretized speech input as an additional translation language. The resulting open-source model, SPIRE, is able to transcribe and translate English speech input while maintaining TOWER's original performance on translation-related tasks, showcasing that discretized speech input integration as an additional language is feasible during LLM adaptation. We make our code and models available to the community.
• 摘要：大型语言模型（LLM）表现出了多种语言和任务的出色性能和概括能力，使它们非常有吸引力的多模式集成目标（例如，图像或语音）。在这项工作中，我们通过语音离散化并继续预训练将现有的LLM扩展到语音方式。特别是，我们对诸如Tower之类的多语言LLM感兴趣，因为它们的预训练设置使我们能够将离散的语音输入视为另一种翻译语言。由此产生的开源模型Spire能够抄录和翻译英语语音输入，同时维护与翻译相关的任务的原始性能，表明在LLM Adaptation期间，离散的语音输入集成为附加语言是可行的。我们将代码和模型提供给社区。