2025-03-20

Title: Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM

• Authors: Yazeed Alnumay, Alexandre Barbet, Anna Bialas, William Darling, Shaan Desai, Joan Devassy, Kyle Duffy, Stephanie Howe, Olivia Lasche, Justin Lee, Anirudh Shrinivason, Jennifer Tracey
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.14603
• Pdf URL: https://arxiv.org/pdf/2503.14603
• Copy Paste: [[2503.14603]] Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM(https://arxiv.org/abs/2503.14603)
• Keywords: language model, llm
• Abstract: Building high-quality large language models (LLMs) for enterprise Arabic applications remains challenging due to the limited availability of digitized Arabic data. In this work, we present a data synthesis and refinement strategy to help address this problem, namely, by leveraging synthetic data generation and human-in-the-loop annotation to expand our Arabic training corpus. We further present our iterative post training recipe that is essential to achieving state-of-the-art performance in aligning the model with human preferences, a critical aspect to enterprise use cases. The culmination of this effort is the release of a small, 7B, open-weight model that outperforms similarly sized peers in head-to-head comparisons and on Arabic-focused benchmarks covering cultural knowledge, instruction following, RAG, and contextual faithfulness.
• 摘要：由于数字化的阿拉伯数据的可用性有限，为企业阿拉伯语应用构建高质量的大语言模型（LLM）仍然具有挑战性。在这项工作中，我们提出了一个数据综合和完善策略，即通过利用合成数据生成和人类在循环注释来扩大我们的阿拉伯语培训语料库来帮助解决此问题。我们进一步介绍了我们的迭代后培训配方，这对于在将模型与人类偏好保持一致，这是企业用例的关键方面。这项工作的高潮是释放一个小的7B开放权重模型，该模型在面对面的比较和以阿拉伯语为中心的基准上优于大小的同龄人，涵盖文化知识，抹布，抹布和上下文忠诚。

Title: Retrieval-Augmented Simulacra: Generative Agents for Up-to-date and Knowledge-Adaptive Simulations

• Authors: Hikaru Shimadzu, Takehito Utsuro, Daisuke Kitayama
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14620
• Pdf URL: https://arxiv.org/pdf/2503.14620
• Copy Paste: [[2503.14620]] Retrieval-Augmented Simulacra: Generative Agents for Up-to-date and Knowledge-Adaptive Simulations(https://arxiv.org/abs/2503.14620)
• Keywords: llm, chat, agent
• Abstract: In the 2023 edition of the White Paper on Information and Communications, it is estimated that the population of social networking services in Japan will exceed 100 million by 2022, and the influence of social networking services in Japan is growing significantly. In addition, marketing using SNS and research on the propagation of emotions and information on SNS are being actively conducted, creating the need for a system for predicting trends in SNS interactions. We have already created a system that simulates the behavior of various communities on SNS by building a virtual SNS environment in which agents post and reply to each other in a chat community created by agents using a LLMs. In this paper, we evaluate the impact of the search extension generation mechanism used to create posts and replies in a virtual SNS environment using a simulation system on the ability to generate posts and replies. As a result of the evaluation, we confirmed that the proposed search extension generation mechanism, which mimics human search behavior, generates the most natural exchange.
• 摘要：在2023年的有关信息和通信的白皮书中，据估计，到2022年，日本的社交网络服务人口将超过1亿，而日本社交网络服务的影响正在显着增长。此外，正在积极地进行使用SNS的营销以及有关情绪和信息传播的传播的研究，从而需要系统来预测SNS相互作用趋势的系统。我们已经创建了一个系统，该系统通过构建虚拟SNS环境来模拟SNS上各个社区的行为，在该环境中，代理商在代理商使用LLMS创建的聊天社区中互相发布并互相回复。在本文中，我们评估了搜索扩展生成机制的影响，用于在虚拟SNS环境中使用仿真系统在生成帖子和答复的能力上创建帖子和答复。作为评估的结果，我们证实了拟议的搜索扩展生成机制，该机制模仿人类搜索行为，产生了最自然的交换。

Title: ConQuer: A Framework for Concept-Based Quiz Generation

• Authors: Yicheng Fu, Zikui Wang, Liuxin Yang, Meiqing Huo, Zhongdongming Dai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.14662
• Pdf URL: https://arxiv.org/pdf/2503.14662
• Copy Paste: [[2503.14662]] ConQuer: A Framework for Concept-Based Quiz Generation(https://arxiv.org/abs/2503.14662)
• Keywords: llm
• Abstract: Quizzes play a crucial role in education by reinforcing students' understanding of key concepts and encouraging self-directed exploration. However, compiling high-quality quizzes can be challenging and require deep expertise and insight into specific subject matter. Although LLMs have greatly enhanced the efficiency of quiz generation, concerns remain regarding the quality of these AI-generated quizzes and their educational impact on students. To address these issues, we introduce ConQuer, a concept-based quiz generation framework that leverages external knowledge sources. We employ comprehensive evaluation dimensions to assess the quality of the generated quizzes, using LLMs as judges. Our experiment results demonstrate a 4.8% improvement in evaluation scores and a 77.52% win rate in pairwise comparisons against baseline quiz sets. Ablation studies further underscore the effectiveness of each component in our framework. Code available at this https URL.
• 摘要：测验通过增强学生对关键概念的理解和鼓励自我指导的探索，在教育中起着至关重要的作用。但是，编译高质量的测验可能具有挑战性，需要深入的专业知识和深入了解特定主题。尽管LLM大大提高了测验的效率，但仍然关注这些AI生成的测验及其对学生的教育影响。为了解决这些问题，我们介绍了征服，这是一个基于概念的测验生成框架，利用外部知识来源。我们使用全面的评估维度来评估生成的测验的质量，并使用LLM作为法官。我们的实验结果表明，在对基线测验集的成对比较中，评估得分提高了4.8％，胜率为77.52％。消融研究进一步强调了每个组件在我们的框架中的有效性。可在此HTTPS URL上找到代码。

Title: Generating Medically-Informed Explanations for Depression Detection using LLMs

• Authors: Xiangyong Chen, Xiaochuan Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14671
• Pdf URL: https://arxiv.org/pdf/2503.14671
• Copy Paste: [[2503.14671]] Generating Medically-Informed Explanations for Depression Detection using LLMs(https://arxiv.org/abs/2503.14671)
• Keywords: language model, llm
• Abstract: Early detection of depression from social media data offers a valuable opportunity for timely intervention. However, this task poses significant challenges, requiring both professional medical knowledge and the development of accurate and explainable models. In this paper, we propose LLM-MTD (Large Language Model for Multi-Task Depression Detection), a novel approach that leverages a pre-trained large language model to simultaneously classify social media posts for depression and generate textual explanations grounded in medical diagnostic criteria. We train our model using a multi-task learning framework with a combined loss function that optimizes both classification accuracy and explanation quality. We evaluate LLM-MTD on the benchmark Reddit Self-Reported Depression Dataset (RSDD) and compare its performance against several competitive baseline methods, including traditional machine learning and fine-tuned BERT. Our experimental results demonstrate that LLM-MTD achieves state-of-the-art performance in depression detection, showing significant improvements in AUPRC and other key metrics. Furthermore, human evaluation of the generated explanations reveals their relevance, completeness, and medical accuracy, highlighting the enhanced interpretability of our approach. This work contributes a novel methodology for depression detection that combines the power of large language models with the crucial aspect of explainability.
• 摘要：从社交媒体数据中对抑郁症的早期发现为及时干预提供了宝贵的机会。但是，这项任务构成了重大挑战，需要专业的医学知识和准确和可解释的模型的发展。在本文中，我们提出了LLM-MTD（用于多任务抑郁症检测的大语言模型），这是一种新型方法，它利用预先训练的大型语言模型同时将社交媒体帖子分类为抑郁症，并在医学诊断标准中产生基于医疗诊断标准的文本解释。我们使用多任务学习框架训练模型，并具有组合的损失功能，可优化分类精度和解释质量。我们在基准REDDIT自我报告的抑郁数据集（RSDD）上评估了LLM-MTD，并将其性能与几种竞争性基线方法（包括传统的机器学习和微调BERT）进行比较。我们的实验结果表明，LLM-MTD在抑郁症检测中实现最先进的表现，显示AUPRC和其他关键指标的显着改善。此外，对生成的解释的人类评估揭示了它们的相关性，完整性和医疗准确性，从而强调了我们方法的增强性。这项工作为抑郁症检测提供了一种新颖的方法，将大语言模型的力量与解释性的关键方面相结合。

Title: HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding

• Authors: Rui Yang, Lin Song, Yicheng Xiao, Runhui Huang, Yixiao Ge, Ying Shan, Hengshuang Zhao
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.14694
• Pdf URL: https://arxiv.org/pdf/2503.14694
• Copy Paste: [[2503.14694]] HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding(https://arxiv.org/abs/2503.14694)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have significantly propelled the development of large multi-modal models (LMMs), highlighting the potential for general and intelligent assistants. However, most LMMs model visual and textual modalities separately, leading to recent efforts to develop native LMMs using a single transformer. Despite the promise, these native models are resource-intensive and often exhibit performance gaps compared to their compositional counterparts. To alleviate this issue, we propose a simple yet efficient method to construct a baseline for the native and end-to-end large multi-modal model in a single transformer. First, we propose a new early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an auto-regressive manner. Second, we devise an efficient training recipe for the proposed model, which harnesses the prior knowledge of the pre-trained models, addressing both the performance limitations and the challenge of resource consumption. The proposed model demonstrates superior performance compared to other LMMs using one transformer and significantly narrows the performance gap with compositional LMMs.
• 摘要：大型语言模型（LLM）的最新进展显着推动了大型多模式模型（LMM）的发展，从而突出了一般和智能助手的潜力。但是，大多数LMMS分别模型的视觉和文本模式分别模型，导致最近使用单个变压器开发天然LMM的努力。尽管有希望，这些本地模型是资源密集的，与其组成相比，这些模型经常表现出性能差距。为了减轻此问题，我们提出了一种简单而有效的方法，以在单个变压器中为天然和端到端大型多模式构建基线。首先，我们提出了一种新的早期融合LMM，该LMM可以在早期阶段融合多模式输入，并以自动回归方式响应视觉说明。其次，我们为提出的模型设计了有效的培训配方，该模型利用了预培训模型的先验知识，可以解决绩效限制和资源消费的挑战。提出的模型与其他LMM相比使用了一个变压器，表现出了卓越的性能，并用组成LMM显着缩小了性能差距。

Title: Second language Korean Universal Dependency treebank v1.2: Focus on data augmentation and annotation scheme refinement

• Authors: Hakyung Sung, Gyu-Ho Shin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14718
• Pdf URL: https://arxiv.org/pdf/2503.14718
• Copy Paste: [[2503.14718]] Second language Korean Universal Dependency treebank v1.2: Focus on data augmentation and annotation scheme refinement(https://arxiv.org/abs/2503.14718)
• Keywords: language model
• Abstract: We expand the second language (L2) Korean Universal Dependencies (UD) treebank with 5,454 manually annotated sentences. The annotation guidelines are also revised to better align with the UD framework. Using this enhanced treebank, we fine-tune three Korean language models and evaluate their performance on in-domain and out-of-domain L2-Korean datasets. The results show that fine-tuning significantly improves their performance across various metrics, thus highlighting the importance of using well-tailored L2 datasets for fine-tuning first-language-based, general-purpose language models for the morphosyntactic analysis of L2 data.
• 摘要：我们扩展了第二语言（L2）韩国通用依赖性（UD）树库，并使用5,454个手动注释的句子。还对注释指南进行了修订，以更好地与UD框架保持一致。使用这种增强的树库，我们调整了三种韩国语言模型，并评估了它们在内域和外域L2-Korean数据集上的性能。结果表明，微调显着提高了各种指标的性能，从而突出了使用良好的L2数据集用于微调第一语言的通用语言模型的重要性，以对L2数据进行形态词法分析。

Title: Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence

• Authors: Sophia Hager, David Mueller, Kevin Duh, Nicholas Andrews
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.14749
• Pdf URL: https://arxiv.org/pdf/2503.14749
• Copy Paste: [[2503.14749]] Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence(https://arxiv.org/abs/2503.14749)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) are increasingly used for factual question-answering, it becomes more important for LLMs to have the capability to communicate the likelihood that their answer is correct. For these verbalized expressions of uncertainty to be meaningful, they should reflect the error rates at the expressed level of confidence. However, when prompted to express confidence, the error rates of current LLMs are inconsistent with their communicated confidences, highlighting the need for uncertainty quantification methods. Many prior methods calculate lexical uncertainty, estimating a model's confidence in the specific string it generated. In some cases, however, it may be more useful to estimate semantic uncertainty, or the model's confidence in the answer regardless of how it is verbalized. We propose a simple procedure, uncertainty distillation, to teach an LLM to verbalize calibrated semantic confidences. Using held-out data to map initial uncertainty estimates to meaningful probabilities, we create examples annotated with verbalized probabilities for supervised fine-tuning. We demonstrate our method yields verbalized confidences that correlate with observed error rates with a small fine-tuned language model as well as with larger instruction-tuned models, and find that our semantic uncertainty correlates well with lexical uncertainty on short answers.
• 摘要：随着大型语言模型（LLM）越来越多地用于事实提问，因此对于LLM而言，有能力传达其答案是正确的可能性。为了使这些不确定性的口头表达有意义，它们应在表达的置信度水平上反映错误率。但是，当提示表达信心时，当前LLM的错误率与他们传达的信心不一致，这突出了对不确定性量化方法的需求。许多先前的方法计算词汇不确定性，估计模型对其生成的特定字符串的信心。但是，在某些情况下，估计语义不确定性或模型对答案的信心可能更有用，无论其口头如何。我们提出了一个简单的程序，不确定性蒸馏，以教授LLM语言校准的语义信心。使用固定数据将初始不确定性估算映射到有意义的概率上，我们创建了带有口头化概率的示例以进行监督微调。我们证明我们的方法产生了口头上的信心，这些信心与观察到的错误率与小型微调语言模型以及较大的指导调整模型相关，并发现我们的语义不确定性与短答案的词汇不确定性息息相关。

Title: MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models

• Authors: Chejian Xu, Jiawei Zhang, Zhaorun Chen, Chulin Xie, Mintong Kang, Yujin Potter, Zhun Wang, Zhuowen Yuan, Alexander Xiong, Zidi Xiong, Chenhui Zhang, Lingzhi Yuan, Yi Zeng, Peiyang Xu, Chengquan Guo, Andy Zhou, Jeffrey Ziwei Tan, Xuandong Zhao, Francesco Pinto, Zhen Xiang, Yu Gai, Zinan Lin, Dan Hendrycks, Bo Li, Dawn Song
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.14827
• Pdf URL: https://arxiv.org/pdf/2503.14827
• Copy Paste: [[2503.14827]] MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models(https://arxiv.org/abs/2503.14827)
• Keywords: hallucination
• Abstract: Multimodal foundation models (MMFMs) play a crucial role in various applications, including autonomous driving, healthcare, and virtual assistants. However, several studies have revealed vulnerabilities in these models, such as generating unsafe content by text-to-image models. Existing benchmarks on multimodal models either predominantly assess the helpfulness of these models, or only focus on limited perspectives such as fairness and privacy. In this paper, we present the first unified platform, MMDT (Multimodal DecodingTrust), designed to provide a comprehensive safety and trustworthiness evaluation for MMFMs. Our platform assesses models from multiple perspectives, including safety, hallucination, fairness/bias, privacy, adversarial robustness, and out-of-distribution (OOD) generalization. We have designed various evaluation scenarios and red teaming algorithms under different tasks for each perspective to generate challenging data, forming a high-quality benchmark. We evaluate a range of multimodal models using MMDT, and our findings reveal a series of vulnerabilities and areas for improvement across these perspectives. This work introduces the first comprehensive and unique safety and trustworthiness evaluation platform for MMFMs, paving the way for developing safer and more reliable MMFMs and systems. Our platform and benchmark are available at this https URL.
• 摘要：多模式基础模型（MMFMS）在各种应用中起着至关重要的作用，包括自主驾驶，医疗保健和虚拟助手。但是，一些研究揭示了这些模型中的漏洞，例如通过文本对图像模型生成不安全的内容。多模型模型的现有基准主要评估这些模型的有益性，或者仅关注有限的观点，例如公平和隐私。在本文中，我们介绍了第一个统一平台MMDT（多模式解码器），旨在为MMFM提供全面的安全性和可信度评估。我们的平台从多个角度评估了模型，包括安全，幻觉，公平/偏见，隐私，对抗性鲁棒性和分布外（OOD）概括。我们已经在不同的任务下设计了各种评估方案和红色团队算法，以生成具有挑战性的数据，从而形成高质量的基准。我们使用MMDT评估了一系列多模型模型，我们的发现揭示了一系列脆弱性和在这些角度进行改进的领域。这项工作介绍了MMFMS的第一个全面，独特的安全和可信度评估平台，为开发更安全，更可靠的MMFM和系统铺平了道路。我们的平台和基准可以在此HTTPS URL上找到。

Title: MetaLadder: Ascending Mathematical Solution Quality via Analogical-Problem Reasoning Transfer

• Authors: Honglin Lin, Zhuoshi Pan, Yu Li, Qizhi Pei, Xin Gao, Mengzhang Cai, Conghui He, Lijun Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.14891
• Pdf URL: https://arxiv.org/pdf/2503.14891
• Copy Paste: [[2503.14891]] MetaLadder: Ascending Mathematical Solution Quality via Analogical-Problem Reasoning Transfer(https://arxiv.org/abs/2503.14891)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have demonstrated promising capabilities in solving mathematical reasoning tasks, leveraging Chain-of-Thought (CoT) data as a vital component in guiding answer generation. Current paradigms typically generate CoT and answers directly for a given problem, diverging from human problem-solving strategies to some extent. Humans often solve problems by recalling analogous cases and leveraging their solutions to reason about the current task. Inspired by this cognitive process, we propose \textbf{MetaLadder}, a novel framework that explicitly prompts LLMs to recall and reflect on meta-problems, those structurally or semantically analogous problems, alongside their CoT solutions before addressing the target problem. Additionally, we introduce a problem-restating mechanism to enhance the model's comprehension of the target problem by regenerating the original question, which further improves reasoning accuracy. Therefore, the model can achieve reasoning transfer from analogical problems, mimicking human-like "learning from examples" and generalization abilities. Extensive experiments on mathematical benchmarks demonstrate that our MetaLadder significantly boosts LLMs' problem-solving accuracy, largely outperforming standard CoT-based methods (\textbf{10.3\%} accuracy gain) and other methods. Our code and data has been released at this https URL.
• 摘要：大型语言模型（LLMS）已经证明了在解决数学推理任务方面具有有希望的能力，利用思考链（COT）数据是指导答案生成的重要组成部分。当前的范例通常会产生COT并直接解决给定问题，从某种程度上与人类解决问题的策略有所不同。人类经常通过召回类似案件并利用其解决方案来解决当前任务来解决问题。受这个认知过程的启发，我们提出了\ textbf {MetalAdder}，这是一个新颖的框架，明确提示LLMS回忆和反思元问题，这些问题在结构或语义上类似的问题以及在解决目标问题之前与COT解决方案一起。此外，我们引入了一种问题纠正的机制，以通过再生原始问题来增强模型对目标问题的理解，从而进一步提高了推理准确性。因此，该模型可以从类似问题中实现推理转移，模仿人类的“从例子学习”和概括能力。关于数学基准测试的广泛实验表明，我们的MetalAdder显着提高了LLMS解决问题的准确性，在很大程度上优于基于标准的COT方法（\ TextBf {10.3 \％}的精度增益）和其他方法。我们的代码和数据已在此HTTPS URL上发布。

Title: Deep Contrastive Unlearning for Language Models

• Authors: Estrid He, Tabinda Sarwar, Ibrahim Khalil, Xun Yi, Ke Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.14900
• Pdf URL: https://arxiv.org/pdf/2503.14900
• Copy Paste: [[2503.14900]] Deep Contrastive Unlearning for Language Models(https://arxiv.org/abs/2503.14900)
• Keywords: language model
• Abstract: The past a few years have witnessed the great success of large language models, demonstrating powerful capabilities in comprehending textual data and generating human-like languages. Large language models achieve success by being trained on vast amounts of textual data, including online sources with copyrighted content and user-generated knowledge. However, this comes at a cost: the potential risk of exposing users' privacy and violating copyright protections. Thus, to safeguard individuals' "right to be forgotten", there has been increasing interests in machine unlearning -- the process of removing information carried by particular training samples from a model while not deteriorating its predictive quality. This is a challenging task due to the black-box nature of language models. Most existing studies focus on mitigating the impact of those forgot samples upon a model's outputs, and do not explicitly consider the geometric distributions of samples in the latent space of a model. To address this issue, we propose a machine unlearning framework, named Deep Contrastive Unlearning for fine-Tuning (DeepCUT) language models. Our proposed model achieves machine unlearning by directly optimizing the latent space of a model. Comprehensive experiments on real-world datasets demonstrate the effectiveness and efficiency of DeepCUT with consistent and significant improvement over baseline methods.
• 摘要：过去的几年见证了大型语言模型的巨大成功，展示了理解文本数据和产生类似人类语言的强大能力。大型语言模型通过接受大量文本数据的培训，包括具有版权内容和用户生成的知识的在线资源来实现成功。但是，这是有代价的：暴露用户隐私和侵犯版权保护的潜在风险。因此，为了维护个人的“被遗忘的权利”，对机器学习的兴趣越来越高 - 从模型中删除特定培训样本的信息的过程，同时并没有恶化其预测性质量。由于语言模型的黑框性质，这是一项具有挑战性的任务。大多数现有研究的重点是减轻这些忘记样本对模型输出的影响，并且没有明确考虑模型潜在空间中样品的几何分布。为了解决这个问题，我们提出了一个机器学习框架，该框架被称为“深度对比”，以进行微调（DEEPCUT）语言模型。我们提出的模型通过直接优化模型的潜在空间来实现机器的学习。对现实世界数据集的综合实验证明了深度曲线的有效性和效率，并且对基线方法一致且显着改善。

Title: MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models

• Authors: Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, Jun Zhou, Yanfang Ye, Chuxu Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.14917
• Pdf URL: https://arxiv.org/pdf/2503.14917
• Copy Paste: [[2503.14917]] MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models(https://arxiv.org/abs/2503.14917)
• Keywords: language model, llm
• Abstract: High-quality data plays a critical role in the pretraining and fine-tuning of large language models (LLMs), even determining their performance ceiling to some degree. Consequently, numerous data selection methods have been proposed to identify subsets of data that can effectively and efficiently enhance model performance. However, most of these methods focus on general data selection and tend to overlook the specific nuances of domain-related data. In this paper, we introduce MASS, a \textbf{MA}thematical data \textbf{S}election framework using the \textbf{S}kill graph for pretraining LLMs in the mathematical reasoning domain. By taking into account the unique characteristics of mathematics and reasoning, we construct a skill graph that captures the mathematical skills and their interrelations from a reference dataset. This skill graph guides us in assigning quality scores to the target dataset, enabling us to select the top-ranked subset which is further used to pretrain LLMs. Experimental results demonstrate the efficiency and effectiveness of MASS across different model sizes (1B and 7B) and pretraining datasets (web data and synthetic data). Specifically, in terms of efficiency, models trained on subsets selected by MASS can achieve similar performance to models trained on the original datasets, with a significant reduction in the number of trained tokens - ranging from 50\% to 70\% fewer tokens. In terms of effectiveness, when trained on the same amount of tokens, models trained on the data selected by MASS outperform those trained on the original datasets by 3.3\% to 5.9\%. These results underscore the potential of MASS to improve both the efficiency and effectiveness of pretraining LLMs.
• 摘要：高质量的数据在大型语言模型（LLMS）的预处理和微调中起着至关重要的作用，甚至在某种程度上确定其性能上限。因此，已经提出了许多数据选择方法来识别可以有效有效地增强模型性能的数据子集。但是，这些方法中的大多数都集中在一般数据选择上，并倾向于忽略与域相关数据的特定细微差别。在本文中，我们介绍了质量，使用\ textbf {s} kill杀死图形在数学推理域中预处理llms。通过考虑数学和推理的独特特征，我们构建了一个技能图，该技能图可以捕获参考数据集中的数学技能及其相互关系。此技能图指导我们将质量得分分配给目标数据集，从而使我们能够选择最高的子集，该子集进一步用于预识LLM。实验结果证明了质量跨不同模型尺寸（1B和7B）以及预处理数据集（Web数据和合成数据）的效率和有效性。具体而言，就效率而言，对质量选择的子集进行训练的模型可以实现与原始数据集中训练的模型相似的性能，而训练的令牌数量显着减少，范围从50 \％\％到70 \％\％\％\％。在有效性方面，当接受相同数量的令牌培训时，对Mass Proferform在原始数据集中训练的数据进行培训的模型在3.3 \％至5.9 \％上进行了培训。这些结果强调了质量提高LLM的效率和有效性的潜力。

Title: ML-Triton, A Multi-Level Compilation and Language Extension to Triton GPU Programming

• Authors: Dewei Wang, Wei Zhu, Liyang Ling, Ettore Tiotto, Quintin Wang, Whitney Tsang, Julian Opperman, Jacky Deng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14985
• Pdf URL: https://arxiv.org/pdf/2503.14985
• Copy Paste: [[2503.14985]] ML-Triton, A Multi-Level Compilation and Language Extension to Triton GPU Programming(https://arxiv.org/abs/2503.14985)
• Keywords: llm
• Abstract: In the era of LLMs, dense operations such as GEMM and MHA are critical components. These operations are well-suited for parallel execution using a tilebased approach. While traditional GPU programming often relies on low level interfaces like CUDA or SYCL, Triton has emerged as a DSL that offers a more user-friendly and portable alternative by programming at a higher level. The current Triton starts at the workgroup (aka threadblock) level, and directly lowers to per-thread level. And then attempt to coalesce and amend through a series of passes, promoting information from low-level representation. We believe this is pre-mature lowering based on the below observations. 1. GPU has a hierarchical structure both physically and logically. Modern GPUs often feature SIMD units capable of directly operating on tiles on a warp or warpgroup basis, such as blocked load and blocked MMA. 2. Multi-level gradual lowering can make compiler decoupled and clean by separating considerations inter and intra a logical layer. 3. Kernel developers often need fine control to get good performance on the latest hardware. FlashAttention2 advocates explicit data partition between warps to make a performance boost. In this context, we propose ML-Triton which features multi-level compilation flow and programming interface. Our approach begins at the workgroup level and progressively lowers to the warp and intrinsic level, implementing a multilevel lowering align with the hierarchical nature of GPU. Additionally, we extend triton language to support user-set compiler hint and warp level programming, enabling researchers to get good out-of-the box performance without awaiting compiler updates. Experimental results demonstrate that our approach achieves performance above 95% of expert-written kernels on Intel GPU, as measured by the geometric mean.
• 摘要：在LLM的时代，诸如GEMM和MHA之类的密集作业是关键组成部分。这些操作非常适合使用基于瓷砖的方法并行执行。尽管传统的GPU编程通常依赖于CUDA或SYCL等低级别接口，但Triton已成为DSL，它通过更高级别的编程提供了更具用户友好和便携式的替代方案。当前的Triton从工作组（又称螺纹块）级别开始，直接降低到每线程级别。然后尝试通过一系列通行证进行合并和修改，从而促进低级代表的信息。我们认为，根据以下观察结果，这是预成熟的降低。 1。GPU在物理和逻辑上具有分层结构。现代GPU通常具有能够以经纱或经纱为基础在瓷砖上直接操作的SIMD单元，例如负载和阻塞的MMA。 2。多级逐渐降低可以通过分开考虑因素和内部逻辑层来使编译器解耦和清洁。 3。内核开发人员通常需要良好的控制才能在最新的硬件上获得良好的性能。 FlashAttention2提倡在WARPS之间明确的数据分区，以提高性能。在这种情况下，我们提出了具有多级汇编流和编程接口的ML-Triton。我们的方法始于工作组级别，并逐渐降低到翘曲和内在级别，实施了与GPU的层次结构的多级降低一致性。此外，我们扩展了Triton语言，以支持用户集编译器提示和翘曲级别的编程，使研究人员能够在不等待编译器更新的情况下获得良好的盒子性能。实验结果表明，通过几何平均值来衡量，我们的方法在英特尔GPU上达到了超过95％的专家写入内核的性能。

Title: Inspecting the Representation Manifold of Differentially-Private Text

• Authors: Stefan Arnold
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14991
• Pdf URL: https://arxiv.org/pdf/2503.14991
• Copy Paste: [[2503.14991]] Inspecting the Representation Manifold of Differentially-Private Text(https://arxiv.org/abs/2503.14991)
• Keywords: language model
• Abstract: Differential Privacy (DP) for text has recently taken the form of text paraphrasing using language models and temperature sampling to better balance privacy and utility. However, the geometric distortion of DP regarding the structure and complexity in the representation space remains unexplored. By estimating the intrinsic dimension of paraphrased text across varying privacy budgets, we find that word-level methods severely raise the representation manifold, while sentence-level methods produce paraphrases whose manifolds are topologically more consistent with human-written paraphrases. Among sentence-level methods, masked paraphrasing, compared to causal paraphrasing, demonstrates superior preservation of structural complexity, suggesting that autoregressive generation propagates distortions from unnatural word choices that cascade and inflate the representation space.
• 摘要：用于文本的差异隐私（DP）最近使用语言模型和温度抽样采取了文本释义的形式，以更好地平衡隐私和效用。但是，DP关于表示空间中的结构和复杂性的几何变形尚未探索。通过估计各种隐私预算中的释义文本的固有维度，我们发现单词级方法严重提高了表示形式的流动性，而句子级别的方法产生的释义酶在其歧义上与人类写的释义更加一致。在句子级别的方法中，与因果释义相比，掩盖的释义表明结构复杂性的保留较高，这表明自回归产生传播了从级联和膨胀表示空间的不自然单词选择中的扭曲。

Title: Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering

• Authors: Francesco Maria Molfese, Luca Moroni, Luca Gioffrè, Alessandro Scirè, Simone Conia, Roberto Navigli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.14996
• Pdf URL: https://arxiv.org/pdf/2503.14996
• Copy Paste: [[2503.14996]] Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering(https://arxiv.org/abs/2503.14996)
• Keywords: language model, llm, prompt
• Abstract: One of the most widely used tasks to evaluate Large Language Models (LLMs) is Multiple-Choice Question Answering (MCQA). While open-ended question answering tasks are more challenging to evaluate, MCQA tasks are, in principle, easier to assess, as the model's answer is thought to be simple to extract and is directly compared to a set of predefined choices. However, recent studies have started to question the reliability of MCQA evaluation, showing that multiple factors can significantly impact the reported performance of LLMs, especially when the model generates free-form text before selecting one of the answer choices. In this work, we shed light on the inconsistencies of MCQA evaluation strategies, which can lead to inaccurate and misleading model comparisons. We systematically analyze whether existing answer extraction methods are aligned with human judgment, and how they are influenced by answer constraints in the prompt across different domains. Our experiments demonstrate that traditional evaluation strategies often underestimate LLM capabilities, while LLM-based answer extractors are prone to systematic errors. Moreover, we reveal a fundamental trade-off between including format constraints in the prompt to simplify answer extraction and allowing models to generate free-form text to improve reasoning. Our findings call for standardized evaluation methodologies and highlight the need for more reliable and consistent MCQA evaluation practices.
• 摘要：评估大语言模型（LLMS）的最广泛使用的任务之一是多项选择问题答案（MCQA）。虽然评估的开放式问题回答任务更具挑战性，但原则上，MCQA任务更容易评估，因为该模型的答案被认为是易于提取的，并且与一组预定义的选择直接进行了比较。但是，最近的研究开始质疑MCQA评估的可靠性，表明多个因素可以显着影响LLM的报告，尤其是当模型在选择答案选择之一之前生成自由形式文本时。在这项工作中，我们阐明了MCQA评估策略的不一致，这可能导致不准确和误导性的模型比较。我们系统地分析现有的答案提取方法是否与人类判断力一致，以及它们如何受到跨不同领域提示中答案约束的影响。我们的实验表明，传统的评估策略经常低估LLM功能，而基于LLM的答案提取器容易出现系统错误。此外，我们揭示了在提示中包括格式约束的基本权衡，以简化答案提取，并允许模型生成自由形式的文本以改善推理。我们的发现要求标准化评估方法，并强调需要更可靠，一致的MCQA评估实践。

Title: LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?

• Authors: Amr Keleg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15003
• Pdf URL: https://arxiv.org/pdf/2503.15003
• Copy Paste: [[2503.15003]] LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?(https://arxiv.org/abs/2503.15003)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have the potential of being useful tools that can automate tasks and assist humans. However, these models are more fluent in English and more aligned with Western cultures, norms, and values. Arabic-specific LLMs are being developed to better capture the nuances of the Arabic language, as well as the views of the Arabs. Yet, Arabs are sometimes assumed to share the same culture. In this position paper, I discuss the limitations of this assumption and provide preliminary thoughts for how to build systems that can better represent the cultural diversity within the Arab world. The invalidity of the cultural homogeneity assumption might seem obvious, yet, it is widely adopted in developing multilingual and Arabic-specific LLMs. I hope that this paper will encourage the NLP community to be considerate of the cultural diversity within various communities speaking the same language.
• 摘要：大型语言模型（LLMS）具有可以自动化任务并协助人类的有用工具的潜力。但是，这些模型的英语更加流利，并且与西方文化，规范和价值观更加一致。正在开发阿拉伯语特异性LLM，以更好地捕捉阿拉伯语的细微差别以及阿拉伯人的观点。然而，有时假定阿拉伯人共享相同的文化。在该立场论文中，我讨论了这一假设的局限性，并为如何建立可以更好地代表阿拉伯世界内文化多样性的系统提供了初步思想。文化同质性假设的无效性似乎很明显，但是在开发多语言和阿拉伯特异性LLM中，它被广泛采用。我希望本文将鼓励NLP社区对讲相同语言的各个社区的文化多样性进行体贴。

Title: SPADE: Systematic Prompt Framework for Automated Dialogue Expansion in Machine-Generated Text Detection

• Authors: Haoyi Li, Angela Yifei Yuan, Soyeon Caren Han, Christopher Leckie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15044
• Pdf URL: https://arxiv.org/pdf/2503.15044
• Copy Paste: [[2503.15044]] SPADE: Systematic Prompt Framework for Automated Dialogue Expansion in Machine-Generated Text Detection(https://arxiv.org/abs/2503.15044)
• Keywords: language model, llm, prompt, chat, agent
• Abstract: The increasing capability of large language models (LLMs) to generate synthetic content has heightened concerns about their misuse, driving the development of Machine-Generated Text (MGT) detection models. However, these detectors face significant challenges due to the lack of systematically generated, high-quality datasets for training. To address this issue, we propose five novel data augmentation frameworks for synthetic user dialogue generation through a structured prompting approach, reducing the costs associated with traditional data collection methods. Our proposed method yields 14 new dialogue datasets, which we benchmark against seven MGT detection models. The results demonstrate improved generalization performance when utilizing a mixed dataset produced by our proposed augmentation framework. Furthermore, considering that real-world agents lack knowledge of future opponent utterances, we simulate online dialogue detection and examine the relationship between chat history length and detection accuracy. We also benchmark online detection performance with limited chat history on our frameworks. Our open-source datasets can be downloaded from this https URL.
• 摘要：大型语言模型（LLMS）生成合成内容的能力提高，人们对它们的滥用的担忧加剧，推动了机器生成的文本（MGT）检测模型的发展。但是，由于缺乏系统生成的高质量培训数据集，这些探测器面临重大挑战。为了解决这个问题，我们通过结构化提示方法提出了五个新颖的数据增强框架，以生成合成用户对话，从而降低了与传统数据收集方法相关的成本。我们提出的方法产生了14个新的对话数据集，我们对七个MGT检测模型进行了基准测试。结果表明，当利用我们提出的增强框架生成的混合数据集时，概括性能得到了改善。此外，考虑到现实世界中的代理缺乏对未来对手话语的了解，我们模拟了在线对话检测并检查聊天历史记录长度和检测准确性之间的关系。我们还基于在线检测性能，在我们的框架上聊天历史记录有限。我们的开源数据集可以从此HTTPS URL下载。

Title: ELTEX: A Framework for Domain-Driven Synthetic Data Generation

• Authors: Arina Razmyslovich, Kseniia Murasheva, Sofia Sedlova, Julien Capitaine, Eugene Dmitriev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15055
• Pdf URL: https://arxiv.org/pdf/2503.15055
• Copy Paste: [[2503.15055]] ELTEX: A Framework for Domain-Driven Synthetic Data Generation(https://arxiv.org/abs/2503.15055)
• Keywords: language model, gpt, llm, prompt
• Abstract: We present ELTEX (Efficient LLM Token Extraction), a domain-driven framework for generating high-quality synthetic training data in specialized domains. While Large Language Models (LLMs) have shown impressive general capabilities, their performance in specialized domains like cybersecurity remains limited by the scarcity of domain-specific training data. ELTEX addresses this challenge by systematically integrating explicit domain indicator extraction with dynamic prompting to preserve critical domain knowledge throughout the generation process. We demonstrate ELTEX's effectiveness in the context of blockchain-related cyberattack detection, where we fine-tune Gemma-2B using various combinations of real and ELTEX-generated data. Our results show that the ELTEX-enhanced model achieves performance competitive with GPT-4 across both standard classification metrics and uncertainty calibration, while requiring significantly fewer computational resources. We release a curated synthetic dataset of social media texts for cyberattack detection in blockchain. Our work demonstrates that domain-driven synthetic data generation can effectively bridge the performance gap between resource-efficient models and larger architectures in specialized domains.
• 摘要：我们提出了ELTEX（有效的LLM令牌提取），这是一个域驱动的框架，用于在专用域中生成高质量的合成训练数据。尽管大型语言模型（LLM）表现出令人印象深刻的一般能力，但它们在网络安全等专业领域的性能仍然受到特定领域特定培训数据的稀缺的限制。 Eletex通过系统地将显式域指标提取与动态提示来解决这一挑战，从而在整个生成过程中保留关键的域知识。我们在与区块链相关的网络攻击检测的背景下展示了ELTEX的有效性，在此，我们使用真实和ELTEX生成的数据的各种组合对Gemma-2b进行了微调。我们的结果表明，ELTEX增强模型在标准分类指标和不确定性校准中都与GPT-4达到了性能竞争，同时需要更少的计算资源。我们发布了社交媒体文本的精选合成数据集，以用于区块链中的网络攻击检测。我们的工作表明，域驱动的合成数据生成可以有效地弥合资源有效模型与专用域中较大体系结构之间的性能差距。

Title: Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification

• Authors: Shichen Li, Zhongqing Wang, Zheyu Zhao, Yue Zhang, Peifeng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15117
• Pdf URL: https://arxiv.org/pdf/2503.15117
• Copy Paste: [[2503.15117]] Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification(https://arxiv.org/abs/2503.15117)
• Keywords: language model, llm
• Abstract: Model editing aims at selectively updating a small subset of a neural model's parameters with an interpretable strategy to achieve desired modifications. It can significantly reduce computational costs to adapt to large language models (LLMs). Given its ability to precisely target critical components within LLMs, model editing shows great potential for efficient fine-tuning applications. In this work, we investigate model editing to serve an efficient method for adapting LLMs to solve aspect-based sentiment classification. Through causal interventions, we trace and determine which neuron hidden states are essential for the prediction of the model. By performing interventions and restorations on each component of an LLM, we identify the importance of these components for aspect-based sentiment classification. Our findings reveal that a distinct set of mid-layer representations is essential for detecting the sentiment polarity of given aspect words. Leveraging these insights, we develop a model editing approach that focuses exclusively on these critical parts of the LLM, leading to a more efficient method for adapting LLMs. Our in-domain and out-of-domain experiments demonstrate that this approach achieves competitive results compared to the currently strongest methods with significantly fewer trainable parameters, highlighting a more efficient and interpretable fine-tuning strategy.
• 摘要：模型编辑旨在选择性地更新神经模型参数的一小部分，并具有可解释的策略以实现所需的修改。它可以大大降低适应大型语言模型（LLM）的计算成本。鉴于其能够精确靶向LLM中关键组件的能力，模型编辑显示出有效的微调应用的巨大潜力。在这项工作中，我们调查了模型编辑，以提供一种有效的方法来调整LLMS解决基于方面的情感分类。通过因果干预，我们追踪并确定哪些神经元隐藏状态对于模型的预测至关重要。通过对LLM的每个组件执行干预措施和修复，我们确定了这些组件对基于方面的情感分类的重要性。我们的发现表明，一组独特的中层表示对于检测给定方面单词的情感极性至关重要。利用这些见解，我们开发了一种模型编辑方法，该方法仅专注于LLM的这些关键部分，从而导致更有效地适应LLM的方法。我们的内域和室外实验表明，与当前最强大的训练参数相比，这种方法可以实现竞争成果，这突出了一种更有效，更可解释的微调策略。

Title: Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors

• Authors: Dominik Macko, Robert Moro, Ivan Srba
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.15128
• Pdf URL: https://arxiv.org/pdf/2503.15128
• Copy Paste: [[2503.15128]] Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors(https://arxiv.org/abs/2503.15128)
• Keywords: llm
• Abstract: Since the proliferation of LLMs, there have been concerns about their misuse for harmful content creation and spreading. Recent studies justify such fears, providing evidence of LLM vulnerabilities and high potential of their misuse. Humans are no longer able to distinguish between high-quality machine-generated and authentic human-written texts. Therefore, it is crucial to develop automated means to accurately detect machine-generated content. It would enable to identify such content in online information space, thus providing an additional information about its credibility. This work addresses the problem by proposing a robust fine-tuning process of LLMs for the detection task, making the detectors more robust against obfuscation and more generalizable to out-of-distribution data.
• 摘要：自从LLM的扩散以来，人们一直担心他们滥用有害内容创建和传播。最近的研究证明了这种恐惧是合理的，提供了LLM漏洞的证据和滥用的高潜力。人类不再能够区分高质量的机器生成和真实的人文所写文本。因此，开发自动化手段以准确检测机器生成的内容至关重要。它可以在在线信息领域中识别此类内容，从而提供有关其信誉的其他信息。这项工作通过为检测任务提出一个强大的LLMS微调过程来解决该问题，从而使检测器更强大地抵抗混淆，并且更易于分发数据。

Title: Comparing Llama3 and DeepSeekR1 on Biomedical Text Classification Tasks

• Authors: Yuting Guo, Abeed Sarker
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.15169
• Pdf URL: https://arxiv.org/pdf/2503.15169
• Copy Paste: [[2503.15169]] Comparing Llama3 and DeepSeekR1 on Biomedical Text Classification Tasks(https://arxiv.org/abs/2503.15169)
• Keywords: language model, llm
• Abstract: This study compares the performance of two open-source large language models (LLMs)-Llama3-70B and DeepSeekR1-distill-Llama3-70B-on six biomedical text classification tasks. Four tasks involve data from social media, while two tasks focus on clinical notes from electronic health records, and all experiments were performed in zero-shot settings. Performance metrics, including precision, recall, and F1 scores, were measured for each task, along with their 95% confidence intervals. Results demonstrated that DeepSeekR1-distill-Llama3-70B generally performs better in terms of precision on most tasks, with mixed results on recall. While the zero-shot LLMs demonstrated high F1 scores for some tasks, they grossly underperformed on others, for data from both sources. The findings suggest that model selection should be guided by the specific requirements of the health-related text classification tasks, particularly when considering the precision-recall trade-offs, and that, in the presence of annotated data, supervised classification approaches may be more reliable than zero-shot LLMs.
• 摘要：这项研究比较了两种开源大语模型（LLMS）-Llama3-70B和DeepSeekr1-Distill-LlAma-llama3-70B-ON六个生物医学文本分类任务的性能。四个任务涉及社交媒体的数据，而两个任务则集中在电子健康记录的临床注释上，所有实验均在零摄影设置中进行。为每个任务以及其95％的置信区间测量了包括精度，召回和F1分数在内的性能指标。结果表明，DeepSeekr1-Distill-llama3-70b通常在大多数任务上的精度方面表现更好，而召回结果混合了。尽管零拍的LLM在某些任务中表现出很高的F1分数，但它们在其他任务上的表现严重不足，而来自两个来源的数据。调查结果表明，应指导模型选择与健康相关的文本分类任务的特定要求，尤其是在考虑精确核心权衡取舍时，并且在存在带注释的数据的情况下，监督分类方法可能比零击LLM更可靠。

Title: Entity-aware Cross-lingual Claim Detection for Automated Fact-checking

• Authors: Rrubaa Panchendrarajan, Arkaitz Zubiaga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15220
• Pdf URL: https://arxiv.org/pdf/2503.15220
• Copy Paste: [[2503.15220]] Entity-aware Cross-lingual Claim Detection for Automated Fact-checking(https://arxiv.org/abs/2503.15220)
• Keywords: language model
• Abstract: Identifying claims requiring verification is a critical task in automated fact-checking, especially given the proliferation of misinformation on social media platforms. Despite significant progress in the task, there remain open challenges such as dealing with multilingual and multimodal data prevalent in online discourse. Addressing the multilingual challenge, recent efforts have focused on fine-tuning pre-trained multilingual language models. While these models can handle multiple languages, their ability to effectively transfer cross-lingual knowledge for detecting claims spreading on social media remains under-explored. In this paper, we introduce \textit{EX-Claim}, an entity-aware cross-lingual claim detection model that generalizes well to handle claims written in any language. The model leverages entity information derived from named entity recognition and entity linking techniques to improve the language-level performance of both seen and unseen languages during training. Extensive experiments conducted on three datasets from different social media platforms demonstrate that our proposed model significantly outperforms the baselines, across 27 languages, and achieves the highest rate of knowledge transfer, even with limited training data.
• 摘要：确定需要验证的主张是自动化事实检查的关键任务，尤其是考虑到社交媒体平台上错误信息的扩散。尽管这项任务取得了重大进展，但仍存在公开挑战，例如处理在线话语中普遍存在的多语言和多模式数据。在应对多语言挑战时，最近的努力集中在微调预训练的多语言模型上。尽管这些模型可以处理多种语言，但它们有效地传递跨语性知识以检测在社交媒体上传播的主张的能力仍未得到充分探索。在本文中，我们介绍了\ textit {ex-claim}，这是一种实体感知的跨语性索赔检测模型，可以很好地处理以任何语言编写的索赔。该模型利用从指定实体识别和实体链接技术得出的实体信息，以提高培训期间可见语言和看不见的语言的语言级别的性能。在不同社交媒体平台的三个数据集上进行的广泛实验表明，我们提出的模型在27种语言上大大优于基线，即使使用有限的培训数据也达到了最高的知识转移率。

Title: Exploring Large Language Models for Word Games:Who is the Spy?

• Authors: Chentian Wei, Jiewei Chen, Jinzhu Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.15235
• Pdf URL: https://arxiv.org/pdf/2503.15235
• Copy Paste: [[2503.15235]] Exploring Large Language Models for Word Games:Who is the Spy?(https://arxiv.org/abs/2503.15235)
• Keywords: language model, llm, chain-of-thought, agent
• Abstract: Word games hold significant research value for natural language processing (NLP), game theory, and related fields due to their rule-based and situational nature. This study explores how large language models (LLMs) can be effectively involved in word games and proposes a training-free framework. "Shei Shi Wo Di" or "Who is the Spy" in English, is a classic word game. Using this game as an example, we introduce a Chain-of-Thought (CoT)-based scheduling framework to enable LLMs to achieve excellent performance in tasks such as inferring role words and disguising their identities. We evaluate the framework's performance based on game success rates and the accuracy of the LLM agents' analytical results. Experimental results affirm the framework's effectiveness, demonstrating notable improvements in LLM performance across multiple datasets. This work highlights the potential of LLMs in mastering situational reasoning and social interactions within structured game environments. Our code is publicly available at this https URL.
• 摘要：文字游戏具有自然语言处理（NLP），游戏理论和相关领域的重要研究价值，这是由于其基于规则和情境性质。这项研究探讨了大型语言模型（LLM）如何有效地参与文字游戏，并提出了无训练的框架。 “ Shei shi wo di”或“是英语的间谍”是一个经典的文字游戏。以此游戏为例，我们介绍了基于经过深思熟虑的（COT）的调度框架，以使LLMS能够在诸如推断角色词和掩盖其身份之类的任务中实现出色的性能。我们根据游戏的成功率和LLM代理分析结果的准确性来评估框架的性能。实验结果肯定了该框架的有效性，证明了多个数据集的LLM性能的显着改善。这项工作突出了LLM在结构化游戏环境中掌握情境推理和社交互动中的潜力。我们的代码在此HTTPS URL上公开可用。

Title: BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?

• Authors: Pierre Chambon, Baptiste Roziere, Benoit Sagot, Gabriel Synnaeve
• Subjects: cs.CL, cs.AI, cs.CC
• Abstract URL: https://arxiv.org/abs/2503.15242
• Pdf URL: https://arxiv.org/pdf/2503.15242
• Copy Paste: [[2503.15242]] BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?(https://arxiv.org/abs/2503.15242)
• Keywords: language model, llm
• Abstract: We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generating code with specified time and space complexities. This benchmark addresses the gap in current evaluations that often overlook the ability of models to comprehend and produce code constrained by computational complexity. BigO(Bench) includes tooling to infer the algorithmic complexity of any Python function from profiling measurements, including human- or LLM-generated solutions. BigO(Bench) also includes of set of 3,105 coding problems and 1,190,250 solutions from Code Contests annotated with inferred (synthetic) time and space complexity labels from the complexity framework, as well as corresponding runtime and memory footprint values for a large set of input sizes. We present results from evaluating multiple state-of-the-art language models on this benchmark, highlighting their strengths and weaknesses in handling complexity requirements. In particular, token-space reasoning models are unrivaled in code generation but not in complexity understanding, hinting that they may not generalize well to tasks for which no reward was given at training time.
• 摘要：我们介绍了Bigo（Bench），这是一种新颖的编码基准测试，旨在评估生成语言模型在理解和生成具有指定时间和空间复杂性的代码方面的功能。该基准测定了当前评估中的差距，这些差距通常忽略了模型理解和产生受计算复杂性约束的代码的能力。 Bigo（台）包括通过分析测量值（包括人类或LLM生成的溶液）来推断任何Python功能的算法复杂性的工具。 Bigo（台）还包括一组3,105个编码问题和1,190,250个解决方案，来自代码竞赛的解决方案，并从复杂性框架中带有推断（合成的）时间和空间复杂性标签，以及相应的运行时和内存足迹值，用于大量输入尺寸。我们在此基准上评估了多种最先进的语言模型，从而提出了结果，突出了它们在处理复杂性要求中的优势和缺点。特别是，代币空间推理模型在代码生成中是无与伦比的，但没有在复杂的理解中，暗示它们可能无法很好地推广到在培训时没有奖励的任务。

Title: MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration

• Authors: David Wan, Justin Chih-Yao Chen, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.15272
• Pdf URL: https://arxiv.org/pdf/2503.15272
• Copy Paste: [[2503.15272]] MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration(https://arxiv.org/abs/2503.15272)
• Keywords: language model, llm, agent
• Abstract: Multi-agent collaboration among models has shown promise in reasoning tasks but is underexplored in long-form generation tasks like summarization and question-answering. We extend multi-agent multi-model reasoning to generation, specifically to improving faithfulness through refinement, i.e., revising model-generated outputs to remove factual inconsistencies. We investigate how iterative collaboration among multiple instances and types of large language models (LLMs) enhances subtasks in the refinement process, such as error detection, critiquing unfaithful sentences, and making corrections based on critiques. We design intrinsic evaluations for each subtask, with our findings indicating that both multi-agent (multiple instances) and multi-model (diverse LLM types) approaches benefit error detection and critiquing. Additionally, reframing critiquing and refinement as reranking rather than generation tasks improves multi-agent performance. We consolidate these insights into a final "recipe" called Multi-Agent Multi-Model Refinement (MAMM-Refine), where multi-agent and multi-model collaboration significantly boosts performance on three summarization datasets as well as on long-form question answering, demonstrating the effectiveness and generalizability of our recipe.
• 摘要：模型之间的多代理协作在推理任务方面表现出了希望，但在诸如摘要和提问的长期生成任务中没有反应。我们将多代理多模型推理扩展到生成，特别是通过改进来提高忠诚，即修订模型生成的输出以消除事实不一致。我们研究了多种实例和类型的大型语言模型（LLMS）之间的迭代协作如何在完善过程中增强子任务，例如错误检测，批评不忠的句子以及基于批评进行更正。我们为每个子任务设计了内在的评估，我们的发现表明多代理（多个实例）和多模型（不同的LLM类型）方法均效果错误检测和批评。此外，将批评和改进的重新定义为重读而不是生成任务可改善多代理性能。我们将这些洞察力巩固在称为多代理多模型改进（Mamm-refine）的最终“食谱”中，其中多代理和多模型协作大大提高了三个摘要数据集的性能，以及在长形式问题上的答案，表明了我们的播放器的有效性和概括性。

Title: TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification

• Authors: Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou, Chengqing Zong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15289
• Pdf URL: https://arxiv.org/pdf/2503.15289
• Copy Paste: [[2503.15289]] TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification(https://arxiv.org/abs/2503.15289)
• Keywords: gpt, llm, prompt
• Abstract: LLMs have achieved remarkable fluency and coherence in text generation, yet their widespread adoption has raised concerns about content reliability and accountability. In high-stakes domains such as healthcare, law, and news, it is crucial to understand where and how the content is created. To address this, we introduce the Text pROVEnance (TROVE) challenge, designed to trace each sentence of a target text back to specific source sentences within potentially lengthy or multi-document inputs. Beyond identifying sources, TROVE annotates the fine-grained relationships (quotation, compression, inference, and others), providing a deep understanding of how each target sentence is formed. To benchmark TROVE, we construct our dataset by leveraging three public datasets covering 11 diverse scenarios (e.g., QA and summarization) in English and Chinese, spanning source texts of varying lengths (0-5k, 5-10k, 10k+), emphasizing the multi-document and long-document settings essential for provenance. To ensure high-quality data, we employ a three-stage annotation process: sentence retrieval, GPT provenance, and human provenance. We evaluate 11 LLMs under direct prompting and retrieval-augmented paradigms, revealing that retrieval is essential for robust performance, larger models perform better in complex relationship classification, and closed-source models often lead, yet open-source models show significant promise, particularly with retrieval augmentation.
• 摘要：LLM在文本生成方面取得了显着的流利性和连贯性，但他们的广泛采用使人们对内容可靠性和问责制引起了人们的关注。在医疗保健，法律和新闻等高风险领域中，了解内容的创建在哪里以及如何创建至关重要。为了解决这个问题，我们介绍了文本出处（TROVE）挑战，旨在将目标文本的每个句子追溯到潜在的冗长或多文件输入中的特定源句子中。除了识别来源外，Trove还注释了细粒度的关系（引号，压缩，推理等），还对每个目标句子的形成方式都深入了解。为了基于基准Trove，我们通过利用三个公共数据集来构建数据集，涵盖英文和中文的11种不同方案（例如，QA和摘要），涵盖长度不同的源文本（0-5k，5-10k，10k+），强调多纪念碑和长期记录的遗址和长期记录量表和长期记录的设置。为了确保高质量的数据，我们采用了三阶段的注释过程：句子检索，GPT出处和人类出处。我们在直接提示和检索式范式下评估了11个LLM，表明检索对于稳健的性能至关重要，较大的模型在复杂的关系分类中表现更好，并且封闭源模型经常引导，但开源模型却表现出了巨大的希望，尤其是在检索增强方面。

Title: Inside-Out: Hidden Factual Knowledge in LLMs

• Authors: Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpector, Jonathan Herzig, Roi Reichart
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15299
• Pdf URL: https://arxiv.org/pdf/2503.15299
• Copy Paste: [[2503.15299]] Inside-Out: Hidden Factual Knowledge in LLMs(https://arxiv.org/abs/2503.15299)
• Keywords: language model, llm
• Abstract: This work presents a framework for assessing whether large language models (LLMs) encode more factual knowledge in their parameters than what they express in their outputs. While a few studies hint at this possibility, none has clearly defined or demonstrated this phenomenon. We first propose a formal definition of knowledge, quantifying it for a given question as the fraction of correct-incorrect answer pairs where the correct one is ranked higher. This gives rise to external and internal knowledge, depending on the information used to score individual answer candidates: either the model's observable token-level probabilities or its intermediate computations. Hidden knowledge arises when internal knowledge exceeds external knowledge. We then present a case study, applying this framework to three popular open-weights LLMs in a closed-book QA setup. Our results indicate that: (1) LLMs consistently encode more factual knowledge internally than what they express externally, with an average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a model can internally know an answer perfectly, yet fail to generate it even once, despite large-scale repeated sampling of 1,000 answers. This reveals fundamental limitations in the generation capabilities of LLMs, which (3) puts a practical constraint on scaling test-time compute via repeated answer sampling in closed-book QA: significant performance improvements remain inaccessible because some answers are practically never sampled, yet if they were, we would be guaranteed to rank them first.
• 摘要：这项工作提出了一个框架，用于评估大型语言模型（LLMS）是否在其参数中编码更多的事实知识，而不是他们在输出中表达的知识。尽管一些研究暗示了这种可能性，但没有一个清楚地定义或证明了这种现象。我们首先提出了对知识的形式定义，将其量​​化为给定的问题，因为正确的答案对的比例是正确的答案对，其中正确的答案对。这引起了外部和内部知识，具体取决于用于评分单个答案候选者的信息：模型可观察到的令牌级别的概率或其中间计算。当内部知识超过外部知识时，就会出现隐藏的知识。然后，我们提出了一个案例研究，将此框架应用于封闭式质量检查设置中的三个流行的开放式LLM。我们的结果表明：（1）LLM始终在内部编码比外部表达的更多事实知识，平均差距为40％。 （2）令人惊讶的是，某些知识是如此深层隐藏，以至于模型可以在内部完美地知道答案，但尽管重复了1,000个答案，但模型甚至无法产生一次。这揭示了LLMS发电能力的根本限制，（3）在闭幕式质量检查中通过重复的答案采样对测试时间计算进行了实际限制：显着的绩效改进仍然无法访问，因为实际上从来没有采样过一些答案，但是，如果它们是确保他们首先排名第一。

Title: SPILL: Domain-Adaptive Intent Clustering based on Selection and Pooling with Large Language Models

• Authors: I-Fan Lin, Faegheh Hasibi, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15351
• Pdf URL: https://arxiv.org/pdf/2503.15351
• Copy Paste: [[2503.15351]] SPILL: Domain-Adaptive Intent Clustering based on Selection and Pooling with Large Language Models(https://arxiv.org/abs/2503.15351)
• Keywords: language model, llm
• Abstract: In this paper, we propose Selection and Pooling with Large Language Models (SPILL), an intuitive and domain-adaptive method for intent clustering without fine-tuning. Existing embeddings-based clustering methods rely on a few labeled examples or unsupervised fine-tuning to optimize results for each new dataset, which makes them less generalizable to multiple datasets. Our goal is to make these existing embedders more generalizable to new domain datasets without further fine-tuning. Inspired by our theoretical derivation and simulation results on the effectiveness of sampling and pooling techniques, we view the clustering task as a small-scale selection problem. A good solution to this problem is associated with better clustering performance. Accordingly, we propose a two-stage approach: First, for each utterance (referred to as the seed), we derive its embedding using an existing embedder. Then, we apply a distance metric to select a pool of candidates close to the seed. Because the embedder is not optimized for new datasets, in the second stage, we use an LLM to further select utterances from these candidates that share the same intent as the seed. Finally, we pool these selected candidates with the seed to derive a refined embedding for the seed. We found that our method generally outperforms directly using an embedder, and it achieves comparable results to other state-of-the-art studies, even those that use much larger models and require fine-tuning, showing its strength and efficiency. Our results indicate that our method enables existing embedders to be further improved without additional fine-tuning, making them more adaptable to new domain datasets. Additionally, viewing the clustering task as a small-scale selection problem gives the potential of using LLMs to customize clustering tasks according to the user's goals.
• 摘要：在本文中，我们提出了使用大语言模型（Spill）的选择和汇总，这是一种直观而域的自适应方法，用于无需微调。现有的基于嵌入的聚类方法依赖一些标记的示例或无监督的微调来优化每个新数据集的结果，这使得它们不太可以在多个数据集中概括。我们的目标是使这些现有的嵌入者更能在不进行微调的情况下对新域数据集进行概括。受我们的理论推导和仿真结果的启发，我们将聚类任务视为一个小规模的选择问题。解决此问题的一个很好的解决方案与更好的聚类性能有关。因此，我们提出了一种两个阶段的方法：首先，对于每个话语（称为种子），我们使用现有嵌入器得出其嵌入。然后，我们将距离度量标准选择接近种子的候选者。由于嵌入器未针对新数据集进行优化，因此在第二阶段，我们使用LLM从这些候选人中进一步选择与种子相同意图的话语。最后，我们将这些选定的候选物与种子汇集，以得出种子的精致嵌入。我们发现，我们的方法通常使用嵌入器直接胜过表现，并且与其他最先进的研究相当，即使是那些使用更大模型并需要微调的结果，也表现出其强度和效率。我们的结果表明，我们的方法使现有的嵌入式可以进一步改进，而无需进行其他微调，从而使它们更适合新域数据集。此外，将聚类任务视为小规模的选择问题，可能会根据用户的目标自定义聚类任务。

Title: SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representation

• Authors: Thomas Pickard, Aline Villavicencio, Maggie Mi, Wei He, Dylan Phelps, Carolina Scarton, Marco Idiart
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.15358
• Pdf URL: https://arxiv.org/pdf/2503.15358
• Copy Paste: [[2503.15358]] SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representation(https://arxiv.org/abs/2503.15358)
• Keywords: language model, llm
• Abstract: Idiomatic expressions present a unique challenge in NLP, as their meanings are often not directly inferable from their constituent words. Despite recent advancements in Large Language Models (LLMs), idiomaticity remains a significant obstacle to robust semantic representation. We present datasets and tasks for SemEval-2025 Task 1: AdMiRe (Advancing Multimodal Idiomaticity Representation), which challenges the community to assess and improve models' ability to interpret idiomatic expressions in multimodal contexts and in multiple languages. Participants competed in two subtasks: ranking images based on their alignment with idiomatic or literal meanings, and predicting the next image in a sequence. The most effective methods achieved human-level performance by leveraging pretrained LLMs and vision-language models in mixture-of-experts settings, with multiple queries used to smooth over the weaknesses in these models' representations of idiomaticity.
• 摘要：惯用表达在NLP中带来了独特的挑战，因为它们的含义通常无法直接从其组成词中推断出来。尽管大型语言模型（LLM）最近取得了进步，但惯用性仍然是强大语义表示的重要障碍。我们介绍了Semeval-2025任务1：欣赏（推进多模式惯用性表示）的数据集和任务，这挑战了社区评估和提高模型在多模式上下文和多种语言中解释惯用表达的能力。参与者参加了两个子任务：根据惯用或字面意义的对齐方式对图像进行排名，并按顺序预测下一个图像。最有效的方法通过利用预验证的LLM和视觉语言模型来实现人级的性能，并使用多个查询来平滑这些模型的惯用性表示中的弱点。

Title: Real-world validation of a multimodal LLM-powered pipeline for High-Accuracy Clinical Trial Patient Matching leveraging EHR data

• Authors: Anatole Callies (Inato), Quentin Bodinier (Inato), Philippe Ravaud (Inato, Université Paris Cité and Université Sorbonne Paris Nord, INSERM, INRAE, Paris, France, Centre d'epidémiologie clinique, AP-HP, Hôpital Hôtel Dieu, Paris, France), Kourosh Davarpanah (Inato)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.15374
• Pdf URL: https://arxiv.org/pdf/2503.15374
• Copy Paste: [[2503.15374]] Real-world validation of a multimodal LLM-powered pipeline for High-Accuracy Clinical Trial Patient Matching leveraging EHR data(https://arxiv.org/abs/2503.15374)
• Keywords: llm
• Abstract: Background: Patient recruitment in clinical trials is hindered by complex eligibility criteria and labor-intensive chart reviews. Prior research using text-only models have struggled to address this problem in a reliable and scalable way due to (1) limited reasoning capabilities, (2) information loss from converting visual records to text, and (3) lack of a generic EHR integration to extract patient data. Methods: We introduce a broadly applicable, integration-free, LLM-powered pipeline that automates patient-trial matching using unprocessed documents extracted from EHRs. Our approach leverages (1) the new reasoning-LLM paradigm, enabling the assessment of even the most complex criteria, (2) visual capabilities of latest LLMs to interpret medical records without lossy image-to-text conversions, and (3) multimodal embeddings for efficient medical record search. The pipeline was validated on the n2c2 2018 cohort selection dataset (288 diabetic patients) and a real-world dataset composed of 485 patients from 30 different sites matched against 36 diverse trials. Results: On the n2c2 dataset, our method achieved a new state-of-the-art criterion-level accuracy of 93\%. In real-world trials, the pipeline yielded an accuracy of 87\%, undermined by the difficulty to replicate human decision-making when medical records lack sufficient information. Nevertheless, users were able to review overall eligibility in under 9 minutes per patient on average, representing an 80\% improvement over traditional manual chart reviews. Conclusion: This pipeline demonstrates robust performance in clinical trial patient matching without requiring custom integration with site systems or trial-specific tailoring, thereby enabling scalable deployment across sites seeking to leverage AI for patient matching.
• 摘要：背景：复杂的资格标准和劳动密集型图表审查，阻碍了临床试验中的患者招募。由于（1）推理功能有限，（2）从转换视觉记录转换为文本，以及（3）缺乏通用的EHR集成来提取患者数据，因此使用仅文本模型的先前研究一直在以可靠且可扩展的方式解决此问题。方法：我们介绍了一种广泛的，无集成的LLM驱动管道，该管道使用从EHRS提取的未经处理的文档自动化了患者进行试验匹配。我们的方法利用（1）新的推理-LLM范式，甚至可以评估最复杂的标准，（2）最新LLM的视觉功能可以解释医疗记录而无需丢失的图像到文本转换，以及（3）多模态嵌入，以进行有效的病理记录搜索。该管道已在N2C2 2018同类选择数据集（288名糖尿病患者）和一个现实世界中的数据集上进行了验证，该数据集由来自30个不同地点的485名患者组成，与36种不同的试验相匹配。结果：在N2C2数据集上，我们的方法达到了93 \％的新标准级准确性。在实际试验中，该管道的准确性为87 \％，这是由于在病历缺乏足够信息时难以复制人类决策的困难而破坏。然而，用户平均能够在每位患者的9分钟以下审查总体资格，这比传统手动图表的评论的改善为80 \％。结论：该管道表明在临床试验患者匹配中表现出良好的性能，而无需与现场系统或特定于试验的裁缝进行自定义集成，从而使寻求利用AI进行患者匹配的站点的可扩展部署。

Title: VenusFactory: A Unified Platform for Protein Engineering Data Retrieval and Language Model Fine-Tuning

• Authors: Yang Tan, Chen Liu, Jingyuan Gao, Banghao Wu, Mingchen Li, Ruilin Wang, Lingrong Zhang, Huiqun Yu, Guisheng Fan, Liang Hong, Bingxin Zhou
• Subjects: cs.CL, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2503.15438
• Pdf URL: https://arxiv.org/pdf/2503.15438
• Copy Paste: [[2503.15438]] VenusFactory: A Unified Platform for Protein Engineering Data Retrieval and Language Model Fine-Tuning(https://arxiv.org/abs/2503.15438)
• Keywords: language model
• Abstract: Natural language processing (NLP) has significantly influenced scientific domains beyond human language, including protein engineering, where pre-trained protein language models (PLMs) have demonstrated remarkable success. However, interdisciplinary adoption remains limited due to challenges in data collection, task benchmarking, and application. This work presents VenusFactory, a versatile engine that integrates biological data retrieval, standardized task benchmarking, and modular fine-tuning of PLMs. VenusFactory supports both computer science and biology communities with choices of both a command-line execution and a Gradio-based no-code interface, integrating $40+$ protein-related datasets and $40+$ popular PLMs. All implementations are open-sourced on this https URL.
• 摘要：自然语言处理（NLP）对包括蛋白质工程的蛋白质工程（包括蛋白质工程）的科学领域极大地影响了，预先训练的蛋白质语言模型（PLMS）取得了巨大的成功。但是，由于数据收集，任务基准和应用的挑战，跨学科的采用仍然有限。这项工作介绍了Venusfactory，这是一种多功能引擎，可集成生物数据检索，标准化的任务基准测试和PLM的模块化微调。 VenusFactory可以选择命令行执行和基于级别的无代码界面的计算机科学和生物学社区，集成了$ 40+$ $ $ $蛋白质相关的数据集和$ 40+$ $ $ $ $ $ $。所有实现都在此HTTPS URL上开源。

Title: SkyLadder: Better and Faster Pretraining via Context Window Scheduling

• Authors: Tongyao Zhu, Qian Liu, Haonan Wang, Shiqi Chen, Xiangming Gu, Tianyu Pang, Min-Yen Kan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15450
• Pdf URL: https://arxiv.org/pdf/2503.15450
• Copy Paste: [[2503.15450]] SkyLadder: Better and Faster Pretraining via Context Window Scheduling(https://arxiv.org/abs/2503.15450)
• Keywords: llm, long context
• Abstract: Recent advancements in LLM pretraining have featured ever-expanding context windows to process longer sequences. However, our pilot study reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. This finding motivates us to explore an optimal context window scheduling strategy to better balance long-context capability with pretraining efficiency. To this end, we propose SkyLadder, a simple yet effective approach that implements a short-to-long context window transition. SkyLadder preserves strong standard benchmark performance, while matching or exceeding baseline results on long context tasks. Through extensive experiments, we pre-train 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating that SkyLadder yields consistent gains of up to 3.7% on common benchmarks, while achieving up to 22% faster training speeds compared to baselines. The code is at this https URL.
• 摘要：LLM预审进的最新进展具有不断扩展的上下文窗口，以处理更长的序列。但是，我们的试点研究表明，在固定的令牌预算下，用较短的上下文窗口预识别的模型始终优于其长篇小说。这一发现激发了我们探索最佳上下文窗口调度策略，以更好地平衡长期文化能力和训练效率。为此，我们提出了Skyladder，这是一种简单而有效的方法，它实现了短期的上下文窗口过渡。 Skyladder可以保留强大的标准基准性能，同时在长上下文任务上匹配或超过基线结果。通过广泛的实验，我们在100B代币上预先培训1B参数模型（最高32K上下文）和3B参数模型（8K上下文），这表明Skyladder在共同基准方面的一致增长均高达3.7％，而与基层相比，高达22％的训练速度最多可实现22％。该代码在此HTTPS URL上。

Title: Evaluating Bias in Retrieval-Augmented Medical Question-Answering Systems

• Authors: Yuelyu Ji, Hang Zhang, Yanshan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.15454
• Pdf URL: https://arxiv.org/pdf/2503.15454
• Copy Paste: [[2503.15454]] Evaluating Bias in Retrieval-Augmented Medical Question-Answering Systems(https://arxiv.org/abs/2503.15454)
• Keywords: llm, retrieval-augmented generation
• Abstract: Medical QA systems powered by Retrieval-Augmented Generation (RAG) models support clinical decision-making but may introduce biases related to race, gender, and social determinants of health. We systematically evaluate biases in RAG-based LLM by examining demographic-sensitive queries and measuring retrieval discrepancies. Using datasets like MMLU and MedMCQA, we analyze retrieval overlap and correctness disparities. Our findings reveal substantial demographic disparities within RAG pipelines, emphasizing the critical need for retrieval methods that explicitly account for fairness to ensure equitable clinical decision-making.
• 摘要：由检索功能增强生成（RAG）模型提供支持的医疗质量检查系统支持临床决策，但可能引入与种族，性别和社会决定因素有关的偏见。我们通过检查对人口敏感的查询并测量检索差异来系统地评估基于抹布的LLM的偏见。使用MMLU和MEDMCQA等数据集，我们分析了检索重叠和正确性差异。我们的发现揭示了抹布管道内的人口统计学差异，强调了对检索方法的关键需求，这些方法明确解释了公平性，以确保公平的临床决策。

Title: From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

• Authors: Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, Rui Yan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.15463
• Pdf URL: https://arxiv.org/pdf/2503.15463
• Copy Paste: [[2503.15463]] From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment(https://arxiv.org/abs/2503.15463)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches that assume uniform human preferences, fundamentally overlooking the diversity in user values and needs. This paper introduces a comprehensive framework for scalable personalized alignment of LLMs. We establish a systematic preference space characterizing psychological and behavioral dimensions, alongside diverse persona representations for robust preference inference in real-world scenarios. Building upon this foundation, we introduce \textsc{AlignX}, a large-scale dataset of over 1.3 million personalized preference examples, and develop two complementary alignment approaches: \textit{in-context alignment} directly conditioning on persona representations and \textit{preference-bridged alignment} modeling intermediate preference distributions. Extensive experiments demonstrate substantial improvements over existing methods, with an average 17.06\% accuracy gain across four benchmarks while exhibiting a strong adaptation capability to novel preferences, robustness to limited user data, and precise preference controllability. These results validate our framework's effectiveness, advancing toward truly user-adaptive AI systems.
• 摘要：传统上，大型语言模型（LLMS）是通过一种适合统一的人类偏好的一种尺寸适合的方法来对齐的，从根本上忽略了用户价值观和需求的多样性。本文介绍了一个综合框架，用于可扩展的LLMS个性化对齐。我们建立了一个系统的偏好空间，表征心理和行为维度，以及在现实世界中的强大偏好推论的各种角色表示。在这个基础的基础上，我们介绍了\ textsc {alignx}，这是一个超过130万个个性化偏好示例的大规模数据集，并开发了两种互补的对准方法：\ textIt {textIt {in-context对齐}直接调理人格代表和\ textit {precference-bridded bridged ailignment}模型的偏好范围。广泛的实验证明了对现有方法的实质性改进，在四个基准测试中的平均精度为17.06 \％的精度增长，同时表现出强大的适应能力，对新型偏好，对有限用户数据的鲁棒性以及精确的偏好可控性。这些结果证明了我们的框架的有效性，并朝着真正的用户自适应AI系统发展。