2025-02-17

Title: CORRECT: Context- and Reference-Augmented Reasoning and Prompting for Fact-Checking

• Authors: Delvin Ce Zhang, Dongwon Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09635
• Pdf URL: https://arxiv.org/pdf/2502.09635
• Copy Paste: [[2502.09635]] CORRECT: Context- and Reference-Augmented Reasoning and Prompting for Fact-Checking(https://arxiv.org/abs/2502.09635)
• Keywords: prompt
• Abstract: Fact-checking the truthfulness of claims usually requires reasoning over multiple evidence sentences. Oftentimes, evidence sentences may not be always self-contained, and may require additional contexts and references from elsewhere to understand coreferential expressions, acronyms, and the scope of a reported finding. For example, evidence sentences from an academic paper may need contextual sentences in the paper and descriptions in its cited papers to determine the scope of a research discovery. However, most fact-checking models mainly focus on the reasoning within evidence sentences, and ignore the auxiliary contexts and references. To address this problem, we propose a novel method, Context- and Reference-augmented Reasoning and Prompting. For evidence reasoning, we construct a three-layer evidence graph with evidence, context, and reference layers. We design intra- and cross-layer reasoning to integrate three graph layers into a unified evidence embedding. For verdict prediction, we design evidence-conditioned prompt encoder, which produces unique prompt embeddings for each claim. These evidence-conditioned prompt embeddings and claims are unified for fact-checking. Experiments verify the strength of our model.
• 摘要：事实检查索赔的真实性通常需要对多个证据判决进行推理。通常，证据句子可能并不总是独立的，可能需要其他地方的其他上下文和参考，以了解核心表达式，首字母缩写和报告的发现的范围。例如，学术论文的证据句子可能需要在论文中的上下文句子和引用论文中的描述，以确定研究发现的范围。但是，大多数事实检查模型主要集中在证据句子中的推理上，而忽略了辅助上下文和参考。为了解决这个问题，我们提出了一种新颖的方法，上下文和引用的推理和提示。对于证据推理，我们构建了一个三层证据图，其中有证据，上下文和参考层。我们设计内部和跨层的推理，将三个图层整合到统一的证据嵌入中。为了预测，我们设计了有证据的及时编码器，该编码器为每种索赔产生独特的及时嵌入。这些有证据的及时嵌入和主张是统一的，以进行事实检查。实验验证了我们模型的强度。

Title: Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?

• Authors: Sougata Saha, Saurabh Kumar Pandey, Harshit Gupta, Monojit Choudhury
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09636
• Pdf URL: https://arxiv.org/pdf/2502.09636
• Copy Paste: [[2502.09636]] Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?(https://arxiv.org/abs/2502.09636)
• Keywords: gpt, llm
• Abstract: In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to the presence of culturally-specific items and elements that might be alien to users from another culture. Our user-study on 57 book reviews from Goodreads reveal that 83\% of the reviews had at least one culture-specific difficult-to-understand element. We also evaluate the efficacy of GPT-4o in identifying such items, given the cultural background of the reader; the results are mixed, implying a significant scope for improvement. Our datasets are available here: this https URL
• 摘要：在一个迅速的全球化和数字世界中，来自不同文化的人们创建的书籍和产品评论等内容被来自世界各个角落的其他人所读和消费。在本文中，我们研究了由于具有特定于文化的项目和元素而与其他文化陌生的文化特定物品和元素，因此书面评论可理解的差距的程度和模式。我们对Goodreads的57本书评论的用户研究表明，有83％的评论至少有一个特定于文化的难以理解的元素。鉴于读者的文化背景，我们还评估了GPT-4O在识别此类项目方面的功效；结果是混合的，这意味着改进的显着范围。我们的数据集可在此处找到：此HTTPS URL

Title: Jailbreaking to Jailbreak

• Authors: Jeremy Kritz, Vaughn Robinson, Robert Vacareanu, Bijan Varjavand, Michael Choi, Bobby Gogov, Scale Red Team, Summer Yue, Willow E. Primack, Zifan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09638
• Pdf URL: https://arxiv.org/pdf/2502.09638
• Copy Paste: [[2502.09638]] Jailbreaking to Jailbreak(https://arxiv.org/abs/2502.09638)
• Keywords: language model, gpt, llm, prompt
• Abstract: Refusal training on Large Language Models (LLMs) prevents harmful outputs, yet this defense remains vulnerable to both automated and human-crafted jailbreaks. We present a novel LLM-as-red-teamer approach in which a human jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or other LLMs. We refer to the jailbroken LLMs as $J_2$ attackers, which can systematically evaluate target models using various red teaming strategies and improve its performance via in-context learning from the previous failures. Our experiments demonstrate that Sonnet 3.5 and Gemini 1.5 pro outperform other LLMs as $J_2$, achieving 93.0% and 91.0% attack success rates (ASRs) respectively against GPT-4o (and similar results across other capable LLMs) on Harmbench. Our work not only introduces a scalable approach to strategic red teaming, drawing inspiration from human red teamers, but also highlights jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard. Specifically, an LLM can bypass its own safeguards by employing a jailbroken version of itself that is willing to assist in further jailbreaking. To prevent any direct misuse with $J_2$, while advancing research in AI safety, we publicly share our methodology while keeping specific prompting details private.
• 摘要：大型语言模型（LLMS）的拒绝培训可以防止有害产出，但这种防御仍然容易受到自动化和人力制作的越狱的攻击。我们提出了一种新颖的LLM-AS-RED团队方法，其中人类越狱拒绝训练LLM，以使其愿意越狱本身或其他LLM。我们将越狱的LLMS称为$ J_2 $攻击者，该攻击者可以使用各种红色小组策略系统地评估目标模型，并通过从以前的失败中学习通过内在学习来提高其性能。我们的实验表明，SONNET 3.5和GEMINI 1.5 Pro的表现优于其他LLMS $ J_2 $，在HarmBench上分别针对GPT-4O（在其他有能力的LLM的LLMS）中分别获得93.0％和91.0％的攻击成功率（ASRS）（ASRS）。我们的工作不仅引入了一种可扩展的战略红色团队的方法，还从人类红色的团队中汲取了灵感，而且还强调了越狱到命运的突破，这是保障措施的被忽视的失败模式。具体来说，LLM可以通过雇用愿意帮助进一步越狱的越狱版来绕开自己的保障措施。为了防止使用$ J_2 $的任何直接滥用，在进行AI安全研究的同时，我们公开分享我们的方法，同时将特定的提示详细信息保密。

Title: Online Social Support Detection in Spanish Social Media Texts

• Authors: Moein Shahiki Tash, Luis Ramos, Zahra Ahani, Raul Monroy, Olga kolesnikova, Hiram Calvo, Grigori Sidorov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09640
• Pdf URL: https://arxiv.org/pdf/2502.09640
• Copy Paste: [[2502.09640]] Online Social Support Detection in Spanish Social Media Texts(https://arxiv.org/abs/2502.09640)
• Keywords: gpt
• Abstract: The advent of social media has transformed communication, enabling individuals to share their experiences, seek support, and participate in diverse discussions. While extensive research has focused on identifying harmful content like hate speech, the recognition and promotion of positive and supportive interactions remain largely unexplored. This study proposes an innovative approach to detecting online social support in Spanish-language social media texts. We introduce the first annotated dataset specifically created for this task, comprising 3,189 YouTube comments classified as supportive or non-supportive. To address data imbalance, we employed GPT-4o to generate paraphrased comments and create a balanced dataset. We then evaluated social support classification using traditional machine learning models, deep learning architectures, and transformer-based models, including GPT-4o, but only on the unbalanced dataset. Subsequently, we utilized a transformer model to compare the performance between the balanced and unbalanced datasets. Our findings indicate that the balanced dataset yielded improved results for Task 2 (Individual and Group) and Task 3 (Nation, Other, LGBTQ, Black Community, Women, Religion), whereas GPT-4o performed best for Task 1 (Social Support and Non-Support). This study highlights the significance of fostering a supportive online environment and lays the groundwork for future research in automated social support detection.
• 摘要：社交媒体的出现改变了沟通，使个人能够分享自己的经验，寻求支持并参与多样化的讨论。尽管广泛的研究集中在确定诸如仇恨言论之类的有害内容上，但对积极和支持性互动的认识和促进仍然在很大程度上没有探索。这项研究提出了一种创新的方法，可以在西班牙语社交媒体文本中检测在线社会支持。我们介绍了专门为此任务创建的第一个注释数据集，其中包含3,189个YouTube注释，分类为支持性或不支持。为了解决数据不平衡，我们使用GPT-4O来生成释义评论并创建平衡的数据集。然后，我们使用传统的机器学习模型，深度学习体系结构和基于变压器的模型（包括GPT-4O）评估了社会支持分类，但仅在不平衡的数据集中。随后，我们利用变压器模型比较平衡数据集和不平衡数据集之间的性能。我们的发现表明，平衡的数据集为任务2（个人和组）和任务3（国家，其他，LGBTQ，黑人社区，妇女，宗教）产生了改进的结果，而GPT-4O最适合任务1（社会支持和非社会支持和非-支持）。这项研究强调了促进支持性在线环境的重要性，并为自动化社会支持检测的未来研究奠定了基础。

Title: Krutrim LLM: Multilingual Foundational Model for over a Billion People

• Authors: Aditya Kallappa, Palash Kamble, Abhinav Ravi, Akshat Patidar, Vinayak Dhruv, Deepak Kumar, Raghav Awasthi, Arveti Manjunath, Shubham Agarwal, Kumar Ashish, Gautam Bhargava, Chandra Khatri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09642
• Pdf URL: https://arxiv.org/pdf/2502.09642
• Copy Paste: [[2502.09642]] Krutrim LLM: Multilingual Foundational Model for over a Billion People(https://arxiv.org/abs/2502.09642)
• Keywords: llm
• Abstract: India is a diverse society with unique challenges in developing AI systems, including linguistic diversity, oral traditions, data accessibility, and scalability. Existing foundation models are primarily trained on English, limiting their effectiveness for India's population. Indic languages comprise only 1 percent of Common Crawl corpora despite India representing 18 percent of the global population, leading to linguistic biases. Thousands of regional languages, dialects, and code mixing create additional representation challenges due to sparse training data. We introduce Krutrim LLM, a 2 trillion token multilingual model designed for India's linguistic landscape. It incorporates the largest known Indic dataset, mitigating data scarcity and ensuring balanced performance across dialects. Krutrim outperforms or matches state-of-the-art models on Indic benchmarks while maintaining competitive English performance. Despite being significantly smaller in training flops, Krutrim LLM matches or exceeds models like LLAMA-2 on 10 out of 16 tasks, with an average score of 0.57 versus 0.55. This evidences Krutrim's flexible multilingual fluency across diverse linguistic contexts. Krutrim is integrated with real-time search to improve factual accuracy in conversational AI applications. This enhances accessibility for over 1 billion users worldwide. Through intentional design choices addressing data imbalances, Krutrim LLM signifies meaningful progress in building ethical, globally representative AI models.
• 摘要：印度是一个多元化的社会，在开发AI系统方面面临独特的挑战，包括语言多样性，口头传统，数据可访问性和可扩展性。现有的基础模型主要接受英语培训，从而限制了印度人口的有效性。尽管印度占全球人口的18％，但只占普通爬网语言的1％，导致语言偏见。由于稀疏的培训数据，成千上万的区域语言，方言和代码混合造成了其他表示挑战。我们介绍了Krutrim LLM，这是一个为印度语言景观设计的200万亿代币多语言模型。它结合了最大的已知IND数据集，减轻数据稀缺性并确保跨方言的平衡性能。 Krutrim的表现优于指示基准的最先进模型，同时保持竞争激烈的英语表现。尽管在训练拖失板中较小，但Krutrim LLM匹配或超过了16个任务中的10个模型，诸如Llama-2之类的模型，平均得分为0.57对0.55。这证明了克鲁特里姆在各种语言环境中的灵活多语言流利度。 Krutrim与实时搜索集成在一起，以提高对话AI应用程序的事实准确性。这可以增强全球超过10亿用户的可访问性。通过解决数据失衡的故意设计选择，Krutrim LLM表示有意义的构建道德，全球代表性的AI模型的进展。

Title: Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification

• Authors: Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09647
• Pdf URL: https://arxiv.org/pdf/2502.09647
• Copy Paste: [[2502.09647]] Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification(https://arxiv.org/abs/2502.09647)
• Keywords: long context
• Abstract: The ability to process long contexts is crucial for many natural language processing tasks, yet it remains a significant challenge. While substantial progress has been made in enhancing the efficiency of attention mechanisms, there is still a gap in understanding how attention heads function in long-context settings. In this paper, we observe that while certain heads consistently attend to local information only, others swing between attending to local and long-context information depending on the query. This raises the question: can we identify which heads require long-context information to predict the next token accurately? We demonstrate that it's possible to predict which heads are crucial for long-context processing using only local keys. The core idea here is to exploit a simple model for the long-context scores via second moment approximations. These findings unveil simple properties of attention in the context of long sequences, and open the door to potentially significant gains in efficiency.
• 摘要：处理长篇小说的能力对于许多自然语言处理任务至关重要，但这仍然是一个重大挑战。尽管在提高注意力机制的效率方面已经取得了重大进展，但了解注意力头在长篇下说设置中的功能仍然存在差距。在本文中，我们观察到，虽然某些头部仅始终如一地参与本地信息，但其他人则根据查询的不同，在参与本地和长篇小说信息之间进行摇摆。这就提出了一个问题：我们可以确定哪些头部需要长篇文化信息才能准确预测下一个令牌？我们证明，有可能仅使用本地密钥来预测哪些头部对于长篇文化处理至关重要。这里的核心思想是通过第二次近似值来利用长篇文章得分的简单模型。这些发现在长序列的背景下揭示了注意力的简单特性，并为效率带来了潜在的显着提高的大门。

Title: Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

• Authors: Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, Zhiqiang Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09650
• Pdf URL: https://arxiv.org/pdf/2502.09650
• Copy Paste: [[2502.09650]] Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples(https://arxiv.org/abs/2502.09650)
• Keywords: language model, llm
• Abstract: The alignment of large language models (LLMs) often assumes that using more clean data yields better outcomes, overlooking the match between model capacity and example difficulty. Challenging this, we propose a new principle: Preference data vary in difficulty, and overly difficult examples hinder alignment, by exceeding the model's capacity. Through systematic experimentation, we validate this principle with three key findings: (1) preference examples vary in difficulty, as evidenced by consistent learning orders across alignment runs; (2) overly difficult examples significantly degrade performance across four LLMs and two datasets; and (3) the capacity of a model dictates its threshold for handling difficult examples, underscoring a critical relationship between data selection and model capacity. Building on this principle, we introduce Selective DPO, which filters out overly difficult examples. This simple adjustment improves alignment performance by 9-16% in win rates on the AlpacaEval 2 benchmark compared to the DPO baseline, suppressing a series of DPO variants with different algorithmic adjustments. Together, these results illuminate the importance of aligning data difficulty with model capacity, offering a transformative perspective for improving alignment strategies in LLMs. Code is available at this https URL.
• 摘要：大型语言模型（LLMS）的一致性通常假设使用更多干净的数据会产生更好的结果，从而忽略模型容量和示例难度之间的匹配。挑战这一点，我们提出了一个新的原则：偏好数据在困难方面有所不同，并且过于困难的例子通过超出模型的能力来阻碍一致性。通过系统的实验，我们通过三个关键的发现来验证这一原理：（1）偏好示例在难度上有所不同，这是通过整个对齐运行的一致学习顺序证明的； （2）在四个LLM和两个数据集中大大降低了示例过于困难的示例； （3）模型的能力决定了其处理困难示例的阈值，强调了数据选择和模型容量之间的关键关系。以此原则为基础，我们介绍了选择性DPO，该DPO过滤了过于困难的例子。与DPO基线相比，这种简单的调整使Alpacaeval 2基准的赢率提高了对齐性能的胜利率9-16％，从而抑制了一系列具有不同算法调整的DPO变体。这些结果共同阐明了将数据难度与模型容量保持一致的重要性，从而提供了改善LLM中对齐策略的变革性观点。代码可在此HTTPS URL上找到。

Title: AI-VERDE: A Gateway for Egalitarian Access to Large Language Model-Based Resources For Educational Institutions

• Authors: Paul Mithun, Enrique Noriega-Atala, Nirav Merchant, Edwin Skidmore
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.09651
• Pdf URL: https://arxiv.org/pdf/2502.09651
• Copy Paste: [[2502.09651]] AI-VERDE: A Gateway for Egalitarian Access to Large Language Model-Based Resources For Educational Institutions(https://arxiv.org/abs/2502.09651)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: We present AI-VERDE, a unified LLM-as-a-platform service designed to facilitate seamless integration of commercial, cloud-hosted, and on-premise open LLMs in academic settings. AI-VERDE streamlines access management for instructional and research groups by providing features such as robust access control, privacy-preserving mechanisms, native Retrieval-Augmented Generation (RAG) support, budget management for third-party LLM services, and both a conversational web interface and API access. In a pilot deployment at a large public university, AI-VERDE demonstrated significant engagement across diverse educational and research groups, enabling activities that would typically require substantial budgets for commercial LLM services with limited user and team management capabilities. To the best of our knowledge, AI-Verde is the first platform to address both academic and research needs for LLMs within an higher education institutional framework.
• 摘要：We present AI-VERDE, a unified LLM-as-a-platform service designed to facilitate seamless integration of commercial, cloud-hosted, and on-premise open LLMs in academic settings. AI-VERDE通过提供诸如强大的访问控制，隐私保护机制，本地检索效果生成（RAG）支持，第三方LLM服务的预算管理以及对话性Web界面等功能来简化教学和研究小组的访问管理。和API访问。在一所大型公立大学的试点部署中，AI-Verde在各种教育和研究小组中表现出了大量参与，这使活动通常需要具有有限用户和团队管理能力的商业LLM服务预算大量预算。据我们所知，AI-Verde是第一个解决高等教育机构框架内LLM的学术和研究需求的平台。

Title: Cancer Vaccine Adjuvant Name Recognition from Biomedical Literature using Large Language Models

• Authors: Hasin Rehana, Jie Zheng, Leo Yeh, Benu Bansal, Nur Bengisu Çam, Christianah Jemiyo, Brett McGregor, Arzucan Özgür, Yongqun He, Junguk Hur
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.09659
• Pdf URL: https://arxiv.org/pdf/2502.09659
• Copy Paste: [[2502.09659]] Cancer Vaccine Adjuvant Name Recognition from Biomedical Literature using Large Language Models(https://arxiv.org/abs/2502.09659)
• Keywords: language model, gpt, llm, prompt
• Abstract: Motivation: An adjuvant is a chemical incorporated into vaccines that enhances their efficacy by improving the immune response. Identifying adjuvant names from cancer vaccine studies is essential for furthering research and enhancing immunotherapies. However, the manual curation from the constantly expanding biomedical literature poses significant challenges. This study explores the automated recognition of vaccine adjuvant names using Large Language Models (LLMs), specifically Generative Pretrained Transformers (GPT) and Large Language Model Meta AI (Llama). Methods: We utilized two datasets: 97 clinical trial records from AdjuvareDB and 290 abstracts annotated with the Vaccine Adjuvant Compendium (VAC). GPT-4o and Llama 3.2 were employed in zero-shot and few-shot learning paradigms with up to four examples per prompt. Prompts explicitly targeted adjuvant names, testing the impact of contextual information such as substances or interventions. Outputs underwent automated and manual validation for accuracy and consistency. Results: GPT-4o attained 100% Precision across all situations while exhibiting notable improve in Recall and F1-scores, particularly with incorporating interventions. On the VAC dataset, GPT-4o achieved a maximum F1-score of 77.32% with interventions, surpassing Llama-3.2-3B by approximately 2%. On the AdjuvareDB dataset, GPT-4o reached an F1-score of 81.67% for three-shot prompting with interventions, surpassing Llama-3.2-3 B's maximum F1-score of 65.62%. Conclusion: Our findings demonstrate that LLMs excel at identifying adjuvant names, including rare variations of naming representation. This study emphasizes the capability of LLMs to enhance cancer vaccine development by efficiently extracting insights. Future work aims to broaden the framework to encompass various biomedical literature and enhance model generalizability across various vaccines and adjuvants.
• 摘要：动机：佐剂是一种掺入疫苗中的化学物质，可通过改善免疫反应来增强其功效。从癌症疫苗研究中鉴定辅助名称对于进一步研究和增强免疫疗法至关重要。但是，不断扩展的生物医学文献的手动策划提出了重大挑战。这项研究探讨了使用大语言模型（LLMS），特别是生成预验证的变压器（GPT）和大语言模型Meta AI（Llama）对疫苗辅助名称的自动识别。方法：我们利用了两个数据集：97个来自SwissuvaredB的临床试验记录，并用疫苗辅助汇编（VAC）注释的290个摘要。 GPT-4O和LLAMA 3.2在零射中使用，每提示最多四个示例。提示明确针对的辅助名称，测试上下文信息（例如物质或干预措施）的影响。输出经过自动化和手动验证，以获得准确性和一致性。结果：GPT-4O在所有情况下均达到100％的精度，同时在召回和F1得分方面表现出显着的改善，尤其是在纳入干预措施中。在VAC数据集上，GPT-4O在干预措施下达到了最高77.32％的F1得分，超过Llama-3.2-3b约2％。在SwixuvaredB数据集上，GPT-4O的F1得分达到了81.67％的三杆提示，并通过干预措施超过了Llama-3.2-3 B的最高F1次数为65.62％。结论：我们的发现表明，LLM在识别辅助名称方面表现出色，包括命名表示的罕见变化。这项研究强调了LLM通过有效提取见解来增强癌症疫苗发展的能力。未来的工作旨在扩大框架，以涵盖各种生物医学文献，并增强各种疫苗和佐剂的概括性。

Title: k-LLMmeans: Summaries as Centroids for Interpretable and Scalable LLM-Based Text Clustering

• Authors: Jairo Diaz-Rodriguez
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.09667
• Pdf URL: https://arxiv.org/pdf/2502.09667
• Copy Paste: [[2502.09667]] k-LLMmeans: Summaries as Centroids for Interpretable and Scalable LLM-Based Text Clustering(https://arxiv.org/abs/2502.09667)
• Keywords: llm
• Abstract: We introduce k-LLMmeans, a novel modification of the k-means clustering algorithm that utilizes LLMs to generate textual summaries as cluster centroids, thereby capturing contextual and semantic nuances often lost when relying on purely numerical means of document embeddings. This modification preserves the properties of k-means while offering greater interpretability: the cluster centroid is represented by an LLM-generated summary, whose embedding guides cluster assignments. We also propose a mini-batch variant, enabling efficient online clustering for streaming text data and providing real-time interpretability of evolving cluster centroids. Through extensive simulations, we show that our methods outperform vanilla k-means on multiple metrics while incurring only modest LLM usage that does not scale with dataset size. Finally, We present a case study showcasing the interpretability of evolving cluster centroids in sequential text streams. As part of our evaluation, we compile a new dataset from StackExchange, offering a benchmark for text-stream clustering.
• 摘要：我们介绍了K-llmmeans，这是对K-均值聚类算法的一种新颖的修改，该算法利用LLMS将文本摘要作为群集质心生成，从而捕获上下文和语义上的细微差别，通常依靠纯粹的数值文档嵌入方式时会丢失。这种修改保留了K-均值的属性，同时提供了更大的解释性：群集质心由LLM生成的摘要表示，其嵌入指南群集分配。我们还提出了一个迷你批次变体，可实现有效的在线聚类，用于流式文本数据并提供不断发展的群集质心的实时可解释性。通过大量的模拟，我们表明我们的方法的表现优于多个指标的香草k-均值，而仅产生不随数据集大小扩展的适度LLM使用情况。最后，我们提出了一项案例研究，展示了顺序文本流中不断发展的集群质心的解释性。作为评估的一部分，我们从Stackexchange编译了一个新数据集，为文本流集群提供了基准。

Title: The Science of Evaluating Foundation Models

• Authors: Jiayi Yuan, Jiamu Zhang, Andrew Wen, Xia Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09670
• Pdf URL: https://arxiv.org/pdf/2502.09670
• Copy Paste: [[2502.09670]] The Science of Evaluating Foundation Models(https://arxiv.org/abs/2502.09670)
• Keywords: llm
• Abstract: The emergent phenomena of large foundation models have revolutionized natural language processing. However, evaluating these models presents significant challenges due to their size, capabilities, and deployment across diverse applications. Existing literature often focuses on individual aspects, such as benchmark performance or specific tasks, but fails to provide a cohesive process that integrates the nuances of diverse use cases with broader ethical and operational considerations. This work focuses on three key aspects: (1) Formalizing the Evaluation Process by providing a structured framework tailored to specific use-case contexts, (2) Offering Actionable Tools and Frameworks such as checklists and templates to ensure thorough, reproducible, and practical evaluations, and (3) Surveying Recent Work with a targeted review of advancements in LLM evaluation, emphasizing real-world applications.
• 摘要：大型基础模型的新兴现象彻底改变了自然语言处理。但是，评估这些模型由于各种应用程序之间的规模，能力和部署而提出了重大挑战。现有文献通常关注各个方面，例如基准绩效或特定任务，但没有提供一个凝聚力的过程，将各种用例的细微差别与更广泛的道德和运营考虑因素相结合。这项工作重点介绍了三个关键方面：（1）通过提供针对特定用例上下文的结构化框架来形式化评估过程，（2）提供可行的工具和框架，例如清单和模板，以确保彻底，可重复的评估和实际评估，以及（3）对LLM评估中进步的有针对性审查的最新工作，强调了现实世界的应用。

Title: Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning

• Authors: Ang Li, Yichuan Mo, Mingjie Li, Yifei Wang, Yisen Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09673
• Pdf URL: https://arxiv.org/pdf/2502.09673
• Copy Paste: [[2502.09673]] Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning(https://arxiv.org/abs/2502.09673)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable success across various NLP benchmarks. However, excelling in complex tasks that require nuanced reasoning and precise decision-making demands more than raw language proficiency--LLMs must reason, i.e., think logically, draw from past experiences, and synthesize information to reach conclusions and take action. To enhance reasoning abilities, approaches such as prompting and fine-tuning have been widely explored. While these methods have led to clear improvements in reasoning, their impact on LLM safety remains less understood. In this work, we investigate the interplay between reasoning and safety in LLMs. We highlight the latent safety risks that arise as reasoning capabilities improve, shedding light on previously overlooked vulnerabilities. At the same time, we explore how reasoning itself can be leveraged to enhance safety, uncovering potential mitigation strategies. By examining both the risks and opportunities in reasoning-driven LLM safety, our study provides valuable insights for developing models that are not only more capable but also more trustworthy in real-world deployments.
• 摘要：大型语言模型（LLMS）在各种NLP基准中都取得了巨大的成功。但是，在需要细微的推理和精确决策的复杂任务中脱颖而出要比原始语言能力更高 - LLLS必须推理，即从逻辑上进行思考，从过去的经验中借鉴，并合成信息以得出结论并采取行动。为了增强推理能力，广泛探索了提示和微调等方法。尽管这些方法导致了推理的明显改善，但它们对LLM安全性的影响仍然不太了解。在这项工作中，我们研究了LLM中推理与安全之间的相互作用。我们强调了随着推理能力的提高，出现的潜在安全风险，阐明了以前被忽视的脆弱性。同时，我们探讨了如何利用推理本身来提高安全性，从而揭示潜在的缓解策略。通过研究以推理驱动的LLM安全性的风险和机会，我们的研究为开发模型提供了宝贵的见解，这些模型不仅能有能力，而且对现实世界部署更加值得信赖。

Title: The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Safety Analysis

• Authors: Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, Xiaohua Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09674
• Pdf URL: https://arxiv.org/pdf/2502.09674
• Copy Paste: [[2502.09674]] The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Safety Analysis(https://arxiv.org/abs/2502.09674)
• Keywords: language model, llm
• Abstract: Large Language Models' safety-aligned behaviors, such as refusing harmful queries, can be represented by linear directions in activation space. Previous research modeled safety behavior with a single direction, limiting mechanistic understanding to an isolated safety feature. In this work, we discover that safety-aligned behavior is jointly controlled by multi-dimensional directions. Namely, we study the vector space of representation shifts during safety fine-tuning on Llama 3 8B for refusing jailbreaks. By studying orthogonal directions in the space, we first find that a dominant direction governs the model's refusal behavior, while multiple smaller directions represent distinct and interpretable features like hypothetical narrative and role-playing. We then measure how different directions promote or suppress the dominant direction, showing the important role of secondary directions in shaping the model's refusal representation. Finally, we demonstrate that removing certain trigger tokens in harmful queries can mitigate these directions to bypass the learned safety capability, providing new insights on understanding safety alignment vulnerability from a multi-dimensional perspective. Code and artifacts are available at this https URL.
• 摘要：大型语言模型的安全一致行为（例如拒绝有害查询）可以用激活空间中的线性方向表示。先前的研究将安全行为模拟了一个方向，将机械理解限制在孤立的安全特征上。在这项工作中，我们发现与安全符合的行为由多维方向共同控制。也就是说，我们研究了代表的矢量空间在安全调查时在Llama 3 8B上进行拒绝越狱的情况。通过研究该空间中的正交方向，我们首先发现主导方向控制模型的拒绝行为，而多个较小的方向代表了不同的和可解释的特征，例如假设的叙事和角色扮演。然后，我们衡量不同方向如何促进或抑制主要方向，显示次要方向在塑造模型拒绝表示方面的重要作用。最后，我们证明，在有害查询中删除某些触发令牌可以减轻这些方向以绕过学习的安全能力，从而从多维的角度提供有关理解安全对齐脆弱性的新见解。代码和工件可在此HTTPS URL上找到。

Title: Mind What You Ask For: Emotional and Rational Faces of Persuasion by Large Language Models

• Authors: Wiktoria Mieleszczenko-Kowszewicz, Beata Bajcar, Jolanta Babiak, Berenika Dyczek, Jakub Świstak, Przemysław Biecek
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2502.09687
• Pdf URL: https://arxiv.org/pdf/2502.09687
• Copy Paste: [[2502.09687]] Mind What You Ask For: Emotional and Rational Faces of Persuasion by Large Language Models(https://arxiv.org/abs/2502.09687)
• Keywords: language model, llm, prompt
• Abstract: Be careful what you ask for, you just might get it. This saying fits with the way large language models (LLMs) are trained, which, instead of being rewarded for correctness, are increasingly rewarded for pleasing the recipient. So, they are increasingly effective at persuading us that their answers are valuable. But what tricks do they use in this persuasion? In this study, we examine what are the psycholinguistic features of the responses used by twelve different language models. By grouping response content according to rational or emotional prompts and exploring social influence principles employed by LLMs, we ask whether and how we can mitigate the risks of LLM-driven mass misinformation. We position this study within the broader discourse on human-centred AI, emphasizing the need for interdisciplinary approaches to mitigate cognitive and societal risks posed by persuasive AI responses.
• 摘要：小心您的要求，您可能会得到它。这句话符合大型语言模型（LLM）的训练方式，而不是为了取悦接收者而获得奖励，而不是被奖励。因此，他们越来越有效地说服我们的答案很有价值。但是，在这种说服力中，他们使用什么技巧？在这项研究中，我们检查了十二种不同语言模型使用的响应的心理语言特征是什么。通过根据理性或情感提示和探索LLMS采用的社会影响原则进行分组，我们询问以及如何减轻LLM驱动的质量错误信息的风险。我们将这项研究定位在更广泛的关于以人为中心的AI的论述中，强调需要跨学科方法来减轻由说服力的AI反应带来的认知和社会风险。

Title: Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories

• Authors: Tomas Peterka, Matyas Bohacek
• Subjects: cs.CL, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.09689
• Pdf URL: https://arxiv.org/pdf/2502.09689
• Copy Paste: [[2502.09689]] Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories(https://arxiv.org/abs/2502.09689)
• Keywords: language model
• Abstract: The most effective misinformation campaigns are multimodal, often combining text with images and videos taken out of context -- or fabricating them entirely -- to support a given narrative. Contemporary methods for detecting misinformation, whether in deepfakes or text articles, often miss the interplay between multiple modalities. Built around a large language model, the system proposed in this paper addresses these challenges. It analyzes both the article's text and the provenance metadata of included images and videos to determine whether they are relevant. We open-source the system prototype and interactive web interface.
• 摘要：最有效的错误信息广告系列是多模式的，通常将文本与图像和视频结合起来，或者完全制造出来，以支持给定的叙述。当代检测错误信息的方法，无论是在深层还是文本文章中，通常都会错过多种方式之间的相互作用。本文提出的系统围绕大型语言模型建立，解决了这些挑战。它分析了文章的文本和包含图像和视频的出处元数据，以确定它们是否相关。我们开源系统原型和交互式Web界面。

Title: Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes

• Authors: Taylan G. Topcu, Mohammed Husain, Max Ofsa, Paul Wach
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09690
• Pdf URL: https://arxiv.org/pdf/2502.09690
• Copy Paste: [[2502.09690]] Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes(https://arxiv.org/abs/2502.09690)
• Keywords: language model, llm, prompt
• Abstract: Multi-purpose Large Language Models (LLMs), a subset of generative Artificial Intelligence (AI), have recently made significant progress. While expectations for LLMs to assist systems engineering (SE) tasks are paramount; the interdisciplinary and complex nature of systems, along with the need to synthesize deep-domain knowledge and operational context, raise questions regarding the efficacy of LLMs to generate SE artifacts, particularly given that they are trained using data that is broadly available on the internet. To that end, we present results from an empirical exploration, where a human expert-generated SE artifact was taken as a benchmark, parsed, and fed into various LLMs through prompt engineering to generate segments of typical SE artifacts. This procedure was applied without any fine-tuning or calibration to document baseline LLM performance. We then adopted a two-fold mixed-methods approach to compare AI generated artifacts against the benchmark. First, we quantitatively compare the artifacts using natural language processing algorithms and find that when prompted carefully, the state-of-the-art algorithms cannot differentiate AI-generated artifacts from the human-expert benchmark. Second, we conduct a qualitative deep dive to investigate how they differ in terms of quality. We document that while the two-material appear very similar, AI generated artifacts exhibit serious failure modes that could be difficult to detect. We characterize these as: premature requirements definition, unsubstantiated numerical estimates, and propensity to overspecify. We contend that this study tells a cautionary tale about why the SE community must be more cautious adopting AI suggested feedback, at least when generated by multi-purpose LLMs.
• 摘要：多功能大语言模型（LLMS）是生成人工智能（AI）的子集，最近取得了重大进展。尽管对LLMS协助系统工程（SE）任务的期望至关重要；系统的跨学科和复杂性质，以及综合深层知识和操作环境的需求，提出了有关LLMS生成SE伪像的功效的问题，尤其是考虑到它们是使用Internet上广泛可用的数据进行培训的问题。为此，我们介绍了经验探索的结果，其中人类专家生成的SE伪像被视为基准，解析并通过迅速的工程进行了各种LLM，以产生典型的SE伪像。在没有任何微调或校准的情况下应用此过程以记录基线LLM性能。然后，我们采用了两倍的混合方法方法，将AI产生的工件与基准进行了比较。首先，我们使用自然语言处理算法进行定量比较工件，并发现当仔细提示时，最先进的算法无法将AI生成的伪像与人类专家基准区分开。其次，我们进行了定性的深度潜水，以研究它们在质量方面的不同。我们记录到，尽管两种物质看起来非常相似，但AI产生的伪像表现出严重的故障模式，可能难以检测。我们将其描述为：过早的需求定义，未经证实的数值估计以及超出指定的倾向。我们认为，这项研究讲述了一个警告性的故事，讲述了为什么SE社区必须更加谨慎地采用AI建议的反馈，至少在多功能LLMS产生时。

Title: FoNE: Precise Single-Token Number Embeddings via Fourier Features

• Authors: Tianyi Zhou, Deqing Fu, Mahdi Soltanolkotabi, Robin Jia, Vatsal Sharan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09741
• Pdf URL: https://arxiv.org/pdf/2502.09741
• Copy Paste: [[2502.09741]] FoNE: Precise Single-Token Number Embeddings via Fourier Features(https://arxiv.org/abs/2502.09741)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) typically represent numbers using multiple tokens, which requires the model to aggregate these tokens to interpret numerical values. This fragmentation makes both training and inference less efficient and adversely affects the model's performance on number-related tasks. Inspired by the observation that pre-trained LLMs internally learn Fourier-like features for number tokens, we propose Fourier Number Embedding (FoNE), a novel method that directly maps numbers into the embedding space with their Fourier features. FoNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. This compact representation accelerates both training and inference. Compared to traditional subword and digit-wise embeddings, FoNE not only reduces computational overhead but also achieves higher accuracy across various numerical tasks including addition, subtraction and multiplication. On 6-digit decimal addition, FoNE requires 64$\times$ less data to achieve 99% accuracy than subword and digit-wise embeddings while using 3$\times$ and 6$\times$ fewer tokens per number, respectively. Furthermore, FoNE is the only method that yields 100% accuracy on over 100,000 test examples for addition, subtraction, and multiplication. The codes and visualization are available at this https URL.
• 摘要：大型语言模型（LLMS）通常使用多个令牌表示数字，这要求模型汇总这些令牌以解释数值。这种分裂使培训和推断效率降低，并且会对模型在与数字相关的任务上的性能产生不利影响。受到观察的启发，即预先训练的LLM在内部学习数字令牌的傅立叶样特征，我们提出了傅立叶数字嵌入（FONE），这是一种新颖的方法，一种新型方法，该方法将数字直接映射到具有其傅立叶功能的嵌入空间中。 fone将每个数字编码为单个令牌，只有两个嵌入尺寸，每位数字有效地捕获数值而不会碎片。这种紧凑的表示可以加速训练和推理。与传统的子字和数字嵌入相比，FONE不仅减少了计算开销，而且还可以在包括加法，减法和乘法在内的各种数值任务中达到更高的精度。在6位十进制添加时，Fone需要64 $ \ tims $ $ $ $ \少于子字的精度和数字嵌入，同时使用3 $ \ times $和6 $ \ times $ $ \ times $ $ \ times $ $ $ \ times $ $每数字。此外，FONE是唯一在超过100,000个测试示例中获得100％准确性的方法，以进行添加，减法和乘法。代码和可视化可在此HTTPS URL上获得。

Title: The Widespread Adoption of Large Language Model-Assisted Writing Across Society

• Authors: Weixin Liang, Yaohui Zhang, Mihai Codreanu, Jiayu Wang, Hancheng Cao, James Zou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09747
• Pdf URL: https://arxiv.org/pdf/2502.09747
• Copy Paste: [[2502.09747]] The Widespread Adoption of Large Language Model-Assisted Writing Across Society(https://arxiv.org/abs/2502.09747)
• Keywords: language model, gpt, llm, chat
• Abstract: The recent advances in large language models (LLMs) attracted significant public and policymaker interest in its adoption patterns. In this paper, we systematically analyze LLM-assisted writing across four domains-consumer complaints, corporate communications, job postings, and international organization press releases-from January 2022 to September 2024. Our dataset includes 687,241 consumer complaints, 537,413 corporate press releases, 304.3 million job postings, and 15,919 United Nations (UN) press releases. Using a robust population-level statistical framework, we find that LLM usage surged following the release of ChatGPT in November 2022. By late 2024, roughly 18% of financial consumer complaint text appears to be LLM-assisted, with adoption patterns spread broadly across regions and slightly higher in urban areas. For corporate press releases, up to 24% of the text is attributable to LLMs. In job postings, LLM-assisted writing accounts for just below 10% in small firms, and is even more common among younger firms. UN press releases also reflect this trend, with nearly 14% of content being generated or modified by LLMs. Although adoption climbed rapidly post-ChatGPT, growth appears to have stabilized by 2024, reflecting either saturation in LLM adoption or increasing subtlety of more advanced models. Our study shows the emergence of a new reality in which firms, consumers and even international organizations substantially rely on generative AI for communications.
• 摘要：大型语言模型（LLM）的最新进展引起了对其采用模式的重大公共和决策者兴趣。在本文中，我们系统地分析了四个域名 - 消费者投诉，企业通讯，职位发布和国际组织新闻稿的LLM辅助写作，从2022年1月至2024年9月。我们的数据集包括687,241个消费者投诉，537,413 Comporatate Presseas，304.34.34.34.34.34.34.34.34.34.344.34.34.34.3。数百万个职位发布和15,919个联合国（联合国）新闻稿。使用强大的人口级统计框架，我们发现2022年11月发布Chatgpt之后的LLM用法飙升。到2024年下半年，大约18％在城市地区略高。对于公司新闻稿，最多24％的文本归因于LLM。在职位发布中，LLM辅助写作账户在小公司中的份额略低于10％，在年轻公司中更为普遍。联合国新闻稿也反映了这一趋势，近14％的内容由LLMS生成或修改。尽管采用率迅速攀升，但在2024年之前，生长似乎已经稳定，反映了LLM采用的饱和度或更高级模型的微妙之处。我们的研究表明了一个新现实的出现，在该现实中，公司，消费者甚至国际组织基本上依靠生成AI进行通信。

Title: Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages

• Authors: Micha Elsner, David Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09778
• Pdf URL: https://arxiv.org/pdf/2502.09778
• Copy Paste: [[2502.09778]] Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages(https://arxiv.org/abs/2502.09778)
• Keywords: llm, prompt
• Abstract: Partly automated creation of interlinear glossed text (IGT) has the potential to assist in linguistic documentation. We argue that LLMs can make this process more accessible to linguists because of their capacity to follow natural-language instructions. We investigate the effectiveness of a retrieval-based LLM prompting approach to glossing, applied to the seven languages from the SIGMORPHON 2023 shared task. Our system beats the BERT-based shared task baseline for every language in the morpheme-level score category, and we show that a simple 3-best oracle has higher word-level scores than the challenge winner (a tuned sequence model) in five languages. In a case study on Tsez, we ask the LLM to automatically create and follow linguistic instructions, reducing errors on a confusing grammatical feature. Our results thus demonstrate the potential contributions which LLMs can make in interactive systems for glossing, both in making suggestions to human annotators and following directions.
• 摘要：部分自动化的插入式光泽文本（IGT）具有协助语言文档的潜力。我们认为，LLM可以使语言学家更容易访问此过程，因为他们遵循自然语言说明。我们研究了基于检索的LLM提示方法对掩饰的有效性，该方法应用于Sigmorphon 2023共享任务的七种语言。我们的系统击败了基于BERT的分数分数中的每种语言的共享任务基线，我们表明，与五种语言中的挑战赢家（一种调谐的序列模型）相比，简单的3好Oracle具有更高的单词级别得分。在一项关于TSEZ的案例研究中，我们要求LLM自动创建和遵循语言说明，从而减少令人困惑的语法特征的错误。因此，我们的结果证明了LLM可以在互动系统中所做的潜在贡献，包括向人类注释和遵循方向提出建议。

Title: INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages

• Authors: Hao Yu, Jesujoba O. Alabi, Andiswa Bukula, Jian Yun Zhuang, En-Shiun Annie Lee, Tadesse Kebede Guge, Israel Abebe Azime, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson K. Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Juliet W. Murage, Dietrich Klakow, David Ifeoluwa Adelani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09814
• Pdf URL: https://arxiv.org/pdf/2502.09814
• Copy Paste: [[2502.09814]] INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages(https://arxiv.org/abs/2502.09814)
• Keywords: language model, gpt, llm, prompt
• Abstract: Slot-filling and intent detection are well-established tasks in Conversational AI. However, current large-scale benchmarks for these tasks often exclude evaluations of low-resource languages and rely on translations from English benchmarks, thereby predominantly reflecting Western-centric concepts. In this paper, we introduce Injongo -- a multicultural, open-source benchmark dataset for 16 African languages with utterances generated by native speakers across diverse domains, including banking, travel, home, and dining. Through extensive experiments, we benchmark the fine-tuning multilingual transformer models and the prompting large language models (LLMs), and show the advantage of leveraging African-cultural utterances over Western-centric utterances for improving cross-lingual transfer from the English language. Experimental results reveal that current LLMs struggle with the slot-filling task, with GPT-4o achieving an average performance of 26 F1-score. In contrast, intent detection performance is notably better, with an average accuracy of 70.6%, though it still falls behind the fine-tuning baselines. Compared to the English language, GPT-4o and fine-tuning baselines perform similarly on intent detection, achieving an accuracy of approximately 81%. Our findings suggest that the performance of LLMs is still behind for many low-resource African languages, and more work is needed to further improve their downstream performance.
• 摘要：插槽填充和意图检测是对话式AI中完善的任务。但是，当前针对这些任务的大规模基准通常排除对低资源语言的评估，并依靠英语基准的翻译，从而主要反映了以西方为中心的概念。在本文中，我们介绍了Ingongo - 一种多元文化的开源基准数据集，用于16种非洲语言，由以跨越银行，旅行，家庭和用餐在内的不同领域中的母语人士产生的话语。通过广泛的实验，我们基准了微调的多语言变压器模型和提示大型语言模型（LLMS），并显示出利用非洲文化话语优于西方以西方语言来改善英语的跨语言转移的优势。实验结果表明，当前的LLM与插槽填充任务斗争，GPT-4O的平均性能为26 F1得分。相比之下，意图检测性能的平均准确性为70.6％，尽管它仍然落后于微调基线。与英语相比，GPT-4O和微调基线在意图检测方面的表现相似，其精度约为81％。我们的发现表明，对于许多低资源的非洲语言，LLM的表现仍然落后，需要更多的工作来进一步提高其下游性能。

Title: Statistical Coherence Alignment for Large Language Model Representation Learning Through Tensor Field Convergence

• Authors: Jonathan Gale, Godfrey Aldington, Harriet Thistlewood, Thomas Tattershall, Basil Wentworth, Vincent Enoasmo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.09815
• Pdf URL: https://arxiv.org/pdf/2502.09815
• Copy Paste: [[2502.09815]] Statistical Coherence Alignment for Large Language Model Representation Learning Through Tensor Field Convergence(https://arxiv.org/abs/2502.09815)
• Keywords: language model
• Abstract: Representation learning plays a central role in structuring internal embeddings to capture the statistical properties of language, influencing the coherence and contextual consistency of generated text. Statistical Coherence Alignment is introduced as a method to enforce structured token representations through tensor field convergence, guiding embeddings to reflect statistical dependencies inherent in linguistic data. A mathematical framework is established to quantify coherence alignment, integrating a loss function that optimizes representational consistency across training iterations. Empirical evaluations demonstrate that applying coherence constraints improves perplexity, enhances classification accuracy, and refines rare word embeddings, contributing to a more stable representation space. Comparative analyses with baseline models reveal that the proposed method fosters a more interpretable internal structure, ensuring that embeddings retain contextual dependencies while mitigating representation collapse. The impact on coherence score distributions suggests that the alignment mechanism strengthens semantic integrity across diverse linguistic constructs, leading to a more balanced organization of learned embeddings. Computational assessments indicate that while the method introduces additional memory and training costs, the structured optimization process justifies the trade-offs in applications requiring heightened contextual fidelity. Experimental results validate the effectiveness of coherence alignment in optimizing token representations, providing insights into how statistical dependencies can be leveraged to improve language model training.
• 摘要：表示学习在构造内部嵌入以捕获语言的统计属性中起着核心作用，从而影响了生成的文本的连贯性和上下文一致性。引入统计相干对准作为一种方法，以通过张量场收敛来强制实现结构的令牌表示形式，从而指导嵌入以反映语言数据固有的统计依赖性。建立了一个数学框架来量化连贯对准，整合了损失函数，该损失函数优化了跨训练迭代的代表性一致性。经验评估表明，应用一致性约束可以提高困惑，提高分类精度并完善稀有单词嵌入，从而有助于更稳定的表示空间。与基线模型的比较分析表明，所提出的方法促进了更容易解释的内部结构，从而确保嵌入在缓解表示崩溃的同时保留上下文依赖性。对相干得分分布的影响表明，对准机制增强了各种语言构建体之间的语义完整性，从而导致了更加平衡的学习嵌入组织。计算评估表明，尽管该方法引入了额外的内存和培训成本，但结构化优化过程证明了需要提高上下文保真度的应用程序的权衡。实验结果证明了相干对准在优化令牌表示方面的有效性，从而提供了有关如何利用统计依赖性以改善语言模型训练的见解。

Title: Efficient Multitask Learning in Small Language Models Through Upside-Down Reinforcement Learning

• Authors: Yu-Chen Lin, Sanat Sharma, Hari Manikandan, Jayant Kumar, Tracy Holloway King, Jing Zheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09854
• Pdf URL: https://arxiv.org/pdf/2502.09854
• Copy Paste: [[2502.09854]] Efficient Multitask Learning in Small Language Models Through Upside-Down Reinforcement Learning(https://arxiv.org/abs/2502.09854)
• Keywords: language model, gpt, llm, prompt
• Abstract: In this work, we demonstrate that small language models (SLMs), specifically a 100M parameter GPT-2 model, can achieve competitive performance in multitask prompt generation tasks while requiring only a fraction of the computational resources needed by large language models (LLMs). Through a novel combination of upside-down reinforcement learning and synthetic data distillation from a powerful LLM, Llama-3, we train an SLM that achieves relevance scores within 5% of state-of-the-art models, including Llama-3, Qwen2, and Mistral, despite being up to 80 times smaller, making it highly suitable for resource-constrained and real-time applications. This study highlights the potential of SLMs as efficient multitask learners in multimodal settings, providing a promising alternative to LLMs for scalable, low-latency deployments.
• 摘要：在这项工作中，我们证明了小语言模型（SLM），特别是100m参数GPT-2模型，可以在多任务及时生成任务中实现竞争性能，而仅需要大语言模型（LLMS）所需的计算资源的一小部分。通过从强大的LLM，Llama-3中颠倒的增强学习和合成数据蒸馏的新型组合，我们训练一个SLM，该SLM在最先进的模型的5％以内，包括Llama-3，Qwen2和Mistral，尽管较小的80倍，使其非常适合资源受限和实时应用程序。这项研究突出了SLM作为多模式设置中有效的多任务学习者的潜力，为LLM提供了有希望的可扩展，低延迟部署的替代方案。

Title: A Preliminary Exploration with GPT-4o Voice Mode

• Authors: Yu-Xiang Lin, Chih-Kai Yang, Wei-Chih Chen, Chen-An Li, Chien-yu Huang, Xuanjun Chen, Hung-yi Lee
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2502.09940
• Pdf URL: https://arxiv.org/pdf/2502.09940
• Copy Paste: [[2502.09940]] A Preliminary Exploration with GPT-4o Voice Mode(https://arxiv.org/abs/2502.09940)
• Keywords: language model, gpt, hallucination
• Abstract: With the rise of multimodal large language models, GPT-4o stands out as a pioneering model, driving us to evaluate its capabilities. This report assesses GPT-4o across various tasks to analyze its audio processing and reasoning abilities. We find that GPT-4o exhibits strong knowledge in audio, speech, and music understanding, performing well in tasks like intent classification, spoken command classification, semantic and grammatical reasoning., multilingual speech recognition, and singing analysis. It also shows greater robustness against hallucinations than other large audio-language models (LALMs). However, it struggles with tasks such as audio duration prediction and instrument classification. Additionally, GPT-4o's safety mechanisms cause it to decline tasks like speaker identification, age classification, MOS prediction, and audio deepfake detection. Notably, the model exhibits a significantly different refusal rate when responding to speaker verification tasks on different datasets. This is likely due to variations in the accompanying instructions or the quality of the input audio, suggesting the sensitivity of its built-in safeguards. Finally, we acknowledge that model performance varies with evaluation protocols. This report only serves as a preliminary exploration of the current state of LALMs.
• 摘要：随着多模式大语言模型的兴起，GPT-4O作为开创性模型脱颖而出，推动我们评估其功能。该报告评估了各种任务中的GPT-4O，以分析其音频处理和推理能力。我们发现，GPT-4O在音频，语音和音乐理解方面表现出很强的知识，在意图分类，口语命令分类，语义和语法推理，多语言语音识别和歌唱分析等任务中表现良好。与其他大型音频语言模型（LALMS）相比，它还显示出对幻觉的鲁棒性。但是，它在音频持续时间预测和仪器分类等任务上挣扎。此外，GPT-4O的安全机制导致其拒绝诸如说话者识别，年龄分类，MOS预测和音频深击检测等任务。值得注意的是，该模型在响应不同数据集上的说话者验证任务时表现出明显不同的拒绝率。这可能是由于随附的说明或输入音频质量的变化所致，这表明其内置保障的敏感性。最后，我们承认模型性能随评估协议而变化。该报告仅是对LALMS现状的初步探索。

Title: KGGen: Extracting Knowledge Graphs from Plain Text with Language Models

• Authors: Belinda Mo, Kyssen Yu, Joshua Kazdan, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, Sanmi Koyejo
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09956
• Pdf URL: https://arxiv.org/pdf/2502.09956
• Copy Paste: [[2502.09956]] KGGen: Extracting Knowledge Graphs from Plain Text with Language Models(https://arxiv.org/abs/2502.09956)
• Keywords: language model
• Abstract: Recent interest in building foundation models for KGs has highlighted a fundamental challenge: knowledge-graph data is relatively scarce. The best-known KGs are primarily human-labeled, created by pattern-matching, or extracted using early NLP techniques. While human-generated KGs are in short supply, automatically extracted KGs are of questionable quality. We present a solution to this data scarcity problem in the form of a text-to-KG generator (KGGen), a package that uses language models to create high-quality graphs from plaintext. Unlike other KG extractors, KGGen clusters related entities to reduce sparsity in extracted KGs. KGGen is available as a Python library (\texttt{pip install kg-gen}), making it accessible to everyone. Along with KGGen, we release the first benchmark, Measure of of Information in Nodes and Edges (MINE), that tests an extractor's ability to produce a useful KG from plain text. We benchmark our new tool against existing extractors and demonstrate far superior performance.
• 摘要：最近对KGS建立基础模型的兴趣强调了一个基本挑战：知识图数据相对稀缺。最著名的kg主要是人体标记，通过图案匹配或使用早期NLP技术提取。尽管人类生成的公斤供应不足，但自动提取的kg质量值得怀疑。我们以文本到千克生成器（KGGEN）的形式提出了这种数据稀缺问题的解决方案，该软件包使用语言模型从专门文本创建高质量的图形。与其他KG提取器不同，KGGEN簇相关的实体可减少提取的kg中的稀疏性。 Kggen可作为python库（\ texttt {pip install kg-gen}）提供，使每个人都可以使用。与Kggen一起，我们释放了第一个基准测试，即节点和边缘（矿山）中信息的度量，以测试提取器从纯文本中产生有用的kg的能力。我们对现有提取器进行基准测试我们的新工具，并表现出卓越的性能。

Title: LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing

• Authors: Kuan Li, Liwen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Shuai Wang, Minhao Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.09977
• Pdf URL: https://arxiv.org/pdf/2502.09977
• Copy Paste: [[2502.09977]] LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing(https://arxiv.org/abs/2502.09977)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Effectively incorporating external knowledge into Large Language Models (LLMs) is crucial for enhancing their capabilities and addressing real-world needs. Retrieval-Augmented Generation (RAG) offers an effective method for achieving this by retrieving the most relevant fragments into LLMs. However, the advancements in context window size for LLMs offer an alternative approach, raising the question of whether RAG remains necessary for effectively handling external knowledge. Several existing studies provide inconclusive comparisons between RAG and long-context (LC) LLMs, largely due to limitations in the benchmark designs. In this paper, we present LaRA, a novel benchmark specifically designed to rigorously compare RAG and LC LLMs. LaRA encompasses 2,326 test cases across four practical QA task categories and three types of naturally occurring long texts. Through systematic evaluation of seven open-source and four proprietary LLMs, we find that the optimal choice between RAG and LC depends on a complex interplay of factors, including the model's parameter size, long-text capabilities, context length, task type, and the characteristics of the retrieved chunks. Our findings provide actionable guidelines for practitioners to effectively leverage both RAG and LC approaches in developing and deploying LLM applications. Our code and dataset is provided at: \href{this https URL}{\textbf{this https URL}}.
• 摘要：有效地将外部知识纳入大语言模型（LLM）对于增强其能力和满足现实世界需求至关重要。检索增强的生成（RAG）提供了一种有效的方法来实现此目的，通过将最相关的片段检索到LLM中。但是，LLM的上下文窗口大小的进步提供了一种替代方法，提出了一个问题，即抹布是否对于有效处理外部知识仍然是必要的。现有的几项研究提供了抹布和长篇小说（LC）LLMS之间的不确定的比较，这在很大程度上是由于基准设计的局限性。在本文中，我们提出了Lara，这是一种专门设计用于比较抹布和LC LLM的新型基准。 LARA包括四个实用的质量检查任务类别和三种天然出现的长文本的2,326例测试用例。通过对七个开源和四个专有LLM的系统评估，我们发现抹布和LC之间的最佳选择取决于复杂的因素相互作用，包括模型的参数大小，长篇文本功能，上下文长度，任务类型，以及检索到的块的特征。我们的发现为从业者提供了可行的指南，以有效利用抹布和LC方法来开发和部署LLM应用程序。我们的代码和数据集提供：\ href {this HTTPS url} {\ textbf {this https url}}。

Title: Large Language Diffusion Models

• Authors: Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.09992
• Pdf URL: https://arxiv.org/pdf/2502.09992
• Copy Paste: [[2502.09992]] Large Language Diffusion Models(https://arxiv.org/abs/2502.09992)
• Keywords: language model, gpt, llm
• Abstract: Autoregressive models (ARMs) are widely regarded as the cornerstone of large language models (LLMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data masking process and a reverse process, parameterized by a vanilla Transformer to predict masked tokens. By optimizing a likelihood bound, it provides a principled generative approach for probabilistic inference. Across extensive benchmarks, LLaDA demonstrates strong scalability, outperforming our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings establish diffusion models as a viable and promising alternative to ARMs, challenging the assumption that key LLM capabilities discussed above are inherently tied to ARMs.
• 摘要：自回旋模型（ARM）被广泛认为是大语言模型（LLMS）的基石。我们通过引入LLADA来挑战这一概念，这是一种在预训练和监督的微调（SFT）范式下从头开始训练的扩散模型。 LLADA通过向前数据掩蔽过程和反向过程进行建模，该过程由Vanilla Transformer进行参数以预测掩盖的令牌。通过优化可能性约束，它为概率推断提供了一种原则上的生成方法。在广泛的基准测试中，LLADA表现出强大的可扩展性，优于我们自我建造的手臂基线。值得注意的是，LLADA 8B具有强大的LLM竞争性LLM，例如Llama3 8b在封闭式学习中，在SFT之后，在诸如多转向对话之类的案例研究中表现出令人印象深刻的跟随能力。此外，LLADA解决了逆转诅咒，超过了逆转诗完成任务中的GPT-4O。我们的发现将扩散模型建立为武器的可行且有前途的替代方案，挑战了以下假设：上面讨论的关键LLM能力与武器固有地息息相关。

Title: EmbBERT-Q: Breaking Memory Barriers in Embedded NLP

• Authors: Riccardo Bravin, Massimo Pavan, Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri
• Subjects: cs.CL, cs.AR, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.10001
• Pdf URL: https://arxiv.org/pdf/2502.10001
• Copy Paste: [[2502.10001]] EmbBERT-Q: Breaking Memory Barriers in Embedded NLP(https://arxiv.org/abs/2502.10001)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing, setting new standards across a wide range of applications. However, their relevant memory and computational demands make them impractical for deployment on technologically-constrained tiny devices such as wearable devices and Internet-of-Things units. To address this limitation, we introduce EmbBERT-Q, a novel tiny language model specifically designed for tiny devices with stringent memory constraints. EmbBERT-Q achieves state-of-the-art (SotA) accuracy in Natural Language Processing tasks in this scenario, with a total memory footprint (weights and activations) of just 781 kB, representing a 25x reduction in size with respect to SotA models. By combining architectural innovations with hardware-compatible 8-bit quantization, EmbBERT-Q consistently outperforms several baseline models scaled down to a 2 MB memory budget (i.e., the maximum memory typically available in tiny devices), including heavily compressed versions of BERT and MAMBA. Extensive experimental evaluations on both a selected benchmark dataset, TinyNLP, specifically curated to evaluate Tiny Language Models in NLP tasks and real-world scenarios, and the GLUE benchmark, demonstrate EmbBERT-Q ability to deliver competitive accuracy with respect to existing approaches, achieving an unmatched balance between memory and performance. To ensure the complete and immediate reproducibility of all our results, we release all code, scripts, and model checkpoints at this https URL.
• 摘要：大型语言模型（LLM）彻底改变了自然语言处理，并在广泛的应用程序中设定了新标准。但是，它们相关的记忆和计算需求使它们在技术约束的小设备（例如可穿戴设备和物质互联网单元）上的部署中不切实际。为了解决此限制，我们介绍了Embbert-Q，这是一种专门为具有严格内存约束的小型设备而设计的新型小语言模型。在这种情况下，Embbert-Q在自然语言处理任务中实现了最先进的（SOTA）精度，总记忆足迹（重量和激活）仅为781 kb，而SOTA型号的大小降低了25倍。通过将架构创新与硬件兼容的8位量化相结合，Embbert-Q始终优于几种基线模型，以缩放到2 MB的内存预算（即，通常在微型设备中可用的最大内存），包括重压的Bert和Mamba的重压版本。对选定基准数据集（TinyNLP）进行的广泛实验评估，专门策划，以评估NLP任务和现实世界中的小型语言模型，以及胶水基准，都证明了Embbert-Q能够相对于现有方法提供竞争力准确性内存和性能之间无与伦比的平衡。为了确保我们所有结果的完整和立即可重复性，我们在此HTTPS URL上发布所有代码，脚本和模型检查点。

Title: Probabilistic Lexical Manifold Construction in Large Language Models via Hierarchical Vector Field Interpolation

• Authors: Clive Pendleton, Ewan Harrington, Giles Fairbrother, Jasper Arkwright, Nigel Fenwick, Richard Katrix
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10013
• Pdf URL: https://arxiv.org/pdf/2502.10013
• Copy Paste: [[2502.10013]] Probabilistic Lexical Manifold Construction in Large Language Models via Hierarchical Vector Field Interpolation(https://arxiv.org/abs/2502.10013)
• Keywords: language model
• Abstract: Hierarchical vector field interpolation introduces a structured probabilistic framework for lexical representation, ensuring that word embeddings transition smoothly across a continuous manifold rather than being constrained to discrete token mappings. The proposed methodology constructs a probabilistic function space where word representations adhere to topological consistency, mitigating representational discontinuities commonly observed in transformer-based embeddings. Empirical evaluations reveal that probabilistic constraints enhance lexical coherence by refining contextual relationships, leading to improvements in semantic stability across multiple linguistic distributions. The application of divergence minimization techniques ensures that interpolated embeddings maintain probabilistic consistency while preserving computational feasibility for large-scale implementations. Experimental findings demonstrate that interpolated lexical manifolds improve representation density alignment, reducing anisotropic distortions in contextual embedding distributions. Comparative analyses with standard transformer-based models highlight that structured interpolation yields more stable representations, particularly in tasks requiring fine-grained semantic differentiation. The statistical evaluation of embedding divergence confirms that probabilistic lexical manifolds reduce representational inconsistencies while maintaining coherence across varying scales of contextual abstraction. An assessment of computational efficiency reveals that while interpolation introduces minor processing overhead, the structured representation learning approach remains scalable for practical deployment.
• 摘要：分层矢量场插值引入了词汇表示的结构化概率框架，以确保单词嵌入在连续的歧管上平稳过渡，而不是被约束到离散的令牌映射。所提出的方法构建了一个概率功能空间，其中单词表示遵守拓扑一致性，从而减轻了基于变压器的嵌入中通常观察到的代表性不连续性。经验评估表明，概率约束通过完善上下文关系增强了词汇连贯性，从而改善了多种语言分布的语义稳定性。差异最小化技术的应用确保插值嵌入保持概率一致性，同时保留用于大规模实施的计算可行性。实验发现表明，插值的词汇歧管改善了表示密度比对，从而减少了上下文嵌入分布中各向异性畸变。基于标准变压器的模型的比较分析突出了结构化插值会产生更稳定的表示形式，尤其是在需要细粒语义分化的任务中。嵌入差异的统计评估证实，概率词汇歧管降低了表示不一致的情况，同时保持了各种范围的上下文抽象范围内的连贯性。对计算效率的评估表明，尽管插值引入了较小的处理开销，但结构化表示方法仍然可以扩展实用部署。

Title: ORI: O Routing Intelligence

• Authors: Ahmad Shadid, Rahul Kumar, Mohit Mayank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10051
• Pdf URL: https://arxiv.org/pdf/2502.10051
• Copy Paste: [[2502.10051]] ORI: O Routing Intelligence(https://arxiv.org/abs/2502.10051)
• Keywords: language model, llm
• Abstract: Single large language models (LLMs) often fall short when faced with the ever-growing range of tasks, making a single-model approach insufficient. We address this challenge by proposing ORI (O Routing Intelligence), a dynamic framework that leverages a set of LLMs. By intelligently routing incoming queries to the most suitable model, ORI not only improves task-specific accuracy, but also maintains efficiency. Comprehensive evaluations across diverse benchmarks demonstrate consistent accuracy gains while controlling computational overhead. By intelligently routing queries, ORI outperforms the strongest individual models by up to 2.7 points on MMLU and 1.8 points on MuSR, ties the top performance on ARC, and on BBH. These results underscore the benefits of a multi-model strategy and demonstrate how ORI's adaptive architecture can more effectively handle diverse tasks, offering a scalable, high-performance solution for a system of multiple large language models.
• 摘要：当面对不断增长的任务范围时，单个大型语言模型（LLMS）通常会缺乏，从而使单模方法不足。我们通过提出ORI（o路由智能）来应对这一挑战，这是一个利用一组LLM的动态框架。通过将传入的查询智能路由到最合适的模型，ORI不仅提高了特定于任务的准确性，而且可以保持效率。跨不同基准测试的全面评估在控制计算开销的同时表现出一致的准确性提高。通过智能路由查询，Ori在MMLU上的表现最高2.7分，而MUSR上的单个模型最高可高达2.7分，并将ARC上的最高性能和BBH连接起来。这些结果强调了多模型策略的好处，并演示了Ori的适应性体系结构如何更有效地处理各种任务，为多个大型语言模型的系统提供了可扩展的高性能解决方案。

Title: MTLM: an Innovative Language Model Training Paradigm for ASR

• Authors: Qingliang Meng, Pengju Ren, Tian Li, Changsong Dai
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2502.10058
• Pdf URL: https://arxiv.org/pdf/2502.10058
• Copy Paste: [[2502.10058]] MTLM: an Innovative Language Model Training Paradigm for ASR(https://arxiv.org/abs/2502.10058)
• Keywords: language model
• Abstract: Pre-training Transformer-based language models (LMs) on a large amount of text has proven crucial for improving automatic speech recognition (ASR) performance. Generally, traditional LMs are unidirectional and unable to access the context on the right. This paper proposes a method for training LMs that enable traditional unidirectional LMs to fully utilize left and right contexts. Compared with the unidirectional LMs, our LM facilitates ASR to transcribe hypotheses more consistently and in a more semantically unambiguous way, as it incorporates richer contextual representations. Finally, our experimental results on the LibriSpeech corpus demonstrate that our model outperforms traditional unidirectional LMs, whether n-best rescoring or shallow fusion is used as the decoding algorithm.
• 摘要：在大量文本上，基于变压器的训练前的语言模型（LMS）对于改善自动语音识别（ASR）性能至关重要。通常，传统的LM是单向的，无法访问右边的上下文。本文提出了一种培训LMS的方法，该方法使传统的单向LMS能够充分利用左右情况。与单向LMS相比，我们的LM促进了ASR以更稳定和语义上明确的方式转录假设，因为它结合了更丰富的上下文表示。最后，我们对Librispeech语料库的实验结果表明，我们的模型的表现优于传统的单向LMS，无论是n-t-t-t-t-最佳融合还是浅融合用作解码算法。

Title: Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages

• Authors: Daniil Gurgurov, Ivan Vykopal, Josef van Genabith, Simon Ostermann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10140
• Pdf URL: https://arxiv.org/pdf/2502.10140
• Copy Paste: [[2502.10140]] Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages(https://arxiv.org/abs/2502.10140)
• Keywords: language model, gpt, llm
• Abstract: Low-resource languages (LRLs) face significant challenges in natural language processing (NLP) due to limited data. While current state-of-the-art large language models (LLMs) still struggle with LRLs, smaller multilingual models (mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of their capacity to low training data sizes. This study systematically investigates parameter-efficient adapter-based methods for adapting mLMs to LRLs, evaluating three architectures: Sequential Bottleneck, Invertible Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and structured knowledge from ConceptNet, we show that small adaptation datasets (e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains in intrinsic (masked language modeling) and extrinsic tasks (topic classification, sentiment analysis, and named entity recognition). We find that Sequential Bottleneck adapters excel in language modeling, while Invertible Bottleneck adapters slightly outperform other methods on downstream tasks due to better embedding alignment and larger parameter counts. Adapter-based methods match or outperform full fine-tuning while using far fewer parameters, and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3, GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves performance, pre-training data size remains the dominant factor, especially for languages with extensive pre-training coverage.
• 摘要：由于数据有限，低资源语言（LRLS）在自然语言处理（NLP）方面面临重大挑战。尽管当前最新的大语言模型（LLMS）仍在与LRL相处挣扎，但较小的多语言模型（MLMS）（例如Mbert和XLM-R）提供了更大的希望，因为它们可以更好地适应低训练数据尺寸的能力。这项研究系统地研究了基于参数的适配器的方法，用于调整MLMS对LRL，评估三个体系结构：顺序瓶颈，可逆瓶颈和低级适应。使用来自GLOTCC的非结构化文本和概念网的结构化知识，我们表明，小型适应数据集（例如，最多1 GB的自由文本或几个MB知识图数据）在内在（掩盖语言建模）和外部任务中产生收益的增长（主题分类，情感分析和命名实体识别）。我们发现，顺序瓶颈适配器在语言建模中表现出色，而可逆的瓶颈适配器由于更好地嵌入对齐和较大的参数计数，因此在下游任务上的其他方法略高于其他方法。基于适配器的方法在使用较少的参数时匹配或胜过完整的微调，而较小的MLMS与LRL相比，比Llama-3，GPT-4，GPT-4和DeepSeek-R1的蒸馏型模型更有效。尽管适应性提高了性能，但培训前数据大小仍然是主要因素，尤其是对于具有广泛培训覆盖率的语言而言。

Title: Prediction hubs are context-informed frequent tokens in LLMs

• Authors: Beatrix M. G. Nielsen, Iuri Macocco, Marco Baroni
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.10201
• Pdf URL: https://arxiv.org/pdf/2502.10201
• Copy Paste: [[2502.10201]] Prediction hubs are context-informed frequent tokens in LLMs(https://arxiv.org/abs/2502.10201)
• Keywords: language model, llm
• Abstract: Hubness, the tendency for few points to be among the nearest neighbours of a disproportionate number of other points, commonly arises when applying standard distance measures to high-dimensional data, often negatively impacting distance-based analysis. As autoregressive large language models (LLMs) operate on high-dimensional representations, we ask whether they are also affected by hubness. We first show, theoretically, that the only representation comparison operation performed by LLMs, namely that between context and unembedding vectors to determine continuation probabilities, is not characterized by the concentration of distances phenomenon that typically causes the appeareance of nuisance hubness. We then empirically show that this comparison still leads to a high degree of hubness, but the hubs in this case do not constitute a disturbance. They are rather the result of context-modulated frequent tokens often appearing in the pool of likely candidates for next token prediction. On the other hand, when other distance computations involving LLM representations are performed, we do not have the same theoretical guarantees, and, indeed, we see nuisance hubs appear. In summary, our work highlights, on the one hand, how hubness, while omnipresent in high-dimensional spaces, is not always a negative property that needs to be mitigated, and, on the other hand, it shows that various widely-used LLMs have developed a guessing strategy that consists in constantly assigning a high probability to frequent tokens.
• 摘要：枢纽是几个点的趋势是其他点不成比例的邻居之一，通常在对高维数据应用标准距离度量时通常会出现，通常会对基于距离的分析产生负面影响。随着自回归的大型语言模型（LLMS）在高维表示上运行，我们询问它们是否也受到枢纽的影响。从理论上讲，我们首先表明，LLMS执行的唯一代表性比较操作，即，在上下文和无型向量以确定延续概率的情况下，并不是以距离现象的浓度来表征，通常会导致nuiisance hub shubness的外观。然后，我们从经验上表明，这种比较仍然导致高度的中心，但是在这种情况下，枢纽并不构成干扰。它们是上下文调制的频繁代币的结果，通常会出现在可能的候选人池中以进行隔壁预测。另一方面，当执行涉及LLM表示的其他距离计算时，我们没有相同的理论保证，实际上，我们看到滋扰枢纽会出现。总而言之，一方面，我们的工作重点是，虽然在高维空间中无所不在的枢纽并不总是需要缓解的负面特性，另一方面，它表明了各种广泛使用的LLMS已经制定了一种猜测策略，该策略包括不断分配频繁代币的可能性。

Title: Can Post-Training Quantization Benefit from an Additional QLoRA Integration?

• Authors: Xiliang Zhu, Elena Khasanova, Cheng Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10202
• Pdf URL: https://arxiv.org/pdf/2502.10202
• Copy Paste: [[2502.10202]] Can Post-Training Quantization Benefit from an Additional QLoRA Integration?(https://arxiv.org/abs/2502.10202)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have transformed natural language processing but pose significant challenges for real-world deployment. These models necessitate considerable computing resources, which can be costly and frequently unavailable. Model compression techniques such as quantization are often leveraged to alleviate resource demand, but they may have a negative impact on the generation quality. In this study, we explore the integration of 4-bit Post-training Quantization (PTQ) with QLoRA to address these issues. We demonstrate through extensive experiments that this integration outperforms standard PTQ, and in some cases even 16-bit full-parameter fine-tuning on LLMs, validated across proprietary and public datasets with different quantization algorithms. The results demonstrate the efficacy of PTQ-QLoRA integration, offering a viable solution for deploying powerful LLMs in resource-constrained environments without compromising on performance.
• 摘要：大型语言模型（LLM）改变了自然语言处理，但对现实部署构成了重大挑战。这些模型需要大量的计算资源，这可能是昂贵且经常不可用的。诸如量化之类的模型压缩技术通常被利用以减轻资源需求，但它们可能会对发电质量产生负面影响。在这项研究中，我们探讨了4位培训后量化（PTQ）与Qlora的整合以解决这些问题。我们通过广泛的实验证明，这种集成的表现优于标准PTQ，在某些情况下，在LLMS上甚至在16位全参数进行了微调，在具有不同量化算法的专有和公共数据集中验证了。结果证明了PTQ-Qlora集成的功效，为在不影响性能的情况下提供了可行的解决方案，用于在资源约束环境中部署强大的LLM。

Title: VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models

• Authors: Gokul Karthik Kumar, Iheb Chaabane, Kebin Wu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.10250
• Pdf URL: https://arxiv.org/pdf/2502.10250
• Copy Paste: [[2502.10250]] VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models(https://arxiv.org/abs/2502.10250)
• Keywords: language model, gpt, llm, chat
• Abstract: Vision-language models (VLMs) excel in various visual benchmarks but are often constrained by the lack of high-quality visual fine-tuning data. To address this challenge, we introduce VisCon-100K, a novel dataset derived from interleaved image-text web documents. Our approach transforms 45K web documents from the OBELICS dataset into 100K image conversation samples. We utilize GPT-4V to generate image-contextual captions and OpenChat 3.5 model to convert these captions into diverse free-form and multiple-choice question-answer pairs. Integrating this dataset for fine-tuning considerably enhances VLM performance across multiple benchmarks. Unlike methods that focus solely on fine-grained visual content, our approach leverages accompanying web context, yielding superior results. We also discover that a `leaky modality mix,' where conversation samples contain questions answerable from both the image and its contextual caption, outperforms non-leaky combinations of captions and Q\&A pairs. VisCon-100k dataset shows strong performance with two popular VLM approaches: text-only large language model (LLM) aligned with a vision encoder using image captions data (ShareGPT4V-7b) and multimodally pretrained LLM (IDEFICS2-8b) using interleaved image-text data. In addition to releasing the VisCon-100K dataset, we provide a contextual captioner trained on this dataset, facilitating scalable fine-tuning data generation for future research and open-source applications. Using the same pipeline, but substituting our trained contextual captioner for GPT-4V, we also release the larger VisCon-1M dataset.
• 摘要：视觉语言模型（VLM）在各种视觉基准中都表现出色，但通常受到缺乏高质量视觉微调数据的限制。为了应对这一挑战，我们介绍了Viscon-100K，这是一种来自交织的图像文本Web文档的新颖数据集。我们的方法将45K Web文档从质子数据集转换为100k图像对话样本。我们利用GPT-4V生成图像上下文字幕和OpenChat 3.5模型，将这些字幕转换为各种自由形式和多项选择的问答对。集成此数据集以进行微调大大提高了多个基准测试的VLM性能。与仅关注细粒视觉内容的方法不同，我们的方法利用了Web上下文，从而产生了卓越的结果。我们还发现一种“泄漏的方式混合”，其中对话样本包含可以从图像及其上下文标题中回答的问题，表现优于标题和Q \＆a对的非裸露组合。 VisCon-100K数据集通过两种流行的VLM方法显示出很强的性能：使用图像标题数据（ShareGPT4V-7B）与视觉编码器排列的仅文本大语模型（LLM）和使用交织织物的Interleaved Image Image-Text-Text-Text-Text（IDEFICS2-8B）对齐数据。除了释放VisCon-100K数据集外，我们还提供了在该数据集上培训的上下文标题者，从​​而促进可扩展的微调数据生成，以实现未来的研究和开源应用程序。使用同一条管道，但将我们训练的上下文标题用GPT-4V代替，我们还释放了较大的Viscon-1M数据集。

Title: Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers

• Authors: Aivin V. Solatorio, Rafael Macalaba, James Liounis
• Subjects: cs.CL, cs.AI, cs.CY, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.10263
• Pdf URL: https://arxiv.org/pdf/2502.10263
• Copy Paste: [[2502.10263]] Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers(https://arxiv.org/abs/2502.10263)
• Keywords: language model, llm, agent
• Abstract: Tracking how data is mentioned and used in research papers provides critical insights for improving data discoverability, quality, and production. However, manually identifying and classifying dataset mentions across vast academic literature is resource-intensive and not scalable. This paper presents a machine learning framework that automates dataset mention detection across research domains by leveraging large language models (LLMs), synthetic data, and a two-stage fine-tuning process. We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset. The Phi-3.5-mini instruct model is pre-fine-tuned on this dataset, followed by fine-tuning on a manually annotated subset. At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall. Evaluated on a held-out manually annotated sample, our fine-tuned model outperforms NuExtract-v1.5 and GLiNER-large-v2.1 in dataset extraction accuracy. Our results highlight how LLM-generated synthetic data can effectively address training data scarcity, improving generalization in low-resource settings. This framework offers a pathway toward scalable monitoring of dataset usage, enhancing transparency, and supporting researchers, funders, and policymakers in identifying data gaps and strengthening data accessibility for informed decision-making.
• 摘要：跟踪研究论文中提到和使用的数据为改善数据可发现性，质量和生产提供了关键见解。但是，手动识别和分类数据集在大量的学术文献中提到的是资源密集的，而不是可扩展的。本文提出了一个机器学习框架，该框架通过利用大型语言模型（LLM），合成数据和两个阶段的微调过程来自动化数据集提及跨研究领域的检测。我们从研究论文中提取零射，法官法官进行质量评估，以及用于产生弱监督的合成数据集的推理剂。 PHI-3.5-MINI指示模型在此数据集上进行了预先调整，然后在手动注释的子集上进行微调。在推断时，现代基于现代的分类器有效地过滤数据集提及，从而减少了计算开销，同时保持了高召回率。在固定的手动注释样本上进行了评估，我们的微调模型在数据集提取精度中优于nuextract-v1.5和Gliner-large-v2.1。我们的结果突出了LLM生成的合成数据如何有效地解决培训数据稀缺性，从而改善低资源设置的概括。该框架为数据集使用，提高透明度以及支持研究人员，资助者和政策制定者识别数据差距并加强数据可访问性以实现知情决策的途径提供了一种途径。

Title: Are Large Language Models the future crowd workers of Linguistics?

• Authors: Iris Ferrazzo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.10266
• Pdf URL: https://arxiv.org/pdf/2502.10266
• Copy Paste: [[2502.10266]] Are Large Language Models the future crowd workers of Linguistics?(https://arxiv.org/abs/2502.10266)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Data elicitation from human participants is one of the core data collection strategies used in empirical linguistic research. The amount of participants in such studies may vary considerably, ranging from a handful to crowdsourcing dimensions. Even if they provide resourceful extensive data, both of these settings come alongside many disadvantages, such as low control of participants' attention during task completion, precarious working conditions in crowdsourcing environments, and time-consuming experimental designs. For these reasons, this research aims to answer the question of whether Large Language Models (LLMs) may overcome those obstacles if included in empirical linguistic pipelines. Two reproduction case studies are conducted to gain clarity into this matter: Cruz (2023) and Lombard et al. (2021). The two forced elicitation tasks, originally designed for human participants, are reproduced in the proposed framework with the help of OpenAI's GPT-4o-mini model. Its performance with our zero-shot prompting baseline shows the effectiveness and high versatility of LLMs, that tend to outperform human informants in linguistic tasks. The findings of the second replication further highlight the need to explore additional prompting techniques, such as Chain-of-Thought (CoT) prompting, which, in a second follow-up experiment, demonstrates higher alignment to human performance on both critical and filler items. Given the limited scale of this study, it is worthwhile to further explore the performance of LLMs in empirical Linguistics and in other future applications in the humanities.
• 摘要：人类参与者的数据启发是经验语言研究中使用的核心数据收集策略之一。此类研究的参与者的数量可能有很大的不同，从少数到众包维度。即使他们提供了足够的广泛数据，这两个设置都与许多缺点伴随着，例如在任务完成期间对参与者的注意力的低控制，众包环境中的不稳定工作条件以及耗时的实验设计。由于这些原因，本研究旨在回答一个问题，即如果包括在经验语言管道中，大型语言模型（LLM）是否可能克服这些障碍。进行了两项繁殖案例研究，以使此事清楚地表明：Cruz（2023）和Lombard等人。 （2021）。在OpenAI的GPT-4O-Mini模型的帮助下，在拟议的框架中重现了最初为人类参与者设计的两项强迫启发任务。它在零射击促使基线的表现表明了LLM的有效性和高多功能性，这往往在语言任务中表现优于人类线人。第二次复制的发现进一步强调了需要探索其他提示技术的需要，例如思考链（COT）提示，在第二次后续实验中，这表明了对关键和填充物品的人类绩效的更高对齐。鉴于这项研究的规模有限，值得进一步探索LLM在经验语言学和人文学科中其他未来应用中的表现。

Title: Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering

• Authors: Nick Ferguson, Liane Guillou, Alan Bundy, Kwabena Nuamah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.10338
• Pdf URL: https://arxiv.org/pdf/2502.10338
• Copy Paste: [[2502.10338]] Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering(https://arxiv.org/abs/2502.10338)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel in natural language tasks but still face challenges in Question Answering (QA) tasks requiring complex, multi-step reasoning. We outline the types of reasoning required in some of these tasks, and reframe them in terms of meta-level reasoning (akin to high-level strategic reasoning or planning) and object-level reasoning (embodied in lower-level tasks such as mathematical reasoning). Franklin, a novel dataset with requirements of meta- and object-level reasoning, is introduced and used along with three other datasets to evaluate four LLMs at question answering tasks requiring multiple steps of reasoning. Results from human annotation studies suggest LLMs demonstrate meta-level reasoning with high frequency, but struggle with object-level reasoning tasks in some of the datasets used. Additionally, evidence suggests that LLMs find the object-level reasoning required for the questions in the Franklin dataset challenging, yet they do exhibit strong performance with respect to the meta-level reasoning requirements.
• 摘要：大型语言模型（LLMS）在自然语言任务中表现出色，但仍面临所涉及的挑战（QA）任务，需要复杂的多步推理。我们概述了其中一些任务中所需的推理类型，并根据元级别的推理（类似于高级战略推理或计划）和对象级别的推理（体现在诸如数学推理之类的低级任务中）。富兰克林（Franklin）是一个具有元和对象级推理要求要求的新型数据集，并与其他三个数据集一起介绍并使用了问题，以评估四个LLMS在问答任务中需要多个推理的任务。人类注释研究的结果表明，LLMS证明了高频的元级推理，但在某些数据集中与对象级推理任务斗争。此外，有证据表明，LLMS在富兰克林数据集中找到问题所需的对象级别的推理，但它们确实在元级推理要求方面表现出很强的性能。

Title: Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

• Authors: Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, Luca Soldaini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10341
• Pdf URL: https://arxiv.org/pdf/2502.10341
• Copy Paste: [[2502.10341]] Organize the Web: Constructing Domains Enhances Pre-Training Data Curation(https://arxiv.org/abs/2502.10341)
• Keywords: language model
• Abstract: Modern language models are trained on large, unstructured datasets consisting of trillions of tokens and obtained by crawling the web. The unstructured nature makes it difficult to reason about their contents and develop systematic approaches to data curation. In this paper, we unpack monolithic web corpora by developing taxonomies of their contents and organizing them into domains. We introduce WebOrganizer, a framework for organizing web pages in terms of both their topic and format. Using these two complementary notions of domains, we automatically annotate pre-training data by distilling annotations from a large language model into efficient classifiers. This allows us to study how data from different domains should be mixed to improve models on downstream tasks, and we show that we can combine insights about effective topics and formats to further boost performance. We demonstrate that our domain mixing also improves existing methods that select data based on quality. Furthermore, we study and compare how quality-based methods will implicitly change the domain mixture. Overall, our work demonstrates that constructing and mixing domains provides a valuable complement to quality-based data curation methods, opening new avenues for effective and insightful pre-training data curation.
• 摘要：现代语言模型经过大型，非结构化的数据集培训，这些数据集由数万亿个代币组成，并通过爬网。非结构化的性质使得很难推理其内容并开发系统的数据策展方法。在本文中，我们通过开发其内容的分类法并将其组织成领域来解开单片网络语料库。我们介绍了Weborganizer，这是一个以其主题和格式来组织网页的框架。使用这两个互补的域概念，我们通过将大语模型从大型语言模型提炼为有效的分类器来自动注释预训练数据。这使我们能够研究如何混合来自不同领域的数据以改善下游任务的模型，我们表明我们可以将有关有效主题和格式的见解结合在一起，以进一步提高性能。我们证明我们的域混合还改善了基于质量选择数据的现有方法。此外，我们研究并比较了基于质量的方法将如何隐式改变域混合物。总体而言，我们的工作表明，构建和混合域为基于质量的数据策展方法提供了有价值的补充，为有效且有见地的预训练数据策划打开了新的途径。

Title: Agentic Verification for Ambiguous Query Disambiguation

• Authors: Youngwon Lee, Seung-won Hwang, Ruofan Wu, Feng Yan, Danmei Xu, Moutasem Akkad, Zhewei Yao, Yuxiong He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10352
• Pdf URL: https://arxiv.org/pdf/2502.10352
• Copy Paste: [[2502.10352]] Agentic Verification for Ambiguous Query Disambiguation(https://arxiv.org/abs/2502.10352)
• Keywords: llm, retrieval-augmented generation, agent
• Abstract: In this work, we tackle the challenge of disambiguating queries in retrieval-augmented generation (RAG) to diverse yet answerable interpretations. State-of-the-arts follow a Diversify-then-Verify (DtV) pipeline, where diverse interpretations are generated by an LLM, later used as search queries to retrieve supporting passages. Such a process may introduce noise in either interpretations or retrieval, particularly in enterprise settings, where LLMs -- trained on static data -- may struggle with domain-specific disambiguations. Thus, a post-hoc verification phase is introduced to prune noises. Our distinction is to unify diversification with verification by incorporating feedback from retriever and generator early on. This joint approach improves both efficiency and robustness by reducing reliance on multiple retrieval and inference steps, which are susceptible to cascading errors. We validate the efficiency and effectiveness of our method, Verified-Diversification with Consolidation (VERDICT), on the widely adopted ASQA benchmark to achieve diverse yet verifiable interpretations. Empirical results show that VERDICT improves grounding-aware F1 score by an average of 23% over the strongest baseline across different backbone LLMs.
• 摘要：在这项工作中，我们应对在检索增强的一代（RAG）中删除查询的挑战，以换取多样化但可回答的解释。最先进的工作遵循多元化的验证（DTV）管道，其中LLM产​​生了各种解释，后来用作检索支持段落的搜索查询。这样的过程可能会在解释或检索中引入噪音，尤其是在企业设置中，在企业设置中，LLM（经过静态数据训练）可能会与特定领域的解剖作用。因此，将事后验证阶段引入修剪噪声。我们的区别是通过早期纳入猎犬和发电机的反馈来统一多元化和验证。这种联合方法通过减少对多个检索和推理步骤的依赖，从而提高效率和鲁棒性，这容易受到级联错误的影响。我们在广泛采用的ASQA基准上验证了我们方法的效率和有效性，并通过合并（判决）进行了验证，以实现多样化但可验证的解释。经验结果表明，在不同的骨干LLM中，判决在最强的基线上平均将接地感知的F1得分提高了23％。

Title: Enhancing Multilingual LLM Pretraining with Model-Based Data Selection

• Authors: Bettina Messmer, Vinko Sabolčec, Martin Jaggi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.10361
• Pdf URL: https://arxiv.org/pdf/2502.10361
• Copy Paste: [[2502.10361]] Enhancing Multilingual LLM Pretraining with Model-Based Data Selection(https://arxiv.org/abs/2502.10361)
• Keywords: language model, llm
• Abstract: Dataset curation has become a basis for strong large language model (LLM) performance. While various rule-based filtering heuristics exist for English and multilingual datasets, model-based filtering techniques have primarily focused on English. To address the disparity stemming from limited research on non-English languages, we propose a model-based filtering framework for multilingual datasets that aims to identify a diverse set of structured and knowledge-rich samples. Our approach emphasizes transparency, simplicity, and efficiency, leveraging Transformer- and FastText-based classifiers to ensure the broad accessibility of our technique and data. We conduct comprehensive ablation studies on the FineWeb-2 web crawl dataset across diverse language families, scripts, and resource availability to demonstrate the effectiveness of our method. Training a 1B-parameter Llama model for 70B and 119B tokens, our approach can match the baseline MMLU score with as little as 15% of the training tokens, while also improving across other benchmarks. These findings provide strong evidence for the generalizability of our approach to other languages. As a result, we extend our framework to 20 languages for which we release the refined pretraining datasets.
• 摘要：数据集策划已成为强大的大语言模型（LLM）性能的基础。尽管存在针对英语和多语言数据集的各种基于规则的过滤术，但基于模型的过滤技术主要集中于英语。为了解决对非英语语言研究有限的研究的差异，我们为多语言数据集提出了一个基于模型的过滤框架，旨在确定一组各种结构化和知识丰富的样本。我们的方法强调了透明度，简单性和效率，利用了基于变压器和FastText的分类器，以确保我们的技术和数据的广泛可访问性。我们对各种语言家族，脚本和资源可用性进行了有关FineWeb-2 Web爬网数据集的全面消融研究，以证明我们方法的有效性。我们的方法培训了1B参数Llama模型，我们的方法可以与基线MMLU得分相匹配，而训练令牌的15％，同时也可以改善其他基准测试。这些发现为我们对其他语言的方法的普遍性提供了有力的证据。结果，我们将我们的框架扩展到20种语言，并为其释放精制的预处理数据集。

Title: Aspect-Oriented Summarization for Psychiatric Short-Term Readmission Prediction

• Authors: WonJin Yoon, Boyu Ren, Spencer Thomas, Chanwhi Kim, Guergana Savova, Mei-Hua Hall, Timothy Miller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.10388
• Pdf URL: https://arxiv.org/pdf/2502.10388
• Copy Paste: [[2502.10388]] Aspect-Oriented Summarization for Psychiatric Short-Term Readmission Prediction(https://arxiv.org/abs/2502.10388)
• Keywords: language model, llm, prompt
• Abstract: Recent progress in large language models (LLMs) has enabled the automated processing of lengthy documents even without supervised training on a task-specific dataset. Yet, their zero-shot performance in complex tasks as opposed to straightforward information extraction tasks remains suboptimal. One feasible approach for tasks with lengthy, complex input is to first summarize the document and then apply supervised fine-tuning to the summary. However, the summarization process inevitably results in some loss of information. In this study we present a method for processing the summaries of long documents aimed to capture different important aspects of the original document. We hypothesize that LLM summaries generated with different aspect-oriented prompts contain different \textit{information signals}, and we propose methods to measure these differences. We introduce approaches to effectively integrate signals from these different summaries for supervised training of transformer models. We validate our hypotheses on a high-impact task -- 30-day readmission prediction from a psychiatric discharge -- using real-world data from four hospitals, and show that our proposed method increases the prediction performance for the complex task of predicting patient outcome.
• 摘要：大型语言模型（LLMS）的最新进展已使得即使没有在特定于任务的数据集中进行监督培训的冗长文档的自动处理。然而，与直接信息提取任务相反，它们在复杂任务中的零击性能仍然是最佳的。对于冗长，复杂的输入的任务的一种可行方法是首先汇总文档，然后将监督的微调应用于摘要。但是，汇总过程不可避免地会导致一些信息丢失。在这项研究中，我们提出了一种处理旨在捕获原始文档不同重要方面的长文档摘要的方法。我们假设使用不同方面的提示生成的LLM摘要包含不同的\ textIt {信息信号}，我们提出了测量这些差异的方法。我们介绍了有效整合来自这些不同摘要的信号的方法，以监督变压器模型的培训。我们使用来自四家医院的现实世界数据来验证有关高影响任务的假设 - 从精神病院出院进行了30天的再入院预测，并表明我们提出的方法提高了预测患者结果的复杂任务的预测性能。

Title: MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

• Authors: Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang, Rong Jin, Tieniu Tan
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.10391
• Pdf URL: https://arxiv.org/pdf/2502.10391
• Copy Paste: [[2502.10391]] MM-RLHF: The Next Step Forward in Multimodal LLM Alignment(https://arxiv.org/abs/2502.10391)
• Keywords: language model, llm, hallucination
• Abstract: Despite notable advancements in Multimodal Large Language Models (MLLMs), most state-of-the-art models have not undergone thorough alignment with human preferences. This gap exists because current alignment research has primarily achieved progress in specific areas (e.g., hallucination reduction), while the broader question of whether aligning models with human preferences can systematically enhance MLLM capability remains largely unexplored. To this end, we introduce MM-RLHF, a dataset containing $\mathbf{120k}$ fine-grained, human-annotated preference comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation granularity, and quality. Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce a Critique-Based Reward Model, which generates critiques of model outputs before assigning scores, offering enhanced interpretability and more informative feedback compared to traditional scalar reward mechanisms. Additionally, we propose Dynamic Reward Scaling, a method that adjusts the loss weight of each sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs. Our approach is rigorously evaluated across $\mathbf{10}$ distinct dimensions and $\mathbf{27}$ benchmarks, with results demonstrating significant and consistent improvements in model performance. Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm leads to a $\mathbf{19.5}$% increase in conversational abilities and a $\mathbf{60}$% improvement in safety. We have open-sourced the preference dataset, reward model, training and evaluation code, as well as reward modeling and safety benchmarks. For more details, please visit our project page: this https URL.
• 摘要：尽管多模式大语言模型（MLLM）的显着进步，但大多数最先进的模型尚未与人类偏好进行彻底的一致性。之所以存在这一差距，是因为当前的一致性研究主要在特定领域（例如减少幻觉）取得了进步，而对与人类偏好的比对模型是否可以系统地增强MLLM能力的更广泛的问题仍然在很大程度上没有探索。为此，我们介绍了MM-RLHF，该数据集包含$ \ MATHBF {120K} $细粒度，人类通知比较比较对。该数据集代表了对现有资源的实质性进步，提供了较高的规模，多样性，注释粒度和质量。利用此数据集，我们提出了几项关键创新，以提高奖励模型的质量和对齐算法的效率。值得注意的是，我们引入了基于批评的奖励模型，该模型在分配分数之前会产生模型输出的评论，与传统的标量奖励机制相比，提供了增强的可解释性和更有信息的反馈。此外，我们提出了动态奖励缩放，该方法可根据奖励信号调节每个样本的减肥重量，从而优化使用高质量比较对。我们的方法在$ \ mathbf {10} $不同的尺寸和$ \ mathbf {27} $基准中进行了严格评估，结果表明模型性能的显着和一致的改进。具体来说，带有MM-RLHF和我们的对齐算法的微调LLAVA-OV-7B导致$ \ MathBf {19.5} $％的会话能力和$ \ MATHBF {60} $％的安全性提高。我们已经开源了偏好数据集，奖励模型，培训和评估法，以及奖励建模和安全基准。有关更多详细信息，请访问我们的项目页面：此HTTPS URL。