2024-12-23

Title: OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Models

• Authors: Kartik Sharma, Peeyush Kumar, Yunqing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15235
• Pdf URL: https://arxiv.org/pdf/2412.15235
• Copy Paste: [[2412.15235]] OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Models(https://arxiv.org/abs/2412.15235)
• Keywords: language model, llm, retrieval augmented generation, retrieval-augmented generation
• Abstract: This paper presents OG-RAG, an Ontology-Grounded Retrieval Augmented Generation method designed to enhance LLM-generated responses by anchoring retrieval processes in domain-specific ontologies. While LLMs are widely used for tasks like question answering and search, they struggle to adapt to specialized knowledge, such as industrial workflows or knowledge work, without expensive fine-tuning or sub-optimal retrieval methods. Existing retrieval-augmented models, such as RAG, offer improvements but fail to account for structured domain knowledge, leading to suboptimal context generation. Ontologies, which conceptually organize domain knowledge by defining entities and their interrelationships, offer a structured representation to address this gap. OG-RAG constructs a hypergraph representation of domain documents, where each hyperedge encapsulates clusters of factual knowledge grounded using domain-specific ontology. An optimization algorithm then retrieves the minimal set of hyperedges that constructs a precise, conceptually grounded context for the LLM. This method enables efficient retrieval while preserving the complex relationships between entities. OG-RAG applies to domains where fact-based reasoning is essential, particularly in tasks that require workflows or decision-making steps to follow predefined rules and procedures. These include industrial workflows in healthcare, legal, and agricultural sectors, as well as knowledge-driven tasks such as news journalism, investigative research, consulting and more. Our evaluations demonstrate that OG-RAG increases the recall of accurate facts by 55% and improves response correctness by 40% across four different LLMs. Additionally, OG-RAG enables 30% faster attribution of responses to context and boosts fact-based reasoning accuracy by 27% compared to baseline methods.
• 摘要：本文介绍了一种基于本体的检索增强生成方法 OG-RAG，旨在通过将检索过程锚定在特定领域的本体中来增强 LLM 生成的响应。虽然 LLM 广泛用于问答和搜索等任务，但它们很难适应专业知识，例如工业工作流程或知识工作，而无需昂贵的微调或次优的检索方法。现有的检索增强模型（例如 RAG）提供了改进，但未能考虑结构化领域知识，导致上下文生成不理想。本体通过定义实体及其相互关系来概念化组织领域知识，它提供了一种结构化表示来解决这一差距。OG-RAG 构建了领域文档的超图表示，其中每个超边都封装了使用特定领域本体建立的事实知识集群。然后，优化算法检索最小的超边集，为 LLM 构建精确的、概念化的上下文。该方法可以实现高效检索，同时保留实体之间的复杂关系。OG-RAG 适用于需要基于事实的推理的领域，特别是需要工作流程或决策步骤遵循预定义规则和程序的任务。这些包括医疗保健、法律和农业领域的工业工作流程，以及新闻报道、调查研究、咨询等知识驱动的任务。我们的评估表明，OG-RAG 在四个不同的 LLM 中将准确事实的召回率提高了 55%，并将响应正确率提高了 40%。此外，与基线方法相比，OG-RAG 可以将响应归因于上下文的速度提高 30%，并将基于事实的推理准确性提高 27%。

Title: CareBot: A Pioneering Full-Process Open-Source Medical Language Model

• Authors: Lulu Zhao, Weihao Zeng, Xiaofeng Shi, Hua Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15236
• Pdf URL: https://arxiv.org/pdf/2412.15236
• Copy Paste: [[2412.15236]] CareBot: A Pioneering Full-Process Open-Source Medical Language Model(https://arxiv.org/abs/2412.15236)
• Keywords: language model, llm
• Abstract: Recently, both closed-source LLMs and open-source communities have made significant strides, outperforming humans in various general domains. However, their performance in specific professional domains such as medicine, especially within the open-source community, remains suboptimal due to the complexity of medical knowledge. In this paper, we propose CareBot, a bilingual medical LLM, which leverages a comprehensive approach integrating continuous pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning with human feedback (RLHF). Our novel two-stage CPT method, comprising Stable CPT and Boost CPT, effectively bridges the gap between general and domain-specific data, facilitating a smooth transition from pre-training to fine-tuning and enhancing domain knowledge progressively. We also introduce DataRater, a model designed to assess data quality during CPT, ensuring that the training data is both accurate and relevant. For SFT, we develope a large and diverse bilingual dataset, along with ConFilter, a metric to enhance multi-turn dialogue quality, which is crucial to improving the model's ability to handle more complex dialogues. The combination of high-quality data sources and innovative techniques significantly improves CareBot's performance across a range of medical applications. Our rigorous evaluations on Chinese and English benchmarks confirm CareBot's effectiveness in medical consultation and education. These advancements not only address current limitations in medical LLMs but also set a new standard for developing effective and reliable open-source models in the medical domain. We will open-source the datasets and models later, contributing valuable resources to the research community.
• 摘要：最近，闭源法学硕士和开源社区都取得了长足进步，在各个通用领域都超越了人类。然而，由于医学知识的复杂性，它们在医学等特定专业领域的表现仍然不尽如人意，尤其是在开源社区中。在本文中，我们提出了双语医学法学硕士 CareBot，它利用了一种综合方法，将持续预训练 (CPT)、监督微调 (SFT) 和强化学习与人工反馈 (RLHF) 相结合。我们新颖的两阶段 CPT 方法包括稳定 CPT 和增强 CPT，有效地弥合了通用数据和领域特定数据之间的差距，促进了从预训练到微调的平稳过渡，并逐步增强领域知识。我们还引入了 DataRater，这是一种旨在评估 CPT 期间数据质量的模型，可确保训练数据既准确又相关。对于 SFT，我们开发了一个庞大而多样化的双语数据集，以及 ConFilter（一种提高多轮对话质量的指标），这对于提高模型处理更复杂对话的能力至关重要。高质量数据源和创新技术的结合显著提高了 CareBot 在一系列医疗应用中的性能。我们对中文和英文基准的严格评估证实了 CareBot 在医疗咨询和教育方面的有效性。这些进步不仅解决了医学法学硕士目前存在的局限性，还为在医学领域开发有效可靠的开源模型树立了新标准。我们稍后将开源数据集和模型，为研究界贡献宝贵的资源。

Title: Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks

• Authors: Gregory Kang Ruey Lau, Wenyang Hu, Diwen Liu, Jizhuo Chen, See-Kiong Ng, Bryan Kian Hsiang Low
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.15238
• Pdf URL: https://arxiv.org/pdf/2412.15238
• Copy Paste: [[2412.15238]] Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks(https://arxiv.org/abs/2412.15238)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models still encounter substantial challenges in reasoning tasks, especially for smaller models, which many users may be restricted to due to resource constraints (e.g. GPU memory restrictions). Inference-time methods to boost LLM performance, such as prompting methods to invoke certain reasoning pathways in responses, have been shown effective in past works, though they largely rely on sequential queries. The ensemble method, which consists of multiple constituent models running in parallel, is a promising approach to achieving better inference-time performance, especially given recent developments that enabled significant speed-ups in LLM batch inference. In this work, we propose a novel, training-free LLM ensemble framework where a single LLM model is fed an optimized, diverse set of prompts in parallel, effectively producing an ensemble at inference time to achieve performance improvement in reasoning tasks. We empirically demonstrate that our method leads to significant gains on math reasoning tasks, e.g., on MATH, where our ensemble consisting of a few small models (e.g., three Qwen2-MATH-1.5B-it models) can outperform a larger model (e.g., Qwen2-MATH-7B-it).
• 摘要：大型语言模型在推理任务中仍然面临巨大挑战，尤其是较小的模型，许多用户可能由于资源限制（例如 GPU 内存限制）而受到限制。在过去的工作中，提高 LLM 性能的推理时间方法（例如在响应中调用某些推理路径的提示方法）已被证明是有效的，尽管它们在很大程度上依赖于顺序查询。集成方法由并行运行的多个组成模型组成，是一种实现更好推理时间性能的有前途的方法，特别是考虑到最近的发展使 LLM 批量推理的速度显著提高。在这项工作中，我们提出了一个新颖的、无需训练的 LLM 集成框架，其中单个 LLM 模型并行输入一组优化的、多样化的提示，有效地在推理时产生一个集成，以实现推理任务的性能改进。我们通过实证研究证明，我们的方法在数学推理任务上取得了显著的进步，例如在 MATH 上，由几个小模型（例如三个 Qwen2-MATH-1.5B-it 模型）组成的集成模型可以胜过一个更大的模型（例如 Qwen2-MATH-7B-it）。

Title: Modeling Story Expectations to Understand Engagement: A Generative Framework Using LLMs

• Authors: Hortense Fong, George Gui
• Subjects: cs.CL, cs.AI, econ.GN, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.15239
• Pdf URL: https://arxiv.org/pdf/2412.15239
• Copy Paste: [[2412.15239]] Modeling Story Expectations to Understand Engagement: A Generative Framework Using LLMs(https://arxiv.org/abs/2412.15239)
• Keywords: language model, llm
• Abstract: Understanding when and why consumers engage with stories is crucial for content creators and platforms. While existing theories suggest that audience beliefs of what is going to happen should play an important role in engagement decisions, empirical work has mostly focused on developing techniques to directly extract features from actual content, rather than capturing forward-looking beliefs, due to the lack of a principled way to model such beliefs in unstructured narrative data. To complement existing feature extraction techniques, this paper introduces a novel framework that leverages large language models to model audience forward-looking beliefs about how stories might unfold. Our method generates multiple potential continuations for each story and extracts features related to expectations, uncertainty, and surprise using established content analysis techniques. Applying our method to over 30,000 book chapters from Wattpad, we demonstrate that our framework complements existing feature engineering techniques by amplifying their marginal explanatory power on average by 31%. The results reveal that different types of engagement-continuing to read, commenting, and voting-are driven by distinct combinations of current and anticipated content features. Our framework provides a novel way to study and explore how audience forward-looking beliefs shape their engagement with narrative media, with implications for marketing strategy in content-focused industries.
• 摘要：对于内容创作者和平台来说，了解消费者何时以及为何参与故事至关重要。虽然现有理论表明，受众对即将发生的事情的信念应该在参与决策中发挥重要作用，但实证研究主要集中在开发直接从实际内容中提取特征的技术，而不是捕捉前瞻性信念，因为缺乏在非结构化叙事数据中建模此类信念的原则性方法。为了补充现有的特征提取技术，本文介绍了一个新颖的框架，该框架利用大型语言模型来模拟受众对故事可能如何展开的前瞻性信念。我们的方法为每个故事生成多个潜在的延续，并使用成熟的内容分析技术提取与期望、不确定性和惊喜相关的特征。将我们的方法应用于 Wattpad 的 30,000 多个书籍章节，我们证明我们的框架补充了现有的特征工程技术，平均将其边际解释力放大了 31%。结果表明，不同类型的参与（继续阅读、评论和投票）是由当前和预期内容特征的不同组合驱动的。我们的框架提供了一种新颖的方法来研究和探索观众的前瞻性信念如何影响他们对叙事媒体的参与，这对以内容为中心的行业的营销策略具有启示。

Title: ChainStream: An LLM-based Framework for Unified Synthetic Sensing

• Authors: Jiacheng Liu, Yuanchun Li, Liangyan Li, Yi Sun, Hao Wen, Xiangyu Li, Yao Guo, Yunxin Liu
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.15240
• Pdf URL: https://arxiv.org/pdf/2412.15240
• Copy Paste: [[2412.15240]] ChainStream: An LLM-based Framework for Unified Synthetic Sensing(https://arxiv.org/abs/2412.15240)
• Keywords: language model, llm
• Abstract: Many applications demand context sensing to offer personalized and timely services. Yet, developing sensing programs can be challenging for developers and using them is privacy-concerning for end-users. In this paper, we propose to use natural language as the unified interface to process personal data and sense user context, which can effectively ease app development and make the data pipeline more transparent. Our work is inspired by large language models (LLMs) and other generative models, while directly applying them does not solve the problem - letting the model directly process the data cannot handle complex sensing requests and letting the model write the data processing program suffers error-prone code generation. We address the problem with 1) a unified data processing framework that makes context-sensing programs simpler and 2) a feedback-guided query optimizer that makes data query more informative. To evaluate the performance of natural language-based context sensing, we create a benchmark that contains 133 context sensing tasks. Extensive evaluation has shown that our approach is able to automatically solve the context-sensing tasks efficiently and precisely. The code is opensourced at this https URL.
• 摘要：许多应用程序需要上下文感知来提供个性化和及时的服务。然而，开发感知程序对开发人员来说可能具有挑战性，而使用它们会危及最终用户的隐私。在本文中，我们建议使用自然语言作为处理个人数据和感知用户上下文的统一接口，这可以有效地简化应用程序开发并使数据管道更加透明。我们的工作受到大型语言模型 (LLM) 和其他生成模型的启发，而直接应用它们并不能解决问题——让模型直接处理数据无法处理复杂的感知请求，而让模型编写数据处理程序会导致容易出错的代码生成。我们通过 1) 统一的数据处理框架来解决这个问题，该框架使上下文感知程序更简单；2) 反馈引导的查询优化器使数据查询更具信息性。为了评估基于自然语言的上下文感知的性能，我们创建了一个包含 133 个上下文感知任务的基准。广泛的评估表明，我们的方法能够自动高效、准确地解决上下文感知任务。代码在此 https URL 上开源。

Title: Script-Based Dialog Policy Planning for LLM-Powered Conversational Agents: A Basic Architecture for an "AI Therapist"

• Authors: Robert Wasenmüller, Kevin Hilbert, Christoph Benzmüller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15242
• Pdf URL: https://arxiv.org/pdf/2412.15242
• Copy Paste: [[2412.15242]] Script-Based Dialog Policy Planning for LLM-Powered Conversational Agents: A Basic Architecture for an "AI Therapist"(https://arxiv.org/abs/2412.15242)
• Keywords: language model, llm, prompt, agent
• Abstract: Large Language Model (LLM)-Powered Conversational Agents have the potential to provide users with scaled behavioral healthcare support, and potentially even deliver full-scale "AI therapy'" in the future. While such agents can already conduct fluent and proactive emotional support conversations, they inherently lack the ability to (a) consistently and reliably act by predefined rules to align their conversation with an overarching therapeutic concept and (b) make their decision paths inspectable for risk management and clinical evaluation -- both essential requirements for an "AI Therapist". In this work, we introduce a novel paradigm for dialog policy planning in conversational agents enabling them to (a) act according to an expert-written "script" that outlines the therapeutic approach and (b) explicitly transition through a finite set of states over the course of the conversation. The script acts as a deterministic component, constraining the LLM's behavior in desirable ways and establishing a basic architecture for an AI Therapist. We implement two variants of Script-Based Dialog Policy Planning using different prompting techniques and synthesize a total of 100 conversations with LLM-simulated patients. The results demonstrate the feasibility of this new technology and provide insights into the efficiency and effectiveness of different implementation variants.
• 摘要：大型语言模型 (LLM) 驱动的对话代理有可能为用户提供规模化的行为医疗支持，甚至可能在未来提供全面的“AI 治疗”。虽然此类代理已经可以进行流畅且主动的情感支持对话，但它们本质上缺乏以下能力：(a) 始终如一地、可靠地按照预定义的规则采取行动，以使他们的对话与总体治疗概念保持一致，以及 (b) 使他们的决策路径可检查以进行风险管理和临床评估——这两者都是“AI 治疗师”的基本要求。在这项工作中，我们引入了一种对话代理对话政策规划的新范式，使它们能够 (a) 根据专家编写的概述治疗方法的“脚本”采取行动，以及 (b) 在对话过程中明确地过渡一组有限的状态。该脚本充当确定性组件，以理想的方式约束 LLM 的行为并为 AI 治疗师建立基本架构。我们使用不同的提示技术实施了两种基于脚本的对话策略规划变体，并合成了与 LLM 模拟患者的总共 100 次对话。结果证明了这项新技术的可行性，并深入了解了不同实施变体的效率和有效性。

Title: MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

• Authors: Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15244
• Pdf URL: https://arxiv.org/pdf/2412.15244
• Copy Paste: [[2412.15244]] MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples(https://arxiv.org/abs/2412.15244)
• Keywords: language model, llm
• Abstract: Aligning Large Language Models (LLMs) with human feedback is crucial for their development. Existing preference optimization methods such as DPO and KTO, while improved based on Reinforcement Learning from Human Feedback (RLHF), are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. Meanwhile, current preference optimization research mainly targets single-question scenarios with two replies, neglecting optimization with multiple replies, which leads to a waste of data in the application. This study introduces the MPPO algorithm, which leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. Through a comparison of Point-wise, Pair-wise, and List-wise implementations, we found that the Pair-wise approach achieves the best performance, significantly enhancing the quality of model responses. Experimental results demonstrate MPPO's outstanding performance across various benchmarks. On MT-Bench, MPPO outperforms DPO, ORPO, and SimPO. Notably, on Arena-Hard, MPPO surpasses DPO and ORPO by substantial margins. These achievements underscore the remarkable advantages of MPPO in preference optimization tasks.
• 摘要：将大型语言模型 (LLM) 与人类反馈对齐是其发展的关键。现有的偏好优化方法如 DPO 和 KTO，虽然基于人类反馈强化学习 (RLHF) 进行了改进，但本质上还是从 PPO 衍生而来，需要参考模型增加 GPU 内存资源，并且严重依赖丰富的偏好数据。同时，目前的偏好优化研究主要针对具有两个回复的单问场景，忽略了具有多个回复的优化，这导致应用中的数据浪费。本研究引入了 MPPO 算法，该算法利用模型响应的平均似然来拟合奖励函数并最大化偏好数据的利用率。通过对 Point-wise、Pair-wise 和 List-wise 实现的比较，我们发现 Pair-wise 方法实现了最佳性能，显著提高了模型响应的质量。实验结果表明 MPPO 在各种基准测试中均表现出色。在 MT-Bench 上，MPPO 优于 DPO、ORPO 和 SimPO。值得注意的是，在 Arena-Hard 上，MPPO 以大幅优势超越了 DPO 和 ORPO。这些成绩凸显了 MPPO 在偏好优化任务中的显著优势。

Title: Accelerating Retrieval-Augmented Generation

• Authors: Derrick Quinn, Mohammad Nouri, Neel Patel, John Salihu, Alireza Salemi, Sukhan Lee, Hamed Zamani, Mohammad Alian
• Subjects: cs.CL, cs.AI, cs.AR, cs.DC, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15246
• Pdf URL: https://arxiv.org/pdf/2412.15246
• Copy Paste: [[2412.15246]] Accelerating Retrieval-Augmented Generation(https://arxiv.org/abs/2412.15246)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: An evolving solution to address hallucination and enhance accuracy in large language models (LLMs) is Retrieval-Augmented Generation (RAG), which involves augmenting LLMs with information retrieved from an external knowledge source, such as the web. This paper profiles several RAG execution pipelines and demystifies the complex interplay between their retrieval and generation phases. We demonstrate that while exact retrieval schemes are expensive, they can reduce inference time compared to approximate retrieval variants because an exact retrieval model can send a smaller but more accurate list of documents to the generative model while maintaining the same end-to-end accuracy. This observation motivates the acceleration of the exact nearest neighbor search for RAG. In this work, we design Intelligent Knowledge Store (IKS), a type-2 CXL device that implements a scale-out near-memory acceleration architecture with a novel cache-coherent interface between the host CPU and near-memory accelerators. IKS offers 13.4-27.9x faster exact nearest neighbor search over a 512GB vector database compared with executing the search on Intel Sapphire Rapids CPUs. This higher search performance translates to 1.7-26.3x lower end-to-end inference time for representative RAG applications. IKS is inherently a memory expander; its internal DRAM can be disaggregated and used for other applications running on the server to prevent DRAM, which is the most expensive component in today's servers, from being stranded.
• 摘要：一种解决幻觉并提高大型语言模型 (LLM) 准确性的不断发展的解决方案是检索增强生成 (RAG)，它涉及使用从外部知识源（例如网络）检索的信息来增强 LLM。本文介绍了几种 RAG 执行管道，并揭开了它们检索和生成阶段之间复杂的相互作用。我们证明，虽然精确检索方案成本高昂，但与近似检索变体相比，它们可以减少推理时间，因为精确检索模型可以向生成模型发送更小但更准确的文档列表，同时保持相同的端到端准确性。这一观察结果促使加速 RAG 的精确最近邻搜索。在这项工作中，我们设计了智能知识存储 (IKS)，这是一种 2 型 CXL 设备，它实现了横向扩展近内存加速架构，在主机 CPU 和近内存加速器之间具有新颖的缓存一致性接口。与在 Intel Sapphire Rapids CPU 上执行搜索相比，IKS 在 512GB 矢量数据库上提供 13.4-27.9 倍的精确最近邻搜索速度。对于代表性 RAG 应用程序，这种更高的搜索性能意味着端到端推理时间缩短了 1.7-26.3 倍。IKS 本质上是一个内存扩展器；其内部 DRAM 可以分解并用于服务器上运行的其他应用程序，以防止 DRAM（当今服务器中最昂贵的组件）被搁置。

Title: Streamlining Systematic Reviews: A Novel Application of Large Language Models

• Authors: Fouad Trad, Ryan Yammine, Jana Charafeddine, Marlene Chakhtoura, Maya Rahme, Ghada El-Hajj Fuleihan, Ali Chehab
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15247
• Pdf URL: https://arxiv.org/pdf/2412.15247
• Copy Paste: [[2412.15247]] Streamlining Systematic Reviews: A Novel Application of Large Language Models(https://arxiv.org/abs/2412.15247)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: Systematic reviews (SRs) are essential for evidence-based guidelines but are often limited by the time-consuming nature of literature screening. We propose and evaluate an in-house system based on Large Language Models (LLMs) for automating both title/abstract and full-text screening, addressing a critical gap in the literature. Using a completed SR on Vitamin D and falls (14,439 articles), the LLM-based system employed prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. The system achieved an article exclusion rate (AER) of 99.5%, specificity of 99.6%, a false negative rate (FNR) of 0%, and a negative predictive value (NPV) of 100%. After screening, only 78 articles required manual review, including all 20 identified by traditional methods, reducing manual screening time by 95.5%. For comparison, Rayyan, a commercial tool for title/abstract screening, achieved an AER of 72.1% and FNR of 5% when including articles Rayyan considered as undecided or likely to include. Lowering Rayyan's inclusion thresholds improved FNR to 0% but increased screening time. By addressing both screening phases, the LLM-based system significantly outperformed Rayyan and traditional methods, reducing total screening time to 25.5 hours while maintaining high accuracy. These findings highlight the transformative potential of LLMs in SR workflows by offering a scalable, efficient, and accurate solution, particularly for the full-text screening phase, which has lacked automation tools.
• 摘要：系统评价 (SR) 对于循证指南至关重要，但往往受到文献筛选耗时性质的限制。我们提出并评估了一种基于大型语言模型 (LLM) 的内部系统，用于自动进行标题/摘要和全文筛选，以解决文献中的一个关键空白。使用关于维生素 D 和跌倒的完整 SR（14,439 篇文章），基于 LLM 的系统采用快速工程进行标题/摘要筛选，并使用检索增强生成 (RAG) 进行全文筛选。该系统实现了 99.5% 的文章排除率 (AER)、99.6% 的特异性、0% 的假阴性率 (FNR) 和 100% 的阴性预测值 (NPV)。筛选后，只有 78 篇文章需要人工审核，其中包括通过传统方法确定的全部 20 篇，将人工筛选时间缩短了 95.5%。相比之下，商业标题/摘要筛选工具 Rayyan 在纳入 Rayyan 认为尚未决定或可能纳入的文章时，实现了 72.1% 的 AER 和 5% 的 FNR。降低 Rayyan 的纳入门槛可将 FNR 提高到 0%，但会增加筛选时间。通过解决两个筛选阶段，基于 LLM 的系统明显优于 Rayyan 和传统方法，将总筛选时间缩短至 25.5 小时，同时保持了高准确度。这些发现凸显了 LLM 在 SR 工作流程中的变革潜力，它提供了一种可扩展、高效且准确的解决方案，尤其是对于缺乏自动化工具的全文筛选阶段。

Title: LLMs for Literature Review: Are we there yet?

• Authors: Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, Christopher Pal
• Subjects: cs.CL, cs.AI, cs.DL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15249
• Pdf URL: https://arxiv.org/pdf/2412.15249
• Copy Paste: [[2412.15249]] LLMs for Literature Review: Are we there yet?(https://arxiv.org/abs/2412.15249)
• Keywords: language model, llm, prompt
• Abstract: Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially due to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: 1. Retrieving related works given a query abstract, and 2. Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods, while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Further, we demonstrate that our planning-based approach achieves higher-quality reviews by minimizing hallucinated references in the generated review by 18-26% compared to existing simpler LLM-based generation methods.
• 摘要：文献综述是科学研究的重要组成部分，但撰写文献综述仍然耗时且具有挑战性，尤其是由于最近大量研究论文的涌入。本文探讨了最近的大型语言模型 (LLM) 在协助基于摘要撰写文献综述方面的零样本能力。我们将任务分解为两个部分：1. 根据查询摘要检索相关作品，2. 根据检索到的结果撰写文献综述。我们分析了 LLM 对这两个部分的有效性。对于检索，我们引入了一种新颖的两步搜索策略，首先使用 LLM 从论文摘要中提取有意义的关键字，然后通过查询外部知识库检索潜在相关的论文。此外，我们研究了一种基于提示的带归因的重新排名机制，并表明与简单的搜索方法相比，重新排名使规范化召回率翻了一番，同时提供了对 LLM 决策过程的洞察。在生成阶段，我们提出了一种两步方法，首先概述评论计划，然后执行计划中的步骤以生成实际评论。为了评估不同的基于 LLM 的文献评论方法，我们使用专为与新发布的 LLM 滚动使用而设计的协议从 arXiv 论文中创建测试集，以避免零样本评估中的测试集污染。我们发布此评估协议以促进这方面的进一步研究和开发。我们的实证结果表明，当任务分解为检索和规划的较小部分时，LLM 在撰写文献评论方面显示出巨大的潜力。此外，我们证明，与现有的更简单的基于 LLM 的生成方法相比，我们的基于规划的方法通过将生成的评论中的幻觉参考最小化 18-26% 来实现更高质量的评论。

Title: An Enhanced Text Compression Approach Using Transformer-based Language Models

• Authors: Chowdhury Mofizur Rahman, Mahbub E Sobhani, Anika Tasnim Rodela, Swakkhar Shatabda
• Subjects: cs.CL, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15250
• Pdf URL: https://arxiv.org/pdf/2412.15250
• Copy Paste: [[2412.15250]] An Enhanced Text Compression Approach Using Transformer-based Language Models(https://arxiv.org/abs/2412.15250)
• Keywords: language model
• Abstract: Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy. The integration of lossless compression techniques with transformer-based text decompression has received negligible attention, despite the increasing volume of English text data in communication. The primary barrier in advancing text compression and restoration involves optimizing transformer-based approaches with efficient pre-processing and integrating lossless compression algorithms, that remained unresolved in the prior attempts. Here, we propose a transformer-based method named RejuvenateForme for text decompression, addressing prior issues by harnessing a new pre-processing technique and a lossless compression method. Our meticulous pre-processing technique incorporating the Lempel-Ziv-Welch algorithm achieves compression ratios of 12.57, 13.38, and 11.42 on the BookCorpus, EN-DE, and EN-FR corpora, thus showing state-of-the-art compression ratios compared to other deep learning and traditional approaches. Furthermore, the RejuvenateForme achieves a BLEU score of 27.31, 25.78, and 50.45 on the EN-DE, EN-FR, and BookCorpus corpora, showcasing its comprehensive efficacy. In contrast, the pre-trained T5-Small exhibits better performance over prior state-of-the-art models.
• 摘要：文本压缩可以在保留关键信息的同时缩小文本数据，从而消除存储、带宽和计算效率方面的限制。尽管通信中英文文本数据量不断增加，但无损压缩技术与基于变压器的文本解压的集成却很少受到关注。推进文本压缩和恢复的主要障碍是优化基于变压器的方法并进行有效的预处理，以及集成无损压缩算法，而这些问题在之前的尝试中仍未得到解决。在这里，我们提出了一种基于变压器的文本解压缩方法，名为 RejuvenateForme，通过利用新的预处理技术和无损压缩方法解决了先前的问题。我们结合 Lempel-Ziv-Welch 算法的细致预处理技术在 BookCorpus、EN-DE 和 EN-FR 语料库上实现了 12.57、13.38 和 11.42 的压缩比，与其他深度学习和传统方法相比，其压缩比最高。此外，RejuvenateForme 在 EN-DE、EN-FR 和 BookCorpus 语料库上取得了 27.31、25.78 和 50.45 的 BLEU 分数，展现了其全面的功效。相比之下，预训练的 T5-Small 表现出比之前最先进的模型更好的性能。

Title: AgentPS: Agentic Process Supervision for Multi-modal Content Quality Assurance through Multi-round QA

• Authors: Gorden Liu, Yu Sun, Ruixiao Sun, Xin Dong, Hongyu Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15251
• Pdf URL: https://arxiv.org/pdf/2412.15251
• Copy Paste: [[2412.15251]] AgentPS: Agentic Process Supervision for Multi-modal Content Quality Assurance through Multi-round QA(https://arxiv.org/abs/2412.15251)
• Keywords: language model, llm, agent
• Abstract: The advanced processing and reasoning capabilities of multimodal large language models (MLLMs) have driven substantial progress in vision-language (VL) understanding tasks. However, while effective for tasks governed by straightforward logic, MLLMs often encounter challenges when reasoning over complex, interdependent logic structures. To address this limitation, we introduce \textit{AgentPS}, a novel framework that integrates Agentic Process Supervision into MLLMs via multi-round question answering during fine-tuning. \textit{AgentPS} demonstrates significant performance improvements over baseline MLLMs on proprietary TikTok datasets, due to its integration of process supervision and structured sequential reasoning. Furthermore, we show that replacing human-annotated labels with LLM-generated labels retains much of the performance gain, highlighting the framework's practical scalability in industrial applications. These results position \textit{AgentPS} as a highly effective and efficient architecture for multimodal classification tasks. Its adaptability and scalability, especially when enhanced by automated annotation generation, make it a powerful tool for handling large-scale, real-world challenges.
• 摘要：多模态大型语言模型 (MLLM) 的高级处理和推理能力推动了视觉语言 (VL) 理解任务的重大进展。然而，虽然 MLLM 对于由简单逻辑控制的任务有效，但在推理复杂、相互依赖的逻辑结构时，它们经常会遇到挑战。为了解决这一限制，我们引入了 \textit{AgentPS}，这是一个新颖的框架，它通过微调期间的多轮问答将代理过程监督集成到 MLLM 中。\textit{AgentPS} 由于集成了过程监督和结构化顺序推理，在专有 TikTok 数据集上表现出比基线 MLLM 显着的性能改进。此外，我们表明，用 LLM 生成的标签替换人工注释的标签保留了大部分性能提升，突出了该框架在工业应用中的实际可扩展性。这些结果将 \textit{AgentPS} 定位为多模态分类任务的高效架构。它的适应性和可扩展性，尤其是通过自动注释生成增强时，使其成为处理大规模现实挑战的有力工具。

Title: NER- RoBERTa: Fine-Tuning RoBERTa for Named Entity Recognition (NER) within low-resource languages

• Authors: Abdulhady Abas Abdullah, Srwa Hasan Abdulla, Dalia Mohammad Toufiq, Halgurd S. Maghdid, Tarik A. Rashid, Pakshan F. Farho, Shadan Sh. Sabr, Akar H. Taher, Darya S. Hamad, Hadi Veisi, Aras T. Asaad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15252
• Pdf URL: https://arxiv.org/pdf/2412.15252
• Copy Paste: [[2412.15252]] NER- RoBERTa: Fine-Tuning RoBERTa for Named Entity Recognition (NER) within low-resource languages(https://arxiv.org/abs/2412.15252)
• Keywords: gpt, chat
• Abstract: Nowadays, Natural Language Processing (NLP) is an important tool for most people's daily life routines, ranging from understanding speech, translation, named entity recognition (NER), and text categorization, to generative text models such as ChatGPT. Due to the existence of big data and consequently large corpora for widely used languages like English, Spanish, Turkish, Persian, and many more, these applications have been developed accurately. However, the Kurdish language still requires more corpora and large datasets to be included in NLP applications. This is because Kurdish has a rich linguistic structure, varied dialects, and a limited dataset, which poses unique challenges for Kurdish NLP (KNLP) application development. While several studies have been conducted in KNLP for various applications, Kurdish NER (KNER) remains a challenge for many KNLP tasks, including text analysis and classification. In this work, we address this limitation by proposing a methodology for fine-tuning the pre-trained RoBERTa model for KNER. To this end, we first create a Kurdish corpus, followed by designing a modified model architecture and implementing the training procedures. To evaluate the trained model, a set of experiments is conducted to demonstrate the performance of the KNER model using different tokenization methods and trained models. The experimental results show that fine-tuned RoBERTa with the SentencePiece tokenization method substantially improves KNER performance, achieving a 12.8% improvement in F1-score compared to traditional models, and consequently establishes a new benchmark for KNLP.
• 摘要：如今，自然语言处理 (NLP) 已成为大多数人日常生活中的重要工具，从理解语音、翻译、命名实体识别 (NER) 和文本分类，到 ChatGPT 等生成文本模型。由于大数据的存在，以及英语、西班牙语、土耳其语、波斯语等广泛使用的语言的大型语料库的存在，这些应用程序得到了准确的开发。然而，库尔德语仍然需要更多的语料库和大型数据集才能包含在 NLP 应用程序中。这是因为库尔德语具有丰富的语言结构、多样的方言和有限的数据集，这对库尔德语 NLP (KNLP) 应用程序开发提出了独特的挑战。虽然已经针对各种应用对 KNLP 进行了多项研究，但库尔德语 NER (KNER) 仍然是许多 KNLP 任务的挑战，包括文本分析和分类。在这项工作中，我们通过提出一种微调 KNER 预训练的 RoBERTa 模型的方法来解决这一限制。为此，我们首先创建了一个库尔德语语料库，然后设计了一个修改后的模型架构并实施了训练程序。为了评估训练后的模型，我们进行了一组实验，以使用不同的标记方法和训练后的模型来展示 KNER 模型的性能。实验结果表明，使用 SentencePiece 标记方法微调的 RoBERTa 显著提高了 KNER 的性能，与传统模型相比，F1 分数提高了 12.8%，从而为 KNLP 建立了新的基准。

Title: Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction

• Authors: Andrea Cristina McGlinchey, Peter J Barclay
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15253
• Pdf URL: https://arxiv.org/pdf/2412.15253
• Copy Paste: [[2412.15253]] Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction(https://arxiv.org/abs/2412.15253)
• Keywords: language model, gpt, chat
• Abstract: Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and "fake news". However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers' output to generate "sham books" in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Naive Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.
• 摘要：随着 ChatGPT 的发布，生成式 AI 系统已普遍可用，大型语言模型创建的欺骗性文本的自动检测主要集中在学术剽窃和“假新闻”等领域。然而，生成式 AI 也对创意作家的生计构成威胁，甚至可能对整个文学文化构成威胁，因为它会降低出版材料的质量。用作家的作品训练大型语言模型来生成特定风格的“假书”似乎构成了一种新的剽窃形式。这个问题很少有人研究。在本研究中，我们训练了机器学习分类器模型来区分人类编写的短篇小说和机器生成的创意小说，重点关注经典侦探小说。我们的结果表明，朴素贝叶斯和多层感知器分类器取得了很高的成功率（准确率 > 95%），明显优于人类评判者（准确率 < 55%）。这种方法对短文本样本（约 100 个单词）效果很好，之前的研究表明这些样本很难分类。我们部署了在线概念验证分类工具 AI Detective，这是开发供编辑和出版商使用的轻量级可靠应用程序的第一步，目的是保护人类作者的经济和文化贡献。

Title: RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts

• Authors: Ali Hamdi, Hozaifa Kassab, Mohamed Bahaa, Marwa Mohamed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15254
• Pdf URL: https://arxiv.org/pdf/2412.15254
• Copy Paste: [[2412.15254]] RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts(https://arxiv.org/abs/2412.15254)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have significantly advanced natural language processing, excelling in areas like text generation, summarization, and question-answering. Despite their capabilities, these models face challenges when fine-tuned on small, domain-specific datasets, often struggling to generalize and deliver accurate results with unfamiliar inputs. To tackle this issue, we introduce RIRO, a novel two-layer architecture designed to improve performance in data-scarce environments. The first layer leverages advanced prompt engineering to reformulate inputs, ensuring better alignment with training data, while the second layer focuses on refining outputs to minimize inconsistencies. Through fine-tuning models like Phi-2, Falcon 7B, and Falcon 1B, with Phi-2 outperforming the others. Additionally, we introduce a benchmark using evaluation metrics such as cosine similarity, Levenshtein distance, BLEU score, ROUGE-1, ROUGE-2, and ROUGE-L. While these advancements improve performance, challenges like computational demands and overfitting persist, limiting the potential of LLMs in data-scarce, high-stakes environments such as healthcare, legal documentation, and software testing.
• 摘要：大型语言模型 (LLM) 显著提高了自然语言处理能力，在文本生成、摘要和问答等领域表现出色。尽管这些模型功能强大，但它们在针对小型、特定领域的数据集进行微调时仍面临挑战，通常难以使​​用不熟悉的输入进行概括并提供准确的结果。为了解决这个问题，我们引入了 RIRO，这是一种新颖的两层架构，旨在提高数据稀缺环境中的性能。第一层利用先进的提示工程来重新制定输入，确保与训练数据更好地对齐，而第二层则专注于优化输出以最大限度地减少不一致性。通过微调 Phi-2、Falcon 7B 和 Falcon 1B 等模型，Phi-2 的表现优于其他模型。此外，我们引入了一个基准，使用余弦相似度、Levenshtein 距离、BLEU 分数、ROUGE-1、ROUGE-2 和 ROUGE-L 等评估指标。虽然这些进步提高了性能，但计算需求和过度拟合等挑战仍然存在，限制了 LLM 在数据稀缺、高风险环境（如医疗保健、法律文档和软件测试）中的潜力。

Title: Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation

• Authors: Jonibek Mansurov, Akhmed Sakip, Alham Fikri Aji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15255
• Pdf URL: https://arxiv.org/pdf/2412.15255
• Copy Paste: [[2412.15255]] Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation(https://arxiv.org/abs/2412.15255)
• Keywords: language model
• Abstract: In this paper, we show that knowledge distillation can be subverted to manipulate language model benchmark scores, revealing a critical vulnerability in current evaluation practices. We introduce "Data Laundering," a three-phase process analogous to financial money laundering, that enables the covert transfer of benchmark-specific knowledge through seemingly legitimate intermediate training steps. Through extensive experiments with a 2-layer BERT student model, we show how this approach can achieve substantial improvements in benchmark accuracy (up to 75\% on GPQA) without developing genuine reasoning capabilities. Notably, this method can be exploited intentionally or even unintentionally, as researchers may inadvertently adopt this method that inflates scores using knowledge distillation without realizing the implications. While our findings demonstrate the effectiveness of this technique, we present them as a cautionary tale highlighting the urgent need for more robust evaluation methods in AI. This work aims to contribute to the ongoing discussion about evaluation integrity in AI development and the need for benchmarks that more accurately reflect true model capabilities. The code is available at \url{this https URL}.
• 摘要：在本文中，我们表明知识蒸馏可以被破坏以操纵语言模型基准分数，揭示了当前评估实践中的一个关键漏洞。我们引入了“数据洗钱”，这是一个类似于金融洗钱的三阶段过程，它可以通过看似合法的中间训练步骤秘密转移基准特定知识。通过对 2 层 BERT 学生模型进行大量实验，我们展示了这种方法如何在不开发真正推理能力的情况下实现基准准确度的大幅提高（在 GPQA 上高达 75\%）。值得注意的是，这种方法可以被有意或无意地利用，因为研究人员可能会无意中采用这种使用知识蒸馏来夸大分数的方法，而没有意识到其影响。虽然我们的研究结果证明了这种技术的有效性，但我们将它们作为一个警示故事，强调迫切需要更强大的 AI 评估方法。这项工作旨在促进关于 AI 开发中评估完整性的持续讨论，以及对更准确反映真实模型能力的基准的需求。代码可在 \url{this https URL} 获得。

Title: Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search

• Authors: Edward Kim, Manil Shrestha, Richard Foty, Tom DeLay, Vicki Seyfert-Margolis
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15256
• Pdf URL: https://arxiv.org/pdf/2412.15256
• Copy Paste: [[2412.15256]] Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search(https://arxiv.org/abs/2412.15256)
• Keywords: language model, llm
• Abstract: Creation and curation of knowledge graphs can accelerate disease discovery and analysis in real-world data. While disease ontologies aid in biological data annotation, codified categories (SNOMED-CT, ICD10, CPT) may not capture patient condition nuances or rare diseases. Multiple disease definitions across data sources complicate ontology mapping and disease clustering. We propose creating patient knowledge graphs using large language model extraction techniques, allowing data extraction via natural language rather than rigid ontological hierarchies. Our method maps to existing ontologies (MeSH, SNOMED-CT, RxNORM, HPO) to ground extracted entities. Using a large ambulatory care EHR database with 33.6M patients, we demonstrate our method through the patient search for Dravet syndrome, which received ICD10 recognition in October 2020. We describe our construction of patient-specific knowledge graphs and symptom-based patient searches. Using confirmed Dravet syndrome ICD10 codes as ground truth, we employ LLM-based entity extraction to characterize patients in grounded ontologies. We then apply this method to identify Beta-propeller protein-associated neurodegeneration (BPAN) patients, demonstrating real-world discovery where no ground truth exists.
• 摘要：创建和管理知识图谱可以加速现实世界数据中的疾病发现和分析。虽然疾病本体有助于生物数据注释，但编码类别（SNOMED-CT、ICD10、CPT）可能无法捕捉患者病情的细微差别或罕见疾病。跨数据源的多种疾病定义使本体映射和疾病聚类变得复杂。我们建议使用大型语言模型提取技术创建患者知识图谱，允许通过自然语言而不是严格的本体层次结构提取数据。我们的方法映射到现有本体（MeSH、SNOMED-CT、RxNORM、HPO）以对提取的实体进行基础处理。使用包含 3360 万患者的大型门诊护理 EHR 数据库，我们通过对 Dravet 综合征的患者搜索展示了我们的方法，该综合征于 2020 年 10 月获得 ICD10 认可。我们描述了我们针对患者的知识图谱的构建和基于症状的患者搜索。使用已确认的 Dravet 综合征 ICD10 代码作为基本事实，我们采用基于 LLM 的实体提取来表征基础本体中的患者。然后，我们应用此方法来识别与 Beta-螺旋桨蛋白相关的神经变性 (BPAN) 患者，展示了在不存在基本事实的情况下在现实世界中的发现。

Title: DisEmbed: Transforming Disease Understanding through Embeddings

• Authors: Salman Faroz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15258
• Pdf URL: https://arxiv.org/pdf/2412.15258
• Copy Paste: [[2412.15258]] DisEmbed: Transforming Disease Understanding through Embeddings(https://arxiv.org/abs/2412.15258)
• Keywords: retrieval-augmented generation
• Abstract: The medical domain is vast and diverse, with many existing embedding models focused on general healthcare applications. However, these models often struggle to capture a deep understanding of diseases due to their broad generalization across the entire medical field. To address this gap, I present DisEmbed, a disease-focused embedding model. DisEmbed is trained on a synthetic dataset specifically curated to include disease descriptions, symptoms, and disease-related Q\&A pairs, making it uniquely suited for disease-related tasks. For evaluation, I benchmarked DisEmbed against existing medical models using disease-specific datasets and the triplet evaluation method. My results demonstrate that DisEmbed outperforms other models, particularly in identifying disease-related contexts and distinguishing between similar diseases. This makes DisEmbed highly valuable for disease-specific use cases, including retrieval-augmented generation (RAG) tasks, where its performance is particularly robust.
• 摘要：医学领域广阔而多样，许多现有的嵌入模型专注于一般医疗保健应用。然而，由于这些模型在整个医学领域的广泛推广，它们往往难以深入了解疾病。为了弥补这一差距，我提出了一种以疾病为中心的嵌入模型 DisEmbed。DisEmbed 是在专门整理的合成数据集上训练的，该数据集包含疾病描述、症状和与疾病相关的问答对，使其特别适合与疾病相关的任务。为了进行评估，我使用疾病特定数据集和三重评估方法将 DisEmbed 与现有医学模型进行了对比。我的结果表明，DisEmbed 优于其他模型，特别是在识别与疾病相关的背景和区分相似疾病方面。这使得 DisEmbed 对于疾病特定用例非常有价值，包括检索增强生成 (RAG) 任务，其性能特别强大。

Title: Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice

• Authors: Hannes Westermann, Jaromir Savelka
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.15260
• Pdf URL: https://arxiv.org/pdf/2412.15260
• Copy Paste: [[2412.15260]] Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice(https://arxiv.org/abs/2412.15260)
• Keywords: language model, llm
• Abstract: Interacting with the legal system and the government requires the assembly and analysis of various pieces of information that can be spread across different (paper) documents, such as forms, certificates and contracts (e.g. leases). This information is required in order to understand one's legal rights, as well as to fill out forms to file claims in court or obtain government benefits. However, finding the right information, locating the correct forms and filling them out can be challenging for laypeople. Large language models (LLMs) have emerged as a powerful technology that has the potential to address this gap, but still rely on the user to provide the correct information, which may be challenging and error-prone if the information is only available in complex paper documents. We present an investigation into utilizing multi-modal LLMs to analyze images of handwritten paper forms, in order to automatically extract relevant information in a structured format. Our initial results are promising, but reveal some limitations (e.g., when the image quality is low). Our work demonstrates the potential of integrating multi-modal LLMs to support laypeople and self-represented litigants in finding and assembling relevant information.
• 摘要：与法律系统和政府的互动需要收集和分析各种信息，这些信息可能分散在不同（纸质）文件中，例如表格、证书和合同（例如租约）。这些信息是了解一个人的合法权利以及填写表格以向法院提出索赔或获得政府福利所必需的。然而，对于外行人来说，找到正确的信息、找到正确的表格并填写它们可能具有挑战性。大型语言模型 (LLM) 已经成为一种强大的技术，有可能解决这一差距，但仍然依赖于用户提供正确的信息，如果信息仅在复杂的纸质文档中可用，这可能具有挑战性且容易出错。我们提出了一项调查，利用多模式 LLM 分析手写纸质表格的图像，以便自动以结构化格式提取相关信息。我们的初步结果很有希望，但也暴露了一些局限性（例如，当图像质量较低时）。我们的工作展示了整合多模式法学硕士 (LLM) 的潜力，以支持外行和自我代理的诉讼当事人查找和收集相关信息。

Title: Advanced ingestion process powered by LLM parsing for RAG system

• Authors: Arnau Perez, Xavier Vizcaino
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15262
• Pdf URL: https://arxiv.org/pdf/2412.15262
• Copy Paste: [[2412.15262]] Advanced ingestion process powered by LLM parsing for RAG system(https://arxiv.org/abs/2412.15262)
• Keywords: llm, retrieval augmented generation, agent
• Abstract: Retrieval Augmented Generation (RAG) systems struggle with processing multimodal documents of varying structural complexity. This paper introduces a novel multi-strategy parsing approach using LLM-powered OCR to extract content from diverse document types, including presentations and high text density files both scanned or not. The methodology employs a node-based extraction technique that creates relationships between different information types and generates context-aware metadata. By implementing a Multimodal Assembler Agent and a flexible embedding strategy, the system enhances document comprehension and retrieval capabilities. Experimental evaluations across multiple knowledge bases demonstrate the approach's effectiveness, showing improvements in answer relevancy and information faithfulness.
• 摘要：检索增强生成 (RAG) 系统难以处理结构复杂程度各异的多模态文档。本文介绍了一种新颖的多策略解析方法，该方法使用 LLM 驱动的 OCR 从各种文档类型中提取内容，包括演示文稿和扫描或未扫描的高文本密度文件。该方法采用基于节点的提取技术，在不同信息类型之间建立关系并生成上下文感知元数据。通过实施多模态汇编器代理和灵活的嵌入策略，该系统增强了文档理解和检索能力。跨多个知识库的实验评估证明了该方法的有效性，表明答案相关性和信息真实性有所提高。

Title: ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports

• Authors: Romain Hardy, Sung Eun Kim, Pranav Rajpurkar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15264
• Pdf URL: https://arxiv.org/pdf/2412.15264
• Copy Paste: [[2412.15264]] ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports(https://arxiv.org/abs/2412.15264)
• Keywords: language model, hallucination
• Abstract: The increasing adoption of AI-generated radiology reports necessitates robust methods for detecting hallucinations--false or unfounded statements that could impact patient care. We present ReXTrust, a novel framework for fine-grained hallucination detection in AI-generated radiology reports. Our approach leverages sequences of hidden states from large vision-language models to produce finding-level hallucination risk scores. We evaluate ReXTrust on a subset of the MIMIC-CXR dataset and demonstrate superior performance compared to existing approaches, achieving an AUROC of 0.8751 across all findings and 0.8963 on clinically significant findings. Our results show that white-box approaches leveraging model hidden states can provide reliable hallucination detection for medical AI systems, potentially improving the safety and reliability of automated radiology reporting.
• 摘要：随着人工智能生成的放射学报告的日益普及，需要有强大的方法来检测幻觉——可能影响患者护理的虚假或毫无根据的陈述。我们提出了 ReXTrust，这是一种用于在人工智能生成的放射学报告中进行细粒度幻觉检测的新型框架。我们的方法利用来自大型视觉语言模型的隐藏状态序列来生成发现级幻觉风险评分。我们在 MIMIC-CXR 数据集的一个子集上评估了 ReXTrust，并展示了与现有方法相比更优异的性能，在所有发现中实现了 0.8751 的 AUROC，在具有临床意义的发现中实现了 0.8963 的 AUROC。我们的结果表明，利用模型隐藏状态的白盒方法可以为医疗人工智能系统提供可靠的幻觉检测，从而有可能提高自动放射学报告的安全性和可靠性。

Title: Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models

• Authors: Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiangyong Zhu, Bo Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15265
• Pdf URL: https://arxiv.org/pdf/2412.15265
• Copy Paste: [[2412.15265]] Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models(https://arxiv.org/abs/2412.15265)
• Keywords: language model, llm
• Abstract: With the rapid advancement of Large Language Models (LLMs), significant safety concerns have emerged. Fundamentally, the safety of large language models is closely linked to the accuracy, comprehensiveness, and clarity of their understanding of safety knowledge, particularly in domains such as law, policy and ethics. This factuality ability is crucial in determining whether these models can be deployed and applied safely and compliantly within specific regions. To address these challenges and better evaluate the factuality ability of LLMs to answer short questions, we introduce the Chinese SafetyQA benchmark. Chinese SafetyQA has several properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate, Safety-related, Harmless). Based on Chinese SafetyQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs and analyze how these capabilities relate to LLM abilities, e.g., RAG ability and robustness against attacks.
• 摘要：随着大型语言模型 (LLM) 的快速发展，重大的安全问题也随之出现。从根本上讲，大型语言模型的安全性与其对安全知识理解的准确性、全面性和清晰性密切相关，特别是在法律、政策和道德等领域。这种事实性能力对于决定这些模型是否可以在特定区域内安全合规地部署和应用至关重要。为了应对这些挑战并更好地评估 LLM 回答简短问题的事实性能力，我们引入了中文 SafetyQA 基准。中文 SafetyQA 具有几个属性（即中文、多样、高质量、静态、易于评估、安全相关、无害）。基于中文 SafetyQA，我们对现有 LLM 的事实性能力进行了全面评估，并分析了这些能力与 LLM 能力（例如 RAG 能力和抗攻击能力）的关系。

Title: On the Structural Memory of LLM Agents

• Authors: Ruihong Zeng, Jinyuan Fang, Siwei Liu, Zaiqiao Meng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15266
• Pdf URL: https://arxiv.org/pdf/2412.15266
• Copy Paste: [[2412.15266]] On the Structural Memory of LLM Agents(https://arxiv.org/abs/2412.15266)
• Keywords: language model, llm, agent
• Abstract: Memory plays a pivotal role in enabling large language model~(LLM)-based agents to engage in complex and long-term interactions, such as question answering (QA) and dialogue systems. While various memory modules have been proposed for these tasks, the impact of different memory structures across tasks remains insufficiently explored. This paper investigates how memory structures and memory retrieval methods affect the performance of LLM-based agents. Specifically, we evaluate four types of memory structures, including chunks, knowledge triples, atomic facts, and summaries, along with mixed memory that combines these components. In addition, we evaluate three widely used memory retrieval methods: single-step retrieval, reranking, and iterative retrieval. Extensive experiments conducted across four tasks and six datasets yield the following key insights: (1) Different memory structures offer distinct advantages, enabling them to be tailored to specific tasks; (2) Mixed memory structures demonstrate remarkable resilience in noisy environments; (3) Iterative retrieval consistently outperforms other methods across various scenarios. Our investigation aims to inspire further research into the design of memory systems for LLM-based agents.
• 摘要：记忆在使基于大型语言模型 (LLM) 的代理能够参与复杂和长期交互（例如问答 (QA) 和对话系统）方面起着关键作用。虽然已经为这些任务提出了各种记忆模块，但不同记忆结构对任务的影响仍未得到充分探索。本文研究了记忆结构和记忆检索方法如何影响基于 LLM 的代理的性能。具体来说，我们评估了四种类型的记忆结构，包括块、知识三元组、原子事实和摘要，以及结合了这些组件的混合记忆。此外，我们还评估了三种广泛使用的记忆检索方法：单步检索、重新排序和迭代检索。在四项任务和六个数据集上进行的大量实验得出了以下关键见解：（1）不同的记忆结构具有不同的优势，使它们能够针对特定任务进行定制；（2）混合记忆结构在嘈杂环境中表现出显着的弹性；（3）迭代检索在各种场景中始终优于其他方法。我们的研究旨在启发对基于 LLM 代理的记忆系统设计的进一步研究。

Title: Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph

• Authors: Yibo Zhao, Jiapeng Zhu, Can Xu, Xiang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15268
• Pdf URL: https://arxiv.org/pdf/2412.15268
• Copy Paste: [[2412.15268]] Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph(https://arxiv.org/abs/2412.15268)
• Keywords: language model, llm
• Abstract: The rapid growth of social media platforms has raised significant concerns regarding online content toxicity. When Large Language Models (LLMs) are used for toxicity detection, two key challenges emerge: 1) the absence of domain-specific toxic knowledge leads to false negatives; 2) the excessive sensitivity of LLMs to toxic speech results in false positives, limiting freedom of speech. To address these issues, we propose a novel method called MetaTox, leveraging graph search on a meta-toxic knowledge graph to enhance hatred and toxicity detection. First, we construct a comprehensive meta-toxic knowledge graph by utilizing LLMs to extract toxic information through a three-step pipeline, with toxic benchmark datasets serving as corpora. Second, we query the graph via retrieval and ranking processes to supplement accurate, relevant toxic knowledge. Extensive experiments and in-depth case studies across multiple datasets demonstrate that our MetaTox significantly decreases the false positive rate while boosting overall toxicity detection performance. Our code will be available soon.
• 摘要：社交媒体平台的快速发展引发了人们对在线内容毒性的极大担忧。当使用大型语言模型 (LLM) 进行毒性检测时，会出现两个关键挑战：1) 缺乏特定领域的毒性知识会导致假阴性；2) LLM 对毒性言论的过度敏感性会导致假阳性，从而限制言论自由。为了解决这些问题，我们提出了一种名为 MetaTox 的新方法，利用元毒性知识图谱上的图形搜索来增强仇恨和毒性检测。首先，我们利用 LLM 通过三步管道提取毒性信息，构建一个全面的元毒性知识图谱，并以毒性基准数据集作为语料库。其次，我们通过检索和排名过程查询图谱，以补充准确、相关的毒性知识。在多个数据集上进行的大量实验和深入案例研究表明，我们的 MetaTox 显着降低了假阳性率，同时提高了整体毒性检测性能。我们的代码即将推出。

Title: The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration

• Authors: Geetanjali Bihani, Julia Rayz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15269
• Pdf URL: https://arxiv.org/pdf/2412.15269
• Copy Paste: [[2412.15269]] The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration(https://arxiv.org/abs/2412.15269)
• Keywords: language model
• Abstract: The advent of pre-trained language models (PLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found PLMs to suffer from miscalibration, indicating a lack of accuracy in the confidence estimates provided by these models. Current evaluation methods for PLM calibration often assume that lower calibration error estimates indicate more reliable predictions. However, fine-tuned PLMs often resort to shortcuts, leading to overconfident predictions that create the illusion of enhanced performance but lack generalizability in their decision rules. The relationship between PLM reliability, as measured by calibration error, and shortcut learning, has not been thoroughly explored thus far. This paper aims to investigate this relationship, studying whether lower calibration error implies reliable decision rules for a language model. Our findings reveal that models with seemingly superior calibration portray higher levels of non-generalizable decision rules. This challenges the prevailing notion that well-calibrated models are inherently reliable. Our study highlights the need to bridge the current gap between language model calibration and generalization objectives, urging the development of comprehensive frameworks to achieve truly robust and reliable language models.
• 摘要：预训练语言模型 (PLM) 的出现使自然语言处理领域的性能显著提升。然而，最近的研究发现 PLM 存在校准错误，表明这些模型提供的置信度估计缺乏准确性。当前 PLM 校准的评估方法通常假设较低的校准误差估计表示更可靠的预测。然而，经过微调的 PLM 通常会走捷径，导致过度自信的预测，造成性能增强的假象，但其决策规则缺乏普遍性。迄今为止，尚未彻底探索 PLM 可靠性（以校准误差衡量）与捷径学习之间的关系。本文旨在研究这种关系，研究较低的校准误差是否意味着语言模型的决策规则可靠。我们的研究结果表明，具有看似优越的校准的模型描绘了更高水平的非普遍性决策规则。这挑战了普遍存在的“校准良好的模型本质上是可靠的”观念。我们的研究强调需要弥合语言模型校准和泛化目标之间的当前差距，敦促开发全面的框架以实现真正强大而可靠的语言模型。

Title: Baichuan4-Finance Technical Report

• Authors: Hanyu Zhang, Boyu Qiu, Yuhao Feng, Shuqi Li, Qian Ma, Xiyuan Zhang, Qiang Ju, Dong Yan, Jian Xie
• Subjects: cs.CL, cs.AI, cs.CE, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15270
• Pdf URL: https://arxiv.org/pdf/2412.15270
• Copy Paste: [[2412.15270]] Baichuan4-Finance Technical Report(https://arxiv.org/abs/2412.15270)
• Keywords: language model, llm, chat
• Abstract: Large language models (LLMs) have demonstrated strong capabilities in language understanding, generation, and reasoning, yet their potential in finance remains underexplored due to the complexity and specialization of financial knowledge. In this work, we report the development of the Baichuan4-Finance series, including a comprehensive suite of foundational Baichuan4-Finance-Base and an aligned language model Baichuan4-Finance, which are built upon Baichuan4-Turbo base model and tailored for finance domain. Firstly, we have dedicated significant effort to building a detailed pipeline for improving data quality. Moreover, in the continual pre-training phase, we propose a novel domain self-constraint training strategy, which enables Baichuan4-Finance-Base to acquire financial knowledge without losing general capabilities. After Supervised Fine-tuning and Reinforcement Learning from Human Feedback and AI Feedback, the chat model Baichuan4-Finance is able to tackle various financial certification questions and real-world scenario applications. We evaluate Baichuan4-Finance on many widely used general datasets and two holistic financial benchmarks. The evaluation results show that Baichuan4-Finance-Base surpasses almost all competitive baselines on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. At the same time, Baichuan4-Finance demonstrates even more impressive performance on financial application scenarios, showcasing its potential to foster community innovation in the financial LLM field.
• 摘要：大型语言模型（LLM）在语言理解、生成和推理方面表现出了强大的能力，但由于金融知识的复杂性和专业性，它们在金融领域的潜力仍未得到充分挖掘。在本文中，我们报告了 Baichuan4-Finance 系列的开发，包括一整套基础 Baichuan4-Finance-Base 和一个对齐的语言模型 Baichuan4-Finance，它们建立在 Baichuan4-Turbo 基础模型之上，并针对金融领域量身定制。首先，我们投入了大量精力构建详细的管道以提高数据质量。此外，在持续的预训练阶段，我们提出了一种新颖的领域自约束训练策略，使 Baichuan4-Finance-Base 能够在不损失一般能力的情况下获取金融知识。经过有监督的微调和从人类反馈和人工智能反馈中进行强化学习后，聊天模型 Baichuan4-Finance 能够解决各种金融认证问题和真实场景应用。我们在多个广泛使用的通用数据集和两个整体金融基准上对 Baichuan4-Finance 进行了评估。评估结果表明，Baichuan4-Finance-Base 在金融任务上以显著优势超越了几乎所有竞争基线，同时没有牺牲通用 LLM 基准上的性能。同时，Baichuan4-Finance 在金融应用场景上的表现更为出色，展现出其在金融 LLM 领域促进社区创新的潜力。

Title: A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models

• Authors: Gongbo Zhang, Zihan Xu, Qiao Jin, Fangyi Chen, Yilu Fang, Yi Liu, Justin F. Rousseau, Ziyang Xu, Zhiyong Lu, Chunhua Weng, Yifan Peng
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15271
• Pdf URL: https://arxiv.org/pdf/2412.15271
• Copy Paste: [[2412.15271]] A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models(https://arxiv.org/abs/2412.15271)
• Keywords: language model, llm, long context, hallucination, retrieval-augmented generation
• Abstract: While holding great promise for improving and facilitating healthcare, large language models (LLMs) struggle to produce up-to-date responses on evolving topics due to outdated knowledge or hallucination. Retrieval-augmented generation (RAG) is a pivotal innovation that improves the accuracy and relevance of LLM responses by integrating LLMs with a search engine and external sources of knowledge. However, the quality of RAG responses can be largely impacted by the rank and density of key information in the retrieval results, such as the "lost-in-the-middle" problem. In this work, we aim to improve the robustness and reliability of the RAG workflow in the medical domain. Specifically, we propose a map-reduce strategy, BriefContext, to combat the "lost-in-the-middle" issue without modifying the model weights. We demonstrated the advantage of the workflow with various LLM backbones and on multiple QA datasets. This method promises to improve the safety and reliability of LLMs deployed in healthcare domains.
• 摘要：虽然大型语言模型 (LLM) 有望改善和促进医疗保健，但由于知识过时或存在幻觉，它们难以对不断发展的主题产生最新的反应。检索增强生成 (RAG) 是一项关键创新，它通过将 LLM 与搜索引擎和外部知识源集成在一起，提高了 LLM 响应的准确性和相关性。然而，RAG 响应的质量在很大程度上受到检索结果中关键信息的排名和密度的影响，例如“中间丢失”问题。在这项工作中，我们旨在提高医疗领域 RAG 工作流程的稳健性和可靠性。具体来说，我们提出了一种 map-reduce 策略 BriefContext，以在不修改模型权重的情况下解决“中间丢失”问题。我们通过各种 LLM 主干和多个 QA 数据集展示了该工作流程的优势。这种方法有望提高部署在医疗保健领域的 LLM 的安全性和可靠性。

Title: SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation

• Authors: Yuzheng Cai, Zhenyue Guo, Yiwen Pei, Wanrui Bian, Weiguo Zheng
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15272
• Pdf URL: https://arxiv.org/pdf/2412.15272
• Copy Paste: [[2412.15272]] SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation(https://arxiv.org/abs/2412.15272)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Recent advancements in large language models (LLMs) have shown impressive versatility across various tasks. To eliminate its hallucinations, retrieval-augmented generation (RAG) has emerged as a powerful approach, leveraging external knowledge sources like knowledge graphs (KGs). In this paper, we study the task of KG-driven RAG and propose a novel Similar Graph Enhanced Retrieval-Augmented Generation (SimGRAG) method. It effectively addresses the challenge of aligning query texts and KG structures through a two-stage process: (1) query-to-pattern, which uses an LLM to transform queries into a desired graph pattern, and (2) pattern-to-subgraph, which quantifies the alignment between the pattern and candidate subgraphs using a graph semantic distance (GSD) metric. We also develop an optimized retrieval algorithm that efficiently identifies the top-$k$ subgraphs within 1-second latency on a 10-million-scale KG. Extensive experiments show that SimGRAG outperforms state-of-the-art KG-driven RAG methods in both question answering and fact verification, offering superior plug-and-play usability and scalability.
• 摘要：大型语言模型 (LLM) 的最新进展已在各种任务中展现出令人印象深刻的多功能性。为了消除其幻觉，检索增强生成 (RAG) 已成为一种强大的方法，它利用知识图谱 (KG) 等外部知识源。在本文中，我们研究了 KG 驱动的 RAG 任务，并提出了一种新颖的相似图增强检索增强生成 (SimGRAG) 方法。它通过两阶段过程有效地解决了对齐查询文本和 KG 结构的挑战：(1) 查询到模式，使用 LLM 将查询转换为所需的图形模式，以及 (2) 模式到子图，使用图语义距离 (GSD) 度量量化模式与候选子图之间的对齐。我们还开发了一种优化的检索算法，可在 1 秒延迟内在 1000 万级 KG 上有效识别前 $k$ 个子图。大量实验表明，SimGRAG 在问答和事实验证方面均优于最先进的 KG 驱动的 RAG 方法，具有卓越的即插即用可用性和可扩展性。

Title: Memory-Augmented Agent Training for Business Document Understanding

• Authors: Jiale Liu, Yifan Zeng, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15274
• Pdf URL: https://arxiv.org/pdf/2412.15274
• Copy Paste: [[2412.15274]] Memory-Augmented Agent Training for Business Document Understanding(https://arxiv.org/abs/2412.15274)
• Keywords: language model, llm, prompt, agent
• Abstract: Traditional enterprises face significant challenges in processing business documents, where tasks like extracting transport references from invoices remain largely manual despite their crucial role in logistics operations. While Large Language Models offer potential automation, their direct application to specialized business domains often yields unsatisfactory results. We introduce Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a novel paradigm that enables LLM agents to progressively build domain expertise through experience-driven memory refinement and iterative learning. To validate this approach, we collaborate with one of the world's largest logistics companies to create a dataset of Universal Business Language format invoice documents, focusing on the task of transport reference extraction. Experiments demonstrate that Matrix outperforms prompting a single LLM by 30.3%, vanilla LLM agent by 35.2%. We further analyze the metrics of the optimized systems and observe that the agent system requires less API calls, fewer costs and can analyze longer documents on average. Our methods establish a new approach to transform general-purpose LLMs into specialized business tools through systematic memory enhancement in document processing tasks.
• 摘要：传统企业在处理业务文档方面面临重大挑战，尽管从发票中提取运输参考信息等任务在物流运营中发挥着至关重要的作用，但这些任务仍然主要是手动的。虽然大型语言模型提供了潜在的自动化，但它们直接应用于专业业务领域往往会产生不令人满意的结果。我们引入了 Matrix（通过推理和迭代探索进行记忆增强代理训练），这是一种新颖的范式，使 LLM 代理能够通过经验驱动的记忆细化和迭代学习逐步建立领域专业知识。为了验证这种方法，我们与世界上最大的物流公司之一合作，创建了一个通用商业语言格式发票文档数据集，重点关注运输参考提取任务。实验表明，Matrix 的表现比单个 LLM 提示高出 30.3%，比普通 LLM 代理高出 35.2%。我们进一步分析了优化系统的指标，并观察到代理系统需要更少的 API 调用、更少的成本，并且平均可以分析更长的文档。我们的方法建立了一种新方法，通过文档处理任务中的系统记忆增强，将通用 LLM 转变为专门的商业工具。

Title: PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models

• Authors: Biao Liu, Wenyi Fang, Xiaoyu Wu, Yang Zheng, Zheng Hu, Bo Yuan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15277
• Pdf URL: https://arxiv.org/pdf/2412.15277
• Copy Paste: [[2412.15277]] PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models(https://arxiv.org/abs/2412.15277)
• Keywords: language model, prompt
• Abstract: Pre-trained Vision-Language (VL) models such as CLIP have demonstrated their excellent performance across numerous downstream tasks. A recent method, Context Optimization (CoOp), further improves the performance of VL models on downstream tasks by introducing prompt learning. CoOp optimizes a set of learnable vectors, aka prompt, and freezes the whole CLIP model. However, relying solely on CLIP loss to fine-tune prompts can lead to models that are prone to overfitting on downstream task. To address this issue, we propose a plug-in prompt-regularization method called PLPP (Prompt Learning with PerPlexity), which use perplexity loss to regularize prompt learning. PLPP designs a two-step operation to compute the perplexity for prompts: (a) calculating cosine similarity between the weight of the embedding layer and prompts to get labels, (b) introducing a language model (LM) head that requires no training behind text encoder to output word probability distribution. Meanwhile, we unveil that the essence of PLPP is inherently a form of self-distillation. To further prevent overfitting as well as to reduce the additional computation introduced by PLPP, we turn the hard label to soft label and choose top-$k$ values for calculating the perplexity loss. For accelerating model convergence, we introduce mutual self-distillation learning, that is perplexity and inverted perplexity loss. The experiments conducted on four classification tasks indicate that PLPP exhibits superior performance compared to existing methods.
• 摘要：预训练的视觉语言 (VL) 模型（例如 CLIP）已在众多下游任务中展现出出色的性能。最近的一种方法，上下文优化 (CoOp)，通过引入提示学习，进一步提高了 VL 模型在下游任务上的性能。CoOp 优化了一组可学习向量（又称提示），并冻结了整个 CLIP 模型。但是，仅依靠 CLIP 损失来微调提示可能会导致模型在下游任务上容易过度拟合。为了解决这个问题，我们提出了一种名为 PLPP（使用 PerPlexity 的提示学习）的插件提示正则化方法，该方法使用困惑度损失来规范提示学习。PLPP 设计了一个两步操作来计算提示的困惑度：（a）计算嵌入层权重和提示之间的余弦相似度以获取标签，（b）引入语言模型 (LM) 头，它不需要在文本编码器后面进行训练即可输出单词概率分布。同时，我们揭示了 PLPP 的本质本质上是一种自我蒸馏。为了进一步防止过拟合，并减少 PLPP 引入的额外计算，我们将硬标签转换为软标签，并选择 top-k$ 值来计算困惑度损失。为了加速模型收敛，我们引入了相互的自我蒸馏学习，即困惑度和反向困惑度损失。在四个分类任务上进行的实验表明，与现有方法相比，PLPP 表现出了卓越的性能。

Title: Context-DPO: Aligning Language Models for Context-Faithfulness

• Authors: Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15280
• Pdf URL: https://arxiv.org/pdf/2412.15280
• Copy Paste: [[2412.15280]] Context-DPO: Aligning Language Models for Context-Faithfulness(https://arxiv.org/abs/2412.15280)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at this https URL
• 摘要：大型语言模型 (LLM) 的可靠响应需要遵守用户指令和检索到的信息。虽然对齐技术可以帮助 LLM 与人类的意图和价值观保持一致，但通过对齐来提高上下文忠诚度仍未得到充分探索。为了解决这个问题，我们提出了 $\textbf{Context-DPO}$，这是第一种专门为增强 LLM 的上下文忠诚度而设计的对齐方法。我们引入了 $\textbf{ConFiQA}$，这是一个基准，它模拟具有知识冲突的检索增强生成 (RAG) 场景来评估上下文忠诚度。通过利用 ConFiQA 提供的上下文对问题的忠实和固执的回答，我们的 Context-DPO 通过直接偏好优化来对齐 LLM。大量实验表明，我们的 Context-DPO 显着提高了上下文忠诚度，在流行的开源模型上实现了 35% 到 280% 的改进。进一步的分析表明，Context-DPO 保留了 LLM 的生成能力，同时提供了对上下文利用的可解释见解。我们的代码和数据发布在这个 https URL 上

Title: A Systematic Examination of Preference Learning through the Lens of Instruction-Following

• Authors: Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, Liang Tan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15282
• Pdf URL: https://arxiv.org/pdf/2412.15282
• Copy Paste: [[2412.15282]] A Systematic Examination of Preference Learning through the Lens of Instruction-Following(https://arxiv.org/abs/2412.15282)
• Keywords: language model, llm, prompt
• Abstract: Preference learning is a widely adopted post-training technique that aligns large language models (LLMs) to human preferences and improves specific downstream task capabilities. In this work we systematically investigate how specific attributes of preference datasets affect the alignment and downstream performance of LLMs in instruction-following tasks. We use a novel synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with combinations of 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS) - to obtain pairs of (chosen, rejected) responses. Then, we perform experiments investigating the effects of (1) the presence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the chosen, rejected responses and (3) the complexity of the training prompts. Our experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements and greater stability across challenging training configurations. High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance by balancing diversity and learning efficiency. Additionally, training on prompts of moderate difficulty leads to better generalization across tasks, even for more complex evaluation scenarios, compared to overly challenging prompts. Our findings provide actionable insights into optimizing preference data curation for instruction-following tasks, offering a scalable and effective framework for enhancing LLM training and alignment.
• 摘要：偏好学习是一种广泛采用的后训练技术，可将大型语言模型 (LLM) 与人类偏好对齐，并提高特定的下游任务能力。在这项工作中，我们系统地研究了偏好数据集的特定属性如何影响 LLM 在指令跟踪任务中的对齐和下游性能。我们使用一种新颖的合成数据生成管道来生成 48,000 个独特的指令跟踪提示，并结合 23 个可验证的约束，从而实现对模型响应的细粒度和自动化质量评估。对于我们的合成提示，我们使用两种偏好数据集管理方法 - 拒绝抽样 (RS) 和蒙特卡洛树搜索 (MCTS) - 来获得成对的 (选定、拒绝) 响应。然后，我们进行实验，研究 (1) 选定和拒绝的响应之间是否存在共享前缀、(2) 选定、拒绝的响应的对比度和质量以及 (3) 训练提示的复杂性的影响。我们的实验表明，由 MCTS 生成的偏好对中的共享前缀在具有挑战性的训练配置中提供了边际但一致的改进和更高的稳定性。高对比度偏好对通常优于低对比度偏好对；然而，将两者结合起来通常可以通过平衡多样性和学习效率来获得最佳性能。此外，与过于具有挑战性的提示相比，对中等难度的提示进行训练可以更好地在任务之间进行泛化，即使对于更复杂的评估场景也是如此。我们的研究结果为优化指令遵循任务的偏好数据管理提供了可行的见解，为增强 LLM 培训和协调提供了一个可扩展且有效的框架。

Title: Channel Merging: Preserving Specialization for Merged Experts

• Authors: Mingyang Zhang, Jing Liu, Ganggui Ding, Xinyi Yu, Linlin Ou, Bohan Zhuang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15283
• Pdf URL: https://arxiv.org/pdf/2412.15283
• Copy Paste: [[2412.15283]] Channel Merging: Preserving Specialization for Merged Experts(https://arxiv.org/abs/2412.15283)
• Keywords: language model, llm
• Abstract: Lately, the practice of utilizing task-specific fine-tuning has been implemented to improve the performance of large language models (LLM) in subsequent tasks. Through the integration of diverse LLMs, the overall competency of LLMs is significantly boosted. Nevertheless, traditional ensemble methods are notably memory-intensive, necessitating the simultaneous loading of all specialized models into GPU memory. To address the inefficiency, model merging strategies have emerged, merging all LLMs into one model to reduce the memory footprint during inference. Despite these advances, model merging often leads to parameter conflicts and performance decline as the number of experts increases. Previous methods to mitigate these conflicts include post-pruning and partial merging. However, both approaches have limitations, particularly in terms of performance and storage efficiency when merged experts increase. To address these challenges, we introduce Channel Merging, a novel strategy designed to minimize parameter conflicts while enhancing storage efficiency. This method clusters and merges channel parameters based on their similarity to form several groups offline. By ensuring that only highly similar parameters are merged within each group, it significantly reduces parameter conflicts. During inference, we can instantly look up the expert parameters from the merged groups, preserving specialized knowledge. Our experiments demonstrate that Channel Merging consistently delivers high performance, matching unmerged models in tasks like English and Chinese reasoning, mathematical reasoning, and code generation. Moreover, it obtains results comparable to model ensemble with just 53% parameters when used with a task-specific router.
• 摘要：最近，人们开始利用特定任务的微调来提高大型语言模型 (LLM) 在后续任务中的性能。通过整合不同的 LLM，LLM 的整体能力得到了显著提升。然而，传统的集成方法明显占用大量内存，需要同时将所有专门的模型加载到 GPU 内存中。为了解决效率低下的问题，出现了模型合并策略，将所有 LLM 合并为一个模型，以减少推理过程中的内存占用。尽管取得了这些进展，但随着专家数量的增加，模型合并通常会导致参数冲突和性能下降。以前缓解这些冲突的方法包括后修剪和部分合并。然而，这两种方法都有局限性，特别是在合并专家增加时在性能和存储效率方面。为了应对这些挑战，我们引入了通道合并，这是一种旨在最大限度地减少参数冲突同时提高存储效率的新策略。该方法根据通道参数的相似性对其进行聚类和合并，以离线形成多个组。通过确保每个组内仅合并高度相似的参数，它可以显著减少参数冲突。在推理过程中，我们可以立即从合并的组中查找专家参数，从而保留专业知识。我们的实验表明，通道合并始终提供高性能，在英语和中文推理、数学推理和代码生成等任务中匹配未合并的模型。此外，当与特定于任务的路由器一起使用时，它仅用 53% 的参数就能获得与模型集成相当的结果。

Title: Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

• Authors: Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15285
• Pdf URL: https://arxiv.org/pdf/2412.15285
• Copy Paste: [[2412.15285]] Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining(https://arxiv.org/abs/2412.15285)
• Keywords: language model, llm
• Abstract: Pretraining large language models effectively requires strategic data selection, blending and ordering. However, key details about data mixtures especially their scalability to longer token horizons and larger model sizes remain underexplored due to limited disclosure by model developers. To address this, we formalize the concept of two-phase pretraining and conduct an extensive systematic study on how to select and mix data to maximize model accuracies for the two phases. Our findings illustrate that a two-phase approach for pretraining outperforms random data ordering and natural distribution of tokens by 3.4% and 17% on average accuracies. We provide in-depth guidance on crafting optimal blends based on quality of the data source and the number of epochs to be seen. We propose to design blends using downsampled data at a smaller scale of 1T tokens and then demonstrate effective scaling of our approach to larger token horizon of 15T tokens and larger model size of 25B model size. These insights provide a series of steps practitioners can follow to design and scale their data blends.
• 摘要：有效地预训练大型语言模型需要战略性的数据选择、混合和排序。然而，由于模型开发人员的披露有限，有关数据混合的关键细节，尤其是它们对更长的标记范围和更大的模型大小的可扩展性仍未得到充分探索。为了解决这个问题，我们正式化了两阶段预训练的概念，并就如何选择和混合数据以最大化两个阶段的模型准确率进行了广泛的系统研究。我们的研究结果表明，两阶段预训练方法的平均准确率比随机数据排序和自然标记分布高出 3.4% 和 17%。我们根据数据源的质量和要看到的时期数提供关于制作最佳混合的深入指导。我们建议使用 1T 标记的较小规模的下采样数据来设计混合，然后展示我们的方法对 15T 标记的更大标记范围和 25B 模型大小的更大模型大小的有效扩展。这些见解为从业者提供了一系列可以遵循的步骤来设计和扩展他们的数据混合。

Title: Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

• Authors: Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, Aleksandra Faust
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15287
• Pdf URL: https://arxiv.org/pdf/2412.15287
• Copy Paste: [[2412.15287]] Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models(https://arxiv.org/abs/2412.15287)
• Keywords: language model, llm
• Abstract: Recent studies have indicated that effectively utilizing inference-time compute is crucial for attaining better performance from large language models (LLMs). In this work, we propose a novel inference-aware fine-tuning paradigm, in which the model is fine-tuned in a manner that directly optimizes the performance of the inference-time strategy. We study this paradigm using the simple yet effective Best-of-N (BoN) inference strategy, in which a verifier selects the best out of a set of LLM-generated responses. We devise the first imitation learning and reinforcement learning~(RL) methods for BoN-aware fine-tuning, overcoming the challenging, non-differentiable argmax operator within BoN. We empirically demonstrate that our BoN-aware models implicitly learn a meta-strategy that interleaves best responses with more diverse responses that might be better suited to a test-time input -- a process reminiscent of the exploration-exploitation trade-off in RL. Our experiments demonstrate the effectiveness of BoN-aware fine-tuning in terms of improved performance and inference-time compute. In particular, we show that our methods improve the Bo32 performance of Gemma 2B on Hendrycks MATH from 26.8% to 30.8%, and pass@32 from 60.0% to 67.0%, as well as the pass@16 on HumanEval from 61.6% to 67.1%.
• 摘要：最近的研究表明，有效利用推理时间计算对于从大型语言模型 (LLM) 中获得更好的性能至关重要。在这项工作中，我们提出了一种新颖的推理感知微调范式，其中模型以直接优化推理时间策略性能的方式进行微调。我们使用简单但有效的 Best-of-N (BoN) 推理策略研究此范式，其中验证者从一组 LLM 生成的响应中选择最佳响应。我们设计了第一个用于 BoN 感知微调的模仿学习和强化学习 (RL) 方法，克服了 BoN 中具有挑战性的不可微分 argmax 运算符。我们通过经验证明，我们的 BoN 感知模型隐式学习了一种元策略，该策略将最佳响应与可能更适合测试时间输入的更多样化的响应交织在一起——这一过程让人想起了 RL 中的探索-利用权衡。我们的实验证明了 BoN 感知微调在提高性能和推理时间计算方面的有效性。具体而言，我们表明我们的方法将 Gemma 2B 在 Hendrycks MATH 上的 Bo32 性能从 26.8% 提高到 30.8%，将 pass@32 从 60.0% 提高到 67.0%，并将 HumanEval 上的 pass@16 从 61.6% 提高到 67.1%。

Title: A Large-scale Empirical Study on Large Language Models for Election Prediction

• Authors: Chenxiao Yu, Zhaotian Weng, Yuangang Li, Zheng Li, Xiyang Hu, Yue Zhao
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.15291
• Pdf URL: https://arxiv.org/pdf/2412.15291
• Copy Paste: [[2412.15291]] A Large-scale Empirical Study on Large Language Models for Election Prediction(https://arxiv.org/abs/2412.15291)
• Keywords: language model, llm
• Abstract: Can Large Language Models (LLMs) accurately predict election outcomes? While LLMs have demonstrated impressive performance in healthcare, legal analysis, and creative applications, their capabilities in election forecasting remain uncertain. Notably, election prediction poses unique challenges: limited voter-level data, evolving political contexts, and the complexity of modeling human behavior. In the first part of this paper, we explore and introduce a multi-step reasoning framework for election prediction, which systematically integrates demographic, ideological, and time-sensitive factors. Validated on 2016 and 2020 real-world data and extensive synthetic personas, our approach adapts to changing political landscapes, reducing bias and significantly improving predictive accuracy. We further apply our pipeline to the 2024 U.S. presidential election, illustrating its ability to generalize beyond observed historical data. Beyond enhancing accuracy, the second part of the paper provides insights into the broader implications of LLM-based election forecasting. We identify potential political biases embedded in pretrained corpora, examine how demographic patterns can become exaggerated, and suggest strategies for mitigating these issues. Together, this project, a large-scale LLM empirical study, advances the accuracy of election predictions and establishes directions for more balanced, transparent, and context-aware modeling in political science research and practice.
• 摘要：大型语言模型 (LLM) 能否准确预测选举结果？虽然 LLM 在医疗保健、法律分析和创意应用方面表现出色，但它们在选举预测方面的能力仍不确定。值得注意的是，选举预测带来了独特的挑战：选民层面的数据有限、政治背景不断演变以及对人类行为建模的复杂性。在本文的第一部分，我们探索并介绍了一个用于选举预测的多步骤推理框架，该框架系统地整合了人口、意识形态和时间敏感因素。经过 2016 年和 2020 年现实世界数据和大量合成角色的验证，我们的方法可以适应不断变化的政治格局，减少偏见并显着提高预测准确性。我们进一步将我们的管道应用于 2024 年美国总统大选，说明其能够超越观察到的历史数据进行推广。除了提高准确性之外，本文的第二部分还深入了解了基于 LLM 的选举预测的更广泛影响。我们识别预训练语料库中嵌入的潜在政治偏见，研究人口模式如何被夸大，并提出缓解这些问题的策略。总之，这个项目是一项大规模的 LLM 实证研究，它提高了选举预测的准确性，并为政治科学研究和实践中更平衡、更透明、更情境感知的建模指明了方向。

Title: Confidence in the Reasoning of Large Language Models

• Authors: Yudi Pawitan, Chris Holmes
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15296
• Pdf URL: https://arxiv.org/pdf/2412.15296
• Copy Paste: [[2412.15296]] Confidence in the Reasoning of Large Language Models(https://arxiv.org/abs/2412.15296)
• Keywords: language model, gpt, llm, prompt
• Abstract: There is a growing literature on reasoning by large language models (LLMs), but the discussion on the uncertainty in their responses is still lacking. Our aim is to assess the extent of confidence that LLMs have in their answers and how it correlates with accuracy. Confidence is measured (i) qualitatively in terms of persistence in keeping their answer when prompted to reconsider, and (ii) quantitatively in terms of self-reported confidence score. We investigate the performance of three LLMs -- GPT4o, GPT4-turbo and Mistral -- on two benchmark sets of questions on causal judgement and formal fallacies and a set of probability and statistical puzzles and paradoxes. Although the LLMs show significantly better performance than random guessing, there is a wide variability in their tendency to change their initial answers. There is a positive correlation between qualitative confidence and accuracy, but the overall accuracy for the second answer is often worse than for the first answer. There is a strong tendency to overstate the self-reported confidence score. Confidence is only partially explained by the underlying token-level probability. The material effects of prompting on qualitative confidence and the strong tendency for overconfidence indicate that current LLMs do not have any internally coherent sense of confidence.
• 摘要：关于大型语言模型 (LLM) 推理的文献越来越多，但对其答案的不确定性的讨论仍然缺乏。我们的目标是评估 LLM 对其答案的信心程度以及它与准确性之间的关系。信心的衡量标准是 (i) 在被要求重新考虑时坚持答案的定性衡量标准，以及 (ii) 在自我报告的信心得分的定量衡量标准。我们调查了三门 LLM（GPT4o、GPT4-turbo 和 Mistral）在两组关于因果判断和形式谬误的基准问题以及一组概率和统计谜题和悖论中的表现。虽然 LLM 的表现明显优于随机猜测，但它们改变初始答案的倾向存在很大的差异。定性信心与准确性之间存在正相关关系，但第二个答案的总体准确性通常比第一个答案差。人们强烈倾向于夸大自我报告的信心得分。信心只能部分地由底层的 token 级概率来解释。提示对定性信心的实质性影响和强烈的过度自信倾向表明，当前的 LLM 没有任何内在一致的信心感。

Title: A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation

• Authors: Bhaskarjit Sarmah, Kriti Dutta, Anna Grigoryan, Sachin Tiwari, Stefano Pasquali, Dhagash Mehta
• Subjects: cs.CL, cs.AI, cs.LG, q-fin.ST, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.15298
• Pdf URL: https://arxiv.org/pdf/2412.15298
• Copy Paste: [[2412.15298]] A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation(https://arxiv.org/abs/2412.15298)
• Keywords: language model, llm, hallucination, prompt
• Abstract: We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments.
• 摘要：我们认为，声明式自改进 Python (DSPy) 优化器是一种将大型语言模型 (LLM) 提示及其评估与人工注释对齐的方法。我们对 DSPy 框架内的五种提词器算法进行了比较分析，即协作提示优化 (COPRO)、多阶段指令提示优化 (MIPRO)、BootstrapFewShot、使用 Optuna 的 BootstrapFewShot 和 K-Nearest Neighbor Few Shot，以了解它们与人工评估的一致性。作为一个具体的例子，我们专注于优化提示，以使幻觉检测（使用 LLM 作为判断标准）与公开可用的基准数据集的人工注释的地面实况标签对齐。我们的实验表明，优化的提示可以胜过各种检测幻觉的基准方法，并且至少在这些实验中，某些提词器的表现优于其他方法。

Title: LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliteration

• Authors: Sangmin Lee, Woo-Jin Chung Hong-Goo Kang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.15299
• Pdf URL: https://arxiv.org/pdf/2412.15299
• Copy Paste: [[2412.15299]] LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliteration(https://arxiv.org/abs/2412.15299)
• Keywords: language model
• Abstract: Building a universal multilingual automatic speech recognition (ASR) model that performs equitably across languages has long been a challenge due to its inherent difficulties. To address this task we introduce a Language-Agnostic Multilingual ASR pipeline through orthography Unification and language-specific Transliteration (LAMA-UT). LAMA-UT operates without any language-specific modules while matching the performance of state-of-the-art models trained on a minimal amount of data. Our pipeline consists of two key steps. First, we utilize a universal transcription generator to unify orthographic features into Romanized form and capture common phonetic characteristics across diverse languages. Second, we utilize a universal converter to transform these universal transcriptions into language-specific ones. In experiments, we demonstrate the effectiveness of our proposed method leveraging universal transcriptions for massively multilingual ASR. Our pipeline achieves a relative error reduction rate of 45% when compared to Whisper and performs comparably to MMS, despite being trained on only 0.1% of Whisper's training data. Furthermore, our pipeline does not rely on any language-specific modules. However, it performs on par with zero-shot ASR approaches which utilize additional language-specific lexicons and language models. We expect this framework to serve as a cornerstone for flexible multilingual ASR systems that are generalizable even to unseen languages.
• 摘要：长期以来，由于其固有的困难，构建一个在各种语言中表现均衡的通用多语言自动语音识别 (ASR) 模型一直是一项挑战。为了解决这一任务，我们通过正字法统一和特定语言的音译 (LAMA-UT) 引入了与语言无关的多语言 ASR 管道。LAMA-UT 无需任何特定语言的模块即可运行，同时可与使用最少数据训练的最先进模型的性能相匹配。我们的管道由两个关键步骤组成。首先，我们使用通用转录生成器将正字法特征统一为罗马化形式，并捕捉不同语言的共同语音特征。其次，我们使用通用转换器将这些通用转录转换为特定语言的转录。在实验中，我们证明了我们提出的利用通用转录进行大规模多语言 ASR 的方法的有效性。与 Whisper 相比，我们的管道实现了 45% 的相对错误减少率，并且性能与 MMS 相当，尽管只对 Whisper 的 0.1% 的训练数据进行了训练。此外，我们的管道不依赖任何特定语言的模块。然而，它的表现与使用额外语言特定词典和语言模型的零样本 ASR 方法相当。我们希望这个框架成为灵活的多语言 ASR 系统的基石，甚至可以推广到从未见过的语言。

Title: Self-Evolution Knowledge Distillation for LLM-based Machine Translation

• Authors: Yuncheng Song, Liang Ding, Changtong Zan, Shujian Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15303
• Pdf URL: https://arxiv.org/pdf/2412.15303
• Copy Paste: [[2412.15303]] Self-Evolution Knowledge Distillation for LLM-based Machine Translation(https://arxiv.org/abs/2412.15303)
• Keywords: language model, llm
• Abstract: Knowledge distillation (KD) has shown great promise in transferring knowledge from larger teacher models to smaller student models. However, existing KD strategies for large language models often minimize output distributions between student and teacher models indiscriminately for each token. This overlooks the imbalanced nature of tokens and their varying transfer difficulties. In response, we propose a distillation strategy called Self-Evolution KD. The core of this approach involves dynamically integrating teacher distribution and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process. It adjusts the ratio of prior knowledge based on token learning difficulty, fully leveraging the teacher model's potential. Experimental results show our method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets. Further analysis indicates that the improvement comes from better knowledge transfer from teachers, confirming our hypothesis.
• 摘要：知识蒸馏 (KD) 在将知识从较大的教师模型转移到较小的学生模型方面表现出了巨大的潜力。然而，现有的大型语言模型的 KD 策略通常会不加区分地最小化每个 token 的学生和教师模型之间的输出分布。这忽略了 token 的不平衡性质及其不同的转移难度。为此，我们提出了一种称为自进化 KD 的蒸馏策略。这种方法的核心是将教师分布和地面实况的独热分布动态地整合到学生分布中作为先验知识，从而促进蒸馏过程。它根据 token 的学习难度调整先验知识的比例，充分利用教师模型的潜力。实验结果表明，我们的方法在 WMT22 测试集的四个翻译方向上平均提高了约 1.4 个 SacreBLEU 点。进一步的分析表明，这种改进来自于教师更好的知识转移，证实了我们的假设。

Title: ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese

• Authors: Tran Thai Hoa, Tran Quang Duy, Khanh Quoc Tran, Kiet Van Nguyen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15308
• Pdf URL: https://arxiv.org/pdf/2412.15308
• Copy Paste: [[2412.15308]] ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese(https://arxiv.org/abs/2412.15308)
• Keywords: language model, prompt
• Abstract: The rapid spread of information in the digital age highlights the critical need for effective fact-checking tools, particularly for languages with limited resources, such as Vietnamese. In response to this challenge, we introduce ViFactCheck, the first publicly available benchmark dataset designed specifically for Vietnamese fact-checking across multiple online news domains. This dataset contains 7,232 human-annotated pairs of claim-evidence combinations sourced from reputable Vietnamese online news, covering 12 diverse topics. It has been subjected to a meticulous annotation process to ensure high quality and reliability, achieving a Fleiss Kappa inter-annotator agreement score of 0.83. Our evaluation leverages state-of-the-art pre-trained and large language models, employing fine-tuning and prompting techniques to assess performance. Notably, the Gemma model demonstrated superior effectiveness, with an impressive macro F1 score of 89.90%, thereby establishing a new standard for fact-checking benchmarks. This result highlights the robust capabilities of Gemma in accurately identifying and verifying facts in Vietnamese. To further promote advances in fact-checking technology and improve the reliability of digital media, we have made the ViFactCheck dataset, model checkpoints, fact-checking pipelines, and source code freely available on GitHub. This initiative aims to inspire further research and enhance the accuracy of information in low-resource languages.
• 摘要：数字时代信息的快速传播凸显了对有效事实核查工具的迫切需求，特别是对于资源有限的语言，例如越南语。为了应对这一挑战，我们推出了 ViFactCheck，这是第一个专门为跨多个在线新闻域的越南语事实核查而设计的公开基准数据集。该数据集包含 7,232 对人工注释的声明-证据组合，这些组合来自信誉良好的越南语在线新闻，涵盖 12 个不同的主题。它经过了细致的注释过程，以确保高质量和可靠性，实现了 0.83 的 Fleiss Kappa 注释者间一致性得分。我们的评估利用了最先进的预训练和大型语言模型，采用微调和提示技术来评估性能。值得注意的是，Gemma 模型表现出卓越的有效性，宏观 F1 得分高达 89.90%，从而为事实核查基准建立了新的标准。这一结果凸显了 Gemma 在准确识别和验证越南语事实方面的强大能力。为了进一步推动事实核查技术的进步并提高数字媒体的可靠性，我们在 GitHub 上免费提供了 ViFactCheck 数据集、模型检查点、事实核查流程和源代码。此举旨在激发进一步的研究并提高资源匮乏语言中信息的准确性。

Title: Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models

• Authors: Nishtha N. Vaidya, Thomas Runkler, Thomas Hubauer, Veronika Haderlein-Hoegberg, Maja Mlicic Brandt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15309
• Pdf URL: https://arxiv.org/pdf/2412.15309
• Copy Paste: [[2412.15309]] Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models(https://arxiv.org/abs/2412.15309)
• Keywords: language model, llm, hallucination, prompt, agent
• Abstract: Science and engineering problems fall in the category of complex conceptual problems that require specific conceptual information (CI) like math/logic -related know-how, process information, or engineering guidelines to solve them. Large Language Models (LLMs) are promising agents to solve such complex conceptual problems due to their implications in advancing engineering and science tasks like assisted problem-solving. But vanilla LLMs, trained on open-world data, lack the necessary CI. In this work, we specifically explore shallow customization methods (SCMs) of LLMs for solving complex conceptual problems. We propose two novel SCM algorithms for LLM, to augment LLMs with CI and enable LLMs to solve complex conceptual problems: Conceptual In-Context Learning (C-ICL) and Chain of Concepts (CoC). The problem tackled in this paper is generation of proprietary data models in the engineering/industry domain based on conceptual information in data modelling guidelines. We evaluate our algorithms on varied sizes of the OpenAI LLMs against four evaluation metrics related to syntactic and semantic correctness, time and cost incurred. The proposed algorithms perform better than currently popular LLM SCMs like In-context Learning (ICL) and Chain of Thoughts (CoT). It was observed that as compared to CoT, response correctness increased by 30.6% and 29.88% for the new SCMs C-ICL and CoC respectively. Qualitative analysis suggests that the proposed new SCMs activate emergent capabilities in LLMs, previously unobserved in the existing SCMs. They make problem-solving processes more transparent and reduce hallucinations and the tendency of model responses to copy examples from prompts (parroting).
• 摘要：科学和工程问题属于复杂概念问题，需要特定的概念信息 (CI)，如数学/逻辑相关的知识、流程信息或工程指南来解决。大型语言模型 (LLM) 是有希望解决此类复杂概念问题的代理，因为它们有助于推进工程和科学任务，如辅助解决问题。但是，在开放世界数据上训练的普通 LLM 缺乏必要的 CI。在这项工作中，我们专门探索了用于解决复杂概念问题的 LLM 的浅层定制方法 (SCM)。我们为 LLM 提出了两种新颖的 SCM 算法，以使用 CI 增强 LLM 并使 LLM 能够解决复杂的概念问题：概念上下文学习 (C-ICL) 和概念链 (CoC)。本文解决的问题是基于数据建模指南中的概念信息在工程/工业领域生成专有数据模型。我们根据与句法和语义正确性、时间和成本相关的四个评估指标，在不同大小的 OpenAI LLM 上评估我们的算法。所提出的算法比目前流行的 LLM SCM（如情境学习 (ICL) 和思维链 (CoT)）表现更好。观察发现，与 CoT 相比，新 SCM C-ICL 和 CoC 的响应正确率分别提高了 30.6% 和 29.88%。定性分析表明，所提出的新 SCM 激活了 LLM 中的新兴能力，而这些能力以前在现有 SCM 中是未观察到的。它们使问题解决过程更加透明，并减少了幻觉和模型响应从提示中复制示例（鹦鹉学舌）的倾向。

Title: Eliciting Causal Abilities in Large Language Models for Reasoning Tasks

• Authors: Yajing Wang, Zongwei Luo, Jingzhe Wang, Zhanke Zhou, Yongqiang Chen, Bo Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15314
• Pdf URL: https://arxiv.org/pdf/2412.15314
• Copy Paste: [[2412.15314]] Eliciting Causal Abilities in Large Language Models for Reasoning Tasks(https://arxiv.org/abs/2412.15314)
• Keywords: language model, llm, prompt
• Abstract: Prompt optimization automatically refines prompting expressions, unlocking the full potential of LLMs in downstream tasks. However, current prompt optimization methods are costly to train and lack sufficient interpretability. This paper proposes enhancing LLMs' reasoning performance by eliciting their causal inference ability from prompting instructions to correct answers. Specifically, we introduce the Self-Causal Instruction Enhancement (SCIE) method, which enables LLMs to generate high-quality, low-quantity observational data, then estimates the causal effect based on these data, and ultimately generates instructions with the optimized causal effect. In SCIE, the instructions are treated as the treatment, and textual features are used to process natural language, establishing causal relationships through treatments between instructions and downstream tasks. Additionally, we propose applying Object-Relational (OR) principles, where the uncovered causal relationships are treated as the inheritable class across task objects, ensuring low-cost reusability. Extensive experiments demonstrate that our method effectively generates instructions that enhance reasoning performance with reduced training cost of prompts, leveraging interpretable textual features to provide actionable insights.
• 摘要：提示优化会自动细化提示表达，释放 LLM 在下游任务中的全部潜力。然而，目前的提示优化方法训练成本高，且缺乏足够的可解释性。本文提出通过引出 LLM 从提示指令到正确答案的因果推理能力来增强其推理性能。具体来说，我们引入了自因果指令增强 (SCIE) 方法，使 LLM 能够生成高质量、低数量的观察数据，然后根据这些数据估计因果效应，最终生成具有优化因果效应的指令。在 SCIE 中，指令被视为处理，文本特征用于处理自然语言，通过指令和下游任务之间的处理建立因果关系。此外，我们建议应用对象关系 (OR) 原则，其中未发现的因果关系被视为跨任务对象的可继承类，确保低成本的可重用性。大量实验表明，我们的方法有效地生成指令，提高了推理性能，降低了提示的训练成本，利用可解释的文本特征提供了可操作的见解。

Title: Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation

• Authors: Joanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15375
• Pdf URL: https://arxiv.org/pdf/2412.15375
• Copy Paste: [[2412.15375]] Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation(https://arxiv.org/abs/2412.15375)
• Keywords: language model, llm
• Abstract: Extracting metaphors and analogies from free text requires high-level reasoning abilities such as abstraction and language understanding. Our study focuses on the extraction of the concepts that form metaphoric analogies in literary texts. To this end, we construct a novel dataset in this domain with the help of domain experts. We compare the out-of-the-box ability of recent large language models (LLMs) to structure metaphoric mappings from fragments of texts containing proportional analogies. The models are further evaluated on the generation of implicit elements of the analogy, which are indirectly suggested in the texts and inferred by human readers. The competitive results obtained by LLMs in our experiments are encouraging and open up new avenues such as automatically extracting analogies and metaphors from text instead of investing resources in domain experts to manually label data.
• 摘要：从自由文本中提取隐喻和类比需要高级推理能力，例如抽象和语言理解。我们的研究重点是提取文学文本中形成隐喻类比的概念。为此，我们在领域专家的帮助下构建了一个该领域的新数据集。我们比较了最近的大型语言模型 (LLM) 从包含比例类比的文本片段构建隐喻映射的开箱即用能力。进一步评估了这些模型在类比的隐含元素生成方面的表现，这些元素在文本中间接暗示并由人类读者推断出来。在我们的实验中，LLM 获得的竞争结果令人鼓舞，并开辟了新的途径，例如自动从文本中提取类比和隐喻，而不是投入资源让领域专家手动标记数据。

Title: Systematic Evaluation of Long-Context LLMs on Financial Concepts

• Authors: Lavanya Gupta, Saket Sharma, Yiyun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15386
• Pdf URL: https://arxiv.org/pdf/2412.15386
• Copy Paste: [[2412.15386]] Systematic Evaluation of Long-Context LLMs on Financial Concepts(https://arxiv.org/abs/2412.15386)
• Keywords: language model, gpt, llm, prompt
• Abstract: Long-context large language models (LC LLMs) promise to increase reliability of LLMs in real-world tasks requiring processing and understanding of long input documents. However, this ability of LC LLMs to reliably utilize their growing context windows remains under investigation. In this work, we evaluate the performance of state-of-the-art GPT-4 suite of LC LLMs in solving a series of progressively challenging tasks, as a function of factors such as context length, task difficulty, and position of key information by creating a real world financial news dataset. Our findings indicate that LC LLMs exhibit brittleness at longer context lengths even for simple tasks, with performance deteriorating sharply as task complexity increases. At longer context lengths, these state-of-the-art models experience catastrophic failures in instruction following resulting in degenerate outputs. Our prompt ablations also reveal unfortunate continued sensitivity to both the placement of the task instruction in the context window as well as minor markdown formatting. Finally, we advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall) and reporting confidence intervals, thereby ensuring robust and conclusive findings.
• 摘要：长上下文大型语言模型 (LC LLM) 有望提高 LLM 在需要处理和理解长输入文档的实际任务中的可靠性。然而，LC LLM 可靠地利用其不断增长的上下文窗口的能力仍在研究中。在这项工作中，我们通过创建现实世界的金融新闻数据集，评估了最先进的 GPT-4 套件 LC LLM 在解决一系列逐渐具有挑战性的任务时的性能，这些性能取决于上下文长度、任务难度和关键信息的位置等因素。我们的研究结果表明，即使对于简单的任务，LC LLM 在较长的上下文长度下也会表现出脆弱性，并且随着任务复杂性的增加，性能会急剧下降。在较长的上下文长度下，这些最先进的模型在指令跟踪中会经历灾难性的失败，导致输出退化。我们的快速消融还揭示了对任务指令在上下文窗口中的位置以及次要 markdown 格式的持续敏感性。最后，我们主张通过采用 F1（而非召回率）等整体指标并报告置信区间对 LC LLM 进行更严格的评估，从而确保获得稳健且确凿的发现。

Title: SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval

• Authors: Aakash Mahalingam, Vinesh Kumar Gande, Aman Chadha, Vinija Jain, Divya Chaudhary
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15443
• Pdf URL: https://arxiv.org/pdf/2412.15443
• Copy Paste: [[2412.15443]] SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval(https://arxiv.org/abs/2412.15443)
• Keywords: language model, hallucination, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems have become pivotal in leveraging vast corpora to generate informed and contextually relevant responses, notably reducing hallucinations in Large Language Models. Despite significant advancements, these systems struggle to efficiently process and retrieve information from large datasets while maintaining a comprehensive understanding of the context. This paper introduces SKETCH, a novel methodology that enhances the RAG retrieval process by integrating semantic text retrieval with knowledge graphs, thereby merging structured and unstructured data for a more holistic comprehension. SKETCH, demonstrates substantial improvements in retrieval performance and maintains superior context integrity compared to traditional methods. Evaluated across four diverse datasets: QuALITY, QASPER, NarrativeQA, and Italian Cuisine-SKETCH consistently outperforms baseline approaches on key RAGAS metrics such as answer_relevancy, faithfulness, context_precision and context_recall. Notably, on the Italian Cuisine dataset, SKETCH achieved an answer relevancy of 0.94 and a context precision of 0.99, representing the highest performance across all evaluated metrics. These results highlight SKETCH's capability in delivering more accurate and contextually relevant responses, setting new benchmarks for future retrieval systems.
• 摘要：检索增强生成 (RAG) 系统已成为利用大量语料库生成知情且与上下文相关的响应的关键，尤其是减少大型语言模型中的幻觉。尽管取得了重大进展，但这些系统仍难以在保持对上下文的全面理解的同时有效地处理和检索大型数据集中的信息。本文介绍了 SKETCH，这是一种新颖的方法，它通过将语义文本检索与知识图谱相结合来增强 RAG 检索过程，从而合并结构化和非结构化数据以实现更全面的理解。与传统方法相比，SKETCH 在检索性能方面表现出显着的改进，并保持了出色的上下文完整性。在四个不同的数据集上进行评估：QuALITY、QASPER、NarrativeQA 和 Italian Cuisine-SKETCH 在关键 RAGAS 指标（例如答案相关性、忠实度、上下文精度和上下文召回率）上始终优于基线方法。值得注意的是，在意大利美食数据集上，SKETCH 的答案相关性达到 0.94，上下文精度达到 0.99，在所有评估指标中均表现最佳。这些结果凸显了 SKETCH 能够提供更准确、更符合上下文的答案，为未来的检索系统树立了新的标杆。

Title: Fietje: An open, efficient LLM for Dutch

• Authors: Bram Vanroy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15450
• Pdf URL: https://arxiv.org/pdf/2412.15450
• Copy Paste: [[2412.15450]] Fietje: An open, efficient LLM for Dutch(https://arxiv.org/abs/2412.15450)
• Keywords: language model, llm
• Abstract: This paper introduces Fietje, a family of small language models (SLMs) specifically designed for the Dutch language. The model is based on Phi 2, an English-centric model of 2.7 billion parameters. Fietje demonstrated competitive results with larger language models upon its release. A core emphasis of this work is transparency and reproducibility: Fietje is fully open-source, with model weights, datasets, training, and evaluation code all publicly accessible. The paper discusses the performance of Fietje and many other models on an extensive evaluation suite of benchmarks on reasoning, sentiment analysis, world knowledge, linguistic acceptability and word sense disambiguation. Evaluation results illustrate the rapid progress in the field of LLMs, where recent small models outperform older, larger models that were fine-tuned for Dutch. This trend signals an exciting future for Dutch language processing, suggesting that even compact LLMs are becoming increasingly capable. Furthermore, ongoing and future efforts to adapt LLMs to Dutch are poised to enhance these models even further, broadening their applicability and accessibility. Fietje is only an intermediate step in improving accessibility to language technology for users of the Dutch language.
• 摘要：本文介绍了专为荷兰语设计的一系列小型语言模型 (SLM) Fietje。该模型基于 Phi 2，这是一个以英语为中心的模型，有 27 亿个参数。Fietje 在发布时就表现出与大型语言模型相媲美的结果。这项工作的核心重点是透明度和可重复性：Fietje 是完全开源的，模型权重、数据集、训练和评估代码都是公开的。本文讨论了 Fietje 和许多其他模型在推理、情感分析、世界知识、语言可接受性和词义消歧等一系列基准测试中的表现。评估结果表明 LLM 领域的进展很快，最近的小型模型优于针对荷兰语进行微调的旧大型模型。这一趋势预示着荷兰语处理的未来令人振奋，表明即使是紧凑型 LLM 也变得越来越强大。此外，正在进行的和未来将 LLM 改编为荷兰语的努力将进一步增强这些模型，扩大其适用性和可访问性。 Fietje 只是提高荷兰语用户使用语言技术可及性的一个中间步骤。

Title: Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization

• Authors: Sahil Wadhwa, Chengtian Xu, Haoming Chen, Aakash Mahalingam, Akankshya Kar, Divya Chaudhary
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15453
• Pdf URL: https://arxiv.org/pdf/2412.15453
• Copy Paste: [[2412.15453]] Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization(https://arxiv.org/abs/2412.15453)
• Keywords: language model, llm
• Abstract: The automatic generation of counter-speech (CS) is a critical strategy for addressing hate speech by providing constructive and informed responses. However, existing methods often fail to generate high-quality, impactful, and scalable CS, particularly across diverse linguistic contexts. In this paper, we propose a novel methodology to enhance CS generation by aligning Large Language Models (LLMs) using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Our approach leverages DPO to align LLM outputs with human preferences, ensuring contextually appropriate and linguistically adaptable responses. Additionally, we incorporate knowledge grounding to enhance the factual accuracy and relevance of generated CS. Experimental results demonstrate that DPO-aligned models significantly outperform SFT baselines on CS benchmarks while scaling effectively to multiple languages. These findings highlight the potential of preference-based alignment techniques to advance CS generation across varied linguistic settings. The model supervision and alignment is done in English and the same model is used for reporting metrics across other languages like Basque, Italian, and Spanish.
• 摘要：自动生成反言论 (CS) 是解决仇恨言论的关键策略，因为它可以提供建设性和明智的回应。然而，现有的方法往往无法生成高质量、有影响力且可扩展的 CS，尤其是在不同的语言环境中。在本文中，我们提出了一种新方法，通过使用监督微调 (SFT) 和直接偏好优化 (DPO) 对齐大型语言模型 (LLM) 来增强 CS 生成。我们的方法利用 DPO 将 LLM 输出与人类偏好对齐，确保响应符合语境且语言适应性强。此外，我们结合知识基础来提高生成的 CS 的事实准确性和相关性。实验结果表明，DPO 对齐模型在 CS 基准上的表现明显优于 SFT 基线，同时可以有效扩展到多种语言。这些发现凸显了基于偏好的对齐技术在各种语言环境中推进 CS 生成的潜力。模型监督和对齐以英语进行，同一模型用于报告其他语言（如巴斯克语、意大利语和西班牙语）的指标。

Title: Continual Learning Using Only Large Language Model Prompting

• Authors: Jiabao Qiu, Zixuan Ke, Bing Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15479
• Pdf URL: https://arxiv.org/pdf/2412.15479
• Copy Paste: [[2412.15479]] Continual Learning Using Only Large Language Model Prompting(https://arxiv.org/abs/2412.15479)
• Keywords: language model, llm, prompt
• Abstract: We introduce CLOB, a novel continual learning (CL) paradigm wherein a large language model (LLM) is regarded as a black box. Learning is done incrementally via only verbal prompting. CLOB does not fine-tune any part of the LLM or add any trainable parameters to it. It is particularly suitable for LLMs that are accessible via APIs. We also propose a new CL technique, called CIS, based on incremental summarization that also overcomes the LLM's input length limit. Experiments show CIS outperforms baselines by a very large margin.
• 摘要：我们引入了 CLOB，一种新颖的持续学习 (CL) 范式，其中大型语言模型 (LLM) 被视为黑匣子。学习仅通过口头提示逐步完成。CLOB 不会微调 LLM 的任何部分或向其添加任何可训练参数。它特别适用于可通过 API 访问的 LLM。我们还提出了一种新的 CL 技术，称为 CIS，它基于增量摘要，也克服了 LLM 的输入长度限制。实验表明 CIS 的表现远远优于基线。

Title: Multi-LLM Text Summarization

• Authors: Jiangnan Fang, Cheng-Tse Liu, Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K. Ahmed, Franck Dernoncourt, Ryan A. Rossi, Hanieh Deilamsalehy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15487
• Pdf URL: https://arxiv.org/pdf/2412.15487
• Copy Paste: [[2412.15487]] Multi-LLM Text Summarization(https://arxiv.org/abs/2412.15487)
• Keywords: llm
• Abstract: In this work, we propose a Multi-LLM summarization framework, and investigate two different multi-LLM strategies including centralized and decentralized. Our multi-LLM summarization framework has two fundamentally important steps at each round of conversation: generation and evaluation. These steps are different depending on whether our multi-LLM decentralized summarization is used or centralized. In both our multi-LLM decentralized and centralized strategies, we have k different LLMs that generate diverse summaries of the text. However, during evaluation, our multi-LLM centralized summarization approach leverages a single LLM to evaluate the summaries and select the best one whereas k LLMs are used for decentralized multi-LLM summarization. Overall, we find that our multi-LLM summarization approaches significantly outperform the baselines that leverage only a single LLM by up to 3x. These results indicate the effectiveness of multi-LLM approaches for summarization.
• 摘要：在本研究中，我们提出了一个多 LLM 摘要框架，并研究了两种不同的多 LLM 策略，包括集中式和分散式。我们的多 LLM 摘要框架在每一轮对话中都有两个基本重要的步骤：生成和评估。这些步骤因我们使用多 LLM 分散式摘要还是集中式摘要而不同。在我们的多 LLM 分散式和集中式策略中，我们都有 k 个不同的 LLM 来生成文本的不同摘要。然而，在评估过程中，我们的多 LLM 集中式摘要方法利用单个 LLM 来评估摘要并选择最佳摘要，而 k 个 LLM 用于分散式多 LLM 摘要。总体而言，我们发现我们的多 LLM 摘要方法比仅利用单个 LLM 的基线方法的性能高出 3 倍。这些结果表明多 LLM 方法用于摘要的有效性。

Title: TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use

• Authors: Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, Zhengyin Du
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15495
• Pdf URL: https://arxiv.org/pdf/2412.15495
• Copy Paste: [[2412.15495]] TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use(https://arxiv.org/abs/2412.15495)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with external environments, a critical step toward generalized AI. However, the standard supervised fine-tuning (SFT) approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use, leading to performance bottlenecks. To address this issue, we analyze three existing LLMs and uncover key insights: training data can inadvertently impede tool-use behavior, token importance is distributed unevenly, and errors in tool calls fall into a small set of distinct categories. Building on these findings, we propose TL-Training, a task-feature-based framework that mitigates the effects of suboptimal training data, dynamically adjusts token weights to prioritize key tokens during SFT, and incorporates a robust reward mechanism tailored to error categories, optimized through proximal policy optimization. We validate TL-Training by training CodeLLaMA-2-7B and evaluating it on four diverse open-source test sets. Our results demonstrate that the LLM trained by our method matches or surpasses both open- and closed-source LLMs in tool-use performance using only 1,217 training data points. Additionally, our method enhances robustness in noisy environments and improves general task performance, offering a scalable and efficient paradigm for tool-use training in LLMs. The code and data are available at this https URL.
• 摘要：大型语言模型 (LLM) 通过利用工具与外部环境交互取得了显著的进步，这是迈向通用 AI 的关键一步。然而，依赖于大规模数据集的标准监督微调 (SFT) 方法通常会忽略工具使用中的任务特定特征，从而导致性能瓶颈。为了解决这个问题，我们分析了三个现有的 LLM 并发现了关键见解：训练数据可能会无意中阻碍工具使用行为，token 重要性分布不均，工具调用中的错误分为一小组不同的类别。基于这些发现，我们提出了 TL-Training，这是一个基于任务特征的框架，可减轻次优训练数据的影响，动态调整 token 权重以在 SFT 期间优先考虑关键 token，并结合针对错误类别量身定制的强大奖励机制，通过近端策略优化进行优化。我们通过训练 CodeLLaMA-2-7B 并在四个不同的开源测试集上对其进行评估来验证 TL-Training。我们的结果表明，仅使用 1,217 个训练数据点，通过我们的方法训练的 LLM 在工具使用性能方面与开源和闭源 LLM 相当甚至超过后者。此外，我们的方法增强了在嘈杂环境中的鲁棒性并提高了一般任务性能，为 LLM 中的工具使用训练提供了可扩展且高效的范例。代码和数据可在此 https URL 上获取。

Title: Lexicography Saves Lives (LSL): Automatically Translating Suicide-Related Language

• Authors: Annika Marie Schoene, John E. Ortega, Rodolfo Joel Zevallos, Laura Haaber Ihle
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15497
• Pdf URL: https://arxiv.org/pdf/2412.15497
• Copy Paste: [[2412.15497]] Lexicography Saves Lives (LSL): Automatically Translating Suicide-Related Language(https://arxiv.org/abs/2412.15497)
• Keywords: language model
• Abstract: Recent years have seen a marked increase in research that aims to identify or predict risk, intention or ideation of suicide. The majority of new tasks, datasets, language models and other resources focus on English and on suicide in the context of Western culture. However, suicide is global issue and reducing suicide rate by 2030 is one of the key goals of the UN's Sustainable Development Goals. Previous work has used English dictionaries related to suicide to translate into different target languages due to lack of other available resources. Naturally, this leads to a variety of ethical tensions (e.g.: linguistic misrepresentation), where discourse around suicide is not present in a particular culture or country. In this work, we introduce the 'Lexicography Saves Lives Project' to address this issue and make three distinct contributions. First, we outline ethical consideration and provide overview guidelines to mitigate harm in developing suicide-related resources. Next, we translate an existing dictionary related to suicidal ideation into 200 different languages and conduct human evaluations on a subset of translated dictionaries. Finally, we introduce a public website to make our resources available and enable community participation.
• 摘要：近年来，旨在识别或预测自杀风险、自杀意图或自杀意念的研究显著增加。大多数新任务、数据集、语言模型和其他资源都集中在英语和西方文化背景下的自杀问题上。然而，自杀是一个全球性问题，到 2030 年降低自杀率是联合国可持续发展目标的主要目标之一。由于缺乏其他可用资源，以前的研究使用与自杀相关的英语词典将其翻译成不同的目标语言。自然，这会导致各种道德紧张（例如：语言误传），因为在特定的文化或国家中不存在关于自杀的讨论。在这项工作中，我们引入了“词典学拯救生命项目”来解决这个问题，并做出了三项独特的贡献。首先，我们概述了道德考虑并提供概述指南，以减轻开发自杀相关资源的危害。接下来，我们将一本现有的与自杀意念相关的词典翻译成 200 种不同的语言，并对翻译词典的子集进行人工评估。最后，我们推出了一个公共网站，以便提供我们的资源并让社区参与。

Title: Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models

• Authors: Zhisheng Tang, Mayank Kejriwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15501
• Pdf URL: https://arxiv.org/pdf/2412.15501
• Copy Paste: [[2412.15501]] Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models(https://arxiv.org/abs/2412.15501)
• Keywords: language model, gpt, llm
• Abstract: Research on emergent patterns in Large Language Models (LLMs) has gained significant traction in both psychology and artificial intelligence, motivating the need for a comprehensive review that offers a synthesis of this complex landscape. In this article, we systematically review LLMs' capabilities across three important cognitive domains: decision-making biases, reasoning, and creativity. We use empirical studies drawing on established psychological tests and compare LLMs' performance to human benchmarks. On decision-making, our synthesis reveals that while LLMs demonstrate several human-like biases, some biases observed in humans are absent, indicating cognitive patterns that only partially align with human decision-making. On reasoning, advanced LLMs like GPT-4 exhibit deliberative reasoning akin to human System-2 thinking, while smaller models fall short of human-level performance. A distinct dichotomy emerges in creativity: while LLMs excel in language-based creative tasks, such as storytelling, they struggle with divergent thinking tasks that require real-world context. Nonetheless, studies suggest that LLMs hold considerable potential as collaborators, augmenting creativity in human-machine problem-solving settings. Discussing key limitations, we also offer guidance for future research in areas such as memory, attention, and open-source model development.
• 摘要：在心理学和人工智能领域，大型语言模型 (LLM) 中出现的模式研究都获得了巨大的关注，这促使我们需要进行全面的审查，以综合这一复杂领域。在本文中，我们系统地审查了 LLM 在三个重要认知领域的能力：决策偏见、推理和创造力。我们使用基于既定心理测试的实证研究，并将 LLM 的表现与人类基准进行比较。在决策方面，我们的综合结果表明，虽然 LLM 表现出几种类似人类的偏见，但人类身上观察到的一些偏见并不存在，这表明认知模式仅部分与人类决策一致。在推理方面，像 GPT-4 这样的高级 LLM 表现出类似于人类系统 2 思维的深思熟虑推理，而较小的模型则达不到人类水平的表现。创造力方面出现了明显的二分法：虽然 LLM 在基于语言的创造性任务（例如讲故事）方面表现出色，但它们在需要现实世界背景的发散思维任务方面却举步维艰。尽管如此，研究表明，法学硕士作为合作者具有相当大的潜力，可以增强人机解决问题环境中的创造力。在讨论关键限制的同时，我们还为记忆、注意力和开源模型开发等领域的未来研究提供了指导。

Title: Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework

• Authors: Zhenjie Xu (1), Wenqing Chen (1), Yi Tang (1), Xuanying Li (2), Cheng Hu (1), Zhixuan Chu (3), Kui Ren (3), Zibin Zheng (1), Zhichao Lu (4) ((1) School of Software Engineering, Sun Yat-sen University, (2) School of Physics and Astronomy, Sun Yat-sen University, (3) School of Cyber Science and Technology, Zhejiang University, (4) Department of Computer Science, City University of Hong Kong)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15504
• Pdf URL: https://arxiv.org/pdf/2412.15504
• Copy Paste: [[2412.15504]] Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework(https://arxiv.org/abs/2412.15504)
• Keywords: language model, llm, prompt, agent
• Abstract: Natural language processing (NLP) has seen remarkable advancements with the development of large language models (LLMs). Despite these advancements, LLMs often produce socially biased outputs. Recent studies have mainly addressed this problem by prompting LLMs to behave ethically, but this approach results in unacceptable performance degradation. In this paper, we propose a multi-objective approach within a multi-agent framework (MOMA) to mitigate social bias in LLMs without significantly compromising their performance. The key idea of MOMA involves deploying multiple agents to perform causal interventions on bias-related contents of the input questions, breaking the shortcut connection between these contents and the corresponding answers. Unlike traditional debiasing techniques leading to performance degradation, MOMA substantially reduces bias while maintaining accuracy in downstream tasks. Our experiments conducted on two datasets and two models demonstrate that MOMA reduces bias scores by up to 87.7%, with only a marginal performance degradation of up to 6.8% in the BBQ dataset. Additionally, it significantly enhances the multi-objective metric icat in the StereoSet dataset by up to 58.1%. Code will be made available at this https URL.
• 摘要：随着大型语言模型 (LLM) 的发展，自然语言处理 (NLP) 取得了显著进步。尽管取得了这些进步，但 LLM 经常会产生带有社会偏见的输出。最近的研究主要通过促使 LLM 行为合乎道德地解决这个问题，但这种方法会导致不可接受的性能下降。在本文中，我们提出了一种多智能体框架 (MOMA) 内的多目标方法，以减轻 LLM 中的社会偏见，而不会显著损害其性能。MOMA 的关键思想是部署多个智能体对输入问题的偏见相关内容进行因果干预，打破这些内容与相应答案之间的快捷连接。与导致性能下降的传统去偏见技术不同，MOMA 可以显着降低偏见，同时保持下游任务的准确性。我们在两个数据集和两个模型上进行的实验表明，MOMA 将偏见分数降低了高达 87.7%，而在 BBQ 数据集中，性能仅略有下降，最高为 6.8%。此外，它还显著提高了 StereoSet 数据集中的多目标指标 icat，最高可达 58.1%。代码将在此 https URL 上提供。

Title: ADEQA: A Question Answer based approach for joint ADE-Suspect Extraction using Sequence-To-Sequence Transformers

• Authors: Vinayak Arannil, Tomal Deb, Atanu Roy
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15510
• Pdf URL: https://arxiv.org/pdf/2412.15510
• Copy Paste: [[2412.15510]] ADEQA: A Question Answer based approach for joint ADE-Suspect Extraction using Sequence-To-Sequence Transformers(https://arxiv.org/abs/2412.15510)
• Keywords: prompt
• Abstract: Early identification of Adverse Drug Events (ADE) is critical for taking prompt actions while introducing new drugs into the market. These ADEs information are available through various unstructured data sources like clinical study reports, patient health records, social media posts, etc. Extracting ADEs and the related suspect drugs using machine learning is a challenging task due to the complex linguistic relations between drug ADE pairs in textual data and unavailability of large corpus of labelled datasets. This paper introduces ADEQA, a question-answer(QA) based approach using quasi supervised labelled data and sequence-to-sequence transformers to extract ADEs, drug suspects and the relationships between them. Unlike traditional QA models, natural language generation (NLG) based models don't require extensive token level labelling and thereby reduces the adoption barrier significantly. On a public ADE corpus, we were able to achieve state-of-the-art results with an F1 score of 94% on establishing the relationships between ADEs and the respective suspects.
• 摘要：在将新药推向市场时，及早识别药品不良事件 (ADE) 对于采取及时行动至关重要。这些 ADE 信息可通过各种非结构化数据源获得，例如临床研究报告、患者健康记录、社交媒体帖子等。使用机器学习提取 ADE 和相关可疑药物是一项具有挑战性的任务，因为文本数据中的药物 ADE 对之间存在复杂的语言关系，并且没有大量标记数据集。本文介绍了 ADEQA，这是一种基于问答 (QA) 的方法，使用准监督标记数据和序列到序列转换器来提取 ADE、药物嫌疑人及其之间的关系。与传统的 QA 模型不同，基于自然语言生成 (NLG) 的模型不需要大量的 token 级别标记，从而大大降低了采用障碍。在公共 ADE 语料库上，我们能够在建立 ADE 与相应嫌疑人之间的关系方面取得最先进的结果，F1 得分为 94%。

Title: HREF: Human Response-Guided Evaluation of Instruction Following in Language Models

• Authors: Xinxi Lyu, Yizhong Wang, Hannaneh Hajishirzi, Pradeep Dasigi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15524
• Pdf URL: https://arxiv.org/pdf/2412.15524
• Copy Paste: [[2412.15524]] HREF: Human Response-Guided Evaluation of Instruction Following in Language Models(https://arxiv.org/abs/2412.15524)
• Keywords: language model, llm, prompt
• Abstract: Evaluating the capability of Large Language Models (LLMs) in following instructions has heavily relied on a powerful LLM as the judge, introducing unresolved biases that deviate the judgments from human judges. In this work, we reevaluate various choices for automatic evaluation on a wide range of instruction-following tasks. We experiment with methods that leverage human-written responses and observe that they enhance the reliability of automatic evaluations across a wide range of tasks, resulting in up to a 3.2% improvement in agreement with human judges. We also discovered that human-written responses offer an orthogonal perspective to model-generated responses in following instructions and should be used as an additional context when comparing model responses. Based on these observations, we develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF), comprising 4,258 samples across 11 task categories with a composite evaluation setup, employing a composite evaluation setup that selects the most reliable method for each category. In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination. Finally, we study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template. We host a live leaderboard that evaluates LLMs on the private evaluation set of HREF.
• 摘要：评估大型语言模型 (LLM) 遵循指令的能力在很大程度上依赖于强大的 LLM 作为评判者，这引入了未解决的偏见，导致判断与人类评判者不同。在这项工作中，我们重新评估了在广泛的指令遵循任务中自动评估的各种选择。我们尝试了利用人工书写响应的方法，并观察到它们提高了各种任务中自动评估的可靠性，从而使与人类评判者的一致性提高了 3.2%。我们还发现，人工书写的响应在遵循指令方面为模型生成的响应提供了正交视角，在比较模型响应时应将其用作附加上下文。基于这些观察，我们开发了一个新的评估基准，即人类响应引导的指令遵循评估 (HREF)，它包含 11 个任务类别的 4,258 个样本，采用复合评估设置，采用复合评估设置为每个类别选择最可靠的方法。除了提供可靠的评估外，HREF 还强调个人任务表现并且不受污染。最后，我们研究了 HREF 中关键设计选择的影响，包括评估集的大小、判断模型、基线模型和提示模板。我们举办了一个实时排行榜，对 HREF 私有评估集上的 LLM 进行评估。

Title: XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation

• Authors: Qianren Mao, Yangyifei Luo, Jinlong Zhang, Hanwen Hao, Zhilong Cao, Xiaolong Wang, Xiao Guan, Zhenting Huang, Weifeng Jiang, Shuyu Guo, Zhentao Han, Qili Zhang, Siyuan Tao, Yujie Liu, Junnan Liu, Zhixing Tan, Jie Sun, Bo Li, Xudong Liu, Richong Zhang, Jianxin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15529
• Pdf URL: https://arxiv.org/pdf/2412.15529
• Copy Paste: [[2412.15529]] XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation(https://arxiv.org/abs/2412.15529)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) synergizes the retrieval of pertinent data with the generative capabilities of Large Language Models (LLMs), ensuring that the generated output is not only contextually relevant but also accurate and this http URL introduce XRAG, an open-source, modular codebase that facilitates exhaustive evaluation of the performance of foundational components of advanced RAG modules. These components are systematically categorized into four core phases: pre-retrieval, retrieval, post-retrieval, and generation. We systematically analyse them across reconfigured datasets, providing a comprehensive benchmark for their effectiveness. Given the escalating complexity of RAG systems, we underscore the necessity of identifying potential failure points of RAG modules. We formulate a suite of experimental methodologies and diagnostic testing protocols to dissect the failure points inherent in the engineering of RAG modules. Subsequently, we proffer bespoke solutions that are designed to augment the validation processes and bolster the overall performance of these modules. Our work thoroughly evaluates the performance of core advanced components in RAG systems, providing insights into optimizations for prevalent failure points.
• 摘要：检索增强生成 (RAG) 将相关数据的检索与大型语言模型 (LLM) 的生成功能相结合，确保生成的输出不仅与上下文相关，而且准确，此 http URL 介绍了 XRAG，这是一个开源的模块化代码库，有助于对高级 RAG 模块基础组件的性能进行详尽评估。这些组件系统地分为四个核心阶段：预检索、检索、后检索和生成。我们在重新配置的数据集中对它们进行系统分析，为它们的有效性提供全面的基准。鉴于 RAG 系统的复杂性不断增加，我们强调识别 RAG 模块潜在故障点的必要性。我们制定了一套实验方法和诊断测试协议来剖析 RAG 模块工程中固有的故障点。随后，我们提供定制解决方案，旨在增强验证过程并增强这些模块的整体性能。我们的工作全面评估了 RAG 系统中核心先进组件的性能，为常见故障点的优化提供了见解。

Title: MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering

• Authors: Zhang Siyue, Xue Yuxiang, Zhang Yiming, Wu Xiaobao, Luu Anh Tuan, Zhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15540
• Pdf URL: https://arxiv.org/pdf/2412.15540
• Copy Paste: [[2412.15540]] MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering(https://arxiv.org/abs/2412.15540)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Understanding temporal relations and answering time-sensitive questions is crucial yet a challenging task for question-answering systems powered by large language models (LLMs). Existing approaches either update the parametric knowledge of LLMs with new facts, which is resource-intensive and often impractical, or integrate LLMs with external knowledge retrieval (i.e., retrieval-augmented generation). However, off-the-shelf retrievers often struggle to identify relevant documents that require intensive temporal reasoning. To systematically study time-sensitive question answering, we introduce the TempRAGEval benchmark, which repurposes existing datasets by incorporating temporal perturbations and gold evidence labels. As anticipated, all existing retrieval methods struggle with these temporal reasoning-intensive questions. We further propose Modular Retrieval (MRAG), a trainless framework that includes three modules: (1) Question Processing that decomposes question into a main content and a temporal constraint; (2) Retrieval and Summarization that retrieves evidence and uses LLMs to summarize according to the main content; (3) Semantic-Temporal Hybrid Ranking that scores each evidence summarization based on both semantic and temporal relevance. On TempRAGEval, MRAG significantly outperforms baseline retrievers in retrieval performance, leading to further improvements in final answer accuracy.
• 摘要：对于由大型语言模型 (LLM) 驱动的问答系统来说，理解时间关系和回答时间敏感的问题至关重要，但也是极具挑战性的任务。现有的方法要么用新事实更新 LLM 的参数知识，但这需要大量资源并且通常不切实际，要么将 LLM 与外部知识检索相结合（即检索增强生成）。然而，现成的检索器通常难以识别需要密集时间推理的相关文档。为了系统地研究时间敏感的问答，我们引入了 TempRAGEval 基准，它通过结合时间扰动和黄金证据标签来重新利用现有数据集。正如预期的那样，所有现有的检索方法都在努力解决这些需要大量时间推理的问题。我们进一步提出了模块化检索 (MRAG)，这是一个无需训练的框架，包括三个模块：（1）问题处理，将问题分解为主要内容和时间约束；（2）检索和总结，检索证据并使用 LLM 根据主要内容进行总结； （3）语义-时间混合排序，根据语义和时间相关性对每个证据摘要进行评分。在 TempRAGEval 上，MRAG 的检索性能明显优于基线检索器，从而进一步提高最终答案的准确性。

Title: NGQA: A Nutritional Graph Question Answering Benchmark for Personalized Health-aware Nutritional Reasoning

• Authors: Zheyuan Zhang, Yiyang Li, Nhi Ha Lan Le, Zehong Wang, Tianyi Ma, Vincent Galassi, Keerthiram Murugesan, Nuno Moniz, Werner Geyer, Nitesh V Chawla, Chuxu Zhang, Yanfang Ye
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15547
• Pdf URL: https://arxiv.org/pdf/2412.15547
• Copy Paste: [[2412.15547]] NGQA: A Nutritional Graph Question Answering Benchmark for Personalized Health-aware Nutritional Reasoning(https://arxiv.org/abs/2412.15547)
• Keywords: language model, llm
• Abstract: Diet plays a critical role in human health, yet tailoring dietary reasoning to individual health conditions remains a major challenge. Nutrition Question Answering (QA) has emerged as a popular method for addressing this problem. However, current research faces two critical limitations. On one hand, the absence of datasets involving user-specific medical information severely limits \textit{personalization}. This challenge is further compounded by the wide variability in individual health needs. On the other hand, while large language models (LLMs), a popular solution for this task, demonstrate strong reasoning abilities, they struggle with the domain-specific complexities of personalized healthy dietary reasoning, and existing benchmarks fail to capture these challenges. To address these gaps, we introduce the Nutritional Graph Question Answering (NGQA) benchmark, the first graph question answering dataset designed for personalized nutritional health reasoning. NGQA leverages data from the National Health and Nutrition Examination Survey (NHANES) and the Food and Nutrient Database for Dietary Studies (FNDDS) to evaluate whether a food is healthy for a specific user, supported by explanations of the key contributing nutrients. The benchmark incorporates three question complexity settings and evaluates reasoning across three downstream tasks. Extensive experiments with LLM backbones and baseline models demonstrate that the NGQA benchmark effectively challenges existing models. In sum, NGQA addresses a critical real-world problem while advancing GraphQA research with a novel domain-specific benchmark.
• 摘要：饮食在人类健康中起着至关重要的作用，但根据个人健康状况定制饮食推理仍然是一项重大挑战。营养问答 (QA) 已成为解决此问题的一种流行方法。然而，当前的研究面临两个关键限制。一方面，缺乏涉及用户特定医疗信息的数据集严重限制了 \textit{个性化}。个人健康需求的巨大差异进一步加剧了这一挑战。另一方面，虽然大型语言模型 (LLM)（此任务的流行解决方案）表现出强大的推理能力，但它们在个性化健康饮食推理的特定领域复杂性方面遇到困难，而现有的基准未能捕捉到这些挑战。为了解决这些差距，我们引入了营养图谱问答 (NGQA) 基准，这是第一个为个性化营养健康推理设计的图谱问答数据集。 NGQA 利用国家健康和营养检查调查 (NHANES) 和膳食研究食品和营养数据库 (FNDDS) 的数据来评估某种食物对特定用户是否健康，并提供关键营养成分的解释。该基准测试结合了三种问题复杂性设置，并评估了三个下游任务的推理能力。对 LLM 主干和基线模型进行的大量实验表明，NGQA 基准测试有效地挑战了现有模型。总之，NGQA 解决了一个关键的现实问题，同时通过一种新颖的领域特定基准测试推进了 GraphQA 研究。

Title: In-context Continual Learning Assisted by an External Continual Learner

• Authors: Saleh Momeni, Sahisnu Mazumder, Zixuan Ke, Bing Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15563
• Pdf URL: https://arxiv.org/pdf/2412.15563
• Copy Paste: [[2412.15563]] In-context Continual Learning Assisted by an External Continual Learner(https://arxiv.org/abs/2412.15563)
• Keywords: language model, llm, prompt
• Abstract: Existing continual learning (CL) methods mainly rely on fine-tuning or adapting large language models (LLMs). They still suffer from catastrophic forgetting (CF). Little work has been done to exploit in-context learning (ICL) to leverage the extensive knowledge within LLMs for CL without updating any parameters. However, incrementally learning each new task in ICL necessitates adding training examples from each class of the task to the prompt, which hampers scalability as the prompt length increases. This issue not only leads to excessively long prompts that exceed the input token limit of the underlying LLM but also degrades the model's performance due to the overextended context. To address this, we introduce InCA, a novel approach that integrates an external continual learner (ECL) with ICL to enable scalable CL without CF. The ECL is built incrementally to pre-select a small subset of likely classes for each test instance. By restricting the ICL prompt to only these selected classes, InCA prevents prompt lengths from becoming excessively long, while maintaining high performance. Experimental results demonstrate that InCA significantly outperforms existing CL baselines, achieving substantial performance gains.
• 摘要：现有的持续学习 (CL) 方法主要依赖于微调或调整大型语言模型 (LLM)。它们仍然受到灾难性遗忘 (CF) 的影响。很少有研究利用上下文学习 (ICL) 来利用 LLM 中的广泛知识进行 CL，而无需更新任何参数。然而，在 ICL 中逐步学习每个新任务需要将任务每个类别的训练示例添加到提示中，这会随着提示长度的增加而妨碍可扩展性。这个问题不仅会导致提示过长，超出底层 LLM 的输入标记限制，而且还会由于上下文过度扩展而降低模型的性能。为了解决这个问题，我们引入了 InCA，这是一种新颖的方法，它将外部持续学习器 (ECL) 与 ICL 集成在一起，以实现无需 CF 的可扩展 CL。ECL 是逐步构建的，以便为每个测试实例预先选择一小部分可能的类。通过将 ICL 提示限制为这些选定的类别，InCA 可防止提示长度过长，同时保持高性能。实验结果表明，InCA 明显优于现有的 CL 基线，实现了显著的性能提升。

Title: NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization

• Authors: Danial Kamali, Elham J. Barezi, Parisa Kordjamshidi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15588
• Pdf URL: https://arxiv.org/pdf/2412.15588
• Copy Paste: [[2412.15588]] NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization(https://arxiv.org/abs/2412.15588)
• Keywords: language model, llm, agent
• Abstract: Compositional generalization is crucial for artificial intelligence agents to solve complex vision-language reasoning tasks. Neuro-symbolic approaches have demonstrated promise in capturing compositional structures, but they face critical challenges: (a) reliance on predefined predicates for symbolic representations that limit adaptability, (b) difficulty in extracting predicates from raw data, and (c) using non-differentiable operations for combining primitive concepts. To address these issues, we propose NeSyCoCo, a neuro-symbolic framework that leverages large language models (LLMs) to generate symbolic representations and map them to differentiable neural computations. NeSyCoCo introduces three innovations: (a) augmenting natural language inputs with dependency structures to enhance the alignment with symbolic representations, (b) employing distributed word representations to link diverse, linguistically motivated logical predicates to neural modules, and (c) using the soft composition of normalized predicate scores to align symbolic and differentiable reasoning. Our framework achieves state-of-the-art results on the ReaSCAN and CLEVR-CoGenT compositional generalization benchmarks and demonstrates robust performance with novel concepts in the CLEVR-SYN benchmark.
• 摘要：组合泛化对于人工智能代理解决复杂的视觉语言推理任务至关重要。神经符号方法在捕捉组合结构方面表现出了良好的前景，但它们面临着严峻的挑战：（a）对符号表示的预定义谓词的依赖限制了适应性，（b）难以从原始数据中提取谓词，以及（c）使用不可微分运算来组合原始概念。为了解决这些问题，我们提出了 NeSyCoCo，这是一个神经符号框架，它利用大型语言模型 (LLM) 来生成符号表示并将其映射到可微分的神经计算。NeSyCoCo 引入了三项创新：（a）使用依赖结构增强自然语言输入以增强与符号表示的对齐，（b）使用分布式词表示将各种语言驱动的逻辑谓词链接到神经模块，以及（c）使用规范化谓词分数的软组合来对齐符号和可微分推理。我们的框架在 ReaSCAN 和 CLEVR-CoGenT 组合泛化基准上取得了最先进的结果，并在 CLEVR-SYN 基准中通过新颖的概念展示了强劲的性能。

Title: Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation

• Authors: Xiaoqiang Kang, Zimu Wang, Xiaobo Jin, Wei Wang, Kaizhu Huang, Qiufeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15594
• Pdf URL: https://arxiv.org/pdf/2412.15594
• Copy Paste: [[2412.15594]] Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation(https://arxiv.org/abs/2412.15594)
• Keywords: language model, llm
• Abstract: Solving tabular math word problems (TMWPs) has become a critical role in evaluating the mathematical reasoning ability of large language models (LLMs), where large-scale TMWP samples are commonly required for LLM fine-tuning. Since the collection of high-quality TMWP datasets is costly and time-consuming, recent research has concentrated on automatic TMWP generation. However, current generated samples usually suffer from issues of either correctness or diversity. In this paper, we propose a Template-driven LLM-paraphrased (TeLL) framework for generating high-quality TMWP samples with diverse backgrounds and accurate tables, questions, answers, and solutions. To this end, we first extract templates from existing real samples to generate initial problems, ensuring correctness. Then, we adopt an LLM to extend templates and paraphrase problems, obtaining diverse TMWP samples. Furthermore, we find the reasoning annotation is important for solving TMWPs. Therefore, we propose to enrich each solution with illustrative reasoning steps. Through the proposed framework, we construct a high-quality dataset TabMWP-TeLL by adhering to the question types in the TabMWP dataset, and we conduct extensive experiments on a variety of LLMs to demonstrate the effectiveness of TabMWP-TeLL in improving TMWP solving performance. The code and data of this paper are available at: this https URL.
• 摘要：解决表格数学应用题 (TMWP) 已成为评估大型语言模型 (LLM) 数学推理能力的关键角色，其中 LLM 微调通常需要大规模 TMWP 样本。由于收集高质量 TMWP 数据集成本高昂且耗时，最近的研究集中在自动 TMWP 生成上。然而，当前生成的样本通常存在正确性或多样性问题。在本文中，我们提出了一个模板驱动的 LLM 释义 (TeLL) 框架，用于生成具有不同背景和准确表格、问题、答案和解决方案的高质量 TMWP 样本。为此，我们首先从现有的真实样本中提取模板以生成初始问题，确保正确性。然后，我们采用 LLM 来扩展模板和释义问题，获得多样化的 TMWP 样本。此外，我们发现推理注释对于解决 TMWP 很重要。因此，我们建议用说明性推理步骤来丰富每个解决方案。通过提出的框架，我们遵循 TabMWP 数据集中的问题类型构建了高质量的数据集 TabMWP-TeLL，并在各种 LLM 上进行了广泛的实验，以证明 TabMWP-TeLL 在提高 TMWP 解决性能方面的有效性。本文的代码和数据可在以下网址获取：此 https URL。

Title: Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification

• Authors: Gyutae Park, Ingeol Baek, ByeongJeong Kim, Joongbo Shin, Hwanhee Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15603
• Pdf URL: https://arxiv.org/pdf/2412.15603
• Copy Paste: [[2412.15603]] Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification(https://arxiv.org/abs/2412.15603)
• Keywords: language model
• Abstract: Dialogue intent classification aims to identify the underlying purpose or intent of a user's input in a conversation. Current intent classification systems encounter considerable challenges, primarily due to the vast number of possible intents and the significant semantic overlap among similar intent classes. In this paper, we propose a novel approach to few-shot dialogue intent classification through in-context learning, incorporating dynamic label refinement to address these challenges. Our method retrieves relevant examples for a test input from the training set and leverages a large language model to dynamically refine intent labels based on semantic understanding, ensuring that intents are clearly distinguishable from one another. Experimental results demonstrate that our approach effectively resolves confusion between semantically similar intents, resulting in significantly enhanced performance across multiple datasets compared to baselines. We also show that our method generates more interpretable intent labels, and has a better semantic coherence in capturing underlying user intents compared to baselines.
• 摘要：对话意图分类旨在识别用户在对话中输入的根本目的或意图。当前的意图分类系统面临相当大的挑战，主要是因为可能的意图数量巨大，并且相似意图类别之间存在显著的语义重叠。在本文中，我们提出了一种通过上下文学习进行小样本对话意图分类的新方法，并结合动态标签细化来解决这些挑战。我们的方法从训练集中检索测试输入的相关示例，并利用大型语言模型根据语义理解动态细化意图标签，确保意图彼此之间可以清楚地区分。实验结果表明，我们的方法有效地解决了语义相似意图之间的混淆，与基线相比，在多个数据集上的性能显著提高。我们还表明，与基线相比，我们的方法生成了更多可解释的意图标签，并且在捕获潜在用户意图方面具有更好的语义一致性。

Title: Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

• Authors: Brian J Chan, Chao-Ting Chen, Jui-Hung Cheng, Hen-Hsen Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15605
• Pdf URL: https://arxiv.org/pdf/2412.15605
• Copy Paste: [[2412.15605]] Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks(https://arxiv.org/abs/2412.15605)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) has gained traction as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG introduces challenges such as retrieval latency, potential errors in document selection, and increased system complexity. With the advent of large language models (LLMs) featuring significantly extended context windows, this paper proposes an alternative paradigm, cache-augmented generation (CAG) that bypasses real-time retrieval. Our method involves preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters. During inference, the model utilizes these preloaded parameters to answer queries without additional retrieval steps. Comparative analyses reveal that CAG eliminates retrieval latency and minimizes retrieval errors while maintaining context relevance. Performance evaluations across multiple benchmarks highlight scenarios where long-context LLMs either outperform or complement traditional RAG pipelines. These findings suggest that, for certain applications, particularly those with a constrained knowledge base, CAG provide a streamlined and efficient alternative to RAG, achieving comparable or superior results with reduced complexity.
• 摘要：检索增强生成 (RAG) 是一种通过集成外部知识源来增强语言模型的强大方法，已获得广泛关注。然而，RAG 带来了检索延迟、文档选择中的潜在错误以及系统复杂性增加等挑战。随着具有显著扩展上下文窗口的大型语言模型 (LLM) 的出现，本文提出了一种替代范式，即绕过实时检索的缓存增强生成 (CAG)。我们的方法涉及将所有相关资源（尤其是当要检索的文档或知识大小有限且可管理时）预加载到 LLM 的扩展上下文中并缓存其运行时参数。在推理过程中，模型利用这些预加载的参数来回答查询，而无需额外的检索步骤。比较分析表明，CAG 消除了检索延迟并最大限度地减少了检索错误，同时保持了上下文相关性。跨多个基准的性能评估突出显示了长上下文 LLM 优于或补充传统 RAG 管道的场景。这些发现表明，对于某些应用，特别是知识库有限的应用，CAG 提供了一种比 RAG 更简化、更高效的替代方案，能够以更低的复杂性实现相当或更优异的结果。

Title: Can Input Attributions Interpret the Inductive Reasoning Process Elicited in In-Context Learning?

• Authors: Mengyu Ye, Tatsuki Kuribayashi, Goro Kobayashi, Jun Suzuki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15628
• Pdf URL: https://arxiv.org/pdf/2412.15628
• Copy Paste: [[2412.15628]] Can Input Attributions Interpret the Inductive Reasoning Process Elicited in In-Context Learning?(https://arxiv.org/abs/2412.15628)
• Keywords: language model, llm, prompt
• Abstract: Elucidating the rationale behind neural models' outputs has been challenging in the machine learning field, which is indeed applicable in this age of large language models (LLMs) and in-context learning (ICL). When it comes to estimating input attributions (IA), ICL poses a new issue of interpreting which example in the prompt, consisting of a set of examples, contributed to identifying the task/rule to be solved. To this end, in this paper, we introduce synthetic diagnostic tasks inspired by the poverty of the stimulus design in inductive reasoning; here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates the task demonstrated. The question is whether conventional IA methods can identify such an example in interpreting the inductive reasoning process in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.
• 摘要：阐明神经模型输出背后的原理一直是机器学习领域的难题，在大型语言模型 (LLM) 和上下文学习 (ICL) 时代，这一难题确实适用。在估计输入归因 (IA) 时，ICL 提出了一个新问题，即解释提示中的哪个示例（由一组示例组成）有助于识别要解决的任务/规则。为此，在本文中，我们引入了受归纳推理中刺激设计贫乏启发的合成诊断任务；在这里，大多数上下文示例相对于其基本规则是模棱两可的，而一个关键示例可以消除所演示任务的歧义。问题是传统的 IA 方法是否可以在解释 ICL 中的归纳推理过程时识别出这样的例子。我们的实验提供了几个实际的发现；例如，某种简单的 IA 方法效果最好，模型越大，使用基于梯度的 IA 方法解释 ICL 通常就越困难。

Title: MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula

• Authors: Sieun Hyeon, Kyudan Jung, Jaehee Won, Nam-Joon Kim, Hyun Gon Ryu, Hyuk-Jae Lee, Jaeyoung Do
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15655
• Pdf URL: https://arxiv.org/pdf/2412.15655
• Copy Paste: [[2412.15655]] MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula(https://arxiv.org/abs/2412.15655)
• Keywords: language model, gpt, llm
• Abstract: In various academic and professional settings, such as mathematics lectures or research presentations, it is often necessary to convey mathematical expressions orally. However, reading mathematical expressions aloud without accompanying visuals can significantly hinder comprehension, especially for those who are hearing-impaired or rely on subtitles due to language barriers. For instance, when a presenter reads Euler's Formula, current Automatic Speech Recognition (ASR) models often produce a verbose and error-prone textual description (e.g., e to the power of i x equals cosine of x plus i $\textit{side}$ of x), instead of the concise $\LaTeX{}$ format (i.e., $ e^{ix} = \cos(x) + i\sin(x) $), which hampers clear understanding and communication. To address this issue, we introduce MathSpeech, a novel pipeline that integrates ASR models with small Language Models (sLMs) to correct errors in mathematical expressions and accurately convert spoken expressions into structured $\LaTeX{}$ representations. Evaluated on a new dataset derived from lecture recordings, MathSpeech demonstrates $\LaTeX{}$ generation capabilities comparable to leading commercial Large Language Models (LLMs), while leveraging fine-tuned small language models of only 120M parameters. Specifically, in terms of CER, BLEU, and ROUGE scores for $\LaTeX{}$ translation, MathSpeech demonstrated significantly superior capabilities compared to GPT-4o. We observed a decrease in CER from 0.390 to 0.298, and higher ROUGE/BLEU scores compared to GPT-4o.
• 摘要：在各种学术和专业场合，例如数学讲座或研究报告，经常需要口头传达数学表达式。然而，大声朗读数学表达式而不附带视觉效果会严重阻碍理解，尤其是对于那些有听力障碍或因语言障碍而依赖字幕的人来说。例如，当演示者朗读欧拉公式时，当前的自动语音识别 (ASR) 模型通常会产生冗长且容易出错的文本描述（例如，e 的 i 次方 x 等于 x 的余弦加 i $\textit{side}$），而不是简洁的 $\LaTeX{}$ 格式（即，$ e^{ix} = \cos(x) + i\sin(x) $），这会妨碍清晰的理解和交流。为了解决这个问题，我们引入了 MathSpeech，这是一种新颖的管道，它将 ASR 模型与小型语言模型 (sLM) 集成在一起，以纠正数学表达式中的错误，并准确地将口语表达转换为结构化的 $\LaTeX{}$ 表示。在从讲座录音中获得的新数据集上进行评估时，MathSpeech 展示了与领先的商业大型语言模型 (LLM) 相当的 $\LaTeX{}$ 生成能力，同时利用了仅 1.2 亿个参数的微调小型语言模型。具体而言，就 $\LaTeX{}$ 翻译的 CER、BLEU 和 ROUGE 分数而言，MathSpeech 表现出比 GPT-4o 明显更优越的能力。我们观察到 CER 从 0.390 下降到 0.298，与 GPT-4o 相比，ROUGE/BLEU 分数更高。

Title: Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses

• Authors: Evgenia Ilia, Wilker Aziz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15683
• Pdf URL: https://arxiv.org/pdf/2412.15683
• Copy Paste: [[2412.15683]] Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses(https://arxiv.org/abs/2412.15683)
• Keywords: language model, prompt
• Abstract: With the broader use of language models (LMs) comes the need to estimate their ability to respond reliably to prompts (e.g., are generated responses likely to be correct?). Uncertainty quantification tools (notions of confidence and entropy, i.a.) can be used to that end (e.g., to reject a response when the model is `uncertain'). For example, Kuhn et al. (semantic entropy; 2022b) regard semantic variation amongst sampled responses as evidence that the model `struggles' with the prompt and that the LM is likely to err. We argue that semantic variability need not imply error--this being especially intuitive in open-ended settings, where prompts elicit multiple adequate but semantically distinct responses. Hence, we propose to annotate sampled responses for their adequacy to the prompt (e.g., using a classifier) and estimate the Probability the model assigns to Adequate Responses (PROBAR), which we then regard as an indicator of the model's reliability at the instance level. We evaluate PROBAR as a measure of confidence in selective prediction with OPT models (in two QA datasets and in next-word prediction, for English) and find PROBAR to outperform semantic entropy across prompts with varying degrees of ambiguity/open-endedness.
• 摘要：随着语言模型 (LM) 的广泛使用，需要评估它们可靠地响应提示的能力（例如，生成的响应是否可能正确？）。不确定性量化工具（置信度和熵的概念等）可用于此目的（例如，当模型“不确定”时拒绝响应）。例如，Kuhn 等人（语义熵；2022b）认为采样响应之间的语义变化是模型“难以”处理提示并且 LM 可能出错的证据。我们认为语义变化并不一定意味着错误——这在开放式设置中尤其直观，其中提示会引发多个充分但语义上不同的响应。因此，我们建议对抽样的响应进行注释，以确定其是否适合提示（例如，使用分类器），并估计模型分配给适当响应的概率 (PROBAR)，然后将其视为模型在实例级别的可靠性指标。我们评估 PROBAR 作为 OPT 模型选择性预测的置信度度量（在两个 QA 数据集和下一个单词预测中，对于英语），我们发现 PROBAR 在具有不同程度的歧义/开放性的提示中的表现优于语义熵。

Title: Contrastive Learning for Task-Independent SpeechLLM-Pretraining

• Authors: Maike Züfle, Jan Niehues
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.15712
• Pdf URL: https://arxiv.org/pdf/2412.15712
• Copy Paste: [[2412.15712]] Contrastive Learning for Task-Independent SpeechLLM-Pretraining(https://arxiv.org/abs/2412.15712)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.
• 摘要：大型语言模型 (LLM) 在自然语言处理方面表现出色，但将这些 LLM 有效地应用于语音处理任务并非易事。直接针对特定任务的微调受到过度拟合风险、数据要求和计算成本的限制。为了应对这些挑战，我们提出了一种可扩展的两阶段训练方法：(1) 独立于任务的语音预训练阶段，使用对比学习在所有层上对齐文本和语音表示，然后是 (2) 需要最少数据的特定于任务的微调阶段。这种方法优于传统的 ASR 预训练，使模型能够超越专门用于语音翻译和问答的模型，同时仅使用 10% 的任务特定数据进行训练。

Title: Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models

• Authors: Shamus Sim, Tyrone Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15748
• Pdf URL: https://arxiv.org/pdf/2412.15748
• Copy Paste: [[2412.15748]] Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models(https://arxiv.org/abs/2412.15748)
• Keywords: language model, llm
• Abstract: Background: Despite the current ubiquity of Large Language Models (LLMs) across the medical domain, there is a surprising lack of studies which address their reasoning behaviour. We emphasise the importance of understanding reasoning behaviour as opposed to high-level prediction accuracies, since it is equivalent to explainable AI (XAI) in this context. In particular, achieving XAI in medical LLMs used in the clinical domain will have a significant impact across the healthcare sector. Results: Therefore, we define the concept of reasoning behaviour in the specific context of medical LLMs. We then categorise and discuss the current state of the art of methods which evaluate reasoning behaviour in medical LLMs. Finally, we propose theoretical frameworks which can empower medical professionals or machine learning engineers to gain insight into the low-level reasoning operations of these previously obscure models. Conclusion: The subsequent increased transparency and trust in medical machine learning models by clinicians as well as patients will accelerate the integration, application as well as further development of medical AI for the healthcare system as a whole
• 摘要：背景：尽管目前大型语言模型 (LLM) 在整个医学领域无处不在，但令人惊讶的是，关于其推理行为的研究却很少。我们强调理解推理行为的重要性，而不是高级预测准确性，因为在这种情况下，它相当于可解释的人工智能 (XAI)。特别是在临床领域使用的医学 LLM 中实现 XAI 将对整个医疗保健行业产生重大影响。结果：因此，我们在医学 LLM 的特定背景下定义了推理行为的概念。然后，我们对评估医学 LLM 中推理行为的方法进行分类和讨论。最后，我们提出了理论框架，使医疗专业人员或机器学习工程师能够深入了解这些以前晦涩难懂的模型的低级推理操作。结论：随后，临床医生和患者对医学机器学习模型的透明度和信任度将提高，这将加速整个医疗保健系统医疗人工智能的整合、应用和进一步发展

Title: Linguistic Features Extracted by GPT-4 Improve Alzheimer's Disease Detection based on Spontaneous Speech

• Authors: Jonathan Heitz, Gerold Schneider, Nicolas Langer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15772
• Pdf URL: https://arxiv.org/pdf/2412.15772
• Copy Paste: [[2412.15772]] Linguistic Features Extracted by GPT-4 Improve Alzheimer's Disease Detection based on Spontaneous Speech(https://arxiv.org/abs/2412.15772)
• Keywords: language model, gpt, llm
• Abstract: Alzheimer's Disease (AD) is a significant and growing public health concern. Investigating alterations in speech and language patterns offers a promising path towards cost-effective and non-invasive early detection of AD on a large scale. Large language models (LLMs), such as GPT, have enabled powerful new possibilities for semantic text analysis. In this study, we leverage GPT-4 to extract five semantic features from transcripts of spontaneous patient speech. The features capture known symptoms of AD, but they are difficult to quantify effectively using traditional methods of computational linguistics. We demonstrate the clinical significance of these features and further validate one of them ("Word-Finding Difficulties") against a proxy measure and human raters. When combined with established linguistic features and a Random Forest classifier, the GPT-derived features significantly improve the detection of AD. Our approach proves effective for both manually transcribed and automatically generated transcripts, representing a novel and impactful use of recent advancements in LLMs for AD speech analysis.
• 摘要：阿尔茨海默病 (AD) 是一个日益严重的重大公共卫生问题。研究语音和语言模式的改变为大规模、经济高效且无创地早期检测 AD 提供了一条有希望的途径。大型语言模型 (LLM)，例如 GPT，为语义文本分析提供了强大的新可能性。在本研究中，我们利用 GPT-4 从自发患者语音的转录本中提取五个语义特征。这些特征捕捉了已知的 AD 症状，但使用传统的计算语言学方法很难有效地量化它们。我们展示了这些特征的临床意义，并进一步根据代理测量和人类评分者验证了其中一个特征（“找词困难”）。当与已建立的语言特征和随机森林分类器相结合时，GPT 衍生的特征可显著提高 AD 的检测率。我们的方法被证明对手动转录和自动生成的转录本都有效，代表了 LLM 在 AD 语音分析方面的最新进展的新颖且有影响力的用途。

Title: Learning from Impairment: Leveraging Insights from Clinical Linguistics in Language Modelling Research

• Authors: Dominique Brunato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15785
• Pdf URL: https://arxiv.org/pdf/2412.15785
• Copy Paste: [[2412.15785]] Learning from Impairment: Leveraging Insights from Clinical Linguistics in Language Modelling Research(https://arxiv.org/abs/2412.15785)
• Keywords: language model
• Abstract: This position paper investigates the potential of integrating insights from language impairment research and its clinical treatment to develop human-inspired learning strategies and evaluation frameworks for language models (LMs). We inspect the theoretical underpinnings underlying some influential linguistically motivated training approaches derived from neurolinguistics and, particularly, aphasiology, aimed at enhancing the recovery and generalization of linguistic skills in aphasia treatment, with a primary focus on those targeting the syntactic domain. We highlight how these insights can inform the design of rigorous assessments for LMs, specifically in their handling of complex syntactic phenomena, as well as their implications for developing human-like learning strategies, aligning with efforts to create more sustainable and cognitively plausible natural language processing (NLP) models.
• 摘要：本立场文件探讨了将语言障碍研究及其临床治疗的见解结合起来，以开发受人类启发的语言模型 (LM) 学习策略和评估框架的潜力。我们研究了一些有影响力的语言激励训练方法背后的理论基础，这些方法源自神经语言学，特别是失语症学，旨在增强失语症治疗中语言技能的恢复和概括，主要关注那些针对句法领域的技能。我们重点介绍了这些见解如何为 LM 的严格评估设计提供信息，特别是在处理复杂句法现象方面，以及它们对开发类似人类的学习策略的影响，与创建更可持续和认知上更合理的自然语言处理 (NLP) 模型的努力相一致。

Title: Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning

• Authors: Sungjin Park, Xiao Liu, Yeyun Gong, Edward Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15797
• Pdf URL: https://arxiv.org/pdf/2412.15797
• Copy Paste: [[2412.15797]] Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning(https://arxiv.org/abs/2412.15797)
• Keywords: language model
• Abstract: Despite recent advances in large language models, open-source models often struggle to consistently perform well on complex reasoning tasks. Existing ensemble methods, whether applied at the token or output levels, fail to address these challenges. In response, we present Language model Ensemble with Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level ensembling of language models. LE-MCTS formulates step-by-step reasoning with an ensemble of language models as a Markov decision process. In this framework, states represent intermediate reasoning paths, while actions consist of generating the next reasoning step using one of the language models selected from a predefined pool. Guided by a process-based reward model, LE-MCTS performs a tree search over the reasoning steps generated by different language models, identifying the most accurate reasoning chain. Experimental results on five mathematical reasoning benchmarks demonstrate that our approach outperforms both single language model decoding algorithms and language model ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the MATH and MQA datasets, respectively, highlighting its effectiveness in solving complex reasoning problems.
• 摘要：尽管大型语言模型最近取得了进展，但开源模型在复杂的推理任务上往往难以始终如一地表现良好。现有的集成方法，无论是应用于标记级别还是输出级别，都无法解决这些挑战。为此，我们提出了蒙特卡洛树搜索语言模型集成 (LE-MCTS)，这是一种用于过程级语言模型集成的新型框架。LE-MCTS 将使用语言模型集成的逐步推理公式化为马尔可夫决策过程。在这个框架中，状态代表中间推理路径，而动作包括使用从预定义池中选择的语言模型之一生成下一个推理步骤。在基于过程的奖励模型的指导下，LE-MCTS 对不同语言模型生成的推理步骤执行树搜索，从而确定最准确的推理链。在五个数学推理基准上的实验结果表明，我们的方法优于单一语言模型解码算法和语言模型集成方法。值得注意的是，LE-MCTS 在 MATH 和 MQA 数据集上的性能分别提高了 3.6% 和 4.3%，凸显了其在解决复杂推理问题方面的有效性。

Title: $\pi$-yalli: un nouveau corpus pour le nahuatl

• Authors: Juan-Manuel Torres-Moreno, Juan-José Guzmán-Landa, Graham Ranger, Martha Lorena Avendaño Garrido, Miguel Figueroa-Saavedra, Ligia Quintana-Torres, Carlos-Emiliano González-Gallardo, Elvys Linhares Pontes, Patricia Velázquez Morales, Luis-Gil Moreno Jiménez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15821
• Pdf URL: https://arxiv.org/pdf/2412.15821
• Copy Paste: [[2412.15821]] $\pi$-yalli: un nouveau corpus pour le nahuatl(https://arxiv.org/abs/2412.15821)
• Keywords: language model
• Abstract: The NAHU$^2$ project is a Franco-Mexican collaboration aimed at building the $\pi$-YALLI corpus adapted to machine learning, which will subsequently be used to develop computer resources for the Nahuatl language. Nahuatl is a language with few computational resources, even though it is a living language spoken by around 2 million people. We have decided to build $\pi$-YALLI, a corpus that will enable to carry out research on Nahuatl in order to develop Language Models (LM), whether dynamic or not, which will make it possible to in turn enable the development of Natural Language Processing (NLP) tools such as: a) a grapheme unifier, b) a word segmenter, c) a POS grammatical analyser, d) a content-based Automatic Text Summarization; and possibly, e) a translator translator (probabilistic or learning-based).
• 摘要：NAHU$^2$ 项目是法国和墨西哥的合作项目，旨在构建适用于机器学习的 $\pi$-YALLI 语料库，随后将用于开发纳瓦特尔语的计算机资源。纳瓦特尔语是一种计算资源很少的语言，尽管它是一种约有 200 万人使用的活语言。我们决定构建 $\pi$-YALLI，这是一个语料库，它将使我们能够对纳瓦特尔语进行研究，以开发语言模型 (LM)，无论是否动态，这反过来又使我们能够开发自然语言处理 (NLP) 工具，例如：a) 字素统一器，b) 词分割器，c) POS 语法分析器，d) 基于内容的自动文本摘要；以及可能的 e) 翻译器（基于概率或学习）。

Title: TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain

• Authors: Camille Barboule, Viet-Phi Huynh, Adrien Bufort, Yoan Chabot, Géraldine Damnati, Gwénolé Lecorvé
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15891
• Pdf URL: https://arxiv.org/pdf/2412.15891
• Copy Paste: [[2412.15891]] TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain(https://arxiv.org/abs/2412.15891)
• Keywords: language model, llm
• Abstract: Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand.
• 摘要：尽管大型语言模型 (LLM) 在许多任务中表现出色，但在处理技术含量高的领域时，其准确性仍然不足。尤其是电信 (telco) 是一个特别具有挑战性的领域，因为它有大量的词汇、语义和概念特性。然而，这个领域有许多有价值的用例，直接与工业需求相关。因此，本文研究了如何将 LLM 适应电信领域。它报告了我们的努力：(i) 收集大量特定领域的数据 (800M 标记、80K 指令)，(ii) 使用各种方法进行适应，以及 (iii) 在需要大量电信知识的下游任务中将它们与更大的通用模型进行对比。我们在 Llama-2-7b 上的实验表明，领域适应模型可以挑战大型通用模型。他们还表明，适应可以限制在一个独特的指令调整步骤中，从而无需事先对原始文本进行任何微调。

Title: On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education

• Authors: Lorenz Wendlinger, Christian Braun, Abdullah Al Zubaer, Simon Alexander Nonn, Sarah Großkopf, Christofer Fellicious, Michael Granitzer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15902
• Pdf URL: https://arxiv.org/pdf/2412.15902
• Copy Paste: [[2412.15902]] On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education(https://arxiv.org/abs/2412.15902)
• Keywords: llm, prompt, retrieval augmented generation, chain-of-thought
• Abstract: We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.
• 摘要：我们表明，当前的开源基础法学硕士拥有教学能力和德国法律背景知识，足以在教育环境中进行一些法律分析。然而，模型能力在非常具体的任务中会失效，例如“Gutachtenstil”评估风格成分的分类，或者在复杂的上下文中，例如完整的法律意见。即使有扩展的上下文和有效的提示策略，它们也无法与词袋基线相匹配。为了解决这个问题，我们引入了一种基于检索增强生成的提示示例选择方法，该方法在高数据可用性场景中显著改善了预测。我们进一步评估了预训练法学硕士在论证挖掘和自动论文评分两个标准任务上的表现，发现它更合适。在整个过程中，预训练法学硕士在标记数据很少或没有标记数据的场景中比基线有所改进，而思想链提示在零样本情况下进一步提供帮助。

Title: Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Model

• Authors: Yosuke Yamagishi, Yuta Nakamura, Tomohiro Kikuchi, Yuki Sonoda, Hiroshi Hirakawa, Shintaro Kano, Satoshi Nakamura, Shouhei Hanaoka, Takeharu Yoshikawa, Osamu Abe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15907
• Pdf URL: https://arxiv.org/pdf/2412.15907
• Copy Paste: [[2412.15907]] Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Model(https://arxiv.org/abs/2412.15907)
• Keywords: language model, gpt
• Abstract: Background: Recent advances in large language models highlight the need for high-quality multilingual medical datasets. While Japan leads globally in CT scanner deployment and utilization, the lack of large-scale Japanese radiology datasets has hindered the development of specialized language models for medical imaging analysis. Objective: To develop a comprehensive Japanese CT report dataset through machine translation and establish a specialized language model for structured finding classification. Additionally, to create a rigorously validated evaluation dataset through expert radiologist review. Methods: We translated the CT-RATE dataset (24,283 CT reports from 21,304 patients) into Japanese using GPT-4o mini. The training dataset consisted of 22,778 machine-translated reports, while the validation dataset included 150 radiologist-revised reports. We developed CT-BERT-JPN based on "tohoku-nlp/bert-base-japanese-v3" architecture for extracting 18 structured findings from Japanese radiology reports. Results: Translation metrics showed strong performance with BLEU scores of 0.731 and 0.690, and ROUGE scores ranging from 0.770 to 0.876 for Findings and from 0.748 to 0.857 for Impression sections. CT-BERT-JPN demonstrated superior performance compared to GPT-4o in 11 out of 18 conditions, including lymphadenopathy (+14.2%), interlobular septal thickening (+10.9%), and atelectasis (+7.4%). The model maintained F1 scores exceeding 0.95 in 14 out of 18 conditions and achieved perfect scores in four conditions. Conclusions: Our study establishes a robust Japanese CT report dataset and demonstrates the effectiveness of a specialized language model for structured finding classification. The hybrid approach of machine translation and expert validation enables the creation of large-scale medical datasets while maintaining high quality.
• 摘要：背景：大型语言模型的最新进展凸显了对高质量多语言医学数据集的需求。虽然日本在 CT 扫描仪部署和使用方面处于全球领先地位，但缺乏大规模的日本放射学数据集阻碍了用于医学影像分析的专门语言模型的开发。目标：通过机器翻译开发全面的日语 CT 报告数据集，并建立用于结构化发现分类的专门语言模型。此外，通过专家放射科医生的审查创建经过严格验证的评估数据集。方法：我们使用 GPT-4o mini 将 CT-RATE 数据集（来自 21,304 名患者的 24,283 份 CT 报告）翻译成日语。训练数据集包括 22,778 份机器翻译的报告，而验证数据集包括 150 份放射科医生修订的报告。我们基于“tohoku-nlp/bert-base-japanese-v3”架构开发了 CT-BERT-JPN，用于从日语放射学报告中提取 18 个结构化发现。结果：翻译指标表现出色，BLEU 得分为 0.731 和 0.690，ROUGE 得分范围为 0.770 至 0.876（发现部分）和 0.748 至 0.857（印象部分）。CT-BERT-JPN 在 18 种情况中的 11 种中表现出优于 GPT-4o 的性能，包括淋巴结肿大（+14.2%）、小叶间隔增厚（+10.9%）和肺不张（+7.4%）。该模型在 18 种情况中的 14 种中保持 F1 得分超过 0.95，并在 4 种情况下获得满分。结论：我们的研究建立了一个强大的日语 CT 报告数据集，并证明了专门的语言模型对结构化发现分类的有效性。机器翻译和专家验证的混合方法可以创建大规模医疗数据集，同时保持高质量。

Title: From General to Specific: Tailoring Large Language Models for Personalized Healthcare

• Authors: Ruize Shi, Hong Huang, Wei Zhou, Kehan Yin, Kai Zhao, Yun Zhao
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15957
• Pdf URL: https://arxiv.org/pdf/2412.15957
• Copy Paste: [[2412.15957]] From General to Specific: Tailoring Large Language Models for Personalized Healthcare(https://arxiv.org/abs/2412.15957)
• Keywords: language model, llm, prompt
• Abstract: The rapid development of large language models (LLMs) has transformed many industries, including healthcare. However, previous medical LLMs have largely focused on leveraging general medical knowledge to provide responses, without accounting for patient variability and lacking true personalization at the individual level. To address this, we propose a novel method called personalized medical language model (PMLM), which explores and optimizes personalized LLMs through recommendation systems and reinforcement learning (RL). Specifically, by utilizing self-informed and peer-informed personalization, PMLM captures changes in behaviors and preferences to design initial personalized prompts tailored to individual needs. We further refine these initial personalized prompts through RL, ultimately enhancing the precision of LLM guidance. Notably, the personalized prompt are hard prompt, which grants PMLM high adaptability and reusability, allowing it to directly leverage high-quality proprietary LLMs. We evaluate PMLM using real-world obstetrics and gynecology data, and the experimental results demonstrate that PMLM achieves personalized responses, and it provides more refined and individualized services, offering a potential way for personalized medical LLMs.
• 摘要：大型语言模型 (LLM) 的快速发展已经改变了许多行业，包括医疗保健。然而，以前的医学 LLM 主要侧重于利用一般医学知识来提供响应，而不考虑患者的差异性，并且缺乏个人层面的真正个性化。为了解决这个问题，我们提出了一种称为个性化医学语言模型 (PMLM) 的新方法，该方法通过推荐系统和强化学习 (RL) 探索和优化个性化 LLM。具体来说，通过利用自我知情和同伴知情的个性化，PMLM 可以捕捉行为和偏好的变化，以设计针对个人需求的初始个性化提示。我们通过 RL 进一步完善这些初始个性化提示，最终提高 LLM 指导的准确性。值得注意的是，个性化提示是硬提示，这赋予了 PMLM 高度的适应性和可重用性，使其能够直接利用高质量的专有 LLM。我们使用现实世界的妇产科数据评估 PMLM，实验结果表明 PMLM 实现了个性化响应，并且它提供了更精细和个性化的服务，为个性化医学 LLM 提供了潜在的途径。

Title: BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models

• Authors: Patrick Haller, Jonas Golde, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15978
• Pdf URL: https://arxiv.org/pdf/2412.15978
• Copy Paste: [[2412.15978]] BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models(https://arxiv.org/abs/2412.15978)
• Keywords: language model
• Abstract: This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 (Qin et al., 2024), a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that BABYHGRN, our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments.
• 摘要：本文探讨了在资源匮乏的语言建模场景中，循环神经网络 (RNN) 和其他次二次架构作为基于 Transformer 的模型的竞争性替代方案的潜力。我们利用最近提出的基于 RNN 的架构 HGRN2 (Qin et al., 2024)，并将其与基于 Transformer 的基线和其他次二次架构 (LSTM、xLSTM、Mamba) 的有效性进行比较评估。我们的实验结果表明，我们的 HGRN2 语言模型 BABYHGRN 在挑战赛的 10M 和 100M 字轨道上均优于基于 Transformer 的模型，这是根据它们在 BLiMP、EWoK、GLUE 和 BEAR 基准上的表现来衡量的。此外，我们展示了知识蒸馏的积极影响。我们的研究结果挑战了当前对 Transformer 架构的关注，并表明了基于 RNN 的模型的可行性，尤其是在资源受限的环境中。

Title: Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs

• Authors: Lynn Greschner, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15993
• Pdf URL: https://arxiv.org/pdf/2412.15993
• Copy Paste: [[2412.15993]] Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs(https://arxiv.org/abs/2412.15993)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Arguments evoke emotions, influencing the effect of the argument itself. Not only the emotional intensity but also the category influence the argument's effects, for instance, the willingness to adapt stances. While binary emotionality has been studied in arguments, there is no work on discrete emotion categories (e.g., "Anger") in such data. To fill this gap, we crowdsource subjective annotations of emotion categories in a German argument corpus and evaluate automatic LLM-based labeling methods. Specifically, we compare three prompting strategies (zero-shot, one-shot, chain-of-thought) on three large instruction-tuned language models (Falcon-7b-instruct, Llama-3.1-8B-instruct, GPT-4o-mini). We further vary the definition of the output space to be binary (is there emotionality in the argument?), closed-domain (which emotion from a given label set is in the argument?), or open-domain (which emotion is in the argument?). We find that emotion categories enhance the prediction of emotionality in arguments, emphasizing the need for discrete emotion annotations in arguments. Across all prompt settings and models, automatic predictions show a high recall but low precision for predicting anger and fear, indicating a strong bias toward negative emotions.
• 摘要：论点会引发情绪，影响论点本身的效果。不仅情绪强度，而且类别也会影响论点的效果，例如，调整立场的意愿。虽然二元情绪在论点中得到了研究，但还没有关于此类数据中离散情绪类别（例如“愤怒”）的研究。为了填补这一空白，我们在德国论点语料库中众包主观情绪类别注释，并评估基于 LLM 的自动标记方法。具体来说，我们在三个大型指令调整语言模型（Falcon-7b-instruct、Llama-3.1-8B-instruct、GPT-4o-mini）上比较了三种提示策略（零样本、单样本、思维链）。我们进一步将输出空间的定义更改为二进制（论点中是否存在情绪？）、闭域（给定标签集中的哪种情绪属于论点？）或开域（哪种情绪属于论点？）。我们发现情绪类别可以增强对争论中情绪的预测，这强调了争论中需要离散的情绪注释。在所有提示设置和模型中，自动预测在预测愤怒和恐惧方面显示出较高的召回率，但准确率较低，这表明对负面情绪存在强烈的偏见。

Title: The Only Way is Ethics: A Guide to Ethical Research with Large Language Models

• Authors: Eddie L. Ungless, Nikolas Vitsakis, Zeerak Talat, James Garforth, Björn Ross, Arno Onken, Atoosa Kasirzadeh, Alexandra Birch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16022
• Pdf URL: https://arxiv.org/pdf/2412.16022
• Copy Paste: [[2412.16022]] The Only Way is Ethics: A Guide to Ethical Research with Large Language Models(https://arxiv.org/abs/2412.16022)
• Keywords: language model, llm
• Abstract: There is a significant body of work looking at the ethical considerations of large language models (LLMs): critiquing tools to measure performance and harms; proposing toolkits to aid in ideation; discussing the risks to workers; considering legislation around privacy and security etc. As yet there is no work that integrates these resources into a single practical guide that focuses on LLMs; we attempt this ambitious goal. We introduce 'LLM Ethics Whitepaper', which we provide as an open and living resource for NLP practitioners, and those tasked with evaluating the ethical implications of others' work. Our goal is to translate ethics literature into concrete recommendations and provocations for thinking with clear first steps, aimed at computer scientists. 'LLM Ethics Whitepaper' distils a thorough literature review into clear Do's and Don'ts, which we present also in this paper. We likewise identify useful toolkits to support ethical work. We refer the interested reader to the full LLM Ethics Whitepaper, which provides a succinct discussion of ethical considerations at each stage in a project lifecycle, as well as citations for the hundreds of papers from which we drew our recommendations. The present paper can be thought of as a pocket guide to conducting ethical research with LLMs.
• 摘要：有大量研究关注大型语言模型 (LLM) 的道德考量：批评衡量绩效和危害的工具；提出有助于构思的工具包；讨论对工人的风险；考虑有关隐私和安全的立法等。目前还没有将这些资源整合成一个专注于 LLM 的实用指南的工作；我们试图实现这个雄心勃勃的目标。我们推出了“LLM 道德白皮书”，作为 NLP 从业人员和那些负责评估他人工作道德影响的人的开放和生活资源。我们的目标是将道德文献转化为针对计算机科学家的具体建议和思考的启发，并采取明确的第一步。“LLM 道德白皮书”将详尽的文献综述提炼为明确的“应该做”和“不应该做”的准则，我们也在本文中进行了介绍。我们同样确定了有用的工具包来支持道德工作。我们建议感兴趣的读者参阅完整的 LLM 道德白皮书，其中简明扼要地讨论了项目生命周期每个阶段的道德考量，并引用了我们提出建议的数百篇论文。本文可以看作是一份使用 LLM 进行道德研究的袖珍指南。

Title: Logical Consistency of Large Language Models in Fact-checking

• Authors: Bishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat, Arijit Khan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16100
• Pdf URL: https://arxiv.org/pdf/2412.16100
• Copy Paste: [[2412.16100]] Logical Consistency of Large Language Models in Fact-checking(https://arxiv.org/abs/2412.16100)
• Keywords: language model, llm, hallucination
• Abstract: In recent years, large language models (LLMs) have demonstrated significant success in performing varied natural language tasks such as language translation, question-answering, summarizing, fact-checking, etc. Despite LLMs' impressive ability to generate human-like texts, LLMs are infamous for their inconsistent responses -- a meaning-preserving change in the input query results in an inconsistent response and attributes to vulnerabilities of LLMs such as hallucination, jailbreaking, etc. Consequently, existing research focuses on simple paraphrasing-based consistency assessment of LLMs, and ignores complex queries that necessitates an even better understanding of logical reasoning by an LLM. Our work therefore addresses the logical inconsistency of LLMs under complex logical queries with primitive logical operators, e.g., negation, conjunction, and disjunction. As a test bed, we consider retrieval-augmented LLMs on a fact-checking task involving propositional logic queries from real-world knowledge graphs (KGs). Our contributions are three-fold. Benchmark: We introduce three logical fact-checking datasets over KGs for community development towards logically consistent LLMs. Assessment: We propose consistency measures of LLMs on propositional logic queries as input and demonstrate that existing LLMs lack logical consistency, specially on complex queries. Improvement: We employ supervised fine-tuning to improve the logical consistency of LLMs on the complex fact-checking task with KG contexts.
• 摘要：近年来，大型语言模型 (LLM) 在执行各种自然语言任务（如语言翻译、问答、总结、事实核查等）方面取得了显著的成功。尽管 LLM 具有生成类似人类文本的令人印象深刻的能力，但 LLM 因其不一致的响应而臭名昭著——输入查询中保留含义的变化会导致不一致的响应，并归因于 LLM 的漏洞，如幻觉、越狱等。因此，现有研究侧重于基于简单释义的 LLM 一致性评估，而忽略了需要 LLM 更好地理解逻辑推理的复杂查询。因此，我们的工作解决了 LLM 在复杂逻辑查询下与原始逻辑运算符（例如否定、合取和析取）的逻辑不一致问题。作为测试平台，我们考虑在涉及来自现实世界知识图谱 (KG) 的命题逻辑查询的事实核查任务上使用检索增强的 LLM。我们的贡献有三方面。基准：我们在 KG 上引入了三个逻辑事实核查数据集，以便社区开发逻辑一致的 LLM。评估：我们提出了 LLM 在命题逻辑查询上的一致性度量作为输入，并证明现有的 LLM 缺乏逻辑一致性，特别是在复杂查询上。改进：我们采用监督微调来提高 LLM 在具有 KG 上下文的复杂事实核查任务上的逻辑一致性。

Title: PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics

• Authors: Daniil Larionov, Steffen Eger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16120
• Pdf URL: https://arxiv.org/pdf/2412.16120
• Copy Paste: [[2412.16120]] PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics(https://arxiv.org/abs/2412.16120)
• Keywords: language model, gpt, llm, prompt
• Abstract: Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model's outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a $2.37\times$ reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.
• 摘要：评估机器生成的自然语言内容的质量是自然语言处理 (NLP) 中的一项艰巨任务。最近，大型语言模型 (LLM)（如 GPT-4）已被用于此目的，但由于复杂的评估提示需要大量使用 token，因此计算成本高昂。在本文中，我们提出了一种提示优化方法，该方法使用较小的、经过微调的语言模型来压缩评估提示的输入数据，从而在使用较大的 LLM 进行下游评估时减少 token 使用量和计算成本。我们的方法涉及一个两阶段的微调过程：监督微调，然后进行偏好优化，以根据人类偏好优化模型的输出。我们专注于机器翻译 (MT) 评估，并使用 GEMBA-MQM 指标作为起点。我们的结果显示 token 使用量减少了 $2.37\times$，而评估质量没有任何损失。这项工作使最先进的基于 LLM 的指标（如 GEMBA-MQM）更具成本效益和效率，增强了它们的可访问性，使其更广泛地使用。