2024-12-11

Title: Guidance is All You Need: Temperature-Guided Reasoning in Large Language Models

• Authors: Eyad Gomaa, Gomaa Salah
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06822
• Pdf URL: https://arxiv.org/pdf/2412.06822
• Copy Paste: [[2412.06822]] Guidance is All You Need: Temperature-Guided Reasoning in Large Language Models(https://arxiv.org/abs/2412.06822)
• Keywords: language model, chain-of-thought
• Abstract: We present Quasar-1, a novel architecture that introduces temperature-guided reasoning to large language models through the Token Temperature Mechanism (TTM) and Guided Sequence of Thought (GSoT). Our approach leverages the concept of hot and cold tokens, where hot tokens are prioritized for their contextual relevance, while cold tokens provide supplementary information. This dynamic modulation of token importance enables the model to achieve superior logical reasoning capabilities compared to traditional chain-of-thought approaches. Through rigorous mathematical analysis, we prove that our temperature-guided attention mechanism converges to optimal reasoning paths with exponential guarantees. Empirical results show significant improvements in reasoning accuracy and computational efficiency across a wide range of tasks, making advanced AI reasoning accessible to a broader range of applications.
• 摘要：我们提出了一种新颖的架构 Quasar-1，它通过标记温度机制 (TTM) 和引导式思维序列 (GSoT) 将温度引导推理引入大型语言模型。我们的方法利用了热标记和冷标记的概念，其中热标记根据其上下文相关性进行优先排序，而冷标记则提供补充信息。与传统的思维链方法相比，这种标记重要性的动态调节使模型能够实现卓越的逻辑推理能力。通过严格的数学分析，我们证明了我们的温度引导注意力机制可以收敛到具有指数保证的最佳推理路径。实证结果表明，在广泛的任务中，推理准确性和计算效率都有显著提高，使高级 AI 推理能够应用于更广泛的应用。

Title: Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System

• Authors: Fang Zeng, Zhiliang Lyu, Quanzheng Li, Xiang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06828
• Pdf URL: https://arxiv.org/pdf/2412.06828
• Copy Paste: [[2412.06828]] Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System(https://arxiv.org/abs/2412.06828)
• Keywords: language model, gpt, llm, agent
• Abstract: This study introduces "RadCouncil," a multi-agent Large Language Model (LLM) framework designed to enhance the generation of impressions in radiology reports from the finding section. RadCouncil comprises three specialized agents: 1) a "Retrieval" Agent that identifies and retrieves similar reports from a vector database, 2) a "Radiologist" Agent that generates impressions based on the finding section of the given report plus the exemplar reports retrieved by the Retrieval Agent, and 3) a "Reviewer" Agent that evaluates the generated impressions and provides feedback. The performance of RadCouncil was evaluated using both quantitative metrics (BLEU, ROUGE, BERTScore) and qualitative criteria assessed by GPT-4, using chest X-ray as a case study. Experiment results show improvements in RadCouncil over the single-agent approach across multiple dimensions, including diagnostic accuracy, stylistic concordance, and clarity. This study highlights the potential of utilizing multiple interacting LLM agents, each with a dedicated task, to enhance performance in specialized medical tasks and the development of more robust and adaptable healthcare AI solutions.
• 摘要：本研究引入了“RadCouncil”，这是一个多智能体大型语言模型 (LLM) 框架，旨在增强从发现部分生成放射学报告中的印象的能力。RadCouncil 由三个专门的智能体组成：1) 一个“检索”智能体，用于从矢量数据库中识别和检索类似的报告；2) 一个“放射科医生”智能体，用于根据给定报告的发现部分以及检索智能体检索到的样本报告生成印象；3) 一个“审阅者”智能体，用于评估生成的印象并提供反馈。使用定量指标（BLEU、ROUGE、BERTScore）和 GPT-4 评估的定性标准来评估 RadCouncil 的性能，并使用胸部 X 光作为案例研究。实验结果表明，与单智能体方法相比，RadCouncil 在多个维度上都有所改进，包括诊断准确性、文体一致性和清晰度。这项研究强调了利用多个交互的 LLM 代理（每个代理都有一项专门的任务）的潜力，以提高专门医疗任务的表现并开发更强大、适应性更强的医疗保健 AI 解决方案。

Title: TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Models

• Authors: Saipraneeth Devunuri, Lewis Lehe
• Subjects: cs.CL, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.06831
• Pdf URL: https://arxiv.org/pdf/2412.06831
• Copy Paste: [[2412.06831]] TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Models(https://arxiv.org/abs/2412.06831)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: This paper introduces a framework that leverages Large Language Models (LLMs) to answer natural language queries about General Transit Feed Specification (GTFS) data. The framework is implemented in a chatbot called TransitGPT with open-source code. TransitGPT works by guiding LLMs to generate Python code that extracts and manipulates GTFS data relevant to a query, which is then executed on a server where the GTFS feed is stored. It can accomplish a wide range of tasks, including data retrieval, calculations, and interactive visualizations, without requiring users to have extensive knowledge of GTFS or programming. The LLMs that produce the code are guided entirely by prompts, without fine-tuning or access to the actual GTFS feeds. We evaluate TransitGPT using GPT-4o and Claude-3.5-Sonnet LLMs on a benchmark dataset of 100 tasks, to demonstrate its effectiveness and versatility. The results show that TransitGPT can significantly enhance the accessibility and usability of transit data.
• 摘要：本文介绍了一个利用大型语言模型 (LLM) 来回答有关通用交通信息规范 (GTFS) 数据的自然语言查询的框架。该框架在一个名为 TransitGPT 的聊天机器人中实现，代码为开源代码。TransitGPT 的工作原理是引导 LLM 生成 Python 代码，该代码提取和处理与查询相关的 GTFS 数据，然后在存储 GTFS 信息源的服务器上执行。它可以完成各种任务，包括数据检索、计算和交互式可视化，而无需用户具备丰富的 GTFS 或编程知识。生成代码的 LLM 完全由提示引导，无需微调或访问实际的 GTFS 信息源。我们使用 GPT-4o 和 Claude-3.5-Sonnet LLM 在 100 个任务的基准数据集上评估 TransitGPT，以证明其有效性和多功能性。结果表明，TransitGPT 可以显著提高交通数据的可访问性和可用性。

Title: Semantic loss guided data efficient supervised fine tuning for Safe Responses in LLMs

• Authors: Yuxiao Lu, Arunesh Sinha, Pradeep Varakantham
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06843
• Pdf URL: https://arxiv.org/pdf/2412.06843
• Copy Paste: [[2412.06843]] Semantic loss guided data efficient supervised fine tuning for Safe Responses in LLMs(https://arxiv.org/abs/2412.06843)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.
• 摘要：大型语言模型 (LLM) 对有毒提示生成不安全响应是其应用中的一个重大问题。虽然各种努力都旨在解决这一安全问题，但以前的方法通常需要大量的人工数据收集，或者依赖于使用另一个 LLM 来生成纠正数据的不太可靠的选项。在本文中，我们旨在解决这个问题并克服需要大量高质量人工数据的限制。我们的方法只需要一小组对有毒提示的不安全响应，这些响应很容易从不安全的 LLM 本身获得。通过采用语义成本与负地球移动距离 (EMD) 损失相结合，我们引导 LLM 避免生成不安全的响应。此外，我们提出了一种新的 EMD 损失下限，从而实现更高效的优化。与基线相比，我们的结果证明了卓越的性能和数据效率，并且我们进一步研究了使用对比数据时过度对齐和语言能力潜在下降的细微影响。

Title: Fully Open Source Moxin-7B Technical Report

• Authors: Pu Zhao, Xuan Shen, Zhenglun Kong, Yixin Shen, Sung-En Chang, Timothy Rupprecht, Lei Lu, Enfu Nan, Changdi Yang, Yumei He, Xingchen Xu, Yu Huang, Wei Wang, Yue Chen, Yong He, Yanzhi Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06845
• Pdf URL: https://arxiv.org/pdf/2412.06845
• Copy Paste: [[2412.06845]] Fully Open Source Moxin-7B Technical Report(https://arxiv.org/abs/2412.06845)
• Keywords: language model, gpt, llm
• Abstract: Recently, Large Language Models (LLMs) have undergone a significant transformation, marked by a rapid rise in both their popularity and capabilities. Leading this evolution are proprietary LLMs like GPT-4 and GPT-o1, which have captured widespread attention in the AI community due to their remarkable performance and versatility. Simultaneously, open-source LLMs, such as LLaMA and Mistral, have made great contributions to the ever-increasing popularity of LLMs due to the ease to customize and deploy the models across diverse applications. Although open-source LLMs present unprecedented opportunities for innovation and research, the commercialization of LLMs has raised concerns about transparency, reproducibility, and safety. Many open-source LLMs fail to meet fundamental transparency requirements by withholding essential components like training code and data, and some use restrictive licenses whilst claiming to be "open-source," which may hinder further innovations on LLMs. To mitigate this issue, we introduce Moxin 7B, a fully open-source LLM developed in accordance with the Model Openness Framework (MOF), a ranked classification system that evaluates AI models based on model completeness and openness, adhering to principles of open science, open source, open data, and open access. Our model achieves the highest MOF classification level of "open science" through the comprehensive release of pre-training code and configurations, training and fine-tuning datasets, and intermediate and final checkpoints. Experiments show that our model achieves superior performance in zero-shot evaluation compared with popular 7B models and performs competitively in few-shot evaluation.
• 摘要：最近，大型语言模型 (LLM) 经历了重大变革，其受欢迎程度和功能都迅速提升。引领这一变革的是 GPT-4 和 GPT-o1 等专有 LLM，它们凭借出色的性能和多功能性在 AI 社区引起了广泛关注。同时，LLaMA 和 Mistral 等开源 LLM 也为 LLM 日益普及做出了巨大贡献，因为它们易于定制和部署模型到各种应用程序中。尽管开源 LLM 为创新和研究提供了前所未有的机会，但 LLM 的商业化引发了人们对透明度、可重复性和安全性的担忧。许多开源 LLM 未能满足基本的透明度要求，因为它们隐瞒了训练代码和数据等基本组件，有些开源 LLM 使用限制性许可证，同时声称是“开源”，这可能会阻碍 LLM 的进一步创新。为了缓解这一问题，我们推出了 Moxin 7B，这是一个完全开源的 LLM，它遵循模型开放框架 (MOF) 开发，MOF 是一个基于模型完整性和开放性评估 AI 模型的排名分类系统，遵循开放科学、开源、开放数据和开放访问的原则。我们的模型通过全面发布预训练代码和配置、训练和微调数据集以及中间和最终检查点，达到了 MOF 分类的最高级别“开放科学”。实验表明，与流行的 7B 模型相比，我们的模型在零样本评估中取得了优异的性能，在少样本评估中也表现出色。

Title: Political-LLM: Large Language Models in Political Science

• Authors: Lincan Li, Jiaqi Li, Catherine Chen, Fred Gui, Hongjia Yang, Chenxiao Yu, Zhengguang Wang, Jianing Cai, Junlong Aaron Zhou, Bolin Shen, Alex Qian, Weixin Chen, Zhongkai Xue, Lichao Sun, Lifang He, Hanjie Chen, Kaize Ding, Zijian Du, Fangzhou Mu, Jiaxin Pei, Jieyu Zhao, Swabha Swayamdipta, Willie Neiswanger, Hua Wei, Xiyang Hu, Shixiang Zhu, Tianlong Chen, Yingzhou Lu, Yang Shi, Lianhui Qin, Tianfan Fu, Zhengzhong Tu, Yuzhe Yang, Jaemin Yoo, Jiaheng Zhang, Ryan Rossi, Liang Zhan, Liang Zhao, Emilio Ferrara, Yan Liu, Furong Huang, Xiangliang Zhang, Lawrence Rothenberg, Shuiwang Ji, Philip S. Yu, Yue Zhao, Yushun Dong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06864
• Pdf URL: https://arxiv.org/pdf/2412.06864
• Copy Paste: [[2412.06864]] Political-LLM: Large Language Models in Political Science(https://arxiv.org/abs/2412.06864)
• Keywords: language model, llm
• Abstract: In recent years, large language models (LLMs) have been widely adopted in political science tasks such as election prediction, sentiment analysis, policy impact assessment, and misinformation detection. Meanwhile, the need to systematically understand how LLMs can further revolutionize the field also becomes urgent. In this work, we--a multidisciplinary team of researchers spanning computer science and political science--present the first principled framework termed Political-LLM to advance the comprehensive understanding of integrating LLMs into computational political science. Specifically, we first introduce a fundamental taxonomy classifying the existing explorations into two perspectives: political science and computational methodologies. In particular, from the political science perspective, we highlight the role of LLMs in automating predictive and generative tasks, simulating behavior dynamics, and improving causal inference through tools like counterfactual generation; from a computational perspective, we introduce advancements in data preparation, fine-tuning, and evaluation methods for LLMs that are tailored to political contexts. We identify key challenges and future directions, emphasizing the development of domain-specific datasets, addressing issues of bias and fairness, incorporating human expertise, and redefining evaluation criteria to align with the unique requirements of computational political science. Political-LLM seeks to serve as a guidebook for researchers to foster an informed, ethical, and impactful use of Artificial Intelligence in political science. Our online resource is available at: this http URL.
• 摘要：近年来，大型语言模型 (LLM) 已广泛应用于选举预测、情绪分析、政策影响评估和错误信息检测等政治科学任务。同时，系统地了解 LLM 如何进一步彻底改变该领域的需求也变得迫切。在这项工作中，我们——一个横跨计算机科学和政治学的多学科研究团队——提出了第一个称为政治 LLM 的原则框架，以增进对将 LLM 融入计算政治科学的全面理解。具体来说，我们首先介绍一个基本分类法，将现有的探索分为两个视角：政治学和计算方法。特别是，从政治学的角度来看，我们强调了 LLM 在自动化预测和生成任务、模拟行为动态以及通过反事实生成等工具改进因果推理方面的作用；从计算的角度来看，我们介绍了针对政治背景的 LLM 的数据准备、微调和评估方法的进步。我们确定了关键挑战和未来方向，强调开发特定领域的数据集，解决偏见和公平问题，融入人类专业知识，并重新定义评估标准以符合计算政治科学的独特要求。Political-LLM 旨在成为研究人员的指南，以促进人工智能在政治科学中的明智、合乎道德和有影响力的使用。我们的在线资源位于：此 http URL。

Title: LLMs for Generalizable Language-Conditioned Policy Learning under Minimal Data Requirements

• Authors: Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06877
• Pdf URL: https://arxiv.org/pdf/2412.06877
• Copy Paste: [[2412.06877]] LLMs for Generalizable Language-Conditioned Policy Learning under Minimal Data Requirements(https://arxiv.org/abs/2412.06877)
• Keywords: language model, llm, agent
• Abstract: To develop autonomous agents capable of executing complex, multi-step decision-making tasks as specified by humans in natural language, existing reinforcement learning approaches typically require expensive labeled datasets or access to real-time experimentation. Moreover, conventional methods often face difficulties in generalizing to unseen goals and states, thereby limiting their practical applicability. This paper presents TEDUO, a novel training pipeline for offline language-conditioned policy learning. TEDUO operates on easy-to-obtain, unlabeled datasets and is suited for the so-called in-the-wild evaluation, wherein the agent encounters previously unseen goals and states. To address the challenges posed by such data and evaluation settings, our method leverages the prior knowledge and instruction-following capabilities of large language models (LLMs) to enhance the fidelity of pre-collected offline data and enable flexible generalization to new goals and states. Empirical results demonstrate that the dual role of LLMs in our framework-as data enhancers and generalizers-facilitates both effective and data-efficient learning of generalizable language-conditioned policies.
• 摘要：为了开发能够执行人类用自然语言指定的复杂、多步骤决策任务的自主代理，现有的强化学习方法通​​常需要昂贵的标记数据集或实时实验。此外，传统方法在推广到未见过的目标和状态时往往面临困难，从而限制了它们的实际适用性。本文介绍了 TEDUO，一种用于离线语言条件策略学习的新型训练管道。TEDUO 在易于获取的未标记数据集上运行，适用于所谓的野外评估，其中代理会遇到以前未见过的目标和状态。为了应对此类数据和评估设置带来的挑战，我们的方法利用大型语言模型 (LLM) 的先验知识和指令遵循能力来增强预先收集的离线数据的保真度，并能够灵活地推广到新的目标和状态。实证结果表明，LLM 在我们的框架中的双重角色——作为数据增强器和泛化器——促进了可泛化的语言条件策略的有效和数据高效学习。

Title: When Every Token Counts: Optimal Segmentation for Low-Resource Language Models

• Authors: Bharath Raj S, Garvit Suri, Vikrant Dewangan, Raghav Sonavane
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06926
• Pdf URL: https://arxiv.org/pdf/2412.06926
• Copy Paste: [[2412.06926]] When Every Token Counts: Optimal Segmentation for Low-Resource Language Models(https://arxiv.org/abs/2412.06926)
• Keywords: language model
• Abstract: Traditional greedy tokenization methods have been a critical step in Natural Language Processing (NLP), influencing how text is converted into tokens and directly impacting model performance. While subword tokenizers like Byte-Pair Encoding (BPE) are widely used, questions remain about their optimality across model scales and languages. In this work, we demonstrate through extensive experiments that an optimal BPE configuration significantly reduces token count compared to greedy segmentation, yielding improvements in token-saving percentages and performance benefits, particularly for smaller models. We evaluate tokenization performance across various intrinsic and extrinsic tasks, including generation and classification. Our findings suggest that compression-optimized tokenization strategies could provide substantial advantages for multilingual and low-resource language applications, highlighting a promising direction for further research and inclusive NLP.
• 摘要：传统的贪婪标记化方法是自然语言处理 (NLP) 中的关键步骤，影响文本转换为标记的方式并直接影响模型性能。虽然字节对编码 (BPE) 等子词标记器被广泛使用，但它们在模型规模和语言中的最优性仍然存在疑问。在这项工作中，我们通过大量实验证明，与贪婪分割相比，最佳 BPE 配置显著减少了标记数量，从而提高了标记节省百分比和性能优势，尤其是对于较小的模型。我们评估了各种内在和外在任务（包括生成和分类）中的标记化性能。我们的研究结果表明，压缩优化的标记化策略可以为多语言和低资源语言应用程序提供显着优势，为进一步的研究和包容性 NLP 指明了一个有希望的方向。

Title: Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning

• Authors: Yingyi Ma, Zhe Liu, Ozlem Kalinli
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.06967
• Pdf URL: https://arxiv.org/pdf/2412.06967
• Copy Paste: [[2412.06967]] Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning(https://arxiv.org/abs/2412.06967)
• Keywords: language model, llm, prompt
• Abstract: The advent of Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR). Prompting LLM with audio embeddings to generate transcriptions becomes the new state-of-the-art ASR. Despite LLMs being trained with an extensive amount of text corpora, high-quality domain-specific text data can still significantly enhance ASR performance on domain adaptation tasks. Although LLM-based ASR can naturally incorporate more text corpora by fine-tuning the LLM decoder, fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge. To mitigate this issue, we propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation. Experimental results show that text adaptation with our proposed method achieved a relative up to 9% Word Error Rate (WER) reduction and up to 18% Entity Error Rate (EER) reduction on the target domain compared to the baseline ASR. Combining this with domain-specific Language Model (LM) fusion can further improve the EER by a relative 2-5%
• 摘要：大型语言模型 (LLM) 的出现改革了自动语音识别 (ASR)。使用音频嵌入提示 LLM 生成转录成为新的最先进的 ASR。尽管 LLM 经过大量文本语料库的训练，但高质量的领域特定文本数据仍然可以显著提高领域自适应任务上的 ASR 性能。虽然基于 LLM 的 ASR 可以通过微调 LLM 解码器自然地合并更多文本语料库，但在没有配对提示的情况下对纯文本数据进行此类 ASR 微调可能会降低领域特定知识的有效性。为了缓解这个问题，我们提出了一种两步软提示微调策略，以增强领域特定文本自适应。实验结果表明，与基线 ASR 相比，使用我们提出的方法进行的文本自适应在目标领域上实现了相对高达 9% 的字错误率 (WER) 降低和高达 18% 的实体错误率 (EER) 降低。将其与领域特定语言模型 (LM) 融合相结合，可以进一步将 EER 提高 2-5%

Title: AutoReason: Automatic Few-Shot Reasoning Decomposition

• Authors: Arda Sevinc, Abdurrahman Gumus
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06975
• Pdf URL: https://arxiv.org/pdf/2412.06975
• Copy Paste: [[2412.06975]] AutoReason: Automatic Few-Shot Reasoning Decomposition(https://arxiv.org/abs/2412.06975)
• Keywords: language model, llm, prompt
• Abstract: Chain of Thought (CoT) was introduced in recent research as a method for improving step-by-step reasoning in Large Language Models. However, CoT has limited applications such as its need for hand-crafted few-shot exemplar prompts and no capability to adjust itself to different queries. In this work, we propose a system to automatically generate rationales using CoT. Our method improves multi-step implicit reasoning capabilities by decomposing the implicit query into several explicit questions. This provides interpretability for the model, improving reasoning in weaker LLMs. We test our approach with two Q\&A datasets: StrategyQA and HotpotQA. We show an increase in accuracy with both, especially on StrategyQA. To facilitate further research in this field, the complete source code for this study has been made publicly available on GitHub: this https URL.
• 摘要：思路链 (CoT) 是近期研究引入的一种方法，用于改进大型语言模型中的逐步推理。然而，CoT 的应用有限，例如它需要手工制作少量样本提示，并且无法根据不同的查询进行调整。在这项工作中，我们提出了一个使用 CoT 自动生成原理的系统。我们的方法通过将隐式查询分解为几个显式问题来提高多步骤隐式推理能力。这为模型提供了可解释性，从而提高了较弱的 LLM 中的推理能力。我们使用两个问答数据集测试了我们的方法：StrategyQA 和 HotpotQA。我们发现两者的准确率都有所提高，尤其是在 StrategyQA 上。为了促进该领域的进一步研究，本研究的完整源代码已在 GitHub 上公开：此 https URL。

Title: Asynchronous LLM Function Calling

• Authors: In Gim, Seung-seob Lee, Lin Zhong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07017
• Pdf URL: https://arxiv.org/pdf/2412.07017
• Copy Paste: [[2412.07017]] Asynchronous LLM Function Calling(https://arxiv.org/abs/2412.07017)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) use function calls to interface with external tools and data source. However, the current approach to LLM function calling is inherently synchronous, where each call blocks LLM inference, limiting LLM operation and concurrent function execution. In this work, we propose AsyncLM, a system for asynchronous LLM function calling. AsyncLM improves LLM's operational efficiency by enabling LLMs to generate and execute function calls concurrently. Instead of waiting for each call's completion, AsyncLM introduces an interrupt mechanism to asynchronously notify the LLM in-flight when function calls return. We design an in-context protocol for function calls and interrupts, provide fine-tuning strategy to adapt LLMs to the interrupt semantics, and implement these mechanisms efficiently on LLM inference process. We demonstrate that AsyncLM can reduce end-to-end task completion latency from 1.6x-5.4x compared to synchronous function calling on a set of benchmark tasks in the Berkeley function calling leaderboard (BFCL). Furthermore, we discuss how interrupt mechanisms can be extended to enable novel human-LLM or LLM-LLM interactions.
• 摘要：大型语言模型 (LLM) 使用函数调用与外部工具和数据源交互。然而，当前的 LLM 函数调用方法本质上是同步的，其中每个调用都会阻止 LLM 推理，从而限制 LLM 操作和并发函数执行。在这项工作中，我们提出了 AsyncLM，一种用于异步 LLM 函数调用的系统。AsyncLM 通过使 LLM 能够并发生成和执行函数调用来提高 LLM 的运行效率。AsyncLM 无需等待每个调用完成，而是引入了一种中断机制，以便在函数调用返回时异步通知正在运行的 LLM。我们为函数调用和中断设计了一个上下文协议，提供了微调策略以使 LLM 适应中断语义，并在 LLM 推理过程中有效地实现了这些机制。我们证明，与伯克利函数调用排行榜 (BFCL) 中的一组基准任务上的同步函数调用相比，AsyncLM 可以将端到端任务完成延迟从 1.6 倍减少到 5.4 倍。此外，我们讨论了如何扩展中断机制以实现新颖的人与 LLM 或 LLM 与 LLM 之间的交互。

Title: Assessing the Impact of Conspiracy Theories Using Large Language Models

• Authors: Bohan Jiang, Dawei Li, Zhen Tan, Xinyi Zhou, Ashwin Rao, Kristina Lerman, H. Russell Bernard, Huan Liu
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.07019
• Pdf URL: https://arxiv.org/pdf/2412.07019
• Copy Paste: [[2412.07019]] Assessing the Impact of Conspiracy Theories Using Large Language Models(https://arxiv.org/abs/2412.07019)
• Keywords: language model, llm, prompt
• Abstract: Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs.
• 摘要：衡量 CT 的相对影响对于确定响应的优先次序和有效分配资源非常重要，尤其是在危机期间。然而，评估 CT 对公众的实际影响带来了独特的挑战。它不仅需要收集 CT 特定的知识，还需要收集来自社会、心理和文化维度的各种信息。大型语言模型 (LLM) 的最新进展表明它们在这种情况下具有潜在的实用性，这不仅是因为它们从大型训练语料库中获得了广泛的知识，还因为它们可以用于复杂的推理。在这项工作中，我们开发了具有人工注释影响的流行 CT 数据集。借鉴人类影响评估过程的见解，我们设计了量身定制的策略来利用 LLM 进行类似人类的 CT 影响评估。通过严格的实验，我们发现使用多步推理来批判性地分析更多与 CT 相关的证据的影响评估模式会产生准确的结果；大多数 LLM 都表现出强烈的偏见，例如将更高的影响分配给提示中较早出现的 CT，而对情绪化和冗长的 CT 产生的影响评估则不太准确。

Title: Defensive Dual Masking for Robust Adversarial Defense

• Authors: Wangli Yang, Jie Yang, Yi Guo, Johan Barthelemy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07078
• Pdf URL: https://arxiv.org/pdf/2412.07078
• Copy Paste: [[2412.07078]] Defensive Dual Masking for Robust Adversarial Defense(https://arxiv.org/abs/2412.07078)
• Keywords: language model, llm
• Abstract: The field of textual adversarial defenses has gained considerable attention in recent years due to the increasing vulnerability of natural language processing (NLP) models to adversarial attacks, which exploit subtle perturbations in input text to deceive models. This paper introduces the Defensive Dual Masking (DDM) algorithm, a novel approach designed to enhance model robustness against such attacks. DDM utilizes a unique adversarial training strategy where [MASK] tokens are strategically inserted into training samples to prepare the model to handle adversarial perturbations more effectively. During inference, potentially adversarial tokens are dynamically replaced with [MASK] tokens to neutralize potential threats while preserving the core semantics of the input. The theoretical foundation of our approach is explored, demonstrating how the selective masking mechanism strengthens the model's ability to identify and mitigate adversarial manipulations. Our empirical evaluation across a diverse set of benchmark datasets and attack mechanisms consistently shows that DDM outperforms state-of-the-art defense techniques, improving model accuracy and robustness. Moreover, when applied to Large Language Models (LLMs), DDM also enhances their resilience to adversarial attacks, providing a scalable defense mechanism for large-scale NLP applications.
• 摘要：近年来，由于自然语言处理 (NLP) 模型越来越容易受到对抗性攻击，文本对抗性防御领域受到了广泛关注，这种攻击利用输入文本中的细微扰动来欺骗模型。本文介绍了防御性双重掩蔽 (DDM) 算法，这是一种旨在增强模型对此类攻击的鲁棒性的新方法。DDM 采用独特的对抗性训练策略，其中 [MASK] 标记被策略性地插入训练样本中，以使模型能够更有效地处理对抗性扰动。在推理过程中，潜在的对抗性标记被动态替换为 [MASK] 标记，以消除潜在威胁，同时保留输入的核心语义。我们探索了方法的理论基础，展示了选择性掩蔽机制如何增强模型识别和减轻对抗性操纵的能力。我们对各种基准数据集和攻击机制进行的经验评估一致表明，DDM 优于最先进的防御技术，提高了模型的准确性和鲁棒性。此外，当应用于大型语言模型 (LLM) 时，DDM 还增强了其对对抗性攻击的抵御能力，为大规模 NLP 应用提供了可扩展的防御机制。

Title: Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need

• Authors: Bo-Wen Zhang, Yan Yan, Boxiang Yang, Yifei Xue, Guang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07111
• Pdf URL: https://arxiv.org/pdf/2412.07111
• Copy Paste: [[2412.07111]] Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need(https://arxiv.org/abs/2412.07111)
• Keywords: language model, llm
• Abstract: While scaling laws optimize training configurations for large language models (LLMs) through experiments on smaller or early-stage models, they fail to predict emergent abilities due to the absence of such capabilities in these models. To address this, we propose a method that predicts emergent abilities by leveraging proxy tasks. We begin by establishing relevance metrics between the target task and candidate tasks based on performance differences across multiple models. These candidate tasks are then validated for robustness with small model ensembles, leading to the selection of the most appropriate proxy tasks. The predicted performance on the target task is then derived by integrating the evaluation results of these proxies. In a case study on tool utilization capabilities, our method demonstrated a strong correlation between predicted and actual performance, confirming its effectiveness.
• 摘要：尽管缩放定律可以通过在较小或早期模型上进行实验来优化大型语言模型 (LLM) 的训练配置，但它们无法预测新兴能力，因为这些模型缺乏此类能力。为了解决这个问题，我们提出了一种利用代理任务来预测新兴能力的方法。我们首先根据多个模型之间的性能差异建立目标任务和候选任务之间的相关性指标。然后使用小型模型集成验证这些候选任务的稳健性，从而选择最合适的代理任务。然后通过整合这些代理的评估结果来得出目标任务的预测性能。在工具利用能力的案例研究中，我们的方法证明了预测性能和实际性能之间的很强的相关性，证实了它的有效性。

Title: Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance

• Authors: Dongjun Kim, Minhyuk Kim, YongChan Chun, Chanjun Park, Heuiseok Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07113
• Pdf URL: https://arxiv.org/pdf/2412.07113
• Copy Paste: [[2412.07113]] Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance(https://arxiv.org/abs/2412.07113)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated notable proficiency in both code generation and comprehension across multiple programming languages. However, the mechanisms underlying this proficiency remain underexplored, particularly with respect to whether distinct programming languages are processed independently or within a shared parametric region. Drawing an analogy to the specialized regions of the brain responsible for distinct cognitive functions, we introduce the concept of Coding Spot, a specialized parametric region within LLMs that facilitates coding capabilities. Our findings identify this Coding Spot and show that targeted modifications to this subset significantly affect performance on coding tasks, while largely preserving non-coding functionalities. This compartmentalization mirrors the functional specialization observed in cognitive neuroscience, where specific brain regions are dedicated to distinct tasks, suggesting that LLMs may similarly employ specialized parameter regions for different knowledge domains.
• 摘要：大型语言模型 (LLM) 已证明在多种编程语言的代码生成和理解方面具有显著的能力。然而，这种能力背后的机制仍未得到充分探索，特别是关于不同的编程语言是独立处理还是在共享参数区域内处理。与负责不同认知功能的大脑专门区域类似，我们引入了编码点的概念，这是 LLM 内促进编码能力的专门参数区域。我们的研究结果确定了这个编码点，并表明针对这个子集的修改会显著影响编码任务的性能，同时在很大程度上保留非编码功能。这种划分反映了认知神经科学中观察到的功能专业化，其中特定的大脑区域专用于不同的任务，这表明 LLM 可能同样为不同的知识领域采用专门的参数区域。

Title: Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models

• Authors: Haoran Lian, Junmin Chen, Wei Huang, Yizhe Xiong, Wenping Hu, Guiguang Ding, Hui Chen, Jianwei Niu, Zijia Lin, Fuzheng Zhang, Di Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07171
• Pdf URL: https://arxiv.org/pdf/2412.07171
• Copy Paste: [[2412.07171]] Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models(https://arxiv.org/abs/2412.07171)
• Keywords: language model, llm, long context
• Abstract: Recently, Large language models (LLMs) have revolutionized Natural Language Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with handling long token sequences, limiting their performance on various downstream tasks. Current solutions toward long context modeling often employ multi-stage continual pertaining, which progressively increases the effective context length through several continual pretraining stages. However, those approaches require extensive manual tuning and human expertise. In this paper, we introduce a novel single-stage continual pretraining method, Head-Adaptive Rotary Position Encoding (HARPE), to equip LLMs with long context modeling capabilities while simplifying the training process. Our HARPE leverages different Rotary Position Encoding (RoPE) base frequency values across different attention heads and directly trains LLMs on the target context length. Extensive experiments on 4 language modeling benchmarks, including the latest RULER benchmark, demonstrate that HARPE excels in understanding and integrating long-context tasks with single-stage training, matching and even outperforming existing multi-stage methods. Our results highlight that HARPE successfully breaks the stage barrier for training LLMs with long context modeling capabilities.
• 摘要：最近，大型语言模型 (LLM) 彻底改变了自然语言处理 (NLP)。由于训练上下文大小有限，预训练的 LLM 难以处理长标记序列，从而限制了它们在各种下游任务上的性能。当前针对长上下文建模的解决方案通常采用多阶段连续关联，通过几个连续预训练阶段逐步增加有效上下文长度。然而，这些方法需要大量的手动调整和人类专业知识。在本文中，我们介绍了一种新颖的单阶段连续预训练方法，头部自适应旋转位置编码 (HARPE)，为 LLM 配备长上下文建模功能，同时简化训练过程。我们的 HARPE 利用不同注意头之间的不同旋转位置编码 (RoPE) 基频值，并直接在目标上下文长度上训练 LLM。在包括最新 RULER 基准在内的 4 个语言建模基准上进行的大量实验表明，HARPE 在理解和整合长上下文任务与单阶段训练方面表现出色，与现有的多阶段方法相当甚至优于它们。我们的结果表明，HARPE 成功突破了训练具有长上下文建模能力的 LLM 的阶段障碍。

Title: A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis

• Authors: Nimisha Ghosh, Daniele Santoni, Indrajit Saha, Giovanni Felici
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07201
• Pdf URL: https://arxiv.org/pdf/2412.07201
• Copy Paste: [[2412.07201]] A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis(https://arxiv.org/abs/2412.07201)
• Keywords: language model
• Abstract: In recent times, Transformer-based language models are making quite an impact in the field of natural language processing. As relevant parallels can be drawn between biological sequences and natural languages, the models used in NLP can be easily extended and adapted for various applications in bioinformatics. In this regard, this paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We have reviewed and analysed a large number of application-based papers on this subject, giving evidence of the main characterizing features and to different approaches that may be adopted to customize such powerful computational machines. We have also provided a structured description of the functioning of Transformers, that may enable even first time users to grab the essence of such complex architectures. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences. This work will motivate the readers to build on these methodologies to tackle also various other problems in the field of bioinformatics.
• 摘要：近年来，基于 Transformer 的语言模型在自然语言处理领域产生了相当大的影响。由于生物序列和自然语言之间存在相关的相似之处，因此 NLP 中使用的模型可以轻松扩展并适应生物信息学的各种应用。在这方面，本文介绍了基于 Transformer 的模型在核苷酸序列背景下的近期主要发展。我们回顾并分析了大量关于这个主题的应用论文，提供了主要特征的证据，以及可以采用的不同方法来定制这种强大的计算机器。我们还提供了 Transformer 功能的结构化描述，即使是初次使用者也可以掌握这种复杂架构的本质。我们相信这篇评论将有助于科学界了解基于 Transformer 的语言模型在核苷酸序列中的各种应用。这项工作将激励读者在这些方法的基础上解决生物信息学领域的各种其他问题。

Title: Speaker effects in spoken language comprehension

• Authors: Hanlin Wu, Zhenguang G. Cai
• Subjects: cs.CL, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2412.07238
• Pdf URL: https://arxiv.org/pdf/2412.07238
• Copy Paste: [[2412.07238]] Speaker effects in spoken language comprehension(https://arxiv.org/abs/2412.07238)
• Keywords: agent
• Abstract: The identity of a speaker significantly influences spoken language comprehension by affecting both perception and expectation. This review explores speaker effects, focusing on how speaker information impacts language processing. We propose an integrative model featuring the interplay between bottom-up perception-based processes driven by acoustic details and top-down expectation-based processes driven by a speaker model. The acoustic details influence lower-level perception, while the speaker model modulates both lower-level and higher-level processes such as meaning interpretation and pragmatic inferences. We define speaker-idiosyncrasy and speaker-demographics effects and demonstrate how bottom-up and top-down processes interact at various levels in different scenarios. This framework contributes to psycholinguistic theory by offering a comprehensive account of how speaker information interacts with linguistic content to shape message construction. We suggest that speaker effects can serve as indices of a language learner's proficiency and an individual's characteristics of social cognition. We encourage future research to extend these findings to AI speakers, probing the universality of speaker effects across humans and artificial agents.
• 摘要：说话者的身份会显著影响口语理解，因为它会影响感知和期望。本综述探讨了说话者效应，重点研究了说话者信息如何影响语言处理。我们提出了一个综合模型，该模型的特点是自下而上的基于感知的过程（由声学细节驱动）与自上而下的基于期望的过程（由说话者模型驱动）之间的相互作用。声学细节影响较低级别的感知，而说话者模型则调节较低级别和较高级别的过程，例如意义解释和语用推理。我们定义了说话者特质和说话者人口统计效应，并展示了自下而上和自上而下的过程如何在不同场景中在各个层面上相互作用。该框架通过全面阐述说话者信息如何与语言内容相互作用以塑造信息构建，为心理语言学理论做出了贡献。我们认为说话者效应可以作为语言学习者熟练程度和个人社会认知特征的指标。我们鼓励未来的研究将这些发现扩展到人工智能说话者，探索说话者效应在人类和人工智能体之间的普遍性。

Title: Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs without Real Data Replay

• Authors: Ruiheng Liu, Jinyu Zhang, Yanqi Song, Yu Zhang, Bailong Yang
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.07246
• Pdf URL: https://arxiv.org/pdf/2412.07246
• Copy Paste: [[2412.07246]] Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs without Real Data Replay(https://arxiv.org/abs/2412.07246)
• Keywords: language model, llm
• Abstract: Continual Semantic Parsing (CSP) aims to train parsers to convert natural language questions into SQL across tasks with limited annotated examples, adapting to the real-world scenario of dynamically updated databases. Previous studies mitigate this challenge by replaying historical data or employing parameter-efficient tuning (PET), but they often violate data privacy or rely on ideal continual learning settings. To address these problems, we propose a new Large Language Model (LLM)-Enhanced Continuous Semantic Parsing method, named LECSP, which alleviates forgetting while encouraging generalization, without requiring real data replay or ideal settings. Specifically, it first analyzes the commonalities and differences between tasks from the SQL syntax perspective to guide LLMs in reconstructing key memories and improving memory accuracy through a calibration strategy. Then, it uses a task-aware dual-teacher distillation framework to promote the accumulation and transfer of knowledge during sequential training. Experimental results on two CSP benchmarks show that our method significantly outperforms existing methods, even those utilizing data replay or ideal settings. Additionally, we achieve generalization performance beyond the upper limits, better adapting to unseen tasks.
• 摘要：持续语义解析 (CSP) 旨在训练解析器在有限的带注释示例的情况下跨任务将自然语言问题转换为 SQL，以适应动态更新数据库的实际场景。先前的研究通过重放历史数据或采用参数高效调整 (PET) 来缓解这一挑战，但它们往往违反数据隐私或依赖于理想的持续学习设置。为了解决这些问题，我们提出了一种新的大型语言模型 (LLM) 增强型持续语义解析方法，称为 LECSP，它在鼓励泛化的同时减轻遗忘，而无需真实数据重放或理想设置。具体来说，它首先从 SQL 语法的角度分析任务之间的共性和差异，以指导 LLM 重建关键记忆并通过校准策略提高记忆准确性。然后，它使用任务感知的双教师蒸馏框架来促进顺序训练过程中知识的积累和转移。在两个 CSP 基准上的实验结果表明，我们的方法明显优于现有方法，即使是那些使用数据重放或理想设置的方法。此外，我们实现了超越上限的泛化性能，更好地适应看不见的任务。

Title: KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context

• Authors: Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, Hansaem Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07251
• Pdf URL: https://arxiv.org/pdf/2412.07251
• Copy Paste: [[2412.07251]] KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context(https://arxiv.org/abs/2412.07251)
• Keywords: language model
• Abstract: Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increases as these models generate more fluent and coherent content. Current multilingual benchmarks often use translated English versions, which may incorporate Western cultural biases that do not accurately assess other languages and cultures. To address this research gap, we introduce KULTURE Bench, an evaluation framework specifically designed for Korean culture that features datasets of cultural news, idioms, and poetry. It is designed to assess language models' cultural comprehension and reasoning capabilities at the word, sentence, and paragraph levels. Using the KULTURE Bench, we assessed the capabilities of models trained with different language corpora and analyzed the results comprehensively. The results show that there is still significant room for improvement in the models' understanding of texts related to the deeper aspects of Korean culture.
• 摘要：大型语言模型在各种任务中都表现出了显著的性能提升。然而，随着这些模型生成的内容更加流畅和连贯，评估的复杂性也随之增加。当前的多语言基准测试通常使用翻译后的英文版本，这可能包含西方文化偏见，无法准确评估其他语言和文化。为了弥补这一研究空白，我们推出了 KULTURE Bench，这是一个专为韩国文化设计的评估框架，其中包含文化新闻、成语和诗歌的数据集。它旨在评估语言模型在单词、句子和段落级别的文化理解和推理能力。使用 KULTURE Bench，我们评估了使用不同语言语料库训练的模型的能力，并对结果进行了全面分析。结果表明，模型在理解与韩国文化深层方面相关的文本方面仍有很大改进空间。

Title: Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation

• Authors: Qinhong Lin, Linna Zhou, Zhongliang Yang, Yuang Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07255
• Pdf URL: https://arxiv.org/pdf/2412.07255
• Copy Paste: [[2412.07255]] Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation(https://arxiv.org/abs/2412.07255)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) display formidable capabilities in generative tasks but also pose potential risks due to their tendency to generate hallucinatory responses. Uncertainty Quantification (UQ), the evaluation of model output reliability, is crucial for ensuring the safety and robustness of AI systems. Recent studies have concentrated on model uncertainty by analyzing the relationship between output entropy under various sampling conditions and the corresponding labels. However, these methods primarily focus on measuring model entropy with precision to capture response characteristics, often neglecting the uncertainties associated with greedy decoding results-the sources of model labels, which can lead to biased classification outcomes. In this paper, we explore the biases introduced by greedy decoding and propose a label-confidence-aware (LCA) uncertainty estimation based on Kullback-Leibler (KL) divergence bridging between samples and label source, thus enhancing the reliability and stability of uncertainty assessments. Our empirical evaluations across a range of popular LLMs and NLP datasets reveal that different label sources can indeed affect classification, and that our approach can effectively capture differences in sampling results and label sources, demonstrating more effective uncertainty estimation.
• 摘要：大型语言模型 (LLM) 在生成任务中表现出强大的能力，但也因其容易产生幻觉反应而带来潜在风险。不确定性量化 (UQ) 是对模型输出可靠性的评估，对于确保人工智能系统的安全性和鲁棒性至关重要。最近的研究集中在模型不确定性上，通过分析不同采样条件下的输出熵与相应标签之间的关系。然而，这些方法主要侧重于精确测量模型熵以捕捉响应特征，往往忽略了与贪婪解码结果相关的不确定性——模型标签的来源，这可能导致有偏差的分类结果。在本文中，我们探讨了贪婪解码引入的偏差，并提出了一种基于 Kullback-Leibler (KL) 散度的标签置信度感知 (LCA) 不确定性估计，在样本和标签源之间架起了桥梁，从而提高了不确定性评估的可靠性和稳定性。我们对一系列流行的 LLM 和 NLP 数据集进行的经验评估表明，不同的标签源确实会影响分类，并且我们的方法可以有效地捕捉采样结果和标签源的差异，从而展示更有效的不确定性估计。

Title: HARP: Hesitation-Aware Reframing in Transformer Inference Pass

• Authors: Romain Storaï, Seung-won Hwang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07282
• Pdf URL: https://arxiv.org/pdf/2412.07282
• Copy Paste: [[2412.07282]] HARP: Hesitation-Aware Reframing in Transformer Inference Pass(https://arxiv.org/abs/2412.07282)
• Keywords: language model
• Abstract: This paper aims to improve the performance of large language models by addressing the variable computational demands in inference steps, where some tokens require more computational resources than others. We present HARP, a simple modification to "off-the-shelf" Transformer forward pass. Drawing from hesitation and the framing effect in decision-making, HARP selectively applies additional computation when the model encounters uncertainty during token generation. Our method mimics human cognitive processes by pausing at difficult decision points and reframing inputs for a different perspective. Unlike other approaches, HARP is model-agnostic, training-free, and easy to implement. We thoroughly evaluate our method across various downstream tasks and model sizes, demonstrating performance improvements up to +5.16%. Notably, HARP achieves these gains while maintaining inference times twice faster than beam search. Simple and yet with significant gains, HARP offers a practical solution for enhancing the performance of Transformer-based language models with minimal computational impact.
• 摘要：本文旨在通过解决推理步骤中可变的计算需求来提高大型语言模型的性能，其中某些标记比其他标记需要更多的计算资源。我们提出了 HARP，这是对“现成的”Transformer 前向传递的简单修改。借鉴决策中的犹豫和框架效应，HARP 在模型在标记生成过程中遇到不确定性时有选择地应用额外计算。我们的方法通过在困难的决策点暂停并从不同的角度重新构建输入来模仿人类的认知过程。与其他方法不同，HARP 与模型无关、无需训练且易于实施。我们在各种下游任务和模型大小中彻底评估了我们的方法，结果显示性能提升高达 +5.16%。值得注意的是，HARP 在实现这些增益的同时，保持了比集束搜索快两倍的推理时间。HARP 简单但增益显著，提供了一种实用的解决方案，可以以最小的计算影响来提高基于 Transformer 的语言模型的性能。

Title: Enhancing Relation Extraction via Supervised Rationale Verification and Feedback

• Authors: Yongqi Li, Xin Miao, Shen Zhou, Mayi Xu, Yuyang Ren, Tieyun Qian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07289
• Pdf URL: https://arxiv.org/pdf/2412.07289
• Copy Paste: [[2412.07289]] Enhancing Relation Extraction via Supervised Rationale Verification and Feedback(https://arxiv.org/abs/2412.07289)
• Keywords: language model, llm
• Abstract: Despite the rapid progress that existing automated feedback methods have made in correcting the output of large language models (LLMs), these methods cannot be well applied to the relation extraction (RE) task due to their designated feedback objectives and correction manner. To address this problem, we propose a novel automated feedback framework for RE, which presents a rationale supervisor to verify the rationale and provide re-selected demonstrations as feedback to correct the initial prediction. Specifically, we first design a causal intervention and observation method for to collect biased/unbiased rationales for contrastive training the rationale supervisor. Then, we present a verification-feedback-correction procedure to iteratively enhance LLMs' capability of handling the RE task. Extensive experiments prove that our proposed framework significantly outperforms existing methods.
• 摘要：尽管现有的自动反馈方法在纠正大型语言模型 (LLM) 的输出方面取得了快速进展，但由于其指定的反馈目标和纠正方式，这些方法不能很好地应用于关系提取 (RE) 任务。为了解决这个问题，我们提出了一种新颖的 RE 自动反馈框架，该框架提出了一个原理监督器来验证原理并提供重新选择的演示作为反馈来纠正初始预测。具体而言，我们首先设计一种因果干预和观察方法来收集有偏/无偏原理，以对比训练原理监督器。然后，我们提出一个验证-反馈-校正程序来迭代增强 LLM 处理 RE 任务的能力。大量实验证明我们提出的框架明显优于现有方法。

Title: The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model

• Authors: Jiawei Chen, Wentao Chen, Jing Su, Jingjing Xu, Hongyu Lin, Mengjie Ren, Yaojie Lu, Xianpei Han, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07298
• Pdf URL: https://arxiv.org/pdf/2412.07298
• Copy Paste: [[2412.07298]] The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model(https://arxiv.org/abs/2412.07298)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown significant multilingual capabilities. However, the mechanisms underlying the development of these capabilities during pre-training are not well understood. In this paper, we use code LLMs as an experimental platform to explore the evolution of multilingual capabilities in LLMs during the pre-training process. Based on our observations, we propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. During the learning process, multiple languages initially share a single knowledge system dominated by the primary language and gradually develop language-specific knowledge systems. We then validate the above hypothesis by tracking the internal states of the LLMs through identifying working languages and language transferring neurons. Experimental results show that the internal state changes of the LLM are consistent with our Babel Tower Hypothesis. Building on these insights, we propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs, which significantly outperforms LLMs trained on the original corpus. The proposed Babel Tower Hypothesis provides new insights into designing pre-training data distributions to achieve optimal multilingual capabilities in LLMs.
• 摘要：大型语言模型 (LLM) 已表现出显著的多语言能力。然而，在预训练过程中，这些能力发展的潜在机制尚不清楚。在本文中，我们使用代码 LLM 作为实验平台，探索 LLM 在预训练过程中多语言能力的演变。基于我们的观察，我们提出了巴别塔假说，该假说描述了 LLM 获得新语言能力的整个过程。在学习过程中，多种语言最初共享一个由主要语言主导的知识系统，并逐渐发展出特定于语言的知识系统。然后，我们通过识别工作语言和语言传输神经元来跟踪 LLM 的内部状态，从而验证上述假设。实验结果表明，LLM 的内部状态变化与我们的巴别塔假说一致。基于这些见解，我们提出了一种新方法来构建优化的多语言代码 LLM 预训练语料库，其性能明显优于在原始语料库上训练的 LLM。提出的巴别塔假说为设计预训练数据分布以实现 LLM 中的最佳多语言能力提供了新的见解。

Title: Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

• Authors: Lance Calvin Lim Gamboa, Mark Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07303
• Pdf URL: https://arxiv.org/pdf/2412.07303
• Copy Paste: [[2412.07303]] Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia(https://arxiv.org/abs/2412.07303)
• Keywords: language model
• Abstract: Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models.
• 摘要：多语言模型的偏见研究证实，在处理具有大量 NLP 资源的语言的掩蔽模型中存在与性别相关的刻板印象。我们通过引入菲律宾 CrowS-Pairs 和菲律宾 WinoQueer 扩展了这一研究方向：这两个基准用于评估处理菲律宾语（一种来自菲律宾的资源较少的语言）文本的预训练语言模型 (PLM) 中的性别歧视和反同性恋偏见。这些基准由 7,074 个新的挑战对组成，这些挑战对是我们对英语偏见评估数据集进行文化改编的结果，我们详细记录了这一过程以指导类似的未来努力。我们将菲律宾语基准应用于掩蔽和因果多语言模型，包括那些在东南亚数据上进行预训练的模型，发现它们包含大量偏见。我们还发现，对于多语言模型，针对特定语言学习到的偏见程度受模型接触该语言的预训练数据量的影响。我们的基准和见解可以作为未来分析和减轻多语言模型偏见的基础。

Title: Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation

• Authors: Pedro H. V. Valois, Lincon S. Souza, Erica K. Shimomoto, Kazuhiro Fukui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07334
• Pdf URL: https://arxiv.org/pdf/2412.07334
• Copy Paste: [[2412.07334]] Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation(https://arxiv.org/abs/2412.07334)
• Keywords: language model, llm
• Abstract: Interpretability is a key challenge in fostering trust for Large Language Models (LLMs), which stems from the complexity of extracting reasoning from model's parameters. We present the Frame Representation Hypothesis, a theoretically robust framework grounded in the Linear Representation Hypothesis (LRH) to interpret and control LLMs by modeling multi-token words. Prior research explored LRH to connect LLM representations with linguistic concepts, but was limited to single token analysis. As most words are composed of several tokens, we extend LRH to multi-token words, thereby enabling usage on any textual data with thousands of concepts. To this end, we propose words can be interpreted as frames, ordered sequences of vectors that better capture token-word relationships. Then, concepts can be represented as the average of word frames sharing a common concept. We showcase these tools through Top-k Concept-Guided Decoding, which can intuitively steer text generation using concepts of choice. We verify said ideas on Llama 3.1, Gemma 2, and Phi 3 families, demonstrating gender and language biases, exposing harmful content, but also potential to remediate them, leading to safer and more transparent LLMs. Code is available at this https URL
• 摘要：可解释性是培养大型语言模型 (LLM) 信任度的关键挑战，这源于从模型参数中提取推理的复杂性。我们提出了框架表示假设，这是一个理论上稳健的框架，以线性表示假设 (LRH) 为基础，通过对多标记词进行建模来解释和控制 LLM。先前的研究探索了 LRH 将 LLM 表示与语言概念联系起来，但仅限于单个标记分析。由于大多数单词由多个标记组成，我们将 LRH 扩展到多标记词，从而可以在任何具有数千个概念的文本数据上使用。为此，我们提出可以将单词解释为框架，即更好地捕捉标记词关系的有序向量序列。然后，概念可以表示为共享共同概念的词框架的平均值。我们通过 Top-k 概念引导解码展示这些工具，它可以直观地使用选择的概念来引导文本生成。我们在 Llama 3.1、Gemma 2 和 Phi 3 系列上验证了上述想法，展示了性别和语言偏见，揭露了有害内容，但也有补救的潜力，从而实现更安全、更透明的 LLM。代码可从此 https URL 获取

Title: My Words Imply Your Opinion: Reader Agent-Based Propagation Enhancement for Personalized Implicit Emotion Analysis

• Authors: Jian Liao, Yu Feng, Xiaoyu Wang, Suge Wang, Jianxing Zheng, Deyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07367
• Pdf URL: https://arxiv.org/pdf/2412.07367
• Copy Paste: [[2412.07367]] My Words Imply Your Opinion: Reader Agent-Based Propagation Enhancement for Personalized Implicit Emotion Analysis(https://arxiv.org/abs/2412.07367)
• Keywords: language model, agent
• Abstract: In implicit emotion analysis (IEA), the subtlety of emotional expressions makes it particularly sensitive to user-specific characteristics. Existing studies often inject personalization into the analysis by focusing on the authorial dimension of the emotional text. However, these methods overlook the potential influence of the intended reader on the reaction of implicit emotions. In this paper, we refine the IEA task to Personalized Implicit Emotion Analysis (PIEA) and introduce the RAPPIE model, a novel framework designed to address the issue of missing user information within this task. In particular, 1) we create reader agents based on the Large Language Model to simulate reader reactions, to address challenges of the spiral of silence and data incompleteness encountered when acquiring reader feedback information. 2) We establish a reader propagation role system and develop a role-aware emotion propagation multi-view graph learning model, which effectively deals with the sparsity of reader information by utilizing the distribution of propagation roles. 3) We annotate two Chinese PIEA datasets with detailed user metadata, thereby addressing the limitation of prior datasets that primarily focus on textual content annotation. Extensive experiments on these datasets indicate that the RAPPIE model outperforms current state-of-the-art baselines, highlighting the significance and efficacy of incorporating reader feedback into the PIEA process.
• 摘要：在隐性情感分析（IEA）中，情感表达的微妙性使其对用户特定特征特别敏感。现有研究通常通过关注情感文本的作者维度将个性化注入分析中。然而，这些方法忽略了目标读者对隐性情感反应的潜在影响。在本文中，我们将 IEA 任务细化为个性化隐性情感分析（PIEA），并引入 RAPPIE 模型，这是一个旨在解决该任务中缺少用户信息问题的新框架。具体而言，1）我们基于大型语言模型创建读者代理来模拟读者反应，以解决获取读者反馈信息时遇到的沉默螺旋和数据不完整的挑战。2）我们建立了读者传播角色系统并开发了角色感知的情感传播多视图图学习模型，该模型利用传播角色的分布有效地处理了读者信息的稀疏性。3）我们用详细的用户元数据注释了两个中文 PIEA 数据集，从而解决了以前主要关注文本内容注释的数据集的局限性。对这些数据集进行的大量实验表明，RAPPIE 模型优于当前最先进的基线，凸显了将读者反馈纳入 PIEA 流程的意义和有效性。

Title: SpecFuse: Ensembling Large Language Models via Next-Segment Prediction

• Authors: Bo Lv, Chen Tang, Yanan Zhang, Xin Liu, Yue Yu, Ping Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07380
• Pdf URL: https://arxiv.org/pdf/2412.07380
• Copy Paste: [[2412.07380]] SpecFuse: Ensembling Large Language Models via Next-Segment Prediction(https://arxiv.org/abs/2412.07380)
• Keywords: language model, llm
• Abstract: Ensembles of generative large language models (LLMs) can integrate the strengths of different LLMs to compensate for the limitations of individual models. However, recent work has focused on training an additional fusion model to combine complete responses from multiple LLMs, failing to tap into their collaborative potential to generate higher-quality responses. Moreover, as the additional fusion model is trained on a specialized dataset, these methods struggle with generalizing to open-domain queries from online users. In this paper, we propose SpecFuse, a novel ensemble framework that outputs the fused result by iteratively producing the next segment through collaboration among LLMs. This is achieved through cyclic execution of its inference and verification components. In each round, the inference component invokes each base LLM to generate candidate segments in parallel, and the verify component calls these LLMs again to predict the ranking of the segments. The top-ranked segment is then broadcast to all LLMs, encouraging them to generate higher-quality segments in the next round. This approach also allows the base LLMs to be plug-and-play, without any training or adaptation, avoiding generalization limitations. Furthermore, to conserve computational resources, we propose a model exit mechanism that dynamically excludes models exhibiting poor performance in previous rounds during each query response. In this way, it effectively reduces the number of model calls while maintaining overall performance.
• 摘要：生成式大型语言模型 (LLM) 的集成可以整合不同 LLM 的优势，以弥补单个模型的局限性。然而，最近的研究主要集中在训练一个额外的融合模型来组合来自多个 LLM 的完整响应，未能利用它们的协作潜力来生成更高质量的响应。此外，由于额外的融合模型是在专门的数据集上训练的，这些方法很难推广到在线用户的开放域查询。在本文中，我们提出了 SpecFuse，这是一个新颖的集成框架，它通过 LLM 之间的协作迭代生成下一个片段来输出融合结果。这是通过循环执行其推理和验证组件来实现的。在每一轮中，推理组件调用每个基础 LLM 来并行生成候选片段，验证组件再次调用这些 LLM 来预测片段的排名。然后，排名靠前的片段被广播给所有 LLM，鼓励它们在下一轮中生成更高质量的片段。这种方法还允许基础 LLM 即插即用，无需任何训练或调整，从而避免了泛化限制。此外，为了节省计算资源，我们提出了一种模型退出机制，该机制在每次查询响应期间动态排除前几轮表现不佳的模型。通过这种方式，它可以有效减少模型调用次数，同时保持整体性能。

Title: Algorithmic Phase Transitions in Language Models: A Mechanistic Case Study of Arithmetic

• Authors: Alan Sun, Ethan Sun, Warren Shepard
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07386
• Pdf URL: https://arxiv.org/pdf/2412.07386
• Copy Paste: [[2412.07386]] Algorithmic Phase Transitions in Language Models: A Mechanistic Case Study of Arithmetic(https://arxiv.org/abs/2412.07386)
• Keywords: language model
• Abstract: Zero-shot capabilities of large language models make them powerful tools for solving a range of tasks without explicit training. It remains unclear, however, how these models achieve such performance, or why they can zero-shot some tasks but not others. In this paper, we shed some light on this phenomenon by defining and investigating algorithmic stability in language models -- changes in problem-solving strategy employed by the model as a result of changes in task specification. We focus on a task where algorithmic stability is needed for generalization: two-operand arithmetic. Surprisingly, we find that Gemma-2-2b employs substantially different computational models on closely related subtasks, i.e. four-digit versus eight-digit addition. Our findings suggest that algorithmic instability may be a contributing factor to language models' poor zero-shot performance across certain logical reasoning tasks, as they struggle to abstract different problem-solving strategies and smoothly transition between them.
• 摘要：大型语言模型的零样本能力使其成为无需明确训练即可解决一系列任务的强大工具。然而，这些模型如何实现这样的性能，或者为什么它们可以对某些任务进行零样本处理，而对其他任务则不能，这一点仍不清楚。在本文中，我们通过定义和研究语言模型中的算法稳定性（由于任务规范的变化，模型所采用的解决问题的策略也发生了变化）来阐明这一现象。我们专注于一项需要算法稳定性才能泛化的任务：双操作数算术。令人惊讶的是，我们发现 Gemma-2-2b 在密切相关的子任务（即四位数和八位数加法）上采用了截然不同的计算模型。我们的研究结果表明，算法不稳定性可能是导致语言模型在某些逻辑推理任务中零样本性能不佳的一个因素，因为它们很难抽象出不同的解决问题的策略并在它们之间平稳过渡。

Title: CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models

• Authors: Dongfang Li, Zetian Sun, Xinshuo Hu, Baotian Hu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07393
• Pdf URL: https://arxiv.org/pdf/2412.07393
• Copy Paste: [[2412.07393]] CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models(https://arxiv.org/abs/2412.07393)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) need to adapt to the continuous changes in data, tasks, and user preferences. Due to their massive size and the high costs associated with training, LLMs are not suitable for frequent retraining. However, updates are necessary to keep them in sync with rapidly evolving human knowledge. To address these challenges, this paper proposes the Compression Memory Training (CMT) method, an efficient and effective online adaptation framework for LLMs that features robust knowledge retention capabilities. Inspired by human memory mechanisms, CMT compresses and extracts information from new documents to be stored in a memory bank. When answering to queries related to these new documents, the model aggregates these document memories from the memory bank to better answer user questions. The parameters of the LLM itself do not change during training and inference, reducing the risk of catastrophic forgetting. To enhance the encoding, retrieval, and aggregation of memory, we further propose three new general and flexible techniques, including memory-aware objective, self-matching and top-aggregation. Extensive experiments conducted on three continual learning datasets (i.e., StreamingQA, SQuAD and ArchivalQA) demonstrate that the proposed method improves model adaptability and robustness across multiple base LLMs (e.g., +4.07 EM & +4.19 F1 in StreamingQA with Llama-2-7b).
• 摘要：大型语言模型 (LLM) 需要适应数据、任务和用户偏好的不断变化。由于其庞大的规模和与训练相关的高成本，LLM 不适合频繁重新训练。然而，更新是必要的，以使它们与快速发展的人类知识保持同步。为了应对这些挑战，本文提出了压缩记忆训练 (CMT) 方法，这是一种高效且有效的 LLM 在线适应框架，具有强大的知识保留能力。受人类记忆机制的启发，CMT 压缩并提取新文档中的信息以存储在记忆库中。在回答与这些新文档相关的查询时，模型会从记忆库中聚合这些文档记忆，以更好地回答用户的问题。LLM 本身的参数在训练和推理过程中不会发生变化，从而降低了灾难性遗忘的风险。为了增强记忆的编码、检索和聚合，我们进一步提出了三种新的通用且灵活的技术，包括记忆感知目标、自我匹配和顶部聚合。在三个持续学习数据集（即 StreamingQA、SQuAD 和 ArchivalQA）上进行的大量实验表明，所提出的方法提高了跨多个基础 LLM 的模型适应性和鲁棒性（例如，在使用 Llama-2-7b 的 StreamingQA 中 +4.07 EM 和 +4.19 F1）。

Title: Generating Knowledge Graphs from Large Language Models: A Comparative Study of GPT-4, LLaMA 2, and BERT

• Authors: Ahan Bhatt, Nandan Vaghela, Kush Dudhia
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.07412
• Pdf URL: https://arxiv.org/pdf/2412.07412
• Copy Paste: [[2412.07412]] Generating Knowledge Graphs from Large Language Models: A Comparative Study of GPT-4, LLaMA 2, and BERT(https://arxiv.org/abs/2412.07412)
• Keywords: language model, gpt, llm
• Abstract: Knowledge Graphs (KGs) are essential for the functionality of GraphRAGs, a form of Retrieval-Augmented Generative Systems (RAGs) that excel in tasks requiring structured reasoning and semantic understanding. However, creating KGs for GraphRAGs remains a significant challenge due to accuracy and scalability limitations of traditional methods. This paper introduces a novel approach leveraging large language models (LLMs) like GPT-4, LLaMA 2 (13B), and BERT to generate KGs directly from unstructured data, bypassing traditional pipelines. Using metrics such as Precision, Recall, F1-Score, Graph Edit Distance, and Semantic Similarity, we evaluate the models' ability to generate high-quality KGs. Results demonstrate that GPT-4 achieves superior semantic fidelity and structural accuracy, LLaMA 2 excels in lightweight, domain-specific graphs, and BERT provides insights into challenges in entity-relationship modeling. This study underscores the potential of LLMs to streamline KG creation and enhance GraphRAG accessibility for real-world applications, while setting a foundation for future advancements.
• 摘要：知识图谱 (KG) 对于 GraphRAG 的功能至关重要，GraphRAG 是一种检索增强生成系统 (RAG)，在需要结构化推理和语义理解的任务中表现出色。然而，由于传统方法的准确性和可扩展性限制，为 GraphRAG 创建 KG 仍然是一项重大挑战。本文介绍了一种新方法，利用 GPT-4、LLaMA 2 (13B) 和 BERT 等大型语言模型 (LLM) 直接从非结构化数据生成 KG，绕过传统管道。使用准确率、召回率、F1 分数、图编辑距离和语义相似度等指标，我们评估模型生成高质量 KG 的能力。结果表明，GPT-4 实现了出色的语义保真度和结构准确性，LLaMA 2 在轻量级、特定领域图方面表现出色，而 BERT 则提供了对实体关系建模挑战的洞察。这项研究强调了 LLM 简化 KG 创建和增强 GraphRAG 对实际应用的可访问性的潜力，同时为未来的发展奠定了基础。

Title: RAG-based Question Answering over Heterogeneous Data and Text

• Authors: Philipp Christmann, Gerhard Weikum
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.07420
• Pdf URL: https://arxiv.org/pdf/2412.07420
• Copy Paste: [[2412.07420]] RAG-based Question Answering over Heterogeneous Data and Text(https://arxiv.org/abs/2412.07420)
• Keywords: language model, gpt
• Abstract: This article presents the QUASAR system for question answering over unstructured text, structured tables, and knowledge graphs, with unified treatment of all sources. The system adopts a RAG-based architecture, with a pipeline of evidence retrieval followed by answer generation, with the latter powered by a moderate-sized language model. Additionally and uniquely, QUASAR has components for question understanding, to derive crisper input for evidence retrieval, and for re-ranking and filtering the retrieved evidence before feeding the most informative pieces into the answer generation. Experiments with three different benchmarks demonstrate the high answering quality of our approach, being on par with or better than large GPT models, while keeping the computational cost and energy consumption orders of magnitude lower.
• 摘要：本文介绍了 QUASAR 系统，该系统针对非结构化文本、结构化表格和知识图谱进行问答，并统一处理所有来源。该系统采用基于 RAG 的架构，具有证据检索和答案生成的流程，后者由中等规模的语言模型提供支持。此外，QUASAR 还具有独特的组件，用于问题理解、获取更清晰的证据检索输入，以及在将最具信息量的信息输入答案生成之前对检索到的证据进行重新排序和过滤。使用三个不同基准进行的实验表明，我们的方法具有很高的回答质量，与大型 GPT 模型相当或更好，同时将计算成本和能耗降低了几个数量级。

Title: Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation

• Authors: Javad Seraj, Mohammad Mahdi Mohajeri, Mohammad Javad Dousti, Majid Nili Ahmadabadi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07429
• Pdf URL: https://arxiv.org/pdf/2412.07429
• Copy Paste: [[2412.07429]] Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation(https://arxiv.org/abs/2412.07429)
• Keywords: language model, llm
• Abstract: Automatic evaluation by large language models (LLMs) is a prominent topic today; however, judgment and evaluation tasks are often subjective and influenced by various factors, making adaptation challenging. While many studies demonstrate the capabilities of state-of-the-art proprietary LLMs in comparison to human evaluators, they often struggle to adapt to reference evaluators over time, a requirement for achieving personalized judgment. Additionally, numerous works have attempted to apply open LLMs as judges or evaluators, but these efforts frequently overlook the limitations of working with scarce data. Personalized judgment is inherently associated with limited data scenarios, which are common in many real-world problems. Our work aims to present a data augmentation technique to select a more effective sample from limited data in order to align an open LLM with human preference. Our work achieves approximately 7% improvements in Pearson correlation with a reference judge over the baseline,and 30% improvement over the base model (Llama3.1-8B-Instruct) in the mathematical reasoning evaluation task. demonstrating that augmenting selecting more effective preference data enables our approach to surpass baseline methods.
• 摘要：大型语言模型 (LLM) 的自动评估是当今的一个热门话题；然而，判断和评估任务往往是主观的，并受到各种因素的影响，因此很难适应。虽然许多研究证明了最先进的专有 LLM 与人类评估者相比的能力，但它们往往难以随着时间的推移适应参考评估者，这是实现个性化判断的必要条件。此外，许多研究都试图将开放式 LLM 用作评判者或评估者，但这些努力往往忽视了处理稀缺数据的局限性。个性化判断本质上与有限数据场景相关，这在许多现实世界的问题中很常见。我们的工作旨在提出一种数据增强技术，从有限的数据中选择更有效的样本，以使开放式 LLM 与人类偏好保持一致。我们的工作使与参考评判者的皮尔逊相关性比基线提高了约 7%，比基础模型 (Llama3.1-8B-Instruct) 在数学推理评估任务中提高了 30%。这表明，通过增强选择更有效的偏好数据，我们的方法可以超越基线方法。

Title: Knowledge Graph Guided Evaluation of Abstention Techniques

• Authors: Kinshuk Vasisht, Navreet Kaur, Danish Pruthi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07430
• Pdf URL: https://arxiv.org/pdf/2412.07430
• Copy Paste: [[2412.07430]] Knowledge Graph Guided Evaluation of Abstention Techniques(https://arxiv.org/abs/2412.07430)
• Keywords: language model
• Abstract: To deploy language models safely, it is crucial that they abstain from responding to inappropriate requests. Several prior studies test the safety promises of models based on their effectiveness in blocking malicious requests. In this work, we focus on evaluating the underlying techniques that cause models to abstain. We create SELECT, a benchmark derived from a set of benign concepts (e.g., "rivers") from a knowledge graph. The nature of SELECT enables us to isolate the effects of abstention techniques from other safety training procedures, as well as evaluate their generalization and specificity. Using SELECT, we benchmark different abstention techniques over six open-weight and closed-source models. We find that the examined techniques indeed cause models to abstain with over $80\%$ abstention rates. However, these techniques are not as effective for descendants of the target concepts, with refusal rates declining by $19\%$. We also characterize the generalization-vs-specificity trade-offs for different techniques. Overall, no single technique is invariably better than the others. Our findings call for a careful evaluation of different aspects of abstention, and hopefully inform practitioners of various trade-offs involved.
• 摘要：为了安全地部署语言模型，至关重要的是它们要避免响应不适当的请求。先前的几项研究根据模型阻止恶意请求的有效性来测试其安全承诺。在这项工作中，我们专注于评估导致模型弃权的底层技术。我们创建了 SELECT，这是一个从知识图谱中的一组良性概念（例如“河流”）派生出的基准。SELECT 的性质使我们能够将弃权技术的影响与其他安全训练程序隔离开来，并评估它们的泛化和特异性。使用 SELECT，我们对六个开源和闭源模型的不同弃权技术进行了基准测试。我们发现，所检查的技术确实导致模型弃权，弃权率超过 $80\%$。然而，这些技术对目标概念的后代不那么有效，拒绝率下降了 $19\%$。我们还描述了不同技术的泛化与特异性的权衡。总的来说，没有一种技术总是比其他技术更好。我们的研究结果要求仔细评估禁欲的不同方面，并希望让从业者了解所涉及的各种权衡。

Title: CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues

• Authors: Sebastian Steindl, Ulrich Schäfer, Bernd Ludwig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07515
• Pdf URL: https://arxiv.org/pdf/2412.07515
• Copy Paste: [[2412.07515]] CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues(https://arxiv.org/abs/2412.07515)
• Keywords: language model, llm
• Abstract: Large-scale Wizard-Of-Oz dialogue datasets have enabled the training of deep learning-based dialogue systems. While they are successful as benchmark datasets, they lack certain types of utterances, which would make them more realistic. In this work, we investigate the creation of synthetic communication errors in an automatic pipeline. Based on linguistic theory, we propose and follow a simple error taxonomy. We focus on three types of miscommunications that could happen in real-world dialogues but are underrepresented in the benchmark dataset: misunderstandings, non-understandings and vaguely related questions. Our two-step approach uses a state-of-the-art Large Language Model (LLM) to first create the error and secondly the repairing utterance. We perform Language Model-based evaluation to ensure the quality of the generated utterances. We apply the method to the MultiWOZ dataset and evaluate it both qualitatively and empirically as well as with human judges. Our results indicate that current LLMs can aid in adding post-hoc miscommunications to benchmark datasets as a form of data augmentation. We publish the resulting dataset, in which nearly 1900 dialogues have been modified, as CoPrUS-MultiWOZ to facilitate future work on dialogue systems.
• 摘要：大规模 Wizard-Of-Oz 对话数据集使基于深度学习的对话系统的训练成为可能。虽然它们作为基准数据集取得了成功，但它们缺少某些类型的话语，而这些话语会使它们更加逼真。在这项工作中，我们研究了自动管道中合成通信错误的产生。基于语言理论，我们提出并遵循一个简单的错误分类法。我们关注三种可能发生在现实世界对话中但在基准数据集中代表性不足的错误沟通：误解、不理解和模糊相关的问题。我们的两步方法使用最先进的大型语言模型 (LLM) 首先创建错误，然后修复话语。我们执行基于语言模型的评估以确保生成的话语的质量。我们将该方法应用于 MultiWOZ 数据集，并对其进行定性和实证评估以及与人类评判者的评估。我们的结果表明，当前的 LLM 可以帮助将事后误传添加到基准数据集中，作为一种数据增强形式。我们将结果数据集（其中近 1900 个对话经过了修改）发布为 CoPrUS-MultiWOZ，以促进未来对话系统的研究。

Title: DRUM: Learning Demonstration Retriever for Large MUlti-modal Models

• Authors: Ellen Yi-Ge, Jiechao Gao, Wei Han, Wei Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07619
• Pdf URL: https://arxiv.org/pdf/2412.07619
• Copy Paste: [[2412.07619]] DRUM: Learning Demonstration Retriever for Large MUlti-modal Models(https://arxiv.org/abs/2412.07619)
• Keywords: language model, llm
• Abstract: Recently, large language models (LLMs) have demonstrated impressive capabilities in dealing with new tasks with the help of in-context learning (ICL). In the study of Large Vision-Language Models (LVLMs), when implementing ICL, researchers usually adopts the naive strategies like fixed demonstrations across different samples, or selecting demonstrations directly via a visual-language embedding model. These methods does not guarantee the configured demonstrations fit the need of the LVLMs. To address this issue, we now propose a novel framework, \underline{d}emonstration \underline{r}etriever for large m\underline{u}lti-modal \underline{m}odel (DRUM), which fine-tunes the visual-language embedding model to better meet the LVLM's needs. First, we discuss the retrieval strategies for a visual-language task, assuming an embedding model is given. And we propose to concate the image and text embeddings to enhance the retrieval performance. Second, we propose to re-rank the demonstrations retrieved by the embedding model via the LVLM's feedbacks, and calculate a list-wise ranking loss for training the embedding model. Third, we propose an iterative demonstration mining strategy to improve the training of the embedding model. Through extensive experiments on 3 types of visual-language tasks, 7 benchmark datasets, our DRUM framework is proven to be effective in boosting the LVLM's in-context learning performance via retrieving more proper demonstrations.
• 摘要：最近，大型语言模型 (LLM) 在上下文学习 (ICL) 的帮助下表现出了处理新任务的出色能力。在大型视觉语言模型 (LVLM) 的研究中，在实现 ICL 时，研究人员通常采用简单的策略，例如在不同样本中固定演示，或直接通过视觉语言嵌入模型选择演示。这些方法不能保证配置的演示符合 LVLM 的需求。为了解决这个问题，我们现在提出了一个新颖的框架 \underline{d}emonstration \underline{r}etriever for large m\underline{u}lti-modal \underline{m}odel (DRUM)，它可以对视觉语言嵌入模型进行微调，以更好地满足 LVLM 的需求。首先，我们讨论视觉语言任务的检索策略，假设给出了一个嵌入模型。我们建议将图像和文本嵌入连接起来以增强检索性能。其次，我们提出通过 LVLM 的反馈对嵌入模型检索到的演示进行重新排序，并计算列表排序损失以训练嵌入模型。第三，我们提出了一种迭代演示挖掘策略来改进嵌入模型的训练。通过对 3 种类型的视觉语言任务、7 个基准数据集进行大量实验，我们的 DRUM 框架被证明能够通过检索更合适的演示来有效提升 LVLM 的上下文学习性能。

Title: Piece of Table: A Divide-and-Conquer Approach for Selecting Sub-Tables in Table Question Answering

• Authors: Wonjin Lee, Kyumin Kim, Sungjae Lee, Jihun Lee, Kwang In KIm
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07629
• Pdf URL: https://arxiv.org/pdf/2412.07629
• Copy Paste: [[2412.07629]] Piece of Table: A Divide-and-Conquer Approach for Selecting Sub-Tables in Table Question Answering(https://arxiv.org/abs/2412.07629)
• Keywords: language model, long context
• Abstract: Applying language models (LMs) to tables is challenging due to the inherent structural differences between two-dimensional tables and one-dimensional text for which the LMs were originally designed. Furthermore, when applying linearized tables to LMs, the maximum token lengths often imposed in self-attention calculations make it difficult to comprehensively understand the context spread across large tables. To address these challenges, we present PieTa (Piece of Table), a new framework for sub-table-based question answering (QA). PieTa operates through an iterative process of dividing tables into smaller windows, using LMs to select relevant cells within each window, and merging these cells into a sub-table. This multi-resolution approach captures dependencies across multiple rows and columns while avoiding the limitations caused by long context inputs. Instantiated as a simple iterative sub-table union algorithm, PieTa demonstrates improved performance over previous sub-table-based QA approaches.
• 摘要：将语言模型 (LM) 应用于表格具有挑战性，因为二维表格和 LM 最初设计用于的一维文本之间存在固有的结构差异。此外，在将线性化表格应用于 LM 时，自注意力计算中经常施加的最大标记长度使得难以全面理解大型表格中的上下文。为了应对这些挑战，我们提出了 PieTa（Piece of Table），这是一种基于子表的问答 (QA) 的新框架。PieTa 通过一个迭代过程来运行，将表格划分为较小的窗口，使用 LM 选择每个窗口内的相关单元格，并将这些单元格合并到子表中。这种多分辨率方法可以捕获跨多行和多列的依赖关系，同时避免长上下文输入造成的限制。PieTa 实例化为一个简单的迭代子表联合算法，与以前基于子表的 QA 方法相比，其性能有所提高。

Title: ChocoLlama: Lessons Learned From Teaching Llamas Dutch

• Authors: Matthieu Meeus, Anthony Rathé, François Remy, Pieter Delobelle, Jens-Joris Decorte, Thomas Demeester
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07633
• Pdf URL: https://arxiv.org/pdf/2412.07633
• Copy Paste: [[2412.07633]] ChocoLlama: Lessons Learned From Teaching Llamas Dutch(https://arxiv.org/abs/2412.07633)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, their performance often lags in lower-resource, non-English languages due to biases in the training data. In this work, we explore strategies for adapting the primarily English LLMs (Llama-2 and Llama-3) to Dutch, a language spoken by 30 million people worldwide yet often underrepresented in LLM development. We collect 104GB of Dutch text ($32$B tokens) from various sources to first apply continued pretraining using low-rank adaptation (LoRA), complemented with Dutch posttraining strategies provided by prior work. For Llama-2, we consider using (i) the tokenizer of the original model, and (ii) training a new, Dutch-specific tokenizer combined with embedding reinitialization. We evaluate our adapted models, ChocoLlama-2, both on standard benchmarks and a novel Dutch benchmark, ChocoLlama-Bench. Our results demonstrate that LoRA can effectively scale for language adaptation, and that tokenizer modification with careful weight reinitialization can improve performance. Notably, Llama-3 was released during the course of this project and, upon evaluation, demonstrated superior Dutch capabilities compared to our Dutch-adapted versions of Llama-2. We hence apply the same adaptation technique to Llama-3, using its original tokenizer. While our adaptation methods enhanced Llama-2's Dutch capabilities, we found limited gains when applying the same techniques to Llama-3. This suggests that for ever improving, multilingual foundation models, language adaptation techniques may benefit more from focusing on language-specific posttraining rather than on continued pretraining. We hope this work contributes to the broader understanding of adapting LLMs to lower-resource languages, and to the development of Dutch LLMs in particular.
• 摘要：虽然大型语言模型 (LLM) 在自然语言理解和生成方面表现出非凡的能力，但由于训练数据存在偏差，它们在资源较少的非英语语言中的表现往往落后。在这项工作中，我们探索了将主要使用英语的 LLM（Llama-2 和 Llama-3）改编为荷兰语的策略，荷兰语是全球 3000 万人使用的语言，但在 LLM 开发中却往往代表性不足。我们从各种来源收集了 104GB 的荷兰语文本（价值 320 亿美元的代币），首先应用低秩自适应 (LoRA) 进行持续预训练，并辅以先前工作提供的荷兰语后训练策略。对于 Llama-2，我们考虑使用 (i) 原始模型的标记器，以及 (ii) 训练一个新的、特定于荷兰语的标记器并结合嵌入重新初始化。我们在标准基准和新的荷兰语基准 ChocoLlama-Bench 上评估了我们改编的模型 ChocoLlama-2。我们的结果表明，LoRA 可以有效地扩展以适应语言，并且通过仔细重新初始化权重来修改标记器可以提高性能。值得注意的是，Llama-3 是在该项目期间发布的，经过评估，与我们的荷兰语改编版 Llama-2 相比，它表现出了更出色的荷兰语能力。因此，我们将相同的适应技术应用于 Llama-3，使用其原始标记器。虽然我们的适应方法增强了 Llama-2 的荷兰语能力，但我们发现将相同技术应用于 Llama-3 时收益有限。这表明，对于不断改进的多语言基础模型，语言适应技术可能更多地受益于专注于特定于语言的后训练而不是持续的预训练。我们希望这项工作有助于更广泛地理解将 LLM 适应资源较少的语言，特别是荷兰语 LLM 的发展。

Title: Searching for Structure: Investigating Emergent Communication with Large Language Models

• Authors: Tom Kouwenhoven, Max Peeperkorn, Tessa Verhoef
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07646
• Pdf URL: https://arxiv.org/pdf/2412.07646
• Copy Paste: [[2412.07646]] Searching for Structure: Investigating Emergent Communication with Large Language Models(https://arxiv.org/abs/2412.07646)
• Keywords: language model, llm, agent
• Abstract: Human languages have evolved to be structured through repeated language learning and use. These processes introduce biases that operate during language acquisition and shape linguistic systems toward communicative efficiency. In this paper, we investigate whether the same happens if artificial languages are optimised for implicit biases of Large Language Models (LLMs). To this end, we simulate a classical referential game in which LLMs learn and use artificial languages. Our results show that initially unstructured holistic languages are indeed shaped to have some structural properties that allow two LLM agents to communicate successfully. Similar to observations in human experiments, generational transmission increases the learnability of languages, but can at the same time result in non-humanlike degenerate vocabularies. Taken together, this work extends experimental findings, shows that LLMs can be used as tools in simulations of language evolution, and opens possibilities for future human-machine experiments in this field.
• 摘要：人类语言通过反复的语言学习和使用而逐渐形成结构。这些过程引入了在语言习得过程中起作用的偏见，并将语言系统塑造成更高效的交流系统。在本文中，我们研究如果针对大型语言模型 (LLM) 的隐性偏见优化人工语言，是否会发生同样的情况。为此，我们模拟了一个经典的指称游戏，其中 LLM 学习和使用人工语言。我们的结果表明，最初非结构化的整体语言确实被塑造成具有一些结构属性，使两个 LLM 代理能够成功通信。与人类实验中的观察结果类似，代际传递增加了语言的可学习性，但同时也会导致非人类的退化词汇。总之，这项工作扩展了实验结果，表明 LLM 可以用作语言进化模拟的工具，并为未来该领域的人机实验开辟了可能性。

Title: RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting

• Authors: Shuo Yang, Bardh Prenkaj, Gjergji Kasneci
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07675
• Pdf URL: https://arxiv.org/pdf/2412.07675
• Copy Paste: [[2412.07675]] RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting(https://arxiv.org/abs/2412.07675)
• Keywords: llm
• Abstract: Despite the widespread use of LLMs due to their superior performance in various tasks, their high computational costs often lead potential users to opt for the pretraining-finetuning pipeline. However, biases prevalent in manually constructed datasets can introduce spurious correlations between tokens and labels, creating so-called shortcuts and hindering the generalizability of fine-tuned models. Existing debiasing methods often rely on prior knowledge of specific dataset biases, which is challenging to acquire a priori. We propose RAZOR (Rewriting And Zero-bias Optimization Refinement), a novel, unsupervised, and data-focused debiasing approach based on text rewriting for shortcut mitigation. RAZOR leverages LLMs to iteratively rewrite potentially biased text segments by replacing them with heuristically selected alternatives in a shortcut space defined by token statistics and positional information. This process aims to align surface-level text features more closely with diverse label distributions, thereby promoting the learning of genuine linguistic patterns. Compared with unsupervised SoTA models, RAZOR improves by 3.5% on the FEVER and 6.5% on MNLI and SNLI datasets according to the F1 score. Additionally, RAZOR effectively mitigates specific known biases, reducing bias-related terms by x2 without requiring prior bias information, a result that is on par with SoTA models that leverage prior information. Our work prioritizes data manipulation over architectural modifications, emphasizing the pivotal role of data quality in enhancing model performance and fairness. This research contributes to developing more robust evaluation benchmarks for debiasing methods by incorporating metrics for bias reduction and overall model efficacy.
• 摘要：尽管 LLM 因其在各种任务中的出色表现而被广泛使用，但其高昂的计算成本往往导致潜在用户选择预训练-微调流程。然而，手动构建数据集中普遍存在的偏差可能会在标记和标签之间引入虚假相关性，从而产生所谓的捷径并阻碍微调模型的普遍性。现有的去偏方法通常依赖于对特定数据集偏差的先验知识，而这种先验知识很难获得。我们提出了 RAZOR（重写和零偏差优化细化），这是一种新颖的、无监督的、以数据为中心的去偏方法，基于文本重写来缓解捷径问题。RAZOR 利用 LLM 迭代重写可能有偏差的文本段，方法是将它们替换为由标记统计数据和位置信息定义的捷径空间中启发式选择的替代方案。此过程旨在使表层文本特征与不同的标签分布更紧密地对齐，从而促进对真正语言模式的学习。与无监督 SoTA 模型相比，RAZOR 在 FEVER 上提高了 3.5%，在 MNLI 和 SNLI 数据集上提高了 6.5%（根据 F1 分数）。此外，RAZOR 有效地减轻了特定的已知偏差，将与偏差相关的术语减少了 2 倍，而无需先前的偏差信息，这一结果与利用先前信息的 SoTA 模型相当。我们的工作优先考虑数据操作而不是架构修改，强调数据质量在提高模型性能和公平性方面的关键作用。这项研究通过结合偏差减少和整体模型功效的指标，有助于为去偏差方法开发更强大的评估基准。

Title: Can linguists better understand DNA?

• Authors: Wang Liang
• Subjects: cs.CL, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2412.07678
• Pdf URL: https://arxiv.org/pdf/2412.07678
• Copy Paste: [[2412.07678]] Can linguists better understand DNA?(https://arxiv.org/abs/2412.07678)
• Keywords: gpt
• Abstract: Multilingual transfer ability, which reflects how well models fine-tuned on one source language can be applied to other languages, has been well studied in multilingual pre-trained models. However, the existence of such capability transfer between natural language and gene sequences/languages remains this http URL study addresses this gap by drawing inspiration from the sentence-pair classification task used for evaluating sentence similarity in natural language. We constructed two analogous tasks: DNA-pair classification(DNA sequence similarity) and DNA-protein-pair classification(gene coding determination). These tasks were designed to validate the transferability of capabilities from natural language to gene sequences. Even a small-scale pre-trained model like GPT-2-small, which was pre-trained on English, achieved an accuracy of 78% on the DNA-pair classification task after being fine-tuned on English sentence-pair classification data(XTREME PAWS-X). While training a BERT model on multilingual text, the precision reached 82%.On the more complex DNA-protein-pair classification task, however, the model's output was barely distinguishable from random this http URL suggest that there may be a capability transfer from natural language to genetic language, but further task testing is needed to confirm this.
• 摘要：多语言迁移能力反映了在一种源语言上微调的模型应用于其他语言的效果如何，这一能力在多语言预训练模型中已经得到了很好的研究。然而，自然语言和基因序列 / 语言之间仍然存在这种能力迁移，本研究从用于评估自然语言中句子相似性的句子对分类任务中汲取灵感，解决了这一空白。我们构建了两个类似的任务：DNA 对分类（DNA 序列相似性）和 DNA 蛋白质对分类（基因编码测定）。这些任务旨在验证能力从自然语言到基因序列的可迁移性。即使是像 GPT-2-small 这样在英语上进行预训练的小规模预训练模型，在对英语句子对分类数据（XTREME PAWS-X）进行微调后，在 DNA 对分类任务上也达到了 78% 的准确率。在对多语言文本进行 BERT 模型训练时，准确率达到了 82%。然而，在更复杂的 DNA-蛋白质对分类任务中，该模型的输出几乎无法与随机数区分，这表明可能存在从自然语言到遗传语言的能力转移，但需要进一步的任务测试来证实这一点。

Title: TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation

• Authors: Alfredo Garrachón Ruiz, Tomás de la Rosa, Daniel Borrajo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07682
• Pdf URL: https://arxiv.org/pdf/2412.07682
• Copy Paste: [[2412.07682]] TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation(https://arxiv.org/abs/2412.07682)
• Keywords: language model, llm, prompt
• Abstract: The inference cost of Large Language Models (LLMs) is a significant challenge due to their computational demands, specially on tasks requiring long outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language-concise outputs that retain essential meaning, when prompted appropriately. We propose a framework for saving computational cost, in which a shorter distilled output from the LLM is reconstructed into a full narrative by a smaller model with lower inference costs. Our experiments show promising results, particularly in general knowledge domains with 20.58% saved tokens on average with tiny decrease in evaluation metrics, hinting that this approach can effectively balance efficiency and accuracy in language processing tasks.
• 摘要：大型语言模型 (LLM) 的推理成本是一项重大挑战，因为它们需要大量计算，尤其是在需要长输出的任务上。然而，自然语言通常包含冗余，这为优化提供了机会。我们观察到，在适当提示时，LLM 可以生成保留基本含义的精简语言输出。我们提出了一个节省计算成本的框架，其中 LLM 的较短精简输出由推理成本较低的较小模型重建为完整叙述。我们的实验显示出令人鼓舞的结果，特别是在一般知识领域，平均节省了 20.58% 的标记，评估指标略有下降，暗示这种方法可以有效地平衡语言处理任务的效率和准确性。

Title: Granite Guardian

• Authors: Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, Prasanna Sattigeri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07724
• Pdf URL: https://arxiv.org/pdf/2412.07724
• Copy Paste: [[2412.07724]] Granite Guardian(https://arxiv.org/abs/2412.07724)
• Keywords: language model, llm, hallucination, prompt, retrieval-augmented generation
• Abstract: We introduce the Granite Guardian models, a suite of safeguards designed to provide risk detection for prompts and responses, enabling safe and responsible use in combination with any large language model (LLM). These models offer comprehensive coverage across multiple risk dimensions, including social bias, profanity, violence, sexual content, unethical behavior, jailbreaking, and hallucination-related risks such as context relevance, groundedness, and answer relevance for retrieval-augmented generation (RAG). Trained on a unique dataset combining human annotations from diverse sources and synthetic data, Granite Guardian models address risks typically overlooked by traditional risk detection models, such as jailbreaks and RAG-specific issues. With AUC scores of 0.871 and 0.854 on harmful content and RAG-hallucination-related benchmarks respectively, Granite Guardian is the most generalizable and competitive model available in the space. Released as open-source, Granite Guardian aims to promote responsible AI development across the community. this https URL
• 摘要：我们推出了 Granite Guardian 模型，这是一套旨在为提示和响应提供风险检测的安全措施，可与任何大型语言模型 (LLM) 结合使用，以实现安全且负责任的使用。这些模型全面覆盖多个风险维度，包括社交偏见、亵渎、暴力、性内容、不道德行为、越狱和幻觉相关风险，例如上下文相关性、接地性和检索增强生成 (RAG) 的答案相关性。Granite Guardian 模型在结合了来自不同来源的人工注释和合成数据的独特数据集上进行训练，解决了传统风险检测模型通常忽略的风险，例如越狱和 RAG 特定问题。Granite Guardian 在有害内容和 RAG 幻觉相关基准上的 AUC 得分分别为 0.871 和 0.854，是该领域最具通用性和竞争力的模型。Granite Guardian 以开源形式发布，旨在促进整个社区负责任的 AI 开发。此 https URL

Title: Zero-Shot ATC Coding with Large Language Models for Clinical Assessments

• Authors: Zijian Chen, John-Michael Gamble, Micaela Jantzi, John P. Hirdes, Jimmy Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07743
• Pdf URL: https://arxiv.org/pdf/2412.07743
• Copy Paste: [[2412.07743]] Zero-Shot ATC Coding with Large Language Models for Clinical Assessments(https://arxiv.org/abs/2412.07743)
• Keywords: language model, gpt, llm
• Abstract: Manual assignment of Anatomical Therapeutic Chemical (ATC) codes to prescription records is a significant bottleneck in healthcare research and operations at Ontario Health and InterRAI Canada, requiring extensive expert time and effort. To automate this process while maintaining data privacy, we develop a practical approach using locally deployable large language models (LLMs). Inspired by recent advances in automatic International Classification of Diseases (ICD) coding, our method frames ATC coding as a hierarchical information extraction task, guiding LLMs through the ATC ontology level by level. We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment. Testing across Health Canada drug product data, the RABBITS benchmark, and real clinical notes from Ontario Health, our method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B. We investigate knowledge grounding through drug definitions, finding modest improvements in accuracy. Further, we show that fine-tuned Llama 3.1 8B matches zero-shot Llama 3.1 70B accuracy, suggesting that effective ATC coding is feasible with smaller models. Our results demonstrate the feasibility of automatic ATC coding in privacy-sensitive healthcare environments, providing a foundation for future deployments.
• 摘要：手动将解剖治疗化学 (ATC) 代码分配给处方记录是安大略健康部和 InterRAI Canada 医疗保健研究和运营的一个重大瓶颈，需要大量专家的时间和精力。为了在保持数据隐私的同时实现此过程的自动化，我们开发了一种使用本地可部署的大型语言模型 (LLM) 的实用方法。受国际疾病分类 (ICD) 自动编码最新进展的启发，我们的方法将 ATC 编码构建为分层信息提取任务，引导 LLM 逐级完成 ATC 本体。我们使用 GPT-4o 作为准确度上限来评估我们的方法，并专注于开发适合隐私敏感部署的开源 Llama 模型。在加拿大卫生部的药品数据、RABBITS 基准和安大略健康部的真实临床记录中进行测试后，我们的方法使用 GPT-4o 实现了 78% 的精确匹配准确率，使用 Llama 3.1 70B 实现了 60% 的精确匹配准确率。我们通过药物定义研究知识基础，发现准确率略有提高。此外，我们表明经过微调的 Llama 3.1 8B 的准确率与零样本 Llama 3.1 70B 相当，这表明使用较小的模型也可以实现有效的 ATC 编码。我们的结果证明了在隐私敏感的医疗保健环境中自动 ATC 编码的可行性，为未来的部署奠定了基础。