2024-12-16

Title: GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers

• Authors: Sarkar Snigdha Sarathi Das, Ryo Kamoi, Bo Pang, Yusen Zhang, Caiming Xiong, Rui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09722
• Pdf URL: https://arxiv.org/pdf/2412.09722
• Copy Paste: [[2412.09722]] GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers(https://arxiv.org/abs/2412.09722)
• Keywords: language model, llm, prompt
• Abstract: The effectiveness of large language models (LLMs) is closely tied to the design of prompts, making prompt optimization essential for enhancing their performance across a wide range of tasks. Many existing approaches to automating prompt engineering rely exclusively on textual feedback, refining prompts based solely on inference errors identified by large, computationally expensive LLMs. Unfortunately, smaller models struggle to generate high-quality feedback, resulting in complete dependence on large LLM judgment. Moreover, these methods fail to leverage more direct and finer-grained information, such as gradients, due to operating purely in text space. To this end, we introduce GReaTer, a novel prompt optimization technique that directly incorporates gradient information over task-specific reasoning. By utilizing task loss gradients, GReaTer enables self-optimization of prompts for open-source, lightweight language models without the need for costly closed-source LLMs. This allows high-performance prompt optimization without dependence on massive LLMs, closing the gap between smaller models and the sophisticated reasoning often needed for prompt refinement. Extensive evaluations across diverse reasoning tasks including BBH, GSM8k, and FOLIO demonstrate that GReaTer consistently outperforms previous state-of-the-art prompt optimization methods, even those reliant on powerful LLMs. Additionally, GReaTer-optimized prompts frequently exhibit better transferability and, in some cases, boost task performance to levels comparable to or surpassing those achieved by larger language models, highlighting the effectiveness of prompt optimization guided by gradients over reasoning. Code of GReaTer is available at this https URL.
• 摘要：大型语言模型 (LLM) 的有效性与提示的设计密切相关，因此提示优化对于提高其在各种任务中的性能至关重要。许多现有的自动化提示工程方法完全依赖于文本反馈，仅根据大型、计算成本高昂的 LLM 识别出的推理错误来改进提示。不幸的是，较小的模型难以生成高质量的反馈，导致完全依赖大型 LLM 判断。此外，由于这些方法纯粹在文本空间中运行，因此无法利用更直接、更细粒度的信息（例如梯度）。为此，我们引入了 GReaTer，这是一种新颖的提示优化技术，它直接结合梯度信息而不是特定于任务的推理。通过利用任务损失梯度，GReaTer 无需昂贵的闭源 LLM，即可实现开源轻量级语言模型的提示自我优化。这允许高性能提示优化而无需依赖大量 LLM，从而缩小了较小模型与提示改进通常所需的复杂推理之间的差距。对 BBH、GSM8k 和 FOLIO 等各种推理任务进行的广泛评估表明，GReaTer 始终优于之前最先进的提示优化方法，即使是那些依赖于强大 LLM 的方法。此外，GReaTer 优化的提示通常表现出更好的可转移性，在某些情况下，将任务性能提升到与大型语言模型相当或超过大型语言模型的水平，凸显了由梯度引导的提示优化比推理更有效。GReaTer 的代码可在此 https URL 上找到。

Title: Memory Layers at Scale

• Authors: Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Gosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09764
• Pdf URL: https://arxiv.org/pdf/2412.09764
• Copy Paste: [[2412.09764]] Memory Layers at Scale(https://arxiv.org/abs/2412.09764)
• Keywords: language model
• Abstract: Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.
• 摘要：记忆层使用可训练的键值查找机制，在不增加 FLOP 的情况下向模型添加额外参数。从概念上讲，稀疏激活的记忆层补充了计算密集型密集前馈层，提供了专用容量来以低成本存储和检索信息。这项工作使记忆层超越了概念验证，证明了它们在当代规模上的实用性。在下游任务中，使用我们改进的记忆层增强的语言模型在计算和参数匹配时，其表现优于计算预算两倍以上的密集模型以及专家混合模型。我们发现对于事实任务而言，收益尤其明显。我们提供了一个完全可并行化的记忆层实现，展示了具有多达 128B 记忆参数的扩展规律，预训练到 1 万亿个标记，而基础模型的参数多达 8B。

Title: AutoPatent: A Multi-Agent Framework for Automatic Patent Generation

• Authors: Qiyao Wang, Shiwen Ni, Huaren Liu, Shule Lu, Guhong Chen, Xi Feng, Chi Wei, Qiang Qu, Hamid Alinejad-Rokny, Yuan Lin, Min Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09796
• Pdf URL: https://arxiv.org/pdf/2412.09796
• Copy Paste: [[2412.09796]] AutoPatent: A Multi-Agent Framework for Automatic Patent Generation(https://arxiv.org/abs/2412.09796)
• Keywords: language model, gpt, llm, agent
• Abstract: As the capabilities of Large Language Models (LLMs) continue to advance, the field of patent processing has garnered increased attention within the natural language processing community. However, the majority of research has been concentrated on classification tasks, such as patent categorization and examination, or on short text generation tasks like patent summarization and patent quizzes. In this paper, we introduce a novel and practical task known as Draft2Patent, along with its corresponding D2P benchmark, which challenges LLMs to generate full-length patents averaging 17K tokens based on initial drafts. Patents present a significant challenge to LLMs due to their specialized nature, standardized terminology, and extensive length. We propose a multi-agent framework called AutoPatent which leverages the LLM-based planner agent, writer agents, and examiner agent with PGTree and RRAG to generate lengthy, intricate, and high-quality complete patent documents. The experimental results demonstrate that our AutoPatent framework significantly enhances the ability to generate comprehensive patents across various LLMs. Furthermore, we have discovered that patents generated solely with the AutoPatent framework based on the Qwen2.5-7B model outperform those produced by larger and more powerful LLMs, such as GPT-4o, Qwen2.5-72B, and LLAMA3.1-70B, in both objective metrics and human evaluations. We will make the data and code available upon acceptance at \url{this https URL}.
• 摘要：随着大型语言模型 (LLM) 的功能不断进步，专利处理领域在自然语言处理社区中引起了越来越多的关注。然而，大多数研究都集中在分类任务上，例如专利分类和审查，或短文本生成任务，例如专利摘要和专利测验。在本文中，我们介绍了一项名为 Draft2Patent 的新颖实用任务，以及其相应的 D2P 基准，该基准要求 LLM 根据初稿生成平均 17K 个标记的全长专利。专利因其专业性、标准化术语和冗长篇幅而对 LLM 提出了重大挑战。我们提出了一个名为 AutoPatent 的多代理框架，该框架利用基于 LLM 的规划代理、作者代理和审查员代理以及 PGTree 和 RRAG 来生成冗长、复杂且高质量的完整专利文件。实验结果表明，我们的 AutoPatent 框架显著增强了跨各种 LLM 生成综合专利的能力。此外，我们发现，仅使用基于 Qwen2.5-7B 模型的 AutoPatent 框架生成的专利在客观指标和人工评估方面都优于由更大、更强大的 LLM（例如 GPT-4o、Qwen2.5-72B 和 LLAMA3.1-70B）生成的专利。我们将在接受后在 \url{此 https URL} 上提供数据和代码。

Title: LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering

• Authors: Patrick Sutanto, Joan Santoso
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09807
• Pdf URL: https://arxiv.org/pdf/2412.09807
• Copy Paste: [[2412.09807]] LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering(https://arxiv.org/abs/2412.09807)
• Keywords: language model, llm
• Abstract: Multiple Choice Question Answering (MCQA) is an important problem with numerous real-world applications, such as medicine, law, and education. The high cost of building MCQA datasets makes few-shot learning pivotal in this domain. While Large Language Models (LLMs) can enable few-shot learning, their direct application in real-world scenarios is often hindered by their high computational cost. To address this challenge, we propose a simple yet effective approach that uses LLMs for data generation and scoring. Our approach utilizes LLMs to create MCQA data which contains questions and choices, and to assign probability scores to the generated choices. We then use the generated data and LLM-assigned scores to finetune a smaller and more efficient encoder-only model, DeBERTa-v3-base by leveraging distillation loss. Extensive experiments on the Massive Multitask Language Understanding (MMLU) benchmark demonstrate that our method improves accuracy from 28.9% to 39.3%, representing a gain of over 10% compared to a baseline finetuned directly on 5-shot examples. This shows the effectiveness of LLM-driven data generation and knowledge distillation for few-shot MCQA.
• 摘要：多项选择题回答 (MCQA) 是医学、法律和教育等众多实际应用中的一个重要问题。构建 MCQA 数据集的高成本使得小样本学习成为这一领域的关键。虽然大型语言模型 (LLM) 可以实现小样本学习，但它们在实际场景中的直接应用往往受到高计算成本的阻碍。为了应对这一挑战，我们提出了一种简单而有效的方法，即使用 LLM 进行数据生成和评分。我们的方法利用 LLM 创建包含问题和选项的 MCQA 数据，并为生成的选项分配概率分数。然后，我们使用生成的数据和 LLM 分配的分数，通过利用蒸馏损失来微调更小、更高效的仅编码器模型 DeBERTa-v3-base。在大规模多任务语言理解 (MMLU) 基准上进行的大量实验表明，我们的方法将准确率从 28.9% 提高到 39.3%，与直接在 5 次样本示例上微调的基线相比，提高了 10% 以上。这证明了 LLM 驱动的数据生成和知识提炼对于少样本 MCQA 的有效性。

Title: ScaleOT: Privacy-utility-scalable Offsite-tuning with Dynamic LayerReplace and Selective Rank Compression

• Authors: Kai Yao, Zhaorui Tan, Tiandi Ye, Lichun Li, Yuan Zhao, Wenyan Liu, Wei Wang, Jianke Zhu
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.09812
• Pdf URL: https://arxiv.org/pdf/2412.09812
• Copy Paste: [[2412.09812]] ScaleOT: Privacy-utility-scalable Offsite-tuning with Dynamic LayerReplace and Selective Rank Compression(https://arxiv.org/abs/2412.09812)
• Keywords: language model, llm
• Abstract: Offsite-tuning is a privacy-preserving method for tuning large language models (LLMs) by sharing a lossy compressed emulator from the LLM owners with data owners for downstream task tuning. This approach protects the privacy of both the model and data owners. However, current offsite tuning methods often suffer from adaptation degradation, high computational costs, and limited protection strength due to uniformly dropping LLM layers or relying on expensive knowledge distillation. To address these issues, we propose ScaleOT, a novel privacy-utility-scalable offsite-tuning framework that effectively balances privacy and utility. ScaleOT introduces a novel layerwise lossy compression algorithm that uses reinforcement learning to obtain the importance of each layer. It employs lightweight networks, termed harmonizers, to replace the raw LLM layers. By combining important original LLM layers and harmonizers in different ratios, ScaleOT generates emulators tailored for optimal performance with various model scales for enhanced privacy protection. Additionally, we present a rank reduction method to further compress the original LLM layers, significantly enhancing privacy with negligible impact on utility. Comprehensive experiments show that ScaleOT can achieve nearly lossless offsite tuning performance compared with full fine-tuning while obtaining better model privacy.
• 摘要：异地调优是一种隐私保护方法，用于调整大型语言模型 (LLM)，方法是将 LLM 所有者的有损压缩模拟器与数据所有者共享，以便进行下游任务调整。这种方法保护了模型和数据所有者的隐私。然而，当前的异地调优方法通常会因统一删除 LLM 层或依赖昂贵的知识提炼而遭受适应性下降、计算成本高和保护强度有限的困扰。为了解决这些问题，我们提出了 ScaleOT，这是一种新颖的隐私效用可扩展的异地调优框架，可以有效地平衡隐私和效用。ScaleOT 引入了一种新颖的逐层有损压缩算法，该算法使用强化学习来获得每一层的重要性。它使用轻量级网络（称为协调器）来替换原始 LLM 层。通过以不同的比例组合重要的原始 LLM 层和协调器，ScaleOT 生成具有各种模型规模的定制模拟器，以实现最佳性能，从而增强隐私保护。此外，我们提出了一种降阶方法来进一步压缩原始的 LLM 层，从而显著增强了隐私性，而对实用性的影响却微不足道。综合实验表明，与完全微调相比，ScaleOT 可以实现几乎无损的异地调优性能，同时获得更好的模型隐私性。

Title: MERaLiON-AudioLLM: Technical Report

• Authors: Yingxu He, Zhuohan Liu, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy F. Chen, Ai Ti Aw
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09818
• Pdf URL: https://arxiv.org/pdf/2412.09818
• Copy Paste: [[2412.09818]] MERaLiON-AudioLLM: Technical Report(https://arxiv.org/abs/2412.09818)
• Keywords: language model, llm
• Abstract: We introduce MERaLiON-AudioLLM (Multimodal Empathetic Reasoning and Learning in One Network), the first speech-text model tailored for Singapore's multilingual and multicultural landscape. Developed under the National Large Language Models Funding Initiative, Singapore, MERaLiON-AudioLLM integrates advanced speech and text processing to address the diverse linguistic nuances of local accents and dialects, enhancing accessibility and usability in complex, multilingual environments. Our results demonstrate improvements in both speech recognition and task-specific understanding, positioning MERaLiON-AudioLLM as a pioneering solution for region specific AI applications. We envision this release to set a precedent for future models designed to address localised linguistic and cultural contexts in a global framework.
• 摘要：我们推出了 MERaLiON-AudioLLM（多模态同理心推理和学习网络），这是第一个针对新加坡多语言和多元文化环境量身定制的语音文本模型。MERaLiON-AudioLLM 是在新加坡国家大型语言模型资助计划下开发的，它集成了先进的语音和文本处理功能，可解决当地口音和方言的各种语言细微差别，从而提高了在复杂多语言环境中的可访问性和可用性。我们的结果表明，语音识别和特定任务理解都有所改善，这使得 MERaLiON-AudioLLM 成为区域特定 AI 应用的开创性解决方案。我们预计，这一版本将为未来旨在解决全球框架中的本地化语言和文化背景的模型树立先例。

Title: Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models

• Authors: Changqun Li, Chaofan Ding, Kexin Luan, Xinhan Di
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.09827
• Pdf URL: https://arxiv.org/pdf/2412.09827
• Copy Paste: [[2412.09827]] Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models(https://arxiv.org/abs/2412.09827)
• Keywords: language model
• Abstract: Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low dimensional. Although LoRA has demonstrated commendable performance, there remains a significant performance gap between LoRA and full fine-tuning when learning new tasks. In this work, we propose Low-Rank Adaptation with Task-Relevant Feature Enhancement(LoRATRF) for enhancing task-relevant features from the perspective of editing neural network representations. To prioritize task-relevant features, a task-aware filter that selectively extracts valuable knowledge from hidden representations for the target or current task is designed. As the experiments on a vareity of datasets including NLU, commonsense reasoning and mathematical reasoning tasks demonstrates, our method reduces 33.71% parameters and achieves better performance on a variety of datasets in comparison with SOTA low-rank methods.
• 摘要：以参数高效的方式对预训练的大型语言模型进行微调因其有效性和效率而被广泛研究。LoRA 是最广泛使用的方法之一，它假设优化过程本质上是低维的。尽管 LoRA 表现出了令人称赞的性能，但在学习新任务时，LoRA 与完全微调之间仍然存在显著的性能差距。在这项工作中，我们提出了具有任务相关特征增强的低秩自适应 (LoRATRF)，以从编辑神经网络表示的角度增强任务相关特征。为了优先考虑与任务相关的特征，设计了一个任务感知过滤器，它可以选择性地从隐藏表示中提取对目标或当前任务有价值的知识。正如在包括 NLU、常识推理和数学推理任务在内的各种数据集上的实验所表明的那样，与 SOTA 低秩方法相比，我们的方法减少了 33.71% 的参数，并在各种数据集上取得了更好的性能。

Title: Byte Latent Transformer: Patches Scale Better Than Tokens

• Authors: Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srinivasan Iyer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09871
• Pdf URL: https://arxiv.org/pdf/2412.09871
• Copy Paste: [[2412.09871]] Byte Latent Transformer: Patches Scale Better Than Tokens(https://arxiv.org/abs/2412.09871)
• Keywords: llm
• Abstract: We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.
• 摘要：我们引入了字节潜在变换器 (BLT)，这是一种新的字节级 LLM 架构，它首次在规模上与基于标记化的 LLM 性能相匹配，同时显著提高了推理效率和鲁棒性。BLT 将字节编码为动态大小的补丁，这些补丁作为计算的主要单位。补丁根据下一个字节的熵进行分段，在数据复杂性增加需要时分配更多的计算和模型容量。我们提出了第一个 FLOP 控制的字节级模型扩展研究，该模型最多有 8B 个参数和 4T 个训练字节。我们的结果证明了在没有固定词汇表的情况下扩展在原始字节上训练的模型的可行性。由于在数据可预测时动态选择长补丁，训练和推理效率都得到了提高，同时推理和长尾泛化也得到了定性改进。总体而言，对于固定的推理成本，BLT 通过同时增加补丁和模型大小，显示出比基于标记化的模型更好的扩展性。

Title: On the Limit of Language Models as Planning Formalizers

• Authors: Cassie Huang, Li Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09879
• Pdf URL: https://arxiv.org/pdf/2412.09879
• Copy Paste: [[2412.09879]] On the Limit of Language Models as Planning Formalizers(https://arxiv.org/abs/2412.09879)
• Keywords: language model, llm
• Abstract: Large Language Models have been shown to fail to create executable and verifiable plans in grounded environments. An emerging line of work shows success in using LLM as a formalizer to generate a formal representation (e.g., PDDL) of the planning domain, which can be deterministically solved to find a plan. We systematically evaluate this methodology while bridging some major gaps. While previous work only generates a partial PDDL representation given templated and thus unrealistic environment descriptions, we generate the complete representation given descriptions of various naturalness levels. Among an array of observations critical to improve LLMs' formal planning ability, we note that large enough models can effectively formalize descriptions as PDDL, outperforming those directly generating plans, while being robust to lexical perturbation. As the descriptions become more natural-sounding, we observe a decrease in performance and provide detailed error analysis.
• 摘要：大型语言模型已被证明无法在有根有据的环境中创建可执行和可验证的计划。一项新兴的研究显示，使用 LLM 作为形式化器来生成规划域的形式化表示（例如 PDDL）是成功的，该表示可以通过确定性地求解来找到计划。我们系统地评估了这种方法，同时弥补了一些重大缺陷。虽然以前的工作只在给定模板化且不切实际的环境描述的情况下生成部分 PDDL 表示，但我们在给定各种自然程度的描述的情况下生成完整的表示。在一系列对提高 LLM 的形式规划能力至关重要的观察中，我们注意到足够大的模型可以有效地将描述形式化为 PDDL，其性能优于直接生成计划的模型，同时对词汇扰动具有鲁棒性。随着描述听起来越来越自然，我们观察到性能下降并提供详细的错误分析。

Title: Benchmarking Table Comprehension In The Wild

• Authors: Yikang Pan, Yi Zhu, Rand Xie, Yizhi Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09884
• Pdf URL: https://arxiv.org/pdf/2412.09884
• Copy Paste: [[2412.09884]] Benchmarking Table Comprehension In The Wild(https://arxiv.org/abs/2412.09884)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs), while being increasingly dominant on a myriad of knowledge-intensive activities, have only had limited success understanding lengthy table-text mixtures, such as academic papers and financial reports. Recent advances of long-context LLMs have opened up new possibilities for this field. Nonetheless, we identify two roadblocks: (1) Prior benchmarks of table question answering (TableQA) have focused on isolated tables without context, making it hard to evaluate models in real-world scenarios. (2) Prior benchmarks have focused on some narrow skill sets of table comprehension such as table recognition, data manipulation/calculation, table summarization etc., while a skilled human employs those skills collectively. In this work, we introduce TableQuest, a new benchmark designed to evaluate the holistic table comprehension capabilities of LLMs in the natural table-rich context of financial reports. We employ a rigorous data processing and filtering procedure to ensure that the question-answer pairs are logical, reasonable, and diverse. We experiment with 7 state-of-the-art models, and find that despite reasonable accuracy in locating facts, they often falter when required to execute more sophisticated reasoning or multi-step calculations. We conclude with a qualitative study of the failure modes and discuss the challenges of constructing a challenging benchmark. We make the evaluation data, judging procedure and results of this study publicly available to facilitate research in this field.
• 摘要：大型语言模型 (LLM) 虽然在大量知识密集型活动中占据着越来越重要的地位，但在理解冗长的表格文本混合内容（如学术论文和财务报告）方面却收效甚微。长上下文 LLM 的最新进展为这一领域开辟了新的可能性。尽管如此，我们还是发现了两个障碍：（1）先前的表格问答 (TableQA) 基准测试侧重于没有上下文的孤立表格，因此很难在现实场景中评估模型。（2）先前的基准测试侧重于一些狭窄的表格理解技能组合，如表格识别、数据操作/计算、表格汇总等，而熟练的人会综合运用这些技能。在这项工作中，我们引入了 TableQuest，这是一个新的基准测试，旨在评估 LLM 在自然的、富含表格的财务报告背景下的整体表格理解能力。我们采用严格的数据处理和过滤程序，以确保问答对合乎逻辑、合理且多样化。我们试验了 7 个最先进的模型，发现尽管它们在定位事实方面具有合理的准确性，但在需要执行更复杂的推理或多步骤计算时，它们往往会失败。最后，我们对故障模式进行了定性研究，并讨论了构建具有挑战性的基准的挑战。我们将评估数据、评判程序和这项研究的结果公开，以促进该领域的研究。

Title: Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning

• Authors: Jing Bi, Yuting Wu, Weiwei Xing, Zhenjie Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09906
• Pdf URL: https://arxiv.org/pdf/2412.09906
• Copy Paste: [[2412.09906]] Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning(https://arxiv.org/abs/2412.09906)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks. Advances in prompt engineering and fine-tuning techniques have further enhanced their ability to address complex reasoning challenges. However, these advanced capabilities are often exclusive to models exceeding 100 billion parameters. Although Chain-of-Thought (CoT) fine-tuning methods have been explored for smaller models (under 10 billion parameters), they typically depend on extensive CoT training data, which can introduce inconsistencies and limit effectiveness in low-data settings. To overcome these limitations, this paper introduce a new reasoning strategy Solution Guidance (SG) and a plug-and-play training paradigm Solution-Guidance Fine-Tuning (SGFT) for enhancing the reasoning capabilities of small language models. SG focuses on problem understanding and decomposition at the semantic and logical levels, rather than specific computations, which can effectively improve the SLMs' generalization and reasoning abilities. With only a small amount of SG training data, SGFT can fine-tune a SLM to produce accurate problem-solving guidances, which can then be flexibly fed to any SLM as prompts, enabling it to generate correct answers directly. Experimental results demonstrate that our method significantly improves the performance of SLMs on various reasoning tasks, enhancing both their practicality and efficiency within resource-constrained environments.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出色。快速工程和微调技术的进步进一步增强了它们应对复杂推理挑战的能力。然而，这些高级功能通常仅限于超过 1000 亿个参数的模型。尽管已经针对较小的模型（100 亿个参数以下）探索了思路链 (CoT) 微调方法，但它们通常依赖于大量 CoT 训练数据，这可能会引入不一致性并限制其在低数据设置中的有效性。为了克服这些限制，本文介绍了一种新的推理策略解决方案指导 (SG) 和即插即用的训练范式解决方案指导微调 (SGFT)，以增强小型语言模型的推理能力。SG 专注于语义和逻辑层面的问题理解和分解，而不是特定的计算，这可以有效地提高 SLM 的泛化和推理能力。只需少量 SG 训练数据，SGFT 便可以对 SLM 进行微调，生成准确的问题解决指导，然后可以灵活地将其作为提示输入到任何 SLM，使其直接生成正确答案。实验结果表明，我们的方法显著提高了 SLM 在各种推理任务上的性能，提高了它们在资源受限环境中的实用性和效率。

Title: Low-Resource Fast Text Classification Based on Intra-Class and Inter-Class Distance Calculation

• Authors: Yanxu Mao, Peipei Liu, Tiehan Cui, Congying Liu, Datao You
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09922
• Pdf URL: https://arxiv.org/pdf/2412.09922
• Copy Paste: [[2412.09922]] Low-Resource Fast Text Classification Based on Intra-Class and Inter-Class Distance Calculation(https://arxiv.org/abs/2412.09922)
• Keywords: language model
• Abstract: In recent years, text classification methods based on neural networks and pre-trained models have gained increasing attention and demonstrated excellent performance. However, these methods still have some limitations in practical applications: (1) They typically focus only on the matching similarity between sentences. However, there exists implicit high-value information both within sentences of the same class and across different classes, which is very crucial for classification tasks. (2) Existing methods such as pre-trained language models and graph-based approaches often consume substantial memory for training and text-graph construction. (3) Although some low-resource methods can achieve good performance, they often suffer from excessively long processing times. To address these challenges, we propose a low-resource and fast text classification model called LFTC. Our approach begins by constructing a compressor list for each class to fully mine the regularity information within intra-class data. We then remove redundant information irrelevant to the target classification to reduce processing time. Finally, we compute the similarity distance between text pairs for classification. We evaluate LFTC on 9 publicly available benchmark datasets, and the results demonstrate significant improvements in performance and processing time, especially under limited computational and data resources, highlighting its superior advantages.
• 摘要：近年来，基于神经网络和预训练模型的文本分类方法受到越来越多的关注，并表现出优异的性能。然而，这些方法在实际应用中仍然存在一些局限性：（1）它们通常只关注句子之间的匹配相似度。然而，同一类别的句子内部和不同类别之间都存在隐含的高价值信息，这对于分类任务非常关键。（2）现有的方法，如预训练语言模型和基于图的方法，通常会消耗大量内存进行训练和文本图构建。（3）虽然一些低资源方法可以实现良好的性能，但它们通常会受到处理时间过长的困扰。为了应对这些挑战，我们提出了一种低资源、快速的文本分类模型 LFTC。我们的方法首先为每个类构建一个压缩器列表，以充分挖掘类内数据中的规律信息。然后，我们删除与目标分类无关的冗余信息以减少处理时间。最后，我们计算文本对之间的相似度距离以进行分类。我们在 9 个公开可用的基准数据集上对 LFTC 进行评估，结果表明性能和处理时间有显著提高，尤其是在有限的计算和数据资源下，凸显了其卓越的优势。

Title: Enhancing Nursing and Elderly Care with Large Language Models: An AI-Driven Framework

• Authors: Qiao Sun, Jiexin Xie, Nanyang Ye, Qinying Gu, Shijie Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09946
• Pdf URL: https://arxiv.org/pdf/2412.09946
• Copy Paste: [[2412.09946]] Enhancing Nursing and Elderly Care with Large Language Models: An AI-Driven Framework(https://arxiv.org/abs/2412.09946)
• Keywords: language model, llm
• Abstract: This paper explores the application of large language models (LLMs) in nursing and elderly care, focusing on AI-driven patient monitoring and interaction. We introduce a novel Chinese nursing dataset and implement incremental pre-training (IPT) and supervised fine-tuning (SFT) techniques to enhance LLM performance in specialized tasks. Using LangChain, we develop a dynamic nursing assistant capable of real-time care and personalized interventions. Experimental results demonstrate significant improvements, paving the way for AI-driven solutions to meet the growing demands of healthcare in aging populations.
• 摘要：本文探讨了大型语言模型 (LLM) 在护理和老年护理中的应用，重点关注人工智能驱动的患者监控和互动。我们引入了一个新颖的中文护理数据集，并实施了增量预训练 (IPT) 和监督微调 (SFT) 技术，以提高 LLM 在专门任务中的表现。使用 LangChain，我们开发了一个动态护理助手，能够进行实时护理和个性化干预。实验结果表明取得了显着的改进，为人工智能驱动的解决方案铺平了道路，以满足老龄化人口日益增长的医疗保健需求。

Title: Small Language Model as Data Prospector for Large Language Model

• Authors: Shiwen Ni, Haihong Wu, Di Yang, Qiang Qu, Hamid Alinejad-Rokny, Min Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09990
• Pdf URL: https://arxiv.org/pdf/2412.09990
• Copy Paste: [[2412.09990]] Small Language Model as Data Prospector for Large Language Model(https://arxiv.org/abs/2412.09990)
• Keywords: language model, llm
• Abstract: The quality of instruction data directly affects the performance of fine-tuned Large Language Models (LLMs). Previously, \cite{li2023one} proposed \texttt{NUGGETS}, which identifies and selects high-quality quality data from a large dataset by identifying those individual instruction examples that can significantly improve the performance of different tasks after being learnt as one-shot instances. In this work, we propose \texttt{SuperNUGGETS}, an improved variant of \texttt{NUGGETS} optimised for efficiency and performance. Our \texttt{SuperNUGGETS} uses a small language model (SLM) instead of a large language model (LLM) to filter the data for outstanding one-shot instances and refines the predefined set of tests. The experimental results show that the performance of \texttt{SuperNUGGETS} only decreases by 1-2% compared to \texttt{NUGGETS}, but the efficiency can be increased by a factor of 58. Compared to the original \texttt{NUGGETS}, our \texttt{SuperNUGGETS} has a higher utility value due to the significantly lower resource consumption.
• 摘要：教学数据的质量直接影响经过微调的大型语言模型 (LLM) 的性能。之前，\cite{li2023one} 提出了 \texttt{NUGGETS}，它通过识别那些在作为一次性实例学习后可以显著提高不同任务性能的单个教学示例，从大型数据集中识别和选择高质量数据。在这项工作中，我们提出了 \texttt{SuperNUGGETS}，这是 \texttt{NUGGETS} 的改进版本，针对效率和性能进行了优化。我们的 \texttt{SuperNUGGETS} 使用小型语言模型 (SLM) 而不是大型语言模型 (LLM) 来过滤数据以查找出色的一次性实例并改进预定义的测试集。实验结果表明，\texttt{SuperNUGGETS}的性能相对于\texttt{NUGGETS}仅下降1-2%，但效率可以提高58倍。与原版\texttt{NUGGETS}相比，我们的\texttt{SuperNUGGETS}由于资源消耗明显降低，具有更高的实用价值。

Title: A Comparative Study of LLMs, NMT Models, and Their Combination in Persian-English Idiom Translation

• Authors: Sara Rezaeimanesh, Faezeh Hosseini, Yadollah Yaghoobzadeh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09993
• Pdf URL: https://arxiv.org/pdf/2412.09993
• Copy Paste: [[2412.09993]] A Comparative Study of LLMs, NMT Models, and Their Combination in Persian-English Idiom Translation(https://arxiv.org/abs/2412.09993)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have shown superior capabilities in translating figurative language compared to neural machine translation (NMT) systems. However, the impact of different prompting methods and LLM-NMT combinations on idiom translation has yet to be thoroughly investigated. This paper introduces two parallel datasets of sentences containing idiomatic expressions for Persian$\rightarrow$English and English$\rightarrow$Persian translations, with Persian idioms sampled from our PersianIdioms resource, a collection of 2,200 idioms and their meanings. Using these datasets, we evaluate various open- and closed-source LLMs, NMT models, and their combinations. Translation quality is assessed through idiom translation accuracy and fluency. We also find that automatic evaluation methods like LLM-as-a-judge, BLEU and BERTScore are effective for comparing different aspects of model performance. Our experiments reveal that Claude-3.5-Sonnet delivers outstanding results in both translation directions. For English$\rightarrow$Persian, combining weaker LLMs with Google Translate improves results, while Persian$\rightarrow$English translations benefit from single prompts for simpler models and complex prompts for advanced ones.
• 摘要：与神经机器翻译 (NMT) 系统相比，大型语言模型 (LLM) 在翻译比喻性语言方面表现出了卓越的能力。然而，不同的提示方法和 LLM-NMT 组合对习语翻译的影响尚未得到彻底研究。本文介绍了两个包含波斯语$\rightarrow$英语和英语$\rightarrow$波斯语翻译的习语表达的平行句子数据集，其中波斯语习语取自我们的 PersianIdioms 资源，该资源收集了 2,200 个习语及其含义。使用这些数据集，我们评估了各种开源和闭源 LLM、NMT 模型及其组合。翻译质量通过习语翻译的准确性和流畅性来评估。我们还发现，LLM-as-a-judge、BLEU 和 BERTScore 等自动评估方法对于比较模型性能的不同方面非常有效。我们的实验表明，Claude-3.5-Sonnet 在两个翻译方向上都取得了出色的成绩。对于英语$\rightarrow$波斯语，将较弱的 LLM 与谷歌翻译相结合可以改善结果，而波斯语$\rightarrow$英语翻译则受益于简单模型的单一提示和高级模型的复杂提示。

Title: Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language

• Authors: Anastasia Zhukova, Christian E. Matt, Bela Gipp
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10008
• Pdf URL: https://arxiv.org/pdf/2412.10008
• Copy Paste: [[2412.10008]] Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language(https://arxiv.org/abs/2412.10008)
• Keywords: llm
• Abstract: Domain-specific languages that use a lot of specific terminology often fall into the category of low-resource languages. Collecting test datasets in a narrow domain is time-consuming and requires skilled human resources with domain knowledge and training for the annotation task. This study addresses the challenge of automated collecting test datasets to evaluate semantic search in low-resource domain-specific German language of the process industry. Our approach proposes an end-to-end annotation pipeline for automated query generation to the score reassessment of query-document pairs. To overcome the lack of text encoders trained in the German chemistry domain, we explore a principle of an ensemble of "weak" text encoders trained on common knowledge datasets. We combine individual relevance scores from diverse models to retrieve document candidates and relevance scores generated by an LLM, aiming to achieve consensus on query-document alignment. Evaluation results demonstrate that the ensemble method significantly improves alignment with human-assigned relevance scores, outperforming individual models in both inter-coder agreement and accuracy metrics. These findings suggest that ensemble learning can effectively adapt semantic search systems for specialized, low-resource languages, offering a practical solution to resource limitations in domain-specific contexts.
• 摘要：使用大量特定术语的领域特定语言通常属于低资源语言类别。在狭窄领域中收集测试数据集非常耗时，需要具有领域知识和注释任务培训的熟练人力资源。本研究解决了自动收集测试数据集以评估过程工业低资源领域特定德语中的语义搜索的挑战。我们的方法提出了一种端到端注释管道，用于自动查询生成和查询文档对的分数重新评估。为了克服在德国化学领域训练的文本编码器的不足，我们探索了在共同知识数据集上训练的“弱”文本编码器集合的原理。我们结合了来自不同模型的单个相关性分数来检索文档候选和由 LLM 生成的相关性分数，旨在就查询文档对齐达成共识。评估结果表明，集成方法显著提高了与人类分配的相关性分数的对齐，在编码器间一致性和准确性指标方面均优于单个模型。这些发现表明，集成学习可以有效地调整专门的、低资源语言的语义搜索系统，为特定领域环境中的资源限制提供切实可行的解决方案。

Title: GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?

• Authors: Zhikai Lei, Tianyi Liang, Hanglei Hu, Jin Zhang, Yunhua Zhou, Yunfan Shao, Linyang Li, Chenchui Li, Changbo Wang, Hang Yan, Qipeng Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10056
• Pdf URL: https://arxiv.org/pdf/2412.10056
• Copy Paste: [[2412.10056]] GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?(https://arxiv.org/abs/2412.10056)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are commonly evaluated using human-crafted benchmarks, under the premise that higher scores implicitly reflect stronger human-like performance. However, there is growing concern that LLMs may ``game" these benchmarks due to data leakage, achieving high scores while struggling with tasks simple for humans. To substantively address the problem, we create GAOKAO-Eval, a comprehensive benchmark based on China's National College Entrance Examination (Gaokao), and conduct ``closed-book" evaluations for representative models released prior to Gaokao. Contrary to prevailing consensus, even after addressing data leakage and comprehensiveness, GAOKAO-Eval reveals that high scores still fail to truly reflect human-aligned capabilities. To better understand this mismatch, We introduce the Rasch model from cognitive psychology to analyze LLM scoring patterns and identify two key discrepancies: 1) anomalous consistent performance across various question difficulties, and 2) high variance in performance on questions of similar difficulty. In addition, We identified inconsistent grading of LLM-generated answers among teachers and recurring mistake patterns. we find that the phenomenons are well-grounded in the motivations behind OpenAI o1, and o1's reasoning-as-difficulties can mitigate the mismatch. These results show that GAOKAO-Eval can reveal limitations in LLM capabilities not captured by current benchmarks and highlight the need for more LLM-aligned difficulty analysis.
• 摘要：大型语言模型 (LLM) 通常使用人工制定的基准进行评估，前提是更高的分数隐含地反映了更强的类人性能。然而，人们越来越担心 LLM 可能会由于数据泄露而“玩弄”这些基准，在完成对人类来说简单的任务时却很难获得高分。为了实质性地解决这个问题，我们创建了 GAOKAO-Eval，这是一个基于中国全国大学入学考试 (高考) 的综合基准，并对高考之前发布的代表性模型进行“闭卷”评估。与普遍的共识相反，即使解决了数据泄露和全面性问题，GAOKAO-Eval 仍然表明高分仍然无法真正反映与人类一致的能力。为了更好地理解这种不匹配，我们引入了认知心理学中的 Rasch 模型来分析 LLM 评分模式，并确定了两个关键差异：1) 在不同问题难度上表现异常一致，2) 在难度相似的问题上表现差异很大。此外，我们发现教师对 LLM 生成的答案的评分不一致，并且错误模式反复出现。我们发现这些现象在 OpenAI o1 背后的动机中是有充分依据的，而 o1 的推理难度可以缓解这种不匹配。这些结果表明，GAOKAO-Eval 可以揭示当前基准未捕捉到的 LLM 能力的局限性，并强调需要进行更多与 LLM 一致的难度分析。

Title: Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

• Authors: George Arthur Baker, Ankush Raut, Sagi Shaier, Lawrence E Hunter, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10079
• Pdf URL: https://arxiv.org/pdf/2412.10079
• Copy Paste: [[2412.10079]] Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA(https://arxiv.org/abs/2412.10079)
• Keywords: language model, long context, prompt, chain-of-thought
• Abstract: Previous work finds that recent long-context language models fail to make equal use of information in the middle of their inputs, preferring pieces of information located at the tail ends which creates an undue bias in situations where we would like models to be equally capable of using different parts of the input. Thus far, the problem has mainly only been considered in settings with single pieces of critical information, leading us to question what happens when multiple necessary pieces of information are spread out over the inputs. Here, we demonstrate the effects of the "lost in the middle" problem in the multi-hop question answering setting -- in which multiple reasoning "hops" over disconnected documents are required -- and show that performance degrades not only with respect to the distance of information from the edges of the context, but also between pieces of information. Additionally, we experiment with means of alleviating the problem by reducing superfluous document contents through knowledge graph triple extraction and summarization, and prompting models to reason more thoroughly using chain-of-thought prompting.
• 摘要：先前的研究发现，最近的长上下文语言模型未能平等地利用输入中间的信息，而是偏向于位于尾端的信息，这在我们希望模型能够平等地使用输入的不同部分的情况下产生了过度的偏见。到目前为止，这个问题主要只在具有单个关键信息的设置中考虑，这让我们质疑当多个必要的信息分散在输入中时会发生什么。在这里，我们展示了“迷失在中间”问题在多跳问答设置中的影响——其中需要在断开连接的文档上进行多个推理“跳跃”——并表明性能不仅会随着信息与上下文边缘的距离而下降，而且会随着信息之间的距离而下降。此外，我们还尝试了通过知识图谱三重提取和总结减少多余文档内容以及使用思路链提示促使模型进行更彻底的推理来缓解问题的方法。

Title: RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector

• Authors: Zhensheng Wang, Wenmian Yang, Kun Zhou, Yiquan Zhang, Weijia Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10104
• Pdf URL: https://arxiv.org/pdf/2412.10104
• Copy Paste: [[2412.10104]] RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector(https://arxiv.org/abs/2412.10104)
• Keywords: language model
• Abstract: The real estate market relies heavily on structured data, such as property details, market trends, and price fluctuations. However, the lack of specialized Tabular Question Answering datasets in this domain limits the development of automated question-answering systems. To fill this gap, we introduce RETQA, the first large-scale open-domain Chinese Tabular Question Answering dataset for Real Estate. RETQA comprises 4,932 tables and 20,762 question-answer pairs across 16 sub-fields within three major domains: property information, real estate company finance information and land auction information. Compared with existing tabular question answering datasets, RETQA poses greater challenges due to three key factors: long-table structures, open-domain retrieval, and multi-domain queries. To tackle these challenges, we propose the SLUTQA framework, which integrates large language models with spoken language understanding tasks to enhance retrieval and answering accuracy. Extensive experiments demonstrate that SLUTQA significantly improves the performance of large language models on RETQA by in-context learning. RETQA and SLUTQA provide essential resources for advancing tabular question answering research in the real estate domain, addressing critical challenges in open-domain and long-table question-answering. The dataset and code are publicly available at \url{this https URL}.
• 摘要：房地产市场高度依赖结构化数据，如房产详情、市场趋势和价格波动。然而，该领域缺乏专门的表格问答数据集，限制了自动问答系统的发展。为了填补这一空白，我们推出了第一个大规模开放领域中文房地产表格问答数据集 RETQA。RETQA 包含 4,932 个表格和 20,762 个问答对，涵盖三大领域的 16 个子领域：房产信息、房地产公司财务信息和土地拍卖信息。与现有的表格问答数据集相比，RETQA 因三个关键因素而面临更大的挑战：长表结构、开放域检索和多域查询。为了应对这些挑战，我们提出了 SLUTQA 框架，它将大型语言模型与口语理解任务相结合，以提高检索和回答的准确性。大量实验表明，SLUTQA 通过上下文学习显著提高了大型语言模型在 RETQA 上的性能。 RETQA 和 SLUTQA 为推进房地产领域的表格问答研究提供了重要资源，解决了开放域和长表问答中的关键挑战。数据集和代码可在 \url{此 https URL} 上公开获取。

Title: MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset

• Authors: Sagi Shaier, George Arthur Baker, Chiranthan Sridhar, Lawrence E Hunter, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10105
• Pdf URL: https://arxiv.org/pdf/2412.10105
• Copy Paste: [[2412.10105]] MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset(https://arxiv.org/abs/2412.10105)
• Keywords: language model, prompt
• Abstract: Language models (LMs) have excelled in various broad domains. However, to ensure their safe and effective integration into real-world educational settings, they must demonstrate proficiency in specific, granular areas of knowledge. Existing cloze-style benchmarks, commonly used to evaluate LMs' knowledge, have three major limitations. They: 1) do not cover the educational domain; 2) typically focus on low-complexity, generic knowledge or broad domains, which do not adequately assess the models' knowledge in specific subjects; and 3) often rely on templates that can bias model predictions. Here, we introduce MALAMUTE, a multilingual, template-free, and highly granular probing dataset comprising expert-written, peer-reviewed probes from 71 university-level textbooks across three languages (English, Spanish, and Polish). MALAMUTE is the first education-based cloze-style dataset. It covers eight domains, each with up to 14 subdomains, further broken down into concepts and concept-based prompts, totaling 33,361 university curriculum concepts and 116,887 prompts. MALAMUTE's fine granularity, educational focus, and inclusion of both sentence-level and paragraph-level prompts make it an ideal tool for evaluating LMs' course-related knowledge. Our evaluation of masked and causal LMs on MALAMUTE shows that despite overall proficiency, they have significant gaps in knowledge when examined closely on specific subjects, hindering their safe use in classrooms and underscoring the need for further development.
• 摘要：语言模型 (LM) 在各个广泛领域都表现出色。然而，为了确保它们能够安全有效地融入现实世界的教育环境，它们必须展示出在特定、细粒度的知识领域的熟练程度。现有的完形填空式基准通常用于评估 LM 的知识，但存在三大局限性。它们：1) 不涵盖教育领域；2) 通常侧重于低复杂度、通用知识或广泛领域，无法充分评估模型在特定学科的知识；3) 通常依赖于可能影响模型预测的模板。在这里，我们介绍了 MALAMUTE，这是一个多语言、无模板且高度细粒度的探测数据集，包含来自三种语言（英语、西班牙语和波兰语）的 71 本大学级教科书的专家编写、同行评审的探测。MALAMUTE 是第一个基于教育的完形填空式数据集。它涵盖八个领域，每个领域最多有 14 个子领域，进一步细分为概念和基于概念的提示，总共有 33,361 个大学课程概念和 116,887 个提示。MALAMUTE 的精细粒度、教育重点以及句子级和段落级提示的包含使其成为评估 LM 课程相关知识的理想工具。我们对 MALAMUTE 上的蒙面和因果 LM 的评估表明，尽管总体上熟练，但在仔细检查特定主题时，他们在知识方面存在重大差距，阻碍了他们在课堂上的安全使用，并强调需要进一步发展。

Title: ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers

• Authors: Junyan Hu, Xue Xiao, Mengqi Zhang, Xiao Chen, Zhaochun Ren, Zhumin Chen, Pengjie Ren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10135
• Pdf URL: https://arxiv.org/pdf/2412.10135
• Copy Paste: [[2412.10135]] ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers(https://arxiv.org/abs/2412.10135)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) grow in size, traditional full fine-tuning becomes increasingly impractical due to its high computational and storage costs. Although popular parameter-efficient fine-tuning methods, such as LoRA, have significantly reduced the number of tunable parameters, there is still room for further optimization. In this work, we propose ASLoRA, a cross-layer parameter-sharing strategy combining global sharing with partial adaptive sharing. Specifically, we share the low-rank matrix A across all layers and adaptively merge matrix B during training. This sharing mechanism not only mitigates overfitting effectively but also captures inter-layer dependencies, significantly enhancing the model's representational capability. We conduct extensive experiments on various NLP tasks, showing that ASLoRA outperforms LoRA while using less than 25% of the parameters, highlighting its flexibility and superior parameter efficiency. Furthermore, in-depth analyses of the adaptive sharing strategy confirm its significant advantages in enhancing both model flexibility and task adaptability.
• 摘要：随着大型语言模型 (LLM) 的规模不断扩大，传统的完全微调由于计算和存储成本高昂而变得越来越不切实际。尽管流行的参数高效微调方法（例如 LoRA）已显着减少了可调参数的数量，但仍有进一步优化的空间。在这项工作中，我们提出了 ASLoRA，这是一种将全局共享与部分自适应共享相结合的跨层参数共享策略。具体来说，我们在所有层之间共享低秩矩阵 A，并在训练期间自适应地合并矩阵 B。这种共享机制不仅可以有效缓解过度拟合，还可以捕获层间依赖关系，从而显着增强模型的表示能力。我们在各种 NLP 任务上进行了广泛的实验，结果表明 ASLoRA 在使用不到 25% 的参数的情况下优于 LoRA，突出了其灵活性和卓越的参数效率。此外，对自适应共享策略的深入分析证实了其在增强模型灵活性和任务适应性方面的显著优势。

Title: Can LLMs Convert Graphs to Text-Attributed Graphs?

• Authors: Zehong Wang, Sidney Liu, Zheyuan Zhang, Tianyi Ma, Chuxu Zhang, Yanfang Ye
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10136
• Pdf URL: https://arxiv.org/pdf/2412.10136
• Copy Paste: [[2412.10136]] Can LLMs Convert Graphs to Text-Attributed Graphs?(https://arxiv.org/abs/2412.10136)
• Keywords: language model, llm
• Abstract: Graphs are ubiquitous data structures found in numerous real-world applications, such as drug discovery, recommender systems, and social network analysis. Graph neural networks (GNNs) have become a popular tool to learn node embeddings through message passing on these structures. However, a significant challenge arises when applying GNNs to multiple graphs with different feature spaces, as existing GNN architectures are not designed for cross-graph feature alignment. To address this, recent approaches introduce text-attributed graphs, where each node is associated with a textual description, enabling the use of a shared textual encoder to project nodes from different graphs into a unified feature space. While promising, this method relies heavily on the availability of text-attributed data, which can be difficult to obtain in practice. To bridge this gap, we propose a novel method named Topology-Aware Node description Synthesis (TANS), which leverages large language models (LLMs) to automatically convert existing graphs into text-attributed graphs. The key idea is to integrate topological information with each node's properties, enhancing the LLMs' ability to explain how graph topology influences node semantics. We evaluate our TANS on text-rich, text-limited, and text-free graphs, demonstrating that it enables a single GNN to operate across diverse graphs. Notably, on text-free graphs, our method significantly outperforms existing approaches that manually design node features, showcasing the potential of LLMs for preprocessing graph-structured data, even in the absence of textual information. The code and data are available at this https URL.
• 摘要：图是许多实际应用中无处不在的数据结构，例如药物发现、推荐系统和社交网络分析。图神经网络 (GNN) 已成为一种流行的工具，通过在这些结构上传递消息来学习节点嵌入。然而，当将 GNN 应用于具有不同特征空间的多个图时，会出现一个重大挑战，因为现有的 GNN 架构不是为跨图特征对齐而设计的。为了解决这个问题，最近的方法引入了文本属性图，其中每个节点都与一个文本描述相关联，从而可以使用共享的文本编码器将不同图中的节点投影到统一的特征空间中。虽然这种方法很有前景，但它严重依赖于文本属性数据的可用性，而这在实践中很难获得。为了弥补这一差距，我们提出了一种名为拓扑感知节点描述合成 (TANS) 的新方法，它利用大型语言模型 (LLM) 将现有图自动转换为文本属性图。关键思想是将拓扑信息与每个节点的属性相结合，从而增强 LLM 解释图拓扑如何影响节点语义的能力。我们在文本丰富、文本有限和无文本图上评估了我们的 TANS，表明它使单个 GNN 能够跨不同的图运行。值得注意的是，在无文本图上，我们的方法明显优于现有的手动设计节点特征的方法，展示了 LLM 在预处理图结构数据方面的潜力，即使在没有文本信息的情况下也是如此。代码和数据可在此 https URL 上找到。

Title: ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL

• Authors: Yang Qin, Chao Chen, Zhihang Fu, Ze Chen, Dezhong Peng, Peng Hu, Jieping Ye
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10138
• Pdf URL: https://arxiv.org/pdf/2412.10138
• Copy Paste: [[2412.10138]] ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL(https://arxiv.org/abs/2412.10138)
• Keywords: language model, gpt, llm, hallucination, prompt, agent
• Abstract: Despite the significant advancements in Text-to-SQL (Text2SQL) facilitated by large language models (LLMs), the latest state-of-the-art techniques are still trapped in the in-context learning of closed-source LLMs (e.g., GPT-4), which limits their applicability in open scenarios. To address this challenge, we propose a novel RObust mUltitask Tuning and collaboration mEthod (ROUTE) to improve the comprehensive capabilities of open-source LLMs for Text2SQL, thereby providing a more practical solution. Our approach begins with multi-task supervised fine-tuning (SFT) using various synthetic training data related to SQL generation. Unlike existing SFT-based Text2SQL methods, we introduced several additional SFT tasks, including schema linking, noise correction, and continuation writing. Engaging in a variety of SQL generation tasks enhances the model's understanding of SQL syntax and improves its ability to generate high-quality SQL queries. Additionally, inspired by the collaborative modes of LLM agents, we introduce a Multitask Collaboration Prompting (MCP) strategy. This strategy leverages collaboration across several SQL-related tasks to reduce hallucinations during SQL generation, thereby maximizing the potential of enhancing Text2SQL performance through explicit multitask capabilities. Extensive experiments and in-depth analyses have been performed on eight open-source LLMs and five widely-used benchmarks. The results demonstrate that our proposal outperforms the latest Text2SQL methods and yields leading performance.
• 摘要：尽管大型语言模型 (LLM) 推动了文本到 SQL (Text2SQL) 的重大进步，但最新的最先进技术仍然局限于闭源 LLM（例如 GPT-4）的上下文学习，这限制了它们在开放场景中的适用性。为了应对这一挑战，我们提出了一种新颖的 RObust mUltitask Tuning 和协作方法 (ROUTE)，以提高开源 LLM 对 Text2SQL 的综合能力，从而提供更实用的解决方案。我们的方法从使用与 SQL 生成相关的各种合成训练数据进行多任务监督微调 (SFT) 开始。与现有的基于 SFT 的 Text2SQL 方法不同，我们引入了几个额外的 SFT 任务，包括模式链接、噪声校正和连续写入。参与各种 SQL 生成任务可增强模型对 SQL 语法的理解，并提高其生成高质量 SQL 查询的能力。此外，受 LLM 代理的协作模式的启发，我们引入了多任务协作提示 (MCP) 策略。此策略利用多个与 SQL 相关的任务之间的协作来减少 SQL 生成过程中的幻觉，从而最大限度地发挥通过显式多任务功能增强 Text2SQL 性能的潜力。我们在八个开源 LLM 和五个广泛使用的基准上进行了广泛的实验和深入分析。结果表明，我们的提案优于最新的 Text2SQL 方法并获得了领先的性能。

Title: TACOMORE: Leveraging the Potential of LLMs in Corpus-based Discourse Analysis with Prompt Engineering

• Authors: Bingru Li, Han Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10139
• Pdf URL: https://arxiv.org/pdf/2412.10139
• Copy Paste: [[2412.10139]] TACOMORE: Leveraging the Potential of LLMs in Corpus-based Discourse Analysis with Prompt Engineering(https://arxiv.org/abs/2412.10139)
• Keywords: gpt, llm, hallucination, prompt
• Abstract: The capacity of LLMs to carry out automated qualitative analysis has been questioned by corpus linguists, and it has been argued that corpus-based discourse analysis incorporating LLMs is hindered by issues of unsatisfying performance, hallucination, and irreproducibility. Our proposed method, TACOMORE, aims to address these concerns by serving as an effective prompting framework in this domain. The framework consists of four principles, i.e., Task, Context, Model and Reproducibility, and specifies five fundamental elements of a good prompt, i.e., Role Description, Task Definition, Task Procedures, Contextual Information and Output Format. We conduct experiments on three LLMs, i.e., GPT-4o, Gemini-1.5-Pro and this http URL, and find that TACOMORE helps improve LLM performance in three representative discourse analysis tasks, i.e., the analysis of keywords, collocates and concordances, based on an open corpus of COVID-19 research articles. Our findings show the efficacy of the proposed prompting framework TACOMORE in corpus-based discourse analysis in terms of Accuracy, Ethicality, Reasoning, and Reproducibility, and provide novel insights into the application and evaluation of LLMs in automated qualitative studies.
• 摘要：语料库语言学家对 LLM 进行自动定性分析的能力提出了质疑，并认为包含 LLM 的基于语料库的话语分析受到性能不令人满意、幻觉和不可重复性问题的阻碍。我们提出的方法 TACOMORE 旨在通过充当该领域的有效提示框架来解决这些问题。该框架由四项原则组成，即任务、上下文、模型和可重复性，并指定了良好提示的五个基本要素，即角色描述、任务定义、任务程序、上下文信息和输出格式。我们对三个 LLM（即 GPT-4o、Gemini-1.5-Pro 和此 http URL）进行了实验，发现 TACOMORE 有助于提高 LLM 在三个代表性话语分析任务（即基于 COVID-19 研究文章开放语料库的关键词分析、搭配和一致性分析）中的表现。我们的研究结果表明，所提出的提示框架 TACOMORE 在基于语料库的话语分析中在准确性、道德性、推理性和可重复性方面的有效性，并为 LLM 在自动化定性研究中的应用和评估提供了新的见解。

Title: Retrieval-Augmented Semantic Parsing: Using Large Language Models to Improve Generalization

• Authors: Xiao Zhang, Qianru Meng, Johan Bos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10207
• Pdf URL: https://arxiv.org/pdf/2412.10207
• Copy Paste: [[2412.10207]] Retrieval-Augmented Semantic Parsing: Using Large Language Models to Improve Generalization(https://arxiv.org/abs/2412.10207)
• Keywords: language model, llm
• Abstract: Open-domain semantic parsing remains a challenging task, as models often rely on heuristics and struggle to handle unseen concepts. In this paper, we investigate the potential of large language models (LLMs) for this task and introduce Retrieval-Augmented Semantic Parsing (RASP), a simple yet effective approach that integrates external lexical knowledge into the parsing process. Our experiments not only show that LLMs outperform previous encoder-decoder baselines for semantic parsing, but that RASP further enhances their ability to predict unseen concepts, nearly doubling the performance of previous models on out-of-distribution concepts. These findings highlight the promise of leveraging large language models and retrieval mechanisms for robust and open-domain semantic parsing.
• 摘要：开放域语义解析仍然是一项具有挑战性的任务，因为模型通常依赖于启发式方法，并且难以处理看不见的概念。在本文中，我们研究了大型语言模型 (LLM) 完成这项任务的潜力，并介绍了检索增强语义解析 (RASP)，这是一种简单而有效的方法，将外部词汇知识集成到解析过程中。我们的实验不仅表明 LLM 在语义解析方面的表现优于以前的编码器-解码器基线，而且 RASP 进一步增强了它们预测看不见的概念的能力，几乎使以前模型在分布外概念上的表现翻了一番。这些发现凸显了利用大型语言模型和检索机制实现稳健和开放域语义解析的前景。

Title: How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives

• Authors: Timour Ichmoukhamedov, James Hinns, David Martens
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10220
• Pdf URL: https://arxiv.org/pdf/2412.10220
• Copy Paste: [[2412.10220]] How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives(https://arxiv.org/abs/2412.10220)
• Keywords: llm, hallucination, prompt
• Abstract: A rapidly developing application of LLMs in XAI is to convert quantitative explanations such as SHAP into user-friendly narratives to explain the decisions made by smaller prediction models. Evaluating the narratives without relying on human preference studies or surveys is becoming increasingly important in this field. In this work we propose a framework and explore several automated metrics to evaluate LLM-generated narratives for explanations of tabular classification tasks. We apply our approach to compare several state-of-the-art LLMs across different datasets and prompt types. As a demonstration of their utility, these metrics allow us to identify new challenges related to LLM hallucinations for XAI narratives.
• 摘要：LLM 在 XAI 中快速发展的一个应用是将定量解释（例如 SHAP）转换为用户友好的叙述，以解释较小预测模型做出的决策。在这一领域，不依赖人类偏好研究或调查来评估叙述变得越来越重要。在这项工作中，我们提出了一个框架并探索了几种自动化指标来评估 LLM 生成的叙述，以解释表格分类任务。我们应用我们的方法来比较不同数据集和提示类型的几种最先进的 LLM。作为其实用性的证明，这些指标使我们能够识别与 XAI 叙述的 LLM 幻觉相关的新挑战。

Title: Efficient Continual Pre-training of LLMs for Low-resource Languages

• Authors: Arijit Nag, Soumen Chakrabarti, Animesh Mukherjee, Niloy Ganguly
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10244
• Pdf URL: https://arxiv.org/pdf/2412.10244
• Copy Paste: [[2412.10244]] Efficient Continual Pre-training of LLMs for Low-resource Languages(https://arxiv.org/abs/2412.10244)
• Keywords: language model, llm
• Abstract: Open-source Large Language models (OsLLMs) propel the democratization of natural language research by giving the flexibility to augment or update model parameters for performance improvement. Nevertheless, like proprietary LLMs, Os-LLMs offer poorer performance on low-resource languages (LRLs) than high-resource languages (HRLs), owing to smaller amounts of training data and underrepresented vocabulary. On the other hand, continual pre-training (CPT) with large amounts of language-specific data is a costly proposition in terms of data acquisition and computational resources. Our goal is to drastically reduce CPT cost. To that end, we first develop a new algorithm to select a subset of texts from a larger corpus. We show the effectiveness of our technique using very little CPT data. In search of further improvement, we design a new algorithm to select tokens to include in the LLM vocabulary. We experiment with the recent Llama-3 model and nine Indian languages with diverse scripts and extent of resource availability. For evaluation, we use IndicGenBench, a generation task benchmark dataset for Indic languages. We experiment with various CPT corpora and augmented vocabulary size and offer insights across language families.
• 摘要：开源大型语言模型 (OsLLM) 推动了自然语言研究的民主化，因为它提供了增强或更新模型参数以提高性能的灵活性。然而，与专有 LLM 一样，由于训练数据量较少且词汇量不足，Os-LLM 在低资源语言 (LRL) 上的表现比高资源语言 (HRL) 差。另一方面，使用大量特定语言数据进行持续预训练 (CPT) 在数据获取和计算资源方面是一项昂贵的提议。我们的目标是大幅降低 CPT 成本。为此，我们首先开发了一种新算法来从更大的语料库中选择文本子集。我们使用很少的 CPT 数据展示了我们技术的有效性。为了进一步改进，我们设计了一种新算法来选择要包含在 LLM 词汇表中的标记。我们尝试了最近的 Llama-3 模型和九种具有不同脚本和资源可用性范围的印度语言。为了进行评估，我们使用了 IndicGenBench，这是针对印度语的生成任务基准数据集。我们尝试了各种 CPT 语料库和增强的词汇量，并提供了跨语系的见解。

Title: Targeted Angular Reversal of Weights (TARS) for Knowledge Removal in Large Language Models

• Authors: Harry J. Davies, Giorgos Iacovides, Danilo P. Mandic
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10257
• Pdf URL: https://arxiv.org/pdf/2412.10257
• Copy Paste: [[2412.10257]] Targeted Angular Reversal of Weights (TARS) for Knowledge Removal in Large Language Models(https://arxiv.org/abs/2412.10257)
• Keywords: language model, llm, prompt
• Abstract: The sheer scale of data required to train modern large language models (LLMs) poses significant risks, as models are likely to gain knowledge of sensitive topics such as bio-security, as well the ability to replicate copyrighted works. Methods designed to remove such knowledge must do so from all prompt directions, in a multi-lingual capacity and without degrading general model performance. To this end, we introduce the targeted angular reversal (TARS) method of knowledge removal from LLMs. The TARS method firstly leverages the LLM in combination with a detailed prompt to aggregate information about a selected concept in the internal representation space of the LLM. It then refines this approximate concept vector to trigger the concept token with high probability, by perturbing the approximate concept vector with noise and transforming it into token scores with the language model head. The feedforward weight vectors in the LLM which operate directly on the internal representation space, and have the highest cosine similarity with this targeting vector, are then replaced by a reversed targeting vector, thus limiting the ability of the concept to propagate through the model. The modularity of the TARS method allows for a sequential removal of concepts from Llama 3.1 8B, such as the famous literary detective Sherlock Holmes, and the planet Saturn. It is demonstrated that the probability of triggering target concepts can be reduced to 0.00 with as few as 1 TARS edit, whilst simultaneously removing the knowledge bi-directionally. Moreover, knowledge is shown to be removed across all languages despite only being targeted in English. Importantly, TARS has minimal impact on the general model capabilities, as after removing 5 diverse concepts in a modular fashion, there is minimal KL divergence in the next token probabilities of the LLM on large corpora of Wikipedia text (median of 0.002).
• 摘要：训练现代大型语言模型 (LLM) 所需的大量数据带来了重大风险，因为模型可能会获得敏感主题（例如生物安全）的知识，以及复制受版权保护的作品的能力。旨在删除此类知识的方法必须从所有提示方向、以多语言能力进行，并且不会降低一般模型性能。为此，我们引入了从 LLM 中删除知识的目标角度反转 (TARS) 方法。TARS 方法首先利用 LLM 与详细提示相结合，在 LLM 的内部表示空间中聚合有关选定概念的信息。然后，它通过用噪声扰动近似概念向量并将其转换为语言模型头的标记分数，细化此近似概念向量以高概率触发概念标记。LLM 中直接在内部表示空间上运行的前馈权重向量与此目标向量具有最高的余弦相似度，然后被反向目标向量替换，从而限制概念在模型中传播的能力。 TARS 方法的模块化允许从 Llama 3.1 8B 中顺序删除概念，例如著名的文学侦探 Sherlock Holmes 和土星。事实证明，只需 1 次 TARS 编辑，触发目标概念的概率就可以降低到 0.00，同时双向删除知识。此外，尽管只针对英语，但事实证明所有语言的知识都被删除了。重要的是，TARS 对一般模型功能的影响很小，因为在以模块化方式删除 5 个不同的概念后，LLM 在大量维基百科文本语料库中的下一个标记概率的 KL 散度很小（中位数为 0.002）。

Title: Reasoner Outperforms: Generative Stance Detection with Rationalization for Social Media

• Authors: Jiaqing Yuan, Ruijie Xi, Munindar P. Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10266
• Pdf URL: https://arxiv.org/pdf/2412.10266
• Copy Paste: [[2412.10266]] Reasoner Outperforms: Generative Stance Detection with Rationalization for Social Media(https://arxiv.org/abs/2412.10266)
• Keywords: language model, gpt, llm
• Abstract: Stance detection is crucial for fostering a human-centric Web by analyzing user-generated content to identify biases and harmful narratives that undermine trust. With the development of Large Language Models (LLMs), existing approaches treat stance detection as a classification problem, providing robust methodologies for modeling complex group interactions and advancing capabilities in natural language tasks. However, these methods often lack interpretability, limiting their ability to offer transparent and understandable justifications for predictions. This study adopts a generative approach, where stance predictions include explicit, interpretable rationales, and integrates them into smaller language models through single-task and multitask learning. We find that incorporating reasoning into stance detection enables the smaller model (FlanT5) to outperform GPT-3.5's zero-shot performance, achieving an improvement of up to 9.57%. Moreover, our results show that reasoning capabilities enhance multitask learning performance but may reduce effectiveness in single-task settings. Crucially, we demonstrate that faithful rationales improve rationale distillation into SLMs, advancing efforts to build interpretable, trustworthy systems for addressing discrimination, fostering trust, and promoting equitable engagement on social media.
• 摘要：立场检测对于促进以人为本的网络至关重要，它通过分析用户生成的内容来识别破坏信任的偏见和有害叙述。随着大型语言模型 (LLM) 的发展，现有方法将立场检测视为分类问题，为建模复杂的群体互动和提高自然语言任务的能力提供了强大的方法。然而，这些方法往往缺乏可解释性，限制了它们为预测提供透明和可理解的理由的能力。本研究采用生成方法，其中立场预测包括明确的、可解释的理由，并通过单任务和多任务学习将它们集成到较小的语言模型中。我们发现，将推理纳入立场检测可以使较小的模型 (FlanT5) 超越 GPT-3.5 的零样本性能，实现高达 9.57% 的提升。此外，我们的结果表明，推理能力可以提高多任务学习性能，但可能会降低单任务设置中的有效性。至关重要的是，我们证明忠实的理由可以改善 SLM 的理由提炼，从而推动建立可解释、可信赖的系统来解决歧视问题、培养信任并促进社交媒体上的平等参与。

Title: Benchmarking Linguistic Diversity of Large Language Models

• Authors: Yanzhu Guo, Guokan Shang, Chloé Clavel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10271
• Pdf URL: https://arxiv.org/pdf/2412.10271
• Copy Paste: [[2412.10271]] Benchmarking Linguistic Diversity of Large Language Models(https://arxiv.org/abs/2412.10271)
• Keywords: language model, llm
• Abstract: The development and evaluation of Large Language Models (LLMs) has primarily focused on their task-solving capabilities, with recent models even surpassing human performance in some areas. However, this focus often neglects whether machine-generated language matches the human level of diversity, in terms of vocabulary choice, syntactic construction, and expression of meaning, raising questions about whether the fundamentals of language generation have been fully addressed. This paper emphasizes the importance of examining the preservation of human linguistic richness by language models, given the concerning surge in online content produced or aided by LLMs. We propose a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives including lexical, syntactic, and semantic dimensions. Using this framework, we benchmark several state-of-the-art LLMs across all diversity dimensions, and conduct an in-depth case study for syntactic diversity. Finally, we analyze how different development and deployment choices impact the linguistic diversity of LLM outputs.
• 摘要：大型语言模型 (LLM) 的开发和评估主要集中在其任务解决能力上，最近的模型甚至在某些领域超越了人类的表现。然而，这种关注往往忽略了机器生成的语言在词汇选择、句法构造和意义表达方面是否与人类的多样性水平相匹配，这引发了人们对语言生成的基本原理是否已得到充分解决的质疑。鉴于 LLM 生成或辅助的在线内容激增，本文强调了研究语言模型对人类语言丰富性的保存的重要性。我们提出了一个全面的框架，从词汇、句法和语义维度等各种语言多样性角度评估 LLM。利用这个框架，我们对所有多样性维度的几种最先进的 LLM 进行了基准测试，并对句法多样性进行了深入的案例研究。最后，我们分析了不同的开发和部署选择如何影响 LLM 输出的语言多样性。

Title: One world, one opinion? The superstar effect in LLM responses

• Authors: Sofie Goethals, Lauren Rhue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10281
• Pdf URL: https://arxiv.org/pdf/2412.10281
• Copy Paste: [[2412.10281]] One world, one opinion? The superstar effect in LLM responses(https://arxiv.org/abs/2412.10281)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) are shaping the way information is shared and accessed online, their opinions have the potential to influence a wide audience. This study examines who the LLMs view as the most prominent figures across various fields, using prompts in ten different languages to explore the influence of linguistic diversity. Our findings reveal low diversity in responses, with a small number of figures dominating recognition across languages (also known as the "superstar effect"). These results highlight the risk of narrowing global knowledge representation when LLMs retrieve subjective information.
• 摘要：由于大型语言模型 (LLM) 正在塑造信息在线共享和访问的方式，因此他们的观点可能会影响广泛的受众。本研究使用十种不同语言的提示来探索语言多样性的影响，以研究 LLM 认为哪些人是各个领域最杰出的人物。我们的研究结果显示，答案的多样性较低，少数人物在各种语言中占据主导地位（也称为“超级明星效应”）。这些结果强调了 LLM 检索主观信息时缩小全球知识表示的风险。

Title: Still "Talking About Large Language Models": Some Clarifications

• Authors: Murray Shanahan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10291
• Pdf URL: https://arxiv.org/pdf/2412.10291
• Copy Paste: [[2412.10291]] Still "Talking About Large Language Models": Some Clarifications(https://arxiv.org/abs/2412.10291)
• Keywords: language model
• Abstract: My paper "Talking About Large Language Models" has more than once been interpreted as advocating a reductionist stance towards large language models. But the paper was not intended that way, and I do not endorse such positions. This short note situates the paper in the context of a larger philosophical project that is concerned with the (mis)use of words rather than metaphysics, in the spirit of Wittgenstein's later writing.
• 摘要：我的论文“谈论大型语言模型”不止一次被解读为提倡对大型语言模型采取还原论立场。但这篇论文的本意并非如此，我也不赞同这种立场。这篇短文将这篇论文置于一个更大的哲学项目的背景下，该项目关注的是词语的（误用）而不是形而上学，这体现了维特根斯坦后期著作的精神。

Title: SCBench: A KV Cache-Centric Analysis of Long-Context Methods

• Authors: Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10319
• Pdf URL: https://arxiv.org/pdf/2412.10319
• Copy Paste: [[2412.10319]] SCBench: A KV Cache-Centric Analysis of Long-Context Methods(https://arxiv.org/abs/2412.10319)
• Keywords: llm, prompt
• Abstract: Long-context LLMs have enabled numerous downstream applications but also introduced significant challenges related to computational and memory efficiency. To address these challenges, optimizations for long-context inference have been developed, centered around the KV cache. However, existing benchmarks often evaluate in single-request, neglecting the full lifecycle of the KV cache in real-world use. This oversight is particularly critical, as KV cache reuse has become widely adopted in LLMs inference frameworks, such as vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft, Google, and Anthropic. To address this gap, we introduce SCBench(SharedContextBench), a comprehensive benchmark for evaluating long-context methods from a KV cachecentric perspective: 1) KV cache generation, 2) KV cache compression, 3) KV cache retrieval, 4) KV cache loading. Specifically, SCBench uses test examples with shared context, ranging 12 tasks with two shared context modes, covering four categories of long-context capabilities: string retrieval, semantic retrieval, global information, and multi-task. With it, we provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs, Mamba-Attention hybrids, and efficient methods such as sparse attention, KV cache dropping, quantization, retrieval, loading, and prompt compression. The evaluation is conducted on 8 long-context LLMs. Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n^2) pre-filling computation perform robustly. Dynamic sparsity yields more expressive KV caches than static patterns, and layer-level sparsity in hybrid architectures reduces memory usage with strong performance. Additionally, we identify attention distribution shift issues in long-generation scenarios. this https URL.
• 摘要：长上下文 LLM 已实现众多下游应用，但也带来了与计算和内存效率相关的重大挑战。为了应对这些挑战，围绕 KV 缓存开发了针对长上下文推理的优化。然而，现有的基准测试通常在单个请求中进行评估，忽略了实际使用中 KV 缓存的整个生命周期。这种疏忽尤其重要，因为 KV 缓存重用已在 LLM 推理框架（例如 vLLM 和 SGLang）以及 LLM 提供商（包括 OpenAI、Microsoft、Google 和 Anthropic）中得到广泛采用。为了弥补这一差距，我们引入了 SCBench（SharedContextBench），这是一个从 KV 缓存中心角度评估长上下文方法的综合基准测试：1) KV 缓存生成，2) KV 缓存压缩，3) KV 缓存检索，4) KV 缓存加载。具体来说，SCBench 使用具有共享上下文的测试示例，涵盖 12 个任务和两种共享上下文模式，涵盖四类长上下文功能：字符串检索、语义检索、全局信息和多任务。借助它，我们对八类长上下文解决方案进行了以 KV 缓存为中心的广泛分析，包括门控线性 RNN、Mamba-Attention 混合，以及稀疏注意、KV 缓存丢弃、量化、检索、加载和即时压缩等高效方法。评估在 8 个长上下文 LLM 上进行。我们的研究结果表明，亚 O(n) 内存方法在多轮场景中会受到影响，而具有 O(n) 内存的稀疏编码和亚 O(n^2) 预填充计算则表现稳健。动态稀疏性比静态模式产生更具表现力的 KV 缓存，混合架构中的层级稀疏性在性能强劲的情况下减少了内存使用量。此外，我们还确定了长代场景中的注意力分布偏移问题。这个 https URL。

Title: A Grounded Typology of Word Classes

• Authors: Coleman Haley, Sharon Goldwater, Edoardo Ponti
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10369
• Pdf URL: https://arxiv.org/pdf/2412.10369
• Copy Paste: [[2412.10369]] A Grounded Typology of Word Classes(https://arxiv.org/abs/2412.10369)
• Keywords: language model
• Abstract: We propose a grounded approach to meaning in language typology. We treat data from perceptual modalities, such as images, as a language-agnostic representation of meaning. Hence, we can quantify the function--form relationship between images and captions across languages. Inspired by information theory, we define "groundedness", an empirical measure of contextual semantic contentfulness (formulated as a difference in surprisal) which can be computed with multilingual multimodal language models. As a proof of concept, we apply this measure to the typology of word classes. Our measure captures the contentfulness asymmetry between functional (grammatical) and lexical (content) classes across languages, but contradicts the view that functional classes do not convey content. Moreover, we find universal trends in the hierarchy of groundedness (e.g., nouns > adjectives > verbs), and show that our measure partly correlates with psycholinguistic concreteness norms in English. We release a dataset of groundedness scores for 30 languages. Our results suggest that the grounded typology approach can provide quantitative evidence about semantic function in language.
• 摘要：我们提出了一种语言类型学中意义的扎根方法。我们将来自感知模态（例如图像）的数据视为与语言无关的意义表示。因此，我们可以量化不同语言中图像和标题之间的功能-形式关系。受信息论的启发，我们定义了“扎根性”，这是上下文语义内容性（表述为惊奇差异）的经验度量，可以使用多语言多模态语言模型来计算。作为概念证明，我们将此度量应用于词类类型学。我们的度量捕捉了不同语言中功能（语法）和词汇（内容）类别之间的内容性不对称，但与功能类别不传达内容的观点相矛盾。此外，我们发现了扎根性层次结构中的普遍趋势（例如，名词 > 形容词 > 动词），并表明我们的度量与英语中的心理语言学具体性规范部分相关。我们发布了 30 种语言的扎根性分数数据集。我们的结果表明，扎根类型学方法可以提供有关语言语义功能的定量证据。