2024-12-25

Title: Inductive Linguistic Reasoning with Large Language Models

• Authors: Raghav Ramji, Keshav Ramji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17819
• Pdf URL: https://arxiv.org/pdf/2412.17819
• Copy Paste: [[2412.17819]] Inductive Linguistic Reasoning with Large Language Models(https://arxiv.org/abs/2412.17819)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1% and Llama-3.1-405B-Instruct by 5.9% over chain-of-thought approaches. These gains are attributable to the analogical demonstrations, both when self-generated as well as when produced by weaker multilingual models. Furthermore, we demonstrate that our method generalizes to other tasks present in Linguistics Olympiad competitions, achieving sizable improvements across all problem types and difficulty levels included in the LINGOLY dataset with GPT-4o. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods.
• 摘要：评估大型语言模型 (LLM) 的语言推理能力是一项重要任务，有助于了解其在大规模采用过程中可能出现的技能差距。在这项工作中，我们通过对资源极其匮乏的语言进行语言谜题的视角，研究了此类模型执行抽象多语言推理的能力。由于这些翻译任务涉及从参考实例进行归纳和演绎推理，我们研究是否可以通过类比提示从种子样本中自动诱导出各种辅助演示。我们采用两阶段程序，首先使用语言模型生成类比样本，然后将它们与提供的目标语言样本一起应用于上下文中。我们在 modelLing 数据集上的结果表明，类比提示可以有效地引出模型对语言语法相似性的了解，与思路链方法相比，GPT-4o 的性能提高了 8.1%，Llama-3.1-405B-Instruct 的性能提高了 5.9%。这些收益归功于类比演示，无论是自生成还是由较弱的多语言模型生成。此外，我们证明了我们的方法可以推广到语言学奥林匹克竞赛中的其他任务，在使用 GPT-4o 的 LINGOLY 数据集中包含的所有问题类型和难度级别上都取得了显着的改进。我们还报告了一些关于推动语言推理性能的有趣现象的发现，表明此类谜题是新推理方法的宝贵基准。

Title: The Rosetta Paradox: Domain-Specific Performance Inversions in Large Language Models

• Authors: Basab Jha, Ujjwal Puri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17821
• Pdf URL: https://arxiv.org/pdf/2412.17821
• Copy Paste: [[2412.17821]] The Rosetta Paradox: Domain-Specific Performance Inversions in Large Language Models(https://arxiv.org/abs/2412.17821)
• Keywords: language model, gpt, llm
• Abstract: While large language models, such as GPT and BERT, have already demonstrated unprecedented skills in everything from natural language processing to domain-specific applications, there came an unexplored phenomenon we term the Rosetta Paradox. The Rosetta Paradox characterizes the counterintuitive performance inversions across domains of knowledge. This paradox captures how such LLMs can excel in highly specialized fields but do poorly on tasks which require general, everyday knowledge. This paper formalizes the definition of the Rosetta Paradox and introduces a panoramic analysis framework that includes both a Domain Specificity Index (DSI) and a Performance Inversion Metric (PIM) for consistent quantification of domain-specific behavior in LLMs. We adopt this paradox and conduct a series of investigations through extensive experiments across diverse models and knowledge domains, ranging from rich technical areas to common-sense reasoning. Our findings indicate that the Rosetta Paradox is likely not a mere artifact of data distribution but an intrinsic architectural and emergent property of deep neural networks. We present comparative analyses across different model architectures, sizes, and training methodologies that shed light into the peculiar ways this paradox manifests itself and challenge the standard evaluation metrics.
• 摘要：虽然 GPT 和 BERT 等大型语言模型已经在从自然语言处理到特定领域应用的各个方面展示了前所未有的技能，但出现了一种未被探索的现象，我们称之为罗塞塔悖论。罗塞塔悖论描述了跨知识领域的反直觉性能反转。这一悖论捕捉到了此类 LLM 如何在高度专业化的领域表现出色，但在需要一般日常知识的任务上却表现不佳。本文形式化了罗塞塔悖论的定义，并引入了一个全景分析框架，该框架包括领域特异性指数 (DSI) 和性能反转指标 (PIM)，用于一致量化 LLM 中的领域特定行为。我们采用这一悖论，通过广泛的实验开展了一系列调查，涉及从丰富的技术领域到常识推理的各种模型和知识领域。我们的研究结果表明，罗塞塔悖论可能不仅仅是数据分布的产物，而是深度神经网络的内在架构和新兴属性。我们对不同的模型架构、大小和训练方法进行了比较分析，揭示了这种悖论的特殊表现方式并对标准评估指标提出了挑战。

Title: Leveraging Sentiment for Offensive Text Classification

• Authors: Khondoker Ittehadul Islam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17825
• Pdf URL: https://arxiv.org/pdf/2412.17825
• Copy Paste: [[2412.17825]] Leveraging Sentiment for Offensive Text Classification(https://arxiv.org/abs/2412.17825)
• Keywords: language model
• Abstract: In this paper, we conduct experiment to analyze whether models can classify offensive texts better with the help of sentiment. We conduct this experiment on the SemEval 2019 task 6, OLID, dataset. First, we utilize pre-trained language models to predict the sentiment of each instance. Later we pick the model that achieved the best performance on the OLID test set, and train it on the augmented OLID set to analyze the performance. Results show that utilizing sentiment increases the overall performance of the model.
• 摘要：在本文中，我们进行了实验，以分析模型是否可以在情感的帮助下更好地对攻击性文本进行分类。我们在 SemEval 2019 任务 6 OLID 数据集上进行了这项实验。首先，我们利用预先训练的语言模型来预测每个实例的情感。然后，我们选择在 OLID 测试集上表现最佳的模型，并在增强的 OLID 集上对其进行训练以分析性能。结果表明，利用情感可以提高模型的整体性能。

Title: Look Ahead Text Understanding and LLM Stitching

• Authors: Junlin Julian Jiang (Piedmont High School, Piedmont, CA, USA), Xin Li (College of Business, City University of Hong Kong, Hong Kong, China)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17836
• Pdf URL: https://arxiv.org/pdf/2412.17836
• Copy Paste: [[2412.17836]] Look Ahead Text Understanding and LLM Stitching(https://arxiv.org/abs/2412.17836)
• Keywords: gpt, llm
• Abstract: This paper proposes a look ahead text understanding problem with look ahead section identification (LASI) as an example. This problem may appear in generative AI as well as human interactions, where we want to understand the direction of a developing text or conversation. We tackle the problem using transformer-based LLMs. We show that LASI is more challenging than classic section identification (SI). We argue that both bidirectional contextual information (e.g., BERT) and unidirectional predictive ability (e.g., GPT) will benefit the task. We propose two approaches to stitch together BERT and GPT. Experiments show that our approach outperforms the established models, especially when there is noise in the text (which is often the case for developing text in generative AI). Our paper sheds light on other look ahead text understanding tasks that are important to social media, such as look ahead sentiment classification, and points out the opportunities to leverage pre-trained LLMs through stitching.
• 摘要：本文提出了一个前瞻性文本理解问题，并以前瞻性章节识别 (LASI) 为例。这个问题可能出现在生成式人工智能以及人类交互中，我们希望了解文本或对话的发展方向。我们使用基于 Transformer 的 LLM 来解决这个问题。我们表明 LASI 比经典的章节识别 (SI) 更具挑战性。我们认为双向上下文信息（例如 BERT）和单向预测能力（例如 GPT）都将有利于这项任务。我们提出了两种将 BERT 和 GPT 拼接在一起的方法。实验表明，我们的方法优于已建立的模型，尤其是当文本中有噪音时（生成式人工智能中开发文本通常就是这种情况）。我们的论文阐明了对社交媒体很重要的其他前瞻性文本理解任务，例如前瞻性情绪分类，并指出了通过拼接利用预训练 LLM 的机会。

Title: Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding

• Authors: Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Grigori Sidorov, Dietrich Klakow, Philipp Slusallek, Olga Kolesnikova, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17837
• Pdf URL: https://arxiv.org/pdf/2412.17837
• Copy Paste: [[2412.17837]] Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding(https://arxiv.org/abs/2412.17837)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) show promising learning and reasoning abilities. Compared to other NLP tasks, multilingual and multi-label emotion evaluation tasks are under-explored in LLMs. In this paper, we present EthioEmo, a multi-label emotion classification dataset for four Ethiopian languages, namely, Amharic (amh), Afan Oromo (orm), Somali (som), and Tigrinya (tir). We perform extensive experiments with an additional English multi-label emotion dataset from SemEval 2018 Task 1. Our evaluation includes encoder-only, encoder-decoder, and decoder-only language models. We compare zero and few-shot approaches of LLMs to fine-tuning smaller language models. The results show that accurate multi-label emotion classification is still insufficient even for high-resource languages such as English, and there is a large gap between the performance of high-resource and low-resource languages. The results also show varying performance levels depending on the language and model type. EthioEmo is available publicly to further improve the understanding of emotions in language models and how people convey emotions through various languages.
• 摘要：大型语言模型 (LLM) 表现出良好的学习和推理能力。与其他 NLP 任务相比，多语言和多标签情感评估任务在 LLM 中尚未得到充分探索。在本文中，我们介绍了 EthioEmo，这是一个针对四种埃塞俄比亚语言（即阿姆哈拉语 (amh)、阿凡奥罗莫语 (orm)、索马里语 (som) 和提格里尼亚语 (tir)）的多标签情感分类数据集。我们使用来自 SemEval 2018 任务 1 的另一个英语多标签情感数据集进行了大量实验。我们的评估包括仅编码器、编码器解码器和仅解码器语言模型。我们将 LLM 的零样本和少样本方法与微调较小的语言模型进行了比较。结果表明，即使对于英语等资源丰富的语言，准确的多标签情感分类仍然不足，并且资源丰富的语言和资源较少的语言之间的性能差距很大。结果还显示，根据语言和模型类型，性能水平各不相同。 EthioEmo 向公众开放，以进一步提高对语言模型中的情感的理解以及人们如何通过各种语言传达情感。

Title: Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting

• Authors: Vijay Goyal, Mustafa Khan, Aprameya Tirupati, Harveer Saini, Michael Lam, Kevin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17846
• Pdf URL: https://arxiv.org/pdf/2412.17846
• Copy Paste: [[2412.17846]] Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting(https://arxiv.org/abs/2412.17846)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing (NLP) tasks. However, these models are often difficult to deploy due to significant computational requirements and resource constraints. Knowledge distillation (KD) is an effective technique for transferring the performance of larger LLMs to smaller models. Traditional KD methods primarily focus on the direct output of the teacher model, with little emphasis on the role of prompting during knowledge transfer. In this paper, we propose a set of novel response-priming prompting strategies applied in the knowledge distillation pipeline to enhance the performance of student models. Our approach fine-tunes a smaller Llama 3.1 8B Instruct model by distilling knowledge from a quantized Llama 3.1 405B Instruct teacher model. We apply LoRA optimization and evaluate on the GSM8K benchmark. Experimental results demonstrate that integrating reasoning-eliciting prompting into the proposed KD pipeline significantly improves student model performance, offering an efficient way to deploy powerful models in resource-constrained environments. We find that Ground Truth prompting results in a 55\% performance increase on GSM8K for a distilled Llama 3.1 8B Instruct compared to the same model distilled without prompting. A thorough investigation into the self-attention layers of the student models indicates that the more successful prompted models tend to exhibit certain positive behaviors inside their attention heads which can be tied to their increased accuracy. Our implementation can be found at this https URL.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理 (NLP) 任务中都表现出色。然而，由于计算要求高和资源限制，这些模型通常难以部署。知识蒸馏 (KD) 是一种将大型 LLM 的性能转移到较小模型的有效技术。传统的 KD 方法主要关注教师模型的直接输出，很少强调提示在知识转移过程中的作用。在本文中，我们提出了一套新颖的反应启动提示策略，应用于知识蒸馏管道，以提高学生模型的性能。我们的方法通过从量化的 Llama 3.1 405B Instruct 教师模型中提炼知识来微调较小的 Llama 3.1 8B Instruct 模型。我们应用 LoRA 优化并在 GSM8K 基准上进行评估。实验结果表明，将推理提示集成到所提出的 KD 管道中可显著提高学生模型的性能，为在资源受限的环境中部署强大的模型提供了一种有效的方法。我们发现，与未经提示而提炼的相同模型相比，Ground Truth 提示可使提炼后的 Llama 3.1 8B Instruct 在 GSM8K 上的性能提高 55%。对学生模型的自我注意层的彻底调查表明，更成功的提示模型往往会在其注意力头中表现出某些积极行为，这可能与其准确性的提高有关。我们的实现可在此 https URL 中找到。

Title: Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

• Authors: Ziming Guo, Chao Ma, Yinggang Sun, Tiancheng Zhao, Guangyao Wang, Hai Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17867
• Pdf URL: https://arxiv.org/pdf/2412.17867
• Copy Paste: [[2412.17867]] Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types(https://arxiv.org/abs/2412.17867)
• Keywords: language model, llm, agent
• Abstract: Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q\&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries.
• 摘要：大型语言模型 (LLM) 的最新进展显著推动了文本到 SQL 系统的发展。然而，大多数基于 LLM 的方法通常只关注 SQL 生成，而忽略了现实世界对话查询的复杂性。这种疏忽可能导致不可靠的响应，特别是对于无法直接用 SQL 解决的模糊问题。为了弥补这一差距，我们提出了 MMSQL，这是一个全面的测试套件，旨在通过模拟具有多种问题类型和多轮问答交互的真实场景来评估 LLM 的问题分类和 SQL 生成能力。使用 MMSQL，我们评估了流行的 LLM（包括开源和闭源模型）的性能，并确定了影响它们在这些场景中性能的关键因素。此外，我们引入了一个基于 LLM 的多智能体框架，该框架使用专门的智能体来识别问题类型并确定适当的回答策略。我们的实验表明，这种方法显著增强了模型应对对话动态复杂性的能力，有效地处理了用户查询的多样性和复杂性。

Title: Joint Knowledge Editing for Information Enrichment and Probability Promotion

• Authors: Wenhang Shi, Yiren Chen, Shuqing Bian, Xinyi Zhang, Zhe Zhao, Pengfei Hu, Wei Lu, Xiaoyong Du
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.17872
• Pdf URL: https://arxiv.org/pdf/2412.17872
• Copy Paste: [[2412.17872]] Joint Knowledge Editing for Information Enrichment and Probability Promotion(https://arxiv.org/abs/2412.17872)
• Keywords: language model, gpt
• Abstract: Knowledge stored in large language models requires timely updates to reflect the dynamic nature of real-world information. To update the knowledge, most knowledge editing methods focus on the low layers, since recent probes into the knowledge recall process reveal that the answer information is enriched in low layers. However, these probes only and could only reveal critical recall stages for the original answers, while the goal of editing is to rectify model's prediction for the target answers. This inconsistency indicates that both the probe approaches and the associated editing methods are deficient. To mitigate the inconsistency and identify critical editing regions, we propose a contrast-based probe approach, and locate two crucial stages where the model behavior diverges between the original and target answers: Information Enrichment in low layers and Probability Promotion in high layers. Building upon the insights, we develop the Joint knowledge Editing for information Enrichment and probability Promotion (JEEP) method, which jointly edits both the low and high layers to modify the two critical recall stages. Considering the mutual interference and growing forgetting due to dual modifications, JEEP is designed to ensure that updates to distinct regions share the same objectives and are complementary. We rigorously evaluate JEEP by editing up to thousands of facts on various models, i.e., GPT-J (6B) and LLaMA (7B), and addressing diverse editing objectives, i.e., adding factual and counterfactual knowledge. In all tested scenarios, JEEP achieves best performances, validating the effectiveness of the revealings of our probe approach and the designs of our editing method. Our code and data are available at this https URL.
• 摘要：大型语言模型中存储的知识需要及时更新，以反映现实世界信息的动态特性。为了更新知识，大多数知识编辑方法都侧重于低层，因为最近对知识回忆过程的探索表明，答案信息在低层得到了丰富。然而，这些探测只能揭示原始答案的关键回忆阶段，而编辑的目标是纠正模型对目标答案的预测。这种不一致表明探测方法和相关的编辑方法都存在缺陷。为了缓解不一致并识别关键的编辑区域，我们提出了一种基于对比的探测方法，并找到了模型行为在原始答案和目标答案之间出现分歧的两个关键阶段：低层的信息丰富和高层的概率提升。基于这些见解，我们开发了联合知识编辑信息丰富和概率提升 (JEEP) 方法，该方法联合编辑低层和高层以修改两个关键的回忆阶段。考虑到双重修改导致的相互干扰和遗忘增加，JEEP 旨在确保对不同区域的更新具有相同的目标并且是互补的。我们通过在各种模型（即 GPT-J (6B) 和 LLaMA (7B)）上编辑多达数千个事实并解决各种编辑目标（即添加事实和反事实知识）来严格评估 JEEP。在所有测试场景中，JEEP 都实现了最佳性能，验证了我们的探测方法的揭示和编辑方法的设计的有效性。我们的代码和数据可在此 https URL 上找到。

Title: Evaluating LLM Reasoning in the Operations Research Domain with ORQA

• Authors: Mahdi Mostajabdaveh, Timothy T. Yu, Samarendra Chandan Bindu Dash, Rindranirina Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17874
• Pdf URL: https://arxiv.org/pdf/2412.17874
• Copy Paste: [[2412.17874]] Evaluating LLM Reasoning in the Operations Research Domain with ORQA(https://arxiv.org/abs/2412.17874)
• Keywords: language model, llm
• Abstract: In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available.
• 摘要：在本文中，我们介绍并应用了运筹学问答系统 (ORQA)，这是一种新的基准，旨在评估大型语言模型 (LLM) 在运筹学 (OR) 专业技术领域的泛化能力。该基准评估 LLM 在面对多样化和复杂的优化问题时是否可以模仿 OR 专家的知识和推理技能。该数据集由 OR 专家开发，以现实世界的优化问题为特色，这些问题需要多步推理来构建其数学模型。我们对各种开源 LLM（如 LLaMA 3.1、DeepSeek 和 Mixtral）的评估表明，它们的性能一般，凸显了它们在泛化到专业技术领域的能力方面存在差距。这项工作有助于持续讨论 LLM 的泛化能力，为该领域的未来研究提供宝贵的见解。数据集和评估代码是公开的。

Title: The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting

• Authors: Shuzhang Cai, Twumasi Mensah-Boateng, Xander Kuksov, Jing Yuan, Shaojie Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17891
• Pdf URL: https://arxiv.org/pdf/2412.17891
• Copy Paste: [[2412.17891]] The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting(https://arxiv.org/abs/2412.17891)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated exceptional abilities across a broad range of language-related tasks, including generating solutions to complex reasoning problems. An effective technique to enhance LLM performance is in-context learning, which encourages a step-by-step reasoning process by including explanatory examples to guide the model's responses. However, selecting appropriate exemplars for the model poses a challenge, as each dataset demands a distinct set of exemplars to enable the LLM to learn effectively and perform well on the test set. Current studies often rely on uncertainty- or diversity-based selection strategies to select exemplars for annotation and to improve model learning. However, these studies typically employ a non-adaptive approach, selecting a set of exemplars all at once. We argue that this non-adaptive strategy may result in a set of exemplars with high redundancy in terms of the knowledge covered, ultimately reducing their overall informativeness. To address this limitation, we propose \textsc{Adaptive-Prompt}, a novel method that adaptively selects exemplars by leveraging model feedback from previously chosen exemplars. Experimental results show that \textsc{Adaptive-Prompt} significantly enhances LLM performance across a variety of reasoning tasks.
• 摘要：大型语言模型 (LLM) 已在广泛的语言相关任务中展现出卓越的能力，包括为复杂的推理问题生成解决方案。增强 LLM 性能的一种有效技术是上下文学习，它通过包含解释性示例来指导模型的响应，从而鼓励逐步推理过程。然而，为模型选择合适的样本是一项挑战，因为每个数据集都需要一组不同的样本，以使 LLM 能够有效学习并在测试集上表现良好。当前的研究通常依靠不确定性或基于多样性的选择策略来选择样本进行注释并改进模型学习。然而，这些研究通常采用非自适应方法，一次选择一组样本。我们认为，这种非自适应策略可能会导致一组样本在所涵盖的知识方面具有高度冗余，最终降低它们的整体信息量。为了解决这一限制，我们提出了 \textsc{Adaptive-Prompt}，这是一种新方法，它通过利用先前选择的样本的模型反馈来自适应地选择样本。实验结果表明，\textsc{Adaptive-Prompt} 显著提高了 LLM 在各种推理任务中的性能。

Title: BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

• Authors: Martin Fajcik, Martin Docekal, Jan Dolezal, Karel Ondrej, Karel Beneš, Jan Kapsa, Pavel Smrz, Alexander Polok, Michal Hradis, Zuzana Neverilova, Ales Horak, Radoslav Sabol, Michal Stefanik, Adam Jirkovsky, David Adamczyk, Petr Hyner, Jan Hula, Hynek Kydlicek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17933
• Pdf URL: https://arxiv.org/pdf/2412.17933
• Copy Paste: [[2412.17933]] BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism(https://arxiv.org/abs/2412.17933)
• Keywords: language model
• Abstract: We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 11 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis, (ii) continuous pretraining of the first Czech-centric 7B language model, with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard, with existing 44 model submissions, where new model submissions can be made at this https URL.
• 摘要：我们推出了 BenCzechMark (BCM)，这是第一个为大型语言模型设计的综合捷克语基准，提供多样化的任务、多种任务格式和多种评估指标。它的评分系统以统计显着性理论为基础，并使用受社会偏好理论启发的任务聚合。我们的基准包含 50 项具有挑战性的任务，以及相应的测试数据集，主要以捷克语为母语，其中 11 个是新收集的。这些任务涵盖 8 个类别，涵盖不同的领域，包括历史捷克新闻、学生或语言学习者的文章和口语。此外，我们收集和清理 BUT-Large Czech Collection，这是最大的公开可用的干净捷克语语料库，并将其用于 (i) 污染分析，(ii) 对第一个以捷克语为中心的 7B 语言模型进行持续预训练，并进行捷克语特定的标记化。我们使用我们的模型作为与公开可用的多语言模型进行比较的基线。最后，我们发布并维护一个排行榜，其中有 44 个模型提交，可以通过此 https URL 提交新的模型。

Title: Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models

• Authors: Ge Zhang, Mohammad Ali Alomrani, Hongjian Gu, Jiaming Zhou, Yaochen Hu, Bin Wang, Qun Liu, Mark Coates, Yingxue Zhang, Jianye Hao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17963
• Pdf URL: https://arxiv.org/pdf/2412.17963
• Copy Paste: [[2412.17963]] Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models(https://arxiv.org/abs/2412.17963)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework designed to tackle relation reasoning by decomposing the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a task-agnostic graph that identifies crucial entities, relations, and attributes within the problem context. Subsequently, PoT identifies relevant reasoning chains within the graph corresponding to the posed question, facilitating inference of potential answers. Experimental evaluations on four benchmark datasets, demanding long reasoning chains, demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (maximum 21.3%) without necessitating fine-tuning or extensive LLM calls. Furthermore, as opposed to prior neuro-symbolic methods, PoT exhibits improved resilience against LLM errors by leveraging the compositional nature of graphs.
• 摘要：大型语言模型 (LLM) 拥有丰富的语义知识，但通常在处理复杂的推理任务时会遇到困难，尤其是在亲属关系或空间推理等关系推理问题中。在本文中，我们提出了“思维路径” (PoT)，这是一个新颖的框架，旨在通过将任务分解为三个关键阶段来解决关系推理：图提取、路径识别和推理。与以前的方法不同，PoT 可以有效地提取与任务无关的图，该图可以识别问题上下文中的关键实体、关系和属性。随后，PoT 识别图中与提出的问题相对应的相关推理链，从而有助于推断潜在的答案。在四个基准数据集上进行的实验评估需要较长的推理链，表明 PoT 远远超过了最先进的基线 (最高 21.3%)，而无需微调或大量 LLM 调用。此外，与之前的神经符号方法相比，PoT 通过利用图的组合性质表现出对 LLM 错误的更高弹性。

Title: CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models

• Authors: Ruibo Tu, Hedvig Kjellström, Gustav Eje Henter, Cheng Zhang
• Subjects: cs.CL, cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.17970
• Pdf URL: https://arxiv.org/pdf/2412.17970
• Copy Paste: [[2412.17970]] CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models(https://arxiv.org/abs/2412.17970)
• Keywords: language model, llm, prompt
• Abstract: Causal reasoning capabilities are essential for large language models (LLMs) in a wide range of applications, such as education and healthcare. But there is still a lack of benchmarks for a better understanding of such capabilities. Current LLM benchmarks are mainly based on conversational tasks, academic math tests, and coding tests. Such benchmarks evaluate LLMs in well-regularized settings, but they are limited in assessing the skills and abilities to solve real-world problems. In this work, we provide a benchmark, named by CARL-GT, which evaluates CAusal Reasoning capabilities of large Language models using Graphs and Tabular data. The benchmark has a diverse range of tasks for evaluating LLMs from causal graph reasoning, knowledge discovery, and decision-making aspects. In addition, effective zero-shot learning prompts are developed for the tasks. In our experiments, we leverage the benchmark for evaluating open-source LLMs and provide a detailed comparison of LLMs for causal reasoning abilities. We found that LLMs are still weak in casual reasoning, especially with tabular data to discover new insights. Furthermore, we investigate and discuss the relationships of different benchmark tasks by analyzing the performance of LLMs. The experimental results show that LLMs have different strength over different tasks and that their performance on tasks in different categories, i.e., causal graph reasoning, knowledge discovery, and decision-making, shows stronger correlation than tasks in the same category.
• 摘要：因果推理能力对于教育和医疗保健等广泛应用中的大型语言模型 (LLM) 至关重要。但仍然缺乏基准来更好地理解这种能力。当前的 LLM 基准主要基于对话任务、学术数学测试和编码测试。这些基准在规范良好的环境中评估 LLM，但它们在评估解决实际问题的技能和能力方面受到限制。在这项工作中，我们提供了一个由 CARL-GT 命名的基准，它使用图形和表格数据评估大型语言模型的因果推理能力。该基准具有多种任务，可从因果图推理、知识发现和决策方面评估 LLM。此外，还为这些任务开发了有效的零样本学习提示。在我们的实验中，我们利用基准来评估开源 LLM，并对 LLM 的因果推理能力进行了详细的比较。我们发现 LLM 在因果推理方面仍然很弱，尤其是使用表格数据来发现新见解。此外，我们通过分析LLM的性能，研究并讨论了不同基准任务之间的关系。实验结果表明，LLM在不同任务上表现出不同的优势，并且它们在不同类别的任务（即因果图推理、知识发现和决策）上的表现比同一类别的任务表现出更强的相关性。

Title: Correctness is not Faithfulness in RAG Attributions

• Authors: Jonas Wallat, Maria Heuss, Maarten de Rijke, Avishek Anand
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18004
• Pdf URL: https://arxiv.org/pdf/2412.18004
• Copy Paste: [[2412.18004]] Correctness is not Faithfulness in RAG Attributions(https://arxiv.org/abs/2412.18004)
• Keywords: language model, hallucination
• Abstract: Retrieving relevant context is a common approach to reduce hallucinations and enhance answer reliability. Explicitly citing source documents allows users to verify generated responses and increases trust. Prior work largely evaluates citation correctness - whether cited documents support the corresponding statements. But citation correctness alone is insufficient. To establish trust in attributed answers, we must examine both citation correctness and citation faithfulness. In this work, we first disentangle the notions of citation correctness and faithfulness, which have been applied inconsistently in previous studies. Faithfulness ensures that the model's reliance on cited documents is genuine, reflecting actual reference use rather than superficial alignment with prior beliefs, which we call post-rationalization. We design an experiment that reveals the prevalent issue of post-rationalization, which undermines reliable attribution and may result in misplaced trust. Our findings suggest that current attributed answers often lack citation faithfulness (up to 57 percent of the citations), highlighting the need to evaluate correctness and faithfulness for trustworthy attribution in language models.
• 摘要：检索相关上下文是减少幻觉和提高答案可靠性的常用方法。明确引用源文档允许用户验证生成的响应并增加信任。先前的工作主要评估引用的正确性 - 引用的文档是否支持相应的陈述。但仅靠引用正确性是不够的。要建立对归因答案的信任，我们必须检查引用的正确性和引用的忠实性。在这项工作中，我们首先理清引用正确性和忠实性的概念，这两个概念在以前的研究中应用不一致。忠实性确保模型对引用文档的依赖是真实的，反映了实际的参考使用，而不是与先前信念的表面一致，我们称之为后合理化。我们设计了一个实验，揭示了后合理化的普遍问题，它破坏了可靠的归因并可能导致错误的信任。我们的研究结果表明，当前的归因答案通常缺乏引用的忠实性（高达 57% 的引用），突出了在语言模型中评估正确性和忠实性以进行可信归因的必要性。

Title: StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs

• Authors: Hailin Chen, Fangkai Jiao, Mathieu Ravaut, Nawshad Farruque, Xuan Phi Nguyen, Chengwei Qin, Manan Dey, Bosheng Ding, Caiming Xiong, Shafiq Joty, Yingbo Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18011
• Pdf URL: https://arxiv.org/pdf/2412.18011
• Copy Paste: [[2412.18011]] StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs(https://arxiv.org/abs/2412.18011)
• Keywords: language model, llm
• Abstract: The rapid development of large language models (LLMs) necessitates robust, unbiased, and scalable methods for evaluating their capabilities. However, human annotations are expensive to scale, model-based evaluations are prone to biases in answer style, while target-answer-based benchmarks are vulnerable to data contamination and cheating. To address these limitations, we propose StructTest, a novel benchmark that evaluates LLMs on their ability to produce compositionally specified structured outputs as an unbiased, cheap-to-run and difficult-to-cheat measure. The evaluation is done deterministically by a rule-based evaluator, which can be easily extended to new tasks. By testing structured outputs across diverse task domains -- including Summarization, Code, HTML and Math -- we demonstrate that StructTest serves as a good proxy for general reasoning abilities, as producing structured outputs often requires internal logical reasoning. We believe that StructTest offers a critical, complementary approach to objective and robust model evaluation.
• 摘要：大型语言模型 (LLM) 的快速发展需要稳健、无偏且可扩展的方法来评估其能力。然而，人工注释的扩展成本很高，基于模型的评估容易在回答风格上出现偏差，而基于目标答案的基准容易受到数据污染和作弊的影响。为了解决这些限制，我们提出了 StructTest，这是一种新颖的基准，它评估 LLM 生成组合指定的结构化输出的能力，是一种无偏、运行成本低且难以作弊的衡量标准。评估由基于规则的评估器确定性地完成，可以轻松扩展到新任务。通过在不同的任务领域（包括摘要、代码、HTML 和数学）测试结构化输出，我们证明了 StructTest 可以作为一般推理能力的良好代理，因为生成结构化输出通常需要内部逻辑推理。我们相信 StructTest 为客观和稳健的模型评估提供了一种关键的互补方法。

Title: Factuality or Fiction? Benchmarking Modern LLMs on Ambiguous QA with Citations

• Authors: Maya Patel, Aditi Anand
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18051
• Pdf URL: https://arxiv.org/pdf/2412.18051
• Copy Paste: [[2412.18051]] Factuality or Fiction? Benchmarking Modern LLMs on Ambiguous QA with Citations(https://arxiv.org/abs/2412.18051)
• Keywords: language model, gpt, llm, prompt
• Abstract: Benchmarking modern large language models (LLMs) on complex and realistic tasks is critical to advancing their development. In this work, we evaluate the factual accuracy and citation performance of state-of-the-art LLMs on the task of Question Answering (QA) in ambiguous settings with source citations. Using three recently published datasets-DisentQA-DupliCite, DisentQA-ParaCite, and AmbigQA-Cite-featuring a range of real-world ambiguities, we analyze the performance of two leading LLMs, GPT-4o-mini and Claude-3.5. Our results show that larger, recent models consistently predict at least one correct answer in ambiguous contexts but fail to handle cases with multiple valid answers. Additionally, all models perform equally poorly in citation generation, with citation accuracy consistently at 0. However, introducing conflict-aware prompting leads to large improvements, enabling models to better address multiple valid answers and improve citation accuracy, while maintaining their ability to predict correct answers. These findings highlight the challenges and opportunities in developing LLMs that can handle ambiguity and provide reliable source citations. Our benchmarking study provides critical insights and sets a foundation for future improvements in trustworthy and interpretable QA systems.
• 摘要：在复杂而现实的任务上对现代大型语言模型 (LLM) 进行基准测试对于推动其发展至关重要。在这项工作中，我们评估了最先进的 LLM 在带有来源引文的模糊设置中问答 (QA) 任务的事实准确性和引用性能。使用三个最近发布的数据集 - DisentQA-DupliCite、DisentQA-ParaCite 和 AmbigQA-Cite - 具有一系列现实世界的歧义性，我们分析了两个领先的 LLM，GPT-4o-mini 和 Claude-3.5 的性能。我们的结果表明，更大的近期模型在模棱两可的上下文中始终可以预测至少一个正确答案，但无法处理具有多个有效答案的情况。此外，所有模型在引文生成方面的表现都同样糟糕，引文准确率始终为 0。然而，引入冲突感知提示会带来很大的改进，使模型能够更好地解决多个有效答案并提高引用准确性，同时保持其预测正确答案的能力。这些发现凸显了开发能够处理歧义并提供可靠来源引文的 LLM 所面临的挑战和机遇。我们的基准研究提供了关键见解，并为未来改进值得信赖且可解释的 QA 系统奠定了基础。

Title: Neuron Empirical Gradient: Connecting Neurons' Linear Controllability and Representational Capacity

• Authors: Xin Zhao, Zehui Jiang, Naoki Yoshinaga
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18053
• Pdf URL: https://arxiv.org/pdf/2412.18053
• Copy Paste: [[2412.18053]] Neuron Empirical Gradient: Connecting Neurons' Linear Controllability and Representational Capacity(https://arxiv.org/abs/2412.18053)
• Keywords: language model
• Abstract: Although neurons in the feed-forward layers of pre-trained language models (PLMs) can store factual knowledge, most prior analyses remain qualitative, leaving the quantitative relationship among knowledge representation, neuron activations, and model output poorly understood. In this study, by performing neuron-wise interventions using factual probing datasets, we first reveal the linear relationship between neuron activations and output token probabilities. We refer to the gradient of this linear relationship as ``neuron empirical gradients.'' and propose NeurGrad, an efficient method for their calculation to facilitate quantitative neuron analysis. We next investigate whether neuron empirical gradients in PLMs encode general task knowledge by probing skill neurons. To this end, we introduce MCEval8k, a multi-choice knowledge evaluation benchmark spanning six genres and 22 tasks. Our experiments confirm that neuron empirical gradients effectively capture knowledge, while skill neurons exhibit efficiency, generality, inclusivity, and interdependency. These findings link knowledge to PLM outputs via neuron empirical gradients, shedding light on how PLMs store knowledge. The code and dataset are released.
• 摘要：尽管预训练语言模型 (PLM) 的前馈层中的神经元可以存储事实知识，但大多数先前的分析仍然是定性的，知识表示、神经元激活和模型输出之间的定量关系尚不清楚。在本研究中，通过使用事实探测数据集执行神经元干预，我们首先揭示了神经元激活和输出标记概率之间的线性关系。我们将这种线性关系的梯度称为“神经元经验梯度”，并提出了 NeurGrad，这是一种有效的计算方法，可促进定量神经元分析。接下来，我们将研究 PLM 中的神经元经验梯度是否通过探测技能神经元来编码一般任务知识。为此，我们引入了 MCEval8k，这是一个涵盖六种类型和 22 个任务的多选知识评估基准。我们的实验证实，神经元经验梯度可以有效地捕获知识，而技能神经元则表现出效率、通用性、包容性和相互依赖性。这些发现通过神经元经验梯度将知识与 PLM 输出联系起来，揭示了 PLM 如何存储知识。代码和数据集已发布。

Title: Improving Factuality with Explicit Working Memory

• Authors: Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Sun, Luke Zettlemoyer, Gargi Gosh, Wen-tau Yih
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18069
• Pdf URL: https://arxiv.org/pdf/2412.18069
• Copy Paste: [[2412.18069]] Improving Factuality with Explicit Working Memory(https://arxiv.org/abs/2412.18069)
• Keywords: language model, hallucination, prompt
• Abstract: Large language models can generate factually inaccurate content, a problem known as hallucination. Recent works have built upon retrieved-augmented generation to improve factuality through iterative prompting but these methods are limited by the traditional RAG design. To address these challenges, we introduce EWE (Explicit Working Memory), a novel approach that enhances factuality in long-form text generation by integrating a working memory that receives real-time feedback from external resources. The memory is refreshed based on online fact-checking and retrieval feedback, allowing EWE to rectify false claims during the generation process and ensure more accurate and reliable outputs. Our experiments demonstrate that Ewe outperforms strong baselines on four fact-seeking long-form generation datasets, increasing the factuality metric, VeriScore, by 2 to 10 points absolute without sacrificing the helpfulness of the responses. Further analysis reveals that the design of rules for memory updates, configurations of memory units, and the quality of the retrieval datastore are crucial factors for influencing model performance.
• 摘要：大型语言模型可能会生成事实不准确的内容，这一问题被称为幻觉。最近的研究基于检索增强生成，通过迭代提示来提高事实性，但这些方法受到传统 RAG 设计的限制。为了应对这些挑战，我们引入了 EWE（显性工作记忆），这是一种新颖的方法，通过集成接收来自外部资源的实时反馈的工作记忆来增强长文本生成中的事实性。记忆会根据在线事实核查和检索反馈进行刷新，从而使 EWE 能够在生成过程中纠正虚假声明并确保更准确、更可靠的输出。我们的实验表明，Ewe 在四个寻求事实的长文本生成数据集上的表现优于强基线，将事实性指标 VeriScore 提高了 2 到 10 分，而不会牺牲响应的有用性。进一步的分析表明，内存更新规则的设计、内存单元的配置和检索数据存储的质量是影响模型性能的关键因素。

Title: Molly: Making Large Language Model Agents Solve Python Problem More Logically

• Authors: Rui Xiao, Jiong Wang, Lu Han, Na Zong, Han Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18093
• Pdf URL: https://arxiv.org/pdf/2412.18093
• Copy Paste: [[2412.18093]] Molly: Making Large Language Model Agents Solve Python Problem More Logically(https://arxiv.org/abs/2412.18093)
• Keywords: language model, llm, retrieval augmented generation, agent
• Abstract: Applying large language models (LLMs) as teaching assists has attracted much attention as an integral part of intelligent education, particularly in computing courses. To reduce the gap between the LLMs and the computer programming education expert, fine-tuning and retrieval augmented generation (RAG) are the two mainstream methods in existing researches. However, fine-tuning for specific tasks is resource-intensive and may diminish the model`s generalization capabilities. RAG can perform well on reducing the illusion of LLMs, but the generation of irrelevant factual content during reasoning can cause significant confusion for learners. To address these problems, we introduce the Molly agent, focusing on solving the proposed problem encountered by learners when learning Python programming language. Our agent automatically parse the learners' questioning intent through a scenario-based interaction, enabling precise retrieval of relevant documents from the constructed knowledge base. At generation stage, the agent reflect on the generated responses to ensure that they not only align with factual content but also effectively answer the user's queries. Extensive experimentation on a constructed Chinese Python QA dataset shows the effectiveness of the Molly agent, indicating an enhancement in its performance for providing useful responses to Python questions.
• 摘要：作为智能教育不可或缺的一部分，将大型语言模型 (LLM) 用作教学辅助手段已引起广泛关注，尤其是在计算机课程中。为了缩小 LLM 与计算机编程教育专家之间的差距，微调和检索增强生成 (RAG) 是现有研究中的两种主流方法。然而，针对特定任务的微调是资源密集型的，并且可能会降低模型的泛化能力。RAG 在减少 LLM 的错觉方面表现良好，但在推理过程中生成不相关的事实内容会给学习者带来很大的困惑。为了解决这些问题，我们引入了 Molly 代理，专注于解决学习者在学习 Python 编程语言时遇到的所提出的问题。我们的代理通过基于场景的交互自动解析学习者的提问意图，从而能够从构建的知识库中精确检索相关文档。在生成阶段，代理会反思生成的响应，以确保它们不仅与事实内容相符，而且还能有效地回答用户的查询。在构建的中文 Python QA 数据集上进行的大量实验证明了 Molly 代理的有效性，表明其在为 Python 问题提供有用答案的性能得到了提升。

Title: Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm

• Authors: Xiaoyang Hu, Richard L. Lewis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18120
• Pdf URL: https://arxiv.org/pdf/2412.18120
• Copy Paste: [[2412.18120]] Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm(https://arxiv.org/abs/2412.18120)
• Keywords: language model, gpt, prompt
• Abstract: Cognitive tasks originally developed for humans are now increasingly used to study language models. While applying these tasks is often straightforward, interpreting their results can be challenging. In particular, when a model underperforms, it's often unclear whether this results from a limitation in the cognitive ability being tested or a failure to understand the task itself. A recent study argued that GPT 3.5's declining performance on 2-back and 3-back tasks reflects a working memory capacity limit similar to humans. By analyzing a range of open-source language models of varying performance levels on these tasks, we show that the poor performance instead reflects a limitation in task comprehension and task set maintenance. In addition, we push the best performing model to higher n values and experiment with alternative prompting strategies, before analyzing model attentions. Our larger aim is to contribute to the ongoing conversation around refining methodologies for the cognitive evaluation of language models.
• 摘要：最初为人类开发的认知任务现在越来越多地用于研究语言模型。虽然应用这些任务通常很简单，但解释它们的结果却很困难。特别是，当一个模型表现不佳时，通常不清楚这是由于被测试的认知能力有限还是未能理解任务本身。最近的一项研究表明，GPT 3.5 在 2-back 和 3-back 任务中的表现下降反映了与人类类似的工作记忆容量限制。通过分析这些任务上性能水平各异的一系列开源语言模型，我们发现糟糕的表现反而反映了任务理解和任务集维护的局限性。此外，在分析模型注意力之前，我们将表现最佳的模型推向更高的 n 值并尝试替代的提示策略。我们更大的目标是为围绕改进语言模型认知评估方法的持续对话做出贡献。

Title: LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment

• Authors: Binrui Zeng, Bin Ji, Xiaodong Liu, Jie Yu, Shasha Li, Jun Ma, Xiaopeng Li, Shangwen Wang, Xinran Hong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18135
• Pdf URL: https://arxiv.org/pdf/2412.18135
• Copy Paste: [[2412.18135]] LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment(https://arxiv.org/abs/2412.18135)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) demonstrate exceptional performance across various domains, the deployment of these models on edge devices has emerged as a new trend. Quantization techniques, which reduce the size and memory footprint of LLMs, are effective for enabling deployment on resource-constrained edge devices. However, existing one-size-fits-all quantization methods often fail to dynamically adjust the memory consumption of LLMs based on specific hardware characteristics and usage scenarios. To address this limitation, we propose LSAQ (Layer-Specific Adaptive Quantization), a system for adaptive quantization and dynamic deployment of LLMs based on layer importance. LSAQ evaluates layer importance by constructing top-k token sets from the inputs and outputs of each layer and calculating their Jaccard coefficient. Using this evaluation, the system adaptively adjusts quantization strategies in real time according to the resource availability of edge devices, assigning different precision levels to layers of varying importance. This approach significantly reduces the storage requirements of LLMs while maintaining model performance, enabling efficient deployment across diverse hardware platforms and usage scenarios.
• 摘要：由于大型语言模型 (LLM) 在各个领域都表现出色，因此将这些模型部署在边缘设备上已成为一种新趋势。量化技术可以减少 LLM 的大小和内存占用，对于在资源受限的边缘设备上部署非常有效。然而，现有的一刀切量化方法往往无法根据特定的硬件特性和使用场景动态调整 LLM 的内存消耗。为了解决这一限制，我们提出了 LSAQ（层特定自适应量化），这是一种基于层重要性的自适应量化和动态部署 LLM 的系统。LSAQ 通过从每层的输入和输出构建前 k 个标记集并计算它们的 Jaccard 系数来评估层重要性。通过这种评估，系统可以根据边缘设备的资源可用性实时自适应地调整量化策略，为不同重要性的层分配不同的精度级别。这种方法在保持模型性能的同时显著降低了 LLM 的存储要求，实现了跨不同硬件平台和使用场景的高效部署。

Title: Ensuring Consistency for In-Image Translation

• Authors: Chengpeng Fu, Xiaocheng Feng, Yichong Huang, Wenshuai Huo, Baohang Li, Zhirui Zhang, Yunfei Lu, Dandan Tu, Duyu Tang, Hui Wang, Bing Qin, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18139
• Pdf URL: https://arxiv.org/pdf/2412.18139
• Copy Paste: [[2412.18139]] Ensuring Consistency for In-Image Translation(https://arxiv.org/abs/2412.18139)
• Keywords: language model
• Abstract: The in-image machine translation task involves translating text embedded within images, with the translated results presented in image format. While this task has numerous applications in various scenarios such as film poster translation and everyday scene image translation, existing methods frequently neglect the aspect of consistency throughout this process. We propose the need to uphold two types of consistency in this task: translation consistency and image generation consistency. The former entails incorporating image information during translation, while the latter involves maintaining consistency between the style of the text-image and the original image, ensuring background integrity. To address these consistency requirements, we introduce a novel two-stage framework named HCIIT (High-Consistency In-Image Translation) which involves text-image translation using a multimodal multilingual large language model in the first stage and image backfilling with a diffusion model in the second stage. Chain of thought learning is utilized in the first stage to enhance the model's ability to leverage image information during translation. Subsequently, a diffusion model trained for style-consistent text-image generation ensures uniformity in text style within images and preserves background details. A dataset comprising 400,000 style-consistent pseudo text-image pairs is curated for model training. Results obtained on both curated test sets and authentic image test sets validate the effectiveness of our framework in ensuring consistency and producing high-quality translated images.
• 摘要：图像内机器翻译任务涉及翻译嵌入在图像中的文本，并将翻译结果以图像格式呈现。虽然这项任务在电影海报翻译和日常场景图像翻译等各种场景中都有广泛的应用，但现有的方法经常忽略整个过程中的一致性。我们提出需要在这个任务中坚持两种一致性：翻译一致性和图像生成一致性。前者需要在翻译过程中融入图像信息，而后者需要在文本图像和原始图像的风格之间保持一致性，确保背景完整性。为了满足这些一致性要求，我们引入了一个新颖的两阶段框架，名为 HCIIT（高一致性图像内翻译），该框架涉及在第一阶段使用多模态多语言大型语言模型进行文本图像翻译，在第二阶段使用扩散模型进行图像回填。在第一阶段使用思路链学习来增强模型在翻译过程中利用图像信息的能力。随后，训练用于风格一致的文本图像生成的扩散模型可确保图像内文本风格的一致性并保留背景细节。为模型训练精心挑选了一个包含 400,000 个风格一致的伪文本-图像对的数据集。精选测试集和真实图像测试集上获得的结果验证了我们的框架在确保一致性和生成高质量翻译图像方面的有效性。

Title: CoAM: Corpus of All-Type Multiword Expressions

• Authors: Yusuke Ide, Joshua Tanner, Adam Nohejl, Jacob Hoffman, Justin Vasselli, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18151
• Pdf URL: https://arxiv.org/pdf/2412.18151
• Copy Paste: [[2412.18151]] CoAM: Corpus of All-Type Multiword Expressions(https://arxiv.org/abs/2412.18151)
• Keywords: language model
• Abstract: Multiword expressions (MWEs) refer to idiomatic sequences of multiple words. MWE identification, i.e., detecting MWEs in text, can play a key role in downstream tasks such as machine translation. Existing datasets for MWE identification are inconsistently annotated, limited to a single type of MWE, or limited in size. To enable reliable and comprehensive evaluation, we created CoAM: Corpus of All-Type Multiword Expressions, a dataset of 1.3K sentences constructed through a multi-step process to enhance data quality consisting of human annotation, human review, and automated consistency checking. MWEs in CoAM are tagged with MWE types, such as Noun and Verb, to enable fine-grained error analysis. Annotations for CoAM were collected using a new interface created with our interface generator, which allows easy and flexible annotation of MWEs in any form, including discontinuous ones. Through experiments using CoAM, we find that a fine-tuned large language model outperforms the current state-of-the-art approach for MWE identification. Furthermore, analysis using our MWE type tagged data reveals that Verb MWEs are easier than Noun MWEs to identify across approaches.
• 摘要：多词表达 (MWE) 是指多个单词的惯用序列。MWE 识别，即检测文本中的 MWE，可以在机器翻译等下游任务中发挥关键作用。现有的 MWE 识别数据集注释不一致、仅限于单一类型的 MWE 或大小有限。为了实现可靠和全面的评估，我们创建了 CoAM：全类型多词表达语料库，这是一个包含 1.3K 个句子的数据集，通过多步骤流程构建而成，以提高数据质量，包括人工注释、人工审核和自动一致性检查。CoAM 中的 MWE 被标记为 MWE 类型，例如名词和动词，以便进行细粒度错误分析。使用我们的界面生成器创建的新界面收集 CoAM 的注释，该界面允许轻松灵活地注释任何形式的 MWE，包括不连续的 MWE。通过使用 CoAM 的实验，我们发现经过微调的大型语言模型优于当前最先进的 MWE 识别方法。此外，使用我们的 MWE 类型标记数据进行分析表明，动词 MWE 比名词 MWE 更容易在各个方法中识别。

Title: An Analysis on Automated Metrics for Evaluating Japanese-English Chat Translation

• Authors: Andre Rusli, Makoto Shishido
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18190
• Pdf URL: https://arxiv.org/pdf/2412.18190
• Copy Paste: [[2412.18190]] An Analysis on Automated Metrics for Evaluating Japanese-English Chat Translation(https://arxiv.org/abs/2412.18190)
• Keywords: chat
• Abstract: This paper analyses how traditional baseline metrics, such as BLEU and TER, and neural-based methods, such as BERTScore and COMET, score several NMT models performance on chat translation and how these metrics perform when compared to human-annotated scores. The results show that for ranking NMT models in chat translations, all metrics seem consistent in deciding which model outperforms the others. This implies that traditional baseline metrics, which are faster and simpler to use, can still be helpful. On the other hand, when it comes to better correlation with human judgment, neural-based metrics outperform traditional metrics, with COMET achieving the highest correlation with the human-annotated score on a chat translation. However, we show that even the best metric struggles when scoring English translations from sentences with anaphoric zero-pronoun in Japanese.
• 摘要：本文分析了传统基线指标（例如 BLEU 和 TER）和基于神经网络的方法（例如 BERTScore 和 COMET）如何对聊天翻译中的几种 NMT 模型进行评分，以及这些指标与人工标注的分数相比的表现如何。结果表明，对于聊天翻译中的 NMT 模型排名，所有指标似乎都一致决定了哪个模型的表现优于其他模型。这意味着更快、更简单的传统基线指标仍然有用。另一方面，在与人类判断的更好相关性方面，基于神经网络的指标优于传统指标，其中 COMET 与聊天翻译中的人工标注分数的相关性最高。然而，我们表明，即使是最好的指标在对日语中带有回指零代词的句子进行英语翻译评分时也会遇到困难。

Title: Robustness-aware Automatic Prompt Optimization

• Authors: Zeru Shi, Zhenting Wang, Yongye Su, Weidi Luo, Fan Yang, Yongfeng Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18196
• Pdf URL: https://arxiv.org/pdf/2412.18196
• Copy Paste: [[2412.18196]] Robustness-aware Automatic Prompt Optimization(https://arxiv.org/abs/2412.18196)
• Keywords: language model, llm, prompt
• Abstract: The performance of Large Language Models (LLMs) is based on the quality of the prompts and the semantic and structural integrity information of the input data. However, current prompt generation methods primarily focus on generating prompts for clean input data, often overlooking the impact of perturbed inputs on prompt performance. To address this limitation, we propose BATprompt (By Adversarial Training prompt), a novel method for prompt generation designed to withstand input perturbations (such as typos in the input). Inspired by adversarial training techniques, BATprompt demonstrates strong performance on a variety of perturbed tasks through a two-step process: adversarial perturbation and iterative optimization on unperturbed input via LLM. Unlike conventional adversarial attack methods, BATprompt avoids reliance on real gradients or model parameters. Instead, it leverages the advanced reasoning, language understanding and self reflection capabilities of LLMs to simulate gradients, guiding the generation of adversarial perturbations and optimizing prompt performance. In our experiments, we evaluate BATprompt on multiple datasets across both language understanding and generation tasks. The results indicate that BATprompt outperforms existing prompt generation methods, delivering superior robustness and performance under diverse perturbation scenarios.
• 摘要：大型语言模型 (LLM) 的性能取决于提示的质量以及输入数据的语义和结构完整性信息。然而，当前的提示生成方法主要侧重于为干净的输入数据生成提示，往往忽略了扰动输入对提示性能的影响。为了解决这一限制，我们提出了 BATprompt（通过对抗训练提示），这是一种新颖的提示生成方法，旨在承受输入扰动（例如输入中的拼写错误）。受到对抗训练技术的启发，BATprompt 通过两步过程在各种扰动任务上表现出色：对抗性扰动和通过 LLM 对未扰动输入进行迭代优化。与传统的对抗性攻击方法不同，BATprompt 避免依赖真实梯度或模型参数。相反，它利用 LLM 的高级推理、语言理解和自我反思能力来模拟梯度，指导对抗性扰动的生成并优化提示性能。在我们的实验中，我们在语言理解和生成任务的多个数据集上评估了 BATprompt。结果表明，BATprompt 的表现优于现有的提示生成方法，在各种干扰场景下都具有出色的稳健性和性能。

Title: Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study

• Authors: Xuefeng Jiang, Lvhua Wu, Sheng Sun, Jia Li, Jingjing Xue, Yuwei Wang, Tingting Wu, Min Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18260
• Pdf URL: https://arxiv.org/pdf/2412.18260
• Copy Paste: [[2412.18260]] Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study(https://arxiv.org/abs/2412.18260)
• Keywords: language model, llm
• Abstract: Code vulnerability detection (CVD) is essential for addressing and preventing system security issues, playing a crucial role in ensuring software security. Previous learning-based vulnerability detection methods rely on either fine-tuning medium-size sequence models or training smaller neural networks from scratch. Recent advancements in large pre-trained language models (LLMs) have showcased remarkable capabilities in various code intelligence tasks including code understanding and generation. However, the effectiveness of LLMs in detecting code vulnerabilities is largely under-explored. This work aims to investigate the gap by fine-tuning LLMs for the CVD task, involving four widely-used open-source LLMs. We also implement other five previous graph-based or medium-size sequence models for comparison. Experiments are conducted on five commonly-used CVD datasets, including both the part of short samples and long samples. In addition, we conduct quantitative experiments to investigate the class imbalance issue and the model's performance on samples of different lengths, which are rarely studied in previous works. To better facilitate communities, we open-source all codes and resources of this study in this https URL and this https URL.
• 摘要：代码漏洞检测 (CVD) 对于解决和预防系统安全问题至关重要，在确保软件安全方面发挥着至关重要的作用。以前基于学习的漏洞检测方法依赖于微调中型序列模型或从头开始训练较小的神经网络。大型预训练语言模型 (LLM) 的最新进展已在各种代码智能任务（包括代码理解和生成）中展示了卓越的能力。然而，LLM 在检测代码漏洞方面的有效性在很大程度上尚未得到充分探索。这项工作旨在通过微调 LLM 进行 CVD 任务来调查差距，涉及四个广泛使用的开源 LLM。我们还实现了其他五个以前的基于图或中型序列模型进行比较。实验在五个常用的 CVD 数据集上进行，包括短样本和长样本的部分。此外，我们进行了定量实验来研究类别不平衡问题和模型在不同长度的样本上的性能，这在以前的工作中很少研究。为了更好地方便社区，我们在此 https URL 和此 https URL 中开源了本研究的所有代码和资源。

Title: GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge

• Authors: Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu, Kaan Efe Keles, Fatema Ahmad, Tasnim Mohiuddin, George Mikros, Firoj Alam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18274
• Pdf URL: https://arxiv.org/pdf/2412.18274
• Copy Paste: [[2412.18274]] GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge(https://arxiv.org/abs/2412.18274)
• Keywords: language model, llm
• Abstract: This paper presents a comprehensive overview of the first edition of the Academic Essay Authenticity Challenge, organized as part of the GenAI Content Detection shared tasks collocated with COLING 2025. This challenge focuses on detecting machine-generated vs. human-authored essays for academic purposes. The task is defined as follows: "Given an essay, identify whether it is generated by a machine or authored by a human.'' The challenge involves two languages: English and Arabic. During the evaluation phase, 25 teams submitted systems for English and 21 teams for Arabic, reflecting substantial interest in the task. Finally, seven teams submitted system description papers. The majority of submissions utilized fine-tuned transformer-based models, with one team employing Large Language Models (LLMs) such as Llama 2 and Llama 3. This paper outlines the task formulation, details the dataset construction process, and explains the evaluation framework. Additionally, we present a summary of the approaches adopted by participating teams. Nearly all submitted systems outperformed the n-gram-based baseline, with the top-performing systems achieving F1 scores exceeding 0.98 for both languages, indicating significant progress in the detection of machine-generated text.
• 摘要：本文全面概述了第一届学术论文真实性挑战赛，该挑战赛是与 COLING 2025 共同举办的 GenAI 内容检测共享任务的一部分。该挑战赛专注于为学术目的检测机器生成的论文与人类撰写的论文。该任务的定义如下：“给定一篇文章，判断它是由机器生成的还是由人类撰写的。”该挑战涉及两种语言：英语和阿拉伯语。在评估阶段，25 个团队提交了英语系统，21 个团队提交了阿拉伯语系统，反映出人们对该任务的浓厚兴趣。最后，七支队伍提交了系统描述论文。大多数提交的论文使用了经过微调的基于 Transformer 的模型，其中一个团队采用了大型语言模型 (LLM)，例如 Llama 2 和 Llama 3。本文概述了任务制定，详细介绍了数据集构建过程，并解释了评估框架。此外，我们还总结了参与团队采用的方法。几乎所有提交的系统都优于基于 n-gram 的基线，表现最好的系统在两种语言中的 F1 分数都超过 0.98，这表明在机器生成文本的检测方面取得了重大进展。

Title: M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models

• Authors: Jiaxin Guo, Daimeng Wei, Yuanchang Luo, Shimin Tao, Hengchao Shang, Zongyao Li, Shaojun Li, Jinlong Yang, Zhanglin Wu, Zhiqiang Rao, Hao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18299
• Pdf URL: https://arxiv.org/pdf/2412.18299
• Copy Paste: [[2412.18299]] M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models(https://arxiv.org/abs/2412.18299)
• Keywords: language model, llm, prompt
• Abstract: With the widespread application of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), enhancing their performance has become a research hotspot. This paper presents a novel multi-prompt ensemble decoding approach designed to bolster the generation quality of LLMs by leveraging the aggregation of outcomes from multiple prompts. Given a unique input $X$, we submit $n$ variations of prompts with $X$ to LLMs in batch mode to decode and derive probability distributions. For each token prediction, we calculate the ensemble probability by averaging the $n$ probability distributions within the batch, utilizing this aggregated probability to generate the token. This technique is dubbed Inner-Batch Ensemble. To facilitate efficient batch inference, we implement a Left-Padding strategy to maintain uniform input lengths across the n prompts. Through extensive experimentation on diverse NLP tasks, including machine translation, code generation, and text simplification, we demonstrate the efficacy of our method in enhancing LLM performance. The results show substantial improvements in BLEU scores, pass@$k$ rates, and LENS metrics over conventional methods.
• 摘要：随着大型语言模型 (LLM) 在自然语言处理 (NLP) 领域的广泛应用，提升其性能成为研究热点。本文提出了一种新颖的多提示集成解码方法，旨在通过利用多个提示结果的聚合来提高 LLM 的生成质量。给定一个唯一的输入 $X$，我们以批处理模式将 $n$ 个带有 $X$ 的提示变体提交给 LLM 以解码并得出概率分布。对于每个标记预测，我们通过平均批次内的 $n$ 个概率分布来计算集成概率，然后利用这个聚合概率生成标记。这种技术被称为批内集成。为了促进高效的批量推理，我们实施了左填充策略以在 n 个提示中保持一致的输入长度。通过对机器翻译、代码生成和文本简化等各种 NLP 任务进行大量实验，我们证明了我们的方法在提升 LLM 性能方面的有效性。结果显示，与传统方法相比，BLEU 分数、pass@$k$ 率和 LENS 指标有显著提高。

Title: Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering

• Authors: Zhongjian Hu, Peng Yang, Bing Li, Zhenqi Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18351
• Pdf URL: https://arxiv.org/pdf/2412.18351
• Copy Paste: [[2412.18351]] Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering(https://arxiv.org/abs/2412.18351)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.
• 摘要：大型语言模型 (LLM) 在基于知识的视觉问答 (VQA) 中取得了令人瞩目的成果。然而，现有方法仍然存在挑战：无法自主使用外部工具，无法团队协作。人类在遇到新问题时往往知道是否需要使用外部工具，例如，他们往往能够直接回答熟悉的问题，而当他们遇到不熟悉的问题时，他们倾向于使用搜索引擎等工具。此外，人类还倾向于与他人合作和讨论以获得更好的答案。受此启发，我们提出了多智能体投票框架。我们设计了三个基于 LLM 的智能体，模拟团队中不同级别的员工，并根据级别分配可用的工具。每个智能体提供相应的答案，最后对智能体提供的所有答案进行投票以获得最终答案。在 OK-VQA 和 A-OKVQA 上的实验表明，我们的方法分别比其他基线高出 2.2 和 1.0。

Title: Extracting triples from dialogues for conversational social agents

• Authors: Piek Vossen, Selene Báez Santamaría, Lenka Bajčetić, Thomas Belluci
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18364
• Pdf URL: https://arxiv.org/pdf/2412.18364
• Copy Paste: [[2412.18364]] Extracting triples from dialogues for conversational social agents(https://arxiv.org/abs/2412.18364)
• Keywords: agent
• Abstract: Obtaining an explicit understanding of communication within a Hybrid Intelligence collaboration is essential to create controllable and transparent agents. In this paper, we describe a number of Natural Language Understanding models that extract explicit symbolic triples from social conversation. Triple extraction has mostly been developed and tested for Knowledge Base Completion using Wikipedia text and data for training and testing. However, social conversation is very different as a genre in which interlocutors exchange information in sequences of utterances that involve statements, questions, and answers. Phenomena such as co-reference, ellipsis, coordination, and implicit and explicit negation or confirmation are more prominent in conversation than in Wikipedia text. We therefore describe an attempt to fill this gap by releasing data sets for training and testing triple extraction from social conversation. We also created five triple extraction models and tested them in our evaluation data. The highest precision is 51.14 for complete triples and 69.32 for triple elements when tested on single utterances. However, scores for conversational triples that span multiple turns are much lower, showing that extracting knowledge from true conversational data is much more challenging.
• 摘要：在混合智能协作中获得对通信的明确理解对于创建可控且透明的代理至关重要。在本文中，我们描述了一些自然语言理解模型，这些模型从社交对话中提取显式符号三元组。三元组提取主要是为知识库完成而开发和测试的，使用维基百科文本和数据进行训练和测试。然而，社交对话是一种非常不同的类型，在这种类型中，对话者以涉及陈述、问题和答案的话语序列交换信息。在对话中，共指、省略、协调以及隐含和显式否定或确认等现象比在维基百科文本中更为突出。因此，我们描述了一种尝试，即通过发布用于训练和测试社交对话三元组提取的数据集来填补这一空白。我们还创建了五个三元组提取模型并在我们的评估数据中对它们进行了测试。在单个话语上测试时，完整三元组的最高精度为 51.14，三元组元素的最高精度为 69.32。然而，跨越多个回合的对话三元组的得分要低得多，这表明从真实的对话数据中提取知识要困难得多。

Title: Towards Global AI Inclusivity: A Large-Scale Multilingual Terminology Dataset

• Authors: Jiarui Liu, Iman Ouzzani, Wenkai Li, Lechen Zhang, Tianyue Ou, Houda Bouamor, Zhijing Jin, Mona Diab
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18367
• Pdf URL: https://arxiv.org/pdf/2412.18367
• Copy Paste: [[2412.18367]] Towards Global AI Inclusivity: A Large-Scale Multilingual Terminology Dataset(https://arxiv.org/abs/2412.18367)
• Keywords: llm, prompt
• Abstract: The field of machine translation has achieved significant advancements, yet domain-specific terminology translation, particularly in AI, remains challenging. We introduced GIST, a large-scale multilingual AI terminology dataset containing 5K terms extracted from top AI conference papers spanning 2000 to 2023. The terms were translated into Arabic, Chinese, French, Japanese, and Russian using a hybrid framework that combines LLMs for extraction with human expertise for translation. The dataset's quality was benchmarked against existing resources, demonstrating superior translation accuracy through crowdsourced evaluation. GIST was integrated into translation workflows using post-translation refinement methods that required no retraining, where LLM prompting consistently improved BLEU and COMET scores. A web demonstration on the ACL Anthology platform highlights its practical application, showcasing improved accessibility for non-English speakers. This work aims to address critical gaps in AI terminology resources and fosters global inclusivity and collaboration in AI research.
• 摘要：机器翻译领域取得了重大进展，但特定领域的术语翻译，尤其是人工智能领域的术语翻译仍然具有挑战性。我们引入了 GIST，这是一个大规模多语言人工智能术语数据集，包含从 2000 年至 2023 年的顶级人工智能会议论文中提取的 5000 个术语。这些术语使用混合框架翻译成阿拉伯语、中文、法语、日语和俄语，该框架结合了提取的 LLM 和翻译的人类专业知识。该数据集的质量以现有资源为基准，通过众包评估展示了卓越的翻译准确性。GIST 使用不需要重新训练的翻译后细化方法集成到翻译工作流程中，其中 LLM 提示持续提高了 BLEU 和 COMET 分数。ACL Anthology 平台上的网络演示重点介绍了它的实际应用，展示了非英语人士的可访问性。这项工作旨在解决人工智能术语资源中的关键差距，并促进人工智能研究的全球包容性和协作。

Title: ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots

• Authors: Shani Goren, Oren Kalinsky, Tomer Stav, Yuri Rapoport, Yaron Fairstein, Ram Yazdy, Nachshon Cohen, Alexander Libov, Guy Kushilevitz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18377
• Pdf URL: https://arxiv.org/pdf/2412.18377
• Copy Paste: [[2412.18377]] ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots(https://arxiv.org/abs/2412.18377)
• Keywords: llm, chat
• Abstract: The rise of LLMs has deflected a growing portion of human-computer interactions towards LLM-based chatbots. The remarkable abilities of these models allow users to interact using long, diverse natural language text covering a wide range of topics and styles. Phrasing these messages is a time and effort consuming task, calling for an autocomplete solution to assist users. We introduce the task of chatbot interaction autocomplete. We present ChaI-TeA: CHat InTEraction Autocomplete; An autcomplete evaluation framework for LLM-based chatbot interactions. The framework includes a formal definition of the task, coupled with suitable datasets and metrics. We use the framework to evaluate After formally defining the task along with suitable datasets and metrics, we test 9 models on the defined auto completion task, finding that while current off-the-shelf models perform fairly, there is still much room for improvement, mainly in ranking of the generated suggestions. We provide insights for practitioners working on this task and open new research directions for researchers in the field. We release our framework to serve as a foundation for future research.
• 摘要：LLM 的兴起使越来越多的人机交互转向基于 LLM 的聊天机器人。这些模型的卓越能力使用户能够使用涵盖广泛主题和风格的长篇、多样化的自然语言文本进行交互。措辞这些消息是一项耗时耗力的任务，需要自动完成解决方案来帮助用户。我们介绍了聊天机器人交互自动完成的任务。我们提出了 ChaI-TeA：聊天自动完成；基于 LLM 的聊天机器人交互的自动完成评估框架。该框架包括任务的正式定义，以及合适的数据集和指标。我们使用该框架进行评估在正式定义任务以及合适的数据集和指标后，我们在定义的自动完成任务上测试了 9 个模型，发现虽然目前现成的模型表现不错，但仍有很大的改进空间，主要是在对生成的建议进行排名方面。我们为从事这项任务的从业者提供了见解，并为该领域的研究人员开辟了新的研究方向。我们发布了我们的框架，作为未来研究的基础。

Title: Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English

• Authors: Avinash Anand, Kritarth Prasad, Chhavi Kirtani, Ashwin R Nair, Manvendra Kumar Nema, Raj Jaiswal, Rajiv Ratn Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18415
• Pdf URL: https://arxiv.org/pdf/2412.18415
• Copy Paste: [[2412.18415]] Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English(https://arxiv.org/abs/2412.18415)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: Large Language Models (LLMs) excel in linguistic tasks but struggle with mathematical reasoning, particularly in non English languages like Hindi. This research aims to enhance the mathematical reasoning skills of smaller, resource efficient open-source LLMs in both Hindi and English. We evaluate models like OpenHathi 7B, LLaMA-2 7B, WizardMath 7B, Mistral 7B, LLeMMa 7B, MAmmoTH 7B, Gemini Pro, and GPT-4 using zero-shot, few-shot chain-of-thought (CoT) methods, and supervised fine-tuning. Our approach incorporates curriculum learning, progressively training models on increasingly difficult problems, a novel Decomposition Strategy to simplify complex arithmetic operations, and a Structured Solution Design that divides solutions into phases. Our experiments result in notable performance enhancements. WizardMath 7B exceeds Gemini's accuracy on English datasets by +6% and matches Gemini's performance on Hindi datasets. Adopting a bilingual approach that combines English and Hindi samples achieves results comparable to individual language models, demonstrating the capability to learn mathematical reasoning in both languages. This research highlights the potential for improving mathematical reasoning in open-source LLMs.
• 摘要：大型语言模型 (LLM) 在语言任务上表现出色，但在数学推理方面却举步维艰，尤其是在印地语等非英语语言中。这项研究旨在提高印地语和英语中较小、资源高效的开源 LLM 的数学推理能力。我们使用零样本、少量样本思维链 (CoT) 方法和监督微调来评估 OpenHathi 7B、LLaMA-2 7B、WizardMath 7B、Mistral 7B、LLeMMa 7B、MAmmoTH 7B、Gemini Pro 和 GPT-4 等模型。我们的方法结合了课程学习、针对越来越困难的问题逐步训练模型、一种简化复杂算术运算的新型分解策略以及将解决方案分为几个阶段的结构化解决方案设计。我们的实验显着提高了性能。WizardMath 7B 在英语数据集上的准确率比 Gemini 高出 6%，在印地语数据集上的表现与 Gemini 相当。采用结合英语和印地语样本的双语方法可实现与单个语言模型相当的结果，表明能够学习两种语言的数学推理。这项研究凸显了在开源法学硕士课程中提高数学推理能力的潜力。

Title: GeAR: Graph-enhanced Agent for Retrieval-augmented Generation

• Authors: Zhili Shen, Chenxin Diao, Pavlos Vougiouklis, Pascual Merita, Shriram Piramanayagam, Damien Graux, Dandan Tu, Zeren Jiang, Ruofei Lai, Yang Ren, Jeff Z. Pan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.18431
• Pdf URL: https://arxiv.org/pdf/2412.18431
• Copy Paste: [[2412.18431]] GeAR: Graph-enhanced Agent for Retrieval-augmented Generation(https://arxiv.org/abs/2412.18431)
• Keywords: retrieval-augmented generation, agent
• Abstract: Retrieval-augmented generation systems rely on effective document retrieval capabilities. By design, conventional sparse or dense retrievers face challenges in multi-hop retrieval scenarios. In this paper, we present GeAR, which advances RAG performance through two key innovations: (i) graph expansion, which enhances any conventional base retriever, such as BM25, and (ii) an agent framework that incorporates graph expansion. Our evaluation demonstrates GeAR's superior retrieval performance on three multi-hop question answering datasets. Additionally, our system achieves state-of-the-art results with improvements exceeding 10% on the challenging MuSiQue dataset, while requiring fewer tokens and iterations compared to other multi-step retrieval systems.
• 摘要：检索增强生成系统依赖于有效的文档检索功能。从设计上讲，传统的稀疏或密集检索器在多跳检索场景中面临挑战。在本文中，我们介绍了 GeAR，它通过两项关键创新提高了 RAG 性能：(i) 图形扩展，增强了任何传统的基础检索器，例如 BM25，以及 (ii) 包含图形扩展的代理框架。我们的评估证明了 GeAR 在三个多跳问答数据集上的卓越检索性能。此外，我们的系统在具有挑战性的 MuSiQue 数据集上实现了最先进的结果，改进超过 10%，同时与其他多步检索系统相比，需要更少的标记和迭代。

Title: Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks

• Authors: Abdullah Khondoker, Enam Ahmed Taufik, Md Iftekhar Islam Tashik, S M Ishtiak mahmud, Antara Firoz Parsa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18440
• Pdf URL: https://arxiv.org/pdf/2412.18440
• Copy Paste: [[2412.18440]] Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks(https://arxiv.org/abs/2412.18440)
• Keywords: language model
• Abstract: Evaluating text comprehension in educational settings is critical for understanding student performance and improving curricular effectiveness. This study investigates the capability of state-of-the-art language models-RoBERTa Base, Bangla-BERT, and BERT Base-in automatically assessing Bangla passage-based question-answering from the National Curriculum and Textbook Board (NCTB) textbooks for classes 6-10. A dataset of approximately 3,000 Bangla passage-based question-answering instances was compiled, and the models were evaluated using F1 Score and Exact Match (EM) metrics across various hyperparameter configurations. Our findings revealed that Bangla-BERT consistently outperformed the other models, achieving the highest F1 (0.75) and EM (0.53) scores, particularly with smaller batch sizes, the inclusion of stop words, and a moderate learning rate. In contrast, RoBERTa Base demonstrated the weakest performance, with the lowest F1 (0.19) and EM (0.27) scores under certain configurations. The results underscore the importance of fine-tuning hyperparameters for optimizing model performance and highlight the potential of machine learning models in evaluating text comprehension in educational contexts. However, limitations such as dataset size, spelling inconsistencies, and computational constraints emphasize the need for further research to enhance the robustness and applicability of these models. This study lays the groundwork for the future development of automated evaluation systems in educational institutions, providing critical insights into model performance in the context of Bangla text comprehension.
• 摘要：在教育环境中评估文本理解能力对于了解学生表现和提高课程效果至关重要。本研究调查了最先进的语言模型 RoBERTa Base、Bangla-BERT 和 BERT Base 在自动评估孟加拉语段落问答方面的能力，这些问答来自国家课程与教科书委员会 (NCTB) 为 6-10 年级编写的教科书。我们编制了一个包含约 3,000 个孟加拉语段落问答实例的数据集，并使用 F1 分数和精确匹配 (EM) 指标在各种超参数配置中对模型进行了评估。我们的研究结果表明，Bangla-BERT 的表现始终优于其他模型，获得了最高的 F1 (0.75) 和 EM (0.53) 分数，尤其是在批次大小较小、包含停用词和学习率适中的情况下。相比之下，RoBERTa Base 的表现最差，在某些配置下 F1 (0.19) 和 EM (0.27) 分数最低。结果强调了微调超参数对于优化模型性能的重要性，并突出了机器学习模型在评估教育环境中的文本理解方面的潜力。然而，数据集大小、拼写不一致和计算约束等限制强调了进一步研究的必要性，以增强这些模型的稳健性和适用性。这项研究为教育机构未来开发自动评估系统奠定了基础，为孟加拉语文本理解背景下的模型性能提供了关键见解。

Title: Is Large Language Model Good at Triple Set Prediction? An Empirical Study

• Authors: Yuan Yuan, Yajing Xu, Wen Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18443
• Pdf URL: https://arxiv.org/pdf/2412.18443
• Copy Paste: [[2412.18443]] Is Large Language Model Good at Triple Set Prediction? An Empirical Study(https://arxiv.org/abs/2412.18443)
• Keywords: language model, llm, hallucination, prompt
• Abstract: The core of the Knowledge Graph Completion (KGC) task is to predict and complete the missing relations or nodes in a KG. Common KGC tasks are mostly about inferring unknown elements with one or two elements being known in a triple. In comparison, the Triple Set Prediction (TSP) task is a more realistic knowledge graph completion task. It aims to predict all elements of unknown triples based on the information from known triples. In recent years, large language models (LLMs) have exhibited significant advancements in language comprehension, demonstrating considerable potential for KGC tasks. However, the potential of LLM on the TSP task has not yet to be investigated. Thus in this paper we proposed a new framework to explore the strengths and limitations of LLM in the TSP task. Specifically, the framework consists of LLM-based rule mining and LLM-based triple set prediction. The relation list of KG embedded within rich semantic information is first leveraged to prompt LLM in the generation of rules. This process is both efficient and independent of statistical information, making it easier to mine effective and realistic rules. For each subgraph, the specified rule is applied in conjunction with the relevant triples within that subgraph to guide the LLM in predicting the missing triples. Subsequently, the predictions from all subgraphs are consolidated to derive the complete set of predicted triples on KG. Finally, the method is evaluated on the relatively complete CFamily dataset. The experimental results indicate that when LLMs are required to adhere to a large amount of factual knowledge to predict missing triples, significant hallucinations occurs, leading to a noticeable decline in performance. To further explore the causes of this phenomenon, this paper presents a comprehensive analysis supported by a detailed case study.
• 摘要：知识图谱补全 (KGC) 任务的核心是预测和补全知识图谱中缺失的关系或节点。常见的知识图谱补全任务主要是推断三元组中一两个已知元素的未知元素。相比之下，三元组预测 (TSP) 任务是一个更现实的知识图谱补全任务。它旨在根据已知三元组的信息预测未知三元组的所有元素。近年来，大型语言模型 (LLM) 在语言理解方面取得了显著的进步，为知识图谱补全任务带来了巨大的潜力。然而，LLM 在 TSP 任务中的潜力尚未得到充分研究。因此，本文提出了一个新的框架来探索 LLM 在 TSP 任务中的优势和局限性。具体来说，该框架包括基于 LLM 的规则挖掘和基于 LLM 的三元组预测。首先利用嵌入丰富语义信息的知识图谱关系列表来提示 LLM 生成规则。这个过程既高效又独立于统计信息，使得挖掘有效和现实的规则变得更加容易。对于每个子图，将指定的规则与该子图中的相关三元组结合使用，以指导 LLM 预测缺失的三元组。随后，合并所有子图的预测以得出 KG 上的完整预测三元组集。最后，在相对完整的 CFamily 数据集上对该方法进行评估。实验结果表明，当 LLM 需要依靠大量事实知识来预测缺失三元组时，会出现明显的幻觉，导致性能明显下降。为了进一步探究这种现象的原因，本文提供了全面的分析，并附有详细的案例研究。

Title: Segment-Based Attention Masking for GPTs

• Authors: Shahar Katz, Liran Ringel, Yaniv Romano, Lior Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18487
• Pdf URL: https://arxiv.org/pdf/2412.18487
• Copy Paste: [[2412.18487]] Segment-Based Attention Masking for GPTs(https://arxiv.org/abs/2412.18487)
• Keywords: language model, gpt, prompt, chat
• Abstract: Modern Language Models (LMs) owe much of their success to masked causal attention, the backbone of Generative Pre-Trained Transformer (GPT) models. Although GPTs can process the entire user prompt at once, the causal masking is applied to all input tokens step-by-step, mimicking the generation process. This imposes an unnecessary constraint during the initial "prefill" phase when the model processes the input prompt and generates the internal representations before producing any output tokens. In this work, attention is masked based on the known block structure at the prefill phase, followed by the conventional token-by-token autoregressive process after that. For example, in a typical chat prompt, the system prompt is treated as one block, and the user prompt as the next one. Each of these is treated as a unit for the purpose of masking, such that the first tokens in each block can access the subsequent tokens in a non-causal manner. Then, the model answer is generated in the conventional causal manner. This Segment-by-Segment scheme entails no additional computational overhead. When integrating it into models such as Llama and Qwen, state-of-the-art performance is consistently achieved.
• 摘要：现代语言模型 (LM) 的成功很大程度上归功于掩蔽因果注意力，这是生成式预训练 Transformer (GPT) 模型的支柱。尽管 GPT 可以一次处理整个用户提示，但因果掩蔽会逐步应用于所有输入标记，模仿生成过程。这在初始“预填充”阶段施加了不必要的约束，此时模型处理输入提示并在生成任何输出标记之前生成内部表示。在这项工作中，注意力在预填充阶段根据已知的块结构进行掩蔽，然后进行传统的逐个标记自回归过程。例如，在典型的聊天提示中，系统提示被视为一个块，用户提示被视为下一个块。为了掩蔽的目的，每个块都被视为一个单元，这样每个块中的第一个标记就可以以非因果方式访问后续标记。然后，以传统的因果方式生成模型答案。这种逐段方案不需要额外的计算开销。当将其集成到 Llama 和 Qwen 等模型中时，始终能够实现最先进的性能。

Title: Generating event descriptions under syntactic and semantic constraints

• Authors: Angela Cao, Faye Holt, Jonas Chan, Stephanie Richter, Lelia Glass, Aaron Steven White
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18496
• Pdf URL: https://arxiv.org/pdf/2412.18496
• Copy Paste: [[2412.18496]] Generating event descriptions under syntactic and semantic constraints(https://arxiv.org/abs/2412.18496)
• Keywords: language model
• Abstract: With the goal of supporting scalable lexical semantic annotation, analysis, and theorizing, we conduct a comprehensive evaluation of different methods for generating event descriptions under both syntactic constraints -- e.g. desired clause structure -- and semantic constraints -- e.g. desired verb sense. We compare three different methods -- (i) manual generation by experts; (ii) sampling from a corpus annotated for syntactic and semantic information; and (iii) sampling from a language model (LM) conditioned on syntactic and semantic information -- along three dimensions of the generated event descriptions: (a) naturalness, (b) typicality, and (c) distinctiveness. We find that all methods reliably produce natural, typical, and distinctive event descriptions, but that manual generation continues to produce event descriptions that are more natural, typical, and distinctive than the automated generation methods. We conclude that the automated methods we consider produce event descriptions of sufficient quality for use in downstream annotation and analysis insofar as the methods used for this annotation and analysis are robust to a small amount of degradation in the resulting event descriptions.
• 摘要：为了支持可扩展的词汇语义注释、分析和理论化，我们对在句法约束（例如所需的子句结构）和语义约束（例如所需的动词意义）下生成事件描述的不同方法进行了全面评估。我们比较了三种不同的方法：（i）专家手动生成；（ii）从注释了句法和语义信息的语料库中抽样；（iii）从以句法和语义信息为条件的语言模型 (LM) 中抽样，沿着生成的事件描述的三个维度：（a）自然性、（b）典型性和（c）独特性。我们发现所有方法都能可靠地生成自然、典型和独特的事件描述，但手动生成仍然比自动生成方法生成更自然、更典型、更独特的事件描述。我们得出结论，我们所考虑的自动化方法可以生成足够质量的事件描述，以用于下游注释和分析，只要用于此注释和分析的方法对生成的事件描述的少量质量下降具有鲁棒性。

Title: Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization

• Authors: Yi-Fu Fu, Yu-Chieh Tu, Tzu-Ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, Shou-De Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18497
• Pdf URL: https://arxiv.org/pdf/2412.18497
• Copy Paste: [[2412.18497]] Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization(https://arxiv.org/abs/2412.18497)
• Keywords: language model, llm
• Abstract: In this paper, we explore the foundational mechanisms of memorization and generalization in Large Language Models (LLMs), inspired by the functional specialization observed in the human brain. Our investigation serves as a case study leveraging specially designed datasets and experimental-scale LLMs to lay the groundwork for understanding these behaviors. Specifically, we aim to first enable LLMs to exhibit both memorization and generalization by training with the designed dataset, then (a) examine whether LLMs exhibit neuron-level spatial differentiation for memorization and generalization, (b) predict these behaviors using model internal representations, and (c) steer the behaviors through inference-time interventions. Our findings reveal that neuron-wise differentiation of memorization and generalization is observable in LLMs, and targeted interventions can successfully direct their behavior.
• 摘要：在本文中，我们探索了大型语言模型 (LLM) 中记忆和泛化的基本机制，这种机制的灵感来自人类大脑中观察到的功能专业化。我们的研究是一个案例研究，利用专门设计的数据集和实验规模的 LLM 为理解这些行为奠定基础。具体来说，我们的目标是首先通过使用设计的数据集进行训练，使 LLM 能够同时表现出记忆和泛化，然后 (a) 检查 LLM 是否表现出神经元级的记忆和泛化空间分化，(b) 使用模型内部表示预测这些行为，以及 (c) 通过推理时间干预来引导行为。我们的研究结果表明，在 LLM 中可以观察到记忆和泛化的神经元分化，有针对性的干预可以成功引导它们的行为。

Title: Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

• Authors: Derong Xu Xinhang Li, Ziheng Zhang, Zhenxi Lin, Zhihong Zhu, Zhi Zheng, Xian Wu, Xiangyu Zhao, Tong Xu, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18537
• Pdf URL: https://arxiv.org/pdf/2412.18537
• Copy Paste: [[2412.18537]] Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation(https://arxiv.org/abs/2412.18537)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities, yet struggle with hallucination and outdated knowledge when tasked with complex knowledge reasoning, resulting in factually incorrect outputs. Previous studies have attempted to mitigate it by retrieving factual knowledge from large-scale knowledge graphs (KGs) to assist LLMs in logical reasoning and prediction of answers. However, this kind of approach often introduces noise and irrelevant data, especially in situations with extensive context from multiple knowledge aspects. In this way, LLM attention can be potentially mislead from question and relevant information. In our study, we introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. The Amar framework comprises two key sub-components: 1) a self-alignment module that aligns commonalities among entities, relations, and subgraphs to enhance retrieved text, thereby reducing noise interference; 2) a relevance gating module that employs a soft gate to learn the relevance score between question and multi-aspect retrieved data, to determine which information should be used to enhance LLMs' output, or even filtered altogether. Our method has achieved state-of-the-art performance on two common datasets, WebQSP and CWQ, showing a 1.9\% improvement in accuracy over its best competitor and a 6.6\% improvement in logical form generation over a method that directly uses retrieved text as context prompts. These results demonstrate the effectiveness of Amar in improving the reasoning of LLMs.
• 摘要：大型语言模型（LLM）表现出卓越的能力，然而在进行复杂的知识推理时却会出现幻觉和知识过时的问题，从而导致输出的结果与事实不符。先前的研究试图通过从大规模知识图谱（KG）中检索事实知识来帮助LLM进行逻辑推理和预测答案，以缓解这一问题。然而，这种方法往往会引入噪音和不相关的数据，尤其是在具有来自多个知识方面的广泛背景的情况下。这样，LLM的注意力可能会被问题和相关信息误导。在我们的研究中，我们引入了一个自适应多方面检索增强的KG（Amar）框架。该方法检索包括实体、关系和子图在内的知识，并将每段检索到的文本转换为提示嵌入。Amar框架包含两个关键子组件：1）自对齐模块，它对齐实体、关系和子图之间的共性以增强检索到的文本，从而减少噪音干扰； 2) 相关性门控模块，采用软门来学习问题与多方面检索数据之间的相关性得分，以确定哪些信息应该用于增强 LLM 的输出，甚至完全过滤。我们的方法在两个常见数据集 WebQSP 和 CWQ 上取得了最先进的性能，与最佳竞争对手相比，其准确度提高了 1.9%，与直接使用检索文本作为上下文提示的方法相比，逻辑形式生成提高了 6.6%。这些结果证明了 Amar 在改进 LLM 推理方面的有效性。

Title: Token-Budget-Aware LLM Reasoning

• Authors: Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, Zhenting Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18547
• Pdf URL: https://arxiv.org/pdf/2412.18547
• Copy Paste: [[2412.18547]] Token-Budget-Aware LLM Reasoning(https://arxiv.org/abs/2412.18547)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Reasoning is critical for large language models (LLMs) to excel in a wide range of tasks. While methods like Chain-of-Thought (CoT) reasoning enhance LLM performance by decomposing problems into intermediate steps, they also incur significant overhead in token usage, leading to increased costs. We find that the reasoning process of current LLMs is unnecessarily lengthy and it can be compressed by including a reasonable token budget in the prompt, but the choice of token budget plays a crucial role in the actual compression effectiveness. We then propose a token-budget-aware LLM reasoning framework, which dynamically estimates token budgets for different problems based on reasoning complexity and uses the estimated token budgets to guide the reasoning process. Experiments show that our method effectively reduces token costs in CoT reasoning with only a slight performance reduction, offering a practical solution to balance efficiency and accuracy in LLM reasoning. Code: this https URL.
• 摘要：推理是大型语言模型 (LLM) 在各种任务中脱颖而出的关键。虽然诸如思维链 (CoT) 推理之类的方法通过将问题分解为中间步骤来提高 LLM 性能，但它们也会在 token 使用方面产生大量开销，从而导致成本增加。我们发现当前 LLM 的推理过程不必要地冗长，可以通过在提示中包含合理的 token 预算来压缩它，但 token 预算的选择对实际压缩效果起着至关重要的作用。然后，我们提出了一个 token 预算感知的 LLM 推理框架，该框架根据推理复杂度动态估计不同问题的 token 预算，并使用估计的 token 预算来指导推理过程。实验表明，我们的方法有效地降低了 CoT 推理中的 token 成本，而性能仅略有降低，为平衡 LLM 推理的效率和准确性提供了一种实用的解决方案。代码：这个 https URL。

Title: Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability

• Authors: Haonan Li, Xudong Han, Zenan Zhai, Honglin Mu, Hao Wang, Zhenxuan Zhang, Yilin Geng, Shom Lin, Renxi Wang, Artem Shelmanov, Xiangyu Qi, Yuxia Wang, Donghai Hong, Youliang Yuan, Meng Chen, Haoqin Tu, Fajri Koto, Tatsuki Kuribayashi, Cong Zeng, Rishabh Bhardwaj, Bingchen Zhao, Yawen Duan, Yi Liu, Emad A. Alghamdi, Yaodong Yang, Yinpeng Dong, Soujanya Poria, Pengfei Liu, Zhengzhong Liu, Xuguang Ren, Eduard Hovy, Iryna Gurevych, Preslav Nakov, Monojit Choudhury, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18551
• Pdf URL: https://arxiv.org/pdf/2412.18551
• Copy Paste: [[2412.18551]] Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability(https://arxiv.org/abs/2412.18551)
• Keywords: llm
• Abstract: To address this gap, we introduce Libra-Leaderboard, a comprehensive framework designed to rank LLMs through a balanced evaluation of performance and safety. Combining a dynamic leaderboard with an interactive LLM arena, Libra-Leaderboard encourages the joint optimization of capability and safety. Unlike traditional approaches that average performance and safety metrics, Libra-Leaderboard uses a distance-to-optimal-score method to calculate the overall rankings. This approach incentivizes models to achieve a balance rather than excelling in one dimension at the expense of some other ones. In the first release, Libra-Leaderboard evaluates 26 mainstream LLMs from 14 leading organizations, identifying critical safety challenges even in state-of-the-art models.
• 摘要：为了弥补这一差距，我们推出了 Libra-Leaderboard，这是一个全面的框架，旨在通过平衡评估性能和安全性来对 LLM 进行排名。Libra-Leaderboard 将动态排行榜与交互式 LLM 竞技场相结合，鼓励对能力和安全性进行联合优化。与平均性能和安全指标的传统方法不同，Libra-Leaderboard 使用距离最优得分方法来计算总体排名。这种方法激励模型实现平衡，而不是在一个维度上表现出色而牺牲其他维度。在第一个版本中，Libra-Leaderboard 评估了来自 14 个领先组织的 26 个主流 LLM，即使在最先进的模型中也发现了关键的安全挑战。

Title: Distilling Fine-grained Sentiment Understanding from Large Language Models

• Authors: Yice Zhang, Guangyu Xie, Hongling Xu, Kaiheng Hou, Jianzhu Bao, Qianlong Wang, Shiwei Chen, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18552
• Pdf URL: https://arxiv.org/pdf/2412.18552
• Copy Paste: [[2412.18552]] Distilling Fine-grained Sentiment Understanding from Large Language Models(https://arxiv.org/abs/2412.18552)
• Keywords: language model, llm, prompt
• Abstract: Fine-grained sentiment analysis (FSA) aims to extract and summarize user opinions from vast opinionated text. Recent studies demonstrate that large language models (LLMs) possess exceptional sentiment understanding capabilities. However, directly deploying LLMs for FSA applications incurs high inference costs. Therefore, this paper investigates the distillation of fine-grained sentiment understanding from LLMs into small language models (SLMs). We prompt LLMs to examine and interpret the sentiments of given reviews and then utilize the generated content to pretrain SLMs. Additionally, we develop a comprehensive FSA benchmark to evaluate both SLMs and LLMs. Extensive experiments on this benchmark reveal that: (1) distillation significantly enhances the performance of SLMs in FSA tasks, achieving a 6.00\% improvement in $F_1$-score, and the distilled model can outperform Llama-2-7b with only 220M parameters; (2) distillation equips SLMs with excellent zero-shot sentiment classification capabilities, enabling them to match or even exceed their teacher models. These results suggest that distillation from LLMs is a highly promising direction for FSA. We will release our code, data, and pretrained model weights at \url{this https URL}.
• 摘要：细粒度情绪分析 (FSA) 旨在从大量带有观点的文本中提取和总结用户意见。最近的研究表明，大型语言模型 (LLM) 具有出色的情绪理解能力。然而，直接将 LLM 部署到 FSA 应用中会产生高昂的推理成本。因此，本文研究了将细粒度情绪理解从 LLM 提炼到小型语言模型 (SLM) 的过程。我们提示 LLM 检查和解释给定评论的情绪，然后利用生成的内容对 SLM 进行预训练。此外，我们开发了一个全面的 FSA 基准来评估 SLM 和 LLM。在这个基准上进行的大量实验表明：(1) 提炼显著提高了 SLM 在 FSA 任务中的表现，$F_1$ 分数提高了 6.00\%，提炼后的模型仅使用 220M 个参数就能胜过 Llama-2-7b；(2) 提炼使 SLM 具有出色的零样本情绪分类能力，使它们能够匹敌甚至超越其教师模型。这些结果表明，从 LLM 进行提炼是 FSA 的一个非常有前途的方向。我们将在 \url{此 https URL} 发布我们的代码、数据和预训练模型权重。

Title: Zero-resource Speech Translation and Recognition with LLMs

• Authors: Karel Mundnich, Xing Niu, Prashant Mathur, Srikanth Ronanki, Brady Houston, Veera Raghavendra Elluru, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Anshu Bhatia, Daniel Garcia-Romero, Kyu J. Han, Katrin Kirchhoff
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.18566
• Pdf URL: https://arxiv.org/pdf/2412.18566
• Copy Paste: [[2412.18566]] Zero-resource Speech Translation and Recognition with LLMs(https://arxiv.org/abs/2412.18566)
• Keywords: language model, llm
• Abstract: Despite recent advancements in speech processing, zero-resource speech translation (ST) and automatic speech recognition (ASR) remain challenging problems. In this work, we propose to leverage a multilingual Large Language Model (LLM) to perform ST and ASR in languages for which the model has never seen paired audio-text data. We achieve this by using a pre-trained multilingual speech encoder, a multilingual LLM, and a lightweight adaptation module that maps the audio representations to the token embedding space of the LLM. We perform several experiments both in ST and ASR to understand how to best train the model and what data has the most impact on performance in previously unseen languages. In ST, our best model is capable to achieve BLEU scores over 23 in CoVoST2 for two previously unseen languages, while in ASR, we achieve WERs of up to 28.2\%. We finally show that the performance of our system is bounded by the ability of the LLM to output text in the desired language.
• 摘要：尽管语音处理领域最近取得了进展，但零资源语音翻译 (ST) 和自动语音识别 (ASR) 仍然是一个具有挑战性的问题。在这项工作中，我们建议利用多语言大型语言模型 (LLM) 来执行 ST 和 ASR，这些语言模型从未见过配对的音频文本数据。我们通过使用预先训练的多语言语音编码器、多语言 LLM 和轻量级自适应模块来实现这一点，该模块将音频表示映射到 LLM 的标记嵌入空间。我们在 ST 和 ASR 中都进行了几次实验，以了解如何最好地训练模型以及哪些数据对以前从未见过的语言的性能影响最大。在 ST 中，我们最好的模型能够在 CoVoST2 中为两种以前从未见过的语言获得超过 23 的 BLEU 分数，而在 ASR 中，我们实现了高达 28.2% 的 WER。我们最终表明，我们系统的性能取决于 LLM 以所需语言输出文本的能力。

Title: Exploring Embedding Priors in Prompt-Tuning for Improved Interpretability and Control

• Authors: Sergey Sedov, Sumanth Bharadwaj Hachalli Karanam, Venu Gopal Kadamba
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18582
• Pdf URL: https://arxiv.org/pdf/2412.18582
• Copy Paste: [[2412.18582]] Exploring Embedding Priors in Prompt-Tuning for Improved Interpretability and Control(https://arxiv.org/abs/2412.18582)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Prompt-Tuning is an efficient method for adapting pre-trained language models to new tasks with minimal computational overhead by modifying prompt embeddings. In this work, we investigate how crucial the phenomenon of embedding collapse, frequently observed in Prompt-Tuning, is for the final performance of the model. To address this question, we designed embedding priors and compared them with posteriors of the converged Soft and Deep Prompt-Tuning methods. Our findings suggest that priors strongly affect the position of the tuned embeddings, and models can effectively work with embeddings from different parts of activation spaces, including completely new regions. As the final Prompt-Tuning capabilities are limited, we hypothesize that controllable Prompt-Tuning posteriors may serve as a good starting point for tasks such as chain-of-thought (COT) distillation. Our experiments also show that generated trajectories are not localized in the activation space of the models. However, there are distinct clusters of activations for distant tasks (e.g., NLP and arithmetic), while activations between NLP tasks (e.g., Question-Answering and MLM) lie in the same cluster. These observations raise questions about the importance of a single activation cluster for the generalization abilities of large language models.
• 摘要：Prompt-Tuning 是一种有效的方法，它通过修改提示嵌入，以最小的计算开销将预训练的语言模型适应新任务。在这项工作中，我们研究了在 Prompt-Tuning 中经常观察到的嵌入崩溃现象对模型的最终性能有多么重要。为了解决这个问题，我们设计了嵌入先验，并将它们与融合的 Soft 和 Deep Prompt-Tuning 方法的后验进行了比较。我们的研究结果表明，先验强烈影响调整后的嵌入的位置，并且模型可以有效地处理来自激活空间不同部分（包括全新区域）的嵌入。由于最终的 Prompt-Tuning 功能有限，我们假设可控的 Prompt-Tuning 后验可以作为思路链 (COT) 提炼等任务的良好起点。我们的实验还表明，生成的轨迹并不局限于模型的激活空间中。然而，对于相距较远的任务（例如 NLP 和算术），存在不同的激活簇，而 NLP 任务（例如问答和 MLM）之间的激活则位于同一簇中。这些观察结果提出了一个问题：单个激活簇对于大型语言模型的泛化能力的重要性。

Title: Long-Form Speech Generation with Spoken Language Models

• Authors: Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, RJ Skerry-Ryan
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.18603
• Pdf URL: https://arxiv.org/pdf/2412.18603
• Copy Paste: [[2412.18603]] Long-Form Speech Generation with Spoken Language Models(https://arxiv.org/abs/2412.18603)
• Keywords: language model, llm
• Abstract: We consider the generative modeling of speech over multiple minutes, a requirement for long-form multimedia generation and audio-native voice assistants. However, current spoken language models struggle to generate plausible speech past tens of seconds, from high temporal resolution of speech tokens causing loss of coherence, to architectural issues with long-sequence training or extrapolation, to memory costs at inference time. With these considerations we propose SpeechSSM, the first speech language model to learn from and sample long-form spoken audio (e.g., 16 minutes of read or extemporaneous speech) in a single decoding session without text intermediates, based on recent advances in linear-time sequence modeling. Furthermore, to address growing challenges in spoken language evaluation, especially in this new long-form setting, we propose: new embedding-based and LLM-judged metrics; quality measurements over length and time; and a new benchmark for long-form speech processing and generation, LibriSpeech-Long. Speech samples and the dataset are released at this https URL
• 摘要：我们考虑对几分钟内的语音进行生成建模，这是长篇多媒体生成和音频原生语音助手的要求。然而，目前的口语语言模型很难生成超过几十秒的可信语音，从语音标记的高时间分辨率导致连贯性丧失，到长序列训练或外推的架构问题，再到推理时的内存成本。考虑到这些因素，我们提出了 SpeechSSM，这是第一个基于线性时间序列建模的最新进展，在单个解码会话中学习和采样长篇口语音频（例如 16 分钟的朗读或即兴演讲）的语音语言模型，无需文本中间体。此外，为了应对口语语言评估中日益增长的挑战，特别是在这种新的长篇环境中，我们提出了：新的基于嵌入和 LLM 评判的指标；长度和时间的质量测量；以及长篇语音处理和生成的新基准 LibriSpeech-Long。语音样本和数据集发布在此 https URL