2024-12-19

Title: In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning

• Authors: Yifei Duan, Liu Li, Zirui Zhai, Jinxia Yao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13243
• Pdf URL: https://arxiv.org/pdf/2412.13243
• Copy Paste: [[2412.13243]] In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning(https://arxiv.org/abs/2412.13243)
• Keywords: prompt
• Abstract: We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning.
• 摘要：我们在自然语言推理任务的 OPT-1.3B 模型上应用了少样本上下文学习，并通过知识蒸馏将上下文信息内化，将模型参数从 1.3B 减少到 125M，大小从 2.5GB 减少到 0.25GB。与在类似大小的模型上单独使用上下文学习相比，这种上下文蒸馏方法实现了域外准确率的近 50% 提升，展示了优于基于提示的方法的知识迁移能力。此外，与传统的基于模式的微调相比，该方法在域外准确率提高 20% 的同时，内存消耗降低了 60%。

Title: Enhancing Persona Classification in Dialogue Systems: A Graph Neural Network Approach

• Authors: Konstantin Zaitsev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13283
• Pdf URL: https://arxiv.org/pdf/2412.13283
• Copy Paste: [[2412.13283]] Enhancing Persona Classification in Dialogue Systems: A Graph Neural Network Approach(https://arxiv.org/abs/2412.13283)
• Keywords: language model, llm, chat
• Abstract: In recent years, Large Language Models (LLMs) gain considerable attention for their potential to enhance personalized experiences in virtual assistants and chatbots. A key area of interest is the integration of personas into LLMs to improve dialogue naturalness and user engagement. This study addresses the challenge of persona classification, a crucial component in dialogue understanding, by proposing a framework that combines text embeddings with Graph Neural Networks (GNNs) for effective persona classification. Given the absence of dedicated persona classification datasets, we create a manually annotated dataset to facilitate model training and evaluation. Our method involves extracting semantic features from persona statements using text embeddings and constructing a graph where nodes represent personas and edges capture their similarities. The GNN component uses this graph structure to propagate relevant information, thereby improving classification performance. Experimental results show that our approach, in particular the integration of GNNs, significantly improves classification performance, especially with limited data. Our contributions include the development of a persona classification framework and the creation of a dataset.
• 摘要：近年来，大型语言模型 (LLM) 因其在虚拟助手和聊天机器人中增强个性化体验的潜力而备受关注。一个关键的关注领域是将角色集成到 LLM 中，以提高对话的自然度和用户参与度。这项研究通过提出一个将文本嵌入与图神经网络 (GNN) 相结合的框架来实现有效的角色分类，解决了角色分类这一对话理解的重要组成部分的挑战。鉴于缺乏专门的角色分类数据集，我们创建了一个手动注释的数据集以促进模型训练和评估。我们的方法包括使用文本嵌入从角色陈述中提取语义特征，并构建一个图，其中节点代表角色，边捕获它们的相似性。GNN 组件使用此图结构传播相关信息，从而提高分类性能。实验结果表明，我们的方法，特别是 GNN 的集成，显着提高了分类性能，尤其是在数据有限的情况下。我们的贡献包括角色分类框架的开发和数据集的创建。

Title: Hint Marginalization for Improved Reasoning in Large Language Models

• Authors: Soumyasundar Pal, Didier Chételat, Yingxue Zhang, Mark Coates
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13292
• Pdf URL: https://arxiv.org/pdf/2412.13292
• Copy Paste: [[2412.13292]] Hint Marginalization for Improved Reasoning in Large Language Models(https://arxiv.org/abs/2412.13292)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have exhibited an impressive capability to perform reasoning tasks, especially if they are encouraged to generate a sequence of intermediate steps. Reasoning performance can be improved by suitably combining multiple LLM responses, generated either in parallel in a single query, or via sequential interactions with LLMs throughout the reasoning process. Existing strategies for combination, such as self-consistency and progressive-hint-prompting, make inefficient usage of the LLM responses. We present Hint Marginalization, a novel and principled algorithmic framework to enhance the reasoning capabilities of LLMs. Our approach can be viewed as an iterative sampling strategy for forming a Monte Carlo approximation of an underlying distribution of answers, with the goal of identifying the mode the most likely answer. Empirical evaluation on several benchmark datasets for arithmetic reasoning demonstrates the superiority of the proposed approach.
• 摘要：大型语言模型 (LLM) 表现出了执行推理任务的出色能力，尤其是在鼓励它们生成一系列中间步骤的情况下。通过适当组合多个 LLM 响应（这些响应可以在单个查询中并行生成，也可以在整个推理过程中通过与 LLM 的顺序交互生成），可以提高推理性能。现有的组合策略（例如自洽和渐进提示提示）使 LLM 响应的使用效率低下。我们提出了提示边缘化，这是一种新颖且有原则的算法框架，可增强 LLM 的推理能力。我们的方法可以看作是一种迭代采样策略，用于形成答案底层分布的蒙特卡罗近似值，目的是确定最有可能的答案模式。对几个算术推理基准数据集的实证评估证明了所提出方法的优越性。

Title: Experience of Training a 1.7B-Parameter LLaMa Model From Scratch

• Authors: Miles Q. Li, Benjamin C. M. Fung, Shih-Chia Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13335
• Pdf URL: https://arxiv.org/pdf/2412.13335
• Copy Paste: [[2412.13335]] Experience of Training a 1.7B-Parameter LLaMa Model From Scratch(https://arxiv.org/abs/2412.13335)
• Keywords: language model
• Abstract: Pretraining large language models is a complex endeavor influenced by multiple factors, including model architecture, data quality, training continuity, and hardware constraints. In this paper, we share insights gained from the experience of training DMaS-LLaMa-Lite, a fully open source, 1.7-billion-parameter, LLaMa-based model, on approximately 20 billion tokens of carefully curated data. We chronicle the full training trajectory, documenting how evolving validation loss levels and downstream benchmarks reflect transitions from incoherent text to fluent, contextually grounded output. Beyond standard quantitative metrics, we highlight practical considerations such as the importance of restoring optimizer states when resuming from checkpoints, and the impact of hardware changes on training stability and throughput. While qualitative evaluation provides an intuitive understanding of model improvements, our analysis extends to various performance benchmarks, demonstrating how high-quality data and thoughtful scaling enable competitive results with significantly fewer training tokens. By detailing these experiences and offering training logs, checkpoints, and sample outputs, we aim to guide future researchers and practitioners in refining their pretraining strategies. The training script is available on Github at this https URL. The model checkpoints are available on Huggingface at this https URL.
• 摘要：大型语言模型的预训练是一项复杂的工作，受多种因素影响，包括模型架构、数据质量、训练连续性和硬件限制。在本文中，我们分享了在约 200 亿个精心策划的数据标记上训练 DMaS-LLaMa-Lite（一个完全开源的、拥有 17 亿个参数的基于 LLaMa 的模型）的经验。我们记录了完整的训练轨迹，记录了不断变化的验证损失水平和下游基准如何反映从不连贯的文本到流畅的、基于上下文的输出的转变。除了标准的定量指标外，我们还强调了一些实际考虑因素，例如从检查点恢复时恢复优化器状态的重要性，以及硬件变化对训练稳定性和吞吐量的影响。虽然定性评估提供了对模型改进的直观理解，但我们的分析扩展到各种性能基准，展示了高质量数据和周到的扩展如何以更少的训练标记实现具有竞争力的结果。通过详细介绍这些经验并提供训练日志、检查点和示例输出，我们旨在指导未来的研究人员和从业者完善他们的预训练策略。训练脚本可在 Github 上的 https URL 上找到。模型检查点可在 Huggingface 上的 https URL 上找到。

Title: Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation

• Authors: Samin Mahdizadeh Sani, Pouya Sadeghi, Thuy-Trang Vu, Yadollah Yaghoobzadeh, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13375
• Pdf URL: https://arxiv.org/pdf/2412.13375
• Copy Paste: [[2412.13375]] Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation(https://arxiv.org/abs/2412.13375)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have made great progress in classification and text generation tasks. However, they are mainly trained on English data and often struggle with low-resource languages. In this study, we explore adding a new language, i.e., Persian, to Llama (a model with a limited understanding of Persian) using parameter-efficient fine-tuning. We employ a multi-stage approach involving pretraining on monolingual Persian data, aligning representations through bilingual pretraining and instruction datasets, and instruction-tuning with task-specific datasets. We evaluate the model's performance at each stage on generation and classification tasks. Our findings suggest that incorporating the Persian language, through bilingual data alignment, can enhance classification accuracy for Persian tasks, with no adverse impact and sometimes even improvements on English tasks. Additionally, the results highlight the model's initial strength as a critical factor when working with limited training data, with cross-lingual alignment offering minimal benefits for the low-resource language. Knowledge transfer from English to Persian has a marginal effect, primarily benefiting simple classification tasks.
• 摘要：大型语言模型 (LLM) 在分类和文本生成任务方面取得了巨大进步。然而，它们主要在英语数据上进行训练，并且经常难以处理资源匮乏的语言。在本研究中，我们探索使用参数高效的微调将一种新语言（即波斯语）添加到 Llama（一种对波斯语理解有限的模型）中。我们采用多阶段方法，包括在单语波斯语数据上进行预训练、通过双语预训练和指令数据集对齐表示以及使用特定于任务的数据集进行指令调整。我们评估模型在生成和分类任务的每个阶段的性能。我们的研究结果表明，通过双语数据对齐加入波斯语可以提高波斯语任务的分类准确性，而不会对英语任务产生不利影响，有时甚至会有所改进。此外，结果强调了模型的初始强度是使用有限训练数据时的关键因素，跨语言对齐对资源匮乏的语言提供的好处微乎其微。从英语到波斯语的知识转移效果微乎其微，主要有利于简单的分类任务。

Title: DateLogicQA: Benchmarking Temporal Biases in Large Language Models

• Authors: Gagan Bhatia, MingZe Tang, Cristina Mahanta, Madiha Kazi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13377
• Pdf URL: https://arxiv.org/pdf/2412.13377
• Copy Paste: [[2412.13377]] DateLogicQA: Benchmarking Temporal Biases in Large Language Models(https://arxiv.org/abs/2412.13377)
• Keywords: language model, llm
• Abstract: This paper introduces DateLogicQA, a benchmark with 190 questions covering diverse date formats, temporal contexts, and reasoning types. We propose the Semantic Integrity Metric to assess tokenization quality and analyse two biases: Representation-Level Bias, affecting embeddings, and Logical-Level Bias, influencing reasoning outputs. Our findings provide a comprehensive evaluation of LLMs' capabilities and limitations in temporal reasoning, highlighting key challenges in handling temporal data accurately. The GitHub repository for our work is available at this https URL
• 摘要：本文介绍了 DateLogicQA，这是一个包含 190 个问题的基准，涵盖各种日期格式、时间背景和推理类型。我们提出了语义完整性指标来评估标记化质量并分析两种偏差：影响嵌入的表示级偏差和影响推理输出的逻辑级偏差。我们的研究结果对 LLM 在时间推理方面的能力和局限性进行了全面评估，突出了准确处理时间数据的关键挑战。我们工作的 GitHub 存储库可在此 https URL 上找到

Title: An Automated Explainable Educational Assessment System Built on LLMs

• Authors: Jiazheng Li, Artem Bobrov, David West, Cesare Aloisi, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13381
• Pdf URL: https://arxiv.org/pdf/2412.13381
• Copy Paste: [[2412.13381]] An Automated Explainable Educational Assessment System Built on LLMs(https://arxiv.org/abs/2412.13381)
• Keywords: language model, llm, chat
• Abstract: In this demo, we present AERA Chat, an automated and explainable educational assessment system designed for interactive and visual evaluations of student responses. This system leverages large language models (LLMs) to generate automated marking and rationale explanations, addressing the challenge of limited explainability in automated educational assessment and the high costs associated with annotation. Our system allows users to input questions and student answers, providing educators and researchers with insights into assessment accuracy and the quality of LLM-assessed rationales. Additionally, it offers advanced visualization and robust evaluation tools, enhancing the usability for educational assessment and facilitating efficient rationale verification. Our demo video can be found at this https URL.
• 摘要：在此演示中，我们展示了 AERA Chat，这是一种自动化且可解释的教育评估系统，旨在对学生的回答进行交互式和视觉评估。该系统利用大型语言模型 (LLM) 生成自动标记和理由解释，解决了自动教育评估中可解释性有限以及注释成本高昂的挑战。我们的系统允许用户输入问题和学生答案，为教育工作者和研究人员提供有关评估准确性和 LLM 评估理由质量的见解。此外，它还提供高级可视化和强大的评估工具，增强了教育评估的可用性并促进了有效的理由验证。我们的演示视频可以在此 https URL 中找到。

Title: Lightweight Safety Classification Using Pruned Language Models

• Authors: Mason Sawtell, Tula Masterman, Sandi Besen, Jim Brown
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13435
• Pdf URL: https://arxiv.org/pdf/2412.13435
• Copy Paste: [[2412.13435]] Lightweight Safety Classification Using Pruned Language Models(https://arxiv.org/abs/2412.13435)
• Keywords: language model, gpt, llm, prompt
• Abstract: In this paper, we introduce a novel technique for content safety and prompt injection classification for Large Language Models. Our technique, Layer Enhanced Classification (LEC), trains a Penalized Logistic Regression (PLR) classifier on the hidden state of an LLM's optimal intermediate transformer layer. By combining the computational efficiency of a streamlined PLR classifier with the sophisticated language understanding of an LLM, our approach delivers superior performance surpassing GPT-4o and special-purpose models fine-tuned for each task. We find that small general-purpose models (Qwen 2.5 sizes 0.5B, 1.5B, and 3B) and other transformer-based architectures like DeBERTa v3 are robust feature extractors allowing simple classifiers to be effectively trained on fewer than 100 high-quality examples. Importantly, the intermediate transformer layers of these models typically outperform the final layer across both classification tasks. Our results indicate that a single general-purpose LLM can be used to classify content safety, detect prompt injections, and simultaneously generate output tokens. Alternatively, these relatively small LLMs can be pruned to the optimal intermediate layer and used exclusively as robust feature extractors. Since our results are consistent on different transformer architectures, we infer that robust feature extraction is an inherent capability of most, if not all, LLMs.
• 摘要：在本文中，我们介绍了一种用于大型语言模型的内容安全和提示注入分类的新技术。我们的技术，即层增强分类 (LEC)，在 LLM 最佳中间 Transformer 层的隐藏状态下训练惩罚逻辑回归 (PLR) 分类器。通过将精简的 PLR 分类器的计算效率与 LLM 的复杂语言理解相结合，我们的方法提供了超越 GPT-4o 和针对每个任务进行微调的专用模型的卓越性能。我们发现小型通用模型（Qwen 2.5 大小 0.5B、1.5B 和 3B）和其他基于 Transformer 的架构（如 DeBERTa v3）是强大的特征提取器，允许在少于 100 个高质量示例上有效地训练简单的分类器。重要的是，这些模型的中间 Transformer 层在两个分类任务中通常都优于最后一层。我们的结果表明，单个通用 LLM 可用于对内容安全进行分类、检测提示注入并同时生成输出标记。或者，这些相对较小的 LLM 可以修剪到最佳中间层，并专门用作稳健的特征提取器。由于我们的结果在不同的 Transformer 架构上是一致的，我们推断，稳健的特征提取是大多数（如果不是全部）LLM 的固有功能。

Title: A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models

• Authors: Bowen Chen, Namgi Han, Yusuke Miyao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13475
• Pdf URL: https://arxiv.org/pdf/2412.13475
• Copy Paste: [[2412.13475]] A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models(https://arxiv.org/abs/2412.13475)
• Keywords: language model, llm
• Abstract: The lack of data transparency in Large Language Models (LLMs) has highlighted the importance of Membership Inference Attack (MIA), which differentiates trained (member) and untrained (non-member) data. Though it shows success in previous studies, recent research reported a near-random performance in different settings, highlighting a significant performance inconsistency. We assume that a single setting doesn't represent the distribution of the vast corpora, causing members and non-members with different distributions to be sampled and causing inconsistency. In this study, instead of a single setting, we statistically revisit MIA methods from various settings with thousands of experiments for each MIA method, along with study in text feature, embedding, threshold decision, and decoding dynamics of members and non-members. We found that (1) MIA performance improves with model size and varies with domains, while most methods do not statistically outperform baselines, (2) Though MIA performance is generally low, a notable amount of differentiable member and non-member outliers exists and vary across MIA methods, (3) Deciding a threshold to separate members and non-members is an overlooked challenge, (4) Text dissimilarity and long text benefit MIA performance, (5) Differentiable or not is reflected in the LLM embedding, (6) Member and non-members show different decoding dynamics.
• 摘要：大型语言模型 (LLM) 缺乏数据透明度，这凸显了成员推理攻击 (MIA) 的重要性，MIA 可区分经过训练的（成员）和未经训练的（非成员）数据。尽管它在之前的研究中显示出成功，但最近的研究报告称，在不同设置下，MIA 的性能接近随机，凸显了显著的性能不一致。我们假设单一设置不能代表庞大语料库的分布，导致对具有不同分布的成员和非成员进行采样并导致不一致。在本研究中，我们不是针对单一设置，而是从统计上重新审视来自各种设置的 MIA 方法，对每种 MIA 方法进行数千次实验，同时研究成员和非成员的文本特征、嵌入、阈值决策和解码动态。我们发现：（1）MIA 性能随着模型大小的提高而提高，并且因领域而异，而大多数方法在统计上并不优于基线；（2）虽然 MIA 性能通常较低，但存在大量可区分的成员和非成员异常值，并且因 MIA 方法而异；（3）确定分离成员和非成员的阈值是一个被忽视的挑战；（4）文本差异和长文本有利于 MIA 性能；（5）是否可区分反映在 LLM 嵌入中；（6）成员和非成员表现出不同的解码动态。

Title: Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

• Authors: Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, Xitong Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13488
• Pdf URL: https://arxiv.org/pdf/2412.13488
• Copy Paste: [[2412.13488]] Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models(https://arxiv.org/abs/2412.13488)
• Keywords: language model, llm
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has gained prominence through low-rank adaptation methods like LoRA. In this paper, we focus on sparsity-based PEFT (SPEFT), which introduces trainable sparse adaptations to the weight matrices in the model, offering greater flexibility in selecting fine-tuned parameters compared to low-rank methods. We conduct the first systematic evaluation of salience metrics for SPEFT, inspired by zero-cost NAS proxies, and identify simple gradient-based metrics is reliable, and results are on par with the best alternatives, offering both computational efficiency and robust performance. Additionally, we compare static and dynamic masking strategies, finding that static masking, which predetermines non-zero entries before training, delivers efficiency without sacrificing performance, while dynamic masking offers no substantial benefits. Across NLP tasks, a simple gradient-based, static SPEFT consistently outperforms other fine-tuning methods for LLMs, providing a simple yet effective baseline for SPEFT. Our work challenges the notion that complexity is necessary for effective PEFT. Our work is open source and available to the community at [this https URL].
• 摘要：参数高效微调 (PEFT) 通过 LoRA 等低秩自适应方法而备受关注。在本文中，我们重点介绍基于稀疏性的 PEFT (SPEFT)，它为模型中的权重矩阵引入了可训练的稀疏自适应，与低秩方法相比，在选择微调参数方面提供了更大的灵活性。受零成本 NAS 代理的启发，我们对 SPEFT 的显着性指标进行了首次系统评估，并确定基于梯度的简单指标是可靠的，结果与最佳替代方案相当，既提供计算效率又提供稳健的性能。此外，我们比较了静态和动态掩码策略，发现静态掩码（在训练之前预先确定非零条目）可在不牺牲性能的情况下提供效率，而动态掩码没有提供实质性的好处。在 NLP 任务中，简单的基于梯度的静态 SPEFT 始终优于其他 LLM 微调方法，为 SPEFT 提供了简单而有效的基线。我们的工作挑战了“复杂性是有效 PEFT 的必要条件”这一观念。我们的工作是开源的，社区可以通过 [此 https URL] 获取。

Title: MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules

• Authors: Kejie Chen, Lin Wang, Qinghai Zhang, Renjun Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13536
• Pdf URL: https://arxiv.org/pdf/2412.13536
• Copy Paste: [[2412.13536]] MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules(https://arxiv.org/abs/2412.13536)
• Keywords: language model, gpt
• Abstract: Recent studies have highlighted the limitations of large language models in mathematical reasoning, particularly their inability to capture the underlying logic. Inspired by meta-learning, we propose that models should acquire not only task-specific knowledge but also transferable problem-solving skills. We introduce MetaRuleGPT, a novel Transformer-based architecture that performs precise numerical calculations and complex logical operations by learning and combining different rules. In contrast with traditional training sets, which are heavily composed of massive raw instance data, MetaRuleGPT is pre-trained on much less abstract datasets containing basic, compound, and iterative rules for mathematical reasoning. Extensive experimental results demonstrate MetaRuleGPT can mimic human's rule-following capabilities, break down complexity, and iteratively derive accurate results for complex mathematical problems. These findings prove the potential of rule learning to enhance the numerical reasoning abilities of language models.
• 摘要：最近的研究强调了大型语言模型在数学推理中的局限性，特别是它们无法捕捉底层逻辑。受元学习的启发，我们提出模型不仅应该获得特定于任务的知识，还应该获得可转移的问题解决技能。我们介绍了 MetaRuleGPT，这是一种基于 Transformer 的新型架构，它通过学习和组合不同的规则来执行精确的数值计算和复杂的逻辑运算。与由大量原始实例数据组成的传统训练集相比，MetaRuleGPT 在抽象程度低得多的数据集上进行了预训练，其中包含用于数学推理的基本、复合和迭代规则。大量实验结果表明，MetaRuleGPT 可以模仿人类的规则遵循能力，分解复杂性，并迭代地为复杂的数学问题得出准确的结果。这些发现证明了规则学习在增强语言模型的数值推理能力方面的潜力。

Title: Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning

• Authors: Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Min Zhang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13540
• Pdf URL: https://arxiv.org/pdf/2412.13540
• Copy Paste: [[2412.13540]] Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning(https://arxiv.org/abs/2412.13540)
• Keywords: language model
• Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across diverse tasks. Despite great success, recent studies show that LVLMs encounter substantial limitations when engaging with visual graphs. To study the reason behind these limitations, we propose VGCure, a comprehensive benchmark covering 22 tasks for examining the fundamental graph understanding and reasoning capacities of LVLMs. Extensive evaluations conducted on 14 LVLMs reveal that LVLMs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. Based on this observation, we propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through 3 self-supervised learning tasks. Experiments validate the effectiveness of our method in improving LVLMs' zero-shot performance on fundamental graph learning tasks, as well as enhancing the robustness of LVLMs against complex visual graphs.
• 摘要：大型视觉语言模型 (LVLM) 已在各种任务中展现出卓越的性能。尽管取得了巨大的成功，但最近的研究表明，LVLM 在处理视觉图时遇到了很大的限制。为了研究这些限制背后的原因，我们提出了 VGCure，这是一个涵盖 22 项任务的综合基准，用于检查 LVLM 的基本图理解和推理能力。对 14 个 LVLM 进行的广泛评估表明，LVLM 在基本图理解和推理任务中表现较弱，特别是涉及关系或结构复杂信息的任务。基于这一观察，我们提出了一个结构感知微调框架，通过 3 个自监督学习任务增强具有结构学习能力的 LVLM。实验验证了我们的方法在提高 LVLM 在基本图学习任务上的零样本性能以及增强 LVLM 对复杂视觉图的鲁棒性方面的有效性。

Title: EscapeBench: Pushing Language Models to Think Outside the Box

• Authors: Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, Heng Ji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13549
• Pdf URL: https://arxiv.org/pdf/2412.13549
• Copy Paste: [[2412.13549]] EscapeBench: Pushing Language Models to Think Outside the Box(https://arxiv.org/abs/2412.13549)
• Keywords: language model, chain-of-thought, agent
• Abstract: Language model agents excel in long-session planning and reasoning, but existing benchmarks primarily focus on goal-oriented tasks with explicit objectives, neglecting creative adaptation in unfamiliar environments. To address this, we introduce EscapeBench, a benchmark suite of room escape game environments designed to challenge agents with creative reasoning, unconventional tool use, and iterative problem-solving to uncover implicit goals. Our results show that current LM models, despite employing working memory and Chain-of-Thought reasoning, achieve only 15% average progress without hints, highlighting their limitations in creativity. To bridge this gap, we propose EscapeAgent, a framework designed to enhance creative reasoning through Foresight (innovative tool use) and Reflection (identifying unsolved tasks). Experiments show that EscapeAgent can execute action chains over 1,000 steps while maintaining logical coherence. It navigates and completes games with up to 40% fewer steps and hints, performs robustly across varying difficulty levels, and achieves higher action success rates with more efficient and innovative puzzle-solving strategies. All the data and codes are released.
• 摘要：语言模型代理擅长长会话规划和推理，但现有的基准测试主要关注具有明确目标的目标导向任务，而忽略了在陌生环境中的创造性适应。为了解决这个问题，我们引入了 EscapeBench，这是一套密室逃脱游戏环境的基准测试套件，旨在挑战代理的创造性推理、非常规工具使用和迭代解决问题以发现隐含目标。我们的结果表明，尽管当前的 LM 模型采用了工作记忆和思维链推理，但在没有提示的情况下，平均进度只有 15%，这凸显了它们在创造力方面的局限性。为了弥补这一差距，我们提出了 EscapeAgent，这是一个旨在通过远见（创新工具使用）和反思（识别未解决的任务）来增强创造性推理的框架。实验表明，EscapeAgent 可以在保持逻辑连贯性的同时执行超过 1,000 个步骤的动作链。它可以以最多减少 40% 的步骤和提示来导航和完成游戏，在不同难度级别上表现稳健，并通过更高效和创新的解谜策略实现更高的动作成功率。所有数据和代码均已发布。

Title: Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancement

• Authors: Qianyue Wang, Jinwu Hu, Zhengping Li, Yufeng Wang, daiyuan li, Yu Hu, Mingkui Tan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13575
• Pdf URL: https://arxiv.org/pdf/2412.13575
• Copy Paste: [[2412.13575]] Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancement(https://arxiv.org/abs/2412.13575)
• Keywords: llm
• Abstract: Long-form story generation task aims to produce coherent and sufficiently lengthy text, essential for applications such as novel writingand interactive storytelling. However, existing methods, including LLMs, rely on rigid outlines or lack macro-level planning, making it difficult to achieve both contextual consistency and coherent plot development in long-form story generation. To address this issues, we propose Dynamic Hierarchical Outlining with Memory-Enhancement long-form story generation method, named DOME, to generate the long-form story with coherent content and plot. Specifically, the Dynamic Hierarchical Outline(DHO) mechanism incorporates the novel writing theory into outline planning and fuses the plan and writing stages together, improving the coherence of the plot by ensuring the plot completeness and adapting to the uncertainty during story generation. A Memory-Enhancement Module (MEM) based on temporal knowledge graphs is introduced to store and access the generated content, reducing contextual conflicts and improving story coherence. Finally, we propose a Temporal Conflict Analyzer leveraging temporal knowledge graphs to automatically evaluate the contextual consistency of long-form story. Experiments demonstrate that DOME significantly improves the fluency, coherence, and overall quality of generated long stories compared to state-of-the-art methods.
• 摘要：长篇故事生成任务旨在生成连贯且足够长的文本，这对于小说写作和交互式讲故事等应用至关重要。然而，包括 LLM 在内的现有方法依赖于僵化的大纲或缺乏宏观规划，这使得在长篇故事生成中难以同时实现上下文一致性和连贯的情节发展。针对这些问题，我们提出了具有记忆增强的动态分层大纲长篇故事生成方法（DOME），以生成具有连贯内容和情节的长篇故事。具体而言，动态分层大纲（DHO）机制将小说写作理论纳入大纲规划中，将计划和写作阶段融合在一起，通过确保情节完整性和适应故事生成过程中的不确定性来提高情节的连贯性。引入基于时间知识图的记忆增强模块（MEM）来存储和访问生成的内容，减少上下文冲突并提高故事连贯性。最后，我们提出了一个时间冲突分析器，利用时间知识图谱自动评估长篇故事的上下文一致性。实验表明，与最先进的方法相比，DOME 显著提高了生成的长篇故事的流畅性、连贯性和整体质量。

Title: Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation

• Authors: Shanu Kumar, Gauri Kholkar, Saish Mendke, Anubhav Sadana, Parag Agrawal, Sandipan Dandapat
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13578
• Pdf URL: https://arxiv.org/pdf/2412.13578
• Copy Paste: [[2412.13578]] Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation(https://arxiv.org/abs/2412.13578)
• Keywords: language model, llm
• Abstract: With the growth of social media and large language models, content moderation has become crucial. Many existing datasets lack adequate representation of different groups, resulting in unreliable assessments. To tackle this, we propose a socio-culturally aware evaluation framework for LLM-driven content moderation and introduce a scalable method for creating diverse datasets using persona-based generation. Our analysis reveals that these datasets provide broader perspectives and pose greater challenges for LLMs than diversity-focused generation methods without personas. This challenge is especially pronounced in smaller LLMs, emphasizing the difficulties they encounter in moderating such diverse content.
• 摘要：随着社交媒体和大型语言模型的发展，内容审核变得至关重要。许多现有数据集缺乏对不同群体的充分代表性，导致评估不可靠。为了解决这个问题，我们提出了一个具有社会文化意识的 LLM 驱动内容审核评估框架，并介绍了一种使用基于角色的生成创建多样化数据集的可扩展方法。我们的分析表明，与没有角色的以多样性为重点的生成方法相比，这些数据集为 LLM 提供了更广阔的视角，也带来了更大的挑战。这一挑战在较小的 LLM 中尤为明显，强调了它们在审核如此多样化的内容时遇到的困难。

Title: EvoWiki: Evaluating LLMs on Evolving Knowledge

• Authors: Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yugang Jiang, Yong Liao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13582
• Pdf URL: https://arxiv.org/pdf/2412.13582
• Copy Paste: [[2412.13582]] EvoWiki: Evaluating LLMs on Evolving Knowledge(https://arxiv.org/abs/2412.13582)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updatable, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Contunual Learning (CL), we evaluate how effectively LLMs adapt to evolving knowledge. Our results indicate that current models often struggle with evolved knowledge, frequently providing outdated or incorrect responses. Moreover, the dataset highlights a synergistic effect between RAG and CL, demonstrating their potential to better adapt to evolving knowledge. EvoWiki provides a robust benchmark for advancing future research on the knowledge evolution capabilities of large language models.
• 摘要：知识利用是 LLM 的一个关键方面，了解它们如何适应不断发展的知识对于有效部署它们至关重要。然而，现有的基准主要是静态的，无法捕捉 LLM 和知识不断发展的性质，导致不准确和易受污染等问题。在本文中，我们介绍了 EvoWiki，这是一个不断发展的数据集，旨在通过将信息分为稳定、发展和未知状态来反映知识的演变。EvoWiki 完全可自动更新，可以精确评估不断变化的知识和新发布的 LLM。通过对检索增强生成 (RAG) 和持续学习 (CL) 的实验，我们评估了 LLM 如何有效地适应不断发展的知识。我们的结果表明，当前的模型经常难以应对不断发展的知识，经常提供过时或不正确的响应。此外，该数据集强调了 RAG 和 CL 之间的协同效应，展示了它们更好地适应不断发展的知识的潜力。EvoWiki 为推进未来对大型语言模型知识演化能力的研究提供了一个强大的基准。

Title: Beyond Outcomes: Transparent Assessment of LLM Reasoning in Games

• Authors: Wenye Lin, Jonathan Roberts, Yunhan Yang, Samuel Albanie, Zongqing Lu, Kai Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13602
• Pdf URL: https://arxiv.org/pdf/2412.13602
• Copy Paste: [[2412.13602]] Beyond Outcomes: Transparent Assessment of LLM Reasoning in Games(https://arxiv.org/abs/2412.13602)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications that demand complex reasoning. To track progress, robust benchmarks are required to evaluate their capabilities beyond superficial pattern recognition. However, current LLM reasoning benchmarks often face challenges such as insufficient interpretability, performance saturation or data contamination. To address these challenges, we introduce GAMEBoT, a gaming arena designed for rigorous and transparent assessment of LLM reasoning capabilities. GAMEBoT decomposes complex reasoning in games into predefined modular subproblems. This decomposition allows us to design a suite of Chain-of-Thought (CoT) prompts that leverage domain knowledge to guide LLMs in addressing these subproblems before action selection. Furthermore, we develop a suite of rule-based algorithms to generate ground truth for these subproblems, enabling rigorous validation of the LLMs' intermediate reasoning steps. This approach facilitates evaluation of both the quality of final actions and the accuracy of the underlying reasoning process. GAMEBoT also naturally alleviates the risk of data contamination through dynamic games and head-to-head LLM competitions. We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics. Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts. Project page: \url{this https URL}
• 摘要：大型语言模型 (LLM) 越来越多地部署在需要复杂推理的现实世界应用中。为了跟踪进展，需要强大的基准来评估它们在表面模式识别之外的能力。然而，当前的 LLM 推理基准通常面临诸如可解释性不足、性能饱和或数据污染等挑战。为了应对这些挑战，我们推出了 GAMEBoT，这是一个专为严格透明地评估 LLM 推理能力而设计的游戏竞技场。GAMEBoT 将游戏中的复杂推理分解为预定义的模块化子问题。这种分解使我们能够设计一套思路链 (CoT) 提示，利用领域知识指导 LLM 在选择动作之前解决这些子问题。此外，我们开发了一套基于规则的算法来为这些子问题生成基本事实，从而能够严格验证 LLM 的中间推理步骤。这种方法有助于评估最终行动的质量和底层推理过程的准确性。 GAMEBoT 还通过动态游戏和面对面的 LLM 竞赛自然地减轻了数据污染的风险。我们对 8 款游戏中的 17 位杰出 LLM 进行了基准测试，涵盖了各种战略能力和游戏特征。我们的结果表明，即使为 LLM 提供了详细的 CoT 提示，GAMEBoT 也带来了巨大的挑战。项目页面：\url{此 https URL}

Title: Are LLMs Good Literature Review Writers? Evaluating the Literature Review Writing Ability of Large Language Models

• Authors: Xuemei Tang, Xufeng Duan, Zhenguang G. Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13612
• Pdf URL: https://arxiv.org/pdf/2412.13612
• Copy Paste: [[2412.13612]] Are LLMs Good Literature Review Writers? Evaluating the Literature Review Writing Ability of Large Language Models(https://arxiv.org/abs/2412.13612)
• Keywords: language model, llm, hallucination
• Abstract: The literature review is a crucial form of academic writing that involves complex processes of literature collection, organization, and summarization. The emergence of large language models (LLMs) has introduced promising tools to automate these processes. However, their actual capabilities in writing comprehensive literature reviews remain underexplored, such as whether they can generate accurate and reliable references. To address this gap, we propose a framework to assess the literature review writing ability of LLMs automatically. We evaluate the performance of LLMs across three tasks: generating references, writing abstracts, and writing literature reviews. We employ external tools for a multidimensional evaluation, which includes assessing hallucination rates in references, semantic coverage, and factual consistency with human-written context. By analyzing the experimental results, we find that, despite advancements, even the most sophisticated models still cannot avoid generating hallucinated references. Additionally, different models exhibit varying performance in literature review writing across different disciplines.
• 摘要：文献综述是学术写作的重要形式，涉及文献收集、组织和总结等复杂过程。大型语言模型 (LLM) 的出现带来了许多有前途的工具来自动化这些过程。然而，它们在撰写全面文献综述方面的实际能力仍未得到充分探索，例如它们是否可以生成准确可靠的参考文献。为了解决这一差距，我们提出了一个框架来自动评估 LLM 的文献综述写作能力。我们评估了 LLM 在三个任务中的表现：生成参考文献、撰写摘要和撰写文献综述。我们使用外部工具进行多维评估，包括评估参考文献中的幻觉率、语义覆盖率以及与人类书写上下文的事实一致性。通过分析实验结果，我们发现，尽管取得了进步，但即使是最复杂的模型仍然无法避免产生幻觉参考文献。此外，不同学科的模型在文献综述写作中表现出不同的表现。

Title: LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning

• Authors: Yansheng Mao, Jiaqi Li, Fanxu Meng, Jing Xiong, Zilong Zheng, Muhan Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13626
• Pdf URL: https://arxiv.org/pdf/2412.13626
• Copy Paste: [[2412.13626]] LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning(https://arxiv.org/abs/2412.13626)
• Keywords: language model, llm, long context
• Abstract: Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling, a novel framework that enhances LLM performance on long-context tasks by adapting model parameters to the context at test time. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation, and can improve the long-context capabilities of arbitrary short-context models. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning. The combination of in-context learning and LIFT enables short-context models like Llama 3 to handle arbitrarily long contexts and consistently improves their performance on popular long-context benchmarks like LooGLE and LongBench. We also provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research.
• 摘要：由于大型语言模型的上下文窗口有限，长上下文理解仍然具有挑战性。本文介绍了用于长上下文建模的长输入微调 (LIFT)，这是一种新颖的框架，通过在测试时根据上下文调整模型参数来提高 LLM 在长上下文任务上的性能。LIFT 可以高效处理长输入，而无需离线长上下文自适应的计算负担，并且可以提高任意短上下文模型的长上下文能力。通过集成上下文学习和 LIFT 前监督微调，该框架得到了进一步增强。上下文学习和 LIFT 的结合使 Llama 3 等短上下文模型能够处理任意长的上下文，并在 LooGLE 和 LongBench 等流行的长上下文基准上持续提高其性能。我们还对 LIFT 在长上下文理解方面的优势和局限性进行了全面分析，为未来的研究提供了有价值的方向。

Title: SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation

• Authors: Jialong Wu, Zhenglin Wang, Linhai Zhang, Yilong Lai, Yulan He, Deyu Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13649
• Pdf URL: https://arxiv.org/pdf/2412.13649
• Copy Paste: [[2412.13649]] SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation(https://arxiv.org/abs/2412.13649)
• Keywords: llm
• Abstract: Key-Value (KV) cache has become a bottleneck of LLMs for long-context generation. Despite the numerous efforts in this area, the optimization for the decoding phase is generally ignored. However, we believe such optimization is crucial, especially for long-output generation tasks based on the following two observations: (i) Excessive compression during the prefill phase, which requires specific full context impairs the comprehension of the reasoning task; (ii) Deviation of heavy hitters occurs in the reasoning tasks with long outputs. Therefore, SCOPE, a simple yet efficient framework that separately performs KV cache optimization during the prefill and decoding phases, is introduced. Specifically, the KV cache during the prefill phase is preserved to maintain the essential information, while a novel strategy based on sliding is proposed to select essential heavy hitters for the decoding phase. Memory usage and memory transfer are further optimized using adaptive and discontinuous strategies. Extensive experiments on LongGenBench show the effectiveness and generalization of SCOPE and its compatibility as a plug-in to other prefill-only KV compression methods.
• 摘要：键值 (KV) 缓存已成为 LLM 生成长上下文的瓶颈。尽管在这方面做出了许多努力，但解码阶段的优化通常被忽略。然而，我们认为这种优化至关重要，特别是对于长输出生成任务，基于以下两个观察：(i) 预填充阶段的过度压缩需要特定的完整上下文，这会损害推理任务的理解；(ii) 重击者偏差发生在具有长输出的推理任务中。因此，引入了一个简单而有效的框架 SCOPE，它在预填充和解码阶段分别执行 KV 缓存优化。具体而言，保留预填充阶段的 KV 缓存以维护基本信息，同时提出一种基于滑动的新策略来为解码阶段选择必要的重击者。使用自适应和不连续策略进一步优化内存使用和内存传输。在 LongGenBench 上进行的大量实验表明了 SCOPE 的有效性和泛化性，以及它作为其他仅预填充 KV 压缩方法的插件的兼容性。

Title: PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling

• Authors: Haojie Xie, Yirong Chen, Xiaofen Xing, Jingkai Lin, Xiangmin Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13660
• Pdf URL: https://arxiv.org/pdf/2412.13660
• Copy Paste: [[2412.13660]] PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling(https://arxiv.org/abs/2412.13660)
• Keywords: language model, gpt, llm
• Abstract: Currently, large language models (LLMs) have made significant progress in the field of psychological counseling. However, existing mental health LLMs overlook a critical issue where they do not consider the fact that different psychological counselors exhibit different personal styles, including linguistic style and therapy techniques, etc. As a result, these LLMs fail to satisfy the individual needs of clients who seek different counseling styles. To help bridge this gap, we propose PsyDT, a novel framework using LLMs to construct the Digital Twin of Psychological counselor with personalized counseling style. Compared to the time-consuming and costly approach of collecting a large number of real-world counseling cases to create a specific counselor's digital twin, our framework offers a faster and more cost-effective solution. To construct PsyDT, we utilize dynamic one-shot learning by using GPT-4 to capture counselor's unique counseling style, mainly focusing on linguistic style and therapy techniques. Subsequently, using existing single-turn long-text dialogues with client's questions, GPT-4 is guided to synthesize multi-turn dialogues of specific counselor. Finally, we fine-tune the LLMs on the synthetic dataset, PsyDTCorpus, to achieve the digital twin of psychological counselor with personalized counseling style. Experimental results indicate that our proposed PsyDT framework can synthesize multi-turn dialogues that closely resemble real-world counseling cases and demonstrate better performance compared to other baselines, thereby show that our framework can effectively construct the digital twin of psychological counselor with a specific counseling style.
• 摘要：目前，大型语言模型（LLM）在心理咨询领域取得了重大进展。然而，现有的心理健康LLM忽略了一个关键问题，即它们没有考虑到不同的心理咨询师表现出不同的个人风格，包括语言风格和治疗技术等。因此，这些LLM无法满足寻求不同咨询风格的客户的个性化需求。为了弥补这一差距，我们提出了PsyDT，这是一个使用LLM构建具有个性化咨询风格的心理咨询师数字孪生的新颖框架。与收集大量现实世界的咨询案例来创建特定咨询师的数字孪生的耗时且昂贵的方法相比，我们的框架提供了更快、更具成本效益的解决方案。为了构建PsyDT，我们利用GPT-4进行动态一次性学习来捕捉咨询师独特的咨询风格，主要关注语言风格和治疗技巧。随后，使用现有的带有客户问题的单轮长文本对话，引导 GPT-4 合成特定咨询师的多轮对话。最后，我们在合成数据集 PsyDTCorpus 上对 LLM 进行微调，以实现具有个性化咨询风格的心理咨询师的数字孪生。实验结果表明，我们提出的 PsyDT 框架可以合成与现实世界咨询案例非常相似的多轮对话，并且与其他基线相比表现出更好的性能，从而表明我们的框架可以有效地构建具有特定咨询风格的心理咨询师的数字孪生。

Title: Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

• Authors: Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13663
• Pdf URL: https://arxiv.org/pdf/2412.13663
• Copy Paste: [[2412.13663]] Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference(https://arxiv.org/abs/2412.13663)
• Keywords: long context
• Abstract: Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.
• 摘要：与较大的仅使用解码器的模型相比，仅使用编码器的 Transformer 模型（例如 BERT）在检索和分类任务中提供了出色的性能和尺寸权衡。尽管 BERT 是众多生产流程的主力，但自发布以来，其帕累托改进有限。在本文中，我们介绍了 ModernBERT，它将现代模型优化引入仅使用编码器的模型，并且与旧编码器相比实现了重大的帕累托改进。ModernBERT 模型在 2 万亿个具有原生 8192 序列长度的标记上进行训练，在大量评估中表现出最佳结果，这些评估涵盖了各种分类任务以及不同域（包括代码）上的单向量和多向量检索。除了强大的下游性能外，ModernBERT 还是速度和内存效率最高的编码器，专为在常见 GPU 上进行推理而设计。

Title: Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation

• Authors: Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopal, Katarina Marcincinova, Matus Mesarcik
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.13666
• Pdf URL: https://arxiv.org/pdf/2412.13666
• Copy Paste: [[2412.13666]] Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation(https://arxiv.org/abs/2412.13666)
• Keywords: language model, llm
• Abstract: The capabilities of recent large language models (LLMs) to generate high-quality content indistinguishable by humans from human-written texts rises many concerns regarding their misuse. Previous research has shown that LLMs can be effectively misused for generating disinformation news articles following predefined narratives. Their capabilities to generate personalized (in various aspects) content have also been evaluated and mostly found usable. However, a combination of personalization and disinformation abilities of LLMs has not been comprehensively studied yet. Such a dangerous combination should trigger integrated safety filters of the LLMs, if there are some. This study fills this gap by evaluation of vulnerabilities of recent open and closed LLMs, and their willingness to generate personalized disinformation news articles in English. We further explore whether the LLMs can reliably meta-evaluate the personalization quality and whether the personalization affects the generated-texts detectability. Our results demonstrate the need for stronger safety-filters and disclaimers, as those are not properly functioning in most of the evaluated LLMs. Additionally, our study revealed that the personalization actually reduces the safety-filter activations; thus effectively functioning as a jailbreak. Such behavior must be urgently addressed by LLM developers and service providers.
• 摘要：最近的大型语言模型 (LLM) 能够生成人类无法区分的高质量内容，这引发了许多关于其滥用的担忧。先前的研究表明，LLM 可能被滥用于按照预定义的叙述生成虚假信息新闻文章。它们生成个性化（各个方面）内容的能力也得到了评估，并且大多被发现可用。然而，LLM 的个性化和虚假信息能力的组合尚未得到全面研究。如果有的话，这种危险的组合应该会触发 LLM 的集成安全过滤器。本研究通过评估最近的开放和封闭 LLM 的漏洞以及它们生成个性化的英文虚假信息新闻文章的意愿来填补这一空白。我们进一步探讨 LLM 是否可以可靠地元评估个性化质量，以及个性化是否会影响生成文本的可检测性。我们的结果表明需要更强大的安全过滤器和免责声明，因为它们在大多数评估的 LLM 中都无法正常运行。此外，我们的研究表明，个性化实际上降低了安全过滤器的激活；从而有效地起到越狱的作用。LLM 开发人员和服务提供商必须紧急解决此类行为。

Title: AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

• Authors: Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, William Yang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13670
• Pdf URL: https://arxiv.org/pdf/2412.13670
• Copy Paste: [[2412.13670]] AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge(https://arxiv.org/abs/2412.13670)
• Keywords: llm
• Abstract: Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.
• 摘要：数据污染通过将测试数据引入新模型的训练集来阻碍公平的 LLM 评估。现有研究通过使用新收集的数据更新基准来解决这一挑战。然而，它们无法保证无污染评估，因为新收集的数据可能包含预先存在的知识，并且它们的基准更新依赖于大量的人力劳动。为了解决这些问题，我们在本文中提出了一种自动化的防泄漏基准测试框架 AntiLeak-Bench。我们不是简单地使用新收集的数据，而是使用 LLM 训练集中没有的明确新知识构建样本，从而确保严格无污染评估。我们进一步设计了一个完全自动化的工作流程来构建和更新我们的基准，而无需人工。这大大降低了基准维护的成本，以适应新兴的 LLM。通过大量实验，我们强调数据污染可能存在于 LLM 截止时间之前，并证明 AntiLeak-Bench 有效地克服了这一挑战。

Title: Towards Efficient and Explainable Hate Speech Detection via Model Distillation

• Authors: Paloma Piot, Javier Parapar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13698
• Pdf URL: https://arxiv.org/pdf/2412.13698
• Copy Paste: [[2412.13698]] Towards Efficient and Explainable Hate Speech Detection via Model Distillation(https://arxiv.org/abs/2412.13698)
• Keywords: language model, llm, chain-of-thought
• Abstract: Automatic detection of hate and abusive language is essential to combat its online spread. Moreover, recognising and explaining hate speech serves to educate people about its negative effects. However, most current detection models operate as black boxes, lacking interpretability and explainability. In this context, Large Language Models (LLMs) have proven effective for hate speech detection and to promote interpretability. Nevertheless, they are computationally costly to run. In this work, we propose distilling big language models by using Chain-of-Thought to extract explanations that support the hate speech classification task. Having small language models for these tasks will contribute to their use in operational settings. In this paper, we demonstrate that distilled models deliver explanations of the same quality as larger models while surpassing them in classification performance. This dual capability, classifying and explaining, advances hate speech detection making it more affordable, understandable and actionable.
• 摘要：自动检测仇恨和辱骂性语言对于打击其在线传播至关重要。此外，识别和解释仇恨言论有助于教育人们了解其负面影响。然而，大多数当前的检测模型都像黑匣子一样运行，缺乏可解释性和可解释性。在这种情况下，大型语言模型 (LLM) 已被证明对仇恨言论检测和促进可解释性有效。然而，它们的运行计算成本很高。在这项工作中，我们建议使用思想链来提炼大型语言模型，以提取支持仇恨言论分类任务的解释。为这些任务提供小型语言模型将有助于它们在操作环境中的使用。在本文中，我们证明提炼后的模型提供的解释质量与大型模型相同，同时在分类性能上超越大型模型。这种分类和解释的双重能力推动了仇恨言论检测的发展，使其更经济实惠、更易于理解和更具可操作性。

Title: Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models

• Authors: Kunat Pipatanakul, Potsawee Manakul, Natapong Nitarach, Warit Sirichotedumrong, Surapon Nonesung, Teetouch Jaknamon, Parinthapat Pengpun, Pittawat Taveekitworachai, Adisai Na-Thalang, Sittipong Sripaisarnmongkol, Krisanapong Jirayoot, Kasima Tharnpipitchai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13702
• Pdf URL: https://arxiv.org/pdf/2412.13702
• Copy Paste: [[2412.13702]] Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models(https://arxiv.org/abs/2412.13702)
• Keywords: language model
• Abstract: This paper introduces Typhoon 2, a series of text and multimodal large language models optimized for the Thai language. The series includes models for text, vision, and audio. Typhoon2-Text builds on state-of-the-art open models, such as Llama 3 and Qwen2, and we perform continual pre-training on a mixture of English and Thai data. We employ various post-training techniques to enhance Thai language performance while preserving the base models' original capabilities. We release text models across a range of sizes, from 1 to 70 billion parameters, available in both base and instruction-tuned variants. Typhoon2-Vision improves Thai document understanding while retaining general visual capabilities, such as image captioning. Typhoon2-Audio introduces an end-to-end speech-to-speech model architecture capable of processing audio, speech, and text inputs and generating both text and speech outputs simultaneously.
• 摘要：本文介绍了 Typhoon 2，这是一系列针对泰语优化的文本和多模态大型语言模型。该系列包括文本、视觉和音频模型。Typhoon2-Text 以最先进的开放模型（例如 Llama 3 和 Qwen2）为基础，我们对英语和泰语混合数据进行持续的预训练。我们采用各种后训练技术来提高泰语性能，同时保留基础模型的原始功能。我们发布了各种大小的文本模型，从 1 到 700 亿个参数，提供基础和指令调整版本。Typhoon2-Vision 提高了泰语文档理解能力，同时保留了图像字幕等一般视觉功能。Typhoon2-Audio 引入了一种端到端语音到语音模型架构，能够处理音频、语音和文本输入并同时生成文本和语音输出。

Title: Federated Learning and RAG Integration: A Scalable Approach for Medical Large Language Models

• Authors: Jincheol Jung, Hongju Jeong, Eui-Nam Huh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13720
• Pdf URL: https://arxiv.org/pdf/2412.13720
• Copy Paste: [[2412.13720]] Federated Learning and RAG Integration: A Scalable Approach for Medical Large Language Models(https://arxiv.org/abs/2412.13720)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: This study analyzes the performance of domain-specific Large Language Models (LLMs) for the medical field by integrating Retrieval-Augmented Generation (RAG) systems within a federated learning framework. Leveraging the inherent advantages of federated learning, such as preserving data privacy and enabling distributed computation, this research explores the integration of RAG systems with models trained under varying client configurations to optimize performance. Experimental results demonstrate that the federated learning-based models integrated with RAG systems consistently outperform their non-integrated counterparts across all evaluation metrics. This study highlights the potential of combining federated learning and RAG systems for developing domain-specific LLMs in the medical field, providing a scalable and privacy-preserving solution for enhancing text generation capabilities.
• 摘要：本研究通过将检索增强生成 (RAG) 系统集成到联邦学习框架中，分析了医学领域特定领域大型语言模型 (LLM) 的性能。利用联邦学习的固有优势，例如保护数据隐私和实现分布式计算，本研究探索了将 RAG 系统与在不同客户端配置下训练的模型集成以优化性能。实验结果表明，与 RAG 系统集成的联邦学习模型在所有评估指标上的表现始终优于未集成的模型。本研究强调了将联邦学习和 RAG 系统结合起来开发医学领域特定领域 LLM 的潜力，为增强文本生成能力提供了一种可扩展且保护隐私的解决方案。

Title: RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

• Authors: Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.13746
• Pdf URL: https://arxiv.org/pdf/2412.13746
• Copy Paste: [[2412.13746]] RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment(https://arxiv.org/abs/2412.13746)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned this http URL release our benchmark and code publicly at this https URL for future work.
• 摘要：尽管现有的检索增强语言模型 (RALM) 在提供可信响应和可靠来源方面取得了重大进展，但它们往往忽视了与人类偏好的有效对齐。在对齐过程中，奖励模型 (RM) 充当人类价值观的重要代理，以指导优化。然而，如何评估和选择可靠的 RM 进行 RALM 中的偏好对齐仍不清楚。为此，我们提出了 RAG-RewardBench，这是第一个在 RAG 设置中评估 RM 的基准。首先，我们设计了四个关键且具有挑战性的 RAG 特定场景来评估 RM，包括多跳推理、细粒度引用、适当弃权和冲突鲁棒性。然后，我们结合了 18 个 RAG 子集、6 个检索器和 24 个 RALM 来增加数据源的多样性。最后，我们采用 LLM-as-a-judge 方法来提高偏好注释的效率和有效性，与人工注释表现出很强的相关性。基于 RAG-RewardBench，我们对 45 个 RM 进行了全面评估，并揭示了它们在 RAG 场景中的局限性。此外，我们还发现，现有的经过训练的 RALM 在偏好对齐方面几乎没有任何改进，这凸显了向偏好对齐转变的必要性。此 http URL 发布我们的基准，并在此 https URL 公开代码以供将来使用。

Title: LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms

• Authors: Ali Hamdi, Ahmed Abdelmoneim Mazrou, Mohamed Shaltout
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13765
• Pdf URL: https://arxiv.org/pdf/2412.13765
• Copy Paste: [[2412.13765]] LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms(https://arxiv.org/abs/2412.13765)
• Keywords: language model, llm
• Abstract: Current methods for analyzing student engagement in e-learning platforms, including automated systems, often struggle with challenges such as handling fuzzy sentiment in text comments and relying on limited metadata. Traditional approaches, such as surveys and questionnaires, also face issues like small sample sizes and scalability. In this paper, we introduce LLM-SEM (Language Model-Based Student Engagement Metric), a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement. By utilizing recent Large Language Models (LLMs), we generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes. Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels. Extensive experiments were conducted to evaluate various LLM models, demonstrating the effectiveness of LLM-SEM in providing a scalable and accurate measure of student engagement. We fine-tuned LLMs, including AraBERT, TXLM-RoBERTa, LLama 3B and Gemma 9B from Ollama, using human-annotated sentiment datasets to enhance prediction accuracy.
• 摘要：目前，用于分析电子学习平台（包括自动化系统）中学生参与度的方法通常会面临诸如处理文本评论中的模糊情绪和依赖有限的元数据等挑战。传统方法（例如调查和问卷）也面临样本量小和可扩展性等问题。在本文中，我们介绍了 LLM-SEM（基于语言模型的学生参与度指标），这是一种利用视频元数据和学生评论的情绪分析来衡量参与度的新方法。通过利用最新的大型语言模型 (LLM)，我们可以生成高质量的情绪预测以减轻文本模糊性并规范化关键特征（例如观看次数和喜欢次数）。我们的整体方法将综合元数据与情绪极性分数相结合，以衡量课程和课堂层面的参与度。进行了大量实验来评估各种 LLM 模型，证明了 LLM-SEM 在提供可扩展且准确的学生参与度衡量标准方面的有效性。我们使用人工注释的情绪数据集对 LLM 进行了微调，包括 AraBERT、TXLM-RoBERTa、LLama 3B 和 Ollama 的 Gemma 9B，以提高预测准确性。

Title: Meta-Reflection: A Feedback-Free Reflection Learning Framework

• Authors: Yaoke Wang, Yun Zhu, Xintong Bao, Wenqiao Zhang, Suyang Dai, Kehan Chen, Wenqiang Li, Gang Huang, Siliang Tang, Yueting Zhuang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13781
• Pdf URL: https://arxiv.org/pdf/2412.13781
• Copy Paste: [[2412.13781]] Meta-Reflection: A Feedback-Free Reflection Learning Framework(https://arxiv.org/abs/2412.13781)
• Keywords: language model, llm, hallucination, agent
• Abstract: Despite the remarkable capabilities of large language models (LLMs) in natural language understanding and reasoning, they often display undesirable behaviors, such as generating hallucinations and unfaithful reasoning. A prevalent strategy to mitigate these issues is the use of reflection, which refines responses through an iterative process. However, while promising, reflection heavily relies on high-quality external feedback and requires iterative multi-agent inference processes, thus hindering its practical application. In this paper, we propose Meta-Reflection, a novel feedback-free reflection mechanism that necessitates only a single inference pass without external feedback. Motivated by the human ability to remember and retrieve reflections from past experiences when encountering similar problems, Meta-Reflection integrates reflective insights into a codebook, allowing the historical insights to be stored, retrieved, and used to guide LLMs in problem-solving. To thoroughly investigate and evaluate the practicality of Meta-Reflection in real-world scenarios, we introduce an industrial e-commerce benchmark named E-commerce Customer Intent Detection (ECID). Extensive experiments conducted on both public datasets and the ECID benchmark highlight the effectiveness and efficiency of our proposed approach.
• 摘要：尽管大型语言模型 (LLM) 在自然语言理解和推理方面具有卓越的能力，但它们经常表现出不良行为，例如产生幻觉和不忠实的推理。缓解这些问题的一种普遍策略是使用反射，它通过迭代过程改进响应。然而，虽然前景光明，但反射严重依赖高质量的外部反馈，并需要迭代的多智能体推理过程，从而阻碍了它的实际应用。在本文中，我们提出了元反射，这是一种新颖的无反馈反射机制，只需要一次推理，而无需外部反馈。受人类在遇到类似问题时记住和检索过去经验中的反思的能力的启发，元反射将反射见解集成到代码本中，允许存储、检索和使用历史见解来指导 LLM 解决问题。为了彻底调查和评估元反射在现实场景中的实用性，我们引入了一个名为电子商务客户意图检测 (ECID) 的工业电子商务基准。在公共数据集和 ECID 基准上进行的大量实验凸显了我们提出的方法的有效性和效率。

Title: Knowledge Editing with Dynamic Knowledge Graphs for Multi-hop Question Answering

• Authors: Yifan Lu, Yigeng Zhou, Jing Li, Yequan Wang, Xuebo Liu, Daojing He, Fangming Liu, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13782
• Pdf URL: https://arxiv.org/pdf/2412.13782
• Copy Paste: [[2412.13782]] Knowledge Editing with Dynamic Knowledge Graphs for Multi-hop Question Answering(https://arxiv.org/abs/2412.13782)
• Keywords: language model, llm
• Abstract: Multi-hop question answering (MHQA) poses a significant challenge for large language models (LLMs) due to the extensive knowledge demands involved. Knowledge editing, which aims to precisely modify the LLMs to incorporate specific knowledge without negatively impacting other unrelated knowledge, offers a potential solution for addressing MHQA challenges with LLMs. However, current solutions struggle to effectively resolve issues of knowledge conflicts. Most parameter-preserving editing methods are hindered by inaccurate retrieval and overlook secondary editing issues, which can introduce noise into the reasoning process of LLMs. In this paper, we introduce KEDKG, a novel knowledge editing method that leverages a dynamic knowledge graph for MHQA, designed to ensure the reliability of answers. KEDKG involves two primary steps: dynamic knowledge graph construction and knowledge graph augmented generation. Initially, KEDKG autonomously constructs a dynamic knowledge graph to store revised information while resolving potential knowledge conflicts. Subsequently, it employs a fine-grained retrieval strategy coupled with an entity and relation detector to enhance the accuracy of graph retrieval for LLM generation. Experimental results on benchmarks show that KEDKG surpasses previous state-of-the-art models, delivering more accurate and reliable answers in environments with dynamic information.
• 摘要：多跳问答 (MHQA) 对大型语言模型 (LLM) 提出了重大挑战，因为其中涉及大量知识需求。知识编辑旨在精确修改 LLM 以纳入特定知识，而不会对其他不相关的知识产生负面影响，这为使用 LLM 解决 MHQA 挑战提供了一种潜在的解决方案。然而，当前的解决方案难以有效解决知识冲突问题。大多数保留参数的编辑方法受到不准确检索的阻碍，并忽略了二次编辑问题，这可能会给 LLM 的推理过程带来噪音。在本文中，我们介绍了 KEDKG，这是一种新颖的知识编辑方法，它利用动态知识图进行 MHQA，旨在确保答案的可靠性。KEDKG 涉及两个主要步骤：动态知识图构建和知识图增强生成。首先，KEDKG 自主构建动态知识图来存储修订的信息，同时解决潜在的知识冲突。随后，它采用了细粒度检索策略，并结合了实体和关系检测器，以提高 LLM 生成的图形检索准确性。基准测试的实验结果表明，KEDKG 超越了之前最先进的模型，在具有动态信息的环境中提供了更准确、更可靠的答案。

Title: Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models

• Authors: Xinyu Pang, Ruixin Hong, Zhanke Zhou, Fangrui Lv, Xinwei Yang, Zhilong Liang, Bo Han, Changshui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13791
• Pdf URL: https://arxiv.org/pdf/2412.13791
• Copy Paste: [[2412.13791]] Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models(https://arxiv.org/abs/2412.13791)
• Keywords: language model, llm
• Abstract: Physics problems constitute a significant aspect of reasoning, necessitating complicated reasoning ability and abundant physics knowledge. However, existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application. To mitigate these issues, we propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs. Specifically, the proposed framework constructs a comprehensive formula set to provide explicit physics knowledge and utilizes checklists containing detailed instructions to guide effective knowledge application. Namely, given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning. During the process, checklists are employed to enhance LLMs' self-improvement in the analysis and reasoning stages. Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.
• 摘要：物理问题是推理的一个重要方面，需要复杂的推理能力和丰富的物理知识。然而，现有的大型语言模型 (LLM) 经常由于知识不足或知识应用不正确而失败。为了缓解这些问题，我们提出了 Physics Reasoner，这是一个使用 LLM 解决物理问题的知识增强框架。具体而言，所提出的框架构建了一个全面的公式集来提供显式的物理知识，并利用包含详细说明的清单来指导有效的知识应用。也就是说，给定一个物理问题，Physics Reasoner 通过三个阶段来解决它：问题分析、公式检索和引导推理。在此过程中，使用清单来增强 LLM 在分析和推理阶段的自我改进。从经验上看，Physics Reasoner 缓解了知识不足和应用不正确的问题，在 SciBench 上取得了最佳性能，平均准确率提高了 5.8%。

Title: Enhancing Rhetorical Figure Annotation: An Ontology-Based Web Application with RAG Integration

• Authors: Ramona Kühn, Jelena Mitrović, Michael Granitzer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13799
• Pdf URL: https://arxiv.org/pdf/2412.13799
• Copy Paste: [[2412.13799]] Enhancing Rhetorical Figure Annotation: An Ontology-Based Web Application with RAG Integration(https://arxiv.org/abs/2412.13799)
• Keywords: retrieval augmented generation
• Abstract: Rhetorical figures play an important role in our communication. They are used to convey subtle, implicit meaning, or to emphasize statements. We notice them in hate speech, fake news, and propaganda. By improving the systems for computational detection of rhetorical figures, we can also improve tasks such as hate speech and fake news detection, sentiment analysis, opinion mining, or argument mining. Unfortunately, there is a lack of annotated data, as well as qualified annotators that would help us build large corpora to train machine learning models for the detection of rhetorical figures. The situation is particularly difficult in languages other than English, and for rhetorical figures other than metaphor, sarcasm, and irony. To overcome this issue, we develop a web application called "Find your Figure" that facilitates the identification and annotation of German rhetorical figures. The application is based on the German Rhetorical ontology GRhOOT which we have specially adapted for this purpose. In addition, we improve the user experience with Retrieval Augmented Generation (RAG). In this paper, we present the restructuring of the ontology, the development of the web application, and the built-in RAG pipeline. We also identify the optimal RAG settings for our application. Our approach is one of the first to practically use rhetorical ontologies in combination with RAG and shows promising results.
• 摘要：修辞手法在我们的交流中发挥着重要作用。它们用于传达微妙、隐含的含义或强调陈述。我们在仇恨言论、虚假新闻和宣传中注意到了它们。通过改进计算检测修辞手法的系统，我们还可以改进仇恨言论和虚假新闻检测、情绪分析、观点挖掘或论证挖掘等任务。不幸的是，缺乏注释数据，以及合格的注释者来帮助我们建立大型语料库来训练机器学习模型以检测修辞手法。这种情况在英语以外的语言中尤其困难，对于隐喻、讽刺和反讽以外的修辞手法也是如此。为了解决这个问题，我们开发了一个名为“Find your Figure”的 Web 应用程序，它有助于识别和注释德语修辞手法。该应用程序基于我们为此目的特别改编的德语修辞本体 GRhOOT。此外，我们还通过检索增强生成 (RAG) 改善了用户体验。在本文中，我们介绍了本体的重构、Web 应用程序的开发以及内置的 RAG 管道。我们还确定了应用程序的最佳 RAG 设置。我们的方法是第一个实际使用修辞本体与 RAG 结合的方法之一，并显示出令人鼓舞的结果。

Title: RACQUET: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs

• Authors: Alberto Testoni, Barbara Plank, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13835
• Pdf URL: https://arxiv.org/pdf/2412.13835
• Copy Paste: [[2412.13835]] RACQUET: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs(https://arxiv.org/abs/2412.13835)
• Keywords: language model, llm
• Abstract: Ambiguity resolution is key to effective communication. While humans effortlessly address ambiguity through conversational grounding strategies, the extent to which current language models can emulate these strategies remains unclear. In this work, we examine referential ambiguity in image-based question answering by introducing RACQUET, a carefully curated dataset targeting distinct aspects of ambiguity. Through a series of evaluations, we reveal significant limitations and problems of overconfidence of state-of-the-art large multimodal language models in addressing ambiguity in their responses. The overconfidence issue becomes particularly relevant for RACQUET-BIAS, a subset designed to analyze a critical yet underexplored problem: failing to address ambiguity leads to stereotypical, socially biased responses. Our results underscore the urgency of equipping models with robust strategies to deal with uncertainty without resorting to undesirable stereotypes.
• 摘要：歧义消除是有效沟通的关键。虽然人类可以通过对话基础策略轻松解决歧义问题，但当前语言模型能够模仿这些策略的程度仍不清楚。在这项工作中，我们通过引入 RACQUET（一个精心策划的数据集，针对歧义的不同方面）来研究基于图像的问答中的指称歧义。通过一系列评估，我们发现最先进的大型多模态语言模型在解决其响应中的歧义方面存在重大局限性和过度自信的问题。过度自信问题对于 RACQUET-BIAS 尤其重要，RACQUET-BIAS 是一个旨在分析关键但尚未充分探索的问题的子集：无法解决歧义问题会导致刻板的、带有社会偏见的响应。我们的结果强调了为模型配备强大策略以应对不确定性的紧迫性，而无需诉诸不良刻板印象。

Title: Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali

• Authors: Sharad Duwal, Suraj Prasai, Suresh Manandhar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13860
• Pdf URL: https://arxiv.org/pdf/2412.13860
• Copy Paste: [[2412.13860]] Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali(https://arxiv.org/abs/2412.13860)
• Keywords: language model, llm
• Abstract: Continual learning has emerged as an important research direction due to the infeasibility of retraining large language models (LLMs) from scratch in the event of new data availability. Of great interest is the domain-adaptive pre-training (DAPT) paradigm, which focuses on continually training a pre-trained language model to adapt it to a domain it was not originally trained on. In this work, we evaluate the feasibility of DAPT in a low-resource setting, namely the Nepali language. We use synthetic data to continue training Llama 3 8B to adapt it to the Nepali language in a 4-bit QLoRA setting. We evaluate the adapted model on its performance, forgetting, and knowledge acquisition. We compare the base model and the final model on their Nepali generation abilities, their performance on popular benchmarks, and run case-studies to probe their linguistic knowledge in Nepali. We see some unsurprising forgetting in the final model, but also surprisingly find that increasing the number of shots during evaluation yields better percent increases in the final model (as high as 19.29% increase) compared to the base model (4.98%), suggesting latent retention. We also explore layer-head self-attention heatmaps to establish dependency resolution abilities of the final model in Nepali.
• 摘要：由于在有新数据可用的情况下无法从头开始重新训练大型语言模型 (LLM)，因此持续学习已成为一个重要的研究方向。领域自适应预训练 (DAPT) 范式引起了极大的兴趣，它专注于不断训练预训练的语言模型，使其适应最初未训练过的领域。在这项工作中，我们评估了 DAPT 在低资源环境（即尼泊尔语）中的可行性。我们使用合成数据继续训练 Llama 3 8B，使其在 4 位 QLoRA 设置中适应尼泊尔语。我们评估了改编后的模型的性能、遗忘和知识获取。我们比较了基础模型和最终模型的尼泊尔语生成能力、它们在流行基准上的表现，并进行了案例研究以探究它们的尼泊尔语语言知识。我们在最终模型中看到了一些意料之中的遗忘，但也令人惊讶地发现，在评估期间增加镜头数量可以使最终模型的百分比增幅更高（增幅高达 19.29%），而基础模型的增幅为 4.98%，这表明存在潜在保留。我们还探索了层头自注意力热图，以建立最终模型在尼泊尔语中的依赖性解析能力。

Title: Crabs: Consuming Resrouce via Auto-generation for LLM-DoS Attack under Black-box Settings

• Authors: Yuanhe Zhang, Zhenhong Zhou, Wei Zhang, Xinyue Wang, Xiaojun Jia, Yang Liu, Sen Su
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13879
• Pdf URL: https://arxiv.org/pdf/2412.13879
• Copy Paste: [[2412.13879]] Crabs: Consuming Resrouce via Auto-generation for LLM-DoS Attack under Black-box Settings(https://arxiv.org/abs/2412.13879)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks. LLMs continue to be vulnerable to external threats, particularly Denial-of-Service (DoS) attacks. Specifically, LLM-DoS attacks aim to exhaust computational resources and block services. However, prior works tend to focus on performing white-box attacks, overlooking black-box settings. In this work, we propose an automated algorithm designed for black-box LLMs, called Auto-Generation for LLM-DoS Attack (AutoDoS). AutoDoS introduces DoS Attack Tree and optimizes the prompt node coverage to enhance effectiveness under black-box conditions. Our method can bypass existing defense with enhanced stealthiness via semantic improvement of prompt nodes. Furthermore, we reveal that implanting Length Trojan in Basic DoS Prompt aids in achieving higher attack efficacy. Experimental results show that AutoDoS amplifies service response latency by over 250 $\times \uparrow$, leading to severe resource consumption in terms of GPU utilization and memory usage. Our code is available at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 已在各种任务中表现出色。LLM 仍然容易受到外部威胁，尤其是拒绝服务 (DoS) 攻击。具体而言，LLM-DoS 攻击旨在耗尽计算资源并阻止服务。然而，先前的研究往往侧重于执行白盒攻击，而忽略了黑盒设置。在这项工作中，我们提出了一种专为黑盒 LLM 设计的自动化算法，称为 LLM-DoS 攻击自动生成 (AutoDoS)。AutoDoS 引入了 DoS 攻击树并优化了提示节点覆盖范围，以提高在黑盒条件下的有效性。我们的方法可以通过提示节点的语义改进来绕过现有防御并增强隐蔽性。此外，我们发现在基本 DoS 提示中植入长度木马有助于实现更高的攻击效力。实验结果表明，AutoDoS 将服务响应延迟放大了 250 $\times \uparrow$ 以上，导致 GPU 利用率和内存使用率方面的资源消耗严重。我们的代码可在 \url{this https URL} 获得。

Title: Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque

• Authors: Ander Corral, Ixak Sarasua, Xabier Saralegi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13922
• Pdf URL: https://arxiv.org/pdf/2412.13922
• Copy Paste: [[2412.13922]] Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque(https://arxiv.org/abs/2412.13922)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are typically optimized for resource-rich languages like English, exacerbating the gap between high-resource and underrepresented languages. This work presents a detailed analysis of strategies for developing a model capable of following instructions in a low-resource language, specifically Basque, by focusing on three key stages: pre-training, instruction tuning, and alignment with human preferences. Our findings demonstrate that continual pre-training with a high-quality Basque corpus of around 600 million words improves natural language understanding (NLU) of the foundational model by over 12 points. Moreover, instruction tuning and human preference alignment using automatically translated datasets proved highly effective, resulting in a 24-point improvement in instruction-following performance. The resulting models, Llama-eus-8B and Llama-eus-8B-instruct, establish a new state-of-the-art for Basque in the sub-10B parameter category.
• 摘要：大型语言模型 (LLM) 通常针对英语等资源丰富的语言进行优化，这加剧了资源丰富的语言与代表性不足的语言之间的差距。这项研究通过关注三个关键阶段，详细分析了开发能够遵循资源匮乏的语言（特别是巴斯克语）指令的模型的策略：预训练、指令调整和与人类偏好保持一致。我们的研究结果表明，使用约 6 亿词的高质量巴斯克语料库进行持续预训练可将基础模型的自然语言理解 (NLU) 提高 12 分以上。此外，使用自动翻译数据集进行指令调整和与人类偏好保持一致被证明非常有效，从而使指令遵循性能提高了 24 分。由此产生的模型 Llama-eus-8B 和 Llama-eus-8B-instruct 在 10B 以下参数类别中为巴斯克语建立了新的领先地位。

Title: Language verY Rare for All

• Authors: Ibrahim Merad, Amos Wolf, Ziad Mazzawi, Yannick Léo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13924
• Pdf URL: https://arxiv.org/pdf/2412.13924
• Copy Paste: [[2412.13924]] Language verY Rare for All(https://arxiv.org/abs/2412.13924)
• Keywords: llm, retrieval-augmented generation
• Abstract: In the quest to overcome language barriers, encoder-decoder models like NLLB have expanded machine translation to rare languages, with some models (e.g., NLLB 1.3B) even trainable on a single GPU. While general-purpose LLMs perform well in translation, open LLMs prove highly competitive when fine-tuned for specific tasks involving unknown corpora. We introduce LYRA (Language verY Rare for All), a novel approach that combines open LLM fine-tuning, retrieval-augmented generation (RAG), and transfer learning from related high-resource languages. This study is exclusively focused on single-GPU training to facilitate ease of adoption. Our study focuses on two-way translation between French and Monégasque, a rare language unsupported by existing translation tools due to limited corpus availability. Our results demonstrate LYRA's effectiveness, frequently surpassing and consistently matching state-of-the-art encoder-decoder models in rare language translation.
• 摘要：为了克服语言障碍，像 NLLB 这样的编码器-解码器模型已经将机器翻译扩展到罕见语言，有些模型（例如 NLLB 1.3B）甚至可以在单个 GPU 上训练。虽然通用 LLM 在翻译方面表现良好，但开放式 LLM 在针对涉及未知语料库的特定任务进行微调时表现出色。我们推出了 LYRA（对所有人来说都非常罕见的语言），这是一种结合了开放式 LLM 微调、检索增强生成 (RAG) 和相关高资源语言的迁移学习的新颖方法。本研究专注于单 GPU 训练，以方便采用。我们的研究重点是法语和摩纳哥语之间的双向翻译，摩纳哥语是一种罕见语言，由于语料库有限，现有翻译工具不支持这种语言。我们的结果证明了 LYRA 的有效性，在罕见语言翻译中经常超越并始终匹配最先进的编码器-解码器模型。

Title: A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI

• Authors: Beiduo Chen, Siyao Peng, Anna Korhonen, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13942
• Pdf URL: https://arxiv.org/pdf/2412.13942
• Copy Paste: [[2412.13942]] A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI(https://arxiv.org/abs/2412.13942)
• Keywords: language model, llm
• Abstract: Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these label-explanations with the goal to approximate human judgment distribution. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJD, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets.
• 摘要：人类标签的分歧无处不在，并且可以在人类判断分布 (HJD) 中捕获。最近的研究表明，解释为理解人类标签变化 (HLV) 提供了有价值的信息，大型语言模型 (LLM) 可以从少数人类提供的标签解释对中近似 HJD。但是，收集每个标签的解释仍然很耗时。本文研究了 LLM 是否可用于替代人类生成近似 HJD 的解释。具体来说，我们使用 LLM 作为注释器来为少数给定的人类标签生成模型解释。我们测试了获取和组合这些标签解释的方法，目标是近似人类判断分布。我们进一步将得到的人类解释与模型生成的解释进行比较，并测试自动和人类解释选择。我们的实验表明 LLM 解释对 NLI 很有前景：为了估计 HJD，生成的解释在提供人类标签时产生与人类相当的结果。重要的是，我们的结果从具有人类解释的数据集推广到i）不可用的数据集和ii）具有挑战性的分布外测试集。

Title: Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence

• Authors: Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13949
• Pdf URL: https://arxiv.org/pdf/2412.13949
• Copy Paste: [[2412.13949]] Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence(https://arxiv.org/abs/2412.13949)
• Keywords: language model, llm, hallucination
• Abstract: Large vision-language models (LVLMs) have made substantial progress in integrating large language models (LLMs) with visual inputs, enabling advanced multimodal reasoning. Despite their success, a persistent challenge is hallucination-where generated text fails to accurately reflect visual content-undermining both accuracy and reliability. Existing methods focus on alignment training or decoding refinements but primarily address symptoms at the generation stage without probing the underlying causes. In this work, we investigate the internal mechanisms driving hallucination in LVLMs, with an emphasis on the multi-head attention module. Specifically, we introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context. Based on this, our findings reveal the presence of vision-aware attention heads that are more attuned to visual information; however, the model's overreliance on its prior language patterns is closely related to hallucinations. Building on these insights, we propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. Extensive experiments demonstrate that our method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations, while maintaining high efficiency with negligible additional time overhead.
• 摘要：大型视觉语言模型 (LVLM) 在将大型语言模型 (LLM) 与视觉输入集成方面取得了重大进展，从而实现了高级多模态推理。尽管取得了成功，但幻觉（生成的文本无法准确反映视觉内容）仍是一个持续存在的挑战，这会损害准确性和可靠性。现有方法侧重于对齐训练或解码改进，但主要解决生成阶段的症状，而不探究根本原因。在这项工作中，我们研究了驱动 LVLM 中幻觉的内部机制，重点是多头注意力模块。具体来说，我们引入了视觉感知头部发散 (VHD)，这是一种量化注意力头输出对视觉环境的敏感度的指标。基于此，我们的研究结果揭示了视觉感知注意力头的存在，它们更适应视觉信息；然而，该模型对其先前语言模式的过度依赖与幻觉密切相关。基于这些见解，我们提出了视觉感知头部强化 (VHR)，这是一种无需训练的方法，通过增强视觉感知注意力头部的作用来缓解幻觉。大量实验表明，与最先进的方法相比，我们的方法在缓解幻觉方面取得了卓越的表现，同时保持了高效率，而额外的时间开销几乎可以忽略不计。

Title: Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlation

• Authors: Eleni Sgouritsa, Virginia Aglietti, Yee Whye Teh, Arnaud Doucet, Arthur Gretton, Silvia Chiappa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13952
• Pdf URL: https://arxiv.org/pdf/2412.13952
• Copy Paste: [[2412.13952]] Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlation(https://arxiv.org/abs/2412.13952)
• Keywords: language model, llm, prompt
• Abstract: The reasoning abilities of Large Language Models (LLMs) are attracting increasing attention. In this work, we focus on causal reasoning and address the task of establishing causal relationships based on correlation information, a highly challenging problem on which several LLMs have shown poor performance. We introduce a prompting strategy for this problem that breaks the original task into fixed subquestions, with each subquestion corresponding to one step of a formal causal discovery algorithm, the PC algorithm. The proposed prompting strategy, PC-SubQ, guides the LLM to follow these algorithmic steps, by sequentially prompting it with one subquestion at a time, augmenting the next subquestion's prompt with the answer to the previous one(s). We evaluate our approach on an existing causal benchmark, Corr2Cause: our experiments indicate a performance improvement across five LLMs when comparing PC-SubQ to baseline prompting strategies. Results are robust to causal query perturbations, when modifying the variable names or paraphrasing the expressions.
• 摘要：大型语言模型 (LLM) 的推理能力正受到越来越多的关注。在这项工作中，我们专注于因果推理，并解决基于相关性信息建立因果关系的任务，这是一个极具挑战性的问题，多个 LLM 都表现不佳。我们为这个问题引入了一种提示策略，将原始任务分解为固定的子问题，每个子问题对应正式因果发现算法（PC 算法）的一个步骤。提出的提示策略 PC-SubQ 引导 LLM 遵循这些算法步骤，通过一次用一个子问题按顺序提示它，用前一个子问题的答案来增强下一个子问题的提示。我们在现有的因果基准 Corr2Cause 上评估了我们的方法：我们的实验表明，当将 PC-SubQ 与基线提示策略进行比较时，五个 LLM 的性能有所提高。当修改变量名称或解释表达式时，结果对因果查询扰动具有鲁棒性。

Title: What makes a good metric? Evaluating automatic metrics for text-to-image consistency

• Authors: Candace Ross, Melissa Hall, Adriana Romero Soriano, Adina Williams
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13989
• Pdf URL: https://arxiv.org/pdf/2412.13989
• Copy Paste: [[2412.13989]] What makes a good metric? Evaluating automatic metrics for text-to-image consistency(https://arxiv.org/abs/2412.13989)
• Keywords: language model, prompt
• Abstract: Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we analyze the construct validity of four recent, commonly used methods for measuring text-to-image consistency - CLIPScore, TIFA, VPEval, and DSG - which rely on language models and/or VQA models as components. We define construct validity for text-image consistency metrics as a set of desiderata that text-image consistency metrics should have, and find that no tested metric satisfies all of them. We find that metrics lack sufficient sensitivity to language and visual properties. Next, we find that TIFA, VPEval and DSG contribute novel information above and beyond CLIPScore, but also that they correlate highly with each other. We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information. Finally, we show that all three VQA-based metrics likely rely on familiar text shortcuts (such as yes-bias in QA) that call their aptitude as quantitative evaluations of model performance into question.
• 摘要：语言模型越来越多地被纳入大型 AI 系统，用于各种目的，从快速优化到自动评估。在这项工作中，我们分析了四种最近常用的测量文本到图像一致性的方法的结构效度 - CLIPScore、TIFA、VPEval 和 DSG - 这些方法依赖语言模型和/或 VQA 模型作为组件。我们将文本-图像一致性指标的结构效度定义为文本-图像一致性指标应具备的一组要求，并发现没有一个测试指标能够满足所有这些要求。我们发现这些指标对语言和视觉属性缺乏足够的敏感性。接下来，我们发现 TIFA、VPEval 和 DSG 贡献了比 CLIPScore 更新颖的信息，但它们之间也高度相关。我们还从不同方面分析了文本-图像一致性指标，发现并非所有模型组件都是绝对必要的，这也是对视觉信息敏感性不足的症状。最后，我们表明，所有三个基于 VQA 的指标都可能依赖于熟悉的文本快捷方式（例如 QA 中的“是偏见”），这使得它们作为模型性能的定量评估的能力受到质疑。

Title: FarExStance: Explainable Stance Detection for Farsi

• Authors: Majid Zarharan, Maryam Hashemi, Malika Behroozrazegh, Sauleh Eetemadi, Mohammad Taher Pilehvar, Jennifer Foster
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14008
• Pdf URL: https://arxiv.org/pdf/2412.14008
• Copy Paste: [[2412.14008]] FarExStance: Explainable Stance Detection for Farsi(https://arxiv.org/abs/2412.14008)
• Keywords: language model, gpt, llm
• Abstract: We introduce FarExStance, a new dataset for explainable stance detection in Farsi. Each instance in this dataset contains a claim, the stance of an article or social media post towards that claim, and an extractive explanation which provides evidence for the stance label. We compare the performance of a fine-tuned multilingual RoBERTa model to several large language models in zero-shot, few-shot, and parameter-efficient fine-tuned settings on our new dataset. On stance detection, the most accurate models are the fine-tuned RoBERTa model, the LLM Aya-23-8B which has been fine-tuned using parameter-efficient fine-tuning, and few-shot Claude-3.5-Sonnet. Regarding the quality of the explanations, our automatic evaluation metrics indicate that few-shot GPT-4o generates the most coherent explanations, while our human evaluation reveals that the best Overall Explanation Score (OES) belongs to few-shot Claude-3.5-Sonnet. The fine-tuned Aya-32-8B model produced explanations most closely aligned with the reference explanations.
• 摘要：我们引入了 FarExStance，这是一个用于波斯语中可解释立场检测的新数据集。此数据集中的每个实例都包含一个主张、一篇文章或社交媒体帖子对该主张的立场，以及为立场标签提供证据的提取解释。我们在新数据集上将微调的多语言 RoBERTa 模型与零样本、小样本和参数高效微调设置中的几种大型语言模型的性能进行了比较。在立场检测方面，最准确的模型是微调的 RoBERTa 模型、使用参数高效微调进行微调的 LLM Aya-23-8B 和小样本 Claude-3.5-Sonnet。关于解释的质量，我们的自动评估指标表明小样本 GPT-4o 生成的解释最连贯，而我们的人工评估表明最佳整体解释分数 (OES) 属于小样本 Claude-3.5-Sonnet。经过微调的 Aya-32-8B 模型给出的解释与参考解释最为接近。

Title: Towards an optimised evaluation of teachers' discourse: The case of engaging messages

• Authors: Samuel Falcon, Jaime Leon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14011
• Pdf URL: https://arxiv.org/pdf/2412.14011
• Copy Paste: [[2412.14011]] Towards an optimised evaluation of teachers' discourse: The case of engaging messages(https://arxiv.org/abs/2412.14011)
• Keywords: language model
• Abstract: Evaluating teachers' skills is crucial for enhancing education quality and student outcomes. Teacher discourse, significantly influencing student performance, is a key component. However, coding this discourse can be laborious. This study addresses this issue by introducing a new methodology for optimising the assessment of teacher discourse. The research consisted of two studies, both within the framework of engaging messages used by secondary education teachers. The first study involved training two large language models on real-world examples from audio-recorded lessons over two academic years to identify and classify the engaging messages from the lessons' transcripts. This resulted in sensitivities of 84.31% and 91.11%, and specificities of 97.69% and 86.36% in identification and classification, respectively. The second study applied these models to transcripts of audio-recorded lessons from a third academic year to examine the frequency and distribution of message types by educational level and moment of the academic year. Results showed teachers predominantly use messages emphasising engagement benefits, linked to improved outcomes, while one-third highlighted non-engagement disadvantages, associated with increased anxiety. The use of engaging messages declined in Grade 12 and towards the academic year's end. These findings suggest potential interventions to optimise engaging message use, enhancing teaching quality and student outcomes.
• 摘要：评估教师的技能对于提高教育质量和学生成绩至关重要。教师话语是其中一个关键因素，对学生成绩有显著影响。然而，对这种话语进行编码可能很费力。本研究通过引入一种优化教师话语评估的新方法来解决这个问题。该研究包括两项研究，均在中学教师使用的引人入胜的信息框架内。第一项研究涉及使用来自两个学年的录音课程的真实示例训练两个大型语言模型，以识别和分类课程记录中的引人入胜的信息。结果显示，识别和分类的敏感性分别为 84.31% 和 91.11%，特异性分别为 97.69% 和 86.36%。第二项研究将这些模型应用于第三学年的录音课程记录，以检查信息类型的频率和分布（按教育水平和学年时刻划分）。结果显示，教师主要使用强调参与度优势的信息，这与提高成绩有关，而三分之一的教师强调非参与度劣势，这会增加焦虑。在 12 年级和学年末，参与度信息的使用有所下降。这些发现表明，可以采取潜在的干预措施来优化参与度信息的使用，提高教学质量和学生成绩。

Title: Hansel: Output Length Controlling Framework for Large Language Models

• Authors: Seoha Song, Junhyun Lee, Hyeonmok Ko
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14033
• Pdf URL: https://arxiv.org/pdf/2412.14033
• Copy Paste: [[2412.14033]] Hansel: Output Length Controlling Framework for Large Language Models(https://arxiv.org/abs/2412.14033)
• Keywords: language model, llm, prompt
• Abstract: Despite the great success of large language models (LLMs), efficiently controlling the length of the output sequence still remains a challenge. In this paper, we propose Hansel, an efficient framework for length control in LLMs without affecting its generation ability. Hansel utilizes periodically outputted hidden special tokens to keep track of the remaining target length of the output sequence. Together with techniques to avoid abrupt termination of the output, this seemingly simple method proved to be efficient and versatile, while not harming the coherency and fluency of the generated text. The framework can be applied to any pre-trained LLMs during the finetuning stage of the model, regardless of its original positional encoding method. We demonstrate this by finetuning four different LLMs with Hansel and show that the mean absolute error of the output sequence decreases significantly in every model and dataset compared to the prompt-based length control finetuning. Moreover, the framework showed a substantially improved ability to extrapolate to target lengths unseen during finetuning, such as long dialog responses or extremely short summaries. This indicates that the model learns the general means of length control, rather than learning to match output lengths to those seen during training.
• 摘要：尽管大型语言模型 (LLM) 取得了巨大成功，但有效控制输出序列的长度仍然是一个挑战。在本文中，我们提出了 Hansel，这是一种有效的框架，可在不影响其生成能力的情况下控制 LLM 的长度。Hansel 利用定期输出的隐藏特殊标记来跟踪输出序列的剩余目标长度。结合避免突然终止输出的技术，这种看似简单的方法被证明是高效且通用的，同时不会损害生成文本的连贯性和流畅性。该框架可以在模型的微调阶段应用于任何预训练的 LLM，无论其原始位置编码方法如何。我们通过使用 Hansel 对四个不同的 LLM 进行微调来证明这一点，并表明与基于提示的长度控制微调相比，每个模型和数据集中的输出序列的平均绝对误差都显著降低。此外，该框架显示出显著提高的推断微调期间未见的目标长度的能力，例如长对话响应或极短摘要。这表明模型学习了长度控制的一般方法，而不是学习将输出长度与训练期间看到的长度相匹配。

Title: Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation

• Authors: Vera Neplenbroek, Arianna Bisazza, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14050
• Pdf URL: https://arxiv.org/pdf/2412.14050
• Copy Paste: [[2412.14050]] Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation(https://arxiv.org/abs/2412.14050)
• Keywords: language model, llm, prompt
• Abstract: Recent generative large language models (LLMs) show remarkable performance in non-English languages, but when prompted in those languages they tend to express higher harmful social biases and toxicity levels. Prior work has shown that finetuning on specialized datasets can mitigate this behavior, and doing so in English can transfer to other languages. In this work, we investigate the impact of different finetuning methods on the model's bias and toxicity, but also on its ability to produce fluent and diverse text. Our results show that finetuning on curated non-harmful text is more effective for mitigating bias, and finetuning on direct preference optimization (DPO) datasets is more effective for mitigating toxicity. The mitigation caused by applying these methods in English also transfers to non-English languages. We find evidence that the extent to which transfer takes place can be predicted by the amount of data in a given language present in the model's pretraining data. However, this transfer of bias and toxicity mitigation often comes at the expense of decreased language generation ability in non-English languages, highlighting the importance of developing language-specific bias and toxicity mitigation methods.
• 摘要：最近的生成式大型语言模型 (LLM) 在非英语语言中表现优异，但当使用这些语言时，它们往往会表现出更高的有害社会偏见和毒性水平。之前的研究表明，对专门的数据集进行微调可以缓解这种行为，并且用英语进行微调可以转移到其他语言。在这项工作中，我们研究了不同的微调方法对模型的偏见和毒性的影响，以及其生成流畅和多样化文本的能力的影响。我们的结果表明，对精选的无害文本进行微调对于缓解偏见更有效，对直接偏好优化 (DPO) 数据集进行微调对于缓解毒性更有效。在英语中应用这些方法所带来的缓解也会转移到非英语语言。我们发现证据表明，可以通过模型预训练数据中存在的特定语言的数据量来预测迁移发生的程度。然而，这种偏见和毒性缓解的转移往往是以非英语语言生成能力的下降为代价的，凸显了开发特定语言的偏见和毒性缓解方法的重要性。

Title: Digestion Algorithm in Hierarchical Symbolic Forests: A Fast Text Normalization Algorithm and Semantic Parsing Framework for Specific Scenarios and Lightweight Deployment

• Authors: Kevin You
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.14054
• Pdf URL: https://arxiv.org/pdf/2412.14054
• Copy Paste: [[2412.14054]] Digestion Algorithm in Hierarchical Symbolic Forests: A Fast Text Normalization Algorithm and Semantic Parsing Framework for Specific Scenarios and Lightweight Deployment(https://arxiv.org/abs/2412.14054)
• Keywords: language model, llm
• Abstract: Text Normalization and Semantic Parsing have numerous applications in natural language processing, such as natural language programming, paraphrasing, data augmentation, constructing expert systems, text matching, and more. Despite the prominent achievements of deep learning in Large Language Models (LLMs), the interpretability of neural network architectures is still poor, which affects their credibility and hence limits the deployments of risk-sensitive scenarios. In certain scenario-specific domains with scarce data, rapidly obtaining a large number of supervised learning labels is challenging, and the workload of manually labeling data would be enormous. Catastrophic forgetting in neural networks further leads to low data utilization rates. In situations where swift responses are vital, the density of the model makes local deployment difficult and the response time long, which is not conducive to local applications of these fields. Inspired by the multiplication rule, a principle of combinatorial mathematics, and human thinking patterns, a multilayer framework along with its algorithm, the Digestion Algorithm in Hierarchical Symbolic Forests (DAHSF), is proposed to address these above issues, combining text normalization and semantic parsing workflows. The Chinese Scripting Language "Fire Bunny Intelligent Development Platform V2.0" is an important test and application of the technology discussed in this paper. DAHSF can run locally in scenario-specific domains on little datasets, with model size and memory usage optimized by at least two orders of magnitude, thus improving the execution speed, and possessing a promising optimization outlook.
• 摘要：文本规范化与语义分析在自然语言处理中有广泛的应用，例如自然语言编程、释义、数据增强、构建专家系统、文本匹配等。尽管深度学习在大型语言模型（LLM）方面取得了显著成就，但神经网络架构的可解释性仍然较差，影响其可信度并限制了其在风险敏感场景的部署。在某些数据稀缺的场景领域，快速获取大量监督学习标签非常困难，而人工标注数据的工作量巨大。神经网络的灾难性遗忘进一步导致数据利用率低。在需要快速响应的场合，模型的密度使得本地部署困难、响应时间长，不利于这些领域的本地应用。受组合数学乘法原理和人类思维模式的启发，针对上述问题，提出了一种多层框架及其算法——分层符号森林中的消化算法（DAHSF），结合文本规范化和语义解析工作流。中文脚本语言“火兔智能开发平台V2.0”是本文讨论技术的重要测试和应用。DAHSF可以在小数据集上在特定场景的领域本地运行，模型大小和内存使用率至少优化了两个数量级，从而提高了执行速度，具有良好的优化前景。

Title: Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models

• Authors: Ido Cohen, Daniela Gottesman, Mor Geva, Raja Giryes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14133
• Pdf URL: https://arxiv.org/pdf/2412.14133
• Copy Paste: [[2412.14133]] Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models(https://arxiv.org/abs/2412.14133)
• Keywords: language model
• Abstract: Vision-language models (VLMs) excel at extracting and reasoning about information from images. Yet, their capacity to leverage internal knowledge about specific entities remains underexplored. This work investigates the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image. Our results reveal a significant accuracy drop --averaging 19%-- when the entity is presented visually instead of textually. We hypothesize that this decline arises from limitations in how information flows from image tokens to query tokens. We use mechanistic interpretability tools to reveal that, although image tokens are preprocessed by the vision encoder, meaningful information flow from these tokens occurs only in the much deeper layers. Furthermore, critical image processing happens in the language model's middle layers, allowing few layers for consecutive reasoning, highlighting a potential inefficiency in how the model utilizes its layers for reasoning. These insights shed light on the internal mechanics of VLMs and offer pathways for enhancing their reasoning capabilities.
• 摘要：视觉语言模型 (VLM) 擅长从图像中提取和推理信息。然而，它们利用特定实体的内部知识的能力仍未得到充分探索。这项研究调查了在回答有关文本中描述的实体与图像中描绘的实体的事实问题时模型性能的差异。我们的结果表明，当实体以视觉而不是文本形式呈现时，准确率会显著下降——平均下降 19%。我们假设这种下降是由于信息从图像标记流向查询标记的方式存在限制。我们使用机械可解释性工具来揭示，尽管图像标记由视觉编码器预处理，但来自这些标记的有意义的信息流仅发生在更深的层中。此外，关键的图像处理发生在语言模型的中间层，允许很少的层进行连续推理，这突显了模型利用其层进行推理的潜在低效率。这些见解揭示了 VLM 的内部机制，并提供了增强其推理能力的途径。

Title: GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking

• Authors: Darshan Deshpande, Selvan Sunitha Ravi, Sky CH-Wang, Bartosz Mielczarek, Anand Kannappan, Rebecca Qian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.14140
• Pdf URL: https://arxiv.org/pdf/2412.14140
• Copy Paste: [[2412.14140]] GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking(https://arxiv.org/abs/2412.14140)
• Keywords: gpt, llm
• Abstract: The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed source LLMs display critical shortcomings when deployed in real world applications due to challenges of fine grained metrics and explainability, while task specific evaluation models lack cross-domain generalization. We introduce GLIDER, a powerful 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. GLIDER shows higher Pearson's correlation than GPT-4o on FLASK and greatly outperforms prior evaluation models, achieving comparable performance to LLMs 17x its size. GLIDER supports fine-grained scoring, multilingual reasoning, span highlighting and was trained on 685 domains and 183 criteria. Extensive qualitative analysis shows that GLIDER scores are highly correlated with human judgments, with 91.3% human agreement. We have open-sourced GLIDER to facilitate future research.
• 摘要：LLM-as-judge 范式越来越多地被用于自动评估模型输出。虽然 LLM 评委在受限评估任务上表现出色，但由于细粒度指标和可解释性的挑战，闭源 LLM 在部署到实际应用中时显示出严重的缺陷，而特定于任务的评估模型缺乏跨域泛化。我们引入了 GLIDER，这是一个强大的 3B 评估器 LLM，可以根据任意用户定义的标准对任何文本输入和相关上下文进行评分。GLIDER 在 FLASK 上显示出比 GPT-4o 更高的 Pearson 相关性，并且大大优于之前的评估模型，实现了与其 17 倍大小的 LLM 相当的性能。GLIDER 支持细粒度评分、多语言推理、跨度突出显示，并在 685 个领域和 183 个标准上进行了训练。广泛的定性分析表明，GLIDER 分数与人类判断高度相关，人类一致性为 91.3%。我们已经开源了 GLIDER 以促进未来的研究。

Title: TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

• Authors: Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14161
• Pdf URL: https://arxiv.org/pdf/2412.14161
• Copy Paste: [[2412.14161]] TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks(https://arxiv.org/abs/2412.14161)
• Keywords: language model, llm, agent
• Abstract: We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at helping to accelerate or even autonomously perform work-related tasks? The answer to this question has important implications for both industry looking to adopt AI into their workflows, and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper, we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that with the most competitive agent, 24% of the tasks can be completed autonomously. This paints a nuanced picture on task automation with LM agents -- in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems.
• 摘要：无论是在日常生活中还是在工作中，我们每天都会与计算机互动，许多工作方面都可以完全通过计算机和互联网来完成。同时，由于大型语言模型 (LLM) 的改进，与周围环境互动并影响周围环境变化的 AI 代理也得到了快速发展。但是，AI 代理在帮助加速甚至自主执行与工作相关的任务方面表现如何？这个问题的答案对于希望将 AI 纳入其工作流程的行业以及了解采用 AI 可能对劳动力市场产生的影响的经济政策都具有重要意义。为了衡量这些 LLM 代理在执行现实世界专业任务方面的表现进展，我们在本文中引入了 TheAgentCompany，这是一个可扩展的基准，用于评估以与数字工作者类似的方式与世界互动的 AI 代理：通过浏览网页、编写代码、运行程序和与其他同事交流。我们构建了一个自给自足的环境，其中包含内部网站和数据，以模拟小型软件公司环境，并创建了此类公司员工可能执行的各种任务。我们测试了由封闭式 API 和开放权重语言模型 (LM) 提供支持的基线代理，发现使用最具竞争力的代理，24% 的任务可以自主完成。这为使用 LM 代理进行任务自动化描绘了一幅细致入微的图景——在模拟真实工作场所的环境中，相当一部分较简单的任务可以自主解决，但更困难的长期任务仍然超出了当前系统的能力范围。