2024-11-19

Title: "On the goals of linguistic theory": Revisiting Chomskyan theories in the era of AI

• Authors: Eva Portelance, Masoud Jasbi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10533
• Pdf URL: https://arxiv.org/pdf/2411.10533
• Copy Paste: [[2411.10533]] "On the goals of linguistic theory": Revisiting Chomskyan theories in the era of AI(https://arxiv.org/abs/2411.10533)
• Keywords: language model
• Abstract: Theoretical linguistics seeks to explain what human language is, and why. Linguists and cognitive scientists have proposed different theoretical models of what language is, as well as cognitive factors that shape it, and allow humans to 'produce', 'understand', and 'acquire' natural languages. However, humans may no longer be the only ones learning to 'generate', 'parse', and 'learn' natural language: artificial intelligence (AI) models such as large language models are proving to have impressive linguistic capabilities. Many are thus questioning what role, if any, such models should play in helping theoretical linguistics reach its ultimate research goals? In this paper, we propose to answer this question, by reiterating the tenets of generative linguistics, a leading school of thought in the field, and by considering how AI models as theories of language relate to each of these important concepts. Specifically, we consider three foundational principles, finding roots in the early works of Noam Chomsky: (1) levels of theoretical adequacy; (2) procedures for linguistic theory development; (3) language learnability and Universal Grammar. In our discussions of each principle, we give special attention to two types of AI models: neural language models and neural grammar induction models. We will argue that such models, in particular neural grammar induction models, do have a role to play, but that this role is largely modulated by the stance one takes regarding each of these three guiding principles.
• 摘要：理论语言学试图解释什么是人类语言以及为什么存在。语言学家和认知科学家提出了不同的理论模型来解释语言是什么，以及塑造语言的认知因素，使人类能够“产生”、“理解”和“获得”自然语言。然而，人类可能不再是唯一学习“生成”、“解析”和“学习”自然语言的物种：人工智能 (AI) 模型（例如大型语言模型）已被证明具有令人印象深刻的语言能力。因此，许多人都在质疑，这些模型在帮助理论语言学实现其最终研究目标方面应该发挥什么作用（如果有的话）？在本文中，我们打算通过重申生成语言学（该领域的领先学派）的原则，并考虑作为语言理论的人工智能模型与这些重要概念之间的关系来回答这个问题。具体来说，我们考虑了三个基本原则，这三个原则源于诺姆乔姆斯基的早期著作：（1）理论充分性水平； (2) 语言理论发展程序；(3) 语言可学习性和通用语法。在讨论每一项原则时，我们特别关注两种类型的人工智能模型：神经语言模型和神经语法归纳模型。我们将论证此类模型（尤其是神经语法归纳模型）确实可以发挥作用，但这种作用在很大程度上受到人们对这三项指导原则所持立场的调节。

Title: Does Prompt Formatting Have Any Impact on LLM Performance?

• Authors: Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, Sadid Hasan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10541
• Pdf URL: https://arxiv.org/pdf/2411.10541
• Copy Paste: [[2411.10541]] Does Prompt Formatting Have Any Impact on LLM Performance?(https://arxiv.org/abs/2411.10541)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance.
• 摘要：在大型语言模型 (LLM) 领域，提示优化对于模型性能至关重要。尽管先前的研究已经探索了重新表述提示上下文、使用各种提示技术（如上下文学习和思路链）以及对少量样本进行排序等方面，但我们对 LLM 对提示模板的敏感性的理解仍然有限。因此，本文研究了不同提示模板对 LLM 性能的影响。我们将相同的上下文格式化为各种人类可读的模板，包括纯文本、Markdown、JSON 和 YAML，并使用 OpenAI 的 GPT 模型评估它们在自然语言推理、代码生成和翻译等任务中的影响。实验表明，GPT-3.5-turbo 的性能在代码翻译任务中根据提示模板的不同而变化高达 40\%，而 GPT-4 等大型模型对这些变化的鲁棒性更强。我们的分析强调需要重新考虑使用固定提示模板，因为不同的格式会显著影响模型性能。

Title: mlan: language-based instruction tuning improves zero-shot generalization of multimodal large language models

• Authors: Jianhong Tu, Zhuohao Ni, Nicholas Crispino, Zihao Yu, Michael Bendersky, Beliz Gunel, Ruoxi Jia, Xin Liu, Lingjuan Lyu, Dawn Song, Chenguang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10557
• Pdf URL: https://arxiv.org/pdf/2411.10557
• Copy Paste: [[2411.10557]] mlan: language-based instruction tuning improves zero-shot generalization of multimodal large language models(https://arxiv.org/abs/2411.10557)
• Keywords: language model
• Abstract: We present a novel instruction tuning recipe to improve the zero-shot task generalization of multimodal large language models. In contrast to existing instruction tuning mechanisms that heavily rely on visual instructions, our approach focuses on language-based instruction tuning, offering a distinct and more training efficient path for multimodal instruction tuning. We evaluate the performance of the proposed approach on 9 unseen datasets across both language and vision modalities. Our results show that our language-only instruction tuning is able to significantly improve the performance of two pretrained multimodal models based on Llama 2 and Vicuna on those unseen datasets. Interestingly, the language instruction following ability also helps unlock the models to follow vision instructions without explicit training. Compared to the state of the art multimodal instruction tuning approaches that are mainly based on visual instructions, our language-based method not only achieves superior performance but also significantly enhances training efficiency. For instance, the language-only instruction tuning produces competitive average performance across the evaluated datasets (with even better performance on language datasets) with significant training efficiency improvements (on average 4x), thanks to the striking reduction in the need for vision data. With a small number of visual instructions, this emerging language instruction following ability transfers well to the unseen vision datasets, outperforming the state of the art with greater training efficiency.
• 摘要：我们提出了一种新颖的指令调优方法，以改进多模态大型语言模型的零样本任务泛化。与严重依赖视觉指令的现有指令调优机制相比，我们的方法专注于基于语言的指令调优，为多模态指令调优提供了一条独特且更高效的训练路径。我们在语言和视觉模态的 9 个未见数据集上评估了所提出方法的性能。我们的结果表明，我们的纯语言指令调优能够显著提高基于 Llama 2 和 Vicuna 的两个预训练多模态模型在这些未见数据集上的性能。有趣的是，语言指令跟踪能力还有助于解锁模型以遵循视觉指令而无需明确训练。与主要基于视觉指令的最先进的多模态指令调优方法相比，我们基于语言的方法不仅实现了卓越的性能，而且还显著提高了训练效率。例如，纯语言指令调优在评估的数据集上产生了具有竞争力的平均性能（在语言数据集上的性能甚至更好），并且训练效率显著提高（平均 4 倍），这要归功于对视觉数据的需求显著减少。通过少量的视觉指令，这种新兴的语言指令跟随能力可以很好地转移到看不见的视觉数据集，以更高的训练效率超越最先进的技术。

Title: A dataset of questions on decision-theoretic reasoning in Newcomb-like problems

• Authors: Caspar Oesterheld, Emery Cooper, Miles Kodama, Linh Chi Nguyen, Ethan Perez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10588
• Pdf URL: https://arxiv.org/pdf/2411.10588
• Copy Paste: [[2411.10588]] A dataset of questions on decision-theoretic reasoning in Newcomb-like problems(https://arxiv.org/abs/2411.10588)
• Keywords: llm, prompt, agent
• Abstract: We introduce a dataset of natural-language questions in the decision theory of so-called Newcomb-like problems. Newcomb-like problems include, for instance, decision problems in which an agent interacts with a similar other agent, and thus has to reason about the fact that the other agent will likely reason in similar ways. Evaluating LLM reasoning about Newcomb-like problems is important because interactions between foundation-model-based agents will often be Newcomb-like. Some ways of reasoning about Newcomb-like problems may allow for greater cooperation between models. Our dataset contains both capabilities questions (i.e., questions with a unique, uncontroversially correct answer) and attitude questions (i.e., questions about which decision theorists would disagree). We use our dataset for an investigation of decision-theoretical capabilities and expressed attitudes and their interplay in existing models (different models by OpenAI, Anthropic, Meta, GDM, Reka, etc.), as well as models under simple prompt-based interventions. We find, among other things, that attitudes vary significantly between existing models; that high capabilities are associated with attitudes more favorable toward so-called evidential decision theory; and that attitudes are consistent across different types of questions.
• 摘要：我们在所谓的 Newcomb 类问题决策理论中引入了一个自然语言问题数据集。Newcomb 类问题包括，例如，一个代理与另一个类似的代理交互的决策问题，因此必须推理另一个代理可能会以类似的方式推理。评估关于 Newcomb 类问题的 LLM 推理非常重要，因为基于基础模型的代理之间的交互通常是 Newcomb 式的。一些关于 Newcomb 类问题的推理方式可能允许模型之间进行更大的合作。我们的数据集包含能力问题（即具有独特、无可争议的正确答案的问题）和态度问题（即决策理论家不同意的问题）。我们使用我们的数据集来调查决策理论能力和表达的态度及其在现有模型（OpenAI、Anthropic、Meta、GDM、Reka 等的不同模型）以及简单提示干预下的模型中的相互作用。除其他外，我们发现现有模型之间态度差异很大；能力强的人对所谓证据决策理论的态度更为青睐；并且这种态度在不同类型的问题上是一致的。

Title: Leveraging large language models for efficient representation learning for entity resolution

• Authors: Xiaowei Xu, Bi T. Foua, Xingqiao Wang, Vivek Gunasekaran, John R. Talburt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10629
• Pdf URL: https://arxiv.org/pdf/2411.10629
• Copy Paste: [[2411.10629]] Leveraging large language models for efficient representation learning for entity resolution(https://arxiv.org/abs/2411.10629)
• Keywords: language model
• Abstract: In this paper, the authors propose TriBERTa, a supervised entity resolution system that utilizes a pre-trained large language model and a triplet loss function to learn representations for entity matching. The system consists of two steps: first, name entity records are fed into a Sentence Bidirectional Encoder Representations from Transformers (SBERT) model to generate vector representations, which are then fine-tuned using contrastive learning based on a triplet loss function. Fine-tuned representations are used as input for entity matching tasks, and the results show that the proposed approach outperforms state-of-the-art representations, including SBERT without fine-tuning and conventional Term Frequency-Inverse Document Frequency (TF-IDF), by a margin of 3 - 19%. Additionally, the representations generated by TriBERTa demonstrated increased robustness, maintaining consistently higher performance across a range of datasets. The authors also discussed the importance of entity resolution in today's data-driven landscape and the challenges that arise when identifying and reconciling duplicate data across different sources. They also described the ER process, which involves several crucial steps, including blocking, entity matching, and clustering.
• 摘要：在本文中，作者提出了 TriBERTa，这是一个监督实体解析系统，它利用预先训练的大型语言模型和三重损失函数来学习实体匹配的表示。该系统包括两个步骤：首先，将名称实体记录输入到 Transformers 的句子双向编码器表示 (SBERT) 模型中以生成向量表示，然后使用基于三重损失函数的对比学习对其进行微调。微调后的表示用作实体匹配任务的输入，结果表明，所提出的方法比最先进的表示（包括没有微调的 SBERT 和传统的词频-逆文档频率 (TF-IDF)）高出 3 - 19%。此外，TriBERTa 生成的表示表现出更高的稳健性，在一系列数据集中始终保持更高的性能。作者还讨论了实体解析在当今数据驱动环境中的重要性，以及在识别和协调不同来源的重复数据时出现的挑战。他们还描述了 ER 过程，其中涉及几个关键步骤，包括阻塞、实体匹配和聚类。

Title: Gender Bias Mitigation for Bangla Classification Tasks

• Authors: Sajib Kumar Saha Joy, Arman Hassan Mahy, Meherin Sultana, Azizah Mamun Abha, MD Piyal Ahmmed, Yue Dong, G M Shahariar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10636
• Pdf URL: https://arxiv.org/pdf/2411.10636
• Copy Paste: [[2411.10636]] Gender Bias Mitigation for Bangla Classification Tasks(https://arxiv.org/abs/2411.10636)
• Keywords: language model
• Abstract: In this study, we investigate gender bias in Bangla pretrained language models, a largely under explored area in low-resource languages. To assess this bias, we applied gender-name swapping techniques to existing datasets, creating four manually annotated, task-specific datasets for sentiment analysis, toxicity detection, hate speech detection, and sarcasm detection. By altering names and gender-specific terms, we ensured these datasets were suitable for detecting and mitigating gender bias. We then proposed a joint loss optimization technique to mitigate gender bias across task-specific pretrained models. Our approach was evaluated against existing bias mitigation methods, with results showing that our technique not only effectively reduces bias but also maintains competitive accuracy compared to other baseline approaches. To promote further research, we have made both our implementation and datasets publicly available this https URL
• 摘要：在本研究中，我们调查了孟加拉语预训练语言模型中的性别偏见，这是资源匮乏的语言中一个尚未得到充分探索的领域。为了评估这种偏见，我们将性别名称交换技术应用于现有数据集，创建了四个手动注释的任务特定数据集，用于情绪分析、毒性检测、仇恨言论检测和讽刺检测。通过更改名称和性别特定术语，我们确保这些数据集适合检测和缓解性别偏见。然后，我们提出了一种联合损失优化技术，以缓解任务特定预训练模型中的性别偏见。我们的方法与现有的偏见缓解方法进行了评估，结果表明，与其他基线方法相比，我们的技术不仅有效地减少了偏见，而且还保持了竞争性准确性。为了促进进一步的研究，我们已通过此 https URL 公开了我们的实现和数据集

Title: SAM Decoding: Speculative Decoding via Suffix Automaton

• Authors: Yuxuan Hu, Ke Wang, Jing Zhang, Cuiping Li, Hong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10666
• Pdf URL: https://arxiv.org/pdf/2411.10666
• Copy Paste: [[2411.10666]] SAM Decoding: Speculative Decoding via Suffix Automaton(https://arxiv.org/abs/2411.10666)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing by unifying tasks into text generation, yet their large parameter sizes and autoregressive nature limit inference speed. SAM-Decoding addresses this by introducing a novel retrieval-based speculative decoding method that uses a suffix automaton for efficient and accurate draft generation. Unlike n-gram matching used by the existing method, SAM-Decoding finds the longest suffix match in generating text and text corpuss, achieving an average time complexity of $O(1)$ per generation step. SAM-Decoding constructs static and dynamic suffix automatons for the text corpus and input prompts, respectively, enabling fast and precise draft generation. Meanwhile, it is designed as an approach that can be combined with existing methods, allowing SAM-Decoding to adaptively select a draft generation strategy based on the matching length, thus increasing the inference speed of the LLM. When combined with Token Recycling, evaluations show SAM-Decoding outperforms existing model-free methods, achieving a speedup of $2.27\times$ over autoregressive decoding on Spec-Bench. When combined with EAGLE2, it reaches a speedup of $2.49\times$, surpassing all current approaches. Our code is available at this https URL.
• 摘要：大型语言模型 (LLM) 通过将任务统一到文本生成中，彻底改变了自然语言处理，但它们的大型参数大小和自回归性质限制了推理速度。SAM-Decoding 通过引入一种新颖的基于检索的推测解码方法解决了这个问题，该方法使用后缀自动机实现高效准确的草稿生成。与现有方法使用的 n-gram 匹配不同，SAM-Decoding 在生成文本和文本语料库中找到最长的后缀匹配，实现每个生成步骤的平均时间复杂度为 $O(1)$。SAM-Decoding 分别为文本语料库和输入提示构建静态和动态后缀自动机，从而实现快速而精确的草稿生成。同时，它被设计为一种可以与现有方法相结合的方法，允许 SAM-Decoding 根据匹配长度自适应地选择草稿生成策略，从而提高 LLM 的推理速度。评估表明，与 Token Recycling 结合使用时，SAM-Decoding 的表现优于现有的无模型方法，在 Spec-Bench 上比自回归解码的速度提高了 $2.27\times$。与 EAGLE2 结合使用时，速度提高了 $2.49\times$，超越了所有当前方法。我们的代码可在此 https URL 上找到。

Title: IntentGPT: Few-shot Intent Discovery with Large Language Models

• Authors: Juan A. Rodriguez, Nicholas Botzer, David Vazquez, Christopher Pal, Marco Pedersoli, Issam Laradji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10670
• Pdf URL: https://arxiv.org/pdf/2411.10670
• Copy Paste: [[2411.10670]] IntentGPT: Few-shot Intent Discovery with Large Language Models(https://arxiv.org/abs/2411.10670)
• Keywords: language model, gpt, llm, prompt
• Abstract: In today's digitally driven world, dialogue systems play a pivotal role in enhancing user interactions, from customer service to virtual assistants. In these dialogues, it is important to identify user's goals automatically to resolve their needs promptly. This has necessitated the integration of models that perform Intent Detection. However, users' intents are diverse and dynamic, making it challenging to maintain a fixed set of predefined intents. As a result, a more practical approach is to develop a model capable of identifying new intents as they emerge. We address the challenge of Intent Discovery, an area that has drawn significant attention in recent research efforts. Existing methods need to train on a substantial amount of data for correctly identifying new intents, demanding significant human effort. To overcome this, we introduce IntentGPT, a novel training-free method that effectively prompts Large Language Models (LLMs) such as GPT-4 to discover new intents with minimal labeled data. IntentGPT comprises an \textit{In-Context Prompt Generator}, which generates informative prompts for In-Context Learning, an \textit{Intent Predictor} for classifying and discovering user intents from utterances, and a \textit{Semantic Few-Shot Sampler} that selects relevant few-shot examples and a set of known intents to be injected into the prompt. Our experiments show that IntentGPT outperforms previous methods that require extensive domain-specific data and fine-tuning, in popular benchmarks, including CLINC and BANKING, among others.
• 摘要：在当今数字化驱动的世界中，从客户服务到虚拟助手，对话系统在增强用户交互方面发挥着关键作用。在这些对话中，自动识别用户的目标以迅速解决他们的需求非常重要。这需要集成执行意图检测的模型。但是，用户的意图是多种多样且动态的，因此很难维护一组固定的预定义意图。因此，更实用的方法是开发一种能够在新意图出现时识别它们的模型。我们解决了意图发现的挑战，这一领域在最近的研究中引起了广泛关注。现有方法需要对大量数据进行训练才能正确识别新意图，这需要大量的人力。为了克服这个问题，我们引入了 IntentGPT，这是一种新颖的免训练方法，可以有效地促使大型语言模型 (LLM)（如 GPT-4）以最少的标记数据发现新意图。 IntentGPT 包含一个 \textit{上下文提示生成器}，用于为上下文学习生成信息提示，一个 \textit{意图预测器}，用于对话语中的用户意图进行分类和发现，以及一个 \textit{语义少量样本采样器}，用于选择相关的少量样本示例和一组已知意图注入提示中。我们的实验表明，在包括 CLINC 和 BANKING 等在内的流行基准测试中，IntentGPT 的表现优于以前需要大量特定领域数据和微调的方法。

Title: Structured Dialogue System for Mental Health: An LLM Chatbot Leveraging the PM+ Guidelines

• Authors: Yixiang Chen, Xinyu Zhang, Jinran Wang, Xurong Xie, Nan Yan, Hui Chen, Lan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10681
• Pdf URL: https://arxiv.org/pdf/2411.10681
• Copy Paste: [[2411.10681]] Structured Dialogue System for Mental Health: An LLM Chatbot Leveraging the PM+ Guidelines(https://arxiv.org/abs/2411.10681)
• Keywords: language model, llm, chat
• Abstract: The Structured Dialogue System, referred to as SuDoSys, is an innovative Large Language Model (LLM)-based chatbot designed to provide psychological counseling. SuDoSys leverages the World Health Organization (WHO)'s Problem Management Plus (PM+) guidelines to deliver stage-aware multi-turn dialogues. Existing methods for employing an LLM in multi-turn psychological counseling typically involve direct fine-tuning using generated dialogues, often neglecting the dynamic stage shifts of counseling sessions. Unlike previous approaches, SuDoSys considers the different stages of counseling and stores essential information throughout the counseling process, ensuring coherent and directed conversations. The system employs an LLM, a stage-aware instruction generator, a response unpacker, a topic database, and a stage controller to maintain dialogue flow. In addition, we propose a novel technique that simulates counseling clients to interact with the evaluated system and evaluate its performance automatically. When assessed using both objective and subjective evaluations, SuDoSys demonstrates its effectiveness in generating logically coherent responses. The system's code and program scripts for evaluation are open-sourced.
• 摘要：结构化对话系统（简称 SuDoSys）是一种创新的基于大型语言模型 (LLM) 的聊天机器人，旨在提供心理咨询。SuDoSys 利用世界卫生组织 (WHO) 的问题管理加 (PM+) 指南提供阶段感知多轮对话。在多轮心理咨询中使用 LLM 的现有方法通常涉及使用生成的对话进行直接微调，通常忽略了咨询会话的动态阶段转变。与以前的方法不同，SuDoSys 考虑了咨询的不同阶段，并在整个咨询过程中存储了基本信息，确保对话连贯且有针对性。该系统采用 LLM、阶段感知指令生成器、响应解包器、主题数据库和阶段控制器来维持对话流。此外，我们提出了一种新技术，模拟咨询客户与评估系统交互并自动评估其性能。当使用客观和主观评估进行评估时，SuDoSys 证明了其在生成逻辑连贯的响应方面的有效性。该系统的代码和评估程序脚本都是开源的。

Title: Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English

• Authors: Omar W. Abdalla, Aditya Joshi, Rahat Masood, Salil S. Kanhere
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.10730
• Pdf URL: https://arxiv.org/pdf/2411.10730
• Copy Paste: [[2411.10730]] Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English(https://arxiv.org/abs/2411.10730)
• Keywords: language model, prompt, chat, chain-of-thought
• Abstract: Satirical news is real news combined with a humorous comment or exaggerated content, and it often mimics the format and style of real news. However, satirical news is often misunderstood as misinformation, especially by individuals from different cultural and social backgrounds. This research addresses the challenge of distinguishing satire from truthful news by leveraging multilingual satire detection methods in English and Arabic. We explore both zero-shot and chain-of-thought (CoT) prompting using two language models, Jais-chat(13B) and LLaMA-2-chat(7B). Our results show that CoT prompting offers a significant advantage for the Jais-chat model over the LLaMA-2-chat model. Specifically, Jais-chat achieved the best performance, with an F1-score of 80\% in English when using CoT prompting. These results highlight the importance of structured reasoning in CoT, which enhances contextual understanding and is vital for complex tasks like satire detection.
• 摘要：讽刺新闻是真实新闻与幽默评论或夸张内容的结合，它通常模仿真实新闻的格式和风格。然而，讽刺新闻经常被误解为错误信息，尤其是来自不同文化和社会背景的人。这项研究利用英语和阿拉伯语的多语言讽刺检测方法解决了区分讽刺和真实新闻的挑战。我们使用两种语言模型 Jais-chat(13B) 和 LLaMA-2-chat(7B) 探索零样本和思维链 (CoT) 提示。我们的结果表明，CoT 提示为 Jais-chat 模型提供了比 LLaMA-2-chat 模型的显著优势。具体来说，Jais-chat 在使用 CoT 提示时取得了最佳表现，英语的 F1 得分为 80\%。这些结果强调了结构化推理在 CoT 中的重要性，它增强了语境理解，对于讽刺检测等复杂任务至关重要。

Title: Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?

• Authors: Tiantian Feng, Anfeng Xu, Rimita Lahiri, Helen Tager-Flusberg, So Hyun Kim, Somer Bishop, Catherine Lord, Shrikanth Narayanan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10761
• Pdf URL: https://arxiv.org/pdf/2411.10761
• Copy Paste: [[2411.10761]] Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?(https://arxiv.org/abs/2411.10761)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown significant potential in understanding human communication and interaction. However, their performance in the domain of child-inclusive interactions, including in clinical settings, remains less explored. In this work, we evaluate generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD. Specifically, we explore LLMs in performing four tasks: classifying child-adult utterances, predicting engaged activities, recognizing language skills and understanding traits that are clinically relevant. Our evaluation shows that generic LLMs are highly capable of analyzing long and complex conversations in clinical observation sessions, often surpassing the performance of non-expert human evaluators. The results show their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.
• 摘要：大型语言模型 (LLM) 在理解人类交流和互动方面表现出巨大潜力。然而，它们在儿童包容性互动领域（包括临床环境）的表现仍未得到充分探索。在这项工作中，我们评估了通用 LLM 在涉及自闭症儿童的临床相关环境中分析儿童与成人二元互动的能力。具体来说，我们探索了 LLM 执行四项任务的能力：对儿童与成人的话语进行分类、预测参与活动、识别语言技能和理解临床相关的特征。我们的评估表明，通用 LLM 能够很好地分析临床观察会议中的长而复杂的对话，通常超过非专家人类评估者的表现。结果显示了它们在细分感兴趣的互动、协助语言技能评估、识别参与活动和提供临床相关评估背景的潜力。

Title: Information Anxiety in Large Language Models

• Authors: Prasoon Bajpai, Sarah Masud, Tanmoy Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10813
• Pdf URL: https://arxiv.org/pdf/2411.10813
• Copy Paste: [[2411.10813]] Information Anxiety in Large Language Models(https://arxiv.org/abs/2411.10813)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated strong performance as knowledge repositories, enabling models to understand user queries and generate accurate and context-aware responses. Extensive evaluation setups have corroborated the positive correlation between the retrieval capability of LLMs and the frequency of entities in their pretraining corpus. We take the investigation further by conducting a comprehensive analysis of the internal reasoning and retrieval mechanisms of LLMs. Our work focuses on three critical dimensions - the impact of entity popularity, the models' sensitivity to lexical variations in query formulation, and the progression of hidden state representations across LLM layers. Our preliminary findings reveal that popular questions facilitate early convergence of internal states toward the correct answer. However, as the popularity of a query increases, retrieved attributes across lexical variations become increasingly dissimilar and less accurate. Interestingly, we find that LLMs struggle to disentangle facts, grounded in distinct relations, from their parametric memory when dealing with highly popular subjects. Through a case study, we explore these latent strains within LLMs when processing highly popular queries, a phenomenon we term information anxiety. The emergence of information anxiety in LLMs underscores the adversarial injection in the form of linguistic variations and calls for a more holistic evaluation of frequently occurring entities.
• 摘要：大型语言模型 (LLM) 作为知识库表现出色，使模型能够理解用户查询并生成准确且具有上下文感知能力的响应。广泛的评估设置证实了 LLM 的检索能力与其预训练语料库中实体的频率之间的正相关性。我们通过对 LLM 的内部推理和检索机制进行全面分析，进一步进行了调查。我们的工作重点是三个关键维度 - 实体流行度的影响、模型对查询表述中的词汇变化的敏感性以及 LLM 层之间隐藏状态表示的进展。我们的初步研究结果表明，热门问题有助于内部状态向正确答案的早期收敛。然而，随着查询的流行度增加，词汇变化中检索到的属性变得越来越不同，准确性也越来越低。有趣的是，我们发现 LLM 在处理非常流行的主题时，很难将基于不同关系的事实与参数记忆区分开来。通过案例研究，我们探讨了法学硕士在处理热门查询时的潜在压力，我们将这种现象称为信息焦虑。法学硕士中信息焦虑的出现凸显了语言变异形式的对抗性注入，并要求对经常出现的实体进行更全面的评估。

Title: Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm

• Authors: Sari Masri, Huthaifa I. Ashqar, Mohammed Elhenawy
• Subjects: cs.CL, cs.CE, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.10869
• Pdf URL: https://arxiv.org/pdf/2411.10869
• Copy Paste: [[2411.10869]] Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm(https://arxiv.org/abs/2411.10869)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: This study introduces a novel approach for traffic control systems by using Large Language Models (LLMs) as traffic controllers. The study utilizes their logical reasoning, scene understanding, and decision-making capabilities to optimize throughput and provide feedback based on traffic conditions in real-time. LLMs centralize traditionally disconnected traffic control processes and can integrate traffic data from diverse sources to provide context-aware decisions. LLMs can also deliver tailored outputs using various means such as wireless signals and visuals to drivers, infrastructures, and autonomous vehicles. To evaluate LLMs ability as traffic controllers, this study proposed a four-stage methodology. The methodology includes data creation and environment initialization, prompt engineering, conflict identification, and fine-tuning. We simulated multi-lane four-leg intersection scenarios and generates detailed datasets to enable conflict detection using LLMs and Python simulation as a ground truth. We used chain-of-thought prompts to lead LLMs in understanding the context, detecting conflicts, resolving them using traffic rules, and delivering context-sensitive traffic management solutions. We evaluated the prformance GPT-mini, Gemini, and Llama as traffic controllers. Results showed that the fine-tuned GPT-mini achieved 83% accuracy and an F1-score of 0.84. GPT-mini model exhibited a promising performance in generating actionable traffic management insights, with high ROUGE-L scores across conflict identification of 0.95, decision-making of 0.91, priority assignment of 0.94, and waiting time optimization of 0.92. We demonstrated that LLMs can offer precise recommendations to drivers in real-time including yielding, slowing, or stopping based on vehicle dynamics.
• 摘要：本研究介绍了一种使用大型语言模型 (LLM) 作为交通控制器的交通控制系统的新方法。该研究利用它们的逻辑推理、场景理解和决策能力来优化吞吐量并根据实时交通状况提供反馈。LLM 集中了传统上不相连的交通控制流程，并可以整合来自不同来源的交通数据以提供情境感知决策。LLM 还可以使用各种方式（例如无线信号和视觉效果）向驾驶员、基础设施和自动驾驶汽车提供定制的输出。为了评估 LLM 作为交通控制器的能力，本研究提出了一种四阶段方法。该方法包括数据创建和环境初始化、提示工程、冲突识别和微调。我们模拟了多车道四路交叉口场景并生成详细的数据集，以便使用 LLM 和 Python 模拟作为基本事实进行冲突检测。我们使用思路链提示来引导 LLM 理解上下文、检测冲突、使用交通规则解决冲突并提供情境敏感的交通管理解决方案。我们评估了 GPT-mini、Gemini 和 Llama 作为交通控制器的性能。结果表明，经过微调的 GPT-mini 实现了 83% 的准确率和 0.84 的 F1 分数。GPT-mini 模型在生成可操作的交通管理见解方面表现出色，在冲突识别、决策、优先级分配和等待时间优化方面均获得了较高的 ROUGE-L 分数，分别为 0.95、0.91、0.94 和 0.92。我们证明了 LLM 可以根据车辆动态实时向驾驶员提供精确的建议，包括让行、减速或停车。

Title: Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis

• Authors: Jawad Ibn Ahad, Rafeed Mohammad Sultan, Abraham Kaikobad, Fuad Rahman, Mohammad Ruhul Amin, Nabeel Mohammed, Shafin Rahman
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.10878
• Pdf URL: https://arxiv.org/pdf/2411.10878
• Copy Paste: [[2411.10878]] Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis(https://arxiv.org/abs/2411.10878)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: This study investigates the automation of meta-analysis in scientific documents using large language models (LLMs). Meta-analysis is a robust statistical method that synthesizes the findings of multiple studies support articles to provide a comprehensive understanding. We know that a meta-article provides a structured analysis of several articles. However, conducting meta-analysis by hand is labor-intensive, time-consuming, and susceptible to human error, highlighting the need for automated pipelines to streamline the process. Our research introduces a novel approach that fine-tunes the LLM on extensive scientific datasets to address challenges in big data handling and structured data extraction. We automate and optimize the meta-analysis process by integrating Retrieval Augmented Generation (RAG). Tailored through prompt engineering and a new loss metric, Inverse Cosine Distance (ICD), designed for fine-tuning on large contextual datasets, LLMs efficiently generate structured meta-analysis content. Human evaluation then assesses relevance and provides information on model performance in key metrics. This research demonstrates that fine-tuned models outperform non-fine-tuned models, with fine-tuned LLMs generating 87.6% relevant meta-analysis abstracts. The relevance of the context, based on human evaluation, shows a reduction in irrelevancy from 4.56% to 1.9%. These experiments were conducted in a low-resource environment, highlighting the study's contribution to enhancing the efficiency and reliability of meta-analysis automation.
• 摘要：本研究调查了使用大型语言模型 (LLM) 对科学文档进行元分析的自动化。元分析是一种强大的统计方法，它综合了多项研究支持文章的结果，以提供全面的理解。我们知道一篇元文章提供了多篇文章的结构化分析。然而，手动进行元分析需要大量劳动力、耗时且容易出现人为错误，这凸显了对自动化流程进行简化流程的需求。我们的研究引入了一种新方法，可以在广泛的科学数据集上微调 LLM，以应对大数据处理和结构化数据提取方面的挑战。我们通过集成检索增强生成 (RAG) 来自动化和优化元分析过程。通过快速工程和新的损失指标反余弦距离 (ICD) 进行定制，LLM 可以高效地生成结构化的元分析内容，该指标专为在大型上下文数据集上进行微调而设计。然后，人工评估将评估相关性并提供有关模型在关键指标方面表现的信息。这项研究表明，微调模型的表现优于非微调模型，微调后的 LLM 生成了 87.6% 相关的元分析摘要。基于人工评估的上下文相关性显示，不相关性从 4.56% 降低到 1.9%。这些实验是在资源匮乏的环境中进行的，凸显了该研究对提高元分析自动化效率和可靠性的贡献。

Title: BanglaDialecto: An End-to-End AI-Powered Regional Speech Standardization

• Authors: Md. Nazmus Sadat Samin, Jawad Ibn Ahad, Tanjila Ahmed Medha, Fuad Rahman, Mohammad Ruhul Amin, Nabeel Mohammed, Shafin Rahman
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.10879
• Pdf URL: https://arxiv.org/pdf/2411.10879
• Copy Paste: [[2411.10879]] BanglaDialecto: An End-to-End AI-Powered Regional Speech Standardization(https://arxiv.org/abs/2411.10879)
• Keywords: language model, llm
• Abstract: This study focuses on recognizing Bangladeshi dialects and converting diverse Bengali accents into standardized formal Bengali speech. Dialects, often referred to as regional languages, are distinctive variations of a language spoken in a particular location and are identified by their phonetics, pronunciations, and lexicon. Subtle changes in pronunciation and intonation are also influenced by geographic location, educational attainment, and socioeconomic status. Dialect standardization is needed to ensure effective communication, educational consistency, access to technology, economic opportunities, and the preservation of linguistic resources while respecting cultural diversity. Being the fifth most spoken language with around 55 distinct dialects spoken by 160 million people, addressing Bangla dialects is crucial for developing inclusive communication tools. However, limited research exists due to a lack of comprehensive datasets and the challenges of handling diverse dialects. With the advancement in multilingual Large Language Models (mLLMs), emerging possibilities have been created to address the challenges of dialectal Automated Speech Recognition (ASR) and Machine Translation (MT). This study presents an end-to-end pipeline for converting dialectal Noakhali speech to standard Bangla speech. This investigation includes constructing a large-scale diverse dataset with dialectal speech signals that tailored the fine-tuning process in ASR and LLM for transcribing the dialect speech to dialect text and translating the dialect text to standard Bangla text. Our experiments demonstrated that fine-tuning the Whisper ASR model achieved a CER of 0.8% and WER of 1.5%, while the BanglaT5 model attained a BLEU score of 41.6% for dialect-to-standard text translation.
• 摘要：本研究的重点是识别孟加拉方言，并将不同的孟加拉语口音转换成标准化的正式孟加拉语。方言通常被称为区域性语言，是特定地区使用的语言的独特变体，通过语音、发音和词汇来识别。发音和语调的细微变化也受到地理位置、教育程度和社会经济地位的影响。方言标准化是确保有效沟通、教育一致性、技术获取、经济机会和语言资源保护的必要条件，同时尊重文化多样性。孟加拉语是第五大语言，有大约 55 种不同的方言，有 1.6 亿人使用，解决方言问题对于开发包容性沟通工具至关重要。然而，由于缺乏全面的数据集和处理不同方言的挑战，研究有限。随着多语言大型语言模型 (mLLM) 的进步，出现了应对方言自动语音识别 (ASR) 和机器翻译 (MT) 挑战的可能性。本研究提出了一种将诺阿卡利方言语音转换为标准孟加拉语的端到端流程。本研究包括构建一个包含方言语音信号的大规模多样化数据集，该数据集定制了 ASR 和 LLM 中的微调过程，以将方言语音转录为方言文本，并将方言文本翻译为标准孟加拉语文本。我们的实验表明，对 Whisper ASR 模型进行微调可实现 0.8% 的 CER 和 1.5% 的 WER，而 BanglaT5 模型在方言到标准文本的翻译中获得了 41.6% 的 BLEU 分数。

Title: SPICA: Retrieving Scenarios for Pluralistic In-Context Alignment

• Authors: Quan Ze Chen, K.J. Kevin Feng, Chan Young Park, Amy X. Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10912
• Pdf URL: https://arxiv.org/pdf/2411.10912
• Copy Paste: [[2411.10912]] SPICA: Retrieving Scenarios for Pluralistic In-Context Alignment(https://arxiv.org/abs/2411.10912)
• Keywords: language model, llm, prompt
• Abstract: Alignment of large language models (LLMs) to societal values should account for pluralistic values from diverse groups. One technique uses in-context learning for inference-time alignment, but only considers similarity when drawing few-shot examples, not accounting for cross-group differences in value prioritization. We propose SPICA, a framework for pluralistic alignment that accounts for group-level differences during in-context example retrieval. SPICA introduces three designs to facilitate pluralistic alignment: scenario banks, group-informed metrics, and in-context alignment prompts. From an evaluation of SPICA on an alignment task collecting inputs from four demographic groups ($n = 544$), our metrics retrieve in-context examples that more closely match observed preferences, with the best prompt configuration using multiple contrastive responses to demonstrate examples. In an end-to-end evaluation ($n = 80$), we observe that SPICA-aligned models are higher rated than a baseline similarity-only retrieval approach, with groups seeing up to a +0.16 point improvement on a 5 point scale. Additionally, gains from SPICA were more uniform, with all groups benefiting from alignment rather than only some. Finally, we find that while a group-agnostic approach can effectively align to aggregated values, it is not most suited for aligning to divergent groups.
• 摘要：大型语言模型 (LLM) 与社会价值观的对齐应考虑来自不同群体的多元价值观。一种技术使用上下文学习进行推理时间对齐，但仅在绘制少量样本示例时考虑相似性，而不考虑跨组价值优先级差异。我们提出了 SPICA，这是一个多元对齐框架，可在上下文示例检索期间考虑组级差异。SPICA 引入了三种设计来促进多元对齐：场景库、群体知情指标和上下文对齐提示。从对 SPICA 的对齐任务的评估中，收集了来自四个人口统计组（n = 544 美元）的输入，我们的指标检索出与观察到的偏好更接近的上下文示例，最佳提示配置使用多个对比响应来演示示例。在端到端评估（n = 80）中，我们观察到 SPICA 对齐模型的评分高于仅基于相似性的基线检索方法，各组在 5 分制中最多可获得 +0.16 分的提升。此外，SPICA 带来的收益更加均匀，所有组都从对齐中受益，而不是只有部分组受益。最后，我们发现，虽然与组无关的方法可以有效地与聚合值对齐，但它并不是最适合与不同组对齐。

Title: BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

• Authors: Sizhe Wang, Yongqi Tong, Hengyuan Zhang, Dawei Li, Xin Zhang, Tianlong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10914
• Pdf URL: https://arxiv.org/pdf/2411.10914
• Copy Paste: [[2411.10914]] BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment(https://arxiv.org/abs/2411.10914)
• Keywords: language model, llm, prompt
• Abstract: Reinforcement Learning with Human Feedback (RLHF) is the key to the success of large language models (LLMs) in recent years. In this work, we first introduce the concepts of knowledge breadth and knowledge depth, which measure the comprehensiveness and depth of an LLM or knowledge source respectively. We reveal that the imbalance in the number of prompts and responses can lead to a potential disparity in breadth and depth learning within alignment tuning datasets by showing that even a simple uniform method for balancing the number of instructions and responses can lead to significant improvements. Building on this, we further propose Balanced Preference Optimization (BPO), designed to dynamically augment the knowledge depth of each sample. BPO is motivated by the observation that the usefulness of knowledge varies across samples, necessitating tailored learning of knowledge depth. To achieve this, we introduce gradient-based clustering, estimating the knowledge informativeness and usefulness of each augmented sample based on the model's optimization direction. Our experimental results across various benchmarks demonstrate that BPO outperforms other baseline methods in alignment tuning while maintaining training efficiency. Furthermore, we conduct a detailed analysis of each component of BPO, providing guidelines for future research in preference data optimization.
• 摘要：带人工反馈的强化学习 (RLHF) 是近年来大型语言模型 (LLM) 成功的关键。在这项工作中，我们首先引入了知识广度和知识深度的概念，它们分别衡量 LLM 或知识源的全面性和深度。我们表明，即使是平衡指令和响应数量的简单统一方法也可以带来显着的改进，从而揭示提示和响应数量的不平衡可能导致对齐调整数据集中广度和深度学习的潜在差异。在此基础上，我们进一步提出了平衡偏好优化 (BPO)，旨在动态增强每个样本的知识深度。BPO 的动机是观察到知识的有用性因样本而异，因此需要定制知识深度的学习。为了实现这一点，我们引入了基于梯度的聚类，根据模型的优化方向估计每个增强样本的知识信息量和有用性。我们在各种基准上的实验结果表明，BPO 在对齐调整方面优于其他基线方法，同时保持了训练效率。此外，我们对BPO的每个组成部分进行了详细的分析，为未来的偏好数据优化研究提供了指导。

Title: Bias in Large Language Models: Origin, Evaluation, and Mitigation

• Authors: Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, Shuo Shuo Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10915
• Pdf URL: https://arxiv.org/pdf/2411.10915
• Copy Paste: [[2411.10915]] Bias in Large Language Models: Origin, Evaluation, and Mitigation(https://arxiv.org/abs/2411.10915)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing, but their susceptibility to biases poses significant challenges. This comprehensive review examines the landscape of bias in LLMs, from its origins to current mitigation strategies. We categorize biases as intrinsic and extrinsic, analyzing their manifestations in various NLP tasks. The review critically assesses a range of bias evaluation methods, including data-level, model-level, and output-level approaches, providing researchers with a robust toolkit for bias detection. We further explore mitigation strategies, categorizing them into pre-model, intra-model, and post-model techniques, highlighting their effectiveness and limitations. Ethical and legal implications of biased LLMs are discussed, emphasizing potential harms in real-world applications such as healthcare and criminal justice. By synthesizing current knowledge on bias in LLMs, this review contributes to the ongoing effort to develop fair and responsible AI systems. Our work serves as a comprehensive resource for researchers and practitioners working towards understanding, evaluating, and mitigating bias in LLMs, fostering the development of more equitable AI technologies.
• 摘要：大型语言模型 (LLM) 彻底改变了自然语言处理，但它们易受偏见的影响，这带来了重大挑战。这篇全面的评论考察了 LLM 中的偏见状况，从其起源到当前的缓解策略。我们将偏见分为内在偏见和外在偏见，并分析了它们在各种 NLP 任务中的表现。这篇评论批判性地评估了一系列偏见评估方法，包括数据级、模型级和输出级方法，为研究人员提供了一套强大的偏见检测工具包。我们进一步探讨了缓解策略，将它们分为模型前、模型内和模型后技术，强调了它们的有效性和局限性。我们讨论了有偏见的 LLM 的伦理和法律影响，强调了在医疗保健和刑事司法等现实应用中的潜在危害。通过综合当前关于 LLM 偏见的知识，这篇评论为持续开发公平和负责任的人工智能系统做出了贡献。我们的工作为致力于理解、评估和减轻法学硕士 (LLM) 中的偏见的研究人员和从业人员提供了全面的资源，促进了更公平的人工智能技术的发展。

Title: Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning

• Authors: Wenke Huang, Jian Liang, Zekun Shi, Didi Zhu, Guancheng Wan, He Li, Bo Du, Dacheng Tao, Mang Ye
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10928
• Pdf URL: https://arxiv.org/pdf/2411.10928
• Copy Paste: [[2411.10928]] Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning(https://arxiv.org/abs/2411.10928)
• Keywords: language model, llm
• Abstract: Multimodal Large Language Model (MLLM) have demonstrated strong generalization capabilities across diverse distributions and tasks, largely due to extensive pre-training datasets. Fine-tuning MLLM has become a common practice to improve performance on specific downstream tasks. However, during fine-tuning, MLLM often faces the risk of forgetting knowledge acquired during pre-training, which can result in a decline in generalization abilities. To balance the trade-off between generalization and specialization, we propose measuring the parameter importance for both pre-trained and fine-tuning distributions, based on frozen pre-trained weight magnitude and accumulated fine-tuning gradient values. We further apply an importance-aware weight allocation strategy, selectively updating relatively important parameters for downstream tasks. We conduct empirical evaluations on both image captioning and visual question-answering tasks using various MLLM architectures. The comprehensive experimental analysis demonstrates the effectiveness of the proposed solution, highlighting the efficiency of the crucial modules in enhancing downstream specialization performance while mitigating generalization degradation in MLLM Fine-Tuning.
• 摘要：多模态大型语言模型 (MLLM) 已在不同分布和任务中表现出强大的泛化能力，这主要归功于广泛的预训练数据集。微调 MLLM 已成为提高特定下游任务性能的常见做法。然而，在微调过程中，MLLM 经常面临忘记预训练期间获得的知识的风险，这可能导致泛化能力下降。为了平衡泛化和专业化之间的权衡，我们建议根据冻结的预训练权重大小和累积的微调梯度值来测量预训练和微调分布的参数重要性。我们进一步应用重要性感知权重分配策略，有选择地更新下游任务相对重要的参数。我们使用各种 MLLM 架构对图像字幕和视觉问答任务进行了实证评估。全面的实验分析证明了所提解决方案的有效性，突出了关键模块在增强下游专业化性能的同时减轻 MLLM 微调中的泛化退化方面的效率。

Title: Analyzing Pok\'emon and Mario Streamers' Twitch Chat with LLM-based User Embeddings

• Authors: Mika Hämäläinen, Jack Rueter, Khalid Alnajjar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10934
• Pdf URL: https://arxiv.org/pdf/2411.10934
• Copy Paste: [[2411.10934]] Analyzing Pok\'emon and Mario Streamers' Twitch Chat with LLM-based User Embeddings(https://arxiv.org/abs/2411.10934)
• Keywords: language model, llm, chat
• Abstract: We present a novel digital humanities method for representing our Twitch chatters as user embeddings created by a large language model (LLM). We cluster these embeddings automatically using affinity propagation and further narrow this clustering down through manual analysis. We analyze the chat of one stream by each Twitch streamer: SmallAnt, DougDoug and PointCrow. Our findings suggest that each streamer has their own type of chatters, however two categories emerge for all of the streamers: supportive viewers and emoji and reaction senders. Repetitive message spammers is a shared chatter category for two of the streamers.
• 摘要：我们提出了一种新颖的数字人文方法，将 Twitch 聊天者表示为由大型语言模型 (LLM) 创建的用户嵌入。我们使用亲和力传播自动对这些嵌入进行聚类，并通过手动分析进一步缩小聚类范围。我们分析了每位 Twitch 主播的一条流的聊天：SmallAnt、DougDoug 和 PointCrow。我们的研究结果表明，每位主播都有自己的聊天者类型，但所有主播都分为两类：支持性观众和表情符号和反应发送者。重复消息垃圾邮件发送者是两位主播的共同聊天者类别。

Title: Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering

• Authors: Zeping Yu, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10950
• Pdf URL: https://arxiv.org/pdf/2411.10950
• Copy Paste: [[2411.10950]] Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering(https://arxiv.org/abs/2411.10950)
• Keywords: language model, llm, hallucination
• Abstract: Understanding the mechanisms behind Large Language Models (LLMs) is crucial for designing improved models and strategies. While recent studies have yielded valuable insights into the mechanisms of textual LLMs, the mechanisms of Multi-modal Large Language Models (MLLMs) remain underexplored. In this paper, we apply mechanistic interpretability methods to analyze the visual question answering (VQA) mechanisms in the first MLLM, Llava. We compare the mechanisms between VQA and textual QA (TQA) in color answering tasks and find that: a) VQA exhibits a mechanism similar to the in-context learning mechanism observed in TQA; b) the visual features exhibit significant interpretability when projecting the visual embeddings into the embedding space; and c) Llava enhances the existing capabilities of the corresponding textual LLM Vicuna during visual instruction tuning. Based on these findings, we develop an interpretability tool to help users and researchers identify important visual locations for final predictions, aiding in the understanding of visual hallucination. Our method demonstrates faster and more effective results compared to existing interpretability approaches. Code: \url{this https URL}
• 摘要：了解大型语言模型 (LLM) 背后的机制对于设计改进的模型和策略至关重要。虽然最近的研究对文本 LLM 的机制产生了有价值的见解，但多模态大型语言模型 (MLLM) 的机制仍未得到充分探索。在本文中，我们应用机械可解释性方法来分析第一个 MLLM Llava 中的视觉问答 (VQA) 机制。我们比较了颜色回答任务中 VQA 和文本 QA (TQA) 之间的机制，发现：a) VQA 表现出与 TQA 中观察到的上下文学习机制类似的机制；b) 将视觉嵌入投影到嵌入空间时，视觉特征表现出显着的可解释性；c) Llava 在视觉指令调整期间增强了相应文本 LLM Vicuna 的现有功能。基于这些发现，我们开发了一种可解释性工具，以帮助用户和研究人员确定最终预测的重要视觉位置，从而有助于理解视觉幻觉。与现有的可解释性方法相比，我们的方法表现出更快、更有效的结果。代码：\url{this https URL}

Title: Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

• Authors: Fahim Faisal, Md Mushfiqur Rahman, Antonios Anastasopoulos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10954
• Pdf URL: https://arxiv.org/pdf/2411.10954
• Copy Paste: [[2411.10954]] Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties(https://arxiv.org/abs/2411.10954)
• Keywords: llm
• Abstract: There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators ("LLM-as-a-judge") is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive in handling both multilingual and dialectal variations. However, if we have to rank the consistency, the weakest area is LLM-human agreement, followed by dialectal consistency. Code repository: \url{this https URL}
• 摘要：关于方言差异如何影响现代 LLM 的毒性检测，目前很少有系统的研究。此外，尽管使用 LLM 作为评估者（“LLM-as-a-judge”）是一个不断发展的研究领域，但它们对方言细微差别的敏感性仍未得到充分探索，需要更加集中的关注。在本文中，我们通过对不同方言的 LLM 进行全面的毒性评估来解决这些差距。我们通过合成转换和人工辅助翻译创建了一个多方言数据集，涵盖 10 个语言集群和 60 个变体。然后，我们评估了三种 LLM 在多语言、方言和 LLM-人类一致性方面评估毒性的能力。我们的研究结果表明，LLM 在处理多语言和方言变化方面都很敏感。然而，如果我们必须对一致性进行排名，最薄弱的领域是 LLM-人类一致性，其次是方言一致性。代码存储库：\url{this https URL}

Title: BianCang: A Traditional Chinese Medicine Large Language Model

• Authors: Sibo Wei, Xueping Peng, Yi-fei Wang, Jiasheng Si, Weiyu Zhang, Wenpeng Lu, Xiaoming Wu, Yinglong Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11027
• Pdf URL: https://arxiv.org/pdf/2411.11027
• Copy Paste: [[2411.11027]] BianCang: A Traditional Chinese Medicine Large Language Model(https://arxiv.org/abs/2411.11027)
• Keywords: language model, llm
• Abstract: The rise of large language models (LLMs) has driven significant progress in medical applications, including traditional Chinese medicine (TCM). However, current medical LLMs struggle with TCM diagnosis and syndrome differentiation due to substantial differences between TCM and modern medical theory, and the scarcity of specialized, high-quality corpora. This paper addresses these challenges by proposing BianCang, a TCM-specific LLM, using a two-stage training process that first injects domain-specific knowledge and then aligns it through targeted stimulation. To enhance diagnostic and differentiation capabilities, we constructed pre-training corpora, instruction-aligned datasets based on real hospital records, and the ChP-TCM dataset derived from the Pharmacopoeia of the People's Republic of China. We compiled extensive TCM and medical corpora for continuous pre-training and supervised fine-tuning, building a comprehensive dataset to refine the model's understanding of TCM. Evaluations across 11 test sets involving 29 models and 4 tasks demonstrate the effectiveness of BianCang, offering valuable insights for future research. Code, datasets, and models are available at this https URL.
• 摘要：大型语言模型 (LLM) 的兴起推动了包括中医 (TCM) 在内的医学应用的重大进步。然而，由于中医与现代医学理论之间存在很大差异，以及专业高质量语料库的稀缺，目前的医学 LLM 在中医诊断和辨证上遇到了困难。本文通过提出 BianCang（一种特定于中医的 LLM）来解决这些挑战，它使用两阶段训练过程，首先注入特定领域的知识，然后通过有针对性的刺激对其进行对齐。为了增强诊断和区分能力，我们构建了预训练语料库、基于真实医院记录的指令对齐数据集以及源自《中华人民共和国药典》的 ChP-TCM 数据集。我们编制了大量的中医和医学语料库，用于持续的预训练和监督微调，构建了一个全面的数据集来完善模型对中医的理解。对涉及 29 个模型和 4 个任务的 11 个测试集的评估证明了 BianCang 的有效性，为未来的研究提供了宝贵的见解。代码、数据集和模型可在此 https URL 上获得。

Title: SRA-MCTS: Self-driven Reasoning Aurmentation with Monte Carlo Tree Search for Enhanced Code Generation

• Authors: Bin Xu, Yiguan Lin, Yinghao Li, YangGao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11053
• Pdf URL: https://arxiv.org/pdf/2411.11053
• Copy Paste: [[2411.11053]] SRA-MCTS: Self-driven Reasoning Aurmentation with Monte Carlo Tree Search for Enhanced Code Generation(https://arxiv.org/abs/2411.11053)
• Keywords: language model, chain-of-thought
• Abstract: Large language models demonstrate exceptional performance in simple code generation tasks but still face challenges in tackling complex problems. These challenges may stem from insufficient reasoning and problem decomposition capabilities. To address this issue, we propose a reasoning-augmented data generation process, SRA-MCTS, which guides the model to autonomously generate high-quality intermediate reasoning paths. This creates a positive feedback loop, enabling continuous improvement. Our method operates entirely through the model itself without requiring additional supervision. By synthesizing natural language reasoning paths and translating them into executable code, the approach ensures analytical accuracy and enhances the success rate in solving complex tasks. Experimental results show that, even without additional supervisory signals, our method achieves performance improvements across different model scales, demonstrating the significant potential of self-improvement in small models. Furthermore, the method remains robust when traditional Chain-of-Thought (CoT) approaches exhibit performance degradation, with notable improvements observed in diversity metrics such as pass@10. We encourage further exploration of reasoning processes within training data to enhance the ability of language models to address complex problems.
• 摘要：大型语言模型在简单的代码生成任务中表现出色，但在解决复杂问题时仍面临挑战。这些挑战可能源于推理和问题分解能力不足。为了解决这个问题，我们提出了一个推理增强数据生成过程 SRA-MCTS，它引导模型自主生成高质量的中间推理路径。这形成了一个正反馈循环，从而实现持续改进。我们的方法完全通过模型本身运行，而无需额外的监督。通过合成自然语言推理路径并将其转换为可执行代码，该方法确保了分析准确性并提高了解决复杂任务的成功率。实验结果表明，即使没有额外的监督信号，我们的方法也能在不同模型规模上实现性能改进，展示了小型模型自我改进的巨大潜力。此外，当传统的思想链 (CoT) 方法表现出性能下降时，该方法仍然保持稳健，并且在 pass@10 等多样性指标方面观察到显着改善。我们鼓励进一步探索训练数据中的推理过程，以增强语言模型解决复杂问题的能力。

Title: FastDraft: How to Train Your Draft

• Authors: Ofir Zafrir, Igor Margulis, Dorin Shteyman, Guy Boudoukh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11055
• Pdf URL: https://arxiv.org/pdf/2411.11055
• Copy Paste: [[2411.11055]] FastDraft: How to Train Your Draft(https://arxiv.org/abs/2411.11055)
• Keywords: language model, llm
• Abstract: Speculative Decoding has gained popularity as an effective technique for accelerating the auto-regressive inference process of Large Language Models (LLMs). However, Speculative Decoding entirely relies on the availability of efficient draft models, which are often lacking for many existing language models due to a stringent constraint of vocabulary incompatibility. In this work we introduce FastDraft, a novel and efficient approach for pre-training and aligning a draft model to any large language model by incorporating efficient pre-training, followed by fine-tuning over synthetic datasets generated by the target model. We demonstrate FastDraft by training two highly parameter efficient drafts for the popular Phi-3-mini and Llama-3.1-8B models. Using FastDraft, we were able to produce a draft with approximately 10 billion tokens on a single server with 8 Intel$^\circledR$ Gaudi$^\circledR$ 2 accelerators in under 24 hours. Our results show that the draft model achieves impressive results in key metrics of acceptance rate, block efficiency and up to 3x memory bound speed up when evaluated on code completion and up to 2x in summarization, text completion and instruction tasks. We validate our theoretical findings through benchmarking on the latest Intel$^\circledR$ Core$^{\tiny \text{TM}}$ Ultra, achieving a wall-clock time speedup of up to 2x, indicating a significant reduction in runtime. Due to its high quality, FastDraft unlocks large language models inference on AI-PC and other edge-devices.
• 摘要：推测解码作为一种有效的加速大型语言模型 (LLM) 自回归推理过程的技术而广受欢迎。然而，推测解码完全依赖于高效草稿模型的可用性，而由于词汇不兼容的严格限制，许多现有语言模型往往缺乏这种模型。在这项工作中，我们引入了 FastDraft，这是一种新颖而有效的方法，通过结合高效的预训练，然后对目标模型生成的合成数据集进行微调，对草稿模型进行预训练并将其与任何大型语言模型对齐。我们通过为流行的 Phi-3-mini 和 Llama-3.1-8B 模型训练两个参数高效的草稿来演示 FastDraft。使用 FastDraft，我们能够在不到 24 小时内在一台服务器上使用 8 个 Intel$^\circledR$ Gaudi$^\circledR$ 2 加速器生成一个包含约 100 亿个 token 的草稿。我们的结果表明，草案模型在代码完成评估中取得了令人印象深刻的成绩，在接受率、块效率和高达 3 倍的内存限制加速等关键指标上取得了令人印象深刻的成绩，在摘要、文本完成和指令任务中则高达 2 倍。我们通过对最新的 Intel$^\circledR$ Core$^{\tiny \text{TM}}$ Ultra 进行基准测试来验证我们的理论发现，实现了高达 2 倍的挂钟时间加速，这表明运行时间显著减少。由于其高质量，FastDraft 解锁了 AI-PC 和其他边缘设备上的大型语言模型推理。

Title: Beyond Human-Like Processing: Large Language Models Perform Equivalently on Forward and Backward Scientific Text

• Authors: Xiaoliang Luo, Michael Ramscar, Bradley C. Love
• Subjects: cs.CL, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2411.11061
• Pdf URL: https://arxiv.org/pdf/2411.11061
• Copy Paste: [[2411.11061]] Beyond Human-Like Processing: Large Language Models Perform Equivalently on Forward and Backward Scientific Text(https://arxiv.org/abs/2411.11061)
• Keywords: language model, llm
• Abstract: The impressive performance of large language models (LLMs) has led to their consideration as models of human language processing. Instead, we suggest that the success of LLMs arises from the flexibility of the transformer learning architecture. To evaluate this conjecture, we trained LLMs on scientific texts that were either in a forward or backward format. Despite backward text being inconsistent with the structure of human languages, we found that LLMs performed equally well in either format on a neuroscience benchmark, eclipsing human expert performance for both forward and backward orders. Our results are consistent with the success of transformers across diverse domains, such as weather prediction and protein design. This widespread success is attributable to LLM's ability to extract predictive patterns from any sufficiently structured input. Given their generality, we suggest caution in interpreting LLM's success in linguistic tasks as evidence for human-like mechanisms.
• 摘要：大型语言模型 (LLM) 的出色表现使其被视为人类语言处理的模型。相反，我们认为 LLM 的成功源于 Transformer 学习架构的灵活性。为了评估这一猜想，我们在正向或反向格式的科学文本上训练了 LLM。尽管反向文本与人类语言的结构不一致，但我们发现 LLM 在神经科学基准上以两种格式表现同样出色，在正向和反向顺序上都超越了人类专家的表现。我们的结果与 Transformer 在天气预报和蛋白质设计等不同领域的成功一致。这种广泛的成功归功于 LLM 能够从任何足够结构化的输入中提取预测模式。鉴于它们的普遍性，我们建议谨慎地将 LLM 在语言任务中的成功解释为类似人类机制的证据。

Title: Multilingual Large Language Models: A Systematic Survey

• Authors: Shaolin Zhu, Supryadi, Shaoyang Xu, Haoran Sun, Leiyu Pan, Menglong Cui, Jiangcun Du, Renren Jin, António Branco, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11072
• Pdf URL: https://arxiv.org/pdf/2411.11072
• Copy Paste: [[2411.11072]] Multilingual Large Language Models: A Systematic Survey(https://arxiv.org/abs/2411.11072)
• Keywords: language model, llm
• Abstract: This paper provides a comprehensive survey of the latest research on multilingual large language models (MLLMs). MLLMs not only are able to understand and generate language across linguistic boundaries, but also represent an important advancement in artificial intelligence. We first discuss the architecture and pre-training objectives of MLLMs, highlighting the key components and methodologies that contribute to their multilingual capabilities. We then discuss the construction of multilingual pre-training and alignment datasets, underscoring the importance of data quality and diversity in enhancing MLLM performance. An important focus of this survey is on the evaluation of MLLMs. We present a detailed taxonomy and roadmap covering the assessment of MLLMs' cross-lingual knowledge, reasoning, alignment with human values, safety, interpretability and specialized applications. Specifically, we extensively discuss multilingual evaluation benchmarks and datasets, and explore the use of LLMs themselves as multilingual evaluators. To enhance MLLMs from black to white boxes, we also address the interpretability of multilingual capabilities, cross-lingual transfer and language bias within these models. Finally, we provide a comprehensive review of real-world applications of MLLMs across diverse domains, including biology, medicine, computer science, mathematics and law. We showcase how these models have driven innovation and improvements in these specialized fields while also highlighting the challenges and opportunities in deploying MLLMs within diverse language communities and application this http URL listed the paper related in this survey and publicly available at this https URL .
• 摘要：本文对多语言大型语言模型 (MLLM) 的最新研究进行了全面调查。MLLM 不仅能够理解和生成跨语言界限的语言，而且代表了人工智能的重要进步。我们首先讨论 MLLM 的架构和预训练目标，重点介绍有助于其多语言能力的关键组件和方法。然后，我们讨论多语言预训练和对齐数据集的构建，强调数据质量和多样性对提高 MLLM 性能的重要性。本次调查的一个重要重点是 MLLM 的评估。我们提出了一个详细的分类法和路线图，涵盖了对 MLLM 的跨语言知识、推理、与人类价值观的一致性、安全性、可解释性和专门应用的评估。具体而言，我们广泛讨论了多语言评估基准和数据集，并探索了 LLM 本身作为多语言评估器的用途。为了将 MLLM 从黑盒增强到白盒，我们还解决了这些模型中多语言能力的可解释性、跨语言迁移和语言偏见。最后，我们全面回顾了 MLLM 在生物学、医学、计算机科学、数学和法律等不同领域的实际应用。我们展示了这些模型如何推动这些专业领域的创新和改进，同时也强调了在不同语言社区和应用中部署 MLLM 的挑战和机遇。此 http URL 列出了本调查中相关的论文，并在此 https URL 上公开发布。

Title: The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection

• Authors: Tomas Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, Timo Spinde
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11081
• Pdf URL: https://arxiv.org/pdf/2411.11081
• Copy Paste: [[2411.11081]] The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection(https://arxiv.org/abs/2411.11081)
• Keywords: language model, llm
• Abstract: High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.
• 摘要：招聘或众包带来的高昂注释成本使创建训练可靠文本分类器所需的大型高质量数据集变得复杂。最近的研究表明，使用大型语言模型 (LLM) 来自动化注释过程，在保持数据质量的同时降低这些成本。LLM 在注释下游任务（如仇恨言论检测和政治框架）方面表现出色。基于这些领域的成功，本研究调查了 LLM 是否适合注释复杂的媒体偏见检测任务，以及下游媒体偏见分类器是否可以在这些数据上进行训练。我们创建了 annolexical，这是第一个用于媒体偏见分类的大型数据集，其中包含超过 48000 个合成注释示例。我们的分类器在此数据集上进行了微调，在马修斯相关系数 (MCC) 方面比所有注释器 LLM 高出 5-9%，并且在两个媒体偏见基准数据集（BAB​​E 和 BASIL）上进行评估时，其表现接近或优于在人工标记数据上训练的模型。这项研究表明我们的方法如何显著降低媒体偏见领域的数据集创建成本，并进而降低分类器的开发成本，而我们随后的行为压力测试揭示了其当前的一些局限性和权衡。

Title: LL\"aMmlein: Compact and Competitive German-Only Language Models from Scratch

• Authors: Jan Pfister, Julia Wunderle, Andreas Hotho
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11171
• Pdf URL: https://arxiv.org/pdf/2411.11171
• Copy Paste: [[2411.11171]] LL\"aMmlein: Compact and Competitive German-Only Language Models from Scratch(https://arxiv.org/abs/2411.11171)
• Keywords: language model
• Abstract: We create two German-only decoder models, LLäMmlein 120M and 1B, transparently from scratch and publish them, along with the training data, for the German NLP research community to use. The model training involved several key steps, including extensive data preprocessing, the creation of a custom German tokenizer, the training itself, as well as the evaluation of the final models on various benchmarks. Throughout the training process, multiple checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor the models' learning dynamics. Compared to state-of-the-art models on the SuperGLEBer benchmark, both LLäMmlein models performed competitively, consistently matching or surpassing models with similar parameter sizes. The results show that the models' quality scales with size as expected, but performance improvements on some tasks plateaued early, offering valuable insights into resource allocation for future model development.
• 摘要：我们从头开始透明地创建了两个仅限德语的解码器模型 LLäMmlein 120M 和 1B，并将它们与训练数据一起发布，供德国 NLP 研究界使用。模型训练涉及几个关键步骤，包括大量数据预处理、创建自定义德语标记器、训练本身以及在各种基准上对最终模型的评估。在整个训练过程中，使用 SuperGLEBer 基准保存和分析了多个检查点，以监控模型的学习动态。与 SuperGLEBer 基准上最先进的模型相比，这两个 LLäMmlein 模型的表现都很有竞争力，始终与具有相似参数大小的模型相匹配或超越它们。结果表明，模型的质量与规模成正比，但某些任务的性能改进在早期就停滞不前，为未来模型开发的资源分配提供了宝贵的见解。

Title: Capturing Sparks of Abstraction for the ARC Challenge

• Authors: Martin Andrews
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11206
• Pdf URL: https://arxiv.org/pdf/2411.11206
• Copy Paste: [[2411.11206]] Capturing Sparks of Abstraction for the ARC Challenge(https://arxiv.org/abs/2411.11206)
• Keywords: language model, llm
• Abstract: Excellent progress has been made recently in solving ARC Challenge problems. However, it seems that new techniques may be required to push beyond 60% accuracy. Even commercial Large Language Models (LLMs) struggle to 'understand' many of the problems (when given the input and output grids), which makes discovering solutions by LLM-lead program search somewhat futile. In this work, LLM 'understanding' is attempted from a stronger starting position : An LLM is given complete solutions to tasks in code, and then asked to explain how the task is being solved at various levels of abstraction. Specifically, the LLM was given code solutions implemented in arc-dsl-llm (an LLM-legible version of Hodel's arc-dsl to obtain: (a) commented code; (b) code refactored into reusable functional chunks; (c) problem solution steps; and (d) high-level problem-solving tactics. We demonstrate that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize. Both the arc-dsl-llm DSL framework (with the re-engineered solutions) and the Gemini LLM-generated data (along with the generation code) are made Open Source.
• 摘要：最近，在解决 ARC 挑战问题方面取得了巨大进展。然而，似乎需要新技术才能将准确率提高到 60% 以上。即使是商用大型语言模型 (LLM) 也难以“理解”许多问题（当给定输入和输出网格时），这使得通过 LLM 主导的程序搜索发现解决方案有些徒劳。在这项工作中，LLM“理解”尝试从一个更强大的起点开始：向 LLM 提供代码中任务的完整解决方案，然后要求其解释如何在各个抽象级别上解决该任务。具体来说，LLM 获得了在 arc-dsl-llm（Hodel 的 arc-dsl 的 LLM 可读版本）中实现的代码解决方案，以获得：（a）注释代码；（b）重构为可重用功能块的代码；（c）问题解决步骤；（d）高级问题解决策略。我们证明可以从 LLM 输出中提取“抽象火花” - 以可以用于下游任务的形式，本地 LLM 有资格参加 ARC 奖。arc-dsl-llm DSL 框架（带有重新设计的解决方案）和 Gemini LLM 生成的数据（以及生成代码）都是开源的。

Title: MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis

• Authors: Yingjie Zhou, Zicheng Zhang, Jiezhang Cao, Jun Jia, Yanwei Jiang, Farong Wen, Xiaohong Liu, Xiongkuo Min, Guangtao Zhai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11235
• Pdf URL: https://arxiv.org/pdf/2411.11235
• Copy Paste: [[2411.11235]] MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis(https://arxiv.org/abs/2411.11235)
• Keywords: language model, llm
• Abstract: Artificial Intelligence (AI) has demonstrated significant capabilities in various fields, and in areas such as human-computer interaction (HCI), embodied intelligence, and the design and animation of virtual digital humans, both practitioners and users are increasingly concerned with AI's ability to understand and express emotion. Consequently, the question of whether AI can accurately interpret human emotions remains a critical challenge. To date, two primary classes of AI models have been involved in human emotion analysis: generative models and Multimodal Large Language Models (MLLMs). To assess the emotional capabilities of these two classes of models, this study introduces MEMO-Bench, a comprehensive benchmark consisting of 7,145 portraits, each depicting one of six different emotions, generated by 12 Text-to-Image (T2I) models. Unlike previous works, MEMO-Bench provides a framework for evaluating both T2I models and MLLMs in the context of sentiment analysis. Additionally, a progressive evaluation approach is employed, moving from coarse-grained to fine-grained metrics, to offer a more detailed and comprehensive assessment of the sentiment analysis capabilities of MLLMs. The experimental results demonstrate that existing T2I models are more effective at generating positive emotions than negative ones. Meanwhile, although MLLMs show a certain degree of effectiveness in distinguishing and recognizing human emotions, they fall short of human-level accuracy, particularly in fine-grained emotion analysis. The MEMO-Bench will be made publicly available to support further research in this area.
• 摘要：人工智能 (AI) 已在各个领域展现出强大的能力，在人机交互 (HCI)、具身智能以及虚拟数字人的设计和动画等领域，从业者和用户都越来越关注 AI 理解和表达情感的能力。因此，AI 是否能够准确解读人类情感仍然是一个关键挑战。迄今为止，人类情感分析涉及两类主要 AI 模型：生成模型和多模态大型语言模型 (MLLM)。为了评估这两类模型的情感能力，本研究引入了 MEMO-Bench，这是一个综合基准，由 12 个文本转图像 (T2I) 模型生成的 7,145 幅肖像组成，每幅肖像描绘了六种不同情感中的一种。与之前的研究不同，MEMO-Bench 提供了一个在情感分析背景下评估 T2I 模型和 MLLM 的框架。此外，还采用了一种渐进式评估方法，从粗粒度指标转向细粒度指标，以提供对 MLLM 情感分析能力的更详细和全面的评估。实验结果表明，现有的 T2I 模型在生成积极情绪方面比生成消极情绪方面更有效。同时，虽然 MLLM 在区分和识别人类情绪方面表现出一定的有效性，但它们的准确率达不到人类水平，特别是在细粒度情绪分析方面。MEMO-Bench 将公开发布，以支持该领域的进一步研究。

Title: ZeFaV: Boosting Large Language Models for Zero-shot Fact Verification

• Authors: Son T. Luu, Hiep Nguyen, Trung Vo, Le-Minh Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11247
• Pdf URL: https://arxiv.org/pdf/2411.11247
• Copy Paste: [[2411.11247]] ZeFaV: Boosting Large Language Models for Zero-shot Fact Verification(https://arxiv.org/abs/2411.11247)
• Keywords: language model
• Abstract: In this paper, we propose ZeFaV - a zero-shot based fact-checking verification framework to enhance the performance on fact verification task of large language models by leveraging the in-context learning ability of large language models to extract the relations among the entities within a claim, re-organized the information from the evidence in a relationally logical form, and combine the above information with the original evidence to generate the context from which our fact-checking model provide verdicts for the input claims. We conducted empirical experiments to evaluate our approach on two multi-hop fact-checking datasets including HoVer and FEVEROUS, and achieved potential results results comparable to other state-of-the-art fact verification task methods.
• 摘要：在本文中，我们提出了一种基于零样本的事实核查验证框架 ZeFaV，利用大型语言模型的上下文学习能力来提取声明中实体之间的关系，以关系逻辑形式重新组织证据中的信息，并将上述信息与原始证据相结合以生成上下文，我们的事实核查模型据此为输入声明提供判决，从而提高大型语言模型在事实核查任务上的性能。我们在包括 HoVer 和 FEVEROUS 在内的两个多跳事实核查数据集上进行了实证实验来评估我们的方法，并取得了与其他最先进的事实核查任务方法相当的潜在结果。

Title: Large corpora and large language models: a replicable method for automating grammatical annotation

• Authors: Cameron Morin, Matti Marttinen Larsson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11260
• Pdf URL: https://arxiv.org/pdf/2411.11260
• Copy Paste: [[2411.11260]] Large corpora and large language models: a replicable method for automating grammatical annotation(https://arxiv.org/abs/2411.11260)
• Keywords: language model, prompt
• Abstract: Much linguistic research relies on annotated datasets of features extracted from text corpora, but the rapid quantitative growth of these corpora has created practical difficulties for linguists to manually annotate large data samples. In this paper, we present a replicable, supervised method that leverages large language models for assisting the linguist in grammatical annotation through prompt engineering, training, and evaluation. We introduce a methodological pipeline applied to the case study of formal variation in the English evaluative verb construction 'consider X (as) (to be) Y', based on the large language model Claude 3.5 Sonnet and corpus data from Davies' NOW and EnTenTen21 (SketchEngine). Overall, we reach a model accuracy of over 90% on our held-out test samples with only a small amount of training data, validating the method for the annotation of very large quantities of tokens of the construction in the future. We discuss the generalisability of our results for a wider range of case studies of grammatical constructions and grammatical variation and change, underlining the value of AI copilots as tools for future linguistic research.
• 摘要：许多语言学研究依赖于从文本语料库中提取的特征的注释数据集，但这些语料库的快速增长给语言学家手动注释大量数据样本带来了实际困难。在本文中，我们提出了一种可复制的监督方法，利用大型语言模型通过快速工程、训练和评估来协助语言学家进行语法注释。我们介绍了一种应用于英语评价动词结构“将 X（视为）（是）Y”形式变化案例研究的方法流水线，该方法基于大型语言模型 Claude 3.5 Sonnet 和 Davies 的 NOW 和 EnTenTen21（SketchEngine）的语料库数据。总体而言，我们仅使用少量训练数据就在我们的保留测试样本上实现了超过 90% 的模型准确率，验证了该方法在未来注释大量结构标记的可行性。我们讨论了我们的研究结果对更广泛的语法结构和语法变异和变化案例研究的普遍性，强调了人工智能副驾驶作为未来语言研究工具的价值。

Title: VersaTune: Fine-Tuning Multi-Ability LLMs Efficiently

• Authors: Keer Lu, Keshi Zhao, Zheng Liang, Da Pan, Shusen Zhang, Xin Wu, Weipeng Chen, Zenan Zhou, Guosheng Dong, Bin Cui, Wentao Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11266
• Pdf URL: https://arxiv.org/pdf/2411.11266
• Copy Paste: [[2411.11266]] VersaTune: Fine-Tuning Multi-Ability LLMs Efficiently(https://arxiv.org/abs/2411.11266)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) exhibit remarkable capabilities in handling multiple tasks across domains due to their emergent properties. These capabilities are further augmented during the Supervised Fine-Tuning (SFT) phase. Despite their potential, existing work mainly focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. In this study, we introduce VersaTune, a novel data composition framework designed for enhancing LLMs' overall multi-ability performances during fine-tuning. We categorize knowledge into distinct domains including law, medicine, finance, science, code. We begin with detecting the distribution of domain-specific knowledge within the base model, followed by the composition of training data that aligns with the model's existing knowledge distribution. During the fine-tuning process, weights of different domains are dynamically adjusted based on their learnable potential and forgetting degree. Experimental results demonstrate that VersaTune achieves significant improvements in multi-domain performance, with a 35.21% enhancement in comprehensive multi-domain tasks. Additionally, in scenarios where specific domain optimization is required, VersaTune reduces the degradation of performance in other domains by 38.77%, without compromising the target domain's training efficacy.
• 摘要：大型语言模型 (LLM) 因其新兴特性，在跨领域处理多项任务方面表现出卓越的能力。这些能力在监督微调 (SFT) 阶段得到进一步增强。尽管它们具有潜力，但现有工作主要侧重于微调过程中特定领域的增强，其挑战在于对其他领域的知识的灾难性遗忘。在本研究中，我们介绍了 VersaTune，这是一种新颖的数据组合框架，旨在增强 LLM 在微调过程中的整体多能力性能。我们将知识分为不同的领域，包括法律、医学、金融、科学、代码。我们首先检测基础模型中特定领域知识的分布，然后组合与模型现有知识分布一致的训练数据。在微调过程中，不同领域的权重会根据其可学习潜力和遗忘程度进行动态调整。实验结果表明，VersaTune 在多领域性能方面取得了显着提升，综合多领域任务的提升幅度为 35.21%。此外，在需要特定领域优化的场景中，VersaTune 将其他领域的性能下降减少了 38.77%，同时不会影响目标领域的训练效果。

Title: LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

• Authors: Yungi Kim, Hyunsoo Ha, Seonghoon Yang, Sukyung Lee, Jihoo Kim, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11289
• Pdf URL: https://arxiv.org/pdf/2411.11289
• Copy Paste: [[2411.11289]] LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models(https://arxiv.org/abs/2411.11289)
• Keywords: language model, llm
• Abstract: Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.
• 摘要：为大型语言模型 (LLM) 创建高质量的大规模数据集通常依赖于资源密集型、GPU 加速的模型进行质量过滤，这使得该过程耗时且成本高昂。这种对 GPU 的依赖限制了缺乏重要计算基础设施的组织的可访问性。为了解决这个问题，我们引入了轻量级、目的驱动 (LP) 数据管道，这是一个完全在 CPU 上运行的框架，可简化数据集提取、过滤和管理的过程。基于我们的四个核心原则，LP 数据管道在保持高数据质量的同时，显著减少了准备时间和成本。重要的是，我们的管道能够创建针对特定领域和语言量身定制的目的驱动数据集，从而增强 LLM 在专业环境中的适用性。我们预计我们的管道将降低 LLM 开发的门槛，使广泛的组织能够更轻松地访问 LLM。

Title: Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translation

• Authors: Peng Shu, Junhao Chen, Zhengliang Liu, Hui Wang, Zihao Wu, Tianyang Zhong, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Constance Owl, Xiaoming Zhai, Ninghao Liu, Claudio Saunt, Tianming Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11295
• Pdf URL: https://arxiv.org/pdf/2411.11295
• Copy Paste: [[2411.11295]] Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translation(https://arxiv.org/abs/2411.11295)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of tasks and domains. However, their performance in low-resource language translation, particularly when translating into these languages, remains underexplored. This gap poses significant challenges, as linguistic barriers hinder the cultural preservation and development of minority communities. To address this issue, this paper introduces a novel retrieval-based method that enhances translation quality for low-resource languages by focusing on key terms, which involves translating keywords and retrieving corresponding examples from existing data. To evaluate the effectiveness of this method, we conducted experiments translating from English into three low-resource languages: Cherokee, a critically endangered indigenous language of North America; Tibetan, a historically and culturally significant language in Asia; and Manchu, a language with few remaining speakers. Our comparison with the zero-shot performance of GPT-4o and LLaMA 3.1 405B, highlights the significant challenges these models face when translating into low-resource languages. In contrast, our retrieval-based method shows promise in improving both word-level accuracy and overall semantic understanding by leveraging existing resources more effectively.
• 摘要：大型语言模型 (LLM) 在广泛的任务和领域中都取得了显著的成功。然而，它们在低资源语言翻译中的表现，尤其是翻译成这些语言时的表现，仍然没有得到充分的探索。这一差距带来了重大挑战，因为语言障碍阻碍了少数民族社区的文化保护和发展。为了解决这个问题，本文介绍了一种基于检索的新方法，该方法通过关注关键词来提高低资源语言的翻译质量，这涉及翻译关键字并从现有数据中检索相应的示例。为了评估这种方法的有效性，我们进行了从英语翻译成三种低资源语言的实验：切诺基语，一种极度濒危的北美土著语言；藏语，一种具有历史和文化意义的亚洲语言；满语，一种很少有人使用的语言。我们与 GPT-4o 和 LLaMA 3.1 405B 的零样本性能的比较突出了这些模型在翻译成低资源语言时面临的重大挑战。相比之下，我们的基于检索的方法通过更有效地利用现有资源，有望提高词级准确性和整体语义理解。

Title: Mitigating Knowledge Conflicts in Language Model-Driven Question Answering

• Authors: Han Cao, Zhaoyang Zhang, Xiangtian Li, Chufan Wu, Hansong Zhang, Wenqing Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11344
• Pdf URL: https://arxiv.org/pdf/2411.11344
• Copy Paste: [[2411.11344]] Mitigating Knowledge Conflicts in Language Model-Driven Question Answering(https://arxiv.org/abs/2411.11344)
• Keywords: language model, hallucination
• Abstract: Knowledge-aware sequence to sequence generation tasks such as document question answering and abstract summarization typically requires two types of knowledge: encoded parametric knowledge and retrieved contextual information. Previous work show improper correlation between parametric knowledge and answers in the training set could cause the model ignore input information at test time, resulting in un-desirable model behaviour such as over-stability and hallucination. In this work, we argue that hallucination could be mitigated via explicit correlation between input source and generated content. We focus on a typical example of hallucination, entity-based knowledge conflicts in question answering, where correlation of entities and their description at training time hinders model behaviour during inference.
• 摘要：知识感知的序列到序列生成任务（例如文档问答和摘要总结）通常需要两种类型的知识：编码的参数知识和检索到的上下文信息。先前的研究表明，训练集中的参数知识和答案之间的不正确关联可能会导致模型在测试时忽略输入信息，从而导致模型出现不良行为，例如过度稳定和幻觉。在这项工作中，我们认为可以通过输入源和生成内容之间的显式关联来减轻幻觉。我们专注于幻觉的一个典型例子，即问答中的基于实体的知识冲突，其中训练时的实体及其描述的相关性会阻碍推理过程中的模型行为。

Title: Rethinking Thinking Tokens: Understanding Why They Underperform in Practice

• Authors: Sreeram Vennam, David Valente, David Herel, Ponnurangam Kumaraguru
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11371
• Pdf URL: https://arxiv.org/pdf/2411.11371
• Copy Paste: [[2411.11371]] Rethinking Thinking Tokens: Understanding Why They Underperform in Practice(https://arxiv.org/abs/2411.11371)
• Keywords: language model, llm, chain-of-thought
• Abstract: Thinking Tokens (TT) have been proposed as an unsupervised method to facilitate reasoning in language models. However, despite their conceptual appeal, our findings show that TTs marginally improves performance and consistently underperforms compared to Chain-of-Thought (CoT) reasoning across multiple benchmarks. We hypothesize that this underperformance stems from the reliance on a single embedding for TTs, which results in inconsistent learning signals and introduces noisy gradients. This paper provides a comprehensive empirical analysis to validate this hypothesis and discusses the implications for future research on unsupervised reasoning in LLMs.
• 摘要：思考标记 (TT) 已被提议作为一种无监督方法，以促进语言模型中的推理。然而，尽管它们的概念很有吸引力，但我们的研究结果表明，与思维链 (CoT) 推理相比，TT 的性能提升幅度很小，并且在多个基准测试中始终表现不佳。我们假设这种表现不佳源于对 TT 的单一嵌入的依赖，这会导致学习信号不一致并引入噪声梯度。本文提供了全面的实证分析来验证这一假设，并讨论了对 LLM 无监督推理未来研究的影响。

Title: Membership Inference Attack against Long-Context Large Language Models

• Authors: Zixiong Wang, Gaoyang Liu, Yang Yang, Chen Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11424
• Pdf URL: https://arxiv.org/pdf/2411.11424
• Copy Paste: [[2411.11424]] Membership Inference Attack against Long-Context Large Language Models(https://arxiv.org/abs/2411.11424)
• Keywords: language model, llm, long context, hallucination, chat
• Abstract: Recent advances in Large Language Models (LLMs) have enabled them to overcome their context window limitations, and demonstrate exceptional retrieval and reasoning capacities on longer context. Quesion-answering systems augmented with Long-Context Language Models (LCLMs) can automatically search massive external data and incorporate it into their contexts, enabling faithful predictions and reducing issues such as hallucinations and knowledge staleness. Existing studies targeting LCLMs mainly concentrate on addressing the so-called lost-in-the-middle problem or improving the inference effiencicy, leaving their privacy risks largely unexplored. In this paper, we aim to bridge this gap and argue that integrating all information into the long context makes it a repository of sensitive information, which often contains private data such as medical records or personal identities. We further investigate the membership privacy within LCLMs external context, with the aim of determining whether a given document or sequence is included in the LCLMs context. Our basic idea is that if a document lies in the context, it will exhibit a low generation loss or a high degree of semantic similarity to the contents generated by LCLMs. We for the first time propose six membership inference attack (MIA) strategies tailored for LCLMs and conduct extensive experiments on various popular models. Empirical results demonstrate that our attacks can accurately infer membership status in most cases, e.g., 90.66% attack F1-score on Multi-document QA datasets with LongChat-7b-v1.5-32k, highlighting significant risks of membership leakage within LCLMs input contexts. Furthermore, we examine the underlying reasons why LCLMs are susceptible to revealing such membership information.
• 摘要：大型语言模型 (LLM) 的最新进展使其能够克服上下文窗口限制，并在较长的上下文中展示出出色的检索和推理能力。增强了长上下文语言模型 (LCLM) 的问答系统可以自动搜索大量外部数据并将其合并到其上下文中，从而实现忠实的预测并减少幻觉和知识陈旧等问题。现有针对 LCLM 的研究主要集中于解决所谓的中间丢失问题或提高推理效率，而其隐私风险基本上未被探究。在本文中，我们旨在弥合这一空白，并认为将所有信息集成到长上下文中会使其成为敏感信息的存储库，其中通常包含私人数据，例如医疗记录或个人身份。我们进一步研究 LCLM 外部上下文中的成员隐私，目的是确定给定的文档或序列是否包含在 LCLM 上下文中。我们的基本思想是，如果文档位于上下文中，它将表现出较低的生成损失或与 LCLM 生成的内容具有较高的语义相似度。我们首次提出了六种针对 LCLM 量身定制的成员推理攻击 (MIA) 策略，并对各种流行模型进行了广泛的实验。实证结果表明，我们的攻击在大多数情况下可以准确推断成员状态，例如，使用 LongChat-7b-v1.5-32k 在多文档 QA 数据集上攻击 F1 分数为 90.66%，突出了 LCLM 输入上下文中成员泄漏的重大风险。此外，我们研究了 LCLM 容易泄露此类成员信息的根本原因。

Title: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts

• Authors: Jingxuan Li, Yuning Yang, Shengqi Yang, Yizhou Zhao, Ying Nian Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11479
• Pdf URL: https://arxiv.org/pdf/2411.11479
• Copy Paste: [[2411.11479]] Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts(https://arxiv.org/abs/2411.11479)
• Keywords: language model, prompt, agent
• Abstract: The rapid advancement of Vision-Language Models (VLMs) has expanded multimodal applications, yet evaluations often focus on basic tasks like object recognition, overlooking abstract aspects such as personalities and values. To address this gap, we introduce Value-Spectrum, a visual question-answering benchmark aimed at assessing VLMs based on Schwartz's value dimensions, which capture core values guiding people's beliefs and actions across cultures. We constructed a vectorized database of over 50,000 short videos sourced from TikTok, YouTube Shorts, and Instagram Reels, covering multiple months and a wide array of topics such as family, health, hobbies, society, and technology. We also developed a VLM agent pipeline to automate video browsing and analysis. Benchmarking representative VLMs on Value-Spectrum reveals significant differences in their responses to value-oriented content, with most models exhibiting a preference for hedonistic topics. Beyond identifying natural preferences, we explored the ability of VLM agents to adopt specific personas when explicitly prompted, revealing insights into the models' adaptability in role-playing scenarios. These findings highlight the potential of Value-Spectrum as a comprehensive evaluation set for tracking VLM advancements in value-based tasks and for developing more sophisticated role-playing AI agents.
• 摘要：视觉语言模型 (VLM) 的快速发展扩大了多模态应用，但评估通常侧重于对象识别等基本任务，而忽略了个性和价值观等抽象方面。为了弥补这一差距，我们引入了 Value-Spectrum，这是一个视觉问答基准，旨在评估基于 Schwartz 价值观维度的 VLM，这些维度捕捉了指导跨文化人们信仰和行为的核心价值观。我们构建了一个矢量化数据库，其中包含来自 TikTok、YouTube Shorts 和 Instagram Reels 的 50,000 多个短视频，涵盖多个月份和各种主题，例如家庭、健康、爱好、社会和技术。我们还开发了一个 VLM 代理管道来自动化视频浏览和分析。在 Value-Spectrum 上对代表性 VLM 进行基准测试发现，它们对价值导向内容的反应存在显著差异，大多数模型都表现出对享乐主义主题的偏好。除了识别自然偏好之外，我们还探索了 VLM 代理在明确提示时采用特定角色的能力，揭示了模型在角色扮演场景中的适应性。这些发现凸显了 Value-Spectrum 作为一个综合评估集的潜力，可用于跟踪基于价值的任务中的 VLM 进展以及开发更复杂的角色扮演 AI 代理。

Title: Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models

• Authors: Chenhang Cui, Gelei Deng, An Zhang, Jingnan Zheng, Yicong Li, Lianli Gao, Tianwei Zhang, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11496
• Pdf URL: https://arxiv.org/pdf/2411.11496
• Copy Paste: [[2411.11496]] Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models(https://arxiv.org/abs/2411.11496)
• Keywords: language model, prompt, agent
• Abstract: Recent advances in Large Vision-Language Models (LVLMs) have showcased strong reasoning abilities across multiple modalities, achieving significant breakthroughs in various real-world applications. Despite this great success, the safety guardrail of LVLMs may not cover the unforeseen domains introduced by the visual modality. Existing studies primarily focus on eliciting LVLMs to generate harmful responses via carefully crafted image-based jailbreaks designed to bypass alignment defenses. In this study, we reveal that a safe image can be exploited to achieve the same jailbreak consequence when combined with additional safe images and prompts. This stems from two fundamental properties of LVLMs: universal reasoning capabilities and safety snowball effect. Building on these insights, we propose Safety Snowball Agent (SSA), a novel agent-based framework leveraging agents' autonomous and tool-using abilities to jailbreak LVLMs. SSA operates through two principal stages: (1) initial response generation, where tools generate or retrieve jailbreak images based on potential harmful intents, and (2) harmful snowballing, where refined subsequent prompts induce progressively harmful outputs. Our experiments demonstrate that \ours can use nearly any image to induce LVLMs to produce unsafe content, achieving high success jailbreaking rates against the latest LVLMs. Unlike prior works that exploit alignment flaws, \ours leverages the inherent properties of LVLMs, presenting a profound challenge for enforcing safety in generative multimodal systems. Our code is avaliable at \url{this https URL}.
• 摘要：大型视觉语言模型 (LVLM) 的最新进展已展示出跨多种模态的强大推理能力，在各种实际应用中取得了重大突破。尽管取得了巨大的成功，但 LVLM 的安全护栏可能无法覆盖视觉模态引入的不可预见的领域。现有研究主要集中于通过精心设计的基于图像的越狱来诱发 LVLM 产生有害反应，这些越狱旨在绕过对齐防御。在这项研究中，我们发现，当结合使用安全图像和其他安全图像和提示时，可以利用安全图像来实现相同的越狱结果。这源于 LVLM 的两个基本特性：通用推理能力和安全滚雪球效应。基于这些见解，我们提出了安全滚雪球代理 (SSA)，这是一种新颖的基于代理的框架，利用代理的自主和使用工具的能力来越狱 LVLM。 SSA 分为两个主要阶段：(1) 初始响应生成，其中工具根据潜在的有害意图生成或检索越狱图像；(2) 有害滚雪球，其中改进的后续提示会逐渐产生有害的输出。我们的实验表明，\ours 可以使用几乎任何图像来诱导 LVLM 产生不安全的内容，从而对最新的 LVLM 实现高越狱成功率。与之前利用对齐缺陷的研究不同，\ours 利用了 LVLM 的固有属性，这对在生成多模态系统中实施安全性提出了严峻挑战。我们的代码可在 \url{此 https URL} 上找到。

Title: Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality

• Authors: Viktoriia Chekalina, Anton Razzigaev, Elizaveta Goncharova, Andrey Kuznetsov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11531
• Pdf URL: https://arxiv.org/pdf/2411.11531
• Copy Paste: [[2411.11531]] Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality(https://arxiv.org/abs/2411.11531)
• Keywords: language model, llm, hallucination, chat
• Abstract: In this paper we present an approach to reduce hallucinations in Large Language Models (LLMs) by incorporating Knowledge Graphs (KGs) as an additional modality. Our method involves transforming input text into a set of KG embeddings and using an adapter to integrate these embeddings into the language model space, without relying on external retrieval processes. To facilitate this, we created WikiEntities, a dataset containing over 3 million Wikipedia texts annotated with entities from Wikidata and their corresponding embeddings from PyTorch-BigGraph. This dataset serves as a valuable resource for training Entity Linking models and adapting the described method to various LLMs using specialized adapters. Our method does not require fine-tuning of the language models themselves; instead, we only train the adapter. This ensures that the model's performance on other tasks is not affected. We trained an adapter for the Mistral 7B, LLaMA 2-7B (chat), and LLaMA 3-8B (instruct) models using this dataset and demonstrated that our approach improves performance on the HaluEval, True-False benchmarks and FEVER dataset. The results indicate that incorporating KGs as a new modality can effectively reduce hallucinations and improve the factual accuracy of language models, all without the need for external retrieval.
• 摘要：在本文中，我们提出了一种通过结合知识图谱 (KG) 作为附加模态来减少大型语言模型 (LLM) 中的幻觉的方法。我们的方法包括将输入文本转换为一组 KG 嵌入，并使用适配器将这些嵌入集成到语言模型空间中，而无需依赖外部检索过程。为了实现这一点，我们创建了 WikiEntities，这是一个包含超过 300 万个维基百科文本的数据集，这些文本使用来自 Wikidata 的实体及其来自 PyTorch-BigGraph 的相应嵌入进行注释。该数据集是训练实体链接模型和使用专用适配器将所述方法适应各种 LLM 的宝贵资源。我们的方法不需要对语言模型本身进行微调；相反，我们只训练适配器。这确保了模型在其他任务上的性能不受影响。我们使用该数据集训练了 Mistral 7B、LLaMA 2-7B（聊天）和 LLaMA 3-8B（指导）模型的适配器，并证明了我们的方法提高了 HaluEval、True-False 基准和 FEVER 数据集上的性能。结果表明，将 KG 作为一种新模式可以有效减少幻觉并提高语言模型的事实准确性，所有这些都无需外部检索。

Title: OASIS: Open Agents Social Interaction Simulations on One Million Agents

• Authors: Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong, Prateek Gupta, Shuyue Hu, Zhenfei Yin, Guohao Li, Xu Jia, Lijun Wang, Bernard Ghanem, Huchuan Lu, Wanli Ouyang, Yu Qiao, Philip Torr, Jing Shao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11581
• Pdf URL: https://arxiv.org/pdf/2411.11581
• Copy Paste: [[2411.11581]] OASIS: Open Agents Social Interaction Simulations on One Million Agents(https://arxiv.org/abs/2411.11581)
• Keywords: language model, llm, agent
• Abstract: There has been a growing interest in enhancing rule-based agent-based models (ABMs) for social media platforms (\emph{i.e.}, X, Reddit) with more realistic large language model (LLM) agents, thereby allowing for a more nuanced study of complex systems. As a result, several LLM-based ABMs have been proposed in the past year. While they hold promise, each simulator is specifically designed to study a particular scenario, making it time-consuming and resource-intensive to explore other phenomena using the same ABM. Additionally, these models simulate only a limited number of agents, whereas real-world social media platforms involve millions of users. To this end, we propose OASIS, a generalizable and scalable social media simulator. OASIS is designed based on real-world social media platforms, incorporating dynamically updated environments (\emph{i.e.}, dynamic social networks and post information), diverse action spaces (\emph{i.e.}, following, commenting), and recommendation systems (\emph{i.e.}, interest-based and hot-score-based). Additionally, OASIS supports large-scale user simulations, capable of modeling up to one million users. With these features, OASIS can be easily extended to different social media platforms to study large-scale group phenomena and behaviors. We replicate various social phenomena, including information spreading, group polarization, and herd effects across X and Reddit platforms. Moreover, we provide observations of social phenomena at different agent group scales. We observe that the larger agent group scale leads to more enhanced group dynamics and more diverse and helpful agents' opinions. These findings demonstrate OASIS's potential as a powerful tool for studying complex systems in digital environments.
• 摘要：人们越来越有兴趣使用更现实的大型语言模型 (LLM) 代理来增强社交媒体平台（\emph{i.e.}、X、Reddit）的基于规则的代理模型 (ABM)，从而可以对复杂系统进行更细致的研究。因此，过去一年提出了几种基于 LLM 的 ABM。虽然它们很有前景，但每个模拟器都是专门为研究特定场景而设计的，因此使用相同的 ABM 探索其他现象会耗费大量时间和资源。此外，这些模型仅模拟有限数量的代理，而现实世界的社交媒体平台涉及数百万用户。为此，我们提出了 OASIS，一种可通用且可扩展的社交媒体模拟器。OASIS 是基于现实世界的社交媒体平台设计的，结合了动态更新的环境（\emph{i.e.}、动态社交网络和帖子信息）、多样化的动作空间（\emph{i.e.}、关注、评论）和推荐系统（\emph{i.e.}、基于兴趣和基于热门评分）。此外，OASIS 支持大规模用户模拟，能够对多达一百万用户进行建模。凭借这些功能，OASIS 可以轻松扩展到不同的社交媒体平台，以研究大规模群体现象和行为。我们在 X 和 Reddit 平台上复制了各种社会现象，包括信息传播、群体极化和从众效应。此外，我们还在不同代理群体规模上提供了对社会现象的观察。我们观察到，代理群体规模越大，群体动态就越强，代理的意见也越多样化、越有帮助。这些发现证明了 OASIS 成为研究数字环境中复杂系统的强大工具的潜力。

Title: Chapter 7 Review of Data-Driven Generative AI Models for Knowledge Extraction from Scientific Literature in Healthcare

• Authors: Leon Kopitar, Primoz Kocbek, Lucija Gosak, Gregor Stiglic
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11635
• Pdf URL: https://arxiv.org/pdf/2411.11635
• Copy Paste: [[2411.11635]] Chapter 7 Review of Data-Driven Generative AI Models for Knowledge Extraction from Scientific Literature in Healthcare(https://arxiv.org/abs/2411.11635)
• Keywords: language model, gpt
• Abstract: This review examines the development of abstractive NLP-based text summarization approaches and compares them to existing techniques for extractive summarization. A brief history of text summarization from the 1950s to the introduction of pre-trained language models such as Bidirectional Encoder Representations from Transformer (BERT) and Generative Pre-training Transformers (GPT) are presented. In total, 60 studies were identified in PubMed and Web of Science, of which 29 were excluded and 24 were read and evaluated for eligibility, resulting in the use of seven studies for further analysis. This chapter also includes a section with examples including an example of a comparison between GPT-3 and state-of-the-art GPT-4 solutions in scientific text summarisation. Natural language processing has not yet reached its full potential in the generation of brief textual summaries. As there are acknowledged concerns that must be addressed, we can expect gradual introduction of such models in practise.
• 摘要：本综述研究了基于 NLP 的抽象文本摘要方法的发展，并将其与现有的提取摘要技术进行了比较。介绍了文本摘要的简要历史，从 1950 年代到引入预训练语言模型，例如 Transformer 的双向编码器表示 (BERT) 和生成式预训练 Transformer (GPT)。总共在 PubMed 和 Web of Science 中确定了 60 项研究，其中 29 项被排除，24 项被阅读并评估其资格，最终使用 7 项研究进行进一步分析。本章还包括一个示例部分，包括一个 GPT-3 和最先进的 GPT-4 解决方案在科学文本摘要中的比较示例。自然语言处理在生成简短文本摘要方面尚未发挥其全部潜力。由于存在必须解决的已知问题，我们可以期待在实践中逐步引入此类模型。

Title: Technical Report: Enhancing LLM Reasoning with Reward-guided Tree Search

• Authors: Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, Zheng Liu, Dong Yan, Jian Xie, Zhongyuan Wang, Ji-Rong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11694
• Pdf URL: https://arxiv.org/pdf/2411.11694
• Copy Paste: [[2411.11694]] Technical Report: Enhancing LLM Reasoning with Reward-guided Tree Search(https://arxiv.org/abs/2411.11694)
• Keywords: language model, llm
• Abstract: Recently, test-time scaling has garnered significant attention from the research community, largely due to the substantial advancements of the o1 model released by OpenAI. By allocating more computational resources during the inference phase, large language models~(LLMs) can extensively explore the solution space by generating more thought tokens or diverse solutions, thereby producing more accurate responses. However, developing an o1-like reasoning approach is challenging, and researchers have been making various attempts to advance this open area of research. In this paper, we present a preliminary exploration into enhancing the reasoning abilities of LLMs through reward-guided tree search algorithms. This framework is implemented by integrating the policy model, reward model, and search algorithm. It is primarily constructed around a tree search algorithm, where the policy model navigates a dynamically expanding tree guided by a specially trained reward model. We thoroughly explore various design considerations necessary for implementing this framework and provide a detailed report of the technical aspects. To assess the effectiveness of our approach, we focus on mathematical reasoning tasks and conduct extensive evaluations on four challenging datasets, significantly enhancing the reasoning abilities of LLMs.
• 摘要：最近，测试时间扩展引起了研究界的极大关注，这在很大程度上要归功于 OpenAI 发布的 o1 模型的重大进步。通过在推理阶段分配更多的计算资源，大型语言模型 (LLM) 可以通过生成更多思维标记或多样化解决方案来广泛探索解决方案空间，从而产生更准确的响应。然而，开发一种类似 o1 的推理方法具有挑战性，研究人员一直在进行各种尝试来推进这一开放的研究领域。在本文中，我们对通过奖励引导的树搜索算法增强 LLM 的推理能力进行了初步探索。该框架通过集成策略模型、奖励模型和搜索算法来实现。它主要围绕树搜索算法构建，其中策略模型在经过专门训练的奖励模型的引导下导航动态扩展的树。我们彻底探讨了实施该框架所需的各种设计考虑因素，并提供了详细的技术方面报告。为了评估我们方法的有效性，我们专注于数学推理任务并对四个具有挑战性的数据集进行了广泛的评估，显著增强了 LLM 的推理能力。

Title: FedCoLLM: A Parameter-Efficient Federated Co-tuning Framework for Large and Small Language Models

• Authors: Tao Fan, Yan Kang, Guoqiang Ma, Lixin Fan, Kai Chen, Qiang Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11707
• Pdf URL: https://arxiv.org/pdf/2411.11707
• Copy Paste: [[2411.11707]] FedCoLLM: A Parameter-Efficient Federated Co-tuning Framework for Large and Small Language Models(https://arxiv.org/abs/2411.11707)
• Keywords: language model, llm
• Abstract: By adapting Large Language Models (LLMs) to domain-specific tasks or enriching them with domain-specific knowledge, we can fully harness the capabilities of LLMs. Nonetheless, a gap persists in achieving simultaneous mutual enhancement between the server's LLM and the downstream clients' Small Language Models (SLMs). To address this, we propose FedCoLLM, a novel and parameter-efficient federated framework designed for co-tuning LLMs and SLMs. This approach is aimed at adaptively transferring server-side LLMs knowledge to clients' SLMs while simultaneously enriching the LLMs with domain insights from the clients. To accomplish this, FedCoLLM utilizes lightweight adapters in conjunction with SLMs, facilitating knowledge exchange between server and clients in a manner that respects data privacy while also minimizing computational and communication overhead. Our evaluation of FedCoLLM, utilizing various public LLMs and SLMs across a range of NLP text generation tasks, reveals that the performance of clients' SLMs experiences notable improvements with the assistance of the LLMs. Simultaneously, the LLMs enhanced via FedCoLLM achieves comparable performance to that obtained through direct fine-tuning on clients' data.
• 摘要：通过将大型语言模型 (LLM) 调整为特定领域的任务或用特定领域的知识丰富它们，我们可以充分利用 LLM 的功能。尽管如此，在实现服务器的 LLM 和下游客户端的小型语言模型 (SLM) 之间的同时相互增强方面仍然存在差距。为了解决这个问题，我们提出了 FedCoLLM，这是一种新颖且参数高效的联合框架，旨在共同调整 LLM 和 SLM。这种方法旨在自适应地将服务器端 LLM 知识传输到客户端的 SLM，同时用来自客户端的领域见解丰富 LLM。为了实现这一点，FedCoLLM 与 SLM 结合使用轻量级适配器，以尊重数据隐私的方式促进服务器和客户端之间的知识交换，同时最大限度地减少计算和通信开销。我们对 FedCoLLM 的评估利用了各种公共 LLM 和 SLM，用于一系列 NLP 文本生成任务，结果表明，在 LLM 的帮助下，客户端的 SLM 的性能得到了显着改善。同时，通过 FedCoLLM 增强的 LLM 实现了与通过直接对客户数据进行微调所获得的性能相当的性能。

Title: Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignment

• Authors: Allison Huang, Yulu Niki Pi, Carlos Mougan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11731
• Pdf URL: https://arxiv.org/pdf/2411.11731
• Copy Paste: [[2411.11731]] Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignment(https://arxiv.org/abs/2411.11731)
• Keywords: language model, llm, prompt, agent
• Abstract: We explore how large language models (LLMs) can be influenced by prompting them to alter their initial decisions and align them with established ethical frameworks. Our study is based on two experiments designed to assess the susceptibility of LLMs to moral persuasion. In the first experiment, we examine the susceptibility to moral ambiguity by evaluating a Base Agent LLM on morally ambiguous scenarios and observing how a Persuader Agent attempts to modify the Base Agent's initial decisions. The second experiment evaluates the susceptibility of LLMs to align with predefined ethical frameworks by prompting them to adopt specific value alignments rooted in established philosophical theories. The results demonstrate that LLMs can indeed be persuaded in morally charged scenarios, with the success of persuasion depending on factors such as the model used, the complexity of the scenario, and the conversation length. Notably, LLMs of distinct sizes but from the same company produced markedly different outcomes, highlighting the variability in their susceptibility to ethical persuasion.
• 摘要：我们探索如何通过促使大型语言模型 (LLM) 改变其初始决策并使其与既定的道德框架保持一致来影响它们。我们的研究基于两个旨在评估 LLM 对道德说服的敏感性的实验。在第一个实验中，我们通过在道德模糊场景中评估基础代理 LLM 并观察说服者代理如何尝试修改基础代理的初始决策来检查对道德模糊性的敏感性。第二个实验通过促使 LLM 采用植根于既定哲学理论的特定价值观取向来评估 LLM 与预定义的道德框架保持一致的敏感性。结果表明，LLM 确实可以在道德场景中被说服，说服的成功取决于所使用的模型、场景的复杂性和对话长度等因素。值得注意的是，来自同一家公司的不同规模的 LLM 产生了明显不同的结果，突显了它们对道德说服的敏感性存在差异。

Title: CNMBert: A Model For Hanyu Pinyin Abbreviation to Character Conversion Task

• Authors: Zishuo Feng, Feng Cao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11770
• Pdf URL: https://arxiv.org/pdf/2411.11770
• Copy Paste: [[2411.11770]] CNMBert: A Model For Hanyu Pinyin Abbreviation to Character Conversion Task(https://arxiv.org/abs/2411.11770)
• Keywords: gpt
• Abstract: The task of converting Hanyu Pinyin abbreviations to Chinese characters represents a significant branch within the domain of Chinese Spelling Correction (CSC). This task is typically one of text-length alignment, however, due to the limited informational content in pinyin abbreviations, achieving accurate conversion is challenging. In this paper, we propose CNMBert which stands for zh-CN Pinyin Multi-mask Bert Model as a solution to this issue. CNMBert surpasses few-shot GPT models, achieving a 59.63% MRR on a 10,424-sample Hanyu Pinyin abbreviation test dataset.
• 摘要：将汉语拼音缩写转换为汉字的任务是中文拼写纠正 (CSC) 领域的一个重要分支。此任务通常是文本长度对齐任务之一，但是，由于拼音缩写中的信息内容有限，实现准确转换具有挑战性。在本文中，我们提出了 CNMBert（代表 zh-CN 拼音多掩码 Bert 模型）作为此问题的解决方案。CNMBert 超越了少样本 GPT 模型，在 10,424 个样本的汉语拼音缩写测试数据集上实现了 59.63% 的 MRR。

Title: Bi-Mamba: Towards Accurate 1-Bit State Space Models

• Authors: Shengkun Tang, Liqun Ma, Haonan Li, Mingjie Sun, Zhiqiang Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11843
• Pdf URL: https://arxiv.org/pdf/2411.11843
• Copy Paste: [[2411.11843]] Bi-Mamba: Towards Accurate 1-Bit State Space Models(https://arxiv.org/abs/2411.11843)
• Keywords: language model, llm
• Abstract: The typical selective state-space model (SSM) of Mamba addresses several limitations of Transformers, such as quadratic computational complexity with sequence length and significant inference-time memory requirements due to the key-value cache. However, the growing size of Mamba models continues to pose training and deployment challenges and raises environmental concerns due to considerable energy consumption. In this work, we introduce Bi-Mamba, a scalable and powerful 1-bit Mamba architecture designed for more efficient large language models with multiple sizes across 780M, 1.3B, and 2.7B. Bi-Mamba models are trained from scratch on data volume as regular LLM pertaining using an autoregressive distillation loss. Extensive experimental results on language modeling demonstrate that Bi-Mamba achieves performance comparable to its full-precision counterparts (e.g., FP16 or BF16) and much better accuracy than post-training-binarization (PTB) Mamba baselines, while significantly reducing memory footprint and energy consumption compared to the original Mamba model. Our study pioneers a new linear computational complexity LLM framework under low-bit representation and facilitates the future design of specialized hardware tailored for efficient 1-bit Mamba-based LLMs.
• 摘要：Mamba 的典型选择性状态空间模型 (SSM) 解决了 Transformers 的几个限制，例如序列长度的二次计算复杂度和由于键值缓存而产生的大量推理时间内存要求。然而，Mamba 模型的不断增长的规模继续带来训练和部署挑战，并因大量能源消耗而引发环境问题。在这项工作中，我们引入了 Bi-Mamba，这是一种可扩展且强大的 1 位 Mamba 架构，旨在实现更高效的大型语言模型，其规模多种多样，涵盖 780M、1.3B 和 2.7B。Bi-Mamba 模型从头开始在数据量上进行训练，就像使用自回归蒸馏损失的常规 LLM 一样。语言建模的大量实验结果表明，Bi-Mamba 实现了与其全精度同类产品（例如 FP16 或 BF16）相当的性能，并且比训练后二值化 (PTB) Mamba 基线具有更好的准确性，同时与原始 Mamba 模型相比，显着减少了内存占用和能耗。我们的研究开创了一种低位表示下的新型线性计算复杂度 LLM 框架，并促进了未来针对高效的基于 1 位 Mamba 的 LLM 量身定制的专用硬件的设计。