2025-01-15

Title: GPT as a Monte Carlo Language Tree: A Probabilistic Perspective

• Authors: Kun-Peng Ning, Jia-Yu Yao, Yu-Yang Liu, Mu-Nan Ning, Li Yuan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07641
• Pdf URL: https://arxiv.org/pdf/2501.07641
• Copy Paste: [[2501.07641]] GPT as a Monte Carlo Language Tree: A Probabilistic Perspective(https://arxiv.org/abs/2501.07641)
• Keywords: language model, gpt, llm, hallucination, chain-of-thought
• Abstract: Large Language Models (LLMs), such as GPT, are considered to learn the latent distributions within large-scale web-crawl datasets and accomplish natural language processing (NLP) tasks by predicting the next token. However, this mechanism of latent distribution modeling lacks quantitative understanding and analysis. In this paper, we propose a novel perspective that any language dataset can be represented by a Monte Carlo Language Tree (abbreviated as ``Data-Tree''), where each node denotes a token, each edge denotes a token transition probability, and each sequence has a unique path. Any GPT-like language model can also be flattened into another Monte Carlo Language Tree (abbreviated as ``GPT-Tree''). Our experiments show that different GPT models trained on the same dataset exhibit significant structural similarity in GPT-Tree visualization, and larger models converge more closely to the Data-Tree. More than 87\% GPT output tokens can be recalled by Data-Tree. These findings may confirm that the reasoning process of LLMs is more likely to be probabilistic pattern-matching rather than formal reasoning, as each model inference seems to find a context pattern with maximum probability from the Data-Tree. Furthermore, we provide deeper insights into issues such as hallucination, Chain-of-Thought (CoT) reasoning, and token bias in LLMs.
• 摘要：大型语言模型 (LLM)，例如 GPT，被认为可以学习大规模网络爬取数据集中的潜在分布，并通过预测下一个标记来完成自然语言处理 (NLP) 任务。然而，这种潜在分布建模机制缺乏定量的理解和分析。在本文中，我们提出了一个新颖的观点，即任何语言数据集都可以用蒙特卡洛语言树 (缩写为“数据树”) 表示，其中每个节点表示一个标记，每条边表示一个标记转换概率，每个序列都有一条唯一的路径。任何类似 GPT 的语言模型也可以展平为另一个蒙特卡洛语言树 (缩写为“GPT 树”)。我们的实验表明，在同一数据集上训练的不同 GPT 模型在 GPT 树可视化中表现出明显的结构相似性，而较大的模型更接近数据树。超过 87% 的 GPT 输出标记可以通过数据树调用。这些发现可能证实，法学硕士的推理过程更可能是概率模式匹配，而不是形式推理，因为每个模型推理似乎都从数据树中找到具有最大概率的上下文模式。此外，我们对法学硕士中的幻觉、思维链 (CoT) 推理和标记偏差等问题提供了更深入的见解。

Title: Enhancing Talent Employment Insights Through Feature Extraction with LLM Finetuning

• Authors: Karishma Thakrar, Nick Young
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07663
• Pdf URL: https://arxiv.org/pdf/2501.07663
• Copy Paste: [[2501.07663]] Enhancing Talent Employment Insights Through Feature Extraction with LLM Finetuning(https://arxiv.org/abs/2501.07663)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: This paper explores the application of large language models (LLMs) to extract nuanced and complex job features from unstructured job postings. Using a dataset of 1.2 million job postings provided by AdeptID, we developed a robust pipeline to identify and classify variables such as remote work availability, remuneration structures, educational requirements, and work experience preferences. Our methodology combines semantic chunking, retrieval-augmented generation (RAG), and fine-tuning DistilBERT models to overcome the limitations of traditional parsing tools. By leveraging these techniques, we achieved significant improvements in identifying variables often mislabeled or overlooked, such as non-salary-based compensation and inferred remote work categories. We present a comprehensive evaluation of our fine-tuned models and analyze their strengths, limitations, and potential for scaling. This work highlights the promise of LLMs in labor market analytics, providing a foundation for more accurate and actionable insights into job data.
• 摘要：本文探讨了大型语言模型 (LLM) 在从非结构化招聘信息中提取细微而复杂的工作特征方面的应用。利用 AdeptID 提供的 120 万个招聘信息数据集，我们开发了一个强大的流程来识别和分类变量，例如远程工作可用性、薪酬结构、教育要求和工作经验偏好。我们的方法结合了语义分块、检索增强生成 (RAG) 和微调 DistilBERT 模型，以克服传统解析工具的局限性。通过利用这些技术，我们在识别经常被错误标记或忽视的变量方面取得了显著的进步，例如非基于工资的薪酬和推断的远程工作类别。我们对微调后的模型进行了全面评估，并分析了它们的优势、局限性和扩展潜力。这项工作突出了 LLM 在劳动力市场分析中的前景，为更准确和可操作的工作数据洞察奠定了基础。

Title: Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles

• Authors: Samia Touileb, Vladislav Mikhailov, Marie Kroka, Lilja Øvrelid, Erik Velldal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07718
• Pdf URL: https://arxiv.org/pdf/2501.07718
• Copy Paste: [[2501.07718]] Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles(https://arxiv.org/abs/2501.07718)
• Keywords: language model, llm
• Abstract: We introduce a dataset of high-quality human-authored summaries of news articles in Norwegian. The dataset is intended for benchmarking the abstractive summarisation capabilities of generative language models. Each document in the dataset is provided with three different candidate gold-standard summaries written by native Norwegian speakers, and all summaries are provided in both of the written variants of Norwegian -- Bokmål and Nynorsk. The paper describes details on the data creation effort as well as an evaluation of existing open LLMs for Norwegian on the dataset. We also provide insights from a manual human evaluation, comparing human-authored to model-generated summaries. Our results indicate that the dataset provides a challenging LLM benchmark for Norwegian summarisation capabilities
• 摘要：我们引入了一个由人工编写的高质量挪威语新闻文章摘要数据集。该数据集旨在对生成语言模型的抽象摘要能力进行基准测试。数据集中的每个文档都提供了由挪威语母语人士编写的三个不同的候选黄金标准摘要，并且所有摘要都以挪威语的两种书面变体（博克马尔语和尼诺斯克语）提供。本文详细介绍了数据创建工作以及对数据集上现有的挪威语开放法学硕士 (LLM) 的评估。我们还提供了人工评估的见解，将人工编写的摘要与模型生成的摘要进行比较。我们的结果表明，该数据集为挪威语摘要能力提供了一个具有挑战性的法学硕士 (LLM) 基准

Title: Entailed Between the Lines: Incorporating Implication into NLI

• Authors: Shreya Havaldar, Hamidreza Alvari, Alex Fabrikant, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07719
• Pdf URL: https://arxiv.org/pdf/2501.07719
• Copy Paste: [[2501.07719]] Entailed Between the Lines: Incorporating Implication into NLI(https://arxiv.org/abs/2501.07719)
• Keywords: llm
• Abstract: Much of human communication depends on implication, conveying meaning beyond literal words to express a wider range of thoughts, intentions, and feelings. For models to better understand and facilitate human communication, they must be responsive to the text's implicit meaning. We focus on Natural Language Inference (NLI), a core tool for many language tasks, and find that state-of-the-art NLI models and datasets struggle to recognize a range of cases where entailment is implied, rather than explicit from the text. We formalize implied entailment as an extension of the NLI task and introduce the Implied NLI dataset (INLI) to help today's LLMs both recognize a broader variety of implied entailments and to distinguish between implicit and explicit entailment. We show how LLMs fine-tuned on INLI understand implied entailment and can generalize this understanding across datasets and domains.
• 摘要：人类交流很大程度上依赖于暗示，传达超越字面意义的含义，以表达更广泛的思想、意图和感受。为了使模型更好地理解和促进人类交流，它们必须响应文本的隐含含义。我们专注于自然语言推理 (NLI)，这是许多语言任务的核心工具，并发现最先进的 NLI 模型和数据集难以识别一系列隐含含义而不是文本中明确含义的情况。我们将隐含含义形式化为 NLI 任务的扩展，并引入隐含 NLI 数据集 (INLI)，以帮助当今的 LLM 识别更广泛的隐含含义并区分隐含和显含含义。我们展示了在 INLI 上微调的 LLM 如何理解隐含含义，并将这种理解推广到数据集和领域。

Title: LLMic: Romanian Foundation Language Model

• Authors: Vlad-Andrei Bădoiu, Mihai-Valentin Dumitru, Alexandru M. Gherghescu, Alexandru Agache, Costin Raiciu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07721
• Pdf URL: https://arxiv.org/pdf/2501.07721
• Copy Paste: [[2501.07721]] LLMic: Romanian Foundation Language Model(https://arxiv.org/abs/2501.07721)
• Keywords: language model, llm
• Abstract: Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks with commercial models leading the way. While open models usually operate at a smaller scale, they maintain competitiveness through specialization and fine-tuning. However, a significant challenge persists: open models often underperform in low-resource languages due to limited representation in the training corpus. In this paper, we present LLMic, a bilingual foundation language model designed specifically for the Romanian Language. We document the complete process of pretraining a foundation model for a low-resource language, including corpus construction, architecture selection, and hyper-parameter optimization. Our evaluation demonstrates that LLMic can be specialized for tasks in the target language, achieving results comparable to other much larger open models. We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks. This opens the path for efficient large-scale processing for the Romanian language community, using the much smaller LLMic model
• 摘要：大型语言模型 (LLM) 的最新进展已在各种任务中展现出卓越的能力，其中商业模型处于领先地位。虽然开放模型通常以较小的规模运行，但它们通过专业化和微调保持竞争力。然而，一个重大挑战仍然存在：由于训练语料库中的代表性有限，开放模型在资源匮乏的语言中往往表现不佳。在本文中，我们介绍了专为罗马尼亚语设计的双语基础语言模型 LLMic。我们记录了为资源匮乏的语言预训练基础模型的完整过程，包括语料库构建、架构选择和超参数优化。我们的评估表明，LLMic 可以专门用于目标语言中的任务，实现与其他更大的开放模型相当的结果。我们表明，在初始预训练阶段之后对 LLMic 进行语言翻译微调的效果优于英语到罗马尼亚语翻译任务中的现有解决方案。这为罗马尼亚语社区使用小得多的 LLMic 模型进行高效的大规模处理开辟了道路

Title: Advancing Student Writing Through Automated Syntax Feedback

• Authors: Kamyar Zeinalipour, Mehak Mehak, Fatemeh Parsamotamed, Marco Maggini, Marco Gori
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07740
• Pdf URL: https://arxiv.org/pdf/2501.07740
• Copy Paste: [[2501.07740]] Advancing Student Writing Through Automated Syntax Feedback(https://arxiv.org/abs/2501.07740)
• Keywords: language model, gpt, llm, chat
• Abstract: This study underscores the pivotal role of syntax feedback in augmenting the syntactic proficiency of students. Recognizing the challenges faced by learners in mastering syntactic nuances, we introduce a specialized dataset named Essay-Syntax-Instruct designed to enhance the understanding and application of English syntax among these students. Leveraging the capabilities of Large Language Models (LLMs) such as GPT3.5-Turbo, Llama-2-7b-chat-hf, Llama-2-13b-chat-hf, and Mistral-7B-Instruct-v0.2, this work embarks on a comprehensive fine-tuning process tailored to the syntax improvement task. Through meticulous evaluation, we demonstrate that the fine-tuned LLMs exhibit a marked improvement in addressing syntax-related challenges, thereby serving as a potent tool for students to identify and rectify their syntactic errors. The findings not only highlight the effectiveness of the proposed dataset in elevating the performance of LLMs for syntax enhancement but also illuminate a promising path for utilizing advanced language models to support language acquisition efforts. This research contributes to the broader field of language learning technology by showcasing the potential of LLMs in facilitating the linguistic development of Students.
• 摘要：本研究强调了句法反馈在提高学生句法能力方面的关键作用。认识到学习者在掌握句法细微差别方面面临的挑战，我们引入了一个名为 Essay-Syntax-Instruct 的专门数据集，旨在提高这些学生对英语句法的理解和应用。利用 GPT3.5-Turbo、Llama-2-7b-chat-hf、Llama-2-13b-chat-hf 和 Mistral-7B-Instruct-v0.2 等大型语言模型 (LLM) 的功能，这项工作开始了针对句法改进任务的全面微调过程。通过细致的评估，我们证明经过微调的 LLM 在解决与语法相关的挑战方面表现出显着的进步，从而成为学生识别和纠正句法错误的有力工具。研究结果不仅凸显了所提出的数据集在提升 LLM 语法增强性能方面的有效性，还阐明了利用高级语言模型支持语言习得努力的有前途的途径。这项研究通过展示 LLM 在促进学生语言发展方面的潜力，为更广泛的语言学习技术领域做出了贡献。

Title: Large Language Models for Knowledge Graph Embedding Techniques, Methods, and Challenges: A Survey

• Authors: Bingchen Liu, Xin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07766
• Pdf URL: https://arxiv.org/pdf/2501.07766
• Copy Paste: [[2501.07766]] Large Language Models for Knowledge Graph Embedding Techniques, Methods, and Challenges: A Survey(https://arxiv.org/abs/2501.07766)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have attracted a lot of attention in various fields due to their superior performance, aiming to train hundreds of millions or more parameters on large amounts of text data to understand and generate natural language. As the superior performance of LLMs becomes apparent, they are increasingly being applied to knowledge graph embedding (KGE) related tasks to improve the processing results. As a deep learning model in the field of Natural Language Processing (NLP), it learns a large amount of textual data to predict the next word or generate content related to a given text. However, LLMs have recently been invoked to varying degrees in different types of KGE related scenarios such as multi-modal KGE and open KGE according to their task characteristics. In this paper, we investigate a wide range of approaches for performing LLMs-related tasks in different types of KGE scenarios. To better compare the various approaches, we summarize each KGE scenario in a classification. In addition to the categorization methods, we provide a tabular overview of the methods and their source code links for a more direct comparison. In the article we also discuss the applications in which the methods are mainly used and suggest several forward-looking directions for the development of this new research area.
• 摘要：大型语言模型 (LLM) 因其卓越的性能而受到各个领域的广泛关注，旨在在大量文本数据上训练数亿或更多参数以理解和生成自然语言。随着 LLM 的卓越性能逐渐显现，它们越来越多地被应用于知识图谱嵌入 (KGE) 相关任务以改善处理结果。作为自然语言处理 (NLP) 领域的深度学习模型，它学习大量文本数据来预测下一个单词或生成与给定文本相关的内容。然而，最近 LLM 已根据其任务特征在不同类型的 KGE 相关场景（如多模态 KGE 和开放 KGE）中被不同程度地调用。在本文中，我们研究了在不同类型的 KGE 场景中执行 LLM 相关任务的各种方法。为了更好地比较各种方法，我们将每个 KGE 场景总结为一个分类。除了分类方法之外，我们还提供了方法的表格概述及其源代码链接，以便进行更直接的比较。在文章中我们还讨论了这些方法的主要应用，并为这一新研究领域的发展提出了几个前瞻性的方向。

Title: A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models

• Authors: Kaustubh D. Dhole
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07818
• Pdf URL: https://arxiv.org/pdf/2501.07818
• Copy Paste: [[2501.07818]] A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models(https://arxiv.org/abs/2501.07818)
• Keywords: language model
• Abstract: Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream performance. We investigate the impact of freezing the decoder in a multi-task setup comprising diverse natural language tasks, aiming to reduce deployment overhead and enhance portability to novel tasks. Our experiments, conducted by fine-tuning both individual and multi-task setups on the AlexaTM model, reveal that freezing decoders is highly effective for tasks with natural language outputs and mitigates catastrophic forgetting in multilingual tasks. However, we find that pairing frozen decoders with a larger model can effectively maintain or even enhance performance in structured and QA tasks, making it a viable strategy for a broader range of task types.
• 摘要：在参数高效的微调方法中，冻结已成为一种流行的策略，用于加快训练速度、减少灾难性遗忘并提高下游性能。我们研究了在包含各种自然语言任务的多任务设置中冻结解码器的影响，旨在减少部署开销并增强向新任务的可移植性。我们的实验通过微调 AlexaTM 模型上的单个和多任务设置进行，结果表明冻结解码器对于具有自然语言输出的任务非常有效，并且可以减轻多语言任务中的灾难性遗忘。然而，我们发现将冻结解码器与更大的模型配对可以有效地保持甚至提高结构化和 QA 任务的性能，使其成为更广泛任务类型的可行策略。

Title: Real-time Verification and Refinement of Language Model Text Generation

• Authors: Joonho Ko, Jinheon Baek, Sung Ju Hwang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07824
• Pdf URL: https://arxiv.org/pdf/2501.07824
• Copy Paste: [[2501.07824]] Real-time Verification and Refinement of Language Model Text Generation(https://arxiv.org/abs/2501.07824)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown remarkable performance across a wide range of natural language tasks. However, a critical challenge remains in that they sometimes generate factually incorrect answers. To address this, while many previous work has focused on identifying errors in their generation and further refining them, they are slow in deployment since they are designed to verify the response from LLMs only after their entire generation (from the first to last tokens) is done. Further, we observe that once LLMs generate incorrect tokens early on, there is a higher likelihood that subsequent tokens will also be factually incorrect. To this end, in this work, we propose Streaming-VR (Streaming Verification and Refinement), a novel approach designed to enhance the efficiency of verification and refinement of LLM outputs. Specifically, the proposed Streaming-VR enables on-the-fly verification and correction of tokens as they are being generated, similar to a streaming process, ensuring that each subset of tokens is checked and refined in real-time by another LLM as the LLM constructs its response. Through comprehensive evaluations on multiple datasets, we demonstrate that our approach not only enhances the factual accuracy of LLMs, but also offers a more efficient solution compared to prior refinement methods.
• 摘要：大型语言模型 (LLM) 在广泛的自然语言任务中表现出色。然而，一个关键的挑战仍然存在，即它们有时会生成事实上不正确的答案。为了解决这个问题，虽然许多以前的工作都集中在识别生成过程中的错误并进一步改进它们，但它们的部署速度很慢，因为它们被设计为仅在整个生成（从第一个到最后一个标记）完成后才验证来自 LLM 的响应。此外，我们观察到，一旦 LLM 在早期生成了错误的标记，后续标记也更有可能在事实上不正确。为此，在这项工作中，我们提出了 Streaming-VR（流式验证和改进），这是一种旨在提高 LLM 输出验证和改进效率的新方法。具体而言，所提出的 Streaming-VR 能够在生成标记时对其进行即时验证和更正，类似于流式处理，确保在 LLM 构建其响应时，每个标记子集都由另一个 LLM 实时检查和改进。通过对多个数据集的全面评估，我们证明我们的方法不仅提高了 LLM 的事实准确性，而且与之前的细化方法相比提供了更有效的解决方案。

Title: Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoning

• Authors: Haoyu Han, Yaochen Xie, Hui Liu, Xianfeng Tang, Sreyashi Nag, William Headden, Hui Liu, Yang Li, Chen Luo, Shuiwang Ji, Qi He, Jiliang Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07845
• Pdf URL: https://arxiv.org/pdf/2501.07845
• Copy Paste: [[2501.07845]] Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoning(https://arxiv.org/abs/2501.07845)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable success across a wide range of tasks; however, they still encounter challenges in reasoning tasks that require understanding and inferring relationships between distinct pieces of information within text sequences. This challenge is particularly pronounced in tasks involving multi-step processes, such as logical reasoning and multi-hop question answering, where understanding implicit relationships between entities and leveraging multi-hop connections in the given context are crucial. Graphs, as fundamental data structures, explicitly represent pairwise relationships between entities, thereby offering the potential to enhance LLMs' reasoning capabilities. External graphs have proven effective in supporting LLMs across multiple tasks. However, in many reasoning tasks, no pre-existing graph structure is provided. Can we structure implicit knowledge derived from context into graphs to assist LLMs in reasoning? In this paper, we propose Reasoning with Graphs (RwG) by first constructing explicit graphs from the context and then leveraging these graphs to enhance LLM reasoning performance on reasoning tasks. Extensive experiments demonstrate the effectiveness of the proposed method in improving both logical reasoning and multi-hop question answering tasks.
• 摘要：大型语言模型 (LLM) 在各种任务中都取得了显著的成功；然而，它们在推理任务中仍然面临挑战，这些任务需要理解和推断文本序列中不同信息之间的关系。这一挑战在涉及多步骤过程的任务中尤为明显，例如逻辑推理和多跳问答，在这些任务中，理解实体之间的隐式关系和利用给定上下文中的多跳连接至关重要。图作为基本数据结构，明确表示实体之间的成对关系，从而有可能增强 LLM 的推理能力。外部图已被证明可有效支持跨多个任务的 LLM。然而，在许多推理任务中，没有提供预先存在的图结构。我们能否将从上下文中获得的隐式知识构造成图来协助 LLM 进行推理？在本文中，我们提出了使用图进行推理 (RwG)，首先从上下文构建显式图，然后利用这些图来增强 LLM 在推理任务上的推理性能。大量实验证明了所提出的方法在改善逻辑推理和多跳问答任务方面的有效性。

Title: Optimizing Language Models for Grammatical Acceptability: A Comparative Study of Fine-Tuning Techniques

• Authors: Shobhit Ratan, Farley Knight, Ghada Jerfel, Sze Chung Ho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07853
• Pdf URL: https://arxiv.org/pdf/2501.07853
• Copy Paste: [[2501.07853]] Optimizing Language Models for Grammatical Acceptability: A Comparative Study of Fine-Tuning Techniques(https://arxiv.org/abs/2501.07853)
• Keywords: language model, llm
• Abstract: This study explores the fine-tuning (FT) of the Open Pre-trained Transformer (OPT-125M) for grammatical acceptability tasks using the CoLA dataset. By comparing Vanilla-Fine-Tuning (VFT), Pattern-Based-Fine-Tuning (PBFT), and Parameter-Efficient Fine-Tuning techniques (PEFT) like Low-Rank Adaptation (LoRA), we demonstrate significant improvements in computational efficiency while maintaining high accuracy. Our experiments reveal that while VFT achieves the highest accuracy (81.2%), LoRA enhancing FT by reducing memory usage and iteration time by more than 50%, and increases accuracy in PBFT case. Context Distillation (CD), though computationally efficient, underperformed with accuracy around 31%. Our findings contribute to democratizing access to large language models (LLM) by reducing computational barriers.
• 摘要：本研究使用 CoLA 数据集探索了开放式预训练 Transformer (OPT-125M) 的微调 (FT)，以完成语法可接受性任务。通过比较 Vanilla 微调 (VFT)、基于模式的微调 (PBFT) 和低秩自适应 (LoRA) 等参数高效微调技术 (PEFT)，我们证明了计算效率的显著提高，同时保持了较高的准确率。我们的实验表明，虽然 VFT 实现了最高的准确率 (81.2%)，但 LoRA 通过将内存使用量和迭代时间减少 50% 以上来增强 FT，并在 PBFT 情况下提高了准确率。上下文蒸馏 (CD) 虽然计算效率高，但准确率较低，约为 31%。我们的研究结果有助于通过减少计算障碍来实现大型语言模型 (LLM) 的民主化访问。

Title: ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding

• Authors: Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Song Yang, Han Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07861
• Pdf URL: https://arxiv.org/pdf/2501.07861
• Copy Paste: [[2501.07861]] ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding(https://arxiv.org/abs/2501.07861)
• Keywords: language model, llm, retrieval-augmented generation, chain-of-thought
• Abstract: Retrieval-Augmented Generation (RAG) systems for Large Language Models (LLMs) hold promise in knowledge-intensive tasks but face limitations in complex multi-step reasoning. While recent methods have integrated RAG with chain-of-thought reasoning or test-time search using Process Reward Models (PRMs), these approaches encounter challenges such as a lack of explanations, bias in PRM training data, early-step bias in PRM scores, and insufficient post-training optimization of reasoning potential. To address these issues, we propose Retrieval-Augmented Reasoning through Trustworthy Process Rewarding (ReARTeR), a framework that enhances RAG systems' reasoning capabilities through post-training and test-time scaling. At test time, ReARTeR introduces Trustworthy Process Rewarding via a Process Reward Model for accurate scalar scoring and a Process Explanation Model (PEM) for generating natural language explanations, enabling step refinement. During post-training, it utilizes Monte Carlo Tree Search guided by Trustworthy Process Rewarding to collect high-quality step-level preference data, optimized through Iterative Preference Optimization. ReARTeR addresses three core challenges: (1) misalignment between PRM and PEM, tackled through off-policy preference learning; (2) bias in PRM training data, mitigated by balanced annotation methods and stronger annotations for challenging examples; and (3) early-step bias in PRM, resolved through a temporal-difference-based look-ahead search strategy. Experimental results on multi-step reasoning benchmarks demonstrate significant improvements, underscoring ReARTeR's potential to advance the reasoning capabilities of RAG systems.
• 摘要：大型语言模型 (LLM) 的检索增强生成 (RAG) 系统在知识密集型任务中大有可为，但在复杂的多步骤推理中却存在局限性。虽然最近的方法已将 RAG 与使用过程奖励模型 (PRM) 的思路链推理或测试时搜索相结合，但这些方法面临着诸多挑战，例如缺乏解释、PRM 训练数据存在偏差、PRM 分数的早期步骤偏差以及推理潜力的训练后优化不足。为了解决这些问题，我们提出了通过可信过程奖励 (ReARTeR) 进行检索增强推理，这是一个通过训练后和测试时扩展来增强 RAG 系统推理能力的框架。在测试时，ReARTeR 通过过程奖励模型引入可信过程奖励，以实现准确的标量评分，并引入过程解释模型 (PEM) 来生成自然语言解释，从而实现步骤细化。在训练后，它利用由可信过程奖励引导的蒙特卡洛树搜索来收集高质量的步骤级偏好数据，并通过迭代偏好优化进行优化。ReARTeR 解决了三个核心挑战：（1）PRM 和 PEM 之间的不一致，通过离线策略偏好学习解决；（2）PRM 训练数据中的偏差，通过平衡的注释方法和针对具有挑战性的示例的更强的注释来缓解；（3）PRM 中的早期步骤偏差，通过基于时间差异的前瞻搜索策略解决。多步推理基准的实验结果显示出显着的改进，凸显了 ReARTeR 提升 RAG 系统推理能力的潜力。

Title: GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism

• Authors: Chen Tang, Bo Lv, Zifan Zheng, Bohao Yang, Kun Zhao, Ning Liao, Xiaoxing Wang, Feiyu Xiong, Zhiyu Li, Nayu Liu, Jingchi Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07890
• Pdf URL: https://arxiv.org/pdf/2501.07890
• Copy Paste: [[2501.07890]] GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism(https://arxiv.org/abs/2501.07890)
• Keywords: language model
• Abstract: Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models.
• 摘要：传统的混合专家 (MoE) 网络受益于利用多个较小的专家模型，而不是单个大型网络。然而，这些专家通常独立运作，因此，将这些模型互连是否可以提高 MoE 网络的性能仍是一个悬而未决的问题。为此，我们引入了 GRAPHMOE，这是一种新方法，旨在通过在伪 GraphMoE 网络上构建的自我重新思考机制来增强语言模型的认知深度。GRAPHMOE 采用循环路由策略来模拟迭代思考步骤，从而促进专家节点之间的信息流动。我们使用低秩自适应技术 (LoRA) 实现 GRAPHMOE 架构，并对各种基准数据集进行了广泛的实验。实验结果表明，GRAPHMOE 优于其他基于 LoRA 的模型，实现了最先进的 (SOTA) 性能。此外，本研究探索了一种新颖的循环路由策略，可能会激发进一步增强语言模型推理能力的进步。

Title: TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

• Authors: Yao Liang, Yuwei Wang, Yi Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08008
• Pdf URL: https://arxiv.org/pdf/2501.08008
• Copy Paste: [[2501.08008]] TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/2501.08008)
• Keywords: language model, llm
• Abstract: The fine-tuning of Large Language Models (LLMs) is pivotal for achieving optimal performance across diverse downstream tasks. However, while full fine-tuning delivers superior results, it entails significant computational and resource costs. Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, address these challenges by reducing the number of trainable parameters, but they often struggle with rank adjustment efficiency and task-specific adaptability. We propose Triangular Adaptive Low-Rank Adaptation (TriAdaptLoRA), a novel PEFT framework inspired by neuroscience principles, which dynamically optimizes the allocation of trainable parameters. TriAdaptLoRA introduces three key innovations: 1) a triangular split of transformation matrices into lower and upper triangular components to maximize parameter utilization, 2) a parameter importance metric based on normalized Frobenius norms for efficient adaptation, and 3) an adaptive rank-growth strategy governed by dynamic thresholds, allowing flexible parameter allocation across training steps. Experiments conducted on a variety of natural language understanding and generation tasks demonstrate that TriAdaptLoRA consistently outperforms existing PEFT methods. It achieves superior performance, enhanced stability, and reduced computational overhead, particularly under linear threshold-driven rank growth. These results highlight its efficacy as a scalable and resource-efficient solution for fine-tuning LLMs.
• 摘要：大型语言模型 (LLM) 的微调对于实现各种下游任务的最佳性能至关重要。然而，虽然完全微调可以提供出色的结果，但它需要大量的计算和资源成本。参数高效微调 (PEFT) 方法（例如 LoRA）通过减少可训练参数的数量来解决这些挑战，但它们通常在等级调整效率和任务特定适应性方面存在困难。我们提出了三角自适应低秩适应 (TriAdaptLoRA)，这是一种受神经科学原理启发的新型 PEFT 框架，可动态优化可训练参数的分配。TriAdaptLoRA 引入了三项关键创新：1) 将变换矩阵三角分割为下三角和上三角分量以最大限度地提高参数利用率，2) 基于规范化 Frobenius 范数的参数重要性度量，以实现高效适应，3) 由动态阈值控制的自适应等级增长策略，允许在训练步骤中灵活分配参数。在各种自然语言理解和生成任务上进行的实验表明，TriAdaptLoRA 始终优于现有的 PEFT 方法。它实现了卓越的性能、增强的稳定性和降低的计算开销，特别是在线性阈值驱动的排名增长下。这些结果凸显了其作为微调 LLM 的可扩展且资源高效的解决方案的有效性。

Title: Exploring Narrative Clustering in Large Language Models: A Layerwise Analysis of BERT

• Authors: Awritrojit Banerjee, Achim Schilling, Patrick Krauss
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08053
• Pdf URL: https://arxiv.org/pdf/2501.08053
• Copy Paste: [[2501.08053]] Exploring Narrative Clustering in Large Language Models: A Layerwise Analysis of BERT(https://arxiv.org/abs/2501.08053)
• Keywords: language model, gpt
• Abstract: This study investigates the internal mechanisms of BERT, a transformer-based large language model, with a focus on its ability to cluster narrative content and authorial style across its layers. Using a dataset of narratives developed via GPT-4, featuring diverse semantic content and stylistic variations, we analyze BERT's layerwise activations to uncover patterns of localized neural processing. Through dimensionality reduction techniques such as Principal Component Analysis (PCA) and Multidimensional Scaling (MDS), we reveal that BERT exhibits strong clustering based on narrative content in its later layers, with progressively compact and distinct clusters. While strong stylistic clustering might occur when narratives are rephrased into different text types (e.g., fables, sci-fi, kids' stories), minimal clustering is observed for authorial style specific to individual writers. These findings highlight BERT's prioritization of semantic content over stylistic features, offering insights into its representational capabilities and processing hierarchy. This study contributes to understanding how transformer models like BERT encode linguistic information, paving the way for future interdisciplinary research in artificial intelligence and cognitive neuroscience.
• 摘要：本研究调查了基于 Transformer 的大型语言模型 BERT 的内部机制，重点研究了其跨层聚类叙事内容和作者风格的能力。使用通过 GPT-4 开发的叙事数据集，该数据集具有多样化的语义内容和风格变化，我们分析了 BERT 的逐层激活，以揭示局部神经处理的模式。通过主成分分析 (PCA) 和多维缩放 (MDS) 等降维技术，我们发现 BERT 在后面的层中表现出基于叙事内容的强聚类，并且聚类逐渐紧凑且独特。虽然当叙事被改写成不同的文本类型（例如寓言、科幻小说、儿童故事）时可能会出现强烈的风格聚类，但对于特定于个别作者的作者风格，观察到的聚类最少。这些发现突出了 BERT 优先考虑语义内容而不是风格特征，从而深入了解了其表征能力和处理层次结构。这项研究有助于理解 BERT 等 Transformer 模型如何编码语言信息，为未来人工智能和认知神经科学的跨学科研究铺平了道路。

Title: Consistency of Responses and Continuations Generated by Large Language Models on Social Media

• Authors: Wenlu Fan, Yuqi Zhu, Chenyang Wang, Bin Wang, Wentao Xu
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.08102
• Pdf URL: https://arxiv.org/pdf/2501.08102
• Copy Paste: [[2501.08102]] Consistency of Responses and Continuations Generated by Large Language Models on Social Media(https://arxiv.org/abs/2501.08102)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities in text generation, yet their emotional consistency and semantic coherence in social media contexts remain insufficiently understood. This study investigates how LLMs handle emotional content and maintain semantic relationships through continuation and response tasks using two open-source models: Gemma and Llama. By analyzing climate change discussions from Twitter and Reddit, we examine emotional transitions, intensity patterns, and semantic similarity between human-authored and LLM-generated content. Our findings reveal that while both models maintain high semantic coherence, they exhibit distinct emotional patterns: Gemma shows a tendency toward negative emotion amplification, particularly anger, while maintaining certain positive emotions like optimism. Llama demonstrates superior emotional preservation across a broader spectrum of affects. Both models systematically generate responses with attenuated emotional intensity compared to human-authored content and show a bias toward positive emotions in response tasks. Additionally, both models maintain strong semantic similarity with original texts, though performance varies between continuation and response tasks. These findings provide insights into LLMs' emotional and semantic processing capabilities, with implications for their deployment in social media contexts and human-AI interaction design.
• 摘要：大型语言模型 (LLM) 在文本生成方面表现出卓越的能力，但它们在社交媒体环境中的情感一致性和语义连贯性仍未得到充分理解。本研究使用两个开源模型：Gemma 和 Llama，研究了 LLM 如何处理情感内容并通过延续和响应任务维持语义关系。通过分析 Twitter 和 Reddit 上的气候变化讨论，我们研究了人类创作的内容和 LLM 生成的内容之间的情感转变、强度模式和语义相似性。我们的研究结果表明，虽然这两种模型都保持了高度的语义连贯性，但它们表现出不同的情感模式：Gemma 表现出放大负面情绪的倾向，尤其是愤怒，同时保持了某些积极情绪，如乐观。Llama 在更广泛的情感范围内表现出卓越的情感保存能力。与人类创作的内容相比，这两种模型系统地生成情感强度减弱的响应，并在响应任务中表现出对积极情绪的偏向。此外，这两种模型都与原始文本保持了很强的语义相似性，尽管延续任务和响应任务之间的性能有所不同。这些发现深入了解了 LLM 的情感和语义处理能力，对其在社交媒体环境和人机交互设计中的部署具有重要意义。

Title: Refusal Behavior in Large Language Models: A Nonlinear Perspective

• Authors: Fabian Hildebrandt, Andreas Maier, Patrick Krauss, Achim Schilling
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08145
• Pdf URL: https://arxiv.org/pdf/2501.08145
• Copy Paste: [[2501.08145]] Refusal Behavior in Large Language Models: A Nonlinear Perspective(https://arxiv.org/abs/2501.08145)
• Keywords: language model, llm, prompt
• Abstract: Refusal behavior in large language models (LLMs) enables them to decline responding to harmful, unethical, or inappropriate prompts, ensuring alignment with ethical standards. This paper investigates refusal behavior across six LLMs from three architectural families. We challenge the assumption of refusal as a linear phenomenon by employing dimensionality reduction techniques, including PCA, t-SNE, and UMAP. Our results reveal that refusal mechanisms exhibit nonlinear, multidimensional characteristics that vary by model architecture and layer. These findings highlight the need for nonlinear interpretability to improve alignment research and inform safer AI deployment strategies.
• 摘要：大型语言模型 (LLM) 中的拒绝行为使它们能够拒绝响应有害、不道德或不适当的提示，从而确保符合道德标准。本文研究了来自三个架构系列的六个 LLM 中的拒绝行为。我们通过采用降维技术（包括 PCA、t-SNE 和 UMAP）挑战了拒绝作为线性现象的假设。我们的结果表明，拒绝机制表现出非线性、多维的特征，这些特征因模型架构和层而异。这些发现强调了非线性可解释性的必要性，以改进对齐研究并为更安全的 AI 部署策略提供信息。

Title: Potential and Perils of Large Language Models as Judges of Unstructured Textual Data

• Authors: Rewina Bedemariam, Natalie Perez, Sreyoshi Bhaduri, Satya Kapoor, Alex Gil, Elizabeth Conjar, Ikkei Itoku, David Theil, Aman Chadha, Naumaan Nayyar
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.08167
• Pdf URL: https://arxiv.org/pdf/2501.08167
• Copy Paste: [[2501.08167]] Potential and Perils of Large Language Models as Judges of Unstructured Textual Data(https://arxiv.org/abs/2501.08167)
• Keywords: language model, llm
• Abstract: Rapid advancements in large language models have unlocked remarkable capabilities when it comes to processing and summarizing unstructured text data. This has implications for the analysis of rich, open-ended datasets, such as survey responses, where LLMs hold the promise of efficiently distilling key themes and sentiments. However, as organizations increasingly turn to these powerful AI systems to make sense of textual feedback, a critical question arises, can we trust LLMs to accurately represent the perspectives contained within these text based datasets? While LLMs excel at generating human-like summaries, there is a risk that their outputs may inadvertently diverge from the true substance of the original responses. Discrepancies between the LLM-generated outputs and the actual themes present in the data could lead to flawed decision-making, with far-reaching consequences for organizations. This research investigates the effectiveness of LLMs as judge models to evaluate the thematic alignment of summaries generated by other LLMs. We utilized an Anthropic Claude model to generate thematic summaries from open-ended survey responses, with Amazon's Titan Express, Nova Pro, and Meta's Llama serving as LLM judges. The LLM-as-judge approach was compared to human evaluations using Cohen's kappa, Spearman's rho, and Krippendorff's alpha, validating a scalable alternative to traditional human centric evaluation methods. Our findings reveal that while LLMs as judges offer a scalable solution comparable to human raters, humans may still excel at detecting subtle, context-specific nuances. This research contributes to the growing body of knowledge on AI assisted text analysis. We discuss limitations and provide recommendations for future research, emphasizing the need for careful consideration when generalizing LLM judge models across various contexts and use cases.
• 摘要：大型语言模型的快速发展在处理和总结非结构化文本数据方面释放了非凡的能力。这对于分析丰富的开放式数据集（例如调查回复）具有重要意义，其中 LLM 有望有效提炼关键主题和情绪。然而，随着组织越来越多地转向这些强大的 AI 系统来理解文本反馈，一个关键问题出现了，我们能否相信 LLM 能够准确地代表这些基于文本的数据集中包含的观点？虽然 LLM 擅长生成类似人类的摘要，但它们的输出可能会无意中偏离原始响应的真实内容。LLM 生成的输出与数据中存在的实际主题之间的差异可能会导致错误的决策，对组织产生深远的影响。本研究调查了 LLM 作为判断模型的有效性，以评估其他 LLM 生成的摘要的主题一致性。我们利用 Anthropic Claude 模型从开放式调查回复中生成主题摘要，并使用亚马逊的 Titan Express、Nova Pro 和 Meta 的 Llama 担任 LLM 评委。使用 Cohen 的 kappa、Spearman 的 rho 和 Krippendorff 的 alpha 将 LLM 评委方法与人工评估进行了比较，验证了传统以人为本的评估方法的可扩展替代方案。我们的研究结果表明，虽然 LLM 作为评委提供了与人类评分者相当的可扩展解决方案，但人类可能仍然擅长检测微妙的、特定于上下文的细微差别。这项研究为人工智能辅助文本分析知识体系的不断增长做出了贡献。我们讨论了局限性并为未来的研究提供了建议，强调在将 LLM 评委模型推广到各种上下文和用例时需要仔细考虑。

Title: A Multi-Modal AI Copilot for Single-Cell Analysis with Instruction Following

• Authors: Yin Fang, Xinle Deng, Kangwei Liu, Ningyu Zhang, Jingyang Qian, Penghui Yang, Xiaohui Fan, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.CE, cs.HC, cs.LG, q-bio.CB
• Abstract URL: https://arxiv.org/abs/2501.08187
• Pdf URL: https://arxiv.org/pdf/2501.08187
• Copy Paste: [[2501.08187]] A Multi-Modal AI Copilot for Single-Cell Analysis with Instruction Following(https://arxiv.org/abs/2501.08187)
• Keywords: language model
• Abstract: Large language models excel at interpreting complex natural language instructions, enabling them to perform a wide range of tasks. In the life sciences, single-cell RNA sequencing (scRNA-seq) data serves as the "language of cellular biology", capturing intricate gene expression patterns at the single-cell level. However, interacting with this "language" through conventional tools is often inefficient and unintuitive, posing challenges for researchers. To address these limitations, we present InstructCell, a multi-modal AI copilot that leverages natural language as a medium for more direct and flexible single-cell analysis. We construct a comprehensive multi-modal instruction dataset that pairs text-based instructions with scRNA-seq profiles from diverse tissues and species. Building on this, we develop a multi-modal cell language architecture capable of simultaneously interpreting and processing both modalities. InstructCell empowers researchers to accomplish critical tasks-such as cell type annotation, conditional pseudo-cell generation, and drug sensitivity prediction-using straightforward natural language commands. Extensive evaluations demonstrate that InstructCell consistently meets or exceeds the performance of existing single-cell foundation models, while adapting to diverse experimental conditions. More importantly, InstructCell provides an accessible and intuitive tool for exploring complex single-cell data, lowering technical barriers and enabling deeper biological insights.
• 摘要：大型语言模型擅长解释复杂的自然语言指令，使它们能够执行广泛的任务。在生命科学中，单细胞 RNA 测序 (scRNA-seq) 数据充当“细胞生物学语言”，在单细胞水平上捕获复杂的基因表达模式。然而，通过传统工具与这种“语言”交互通常效率低下且不直观，给研究人员带来了挑战。为了解决这些限制，我们提出了 InstructCell，这是一种多模态 AI 副驾驶，它利用自然语言作为更直接、更灵活的单细胞分析媒介。我们构建了一个全面的多模态指令数据集，将基于文本的指令与来自不同组织和物种的 scRNA-seq 配置文件配对。在此基础上，我们开发了一种多模态细胞语言架构，能够同时解释和处理这两种模态。InstructCell 使研究人员能够使用简单的自然语言命令完成关键任务，例如细胞类型注释、条件伪细胞生成和药物敏感性预测。大量评估表明，InstructCell 始终达到或超过现有单细胞基础模型的性能，同时适应不同的实验条件。更重要的是，InstructCell 提供了一种易于使用且直观的工具来探索复杂的单细胞数据，降低技术门槛并实现更深入的生物学洞察。

Title: OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training

• Authors: Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, Ji Pei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08197
• Pdf URL: https://arxiv.org/pdf/2501.08197
• Copy Paste: [[2501.08197]] OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training(https://arxiv.org/abs/2501.08197)
• Keywords: language model, llm, chat
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, but their success heavily relies on the quality of pretraining corpora. For Chinese LLMs, the scarcity of high-quality Chinese datasets presents a significant challenge, often limiting their performance. To address this issue, we propose the OpenCSG Chinese Corpus, a series of high-quality datasets specifically designed for LLM pretraining, post-training, and fine-tuning. This corpus includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets focus on filtered, high-quality content derived from diverse Chinese web sources; Cosmopedia-chinese provides synthetic, textbook-style data for knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its high-quality text, diverse coverage across domains, and scalable, reproducible data curation processes. Additionally, we conducted extensive experimental analyses, including evaluations on smaller parameter models, which demonstrated significant performance improvements in tasks such as C-Eval, showcasing the effectiveness of the corpus for training Chinese LLMs.
• 摘要：大型语言模型 (LLM) 已展现出卓越的能力，但它们的成功在很大程度上依赖于预训练语料库的质量。对于中文 LLM 来说，高质量中文数据集的稀缺是一大挑战，往往会限制其性能。为了解决这个问题，我们提出了 OpenCSG 中文语料库，这是一系列专为 LLM 预训练、后训练和微调而设计的高质量数据集。该语料库包括 Fineweb-edu-chinese、Fineweb-edu-chinese-v2、Cosmopedia-chinese 和 Smoltalk-chinese，每个语料库都有不同的特点：Fineweb-edu 数据集专注于从各种中文网络资源中筛选出的高质量内容；Cosmopedia-chinese 提供用于知识密集型训练的合成教科书式数据；Smoltalk-chinese 强调风格多样的聊天格式数据。OpenCSG 中文语料库的特点是文本质量高、领域覆盖面广、数据管理流程可扩展且可重复。此外，我们还进行了大量的实验分析，包括对较小参数模型的评估，结果表明，该语料库在 C-Eval 等任务中取得了显著的性能提升，展示了该语料库对于训练中文 LLM 的有效性。

Title: ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving

• Authors: Zain Ul Abedin, Shahzeb Qamar, Lucie Flek, Akbar Karimi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08203
• Pdf URL: https://arxiv.org/pdf/2501.08203
• Copy Paste: [[2501.08203]] ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving(https://arxiv.org/abs/2501.08203)
• Keywords: language model, llm, prompt
• Abstract: While Large Language Models (LLMs) have shown impressive capabilities in math problem-solving tasks, their robustness to noisy inputs is not well-studied. In this work, we propose ArithmAttack to examine how robust the LLMs are when they encounter noisy prompts that contain extra noise in the form of punctuation marks. While being easy to implement, ArithmAttack does not cause any information loss since words are not added or deleted from the context. We evaluate the robustness of seven LLMs, including LLama3, Mistral, and Mathstral, on noisy GSM8K and MultiArith datasets. Our experiments suggest that all the studied models show vulnerability to such noise, with more noise leading to poorer performances.
• 摘要：虽然大型语言模型 (LLM) 在数学问题解决任务中表现出了令人印象深刻的能力，但它们对噪声输入的鲁棒性尚未得到充分研究。在这项工作中，我们提出了 ArithmAttack 来检查 LLM 在遇到包含标点符号形式的额外噪声的噪声提示时的鲁棒性。ArithmAttack 虽然易于实现，但不会导致任何信息丢失，因为不会在上下文中添加或删除单词。我们在嘈杂的 GSM8K 和 MultiArith 数据集上评估了包括 LLama3、Mistral 和 Mathstral 在内的七个 LLM 的鲁棒性。我们的实验表明，所有研究的模型都表现出对此类噪声的脆弱性，噪声越多，性能越差。

Title: ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems

• Authors: Mohita Chowdhury, Yajie Vera He, Aisling Higham, Ernest Lim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08208
• Pdf URL: https://arxiv.org/pdf/2501.08208
• Copy Paste: [[2501.08208]] ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems(https://arxiv.org/abs/2501.08208)
• Keywords: language model, llm, prompt, retrieval augmented generation, agent
• Abstract: Large Language Models (LLMs) have shown impressive potential in clinical question answering (QA), with Retrieval Augmented Generation (RAG) emerging as a leading approach for ensuring the factual accuracy of model responses. However, current automated RAG metrics perform poorly in clinical and conversational use cases. Using clinical human evaluations of responses is expensive, unscalable, and not conducive to the continuous iterative development of RAG systems. To address these challenges, we introduce ASTRID - an Automated and Scalable TRIaD for evaluating clinical QA systems leveraging RAG - consisting of three metrics: Context Relevance (CR), Refusal Accuracy (RA), and Conversational Faithfulness (CF). Our novel evaluation metric, CF, is designed to better capture the faithfulness of a model's response to the knowledge base without penalising conversational elements. To validate our triad, we curate a dataset of over 200 real-world patient questions posed to an LLM-based QA agent during surgical follow-up for cataract surgery - the highest volume operation in the world - augmented with clinician-selected questions for emergency, clinical, and non-clinical out-of-domain scenarios. We demonstrate that CF can predict human ratings of faithfulness better than existing definitions for conversational use cases. Furthermore, we show that evaluation using our triad consisting of CF, RA, and CR exhibits alignment with clinician assessment for inappropriate, harmful, or unhelpful responses. Finally, using nine different LLMs, we demonstrate that the three metrics can closely agree with human evaluations, highlighting the potential of these metrics for use in LLM-driven automated evaluation pipelines. We also publish the prompts and datasets for these experiments, providing valuable resources for further research and development.
• 摘要：大型语言模型 (LLM) 在临床问答 (QA) 中展现出令人印象深刻的潜力，其中检索增强生成 (RAG) 正在成为确保模型响应事实准确性的主要方法。但是，当前的自动化 RAG 指标在临床和对话用例中表现不佳。使用临床人工评估反应成本高昂、不可扩展，并且不利于 RAG 系统的持续迭代开发。为了应对这些挑战，我们推出了 ASTRID - 一种用于评估利用 RAG 的临床 QA 系统的自动化和可扩展的 TRIaD - 包含三个指标：上下文相关性 (CR)、拒绝准确度 (RA) 和对话忠实度 (CF)。我们新颖的评估指标 CF 旨在更好地捕捉模型对知识库的响应的忠实度，而不会惩罚对话元素。为了验证我们的三元组，我们整理了一个数据集，其中包含 200 多个真实患者问题，这些问题是在白内障手术（世界上手术量最大的手术）的术后随访期间向基于 LLM 的 QA 代理提出的，并增加了临床医生选择的紧急、临床和非临床领域外场景的问题。我们证明 CF 可以比现有的对话用例定义更好地预测人类对忠诚度的评级。此外，我们表明，使用由 CF、RA 和 CR 组成的三元组进行的评估与临床医生对不适当、有害或无益反应的评估一致。最后，使用九种不同的 LLM，我们证明这三个指标可以与人工评估非常吻合，突出了这些指标在 LLM 驱动的自动评估流程中的使用潜力。我们还发布了这些实验的提示和数据集，为进一步的研究和开发提供了宝贵的资源。

Title: Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models

• Authors: Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B. Cohen, Benjamin Han
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08248
• Pdf URL: https://arxiv.org/pdf/2501.08248
• Copy Paste: [[2501.08248]] Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models(https://arxiv.org/abs/2501.08248)
• Keywords: language model, gpt, long context, retrieval-augmented generation
• Abstract: Recent advancements in long-context language models (LCLMs) promise to transform Retrieval-Augmented Generation (RAG) by simplifying pipelines. With their expanded context windows, LCLMs can process entire knowledge bases and perform retrieval and reasoning directly -- a capability we define as In-Context Retrieval and Reasoning (ICR^2). However, existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts. To address this, we introduce ICR^2, a benchmark that evaluates LCLMs in more realistic scenarios by including confounding passages retrieved with strong retrievers. We then propose three methods to enhance LCLM performance: (1) retrieve-then-generate fine-tuning, (2) retrieval-attention-probing, which uses attention heads to filter and de-noise long contexts during decoding, and (3) joint retrieval head training alongside the generation head. Our evaluation of five well-known LCLMs on LOFT and ICR^2 demonstrates significant gains with our best approach applied to Mistral-7B: +17 and +15 points by Exact Match on LOFT, and +13 and +2 points on ICR^2, compared to vanilla RAG and supervised fine-tuning, respectively. It even outperforms GPT-4-Turbo on most tasks despite being a much smaller model.
• 摘要：长上下文语言模型 (LCLM) 的最新进展有望通过简化流程来改变检索增强生成 (RAG)。借助扩展的上下文窗口，LCLM 可以处理整个知识库并直接执行检索和推理 - 我们将这种能力定义为上下文检索和推理 (ICR^2)。然而，现有的基准测试（如 LOFT）通常会通过提供过于简单的上下文而高估 LCLM 的性能。为了解决这个问题，我们引入了 ICR^2，这是一个基准测试，它通过包括使用强检索器检索的混杂段落来评估更现实场景中的 LCLM。然后，我们提出了三种方法来提高 LCLM 的性能：(1) 检索然后生成微调，(2) 检索注意探测，使用注意头在解码过程中过滤和去噪长上下文，以及 (3) 与生成头一起进行联合检索头训练。我们对 LOFT 和 ICR^2 上五个知名 LCLM 的评估表明，将我们的最佳方法应用于 Mistral-7B 后，取得了显著的进步：与 vanilla RAG 和监督微调相比，LOFT 上的 Exact Match 分别提高了 +17 和 +15 分，ICR^2 上的 Exact Match 提高了 +13 和 +2 分。尽管模型小得多，但它在大多数任务上的表现甚至优于 GPT-4-Turbo。

Title: Exploring Robustness of LLMs to Sociodemographically-Conditioned Paraphrasing

• Authors: Pulkit Arora, Akbar Karimi, Lucie Flek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08276
• Pdf URL: https://arxiv.org/pdf/2501.08276
• Copy Paste: [[2501.08276]] Exploring Robustness of LLMs to Sociodemographically-Conditioned Paraphrasing(https://arxiv.org/abs/2501.08276)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have shown impressive performance in various NLP tasks. However, there are concerns about their reliability in different domains of linguistic variations. Many works have proposed robustness evaluation measures for local adversarial attacks, but we need globally robust models unbiased to different language styles. We take a broader approach to explore a wider range of variations across sociodemographic dimensions to perform structured reliability tests on the reasoning capacity of language models. We extend the SocialIQA dataset to create diverse paraphrased sets conditioned on sociodemographic styles. The assessment aims to provide a deeper understanding of LLMs in (a) their capability of generating demographic paraphrases with engineered prompts and (b) their reasoning capabilities in real-world, complex language scenarios. We also explore measures such as perplexity, explainability, and ATOMIC performance of paraphrases for fine-grained reliability analysis of LLMs on these sets. We find that demographic-specific paraphrasing significantly impacts the performance of language models, indicating that the subtleties of language variations remain a significant challenge. The code and dataset will be made available for reproducibility and future research.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务中表现出色。然而，人们担心它们在不同语言变化领域的可靠性。许多研究提出了针对局部对抗攻击的稳健性评估措施，但我们需要对不同语言风格不偏不倚的全局稳健模型。我们采取更广泛的方法来探索社会人口统计学维度的更广泛变化，以对语言模型的推理能力进行结构化可靠性测试。我们扩展了 SocialIQA 数据集，以创建以社会人口统计学风格为条件的多样化释义集。评估旨在更深入地了解 LLM 在 (a) 使用工程提示生成人口统计学释义的能力和 (b) 在现实世界复杂语言场景中的推理能力。我们还探索了释义的困惑度、可解释性和原子性能等指标，以便在这些集合上对 LLM 进行细粒度的可靠性分析。我们发现特定于人口统计学的释义会显著影响语言模型的性能，这表明语言变化的微妙之处仍然是一个重大挑战。该代码和数据集将提供给可重复性和未来的研究。

Title: AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages

• Authors: Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, David Ifeoluwa Adelani, Ibrahim Said Ahmad, Saminu Mohammad Aliyu, Nelson Odhiambo Onyango, Lilian D. A. Wanzare, Samuel Rutunda, Lukman Jibril Aliyu, Esubalew Alemneh, Oumaima Hourrane, Hagos Tesfahun Gebremichael, Elyas Abdi Ismail, Meriem Beloucif, Ebrahim Chekol Jibril, Andiswa Bukula, Rooweither Mabuya, Salomey Osei, Abigail Oppong, Tadesse Destaw Belay, Tadesse Kebede Guge, Tesfa Tegegne Asfaw, Chiamaka Ijeoma Chukwuneke, Paul Röttger, Seid Muhie Yimam, Nedjma Ousidhoum
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08284
• Pdf URL: https://arxiv.org/pdf/2501.08284
• Copy Paste: [[2501.08284]] AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages(https://arxiv.org/abs/2501.08284)
• Keywords: llm
• Abstract: Hate speech and abusive language are global phenomena that need socio-cultural background knowledge to be understood, identified, and moderated. However, in many regions of the Global South, there have been several documented occurrences of (1) absence of moderation and (2) censorship due to the reliance on keyword spotting out of context. Further, high-profile individuals have frequently been at the center of the moderation process, while large and targeted hate speech campaigns against minorities have been overlooked. These limitations are mainly due to the lack of high-quality data in the local languages and the failure to include local communities in the collection, annotation, and moderation processes. To address this issue, we present AfriHate: a multilingual collection of hate speech and abusive language datasets in 15 African languages. Each instance in AfriHate is annotated by native speakers familiar with the local culture. We report the challenges related to the construction of the datasets and present various classification baseline results with and without using LLMs. The datasets, individual annotations, and hate speech and offensive language lexicons are available on this https URL
• 摘要：仇恨言论和辱骂性语言是一种全球现象，需要社会文化背景知识才能理解、识别和缓和。然而，在全球南方的许多地区，有记录显示，由于依赖脱离上下文的关键词识别，存在 (1) 缺乏审核和 (2) 审查的现象。此外，知名人士经常成为审核过程的中心，而针对少数群体的大规模、有针对性的仇恨言论运动却被忽视。这些限制主要是由于缺乏当地语言的高质量数据，以及未能将当地社区纳入收集、注释和审核过程。为了解决这个问题，我们推出了 AfriHate：一个包含 15 种非洲语言的仇恨言论和辱骂性语言数据集的多语言集合。AfriHate 中的每个实例都由熟悉当地文化的母语人士注释。我们报告了与数据集构建相关的挑战，并展示了使用和不使用 LLM 的各种分类基线结果。数据集、个人注释以及仇恨言论和攻击性语言词典可在此 https URL 上找到

Title: HALoGEN: Fantastic LLM Hallucinations and Where to Find Them

• Authors: Abhilasha Ravichander, Shrusti Ghela, David Wadden, Yejin Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08292
• Pdf URL: https://arxiv.org/pdf/2501.08292
• Copy Paste: [[2501.08292]] HALoGEN: Fantastic LLM Hallucinations and Where to Find Them(https://arxiv.org/abs/2501.08292)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Despite their impressive ability to generate high-quality and fluent text, generative large language models (LLMs) also produce hallucinations: statements that are misaligned with established world knowledge or provided input context. However, measuring hallucination can be challenging, as having humans verify model generations on-the-fly is both expensive and time-consuming. In this work, we release HALoGEN, a comprehensive hallucination benchmark consisting of: (1) 10,923 prompts for generative models spanning nine domains including programming, scientific attribution, and summarization, and (2) automatic high-precision verifiers for each use case that decompose LLM generations into atomic units, and verify each unit against a high-quality knowledge source. We use this framework to evaluate ~150,000 generations from 14 language models, finding that even the best-performing models are riddled with hallucinations (sometimes up to 86% of generated atomic facts depending on the domain). We further define a novel error classification for LLM hallucinations based on whether they likely stem from incorrect recollection of training data (Type A errors), or incorrect knowledge in training data (Type B errors), or are fabrication (Type C errors). We hope our framework provides a foundation to enable the principled study of why generative models hallucinate, and advances the development of trustworthy large language models.
• 摘要：尽管生成式大型语言模型 (LLM) 具有生成高质量流畅文本的出色能力，但它们也会产生幻觉：与既定的世界知识或提供的输入上下文不一致的陈述。然而，测量幻觉可能具有挑战性，因为让人类即时验证模型生成既昂贵又耗时。在这项工作中，我们发布了 HALoGEN，这是一个全面的幻觉基准，包括：(1) 10,923 个生成模型提示，涵盖编程、科学归因和摘要等九个领域，以及 (2) 针对每个用例的自动高精度验证器，将 LLM 生成分解为原子单元，并根据高质量知识源验证每个单元。我们使用此框架评估了来自 14 个语言模型的约 150,000 个生成，发现即使是表现最佳的模型也充斥着幻觉（有时生成的原子事实高达 86%，具体取决于领域）。我们进一步定义了一种新的 LLM 幻觉错误分类，基于它们是可能源于对训练数据的错误回忆（A 类错误），还是训练数据中的错误知识（B 类错误），或者是虚构（C 类错误）。我们希望我们的框架能够为研究生成模型产生幻觉的原因奠定基础，并推动开发值得信赖的大型语言模型。

Title: MiniMax-01: Scaling Foundation Models with Lightning Attention

• Authors: MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, Zijia Wu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08313
• Pdf URL: https://arxiv.org/pdf/2501.08313
• Copy Paste: [[2501.08313]] MiniMax-01: Scaling Foundation Models with Lightning Attention(https://arxiv.org/abs/2501.08313)
• Keywords: language model, gpt
• Abstract: We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at this https URL.
• 摘要：我们推出了 MiniMax-01 系列，包括 MiniMax-Text-01 和 MiniMax-VL-01，它们可与顶级模型相媲美，同时在处理更长的上下文方面具有卓越的能力。核心在于闪电注意力及其高效的扩展。为了最大限度地提高计算能力，我们将其与混合专家 (MoE) 相结合，创建一个拥有 32 位专家和 4560 亿个参数的模型，其中每个 token 激活 459 亿个参数。我们为 MoE 和闪电注意力开发了一种优化的并行策略和高效的计算通信重叠技术。这种方法使我们能够在涵盖数百万个 token 的上下文中对具有数千亿个参数的模型进行高效的训练和推理。MiniMax-Text-01 的上下文窗口在训练期间最多可以达到 100 万个 token，在推理期间可以推断出 400 万个 token，而且成本低廉。我们的视觉语言模型 MiniMax-VL-01 是通过对 5120 亿个视觉语言 token 进行持续训练而建立的。在标准和内部基准测试中的实验表明，我们的模型与 GPT-4o 和 Claude-3.5-Sonnet 等最先进的模型的性能相当，同时提供了 20-32 倍长的上下文窗口。我们在此 https URL 上公开发布了 MiniMax-01。

Title: Enhancing Automated Interpretability with Output-Centric Feature Descriptions

• Authors: Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, Mor Geva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08319
• Pdf URL: https://arxiv.org/pdf/2501.08319
• Copy Paste: [[2501.08319]] Enhancing Automated Interpretability with Output-Centric Feature Descriptions(https://arxiv.org/abs/2501.08319)
• Keywords: language model, llm
• Abstract: Automated interpretability pipelines generate natural language descriptions for the concepts represented by features in large language models (LLMs), such as plants or the first word in a sentence. These descriptions are derived using inputs that activate the feature, which may be a dimension or a direction in the model's representation space. However, identifying activating inputs is costly, and the mechanistic role of a feature in model behavior is determined both by how inputs cause a feature to activate and by how feature activation affects outputs. Using steering evaluations, we reveal that current pipelines provide descriptions that fail to capture the causal effect of the feature on outputs. To fix this, we propose efficient, output-centric methods for automatically generating feature descriptions. These methods use the tokens weighted higher after feature stimulation or the highest weight tokens after applying the vocabulary "unembedding" head directly to the feature. Our output-centric descriptions better capture the causal effect of a feature on model outputs than input-centric descriptions, but combining the two leads to the best performance on both input and output evaluations. Lastly, we show that output-centric descriptions can be used to find inputs that activate features previously thought to be "dead".
• 摘要：自动可解释性管道为大型语言模型 (LLM) 中的特征所表示的概念生成自然语言描述，例如植物或句子中的第一个单词。这些描述是使用激活特征的输入得出的，这些输入可能是模型表示空间中的维度或方向。然而，识别激活输入的成本很高，并且特征在模型行为中的机制作用取决于输入如何导致特征激活以及特征激活如何影响输出。通过转向评估，我们发现当前管道提供的描述无法捕捉特征对输出的因果影响。为了解决这个问题，我们提出了有效的、以输出为中心的方法来自动生成特征描述。这些方法使用特征刺激后权重较高的标记或在将词汇表“解嵌”头直接应用于特征后使用权重最高的标记。与以输入为中心的描述相比，我们的以输出为中心的描述更好地捕捉了特征对模型输出的因果影响，但将两者结合起来可以在输入和输出评估中取得最佳性能。最后，我们表明以输出为中心的描述可用于查找激活以前被认为“死亡”的特征的输入。

Title: Exploring Robustness of Multilingual LLMs on Real-World Noisy Data

• Authors: Amirhossein Aliakbarzadeh, Lucie Flek, Akbar Karimi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08322
• Pdf URL: https://arxiv.org/pdf/2501.08322
• Copy Paste: [[2501.08322]] Exploring Robustness of Multilingual LLMs on Real-World Noisy Data(https://arxiv.org/abs/2501.08322)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are trained on Web data that might contain spelling errors made by humans. But do they become robust to similar real-world noise? In this paper, we investigate the effect of real-world spelling mistakes on the performance of 9 language models, with parameters ranging from 0.2B to 13B, in 3 different NLP tasks, namely Natural Language Inference (NLI), Name Entity Recognition (NER), and Intent Classification (IC). We perform our experiments on 6 different languages and build a dictionary of real-world noise for them using the Wikipedia edit history. We show that the performance gap of the studied models on the clean and noisy test data averaged across all the datasets and languages ranges from 2.3 to 4.3 absolute percentage points. In addition, mT5 models, in general, show more robustness compared to BLOOM, Falcon, and BERT-like models. In particular, mT5 (13B), was the most robust on average overall, across the 3 tasks, and in 4 of the 6 languages.
• 摘要：大型语言模型 (LLM) 是在可能包含人为拼写错误的 Web 数据上训练的。但它们能对类似的现实世界噪音保持稳健吗？在本文中，我们研究了现实世界拼写错误对 9 种语言模型性能的影响，这些模型的参数范围从 0.2B 到 13B，涉及 3 个不同的 NLP 任务，即自然语言推理 (NLI)、名称实体识别 (NER) 和意图分类 (IC)。我们对 6 种不同的语言进行了实验，并使用维基百科编辑历史为它们构建了一个现实世界噪音词典。我们表明，所研究的模型在所有数据集和语言的平均干净和嘈杂测试数据上的性能差距在 2.3 到 4.3 绝对个百分点之间。此外，与 BLOOM、Falcon 和 BERT 类模型相比，mT5 模型总体上表现出更高的稳健性。特别是，mT5 (13B) 在 3 项任务中以及在 6 种语言中的 4 种中总体上平均最为稳健。

Title: PokerBench: Training Large Language Models to become Professional Poker Players

• Authors: Richard Zhuang, Akshat Gupta, Richard Yang, Aniket Rahane, Zhengyu Li, Gopala Anumanchipalli
• Subjects: cs.CL, cs.AI, cs.GT
• Abstract URL: https://arxiv.org/abs/2501.08328
• Pdf URL: https://arxiv.org/pdf/2501.08328
• Copy Paste: [[2501.08328]] PokerBench: Training Large Language Models to become Professional Poker Players(https://arxiv.org/abs/2501.08328)
• Keywords: language model, gpt, llm, chat
• Abstract: We introduce PokerBench - a benchmark for evaluating the poker-playing abilities of large language models (LLMs). As LLMs excel in traditional NLP tasks, their application to complex, strategic games like poker poses a new challenge. Poker, an incomplete information game, demands a multitude of skills such as mathematics, reasoning, planning, strategy, and a deep understanding of game theory and human psychology. This makes Poker the ideal next frontier for large language models. PokerBench consists of a comprehensive compilation of 11,000 most important scenarios, split between pre-flop and post-flop play, developed in collaboration with trained poker players. We evaluate prominent models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models, finding that all state-of-the-art LLMs underperform in playing optimal poker. However, after fine-tuning, these models show marked improvements. We validate PokerBench by having models with different scores compete with each other, demonstrating that higher scores on PokerBench lead to higher win rates in actual poker games. Through gameplay between our fine-tuned model and GPT-4, we also identify limitations of simple supervised fine-tuning for learning optimal playing strategy, suggesting the need for more advanced methodologies for effectively training language models to excel in games. PokerBench thus presents a unique benchmark for a quick and reliable evaluation of the poker-playing ability of LLMs as well as a comprehensive benchmark to study the progress of LLMs in complex game-playing scenarios. The dataset and code will be made available at: \url{this https URL}.
• 摘要：我们引入了 PokerBench——用于评估大型语言模型 (LLM) 玩扑克的能力的基准。由于 LLM 在传统的 NLP 任务中表现出色，因此将其应用于扑克等复杂的战略游戏提出了新的挑战。扑克是一种不完全信息游戏，需要多种技能，例如数学、推理、规划、策略以及对博弈论和人类心理学的深刻理解。这使得扑克成为大型语言模型的理想下一个前沿。PokerBench 包含 11,000 个最重要的场景的综合汇编，分为翻牌前和翻牌后游戏，与训练有素的扑克玩家合作开发。我们评估了包括 GPT-4、ChatGPT 3.5 和各种 Llama 和 Gemma 系列模型在内的著名模型，发现所有最先进的 LLM 在玩最佳扑克方面表现不佳。然而，经过微调后，这些模型显示出显着的改进。我们通过让具有不同分数的模型相互竞争来验证 PokerBench，证明 PokerBench 上的分数越高，实际扑克游戏中的获胜率就越高。通过我们微调后的模型与 GPT-4 之间的对战，我们还发现了简单的监督微调在学习最佳游戏策略方面的局限性，这表明需要更先进的方法来有效地训练语言模型以在游戏中脱颖而出。因此，PokerBench 提供了一个独特的基准，可以快速可靠地评估 LLM 的扑克游戏能力，同时也提供了一个全面的基准，可以研究 LLM 在复杂游戏场景中的进展。数据集和代码将在 \url{此 https URL} 处提供。