2025-02-03

Title: Divergent Emotional Patterns in Disinformation on Social Media? An Analysis of Tweets and TikToks about the DANA in Valencia

• Authors: Iván Arcos, Paolo Rosso, Ramón Salaverría
• Subjects: cs.CL, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.18640
• Pdf URL: https://arxiv.org/pdf/2501.18640
• Copy Paste: [[2501.18640]] Divergent Emotional Patterns in Disinformation on Social Media? An Analysis of Tweets and TikToks about the DANA in Valencia(https://arxiv.org/abs/2501.18640)
• Keywords: language model, gpt
• Abstract: This study investigates the dissemination of disinformation on social media platforms during the DANA event (DANA is a Spanish acronym for Depresion Aislada en Niveles Altos, translating to high-altitude isolated depression) that resulted in extremely heavy rainfall and devastating floods in Valencia, Spain, on October 29, 2024. We created a novel dataset of 650 TikTok and X posts, which was manually annotated to differentiate between disinformation and trustworthy content. Additionally, a Few-Shot annotation approach with GPT-4o achieved substantial agreement (Cohen's kappa of 0.684) with manual labels. Emotion analysis revealed that disinformation on X is mainly associated with increased sadness and fear, while on TikTok, it correlates with higher levels of anger and disgust. Linguistic analysis using the LIWC dictionary showed that trustworthy content utilizes more articulate and factual language, whereas disinformation employs negations, perceptual words, and personal anecdotes to appear credible. Audio analysis of TikTok posts highlighted distinct patterns: trustworthy audios featured brighter tones and robotic or monotone narration, promoting clarity and credibility, while disinformation audios leveraged tonal variation, emotional depth, and manipulative musical elements to amplify engagement. In detection models, SVM+TF-IDF achieved the highest F1-Score, excelling with limited data. Incorporating audio features into roberta-large-bne improved both Accuracy and F1-Score, surpassing its text-only counterpart and SVM in Accuracy. GPT-4o Few-Shot also performed well, showcasing the potential of large language models for automated disinformation detection. These findings demonstrate the importance of leveraging both textual and audio features for improved disinformation detection on multimodal platforms like TikTok.
• 摘要：本研究调查了 DANA 事件（DANA 是西班牙语 Depresion Aislada en Niveles Altos 的缩写，意为高海拔孤立性低气压）期间社交媒体平台上虚假信息的传播情况，该事件于 2024 年 10 月 29 日在西班牙瓦伦西亚造成极强降雨和毁灭性的洪水。我们创建了一个包含 650 条 TikTok 和 X 帖子的新数据集，并对其进行了手动注释以区分虚假信息和可信内容。此外，使用 GPT-4o 的 Few-Shot 注释方法与手动标签实现了高度一致性（Cohen's kappa 为 0.684）。情绪分析显示，X 上的虚假信息主要与悲伤和恐惧的增加有关，而在 TikTok 上，它与更高程度的愤怒和厌恶相关。使用 LIWC 词典进行的语言分析表明，值得信赖的内容使用更清晰、更真实的语言，而虚假信息则使用否定、感知词和个人轶事来显得可信。对 TikTok 帖子的音频分析突出了不同的模式：值得信赖的音频以更明亮的音调和机械或单调的叙述为特色，以提升清晰度和可信度，而虚假信息音频则利用音调变化、情感深度和操纵性音乐元素来增强参与度。在检测模型中，SVM+TF-IDF 获得了最高的 F1 分数，在有限的数据下表现出色。将音频特征纳入 roberta-large-bne 可提高准确度和 F1 分数，在准确度上超越纯文本对应模型和 SVM。GPT-4o Few-Shot 也表现良好，展示了大型语言模型在自动检测虚假信息方面的潜力。这些发现证明了利用文本和音频特征对于在 TikTok 等多模式平台上改进虚假信息检测的重要性。

Title: Prompt-oriented Output of Culture-Specific Items in Translated African Poetry by Large Language Model: An Initial Multi-layered Tabular Review

• Authors: Adeyola Opaluwah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18644
• Pdf URL: https://arxiv.org/pdf/2501.18644
• Copy Paste: [[2501.18644]] Prompt-oriented Output of Culture-Specific Items in Translated African Poetry by Large Language Model: An Initial Multi-layered Tabular Review(https://arxiv.org/abs/2501.18644)
• Keywords: language model, prompt, chat
• Abstract: This paper examines the output of cultural items generated by Chat Generative PreTrained Transformer Pro in response to three structured prompts to translate three anthologies of African poetry. The first prompt was broad, the second focused on poetic structure, and the third prompt emphasized cultural specificity. To support this analysis, four comparative tables were created. The first table presents the results of the cultural items produced after the three prompts, the second categorizes these outputs based on Aixela framework of Proper nouns and Common expressions, the third table summarizes the cultural items generated by human translators, a custom translation engine, and a Large Language Model. The final table outlines the strategies employed by Chat Generative PreTrained Transformer Pro following the culture specific prompt. Compared to the outputs of cultural items from reference human translation and the custom translation engine in prior studies the findings indicate that the culture oriented prompts used with Chat Generative PreTrained Transformer Pro did not yield significant enhancements of cultural items during the translation of African poetry from English to French. Among the fifty four cultural items, the human translation produced thirty three cultural items in repetition, the custom translation engine generated Thirty eight cultural items in repetition while Chat Generative PreTrained Transformer Pro produced forty one cultural items in repetition. The untranslated cultural items revealed inconsistencies in Large language models approach to translating cultural items in African poetry from English to French.
• 摘要：本文研究了 Chat Generative PreTrained Transformer Pro 在翻译三本非洲诗歌选集时，根据三个结构化提示生成的文化项目的输出。第一个提示很宽泛，第二个提示侧重于诗歌结构，第三个提示强调文化特异性。为了支持这一分析，我们创建了四个比较表。第一个表展示了三个提示后产生的文化项目的结果，第二个表根据专有名词和常用表达的 Aixela 框架对这些输出进行分类，第三个表总结了人工翻译、自定义翻译引擎和大型语言模型生成的文化项目。最后一张表概述了 Chat Generative PreTrained Transformer Pro 在特定文化提示之后采用的策略。与先前研究中参考人工翻译和自定义翻译引擎的文化项目输出相比，研究结果表明，在将非洲诗歌从英语翻译成法语的过程中，Chat Generative PreTrained Transformer Pro 使用的文化导向提示并没有显著增强文化项目。在 54 个文化项目中，人工翻译重复生成了 33 个文化项目，自定义翻译引擎重复生成了 38 个文化项目，而 Chat Generative PreTrained Transformer Pro 重复生成了 41 个文化项目。未翻译的文化项目揭示了大型语言模型方法在将非洲诗歌中的文化项目从英语翻译成法语时存在的不一致之处。

Title: Layered Chain-of-Thought Prompting for Multi-Agent LLM Systems: A Comprehensive Approach to Explainable Large Language Models

• Authors: Manish Sanwal
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.18645
• Pdf URL: https://arxiv.org/pdf/2501.18645
• Copy Paste: [[2501.18645]] Layered Chain-of-Thought Prompting for Multi-Agent LLM Systems: A Comprehensive Approach to Explainable Large Language Models(https://arxiv.org/abs/2501.18645)
• Keywords: language model, llm, prompt, chain-of-thought, agent
• Abstract: Large Language Models (LLMs) leverage chain-of-thought (CoT) prompting to provide step-by-step rationales, improving performance on complex tasks. Despite its benefits, vanilla CoT often fails to fully verify intermediate inferences and can produce misleading explanations. In this work, we propose Layered Chain-of-Thought (Layered-CoT) Prompting, a novel framework that systematically segments the reasoning process into multiple layers, each subjected to external checks and optional user feedback. We expand on the key concepts, present three scenarios -- medical triage, financial risk assessment, and agile engineering -- and demonstrate how Layered-CoT surpasses vanilla CoT in terms of transparency, correctness, and user engagement. By integrating references from recent arXiv papers on interactive explainability, multi-agent frameworks, and agent-based collaboration, we illustrate how Layered-CoT paves the way for more reliable and grounded explanations in high-stakes domains.
• 摘要：大型语言模型 (LLM) 利用思路链 (CoT) 提示来提供分步原理，从而提高复杂任务的性能。尽管有诸多好处，但原始 CoT 通常无法完全验证中间推理，并且会产生误导性解释。在这项工作中，我们提出了分层思路链 (Layered-CoT) 提示，这是一种新颖的框架，它系统地将推理过程划分为多个层，每个层都接受外部检查和可选的用户反馈。我们扩展了关键概念，提出了三种场景——医疗分诊、财务风险评估和敏捷工程——并展示了分层 CoT 在透明度、正确性和用户参与度方面如何超越原始 CoT。通过整合最近关于交互式可解释性、多智能体框架和基于智能体的协作的 arXiv 论文中的参考文献，我们说明了分层 CoT 如何为高风险领域中更可靠、更扎实的解释铺平道路。

Title: Fake News Detection After LLM Laundering: Measurement and Explanation

• Authors: Rupak Kumar Das, Jonathan Dodge
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18649
• Pdf URL: https://arxiv.org/pdf/2501.18649
• Copy Paste: [[2501.18649]] Fake News Detection After LLM Laundering: Measurement and Explanation(https://arxiv.org/abs/2501.18649)
• Keywords: language model, llm
• Abstract: With their advanced capabilities, Large Language Models (LLMs) can generate highly convincing and contextually relevant fake news, which can contribute to disseminating misinformation. Though there is much research on fake news detection for human-written text, the field of detecting LLM-generated fake news is still under-explored. This research measures the efficacy of detectors in identifying LLM-paraphrased fake news, in particular, determining whether adding a paraphrase step in the detection pipeline helps or impedes detection. This study contributes: (1) Detectors struggle to detect LLM-paraphrased fake news more than human-written text, (2) We find which models excel at which tasks (evading detection, paraphrasing to evade detection, and paraphrasing for semantic similarity). (3) Via LIME explanations, we discovered a possible reason for detection failures: sentiment shift. (4) We discover a worrisome trend for paraphrase quality measurement: samples that exhibit sentiment shift despite a high BERTSCORE. (5) We provide a pair of datasets augmenting existing datasets with paraphrase outputs and scores. The dataset is available on GitHub
• 摘要：大型语言模型 (LLM) 凭借其先进的功能，可以生成极具说服力且与上下文相关的虚假新闻，从而有助于传播错误信息。尽管针对人工编写的文本的虚假新闻检测已经有很多研究，但检测 LLM 生成的虚假新闻的领域仍未得到充分探索。这项研究衡量了检测器在识别 LLM 释义的虚假新闻方面的有效性，特别是确定在检测流程中添加释义步骤是有助于还是阻碍检测。这项研究的贡献包括：(1) 与人工编写的文本相比，检测器更难检测到 LLM 释义的虚假新闻，(2) 我们发现哪些模型在哪些任务上表现出色（逃避检测、释义以逃避检测以及释义以进行语义相似性）。(3) 通过 LIME 解释，我们发现了检测失败的一个可能原因：情绪转变。(4) 我们发现释义质量测量的一个令人担忧的趋势：尽管 BERTSCORE 很高，但样本仍表现出情绪转变。 （5）我们提供了一对数据集，用释义输出和分数扩充现有数据集。该数据集可在 GitHub 上找到

Title: Zero-shot Large Language Models for Long Clinical Text Summarization with Temporal Reasoning

• Authors: Maya Kruse, Shiyue Hu, Nicholas Derby, Yifu Wu, Samantha Stonbraker, Bingsheng Yao, Dakuo Wang, Elizabeth Goldberg, Yanjun Gao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18724
• Pdf URL: https://arxiv.org/pdf/2501.18724
• Copy Paste: [[2501.18724]] Zero-shot Large Language Models for Long Clinical Text Summarization with Temporal Reasoning(https://arxiv.org/abs/2501.18724)
• Keywords: language model, llm, long context
• Abstract: Recent advancements in large language models (LLMs) have shown potential for transforming data processing in healthcare, particularly in understanding complex clinical narratives. This study evaluates the efficacy of zero-shot LLMs in summarizing long clinical texts that require temporal reasoning, a critical aspect for comprehensively capturing patient histories and treatment trajectories. We applied a series of advanced zero-shot LLMs to extensive clinical documents, assessing their ability to integrate and accurately reflect temporal dynamics without prior task-specific training. While the models efficiently identified key temporal events, they struggled with chronological coherence over prolonged narratives. The evaluation, combining quantitative and qualitative methods, highlights the strengths and limitations of zero-shot LLMs in clinical text summarization. The results suggest that while promising, zero-shot LLMs require further refinement to effectively support clinical decision-making processes, underscoring the need for enhanced model training approaches that better capture the nuances of temporal information in long context medical documents.
• 摘要：大型语言模型 (LLM) 的最新进展已显示出改变医疗保健数据处理的潜力，特别是在理解复杂的临床叙述方面。本研究评估了零样本 LLM 在总结需要时间推理的长篇临床文本方面的有效性，时间推理是全面捕捉患者病史和治疗轨迹的关键方面。我们将一系列先进的零样本 LLM 应用于大量临床文档，评估它们在没有事先进行特定任务训练的情况下整合和准确反映时间动态的能力。虽然这些模型有效地识别了关键的时间事件，但它们在长时间叙述中难以保持时间连贯性。评估结合了定量和定性方法，突出了零样本 LLM 在临床文本摘要方面的优势和局限性。结果表明，虽然零样本 LLM 很有前景，但需要进一步改进才能有效支持临床决策过程，这强调了需要增强模型训练方法，以更好地捕捉长篇医疗文档中时间信息的细微差别。

Title: Examining the Robustness of Large Language Models across Language Complexity

• Authors: Jiayi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18738
• Pdf URL: https://arxiv.org/pdf/2501.18738
• Copy Paste: [[2501.18738]] Examining the Robustness of Large Language Models across Language Complexity(https://arxiv.org/abs/2501.18738)
• Keywords: language model, llm
• Abstract: With the advancement of large language models (LLMs), an increasing number of student models have leveraged LLMs to analyze textual artifacts generated by students to understand and evaluate their learning. These student models typically employ pre-trained LLMs to vectorize text inputs into embeddings and then use the embeddings to train models to detect the presence or absence of a construct of interest. However, how reliable and robust are these models at processing language with different levels of complexity? In the context of learning where students may have different language backgrounds with various levels of writing skills, it is critical to examine the robustness of such models to ensure that these models work equally well for text with varying levels of language complexity. Coincidentally, a few (but limited) research studies show that the use of language can indeed impact the performance of LLMs. As such, in the current study, we examined the robustness of several LLM-based student models that detect student self-regulated learning (SRL) in math problem-solving. Specifically, we compared how the performance of these models vary using texts with high and low lexical, syntactic, and semantic complexity measured by three linguistic measures.
• 摘要：随着大型语言模型 (LLM) 的进步，越来越多的学生模型利用 LLM 分析学生生成的文本工件，以了解和评估他们的学习情况。这些学生模型通常使用预先训练的 LLM 将文本输入矢量化为嵌入，然后使用嵌入来训练模型以检测感兴趣的构造的存在与否。但是，这些模型在处理不同复杂程度的语言时有多可靠和稳健？在学习的背景下，学生可能具有不同的语言背景和不同水平的写作技能，因此检查此类模型的稳健性至关重要，以确保这些模型对于具有不同语言复杂程度的文本同样有效。巧合的是，一些（但有限）研究表明，语言的使用确实会影响 LLM 的性能。因此，在当前的研究中，我们检查了几种基于 LLM 的学生模型的稳健性，这些模型可以检测学生在数学问题解决中的自我调节学习 (SRL)。具体来说，我们比较了这些模型在使用通过三种语言测量方法测量的词汇、句法和语义复杂度高和低的文本时的性能变化。

Title: Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languages

• Authors: Andrei Politov, Oleh Shkalikov, René Jäkel, Michael Färber
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.18750
• Pdf URL: https://arxiv.org/pdf/2501.18750
• Copy Paste: [[2501.18750]] Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languages(https://arxiv.org/abs/2501.18750)
• Keywords: language model
• Abstract: Cross-lingual Named Entity Recognition (NER) leverages knowledge transfer between languages to identify and classify named entities, making it particularly useful for low-resource languages. We show that the data-based cross-lingual transfer method is an effective technique for crosslingual NER and can outperform multilingual language models for low-resource languages. This paper introduces two key enhancements to the annotation projection step in cross-lingual NER for low-resource languages. First, we explore refining word alignments using back-translation to improve accuracy. Second, we present a novel formalized projection approach of matching source entities with extracted target candidates. Through extensive experiments on two datasets spanning 57 languages, we demonstrated that our approach surpasses existing projectionbased methods in low-resource settings. These findings highlight the robustness of projection-based data transfer as an alternative to model-based methods for crosslingual named entity recognition in lowresource languages.
• 摘要：跨语言命名实体识别 (NER) 利用语言之间的知识转移来识别和分类命名实体，这使其对于资源匮乏的语言特别有用。我们表明，基于数据的跨语言传输方法是跨语言 NER 的有效技术，并且对于资源匮乏的语言，其表现可以优于多语言语言模型。本文介绍了针对资源匮乏的语言的跨语言 NER 中注释投影步骤的两个关键增强功能。首先，我们探索使用反向翻译来细化词对齐以提高准确性。其次，我们提出了一种新颖的形式化投影方法，将源实体与提取的目标候选实体进行匹配。通过对两个涵盖 57 种语言的数据集进行大量实验，我们证明了我们的方法在资源匮乏的环境中优于现有的基于投影的方法。这些发现凸显了基于投影的数据传输作为基于模型的低资源语言跨语言命名实体识别方法的替代方案的稳健性。

Title: Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation

• Authors: Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18771
• Pdf URL: https://arxiv.org/pdf/2501.18771
• Copy Paste: [[2501.18771]] Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation(https://arxiv.org/abs/2501.18771)
• Keywords: language model, llm
• Abstract: Data contamination -- the accidental consumption of evaluation examples within the pre-training data -- can undermine the validity of evaluation benchmarks. In this paper, we present a rigorous analysis of the effects of contamination on language models at 1B and 8B scales on the machine translation task. Starting from a carefully decontaminated train-test split, we systematically introduce contamination at various stages, scales, and data formats to isolate its effect and measure its impact on performance metrics. Our experiments reveal that contamination with both source and target substantially inflates BLEU scores, and this inflation is 2.5 times larger (up to 30 BLEU points) for 8B compared to 1B models. In contrast, source-only and target-only contamination generally produce smaller, less consistent over-estimations. Finally, we study how the temporal distribution and frequency of contaminated samples influence performance over-estimation across languages with varying degrees of data resources.
• 摘要：数据污染（即意外使用预训练数据中的评估示例）会破坏评估基准的有效性。在本文中，我们对污染对机器翻译任务中 1B 和 8B 规模的语言模型的影响进行了严格的分析。从精心净化的训练测试分割开始，我们系统地在各个阶段、规模和数据格式中引入污染，以隔离其影响并衡量其对性能指标的影响。我们的实验表明，源和目标的污染都会大大提高 BLEU 分数，与 1B 模型相比，8B 模型的膨胀率是 2.5 倍（高达 30 个 BLEU 分）。相比之下，仅有源和仅有目标的污染通常会产生较小、不太一致的高估。最后，我们研究了受污染样本的时间分布和频率如何影响具有不同数据资源程度的语言的性能高估。

Title: Rope to Nope and Back Again: A New Hybrid Attention Strategy

• Authors: Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, Acyr Locatelli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18795
• Pdf URL: https://arxiv.org/pdf/2501.18795
• Copy Paste: [[2501.18795]] Rope to Nope and Back Again: A New Hybrid Attention Strategy(https://arxiv.org/abs/2501.18795)
• Keywords: language model, llm, long context
• Abstract: Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architectural based on a hybrid attention mechanism that not only surpasses conventional RoPE-based transformer models in long context tasks but also achieves competitive performance on benchmarks requiring shorter context lengths.
• 摘要：长上下文大型语言模型 (LLM) 取得了显著的进步，这得益于旋转位置嵌入 (RoPE) (Su et al., 2023) 及其扩展 (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023) 等技术。通过调整 RoPE 参数并将训练数据与扩展上下文相结合，我们可以使用更长的输入序列训练高性能模型。然而，现有的基于 RoPE 的方法在应用于扩展上下文长度时表现出性能限制。本文对各种注意力机制进行了全面分析，包括 RoPE、无位置嵌入 (NoPE) 和查询键规范化 (QK-Norm)，确定了它们在长上下文建模中的优势和缺点。我们的研究确定了这些方法中独特的注意力模式，并强调了它们对长上下文性能的影响，为架构设计提供了宝贵的见解。基于这些发现，我们提出了一种基于混合注意力机制的新型架构，该架构不仅在长上下文任务中超越了传统的基于 RoPE 的 Transformer 模型，而且在需要较短上下文长度的基准测试中也取得了具有竞争力的性能。

Title: Large Language Models as Common-Sense Heuristics

• Authors: Andrey Borro, Patricia J Riddle, Michael W Barley, Michael J Witbrock
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18816
• Pdf URL: https://arxiv.org/pdf/2501.18816
• Copy Paste: [[2501.18816]] Large Language Models as Common-Sense Heuristics(https://arxiv.org/abs/2501.18816)
• Keywords: language model, llm, prompt
• Abstract: While systems designed for solving planning tasks vastly outperform Large Language Models (LLMs) in this domain, they usually discard the rich semantic information embedded within task descriptions. In contrast, LLMs possess parametrised knowledge across a wide range of topics, enabling them to leverage the natural language descriptions of planning tasks in their solutions. However, current research in this direction faces challenges in generating correct and executable plans. Furthermore, these approaches depend on the LLM to output solutions in an intermediate language, which must be translated into the representation language of the planning task. We introduce a novel planning method, which leverages the parametrised knowledge of LLMs by using their output as a heuristic for Hill-Climbing Search. This approach is further enhanced by prompting the LLM to generate a solution estimate to guide the search. Our method outperforms the task success rate of similar systems within a common household environment by 22 percentage points, with consistently executable plans. All actions are encoded in their original representation, demonstrating that strong results can be achieved without an intermediate language, thus eliminating the need for a translation step.
• 摘要：虽然为解决规划任务而设计的系统在此领域的表现远远优于大型语言模型 (LLM)，但它们通常会丢弃任务描述中嵌入的丰富语义信息。相比之下，LLM 拥有广泛主题的参数化知识，使它们能够在解决方案中利用规划任务的自然语言描述。然而，目前这方面的研究在生成正确且可执行的计划方面面临挑战。此外，这些方法依赖于 LLM 以中间语言输出解决方案，而这些解决方案必须翻译成规划任务的表示语言。我们引入了一种新颖的规划方法，该方法利用 LLM 的参数化知识，将其输出用作爬山搜索的启发式方法。通过提示 LLM 生成解决方案估计来指导搜索，这种方法得到了进一步增强。我们的方法比普通家庭环境中的类似系统的任务成功率高出 22 个百分点，并且具有一致可执行的计划。所有操作都以原始表示形式进行编码，表明无需中间语言即可获得强大的结果，从而消除了翻译步骤的需要。

Title: Memory-Efficient Fine-Tuning of Transformers via Token Selection

• Authors: Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey Yang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18824
• Pdf URL: https://arxiv.org/pdf/2501.18824
• Copy Paste: [[2501.18824]] Memory-Efficient Fine-Tuning of Transformers via Token Selection(https://arxiv.org/abs/2501.18824)
• Keywords: llm
• Abstract: Fine-tuning provides an effective means to specialize pre-trained models for various downstream tasks. However, fine-tuning often incurs high memory overhead, especially for large transformer-based models, such as LLMs. While existing methods may reduce certain parts of the memory required for fine-tuning, they still require caching all intermediate activations computed in the forward pass to update weights during the backward pass. In this work, we develop TokenTune, a method to reduce memory usage, specifically the memory to store intermediate activations, in the fine-tuning of transformer-based models. During the backward pass, TokenTune approximates the gradient computation by backpropagating through just a subset of input tokens. Thus, with TokenTune, only a subset of intermediate activations are cached during the forward pass. Also, TokenTune can be easily combined with existing methods like LoRA, further reducing the memory cost. We evaluate our approach on pre-trained transformer models with up to billions of parameters, considering the performance on multiple downstream tasks such as text classification and question answering in a few-shot learning setup. Overall, TokenTune achieves performance on par with full fine-tuning or representative memory-efficient fine-tuning methods, while greatly reducing the memory footprint, especially when combined with other methods with complementary memory reduction mechanisms. We hope that our approach will facilitate the fine-tuning of large transformers, in specializing them for specific domains or co-training them with other neural components from a larger system. Our code is available at this https URL.
• 摘要：微调提供了一种有效的方法，可以将预训练模型专门用于各种下游任务。然而，微调通常会产生高内存开销，尤其是对于大型基于 Transformer 的模型，例如 LLM。虽然现有方法可能会减少微调所需的某些内存，但它们仍然需要缓存前向传递中计算的所有中间激活，以便在后向传递期间更新权重。在这项工作中，我们开发了 TokenTune，这是一种在基于 Transformer 的模型的微调中减少内存使用量的方法，特别是用于存储中间激活的内存。在后向传递期间，TokenTune 通过仅通过输入 token 的子集进行反向传播来近似梯度计算。因此，使用 TokenTune，在前向传递期间仅缓存中间激活的子集。此外，TokenTune 可以轻松地与 LoRA 等现有方法结合使用，从而进一步降低内存成本。我们在具有多达数十亿个参数的预训练 Transformer 模型上评估了我们的方法，并考虑了在少数样本学习设置中在多个下游任务（例如文本分类和问答）上的性能。总体而言，TokenTune 的性能与完全微调或代表性的内存高效微调方法相当，同时大大减少了内存占用，尤其是与具有互补内存减少机制的其他方法结合使用时。我们希望我们的方法能够促进大型变压器的微调，使其专门用于特定领域或与来自更大系统的其他神经组件一起训练它们。我们的代码可在此 https URL 上找到。

Title: Structural Embedding Projection for Contextual Large Language Model Inference

• Authors: Vincent Enoasmo, Cedric Featherstonehaugh, Xavier Konstantinopoulos, Zacharias Huntington
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18826
• Pdf URL: https://arxiv.org/pdf/2501.18826
• Copy Paste: [[2501.18826]] Structural Embedding Projection for Contextual Large Language Model Inference(https://arxiv.org/abs/2501.18826)
• Keywords: language model
• Abstract: Structured embedding transformations offer a promising approach for enhancing the efficiency and coherence of language model inference. The introduction of Structural Embedding Projection (SEP) provides a mechanism for refining token representations through projection matrices that integrate hierarchical and relational dependencies. The mathematical formulation of SEP enables embedding spaces to capture structured contextual relationships, thereby improving semantic fidelity without significantly increasing computational overhead. Experimental evaluations conducted on a range of linguistic datasets revealed that SEP contributed to reductions in perplexity and enhanced contextual coherence, demonstrating its potential to refine language model outputs. Computational efficiency assessments highlighted variations across different datasets, suggesting that the integration of structured embeddings introduced dataset-dependent trade-offs between inference speed and representational richness. The qualitative analysis of generated responses indicated that SEP enhanced narrative consistency and topic alignment, leading to improved fluency in multi-sentence text generation. The modifications to embedding layers required precise optimization to ensure stable training dynamics, as the introduction of structured transformations altered the traditional representation-learning process. The architectural adjustments necessary for SEP implementation influenced inference latency and memory consumption, requiring a balance between efficiency gains and additional processing demands. The impact of SEP on lexical diversity suggested that embedding modifications influenced the model's vocabulary usage, reflecting a more context-aware selection of generated tokens.
• 摘要：结构化嵌入转换提供了一种有前途的方法，可提高语言模型推理的效率和连贯性。结构化嵌入投影 (SEP) 的引入提供了一种通过集成层次和关系依赖关系的投影矩阵来细化标记表示的机制。SEP 的数学公式使嵌入空间能够捕获结构化的上下文关系，从而提高语义保真度而不会显着增加计算开销。对一系列语言数据集进行的实验评估表明，SEP 有助于减少困惑并增强上下文连贯性，从而展示了其改进语言模型输出的潜力。计算效率评估强调了不同数据集之间的差异，这表明结构化嵌入的集成引入了依赖于数据集的推理速度和表示丰富性之间的权衡。对生成的响应的定性分析表明，SEP 增强了叙述一致性和主题一致性，从而提高了多句文本生成的流畅性。对嵌入层的修改需要精确优化，以确保稳定的训练动态，因为结构化转换的引入改变了传统的表示学习过程。SEP 实现所需的架构调整影响了推理延迟和内存消耗，需要在效率提升和额外处理需求之间取得平衡。SEP 对词汇多样性的影响表明，嵌入修改影响了模型的词汇使用，反映了对生成的标记进行更具上下文感知的选择。

Title: Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

• Authors: Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18837
• Pdf URL: https://arxiv.org/pdf/2501.18837
• Copy Paste: [[2501.18837]] Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming(https://arxiv.org/abs/2501.18837)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are vulnerable to universal jailbreaks-prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers: safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.
• 摘要：大型语言模型 (LLM) 容易受到通用越狱提示策略的攻击，这些策略会系统地绕过模型保护措施，使用户能够执行需要许多模型交互的有害过程，例如大规模制造非法物质。为了防御这些攻击，我们引入了宪法分类器：使用合成数据训练的保护措施，通过使用指定允许和限制内容的自然语言规则（即宪法）提示 LLM 来生成。在估计超过 3,000 小时的红队演练中，没有红队成员发现通用越狱可以从早期分类器保护的 LLM 中提取与大多数目标查询中不受保护的模型类似的详细程度的信息。在自动评估中，增强型分类器表现出对特定领域越狱的强大防御能力。这些分类器还保持了部署可行性，生产流量拒绝绝对增加了 0.38%，推理开销增加了 23.7%。我们的工作表明，在保持实际部署可行性的同时防御通用越狱是可行的。

Title: Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities

• Authors: Yaping Chai, Haoran Xie, Joe S. Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18845
• Pdf URL: https://arxiv.org/pdf/2501.18845
• Copy Paste: [[2501.18845]] Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities(https://arxiv.org/abs/2501.18845)
• Keywords: language model, llm, prompt
• Abstract: The increasing size and complexity of pre-trained language models have demonstrated superior performance in many applications, but they usually require large training datasets to be adequately trained. Insufficient training sets could unexpectedly make the model overfit and fail to cope with complex tasks. Large language models (LLMs) trained on extensive corpora have prominent text generation capabilities, which improve the quality and quantity of data and play a crucial role in data augmentation. Specifically, distinctive prompt templates are given in personalised tasks to guide LLMs in generating the required content. Recent promising retrieval-based techniques further improve the expressive performance of LLMs in data augmentation by introducing external knowledge to enable them to produce more grounded-truth data. This survey provides an in-depth analysis of data augmentation in LLMs, classifying the techniques into Simple Augmentation, Prompt-based Augmentation, Retrieval-based Augmentation and Hybrid Augmentation. We summarise the post-processing approaches in data augmentation, which contributes significantly to refining the augmented data and enabling the model to filter out unfaithful content. Then, we provide the common tasks and evaluation metrics. Finally, we introduce existing challenges and future opportunities that could bring further improvement to data augmentation.
• 摘要：预训练语言模型的规模和复杂度不断增加，在许多应用中表现出色，但它们通常需要大量的训练数据集才能得到充分训练。训练集不足可能会意外地导致模型过拟合，无法应对复杂任务。在大量语料库上训练的大型语言模型 (LLM) 具有突出的文本生成能力，这可以提高数据的质量和数量，在数据增强中发挥着至关重要的作用。具体而言，在个性化任务中给出独特的提示模板，指导 LLM 生成所需的内容。最近有前景的基于检索的技术通过引入外部知识使 LLM 能够生成更多基于事实的数据，进一步提高了 LLM 在数据增强方面的表达性能。本综述对 LLM 中的数据增强进行了深入分析，将技术分为简单增强、基于提示的增强、基于检索的增强和混合增强。我们总结了数据增强中的后处理方法，这些方法有助于完善增强数据并使模型能够过滤掉不真实的内容。然后，我们提供了常见的任务和评估指标。最后，我们介绍了现有的挑战和未来的机会，这些机会可能会进一步改善数据增强。

Title: KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search

• Authors: Haoran Luo, Haihong E, Yikai Guo, Qika Lin, Xiaobao Wu, Xinyu Mu, Wenhao Liu, Meina Song, Yifan Zhu, Luu Anh Tuan
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2501.18922
• Pdf URL: https://arxiv.org/pdf/2501.18922
• Copy Paste: [[2501.18922]] KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search(https://arxiv.org/abs/2501.18922)
• Keywords: language model, gpt, llm, agent
• Abstract: Knowledge Base Question Answering (KBQA) aims to answer natural language questions with a large-scale structured knowledge base (KB). Despite advancements with large language models (LLMs), KBQA still faces challenges in weak KB awareness, imbalance between effectiveness and efficiency, and high reliance on annotated data. To address these challenges, we propose KBQA-o1, a novel agentic KBQA method with Monte Carlo Tree Search (MCTS). It introduces a ReAct-based agent process for stepwise logical form generation with KB environment exploration. Moreover, it employs MCTS, a heuristic search method driven by policy and reward models, to balance agentic exploration's performance and search space. With heuristic exploration, KBQA-o1 generates high-quality annotations for further improvement by incremental fine-tuning. Experimental results show that KBQA-o1 outperforms previous low-resource KBQA methods with limited annotated data, boosting Llama-3.1-8B model's GrailQA F1 performance to 78.5% compared to 48.5% of the previous sota method with GPT-3.5-turbo.
• 摘要：知识库问答 (KBQA) 旨在使用大规模结构化知识库 (KB) 来回答自然语言问题。尽管大型语言模型 (LLM) 取得了进步，但 KBQA 仍然面临着知识库意识薄弱、有效性和效率不平衡以及高度依赖注释数据的挑战。为了应对这些挑战，我们提出了 KBQA-o1，一种具有蒙特卡洛树搜索 (MCTS) 的新型代理 KBQA 方法。它引入了一种基于 ReAct 的代理流程，用于通过知识库环境探索分步生成逻辑形式。此外，它采用了 MCTS，一种由策略和奖励模型驱动的启发式搜索方法，以平衡代理探索的性能和搜索空间。通过启发式探索，KBQA-o1 可以生成高质量注释，并通过增量微调进一步改进。实验结果表明，KBQA-o1 优于以前在注释数据有限的情况下的低资源 KBQA 方法，将 Llama-3.1-8B 模型的 GrailQA F1 性能提升至 78.5%，而以前采用 GPT-3.5-turbo 的 sota 方法仅为 48.5%。

Title: Intrinsic Tensor Field Propagation in Large Language Models: A Novel Approach to Contextual Information Flow

• Authors: Alfred Bexley, Lukas Radcliffe, Giles Weatherstone, Joseph Sakau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18957
• Pdf URL: https://arxiv.org/pdf/2501.18957
• Copy Paste: [[2501.18957]] Intrinsic Tensor Field Propagation in Large Language Models: A Novel Approach to Contextual Information Flow(https://arxiv.org/abs/2501.18957)
• Keywords: language model
• Abstract: Context propagation remains a central challenge in language model architectures, particularly in tasks requiring the retention of long-range dependencies. Conventional attention mechanisms, while effective in many applications, exhibit limitations in maintaining coherent contextual representations over extended sequences due to their reliance on discrete token interactions. A novel approach is introduced through the formulation of Intrinsic Tensor Field Propagation (ITFP), which models contextual relationships as continuous tensor fields distributed across token embeddings. The propagation dynamics are governed through differential equations that enable a structured flow of contextual information, augmenting the standard attention mechanism to enhance coherence and recall. A series of experiments conducted on an open-source transformer-based model demonstrate that ITFP provides measurable improvements in contextual retention, dependency resolution, and inference stability across various linguistic structures. Comparisons with baseline models reveal a reduction in syntactic inconsistencies and factual errors, while ablation studies indicate that the choice of propagation depth and integration strength significantly impacts model performance. Additional evaluations assessing domain generalization suggest that ITFP effectively adapts across different text genres, reinforcing its applicability beyond conventional language modeling tasks. Although computational trade-offs are introduced through the inclusion of tensor field computations, empirical findings suggest that the benefits in accuracy and coherence outweigh the increased processing demands.
• 摘要：上下文传播仍然是语言模型架构中的一个核心挑战，特别是在需要保留长距离依赖关系的任务中。传统的注意力机制虽然在许多应用中都很有效，但由于依赖于离散的 token 交互，因此在保持扩展序列上连贯的上下文表示方面表现出局限性。通过制定内在张量场传播 (ITFP)，引入了一种新方法，该方法将上下文关系建模为分布在 token 嵌入中的连续张量场。传播动力学由微分方程控制，微分方程可实现上下文信息的结构化流动，增强标准注意力机制以增强连贯性和回忆性。在基于开源转换器的模型上进行的一系列实验表明，ITFP 在各种语言结构中提供了上下文保留、依赖解析和推理稳定性方面的可衡量改进。与基线模型的比较表明句法不一致和事实错误的减少，而消融研究表明传播深度和集成强度的选择会显著影响模型性能。对领域泛化能力的额外评估表明，ITFP 能够有效适应不同的文本类型，从而增强了其在传统语言建模任务之外的适用性。尽管通过纳入张量场计算引入了计算权衡，但实证结果表明，准确性和连贯性方面的好处超过了增加的处理需求。

Title: Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings

• Authors: Ahmed K. Kadhim, Lei Jiao, Rishad Shafik, Ole-Christoffer Granmo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18998
• Pdf URL: https://arxiv.org/pdf/2501.18998
• Copy Paste: [[2501.18998]] Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings(https://arxiv.org/abs/2501.18998)
• Keywords: gpt, prompt
• Abstract: In recent years, text generation tools utilizing Artificial Intelligence (AI) have occasionally been misused across various domains, such as generating student reports or creative writings. This issue prompts plagiarism detection services to enhance their capabilities in identifying AI-generated content. Adversarial attacks are often used to test the robustness of AI-text generated detectors. This work proposes a novel textual adversarial attack on the detection models such as Fast-DetectGPT. The method employs embedding models for data perturbation, aiming at reconstructing the AI generated texts to reduce the likelihood of detection of the true origin of the texts. Specifically, we employ different embedding techniques, including the Tsetlin Machine (TM), an interpretable approach in machine learning for this purpose. By combining synonyms and embedding similarity vectors, we demonstrates the state-of-the-art reduction in detection scores against Fast-DetectGPT. Particularly, in the XSum dataset, the detection score decreased from 0.4431 to 0.2744 AUROC, and in the SQuAD dataset, it dropped from 0.5068 to 0.3532 AUROC.
• 摘要：近年来，利用人工智能 (AI) 的文本生成工具偶尔会在各个领域被滥用，例如生成学生报告或创意写作。这个问题促使抄袭检测服务增强其识别 AI 生成内容的能力。对抗性攻击通常用于测试 AI 文本生成检测器的稳健性。这项工作提出了一种针对 Fast-DetectGPT 等检测模型的新型文本对抗性攻击。该方法采用嵌入模型进行数据扰动，旨在重建 AI 生成的文本，以降低检测到文本真实来源的可能性。具体来说，我们为此采用了不同的嵌入技术，包括 Tsetlin Machine (TM)，这是一种可解释的机器学习方法。通过结合同义词和嵌入相似性向量，我们展示了针对 Fast-DetectGPT 的最先进的检测分数降低。具体来说，在 XSum 数据集中，检测分数从 0.4431 下降到 0.2744 AUROC，而在 SQuAD 数据集中，检测分数从 0.5068 下降到 0.3532 AUROC。

Title: Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation

• Authors: Bin Zhu, Hui yan Qi, Yinxuan Gui, Jingjing Chen, Chong-Wah Ngo, Ee Peng Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19017
• Pdf URL: https://arxiv.org/pdf/2501.19017
• Copy Paste: [[2501.19017]] Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation(https://arxiv.org/abs/2501.19017)
• Keywords: language model, gpt, llm
• Abstract: Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities, excelling in complex understanding and generation tasks. Despite their success, MLLMs remain vulnerable to conversational adversarial inputs, particularly negation arguments. This paper systematically evaluates state-of-the-art MLLMs across diverse benchmarks, revealing significant performance drops when negation arguments are introduced to initially correct responses. We show critical vulnerabilities in the reasoning and alignment mechanisms of these models. Proprietary models such as GPT-4o and Claude-3.5-Sonnet demonstrate better resilience compared to open-source counterparts like Qwen2-VL and LLaVA. However, all evaluated MLLMs struggle to maintain logical consistency under negation arguments during conversation. This paper aims to offer valuable insights for improving the robustness of MLLMs against adversarial inputs, contributing to the development of more reliable and trustworthy multimodal AI systems.
• 摘要：多模态大型语言模型 (MLLM) 在整合不同模态方面表现出色，在复杂的理解和生成任务中表现出色。尽管取得了成功，但 MLLM 仍然容易受到对话对抗性输入的影响，尤其是否定论证。本文系统地评估了不同基准中最先进的 MLLM，发现当将否定论证引入最初正确的响应时，性能会显著下降。我们展示了这些模型的推理和对齐机制中的关键漏洞。与 Qwen2-VL 和 LLaVA 等开源模型相比，GPT-4o 和 Claude-3.5-Sonnet 等专有模型表现出更好的弹性。然而，所有评估过的 MLLM 在对话过程中都难以在否定论证下保持逻辑一致性。本文旨在为提高 MLLM 对对抗性输入的鲁棒性提供有价值的见解，为开发更可靠、更值得信赖的多模态 AI 系统做出贡献。

Title: Improving Low-Resource Sequence Labeling with Knowledge Fusion and Contextual Label Explanations

• Authors: Peichao Lai, Jiaxin Gan, Feiyang Ye, Yilei Wang, Bin Cui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19093
• Pdf URL: https://arxiv.org/pdf/2501.19093
• Copy Paste: [[2501.19093]] Improving Low-Resource Sequence Labeling with Knowledge Fusion and Contextual Label Explanations(https://arxiv.org/abs/2501.19093)
• Keywords: llm, prompt
• Abstract: Sequence labeling remains a significant challenge in low-resource, domain-specific scenarios, particularly for character-dense languages like Chinese. Existing methods primarily focus on enhancing model comprehension and improving data diversity to boost performance. However, these approaches still struggle with inadequate model applicability and semantic distribution biases in domain-specific contexts. To overcome these limitations, we propose a novel framework that combines an LLM-based knowledge enhancement workflow with a span-based Knowledge Fusion for Rich and Efficient Extraction (KnowFREE) model. Our workflow employs explanation prompts to generate precise contextual interpretations of target entities, effectively mitigating semantic biases and enriching the model's contextual understanding. The KnowFREE model further integrates extension label features, enabling efficient nested entity extraction without relying on external knowledge during inference. Experiments on multiple Chinese domain-specific sequence labeling datasets demonstrate that our approach achieves state-of-the-art performance, effectively addressing the challenges posed by low-resource settings.
• 摘要：在资源匮乏、领域特定场景下，序列标记仍然是一项重大挑战，尤其是对于像中文这样的字符密集型语言。现有方法主要侧重于增强模型理解能力和改善数据多样性以提高性能。然而，这些方法仍然存在模型适用性不足和领域特定上下文中的语义分布偏差的问题。为了克服这些限制，我们提出了一个新颖的框架，将基于 LLM 的知识增强工作流与基于跨度的知识融合丰富高效提取 (KnowFREE) 模型相结合。我们的工作流程采用解释提示来生成目标实体的精确上下文解释，有效地减轻语义偏差并丰富模型的上下文理解。KnowFREE 模型进一步集成了扩展标签特征，实现了高效的嵌套实体提取，而无需在推理过程中依赖外部知识。在多个中文领域特定序列标记数据集上的实验表明，我们的方法实现了最先进的性能，有效地解决了资源匮乏的设置带来的挑战。

Title: Efficient Reasoning with Hidden Thinking

• Authors: Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, Jiuxiang Gu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19201
• Pdf URL: https://arxiv.org/pdf/2501.19201
• Copy Paste: [[2501.19201]] Efficient Reasoning with Hidden Thinking(https://arxiv.org/abs/2501.19201)
• Keywords: language model, llm, chain-of-thought
• Abstract: Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities in Multimodal Large Language Models (MLLMs). However, the verbose nature of textual reasoning introduces significant inefficiencies. In this work, we propose $\textbf{Heima}$ (as hidden llama), an efficient reasoning framework that leverages reasoning CoTs at hidden latent space. We design the Heima Encoder to condense each intermediate CoT into a compact, higher-level hidden representation using a single thinking token, effectively minimizing verbosity and reducing the overall number of tokens required during the reasoning process. Meanwhile, we design corresponding Heima Decoder with traditional Large Language Models (LLMs) to adaptively interpret the hidden representations into variable-length textual sequence, reconstructing reasoning processes that closely resemble the original CoTs. Experimental results across diverse reasoning MLLM benchmarks demonstrate that Heima model achieves higher generation efficiency while maintaining or even better zero-shot task accuracy. Moreover, the effective reconstruction of multimodal reasoning processes with Heima Decoder validates both the robustness and interpretability of our approach.
• 摘要：思路链 (CoT) 推理已成为多模态大型语言模型 (MLLM) 中提高复杂问题解决能力的强大框架。然而，文本推理的冗长性质带来了严重的低效率。在这项工作中，我们提出了 $\textbf{Heima}$（隐藏的骆驼），这是一个高效的推理框架，它利用隐藏潜在空间中的推理 CoT。我们设计了 Heima 编码器，使用单个思考标记将每个中间 CoT 压缩为紧凑的高级隐藏表示，有效地最大限度地减少了冗长性并减少了推理过程中所需的总标记数量。同时，我们设计了与传统大型语言模型 (LLM) 相对应的 Heima 解码器，以自适应地将隐藏表示解释为可变长度的文本序列，重建与原始 CoT 非常相似的推理过程。跨不同推理 MLLM 基准的实验结果表明，Heima 模型实现了更高的生成效率，同时保持甚至更好的零样本任务准确率。此外，使用 Heima Decoder 对多模态推理过程的有效重建验证了我们方法的稳健性和可解释性。

Title: Improving the Robustness of Representation Misdirection for Large Language Model Unlearning

• Authors: Dang Huu-Tien, Hoang Thanh-Tung, Le-Minh Nguyen, Naoya Inoue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19202
• Pdf URL: https://arxiv.org/pdf/2501.19202
• Copy Paste: [[2501.19202]] Improving the Robustness of Representation Misdirection for Large Language Model Unlearning(https://arxiv.org/abs/2501.19202)
• Keywords: language model, llm
• Abstract: Representation Misdirection (RM) and variants are established large language model (LLM) unlearning methods with state-of-the-art performance. In this paper, we show that RM methods inherently reduce models' robustness, causing them to misbehave even when a single non-adversarial forget-token is in the retain-query. Toward understanding underlying causes, we reframe the unlearning process as backdoor attacks and defenses: forget-tokens act as backdoor triggers that, when activated in retain-queries, cause disruptions in RM models' behaviors, similar to successful backdoor attacks. To mitigate this vulnerability, we propose Random Noise Augmentation -- a model and method agnostic approach with theoretical guarantees for improving the robustness of RM methods. Extensive experiments demonstrate that RNA significantly improves the robustness of RM models while enhancing the unlearning performances.
• 摘要：表征误导 (RM) 及其变体是具有最佳性能的成熟大型语言模型 (LLM) 反学习方法。在本文中，我们表明 RM 方法本质上会降低模型的鲁棒性，导致它们即使在保留查询中有一个非对抗性遗忘标记时也会出现错误行为。为了理解根本原因，我们将反学习过程重新定义为后门攻击和防御：遗忘标记充当后门触发器，当在保留查询中激活时，会导致 RM 模型行为中断，类似于成功的后门攻击。为了缓解这种脆弱性，我们提出了随机噪声增强——一种与模型和方法无关的方法，具有提高 RM 方法鲁棒性的理论保证。大量实验表明，RNA 显着提高了 RM 模型的鲁棒性，同时增强了反学习性能。

Title: Pheromone-based Learning of Optimal Reasoning Paths

• Authors: Anirudh Chari, Aditya Tiwari, Richard Lian, Suraj Reddy, Brian Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19278
• Pdf URL: https://arxiv.org/pdf/2501.19278
• Copy Paste: [[2501.19278]] Pheromone-based Learning of Optimal Reasoning Paths(https://arxiv.org/abs/2501.19278)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through chain-of-thought prompting, yet discovering effective reasoning methods for complex problems remains challenging due to the vast space of possible intermediate steps. We introduce Ant Colony Optimization-guided Tree of Thought (ACO-ToT), a novel algorithm that combines ACO with LLMs to discover optimal reasoning paths for complex problems efficiently. Drawing inspiration from Hebbian learning in neurological systems, our method employs a collection of distinctly fine-tuned LLM "ants" to traverse and lay pheromone trails through a centralized tree of thought, with each ant's movement governed by a weighted combination of existing pheromone trails and its own specialized expertise. The algorithm evaluates complete reasoning paths using a mixture-of-experts-based scoring function, with pheromones reinforcing productive reasoning paths across iterations. Experiments on three challenging reasoning tasks (GSM8K, ARC-Challenge, and MATH) demonstrate that ACO-ToT performs significantly better than existing chain-of-thought optimization approaches, suggesting that incorporating biologically inspired collective search mechanisms into LLM inference can substantially enhance reasoning capabilities.
• 摘要：大型语言模型 (LLM) 通过思维链提示展示了卓越的推理能力，但由于可能的中间步骤空间巨大，发现复杂问题的有效推理方法仍然具有挑战性。我们引入了蚁群优化引导的思维树 (ACO-ToT)，这是一种将 ACO 与 LLM 相结合的新型算法，可有效地发现复杂问题的最佳推理路径。从神经系统中的赫布学习中汲取灵感，我们的方法采用了一组经过明显微调的 LLM“蚂蚁”来穿越和铺设集中思维树中的信息素轨迹，每只蚂蚁的移动都由现有信息素轨迹和其自身专业知识的加权组合控制。该算法使用基于专家混合的评分函数评估完整的推理路径，信息素在迭代过程中强化了富有成效的推理路径。在三个具有挑战性的推理任务（GSM8K、ARC-Challenge 和 MATH）上的实验表明，ACO-ToT 的表现明显优于现有的思路链优化方法，这表明将受生物启发的集体搜索机制融入 LLM 推理中可以显著增强推理能力。

Title: Beyond checkmate: exploring the creative chokepoints in AI text

• Authors: Nafis Irtiza Tripto, Saranya Venkatraman, Mahjabin Nahar, Dongwon Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19301
• Pdf URL: https://arxiv.org/pdf/2501.19301
• Copy Paste: [[2501.19301]] Beyond checkmate: exploring the creative chokepoints in AI text(https://arxiv.org/abs/2501.19301)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) and Artificial Intelligence (AI), unlocking unprecedented capabilities. This rapid advancement has spurred research into various aspects of LLMs, their text generation & reasoning capability, and potential misuse, fueling the necessity for robust detection methods. While numerous prior research has focused on detecting LLM-generated text (AI text) and thus checkmating them, our study investigates a relatively unexplored territory: portraying the nuanced distinctions between human and AI texts across text segments. Whether LLMs struggle with or excel at incorporating linguistic ingenuity across different text segments carries substantial implications for determining their potential as effective creative assistants to humans. Through an analogy with the structure of chess games-comprising opening, middle, and end games-we analyze text segments (introduction, body, and conclusion) to determine where the most significant distinctions between human and AI texts exist. While AI texts can approximate the body segment better due to its increased length, a closer examination reveals a pronounced disparity, highlighting the importance of this segment in AI text detection. Additionally, human texts exhibit higher cross-segment differences compared to AI texts. Overall, our research can shed light on the intricacies of human-AI text distinctions, offering novel insights for text detection and understanding.
• 摘要：大型语言模型 (LLM) 彻底改变了自然语言处理 (NLP) 和人工智能 (AI)，释放了前所未有的能力。这一快速发展推动了对 LLM 各个方面的研究，包括其文本生成和推理能力以及潜在的滥用，从而推动了对强大检测方法的需求。虽然许多先前的研究都集中在检测 LLM 生成的文本（AI 文本）并因此将其淘汰，但我们的研究调查了一个相对未开发的领域：描绘人类和 AI 文本在文本片段之间的细微差别。LLM 在不同文本片段中融入语言独创性的能力是困难还是出色，对于确定它们作为人类有效创意助手的潜力具有重大意义。通过类比国际象棋游戏的结构（包括开局、中局和残局），我们分析文本片段（引言、正文和结论），以确定人类和 AI 文本之间最显着的区别在哪里。虽然 AI 文本可以更好地近似正文部分，因为正文部分的长度增加了，但仔细检查就会发现存在明显差异，这凸显了该部分在 AI 文本检测中的重要性。此外，与 AI 文本相比，人类文本表现出更高的跨段差异。总体而言，我们的研究可以揭示人类与 AI 文本区别的复杂性，为文本检测和理解提供新颖的见解。

Title: An Efficient Approach for Machine Translation on Low-resource Languages: A Case Study in Vietnamese-Chinese

• Authors: Tran Ngoc Son, Nguyen Anh Tu, Nguyen Minh Tri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19314
• Pdf URL: https://arxiv.org/pdf/2501.19314
• Copy Paste: [[2501.19314]] An Efficient Approach for Machine Translation on Low-resource Languages: A Case Study in Vietnamese-Chinese(https://arxiv.org/abs/2501.19314)
• Keywords: language model
• Abstract: Despite the rise of recent neural networks in machine translation, those networks do not work well if the training data is insufficient. In this paper, we proposed an approach for machine translation in low-resource languages such as Vietnamese-Chinese. Our proposed method leveraged the power of the multilingual pre-trained language model (mBART) and both Vietnamese and Chinese monolingual corpus. Firstly, we built an early bird machine translation model using the bilingual training dataset. Secondly, we used TF-IDF technique to select sentences from the monolingual corpus which are the most related to domains of the parallel dataset. Finally, the first model was used to synthesize the augmented training data from the selected monolingual corpus for the translation model. Our proposed scheme showed that it outperformed 8% compared to the transformer model. The augmented dataset also pushed the model performance.
• 摘要：尽管近年来神经网络在机器翻译领域的应用不断涌现，但如果训练数据不足，这些网络的效果就会不佳。在本文中，我们提出了一种用于越南语-中文等低资源语言的机器翻译方法。我们提出的方法利用了多语言预训练语言模型 (mBART) 以及越南语和中文单语语料库的强大功能。首先，我们使用双语训练数据集构建了一个早期的机器翻译模型。其次，我们使用 TF-IDF 技术从单语语料库中选择与并行数据集领域最相关的句子。最后，第一个模型用于从选定的单语语料库中合成增强的训练数据用于翻译模型。我们提出的方案表明，与 Transformer 模型相比，它的性能提高了 8%。增强的数据集也提高了模型性能。

Title: LLM-based Affective Text Generation Quality Based on Different Quantization Values

• Authors: Yarik Menchaca Resendiz, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19317
• Pdf URL: https://arxiv.org/pdf/2501.19317
• Copy Paste: [[2501.19317]] LLM-based Affective Text Generation Quality Based on Different Quantization Values(https://arxiv.org/abs/2501.19317)
• Keywords: language model, llm, prompt
• Abstract: Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a large number of parameters, requiring significant computational resources for training and inference. In some scenarios, accessing these resources can be challenging (e.g., budget or hardware limitations). Techniques like reducing precision bits can make models more memory-efficient, reducing the computational resources needed, at the cost of reduced accuracy. This paper addresses the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation (e.g., "I really enjoy running in the snow-covered forest"). To evaluate, we use an emotion classifier and ten seed prompts to generate affective text. We test three setups of precision bits (8, 16, and 32) across five open-weight language models from two different families. Our findings demonstrate that bit reductions lead to memory savings, achieving a reduction of 76%. However, this optimization comes with a trade-off, leading to a decrease of up to 10 pp in F1 score for larger models and an increase of 10 pp for smaller models, along with roughly double the inference time. In terms of text quality, larger models at lower quantization levels generally outperform smaller, higher-precision models -- while requiring similar memory.
• 摘要：大型语言模型在语言生成和理解方面表现出非凡的能力。这些进步使人工智能系统能够生成更像人类、更具情感吸引力的文本。然而，这些模型依赖于大量参数，需要大量计算资源进行训练和推理。在某些情况下，访问这些资源可能具有挑战性（例如，预算或硬件限制）。降低精度位等技术可以使模型更节省内存，减少所需的计算资源，但代价是降低准确性。本文讨论了情感文本生成中不同量化值、GPU RAM 利用率和文本质量之间的权衡（例如，“我真的很喜欢在白雪覆盖的森林里奔跑”）。为了进行评估，我们使用情感分类器和十个种子提示来生成情感文本。我们在来自两个不同系列的五个开放权重语言模型中测试了三种精度位设置（8、16 和 32）。我们的研究结果表明，减少位数可以节省内存，实现 76% 的减少。然而，这种优化是有代价的，会导致较大模型的 F1 得分下降最多 10 pp，较小模型的 F1 得分上升 10 pp，同时推理时间大约增加一倍。就文本质量而言，较低量化水平的较大模型通常优于较小、更高精度的模型——同时需要的内存相似。

Title: Reward-Guided Speculative Decoding for Efficient LLM Reasoning

• Authors: Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, Caiming Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19324
• Pdf URL: https://arxiv.org/pdf/2501.19324
• Copy Paste: [[2501.19324]] Reward-Guided Speculative Decoding for Efficient LLM Reasoning(https://arxiv.org/abs/2501.19324)
• Keywords: language model, llm
• Abstract: We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios.
• 摘要：我们引入了奖励引导推测解码 (RSD)，这是一种旨在提高大型语言模型 (LLM) 推理效率的新框架。RSD 协同结合了轻量级草稿模型和更强大的目标模型，结合受控偏差来优先考虑高奖励输出，这与强制严格无偏的现有推测解码方法形成鲜明对比。RSD 采用过程奖励模型来评估中间解码步骤并动态决定是否调用目标模型，从而优化计算成本和输出质量之间的权衡。我们从理论上证明了基于阈值的混合策略在资源利用率和性能之间实现了最佳平衡。对包括奥林匹克级任务在内的具有挑战性的推理基准进行的广泛评估表明，与仅使用目标模型进行解码相比，RSD 可显著提高效率（最多减少 4.4 倍的 FLOP），同时平均比并行解码方法实现更高的准确度（最多 +3.5）。这些结果突出表明，RSD 是一种在资源密集型场景中部署 LLM 的稳健且经济高效的方法。

Title: Homogeneity Bias as Differential Sampling Uncertainty in Language Models

• Authors: Messi H.J. Lee, Soyeon Jeon
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19337
• Pdf URL: https://arxiv.org/pdf/2501.19337
• Copy Paste: [[2501.19337]] Homogeneity Bias as Differential Sampling Uncertainty in Language Models(https://arxiv.org/abs/2501.19337)
• Keywords: language model, gpt, llm
• Abstract: Prior research show that Large Language Models (LLMs) and Vision-Language Models (VLMs) represent marginalized groups more homogeneously than dominant groups. However, the mechanisms underlying this homogeneity bias remain relatively unexplored. We propose that this bias emerges from systematic differences in the probability distributions from which tokens are sampled at inference-time. Analyzing three measures of uncertainty in token sampling distributions-entropy, perplexity, and probability of differentiation-we find that in some models, specifically GPT-4 Turbo and Llama-3.2, tokens are sampled more deterministically when generating texts about marginalized groups (i.e., Black Americans and women) compared to their dominant group counterparts (i.e., White Americans and men). While these findings may help explain homogeneity bias in certain models, the patterns did not replicate across all VLMs tested, suggesting multiple mechanisms may contribute to homogeneity bias in AI.
• 摘要：先前的研究表明，大型语言模型 (LLM) 和视觉语言模型 (VLM) 对边缘化群体的表示比对主导群体的表示更同质。然而，这种同质性偏差背后的机制仍然相对未被探索。我们认为，这种偏差源于在推理时对 token 进行采样的概率分布的系统性差异。通过分析 token 采样分布中的三个不确定性度量——熵、困惑度和分化概率——我们发现，在某些模型中，特别是 GPT-4 Turbo 和 Llama-3.2，在生成关于边缘化群体（即美国黑人和女性）的文本时，与主流群体（即美国白人和男性）相比，token 的采样更具确定性。虽然这些发现可能有助于解释某些模型中的同质性偏差，但这些模式并未在所有测试的 VLM 中重现，这表明多种机制可能导致人工智能中的同质性偏差。

Title: Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SCICAP Challenge 2023

• Authors: Ting-Yao E. Hsu, Yi-Li Hsu, Shaurya Rohatgi, Chieh-Yang Huang, Ho Yin Sam Ng, Ryan Rossi, Sungchul Kim, Tong Yu, Lun-Wei Ku, C. Lee Giles, Ting-Hao K. Huang
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19353
• Pdf URL: https://arxiv.org/pdf/2501.19353
• Copy Paste: [[2501.19353]] Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SCICAP Challenge 2023(https://arxiv.org/abs/2501.19353)
• Keywords: gpt
• Abstract: Since the SCICAP datasets launch in 2021, the research community has made significant progress in generating captions for scientific figures in scholarly articles. In 2023, the first SCICAP Challenge took place, inviting global teams to use an expanded SCICAP dataset to develop models for captioning diverse figure types across various academic fields. At the same time, text generation models advanced quickly, with many powerful pre-trained large multimodal models (LMMs) emerging that showed impressive capabilities in various vision-and-language tasks. This paper presents an overview of the first SCICAP Challenge and details the performance of various models on its data, capturing a snapshot of the fields state. We found that professional editors overwhelmingly preferred figure captions generated by GPT-4V over those from all other models and even the original captions written by authors. Following this key finding, we conducted detailed analyses to answer this question: Have advanced LMMs solved the task of generating captions for scientific figures?
• 摘要：自 2021 年 SCICAP 数据集推出以来，研究界在为学术文章中的科学图形生成标题方面取得了重大进展。2023 年，首届 SCICAP 挑战赛举行，邀请全球团队使用扩展的 SCICAP 数据集开发用于为各个学术领域的各种图形类型添加标题的模型。与此同时，文本生成模型发展迅速，许多功能强大的预训练大型多模态模型 (LMM) 应运而生，在各种视觉和语言任务中表现出令人印象深刻的能力。本文概述了第一届 SCICAP 挑战赛，并详细介绍了各种模型在其数据上的表现，捕捉了该领域状态的快照。我们发现，专业编辑绝大多数更喜欢 GPT-4V 生成的图形标题，而不是所有其他模型的标题，甚至是作者编写的原始标题。根据这一关键发现，我们进行了详细分析以回答这个问题：先进的 LMM 是否解决了为科学图形生成标题的任务？

Title: TableMaster: A Recipe to Advance Table Understanding with Language Models

• Authors: Lang Cao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19378
• Pdf URL: https://arxiv.org/pdf/2501.19378
• Copy Paste: [[2501.19378]] TableMaster: A Recipe to Advance Table Understanding with Language Models(https://arxiv.org/abs/2501.19378)
• Keywords: language model, gpt
• Abstract: Tables serve as a fundamental format for representing structured relational data. While current language models (LMs) excel at many text-based tasks, they still face challenges in table understanding due to the complex characteristics of tabular data, such as their structured nature. In this paper, we aim to enhance LMs for improved table understanding. We identify four key challenges: 1) difficulty in locating target data, 2) deficiency in table semantics, 3) numerical inaccuracies in textual reasoning, and 4) semantic inflexibility in symbolic reasoning. To address these issues, we propose TableMaster, a recipe and comprehensive framework that integrates multiple solutions to overcome these obstacles. TableMaster first extracts relevant table content and verbalizes it with enriched semantic context. Additionally, we introduce adaptive reasoning, a flexible approach that dynamically adjusts between textual and symbolic reasoning, tailoring the reasoning process to each query. Extensive analyses and experiments demonstrate our findings and the effectiveness of TableMaster. On the WikiTQ dataset, TableMaster achieves an accuracy of 78.13% using GPT-4o-mini, surpassing existing baselines.
• 摘要：表格是表示结构化关系数据的基本格式。虽然当前的语言模型 (LM) 在许多基于文本的任务上表现出色，但由于表格数据的复杂特征（例如其结构化性质），它们在表格理解方面仍然面临挑战。在本文中，我们旨在增强 LM 以改善表格理解。我们发现了四个主要挑战：1) 难以定位目标数据，2) 表格语义不足，3) 文本推理中的数字不准确，以及 4) 符号推理中的语义不灵活性。为了解决这些问题，我们提出了 TableMaster，这是一个集成多种解决方案来克服这些障碍的配方和综合框架。TableMaster 首先提取相关的表格内容，并使用丰富的语义上下文将其语言化。此外，我们引入了自适应推理，这是一种灵活的方法，可以在文本和符号推理之间动态调整，根据每个查询定制推理过程。大量的分析和实验证明了我们的发现和 TableMaster 的有效性。在 WikiTQ 数据集上，TableMaster 使用 GPT-4o-mini 实现了 78.13% 的准确率，超越了现有基线。

Title: s1: Simple test-time scaling

• Authors: Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, Tatsunori Hashimoto
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19393
• Pdf URL: https://arxiv.org/pdf/2501.19393
• Copy Paste: [[2501.19393]] s1: Simple test-time scaling(https://arxiv.org/abs/2501.19393)
• Keywords: language model
• Abstract: Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset s1K of 1,000 questions paired with reasoning traces relying on three criteria we validate through ablations: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending "Wait" multiple times to the model's generation when it tries to end. This can lead the model to double-check its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1 exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). Further, scaling s1 with budget forcing allows extrapolating beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at this https URL.
• 摘要：测试时间扩展是一种很有前途的语言建模新方法，它使用额外的测试时间计算来提高性能。最近，OpenAI 的 o1 模型展示了这种能力，但并未公开分享其方法，导致了许多复制工作。我们寻求最简单的方法来实现测试时间扩展和强大的推理性能。首先，我们策划了一个小型数据集 s1K，其中包含 1,000 个问题，并根据我们通过消融验证的三个标准与推理轨迹配对：难度、多样性和质量。其次，我们开发了预算强制来控制测试时间计算，方法是强制终止模型的思考过程，或者在模型试图结束时多次将“等待”附加到模型的生成中以延长它。这可能会导致模型仔细检查其答案，通常可以修复不正确的推理步骤。在 s1K 上对 Qwen2.5-32B-Instruct 语言模型进行监督微调并为其配备预算强制后，我们的模型 s1 在竞赛数学问题上的表现比 o1-preview 高出多达 27%（MATH 和 AIME24）。此外，使用预算强制扩展 s1 可以在没有测试时间干预的情况下推断出其性能：在 AIME24 上从 50% 提高到 57%。我们的模型、数据和代码在此 https URL 上开源。

Title: Scalable-Softmax Is Superior for Attention

• Authors: Ken M. Nakanishi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19399
• Pdf URL: https://arxiv.org/pdf/2501.19399
• Copy Paste: [[2501.19399]] Scalable-Softmax Is Superior for Attention(https://arxiv.org/abs/2501.19399)
• Keywords: language model, long context
• Abstract: The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.
• 摘要：Softmax函数输出的向量的最大元素随着输入向量大小的增加而趋近于零。基于Transformer的语言模型依赖Softmax来计算注意力分数，这导致注意力分布随着上下文大小的增加而趋于平坦，从而降低模型对关键信息进行有效优先排序的能力，并潜在地限制了其长度泛化。针对该问题，我们提出了Scalable-Softmax（SSMax），它在输入向量大小变化的场景中替代Softmax。SSMax可以无缝集成到现有的基于Transformer的架构中。语言建模中的实验结果表明，使用SSMax的模型不仅在预训练过程中实现了更快的loss降低，而且在长上下文和关键信息检索中的表现也显著提升。此外，对注意力分数的分析表明，SSMax可以让模型在长上下文中也把注意力集中在关键信息上。此外，虽然从预训练开始就使用 SSMax 的模型实现了更好的长度泛化，但是已经开始预训练的模型仍然可以通过在预训练期间或之后用 SSMax 替换注意力层中的 Softmax 来获得部分这种能力。