2024-11-08

Title: Diversity Helps Jailbreak Large Language Models

• Authors: Weiliang Zhao, Daniel Ben-Levi, Junfeng Yang, Chengzhi Mao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04223
• Pdf URL: https://arxiv.org/pdf/2411.04223
• Copy Paste: [[2411.04223]] Diversity Helps Jailbreak Large Language Models(https://arxiv.org/abs/2411.04223)
• Keywords: language model, gpt, llm, chat
• Abstract: We have uncovered a powerful jailbreak technique that leverages large language models' ability to diverge from prior context, enabling them to bypass safety constraints and generate harmful outputs. By simply instructing the LLM to deviate and obfuscate previous attacks, our method dramatically outperforms existing approaches, achieving up to a 62% higher success rate in compromising nine leading chatbots, including GPT-4, Gemini, and Llama, while using only 13% of the queries. This revelation exposes a critical flaw in current LLM safety training, suggesting that existing methods may merely mask vulnerabilities rather than eliminate them. Our findings sound an urgent alarm for the need to revolutionize testing methodologies to ensure robust and reliable LLM security.
• 摘要：我们发现了一种强大的越狱技术，它利用大型语言模型偏离先前上下文的能力，使它们能够绕过安全约束并产生有害输出。通过简单地指示 LLM 偏离和混淆先前的攻击，我们的方法大大优于现有方法，在仅使用 13% 的查询的情况下，在入侵包括 GPT-4、Gemini 和 Llama 在内的九个领先聊天机器人时，成功率提高了 62%。这一发现暴露了当前 LLM 安全培训中的一个关键缺陷，表明现有方法可能只是掩盖了漏洞，而不是消除漏洞。我们的发现敲响了紧急警钟，需要彻底改变测试方法，以确保 LLM 安全可靠。

Title: Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models

• Authors: Saketh Bachu, Erfan Shayegani, Trishna Chakraborty, Rohit Lal, Arindam Dutta, Chengyu Song, Yue Dong, Nael Abu-Ghazaleh, Amit K. Roy-Chowdhury
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04291
• Pdf URL: https://arxiv.org/pdf/2411.04291
• Copy Paste: [[2411.04291]] Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models(https://arxiv.org/abs/2411.04291)
• Keywords: language model, llm
• Abstract: Vision-language models (VLMs) have improved significantly in multi-modal tasks, but their more complex architecture makes their safety alignment more challenging than the alignment of large language models (LLMs). In this paper, we reveal an unfair distribution of safety across the layers of VLM's vision encoder, with earlier and middle layers being disproportionately vulnerable to malicious inputs compared to the more robust final layers. This 'cross-layer' vulnerability stems from the model's inability to generalize its safety training from the default architectural settings used during training to unseen or out-of-distribution scenarios, leaving certain layers exposed. We conduct a comprehensive analysis by projecting activations from various intermediate layers and demonstrate that these layers are more likely to generate harmful outputs when exposed to malicious inputs. Our experiments with LLaVA-1.5 and Llama 3.2 show discrepancies in attack success rates and toxicity scores across layers, indicating that current safety alignment strategies focused on a single default layer are insufficient.
• 摘要：视觉语言模型 (VLM) 在多模态任务中取得了显著的进步，但它们更复杂的架构使得它们的安全性对齐比大型语言模型 (LLM) 的对齐更具挑战性。在本文中，我们揭示了 VLM 视觉编码器各层之间的安全性分布不公平，与更强大的最终层相比，早期层和中间层更容易受到恶意输入的攻击。这种“跨层”漏洞源于模型无法将其安全性训练从训练期间使用的默认架构设置推广到未见过或分布外的场景，从而使某些层暴露在外。我们通过投射来自各个中间层的激活来进行全面分析，并证明这些层在暴露于恶意输入时更有可能产生有害输出。我们对 LLaVA-1.5 和 Llama 3.2 的实验表明，各层之间的攻击成功率和毒性分数存在差异，这表明当前专注于单个默认层的安全对齐策略是不够的。

Title: Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

• Authors: Anand Syamkumar, Nora Tseng, Kaycie Barron, Shanglin Yang, Shamya Karumbaiah, Rheeya Uppal, Junjie Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04308
• Pdf URL: https://arxiv.org/pdf/2411.04308
• Copy Paste: [[2411.04308]] Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education(https://arxiv.org/abs/2411.04308)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) offer promise in generating educational content, providing instructor feedback, and reducing teacher workload on assessments. While prior studies have focused on studying LLM-powered learning analytics, limited research has examined how effective LLMs are in a bilingual context. In this paper, we study the effectiveness of multilingual large language models (MLLMs) across monolingual (English-only, Spanish-only) and bilingual (Spanglish) student writing. We present a learning analytics use case that details LLM performance in assessing acceptable and unacceptable explanations of Science and Social Science concepts. Our findings reveal a significant bias in the grading performance of pre-trained models for bilingual writing compared to English-only and Spanish-only writing. Following this, we fine-tune open-source MLLMs including Llama 3.1 and Mistral NeMo using synthetic datasets generated in English, Spanish, and Spanglish. Our experiments indicate that the models perform significantly better for all three languages after fine-tuning with bilingual data. This study highlights the potential of enhancing MLLM effectiveness to support authentic language practices amongst bilingual learners. It also aims to illustrate the value of incorporating non-English languages into the design and implementation of language models in education.
• 摘要：大型语言模型 (LLM) 有望生成教育内容、提供教师反馈并减轻教师的评估工作量。虽然先前的研究主要集中在研究 LLM 驱动的学习分析，但很少有研究考察 LLM 在双语环境中的有效性。在本文中，我们研究了多语言大型语言模型 (MLLM) 在单语（仅英语、仅西班牙语）和双语（西班牙语英语）学生写作中的有效性。我们提出了一个学习分析用例，详细说明了 LLM 在评估科学和社会科学概念的可接受和不可接受的解释方面的表现。我们的研究结果表明，与仅英语和仅西班牙语写作相比，预训练模型在双语写作的评分表现方面存在显著偏差。随后，我们使用以英语、西班牙语和西班牙语英语生成的合成数据集对开源 MLLM（包括 Llama 3.1 和 Mistral NeMo）进行了微调。我们的实验表明，在使用双语数据进行微调后，模型在这三种语言中的表现明显更好。本研究强调了提高 MLLM 有效性以支持双语学习者进行真实语言练习的潜力。它还旨在说明将非英语语言纳入教育语言模型的设计和实施中的价值。

Title: A Multilingual Sentiment Lexicon for Low-Resource Language Translation using Large Languages Models and Explainable AI

• Authors: Melusi Malinga, Isaac Lupanda, Mike Wa Nkongolo, Phil van Deventer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04316
• Pdf URL: https://arxiv.org/pdf/2411.04316
• Copy Paste: [[2411.04316]] A Multilingual Sentiment Lexicon for Low-Resource Language Translation using Large Languages Models and Explainable AI(https://arxiv.org/abs/2411.04316)
• Keywords: language model, llm
• Abstract: South Africa and the Democratic Republic of Congo (DRC) present a complex linguistic landscape with languages such as Zulu, Sepedi, Afrikaans, French, English, and Tshiluba (Ciluba), which creates unique challenges for AI-driven translation and sentiment analysis systems due to a lack of accurately labeled data. This study seeks to address these challenges by developing a multilingual lexicon designed for French and Tshiluba, now expanded to include translations in English, Afrikaans, Sepedi, and Zulu. The lexicon enhances cultural relevance in sentiment classification by integrating language-specific sentiment scores. A comprehensive testing corpus is created to support translation and sentiment analysis tasks, with machine learning models such as Random Forest, Support Vector Machine (SVM), Decision Trees, and Gaussian Naive Bayes (GNB) trained to predict sentiment across low resource languages (LRLs). Among them, the Random Forest model performed particularly well, capturing sentiment polarity and handling language-specific nuances effectively. Furthermore, Bidirectional Encoder Representations from Transformers (BERT), a Large Language Model (LLM), is applied to predict context-based sentiment with high accuracy, achieving 99% accuracy and 98% precision, outperforming other models. The BERT predictions were clarified using Explainable AI (XAI), improving transparency and fostering confidence in sentiment classification. Overall, findings demonstrate that the proposed lexicon and machine learning models significantly enhance translation and sentiment analysis for LRLs in South Africa and the DRC, laying a foundation for future AI models that support underrepresented languages, with applications across education, governance, and business in multilingual contexts.
• 摘要：南非和刚果民主共和国 (DRC) 的语言环境复杂，包含祖鲁语、塞佩迪语、南非荷兰语、法语、英语和奇卢巴语 (Ciluba) 等语言，由于缺乏准确标记的数据，这给人工智能驱动的翻译和情感分析系统带来了独特的挑战。本研究旨在通过开发专为法语和奇卢巴语设计的多语言词典来应对这些挑战，现在该词典已扩展到包括英语、南非荷兰语、塞佩迪语和祖鲁语的翻译。该词典通过整合特定语言的情感分数来增强情感分类中的文化相关性。创建了一个全面的测试语料库来支持翻译和情感分析任务，并使用机器学习模型（例如随机森林、支持向量机 (SVM)、决策树和高斯朴素贝叶斯 (GNB)）来训练它们以预测低资源语言 (LRL) 中的情感。其中，随机森林模型表现尤为出色，能够捕捉情绪极性并有效处理特定语言的细微差别。此外，大型语言模型 (LLM) Transformers 的双向编码器表示 (BERT) 被用于预测基于上下文的情绪，准确率高达 99%，精确度高达 98%，优于其他模型。BERT 预测使用可解释人工智能 (XAI) 进行了澄清，提高了透明度并增强了对情绪分类的信心。总体而言，研究结果表明，所提出的词典和机器学习模型显著增强了南非和刚果民主共和国 LRL 的翻译和情绪分析能力，为未来支持代表性不足的语言的人工智能模型奠定了基础，并应用于多语言环境中的教育、治理和商业领域。

Title: CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models

• Authors: Jierui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, Doyen Sahoo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04329
• Pdf URL: https://arxiv.org/pdf/2411.04329
• Copy Paste: [[2411.04329]] CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models(https://arxiv.org/abs/2411.04329)
• Keywords: language model, gpt, llm, agent
• Abstract: Pre-trained on massive amounts of code and text data, large language models (LLMs) have demonstrated remarkable achievements in performing code generation tasks. With additional execution-based feedback, these models can act as agents with capabilities to self-refine and improve generated code autonomously. However, on challenging coding tasks with extremely large search space, current agentic approaches still struggle with multi-stage planning, generating, and debugging. To address this problem, we propose CodeTree, a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. Specifically, we adopted a unified tree structure to explicitly explore different coding strategies, generate corresponding coding solutions, and subsequently refine the solutions. In each stage, critical decision-making (ranking, termination, expanding) of the exploration process is guided by both the environmental execution-based feedback and LLM-agent-generated feedback. We comprehensively evaluated CodeTree on 7 code generation benchmarks and demonstrated the significant performance gains of CodeTree against strong baselines. Using GPT-4o as the base model, we consistently achieved top results of 95.1 on HumanEval, 98.7 on MBPP, and 43.0 on CodeContests. On the challenging SWEBench benchmark, our approach led to significant performance gains.
• 摘要：大型语言模型 (LLM) 经过大量代码和文本数据的预训练，在执行代码生成任务方面取得了显著的成就。通过额外的基于执行的反馈，这些模型可以充当具有自我改进和自主改进生成代码能力的代理。然而，在具有极大搜索空间的具有挑战性的编码任务中，当前的代理方法仍然难以进行多阶段规划、生成和调试。为了解决这个问题，我们提出了 CodeTree，这是一个 LLM 代理框架，可以在代码生成过程的不同阶段有效地探索搜索空间。具体来说，我们采用了统一的树结构来明确探索不同的编码策略，生成相应的编码解决方案，然后改进解决方案。在每个阶段，探索过程的关键决策（排名、终止、扩展）都由基于环境执行的反馈和 LLM 代理生成的反馈指导。我们在 7 个代码生成基准上全面评估了 CodeTree，并证明了 CodeTree 相对于强大基线的显著性能提升。使用 GPT-4o 作为基础模型，我们在 HumanEval 上始终取得最高成绩 95.1，在 MBPP 上取得 98.7，在 CodeContests 上取得 43.0。在具有挑战性的 SWEBench 基准测试中，我们的方法显著提高了性能。

Title: Measuring short-form factuality in large language models

• Authors: Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04368
• Pdf URL: https://arxiv.org/pdf/2411.04368
• Copy Paste: [[2411.04368]] Measuring short-form factuality in large language models(https://arxiv.org/abs/2411.04368)
• Keywords: language model, gpt
• Abstract: We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at this https URL.
• 摘要：我们提出了 SimpleQA，这是一个评估语言模型回答简短、寻求事实的问题的能力的基准。在设计此评估时，我们优先考虑了两个属性。首先，SimpleQA 具有挑战性，因为它是针对 GPT-4 响应进行对抗性收集的。其次，响应易于评分，因为问题的创建方式使得只有一个无可争议的答案。SimpleQA 中的每个答案都被评为正确、不正确或未尝试。具有理想行为的模型会尽可能多地回答正确问题，而不会尝试回答它不确定自己知道正确答案的问题。SimpleQA 是一种简单、有针对性的评估，用于评估模型是否“知道它们所知道的”，我们希望这个基准将在未来几代前沿模型中保持相关性。SimpleQA 可在此 https URL 中找到。

Title: Bayesian Calibration of Win Rate Estimation with LLM Evaluators

• Authors: Yicheng Gao, Gonghan Xu, Zhe Wang, Arman Cohan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04424
• Pdf URL: https://arxiv.org/pdf/2411.04424
• Copy Paste: [[2411.04424]] Bayesian Calibration of Win Rate Estimation with LLM Evaluators(https://arxiv.org/abs/2411.04424)
• Keywords: language model, llm
• Abstract: Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation.
• 摘要：大型语言模型 (LLM) 的最新进展表明，使用 LLM 作为评估器来评估 LLM 文本生成质量的潜力。然而，由于 LLM 评估器固有的胜率估计偏差，天真地应用 LLM 评估器来比较或判断不同的系统可能会导致不可靠的结果。为了缓解这个问题，我们提出了两种校准方法，即贝叶斯胜率抽样 (BWRS) 和贝叶斯 Dawid-Skene，这两种方法都利用贝叶斯推理来更准确地推断生成语言模型的真实胜率。我们在六个数据集上对我们的方法进行了实证验证，这些数据集涵盖故事生成、摘要和指令遵循任务。我们表明，我们的两种方法都能有效提高使用 LLM 作为评估器的胜率估计的准确性，为可靠的自动文本质量评估提供了一个有希望的方向。

Title: DELIFT: Data Efficient Language model Instruction Fine Tuning

• Authors: Ishika Agarwal, Krishna Killamsetty, Lucian Popa, Marina Danilevksy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04425
• Pdf URL: https://arxiv.org/pdf/2411.04425
• Copy Paste: [[2411.04425]] DELIFT: Data Efficient Language model Instruction Fine Tuning(https://arxiv.org/abs/2411.04425)
• Keywords: language model, llm
• Abstract: Fine-tuning large language models (LLMs) is essential for enhancing their performance on specific tasks but is often resource-intensive due to redundant or uninformative data. To address this inefficiency, we introduce DELIFT (Data Efficient Language model Instruction Fine-Tuning), a novel algorithm that systematically optimizes data selection across the three key stages of fine-tuning: (1) instruction tuning, (2) task-specific fine-tuning (e.g., reasoning, question-answering), and (3) continual fine-tuning (e.g., incorporating new data versions). Unlike existing methods that focus on single-stage optimization or rely on computationally intensive gradient calculations, DELIFT operates efficiently across all stages. Central to our approach is a pairwise utility metric that quantifies how beneficial a data sample is for improving the model's responses to other samples, effectively measuring the informational value relative to the model's current capabilities. By leveraging different submodular functions applied to this metric, DELIFT selects diverse and optimal subsets that are useful across all stages of fine-tuning. Experiments across various tasks and model scales demonstrate that DELIFT can reduce the fine-tuning data size by up to 70% without compromising performance, offering significant computational savings and outperforming existing methods in both efficiency and efficacy.
• 摘要：对大型语言模型 (LLM) 进行微调对于提高其在特定任务上的性能至关重要，但由于数据冗余或信息量不足，通常会​​耗费大量资源。为了解决这种低效率问题，我们引入了 DELIFT（数据高效语言模型指令微调），这是一种新颖的算法，可系统地优化微调三个关键阶段的数据选择：（1）指令调整、（2）特定于任务的微调（例如推理、问答）和（3）持续微调（例如合并新数据版本）。与专注于单阶段优化或依赖计算密集型梯度计算的现有方法不同，DELIFT 在所有阶段都能有效运行。我们方法的核心是成对效用指标，它量化了数据样本对改善模型对其他样本的响应有多大益处，有效地衡量了相对于模型当前功能的信息价值。通过利用应用于该指标的不同子模块函数，DELIFT 选择了适用于所有微调阶段的多样化最佳子集。跨各种任务和模型规模的实验表明，DELIFT 可以在不影响性能的情况下将微调数据大小减少高达 70%，从而显著节省计算量，并在效率和功效方面均优于现有方法。

Title: One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity

• Authors: Sonia K. Murthy, Tomer Ullman, Jennifer Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04427
• Pdf URL: https://arxiv.org/pdf/2411.04427
• Copy Paste: [[2411.04427]] One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity(https://arxiv.org/abs/2411.04427)
• Keywords: language model, llm
• Abstract: Researchers in social science and psychology have recently proposed using large language models (LLMs) as replacements for humans in behavioral research. In addition to arguments about whether LLMs accurately capture population-level patterns, this has raised questions about whether LLMs capture human-like conceptual diversity. Separately, it is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity. Inspired by human studies, we use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data. While no model reaches human-like diversity, aligned models generally display less diversity than their instruction fine-tuned counterparts. Our findings highlight potential trade-offs between increasing models' value alignment and decreasing the diversity of their conceptual representations.
• 摘要：社会科学和心理学领域的研究人员最近提出使用大型语言模型 (LLM) 代替人类进行行为研究。除了关于 LLM 是否准确捕捉到人口水平模式的争论之外，这还引发了关于 LLM 是否捕捉到类似人类的概念多样性的问题。另外，关于训练后对齐 (RLHF 或 RLAIF) 是否会影响模型的内部多样性也存在争议。受人类研究的启发，我们使用一种新方法来衡量合成生成的 LLM“种群”的概念多样性，即将模拟个体的内部变异性与种群水平的变异性联系起来。我们使用这种方法在两个具有丰富人类行为数据的领域评估非对齐和对齐的 LLM。虽然没有一个模型达到类似人类的多样性，但对齐模型通常比其指令微调模型表现出更少的多样性。我们的研究结果强调了增加模型的价值对齐和降低其概念表示多样性之间的潜在权衡。

Title: ACCIO: Table Understanding Enhanced via Contrastive Learning with Aggregations

• Authors: Whanhee Cho
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2411.04443
• Pdf URL: https://arxiv.org/pdf/2411.04443
• Copy Paste: [[2411.04443]] ACCIO: Table Understanding Enhanced via Contrastive Learning with Aggregations(https://arxiv.org/abs/2411.04443)
• Keywords: language model
• Abstract: The attention to table understanding using recent natural language models has been growing. However, most related works tend to focus on learning the structure of the table directly. Just as humans improve their understanding of sentences by comparing them, they can also enhance their understanding by comparing tables. With this idea, in this paper, we introduce ACCIO, tAble understanding enhanCed via Contrastive learnIng with aggregatiOns, a novel approach to enhancing table understanding by contrasting original tables with their pivot summaries through contrastive learning. ACCIO trains an encoder to bring these table pairs closer together. Through validation via column type annotation, ACCIO achieves competitive performance with a macro F1 score of 91.1 compared to state-of-the-art methods. This work represents the first attempt to utilize pairs of tables for table embedding, promising significant advancements in table comprehension. Our code is available at this https URL.
• 摘要：使用最近的自然语言模型对表格理解的关注度日益增长。然而，大多数相关工作往往侧重于直接学习表格的结构。正如人类通过比较句子来提高对句子的理解一样，他们也可以通过比较表格来增强理解。基于这个想法，在本文中，我们介绍了 ACCIO，即通过聚合对比学习增强的表格理解，这是一种通过对比学习将原始表格与其枢轴摘要进行对比来增强表格理解的新方法。ACCIO 训练编码器以使这些表格对更紧密地结合在一起。通过列类型注释的验证，ACCIO 实现了与最先进方法相比具有竞争力的性能，宏 F1 得分为 91.1。这项工作代表了首次尝试利用表格对进行表格嵌入，有望在表格理解方面取得重大进展。我们的代码可在此 https URL 上找到。

Title: Gradient Localization Improves Lifelong Pretraining of Language Models

• Authors: Jared Fernandez, Yonatan Bisk, Emma Strubell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04448
• Pdf URL: https://arxiv.org/pdf/2411.04448
• Copy Paste: [[2411.04448]] Gradient Localization Improves Lifelong Pretraining of Language Models(https://arxiv.org/abs/2411.04448)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) trained on web-scale text corpora have been shown to capture world knowledge in their parameters. However, the mechanism by which language models store different types of knowledge is poorly understood. In this work, we examine two types of knowledge relating to temporally sensitive entities and demonstrate that each type is localized to different sets of parameters within the LLMs. We hypothesize that the lack of consideration of the locality of knowledge in existing continual learning methods contributes to both: the failed uptake of new information, and catastrophic forgetting of previously learned information. We observe that sequences containing references to updated and newly mentioned entities exhibit larger gradient norms in a subset of layers. We demonstrate that targeting parameter updates to these relevant layers can improve the performance of continually pretraining on language containing temporal drift.
• 摘要：研究表明，在网络规模的文本语料库上训练的大型语言模型 (LLM) 可以在其参数中捕获世界知识。然而，语言模型存储不同类型知识的机制尚不明确。在这项工作中，我们研究了两种与时间敏感实体相关的知识，并证明每种知识都位于 LLM 内不同的参数集上。我们假设，现有的持续学习方法缺乏对知识局部性的考虑，这导致了以下两种情况：无法吸收新信息，以及彻底遗忘以前学过的信息。我们观察到，包含对更新和新提及实体的引用的序列在层子集中表现出更大的梯度范数。我们证明，将参数更新定位到这些相关层可以提高对包含时间漂移的语言进行持续预训练的性能。

Title: ML-Promise: A Multilingual Dataset for Corporate Promise Verification

• Authors: Yohei Seki, Hakusen Shu, Anaïs Lhuissier, Hanwool Lee, Juyeon Kang, Min-Yuh Day, Chung-Chi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04473
• Pdf URL: https://arxiv.org/pdf/2411.04473
• Copy Paste: [[2411.04473]] ML-Promise: A Multilingual Dataset for Corporate Promise Verification(https://arxiv.org/abs/2411.04473)
• Keywords: llm, retrieval-augmented generation
• Abstract: Promises made by politicians, corporate leaders, and public figures have a significant impact on public perception, trust, and institutional reputation. However, the complexity and volume of such commitments, coupled with difficulties in verifying their fulfillment, necessitate innovative methods for assessing their credibility. This paper introduces the concept of Promise Verification, a systematic approach involving steps such as promise identification, evidence assessment, and the evaluation of timing for verification. We propose the first multilingual dataset, ML-Promise, which includes English, French, Chinese, Japanese, and Korean, aimed at facilitating in-depth verification of promises, particularly in the context of Environmental, Social, and Governance (ESG) reports. Given the growing emphasis on corporate environmental contributions, this dataset addresses the challenge of evaluating corporate promises, especially in light of practices like greenwashing. Our findings also explore textual and image-based baselines, with promising results from retrieval-augmented generation (RAG) approaches. This work aims to foster further discourse on the accountability of public commitments across multiple languages and domains.
• 摘要：政客、企业领导人和公众人物做出的承诺对公众的看法、信任和机构声誉有重大影响。然而，这些承诺的复杂性和数量，加上难以核实其履行情况，需要创新方法来评估其可信度。本文介绍了承诺验证的概念，这是一种系统的方法，涉及承诺识别、证据评估和验证时间评估等步骤。我们提出了第一个多语言数据集 ML-Promise，其中包括英语、法语、中文、日语和韩语，旨在促进对承诺的深入验证，特别是在环境、社会和治理 (ESG) 报告的背景下。鉴于企业对环境的贡献越来越受到重视，该数据集解决了评估企业承诺的挑战，特别是在漂绿等做法的背景下。我们的研究结果还探索了基于文本和图像的基线，检索增强生成 (RAG) 方法取得了有希望的结果。这项工作旨在促进跨多种语言和领域的公共承诺问责制的进一步讨论。

Title: Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model

• Authors: Young-Jun Lee, Dokyong Lee, Junyoung Youn, Kyeongjin Oh, Ho-Jin Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04496
• Pdf URL: https://arxiv.org/pdf/2411.04496
• Copy Paste: [[2411.04496]] Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model(https://arxiv.org/abs/2411.04496)
• Keywords: language model, llm, agent
• Abstract: To increase social bonding with interlocutors, humans naturally acquire the ability to respond appropriately in a given situation by considering which conversational skill is most suitable for the response - a process we call skill-of-mind. For large language model (LLM)-based conversational agents, planning appropriate conversational skills, as humans do, is challenging due to the complexity of social dialogue, especially in interactive scenarios. To address this, we propose a skill-of-mind-annotated conversation dataset, named Multifaceted Skill-of-Mind, which includes multi-turn and multifaceted conversational skills across various interactive scenarios (e.g., long-term, counseling, task-oriented), grounded in diverse social contexts (e.g., demographics, persona, rules of thumb). This dataset consists of roughly 100K conversations. Using this dataset, we introduce a new family of skill-of-mind-infused LLMs, named Thanos, with model sizes of 1B, 3B, and 8B parameters. With extensive experiments, these models successfully demonstrate the skill-of-mind process and exhibit strong generalizability in inferring multifaceted skills across a variety of domains. Moreover, we show that Thanos significantly enhances the quality of responses generated by LLM-based conversational agents and promotes prosocial behavior in human evaluations.
• 摘要：为了增进与对话者之间的社交联系，人类自然而然地获得了在特定情况下做出适当回应的能力，即通过考虑哪种对话技巧最适合回应 - 我们将这一过程称为心智技能。对于基于大型语言模型 (LLM) 的对话代理，由于社交对话的复杂性，尤其是在交互式场景中，像人类一样规划适当的对话技巧是一项挑战。为了解决这个问题，我们提出了一个带有心智技能注释的对话数据集，名为多面心智技能，其中包括各种交互式场景（例如长期、咨询、面向任务）的多轮和多面对话技巧，这些技巧基于不同的社交背景（例如人口统计、角色、经验法则）。该数据集包含大约 100K 次对话。使用此数据集，我们引入了一个名为 Thanos 的新型心智技能注入 LLM 系列，其模型大小为 1B、3B 和 8B 个参数。经过大量实验，这些模型成功展示了思维技能过程，并在推断各个领域的多方面技能方面表现出很强的通用性。此外，我们表明 Thanos 显著提高了基于 LLM 的对话代理生成的响应质量，并在人类评估中促进了亲社会行为。

Title: Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

• Authors: Xinyu Zhang, Jing Lu, Vinh Q. Tran, Tal Schuster, Donald Metzler, Jimmy Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04530
• Pdf URL: https://arxiv.org/pdf/2411.04530
• Copy Paste: [[2411.04530]] Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models(https://arxiv.org/abs/2411.04530)
• Keywords: language model
• Abstract: Human understanding of language is robust to different word choices as far as they represent similar semantic concepts. To what extent does our human intuition transfer to language models, which represent all subwords as distinct embeddings? In this work, we take an initial step on measuring the role of shared semantics among subwords in the encoder-only multilingual language models (mLMs). To this end, we form "semantic tokens" by merging the semantically similar subwords and their embeddings, and evaluate the updated mLMs on 5 heterogeneous multilingual downstream tasks. Results show that the general shared semantics could get the models a long way in making the predictions on mLMs with different tokenizers and model sizes. Inspections on the grouped subwords show that they exhibit a wide range of semantic similarities, including synonyms and translations across many languages and scripts. Lastly, we found the zero-shot results with semantic tokens are on par or even better than the original models on certain classification tasks, suggesting that the shared subword-level semantics may serve as the anchors for cross-lingual transferring.
• 摘要：只要不同的词语代表相似的语义概念，人类对语言的理解就对它们具有鲁棒性。我们的人类直觉在多大程度上转移到将所有子词表示为不同嵌入的语言模型？在这项工作中，我们迈出了第一步，测量了仅编码器的多语言语言模型 (mLM) 中子词之间共享语义的作用。为此，我们通过合并语义相似的子词及其嵌入来形成“语义标记”，并在 5 个异构多语言下游任务上评估更新后的 mLM。结果表明，一般的共享语义可以使模型在对具有不同标记器和模型大小的 mLM 进行预测方面取得很大进展。对分组子词的检查表明，它们表现出广泛的语义相似性，包括跨多种语言和脚本的同义词和翻译。最后，我们发现在某些分类任务上，具有语义标记的零样本结果与原始模型相当甚至更好，这表明共享的子字级语义可以作为跨语言转换的锚点。

Title: Meta-Reasoning Improves Tool Use in Large Language Models

• Authors: Lisa Alazraki, Marek Rei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04535
• Pdf URL: https://arxiv.org/pdf/2411.04535
• Copy Paste: [[2411.04535]] Meta-Reasoning Improves Tool Use in Large Language Models(https://arxiv.org/abs/2411.04535)
• Keywords: language model, llm
• Abstract: External tools help large language models (LLMs) succeed at tasks where they would otherwise typically fail. In existing frameworks, LLMs learn tool use either by in-context demonstrations or via full model fine-tuning on annotated data. As these approaches do not easily scale, a recent trend is to abandon them in favor of lightweight, parameter-efficient tuning paradigms. These methods allow quickly alternating between the frozen LLM and its specialised fine-tuned version, by switching on or off a handful of additional custom parameters. Hence, we postulate that the generalization ability of the frozen model can be leveraged to improve tool selection. We present Tool selECTion via meta-reasONing (TECTON), a two-phase system that first reasons over a task using a custom fine-tuned LM head and outputs candidate tools. Then, with the custom head disabled, it meta-reasons (i.e., it reasons over the previous reasoning process) to make a final choice. We show that TECTON results in substantial gains - both in-distribution and out-of-distribution - on a range of math reasoning datasets.
• 摘要：外部工具可帮助大型语言模型 (LLM) 成功完成原本通常会失败的任务。在现有框架中，LLM 通过上下文演示或对带注释的数据进行完整模型微调来学习工具的使用。由于这些方法不易扩展，最近的趋势是放弃它们，转而采用轻量级、参数高效的调整范例。这些方法允许通过打开或关闭一些额外的自定义参数在冻结的 LLM 及其专门的微调版本之间快速切换。因此，我们假设可以利用冻结模型的泛化能力来改进工具选择。我们提出了通过元推理进行工具选择 (TECTON)，这是一个两阶段系统，首先使用自定义微调的 LM 头对任务进行推理并输出候选工具。然后，在禁用自定义头的情况下，它会进行元推理（即，它对之前的推理过程进行推理）以做出最终选择。我们表明，TECTON 在一系列数学推理数据集上取得了显著的进步——包括分布内和分布外的进步。

Title: Tibyan Corpus: Balanced and Comprehensive Error Coverage Corpus Using ChatGPT for Arabic Grammatical Error Correction

• Authors: Ahlam Alrehili, Areej Alhothali
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04588
• Pdf URL: https://arxiv.org/pdf/2411.04588
• Copy Paste: [[2411.04588]] Tibyan Corpus: Balanced and Comprehensive Error Coverage Corpus Using ChatGPT for Arabic Grammatical Error Correction(https://arxiv.org/abs/2411.04588)
• Keywords: gpt, chat
• Abstract: Natural language processing (NLP) utilizes text data augmentation to overcome sample size constraints. Increasing the sample size is a natural and widely used strategy for alleviating these challenges. In this study, we chose Arabic to increase the sample size and correct grammatical errors. Arabic is considered one of the languages with limited resources for grammatical error correction (GEC). Furthermore, QALB-14 and QALB-15 are the only datasets used in most Arabic grammatical error correction research, with approximately 20,500 parallel examples, which is considered low compared with other languages. Therefore, this study aims to develop an Arabic corpus called "Tibyan" for grammatical error correction using ChatGPT. ChatGPT is used as a data augmenter tool based on a pair of Arabic sentences containing grammatical errors matched with a sentence free of errors extracted from Arabic books, called guide sentences. Multiple steps were involved in establishing our corpus, including the collection and pre-processing of a pair of Arabic texts from various sources, such as books and open-access corpora. We then used ChatGPT to generate a parallel corpus based on the text collected previously, as a guide for generating sentences with multiple types of errors. By engaging linguistic experts to review and validate the automatically generated sentences, we ensured that they were correct and error-free. The corpus was validated and refined iteratively based on feedback provided by linguistic experts to improve its accuracy. Finally, we used the Arabic Error Type Annotation tool (ARETA) to analyze the types of errors in the Tibyan corpus. Our corpus contained 49 of errors, including seven types: orthography, morphology, syntax, semantics, punctuation, merge, and split. The Tibyan corpus contains approximately 600 K tokens.
• 摘要：自然语言处理 (NLP) 利用文本数据增强来克服样本量限制。增加样本量是一种自然且广泛使用的缓解这些挑战的策略。在本研究中，我们选择阿拉伯语来增加样本量并纠正语法错误。阿拉伯语被认为是语法错误纠正 (GEC) 资源有限的语言之一。此外，QALB-14 和 QALB-15 是大多数阿拉伯语法错误纠正研究中使用的唯一数据集，大约有 20,500 个并行示例，与其他语言相比，这个数字被认为是较低的。因此，本研究旨在开发一个名为“Tibyan”的阿拉伯语语料库，用于使用 ChatGPT 进行语法错误纠正。ChatGPT 用作数据增强工具，基于一对包含语法错误的阿拉伯语句子与从阿拉伯语书籍中提取的无错误的句子（称为引导句）相匹配。建立我们的语料库涉及多个步骤，包括从各种来源（例如书籍和开放获取语料库）收集和预处理一对阿拉伯语文本。然后，我们使用 ChatGPT 根据之前收集的文本生成一个平行语料库，作为生成具有多种错误类型的句子的指南。通过聘请语言专家审查和验证自动生成的句子，我们确保它们是正确且无错误的。根据语言专家提供的反馈，我们反复验证和完善语料库，以提高其准确性。最后，我们使用阿拉伯语错误类型注释工具 (ARETA) 分析 Tibyan 语料库中的错误类型。我们的语料库包含 49 种错误，包括七种类型：正字法、形态学、语法、语义、标点符号、合并和拆分。Tibyan 语料库包含大约 600 K 个标记。

Title: Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop

• Authors: Ekaterina Artemova, Akim Tsvigun, Dominik Schlechtweg, Natalia Fedorova, Sergei Tilga, Boris Obmoroshev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04637
• Pdf URL: https://arxiv.org/pdf/2411.04637
• Copy Paste: [[2411.04637]] Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop(https://arxiv.org/abs/2411.04637)
• Keywords: llm
• Abstract: Training and deploying machine learning models relies on a large amount of human-annotated data. As human labeling becomes increasingly expensive and time-consuming, recent research has developed multiple strategies to speed up annotation and reduce costs and human workload: generating synthetic training data, active learning, and hybrid labeling. This tutorial is oriented toward practical applications: we will present the basics of each strategy, highlight their benefits and limitations, and discuss in detail real-life case studies. Additionally, we will walk through best practices for managing human annotators and controlling the quality of the final dataset. The tutorial includes a hands-on workshop, where attendees will be guided in implementing a hybrid annotation setup. This tutorial is designed for NLP practitioners from both research and industry backgrounds who are involved in or interested in optimizing data labeling projects.
• 摘要：训练和部署机器学习模型依赖于大量人工注释的数据。随着人工标记变得越来越昂贵和耗时，最近的研究已经开发出多种策略来加速注释并降低成本和人工工作量：生成合成训练数据、主动学习和混合标记。本教程面向实际应用：我们将介绍每种策略的基础知识，重点介绍它们的优点和局限性，并详细讨论实际案例研究。此外，我们将介绍管理人工注释者和控制最终数据集质量的最佳实践。本教程包括一个实践研讨会，与会者将在研讨会上得到混合注释设置的指导。本教程面向参与或有兴趣优化数据标记项目的研究和行业背景的 NLP 从业者。

Title: RetrieveGPT: Merging Prompts and Mathematical Models for Enhanced Code-Mixed Information Retrieval

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04752
• Pdf URL: https://arxiv.org/pdf/2411.04752
• Copy Paste: [[2411.04752]] RetrieveGPT: Merging Prompts and Mathematical Models for Enhanced Code-Mixed Information Retrieval(https://arxiv.org/abs/2411.04752)
• Keywords: gpt, prompt
• Abstract: Code-mixing, the integration of lexical and grammatical elements from multiple languages within a single sentence, is a widespread linguistic phenomenon, particularly prevalent in multilingual societies. In India, social media users frequently engage in code-mixed conversations using the Roman script, especially among migrant communities who form online groups to share relevant local information. This paper focuses on the challenges of extracting relevant information from code-mixed conversations, specifically within Roman transliterated Bengali mixed with English. This study presents a novel approach to address these challenges by developing a mechanism to automatically identify the most relevant answers from code-mixed conversations. We have experimented with a dataset comprising of queries and documents from Facebook, and Query Relevance files (QRels) to aid in this task. Our results demonstrate the effectiveness of our approach in extracting pertinent information from complex, code-mixed digital conversations, contributing to the broader field of natural language processing in multilingual and informal text environments. We use GPT-3.5 Turbo via prompting alongwith using the sequential nature of relevant documents to frame a mathematical model which helps to detect relevant documents corresponding to a query.
• 摘要：代码混合是指将多种语言的词汇和语法元素整合到一个句子中，这是一种普遍的语言现象，在多语言社会中尤为普遍。在印度，社交媒体用户经常使用罗马文字进行代码混合对话，尤其是在组成在线群组以分享相关本地信息的移民社区中。本文重点介绍从代码混合对话中提取相关信息的挑战，特别是在罗马音译孟加拉语与英语混合的对话中。本研究提出了一种应对这些挑战的新方法，即开发一种机制来自动识别代码混合对话中最相关的答案。我们已经尝试了一个由来自 Facebook 的查询和文档组成的数据集，以及查询相关性文件 (QRels) 来帮助完成这项任务。我们的结果证明了我们的方法在从复杂的代码混合数字对话中提取相关信息方面的有效性，为多语言和非正式文本环境中更广泛的自然语言处理领域做出了贡献。我们通过提示使用 GPT-3.5 Turbo，并利用相关文档的顺序性来构建数学模型，有助于检测与查询相对应的相关文档。

Title: A study of Vietnamese readability assessing through semantic and statistical features

• Authors: Hung Tuan Le, Long Truong To, Manh Trong Nguyen, Quyen Nguyen, Trong-Hop Do
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04756
• Pdf URL: https://arxiv.org/pdf/2411.04756
• Copy Paste: [[2411.04756]] A study of Vietnamese readability assessing through semantic and statistical features(https://arxiv.org/abs/2411.04756)
• Keywords: language model
• Abstract: Determining the difficulty of a text involves assessing various textual features that may impact the reader's text comprehension, yet current research in Vietnamese has only focused on statistical features. This paper introduces a new approach that integrates statistical and semantic approaches to assessing text readability. Our research utilized three distinct datasets: the Vietnamese Text Readability Dataset (ViRead), OneStopEnglish, and RACE, with the latter two translated into Vietnamese. Advanced semantic analysis methods were employed for the semantic aspect using state-of-the-art language models such as PhoBERT, ViDeBERTa, and ViBERT. In addition, statistical methods were incorporated to extract syntactic and lexical features of the text. We conducted experiments using various machine learning models, including Support Vector Machine (SVM), Random Forest, and Extra Trees and evaluated their performance using accuracy and F1 score metrics. Our results indicate that a joint approach that combines semantic and statistical features significantly enhances the accuracy of readability classification compared to using each method in isolation. The current study emphasizes the importance of considering both statistical and semantic aspects for a more accurate assessment of text difficulty in Vietnamese. This contribution to the field provides insights into the adaptability of advanced language models in the context of Vietnamese text readability. It lays the groundwork for future research in this area.
• 摘要：确定文本的难度需要评估可能影响读者文本理解的各种文本特征，但目前越南语研究仅侧重于统计特征。本文介绍了一种结合统计和语义方法来评估文本可读性的新方法。我们的研究使用了三个不同的数据集：越南语文本可读性数据集 (ViRead)、OneStopEnglish 和 RACE，后两者已翻译成越南语。我们使用最先进的语言模型（如 PhoBERT、ViDeBERTa 和 ViBERT）对语义方面采用了高级语义分析方法。此外，还结合了统计方法来提取文本的句法和词汇特征。我们使用各种机器学习模型（包括支持向量机 (SVM)、随机森林和 Extra Trees）进行了实验，并使用准确度和 F1 分数指标评估了它们的性能。我们的结果表明，与单独使用每种方法相比，结合语义和统计特征的联合方法显著提高了可读性分类的准确性。本研究强调了考虑统计和语义方面的重要性，以便更准确地评估越南语文本难度。该研究为该领域的贡献提供了对高级语言模型在越南语文本可读性方面的适应性的见解。它为该领域的未来研究奠定了基础。

Title: AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment

• Authors: Yuxin Zuo, Wenxuan Jiang, Wenxuan Liu, Zixuan Li, Long Bai, Hanbin Wang, Yutao Zeng, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04794
• Pdf URL: https://arxiv.org/pdf/2411.04794
• Copy Paste: [[2411.04794]] AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment(https://arxiv.org/abs/2411.04794)
• Keywords: gpt, llm, chat
• Abstract: Empirical evidence suggests that LLMs exhibit spontaneous cross-lingual alignment. Our findings suggest that although LLMs also demonstrate promising cross-lingual alignment in Information Extraction, there remains significant imbalance across languages, revealing an underlying deficiency in the IE alignment. To address this issue, we propose AlignXIE, a powerful code-based LLM that significantly enhances cross-lingual IE alignment through two strategies. Firstly, AlignXIE formulates IE across different languages, especially non-English ones, as code generation tasks, standardizing the representation of various schemas using Python classes to ensure consistency of the same ontology in different languages and align the schema. Secondly, it incorporates an IE cross-lingual alignment phase through a translated instance prediction task proposed in this paper to align the extraction process, utilizing ParallelNER, an IE bilingual parallel dataset with 257,190 samples, generated by our proposed LLM-based automatic pipeline for IE parallel data construction, with manual annotation to ensure quality. Ultimately, we obtain AlignXIE through multilingual IE instruction tuning. Although without training in 9 unseen languages, AlignXIE surpasses ChatGPT by $30.17\%$ and SoTA by $20.03\%$, thereby demonstrating superior cross-lingual IE capabilities. Comprehensive evaluations on 63 IE benchmarks in Chinese and English under various settings, demonstrate that AlignXIE significantly enhances cross-lingual and multilingual IE through boosting the IE alignment.
• 摘要：经验证据表明，LLM 表现出自发的跨语言对齐。我们的研究结果表明，尽管 LLM 在信息提取中也表现出良好的跨语言对齐效果，但不同语言之间仍然存在显著的不平衡，揭示了 IE 对齐的潜在缺陷。为了解决这个问题，我们提出了 AlignXIE，这是一个强大的基于代码的 LLM，它通过两种策略显著增强了跨语言 IE 对齐。首先，AlignXIE 将不同语言（尤其是非英语语言）的 IE 制定为代码生成任务，使用 Python 类标准化各种模式的表示，以确保同一本体在不同语言中的一致性并对齐模式。其次，它通过本文提出的翻译实例预测任务结合 IE 跨语言对齐阶段来对齐提取过程，利用 ParallelNER，这是一个包含 257,190 个样本的 IE 双语并行数据集，由我们提出的基于 LLM 的 IE 并行数据构建自动管道生成，并进行手动注释以确保质量。最终，我们通过多语言 IE 指令调整获得 AlignXIE。尽管没有接受过 9 种未知语言的训练，AlignXIE 仍比 ChatGPT 高出 $30.17\%$，比 SoTA 高出 $20.03\%$，从而展示了卓越的跨语言 IE 能力。在各种环境下对 63 个中文和英文 IE 基准进行综合评估，表明 AlignXIE 通过增强 IE 对齐，显著提高了跨语言和多语言 IE。

Title: Kwai-STaR: Transform LLMs into State-Transition Reasoners

• Authors: Xingyu Lu, Yuhang Hu, Changyi Liu, Tianke Zhang, Zhenyu Yang, Zhixiang Ding, Shengsheng Qian, Meng Du, Ruiwen Kang, Kaiyu Tang, Fan Yang, Tingting Gao, Di Zhang, Hai-Tao Zheng, Bin Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04799
• Pdf URL: https://arxiv.org/pdf/2411.04799
• Copy Paste: [[2411.04799]] Kwai-STaR: Transform LLMs into State-Transition Reasoners(https://arxiv.org/abs/2411.04799)
• Keywords: llm
• Abstract: Mathematical reasoning presents a significant challenge to the cognitive capabilities of LLMs. Various methods have been proposed to enhance the mathematical ability of LLMs. However, few recognize the value of state transition for LLM reasoning. In this work, we define mathematical problem-solving as a process of transiting from an initial unsolved state to the final resolved state, and propose Kwai-STaR framework, which transforms LLMs into State-Transition Reasoners to improve their intuitive reasoning capabilities. Our approach comprises three main steps: (1) Define the state space tailored to the mathematical reasoning. (2) Generate state-transition data based on the state space. (3) Convert original LLMs into State-Transition Reasoners via a curricular training strategy. Our experiments validate the effectiveness of Kwai-STaR in enhancing mathematical reasoning: After training on the small-scale Kwai-STaR dataset, general LLMs, including Mistral-7B and LLaMA-3, achieve considerable performance gain on the GSM8K and GSM-Hard dataset. Additionally, the state transition-based design endows Kwai-STaR with remarkable training and inference efficiency. Further experiments are underway to establish the generality of Kwai-STaR.
• 摘要：数学推理对 LLM 的认知能力提出了重大挑战。已经提出了各种方法来增强 LLM 的数学能力。然而，很少有人认识到状态转换对 LLM 推理的价值。在本文中，我们将数学问题解决定义为从初始未解决状态过渡到最终解决状态的过程，并提出了 Kwai-STaR 框架，将 LLM 转换为状态转换推理器，以提高其直观推理能力。我们的方法包括三个主要步骤：（1）定义适合数学推理的状态空间。（2）基于状态空间生成状态转换数据。（3）通过课程训练策略将原始 LLM 转换为状态转换推理器。我们的实验验证了 Kwai-STaR 在增强数学推理方面的有效性：在小规模 Kwai-STaR 数据集上训练后，包括 Mistral-7B 和 LLaMA-3 在内的通用 LLM 在 GSM8K 和 GSM-Hard 数据集上实现了显著的性能提升。此外，基于状态转换的设计使 Kwai-STaR 具有出色的训练和推理效率。目前正在进行进一步的实验以确定 Kwai-STaR 的通用性。

Title: LuxBank: The First Universal Dependency Treebank for Luxembourgish

• Authors: Alistair Plum, Caroline Döhmer, Emilia Milano, Anne-Marie Lutgen, Christoph Purschke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04813
• Pdf URL: https://arxiv.org/pdf/2411.04813
• Copy Paste: [[2411.04813]] LuxBank: The First Universal Dependency Treebank for Luxembourgish(https://arxiv.org/abs/2411.04813)
• Keywords: language model
• Abstract: The Universal Dependencies (UD) project has significantly expanded linguistic coverage across 161 languages, yet Luxembourgish, a West Germanic language spoken by approximately 400,000 people, has remained absent until now. In this paper, we introduce LuxBank, the first UD Treebank for Luxembourgish, addressing the gap in syntactic annotation and analysis for this `low-research' language. We establish formal guidelines for Luxembourgish language annotation, providing the foundation for the first large-scale quantitative analysis of its syntax. LuxBank serves not only as a resource for linguists and language learners but also as a tool for developing spell checkers and grammar checkers, organising existing text archives and even training large language models. By incorporating Luxembourgish into the UD framework, we aim to enhance the understanding of syntactic variation within West Germanic languages and offer a model for documenting smaller, semi-standardised languages. This work positions Luxembourgish as a valuable resource in the broader linguistic and NLP communities, contributing to the study of languages with limited research and resources.
• 摘要：通用依存关系 (UD) 项目已显著扩大了 161 种语言的语言覆盖范围，但卢森堡语（一种约有 40 万人使用的西日耳曼语）至今仍未出现在该项目中。在本文中，我们介绍了卢森堡语的第一个 UD 树库 LuxBank，以解决这种“研究程度较低”的语言在句法注释和分析方面的差距。我们为卢森堡语注释制定了正式的指导方针，为其句法的首次大规模定量分析奠定了基础。LuxBank 不仅是语言学家和语言学习者的资源，也是开发拼写检查器和语法检查器、组织现有文本档案甚至训练大型语言模型的工具。通过将卢森堡语纳入 UD 框架，我们旨在增强对西日耳曼语句法变化的理解，并提供一个用于记录较小的半标准化语言的模型。这项工作将卢森堡语定位为更广泛的语言学和 NLP 社区中的宝贵资源，为研究和资源有限的语言研究做出了贡献。

Title: When Does Classical Chinese Help? Quantifying Cross-Lingual Transfer in Hanja and Kanbun

• Authors: Seyoung Song, Haneul Yoo, Jiho Jin, Kyunghyun Cho, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04822
• Pdf URL: https://arxiv.org/pdf/2411.04822
• Copy Paste: [[2411.04822]] When Does Classical Chinese Help? Quantifying Cross-Lingual Transfer in Hanja and Kanbun(https://arxiv.org/abs/2411.04822)
• Keywords: language model
• Abstract: Historical and linguistic connections within the Sinosphere have led researchers to use Classical Chinese resources for cross-lingual transfer when processing historical documents from Korea and Japan. In this paper, we question the assumption of cross-lingual transferability from Classical Chinese to Hanja and Kanbun, the ancient written languages of Korea and Japan, respectively. Our experiments across machine translation, named entity recognition, and punctuation restoration tasks show minimal impact of Classical Chinese datasets on language model performance for ancient Korean documents written in Hanja, with performance differences within $\pm{}0.0068$ F1-score for sequence labeling tasks and up to $+0.84$ BLEU score for translation. These limitations persist consistently across various model sizes, architectures, and domain-specific datasets. Our analysis reveals that the benefits of Classical Chinese resources diminish rapidly as local language data increases for Hanja, while showing substantial improvements only in extremely low-resource scenarios for both Korean and Japanese historical documents. These mixed results emphasize the need for careful empirical validation rather than assuming benefits from indiscriminate cross-lingual transfer.
• 摘要：汉字文化圈内的历史和语言联系促使研究人员在处理来自韩国和日本的历史文献时使用古典汉语资源进行跨语言迁移。在本文中，我们质疑从古典汉语到朝鲜和日本古代书面语言汉字和汉文的跨语言可迁移性假设。我们在机器翻译、命名实体识别和标点符号恢复任务中的实验表明，古典汉语数据集对用汉字书写的古代韩国文献的语言模型性能影响很小，序列标记任务的性能差异在 $\pm{}0.0068$ F1 分数以内，翻译的性能差异高达 $+0.84$ BLEU 分数。这些限制在各种模型大小、架构和特定领域的数据集中始终存在。我们的分析表明，随着汉字的本地语言数据的增加，古典汉语资源的优势迅速减弱，而对于韩语和日语历史文献，只有在资源极其匮乏的情况下才会显示出显着的改善。这些好坏参半的结果强调需要仔细进行实证验证，而不是假设不加区分的跨语言迁移会带来好处。

Title: VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models

• Authors: Ming Cheng, Jiaying Gong, Chenhan Yuan, William A. Ingram, Edward Fox, Hoda Eldardiry
• Subjects: cs.CL, cs.DL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04825
• Pdf URL: https://arxiv.org/pdf/2411.04825
• Copy Paste: [[2411.04825]] VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models(https://arxiv.org/abs/2411.04825)
• Keywords: language model, llm, prompt
• Abstract: Existing text simplification or paraphrase datasets mainly focus on sentence-level text generation in a general domain. These datasets are typically developed without using domain knowledge. In this paper, we release a novel dataset, VTechAGP, which is the first academic-to-general-audience text paraphrase dataset consisting of 4,938 document-level these and dissertation academic and general-audience abstract pairs from 8 colleges authored over 25 years. We also propose a novel dynamic soft prompt generative language model, DSPT5. For training, we leverage a contrastive-generative loss function to learn the keyword vectors in the dynamic prompt. For inference, we adopt a crowd-sampling decoding strategy at both semantic and structural levels to further select the best output candidate. We evaluate DSPT5 and various state-of-the-art large language models (LLMs) from multiple perspectives. Results demonstrate that the SOTA LLMs does not provide satisfactory outcomes, while the lightweight DSPT5 can achieve competitive results. To the best of our knowledge, we are the first to build a benchmark dataset and solutions for academic-to-general-audience text paraphrase dataset.
• 摘要：现有的文本简化或释义数据集主要侧重于通用领域的句子级文本生成。这些数据集通常是在不使用领域知识的情况下开发的。在本文中，我们发布了一个新数据集 VTechAGP，这是第一个学术到普通受众的文本释义数据集，由 8 所大学 25 年来撰写的 4,938 个文档级论文和论文学术和普通受众摘要对组成。我们还提出了一种新颖的动态软提示生成语言模型 DSPT5。对于训练，我们利用对比生成损失函数来学习动态提示中的关键字向量。对于推理，我们在语义和结构层面采用众包采样解码策略来进一步选择最佳输出候选者。我们从多个角度评估了 DSPT5 和各种最先进的大型语言模型 (LLM)。结果表明，SOTA LLM 不能提供令人满意的结果，而轻量级 DSPT5 可以实现具有竞争力的结果。据我们所知，我们是第一个为学术到一般受众的文本释义数据集构建基准数据集和解决方案的人。

Title: Prompt-Guided Internal States for Hallucination Detection of Large Language Models

• Authors: Fujie Zhang, Peiqi Yu, Biao Yi, Baolei Zhang, Tong Li, Zheli Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04847
• Pdf URL: https://arxiv.org/pdf/2411.04847
• Copy Paste: [[2411.04847]] Prompt-Guided Internal States for Hallucination Detection of Large Language Models(https://arxiv.org/abs/2411.04847)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of tasks in different domains. However, they sometimes generate responses that are logically coherent but factually incorrect or misleading, which is known as LLM hallucinations. Data-driven supervised methods train hallucination detectors by leveraging the internal states of LLMs, but detectors trained on specific domains often struggle to generalize well to other domains. In this paper, we aim to enhance the cross-domain performance of supervised detectors with only in-domain data. We propose a novel framework, prompt-guided internal states for hallucination detection of LLMs, namely PRISM. By utilizing appropriate prompts to guide changes in the structure related to text truthfulness within the LLM's internal states, we make this structure more salient and consistent across texts from different domains. We integrated our framework with existing hallucination detection methods and conducted experiments on datasets from different domains. The experimental results indicate that our framework significantly enhances the cross-domain generalization of existing hallucination detection methods.
• 摘要：大型语言模型 (LLM) 已在不同领域的各种任务中展现出卓越的能力。然而，它们有时会产生逻辑上连贯但事实上不正确或误导的响应，这被称为 LLM 幻觉。数据驱动的监督方法通过利用 LLM 的内部状态来训练幻觉检测器，但在特定领域训练的检测器通常很难很好地推广到其他领域。在本文中，我们旨在仅使用域内数据来增强监督检测器的跨域性能。我们提出了一个用于 LLM 幻觉检测的新型框架，即提示引导的内部状态，即 PRISM。通过利用适当的提示来引导 LLM 内部状态中与文本真实性相关的结构的变化，我们使这种结构在不同领域的文本中更加显着和一致。我们将我们的框架与现有的幻觉检测方法相结合，并对来自不同领域的数据集进行了实验。实验结果表明，我们的框架显着增强了现有幻觉检测方法的跨域泛化。

Title: Sentiment Analysis of Spanish Political Party Tweets Using Pre-trained Language Models

• Authors: Chuqiao Song, Shunzhang Chen, Xinyi Cai, Hao Chen
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.04862
• Pdf URL: https://arxiv.org/pdf/2411.04862
• Copy Paste: [[2411.04862]] Sentiment Analysis of Spanish Political Party Tweets Using Pre-trained Language Models(https://arxiv.org/abs/2411.04862)
• Keywords: language model
• Abstract: Title: Sentiment Analysis of Spanish Political Party Communications on Twitter Using Pre-trained Language Models Authors: Chuqiao Song, Shunzhang Chen, Xinyi Cai, Hao Chen Comments: 21 pages, 6 figures Abstract: This study investigates sentiment patterns within Spanish political party communications on Twitter by leveraging BETO and RoBERTuito, two pre-trained language models optimized for Spanish text. Using a dataset of tweets from major Spanish political parties: PSOE, PP, Vox, Podemos, and Ciudadanos, spanning 2019 to 2024, this research analyzes sentiment distributions and explores the relationship between sentiment expression and party ideology. The findings indicate that both models consistently identify a predominant Neutral sentiment across all parties, with significant variations in Negative and Positive sentiments that align with ideological distinctions. Specifically, Vox exhibits higher levels of Negative sentiment, while PSOE demonstrates relatively high Positive sentiment, supporting the hypothesis that emotional appeals in political messaging reflect ideological stances. This study underscores the potential of pre-trained language models for non-English sentiment analysis on social media, providing insights into sentiment dynamics that shape public discourse within Spain's multi-party political system. Keywords: Spanish politics, sentiment analysis, pre-trained language models, Twitter, BETO, RoBERTuito, political ideology, multi-party system
• 摘要：标题：使用预训练语言模型对西班牙政党在推特上的交流进行情绪分析 作者：宋楚乔、陈顺章、蔡欣怡、陈浩 评论：21 页，6 幅图 摘要：本研究利用 BETO 和 RoBERTuito（两个针对西班牙语文本优化的预训练语言模型）研究西班牙政党在推特上的交流中的情绪模式。本研究使用来自西班牙主要政党（PSOE、PP、Vox、Podemos 和 Ciudadanos）的推文数据集（时间跨度为 2019 年至 2024 年），分析情绪分布并探讨情绪表达与政党意识形态之间的关系。研究结果表明，这两个模型一致地识别出所有政党的主导中立情绪，负面和正面情绪存在显着差异，与意识形态区别相符。具体而言，Vox 表现出更高水平的负面情绪，而 PSOE 表现出相对较高的正面情绪，支持政治信息中的情感诉求反映意识形态立场的假设。这项研究强调了预训练语言模型在社交媒体上进行非英语情绪分析的潜力，为影响西班牙多党政治体系内公共话语的情绪动态提供了见解。关键词：西班牙政治、情绪分析、预训练语言模型、推特、BETO、RoBERTuito、政治意识形态、多党制

Title: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

• Authors: Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J.H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, Wei Chu
• Subjects: cs.CL, cs.PL
• Abstract URL: https://arxiv.org/abs/2411.04905
• Pdf URL: https://arxiv.org/pdf/2411.04905
• Copy Paste: [[2411.04905]] OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models(https://arxiv.org/abs/2411.04905)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent this http URL open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an ``open cookbook'' for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.
• 摘要：代码的大型语言模型 (LLM) 已成为各个领域不可或缺的一部分，包括代码生成、推理任务和代理此 http URL 开放访问代码 LLM 越来越接近专有模型的性能水平，但是适合严格科学研究的高质量代码 LLM，特别是那些具有可重现数据处理管道和透明训练协议的代码 LLM 仍然有限。这种稀缺性是由于各种挑战造成的，包括资源限制、道德考虑以及保持模型先进的竞争优势。为了解决这一差距，我们推出了 OpenCoder，这是一个顶级代码 LLM，它不仅实现了与领先模型相当的性能，而且还可以作为研究界的“开放食谱”。与大多数先前的努力不同，我们不仅发布模型权重和推理代码，还发布可重现的训练数据、完整的数据处理管道、严格的实验消融结果和用于开放科学研究的详细训练协议。通过这次全面的发布，我们确定了构建顶级代码 LLM 的关键要素：(1) 用于数据清理的代码优化启发式规则和数据重复数据删除方法，(2) 与代码相关的文本语料库的召回，以及 (3) 退火和监督微调阶段的高质量合成数据。通过提供这种程度的开放性，我们旨在扩大对顶级代码 LLM 各个方面的访问，OpenCoder 既是一个强大的模型，也是一个开放的基础，以加速研究，并实现代码 AI 的可重复进步。

Title: GPTKB: Building Very Large Knowledge Bases from Language Models

• Authors: Yujia Hu, Shrestha Ghosh, Tuan-Phong Nugyen, Simon Razniewski
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2411.04920
• Pdf URL: https://arxiv.org/pdf/2411.04920
• Copy Paste: [[2411.04920]] GPTKB: Building Very Large Knowledge Bases from Language Models(https://arxiv.org/abs/2411.04920)
• Keywords: language model, gpt, llm
• Abstract: General-domain knowledge bases (KB), in particular the "big three" -- Wikidata, Yago and DBpedia -- are the backbone of many intelligent applications. While these three have seen steady development, comprehensive KB construction at large has seen few fresh attempts. In this work, we propose to build a large general-domain KB entirely from a large language model (LLM). We demonstrate the feasibility of large-scale KB construction from LLMs, while highlighting specific challenges arising around entity recognition, entity and property canonicalization, and taxonomy construction. As a prototype, we use GPT-4o-mini to construct GPTKB, which contains 105 million triples for more than 2.9 million entities, at a cost 100x less than previous KBC projects. Our work is a landmark for two fields: For NLP, for the first time, it provides \textit{constructive} insights into the knowledge (or beliefs) of LLMs. For the Semantic Web, it shows novel ways forward for the long-standing challenge of general-domain KB construction. GPTKB is accessible at this https URL.
• 摘要：通用领域知识库 (KB)，尤其是“三巨头”——Wikidata、Yago 和 DBpedia——是许多智能应用程序的支柱。虽然这三个领域都取得了稳步发展，但全面的知识库建设却鲜有新的尝试。在这项工作中，我们建议完全从大型语言模型 (LLM) 构建大型通用领域知识库。我们展示了从 LLM 构建大规模知识库的可行性，同时强调了围绕实体识别、实体和属性规范化以及分类构建出现的特定挑战。作为原型，我们使用 GPT-4o-mini 构建 GPTKB，其中包含 290 多万个实体的 1.05 亿个三元组，成本比以前的 KBC 项目低 100 倍。我们的工作是两个领域的里程碑：对于 NLP，它首次提供了对 LLM 知识（或信念）的 \textit{建设性} 见解。对于语义网，它为长期存在的通用领域知识库建设挑战展示了新的前进方向。可以通过此 https URL 访问 GPTKB。

Title: BitNet a4.8: 4-bit Activations for 1-bit LLMs

• Authors: Hongyu Wang, Shuming Ma, Furu Wei
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04965
• Pdf URL: https://arxiv.org/pdf/2411.04965
• Copy Paste: [[2411.04965]] BitNet a4.8: 4-bit Activations for 1-bit LLMs(https://arxiv.org/abs/2411.04965)
• Keywords: language model, llm
• Abstract: Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.
• 摘要：最近对 1 位大型语言模型 (LLM)（例如 BitNet b1.58）的研究为降低 LLM 的推理成本同时保持其性能提供了一个有希望的方向。在这项工作中，我们引入了 BitNet a4.8，为 1 位 LLM 启用 4 位激活。BitNet a4.8 采用混合量化和稀疏化策略来减轻由异常通道引入的量化误差。具体来说，我们利用 4 位激活作为注意力和前馈网络层的输入，同时稀疏中间状态，然后进行 8 位量化。大量实验表明，BitNet a4.8 在同等训练成本的情况下实现了与 BitNet b1.58 相当的性能，同时在启用 4 位（INT4/FP4）内核的情况下推理速度更快。此外，BitNet a4.8 仅激活 55% 的参数并支持 3 位 KV 缓存，进一步提高了大规模 LLM 部署和推理的效率。

Title: SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference

• Authors: Gabriele Oliaro, Zhihao Jia, Daniel Campos, Aurick Qiao
• Subjects: cs.CL, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04975
• Pdf URL: https://arxiv.org/pdf/2411.04975
• Copy Paste: [[2411.04975]] SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference(https://arxiv.org/abs/2411.04975)
• Keywords: language model, llm, chat
• Abstract: We present SuffixDecoding, a novel model-free approach to accelerating large language model (LLM) inference through speculative decoding. Unlike existing methods that rely on draft models or specialized decoding heads, SuffixDecoding leverages suffix trees built from previously generated outputs to efficiently predict candidate token sequences. Our approach enables flexible tree-structured speculation without the overhead of maintaining and orchestrating additional models. SuffixDecoding builds and dynamically updates suffix trees to capture patterns in the generated text, using them to construct speculation trees through a principled scoring mechanism based on empirical token frequencies. SuffixDecoding requires only CPU memory which is plentiful and underutilized on typical LLM serving nodes. We demonstrate that SuffixDecoding achieves competitive speedups compared to model-based approaches across diverse workloads including open-domain chat, code generation, and text-to-SQL tasks. For open-ended chat and code generation tasks, SuffixDecoding achieves up to $1.4\times$ higher output throughput than SpecInfer and up to $1.1\times$ lower time-per-token (TPOT) latency. For a proprietary multi-LLM text-to-SQL application, SuffixDecoding achieves up to $2.9\times$ higher output throughput and $3\times$ lower latency than speculative decoding. Our evaluation shows that SuffixDecoding maintains high acceptance rates even with small reference corpora of 256 examples, while continuing to improve performance as more historical outputs are incorporated.
• 摘要：我们提出了 SuffixDecoding，这是一种新颖的无模型方法，可通过推测解码加速大型语言模型 (LLM) 推理。与依赖草稿模型或专用解码头的现有方法不同，SuffixDecoding 利用从先前生成的输出构建的后缀树来有效预测候选标记序列。我们的方法可以实现灵活的树结构推测，而无需维护和协调其他模型的开销。SuffixDecoding 构建并动态更新后缀树以捕获生成文本中的模式，并使用它们通过基于经验标记频率的原则性评分机制构建推测树。SuffixDecoding 只需要 CPU 内存，而这在典型的 LLM 服务节点上是充足的且未充分利用的。我们证明，与基于模型的方法相比，SuffixDecoding 在包括开放域聊天、代码生成和文本到 SQL 任务在内的各种工作负载中实现了具有竞争力的加速。对于开放式聊天和代码生成任务，SuffixDecoding 的输出吞吐量比 SpecInfer 高出 $1.4\times$，每令牌时间 (TPOT) 延迟低 $1.1\times$。对于专有多 LLM 文本到 SQL 应用程序，SuffixDecoding 的输出吞吐量比推测解码高出 $2.9\times$，延迟低 $3\times$。我们的评估表明，即使使用 256 个示例的小型参考语料库，SuffixDecoding 也能保持较高的接受率，同时随着更多历史输出的纳入，性能继续提高。

Title: The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

• Authors: Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, Yoon Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04986
• Pdf URL: https://arxiv.org/pdf/2411.04986
• Copy Paste: [[2411.04986]] The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities(https://arxiv.org/abs/2411.04986)
• Keywords: language model
• Abstract: Modern language models can process inputs across diverse languages and modalities. We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e.g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages. We term this the semantic hub hypothesis, following the hub-and-spoke model from neuroscience (Patterson et al., 2007) which posits that semantic knowledge in the human brain is organized through a transmodal semantic "hub" which integrates information from various modality-specific "spokes" regions. We first show that model representations for semantically equivalent inputs in different languages are similar in the intermediate layers, and that this space can be interpreted using the model's dominant pretraining language via the logit lens. This tendency extends to other data types, including arithmetic expressions, code, and visual/audio inputs. Interventions in the shared representation space in one data type also predictably affect model outputs in other data types, suggesting that this shared representations space is not simply a vestigial byproduct of large-scale training on broad data, but something that is actively utilized by the model during input processing.
• 摘要：现代语言模型可以处理跨多种语言和模态的输入。我们假设模型通过学习跨异构数据类型（例如，不同的语言和模态）的共享表示空间来获得这种能力，该空间将语义上相似的输入放在彼此附近，即使它们来自不同的模态/语言。我们将其称为语义中心假设，遵循神经科学的轮毂辐射模型（Patterson 等人，2007 年），该模型假定人类大脑中的语义知识是通过跨模态语义“中心”组织的，该“中心”整合了来自各种模态特定“辐条”区域的信息。我们首先表明，不同语言中语义等效输入的模型表示在中间层中是相似的，并且可以通过 logit 镜头使用模型的主要预训练语言来解释这个空间。这种趋势扩展到其他数据类型，包括算术表达式、代码和视觉/音频输入。对一种数据类型的共享表示空间的干预也会可预测地影响其他数据类型的模型输出，这表明这种共享表示空间不仅仅是对广泛数据进行大规模训练的残留副产品，而且是模型在输入处理过程中积极利用的东西。

Title: Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

• Authors: Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Xi Victoria Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04996
• Pdf URL: https://arxiv.org/pdf/2411.04996
• Copy Paste: [[2411.04996]] Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models(https://arxiv.org/abs/2411.04996)
• Keywords: language model, llm
• Abstract: The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8\% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2\% of the wall-clock time and text quality in 75.6\% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).
• 摘要：大型语言模型 (LLM) 的开发已扩展到能够在统一框架内处理文本、图像和语音的多模态系统。与纯文本 LLM 相比，训练这些模型需要更大的数据集和计算资源。为了应对扩展挑战，我们引入了 Mixture-of-Transformers (MoT)，这是一种稀疏多模态变压器架构，可显著降低预训练计算成本。MoT 按模态（包括前馈网络、注意矩阵和层规范化）解耦模型的非嵌入参数，从而实现对整个输入序列的全局自注意的模态特定处理。我们在多种设置和模型尺度上评估了 MoT。在 Chameleon 7B 设置（自回归文本和图像生成）中，MoT 仅使用 55.8% 的 FLOP 即可达到密集基线的性能。当扩展到包括语音时，MoT 仅使用 37.2% 的 FLOP 即可达到与密集基线相当的语音性能。在输血设置中，文本和图像使用不同的目标进行训练，7B MoT 模型以三分之一的 FLOP 达到密集基线的图像模态性能，而 760M MoT 模型在关键图像生成指标方面优于 1.4B 密集基线。系统分析进一步凸显了 MoT 的实际优势，在 47.2% 的挂钟时间内实现密集基线图像质量，在 75.6% 的挂钟时间内实现文本质量（在配备 NVIDIA A100 GPU 的 AWS p4de.24xlarge 实例上测量）。

Title: Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?

• Authors: Jonathan Roberts, Kai Han, Samuel Albanie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05000
• Pdf URL: https://arxiv.org/pdf/2411.05000
• Copy Paste: [[2411.05000]] Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?(https://arxiv.org/abs/2411.05000)
• Keywords: language model, llm
• Abstract: As the context limits of Large Language Models (LLMs) increase, the range of possible applications and downstream functions broadens. In many real-world tasks, decisions depend on details scattered across collections of often disparate documents containing mostly irrelevant information. Long-context LLMs appear well-suited to this form of complex information retrieval and reasoning, which has traditionally proven costly and time-consuming. However, although the development of longer context models has seen rapid gains in recent years, our understanding of how effectively LLMs use their context has not kept pace. To address this, we conduct a set of retrieval experiments designed to evaluate the capabilities of 17 leading LLMs, such as their ability to follow threads of information through the context window. Strikingly, we find that many models are remarkably threadsafe: capable of simultaneously following multiple threads without significant loss in performance. Still, for many models, we find the effective context limit is significantly shorter than the supported context length, with accuracy decreasing as the context window grows. Our study also highlights the important point that token counts from different tokenizers should not be directly compared -- they often correspond to substantially different numbers of written characters. We release our code and long-context experimental data.
• 摘要：随着大型语言模型 (LLM) 上下文限制的增加，可能的应用和下游功能的范围也随之扩大。在许多现实世界的任务中，决策取决于分散在通常包含大部分不相关信息的不同文档集合中的细节。长上下文 LLM 似乎非常适合这种形式的复杂信息检索和推理，而这种形式传统上已被证明成本高昂且耗时。然而，尽管近年来较长上下文模型的开发取得了快速进展，但我们对 LLM 如何有效使用其上下文的理解却没有跟上步伐。为了解决这个问题，我们进行了一组检索实验，旨在评估 17 个领先 LLM 的功能，例如它们通过上下文窗口跟踪信息线程的能力。令人惊讶的是，我们发现许多模型具有出色的线程安全性：能够同时跟踪多个线程而不会显著降低性能。不过，对于许多模型，我们发现有效上下文限制明显短于支持的上下文长度，并且随着上下文窗口的增大，准确度会降低。我们的研究还强调了重要的一点，即不应直接比较来自不同标记器的标记计数——它们通常对应于截然不同的书写字符数。我们发布了我们的代码和长上下文实验数据。