2024-05-01

Title: Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking

• Authors: Zacharias Chrysidis, Stefanos-Iordanis Papadopoulos, Symeon Papadopoulos, Panagiotis C. Petrantonakis
• Subjects: cs.CL, cs.CY, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.18971
• Pdf URL: https://arxiv.org/pdf/2404.18971
• Copy Paste: [[2404.18971]] Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking(https://arxiv.org/abs/2404.18971)
• Keywords: language model
• Abstract: Automated fact-checking (AFC) is garnering increasing attention by researchers aiming to help fact-checkers combat the increasing spread of misinformation online. While many existing AFC methods incorporate external information from the Web to help examine the veracity of claims, they often overlook the importance of verifying the source and quality of collected "evidence". One overlooked challenge involves the reliance on "leaked evidence", information gathered directly from fact-checking websites and used to train AFC systems, resulting in an unrealistic setting for early misinformation detection. Similarly, the inclusion of information from unreliable sources can undermine the effectiveness of AFC systems. To address these challenges, we present a comprehensive approach to evidence verification and filtering. We create the "CREDible, Unreliable or LEaked" (CREDULE) dataset, which consists of 91,632 articles classified as Credible, Unreliable and Fact checked (Leaked). Additionally, we introduce the EVidence VERification Network (EVVER-Net), trained on CREDULE to detect leaked and unreliable evidence in both short and long texts. EVVER-Net can be used to filter evidence collected from the Web, thus enhancing the robustness of end-to-end AFC systems. We experiment with various language models and show that EVVER-Net can demonstrate impressive performance of up to 91.5% and 94.4% accuracy, while leveraging domain credibility scores along with short or long texts, respectively. Finally, we assess the evidence provided by widely-used fact-checking datasets including LIAR-PLUS, MOCHEG, FACTIFY, NewsCLIPpings+ and VERITE, some of which exhibit concerning rates of leaked and unreliable evidence.
• 摘要：自动事实核查（AFC）越来越受到研究人员的关注，旨在帮助事实核查人员打击网上日益传播的错误信息。虽然许多现有的 AFC 方法结合了来自网络的外部信息来帮助检查索赔的真实性，但它们常常忽视验证所收集“证据”的来源和质量的重要性。一个被忽视的挑战涉及对“泄露证据”的依赖，即直接从事实核查网站收集并用于训练亚足联系统的信息，导致早期错误信息检测的环境不切实际。同样，包含来自不可靠来源的信息可能会破坏 AFC 系统的有效性。为了应对这些挑战，我们提出了一种全面的证据验证和过滤方法。我们创建了“CREDible、Unreliable 或 LEaked”(CREDULE) 数据集，其中包含 91,632 篇分类为 Credible、Unreliable 和事实已检查（Leaked）的文章。此外，我们还引入了证据验证网络（EVVER-Net），该网络在 CREDULE 上进行训练，以检测短文本和长文本中泄露和不可靠的证据。 EVVER-Net 可用于过滤从网络收集的证据，从而增强端到端 AFC 系统的稳健性。我们对各种语言模型进行了实验，结果表明，EVVER-Net 可以展示高达 91.5% 和 94.4% 准确率的令人印象深刻的性能，同时分别利用域可信度分数以及短文本或长文本。最后，我们评估了广泛使用的事实核查数据集提供的证据，包括 LIAR-PLUS、MOCHEG、FACTIFY、NewsCLIPpings+ 和 VERITE，其中一些数据显示出有关泄露和不可靠证据的比率。

Title: Computational Job Market Analysis with Natural Language Processing

• Authors: Mike Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18977
• Pdf URL: https://arxiv.org/pdf/2404.18977
• Copy Paste: [[2404.18977]] Computational Job Market Analysis with Natural Language Processing(https://arxiv.org/abs/2404.18977)
• Keywords: language model
• Abstract: [Abridged Abstract] Recent technological advances underscore labor market dynamics, yielding significant consequences for employment prospects and increasing job vacancy data across platforms and languages. Aggregating such data holds potential for valuable insights into labor market demands, new skills emergence, and facilitating job matching for various stakeholders. However, despite prevalent insights in the private sector, transparent language technology systems and data for this domain are lacking. This thesis investigates Natural Language Processing (NLP) technology for extracting relevant information from job descriptions, identifying challenges including scarcity of training data, lack of standardized annotation guidelines, and shortage of effective extraction methods from job ads. We frame the problem, obtaining annotated data, and introducing extraction methodologies. Our contributions include job description datasets, a de-identification dataset, and a novel active learning algorithm for efficient model training. We propose skill extraction using weak supervision, a taxonomy-aware pre-training methodology adapting multilingual language models to the job market domain, and a retrieval-augmented model leveraging multiple skill extraction datasets to enhance overall performance. Finally, we ground extracted information within a designated taxonomy.
• 摘要：[摘要]最近的技术进步凸显了劳动力市场的动态，对就业前景产生了重大影响，并增加了跨平台和语言的职位空缺数据。汇总这些数据可以为劳动力市场需求、新技能的出现以及促进各个利益相关者的工作匹配提供有价值的见解。然而，尽管私营部门普遍存在见解，但该领域缺乏透明的语言技术系统和数据。本论文研究了从职位描述中提取相关信息的自然语言处理（NLP）技术，识别了培训数据稀缺、缺乏标准化注释指南以及缺乏有效的职位描述提取方法等挑战。我们构建问题，获取带注释的数据，并引入提取方法。我们的贡献包括职位描述数据集、去识别数据集和用于高效模型训练的新型主动学习算法。我们提出使用弱监督进行技能提取、使多语言语言模型适应就业市场领域的分类感知预训练方法以及利用多个技能提取数据集来提高整体性能的检索增强模型。最后，我们将提取的信息归入指定的分类法中。

Title: Markovian Agents for Truthful Language Modeling

• Authors: Scott Viteri, Max Lamparth, Peter Chatain, Clark Barrett
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18988
• Pdf URL: https://arxiv.org/pdf/2404.18988
• Copy Paste: [[2404.18988]] Markovian Agents for Truthful Language Modeling(https://arxiv.org/abs/2404.18988)
• Keywords: language model, chain-of-thought, agent
• Abstract: Chain-of-Thought (CoT) reasoning could in principle enable a deeper understanding of a language model's (LM) internal reasoning. However, prior work suggests that some LMs answer questions similarly despite changes in their CoT, suggesting that those models are not truly using the CoT. We propose a training method to produce CoTs that are sufficient alone for predicting future text, independent of other context. This methodology gives a guarantee that if the LM can predict future tokens, then it must have used the CoT to understand its context. We formalize the idea that the truthfulness of a sender to a receiver LM is the degree to which the sender helps the receiver predict their future observations. Then we define a "Markovian" LM as one which predicts future text given only a CoT as context. We derive a "Markovian training" procedure by applying our definition of truthfulness to a Markovian LM and optimizing via policy gradient and Proximal Policy Optimization (PPO). We demonstrate the effectiveness of our training algorithm on long-context arithmetic problems, show that the model utilizes the CoT, and validate that the generated CoT is meaningful and usable by other models.
• 摘要：思想链 (CoT) 推理原则上可以更深入地理解语言模型 (LM) 的内部推理。然而，之前的工作表明，尽管 CoT 发生了变化，一些 LM 仍会回答类似的问题，这表明这些模型并未真正使用 CoT。我们提出了一种训练方法来生成 CoT，该 CoT 足以独立于其他上下文来预测未来文本。这种方法保证了如果 LM 可以预测未来的代币，那么它一定使用 CoT 来理解其上下文。我们形式化了这样的想法：发送者对接收者 LM 的真实性是发送者帮助接收者预测他们未来观察的程度。然后，我们将“马尔可夫”LM 定义为仅在 CoT 作为上下文的情况下预测未来文本的LM。我们通过将真实性定义应用于马尔可夫 LM 并通过策略梯度和近端策略优化 (PPO) 进行优化，得出“马尔可夫训练”程序。我们证明了我们的训练算法在长上下文算术问题上的有效性，表明该模型利用了 CoT，并验证生成的 CoT 是否有意义且可供其他模型使用。

Title: A Framework for Real-time Safeguarding the Text Generation of Large Language

• Authors: Ximing Dong, Dayi Lin, Shaowei Wang, Ahmed E. Hassan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19048
• Pdf URL: https://arxiv.org/pdf/2404.19048
• Copy Paste: [[2404.19048]] A Framework for Real-time Safeguarding the Text Generation of Large Language(https://arxiv.org/abs/2404.19048)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have significantly advanced natural language processing (NLP) tasks but also pose ethical and societal risks due to their propensity to generate harmful content. To address this, various approaches have been developed to safeguard LLMs from producing unsafe content. However, existing methods have limitations, including the need for training specific control models and proactive intervention during text generation, that lead to quality degradation and increased computational overhead. To mitigate those limitations, we propose LLMSafeGuard, a lightweight framework to safeguard LLM text generation in real-time. LLMSafeGuard integrates an external validator into the beam search algorithm during decoding, rejecting candidates that violate safety constraints while allowing valid ones to proceed. We introduce a similarity based validation approach, simplifying constraint introduction and eliminating the need for control model training. Additionally, LLMSafeGuard employs a context-wise timing selection strategy, intervening LLMs only when necessary. We evaluate LLMSafe-Guard on two tasks, detoxification and copyright safeguarding, and demonstrate its superior performance over SOTA baselines. For instance, LLMSafeGuard reduces the average toxic score of. LLM output by 29.7% compared to the best baseline meanwhile preserving similar linguistic quality as natural output in detoxification task. Similarly, in the copyright task, LLMSafeGuard decreases the Longest Common Subsequence (LCS) by 56.2% compared to baselines. Moreover, our context-wise timing selection strategy reduces inference time by at least 24% meanwhile maintaining comparable effectiveness as validating each time step. LLMSafeGuard also offers tunable parameters to balance its effectiveness and efficiency.
• 摘要：大型语言模型 (LLM) 具有显着先进的自然语言处理 (NLP) 任务，但由于它们倾向于生成有害内容，因此也带来道德和社会风险。为了解决这个问题，人们开发了各种方法来保护法学硕士免于产生不安全的内容。然而，现有方法存在局限性，包括需要训练特定的控制模型和在文本生成过程中主动干预，从而导致质量下降和计算开销增加。为了缓解这些限制，我们提出了 LLMSafeGuard，这是一个轻量级框架，用于实时保护 LLM 文本生成。 LLMSafeGuard 在解码过程中将外部验证器集成到波束搜索算法中，拒绝违反安全约束的候选者，同时允许有效的候选者继续进行。我们引入了基于相似性的验证方法，简化了约束引入并消除了控制模型训练的需要。此外，LLMSafeGuard 采用上下文相关的时序选择策略，仅在必要时干预 LLM。我们在解毒和版权保护这两项任务上评估了 LLMSafe-Guard，并展示了其相对于 SOTA 基线的卓越性能。例如，LLMSafeGuard 降低了平均毒性分数。与最佳基线相比，LLM 输出提高了 29.7%，同时在解毒任务中保持与自然输出相似的语言质量。同样，在版权任务中，LLMSafeGuard 与基线相比，最长公共子序列 (LCS) 减少了 56.2%。此外，我们的上下文明智的时序选择策略将推理时间减少了至少 24%，同时保持了与验证每个时间步骤相当的有效性。 LLMSafeGuard 还提供可调参数来平衡其有效性和效率。

Title: Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models

• Authors: Houjun Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19055
• Pdf URL: https://arxiv.org/pdf/2404.19055
• Copy Paste: [[2404.19055]] Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models(https://arxiv.org/abs/2404.19055)
• Keywords: language model
• Abstract: While language models (LMs) offer significant capability in zero-shot reasoning tasks across a wide range of domains, they do not perform satisfactorily in problems which requires multi-step reasoning. Previous approaches to mitigate this involves breaking a larger, multi-step task into sub-tasks and asking the language model to generate proposals ("thoughts") for each sub-task and using exhaustive planning approaches such as DFS to compose a solution. In this work, we leverage this idea to introduce two new contributions: first, we formalize a planning-based approach to perform multi-step problem solving with LMs via Partially Observable Markov Decision Processes (POMDPs), with the LM's own reflections about the value of a state used as a search heuristic; second, leveraging the online POMDP solver POMCP, we demonstrate a superior success rate of 89.4% on the Game of 24 task as compared to existing approaches while also offering better anytime performance characteristics than fixed tree-search which is used previously. Taken together, these contributions allow modern LMs to decompose and solve larger-scale reasoning tasks more effectively.
• 摘要：虽然语言模型（LM）在跨广泛领域的零样本推理任务中提供了重要的能力，但它们在需要多步推理的问题上表现不佳。以前缓解这种情况的方法包括将较大的多步骤任务分解为子任务，并要求语言模型为每个子任务生成建议（“想法”），并使用 DFS 等详尽的规划方法来组成解决方案。在这项工作中，我们利用这个想法引入了两个新的贡献：首先，我们形式化了一种基于规划的方法，通过部分可观察马尔可夫决策过程（POMDP）使用 LM 执行多步骤问题解决，并结合 LM 自己对价值的反思用作搜索启发式的状态；其次，利用在线 POMDP 求解器 POMCP，与现有方法相比，我们在 24 人游戏任务中展示了 89.4% 的卓越成功率，同时还提供比之前使用的固定树搜索更好的随时性能特征。总而言之，这些贡献使现代 LM 能够更有效地分解和解决更大规模的推理任务。

Title: SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications

• Authors: Liang Xu, Lei Zhu, Yaotong Wu, Hang Xue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19063
• Pdf URL: https://arxiv.org/pdf/2404.19063
• Copy Paste: [[2404.19063]] SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications(https://arxiv.org/abs/2404.19063)
• Keywords: language model, llm
• Abstract: The SuperCLUE-Fin (SC-Fin) benchmark is a pioneering evaluation framework tailored for Chinese-native financial large language models (FLMs). It assesses FLMs across six financial application domains and twenty-five specialized tasks, encompassing theoretical knowledge and practical applications such as compliance, risk management, and investment analysis. Using multi-turn, open-ended conversations that mimic real-life scenarios, SC-Fin measures models on a range of criteria, including accurate financial understanding, logical reasoning, clarity, computational efficiency, business acumen, risk perception, and compliance with Chinese regulations. In a rigorous evaluation involving over a thousand questions, SC-Fin identifies a performance hierarchy where domestic models like GLM-4 and MoonShot-v1-128k outperform others with an A-grade, highlighting the potential for further development in transforming theoretical knowledge into pragmatic financial solutions. This benchmark serves as a critical tool for refining FLMs in the Chinese context, directing improvements in financial knowledge databases, standardizing financial interpretations, and promoting models that prioritize compliance, risk management, and secure practices. We create a contextually relevant and comprehensive benchmark that drives the development of AI in the Chinese financial sector. SC-Fin facilitates the advancement and responsible deployment of FLMs, offering valuable insights for enhancing model performance and usability for both individual and institutional users in the Chinese market..~\footnote{Our benchmark can be found at \url{https://www.CLUEbenchmarks.com}}.
• 摘要：SuperCLUE-Fin (SC-Fin) 基准是一个为中文母语金融大型语言模型 (FLM) 量身定制的开创性评估框架。它评估六个金融应用领域和二十五个专业任务中的 FLM，涵盖理论知识和合规、风险管理和投资分析等实际应用。SC-Fin 使用模拟现实场景的多轮开放式对话，根据一系列标准对模型进行衡量，包括准确的金融理解、逻辑推理、清晰度、计算效率、商业敏锐度、风险感知以及对中国法规的遵守。在涉及一千多个问题的严格评估中，SC-Fin 确定了一个性能等级，其中 GLM-4 和 MoonShot-v1-128k 等国内模型以 A 级的表现优于其他模型，凸显了将理论知识转化为实用金融解决方案的进一步发展潜力。该基准是完善中国背景下的 FLM、指导金融知识数据库改进、标准化金融解释以及推广优先考虑合规、风险管理和安全实践的模型的重要工具。我们创建了与背景相关的综合基准，推动中国金融领域人工智能的发展。SC-Fin 促进了 FLM 的进步和负责任的部署，为提高中国市场个人和机构用户的模型性能和可用性提供了宝贵的见解。~\footnote{我们的基准可以在 \url{https://www.CLUEbenchmarks.com}} 找到。

Title: In-Context Symbolic Regression: Leveraging Language Models for Function Discovery

• Authors: Matteo Merler, Nicola Dainese, Katsiaryna Haitsiukevich
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19094
• Pdf URL: https://arxiv.org/pdf/2404.19094
• Copy Paste: [[2404.19094]] In-Context Symbolic Regression: Leveraging Language Models for Function Discovery(https://arxiv.org/abs/2404.19094)
• Keywords: language model, llm
• Abstract: Symbolic Regression (SR) is a task which aims to extract the mathematical expression underlying a set of empirical observations. Transformer-based methods trained on SR datasets detain the current state-of-the-art in this task, while the application of Large Language Models (LLMs) to SR remains unexplored. This work investigates the integration of pre-trained LLMs into the SR pipeline, utilizing an approach that iteratively refines a functional form based on the prediction error it achieves on the observation set, until it reaches convergence. Our method leverages LLMs to propose an initial set of possible functions based on the observations, exploiting their strong pre-training prior. These functions are then iteratively refined by the model itself and by an external optimizer for their coefficients. The process is repeated until the results are satisfactory. We then analyze Vision-Language Models in this context, exploring the inclusion of plots as visual inputs to aid the optimization process. Our findings reveal that LLMs are able to successfully recover good symbolic equations that fit the given data, outperforming SR baselines based on Genetic Programming, with the addition of images in the input showing promising results for the most complex benchmarks.
• 摘要：符号回归（SR）是一项旨在提取一组经验观察结果背后的数学表达式的任务。在 SR 数据集上训练的基于 Transformer 的方法保留了该任务中当前最先进的技术，而大型语言模型 (LLM) 在 SR 中的应用仍未得到探索。这项工作研究了将预训练的 LLM 集成到 SR 管道中，利用一种方法，根据在观察集上实现的预测误差迭代地细化函数形式，直到达到收敛。我们的方法利用法学硕士根据观察结果提出一组初始可能的函数，利用其强大的预训练先验。然后，这些函数由模型本身和外部优化器对其系数进行迭代细化。重复该过程直到结果令人满意。然后，我们在此背景下分析视觉语言模型，探索将绘图作为视觉输入以帮助优化过程。我们的研究结果表明，法学硕士能够成功恢复适合给定数据的良好符号方程，优于基于遗传编程的 SR 基线，并且在输入中添加图像，显示出最复杂基准的有希望的结果。

Title: Accelerating Production LLMs with Combined Token/Embedding Speculators

• Authors: Davis Wertheimer, Joshua Rosenkranz, Thomas Parnell, Sahil Suneja, Pavithra Ranganathan, Raghu Ganti, Mudhakar Srivatsa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19124
• Pdf URL: https://arxiv.org/pdf/2404.19124
• Copy Paste: [[2404.19124]] Accelerating Production LLMs with Combined Token/Embedding Speculators(https://arxiv.org/abs/2404.19124)
• Keywords: language model, llm
• Abstract: This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements.
• 摘要：该技术报告描述了新型推测解码草稿模型的设计和训练，以加快生产环境中大型语言模型的推理速度。通过根据上下文向量和采样标记调整草稿预测，我们可以训练我们的投机者有效地预测高质量的 n 元语法，然后基础模型会接受或拒绝。这使我们能够有效地预测每次推理前向传递的多个标记，将高度优化的基本模型实现的挂钟推理速度加快 2-3 倍。我们探索这些初步结果并描述进一步改进的后续步骤。

Title: What Drives Performance in Multilingual Language Models?

• Authors: Sina Bagheri Nezhad, Ameeta Agrawal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19159
• Pdf URL: https://arxiv.org/pdf/2404.19159
• Copy Paste: [[2404.19159]] What Drives Performance in Multilingual Language Models?(https://arxiv.org/abs/2404.19159)
• Keywords: language model, llm
• Abstract: This study investigates the factors influencing the performance of multilingual large language models (MLLMs) across diverse languages. We study 6 MLLMs, including masked language models, autoregressive models, and instruction-tuned LLMs, on the SIB-200 dataset, a topic classification dataset encompassing 204 languages. Our analysis considers three scenarios: ALL languages, SEEN languages (present in the model's pretraining data), and UNSEEN languages (not present or documented in the model's pretraining data in any meaningful way). We examine the impact of factors such as pretraining data size, general resource availability, language family, and script type on model performance. Decision tree analysis reveals that pretraining data size is the most influential factor for SEEN languages. However, interestingly, script type and language family are crucial for UNSEEN languages, highlighting the importance of cross-lingual transfer learning. Notably, model size and architecture do not significantly alter the most important features identified. Our findings provide valuable insights into the strengths and limitations of current MLLMs and hope to guide the development of more effective and equitable multilingual NLP systems.
• 摘要：本研究调查了影响不同语言的多语言大语言模型（MLLM）性能的因素。我们在 SIB-200 数据集（包含 204 种语言的主题分类数据集）上研究了 6 个 MLLM，包括屏蔽语言模型、自回归模型和指令调整的 LLM。我们的分析考虑了三种场景：所有语言、SEEN 语言（存在于模型的预训练数据中）和 UNSEEN 语言（不以任何有意义的方式存在或记录在模型的预训练数据中）。我们研究了预训练数据大小、一般资源可用性、语言族和脚本类型等因素对模型性能的影响。决策树分析表明，预训练数据大小是对 SEEN 语言影响最大的因素。然而，有趣的是，脚本类型和语言家族对于 UNSEEN 语言至关重要，凸显了跨语言迁移学习的重要性。值得注意的是，模型大小和架构不会显着改变已识别的最重要特征。我们的研究结果为当前 MLLM 的优势和局限性提供了宝贵的见解，并希望指导更有效、更公平的多语言 NLP 系统的开发。

Title: Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

• Authors: James A. Michaelov, Catherine Arnett, Benjamin K. Bergen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19178
• Pdf URL: https://arxiv.org/pdf/2404.19178
• Copy Paste: [[2404.19178]] Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics(https://arxiv.org/abs/2404.19178)
• Keywords: language model
• Abstract: Transformers have supplanted Recurrent Neural Networks as the dominant architecture for both natural language processing tasks and, despite criticisms of cognitive implausibility, for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent neural network architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match - and in some cases, exceed - performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.
• 摘要：变形金刚已经取代循环神经网络成为自然语言处理任务的主导架构，尽管有人批评认知不可信，但它也用于模拟可预测性对在线人类语言理解的影响。然而，最近开发的两种循环神经网络架构 RWKV 和 Mamba 在执行自然语言任务方面似乎与同等规模的 Transformer 相当或更好。在本文中，我们表明，当代的循环模型现在也能够在在线人类语言理解建模方面与同等大小的变压器相匹配，甚至在某些情况下超过性能。这表明 Transformer 语言模型并不是唯一适合这项任务，并且为关于语言模型的架构特征在多大程度上使其成为更好或更差的人类语言理解模型的争论开辟了新的方向。

Title: Mix of Experts Language Model for Named Entity Recognition

• Authors: Xinwei Chen, Kun Li, Tianyou Song, Jiangjian Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19192
• Pdf URL: https://arxiv.org/pdf/2404.19192
• Copy Paste: [[2404.19192]] Mix of Experts Language Model for Named Entity Recognition(https://arxiv.org/abs/2404.19192)
• Keywords: language model
• Abstract: Named Entity Recognition (NER) is an essential steppingstone in the field of natural language processing. Although promising performance has been achieved by various distantly supervised models, we argue that distant supervision inevitably introduces incomplete and noisy annotations, which may mislead the model training process. To address this issue, we propose a robust NER model named BOND-MoE based on Mixture of Experts (MoE). Instead of relying on a single model for NER prediction, multiple models are trained and ensembled under the Expectation-Maximization (EM) framework, so that noisy supervision can be dramatically alleviated. In addition, we introduce a fair assignment module to balance the document-model assignment process. Extensive experiments on real-world datasets show that the proposed method achieves state-of-the-art performance compared with other distantly supervised NER.
• 摘要：命名实体识别（NER）是自然语言处理领域的重要基石。尽管各种远程监督模型已经取得了良好的性能，但我们认为远程监督不可避免地会引入不完整和嘈杂的注释，这可能会误导模型训练过程。为了解决这个问题，我们提出了一个基于混合专家（MoE）的强大的 NER 模型，名为 BOND-MoE。不再依赖单一模型进行 NER 预测，而是在期望最大化（EM）框架下训练和集成多个模型，从而可以显着减轻噪声监督。此外，我们引入了公平分配模块来平衡文档模型分配过程。对现实世界数据集的大量实验表明，与其他远程监督 NER 相比，所提出的方法实现了最先进的性能。

Title: GRAMMAR: Grounded and Modular Evaluation of Domain-Specific Retrieval-Augmented Language Models

• Authors: Xinzhe Li, Ming Liu, Shang Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19232
• Pdf URL: https://arxiv.org/pdf/2404.19232
• Copy Paste: [[2404.19232]] GRAMMAR: Grounded and Modular Evaluation of Domain-Specific Retrieval-Augmented Language Models(https://arxiv.org/abs/2404.19232)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented Generation (RAG) systems have been actively studied and deployed across various industries to query on domain-specific knowledge base. However, evaluating these systems presents unique challenges due to the scarcity of domain-specific queries and corresponding ground truths, as well as a lack of systematic approaches to diagnosing the cause of failure cases -- whether they stem from knowledge deficits or issues related to system robustness. To address these challenges, we introduce GRAMMAR (GRounded And Modular Methodology for Assessment of RAG), an evaluation framework comprising two key elements: 1) a data generation process that leverages relational databases and LLMs to efficiently produce scalable query-answer pairs. This method facilitates the separation of query logic from linguistic variations for enhanced debugging capabilities; and 2) an evaluation framework that differentiates knowledge gaps from robustness and enables the identification of defective modules. Our empirical results underscore the limitations of current reference-free evaluation approaches and the reliability of GRAMMAR to accurately identify model vulnerabilities.
• 摘要：检索增强生成（RAG）系统已被积极研究并部署在各个行业中，以查询特定领域的知识库。然而，由于缺乏特定领域的查询和相应的基本事实，以及缺乏诊断失败案例原因的系统方法（无论它们是源于知识缺陷还是与系统相关的问题），评估这些系统提出了独特的挑战鲁棒性。为了应对这些挑战，我们引入了 GRAMMAR（RAG 评估的基础模块化方法），这是一个评估框架，包含两个关键要素：1）利用关系数据库和法学硕士有效生成可扩展查询-答案对的数据生成过程。该方法有利于将查询逻辑与语言变化分离，以增强调试能力； 2）一个评估框架，可以区分知识差距和鲁棒性，并能够识别有缺陷的模块。我们的实证结果强调了当前无参考评估方法的局限性以及 GRAMMAR 准确识别模型漏洞的可靠性。

Title: HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

• Authors: Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, Chengzhong Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19245
• Pdf URL: https://arxiv.org/pdf/2404.19245
• Copy Paste: [[2404.19245]] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning(https://arxiv.org/abs/2404.19245)
• Keywords: language model, llm
• Abstract: Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. \href{https://github.com/Clin0212/HydraLoRA}{Code}.
• 摘要：通过引入 LoRA 等参数高效微调 (PEFT) 技术，通过微调使大型语言模型 (LLM) 适应新任务变得更加高效。然而，与完全微调相比，这些方法通常表现不佳，特别是在涉及复杂数据集的场景中。这个问题在复杂领域变得更加明显，突出表明需要改进 PEFT 方法以实现更好的性能。通过一系列实验，我们发现了两个关键见解，揭示了 LoRA 的训练和参数效率低下的问题。基于这些见解，我们开发了 HydraLoRA，这是一种具有非对称结构的 LoRA 框架，无需领域专业知识。我们的实验表明，HydraLoRA 优于其他 PEFT 方法，甚至是那些在训练和推理阶段依赖领域知识的方法。 \href{https://github.com/Clin0212/HydraLoRA}{Code}。

Title: Exploiting Hatred by Targets for Hate Speech Detection on Vietnamese Social Media Texts

• Authors: Cuong Nhat Vo, Khanh Bao Huynh, Son T. Luu, Trong-Hop Do
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19252
• Pdf URL: https://arxiv.org/pdf/2404.19252
• Copy Paste: [[2404.19252]] Exploiting Hatred by Targets for Hate Speech Detection on Vietnamese Social Media Texts(https://arxiv.org/abs/2404.19252)
• Keywords: language model
• Abstract: The growth of social networks makes toxic content spread rapidly. Hate speech detection is a task to help decrease the number of harmful comments. With the diversity in the hate speech created by users, it is necessary to interpret the hate speech besides detecting it. Hence, we propose a methodology to construct a system for targeted hate speech detection from online streaming texts from social media. We first introduce the ViTHSD - a targeted hate speech detection dataset for Vietnamese Social Media Texts. The dataset contains 10K comments, each comment is labeled to specific targets with three levels: clean, offensive, and hate. There are 5 targets in the dataset, and each target is labeled with the corresponding level manually by humans with strict annotation guidelines. The inter-annotator agreement obtained from the dataset is 0.45 by Cohen's Kappa index, which is indicated as a moderate level. Then, we construct a baseline for this task by combining the Bi-GRU-LSTM-CNN with the pre-trained language model to leverage the power of text representation of BERTology. Finally, we suggest a methodology to integrate the baseline model for targeted hate speech detection into the online streaming system for practical application in preventing hateful and offensive content on social media.
• 摘要：社交网络的发展使得有毒内容迅速传播。仇恨言论检测是一项帮助减少有害评论数量的任务。由于用户创建的仇恨言论具有多样性，除了检测仇恨言论之外，还需要对其进行解读。因此，我们提出了一种方法来构建一个系统，用于从社交媒体的在线流文本中进行有针对性的仇恨言论检测。我们首先介绍 ViTHSD——越南社交媒体文本的有针对性的仇恨言论检测数据集。该数据集包含 10K 条评论，每条评论都被标记为特定目标，分为三个级别：干净、攻击性和仇恨。数据集中有 5 个目标，每个目标都由人类按照严格的注释准则手动标记为相应的级别。根据 Cohen 的 Kappa 指数，从数据集中获得的注释者间一致性为 0.45，处于中等水平。然后，我们通过将 Bi-GRU-LSTM-CNN 与预训练的语言模型相结合来构建此任务的基线，以利用 BERTology 文本表示的强大功能。最后，我们提出了一种方法，将有针对性的仇恨言论检测的基线模型集成到在线流媒体系统中，以实际应用于防止社交媒体上的仇恨和攻击性内容。

Title: Suvach -- Generated Hindi QA benchmark

• Authors: Vaishak Narayanan, Prabin Raj KP, Saifudheen Nouphal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19254
• Pdf URL: https://arxiv.org/pdf/2404.19254
• Copy Paste: [[2404.19254]] Suvach -- Generated Hindi QA benchmark(https://arxiv.org/abs/2404.19254)
• Keywords: language model, llm
• Abstract: Current evaluation benchmarks for question answering (QA) in Indic languages often rely on machine translation of existing English datasets. This approach suffers from bias and inaccuracies inherent in machine translation, leading to datasets that may not reflect the true capabilities of EQA models for Indic languages. This paper proposes a new benchmark specifically designed for evaluating Hindi EQA models and discusses the methodology to do the same for any task. This method leverages large language models (LLMs) to generate a high-quality dataset in an extractive setting, ensuring its relevance for the target language. We believe this new resource will foster advancements in Hindi NLP research by providing a more accurate and reliable evaluation tool.
• 摘要：当前印度语言问答 (QA) 的评估基准通常依赖于现有英语数据集的机器翻译。这种方法存在机器翻译固有的偏差和不准确性，导致数据集可能无法反映印度语言 EQA 模型的真实功能。本文提出了一个专门为评估印地语 EQA 模型而设计的新基准，并讨论了对任何任务执行相同操作的方法。该方法利用大型语言模型 (LLM) 在提取设置中生成高质量的数据集，确保其与目标语言的相关性。我们相信这一新资源将通过提供更准确、更可靠的评估工具来促进印地语 NLP 研究的进步。

Title: Octopus v4: Graph of language models

• Authors: Wei Chen, Zhiyuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19296
• Pdf URL: https://arxiv.org/pdf/2404.19296
• Copy Paste: [[2404.19296]] Octopus v4: Graph of language models(https://arxiv.org/abs/2404.19296)
• Keywords: language model, gpt
• Abstract: Language models have been effective in a wide range of applications, yet the most sophisticated models are often proprietary. For example, GPT-4 by OpenAI and various models by Anthropic are expensive and consume substantial energy. In contrast, the open-source community has produced competitive models, like Llama3. Furthermore, niche-specific smaller language models, such as those tailored for legal, medical or financial tasks, have outperformed their proprietary counterparts. This paper introduces a novel approach that employs \textit{functional tokens} to integrate \textbf{multiple open-source models}, each optimized for particular tasks. Our newly developed Octopus v4 model leverages \textit{functional tokens} to intelligently direct user queries to the most appropriate vertical model and reformat the query to achieve the best performance. Octopus v4, an evolution of the Octopus v1, v2, and v3 models, excels in selection and parameter understanding and reformatting. Additionally, we explore the use of graph as a versatile data structure that effectively coordinates multiple open-source models by harnessing the capabilities of the Octopus model and \textit{functional tokens}. Use our open-sourced GitHub (\url{https://www.nexa4ai.com/}) to try Octopus v4 models (\url{https://huggingface.co/NexaAIDev/Octopus-v4}), and contrite to a larger graph of language models. By activating models less than 10B parameters, we achieved SOTA MMLU score of 74.8 among the same level models.
• 摘要：语言模型在广泛的应用中都很有效，但最复杂的模型通常是专有的。例如，OpenAI 的 GPT-4 和 Anthropic 的各种模型价格昂贵且消耗大量能源。相比之下，开源社区已经产生了具有竞争力的模型，例如 Llama3。此外，针对特定领域的较小语言模型，例如为法律、医疗或金融任务量身定制的语言模型，其性能优于其专有同行。本文介绍了一种新颖的方法，该方法采用 \textit{功能标记} 来集成 \textbf{多个开源模型}，每个模型都针对特定任务进行了优化。我们新开发的 Octopus v4 模型利用 \textit{功能标记} 智能地将用户查询引导到最合适的垂直模型，并重新格式化查询以实现最佳性能。 Octopus v4 是 Octopus v1、v2 和 v3 模型的演变，在选择、参数理解和重新格式化方面表现出色。此外，我们探索使用图作为一种多功能数据结构，通过利用 Octopus 模型和 \textit{功能令牌} 的功能来有效协调多个开源模型。使用我们的开源 GitHub (\url{https://www.nexa4ai.com/}) 尝试 Octopus v4 模型 (\url{https://huggingface.co/NexaAIDev/Octopus-v4})，并忏悔更大的语言模型图。通过激活少于 10B 参数的模型，我们在同级别模型中获得了 74.8 的 SOTA MMLU 分数。

Title: Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget

• Authors: Minh Duc Bui, Fabian David Schmidt, Goran Glavaš, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19319
• Pdf URL: https://arxiv.org/pdf/2404.19319
• Copy Paste: [[2404.19319]] Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget(https://arxiv.org/abs/2404.19319)
• Keywords: language model
• Abstract: Compared to standard language model (LM) pretraining (i.e., from scratch), Knowledge Distillation (KD) entails an additional forward pass through a teacher model that is typically substantially larger than the target student model. As such, KD in LM pretraining materially slows down throughput of pretraining instances vis-a-vis pretraining from scratch. Scaling laws of LM pretraining suggest that smaller models can close the gap to larger counterparts if trained on more data (i.e., processing more tokens)-and under a fixed computation budget, smaller models are able be process more data than larger models. We thus hypothesize that KD might, in fact, be suboptimal to pretraining from scratch for obtaining smaller LMs, when appropriately accounting for the compute budget. To test this, we compare pretraining from scratch against several KD strategies for masked language modeling (MLM) in a fair experimental setup, with respect to amount of computation as well as pretraining data. Downstream results on GLUE, however, do not confirm our hypothesis: while pretraining from scratch performs comparably to ordinary KD under a fixed computation budget, more sophisticated KD strategies, namely TinyBERT (Jiao et al., 2020) and MiniLM (Wang et al., 2023), outperform it by a notable margin. We further find that KD yields larger gains over pretraining from scratch when the data must be repeated under the fixed computation budget.
• 摘要：与标准语言模型 (LM) 预训练（即从头开始）相比，知识蒸馏 (KD) 需要对教师模型进行额外的前向传递，该模型通常比目标学生模型大得多。因此，与从头开始的预训练相比，LM 预训练中的 KD 大大降低了预训练实例的吞吐量。 LM 预训练的缩放定律表明，如果使用更多数据进行训练（即处理更多标记），较小的模型可以缩小与较大模型的差距，并且在固定的计算预算下，较小的模型能够比较较大的模型处理更多的数据。因此，我们假设，在适当考虑计算预算时，KD 实际上可能不是从头开始预训练以获得更小的 LM 的最佳选择。为了测试这一点，我们在公平的实验设置中将从头开始的预训练与掩码语言建模 (MLM) 的几种 KD 策略进行比较，包括计算量和预训练数据。然而，GLUE 上的下游结果并没有证实我们的假设：虽然从头开始的预训练在固定计算预算下的性能与普通 KD 相当，但更复杂的 KD 策略，即 TinyBERT（Jiao 等人，2020 年）和 MiniLM（Wang 等人，2020 年）。 ，2023），其表现明显优于它。我们进一步发现，当数据必须在固定计算预算下重复时，KD 比从头开始预训练能产生更大的收益。

Title: StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation

• Authors: Xiaoming Liu, Chen Liu, Zhaohan Zhang, Chengzhengxu Li, Longtian Wang, Yu Lan, Chao Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19335
• Pdf URL: https://arxiv.org/pdf/2404.19335
• Copy Paste: [[2404.19335]] StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation(https://arxiv.org/abs/2404.19335)
• Keywords: language model, prompt
• Abstract: Large language models have shown their ability to become effective few-shot learners with prompting, revoluting the paradigm of learning with data scarcity. However, this approach largely depends on the quality of prompt initialization, and always exhibits large variability among different runs. Such property makes prompt tuning highly unreliable and vulnerable to poorly constructed prompts, which limits its extension to more real-world applications. To tackle this issue, we propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by the prompt initialization. Furthermore, we optimize soft prompts with contrastive learning for utilizing class-aware information in the training process to maintain model performance. Experimental results demonstrate that \sysname outperforms state-of-the-art methods by 7.20% in accuracy and reduces the standard deviation by 2.02 on average. Furthermore, extensive experiments underscore its robustness and stability across 7 datasets covering various tasks.
• 摘要：大型语言模型已经显示出它们能够通过提示、彻底改变数据稀缺的学习范式，成为有效的小样本学习者。然而，这种方法在很大程度上取决于提示初始化的质量，并且在不同的运行之间总是表现出很大的变化。这种属性使得提示调整非常不可靠，并且容易受到构造不良的提示的影响，这限制了其扩展到更多现实世界的应用程序。为了解决这个问题，我们建议将硬提示和软提示视为单独的输入，以减轻提示初始化带来的噪音。此外，我们通过对比学习来优化软提示，以便在训练过程中利用类别感知信息来维持模型性能。实验结果表明，\sysname 的准确率比最先进的方法高出 7.20%，平均标准差降低了 2.02。此外，广泛的实验强调了它在涵盖各种任务的 7 个数据集上的鲁棒性和稳定性。

Title: Evaluating Lexicon Incorporation for Depression Symptom Estimation

• Authors: Kirill Milintsevich, Gaël Dias, Kairit Sirts
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19359
• Pdf URL: https://arxiv.org/pdf/2404.19359
• Copy Paste: [[2404.19359]] Evaluating Lexicon Incorporation for Depression Symptom Estimation(https://arxiv.org/abs/2404.19359)
• Keywords: language model
• Abstract: This paper explores the impact of incorporating sentiment, emotion, and domain-specific lexicons into a transformer-based model for depression symptom estimation. Lexicon information is added by marking the words in the input transcripts of patient-therapist conversations as well as in social media posts. Overall results show that the introduction of external knowledge within pre-trained language models can be beneficial for prediction performance, while different lexicons show distinct behaviours depending on the targeted task. Additionally, new state-of-the-art results are obtained for the estimation of depression level over patient-therapist interviews.
• 摘要：本文探讨了将情绪、情感和特定领域词汇纳入基于变压器的抑郁症状估计模型的影响。通过标记患者与治疗师对话的输入记录以及社交媒体帖子中的单词来添加词典信息。总体结果表明，在预训练的语言模型中引入外部知识可能有利于预测性能，而不同的词典根据目标任务表现出不同的行为。此外，还获得了通过患者与治疗师访谈评估抑郁水平的最新最先进结果。

Title: Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models

• Authors: Yunhao Zhang, Shaonan Wang, Xinyi Dong, Jiajun Yu, Chengqing Zong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19364
• Pdf URL: https://arxiv.org/pdf/2404.19364
• Copy Paste: [[2404.19364]] Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models(https://arxiv.org/abs/2404.19364)
• Keywords: language model
• Abstract: Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.
• 摘要：神经语言模型，特别是大规模的神经语言模型，在一系列研究中已被一致证明在预测大脑神经活动方面最有效。然而，之前的研究忽略了这些模型与心理上合理的模型的比较。此外，评估依赖于有限的、单一模式的英语认知数据集。为了解决这些问题，我们进行了一项分析，比较了各种神经语言模型和心理上合理的模型的编码性能。我们的研究利用了广泛的多模式认知数据集，检查双语单词和话语水平。令人惊讶的是，我们的研究结果表明，在不同的背景下，心理上合理的模型都优于神经语言模型，涵盖功能磁共振成像和眼动追踪等不同模式，并且涵盖从英语到中文的语言。在心理学上合理的模型中，包含具体信息的模型显得尤为特殊。该模型在单词和话语层面都表现出了卓越的性能，对英语和汉语多个区域的大脑激活表现出稳健的预测。

Title: Evaluating Telugu Proficiency in Large Language Models_ A Comparative Analysis of ChatGPT and Gemini

• Authors: Katikela Sreeharsha Kishore, Rahimanuddin Shaik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19369
• Pdf URL: https://arxiv.org/pdf/2404.19369
• Copy Paste: [[2404.19369]] Evaluating Telugu Proficiency in Large Language Models_ A Comparative Analysis of ChatGPT and Gemini(https://arxiv.org/abs/2404.19369)
• Keywords: language model, gpt, llm, chat
• Abstract: The growing prominence of large language models (LLMs) necessitates the exploration of their capabilities beyond English. This research investigates the Telugu language proficiency of ChatGPT and Gemini, two leading LLMs. Through a designed set of 20 questions encompassing greetings, grammar, vocabulary, common phrases, task completion, and situational reasoning, the study delves into their strengths and weaknesses in handling Telugu. The analysis aims to identify the LLM that demonstrates a deeper understanding of Telugu grammatical structures, possesses a broader vocabulary, and exhibits superior performance in tasks like writing and reasoning. By comparing their ability to comprehend and use everyday Telugu expressions, the research sheds light on their suitability for real-world language interaction. Furthermore, the evaluation of adaptability and reasoning capabilities provides insights into how each LLM leverages Telugu to respond to dynamic situations. This comparative analysis contributes to the ongoing discussion on multilingual capabilities in AI and paves the way for future research in developing LLMs that can seamlessly integrate with Telugu-speaking communities.
• 摘要：大型语言模型 (LLM) 的地位日益突出，因此需要探索其英语以外的功能。这项研究调查了两位领先的法学硕士 ChatGPT 和 Gemini 的泰卢固语语言能力。通过设计的 20 个问题，涵盖问候语、语法、词汇、常用短语、任务完成和情境推理，该研究深入探讨了他们在处理泰卢固语方面的优势和劣势。该分析旨在确定法学硕士能够对泰卢固语语法结构有更深入的理解，拥有更广泛的词汇量，并在写作和推理等任务中表现出卓越的表现。通过比较他们理解和使用日常泰卢固语表达的能力，该研究揭示了他们对现实世界语言互动的适用性。此外，对适应性和推理能力的评估可以深入了解每个法学硕士如何利用泰卢固语来应对动态情况。这种比较分析有助于正在进行的关于人工智能多语言能力的讨论，并为未来开发能够与泰卢固语社区无缝集成的法学硕士的研究铺平道路。

Title: Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning

• Authors: Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, Olivier Pietquin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19409
• Pdf URL: https://arxiv.org/pdf/2404.19409
• Copy Paste: [[2404.19409]] Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning(https://arxiv.org/abs/2404.19409)
• Keywords: language model, llm, prompt
• Abstract: While Reinforcement Learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations' and LLM's rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation. We show the effectiveness of RCfD on three language tasks, which achieves comparable performance to carefully tuned baselines while mitigating ROO.
• 摘要：虽然强化学习 (RL) 已被证明对于调整大型语言模型 (LLM) 至关重要，但它可能导致奖励过度优化 (ROO)。现有方法通过添加 KL 正则化来解决 ROO，这需要计算成本高昂的超参数调整。此外，KL 正则化仅侧重于规范语言策略，而忽略了正则化的潜在来源：奖励函数本身。受演示引导 RL 的启发，我们在此引入了演示奖励校准 (RCfD)，它利用人类演示和奖励模型来重新校准奖励目标。正式地，给定一个提示，RCfD 目标会最小化演示和 LLM 奖励之间的距离，而不是直接最大化奖励函数。这种目标转变避免激励 LLM 利用奖励模型，并促进更自然和多样化的语言生成。我们展示了 RCfD 在三个语言任务上的有效性，它在减轻 ROO 的同时实现了与精心调整的基线相当的性能。

Title: Sõnajaht: Definition Embeddings and Semantic Search for Reverse Dictionary Creation

• Authors: Aleksei Dorkin, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19430
• Pdf URL: https://arxiv.org/pdf/2404.19430
• Copy Paste: [[2404.19430]] Sõnajaht: Definition Embeddings and Semantic Search for Reverse Dictionary Creation(https://arxiv.org/abs/2404.19430)
• Keywords: language model
• Abstract: We present an information retrieval based reverse dictionary system using modern pre-trained language models and approximate nearest neighbors search algorithms. The proposed approach is applied to an existing Estonian language lexicon resource, S\~onaveeb (word web), with the purpose of enhancing and enriching it by introducing cross-lingual reverse dictionary functionality powered by semantic search. The performance of the system is evaluated using both an existing labeled English dataset of words and definitions that is extended to contain also Estonian and Russian translations, and a novel unlabeled evaluation approach that extracts the evaluation data from the lexicon resource itself using synonymy relations. Evaluation results indicate that the information retrieval based semantic search approach without any model training is feasible, producing median rank of 1 in the monolingual setting and median rank of 2 in the cross-lingual setting using the unlabeled evaluation approach, with models trained for cross-lingual retrieval and including Estonian in their training data showing superior performance in our particular task.
• 摘要：我们提出了一种基于信息检索的反向词典系统，使用现代预训练语言模型和近似最近邻搜索算法。所提出的方法应用于现有的爱沙尼亚语言词典资源 S\~onaveeb（词网），目的是通过引入由语义搜索支持的跨语言反向词典功能来增强和丰富它。使用现有的标记英语单词和定义数据集（扩展为还包含爱沙尼亚语和俄语翻译）和新颖的未标记评估方法（使用同义词关系从词典资源本身提取评估数据）来评估系统的性能。评估结果表明，基于信息检索的语义搜索方法无需任何模型训练是可行的，使用无标签评估方法，在单语言环境中产生中位数 1，在跨语言环境中产生中位数 2，模型经过跨语言训练。语言检索并将爱沙尼亚语纳入其训练数据，在我们的特定任务中表现出卓越的性能。

Title: Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships

• Authors: D. Panas, S. Seth, V. Belle
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19432
• Pdf URL: https://arxiv.org/pdf/2404.19432
• Copy Paste: [[2404.19432]] Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships(https://arxiv.org/abs/2404.19432)
• Keywords: language model, gpt, llm
• Abstract: Two major areas of interest in the era of Large Language Models regard questions of what do LLMs know, and if and how they may be able to reason (or rather, approximately reason). Since to date these lines of work progressed largely in parallel (with notable exceptions), we are interested in investigating the intersection: probing for reasoning about the implicitly-held knowledge. Suspecting the performance to be lacking in this area, we use a very simple set-up of comparisons between cardinalities associated with elements of various subjects (e.g. the number of legs a bird has versus the number of wheels on a tricycle). We empirically demonstrate that although LLMs make steady progress in knowledge acquisition and (pseudo)reasoning with each new GPT release, their capabilities are limited to statistical inference only. It is difficult to argue that pure statistical learning can cope with the combinatorial explosion inherent in many commonsense reasoning tasks, especially once arithmetical notions are involved. Further, we argue that bigger is not always better and chasing purely statistical improvements is flawed at the core, since it only exacerbates the dangerous conflation of the production of correct answers with genuine reasoning ability.
• 摘要：大型语言模型时代的两个主要兴趣领域涉及法学硕士知道什么以及他们是否以及如何能够推理（或者更确切地说，近似推理）的问题。由于迄今为止，这些工作在很大程度上是并行进行的（有明显的例外），因此我们有兴趣研究交叉点：探索对隐含知识的推理。由于怀疑该领域的表现不足，我们使用了一种非常简单的设置，对与不同主题的元素相关的基数进行比较（例如，鸟的腿数与三轮车的轮子数）。我们凭经验证明，尽管法学硕士在每个新的 GPT 版本中在知识获取和（伪）推理方面取得了稳步进展，但他们的能力仅限于统计推理。很难说纯统计学习可以应对许多常识推理任务中固有的组合爆炸，特别是当涉及算术概念时。此外，我们认为，越大并不总是越好，追求纯粹的统计改进在核心上是有缺陷的，因为它只会加剧正确答案的产生与真正的推理能力的危险混合。

Title: FactCheck Editor: Multilingual Text Editor with End-to-End fact-checking

• Authors: Vinay Setty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19482
• Pdf URL: https://arxiv.org/pdf/2404.19482
• Copy Paste: [[2404.19482]] FactCheck Editor: Multilingual Text Editor with End-to-End fact-checking(https://arxiv.org/abs/2404.19482)
• Keywords: llm
• Abstract: We introduce 'FactCheck Editor', an advanced text editor designed to automate fact-checking and correct factual inaccuracies. Given the widespread issue of misinformation, often a result of unintentional mistakes by content creators, our tool aims to address this challenge. It supports over 90 languages and utilizes transformer models to assist humans in the labor-intensive process of fact verification. This demonstration showcases a complete workflow that detects text claims in need of verification, generates relevant search engine queries, and retrieves appropriate documents from the web. It employs Natural Language Inference (NLI) to predict the veracity of claims and uses LLMs to summarize the evidence and suggest textual revisions to correct any errors in the text. Additionally, the effectiveness of models used in claim detection and veracity assessment is evaluated across multiple languages.
• 摘要：我们推出“FactCheck Editor”，这是一种高级文本编辑器，旨在自动进行事实检查并纠正事实不准确之处。鉴于广泛存在的错误信息问题（通常是内容创建者无意的错误造成的），我们的工具旨在解决这一挑战。它支持 90 多种语言，并利用 Transformer 模型来协助人类进行劳动密集型的事实验证过程。该演示展示了一个完整的工作流程，可检测需要验证的文本声明、生成相关搜索引擎查询并从网络检索适当的文档。它采用自然语言推理 (NLI) 来预测声明的准确性，并使用法学硕士来总结证据并建议文本修改以纠正文本中的任何错误。此外，还跨多种语言评估了索赔检测和准确性评估中使用的模型的有效性。

Title: Safe Training with Sensitive In-domain Data: Leveraging Data Fragmentation To Mitigate Linkage Attacks

• Authors: Mariia Ignashina, Julia Ive
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19486
• Pdf URL: https://arxiv.org/pdf/2404.19486
• Copy Paste: [[2404.19486]] Safe Training with Sensitive In-domain Data: Leveraging Data Fragmentation To Mitigate Linkage Attacks(https://arxiv.org/abs/2404.19486)
• Keywords: llm
• Abstract: Current text generation models are trained using real data which can potentially contain sensitive information, such as confidential patient information and the like. Under certain conditions output of the training data which they have memorised can be triggered, exposing sensitive data. To mitigate against this risk we propose a safer alternative which sees fragmented data in the form of domain-specific short phrases randomly grouped together shared instead of full texts. Thus, text fragments that could re-identify an individual cannot be reproduced by the model in one sequence, giving significant protection against linkage attacks. We fine-tune several state-of-the-art LLMs using meaningful syntactic chunks to explore their utility. In particular, we fine-tune BERT-based models to predict two cardiovascular diagnoses. Our results demonstrate the capacity of LLMs to benefit from the pre-trained knowledge and deliver classification results when fine-tuned with fragmented data comparable to fine-tuning with full training data.
• 摘要：当前的文本生成模型是使用可能包含敏感信息（例如机密患者信息等）的真实数据进行训练的。在某些条件下，可以触发他们记忆的训练数据的输出，从而暴露敏感数据。为了减轻这种风险，我们提出了一种更安全的替代方案，它将特定领域的短语形式的碎片数据随机分组在一起共享，而不是全文。因此，模型无法以一个序列再现可以重新识别个体的文本片段，从而提供了针对链接攻击的重要保护。我们使用有意义的句法块对几个最先进的法学硕士进行微调，以探索其实用性。特别是，我们微调基于 BERT 的模型来预测两种心血管诊断。我们的结果证明了法学硕士有能力从预先训练的知识中受益，并在使用碎片数据进行微调时提供分类结果，这与使用完整训练数据进行微调相当。

Title: Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom

• Authors: Shisen Yue, Siyuan Song, Xinyuan Cheng, Hai Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19509
• Pdf URL: https://arxiv.org/pdf/2404.19509
• Copy Paste: [[2404.19509]] Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom(https://arxiv.org/abs/2404.19509)
• Keywords: language model, gpt, llm
• Abstract: Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.
• 摘要：理解话语的非字面意义对于大型语言模型（LLM）成为类人的社交沟通者至关重要。在这项工作中，我们介绍了 SwordsmanImp，这是第一个针对会话含义的基于多回合对话的中文数据集，源自中国情景喜剧 $\textit{武林外传}$ 中的对话。它包括 200 个精心设计的问题，所有问题都注释了违反格莱斯格言的问题。我们在两项任务下测试了八个闭源和开源法学硕士：多项选择题任务和含义解释任务。我们的结果表明，GPT-4 在多项选择题上达到了人类水平的准确率 (94%)。 CausalLM 在 GPT-4 后表现出 78.5% 的准确率。其他模型，包括 GPT-3.5 和几个开源模型，在多项选择题上的准确率较低，从 20% 到 60% 不等。人类评分者被要求对法学硕士产生的含义的解释的合理性、逻辑性和流畅性进行评分。虽然所有模型都能生成基本流畅且自洽的文本，但除 GPT-4 之外，它们的解释在合理性方面得分较低，这表明大多数法学硕士无法对对话中的含义给出令人满意的解释。此外，我们发现法学硕士的表现并没有因格莱斯格言而显着变化，这表明法学硕士似乎并没有以不同的方式处理从不同格言衍生的含义。我们的数据和代码可在 https://github.com/sjtu-compling/llm-pragmatics 获取。

Title: RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing

• Authors: Yucheng Hu, Yuxing Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19543
• Pdf URL: https://arxiv.org/pdf/2404.19543
• Copy Paste: [[2404.19543]] RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing(https://arxiv.org/abs/2404.19543)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have catalyzed significant advancements in Natural Language Processing (NLP), yet they encounter challenges such as hallucination and the need for domain-specific knowledge. To mitigate these, recent methodologies have integrated information retrieved from external resources with LLMs, substantially enhancing their performance across NLP tasks. This survey paper addresses the absence of a comprehensive overview on Retrieval-Augmented Language Models (RALMs), both Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU), providing an in-depth examination of their paradigm, evolution, taxonomy, and applications. The paper discusses the essential components of RALMs, including Retrievers, Language Models, and Augmentations, and how their interactions lead to diverse model structures and applications. RALMs demonstrate utility in a spectrum of tasks, from translation and dialogue systems to knowledge-intensive applications. The survey includes several evaluation methods of RALMs, emphasizing the importance of robustness, accuracy, and relevance in their assessment. It also acknowledges the limitations of RALMs, particularly in retrieval quality and computational efficiency, offering directions for future research. In conclusion, this survey aims to offer a structured insight into RALMs, their potential, and the avenues for their future development in NLP. The paper is supplemented with a Github Repository containing the surveyed works and resources for further study: https://github.com/2471023025/RALM_Survey.
• 摘要：大型语言模型 (LLM) 促进了自然语言处理 (NLP) 领域的重大进步，但它们也遇到了诸如幻觉和对特定领域知识的需求等挑战。为了缓解这些问题，最近的方法将从外部资源检索到的信息与法学硕士相结合，大大提高了它们在 NLP 任务中的表现。本调查论文解决了检索增强语言模型 (RALM)、检索增强生成 (RAG) 和检索增强理解 (RAU) 缺乏全面概述的问题，深入研究了它们的范式、演变、分类和应用程序。本文讨论了 RALM 的基本组件，包括检索器、语言模型和增强，以及它们的交互如何导致不同的模型结构和应用。 RALM 在从翻译和对话系统到知识密集型应用程序的一系列任务中展示了实用性。该调查包括 RALM 的多种评估方法，强调评估中稳健性、准确性和相关性的重要性。它还承认 RALM 的局限性，特别是在检索质量和计算效率方面，为未来的研究提供了方向。总之，本次调查旨在提供对 RALM、其潜力以及其在 NLP 领域未来发展途径的结构化见解。本文还补充了一个 Github 存储库，其中包含调查的作品和供进一步研究的资源：https://github.com/2471023025/RALM_Survey。

Title: Extending Llama-3's Context Ten-Fold Overnight

• Authors: Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, Zhicheng Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19553
• Pdf URL: https://arxiv.org/pdf/2404.19553
• Copy Paste: [[2404.19553]] Extending Llama-3's Context Ten-Fold Overnight(https://arxiv.org/abs/2404.19553)
• Keywords: gpt, llm
• Abstract: We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs' inherent (yet largely underestimated) potential to extend its original context length. In fact, the context length could be extended far beyond 80K with more computation resources. Therefore, the team will publicly release the entire resources (including data, model, data generation pipeline, training code) so as to facilitate the future research from the community: \url{https://github.com/FlagOpen/FlagEmbedding}.
• 摘要：我们通过 QLoRA 微调将 Llama-3-8B-Instruct 的上下文长度从 8K 扩展到 80K。整个训练周期非常高效，在一台 8xA800 (80G) GPU 机器上需要 8 个小时。所得到的模型在广泛的评估任务中表现出卓越的性能，例如 NIHS、主题检索和长上下文语言理解；同时，它也很好地保留了短上下文中的原始能力。戏剧性的上下文扩展主要归因于 GPT-4 生成的仅 3.5K 合成训练样本，这表明法学硕士具有扩展其原始上下文长度的固有（但在很大程度上被低估）潜力。事实上，使用更多的计算资源，上下文长度可以远远超出 80K。因此，团队将公开发布全部资源（包括数据、模型、数据生成管道、训练代码），以方便社区未来的研究：\url{https://github.com/FlagOpen/FlagEmbedding}。

Title: RepEval: Effective Text Evaluation with LLM Representation

• Authors: Shuqian Sheng, Yi Xu, Tianhang Zhang, Zanwei Shen, Luoyi Fu, Jiaxin Ding, Lei Zhou, Xinbing Wang, Chenghu Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19563
• Pdf URL: https://arxiv.org/pdf/2404.19563
• Copy Paste: [[2404.19563]] RepEval: Effective Text Evaluation with LLM Representation(https://arxiv.org/abs/2404.19563)
• Keywords: gpt, llm, prompt
• Abstract: Automatic evaluation metrics for generated texts play an important role in the NLG field, especially with the rapid growth of LLMs. However, existing metrics are often limited to specific scenarios, making it challenging to meet the evaluation requirements of expanding LLM applications. Therefore, there is a demand for new, flexible, and effective metrics. In this study, we introduce RepEval, the first metric leveraging the projection of LLM representations for evaluation. RepEval requires minimal sample pairs for training, and through simple prompt modifications, it can easily transition to various tasks. Results on ten datasets from three tasks demonstrate the high effectiveness of our method, which exhibits stronger correlations with human judgments compared to previous metrics, even outperforming GPT-4. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
• 摘要：生成文本的自动评估指标在 NLG 领域发挥着重要作用，尤其是随着法学硕士的快速增长。然而，现有的指标往往仅限于特定场景，这使得满足扩展的LLM申请的评估要求具有挑战性。因此，需要新的、灵活的、有效的指标。在本研究中，我们引入了 RepEval，这是第一个利用 LLM 表示投影进行评估的指标。 RepEval 需要最少的样本对进行训练，通过简单的提示修改，可以轻松过渡到各种任务。三项任务的十个数据集的结果证明了我们的方法的高效性，与之前的指标相比，该方法与人类判断的相关性更强，甚至优于 GPT-4。我们的工作强调了法学硕士表示中嵌入的有关文本质量的丰富信息，为新指标的开发提供了见解。

Title: Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning

• Authors: Xuanli He, Jun Wang, Qiongkai Xu, Pasquale Minervini, Pontus Stenetorp, Benjamin I. P. Rubinstein, Trevor Cohn
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19597
• Pdf URL: https://arxiv.org/pdf/2404.19597
• Copy Paste: [[2404.19597]] Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning(https://arxiv.org/abs/2404.19597)
• Keywords: language model, gpt, llm
• Abstract: The implications of backdoor attacks on English-centric large language models (LLMs) have been widely examined - such attacks can be achieved by embedding malicious behaviors during training and activated under specific conditions that trigger malicious outputs. However, the impact of backdoor attacks on multilingual models remains under-explored. Our research focuses on cross-lingual backdoor attacks against multilingual LLMs, particularly investigating how poisoning the instruction-tuning data in one or two languages can affect the outputs in languages whose instruction-tuning data was not poisoned. Despite its simplicity, our empirical analysis reveals that our method exhibits remarkable efficacy in models like mT5, BLOOM, and GPT-3.5-turbo, with high attack success rates, surpassing 95% in several languages across various scenarios. Alarmingly, our findings also indicate that larger models show increased susceptibility to transferable cross-lingual backdoor attacks, which also applies to LLMs predominantly pre-trained on English data, such as Llama2, Llama3, and Gemma. Moreover, our experiments show that triggers can still work even after paraphrasing, and the backdoor mechanism proves highly effective in cross-lingual response settings across 25 languages, achieving an average attack success rate of 50%. Our study aims to highlight the vulnerabilities and significant security risks present in current multilingual LLMs, underscoring the emergent need for targeted security measures.
• 摘要：后门攻击对以英语为中心的大语言模型 (LLM) 的影响已被广泛研究——此类攻击可以通过在训练期间嵌入恶意行为并在触发恶意输出的特定条件下激活来实现。然而，后门攻击对多语言模型的影响仍未得到充分研究。我们的研究重点是针对多语言法学硕士的跨语言后门攻击，特别是调查一两种语言的指令调优数据中毒如何影响指令调优数据未中毒的语言的输出。尽管很简单，但我们的实证分析表明，我们的方法在 mT5、BLOOM 和 GPT-3.5-turbo 等模型中表现出显着的功效，攻击成功率很高，在多种语言和各种场景下，攻击成功率超过 95%。令人担忧的是，我们的研究结果还表明，较大的模型对可转移的跨语言后门攻击的敏感性增加，这也适用于主要使用英语数据进行预训练的法学硕士，例如 Llama2、Llama3 和 Gemma。此外，我们的实验表明，即使在释义后，触发器仍然可以工作，并且后门机制在25种语言的跨语言响应设置中证明非常有效，平均攻击​​成功率达到50%。我们的研究旨在强调当前多语言法学硕士中存在的漏洞和重大安全风险，强调迫切需要有针对性的安全措施。

Title: When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively

• Authors: Tiziano Labruna, Jon Ander Campos, Gorka Azkune
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.19705
• Pdf URL: https://arxiv.org/pdf/2404.19705
• Copy Paste: [[2404.19705]] When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively(https://arxiv.org/abs/2404.19705)
• Keywords: language model, llm
• Abstract: In this paper, we demonstrate how Large Language Models (LLMs) can effectively learn to use an off-the-shelf information retrieval (IR) system specifically when additional context is required to answer a given question. Given the performance of IR systems, the optimal strategy for question answering does not always entail external information retrieval; rather, it often involves leveraging the parametric memory of the LLM itself. Prior research has identified this phenomenon in the PopQA dataset, wherein the most popular questions are effectively addressed using the LLM's parametric memory, while less popular ones require IR system usage. Following this, we propose a tailored training approach for LLMs, leveraging existing open-domain question answering datasets. Here, LLMs are trained to generate a special token, , when they do not know the answer to a question. Our evaluation of the Adaptive Retrieval LLM (Adapt-LLM) on the PopQA dataset showcases improvements over the same LLM under three configurations: (i) retrieving information for all the questions, (ii) using always the parametric memory of the LLM, and (iii) using a popularity threshold to decide when to use a retriever. Through our analysis, we demonstrate that Adapt-LLM is able to generate the token when it determines that it does not know how to answer a question, indicating the need for IR, while it achieves notably high accuracy levels when it chooses to rely only on its parametric memory.
• 摘要：在本文中，我们展示了大型语言模型 (LLM) 如何有效地学习使用现成的信息检索 (IR) 系统，特别是在需要额外背景来回答给定问题时。考虑到 IR 系统的性能，问答的最佳策略并不总是需要外部信息检索；相反，它通常涉及利用 LLM 本身的参数记忆。先前的研究已经在 PopQA 数据集中发现了这种现象，其中最受欢迎的问题可以使用 LLM 的参数记忆有效地解决，而不太流行的问题则需要使用 IR 系统。在此之后，我们提出了一种针对 LLM 的定制训练方法，利用现有的开放域问答数据集。在这里，当 LLM 不知道问题的答案时，它们被训练来生成一个特殊的标记 。我们对 PopQA 数据集上的自适应检索 LLM (Adapt-LLM) 的评估表明，在以下三种配置下，该 LLM 比相同的 LLM 有所改进：(i) 检索所有问题的信息，(ii) 始终使用 LLM 的参数记忆，以及 (iii) 使用流行度阈值来决定何时使用检索器。通过我们的分析，我们证明 Adapt-LLM 在确定它不知道如何回答问题时能够生成 标记，这表明需要 IR，而当它选择仅依赖其参数记忆时，它可以实现非常高的准确度。

Title: Automated Generation of High-Quality Medical Simulation Scenarios Through Integration of Semi-Structured Data and Large Language Models

• Authors: Scott Sumpter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19713
• Pdf URL: https://arxiv.org/pdf/2404.19713
• Copy Paste: [[2404.19713]] Automated Generation of High-Quality Medical Simulation Scenarios Through Integration of Semi-Structured Data and Large Language Models(https://arxiv.org/abs/2404.19713)
• Keywords: language model, gpt, llm, chat
• Abstract: This study introduces a transformative framework for medical education by integrating semi-structured data with Large Language Models (LLMs), primarily OpenAIs ChatGPT3.5, to automate the creation of medical simulation scenarios. Traditionally, developing these scenarios was a time-intensive process with limited flexibility to meet diverse educational needs. The proposed approach utilizes AI to efficiently generate detailed, clinically relevant scenarios that are tailored to specific educational objectives. This innovation has significantly reduced the time and resources required for scenario development, allowing for a broader variety of simulations. Preliminary feedback from educators and learners has shown enhanced engagement and improved knowledge acquisition, confirming the effectiveness of this AI-enhanced methodology in simulation-based learning. The integration of structured data with LLMs not only streamlines the creation process but also offers a scalable, dynamic solution that could revolutionize medical training, highlighting the critical role of AI in advancing educational outcomes and patient care standards.
• 摘要：本研究通过将半结构化数据与大型语言模型 (LLM)（主要是 OpenAI ChatGPT3.5）集成，引入了医学教育变革框架，以自动创建医学模拟场景。传统上，开发这些场景是一个耗时的过程，满足不同教育需求的灵活性有限。所提出的方法利用人工智能有效地生成针对特定教育目标定制的详细的、临床相关的场景。这项创新显着减少了场景开发所需的时间和资源，从而可以进行更广泛的模拟。教育工作者和学习者的初步反馈表明，参与度有所提高，知识获取也有所改善，证实了这种人工智能增强方法在基于模拟的学习中的有效性。结构化数据与法学硕士的集成不仅简化了创建过程，还提供了可扩展的动态解决方案，可以彻底改变医疗培训，凸显人工智能在提高教育成果和患者护理标准方面的关键作用。

Title: Iterative Reasoning Preference Optimization

• Authors: Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, Jason Weston
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19733
• Pdf URL: https://arxiv.org/pdf/2404.19733
• Copy Paste: [[2404.19733]] Iterative Reasoning Preference Optimization(https://arxiv.org/abs/2404.19733)
• Keywords: chat, chain-of-thought
• Abstract: Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy for Llama-2-70B-Chat from 55.6% to 81.6% on GSM8K (and 88.7% with majority voting out of 32 samples), from 12.5% to 20.8% on MATH, and from 77.8% to 86.7% on ARC-Challenge, which outperforms other Llama-2-based models not relying on additionally sourced datasets.
• 摘要：迭代偏好优化方法最近被证明对于一般指令调整任务表现良好，但通常对推理任务几乎没有改进（Yuan 等人，2024；Chen 等人，2024）。在这项工作中，我们开发了一种迭代方法，通过优化导致正确答案的获胜与失败推理步骤来优化竞争生成的思想链 (CoT) 候选者之间的偏好。我们使用修改后的 DPO 损失（Rafailov 等人，2023）和附加的负对数似然项进行训练，我们发现这至关重要。我们展示了该方案的重复迭代中推理能力的提高。虽然仅依赖于训练集中的示例，但我们的方法使 Llama-2-70B-Chat 在 GSM8K 上的准确率从 55.6% 提高到 81.6%（32 个样本中的多数投票为 88.7%），从 12.5% 提高到 20.8 MATH 上的得分从 77.8% 上升到 86.7%，ARC-Challenge 的得分优于其他基于 Llama-2 且不依赖额外来源数据集的模型。

Title: Better & Faster Large Language Models via Multi-token Prediction

• Authors: Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19737
• Pdf URL: https://arxiv.org/pdf/2404.19737
• Copy Paste: [[2404.19737]] Better & Faster Large Language Models via Multi-token Prediction(https://arxiv.org/abs/2404.19737)
• Keywords: language model, gpt
• Abstract: Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes, and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes.
• 摘要：GPT 和 Llama 等大型语言模型是通过下一个令牌预测损失进行训练的。在这项工作中，我们建议训练语言模型来同时预测多个未来标记，从而提高样本效率。更具体地说，在训练语料库中的每个位置，我们要求模型使用在共享模型主干上运行的 n 个独立输出头来预测以下 n 个标记。将多标记预测作为辅助训练任务，我们测量了改进的下游能力，而代码和自然语言模型的训练时间没有开销。该方法对于较大的模型尺寸越来越有用，并且在训练多个时期时保持其吸引力。在编码等生成基准上，收益尤其明显，我们的模型始终比强大的基准高出几个百分点。与同类 next-token 模型相比，我们的 13B 参数模型在 HumanEval 上解决的问题多解决了 12%，在 MBPP 上解决的问题多解决了 17%。小算法任务的实验表明，多token预测有利于归纳头和算法推理能力的发展。另一个好处是，即使批量大小较大，使用 4 令牌预测训练的模型的推理速度也可提高 3 倍。