2024-08-02

Title: ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget

• Authors: Riccardo Orlando, Pere-Lluis Huguet-Cabot, Edoardo Barba, Roberto Navigli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00103
• Pdf URL: https://arxiv.org/pdf/2408.00103
• Copy Paste: [[2408.00103]] ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget(https://arxiv.org/abs/2408.00103)
• Keywords: language model
• Abstract: Entity Linking (EL) and Relation Extraction (RE) are fundamental tasks in Natural Language Processing, serving as critical components in a wide range of applications. In this paper, we propose ReLiK, a Retriever-Reader architecture for both EL and RE, where, given an input text, the Retriever module undertakes the identification of candidate entities or relations that could potentially appear within the text. Subsequently, the Reader module is tasked to discern the pertinent retrieved entities or relations and establish their alignment with the corresponding textual spans. Notably, we put forward an innovative input representation that incorporates the candidate entities or relations alongside the text, making it possible to link entities or extract relations in a single forward pass and to fully leverage pre-trained language models contextualization capabilities, in contrast with previous Retriever-Reader-based methods, which require a forward pass for each candidate. Our formulation of EL and RE achieves state-of-the-art performance in both in-domain and out-of-domain benchmarks while using academic budget training and with up to 40x inference speed compared to competitors. Finally, we show how our architecture can be used seamlessly for Information Extraction (cIE), i.e. EL + RE, and setting a new state of the art by employing a shared Reader that simultaneously extracts entities and relations.
• 摘要：实体链接 (EL) 和关系提取 (RE) 是自然语言处理中的基本任务，是各种应用中的关键组件。在本文中，我们提出了 ReLiK，一种用于 EL 和 RE 的检索器-阅读器架构，其中，给定输入文本，检索器模块负责识别可能出现在文本中的候选实体或关系。随后，阅读器模块负责辨别相关的检索到的实体或关系，并建立它们与相应文本跨度的对齐。值得注意的是，我们提出了一种创新的输入表示，将候选实体或关系与文本结合在一起，从而可以在一次前向传递中链接实体或提取关系，并充分利用预训练语言模型的语境化功能，这与以前基于检索器-阅读器的方法不同，后者需要对每个候选进行前向传递。我们的 EL 和 RE 公式在使用学术预算培训的同时，在域内和域外基准测试中实现了最先进的性能，并且推理速度比竞争对手快 40 倍。最后，我们展示了如何将我们的架构无缝地用于信息提取（cIE），即 EL + RE，并通过使用同时提取实体和关系的共享读取器来设定新的最先进水平。

Title: Gemma 2: Improving Open Language Models at a Practical Size

• Authors: Gemma Team: Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00118
• Pdf URL: https://arxiv.org/pdf/2408.00118
• Copy Paste: [[2408.00118]] Gemma 2: Improving Open Language Models at a Practical Size(https://arxiv.org/abs/2408.00118)
• Keywords: language model
• Abstract: In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.
• 摘要：在这项工作中，我们推出了 Gemma 2，这是 Gemma 系列轻量级、最先进的开放模型的新成员，其规模从 20 亿到 270 亿个参数不等。在这个新版本中，我们对 Transformer 架构应用了几项已知的技术修改，例如交错局部全局注意力（Beltagy 等人，2020a）和组查询注意力（Ainslie 等人，2023）。我们还使用知识蒸馏（Hinton 等人，2015）而不是下一个标记预测来训练 2B 和 9B 模型。生成的模型提供了与其规模相称的最佳性能，甚至可以提供比大 2-3 倍的模型更具竞争力的替代方案。我们向社区发布了所有模型。

Title: A Course Shared Task on Evaluating LLM Output for Clinical Questions

• Authors: Yufang Hou, Thy Thy Tran, Doan Nam Long Vu, Yiwen Cao, Kai Li, Lukas Rohde, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00122
• Pdf URL: https://arxiv.org/pdf/2408.00122
• Copy Paste: [[2408.00122]] A Course Shared Task on Evaluating LLM Output for Clinical Questions(https://arxiv.org/abs/2408.00122)
• Keywords: language model, llm
• Abstract: This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.
• 摘要：本文介绍了我们在达姆施塔特工业大学 2023/2024 年语言技术基础 (FoLT) 课程中组织的一项共享任务，该任务侧重于评估大型语言模型 (LLM) 在生成与健康相关的临床问题的有害答案方面的输出。我们描述了任务设计注意事项并报告了我们从学生那里收到的反馈。我们希望本文中报告的任务和发现与教授自然语言处理 (NLP) 和设计课程作业的教师相关。

Title: Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment

• Authors: Sangwon Yu, Jongyoon Song, Bongkyu Hwang, Hoyoung Kang, Sooah Cho, Junhwa Choi, Seongho Joe, Taehee Lee, Youngjune L. Gwon, Sungroh Yoon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00137
• Pdf URL: https://arxiv.org/pdf/2408.00137
• Copy Paste: [[2408.00137]] Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment(https://arxiv.org/abs/2408.00137)
• Keywords: language model, prompt
• Abstract: A binary decision task, like yes-no questions or answer verification, reflects a significant real-world scenario such as where users look for confirmation about the correctness of their decisions on specific issues. In this work, we observe that language models exhibit a negative bias in the binary decisions of complex reasoning tasks. Based on our observations and the rationale about attention-based model dynamics, we propose a negative attention score (NAS) to systematically and quantitatively formulate negative bias. Based on NAS, we identify attention heads that attend to negative tokens provided in the instructions as answer candidate of binary decisions, regardless of the question in the prompt, and validate their association with the negative bias. Additionally, we propose the negative attention score alignment (NASA) method, which is a parameter-efficient fine-tuning technique to address the extracted negatively biased attention heads. Experimental results from various domains of reasoning tasks and large model search space demonstrate that NASA significantly reduces the gap between precision and recall caused by negative bias while preserving their generalization abilities. Our codes are available at \url{this https URL}.
• 摘要：二元决策任务（例如是非问题或答案验证）反映了现实世界中的一个重要场景，例如用户寻求确认其在特定问题上的决策是否正确。在这项工作中，我们观察到语言模型在复杂推理任务的二元决策中表现出负面偏见。根据我们的观察和基于注意力的模型动力学原理，我们提出了一个负注意力分数（NAS）来系统地定量地制定负面偏见。基于 NAS，我们识别出关注说明中提供的负面标记作为二元决策的答案候选的注意力头，而不管提示中的问题是什么，并验证它们与负面偏见的关联。此外，我们提出了负注意力分数对齐（NASA）方法，这是一种参数高效的微调技术，用于解决提取的负偏差注意力头。来自各种推理任务领域和大型模型搜索空间的实验结果表明，NAS 显着减少了由负偏差引起的精确度和召回率之间的差距，同时保留了它们的泛化能力。我们的代码可在 \url{this https URL} 获得。

Title: Distributed In-Context Learning under Non-IID Among Clients

• Authors: Siqi Liang, Sumyeong Ahn, Jiayu Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00144
• Pdf URL: https://arxiv.org/pdf/2408.00144
• Copy Paste: [[2408.00144]] Distributed In-Context Learning under Non-IID Among Clients(https://arxiv.org/abs/2408.00144)
• Keywords: language model, llm
• Abstract: Advancements in large language models (LLMs) have shown their effectiveness in multiple complicated natural language reasoning tasks. A key challenge remains in adapting these models efficiently to new or unfamiliar tasks. In-context learning (ICL) provides a promising solution for few-shot adaptation by retrieving a set of data points relevant to a query, called in-context examples (ICE), from a training dataset and providing them during the inference as context. Most existing studies utilize a centralized training dataset, yet many real-world datasets may be distributed among multiple clients, and remote data retrieval can be associated with costs. Especially when the client data are non-identical independent distributions (non-IID), retrieving from clients a proper set of ICEs needed for a test query presents critical challenges. In this paper, we first show that in this challenging setting, test queries will have different preferences among clients because of non-IIDness, and equal contribution often leads to suboptimal performance. We then introduce a novel approach to tackle the distributed non-IID ICL problem when a data usage budget is present. The principle is that each client's proper contribution (budget) should be designed according to the preference of each query for that client. Our approach uses a data-driven manner to allocate a budget for each client, tailored to each test query. Through extensive empirical studies on diverse datasets, our framework demonstrates superior performance relative to competing baselines.
• 摘要：大型语言模型 (LLM) 的进步已在多个复杂的自然语言推理任务中显示出其有效性。如何有效地将这些模型适应新的或不熟悉的任务仍然是一个关键挑战。上下文学习 (ICL) 通过从训练数据集中检索与查询相关的一组数据点（称为上下文示例 (ICE)）并在推理过程中将其作为上下文提供，为少样本适应提供了一种有希望的解决方案。大多数现有研究使用集中式训练数据集，但许多现实世界的数据集可能分布在多个客户端之间，并且远程数据检索可能与成本相关。特别是当客户端数据是非相同独立分布 (non-IID) 时，从客户端检索测试查询所需的一组适当的 ICE 提出了严峻的挑战。在本文中，我们首先表明，在这种具有挑战性的环境中，由于非 IID 性，测试查询在客户端之间会有不同的偏好，而同等的贡献通常会导致性能不佳。然后，我们介绍了一种新方法来解决存在数据使用预算时的分布式非 IID ICL 问题。原则是，每个客户的适当贡献（预算）应根据该客户的每个查询的偏好来设计。我们的方法使用数据驱动的方式为每个客户分配预算，并针对每个测试查询进行量身定制。通过对各种数据集的广泛实证研究，我们的框架相对于竞争基线表现出卓越的性能。

Title: Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting

• Authors: Ying Li, Rahul Singh, Tarun Joshi, Agus Sudjianto
• Subjects: cs.CL, cs.AI, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00161
• Pdf URL: https://arxiv.org/pdf/2408.00161
• Copy Paste: [[2408.00161]] Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting(https://arxiv.org/abs/2408.00161)
• Keywords: language model, prompt
• Abstract: Recent work in behavioral testing for natural language processing (NLP) models, such as Checklist, is inspired by related paradigms in software engineering testing. They allow evaluation of general linguistic capabilities and domain understanding, hence can help evaluate conceptual soundness and identify model weaknesses. However, a major challenge is the creation of test cases. The current packages rely on semi-automated approach using manual development which requires domain expertise and can be time consuming. This paper introduces an automated approach to develop test cases by exploiting the power of large language models and statistical techniques. It clusters the text representations to carefully construct meaningful groups and then apply prompting techniques to automatically generate Minimal Functionality Tests (MFT). The well-known Amazon Reviews corpus is used to demonstrate our approach. We analyze the behavioral test profiles across four different classification algorithms and discuss the limitations and strengths of those models.
• 摘要：自然语言处理 (NLP) 模型（例如 Checklist）的行为测试的最新工作受到软件工程测试中相关范例的启发。它们允许评估一般语言能力和领域理解，因此可以帮助评估概念健全性并识别模型弱点。然而，一个主要的挑战是测试用例的创建。当前的软件包依赖于使用手动开发的半自动化方法，这需要领域专业知识并且可能很耗时。本文介绍了一种通过利用大型语言模型和统计技术的强大功能来开发测试用例的自动化方法。它对文本表示进行聚类以精心构建有意义的组，然后应用提示技术自动生成最小功能测试 (MFT)。著名的 Amazon Reviews 语料库用于演示我们的方法。我们分析了四种不同分类算法的行为测试配置文件，并讨论了这些模型的局限性和优势。

Title: Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms

• Authors: Tian Meng, Yang Tao, Wuliang Yin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00244
• Pdf URL: https://arxiv.org/pdf/2408.00244
• Copy Paste: [[2408.00244]] Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms(https://arxiv.org/abs/2408.00244)
• Keywords: language model
• Abstract: Structured State Space Models (SSMs) have emerged as compelling alternatives to Transformer architectures, offering linear-time complexity and superior performance in various sequence modeling tasks. Despite their advantages, SSMs like the original Mamba-2 face training difficulties due to the sensitivities introduced by the extended series of recurrent matrix multiplications. In this paper, we propose an advanced architecture that mitigates these challenges by decomposing A-multiplications into multiple groups and optimizing positional encoding through Grouped Finite Impulse Response (FIR) filtering. This new structure, denoted as Grouped FIR-enhanced SSM (GFSSM), employs semiseparable matrices for efficient computation. Furthermore, inspired by the "attention sink" phenomenon identified in streaming language models, we incorporate a similar mechanism to enhance the stability and performance of our model over extended sequences. Our approach further bridges the gap between SSMs and Transformer architectures, offering a viable path forward for scalable and high-performing sequence modeling.
• 摘要：结构化状态空间模型 (SSM) 已成为 Transformer 架构的有力替代品，在各种序列建模任务中提供线性时间复杂度和卓越性能。尽管 SSM 具有优势，但由于扩展的一系列递归矩阵乘法引入的敏感性，像原始的 Mamba-2 这样的 SSM 面临着训练困难。在本文中，我们提出了一种先进的架构，通过将 A 乘法分解为多个组并通过分组有限脉冲响应 (FIR) 滤波优化位置编码来缓解这些挑战。这种新结构称为分组 FIR 增强型 SSM (GFSSM)，采用半可分矩阵进行高效计算。此外，受流式语言模型中发现的“注意力下沉”现象的启发，我们采用了类似的机制来增强我们的模型在扩展序列上的稳定性和性能。我们的方法进一步弥合了 SSM 和 Transformer 架构之间的差距，为可扩展和高性能序列建模提供了一条可行的前进道路。

Title: Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding

• Authors: Bin Xiao, Lujun Gui, Lei Su, Weipeng Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00264
• Pdf URL: https://arxiv.org/pdf/2408.00264
• Copy Paste: [[2408.00264]] Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding(https://arxiv.org/abs/2408.00264)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.
• 摘要：大型语言模型 (LLM) 经常出现效率低下的问题，这主要归因于自回归解码的要求与当代 GPU 的架构之间的不一致。最近，回归轻量级推测解码因其在文本生成任务中的显著效率改进而备受关注。这种方法利用轻量级回归草稿模型，如循环神经网络 (RNN) 或单个转换器解码器层，利用顺序信息迭代预测潜在标记。具体而言，RNN 草稿模型在计算上经济实惠，但往往准确度较低，而注意解码器层模型则表现出相反的特征。本文介绍了 Clover-2，这是 Clover 的高级迭代，Clover 是一种基于 RNN 的草稿模型，旨在实现与注意解码器层模型相当的准确度，同时保持最小的计算开销。Clover-2 增强了模型架构并结合了知识提炼，以提高 Clover 的准确性并提高整体效率。我们使用开源 Vicuna 7B 和 LLaMA3-Instruct 8B 模型进行了实验。结果表明，Clover-2 在各种模型架构中超越了现有的方法，展示了其有效性和稳健性。

Title: QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression

• Authors: Wenshan Wang, Yihang Wang, Yixing Fan, Huaming Liao, Jiafeng Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00274
• Pdf URL: https://arxiv.org/pdf/2408.00274
• Copy Paste: [[2408.00274]] QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression(https://arxiv.org/abs/2408.00274)
• Keywords: language model, llm
• Abstract: In-context learning (ICL) capabilities are foundational to the success of large language models (LLMs). Recently, context compression has attracted growing interest since it can largely reduce reasoning complexities and computation costs of LLMs. In this paper, we introduce a novel Query-gUIded aTtention cOmpression (QUITO) method, which leverages attention of the question over the contexts to filter useless information. Specifically, we take a trigger token to calculate the attention distribution of the context in response to the question. Based on the distribution, we propose three different filtering methods to satisfy the budget constraints of the context length. We evaluate the QUITO using two widely-used datasets, namely, NaturalQuestions and ASQA. Experimental results demonstrate that QUITO significantly outperforms established baselines across various datasets and downstream LLMs, underscoring its effectiveness. Our code is available at this https URL.
• 摘要：上下文学习 (ICL) 功能是大型语言模型 (LLM) 成功的基础。最近，上下文压缩引起了越来越多的关注，因为它可以大大降低 LLM 的推理复杂性和计算成本。在本文中，我们介绍了一种新颖的查询引导注意力压缩 (QUITO) 方法，该方法利用问题对上下文的注意力来过滤无用信息。具体来说，我们采用触发标记来计算上下文对问题的注意力分布。基于分布，我们提出了三种不同的过滤方法来满足上下文长度的预算约束。我们使用两个广泛使用的数据集 NaturalQuestions 和 ASQA 来评估 QUITO。实验结果表明，QUITO 在各种数据集和下游 LLM 中的表现明显优于既定基线，凸显了其有效性。我们的代码可在此 https URL 上找到。

Title: DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model

• Authors: Nan Xie, Yuelin Bai, Hengyuan Gao, Feiteng Fang, Qixuan Zhao, Zhijian Li, Ziqiang Xue, Liang Zhu, Shiwen Ni, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00357
• Pdf URL: https://arxiv.org/pdf/2408.00357
• Copy Paste: [[2408.00357]] DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model(https://arxiv.org/abs/2408.00357)
• Keywords: language model, llm, hallucination
• Abstract: Traditional legal retrieval systems designed to retrieve legal documents, statutes, precedents, and other legal information are unable to give satisfactory answers due to lack of semantic understanding of specific questions. Large Language Models (LLMs) have achieved excellent results in a variety of natural language processing tasks, which inspired us that we train a LLM in the legal domain to help legal retrieval. However, in the Chinese legal domain, due to the complexity of legal questions and the rigour of legal articles, there is no legal large model with satisfactory practical application yet. In this paper, we present DeliLaw, a Chinese legal counselling system based on a large language model. DeliLaw integrates a legal retrieval module and a case retrieval module to overcome the model hallucination. Users can consult professional legal questions, search for legal articles and relevant judgement cases, etc. on the DeliLaw system in a dialogue mode. In addition, DeliLaw supports the use of English for counseling. we provide the address of the system: this https URL.
• 摘要：传统法律检索系统旨在检索法律文件、法规、判例等法律信息，由于缺乏对具体问题的语义理解，无法给出满意的答案。大型语言模型（LLM）在各种自然语言处理任务中都取得了优异的效果，这启发我们训练一个法律领域的LLM来帮助法律检索。但在中文法律领域，由于法律问题的复杂性和法律文章的严谨性，目前尚无令人满意的实际应用的法律大模型。本文提出了一个基于大型语言模型的中文法律咨询系统DeliLaw。DeliLaw集成了法律检索模块和案例检索模块，克服了模型幻觉。用户可以在DeliLaw系统上以对话模式进行专业法律问题咨询、搜索法律文章和相关裁判案例等。此外，DeliLaw支持使用英语进行咨询。我们提供系统的地址：这个https网址。

Title: In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation

• Authors: Armel Zebaze, Benoît Sagot, Rachel Bawden
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00397
• Pdf URL: https://arxiv.org/pdf/2408.00397
• Copy Paste: [[2408.00397]] In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation(https://arxiv.org/abs/2408.00397)
• Keywords: language model, llm, prompt
• Abstract: The ability of generative large language models (LLMs) to perform in-context learning has given rise to a large body of research into how best to prompt models for various natural language processing tasks. In this paper, we focus on machine translation (MT), a task that has been shown to benefit from in-context translation examples. However no systematic studies have been published on how best to select examples, and mixed results have been reported on the usefulness of similarity-based selection over random selection. We provide a study covering multiple LLMs and multiple in-context example retrieval strategies, comparing multilingual sentence embeddings. We cover several language directions, representing different levels of language resourcedness (English into French, German, Swahili and Wolof). Contrarily to previously published results, we find that sentence embedding similarity can improve MT, especially for low-resource language directions, and discuss the balance between selection pool diversity and quality. We also highlight potential problems with the evaluation of LLM-based MT and suggest a more appropriate evaluation protocol, adapting the COMET metric to the evaluation of LLMs. Code and outputs are freely available at this https URL.
• 摘要：生成式大型语言模型 (LLM) 执行上下文学习的能力引发了大量研究，旨在研究如何最好地为各种自然语言处理任务提示模型。在本文中，我们重点关注机器翻译 (MT)，这项任务已被证明可以从上下文翻译示例中受益。然而，尚未发表关于如何最好地选择示例的系统研究，并且关于基于相似性的选择相对于随机选择的有用性，报告的结果不一。我们提供一项涵盖多个 LLM 和多个上下文示例检索策略的研究，比较多语言句子嵌入。我们涵盖了几个语言方向，代表了不同级别的语言资源（英语到法语、德语、斯瓦希里语和沃洛夫语）。与之前发表的结果相反，我们发现句子嵌入相似性可以改善 MT，尤其是对于资源较少的语言方向，并讨论了选择池多样性和质量之间的平衡。我们还强调了基于 LLM 的 MT 评估的潜在问题，并提出了一种更合适的评估协议，将 COMET 指标调整为 LLM 的评估。代码和输出可通过此 https URL 免费获取。

Title: GalleryGPT: Analyzing Paintings with Large Multimodal Models

• Authors: Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, Heng Tao Shen
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.00491
• Pdf URL: https://arxiv.org/pdf/2408.00491
• Copy Paste: [[2408.00491]] GalleryGPT: Analyzing Paintings with Large Multimodal Models(https://arxiv.org/abs/2408.00491)
• Keywords: gpt
• Abstract: Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, i.e., painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. \textcolor{blue}{The codes and model are available at: this https URL.
• 摘要：艺术品分析是艺术欣赏的重要基本技能，可以丰富个人的审美情趣，促进批判性思维能力的培养。由于艺术品的主观性、解读的多样性、复杂的视觉元素，理解艺术品是一项具有挑战性的任务，需要具备艺术史、文化背景和美学理论方面的专业知识。然而，受限于数据收集和模型能力，以往对艺术品的自动分析主要集中在分类、检索等简单任务上，距离人工智能的目标还很远。为了促进研究进展，本文受大型多模态模型卓越的感知和生成能力的启发，更进一步进行综合分析。具体而言，我们首先提出了一项针对艺术品（本文中为绘画）撰写段落分析的任务，只关注视觉特征以形成对艺术品更全面的理解。为了支持形式分析的研究，我们收集了一个大型数据集 PaintingForm，约有 19k 张绘画图像和 50k 个分析段落。我们进一步介绍了一种用于绘画分析创作的卓越大型多模态模型，称为 GalleryGPT，该模型基于 LLaVA 架构进行了少许修改和微调，并利用了我们收集的数据。我们在多个数据集上进行了正式分析生成和零样本实验，以评估我们模型的容量。与强大的基线 LMM 相比，结果显示性能显着提升，展示了其出色的艺术分析和泛化能力。 \textcolor{blue}{代码和模型可在以下网址获得：此 https URL。

Title: Intermittent Semi-working Mask: A New Masking Paradigm for LLMs

• Authors: Mingcong Lu, Jiangcai Zhu, Wang Hao, Zheng Li, Shusheng Zhang, Kailai Shao, Chao Chen, Nan Li, Feng Wang, Xin Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00539
• Pdf URL: https://arxiv.org/pdf/2408.00539
• Copy Paste: [[2408.00539]] Intermittent Semi-working Mask: A New Masking Paradigm for LLMs(https://arxiv.org/abs/2408.00539)
• Keywords: language model, llm
• Abstract: Multi-turn dialogues are a key interaction method between humans and Large Language Models (LLMs), as conversations extend over multiple rounds, keeping LLMs' high generation quality and low latency is a challenge. Mainstream LLMs can be grouped into two categories based on masking strategy: causal LLM and prefix LLM. Several works have demonstrated that prefix LLMs tend to outperform causal ones in scenarios that heavily depend on historical context such as multi-turn dialogues or in-context learning, thanks to their bidirectional attention on prefix sequences. However, prefix LLMs have an inherent inefficient training problem in multi-turn dialogue datasets. In addition, the attention mechanism of prefix LLM makes it unable to reuse Key-Value Cache (KV Cache) across dialogue rounds to reduce generation latency. In this paper, we propose a novel masking scheme called Intermittent Semi-working Mask (ISM) to address these problems. Specifically, we apply alternate bidirectional and unidirectional attention on queries and answers in the dialogue history. In this way, ISM is able to maintain the high quality of prefix LLM and low generation latency of causal LLM, simultaneously. Extensive experiments illustrate that our ISM achieves significant performance.
• 摘要：多轮对话是人机与大型语言模型 (LLM) 之间的关键交互方式，由于对话跨越多轮，保持 LLM 的高生成质量和低延迟是一项挑战。主流 LLM 可以根据掩蔽策略分为两类：因果 LLM 和前缀 LLM。一些研究表明，由于前缀序列的双向注意力，前缀 LLM 在严重依赖历史上下文的场景（例如多轮对话或上下文学习）中往往优于因果 LLM。然而，前缀 LLM 在多轮对话数据集中存在固有的训练效率低下的问题。此外，前缀 LLM 的注意力机制使其无法在对话轮次之间重用键值缓存 (KV 缓存) 以减少生成延迟。在本文中，我们提出了一种称为间歇性半工作掩蔽 (ISM) 的新型掩蔽方案来解决这些问题。具体而言，我们对对话历史中的查询和答案交替应用双向和单向注意力。这样，ISM 能够同时保持前缀 LLM 的高质量和因果 LLM 的低生成延迟。大量实验表明，我们的 ISM 取得了显著的性能。

Title: Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses

• Authors: Gabriele Sarti, Tommaso Caselli, Malvina Nissim, Arianna Bisazza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00584
• Pdf URL: https://arxiv.org/pdf/2408.00584
• Copy Paste: [[2408.00584]] Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses(https://arxiv.org/abs/2408.00584)
• Keywords: language model, gpt
• Abstract: Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills.
• 摘要：字谜是一种需要进行约束性多步推理才能从一组图像和字母中识别出隐藏短语的谜题。在这项研究中，我们引入了大量意大利语的口头字谜，并用它来评估最先进的大型语言模型的字谜解决能力。虽然 LLaMA-3 和 GPT-4o 等通用系统在这项任务上表现不佳，但临时微调似乎可以提高模型的性能。然而，我们发现训练带来的性能提升主要是由记忆驱动的。我们的结果表明，字谜解决仍然是评估大型语言模型的语言能力和顺序指令遵循技能的具有挑战性的试验台。

Title: Closing the gap between open-source and commercial large language models for medical evidence summarization

• Authors: Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina R. Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion, Zhiyong Lu, Chunhua Weng, Yifan Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00588
• Pdf URL: https://arxiv.org/pdf/2408.00588
• Copy Paste: [[2408.00588]] Closing the gap between open-source and commercial large language models for medical evidence summarization(https://arxiv.org/abs/2408.00588)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance in summarizing medical evidence. Utilizing a benchmark dataset, MedReview, consisting of 8,161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the fine-tuned LLMs obtained an increase of 9.89 in ROUGE-L (95% confidence interval: 8.94-10.81), 13.21 in METEOR score (95% confidence interval: 12.05-14.37), and 15.82 in CHRF score (95% confidence interval: 13.89-16.44). The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were also manifested in both human and GPT4-simulated evaluations. Our results can be applied to guide model selection for tasks demanding particular domain knowledge, such as medical evidence summarization.
• 摘要：大型语言模型 (LLM) 在总结医学证据方面大有可为。最近的研究主要关注专有 LLM 的应用。使用专有 LLM 会带来多种风险因素，包括缺乏透明度和依赖供应商。虽然开源 LLM 具有更好的透明度和定制性，但其性能不如专有 LLM。在本研究中，我们调查了微调开源 LLM 可以在多大程度上进一步提高其总结医学证据的性能。利用由 8,161 对系统评价和摘要组成的基准数据集 MedReview，我们对三个广泛使用的开源 LLM（即 PRIMERA、LongT5 和 Llama-2）进行了微调。总体而言，经过微调的 LLM 在 ROUGE-L 中获得了 9.89 的提升（95% 置信区间：8.94-10.81），在 METEOR 得分中获得了 13.21 的提升（95% 置信区间：12.05-14.37），在 CHRF 得分中获得了 15.82 的提升（95% 置信区间：13.89-16.44）。经过微调的 LongT5 的性能接近零样本设置的 GPT-3.5。此外，较小的微调模型有时甚至比较大的零样本模型表现出更好的性能。上述改进趋势也体现在人类和 GPT4 模拟评估中。我们的结果可用于指导需要特定领域知识的任务（例如医学证据摘要）的模型选择。

Title: Downstream bias mitigation is all you need

• Authors: Arkadeep Baksi, Rahul Singh, Tarun Joshi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00612
• Pdf URL: https://arxiv.org/pdf/2408.00612
• Copy Paste: [[2408.00612]] Downstream bias mitigation is all you need(https://arxiv.org/abs/2408.00612)
• Keywords: language model, llm
• Abstract: The advent of transformer-based architectures and large language models (LLMs) have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and other sources, there has been a major concern about harmful prejudices that may potentially be transferred from the data. In many applications, these pre-trained LLMs are fine-tuned on task specific datasets, which can further contribute to biases. This paper studies the extent of biases absorbed by LLMs during pre-training as well as task-specific behaviour after fine-tuning. We found that controlled interventions on pre-trained LLMs, prior to fine-tuning, have minimal effect on lowering biases in classifiers. However, the biases present in domain-specific datasets play a much bigger role, and hence mitigating them at this stage has a bigger impact. While pre-training does matter, but after the model has been pre-trained, even slight changes to co-occurrence rates in the fine-tuning dataset has a significant effect on the bias of the model.
• 摘要：基于 Transformer 的架构和大型语言模型 (LLM) 的出现显著提高了自然语言处理 (NLP) 模型的性能。由于这些 LLM 是在来自网络和其他来源的大量数据上训练的，因此人们非常担心可能从数据中转移出来的有害偏见。在许多应用中，这些预训练的 LLM 会在特定于任务的数据集上进行微调，这可能会进一步导致偏见。本文研究了 LLM 在预训练期间吸收偏见的程度以及微调后的任务特定行为。我们发现，在微调之前对预训练的 LLM 进行控制干预对降低分类器的偏见效果微乎其微。然而，领域特定数据集中存在的偏见起着更大的作用，因此在此阶段减轻它们的影响更大。虽然预训练确实很重要，但是在模型预训练之后，微调数据集中共现率的微小变化也会对模型的偏差产生显著影响。

Title: Leveraging Entailment Judgements in Cross-Lingual Summarisation

• Authors: Huajian Zhang, Laura Perez-Beltrachini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00675
• Pdf URL: https://arxiv.org/pdf/2408.00675
• Copy Paste: [[2408.00675]] Leveraging Entailment Judgements in Cross-Lingual Summarisation(https://arxiv.org/abs/2408.00675)
• Keywords: hallucination
• Abstract: Synthetically created Cross-Lingual Summarisation (CLS) datasets are prone to include document-summary pairs where the reference summary is unfaithful to the corresponding document as it contains content not supported by the document (i.e., hallucinated content). This low data quality misleads model learning and obscures evaluation results. Automatic ways to assess hallucinations and improve training have been proposed for monolingual summarisation, predominantly in English. For CLS, we propose to use off-the-shelf cross-lingual Natural Language Inference (X-NLI) to evaluate faithfulness of reference and model generated summaries. Then, we study training approaches that are aware of faithfulness issues in the training data and propose an approach that uses unlikelihood loss to teach a model about unfaithful summary sequences. Our results show that it is possible to train CLS models that yield more faithful summaries while maintaining comparable or better informativess.
• 摘要：人工创建的跨语言摘要 (CLS) 数据集容易包含文档摘要对，其中参考摘要与相应文档不符，因为它包含文档不支持的内容（即幻觉内容）。这种低数据质量会误导模型学习并掩盖评估结果。已经提出了用于单语摘要的自动评估幻觉和改进训练的方法，主要针对英语。对于 CLS，我们建议使用现成的跨语言自然语言推理 (X-NLI) 来评估参考和模型生成的摘要的忠实度。然后，我们研究意识到训练数据中的忠实度问题的训练方法，并提出一种使用不可能损失来教模型关于不忠实摘要序列的方法。我们的结果表明，可以训练 CLS 模型以产生更忠实的摘要，同时保持可比或更好的信息量。

Title: Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning

• Authors: Trapoom Ukarapol, Zhicheng Lee, Amy Xin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00690
• Pdf URL: https://arxiv.org/pdf/2408.00690
• Copy Paste: [[2408.00690]] Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning(https://arxiv.org/abs/2408.00690)
• Keywords: language model
• Abstract: While Large Language Models show remarkable performance in natural language understanding, their resource-intensive nature makes them less accessible. In contrast, smaller language models such as MiniCPM offer more sustainable scalability, but often underperform without specialized optimization. In this paper, we explore the enhancement of smaller language models through the improvement of their text embeddings. We select three language models, MiniCPM, Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our results demonstrate that this fine-tuning method enhances the quality of text embeddings for all three models across various benchmarks, with MiniCPM showing the most significant improvements of an average 56.33\% performance gain. The contrastive fine-tuning code is publicly available at this https URL.
• 摘要：虽然大型语言模型在自然语言理解方面表现出色，但其资源密集型特性使其不易获得。相比之下，小型语言模型（如 MiniCPM）提供了更可持续的可扩展性，但如果没有专门的优化，通常表现不佳。在本文中，我们探索通过改进文本嵌入来增强小型语言模型的性能。我们选择了三种语言模型，MiniCPM、Phi-2 和 Gemma，对 NLI 数据集进行对比微调。我们的结果表明，这种微调方法提高了所有三个模型在各种基准测试中的文本嵌入质量，其中 MiniCPM 显示出最显著的改进，平均性能提高了 56.33\%。对比微调代码可在此 https URL 上公开获取。

Title: Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions

• Authors: Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00727
• Pdf URL: https://arxiv.org/pdf/2408.00727
• Copy Paste: [[2408.00727]] Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions(https://arxiv.org/abs/2408.00727)
• Keywords: language model, gpt, llm, prompt, retrieval-augmented generation
• Abstract: The emergent abilities of large language models (LLMs) have demonstrated great potential in solving medical questions. They can possess considerable medical knowledge, but may still hallucinate and are inflexible in the knowledge updates. While Retrieval-Augmented Generation (RAG) has been proposed to enhance the medical question-answering capabilities of LLMs with external knowledge bases, it may still fail in complex cases where multiple rounds of information-seeking are required. To address such an issue, we propose iterative RAG for medicine (i-MedRAG), where LLMs can iteratively ask follow-up queries based on previous information-seeking attempts. In each iteration of i-MedRAG, the follow-up queries will be answered by a vanilla RAG system and they will be further used to guide the query generation in the next iteration. Our experiments show the improved performance of various LLMs brought by i-MedRAG compared with vanilla RAG on complex questions from clinical vignettes in the United States Medical Licensing Examination (USMLE), as well as various knowledge tests in the Massive Multitask Language Understanding (MMLU) dataset. Notably, our zero-shot i-MedRAG outperforms all existing prompt engineering and fine-tuning methods on GPT-3.5, achieving an accuracy of 69.68\% on the MedQA dataset. In addition, we characterize the scaling properties of i-MedRAG with different iterations of follow-up queries and different numbers of queries per iteration. Our case studies show that i-MedRAG can flexibly ask follow-up queries to form reasoning chains, providing an in-depth analysis of medical questions. To the best of our knowledge, this is the first-of-its-kind study on incorporating follow-up queries into medical RAG.
• 摘要：大型语言模型 (LLM) 的新兴能力在解决医学问题方面已显示出巨大的潜力。它们可以拥有大量医学知识，但仍可能产生幻觉，并且在知识更新方面缺乏灵活性。虽然已经提出了检索增强生成 (RAG) 来增强具有外部知识库的 LLM 的医学问答能力，但在需要多轮信息搜索的复杂情况下，它仍可能失败。为了解决这个问题，我们提出了用于医学的迭代 RAG (i-MedRAG)，其中 LLM 可以根据先前的信息搜索尝试迭代地提出后续查询。在 i-MedRAG 的每次迭代中，后续查询将由原始 RAG 系统回答，并且它们将进一步用于指导下一次迭代中的查询生成。我们的实验表明，与 vanilla RAG 相比，i-MedRAG 为各种 LLM 带来了更高的性能，这些性能体现在美国医师执照考试 (USMLE) 临床案例中的复杂问题以及大规模多任务语言理解 (MMLU) 数据集中的各种知识测试上。值得注意的是，我们的零样本 i-MedRAG 优于 GPT-3.5 上所有现有的即时工程和微调方法，在 MedQA 数据集上的准确率达到 69.68%。此外，我们通过不同的后续查询迭代和每次迭代的不同查询数来表征 i-MedRAG 的扩展属性。我们的案例研究表明，i-MedRAG 可以灵活地提出后续查询以形成推理链，从而对医学问题进行深入分析。据我们所知，这是首次将后续查询纳入医学 RAG 的研究。

Title: AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation

• Authors: Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, Dongmei Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00764
• Pdf URL: https://arxiv.org/pdf/2408.00764
• Copy Paste: [[2408.00764]] AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation(https://arxiv.org/abs/2408.00764)
• Keywords: language model, gpt, llm, agent
• Abstract: Large Language Model (LLM) based agents have garnered significant attention and are becoming increasingly popular. Furthermore, planning ability is a crucial component of an LLM-based agent, involving interaction with the environment and executing actions to complete a planning task, which generally entails achieving a desired goal from an initial state. This paper investigates enhancing the planning abilities of LLMs through instruction tuning, referred to as agent training. Recent studies have demonstrated that utilizing expert-level trajectory for instruction-tuning LLMs effectively enhances their planning capabilities. However, existing work primarily focuses on synthesizing trajectories from manually designed planning tasks and environments. The labor-intensive nature of creating these environments and tasks impedes the generation of sufficiently varied and extensive trajectories. To address this limitation, this paper explores the automated synthesis of diverse environments and a gradual range of planning tasks, from easy to difficult. We introduce a framework, AgentGen, that leverages LLMs first to generate environments and subsequently generate planning tasks conditioned on these environments. Specifically, to improve environmental diversity, we propose using an inspiration corpus composed of various domain-specific text segments as the context for synthesizing environments. Moreover, to increase the difficulty diversity of generated planning tasks, we propose a bidirectional evolution method, Bi-Evol, that evolves planning tasks from easier and harder directions to synthesize a task set with a smoother difficulty curve. The evaluation results derived from AgentBoard show that AgentGen greatly improves LLMs' planning ability, e.g., the AgentGen instruction-tuned Llama-3 8B surpasses GPT-3.5 in overall performance. Moreover, in certain tasks, it even outperforms GPT-4.
• 摘要：基于大型语言模型 (LLM) 的代理已引起广泛关注，并且越来越受欢迎。此外，规划能力是基于 LLM 的代理的一个关键组成部分，涉及与环境的交互以及执行操作以完成规划任务，这通常需要从初始状态实现期望的目标。本文研究通过指令调整（称为代理训练）来增强 LLM 的规划能力。最近的研究表明，利用专家级轨迹进行指令调整 LLM 可有效增强其规划能力。然而，现有的工作主要侧重于从手动设计的规划任务和环境中合成轨迹。创建这些环境和任务的劳动密集型性质阻碍了足够多样化和广泛的轨迹的生成。为了解决这一限制，本文探讨了从易到难的多样化环境和逐步规划任务的自动合成。我们引入了一个框架 AgentGen，它首先利用 LLM 生成环境，然后生成以这些环境为条件的规划任务。具体来说，为了提高环境多样性，我们提出使用由各种领域特定文本片段组成的灵感语料库作为合成环境的上下文。此外，为了增加生成的规划任务的难度多样性，我们提出了一种双向进化方法 Bi-Evol，该方法从更容易和更难的方向进化规划任务，以合成具有更平滑难度曲线的任务集。来自 AgentBoard 的评估结果表明 AgentGen 大大提高了 LLM 的规划能力，例如，经过 AgentGen 指令调整的 Llama-3 8B 在整体性能上超越了 GPT-3.5。此外，在某些任务中，它甚至优于 GPT-4。