2024-08-06

Title: MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts

• Authors: Lin Ning, Harsh Lara, Meiqi Guo, Abhinav Rastogi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01505
• Pdf URL: https://arxiv.org/pdf/2408.01505
• Copy Paste: [[2408.01505]] MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts(https://arxiv.org/abs/2408.01505)
• Keywords: language model, llm
• Abstract: Parameter-efficient fine-tuning techniques like Low-Rank Adaptation (LoRA) have revolutionized the adaptation of large language models (LLMs) to diverse tasks. Recent efforts have explored mixtures of LoRA modules for multi-task settings. However, our analysis reveals redundancy in the down-projection matrices of these architectures. This observation motivates our proposed method, Mixture of Dyadic Experts (MoDE), which introduces a novel design for efficient multi-task adaptation. This is done by sharing the down-projection matrix across tasks and employing atomic rank-one adapters, coupled with routers that allow more sophisticated task-level specialization. Our design allows for more fine-grained mixing, thereby increasing the model's ability to jointly handle multiple tasks. We evaluate MoDE on the Supernatural Instructions (SNI) benchmark consisting of a diverse set of 700+ tasks and demonstrate that it outperforms state-of-the-art multi-task parameter-efficient fine-tuning (PEFT) methods, without introducing additional parameters. Our findings contribute to a deeper understanding of parameter efficiency in multi-task LLM adaptation and provide a practical solution for deploying high-performing, lightweight models.
• 摘要：低秩自适应 (LoRA) 等参数高效微调技术彻底改变了大型语言模型 (LLM) 对各种任务的适应性。最近的努力探索了用于多任务设置的 LoRA 模块混合。然而，我们的分析揭示了这些架构的下投影矩阵中的冗余。这一观察促使我们提出了混合二元专家 (MoDE) 方法，该方法引入了一种用于高效多任务适应的新颖设计。这是通过在任务之间共享下投影矩阵并使用原子秩一适配器以及允许更复杂的任务级专业化的路由器来实现的。我们的设计允许更细粒度的混合，从而提高了模型联合处理多个任务的能力。我们在由 700 多个任务组成的超自然指令 (SNI) 基准上评估了 MoDE，并证明它优于最先进的多任务参数高效微调 (PEFT) 方法，而无需引入额外的参数。我们的研究有助于更深入地理解多任务 LLM 适应中的参数效率，并为部署高性能、轻量级模型提供实用的解决方案。

Title: Analyzing LLMs' Capabilities to Establish Implicit User Sentiment of Software Desirability

• Authors: Sherri Weitl-Harms, John D. Hastings, Jonah Lum
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.01527
• Pdf URL: https://arxiv.org/pdf/2408.01527
• Copy Paste: [[2408.01527]] Analyzing LLMs' Capabilities to Establish Implicit User Sentiment of Software Desirability(https://arxiv.org/abs/2408.01527)
• Keywords: gpt, llm
• Abstract: This study explores the use of several LLMs for providing quantitative zero-shot sentiment analysis of implicit software desirability expressed by users. The study provides scaled numerical sentiment analysis unlike other methods that simply classify sentiment as positive, neutral, or negative. Numerical analysis provides deeper insights into the magnitude of sentiment, to drive better decisions regarding product desirability. Data is collected through the use of the Microsoft Product Desirability Toolkit (PDT), a well-known qualitative user experience analysis tool. For initial exploration, the PDT metric was given to users of ZORQ, a gamification system used in undergraduate computer science education. The PDT data collected was fed through several LLMs (Claude Sonnet 3 and 3.5, GPT4, and GPT4o) and through a leading transfer learning technique, Twitter-Roberta-Base-Sentiment (TRBS), and through Vader, a leading sentiment analysis tool, for quantitative sentiment analysis. Each system was asked to evaluate the data in two ways, first by looking at the sentiment expressed in the PDT word/explanation pairs; and by looking at the sentiment expressed by the users in their grouped selection of five words and explanations, as a whole. Each LLM was also asked to provide its confidence (low, medium, high) in its sentiment score, along with an explanation of why it selected the sentiment value. All LLMs tested were able to statistically detect user sentiment from the users' grouped data, whereas TRBS and Vader were not. The confidence and explanation of confidence provided by the LLMs assisted in understanding the user sentiment. This study adds to a deeper understanding of evaluating user experiences, toward the goal of creating a universal tool that quantifies implicit sentiment expressed.
• 摘要：本研究探讨了如何使用多个 LLM 对用户表达的隐性软件可取性进行定量的零样本情绪分析。本研究提供了缩放的数值情绪分析，这与其他简单地将情绪分为积极、中性或消极的方法不同。数值分析可以更深入地洞察情绪的幅度，从而推动有关产品可取性的更好决策。数据是通过使用著名的定性用户体验分析工具 Microsoft 产品可取性工具包 (PDT) 收集的。对于初步探索，PDT 指标提供给 ZORQ 的用户，ZORQ 是本科计算机科学教育中使用的游戏化系统。收集的 PDT 数据通过多个 LLM（Claude Sonnet 3 和 3.5、GPT4 和 GPT4o）和领先的迁移学习技术 Twitter-Roberta-Base-Sentiment (TRBS) 以及领先的情绪分析工具 Vader 进行定量情绪分析。要求每个系统以两种方式评估数据，首先查看 PDT 单词/解释对中表达的情绪；并通过查看用户在五个单词和解释的分组选择中表达的情绪，作为一个整体。还要求每个 LLM 提供其情绪分数的置信度（低、中、高），以及选择情绪值的原因的解释。所有测试的 LLM 都能够从用户的分组数据中统计出用户情绪，而 TRBS 和 Vader 则不能。LLM 提供的信心和信心解释有助于理解用户情绪。这项研究加深了对评估用户体验的理解，朝着创建量化表达的隐性情绪的通用工具的目标迈进。

Title: Dialog Flow Induction for Constrainable LLM-Based Chatbots

• Authors: Stuti Agrawal, Nishi Uppuluri, Pranav Pillai, Revanth Gangi Reddy, Zoey Li, Gokhan Tur, Dilek Hakkani-Tur, Heng Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01623
• Pdf URL: https://arxiv.org/pdf/2408.01623
• Copy Paste: [[2408.01623]] Dialog Flow Induction for Constrainable LLM-Based Chatbots(https://arxiv.org/abs/2408.01623)
• Keywords: llm, chat
• Abstract: LLM-driven dialog systems are used in a diverse set of applications, ranging from healthcare to customer service. However, given their generalization capability, it is difficult to ensure that these chatbots stay within the boundaries of the specialized domains, potentially resulting in inaccurate information and irrelevant responses. This paper introduces an unsupervised approach for automatically inducing domain-specific dialog flows that can be used to constrain LLM-based chatbots. We introduce two variants of dialog flow based on the availability of in-domain conversation instances. Through human and automatic evaluation over various dialog domains, we demonstrate that our high-quality data-guided dialog flows achieve better domain coverage, thereby overcoming the need for extensive manual crafting of such flows.
• 摘要：LLM 驱动的对话系统用于从医疗保健到客户服务等各种应用。然而，鉴于它们的泛化能力，很难确保这些聊天机器人停留在专门领域的边界内，这可能会导致信息不准确和响应不相关。本文介绍了一种无监督方法，用于自动诱导领域特定对话流，可用于约束基于 LLM 的聊天机器人。我们根据领域内对话实例的可用性引入​​了两种对话流变体。通过对各种对话领域的人工和自动评估，我们证明了我们的高质量数据引导对话流实现了更好的领域覆盖，从而克服了对此类流程进行大量手动制作的需求。

Title: Multi-Frame Vision-Language Model for Long-form Reasoning in Driver Behavior Analysis

• Authors: Hiroshi Takato, Hiroshi Tsutsui, Komei Soda, Hidetaka Kamigaito
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01682
• Pdf URL: https://arxiv.org/pdf/2408.01682
• Copy Paste: [[2408.01682]] Multi-Frame Vision-Language Model for Long-form Reasoning in Driver Behavior Analysis(https://arxiv.org/abs/2408.01682)
• Keywords: language model
• Abstract: Identifying risky driving behavior in real-world situations is essential for the safety of both drivers and pedestrians. However, integrating natural language models in this field remains relatively untapped. To address this, we created a novel multi-modal instruction tuning dataset and driver coaching inference system. Our primary use case is dashcam-based coaching for commercial drivers. The North American Dashcam Market is expected to register a CAGR of 15.4 percent from 2022 to 2027. Our dataset enables language models to learn visual instructions across various risky driving scenarios, emphasizing detailed reasoning crucial for effective driver coaching and managerial comprehension. Our model is trained on road-facing and driver-facing RGB camera footage, capturing the comprehensive scope of driving behavior in vehicles equipped with dashcams.
• 摘要：在现实情况下识别危险驾驶行为对于驾驶员和行人的安全至关重要。然而，将自然语言模型集成到这一领域仍相对尚未开发。为了解决这个问题，我们创建了一个新颖的多模式指令调整数据集和驾驶员指导推理系统。我们的主要用例是基于行车记录仪的商业驾驶员指导。北美行车记录仪市场预计在 2022 年至 2027 年期间的复合年增长率为 15.4%。我们的数据集使语言模型能够在各种危险驾驶场景中学习视觉指令，强调对有效驾驶员指导和管理理解至关重要的详细推理。我们的模型是在面向道路和面向驾驶员的 RGB 摄像头镜头上进行训练的，可以捕捉配备行车记录仪的车辆的驾驶行为的全面范围。

Title: MathLearner: A Large Language Model Agent Framework for Learning to Solve Mathematical Problems

• Authors: Wenbei Xie, Donglin Liu, Haoran Yan, Wenjie Wu, Zongyang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01779
• Pdf URL: https://arxiv.org/pdf/2408.01779
• Copy Paste: [[2408.01779]] MathLearner: A Large Language Model Agent Framework for Learning to Solve Mathematical Problems(https://arxiv.org/abs/2408.01779)
• Keywords: language model, llm, chain-of-thought, agent
• Abstract: With the development of artificial intelligence (AI), large language models (LLM) are widely used in many fields. However, the reasoning ability of LLM is still very limited when it comes to mathematical reasoning. Mathematics plays an important role in all aspects of human society and is a technical guarantee in the fields of healthcare, transport and aerospace, for this reason, the development of AI big language models in the field of mathematics has great potential significance. To improve the mathematical reasoning ability of large language models, we proposed an agent framework for learning to solve mathematical problems based on inductive reasoning. By emulating the human learning process of generalization of learned information and effective application of previous knowledge in new reasoning tasks, this framework has great performance in the mathematical reasoning process. It improves global accuracy over the baseline method (chain-of-thought) by 20.96% and solves 17.54% of the mathematical problems that the baseline cannot solve. Benefiting from the efficient RETRIEVAL method, our model improves the ability of large language models to efficiently use external knowledge, i.e., the mathematical computation of the model can be based on written procedures. In education, our model can be used as a personalised learning aid, thus reducing the inequality of educational resources.
• 摘要：随着人工智能的发展，大型语言模型（LLM）被广泛应用于许多领域。然而，当涉及到数学推理时，LLM的推理能力仍然非常有限。数学在人类社会的各个方面都发挥着重要作用，是医疗、交通、航空航天等领域的技术保障，因此，在数学领域开发人工智能大语言模型具有巨大的潜在意义。为了提高大型语言模型的数学推理能力，我们提出了一种基于归纳推理的学习解决数学问题的代理框架。通过模拟人类学习信息泛化和在新的推理任务中有效应用先前知识的学习过程，该框架在数学推理过程中表现出色。它比基线方法（思路链）提高了20.96%的全局准确率，并解决了17.54%的基线无法解决的数学问题。得益于高效的检索方法，我们的模型提高了大型语言模型高效利用外部知识的能力，即模型的数学计算可以基于书面程序。在教育领域，我们的模型可以作为个性化学习辅助手段，从而减少教育资源的不平等。

Title: Tracking Emotional Dynamics in Chat Conversations: A Hybrid Approach using DistilBERT and Emoji Sentiment Analysis

• Authors: Ayan Igali, Abdulkhak Abdrakhman, Yerdaut Torekhan, Pakizar Shamoi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01838
• Pdf URL: https://arxiv.org/pdf/2408.01838
• Copy Paste: [[2408.01838]] Tracking Emotional Dynamics in Chat Conversations: A Hybrid Approach using DistilBERT and Emoji Sentiment Analysis(https://arxiv.org/abs/2408.01838)
• Keywords: chat
• Abstract: Computer-mediated communication has become more important than face-to-face communication in many contexts. Tracking emotional dynamics in chat conversations can enhance communication, improve services, and support well-being in various contexts. This paper explores a hybrid approach to tracking emotional dynamics in chat conversations by combining DistilBERT-based text emotion detection and emoji sentiment analysis. A Twitter dataset was analyzed using various machine learning algorithms, including SVM, Random Forest, and AdaBoost. We contrasted their performance with DistilBERT. Results reveal DistilBERT's superior performance in emotion recognition. Our approach accounts for emotive expressions conveyed through emojis to better understand participants' emotions during chats. We demonstrate how this approach can effectively capture and analyze emotional shifts in real-time conversations. Our findings show that integrating text and emoji analysis is an effective way of tracking chat emotion, with possible applications in customer service, work chats, and social media interactions.
• 摘要：在许多情况下，计算机中介交流已经变得比面对面交流更重要。在各种情况下，跟踪聊天对话中的情绪动态可以增强沟通、改善服务并支持福祉。本文探讨了一种通过结合基于 DistilBERT 的文本情绪检测和表情符号情绪分析来跟踪聊天对话中情绪动态的混合方法。使用各种机器学习算法（包括 SVM、随机森林和 AdaBoost）分析了 Twitter 数据集。我们将它们的性能与 DistilBERT 进行了对比。结果显示 DistilBERT 在情绪识别方面表现优异。我们的方法考虑了通过表情符号传达的情绪表达，以便更好地理解参与者在聊天过程中的情绪。我们展示了这种方法如何有效捕捉和分析实时对话中的情绪变化。我们的研究结果表明，整合文本和表情符号分析是跟踪聊天情绪的有效方法，可能应用于客户服务、工作聊天和社交媒体互动。

Title: S\'olo Esc\'uchame: Spanish Emotional Accompaniment Chatbot

• Authors: Bruno Gil Ramírez, Jessica López Espejel, María del Carmen Santiago Díaz, Gustavo Trinidad Rubín Linares
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01852
• Pdf URL: https://arxiv.org/pdf/2408.01852
• Copy Paste: [[2408.01852]] S\'olo Esc\'uchame: Spanish Emotional Accompaniment Chatbot(https://arxiv.org/abs/2408.01852)
• Keywords: gpt, chat
• Abstract: According to the World Health Organization (WHO), suicide was the fourth leading cause of death in the world for individuals aged 15 to 29 in 2019. Given the rapid increase in mental health issues, providing psychological support is both crucial and urgent. In this paper: (1) we propose Sólo Escúchame, the first open-source Spanish emotional assistance chatbot, based on LLaMA-2-7b-Chat. (2) We introduced the HEAR (Hispanic Emotional Accompaniment Responses) dataset, compiled from multiple English sources translated into Spanish, as well as generic data generated using ChatGPT-3.5-Turbo. Finally, (3) we propose an evaluation metric based on two semi-automatic assessment methods. Our system outperforms a range of state-of-the-art models in providing psychological assistance in Spanish. Our models and datasets are publicly available to facilitate reproducibility.
• 摘要：根据世界卫生组织 (WHO) 的数据，2019 年自杀是全球 15 至 29 岁人群的第四大死因。鉴于心理健康问题的迅速增加，提供心理支持既至关重要又紧迫。在本文中：（1）我们提出了基于 LLaMA-2-7b-Chat 的首个开源西班牙语情感援助聊天机器人 Sólo Escúchame。（2）我们引入了 HEAR（西班牙裔情感陪伴反应）数据集，该数据集由多个英语翻译成西班牙语的资料汇编而成，以及使用 ChatGPT-3.5-Turbo 生成的通用数据。最后，（3）我们提出了一种基于两种半自动评估方法的评估指标。在用西班牙语提供心理援助方面，我们的系统优于一系列最先进的模型。我们的模型和数据集都是公开的，以便于重现。

Title: Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

• Authors: Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi, Matthew Purver
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01866
• Pdf URL: https://arxiv.org/pdf/2408.01866
• Copy Paste: [[2408.01866]] Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly(https://arxiv.org/abs/2408.01866)
• Keywords: language model, gpt, llm, long context
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs, owing to their extensive context windows that allow processing millions of tokens in a single forward pass. However, this paper uncovers a surprising limitation: LLMs fall short when handling long input sequences. We investigate this issue using three datasets and two tasks (sentiment analysis and news categorization) across various LLMs, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct, and Mistral Instruct models. To address this limitation, we propose and evaluate ad-hoc solutions that substantially enhance LLMs' performance on long input sequences by up to 50%, while reducing API cost and latency by up to 93% and 50%, respectively.
• 摘要：大型语言模型 (LLM) 已展现出理解和分析长序列输入的卓越能力，这归功于其广泛的上下文窗口，允许在一次前向传递中处理数百万个标记。然而，本文揭示了一个令人惊讶的局限性：LLM 在处理长输入序列时会有所不足。我们使用三个数据集和两个任务（情感分析和新闻分类）调查了这个问题，这些任务涉及各种 LLM，包括 Claude 3、Gemini Pro、GPT 3.5 Turbo、Llama 3 Instruct 和 Mistral Instruct 模型。为了解决这一限制，我们提出并评估了临时解决方案，这些解决方案可将 LLM 在长输入序列上的性能大幅提高 50%，同时将 API 成本和延迟分别降低 93% 和 50%。

Title: MALADE: Orchestration of LLM-powered Agents with Retrieval Augmented Generation for Pharmacovigilance

• Authors: Jihye Choi, Nils Palumbo, Prasad Chalasani, Matthew M. Engelhard, Somesh Jha, Anivarya Kumar, David Page
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG, cs.MA, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2408.01869
• Pdf URL: https://arxiv.org/pdf/2408.01869
• Copy Paste: [[2408.01869]] MALADE: Orchestration of LLM-powered Agents with Retrieval Augmented Generation for Pharmacovigilance(https://arxiv.org/abs/2408.01869)
• Keywords: language model, gpt, llm, retrieval augmented generation, agent
• Abstract: In the era of Large Language Models (LLMs), given their remarkable text understanding and generation abilities, there is an unprecedented opportunity to develop new, LLM-based methods for trustworthy medical knowledge synthesis, extraction and summarization. This paper focuses on the problem of Pharmacovigilance (PhV), where the significance and challenges lie in identifying Adverse Drug Events (ADEs) from diverse text sources, such as medical literature, clinical notes, and drug labels. Unfortunately, this task is hindered by factors including variations in the terminologies of drugs and outcomes, and ADE descriptions often being buried in large amounts of narrative text. We present MALADE, the first effective collaborative multi-agent system powered by LLM with Retrieval Augmented Generation for ADE extraction from drug label data. This technique involves augmenting a query to an LLM with relevant information extracted from text resources, and instructing the LLM to compose a response consistent with the augmented data. MALADE is a general LLM-agnostic architecture, and its unique capabilities are: (1) leveraging a variety of external sources, such as medical literature, drug labels, and FDA tools (e.g., OpenFDA drug information API), (2) extracting drug-outcome association in a structured format along with the strength of the association, and (3) providing explanations for established associations. Instantiated with GPT-4 Turbo or GPT-4o, and FDA drug label data, MALADE demonstrates its efficacy with an Area Under ROC Curve of 0.90 against the OMOP Ground Truth table of ADEs. Our implementation leverages the Langroid multi-agent LLM framework and can be found at this https URL.
• 摘要：在大型语言模型 (LLM) 时代，鉴于其出色的文本理解和生成能力，我们迎来了前所未有的机会来开发新的基于 LLM 的可靠医学知识合成、提取和总结方法。本文重点关注药物警戒 (PhV) 问题，其重要性和挑战在于从各种文本来源（如医学文献、临床记录和药品标签）中识别不良药物事件 (ADE)。不幸的是，这项任务受到多种因素的阻碍，包括药物和结果术语的差异，以及 ADE 描述通常被埋在大量叙述文本中。我们介绍了 MALADE，这是第一个有效的协作多智能体系统，由 LLM 提供支持，具有检索增强生成功能，用于从药品标签数据中提取 ADE。该技术涉及使用从文本资源中提取的相关信息增强对 LLM 的查询，并指示 LLM 编写与增强数据一致的响应。 MALADE 是一种通用的 LLM 无关架构，其独特功能包括：(1) 利用各种外部来源，例如医学文献、药品标签和 FDA 工具（例如 OpenFDA 药物信息 API），(2) 以结构化格式提取药物结果关联以及关联强度，以及 (3) 为已建立的关联提供解释。使用 GPT-4 Turbo 或 GPT-4o 和 FDA 药品标签数据实例化后，MALADE 证明了其有效性，其 ROC 曲线下面积为 0.90，与 ADE 的 OMOP 真实值表相当。我们的实现利用了 Langroid 多智能体 LLM 框架，可在此 https URL 中找到。

Title: Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval

• Authors: Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, Mohammadhossein Bateni, Chen-Yu Lee, Tomas Pfister
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01875
• Pdf URL: https://arxiv.org/pdf/2408.01875
• Copy Paste: [[2408.01875]] Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval(https://arxiv.org/abs/2408.01875)
• Keywords: language model, llm, agent
• Abstract: Recent advances in large language models (LLMs) have enabled autonomous agents with complex reasoning and task-fulfillment capabilities using a wide range of tools. However, effectively identifying the most relevant tools for a given task becomes a key bottleneck as the toolset size grows, hindering reliable tool utilization. To address this, we introduce Re-Invoke, an unsupervised tool retrieval method designed to scale effectively to large toolsets without training. Specifically, we first generate a diverse set of synthetic queries that comprehensively cover different aspects of the query space associated with each tool document during the tool indexing phase. Second, we leverage LLM's query understanding capabilities to extract key tool-related context and underlying intents from user queries during the inference phase. Finally, we employ a novel multi-view similarity ranking strategy based on intents to pinpoint the most relevant tools for each query. Our evaluation demonstrates that Re-Invoke significantly outperforms state-of-the-art alternatives in both single-tool and multi-tool scenarios, all within a fully unsupervised setting. Notably, on the ToolE datasets, we achieve a 20% relative improvement in nDCG@5 for single-tool retrieval and a 39% improvement for multi-tool retrieval.
• 摘要：大型语言模型 (LLM) 的最新进展使自主代理能够使用各种工具进行复杂的推理和任务完成。然而，随着工具集规模的增长，有效地识别与给定任务最相关的工具成为一个关键瓶颈，阻碍了可靠的工具利用。为了解决这个问题，我们引入了 Re-Invoke，这是一种无监督的工具检索方法，旨在有效地扩展到大型工具集而无需训练。具体来说，我们首先在工具索引阶段生成一组多样化的合成查询，这些查询全面涵盖与每个工具文档相关的查询空间的不同方面。其次，我们利用 LLM 的查询理解功能在推理阶段从用户查询中提取关键的工具相关上下文和潜在意图。最后，我们采用一种基于意图的新型多视图相似性排名策略来精确定位每个查询的最相关工具。我们的评估表明，在完全无监督的设置中，Re-Invoke 在单工具和多工具场景中的表现都明显优于最先进的替代方案。值得注意的是，在 ToolE 数据集上，我们在单工具检索的 nDCG@5 中实现了 20% 的相对改进，在多工具检索中实现了 39% 的相对改进。

Title: Cross-layer Attention Sharing for Large Language Models

• Authors: Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Qiaozhi He, Murun Yang, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01890
• Pdf URL: https://arxiv.org/pdf/2408.01890
• Copy Paste: [[2408.01890]] Cross-layer Attention Sharing for Large Language Models(https://arxiv.org/abs/2408.01890)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) evolve, the increase in model depth and parameter number leads to substantial redundancy. To enhance the efficiency of the attention mechanism, previous works primarily compress the KV cache or group attention heads, while largely overlooking redundancy between layers. Our comprehensive analyses across various LLMs show that highly similar attention patterns persist within most layers. It's intuitive to save the computation by sharing attention weights across layers. However, further analysis reveals two challenges: (1) Directly sharing the weight matrix without carefully rearranging the attention heads proves to be ineffective; (2) Shallow layers are vulnerable to small deviations in attention weights. Driven by these insights, we introduce LiSA, a lightweight substitute for self-attention in well-trained LLMs. LiSA employs tiny feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. Evaluations encompassing 13 typical benchmarks demonstrate that LiSA maintains high response quality in terms of accuracy and perplexity while reducing redundant attention calculations within 53-84% of the total layers. Our implementations of LiSA achieve a 6X compression of Q and K, with maximum throughput improvements of 19.5% for LLaMA3-8B and 32.3% for LLaMA2-7B.
• 摘要：随着大型语言模型 (LLM) 的发展，模型深度和参数数量的增加导致了大量冗余。为了提高注意力机制的效率，以前的工作主要压缩 KV 缓存或组注意力头，而很大程度上忽略了层之间的冗余。我们对各种 LLM 的全面分析表明，大多数层中都存在高度相似的注意力模式。通过跨层共享注意力权重来节省计算量是直观的。然而，进一步的分析揭示了两个挑战：（1）直接共享权重矩阵而不仔细重新排列注意力头被证明是无效的；（2）浅层容易受到注意力权重的细微偏差的影响。基于这些见解，我们引入了 LiSA，它是经过良好训练的 LLM 中自注意力的轻量级替代品。LiSA 采用微小的前馈网络来对齐相邻层之间的注意力头，并使用低秩矩阵来近似逐层注意力权重的差异。涵盖 13 个典型基准的评估表明，LiSA 在准确率和困惑度方面保持了较高的响应质量，同时在总层数的 53-84% 内减少了冗余注意力计算。我们对 LiSA 的实现实现了 Q 和 K 的 6 倍压缩，LLaMA3-8B 的最大吞吐量提升了 19.5%，LLaMA2-7B 的最大吞吐量提升了 32.3%。

Title: DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models

• Authors: Bowen Wang, Jiuyang Chang, Yiming Qian, Guoxin Chen, Junhao Chen, Zhouqiang Jiang, Jiahao Zhang, Yuta Nakashima, Hajime Nagahara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01933
• Pdf URL: https://arxiv.org/pdf/2408.01933
• Copy Paste: [[2408.01933]] DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models(https://arxiv.org/abs/2408.01933)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have recently showcased remarkable capabilities, spanning a wide range of tasks and applications, including those in the medical domain. Models like GPT-4 excel in medical question answering but may face challenges in the lack of interpretability when handling complex tasks in real clinical settings. We thus introduce the diagnostic reasoning dataset for clinical notes (DiReCT), aiming at evaluating the reasoning ability and interpretability of LLMs compared to human doctors. It contains 521 clinical notes, each meticulously annotated by physicians, detailing the diagnostic reasoning process from observations in a clinical note to the final diagnosis. Additionally, a diagnostic knowledge graph is provided to offer essential knowledge for reasoning, which may not be covered in the training data of existing LLMs. Evaluations of leading LLMs on DiReCT bring out a significant gap between their reasoning ability and that of human doctors, highlighting the critical need for models that can reason effectively in real-world clinical scenarios.
• 摘要：大型语言模型 (LLM) 最近展示了卓越的能力，涵盖了包括医学领域在内的广泛任务和应用。GPT-4 等模型在医学问答方面表现出色，但在处理真实临床环境中的复杂任务时，可能会面临缺乏可解释性的挑战。因此，我们引入了临床笔记诊断推理数据集 (DiReCT)，旨在评估 LLM 与人类医生相比的推理能力和可解释性。它包含 521 份临床笔记，每份都由医生精心注释，详细说明了从临床笔记中的观察到最终诊断的诊断推理过程。此外，还提供了诊断知识图谱，以提供推理的基本知识，而现有 LLM 的训练数据可能未涵盖这些知识。对领先的 LLM 在 DiReCT 上的评估表明，它们的推理能力与人类医生之间存在显著差距，凸显了对能够在现实世界的临床场景中有效推理的模型的迫切需求。

Title: Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inference

• Authors: Ke Shen, Mayank Kejriwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01935
• Pdf URL: https://arxiv.org/pdf/2408.01935
• Copy Paste: [[2408.01935]] Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inference(https://arxiv.org/abs/2408.01935)
• Keywords: language model, gpt, llm, chat
• Abstract: Despite their impressive performance, large language models (LLMs) such as ChatGPT are known to pose important risks. One such set of risks arises from misplaced confidence, whether over-confidence or under-confidence, that the models have in their inference. While the former is well studied, the latter is not, leading to an asymmetry in understanding the comprehensive risk of the model based on misplaced confidence. In this paper, we address this asymmetry by defining two types of risk (decision and composite risk), and proposing an experimental framework consisting of a two-level inference architecture and appropriate metrics for measuring such risks in both discriminative and generative LLMs. The first level relies on a decision rule that determines whether the underlying language model should abstain from inference. The second level (which applies if the model does not abstain) is the model's inference. Detailed experiments on four natural language commonsense reasoning datasets using both an open-source ensemble-based RoBERTa model and ChatGPT, demonstrate the practical utility of the evaluation framework. For example, our results show that our framework can get an LLM to confidently respond to an extra 20.1% of low-risk inference tasks that other methods might misclassify as high-risk, and skip 19.8% of high-risk tasks, which would have been answered incorrectly.
• 摘要：尽管性能出色，但大型语言模型 (LLM)（例如 ChatGPT）也存在重大风险。这类风险之一源于模型在推理中信心过剩，无论是信心过剩还是信心不足。前者已经得到充分研究，而后者则尚未得到充分研究，这导致对基于信心过剩的模型综合风险的理解不对称。在本文中，我们通过定义两种风险（决策风险和复合风险）来解决这种不对称问题，并提出一个实验框架，该框架由两级推理架构和适当的指标组成，用于衡量判别性和生成性 LLM 中的此类风险。第一级依赖于决策规则，该规则确定底层语言模型是否应放弃推理。第二级（如果模型不放弃推理则适用）是模型的推理。使用基于开源集成的 RoBERTa 模型和 ChatGPT 在四个自然语言常识推理数据集上进行的详细实验证明了评估框架的实用性。例如，我们的结果表明，我们的框架可以让 LLM 自信地响应额外的 20.1% 的低风险推理任务（其他方法可能会错误地将其归类为高风险），并跳过 19.8% 的高风险任务（这些任务可能会被错误回答）。

Title: A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios

• Authors: Samuel Ackerman, Ella Rabinovich, Eitan Farchi, Ateret Anaby-Tavor
• Subjects: cs.CL, stat.AP
• Abstract URL: https://arxiv.org/abs/2408.01963
• Pdf URL: https://arxiv.org/pdf/2408.01963
• Copy Paste: [[2408.01963]] A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios(https://arxiv.org/abs/2408.01963)
• Keywords: language model
• Abstract: We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets.
• 摘要：我们在多个数据集上评估了几种大型语言模型的稳健性。这里的稳健性是指模型的答案对其输入的保留意义的变体的相对不敏感性。基准数据集是通过引入自然发生的、非恶意的扰动，或通过生成输入问题或语句的语义等效释义来构建的。我们进一步提出了一种评估模型稳健性的新指标，并通过在创建的数据集上对几种模型进行实证评估，证明了其在非对抗场景中的优势。

Title: ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science

• Authors: Robert Wolfe, Alexis Hiniker, Bill Howe
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.01966
• Pdf URL: https://arxiv.org/pdf/2408.01966
• Copy Paste: [[2408.01966]] ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science(https://arxiv.org/abs/2408.01966)
• Keywords: language model, gpt, prompt
• Abstract: This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in interpreting the traditional EAT measurement by quantifying bias at three levels of increasing granularity: the differential association between two target concepts with two attribute concepts; the individual effect size of each target concept with two attribute concepts; and the association between each individual target concept and each individual attribute concept. Using the ML-EAT, this research defines a taxonomy of EAT patterns describing the nine possible outcomes of an embedding association test, each of which is associated with a unique EAT-Map, a novel four-quadrant visualization for interpreting the ML-EAT. Empirical analysis of static and diachronic word embeddings, GPT-2 language models, and a CLIP language-and-image model shows that EAT patterns add otherwise unobservable information about the component biases that make up an EAT; reveal the effects of prompting in zero-shot models; and can also identify situations when cosine similarity is an ineffective metric, rendering an EAT unreliable. Our work contributes a method for rendering bias more observable and interpretable, improving the transparency of computational investigations into human minds and societies.
• 摘要：本研究介绍了多级嵌入关联测试 (ML-EAT)，这是一种用于可解释和透明地测量语言技术中内在偏见的方法。ML-EAT 通过在三个越来越精细的层次上量化偏见解决了传统 EAT 测量中解释的模糊性和困难性问题：两个目标概念与两个属性概念之间的差异关联；每个目标概念与两个属性概念的个体效应大小；以及每个单独的目标概念和每个单独的属性概念之间的关联。使用 ML-EAT，本研究定义了一种 EAT 模式的分类法，描述了嵌入关联测试的九种可能结果，每种结果都与一个独特的 EAT-Map 相关联，EAT-Map 是一种用于解释 ML-EAT 的新型四象限可视化方法。对静态和历时词嵌入、GPT-2 语言模型和 CLIP 语言与图像模型的实证分析表明，EAT 模式增加了关于构成 EAT 的组件偏见的原本无法观察的信息；揭示了提示在零样本模型中的影响；还可以识别余弦相似度无效的情况，从而导致 EAT 不可靠。我们的工作提供了一种使偏见更易于观察和解释的方法，提高了计算研究对人类思想和社会的透明度​​。

Title: LLaSA: Large Language and E-Commerce Shopping Assistant

• Authors: Shuo Zhang, Boci Peng, Xinping Zhao, Boren Hu, Yun Zhu, Yanjia Zeng, Xuming Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02006
• Pdf URL: https://arxiv.org/pdf/2408.02006
• Copy Paste: [[2408.02006]] LLaSA: Large Language and E-Commerce Shopping Assistant(https://arxiv.org/abs/2408.02006)
• Keywords: language model, llm
• Abstract: The e-commerce platform has evolved rapidly due to its widespread popularity and convenience. Developing an e-commerce shopping assistant for customers is crucial to aiding them in quickly finding desired products and recommending precisely what they need. However, most previous shopping assistants face two main problems: (1) task-specificity, which necessitates the development of different models for various tasks, thereby increasing development costs and limiting effectiveness; and (2) poor generalization, where the trained model performs inadequately on up-to-date products. To resolve these issues, we employ Large Language Models (LLMs) to construct an omnipotent assistant, leveraging their adeptness at handling multiple tasks and their superior generalization capability. Nonetheless, LLMs lack inherent knowledge of e-commerce concepts. To address this, we create an instruction dataset comprising 65,000 samples and diverse tasks, termed as EshopInstruct. Through instruction tuning on our dataset, the assistant, named LLaSA, demonstrates the potential to function as an omnipotent assistant. Additionally, we propose various inference optimization strategies to enhance performance with limited inference resources. In the Amazon KDD Cup 2024 Challenge, our proposed method, LLaSA, achieved an overall ranking of 3rd place on ShopBench, including 57 tasks and approximately 20,000 questions, and we secured top-5 rankings in each track, especially in track4, where we achieved the best performance result among all student teams. Our extensive practices fully demonstrate that LLMs possess the great potential to be competent e-commerce shopping assistants.
• 摘要：电子商务平台因其广泛的普及性和便利性而迅速发展。为客户开发电子商务购物助手对于帮助他们快速找到所需产品并准确推荐他们所需的产品至关重要。然而，大多数以前的购物助手面临两个主要问题：（1）任务特异性，这需要为各种任务开发不同的模型，从而增加了开发成本并限制了效率；（2）泛化能力差，训练后的模型在最新产品上表现不佳。为了解决这些问题，我们使用大型语言模型 (LLM) 来构建一个万能的助手，利用它们处理多项任务的熟练程度和卓越的泛化能力。尽管如此，LLM 缺乏对电子商务概念的固有知识。为了解决这个问题，我们创建了一个包含 65,000 个样本和各种任务的指令数据集，称为 EshopInstruct。通过对我们的数据集进行指令调整，名为 LLaSA 的助手展示了作为万能助手的潜力。此外，我们提出了各种推理优化策略，以在有限的推理资源下提高性能。在亚马逊 KDD Cup 2024 挑战赛中，我们提出的方法 LLaSA 在 ShopBench 上取得了总排名第三的成绩，包括 57 个任务和大约 20,000 个问题，并且我们在每个 track 中都获得了前五名的排名，特别是在 track4 中，我们取得了所有学生团队中最好的性能结果。我们广泛的实践充分证明了 LLM 具有成为称职的电子商务购物助手的巨大潜力。

Title: Fine-tuning multilingual language models in Twitter/X sentiment analysis: a study on Eastern-European V4 languages

• Authors: Tomáš Filip, Martin Pavlíček, Petr Sosík
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02044
• Pdf URL: https://arxiv.org/pdf/2408.02044
• Copy Paste: [[2408.02044]] Fine-tuning multilingual language models in Twitter/X sentiment analysis: a study on Eastern-European V4 languages(https://arxiv.org/abs/2408.02044)
• Keywords: language model, gpt, llm
• Abstract: The aspect-based sentiment analysis (ABSA) is a standard NLP task with numerous approaches and benchmarks, where large language models (LLM) represent the current state-of-the-art. We focus on ABSA subtasks based on Twitter/X data in underrepresented languages. On such narrow tasks, small tuned language models can often outperform universal large ones, providing available and cheap solutions. We fine-tune several LLMs (BERT, BERTweet, Llama2, Llama3, Mistral) for classification of sentiment towards Russia and Ukraine in the context of the ongoing military conflict. The training/testing dataset was obtained from the academic API from Twitter/X during 2023, narrowed to the languages of the V4 countries (Czech Republic, Slovakia, Poland, Hungary). Then we measure their performance under a variety of settings including translations, sentiment targets, in-context learning and more, using GPT4 as a reference model. We document several interesting phenomena demonstrating, among others, that some models are much better fine-tunable on multilingual Twitter tasks than others, and that they can reach the SOTA level with a very small training set. Finally we identify combinations of settings providing the best results.
• 摘要：基于方面的情绪分析 (ABSA) 是一项标准的 NLP 任务，具有多种方法和基准，其中大型语言模型 (LLM) 代表了当前的最新技术。我们专注于基于 Twitter/X 数据的 ABSA 子任务，这些子任务使用代表性不足的语言。在这种狭窄的任务中，小型调整的语言模型通常可以胜过通用大型模型，从而提供可用且廉价的解决方案。我们对几个 LLM（BERT、BERTweet、Llama2、Llama3、Mistral）进行了微调，以在持续的军事冲突背景下对俄罗斯和乌克兰的情绪进行分类。训练/测试数据集是在 2023 年从 Twitter/X 的学术 API 中获得的，范围缩小到 V4 国家（捷克共和国、斯洛伐克、波兰、匈牙利）的语言。然后，我们使用 GPT4 作为参考模型，在各种设置（包括翻译、情绪目标、上下文学习等）下测量它们的表现。我们记录了几个有趣的现象，这些现象表明，一些模型在多语言 Twitter 任务上的微调能力比其他模型好得多，并且它们可以用非常小的训练集达到 SOTA 级别。最后，我们确定了提供最佳结果的设置组合。

Title: MedSyn: LLM-based Synthetic Medical Text Generation Framework

• Authors: Gleb Kumichev, Pavel Blinov, Yulia Kuzkina, Vasily Goncharov, Galina Zubkova, Nikolai Zenovkin, Aleksei Goncharov, Andrey Savchenko
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02056
• Pdf URL: https://arxiv.org/pdf/2408.02056
• Copy Paste: [[2408.02056]] MedSyn: LLM-based Synthetic Medical Text Generation Framework(https://arxiv.org/abs/2408.02056)
• Keywords: language model, gpt, llm, prompt
• Abstract: Generating synthetic text addresses the challenge of data availability in privacy-sensitive domains such as healthcare. This study explores the applicability of synthetic data in real-world medical settings. We introduce MedSyn, a novel medical text generation framework that integrates large language models with a Medical Knowledge Graph (MKG). We use MKG to sample prior medical information for the prompt and generate synthetic clinical notes with GPT-4 and fine-tuned LLaMA models. We assess the benefit of synthetic data through application in the ICD code prediction task. Our research indicates that synthetic data can increase the classification accuracy of vital and challenging codes by up to 17.8% compared to settings without synthetic data. Furthermore, to provide new data for further research in the healthcare domain, we present the largest open-source synthetic dataset of clinical notes for the Russian language, comprising over 41k samples covering 219 ICD-10 codes.
• 摘要：生成合成文本解决了隐私敏感领域（例如医疗保健）中数据可用性的挑战。本研究探讨了合成数据在现实医疗环境中的适用性。我们引入了 MedSyn，这是一种新颖的医学文本生成框架，它将大型语言模型与医学知识图谱 (MKG) 集成在一起。我们使用 MKG 对提示的先前医疗信息进行采样，并使用 GPT-4 和微调的 LLaMA 模型生成合成临床记录。我们通过在 ICD 代码预测任务中的应用来评估合成数据的好处。我们的研究表明，与没有合成数据的设置相比，合成数据可以将重要和具有挑战性的代码的分类准确率提高多达 17.8%。此外，为了为医疗保健领域的进一步研究提供新数据，我们提供了俄语最大的开源合成临床记录数据集，包含超过 41,000 个样本，涵盖 219 个 ICD-10 代码。

Title: Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process

• Authors: Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, Yong Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02103
• Pdf URL: https://arxiv.org/pdf/2408.02103
• Copy Paste: [[2408.02103]] Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process(https://arxiv.org/abs/2408.02103)
• Keywords: language model, gpt, llm
• Abstract: In-context learning (ICL) is a few-shot learning paradigm that involves learning mappings through input-output pairs and appropriately applying them to new instances. Despite the remarkable ICL capabilities demonstrated by Large Language Models (LLMs), existing works are highly dependent on large-scale labeled support sets, not always feasible in practical scenarios. To refine this approach, we focus primarily on an innovative selective annotation mechanism, which precedes the standard demonstration retrieval. We introduce the Language Model-based Determinant Point Process (LM-DPP) that simultaneously considers the uncertainty and diversity of unlabeled instances for optimal selection. Consequently, this yields a subset for annotation that strikes a trade-off between the two factors. We apply LM-DPP to various language models, including GPT-J, LlaMA, and GPT-3. Experimental results on 9 NLU and 2 Generation datasets demonstrate that LM-DPP can effectively select canonical examples. Further analysis reveals that LLMs benefit most significantly from subsets that are both low uncertainty and high diversity.
• 摘要：上下文学习 (ICL) 是一种少样本学习范式，涉及通过输入输出对学习映射并适当地将它们应用于新实例。尽管大型语言模型 (LLM) 展示了卓越的 ICL 能力，但现有工作高度依赖于大规模标记支持集，这在实际场景中并不总是可行的。为了改进这种方法，我们主要关注一种创新的选择性注释机制，该机制先于标准演示检索。我们引入了基于语言模型的行列式点过程 (LM-DPP)，它同时考虑了未标记实例的不确定性和多样性以进行最佳选择。因此，这产生了一个在两个因素之间取得权衡的注释子集。我们将 LM-DPP 应用于各种语言模型，包括 GPT-J、LlaMA 和 GPT-3。在 9 个 NLU 和 2 代数据集上的实验结果表明，LM-DPP 可以有效地选择规范示例。进一步的分析表明，LLM 从低不确定性和高多样性的子集中受益最大。

Title: Recent Advances in Multi-Choice Machine Reading Comprehension: A Survey on Methods and Datasets

• Authors: Shima Foolad, Kourosh Kiani, Razieh Rastgoo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02114
• Pdf URL: https://arxiv.org/pdf/2408.02114
• Copy Paste: [[2408.02114]] Recent Advances in Multi-Choice Machine Reading Comprehension: A Survey on Methods and Datasets(https://arxiv.org/abs/2408.02114)
• Keywords: language model, prompt
• Abstract: This paper provides a thorough examination of recent developments in the field of multi-choice Machine Reading Comprehension (MRC). Focused on benchmark datasets, methodologies, challenges, and future trajectories, our goal is to offer researchers a comprehensive overview of the current landscape in multi-choice MRC. The analysis delves into 30 existing cloze-style and multiple-choice MRC benchmark datasets, employing a refined classification method based on attributes such as corpus style, domain, complexity, context style, question style, and answer style. This classification system enhances our understanding of each dataset's diverse attributes and categorizes them based on their complexity. Furthermore, the paper categorizes recent methodologies into Fine-tuned and Prompt-tuned methods. Fine-tuned methods involve adapting pre-trained language models (PLMs) to a specific task through retraining on domain-specific datasets, while prompt-tuned methods use prompts to guide PLM response generation, presenting potential applications in zero-shot or few-shot learning scenarios. By contributing to ongoing discussions, inspiring future research directions, and fostering innovations, this paper aims to propel multi-choice MRC towards new frontiers of achievement.
• 摘要：本文全面分析了多项选择机器阅读理解 (MRC) 领域的最新发展。我们的目标是重点关注基准数据集、方法、挑战和未来发展轨迹，为研究人员提供多项选择 MRC 当前形势的全面概述。分析深入研究了 30 个现有的完形填空和多项选择 MRC 基准数据集，采用基于语料库风格、领域、复杂性、上下文风格、问题风格和答案风格等属性的精细分类方法。该分类系统增强了我们对每个数据集不同属性的理解，并根据其复杂性对其进行分类。此外，本文将最近的方法分为微调和快速调整方法。微调方法涉及通过在特定领域的数据集上进行重新训练来使预训练语言模型 (PLM) 适应特定任务，而快速调整方法使用提示来指导 PLM 响应生成，从而展示零样本或少样本学习场景中的潜在应用。通过参与正在进行的讨论、启发未来的研究方向以及促进创新，本文旨在推动多项选择 MRC 迈向新的成就前沿。

Title: Table Transformers for Imputing Textual Attributes

• Authors: Ting-Ruen Wei, Yuan Wang, Yoshitaka Inoue, Hsin-Tai Wu, Yi Fang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02128
• Pdf URL: https://arxiv.org/pdf/2408.02128
• Copy Paste: [[2408.02128]] Table Transformers for Imputing Textual Attributes(https://arxiv.org/abs/2408.02128)
• Keywords: gpt, chat
• Abstract: Missing data in tabular dataset is a common issue as the performance of downstream tasks usually depends on the completeness of the training dataset. Previous missing data imputation methods focus on numeric and categorical columns, but we propose a novel end-to-end approach called Table Transformers for Imputing Textual Attributes (TTITA) based on the transformer to impute unstructured textual columns using other columns in the table. We conduct extensive experiments on two Amazon Reviews datasets, and our approach shows competitive performance outperforming baseline models such as recurrent neural networks and Llama2. The performance improvement is more significant when the target sequence has a longer length. Additionally, we incorporated multi-task learning to simultaneously impute for heterogeneous columns, boosting the performance for text imputation. We also qualitatively compare with ChatGPT for realistic applications.
• 摘要：表格数据集中缺失数据是一个常见问题，因为下游任务的性能通常取决于训练数据集的完整性。以前的缺失数据插补方法侧重于数字和分类列，但我们提出了一种新颖的端到端方法，称为用于插补文本属性的表转换器 (TTITA)，该方法基于转换器使用表中的其他列来插补非结构化文本列。我们对两个 Amazon Reviews 数据集进行了广泛实验，我们的方法表现出优于循环神经网络和 Llama2 等基线模型的竞争性能。当目标序列长度较长时，性能提升更为显著。此外，我们结合多任务学习来同时对异构列进行插补，从而提高了文本插补的性能。我们还对实际应用与 ChatGPT 进行了定性比较。

Title: Analyzing Cultural Representations of Emotions in LLMs through Mixed Emotion Survey

• Authors: Shiran Dudy, Ibrahim Said Ahmad, Ryoko Kitajima, Agata Lapedriza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02143
• Pdf URL: https://arxiv.org/pdf/2408.02143
• Copy Paste: [[2408.02143]] Analyzing Cultural Representations of Emotions in LLMs through Mixed Emotion Survey(https://arxiv.org/abs/2408.02143)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have gained widespread global adoption, showcasing advanced linguistic capabilities across multiple of languages. There is a growing interest in academia to use these models to simulate and study human behaviors. However, it is crucial to acknowledge that an LLM's proficiency in a specific language might not fully encapsulate the norms and values associated with its culture. Concerns have emerged regarding potential biases towards Anglo-centric cultures and values due to the predominance of Western and US-based training data. This study focuses on analyzing the cultural representations of emotions in LLMs, in the specific case of mixed-emotion situations. Our methodology is based on the studies of Miyamoto et al. (2010), which identified distinctive emotional indicators in Japanese and American human responses. We first administer their mixed emotion survey to five different LLMs and analyze their outputs. Second, we experiment with contextual variables to explore variations in responses considering both language and speaker origin. Thirdly, we expand our investigation to encompass additional East Asian and Western European origin languages to gauge their alignment with their respective cultures, anticipating a closer fit. We find that (1) models have limited alignment with the evidence in the literature; (2) written language has greater effect on LLMs' response than information on participants origin; and (3) LLMs responses were found more similar for East Asian languages than Western European languages.
• 摘要：大型语言模型 (LLM) 已在全球范围内得到广泛采用，展示了跨多种语言的高级语言能力。学术界越来越有兴趣使用这些模型来模拟和研究人类行为。然而，必须承认，LLM 对特定语言的熟练程度可能无法完全体现与其文化相关的规范和价值观。由于西方和美国的训练数据占主导地位，人们开始担心可能存在对以英国为中心的文化和价值观的偏见。本研究侧重于分析 LLM 中情绪的文化表现，具体情况是混合情绪的情况。我们的方法基于 Miyamoto 等人 (2010) 的研究，他们确定了日本和美国人类反应中不同的情绪指标。我们首先对五个不同的 LLM 进行混合情绪调查，并分析它们的输出。其次，我们尝试使用上下文变量来探索考虑语言和说话者来源的响应变化。第三，我们扩大调查范围，涵盖更多东亚和西欧起源的语言，以衡量它们与各自文化的契合度，以期实现更紧密的契合。我们发现 (1) 模型与文献中的证据的契合度有限；(2) 书面语言对法学硕士的回答的影响大于参与者的来源信息；(3) 法学硕士对东亚语言的回答比对西欧语言的回答更相似。

Title: CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs

• Authors: Weijie Lv, Xuan Xia, Sheng-Jun Huang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02193
• Pdf URL: https://arxiv.org/pdf/2408.02193
• Copy Paste: [[2408.02193]] CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs(https://arxiv.org/abs/2408.02193)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown great potential in code-related tasks, yet open-source models lag behind their closed-source counterparts. To bridge this performance gap, existing methods generate vast amounts of synthetic data for fine-tuning, leading to inefficiencies in training. Motivated by the need for more effective and efficient training, we propose the Code Adaptive Compute-efficient Tuning (CodeACT) framework. CodeACT introduces the Complexity and Diversity Aware Sampling (CDAS) method to select high-quality training data based on complexity and diversity, and the Dynamic Pack padding strategy to reduce computational resource usage by minimizing padding tokens during training. Experimental results demonstrate that CodeACT-DeepSeek-Coder-6.7B, fine-tuned on only 40% of the EVOL-Instruct data, achieves an 8.6% performance increase on HumanEval, reduces training time by 78%, and decreases peak GPU memory usage by 27%. These findings underscore CodeACT's ability to enhance the performance and efficiency of open-source models. By optimizing both the data selection and training processes, CodeACT offers a comprehensive approach to improving the capabilities of open-source LLMs while significantly reducing computational requirements, addressing the dual challenges of data quality and training efficiency, and paving the way for more resource-efficient and performant models.
• 摘要：大型语言模型 (LLM) 在代码相关任务中显示出巨大潜力，但开源模型却落后于闭源模型。为了弥补这种性能差距，现有方法会生成大量合成数据进行微调，从而导致训练效率低下。为了满足更有效、更高效的训练需求，我们提出了代码自适应计算高效调优 (CodeACT) 框架。CodeACT 引入了复杂性和多样性感知采样 (CDAS) 方法，根据复杂性和多样性选择高质量训练数据，并引入了动态填充策略，通过最小化训练期间的填充标记来减少计算资源使用量。实验结果表明，仅对 40% 的 EVOL-Instruct 数据进行微调的 CodeACT-DeepSeek-Coder-6.7B 在 HumanEval 上实现了 8.6% 的性能提升，训练时间减少了 78%，峰值 GPU 内存使用量降低了 27%。这些发现凸显了 CodeACT 能够提高开源模型的性能和效率。通过优化数据选择和训练过程，CodeACT 提供了一种全面的方法来提高开源 LLM 的功能，同时显著降低计算要求，解决数据质量和训练效率的双重挑战，并为更节省资源和性能更高的模型铺平道路。

Title: A Multi-Source Heterogeneous Knowledge Injected Prompt Learning Method for Legal Charge Prediction

• Authors: Jingyun Sun, Chi Wei, Yang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02233
• Pdf URL: https://arxiv.org/pdf/2408.02233
• Copy Paste: [[2408.02233]] A Multi-Source Heterogeneous Knowledge Injected Prompt Learning Method for Legal Charge Prediction(https://arxiv.org/abs/2408.02233)
• Keywords: llm, prompt
• Abstract: Legal charge prediction, an essential task in legal AI, seeks to assign accurate charge labels to case descriptions, attracting significant recent interest. Existing methods primarily employ diverse neural network structures for modeling case descriptions directly, failing to effectively leverage multi-source external knowledge. We propose a prompt learning framework-based method that simultaneously leverages multi-source heterogeneous external knowledge from a legal knowledge base, a conversational LLM, and related legal articles. Specifically, we match knowledge snippets in case descriptions via the legal knowledge base and encapsulate them into the input through a hard prompt template. Additionally, we retrieve legal articles related to a given case description through contrastive learning, and then obtain factual elements within the case description through a conversational LLM. We fuse the embedding vectors of soft prompt tokens with the encoding vector of factual elements to achieve knowledge-enhanced model forward inference. Experimental results show that our method achieved state-of-the-art results on CAIL-2018, the largest legal charge prediction dataset, and our method has lower data dependency. Case studies also demonstrate our method's strong interpretability.
• 摘要：法律指控预测是法律人工智能中的一项重要任务，旨在为案例描述分配准确的指控标签，最近引起了广泛关注。现有方法主要采用多种神经网络结构直接对案例描述进行建模，无法有效利用多源外部知识。我们提出了一种基于提示学习框架的方法，该方法同时利用来自法律知识库、对话式 LLM 和相关法律文章的多源异构外部知识。具体而言，我们通过法律知识库匹配案例描述中的知识片段，并通过硬提示模板将其封装到输入中。此外，我们通过对比学习检索与给定案例描述相关的法律文章，然后通过对话式 LLM 获取案例描述中的事实元素。我们将软提示标记的嵌入向量与事实元素的编码向量融合，以实现知识增强的模型前向推理。实验结果表明，我们的方法在最大的法律指控预测数据集 CAIL-2018 上取得了最佳结果，并且我们的方法具有较低的数据依赖性。案例研究也证明了我们的方法具有很强的可解释性。

Title: Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings

• Authors: Md. Arid Hasan, Prerona Tarannum, Krishno Dey, Imran Razzak, Usman Naseem
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02237
• Pdf URL: https://arxiv.org/pdf/2408.02237
• Copy Paste: [[2408.02237]] Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings(https://arxiv.org/abs/2408.02237)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have garnered significant interest in natural language processing (NLP), particularly their remarkable performance in various downstream tasks in resource-rich languages. Recent studies have highlighted the limitations of LLMs in low-resource languages, primarily focusing on binary classification tasks and giving minimal attention to South Asian languages. These limitations are primarily attributed to constraints such as dataset scarcity, computational costs, and research gaps specific to low-resource languages. To address this gap, we present datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, facilitating research in low-resource language processing. Further, we comprehensively examine zero-shot learning using multiple LLMs in English and widely spoken South Asian languages. Our findings indicate that GPT-4 consistently outperforms Llama 2 and Gemini, with English consistently demonstrating superior performance across diverse tasks compared to low-resource languages. Furthermore, our analysis reveals that natural language inference (NLI) exhibits the highest performance among the evaluated tasks, with GPT-4 demonstrating superior capabilities.
• 摘要：大型语言模型 (LLM) 引起了人们对自然语言处理 (NLP) 的极大兴趣，尤其是它们在资源丰富的语言的各种下游任务中的出色表现。最近的研究强调了 LLM 在低资源语言中的局限性，主要关注二元分类任务，而对南亚语言的关注很少。这些限制主要归因于数据集稀缺、计算成本和特定于低资源语言的研究差距等限制。为了解决这一差距，我们通过将英语翻译成孟加拉语、印地语和乌尔都语来提供情绪和仇恨言论任务的数据集，以促进低资源语言处理的研究。此外，我们使用英语和广泛使用的南亚语言中的多个 LLM 全面检查零样本学习。我们的研究结果表明，GPT-4 的表现始终优于 Llama 2 和 Gemini，与低资源语言相比，英语在不同任务中始终表现出优异的性能。此外，我们的分析表明，自然语言推理 (NLI) 在评估的任务中表现出最高的性能，GPT-4 表现出卓越的能力。

Title: BOTS-LM: Training Large Language Models for Setswana

• Authors: Nathan Brown, Vukosi Marivate
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02239
• Pdf URL: https://arxiv.org/pdf/2408.02239
• Copy Paste: [[2408.02239]] BOTS-LM: Training Large Language Models for Setswana(https://arxiv.org/abs/2408.02239)
• Keywords: language model
• Abstract: In this work we present BOTS-LM, a series of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, BOTS-LM achieves performance similar to models significantly larger than itself while maintaining computational efficiency. Our initial release features an 8 billion parameter generative large language model, with upcoming 0.5 billion and 1 billion parameter large language models and a 278 million parameter encoder-only model soon to be released. We find the 8 billion parameter model significantly outperforms Llama-3-70B and Aya 23 on English-Setswana translation tasks, approaching the performance of dedicated machine translation models, while approaching 70B parameter performance on Setswana reasoning as measured by a machine translated subset of the MMLU benchmark. To accompany the BOTS-LM series of language models, we release the largest Setswana web dataset, SetsText, totalling over 267 million tokens. In addition, we release the largest machine translated Setswana dataset, the first and largest synthetic Setswana dataset, training and evaluation code, training logs, and MMLU-tsn, a machine translated subset of MMLU.
• 摘要：在本研究中，我们介绍了 BOTS-LM，这是一系列精通塞茨瓦纳语和英语的双语语言模型。利用数据可用性和高效微调方面的最新进展，BOTS-LM 在保持计算效率的同时实现了与比其大得多的模型相似的性能。我们最初发布的版本具有 80 亿参数生成大型语言模型，即将推出 5 亿和 10 亿参数大型语言模型和 2.78 亿参数编码器专用模型。我们发现，80 亿参数模型在英语-塞茨瓦纳语翻译任务上的表现明显优于 Llama-3-70B 和 Aya 23，接近专用机器翻译模型的性能，同时在塞茨瓦纳语推理方面接近 70B 参数性能，这是由 MMLU 基准的机器翻译子集测量的。为了配合 BOTS-LM 系列语言模型，我们发布了最大的塞茨瓦纳语网络数据集 SetsText，总计超过 2.67 亿个标记。此外，我们还发布了最大的机器翻译塞茨瓦纳语数据集、第一个也是最大的合成塞茨瓦纳语数据集、训练和评估代码、训练日志以及 MMLU 的机器翻译子集 MMLU-tsn。

Title: ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems

• Authors: Andrew Zhu, Liam Dugan, Chris Callison-Burch
• Subjects: cs.CL, cs.MA, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.02248
• Pdf URL: https://arxiv.org/pdf/2408.02248
• Copy Paste: [[2408.02248]] ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems(https://arxiv.org/abs/2408.02248)
• Keywords: language model, llm, agent
• Abstract: Recently, there has been increasing interest in using Large Language Models (LLMs) to construct complex multi-agent systems to perform tasks such as compiling literature reviews, drafting consumer reports, and planning vacations. Many tools and libraries exist for helping create such systems, however none support recursive multi-agent systems -- where the models themselves flexibly decide when to delegate tasks and how to organize their delegation structure. In this work, we introduce ReDel: a toolkit for recursive multi-agent systems that supports custom tool-use, delegation schemes, event-based logging, and interactive replay in an easy-to-use web interface. We show that, using ReDel, we are able to achieve significant performance gains on agentic benchmarks and easily identify potential areas of improvements through the visualization and debugging tools. Our code, documentation, and PyPI package are open-source and free to use under the MIT license.
• 摘要：最近，人们越来越有兴趣使用大型语言模型 (LLM) 来构建复杂的多智能体系统，以执行诸如编写文献综述、起草消费者报告和计划假期等任务。有许多工具和库可以帮助创建这样的系统，但没有一个支持递归多智能体系统——模型本身可以灵活地决定何时委派任务以及如何组织其委派结构。在这项工作中，我们引入了 ReDel：一个用于递归多智能体系统的工具包，它支持自定义工具使用、委派方案、基于事件的日志记录和易于使用的 Web 界面中的交互式重放。我们表明，使用 ReDel，我们能够在智能体基准上实现显著的性能提升，并通过可视化和调试工具轻松识别潜在的改进领域。我们的代码、文档和 PyPI 包都是开源的，可以在 MIT 许可下免费使用。

Title: SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models

• Authors: Shujuan Zhao, Lingfeng Qiao, Kangyang Luo, Qian-Wen Zhang, Junru Lu, Di Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02302
• Pdf URL: https://arxiv.org/pdf/2408.02302
• Copy Paste: [[2408.02302]] SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models(https://arxiv.org/abs/2408.02302)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have become powerful tools for advancing natural language processing applications in the financial industry. However, existing financial LLMs often face challenges such as hallucinations or superficial parameter training, resulting in suboptimal performance, particularly in financial computing and machine reading comprehension (MRC). To address these issues, we propose a novel large language model specifically designed for the Chinese financial domain, named SNFinLLM. SNFinLLM excels in domain-specific tasks such as answering questions, summarizing financial research reports, analyzing sentiment, and executing financial calculations. We then perform the supervised fine-tuning (SFT) to enhance the model's proficiency across various financial domains. Specifically, we gather extensive financial data and create a high-quality instruction dataset composed of news articles, professional papers, and research reports of finance domain. Utilizing both domain-specific and general datasets, we proceed with continuous pre-training on an established open-source base model, resulting in SNFinLLM-base. Following this, we engage in supervised fine-tuning (SFT) to bolster the model's capability across multiple financial tasks. Crucially, we employ a straightforward Direct Preference Optimization (DPO) method to better align the model with human preferences. Extensive experiments conducted on finance benchmarks and our evaluation dataset demonstrate that SNFinLLM markedly outperforms other state-of-the-art financial language models. For more details, check out our demo video here: this https URL.
• 摘要：大型语言模型 (LLM) 已成为推动金融行业自然语言处理应用的有力工具。然而，现有的金融 LLM 经常面临幻觉或肤浅的参数训练等挑战，导致性能不佳，特别是在金融计算和机器阅读理解 (MRC) 方面。为了解决这些问题，我们提出了一种专门为中文金融领域设计的新型大型语言模型，名为 SNFinLLM。SNFinLLM 擅长于回答问题、总结金融研究报告、分析情绪和执行金融计算等领域特定任务。然后，我们执行监督微调 (SFT) 以增强模型在各个金融领域的熟练程度。具体来说，我们收集大量金融数据并创建由新闻文章、专业论文和金融领域研究报告组成的高质量指令数据集。利用领域特定数据集和通用数据集，我们在已建立的开源基础模型上进行持续预训练，从而得到 SNFinLLM-base。随后，我们进行监督微调 (SFT)，以增强模型在多个金融任务中的能力。至关重要的是，我们采用直接的直接偏好优化 (DPO) 方法来更好地使模型与人类偏好保持一致。在金融基准和我们的评估数据集上进行的大量实验表明，SNFinLLM 明显优于其他最先进的金融语言模型。有关更多详细信息，请查看此处的演示视频：此 https URL。

Title: Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding

• Authors: Renato Vukovic, David Arps, Carel van Niekerk, Benjamin Matthias Ruppik, Hsien-Chin Lin, Michael Heck, Milica Gašić
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02361
• Pdf URL: https://arxiv.org/pdf/2408.02361
• Copy Paste: [[2408.02361]] Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding(https://arxiv.org/abs/2408.02361)
• Keywords: language model, hallucination, prompt, chain-of-thought
• Abstract: State-of-the-art task-oriented dialogue systems typically rely on task-specific ontologies for fulfilling user queries. The majority of task-oriented dialogue data, such as customer service recordings, comes without ontology and annotation. Such ontologies are normally built manually, limiting the application of specialised systems. Dialogue ontology construction is an approach for automating that process and typically consists of two steps: term extraction and relation extraction. In this work, we focus on relation extraction in a transfer learning set-up. To improve the generalisation, we propose an extension to the decoding mechanism of large language models. We adapt Chain-of-Thought (CoT) decoding, recently developed for reasoning problems, to generative relation extraction. Here, we generate multiple branches in the decoding space and select the relations based on a confidence threshold. By constraining the decoding to ontology terms and relations, we aim to decrease the risk of hallucination. We conduct extensive experimentation on two widely used datasets and find improvements in performance on target ontology for source fine-tuned and one-shot prompted large language models.
• 摘要：最先进的面向任务的对话系统通常依赖于特定于任务的本体来满足用户查询。大多数面向任务的对话数据（例如客户服务记录）都没有本体和注释。此类本体通常是手动构建的，限制了专用系统的应用。对话本体构建是一种自动化该过程的方法，通常包括两个步骤：术语提取和关系提取。在这项工作中，我们专注于迁移学习设置中的关系提取。为了提高泛化能力，我们提出了对大型语言模型解码机制的扩展。我们将最近为推理问题开发的思路链 (CoT) 解码应用于生成关系提取。在这里，我们在解码空间中生成多个分支并根据置信度阈值选择关系。通过将解码限制在本体术语和关系上，我们旨在降低幻觉的风险。我们对两个广泛使用的数据集进行了广泛的实验，发现源微调和一次性提示的大型语言模型在目标本体上的性能有所提高。

Title: A Few-Shot Approach for Relation Extraction Domain Adaptation using Large Language Models

• Authors: Vanni Zavarella, Juan Carlos Gamero-Salinas, Sergio Consoli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02377
• Pdf URL: https://arxiv.org/pdf/2408.02377
• Copy Paste: [[2408.02377]] A Few-Shot Approach for Relation Extraction Domain Adaptation using Large Language Models(https://arxiv.org/abs/2408.02377)
• Keywords: language model, prompt
• Abstract: Knowledge graphs (KGs) have been successfully applied to the analysis of complex scientific and technological domains, with automatic KG generation methods typically building upon relation extraction models capturing fine-grained relations between domain entities in text. While these relations are fully applicable across scientific areas, existing models are trained on few domain-specific datasets such as SciERC and do not perform well on new target domains. In this paper, we experiment with leveraging in-context learning capabilities of Large Language Models to perform schema-constrained data annotation, collecting in-domain training instances for a Transformer-based relation extraction model deployed on titles and abstracts of research papers in the Architecture, Construction, Engineering and Operations (AECO) domain. By assessing the performance gain with respect to a baseline Deep Learning architecture trained on off-domain data, we show that by using a few-shot learning strategy with structured prompts and only minimal expert annotation the presented approach can potentially support domain adaptation of a science KG generation model.
• 摘要：知识图谱 (KG) 已成功应用于复杂科学技术领域的分析，自动 KG 生成方法通常建立在关系提取模型之上，该模型可捕获文本中领域实体之间的细粒度关系。虽然这些关系完全适用于各个科学领域，但现有模型是在少数领域特定数据集（如 SciERC）上训练的，在新的目标领域表现不佳。在本文中，我们尝试利用大型语言模型的上下文学习功能来执行模式约束的数据注释，为基于 Transformer 的关系提取模型收集领域训练实例，该模型部署在建筑、施工、工程和运营 (AECO) 领域的研究论文标题和摘要上。通过评估相对于在域外数据上训练的基线深度学习架构的性能提升，我们表明，通过使用结构化提示和仅使用最少专家注释的少量学习策略，所提出的方法可以潜在地支持科学 KG 生成模型的领域适应。

Title: Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models

• Authors: Zi Liang, Haibo Hu, Qingqing Ye, Yaxin Xiao, Haoyang Li
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02416
• Pdf URL: https://arxiv.org/pdf/2408.02416
• Copy Paste: [[2408.02416]] Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models(https://arxiv.org/abs/2408.02416)
• Keywords: language model, gpt, llm, prompt
• Abstract: The drastic increase of large language models' (LLMs) parameters has led to a new research direction of fine-tuning-free downstream customization by prompts, i.e., task descriptions. While these prompt-based services (e.g. OpenAI's GPTs) play an important role in many businesses, there has emerged growing concerns about the prompt leakage, which undermines the intellectual properties of these services and causes downstream attacks. In this paper, we analyze the underlying mechanism of prompt leakage, which we refer to as prompt memorization, and develop corresponding defending strategies. By exploring the scaling laws in prompt extraction, we analyze key attributes that influence prompt extraction, including model sizes, prompt lengths, as well as the types of prompts. Then we propose two hypotheses that explain how LLMs expose their prompts. The first is attributed to the perplexity, i.e. the familiarity of LLMs to texts, whereas the second is based on the straightforward token translation path in attention matrices. To defend against such threats, we investigate whether alignments can undermine the extraction of prompts. We find that current LLMs, even those with safety alignments like GPT-4, are highly vulnerable to prompt extraction attacks, even under the most straightforward user attacks. Therefore, we put forward several defense strategies with the inspiration of our findings, which achieve 83.8\% and 71.0\% drop in the prompt extraction rate for Llama2-7B and GPT-3.5, respectively. Source code is avaliable at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 参数的急剧增加导致了一个新的研究方向，即通过提示（即任务描述）进行无需微调的下游定制。虽然这些基于提示的服务（例如 OpenAI 的 GPT）在许多业务中发挥着重要作用，但人们越来越担心提示泄漏，这会破坏这些服务的知识产权并导致下游攻击。在本文中，我们分析了提示泄漏的潜在机制（我们称之为提示记忆），并制定了相应的防御策略。通过探索提示提取中的缩放规律，我们分析了影响提示提取的关键属性，包括模型大小、提示长度以及提示类型。然后，我们提出了两个假设来解释 LLM 如何暴露它们的提示。第一个假设归因于困惑度，即 LLM 对文本的熟悉度，而第二个假设基于注意矩阵中直接的标记翻译路径。为了防御此类威胁，我们研究了对齐是否会破坏提示的提取。我们发现，当前的 LLM，即使是像 GPT-4 这样具有安全性的 LLM，也极易受到即时提取攻击，即使在最直接的用户攻击下也是如此。因此，我们根据研究结果提出了几种防御策略，分别将 Llama2-7B 和 GPT-3.5 的即时提取率降低了 83.8% 和 71.0%。源代码可在 \url{此 https URL} 处获取。

Title: Long Input Benchmark for Russian Analysis

• Authors: Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergej Averkiev, Alena Fenogenova
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02439
• Pdf URL: https://arxiv.org/pdf/2408.02439
• Copy Paste: [[2408.02439]] Long Input Benchmark for Russian Analysis(https://arxiv.org/abs/2408.02439)
• Keywords: language model, llm
• Abstract: Recent advancements in Natural Language Processing (NLP) have fostered the development of Large Language Models (LLMs) that can solve an immense variety of tasks. One of the key aspects of their application is their ability to work with long text documents and to process long sequences of tokens. This has created a demand for proper evaluation of long-context understanding. To address this need for the Russian language, we propose LIBRA (Long Input Benchmark for Russian Analysis), which comprises 21 adapted datasets to study the LLM's abilities to understand long texts thoroughly. The tests are divided into four complexity groups and allow the evaluation of models across various context lengths ranging from 4k up to 128k tokens. We provide the open-source datasets, codebase, and public leaderboard for LIBRA to guide forthcoming research.
• 摘要：自然语言处理 (NLP) 领域的最新进展促进了大型语言模型 (LLM) 的发展，这种模型可以解决各种各样的任务。大型语言模型应用的关键方面之一是它们能够处理长文本文档和长序列的标记。这就需要对长上下文理解进行适当的评估。为了满足俄语的这一需求，我们提出了 LIBRA（俄语分析的长输入基准），它包含 21 个经过调整的数据集，用于研究 LLM 彻底理解长文本的能力。测试分为四个复杂度组，允许对各种上下文长度的模型进行评估，范围从 4k 到 128k 个标记。我们为 LIBRA 提供开源数据集、代码库和公共排行榜，以指导即将开展的研究。

Title: Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

• Authors: Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02442
• Pdf URL: https://arxiv.org/pdf/2408.02442
• Copy Paste: [[2408.02442]] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models(https://arxiv.org/abs/2408.02442)
• Keywords: language model, llm
• Abstract: Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs' abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs' performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs' reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.
• 摘要：结构化生成是一种以 JSON 和 XML 等标准化格式生成内容的过程，在实际应用中被广泛用于从大型语言模型 (LLM) 中提取关键输出信息。本研究调查了这种对生成空间的限制是否会影响 LLM 的能力，包括推理和领域知识理解。具体来说，我们评估了 LLM 在遵守结构化格式和在各种常见任务中生成自由格式响应时的性能。令人惊讶的是，我们观察到 LLM 在格式限制下推理能力显著下降。此外，我们发现更严格的格式约束通常会导致推理任务的性能下降幅度更大。

Title: UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

• Authors: Zhaowei Li, Wei Wang, YiQing Cai, Xu Qi, Pengyu Wang, Dong Zhang, Hang Song, Botian Jiang, Zhida Huang, Tao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02503
• Pdf URL: https://arxiv.org/pdf/2408.02503
• Copy Paste: [[2408.02503]] UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model(https://arxiv.org/abs/2408.02503)
• Keywords: language model, llm
• Abstract: Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental question: Can we develop a unified approach to represent and handle different multi-modal tasks to maximize the generalizability of MLLMs? In this paper, we propose UnifiedMLLM, a comprehensive model designed to represent various tasks using a unified representation. Our model exhibits strong capabilities in comprehending the implicit intent of user instructions and preforming reasoning. In addition to generating textual responses, our model also outputs task tokens and grounding tokens, serving as indicators of task types and task granularity. These outputs are subsequently routed through the task router and directed to specific expert models for task completion. To train our model, we construct a task-specific dataset and an 100k multi-task dataset encompassing complex scenarios. Employing a three-stage training strategy, we equip our model with robust reasoning and task processing capabilities while preserving its generalization capacity and knowledge reservoir. Extensive experiments showcase the impressive performance of our unified representation approach across various tasks, surpassing existing methodologies. Furthermore, our approach exhibits exceptional scalability and generality. Our code, model, and dataset will be available at \url{this https URL}.
• 摘要：最近，多模态大型语言模型 (MLLM) 领域取得了重大进展，展示了它们在理解和推理各种任务方面的卓越能力。然而，这些模型通常针对特定任务进行训练，并依赖于特定于任务的输入输出格式，从而限制了它们在更广泛的任务中的适用性。这提出了一个基本问题：我们能否开发一种统一的方法来表示和处理不同的多模态任务，以最大限度地提高 MLLM 的通用性？在本文中，我们提出了 UnifiedMLLM，这是一个全面的模型，旨在使用统一的表示来表示各种任务。我们的模型在理解用户指令的隐含意图和进行推理方面表现出强大的能力。除了生成文本响应外，我们的模型还输出任务标记和基础标记，作为任务类型和任务粒度的指标。这些输出随后通过任务路由器路由并定向到特定的专家模型以完成任务。为了训练我们的模型，我们构建了一个特定于任务的数据集和一个包含复杂场景的 100k 多任务数据集。通过采用三阶段训练策略，我们为模型配备了强大的推理和任务处理能力，同时保留了其泛化能力和知识库。大量实验展示了我们的统一表示方法在各种任务中的出色表现，超越了现有方法。此外，我们的方法表现出卓越的可扩展性和通用性。我们的代码、模型和数据集将在 \url{此 https URL} 上提供。

Title: OneLove beyond the field -- A few-shot pipeline for topic and sentiment analysis during the FIFA World Cup in Qatar

• Authors: Christoph Rauchegger, Sonja Mei Wang, Pieter Delobelle
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02520
• Pdf URL: https://arxiv.org/pdf/2408.02520
• Copy Paste: [[2408.02520]] OneLove beyond the field -- A few-shot pipeline for topic and sentiment analysis during the FIFA World Cup in Qatar(https://arxiv.org/abs/2408.02520)
• Keywords: llm
• Abstract: The FIFA World Cup in Qatar was discussed extensively in the news and on social media. Due to news reports with allegations of human rights violations, there were calls to boycott it. Wearing a OneLove armband was part of a planned protest activity. Controversy around the armband arose when FIFA threatened to sanction captains who wear it. To understand what topics Twitter users Tweeted about and what the opinion of German Twitter users was towards the OneLove armband, we performed an analysis of German Tweets published during the World Cup using in-context learning with LLMs. We validated the labels on human annotations. We found that Twitter users initially discussed the armband's impact, LGBT rights, and politics; after the ban, the conversation shifted towards politics in sports in general, accompanied by a subtle shift in sentiment towards neutrality. Our evaluation serves as a framework for future research to explore the impact of sports activism and evolving public sentiment. This is especially useful in settings where labeling datasets for specific opinions is unfeasible, such as when events are unfolding.
• 摘要：新闻和社交媒体广泛讨论了卡塔尔世界杯。由于有新闻报道指控侵犯人权，有人呼吁抵制它。佩戴 OneLove 臂章是计划中的抗议活动的一部分。当国际足联威胁要制裁佩戴臂章的队长时，臂章引发了争议。为了了解 Twitter 用户发推文的主题以及德国 Twitter 用户对 OneLove 臂章的看法，我们使用 LLM 的上下文学习对世界杯期间发布的德国推文进行了分析。我们验证了人工注释的标签。我们发现 Twitter 用户最初讨论的是臂章的影响、LGBT 权利和政治；禁令颁布后，谈话转向了体育界的政治，同时情绪也微妙地转向中立。我们的评估为未来研究提供了一个框架，以探索体育激进主义和不断变化的公众情绪的影响。这在无法为特定意见标记数据集的情况下尤其有用，例如当事件正在展开时。

Title: Caution for the Environment: Multimodal Agents are Susceptible to Environmental Distractions

• Authors: Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, Hai Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02544
• Pdf URL: https://arxiv.org/pdf/2408.02544
• Copy Paste: [[2408.02544]] Caution for the Environment: Multimodal Agents are Susceptible to Environmental Distractions(https://arxiv.org/abs/2408.02544)
• Keywords: language model, llm, agent
• Abstract: This paper investigates the faithfulness of multimodal large language model (MLLM) agents in the graphical user interface (GUI) environment, aiming to address the research question of whether multimodal GUI agents can be distracted by environmental context. A general setting is proposed where both the user and the agent are benign, and the environment, while not malicious, contains unrelated content. A wide range of MLLMs are evaluated as GUI agents using our simulated dataset, following three working patterns with different levels of perception. Experimental results reveal that even the most powerful models, whether generalist agents or specialist GUI agents, are susceptible to distractions. While recent studies predominantly focus on the helpfulness (i.e., action accuracy) of multimodal agents, our findings indicate that these agents are prone to environmental distractions, resulting in unfaithful behaviors. Furthermore, we switch to the adversarial perspective and implement environment injection, demonstrating that such unfaithfulness can be exploited, leading to unexpected risks.
• 摘要：本文研究了多模态大型语言模型 (MLLM) 代理在图形用户界面 (GUI) 环境中的忠诚度，旨在解决多模态 GUI 代理是否会受到环境背景干扰的研究问题。提出了一种一般设置，其中用户和代理都是良性的，环境虽然不是恶意的，但包含不相关的内容。使用我们的模拟数据集对各种 MLLM 作为 GUI 代理进行评估，遵循三种具有不同感知水平的工作模式。实验结果表明，即使是最强大的模型，无论是通用代理还是专业 GUI 代理，也容易受到干扰。虽然最近的研究主要关注多模态代理的有用性（即动作准确性），但我们的研究结果表明，这些代理容易受到环境干扰，导致不忠诚的行为。此外，我们切换到对抗视角并实施环境注入，表明这种不忠诚可以被利用，从而导致意想不到的风险。

Title: RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation

• Authors: Daniel Fleischer, Moshe Berchansky, Moshe Wasserblat, Peter Izsak
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02545
• Pdf URL: https://arxiv.org/pdf/2408.02545
• Copy Paste: [[2408.02545]] RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation(https://arxiv.org/abs/2408.02545)
• Keywords: language model, llm, retrieval augmented generation, retrieval-augmented generation
• Abstract: Implementing Retrieval-Augmented Generation (RAG) systems is inherently complex, requiring deep understanding of data, use cases, and intricate design decisions. Additionally, evaluating these systems presents significant challenges, necessitating assessment of both retrieval accuracy and generative quality through a multi-faceted approach. We introduce RAG Foundry, an open-source framework for augmenting large language models for RAG use cases. RAG Foundry integrates data creation, training, inference and evaluation into a single workflow, facilitating the creation of data-augmented datasets for training and evaluating large language models in RAG settings. This integration enables rapid prototyping and experimentation with various RAG techniques, allowing users to easily generate datasets and train RAG models using internal or specialized knowledge sources. We demonstrate the framework effectiveness by augmenting and fine-tuning Llama-3 and Phi-3 models with diverse RAG configurations, showcasing consistent improvements across three knowledge-intensive datasets. Code is released as open-source in this https URL.
• 摘要：实现检索增强生成 (RAG) 系统本质上很复杂，需要深入了解数据、用例和复杂的设计决策。此外，评估这些系统也带来了重大挑战，需要通过多方面的方法评估检索准确性和生成质量。我们引入了 RAG Foundry，这是一个用于增强 RAG 用例大型语言模型的开源框架。RAG Foundry 将数据创建、训练、推理和评估集成到一个工作流程中，有助于创建数据增强数据集，以便在 RAG 设置中训练和评估大型语言模型。这种集成支持快速原型设计和各种 RAG 技术的实验，使用户能够轻松生成数据集并使用内部或专业知识源训练 RAG 模型。我们通过使用不同的 RAG 配置增强和微调 Llama-3 和 Phi-3 模型来展示框架的有效性，展示了三个知识密集型数据集的持续改进。代码在此 https URL 中作为开源发布。

Title: Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information

• Authors: Yauwai Yim, Chunkit Chan, Tianyu Shi, Zheye Deng, Wei Fan, Tianshi Zheng, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02559
• Pdf URL: https://arxiv.org/pdf/2408.02559
• Copy Paste: [[2408.02559]] Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information(https://arxiv.org/abs/2408.02559)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have shown success in handling simple games with imperfect information and enabling multi-agent coordination, but their ability to facilitate practical collaboration against other agents in complex, imperfect information environments, especially in a non-English environment, still needs to be explored. This study investigates the applicability of knowledge acquired by open-source and API-based LLMs to sophisticated text-based games requiring agent collaboration under imperfect information, comparing their performance to established baselines using other types of agents. We propose a Theory of Mind (ToM) planning technique that allows LLM agents to adapt their strategy against various adversaries using only game rules, current state, and historical context as input. An external tool was incorporated to mitigate the challenge of dynamic and extensive action spaces in this card game. Our results show that although a performance gap exists between current LLMs and state-of-the-art reinforcement learning (RL) models, LLMs demonstrate ToM capabilities in this game setting. It consistently improves their performance against opposing agents, suggesting their ability to understand the actions of allies and adversaries and establish collaboration with allies. To encourage further research and understanding, we have made our codebase openly accessible.
• 摘要：大型语言模型 (LLM) 已成功处理具有不完美信息的简单游戏并实现多智能体协调，但它们在复杂、不完美信息环境（尤其是在非英语环境中）中促进与其他智能体进行实际协作的能力仍有待探索。本研究调查了开源和基于 API 的 LLM 所获得的知识在需要不完美信息下的智能体协作的复杂基于文本的游戏中的应用，并将其性能与使用其他类型智能体的既定基线进行比较。我们提出了一种心智理论 (ToM) 规划技术，该技术允许 LLM 智能体仅使用游戏规则、当前状态和历史背景作为输入来调整其针对各种对手的策略。加入了一个外部工具来减轻此纸牌游戏中动态和广泛动作空间的挑战。我们的结果表明，尽管当前的 LLM 与最先进的强化学习 (RL) 模型之间存在性能差距，但 LLM 在这种游戏环境中展示了 ToM 能力。它不断提高它们对抗对方智能体的表现，表明它们能够理解盟友和对手的行动并与盟友建立合作。为了鼓励进一步的研究和理解，我们已经公开了我们的代码库。

Title: Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarization

• Authors: Ankan Mullick, Sombit Bose, Rounak Saha, Ayan Kumar Bhowmick, Aditya Vempaty, Pawan Goyal, Niloy Ganguly, Prasenjit Dey, Ravi Kokku
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.02584
• Pdf URL: https://arxiv.org/pdf/2408.02584
• Copy Paste: [[2408.02584]] Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarization(https://arxiv.org/abs/2408.02584)
• Keywords: language model, llm
• Abstract: The ever-increasing volume of digital information necessitates efficient methods for users to extract key insights from lengthy documents. Aspect-based summarization offers a targeted approach, generating summaries focused on specific aspects within a document. Despite advancements in aspect-based summarization research, there is a continuous quest for improved model performance. Given that large language models (LLMs) have demonstrated the potential to revolutionize diverse tasks within natural language processing, particularly in the problem of summarization, this paper explores the potential of fine-tuning LLMs for the aspect-based summarization task. We evaluate the impact of fine-tuning open-source foundation LLMs, including Llama2, Mistral, Gemma and Aya, on a publicly available domain-specific aspect based summary dataset. We hypothesize that this approach will enable these models to effectively identify and extract aspect-related information, leading to superior quality aspect-based summaries compared to the state-of-the-art. We establish a comprehensive evaluation framework to compare the performance of fine-tuned LLMs against competing aspect-based summarization methods and vanilla counterparts of the fine-tuned LLMs. Our work contributes to the field of aspect-based summarization by demonstrating the efficacy of fine-tuning LLMs for generating high-quality aspect-based summaries. Furthermore, it opens doors for further exploration of using LLMs for targeted information extraction tasks across various NLP domains.
• 摘要：数字信息量不断增加，需要有效的方法来帮助用户从冗长的文档中提取关键见解。基于方面的摘要提供了一种有针对性的方法，可以生成专注于文档中特定方面的摘要。尽管基于方面的摘要研究取得了进展，但人们仍在不断寻求提高模型性能。鉴于大型语言模型 (LLM) 已显示出彻底改变自然语言处理中各种任务的潜力，特别是在摘要问题中，本文探讨了微调 LLM 以完成基于方面的摘要任务的潜力。我们评估了微调开源基础 LLM（包括 Llama2、Mistral、Gemma 和 Aya）对公开可用的领域特定方面摘要数据集的影响。我们假设这种方法将使这些模型能够有效地识别和提取与方面相关的信息，从而产生与最新技术相比质量更高的基于方面的摘要。我们建立了一个全面的评估框架，用于比较微调 LLM 与竞争的基于方面的摘要方法以及微调 LLM 的原始方法的性能。我们的工作通过展示微调 LLM 生成高质量基于方面的摘要的有效性，为基于方面的摘要领域做出了贡献。此外，它为进一步探索在各个 NLP 领域中使用 LLM 进行有针对性的信息提取任务打开了大门。

Title: Progressively Selective Label Enhancement for Language Model Alignment

• Authors: Biao Liu, Ning Xu, Xin Geng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02599
• Pdf URL: https://arxiv.org/pdf/2408.02599
• Copy Paste: [[2408.02599]] Progressively Selective Label Enhancement for Language Model Alignment(https://arxiv.org/abs/2408.02599)
• Keywords: language model
• Abstract: Large Language Models have demonstrated impressive capabilities in various language tasks but may produce content that misaligns with human expectations, raising ethical and legal concerns. Therefore, it is important to explore the limitations and implement restrictions on the models to ensure safety and compliance, with Reinforcement Learning from Human Feedback (RLHF) being the primary method. Due to challenges in stability and scalability with the RLHF stages, researchers are exploring alternative methods to achieve effects comparable to those of RLHF. However, these methods often depend on large high-quality datasets and inefficiently utilize generated data. To deal with this problem, we propose PSLE, i.e., Progressively Selective Label Enhancement for Language Model Alignment, a framework that fully utilizes all generated data by guiding the model with principles to align outputs with human expectations. Using a dynamically updated threshold, our approach ensures efficient data utilization by incorporating all generated responses and weighting them based on their corresponding reward scores. Experimental results on multiple datasets demonstrate the effectiveness of PSLE compared to existing language model alignment methods.
• 摘要：大型语言模型在各种语言任务中表现出了令人印象深刻的能力，但可能会产生与人类期望不符的内容，从而引发道德和法律问题。因此，探索模型的局限性并实施限制以确保安全性和合规性非常重要，其中强化学习人类反馈 (RLHF) 是主要方法。由于 RLHF 阶段在稳定性和可扩展性方面存在挑战，研究人员正在探索替代方法以实现与 RLHF 相当的效果。然而，这些方法通常依赖于大量高质量数据集，并且无法高效地利用生成的数据。为了解决这个问题，我们提出了 PSLE，即语言模型对齐的渐进式选择性标签增强，这是一个充分利用所有生成的数据的框架，通过用原则指导模型以使输出与人类期望保持一致。使用动态更新的阈值，我们的方法通过合并所有生成的响应并根据其相应的奖励分数对其进行加权，从而确保高效的数据利用。与现有的语言模型对齐方法相比，多个数据集上的实验结果证明了 PSLE 的有效性。

Title: Language Model Can Listen While Speaking

• Authors: Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen
• Subjects: cs.CL, cs.AI, cs.HC, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.02622
• Pdf URL: https://arxiv.org/pdf/2408.02622
• Copy Paste: [[2408.02622]] Language Model Can Listen While Speaking(https://arxiv.org/abs/2408.02622)
• Keywords: language model
• Abstract: Dialogue serves as the most natural manner of human-computer interaction (HCI). Recent advancements in speech language models (SLM) have significantly enhanced speech-based conversational AI. However, these models are limited to turn-based conversation, lacking the ability to interact with humans in real-time spoken scenarios, for example, being interrupted when the generated content is not satisfactory. To address these limitations, we explore full duplex modeling (FDM) in interactive speech language models (iSLM), focusing on enhancing real-time interaction and, more explicitly, exploring the quintessential ability of interruption. We introduce a novel model design, namely listening-while-speaking language model (LSLM), an end-to-end system equipped with both listening and speaking channels. Our LSLM employs a token-based decoder-only TTS for speech generation and a streaming self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses both channels for autoregressive generation and detects turn-taking in real time. Three fusion strategies -- early fusion, middle fusion, and late fusion -- are explored, with middle fusion achieving an optimal balance between speech generation and real-time interaction. Two experimental settings, command-based FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity to diverse instructions. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems. This study aims to advance the development of interactive speech dialogue systems, enhancing their applicability in real-world contexts.
• 摘要：对话是人机交互 (HCI) 最自然的方式。语音语言模型 (SLM) 的最新进展显著增强了基于语音的对话式 AI。然而，这些模型仅限于轮流对话，缺乏在实时口语场景中与人类交互的能力，例如，当生成的内容不令人满意时会被打断。为了解决这些限制，我们探索了交互式语音语言模型 (iSLM) 中的全双工建模 (FDM)，重点是增强实时交互，更明确地说，探索打断的典型能力。我们引入了一种新颖的模型设计，即边听边说语言模型 (LSLM)，这是一个配备听和说通道的端到端系统。我们的 LSLM 采用基于 token 的解码器专用 TTS 进行语音生成，并采用流式自监督学习 (SSL) 编码器进行实时音频输入。LSLM 融合两个通道进行自回归生成并实时检测轮流。我们探索了三种融合策略——早期融合、中期融合和晚期融合，其中中期融合在语音生成和实时交互之间实现了最佳平衡。两种实验设置，基于命令的 FDM 和基于语音的 FDM，展示了 LSLM 对噪声的鲁棒性和对不同指令的敏感性。我们的结果突出了 LSLM 实现双工通信的能力，对现有系统的影响最小。这项研究旨在推动交互式语音对话系统的发展，提高它们在现实世界中的适用性。

Title: SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Models

• Authors: Muxi Diao, Rumei Li, Shiyang Liu, Guogang Liao, Jingang Wang, Xunliang Cai, Weiran Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02632
• Pdf URL: https://arxiv.org/pdf/2408.02632
• Copy Paste: [[2408.02632]] SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Models(https://arxiv.org/abs/2408.02632)
• Keywords: language model, gpt, llm, prompt
• Abstract: As large language models (LLMs) continue to advance in capability and influence, ensuring their security and preventing harmful outputs has become crucial. A promising approach to address these concerns involves training models to automatically generate adversarial prompts for red teaming. However, the evolving subtlety of vulnerabilities in LLMs challenges the effectiveness of current adversarial methods, which struggle to specifically target and explore the weaknesses of these models. To tackle these challenges, we introduce the $\mathbf{S}\text{elf-}\mathbf{E}\text{volving }\mathbf{A}\text{dversarial }\mathbf{S}\text{afety }\mathbf{(SEAS)}$ optimization framework, which enhances security by leveraging data generated by the model itself. SEAS operates through three iterative stages: Initialization, Attack, and Adversarial Optimization, refining both the Red Team and Target models to improve robustness and safety. This framework reduces reliance on manual testing and significantly enhances the security capabilities of LLMs. Our contributions include a novel adversarial framework, a comprehensive safety dataset, and after three iterations, the Target model achieves a security level comparable to GPT-4, while the Red Team model shows a marked increase in attack success rate (ASR) against advanced models.
• 摘要：随着大型语言模型 (LLM) 的功能和影响力不断提升，确保其安全性并防止有害输出变得至关重要。解决这些问题的一个有前途的方法是训练模型以自动生成红队对抗提示。然而，LLM 中漏洞的微妙性不断提高，这对当前对抗方法的有效性提出了挑战，这些方法难以专门针对和探索这些模型的弱点。为了应对这些挑战，我们引入了 $\mathbf{S}\text{elf-}\mathbf{E}\text{volving }\mathbf{A}\text{dversarial }\mathbf{S}\text{afety }\mathbf{(SEAS)}$ 优化框架，该框架通过利用模型本身生成的数据来增强安全性。SEAS 通过三个迭代阶段运行：初始化、攻击和对抗优化，改进红队和目标模型以提高稳健性和安全性。该框架减少了对手动测试的依赖，并显著增强了 LLM 的安全功能。我们的贡献包括一个新的对抗框架、一个全面的安全数据集，经过三次迭代，Target 模型达到了与 GPT-4 相当的安全级别，而 Red Team 模型对高级模型的攻击成功率 (ASR) 显著提高。

Title: Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?

• Authors: Mohammad Bahrami Karkevandi, Nishant Vishwamitra, Peyman Najafirad
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02651
• Pdf URL: https://arxiv.org/pdf/2408.02651
• Copy Paste: [[2408.02651]] Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?(https://arxiv.org/abs/2408.02651)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in natural language tasks, but their safety and morality remain contentious due to their training on internet text corpora. To address these concerns, alignment techniques have been developed to improve the public usability and safety of LLMs. Yet, the potential for generating harmful content through these models seems to persist. This paper explores the concept of jailbreaking LLMs-reversing their alignment through adversarial triggers. Previous methods, such as soft embedding prompts, manually crafted prompts, and gradient-based automatic prompts, have had limited success on black-box models due to their requirements for model access and for producing a low variety of manually crafted prompts, making them susceptible to being blocked. This paper introduces a novel approach using reinforcement learning to optimize adversarial triggers, requiring only inference API access to the target model and a small surrogate model. Our method, which leverages a BERTScore-based reward function, enhances the transferability and effectiveness of adversarial triggers on new black-box models. We demonstrate that this approach improves the performance of adversarial triggers on a previously untested language model.
• 摘要：大型语言模型 (LLM) 在自然语言任务中表现出了令人印象深刻的能力，但由于它们是在互联网文本语料库上训练的，因此其安全性和道德性仍然存在争议。为了解决这些问题，已经开发了对齐技术来提高 LLM 的公共可用性和安全性。然而，通过这些模型生成有害内容的可能性似乎仍然存在。本文探讨了越狱 LLM 的概念——通过对抗性触发器逆转其对齐。以前的方法，例如软嵌入提示、手工制作的提示和基于梯度的自动提示，在黑盒模型上取得了有限的成功，因为它们需要模型访问并且需要生成少量手工制作的提示，因此容易被阻止。本文介绍了一种使用强化学习来优化对抗性触发器的新方法，只需要对目标模型和小型代理模型进行推理 API 访问。我们的方法利用基于 BERTScore 的奖励函数，增强了对抗性触发器在新黑盒模型上的可转移性和有效性。我们证明这种方法可以提高以前未经测试的语言模型上对抗触发器的性能。

Title: Self-Taught Evaluators

• Authors: Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, Xian Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02666
• Pdf URL: https://arxiv.org/pdf/2408.02666
• Copy Paste: [[2408.02666]] Self-Taught Evaluators(https://arxiv.org/abs/2408.02666)
• Keywords: gpt, llm
• Abstract: Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples.
• 摘要：基于模型的评估是成功模型开发的核心——作为训练的奖励模型，以及作为人工评估的替代品。为了训练这样的评估者，标准方法是收集大量人类对模型响应的偏好判断，这种方法成本高昂，而且随着模型的改进，数据会变得陈旧。在这项工作中，我们提出了一种旨在改进评估者的方法，无需人工注释，仅使用合成训练数据。从未标记的指令开始，我们的迭代自我改进方案生成对比模型输出并训练 LLM-as-a-Judge 以产生推理轨迹和最终判断，使用改进的预测在每次新迭代中重复此训练。没有任何标记的偏好数据，我们的自学评估者可以在 RewardBench 上将强大的 LLM（Llama3-70B-Instruct）从 75.4 提高到 88.3（多数票为 88.7）。这优于 GPT-4 等常用的 LLM 评判器，并与使用标记示例训练的表现最佳的奖励模型的性能相匹配。