2024-03-06

Title: adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds

• Authors: Séamus Lankford, Haithem Afli, Andy Way
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02370
• Pdf URL: https://arxiv.org/pdf/2403.02370
• Copy Paste: [[2403.02370]] adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds(https://arxiv.org/abs/2403.02370)
• Keywords: language model, llm
• Abstract: The advent of Multilingual Language Models (MLLMs) and Large Language Models has spawned innovation in many areas of natural language processing. Despite the exciting potential of this technology, its impact on developing high-quality Machine Translation (MT) outputs for low-resource languages remains relatively under-explored. Furthermore, an open-source application, dedicated to both fine-tuning MLLMs and managing the complete MT workflow for low-resources languages, remains unavailable. We aim to address these imbalances through the development of adaptMLLM, which streamlines all processes involved in the fine-tuning of MLLMs for MT. This open-source application is tailored for developers, translators, and users who are engaged in MT. An intuitive interface allows for easy customisation of hyperparameters, and the application offers a range of metrics for model evaluation and the capability to deploy models as a translation service directly within the application. As a multilingual tool, we used adaptMLLM to fine-tune models for two low-resource language pairs: English to Irish (EN$\leftrightarrow$GA) and English to Marathi (EN$\leftrightarrow$MR). Compared with baselines from the LoResMT2021 Shared Task, the adaptMLLM system demonstrated significant improvements. In the EN$\rightarrow$GA direction, an improvement of 5.2 BLEU points was observed and an increase of 40.5 BLEU points was recorded in the GA$\rightarrow$EN direction. Significant improvements in the translation performance of the EN$\leftrightarrow$MR pair were also observed notably in the MR$\rightarrow$EN direction with an increase of 21.3 BLEU points. Finally, a fine-grained human evaluation of the MLLM output on the EN$\rightarrow$GA pair was conducted using the Multidimensional Quality Metrics and Scalar Quality Metrics error taxonomies. The application and models are freely available.
• 摘要：多语言语言模型 (MLLM) 和大型语言模型的出现催生了自然语言处理许多领域的创新。尽管这项技术具有令人兴奋的潜力，但它对为资源匮乏的语言开发高质量机器翻译 (MT) 输出的影响仍然相对未被充分探索。此外，专门用于微调 MLLM 和管理低资源语言的完整 MT 工作流程的开源应用程序仍然不可用。我们的目标是通过开发适应MLLM来解决这些不平衡问题，它简化了机器翻译MLLM微调所涉及的所有流程。这个开源应用程序是为从事机器翻译的开发人员、翻译人员和用户量身定制的。直观的界面可以轻松自定义超参数，并且该应用程序提供了一系列用于模型评估的指标以及直接在应用程序中将模型部署为翻译服务的功能。作为多语言工具，我们使用 AdaptMLLM 来微调两种低资源语言对的模型：英语到爱尔兰语 (EN$\leftrightarrow$GA) 和英语到马拉地语 (EN$\leftrightarrow$MR)。与 LoResMT2021 共享任务的基线相比，adaptMLLM 系统表现出显着的改进。在 EN$\rightarrow$GA 方向上，观察到 5.2 BLEU 点的改进，在 GA$\rightarrow$EN 方向上记录了 40.5 BLEU 点的增加。 EN$\leftrightarrow$MR 对的翻译性能也显着改善，尤其是在 MR$\rightarrow$EN 方向，增加了 21.3 BLEU 点。最后，使用多维质量指标和标量质量指标误差分类法对 EN$\rightarrow$GA 对上的 MLLM 输出进行细粒度的人工评估。该应用程序和模型可免费获得。

Title: How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider Transformer Models

• Authors: Xin Lu, Yanyan Zhao, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02436
• Pdf URL: https://arxiv.org/pdf/2403.02436
• Copy Paste: [[2403.02436]] How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider Transformer Models(https://arxiv.org/abs/2403.02436)
• Keywords: language model
• Abstract: Pre-trained language models have been proven to possess strong base capabilities, which not only excel in in-distribution language modeling but also show powerful abilities in out-of-distribution language modeling, transfer learning and few-shot learning. Unlike existing work focusing on the influence of scale on base capabilities, our work examines the influence of architecture on those. Specifically, our concern is: How does architecture influence the base capabilities of pre-trained language models? In this work, we attempt to explain and reverse the decline in base capabilities caused by the architecture of FFN-Wider Transformers, seeking to provide some insights. Through analysis, we found the contribution ratio of Multi-Head Attention (a combination function) to pre-trained language modeling is a key factor affecting base capabilities. FFN-Wider Transformers reduce the contribution ratio of this combination function, leading to a decline in base capabilities. We confirmed this by experiments and proposed Combination Enhancement Architecture (CEA) to address the decline in base capabilities of such models. Significantly, we extended our explanation and CEA to Mixture of Experts (MoE) architecture Transformers, which also alleviated their decline in base capabilities to some extent, proving our work can offer useful guidance for architecture analysis, architecture improvement and architecture design.
• 摘要：预训练语言模型已被证明具有强大的基础能力，不仅在分布内语言建模方面表现出色，而且在分布外语言建模、迁移学习和少样本学习方面也表现出强大的能力。与关注规模对基础能力影响的现有工作不同，我们的工作研究了架构对基础能力的影响。具体来说，我们关心的是：架构如何影响预训练语言模型的基本能力？在这项工作中，我们试图解释并扭转由 FFN-Wider Transformers 架构引起的基础能力下降，寻求提供一些见解。通过分析，我们发现Multi-Head Attention（一种组合函数）对预训练语言模型的贡献率是影响基础能力的关键因素。 FFN-Wider Transformers降低了该组合功能的贡献比例，导致基础能力下降。我们通过实验证实了这一点，并提出了组合增强架构（CEA）来解决此类模型基础能力下降的问题。值得注意的是，我们将我们的解释和CEA扩展到了Mixture of Experts (MoE)架构Transformers，这也在一定程度上缓解了其基础能力的下降，证明我们的工作可以为架构分析、架构改进和架构设计提供有用的指导。

Title: Views Are My Own, But Also Yours: Benchmarking Theory of Mind using Common Ground

• Authors: Adil Soubki, John Murzaku, Arash Yousefi Jordehi, Peter Zeng, Magdalena Markowska, Seyed Abolghasem Mirroshandel, Owen Rambow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02451
• Pdf URL: https://arxiv.org/pdf/2403.02451
• Copy Paste: [[2403.02451]] Views Are My Own, But Also Yours: Benchmarking Theory of Mind using Common Ground(https://arxiv.org/abs/2403.02451)
• Keywords: language model
• Abstract: Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received much attention. However, many existing benchmarks rely on synthetic data which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM.
• 摘要：评估语言模型 (LM) 的思维理论 (ToM) 能力最近受到了广泛关注。然而，许多现有的基准依赖于合成数据，这可能导致实验结果与人类行为不一致。我们引入了第一个基于自然发生的口语对话的 ToM 数据集 Common-ToM，并表明 LM 很难展示 ToM。然后我们证明，集成简单、明确的信念表示可以提高 Common-ToM 上的 LM 性能。

Title: OffLanDat: A Community Based Implicit Offensive Language Dataset Generated by Large Language Model Through Prompt Engineering

• Authors: Amit Das, Mostafa Rahgouy, Dongji Feng, Zheng Zhang, Tathagata Bhattacharya, Nilanjana Raychawdhary, Mary Sandage, Lauramarie Pope, Gerry Dozier, Cheryl Seals
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02472
• Pdf URL: https://arxiv.org/pdf/2403.02472
• Copy Paste: [[2403.02472]] OffLanDat: A Community Based Implicit Offensive Language Dataset Generated by Large Language Model Through Prompt Engineering(https://arxiv.org/abs/2403.02472)
• Keywords: language model, gpt, prompt, chat
• Abstract: The widespread presence of offensive languages on social media has resulted in adverse effects on societal well-being. As a result, it has become very important to address this issue with high priority. Offensive languages exist in both explicit and implicit forms, with the latter being more challenging to detect. Current research in this domain encounters several challenges. Firstly, the existing datasets primarily rely on the collection of texts containing explicit offensive keywords, making it challenging to capture implicitly offensive contents that are devoid of these keywords. Secondly, usual methodologies tend to focus solely on textual analysis, neglecting the valuable insights that community information can provide. In this research paper, we introduce a novel dataset OffLanDat, a community based implicit offensive language dataset generated by ChatGPT containing data for 38 different target groups. Despite limitations in generating offensive texts using ChatGPT due to ethical constraints, we present a prompt-based approach that effectively generates implicit offensive languages. To ensure data quality, we evaluate our data with human. Additionally, we employ a prompt-based Zero-Shot method with ChatGPT and compare the detection results between human annotation and ChatGPT annotation. We utilize existing state-of-the-art models to see how effective they are in detecting such languages. We will make our code and dataset public for other researchers.
• 摘要：社交媒体上广泛存在的攻击性语言对社会福祉造成了不利影响。因此，高度优先解决这个问题变得非常重要。攻击性语言有显式和隐式两种形式，后者更难以检测。目前该领域的研究遇到了一些挑战。首先，现有数据集主要依赖于包含显式攻击性关键字的文本集合，这使得捕获缺乏这些关键字的隐式攻击性内容变得具有挑战性。其次，通常的方法往往只关注文本分析，而忽略了社区信息可以提供的有价值的见解。在这篇研究论文中，我们介绍了一个新颖的数据集 OffLanDat，这是一个由 ChatGPT 生成的基于社区的隐式攻击性语言数据集，其中包含 38 个不同目标群体的数据。尽管由于道德限制，使用 ChatGPT 生成攻击性文本存在局限性，但我们提出了一种基于提示的方法，可以有效生成隐式攻击性语言。为了确保数据质量，我们由人工评估我们的数据。此外，我们采用基于提示的零射击方法和 ChatGPT，并比较人工注释和 ChatGPT 注释之间的检测结果。我们利用现有的最先进的模型来看看它们在检测此类语言方面有多有效。我们将向其他研究人员公开我们的代码和数据集。

Title: Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents

• Authors: Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, Bill Yuchen Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02502
• Pdf URL: https://arxiv.org/pdf/2403.02502
• Copy Paste: [[2403.02502]] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents(https://arxiv.org/abs/2403.02502)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO. This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.
• 摘要：大型语言模型（LLM）已成为各种自治代理系统中不可或缺的组成部分。在这项研究中，我们提出了一种基于探索的轨迹优化方法，称为 ETO。这种学习方法旨在提高开放式LLM代理的性能。与之前专门训练成功的专家轨迹的研究相反，我们的方法允许智能体从探索失败中学习。这可以通过迭代优化框架提高性能。在探索阶段，智能体在完成给定任务的同时与环境进行交互，收集故障轨迹以创建对比轨迹对。在随后的训练阶段，代理利用这些轨迹偏好对，使用 DPO 等对比学习方法来更新其策略。这种探索和训练的迭代循环促进了智能体的持续改进。我们对三个复杂任务的实验表明，ETO 始终大幅超越基线性能。此外，在缺乏专家轨迹的情况下对任务解决效率和潜力的检查强调了我们方法的有效性。

Title: A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processing

• Authors: Yu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02504
• Pdf URL: https://arxiv.org/pdf/2403.02504
• Copy Paste: [[2403.02504]] A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processing(https://arxiv.org/abs/2403.02504)
• Keywords: language model
• Abstract: The pretrain-finetune paradigm represents a transformative approach in natural language processing (NLP). This paradigm distinguishes itself through the use of large pretrained language models, demonstrating remarkable efficiency in finetuning tasks, even with limited training data. This efficiency is especially beneficial for research in social sciences, where the number of annotated samples is often quite limited. Our tutorial offers a comprehensive introduction to the pretrain-finetune paradigm. We first delve into the fundamental concepts of pretraining and finetuning, followed by practical exercises using real-world applications. We demonstrate the application of the paradigm across various tasks, including multi-class classification and regression. Emphasizing its efficacy and user-friendliness, the tutorial aims to encourage broader adoption of this paradigm. To this end, we have provided open access to all our code and datasets. The tutorial is particularly valuable for quantitative researchers in psychology, offering them an insightful guide into this innovative approach.
• 摘要：预训练微调范式代表了自然语言处理 (NLP) 中的一种变革性方法。该范例通过使用大型预训练语言模型而脱颖而出，即使在训练数据有限的情况下，也能在微调任务方面表现出显着的效率。这种效率对于社会科学研究尤其有益，因为社会科学研究中带注释的样本数量通常非常有限。我们的教程全面介绍了预训练微调范例。我们首先深入研究预训练和微调的基本概念，然后使用实际应用进行实际练习。我们演示了该范例在各种任务中的应用，包括多类分类和回归。本教程强调其功效和用户友好性，旨在鼓励更广泛地采用这种范例。为此，我们提供了对所有代码和数据集的开放访问。该教程对于心理学定量研究人员特别有价值，为他们提供了这种创新方法的富有洞察力的指南。

Title: SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models

• Authors: Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, Kamalika Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02509
• Pdf URL: https://arxiv.org/pdf/2403.02509
• Copy Paste: [[2403.02509]] SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models(https://arxiv.org/abs/2403.02509)
• Keywords: language model, llm
• Abstract: In recent years, large language models (LLMs) have become increasingly prevalent, offering remarkable text generation capabilities. However, a pressing challenge is their tendency to make confidently wrong predictions, highlighting the critical need for uncertainty quantification (UQ) in LLMs. While previous works have mainly focused on addressing aleatoric uncertainty, the full spectrum of uncertainties, including epistemic, remains inadequately explored. Motivated by this gap, we introduce a novel UQ method, sampling with perturbation for UQ (SPUQ), designed to tackle both aleatoric and epistemic uncertainties. The method entails generating a set of perturbations for LLM inputs, sampling outputs for each perturbation, and incorporating an aggregation module that generalizes the sampling uncertainty approach for text generation tasks. Through extensive experiments on various datasets, we investigated different perturbation and aggregation techniques. Our findings show a substantial improvement in model uncertainty calibration, with a reduction in Expected Calibration Error (ECE) by 50\% on average. Our findings suggest that our proposed UQ method offers promising steps toward enhancing the reliability and trustworthiness of LLMs.
• 摘要：近年来，大型语言模型（LLM）变得越来越流行，提供了卓越的文本生成功能。然而，一个紧迫的挑战是他们倾向于做出自信的错误预测，这凸显了法学硕士对不确定性量化（UQ）的迫切需求。虽然以前的工作主要集中在解决任意不确定性，但对包括认知在内的所有不确定性的探索仍然不够。受这一差距的启发，我们引入了一种新颖的 UQ 方法，即 UQ 扰动采样（SPUQ），旨在解决任意和认知不确定性。该方法需要为 LLM 输入生成一组扰动，对每个扰动的输出进行采样，并结合一个聚合模块来概括文本生成任务的采样不确定性方法。通过对各种数据集进行广泛的实验，我们研究了不同的扰动和聚合技术。我们的研究结果表明模型不确定性校准有了显着改善，预期校准误差 (ECE) 平均减少了 50%。我们的研究结果表明，我们提出的昆士兰大学方法为提高法学硕士的可靠性和可信度提供了有希望的步骤。

Title: Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF

• Authors: Chen Zheng, Ke Sun, Hang Wu, Chenguang Xi, Xun Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02513
• Pdf URL: https://arxiv.org/pdf/2403.02513
• Copy Paste: [[2403.02513]] Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF(https://arxiv.org/abs/2403.02513)
• Keywords: language model, llm, prompt
• Abstract: In recent advancements in Conversational Large Language Models (LLMs), a concerning trend has emerged, showing that many new base LLMs experience a knowledge reduction in their foundational capabilities following Supervised Fine-Tuning (SFT). This process often leads to issues such as forgetting or a decrease in the base model's abilities. Moreover, fine-tuned models struggle to align with user preferences, inadvertently increasing the generation of toxic outputs when specifically prompted. To overcome these challenges, we adopted an innovative approach by completely bypassing SFT and directly implementing Harmless Reinforcement Learning from Human Feedback (RLHF). Our method not only preserves the base model's general capabilities but also significantly enhances its conversational abilities, while notably reducing the generation of toxic outputs. Our approach holds significant implications for fields that demand a nuanced understanding and generation of responses, such as customer service. We applied this methodology to Mistral, the most popular base model, thereby creating Mistral-Plus. Our validation across 11 general tasks demonstrates that Mistral-Plus outperforms similarly sized open-source base models and their corresponding instruct versions. Importantly, the conversational abilities of Mistral-Plus were significantly improved, indicating a substantial advancement over traditional SFT models in both safety and user preference alignment.
• 摘要：在会话大型语言模型 (LLM) 的最新进展中，出现了一个令人担忧的趋势，表明许多新的基础 LLM 在监督微调 (SFT) 后，其基础能力出现了知识减少。这个过程通常会导致诸如遗忘或基础模型能力下降等问题。此外，经过微调的模型很难与用户偏好保持一致，在特别提示时会无意中增加有毒输出的产生。为了克服这些挑战，我们采用了一种创新方法，完全绕过 SFT，直接实施来自人类反馈的无害强化学习 (RLHF)。我们的方法不仅保留了基本模型的一般功能，而且还显着增强了其对话能力，同时显着减少了有毒输出的产生。我们的方法对于需要细致入微的理解和生成响应的领域具有重大影响，例如客户服务。我们将此方法应用于最流行的基础模型 Mistral，从而创建了 Mistral-Plus。我们对 11 个一般任务的验证表明，Mistral-Plus 的性能优于类似大小的开源基础模型及其相应的指令版本。重要的是，Mistral-Plus 的对话能力得到了显着提高，这表明在安全性和用户偏好一致性方面比传统 SFT 模型有了实质性的进步。

Title: DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation

• Authors: Xueqing Wu, Rui Zheng, Jingzhen Sha, Te-Lin Wu, Hanyu Zhou, Mohan Tang, Kai-Wei Chang, Nanyun Peng, Haoran Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02528
• Pdf URL: https://arxiv.org/pdf/2403.02528
• Copy Paste: [[2403.02528]] DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation(https://arxiv.org/abs/2403.02528)
• Keywords: llm, prompt
• Abstract: Data analysis is a crucial analytical process to generate in-depth studies and conclusive insights to comprehensively answer a given user query for tabular data. In this work, we aim to propose new resources and benchmarks to inspire future research on this crucial yet challenging and under-explored task. However, collecting data analysis annotations curated by experts can be prohibitively expensive. We propose to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs with a multi-turn prompting technique. We construct the DACO dataset, containing (1) 440 databases (of tabular data) collected from real-world scenarios, (2) ~2k query-answer pairs that can serve as weak supervision for model training, and (3) a concentrated but high-quality test set with human refined annotations that serves as our main evaluation benchmark. We train a 6B supervised fine-tuning (SFT) model on DACO dataset, and find that the SFT model learns reasonable data analysis capabilities. To further align the models with human preference, we use reinforcement learning to encourage generating analysis perceived by human as helpful, and design a set of dense rewards to propagate the sparse human preference reward to intermediate code generation steps. Our DACO-RL algorithm is evaluated by human annotators to produce more helpful answers than SFT model in 57.72% cases, validating the effectiveness of our proposed algorithm. Data and code are released at https://github.com/shirley-wu/daco
• 摘要：数据分析是生成深入研究和结论性见解以全面回答给定用户对表格数据的查询的关键分析过程。在这项工作中，我们的目标是提出新的资源和基准，以激发未来对这一至关重要但具有挑战性且尚未探索的任务的研究。然而，收集专家整理的数据分析注释可能会非常昂贵。我们建议利用法学硕士的代码生成功能和多轮提示技术自动生成高质量的答案注释。我们构建了 DACO 数据集，其中包含 (1) 从现实场景收集的 440 个数据库（表格数据），(2) 约 2k 个查询-答案对，可以作为模型训练的弱监督，以及 (3) 一个集中但具有人工精细注释的高质量测试集作为我们的主要评估基准。我们在 DACO 数据集上训练了 6B 监督微调（SFT）模型，发现 SFT 模型学习了合理的数据分析能力。为了进一步使模型与人类偏好保持一致，我们使用强化学习来鼓励生成人类认为有帮助的分析，并设计一组密集奖励将稀疏的人类偏好奖励传播到中间代码生成步骤。我们的 DACO-RL 算法经过人类注释者的评估，在 57.72% 的情况下产生比 SFT 模型更有用的答案，验证了我们提出的算法的有效性。数据和代码发布于https://github.com/shirley-wu/daco

Title: Updating the Minimum Information about CLinical Artificial Intelligence (MI-CLAIM) checklist for generative modeling research

• Authors: Brenda Y. Miao, Irene Y. Chen, Christopher YK Williams, Jaysón Davidson, Augusto Garcia-Agundez, Harry Sun, Travis Zack, Atul J. Butte, Madhumita Sushil
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02558
• Pdf URL: https://arxiv.org/pdf/2403.02558
• Copy Paste: [[2403.02558]] Updating the Minimum Information about CLinical Artificial Intelligence (MI-CLAIM) checklist for generative modeling research(https://arxiv.org/abs/2403.02558)
• Keywords: language model, llm
• Abstract: Recent advances in generative models, including large language models (LLMs), vision language models (VLMs), and diffusion models, have accelerated the field of natural language and image processing in medicine and marked a significant paradigm shift in how biomedical models can be developed and deployed. While these models are highly adaptable to new tasks, scaling and evaluating their usage presents new challenges not addressed in previous frameworks. In particular, the ability of these models to produce useful outputs with little to no specialized training data ("zero-" or "few-shot" approaches), as well as the open-ended nature of their outputs, necessitate the development of updated guidelines in using and evaluating these models. In response to gaps in standards and best practices for the development of clinical AI tools identified by US Executive Order 141103 and several emerging national networks for clinical AI evaluation, we begin to formalize some of these guidelines by building on the "Minimum information about clinical artificial intelligence modeling" (MI-CLAIM) checklist. The MI-CLAIM checklist, originally developed in 2020, provided a set of six steps with guidelines on the minimum information necessary to encourage transparent, reproducible research for artificial intelligence (AI) in medicine. Here, we propose modifications to the original checklist that highlight differences in training, evaluation, interpretability, and reproducibility of generative models compared to traditional AI models for clinical research. This updated checklist also seeks to clarify cohort selection reporting and adds additional items on alignment with ethical standards.
• 摘要：生成模型（包括大语言模型（LLM）、视觉语言模型（VLM）和扩散模型）的最新进展加速了医学中自然语言和图像处理领域的发展，并标志着生物医学模型开发方式的重大范式转变并部署。虽然这些模型高度适应新任务，但扩展和评估它们的使用提出了先前框架中未解决的新挑战。特别是，这些模型在几乎不需要专门训练数据（“零”或“少样本”方法）的情况下产生有用输出的能力，以及其输出的开放性，需要开发更新的模型使用和评估这些模型的指南。为了应对美国第 141103 号行政命令和几个新兴国家临床人工智能评估网络确定的临床人工智能工具开发标准和最佳实践方面的差距，我们开始通过建立在“有关临床人工智能的最低信息”的基础上，正式制定其中一些指南。情报建模”（MI-CLAIM）清单。 MI-CLAIM 清单最初于 2020 年制定，提供了一组六个步骤，并提供了鼓励透明、可重复的医学人工智能 (AI) 研究所需的最低限度信息的指南。在这里，我们建议对原始清单进行修改，突出显示生成模型与传统临床研究人工智能模型相比在训练、评估、可解释性和可重复性方面的差异。此更新的清单还旨在澄清队列选择报告，并添加与道德标准保持一致的其他项目。

Title: Eliciting Better Multilingual Structured Reasoning from LLMs through Code

• Authors: Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02567
• Pdf URL: https://arxiv.org/pdf/2403.02567
• Copy Paste: [[2403.02567]] Eliciting Better Multilingual Structured Reasoning from LLMs through Code(https://arxiv.org/abs/2403.02567)
• Keywords: language model, llm, prompt
• Abstract: Development of large language models (LLM) have shown progress on reasoning, though studies have been limited to English or simple reasoning tasks. We thus introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks. We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multi-lingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus showing our techniques maintain general-purpose abilities.
• 摘要：大语言模型（LLM）的开发已经在推理方面取得了进展，尽管研究仅限于英语或简单的推理任务。因此，我们引入了一个多语言结构化推理和解释数据集，称为 xSTREET，涵盖六种语言的四项任务。 xSTREET 暴露了英语和非英语推理任务之间法学硕士基础表现的差距。然后，我们提出了两种方法来弥补这一差距，基于这样的见解：受过代码训练的法学硕士是更好的推理者。首先，在训练时，我们使用机器翻译通过多语言注释来扩充代码数据集，同时保持程序代码不变。其次，在推理时，我们通过采用包含分步代码原语的提示结构来弥合训练和推理之间的差距，以得出新事实并找到解决方案。我们的方法在 xSTREET 上显示出改进的多语言性能，尤其是在科学常识推理子任务上。此外，模型在非推理任务上没有表现出回归，从而表明我们的技术保持了通用能力。

Title: Improving Event Definition Following For Zero-Shot Event Detection

• Authors: Zefan Cai, Po-Nien Kung, Ashima Suvarna, Mingyu Derek Ma, Hritik Bansal, Baobao Chang, P. Jeffrey Brantingham, Wei Wang, Nanyun Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02586
• Pdf URL: https://arxiv.org/pdf/2403.02586
• Copy Paste: [[2403.02586]] Improving Event Definition Following For Zero-Shot Event Detection(https://arxiv.org/abs/2403.02586)
• Keywords: language model, gpt, prompt
• Abstract: Existing approaches on zero-shot event detection usually train models on datasets annotated with known event types, and prompt them with unseen event definitions. These approaches yield sporadic successes, yet generally fall short of expectations. In this work, we aim to improve zero-shot event detection by training models to better follow event definitions. We hypothesize that a diverse set of event types and definitions are the key for models to learn to follow event definitions while existing event extraction datasets focus on annotating many high-quality examples for a few event types. To verify our hypothesis, we construct an automatically generated Diverse Event Definition (DivED) dataset and conduct comparative studies. Our experiments reveal that a large number of event types (200) and diverse event definitions can significantly boost event extraction performance; on the other hand, the performance does not scale with over ten examples per event type. Beyond scaling, we incorporate event ontology information and hard-negative samples during training, further boosting the performance. Based on these findings, we fine-tuned a LLaMA-2-7B model on our DivED dataset, yielding performance that surpasses SOTA large language models like GPT-3.5 across three open benchmarks on zero-shot event detection.
• 摘要：现有的零样本事件检测方法通常在用已知事件类型注释的数据集上训练模型，并用未见过的事件定义提示它们。这些方法取得了零星的成功，但总体上没有达到预期。在这项工作中，我们的目标是通过训练模型来改进零样本事件检测，以更好地遵循事件定义。我们假设一组不同的事件类型和定义是模型学习遵循事件定义的关键，而现有的事件提取数据集专注于注释少数事件类型的许多高质量示例。为了验证我们的假设，我们构建了一个自动生成的多样化事件定义（DivED）数据集并进行比较研究。我们的实验表明，大量的事件类型（200）和多样化的事件定义可以显着提高事件提取性能；另一方面，对于每种事件类型超过十个示例，性能无法扩展。除了扩展之外，我们还在训练过程中结合了事件本体信息和硬负样本，进一步提高了性能。基于这些发现，我们在 DivED 数据集上对 LLaMA-2-7B 模型进行了微调，在零样本事件检测的三个开放基准测试中，其性能超越了 GPT-3.5 等 SOTA 大型语言模型。

Title: Exploring the Limitations of Large Language Models in Compositional Relation Reasoning

• Authors: Jinman Zhao, Xueyan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02615
• Pdf URL: https://arxiv.org/pdf/2403.02615
• Copy Paste: [[2403.02615]] Exploring the Limitations of Large Language Models in Compositional Relation Reasoning(https://arxiv.org/abs/2403.02615)
• Keywords: language model, llm
• Abstract: We present a comprehensive evaluation of large language models(LLMs)' ability to reason about composition relations through a benchmark encompassing 1,500 test cases in English, designed to cover six distinct types of composition relations: Positional, Comparative, Personal, Mathematical, Identity, and Other. Acknowledging the significance of multilingual capabilities, we expanded our assessment to include translations of these cases into Chinese, Japanese, French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims at investigating the robustness and adaptability of LLMs in handling composition relation reasoning across diverse linguistic contexts.
• 摘要：我们通过包含 1,500 个英语测试用例的基准，对大型语言模型 (LLM) 推理组合关系的能力进行了全面评估，旨在涵盖六种不同类型的组合关系：位置、比较、个人、数学、同一性和其他。认识到多语言能力的重要性，我们扩大了评估范围，将这些案例翻译成中文、日语、法语和韩语。我们的多语言构图关系 (MCR) 基准旨在调查法学硕士在跨不同语言环境处理构图关系推理方面的鲁棒性和适应性。

Title: FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model

• Authors: Xiangyu Li, Xinjie Shen, Yawen Zeng, Xiaofen Xing, Jin Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02647
• Pdf URL: https://arxiv.org/pdf/2403.02647
• Copy Paste: [[2403.02647]] FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model(https://arxiv.org/abs/2403.02647)
• Keywords: language model
• Abstract: The task of stock earnings forecasting has received considerable attention due to the demand investors in real-world scenarios. However, compared with financial institutions, it is not easy for ordinary investors to mine factors and analyze news. On the other hand, although large language models in the financial field can serve users in the form of dialogue robots, it still requires users to have financial knowledge to ask reasonable questions. To serve the user experience, we aim to build an automatic system, FinReport, for ordinary investors to collect information, analyze it, and generate reports after summarizing. Specifically, our FinReport is based on financial news announcements and a multi-factor model to ensure the professionalism of the report. The FinReport consists of three modules: news factorization module, return forecasting module, risk assessment module. The news factorization module involves understanding news information and combining it with stock factors, the return forecasting module aim to analysis the impact of news on market sentiment, and the risk assessment module is adopted to control investment risk. Extensive experiments on real-world datasets have well verified the effectiveness and explainability of our proposed FinReport. Our codes and datasets are available at https://github.com/frinkleko/FinReport.
• 摘要：由于现实场景中投资者的需求，股票收益预测任务受到了相当多的关注。但与金融机构相比，普通投资者挖掘因素、分析新闻并不容易。另一方面，金融领域的大型语言模型虽然可以以对话机器人的形式为用户服务，但仍然需要用户具备金融知识才能提出合理的问题。为了服务用户体验，我们的目标是建立一个自动化系统FinReport，供普通投资者收集信息、分析信息、汇总后生成报告。具体来说，我们的FinReport以财经新闻公告为基础，采用多因素模型，保证报告的专业性。 FinReport由三个模块组成：新闻分解模块、收益预测模块、风险评估模块。新闻分解模块涉及了解新闻信息并将其与股票因素相结合，收益预测模块旨在分析新闻对市场情绪的影响，风险评估模块用于控制投资风险。对现实世界数据集的大量实验很好地验证了我们提出的 FinReport 的有效性和可解释性。我们的代码和数据集可在 https://github.com/frinkleko/FinReport 获取。

Title: Revisiting Meta-evaluation for Grammatical Error Correction

• Authors: Masamune Kobayashi, Masato Mita, Mamoru Komachi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02674
• Pdf URL: https://arxiv.org/pdf/2403.02674
• Copy Paste: [[2403.02674]] Revisiting Meta-evaluation for Grammatical Error Correction(https://arxiv.org/abs/2403.02674)
• Keywords: language model, llm
• Abstract: Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges including biases caused by inconsistencies in evaluation granularity, and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models (LLMs), and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation, suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.
• 摘要：指标是语法错误纠正（GEC）自动评估的基础，对指标的评估（元评估）依赖于它们与人类判断的相关性。然而，英语 GEC 中的传统元评估遇到了一些挑战，包括评估粒度不一致导致的偏差以及使用经典系统的过时设置。这些问题可能会导致对指标的误解，并可能阻碍 GEC 技术的适用性。为了解决这些问题，本文提出了 SEEDA，一个用于 GEC 元评估的新数据集。 SEEDA 包含两种不同粒度的人工评分修正：基于编辑和基于句子，涵盖 12 个最先进的系统，包括大型语言模型 (LLM) 和两种不同侧重点的人工修正。通过调整句子级元评估的粒度来改善相关性的结果表明，现有研究中基于编辑的指标可能被低估了。此外，当从经典系统转变为神经系统时，大多数指标的相关性都会降低，这表明传统指标在评估经过多次编辑的流畅纠正句子方面相对较差。

Title: InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

• Authors: Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02691
• Pdf URL: https://arxiv.org/pdf/2403.02691
• Copy Paste: [[2403.02691]] InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents(https://arxiv.org/abs/2403.02691)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites). However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users. Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative. In this work, we introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data. We evaluate 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time. Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates, nearly doubling the attack success rate on the ReAct-prompted GPT-4. Our findings raise questions about the widespread deployment of LLM Agents. Our benchmark is available at https://github.com/uiuc-kang-lab/InjecAgent.
• 摘要：最近的工作将法学硕士作为代理，允许他们访问工具、执行操作并与外部内容（例如电子邮件或网站）进行交互。然而，外部内容引入了间接提示注入 (IPI) 攻击的风险，其中恶意指令嵌入到 LLM 处理的内容中，旨在操纵这些代理对用户执行有害操作。鉴于此类攻击可能造成严重后果，建立评估和减轻这些风险的基准势在必行。在这项工作中，我们引入了 InjecAgent，这是一个旨在评估工具集成的 LLM 代理对 IPI 攻击的脆弱性的基准。 InjecAgent 包含 1,054 个测试用例，涵盖 17 种不同的用户工具和 62 种攻击者工具。我们将攻击意图分为两种主要类型：对用户的直接伤害和私人数据的泄露。我们评估了 30 种不同的 LLM 代理，结果表明这些代理容易受到 IPI 攻击，其中 ReAct 提示的 GPT-4 在 24% 的情况下容易受到攻击。对增强设置的进一步调查显示，攻击者指令通过黑客提示得到加强，成功率进一步提高，ReAct 提示的 GPT-4 上的攻击成功率几乎翻了一番。我们的研究结果对法学硕士代理的广泛部署提出了质疑。我们的基准测试可在 https://github.com/uiuc-kang-lab/InjecAgent 上找到。

Title: Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment

• Authors: Congzhi Zhang, Linhai Zhang, Deyu Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02698
• Pdf URL: https://arxiv.org/pdf/2403.02698
• Copy Paste: [[2403.02698]] Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment(https://arxiv.org/abs/2403.02698)
• Keywords: language model
• Abstract: Conventional multi-hop fact verification models are prone to rely on spurious correlations from the annotation artifacts, leading to an obvious performance decline on unbiased datasets. Among the various debiasing works, the causal inference-based methods become popular by performing theoretically guaranteed debiasing such as casual intervention or counterfactual reasoning. However, existing causal inference-based debiasing methods, which mainly formulate fact verification as a single-hop reasoning task to tackle shallow bias patterns, cannot deal with the complicated bias patterns hidden in multiple hops of evidence. To address the challenge, we propose Causal Walk, a novel method for debiasing multi-hop fact verification from a causal perspective with front-door adjustment. Specifically, in the structural causal model, the reasoning path between the treatment (the input claim-evidence graph) and the outcome (the veracity label) is introduced as the mediator to block the confounder. With the front-door adjustment, the causal effect between the treatment and the outcome is decomposed into the causal effect between the treatment and the mediator, which is estimated by applying the idea of random walk, and the causal effect between the mediator and the outcome, which is estimated with normalized weighted geometric mean approximation. To investigate the effectiveness of the proposed method, an adversarial multi-hop fact verification dataset and a symmetric multi-hop fact verification dataset are proposed with the help of the large language model. Experimental results show that Causal Walk outperforms some previous debiasing methods on both existing datasets and the newly constructed datasets. Code and data will be released at https://github.com/zcccccz/CausalWalk.
• 摘要：传统的多跳事实验证模型很容易依赖于注释工件的虚假相关性，导致无偏数据集上的性能明显下降。在各种去偏见工作中，基于因果推理的方法通过执行理论上保证的去偏见（例如随意干预或反事实推理）而变得流行。然而，现有的基于因果推理的去偏差方法主要将事实验证作为单跳推理任务来处理浅层偏差模式，无法处理隐藏在多跳证据中的复杂偏差模式。为了应对这一挑战，我们提出了 Causal Walk，这是一种通过前门调整从因果角度消除多跳事实验证偏差的新方法。具体来说，在结构因果模型中，引入处理（输入主张证据图）和结果（真实性标签）之间的推理路径作为中介来阻止混杂因素。通过前门调整，将处理与结果之间的因果效应分解为应用随机游走思想估计的处理与中介变量之间的因果效应，以及中介变量与结果之间的因果效应，这是用归一化加权几何平均近似估计的。为了研究所提出方法的有效性，借助大语言模型提出了对抗性多跳事实验证数据集和对称多跳事实验证数据集。实验结果表明，Causal Walk 在现有数据集和新构建的数据集上都优于以前的一些去偏方法。代码和数据将在https://github.com/zcccccz/CausalWalk发布。

Title: Breeze-7B Technical Report

• Authors: Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, Da-Shan Shiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02712
• Pdf URL: https://arxiv.org/pdf/2403.02712
• Copy Paste: [[2403.02712]] Breeze-7B Technical Report(https://arxiv.org/abs/2403.02712)
• Keywords: language model, chat
• Abstract: Breeze-7B is an open-source language model based on Mistral-7B, designed to address the need for improved language comprehension and chatbot-oriented capabilities in Traditional Chinese. This technical report provides an overview of the additional pretraining, finetuning, and evaluation stages for the Breeze-7B model. The Breeze-7B family of base and chat models exhibits good performance on language comprehension and chatbot-oriented tasks, reaching the top in several benchmarks among models comparable in its complexity class.
• 摘要：Breeze-7B 是基于 Mistral-7B 的开源语言模型，旨在满足改进繁体中文语言理解和面向聊天机器人的功能的需求。本技术报告概述了 Breeze-7B 模型的额外预训练、微调和评估阶段。 Breeze-7B 系列基础模型和聊天模型在语言理解和面向聊天机器人的任务上表现出良好的性能，在其复杂性级别的同类模型中的多项基准测试中名列前茅。

Title: Android in the Zoo: Chain-of-Action-Thought for GUI Agents

• Authors: Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang
• Subjects: cs.CL, cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02713
• Pdf URL: https://arxiv.org/pdf/2403.02713
• Copy Paste: [[2403.02713]] Android in the Zoo: Chain-of-Action-Thought for GUI Agents(https://arxiv.org/abs/2403.02713)
• Keywords: language model, llm, chat, agent
• Abstract: Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typical consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon an off-the-shell LLM, CoAT significantly improves the goal progress compared to standard context modeling. To further facilitate the research in this line, we construct a benchmark Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 200M model on our AitZ dataset achieves on par performance with CogAgent-Chat-18B.
• 摘要：大语言模型（LLM）导致智能手机自主GUI代理的激增，它们通过预测API的一系列动作来完成由自然语言触发的任务。尽管该任务高度依赖于过去的动作和视觉观察，但现有的研究通常很少考虑中间屏幕截图和屏幕操作所执行的语义信息。为了解决这个问题，这项工作提出了行动思想链（称为 CoAT），它描述了先前的行动、当前的屏幕，更重要的是，行动思考应该执行什么行动以及由当前行动导致的结果。选择的行动。我们证明，在现成的法学硕士的零样本设置中，与标准上下文建模相比，CoAT 显着提高了目标进度。为了进一步促进这方面的研究，我们构建了一个基准 Android-In-The-Zoo (AitZ)，其中包含 18,643 个屏幕动作对以及动作思想注释链。实验表明，在 AitZ 数据集上微调 200M 模型可达到与 CogAgent-Chat-18B 相当的性能。

Title: Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

• Authors: Sang T. Truong, Duc Q. Nguyen, Toan Nguyen, Dong D. Le, Nhi N. Truong, Tho Quan, Sanmi Koyejo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02715
• Pdf URL: https://arxiv.org/pdf/2403.02715
• Copy Paste: [[2403.02715]] Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models(https://arxiv.org/abs/2403.02715)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretraining on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 common tasks and 31 metrics. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese. Moreover, our analysis indicates that models with more parameters can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or fine-tuning datasets. These insights underscore the significance of meticulous fine-tuning with high-quality datasets in enhancing LLM performance.
• 摘要：大型语言模型（LLM）的最新进展凸显了它们在人工智能发展中的重要性。然而，尽管对多语言数据集进行了广泛的预训练，可用的开源法学硕士在处理越南语方面的效果有限。由于缺乏为越南法学硕士评估量身定制的系统基准数据集和指标，这一挑战变得更加严重。为了缓解这些问题，我们专门针对越南语对法学硕士进行了微调，并开发了一个包含 10 项常见任务和 31 项指标的综合评估框架。我们的评估结果表明，经过微调的法学硕士表现出增强的越南语理解和生成能力。此外，我们的分析表明，具有更多参数的模型可能会引入更多偏差和未校准的输出，而影响 LLM 性能的关键因素是训练或微调数据集的质量。这些见解强调了使用高质量数据集进行细致微调对于提高法学硕士成绩的重要性。

Title: HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?

• Authors: Sijie Ji, Xinzhe Zheng, Chenshu Wu
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.02727
• Pdf URL: https://arxiv.org/pdf/2403.02727
• Copy Paste: [[2403.02727]] HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?(https://arxiv.org/abs/2403.02727)
• Keywords: language model, gpt, llm, prompt
• Abstract: There is an ongoing debate regarding the potential of Large Language Models (LLMs) as foundational models seamlessly integrated with Cyber-Physical Systems (CPS) for interpreting the physical world. In this paper, we carry out a case study to answer the following question: Are LLMs capable of zero-shot human activity recognition (HAR). Our study, HARGPT, presents an affirmative answer by demonstrating that LLMs can comprehend raw IMU data and perform HAR tasks in a zero-shot manner, with only appropriate prompts. HARGPT inputs raw IMU data into LLMs and utilizes the role-play and think step-by-step strategies for prompting. We benchmark HARGPT on GPT4 using two public datasets of different inter-class similarities and compare various baselines both based on traditional machine learning and state-of-the-art deep classification models. Remarkably, LLMs successfully recognize human activities from raw IMU data and consistently outperform all the baselines on both datasets. Our findings indicate that by effective prompting, LLMs can interpret raw IMU data based on their knowledge base, possessing a promising potential to analyze raw sensor data of the physical world effectively.
• 摘要：关于大型语言模型 (LLM) 作为与网络物理系统 (CPS) 无缝集成以解释物理世界的基础模型的潜力，一直存在争论。在本文中，我们进行了一个案例研究来回答以下问题：法学硕士是否能够进行零样本人类活动识别（HAR）。我们的研究 HARGPT 通过证明法学硕士可以理解原始 IMU 数据并仅在适当的提示下以零样本方式执行 HAR 任务，从而给出了肯定的答案。 HARGPT 将原始 IMU 数据输入 LLM，并利用角色扮演和逐步思考策略进行提示。我们使用两个具有不同类间相似性的公共数据集在 GPT4 上对 HARGPT 进行基准测试，并比较基于传统机器学习和最先进的深度分类模型的各种基线。值得注意的是，法学硕士成功地从原始 IMU 数据中识别人类活动，并始终优于两个数据集的所有基线。我们的研究结果表明，通过有效的提示，法学硕士可以根据其知识库解释原始 IMU 数据，具有有效分析物理世界的原始传感器数据的巨大潜力。

Title: Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment

• Authors: Congzhi Zhang, Linhai Zhang, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02738
• Pdf URL: https://arxiv.org/pdf/2403.02738
• Copy Paste: [[2403.02738]] Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment(https://arxiv.org/abs/2403.02738)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Despite the significant achievements of existing prompting methods such as in-context learning and chain-of-thought for large language models (LLMs), they still face challenges of various biases. Traditional debiasing methods primarily focus on the model training stage, including data augmentation-based and reweight-based approaches, with the limitations of addressing the complex biases of LLMs. To address such limitations, the causal relationship behind the prompting methods is uncovered using a structural causal model, and a novel causal prompting method based on front-door adjustment is proposed to effectively mitigate the bias of LLMs. In specific, causal intervention is implemented by designing the prompts without accessing the parameters and logits of LLMs.The chain-of-thoughts generated by LLMs are employed as the mediator variable and the causal effect between the input prompt and the output answers is calculated through front-door adjustment to mitigate model biases. Moreover, to obtain the representation of the samples precisely and estimate the causal effect more accurately, contrastive learning is used to fine-tune the encoder of the samples by aligning the space of the encoder with the LLM. Experimental results show that the proposed causal prompting approach achieves excellent performance on 3 natural language processing datasets on both open-source and closed-source LLMs.
• 摘要：尽管现有的提示方法（例如上下文学习和大语言模型（LLM）的思维链）取得了显着的成就，但它们仍然面临着各种偏见的挑战。传统的去偏差方法主要集中在模型训练阶段，包括基于数据增强和基于重新加权的方法，但在解决法学硕士的复杂偏差方面存在局限性。为了解决这些局限性，利用结构因果模型揭示了提示方法背后的因果关系，并提出了一种基于前门调整的新型因果提示方法，以有效减轻法学硕士的偏差。具体来说，因果干预是通过在不访问LLM的参数和逻辑的情况下设计提示来实现的。采用LLM生成的思维链作为中介变量，并通过以下方式计算输入提示和输出答案之间的因果效应：前门调整以减轻模型偏差。此外，为了精确地获得样本的表示并更准确地估计因果效应，通过将编码器的空间与LLM对齐，使用对比学习来微调样本的编码器。实验结果表明，所提出的因果提示方法在开源和闭源法学硕士的 3 个自然语言处理数据集上均取得了优异的性能。

Title: Towards Training A Chinese Large Language Model for Anesthesiology

• Authors: Zhonghai Wang, Jie Jiang, Yibing Zhan, Bohao Zhou, Yanhong Li, Chong Zhang, Liang Ding, Hua Jin, Jun Peng, Xu Lin, Weifeng Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02742
• Pdf URL: https://arxiv.org/pdf/2403.02742
• Copy Paste: [[2403.02742]] Towards Training A Chinese Large Language Model for Anesthesiology(https://arxiv.org/abs/2403.02742)
• Keywords: language model, gpt, llm
• Abstract: Medical large language models (LLMs) have gained popularity recently due to their significant practical utility. However, most existing research focuses on general medicine, and there is a need for in-depth study of LLMs in specific fields like anesthesiology. To fill the gap, we introduce Hypnos, a Chinese Anesthesia model built upon existing LLMs, e.g., Llama. Hypnos' contributions have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from current LLMs likely includes inaccuracies. Hypnos implements a cross-filtering strategy to improve the data quality. This strategy involves using one LLM to assess the quality of the generated data from another LLM and filtering out the data with low quality. 2) Hypnos employs a general-to-specific training strategy that starts by fine-tuning LLMs using the general medicine data and subsequently improving the fine-tuned LLMs using data specifically from Anesthesiology. The general medical data supplement the medical expertise in Anesthesiology and enhance the effectiveness of Hypnos' generation. 3) We introduce a standardized benchmark for evaluating medical LLM in Anesthesiology. Our benchmark includes both publicly available instances from the Internet and privately obtained cases from the Hospital. Hypnos outperforms other medical LLMs in anesthesiology in metrics, GPT-4, and human evaluation on the benchmark dataset.
• 摘要：医学大语言模型（LLM）最近因其显着的实用性而受到欢迎。然而，现有的研究大多集中在普通医学，需要对麻醉学等特定领域的法学硕士进行深入研究。为了填补这一空白，我们引入了 Hypnos，这是一种基于现有法学硕士（例如 Llama）的中国麻醉模型。 Hypnos 的贡献有三个方面： 1) 从当前法学硕士获得的数据（例如利用自我指导）可能包含不准确之处。 Hypnos 实施交叉过滤策略来提高数据质量。该策略涉及使用一个法学硕士来评估另一个法学硕士生成的数据的质量并过滤掉低质量的数据。 2) Hypnos 采用从一般到具体的培训策略，首先使用一般医学数据微调法学硕士，然后使用专门来自麻醉学的​​数据改进微调的法学硕士。一般医学数据补充了麻醉学的医学专业知识，并提高了 Hypnos 生成的有效性。 3) 我们引入了评估麻醉学医学法学硕士的标准化基准。我们的基准包括来自互联网的公开实例和来自医院的私人获取的实例。 Hypnos 在指标、GPT-4 和基准数据集的人类评估方面优于麻醉学领域的其他医学法学硕士。

Title: Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models

• Authors: Rui Wang, Fei Mi, Yi Chen, Boyang Xue, Hongru Wang, Qi Zhu, Kam-Fai Wong, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02756
• Pdf URL: https://arxiv.org/pdf/2403.02756
• Copy Paste: [[2403.02756]] Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models(https://arxiv.org/abs/2403.02756)
• Keywords: language model, llm, prompt
• Abstract: The growing interest in Large Language Models (LLMs) for specialized applications has revealed a significant challenge: when tailored to specific domains, LLMs tend to experience catastrophic forgetting, compromising their general capabilities and leading to a suboptimal user experience. Additionally, crafting a versatile model for multiple domains simultaneously often results in a decline in overall performance due to confusion between domains. In response to these issues, we present the RolE Prompting Guided Multi-Domain Adaptation (REGA) strategy. This novel approach effectively manages multi-domain LLM adaptation through three key components: 1) Self-Distillation constructs and replays general-domain exemplars to alleviate catastrophic forgetting. 2) Role Prompting assigns a central prompt to the general domain and a unique role prompt to each specific domain to minimize inter-domain confusion during training. 3) Role Integration reuses and integrates a small portion of domain-specific data to the general-domain data, which are trained under the guidance of the central prompt. The central prompt is used for a streamlined inference process, removing the necessity to switch prompts for different domains. Empirical results demonstrate that REGA effectively alleviates catastrophic forgetting and inter-domain confusion. This leads to improved domain-specific performance compared to standard fine-tuned models, while still preserving robust general capabilities.
• 摘要：人们对专业应用程序的大型语言模型 (LLM) 的兴趣日益增长，这揭示了一个重大挑战：当针对特定领域进行定制时，LLM 往往会经历灾难性的遗忘，从而损害其一般功能并导致次优的用户体验。此外，同时为多个领域构建通用模型通常会由于领域之间的混乱而导致整体性能下降。针对这些问题，我们提出了角色提示引导多域适应（REGA）策略。这种新颖的方法通过三个关键组件有效地管理多领域 LLM 适应：1）自蒸馏构建和重放通用领域范例以减轻灾难性遗忘。 2）角色提示为一般域分配一个中心提示，为每个特定域分配一个独特的角色提示，以最大限度地减少训练过程中域间的混乱。 3）角色集成将一小部分特定领域数据复用并集成到通用领域数据中，在中央提示的指导下进行训练。中央提示用于简化推理过程，无需为不同领域切换提示。实证结果表明，REGA 有效缓解了灾难性遗忘和域间混乱。与标准微调模型相比，这可以提高特定领域的性能，同时仍然保留强大的通用功能。

Title: In-Memory Learning: A Declarative Learning Framework for Large Language Models

• Authors: Bo Wang, Tianxiang Sun, Hang Yan, Siyin Wang, Qingyuan Cheng, Xipeng Qiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02757
• Pdf URL: https://arxiv.org/pdf/2403.02757
• Copy Paste: [[2403.02757]] In-Memory Learning: A Declarative Learning Framework for Large Language Models(https://arxiv.org/abs/2403.02757)
• Keywords: agent
• Abstract: The exploration of whether agents can align with their environment without relying on human-labeled data presents an intriguing research topic. Drawing inspiration from the alignment process observed in intelligent organisms, where declarative memory plays a pivotal role in summarizing past experiences, we propose a novel learning framework. The agents adeptly distill insights from past experiences, refining and updating existing notes to enhance their performance in the environment. This entire process transpires within the memory components and is implemented through natural language, so we character this framework as In-memory Learning. We also delve into the key features of benchmarks designed to evaluate the self-improvement process. Through systematic experiments, we demonstrate the effectiveness of our framework and provide insights into this problem.
• 摘要：探索智能体是否可以在不依赖人类标记数据的情况下与环境保持一致，这是一个有趣的研究课题。从智能生物体中观察到的对齐过程中汲取灵感，其中陈述性记忆在总结过去的经验中发挥着关键作用，我们提出了一种新颖的学习框架。代理熟练地从过去的经验中提取见解，完善和更新现有笔记，以提高他们在环境中的表现。整个过程发生在记忆组件内，并通过自然语言实现，因此我们将这个框架称为内存学习。我们还深入研究了旨在评估自我改进过程的基准的主要特征。通过系统的实验，我们证明了我们的框架的有效性并提供了对这个问题的见解。

Title: DPPA: Pruning Method for Large Language Model to Model Merging

• Authors: Yaochen Zhu, Rui Xia, Jiajun Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02799
• Pdf URL: https://arxiv.org/pdf/2403.02799
• Copy Paste: [[2403.02799]] DPPA: Pruning Method for Large Language Model to Model Merging(https://arxiv.org/abs/2403.02799)
• Keywords: language model
• Abstract: Model merging is to combine fine-tuned models derived from multiple domains, with the intent of enhancing the model's proficiency across various domains. The principal concern is the resolution of parameter conflicts. A substantial amount of existing research remedy this issue during the merging stage, with the latest study focusing on resolving this issue throughout the pruning stage. The DARE approach has exhibited promising outcomes when applied to a simplistic fine-tuned model. However, the efficacy of this method tends to wane when employed on complex fine-tuned models that show a significant parameter bias relative to the baseline model. In this paper, we introduce a dual-stage method termed Dynamic Pruning Partition Amplification (DPPA), devised to tackle the challenge of merging complex fine-tuned models. Initially, we introduce Dynamically Pruning (DP), an improved approach based on magnitude pruning, which aim is to enhance performance at higher pruning rates. Subsequently, we propose Dynamically Partition Amplification (DPA), a rescaling strategy, is designed to dynamically amplify parameter partitions in relation to their significance levels. The experimental results show that our method maintains a mere 20% of domain-specific parameters and yet delivers a performance comparable to other methodologies that preserve up to 90% of parameters. Furthermore, our method displays outstanding performance post-pruning, leading to a significant improvement of nearly 20% performance in model merging. We make our code on Github.
• 摘要：模型合并是将来自多个领域的经过微调的模型组合起来，旨在增强模型跨多个领域的熟练程度。主要关注的是参数冲突的解决。大量现有研究在合并阶段解决了这个问题，最新的研究重点是在整个修剪阶段解决这个问题。当 DARE 方法应用于简单的微调模型时，它表现出了有希望的结果。然而，当用于复杂的微调模型时，该方法的功效往往会减弱，这些模型相对于基线模型显示出显着的参数偏差。在本文中，我们介绍了一种称为动态修剪分区放大（DPPA）的双阶段方法，旨在解决合并复杂微调模型的挑战。首先，我们引入动态剪枝（DP），这是一种基于幅度剪枝的改进方法，其目的是在更高的剪枝率下提高性能。随后，我们提出动态分区放大（DPA），一种重新调整策略，旨在动态放大与其显着性水平相关的参数分区。实验结果表明，我们的方法仅保留了 20% 的特定领域参数，但其性能与保留高达 90% 参数的其他方法相当。此外，我们的方法在剪枝后表现出出色的性能，使模型合并的性能显着提高了近 20%。我们在 Github 上编写代码。

Title: An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Models are Task-specific Classifiers

• Authors: Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02839
• Pdf URL: https://arxiv.org/pdf/2403.02839
• Copy Paste: [[2403.02839]] An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Models are Task-specific Classifiers(https://arxiv.org/abs/2403.02839)
• Keywords: language model, gpt, llm
• Abstract: Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have employed proprietary close-source models, especially GPT4, as the evaluator. Alternatively, other works have fine-tuned judge models based on open-source LLMs as the evaluator. In this study, we conduct an empirical study of different judge models on their evaluation capability. Our findings indicate that although the fine-tuned judge models achieve high accuracy on in-domain test sets, even surpassing GPT4, they are inherently task-specific classifiers, and their generalizability and fairness severely underperform GPT4.
• 摘要：最近，利用大型语言模型（LLM）来评估其他LLM的质量的趋势越来越明显。许多研究都采用专有的闭源模型（尤其是 GPT4）作为评估器。或者，其他作品也有基于开源法学硕士作为评估者的微调判断模型。在本研究中，我们对不同法官模型的评估能力进行了实证研究。我们的研究结果表明，尽管经过微调的判断模型在域内测试集上实现了高精度，甚至超越了 GPT4，但它们本质上是特定于任务的分类器，其泛化性和公平性严重低于 GPT4。

Title: MathScale: Scaling Instruction Tuning for Mathematical Reasoning

• Authors: Zhengyang Tang, Xingxing Zhang, Benyou Wan, Furu Wei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02884
• Pdf URL: https://arxiv.org/pdf/2403.02884
• Copy Paste: [[2403.02884]] MathScale: Scaling Instruction Tuning for Mathematical Reasoning(https://arxiv.org/abs/2403.02884)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in problem-solving. However, their proficiency in solving mathematical problems remains inadequate. We propose MathScale, a simple and scalable method to create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning, it first extracts topics and knowledge points from seed math questions and then build a concept graph, which is subsequently used to generate new math questions. MathScale exhibits effective scalability along the size axis of the math dataset that we generate. As a result, we create a mathematical reasoning dataset (MathScaleQA) containing two million math question-answer pairs. To evaluate mathematical reasoning abilities of LLMs comprehensively, we construct {\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten datasets (including GSM8K and MATH) covering K-12, college, and competition level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g., LLaMA-2 and Mistral), resulting in significantly improved capabilities in mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves state-of-the-art performance across all datasets, surpassing its best peers of equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average accuracy, respectively.
• 摘要：大型语言模型（LLM）在解决问题方面表现出了卓越的能力。然而，他们解决数学问题的能力仍然不足。我们提出了 MathScale，这是一种简单且可扩展的方法，可以使用前沿 LLM（例如 {\tt GPT-3.5}）创建高质量的数学推理数据。受人类数学学习认知机制的启发，它首先从种子数学问题中提取主题和知识点，然后构建概念图，随后用于生成新的数学问题。 MathScale 沿着我们生成的数学数据集的大小轴展示了有效的可扩展性。因此，我们创建了一个包含 200 万数学问答对的数学推理数据集 (MathScaleQA)。为了全面评估法学硕士的数学推理能力，我们构建了数学应用题基准 {\sc MwpBench}，它是涵盖 K-12、大学和竞赛级别数学问题的 10 个数据集（包括 GSM8K 和 MATH）的集合。我们应用 MathScaleQA 来微调开源 LLM（例如 LLaMA-2 和 Mistral），从而显着提高数学推理能力。在 {\sc MwpBench} 上进行评估，MathScale-7B 在所有数据集上都实现了最先进的性能，在微观平均精度和宏观平均精度方面分别超过同等大小的最佳同行 42.9% 和 43.7%。

Title: In Search of Truth: An Interrogation Approach to Hallucination Detection

• Authors: Yakir Yehuda, Itzik Malkiel, Oren Barkan, Jonathan Weill, Royi Ronen, Noam Koenigstein
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02889
• Pdf URL: https://arxiv.org/pdf/2403.02889
• Copy Paste: [[2403.02889]] In Search of Truth: An Interrogation Approach to Hallucination Detection(https://arxiv.org/abs/2403.02889)
• Keywords: language model, llm, hallucination
• Abstract: Despite the many advances of Large Language Models (LLMs) and their unprecedented rapid evolution, their impact and integration into every facet of our daily lives is limited due to various reasons. One critical factor hindering their widespread adoption is the occurrence of hallucinations, where LLMs invent answers that sound realistic, yet drift away from factual truth. In this paper, we present a novel method for detecting hallucinations in large language models, which tackles a critical issue in the adoption of these models in various real-world scenarios. Through extensive evaluations across multiple datasets and LLMs, including Llama-2, we study the hallucination levels of various recent LLMs and demonstrate the effectiveness of our method to automatically detect them. Notably, we observe up to 62% hallucinations for Llama-2 in a specific experiment, where our method achieves a Balanced Accuracy (B-ACC) of 87%, all without relying on external knowledge.
• 摘要：尽管大型语言模型 (LLM) 取得了许多进步，并且发展速度前所未有，但由于各种原因，它们对我们日常生活各个方面的影响和融入仍然有限。阻碍其广泛采用的一个关键因素是幻觉的出现，法学硕士发明的答案听起来很现实，但却偏离了事实真相。在本文中，我们提出了一种在大型语言模型中检测幻觉的新方法，该方法解决了在各种现实场景中采用这些模型的关键问题。通过对多个数据集和法学硕士（包括 Llama-2）的广泛评估，我们研究了各种近期法学硕士的幻觉水平，并证明了我们自动检测它们的方法的有效性。值得注意的是，我们在特定实验中观察到 Llama-2 高达 62% 的幻觉，我们的方法实现了 87% 的平衡准确度 (B-ACC)，所有这些都无需依赖外部知识。

Title: Zero-Shot Cross-Lingual Document-Level Event Causality Identification with Heterogeneous Graph Contrastive Transfer Learning

• Authors: Zhitao He, Pengfei Cao, Yubo Chen, Kang Liu, Zhiqiang Zhang, Mengshu Sun, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02893
• Pdf URL: https://arxiv.org/pdf/2403.02893
• Copy Paste: [[2403.02893]] Zero-Shot Cross-Lingual Document-Level Event Causality Identification with Heterogeneous Graph Contrastive Transfer Learning(https://arxiv.org/abs/2403.02893)
• Keywords: gpt
• Abstract: Event Causality Identification (ECI) refers to detect causal relations between events in texts. However, most existing studies focus on sentence-level ECI with high-resource language, leaving more challenging document-level ECI (DECI) with low-resource languages under-explored. In this paper, we propose a Heterogeneous Graph Interaction Model with Multi-granularity Contrastive Transfer Learning (GIMC) for zero-shot cross-lingual document-level ECI. Specifically, we introduce a heterogeneous graph interaction network to model the long-distance dependencies between events that are scattered over document. Then, to improve cross-lingual transferability of causal knowledge learned from source language, we propose a multi-granularity contrastive transfer learning module to align the causal representations across languages. Extensive experiments show our framework outperforms previous state-of-the-art model by 9.4% and 8.2% of average F1 score on monolingual and multilingual scenarios respectively. Notably, in multilingual scenario, our zero-shot framework even exceeds GPT-3.5 with few-shot learning by 24.3% in overall performance.
• 摘要：事件因果关系识别（ECI）是指检测文本中事件之间的因果关系。然而，大多数现有研究都集中在高资源语言的句子级 ECI 上，而对低资源语言中更具挑战性的文档级 ECI (DECI) 的研究还不够充分。在本文中，我们提出了一种用于零样本跨语言文档级 ECI 的具有多粒度对比迁移学习（GIMC）的异构图交互模型。具体来说，我们引入了异构图交互网络来对分散在文档中的事件之间的长距离依赖关系进行建模。然后，为了提高从源语言学习的因果知识的跨语言可迁移性，我们提出了一种多粒度对比迁移学习模块来对齐跨语言的因果表示。大量实验表明，我们的框架在单语言和多语言场景下的平均 F1 分数分别比之前最先进的模型高出 9.4% 和 8.2%。值得注意的是，在多语言场景中，我们的零样本框架的整体性能甚至超过了少样本学习的 GPT-3.5 24.3%。

Title: Demonstrating Mutual Reinforcement Effect through Information Flow

• Authors: Chengguang Gan, Xuzheng He, Qinghao Zhang, Tatsunori Mori
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02902
• Pdf URL: https://arxiv.org/pdf/2403.02902
• Copy Paste: [[2403.02902]] Demonstrating Mutual Reinforcement Effect through Information Flow(https://arxiv.org/abs/2403.02902)
• Keywords: language model, prompt
• Abstract: The Mutual Reinforcement Effect (MRE) investigates the synergistic relationship between word-level and text-level classifications in text classification tasks. It posits that the performance of both classification levels can be mutually enhanced. However, this mechanism has not been adequately demonstrated or explained in prior research. To address this gap, we employ information flow analysis to observe and substantiate the MRE theory. Our experiments on six MRE hybrid datasets revealed the presence of MRE in the model and its impact. Additionally, we conducted fine-tuning experiments, whose results were consistent with those of the information flow experiments. The convergence of findings from both experiments corroborates the existence of MRE. Furthermore, we extended the application of MRE to prompt learning, utilizing word-level information as a verbalizer to bolster the model's prediction of text-level classification labels. In our final experiment, the F1-score significantly surpassed the baseline in five out of six datasets, further validating the notion that word-level information enhances the language model's comprehension of the text as a whole.
• 摘要：相互强化效应（MRE）研究文本分类任务中单词级和文本级分类之间的协同关系。它假设两个分类级别的性能可以相互增强。然而，这种机制在先前的研究中尚未得到充分证明或解释。为了解决这一差距，我们采用信息流分析来观察和证实 MRE 理论。我们对六个 MRE 混合数据集的实验揭示了模型中 MRE 的存在及其影响。此外，我们还进行了微调实验，其结果与信息流实验的结果一致。两个实验结果的融合证实了 MRE 的存在。此外，我们将 MRE 的应用扩展到促进学习，利用单词级信息作为语言表达器来支持模型对文本级分类标签的预测。在我们的最终实验中，六个数据集中有五个数据集的 F1 分数显着超过了基线，进一步验证了单词级信息增强了语言模型对整个文本的理解的概念。

Title: RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules

• Authors: Miaomiao Li, Jiaqi Zhu, Yang Wang, Yi Yang, Yilin Li, Hongan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02932
• Pdf URL: https://arxiv.org/pdf/2403.02932
• Copy Paste: [[2403.02932]] RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules(https://arxiv.org/abs/2403.02932)
• Keywords: language model, prompt
• Abstract: Weakly supervised text classification (WSTC), also called zero-shot or dataless text classification, has attracted increasing attention due to its applicability in classifying a mass of texts within the dynamic and open Web environment, since it requires only a limited set of seed words (label names) for each category instead of labeled data. With the help of recently popular prompting Pre-trained Language Models (PLMs), many studies leveraged manually crafted and/or automatically identified verbalizers to estimate the likelihood of categories, but they failed to differentiate the effects of these category-indicative words, let alone capture their correlations and realize adaptive adjustments according to the unlabeled corpus. In this paper, in order to let the PLM effectively understand each category, we at first propose a novel form of rule-based knowledge using logical expressions to characterize the meanings of categories. Then, we develop a prompting PLM-based approach named RulePrompt for the WSTC task, consisting of a rule mining module and a rule-enhanced pseudo label generation module, plus a self-supervised fine-tuning module to make the PLM align with this task. Within this framework, the inaccurate pseudo labels assigned to texts and the imprecise logical rules associated with categories mutually enhance each other in an alternative manner. That establishes a self-iterative closed loop of knowledge (rule) acquisition and utilization, with seed words serving as the starting point. Extensive experiments validate the effectiveness and robustness of our approach, which markedly outperforms state-of-the-art weakly supervised methods. What is more, our approach yields interpretable category rules, proving its advantage in disambiguating easily-confused categories.
• 摘要：弱监督文本分类（WSTC），也称为零样本或无数据文本分类，由于其仅需要一组有限的种子词而适用于在动态和开放的 Web 环境中对大量文本进行分类，因此引起了越来越多的关注（标签名称）每个类别而不是标记数据。在最近流行的提示预训练语言模型（PLM）的帮助下，许多研究利用手工制作和/或自动识别的语言器来估计类别的可能性，但他们未能区分这些类别指示词的效果，更不用说捕获它们的相关性并根据未标记的语料库实现自适应调整。在本文中，为了让 PLM 有效地理解每个类别，我们首先提出了一种基于规则的知识的新形式，使用逻辑表达式来表征类别的含义。然后，我们为 WSTC 任务开发了一种名为 RulePrompt 的基于 PLM 的提示方法，由规则挖掘模块和规则增强的伪标签生成模块组成，再加上一个自监督微调模块以使 PLM 与该任务保持一致。在这个框架内，分配给文本的不准确的伪标签和与类别相关的不精确的逻辑规则以另一种方式相互增强。这就建立了一个以种子词为起点的知识（规则）获取和利用的自我迭代闭环。大量的实验验证了我们方法的有效性和鲁棒性，该方法明显优于最先进的弱监督方法。更重要的是，我们的方法产生了可解释的类别规则，证明了其在消除容易混淆的类别方面的优势。

Title: Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation

• Authors: Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, Hangyu Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02951
• Pdf URL: https://arxiv.org/pdf/2403.02951
• Copy Paste: [[2403.02951]] Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation(https://arxiv.org/abs/2403.02951)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods. Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions.To address the aforementioned issues, we firstly construct a new dataset designed to mitigate the risk of overfitting in LLMs. Then we formulate five evaluation tasks to comprehensively assess the performance of diverse methods across various LLMs throughout the Text-to-SQL process.Our study highlights the performance disparities among LLMs and proposes optimal in-context learning solutions tailored to each task. These findings offer valuable insights for enhancing the development of LLM-based Text-to-SQL systems.
• 摘要：大型语言模型 (LLM) 已成为推进文本到 SQL 任务的强大工具，其性能显着优于传统方法。然而，作为一个新兴的研究领域，最佳的提示模板和设计框架仍然没有达成共识。此外，现有的基准测试没有充分探索LLM在文本到SQL过程的各个子任务中的性能，这阻碍了LLM认知能力的评估和基于LLM的解决方案的优化。为了解决上述问题，我们首先构建一个新的数据集，旨在减轻法学硕士中过度拟合的风险。然后，我们制定了五个评估任务，以全面评估整个文本到 SQL 过程中各种法学硕士的不同方法的性能。我们的研究强调了法学硕士之间的性能差异，并提出了针对每个任务量身定制的最佳上下文学习解决方案。这些发现为加强基于 LLM 的文本到 SQL 系统的开发提供了宝贵的见解。

Title: SimuCourt: Building Judicial Decision-Making Agents with Real-world Judgement Documents

• Authors: Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02959
• Pdf URL: https://arxiv.org/pdf/2403.02959
• Copy Paste: [[2403.02959]] SimuCourt: Building Judicial Decision-Making Agents with Real-world Judgement Documents(https://arxiv.org/abs/2403.02959)
• Keywords: language model, agent
• Abstract: With the development of deep learning, natural language processing technology has effectively improved the efficiency of various aspects of the traditional judicial industry. However, most current efforts focus solely on individual judicial stage, overlooking cross-stage collaboration. As the autonomous agents powered by large language models are becoming increasingly smart and able to make complex decisions in real-world settings, offering new insights for judicial intelligence. In this paper, (1) we introduce SimuCourt, a judicial benchmark that encompasses 420 judgment documents from real-world, spanning the three most common types of judicial cases, and a novel task Judicial Decision-Making to evaluate the judicial analysis and decision-making power of agents. To support this task, we construct a large-scale judicial knowledge base, JudicialKB, with multiple legal knowledge. (2) we propose a novel multi-agent framework, AgentsCourt. Our framework follows the real-world classic court trial process, consisting of court debate simulation, legal information retrieval and judgement refinement to simulate the decision-making of judge. (3) we perform extensive experiments, the results demonstrate that, our framework outperforms the existing advanced methods in various aspects, especially in generating legal grounds, where our model achieves significant improvements of 8.6% and 9.1% F1 score in the first and second instance settings, respectively.
• 摘要：随着深度学习的发展，自然语言处理技术有效提升了传统司法行业各方面的效率。然而，目前大多数努力只关注个别司法阶段，忽视了跨阶段合作。随着大型语言模型支持的自主代理变得越来越聪明，能够在现实世界中做出复杂的决策，为司法情报提供了新的见解。在本文中，（1）我们介绍了 SimuCourt，这是一个司法基准，包含来自现实世界的 420 份判决文件，涵盖三种最常见的司法案件类型，以及一个新的任务“司法决策”来评估司法分析和决策 -发挥代理人的力量。为了支持这项任务，我们构建了一个包含多种法律知识的大型司法知识库 JudicialKB。 (2)我们提出了一种新颖的多智能体框架，AgentsCourt。我们的框架遵循现实世界的经典法庭审判流程，由法庭辩论模拟、法律信息检索和判决细化组成，以模拟法官的决策。 （3）我们进行了大量的实验，结果表明，我们的框架在各个方面都优于现有的先进方法，特别是在生成法律依据方面，我们的模型在第一和第二个实例中实现了 8.6% 和 9.1% F1 分数的显着改进分别设置。

Title: Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

• Authors: Sungho Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02966
• Pdf URL: https://arxiv.org/pdf/2403.02966
• Copy Paste: [[2403.02966]] Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering(https://arxiv.org/abs/2403.02966)
• Keywords: language model, llm
• Abstract: Recent studies have investigated utilizing Knowledge Graphs (KGs) to enhance Quesetion Answering (QA) performance of Large Language Models (LLMs), yet structured KG verbalization remains challengin. Existing methods, such as triple-form or free-form textual conversion of triple-form facts, encounter several issues. These include reduced evidence density due to duplicated entities or relationships, and reduced evidence clarity due to an inability to emphasize crucial evidence. To address these issues, we propose EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs. We optimize an open-source LLM as a fact summarizer through distillation and preference alignment. Our extensive experiments show that EFSum improves LLM's zero-shot QA performance, and it is possible to ensure both the helpfulness and faithfulness of the summary.
• 摘要：最近的研究调查了利用知识图 (KG) 来增强大型语言模型 (LLM) 的问答 (QA) 性能，但结构化 KG 语言表达仍然具有挑战性。现有的方法，例如三重形式事实的三重形式或自由形式文本转换，遇到了几个问题。其中包括由于重复的实体或关系而导致证据密度降低，以及由于无法强调关键证据而降低证据清晰度。为了解决这些问题，我们提出了 EFSum，这是一个以证据为中心的事实总结框架，用于通过知识增强的法学硕士来增强质量保证。我们通过蒸馏和偏好调整来优化开源法学硕士作为事实总结器。我们大量的实验表明，EFSum 提高了 LLM 的零样本 QA 性能，并且可以确保摘要的有用性和忠实性。

Title: A General and Flexible Multi-concept Parsing Framework for Multilingual Semantic Matching

• Authors: Dong Yao, Asaad Alghamdi, Qingrong Xia, Xiaoye Qu, Xinyu Duan, Zhefeng Wang, Yi Zheng, Baoxing Huai, Peilun Cheng, Zhou Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02975
• Pdf URL: https://arxiv.org/pdf/2403.02975
• Copy Paste: [[2403.02975]] A General and Flexible Multi-concept Parsing Framework for Multilingual Semantic Matching(https://arxiv.org/abs/2403.02975)
• Keywords: language model, chat
• Abstract: Sentence semantic matching is a research hotspot in natural language processing, which is considerably significant in various key scenarios, such as community question answering, searching, chatbot, and recommendation. Since most of the advanced models directly model the semantic relevance among words between two sentences while neglecting the \textit{keywords} and \textit{intents} concepts of them, DC-Match is proposed to disentangle keywords from intents and utilizes them to optimize the matching performance. Although DC-Match is a simple yet effective method for semantic matching, it highly depends on the external NER techniques to identify the keywords of sentences, which limits the performance of semantic matching for minor languages since satisfactory NER tools are usually hard to obtain. In this paper, we propose to generally and flexibly resolve the text into multi concepts for multilingual semantic matching to liberate the model from the reliance on NER models. To this end, we devise a \underline{M}ulti-\underline{C}oncept \underline{P}arsed \underline{S}emantic \underline{M}atching framework based on the pre-trained language models, abbreviated as \textbf{MCP-SM}, to extract various concepts and infuse them into the classification tokens. We conduct comprehensive experiments on English datasets QQP and MRPC, and Chinese dataset Medical-SM. Besides, we experiment on Arabic datasets MQ2Q and XNLI, the outstanding performance further prove MCP-SM's applicability in low-resource languages.
• 摘要：句子语义匹配是自然语言处理的研究热点，在社区问答、搜索、聊天机器人、推荐等各种关键场景中具有重要意义。由于大多数先进模型直接对两个句子之间的单词之间的语义相关性进行建模，而忽略了它们的 \textit{keywords} 和 \textit{intents} 概念，因此提出了 DC-Match 将关键字与意图分开，并利用它们来优化匹配性能。尽管DC-Match是一种简单而有效的语义匹配方法，但它高度依赖外部NER技术来识别句子的关键词，这限制了小语种语义匹配的性能，因为通常很难获得令人满意的NER工具。在本文中，我们建议通用且灵活地将文本解析为多概念以进行多语言语义匹配，以将模型从对NER模型的依赖中解放出来。为此，我们基于预先训练的语言模型设计了一个 \underline{M}ulti-\underline{C}oncept \underline{P}arsed \underline{S}emantic \underline{M}atching 框架，缩写为\textbf{MCP-SM}，提取各种概念并将其注入分类标记中。我们对英文数据集 QQP 和 MRPC 以及中文数据集 Medical-SM 进行了全面的实验。此外，我们在阿拉伯语数据集MQ2Q和XNLI上进行了实验，出色的性能进一步证明了MCP-SM在低资源语言中的适用性。

Title: Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges

• Authors: Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, Shafiq Joty
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02990
• Pdf URL: https://arxiv.org/pdf/2403.02990
• Copy Paste: [[2403.02990]] Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges(https://arxiv.org/abs/2403.02990)
• Keywords: language model, llm
• Abstract: In the rapidly evolving field of machine learning (ML), data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of Large Language Models (LLMs) on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond. From a data perspective and a learning perspective, we examine various strategies that utilize Large Language Models for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for further training. Additionally, this paper delineates the primary challenges faced in this domain, ranging from controllable data augmentation to multi modal data augmentation. This survey highlights the paradigm shift introduced by LLMs in DA, aims to serve as a foundational guide for researchers and practitioners in this field.
• 摘要：在快速发展的机器学习 (ML) 领域，数据增强 (DA) 已成为一种关键技术，可通过多样化训练示例来增强模型性能，而无需额外的数据收集。本调查探讨了大型语言模型 (LLM) 对 DA 的变革性影响，特别是解决它们在自然语言处理 (NLP) 及其他领域带来的独特挑战和机遇。从数据角度和学习角度，我们研究了利用大型语言模型进行数据增强的各种策略，包括对学习范式的新颖探索，其中法学硕士生成的数据用于进一步训练。此外，本文还描述了该领域面临的主要挑战，从可控数据增强到多模态数据增强。这项调查强调了法学硕士在 DA 中引入的范式转变，旨在为该领域的研究人员和从业者提供基础指南。

Title: Socratic Reasoning Improves Positive Text Rewriting

• Authors: Anmol Goel, Nico Daheim, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03029
• Pdf URL: https://arxiv.org/pdf/2403.03029
• Copy Paste: [[2403.03029]] Socratic Reasoning Improves Positive Text Rewriting(https://arxiv.org/abs/2403.03029)
• Keywords: language model, llm
• Abstract: Reframing a negative into a positive thought is at the crux of several cognitive approaches to mental health and psychotherapy that could be made more accessible by large language model-based solutions. Such reframing is typically non-trivial and requires multiple rationalization steps to uncover the underlying issue of a negative thought and transform it to be more positive. However, this rationalization process is currently neglected by both datasets and models which reframe thoughts in one step. In this work, we address this gap by augmenting open-source datasets for positive text rewriting with synthetically-generated Socratic rationales using a novel framework called \textsc{SocraticReframe}. \textsc{SocraticReframe} uses a sequence of question-answer pairs to rationalize the thought rewriting process. We show that such Socratic rationales significantly improve positive text rewriting for different open-source LLMs according to both automatic and human evaluations guided by criteria from psychotherapy research.
• 摘要：将消极想法重新定义为积极想法是心理健康和心理治疗的几种认知方法的关键，通过基于大型语言模型的解决方案可以使这些方法变得更容易实现。这种重新构建通常非常重要，需要多个合理化步骤来揭示消极想法的根本问题并将其转变为更加积极的想法。然而，这一合理化过程目前被一步重构思想的数据集和模型所忽视。在这项工作中，我们通过使用名为 \textsc{SocraticReframe} 的新颖框架，通过综合生成的苏格拉底原理来增强开源数据集以进行积极的文本重写，从而解决了这一差距。 \textsc{SocraticReframe} 使用一系列问答对来合理化思维重写过程。我们表明，根据心理治疗研究标准指导下的自动评估和人工评估，这种苏格拉底式的基本原理显着改善了不同开源法学硕士的积极文本重写。

Title: Learning to Use Tools via Cooperative and Interactive Agents

• Authors: Zhengliang Shi, Shen Gao, Xiuyi Chen, Lingyong Yan, Haibo Shi, Dawei Yin, Zhumin Chen, Pengjie Ren, Suzan Verberne, Zhaochun Ren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03031
• Pdf URL: https://arxiv.org/pdf/2403.03031
• Copy Paste: [[2403.03031]] Learning to Use Tools via Cooperative and Interactive Agents(https://arxiv.org/abs/2403.03031)
• Keywords: language model, llm, agent
• Abstract: Tool learning empowers large language models (LLMs) as agents to use external tools to extend their capability. Existing methods employ one single LLM-based agent to iteratively select and execute tools, thereafter incorporating the result into the next action prediction. However, they still suffer from potential performance degradation when addressing complex tasks due to: (1) the limitation of the inherent capability of a single LLM to perform diverse actions, and (2) the struggle to adaptively correct mistakes when the task fails. To mitigate these problems, we propose the ConAgents, a Cooperative and interactive Agents framework, which modularizes the workflow of tool learning into Grounding, Execution, and Observing agents. We also introduce an iterative calibration (IterCali) method, enabling the agents to adapt themselves based on the feedback from the tool environment. Experiments conducted on three datasets demonstrate the superiority of our ConAgents (e.g., 6 point improvement over the SOTA baseline). We further provide fine-granularity analysis for the efficiency and consistency of our framework.
• 摘要：工具学习使大型语言模型（LLM）能够作为代理使用外部工具来扩展其能力。现有方法采用一个基于 LLM 的代理来迭代选择和执行工具，然后将结果合并到下一个动作预测中。然而，在处理复杂任务时，它们仍然面临潜在的性能下降，原因是：（1）单个法学硕士执行不同操作的固有能力有限，以及（2）任务失败时难以自适应地纠正错误。为了缓解这些问题，我们提出了 ConAgents，这是一个协作和交互式代理框架，它将工具学习的工作流程模块化为基础代理、执行代理和观察代理。我们还引入了迭代校准（IterCali）方法，使代理能够根据工具环境的反馈进行自我调整。在三个数据集上进行的实验证明了我们的 ConAgents 的优越性（例如，比 SOTA 基线提高了 6 个点）。我们进一步为我们框架的效率和一致性提供细粒度分析。

Title: KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

• Authors: Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Ningyu Zhang, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.03101
• Pdf URL: https://arxiv.org/pdf/2403.03101
• Copy Paste: [[2403.03101]] KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents(https://arxiv.org/abs/2403.03101)
• Keywords: language model, llm, hallucination, agent
• Abstract: Large Language Models (LLMs) have demonstrated great potential in complex reasoning tasks, yet they fall short when tackling more sophisticated challenges, especially when interacting with environments through generating executable actions. This inadequacy primarily stems from the lack of built-in action knowledge in language agents, which fails to effectively guide the planning trajectories during task solving and results in planning hallucination. To address this issue, we introduce KnowAgent, a novel approach designed to enhance the planning capabilities of LLMs by incorporating explicit action knowledge. Specifically, KnowAgent employs an action knowledge base and a knowledgeable self-learning strategy to constrain the action path during planning, enabling more reasonable trajectory synthesis, and thereby enhancing the planning performance of language agents. Experimental results on HotpotQA and ALFWorld based on various backbone models demonstrate that KnowAgent can achieve comparable or superior performance to existing baselines. Further analysis indicates the effectiveness of KnowAgent in terms of planning hallucinations mitigation. Code is available in https://github.com/zjunlp/KnowAgent.
• 摘要：大型语言模型 (LLM) 在复杂的推理任务中表现出了巨大的潜力，但在应对更复杂的挑战时，尤其是在通过生成可执行操作与环境交互时，它们表现不佳。这种不足主要源于语言智能体缺乏内置的动作知识，无法有效指导任务解决过程中的规划轨迹，导致规划幻觉。为了解决这个问题，我们引入了 KnowAgent，这是一种新颖的方法，旨在通过结合明确的行动知识来增强法学硕士的规划能力。具体来说，KnowAgent采用动作知识库和知识自学习策略来约束规划过程中的动作路径，从而实现更合理的轨迹合成，从而提高语言代理的规划性能。基于各种骨干模型的 HotpotQA 和 ALFWorld 的实验结果表明，KnowAgent 可以实现与现有基线相当或更好的性能。进一步的分析表明 KnowAgent 在规划幻觉缓解方面的有效性。代码可在 https://github.com/zjunlp/KnowAgent 中找到。

Title: "In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning

• Authors: Chuanqi Cheng, Quan Tu, Wei Wu, Shuo Shang, Cunli Mao, Zhengtao Yu, Rui Yan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.03102
• Pdf URL: https://arxiv.org/pdf/2403.03102
• Copy Paste: [[2403.03102]] "In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning(https://arxiv.org/abs/2403.03102)
• Keywords: language model
• Abstract: Personalized dialogue systems have gained significant attention in recent years for their ability to generate responses in alignment with different personas. However, most existing approaches rely on pre-defined personal profiles, which are not only time-consuming and labor-intensive to create but also lack flexibility. We propose In-Dialogue Learning (IDL), a fine-tuning framework that enhances the ability of pre-trained large language models to leverage dialogue history to characterize persona for completing personalized dialogue generation tasks without pre-defined profiles. Our experiments on three datasets demonstrate that IDL brings substantial improvements, with BLEU and ROUGE scores increasing by up to 200% and 247%, respectively. Additionally, the results of human evaluations further validate the efficacy of our proposed method.
• 摘要：近年来，个性化对话系统因其能够根据不同角色生成响应的能力而受到广泛关注。然而，大多数现有方法依赖于预先定义的个人档案，这不仅创建起来费时费力，而且缺乏灵活性。我们提出了对话中学习（IDL），这是一种微调框架，可增强预训练大型语言模型的能力，利用对话历史来表征人物角色，从而在没有预定义配置文件的情况下完成个性化对话生成任务。我们在三个数据集上的实验表明，IDL 带来了显着的改进，BLEU 和 ROUGE 分数分别提高了 200% 和 247%。此外，人类评估的结果进一步验证了我们提出的方法的有效性。

Title: Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution

• Authors: Flor Miriam Plaza-del-Arco, Amanda Cercas Curry, Alba Curry, Gavin Abercrombie, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03121
• Pdf URL: https://arxiv.org/pdf/2403.03121
• Copy Paste: [[2403.03121]] Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution(https://arxiv.org/abs/2403.03121)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) reflect societal norms and biases, especially about gender. While societal biases and stereotypes have been extensively researched in various NLP applications, there is a surprising gap for emotion analysis. However, emotion and gender are closely linked in societal discourse. E.g., women are often thought of as more empathetic, while men's anger is more socially accepted. To fill this gap, we present the first comprehensive study of gendered emotion attribution in five state-of-the-art LLMs (open- and closed-source). We investigate whether emotions are gendered, and whether these variations are based on societal stereotypes. We prompt the models to adopt a gendered persona and attribute emotions to an event like 'When I had a serious argument with a dear person'. We then analyze the emotions generated by the models in relation to the gender-event pairs. We find that all models consistently exhibit gendered emotions, influenced by gender stereotypes. These findings are in line with established research in psychology and gender studies. Our study sheds light on the complex societal interplay between language, gender, and emotion. The reproduction of emotion stereotypes in LLMs allows us to use those models to study the topic in detail, but raises questions about the predictive use of those same LLMs for emotion applications.
• 摘要：大型语言模型（LLM）反映了社会规范和偏见，尤其是关于性别的规范和偏见。尽管社会偏见和刻板印象已在各种 NLP 应用中得到广泛研究，但情感分析仍存在令人惊讶的差距。然而，情感和性别在社会话语中密切相关。例如，女性通常被认为更具同理心，而男性的愤怒更容易被社会接受。为了填补这一空白，我们在五个最先进的法学硕士（开源和闭源）中首次对性别情感归因进行了全面研究。我们调查情绪是否有性别差异，以及这些差异是否基于社会刻板印象。我们提示模型采用性别角色，并将情绪归因于诸如“当我与亲爱的人发生严重争执时”之类的事件。然后，我们分析模型产生的与性别事件对相关的情绪。我们发现，受性别刻板印象的影响，所有模特都始终表现出性别情感。这些发现与心理学和性别研究的既定研究一致。我们的研究揭示了语言、性别和情感之间复杂的社会相互作用。法学硕士中情感刻板印象的再现使我们能够使用这些模型来详细研究该主题，但也提出了关于这些相同的法学硕士在情感应用中的预测性使用的问题。

Title: CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following

• Authors: Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi, Ning Ding, Bowen Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03129
• Pdf URL: https://arxiv.org/pdf/2403.03129
• Copy Paste: [[2403.03129]] CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following(https://arxiv.org/abs/2403.03129)
• Keywords: language model
• Abstract: With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.
• 摘要：随着语言模型 (LM) 的进步，它们对私人数据的暴露越来越不可避免，并且它们在个人设备（例如 PC 和智能手机）上的部署（尤其是较小的模型）已成为流行趋势。在充满用户信息的环境中，使模型能够保护用户隐私并有效执行命令成为一项重要的研究当务之急。在本文中，我们提出了 CoGenesis，一个集成大型（托管在云基础设施上）和小型模型（部署在本地设备上）的协作生成框架，以逻辑地解决隐私问题。最初，我们设计了一个管道来创建个性化的写作指导数据集，其中包含丰富的上下文细节，作为本研究问题的测试平台。随后，我们介绍了分别基于 sketch 和 logits 的 CoGenesis 的两个变体。我们的实验结果基于我们的合成数据集和两个额外的开源数据集，表明：1）大型模型在提供用户上下文时表现良好，但在缺乏此类上下文时表现不佳。 2）虽然在合成数据集上进行微调的专用较小模型显示出了希望，但它们仍然落后于较大的模型。 3）我们的CoGenesis框架利用混合规模模型，展示了具有竞争力的性能，为隐私问题提供了可行的解决方案。

Title: Language Guided Exploration for RL Agents in Text Environments

• Authors: Hitesh Golchha, Sahil Yerawar, Dhruvesh Patel, Soham Dan, Keerthiram Murugesan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03141
• Pdf URL: https://arxiv.org/pdf/2403.03141
• Copy Paste: [[2403.03141]] Language Guided Exploration for RL Agents in Text Environments(https://arxiv.org/abs/2403.03141)
• Keywords: language model, llm, agent
• Abstract: Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like $\textit{tabula rasa}$ reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE ) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al.,2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.
• 摘要：现实世界的顺序决策的特点是奖励稀疏和决策空间大，这给像$\textit{tabula rasa}$强化学习（RL）代理这样的体验式学习系统带来了很大的困难。大型语言模型（LLM）具有丰富的世界知识，可以帮助强化学习智能体快速学习并适应分布变化。在这项工作中，我们引入了语言引导探索（LGE）框架，它使用预先训练的语言模型（称为 GUIDE ）为 RL 代理（称为 EXPLORER ）提供决策级指导。我们观察到，在具有挑战性的文本环境 ScienceWorld（Wang 等人，2022）上，LGE 显着优于普通 RL 代理，并且也优于其他复杂方法，例如行为克隆和文本决策转换器。

Title: Design2Code: How Far Are We From Automating Front-End Engineering?

• Authors: Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, Diyi Yang
• Subjects: cs.CL, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.03163
• Pdf URL: https://arxiv.org/pdf/2403.03163
• Copy Paste: [[2403.03163]] Design2Code: How Far Are We From Automating Front-End Engineering?(https://arxiv.org/abs/2403.03163)
• Keywords: gpt, llm, prompt
• Abstract: Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in which multimodal LLMs might directly convert visual designs into code implementations. In this work, we formalize this as a Design2Code task and conduct comprehensive benchmarking. Specifically, we manually curate a benchmark of 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations. We develop a suite of multimodal prompting methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We further finetune an open-source Design2Code-18B model that successfully matches the performance of Gemini Pro Vision. Both human evaluation and automatic metrics show that GPT-4V performs the best on this task compared to other models. Moreover, annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages are considered better than the original reference webpages. Our fine-grained break-down metrics indicate that open-source models mostly lag in recalling visual elements from the input webpages and in generating correct layout designs, while aspects like text content and coloring can be drastically improved with proper finetuning.
• 摘要：近年来，生成式人工智能取得了快速发展，在多模式理解和代码生成方面实现了前所未有的能力。这可以实现前端开发的新范例，其中多模式法学硕士可以直接将视觉设计转换为代码实现。在这项工作中，我们将其形式化为 Design2Code 任务并进行全面的基准测试。具体来说，我们手动策划了 484 个不同的现实世界网页作为测试用例的基准，并开发了一组自动评估指标，以评估当前的多模式 LLM 生成直接渲染到给定参考网页的代码实现的能力，给定的屏幕截图如下输入。我们还通过全面的人工评估来补充自动指标。我们开发了一套多模式提示方法，并在 GPT-4V 和 Gemini Pro Vision 上展示了它们的有效性。我们进一步微调开源 Design2Code-18B 模型，成功匹配 Gemini Pro Vision 的性能。人工评估和自动指标都表明，与其他模型相比，GPT-4V 在这项任务上表现最好。此外，注释者认为 GPT-4V 生成的网页在视觉外观和内容方面可以在 49% 的情况下取代原始参考网页；也许令人惊讶的是，在 64% 的情况下，GPT-4V 生成的网页被认为比原始参考网页更好。我们的细粒度细分指标表明，开源模型在从输入网页调用视觉元素和生成正确的布局设计方面大多滞后，而文本内容和着色等方面可以通过适当的微调得到显着改善。

Title: PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset

• Authors: Arda Uzunoğlu, Abdalfatah Rashid Safa, Gözde Gül Şahin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03167
• Pdf URL: https://arxiv.org/pdf/2403.03167
• Copy Paste: [[2403.03167]] PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset(https://arxiv.org/abs/2403.03167)
• Keywords: language model, gpt, llm, prompt
• Abstract: Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q\&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://github.com/GGLAB-KU/paradise.
• 摘要：最近，社区内对于大型语言模型是否能够规划或执行计划越来越感兴趣。然而，大多数先前的研究使用法学硕士为缺乏语言复杂性和领域多样性的简化场景生成高级计划，限制了对其规划能力的分析。这些设置限制了评估方法（例如，预定义的动作空间）、架构选择（例如，仅生成模型），并且忽略了现实分析所必需的语言细微差别。为了解决这个问题，我们提出了 PARADISE，这是一项对源自 wikiHow 的实际程序文本使用 Q\&A 格式的溯因推理任务。它涉及与目标直接相关的警告和提示推理任务，不包括中间步骤，目的是测试模型仅从给定目标推断计划隐式知识的能力。我们的实验利用微调语言模型和零样本提示，揭示了在大多数情况下特定于任务的小模型相对于大型语言模型的有效性。尽管取得了进步，但所有模型都未能达到人类的表现。值得注意的是，我们的分析揭示了有趣的见解，例如模型行为因关键词被删除而发生的变化、BERT-family 和 GPT-4 在物理和抽象目标方面的挣扎，以及所提出的任务为其他看不见的程序任务提供了宝贵的先验知识。 PARADISE 数据集和相关资源可通过 https://github.com/GGLAB-KU/paradise 公开获取，以供进一步的研究探索。

Title: Reliable, Adaptable, and Attributable Language Models with Retrieval

• Authors: Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, Wen-tau Yih
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03187
• Pdf URL: https://arxiv.org/pdf/2403.03187
• Copy Paste: [[2403.03187]] Reliable, Adaptable, and Attributable Language Models with Retrieval(https://arxiv.org/abs/2403.03187)
• Keywords: language model, hallucination
• Abstract: Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.
• 摘要：参数语言模型 (LM) 经过大量网络数据的训练，表现出卓越的灵活性和功能。然而，他们仍然面临着幻觉、难以适应新数据分布、缺乏可验证性等实际挑战。在这篇立场文件中，我们主张使用检索增强型语言模型来取代参数型语言模型，成为下一代语言模型。通过在推理过程中合并大规模数据存储，检索增强的 LM 可以更加可靠、适应性强和可归因。尽管具有潜力，但由于一些障碍，检索增强型语言模型尚未得到广泛采用：具体来说，当前的检索增强型语言模型很难利用知识密集型任务（例如回答问题）之外的有用文本，检索和语言模型组件之间的交互有限，并且缺乏扩展的基础设施。为了解决这些问题，我们提出了开发通用检索增强型语言模型的路线图。这涉及到重新考虑数据存储和检索器、探索改进检索器与LM交互的管道，以及对基础设施进行大量投资以实现高效的训练和推理。

Title: MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets

• Authors: Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Justin Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03194
• Pdf URL: https://arxiv.org/pdf/2403.03194
• Copy Paste: [[2403.03194]] MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets(https://arxiv.org/abs/2403.03194)
• Keywords: llm
• Abstract: Development of multimodal interactive systems is hindered by the lack of rich, multimodal (text, images) conversational data, which is needed in large quantities for LLMs. Previous approaches augment textual dialogues with retrieved images, posing privacy, diversity, and quality constraints. In this work, we introduce \textbf{M}ultimodal \textbf{A}ugmented \textbf{G}enerative \textbf{I}mages \textbf{D}ialogues (MAGID), a framework to augment text-only dialogues with diverse and high-quality images. Subsequently, a diffusion model is applied to craft corresponding images, ensuring alignment with the identified text. Finally, MAGID incorporates an innovative feedback loop between an image description generation module (textual LLM) and image quality modules (addressing aesthetics, image-text matching, and safety), that work in tandem to generate high-quality and multi-modal dialogues. We compare MAGID to other SOTA baselines on three dialogue datasets, using automated and human evaluation. Our results show that MAGID is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small.
• 摘要：缺乏丰富的多模式（文本、图像）对话数据阻碍了多模式交互系统的发展，而法学硕士需要大量的数据。以前的方法通过检索到的图像来增强文本对话，从而带来隐私、多样性和质量限制。在这项工作中，我们引入了 \textbf{M}ultimodal \textbf{A}ugmented \textbf{G}enerative \textbf{I}mages \textbf{D}ialogues (MAGID)，一个框架，用于增强纯文本对话与各种和高质量的图像。随后，应用扩散模型来制作相应的图像，确保与识别的文本对齐。最后，MAGID 在图像描述生成模块（文本 LLM）和图像质量模块（解决美观、图像文本匹配和安全性）之间融入了创新的反馈循环，它们协同工作以生成高质量的多模式对话。我们使用自动和人工评估，在三个对话数据集上将 MAGID 与其他 SOTA 基线进行比较。我们的结果表明，MAGID 与基线相当或更好，在人类评估方面有显着改进，特别是相对于图像数据库较小的检索基线。