2024-09-12

Title: Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts

• Authors: Eleftheria Briakou, Jiaming Luo, Colin Cherry, Markus Freitag
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06790
• Pdf URL: https://arxiv.org/pdf/2409.06790
• Copy Paste: [[2409.06790]] Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts(https://arxiv.org/abs/2409.06790)
• Keywords: language model, prompt
• Abstract: In this paper we present a step-by-step approach to long-form text translation, drawing on established processes in translation studies. Instead of viewing machine translation as a single, monolithic task, we propose a framework that engages language models in a multi-turn interaction, encompassing pre-translation research, drafting, refining, and proofreading, resulting in progressively improved translations. Extensive automatic evaluations using Gemini 1.5 Pro across ten language pairs show that translating step-by-step yields large translation quality improvements over conventional zero-shot prompting approaches and earlier human-like baseline strategies, resulting in state-of-the-art results on WMT2024.
• 摘要：在本文中，我们借鉴翻译研究中已建立的流程，提出了一种长篇文本翻译的分步方法。我们并不将机器翻译视为一项单一的整体任务，而是提出了一个框架，让语言模型参与多轮交互，包括翻译前研究、起草、改进和校对，从而逐步改进翻译。使用 Gemini 1.5 Pro 对十种语言对进行广泛的自动评估表明，与传统的零样本提示方法和早期的类人基线策略相比，分步翻译可以大大提高翻译质量，从而在 WMT2024 上取得最佳结果。

Title: PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation

• Authors: Ilya Gusev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06820
• Pdf URL: https://arxiv.org/pdf/2409.06820
• Copy Paste: [[2409.06820]] PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation(https://arxiv.org/abs/2409.06820)
• Keywords: language model
• Abstract: We introduce a novel benchmark for evaluating the role-playing capabilities of language models. Our approach leverages language models themselves to emulate users in dynamic, multi-turn conversations and to assess the resulting dialogues. The framework consists of three main components: a player model assuming a specific character role, an interrogator model simulating user behavior, and a judge model evaluating conversation quality. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of model capabilities in interactive scenarios.
• 摘要：我们引入了一种用于评估语言模型角色扮演能力的新基准。我们的方法利用语言模型本身来模拟动态、多轮对话中的用户并评估由此产生的对话。该框架由三个主要组件组成：假设特定角色的玩家模型、模拟用户行为的询问者模型和评估对话质量的判断模型。我们进行了实验，将自动评估与人工注释进行比较以验证我们的方法，结果显示多个标准之间存在很强的相关性。这项工作为在交互式场景中对模型能力进行稳健而动态的评估奠定了基础。

Title: What is the Role of Small Models in the LLM Era: A Survey

• Authors: Lihu Chen, Gaël Varoquaux
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06857
• Pdf URL: https://arxiv.org/pdf/2409.06857
• Copy Paste: [[2409.06857]] What is the Role of Small Models in the LLM Era: A Survey(https://arxiv.org/abs/2409.06857)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at this https URL
• 摘要：大型语言模型 (LLM) 在推进通用人工智能 (AGI) 方面取得了重大进展，从而催生出越来越大的模型，例如 GPT-4 和 LLaMA-405B。然而，扩大模型规模会导致计算成本和能耗呈指数级增长，使得这些模型对于资源有限的学术研究人员和企业来说不切实际。与此同时，小型模型 (SM) 经常用于实际环境，尽管它们的重要性目前被低估了。这引发了关于小型模型在 LLM 时代的作用的重要问题，这一主题在先前的研究中受到的关注有限。在这项工作中，我们从两个关键角度系统地研究了 LLM 和 SM 之间的关系：协作和竞争。我们希望这项调查能为从业者提供有价值的见解，加深对小型模型贡献的理解，并促进更有效地利用计算资源。代码可在此 https URL 上获得

Title: A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task

• Authors: Yuya Fujisaki, Shiro Takagi, Hideki Asoh, Wataru Kumagai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06883
• Pdf URL: https://arxiv.org/pdf/2409.06883
• Copy Paste: [[2409.06883]] A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task(https://arxiv.org/abs/2409.06883)
• Keywords: gpt, llm
• Abstract: The progress in text summarization techniques has been remarkable. However the task of accurately extracting and summarizing necessary information from highly specialized documents such as research papers has not been sufficiently investigated. We are focusing on the task of extracting research questions (RQ) from research papers and construct a new dataset consisting of machine learning papers, RQ extracted from these papers by GPT-4, and human evaluations of the extracted RQ from multiple perspectives. Using this dataset, we systematically compared recently proposed LLM-based evaluation functions for summarizations, and found that none of the functions showed sufficiently high correlations with human evaluations. We expect our dataset provides a foundation for further research on developing better evaluation functions tailored to the RQ extraction task, and contribute to enhance the performance of the task. The dataset is available at this https URL.
• 摘要：文本摘要技术取得了显著进展。然而，从研究论文等高度专业化的文档中准确提取和总结必要信息的任务尚未得到充分研究。我们专注于从研究论文中提取研究问题 (RQ) 的任务，并构建了一个由机器学习论文、GPT-4 从这些论文中提取的 RQ 以及从多个角度对提取的 RQ 的人工评估组成的新数据集。利用该数据集，我们系统地比较了最近提出的基于 LLM 的摘要评估函数，发现没有一个函数与人工评估显示出足够高的相关性。我们希望我们的数据集为进一步研究开发针对 RQ 提取任务的更好的评估函数奠定基础，并有助于提高任务的性能。数据集可在此 https URL 上找到。

Title: You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling

• Authors: Jaewoo Song, Andrew Zhu, Chris Callison-Burch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06949
• Pdf URL: https://arxiv.org/pdf/2409.06949
• Copy Paste: [[2409.06949]] You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling(https://arxiv.org/abs/2409.06949)
• Keywords: language model, llm
• Abstract: Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game master's role. This paper presents a novel approach to enhance AI game masters by leveraging function calling in the context of the table-top role-playing game "Jim Henson's Labyrinth: The Adventure Game." Our methodology involves integrating game-specific controls through functions, which we show improves the narrative quality and state update consistency of the AI game master. The experimental results, based on human evaluations and unit tests, demonstrate the effectiveness of our approach in enhancing gameplay experience and maintaining coherence with the game state. This work contributes to the advancement of game AI and interactive storytelling, offering insights into the design of more engaging and consistent AI-driven game masters.
• 摘要：由于大型语言模型 (LLM) 的局限性以及游戏管理员角色的复杂性，为基于文本的游戏开发一致且可靠的 AI 游戏管理员是一项艰巨的任务。本文介绍了一种通过利用桌面角色扮演游戏“吉姆·汉森的迷宫：冒险游戏”中的函数调用来增强 AI 游戏管理员的新方法。我们的方法涉及通过函数集成特定于游戏的控件，我们表明这可以提高 AI 游戏管理员的叙事质量和状态更新一致性。基于人工评估和单元测试的实验结果证明了我们的方法在增强游戏体验和保持与游戏状态的一致性方面的有效性。这项工作有助于游戏 AI 和交互式故事叙述的进步，为设计更具吸引力和一致性的 AI 驱动的游戏管理员提供了见解。

Title: Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency

• Authors: Hanyu Zhao, Li Du, Yiming Ju, Chengwei Wu, Tengfei Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07045
• Pdf URL: https://arxiv.org/pdf/2409.07045
• Copy Paste: [[2409.07045]] Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency(https://arxiv.org/abs/2409.07045)
• Keywords: language model, llm
• Abstract: With the availability of various instruction datasets, a pivotal challenge is how to effectively select and integrate these instructions to fine-tune large language models (LLMs). Previous research mainly focuses on selecting individual high-quality instructions. However, these works overlooked the joint interactions and dependencies between different categories of instructions, leading to suboptimal selection strategies. Moreover, the nature of these interaction patterns remains largely unexplored, let alone optimize the instruction set with regard to them. To fill these gaps, in this paper, we: (1) systemically investigate interaction and dependency patterns between different categories of instructions, (2) manage to optimize the instruction set concerning the interaction patterns using a linear programming-based method, and optimize the learning schema of SFT using an instruction dependency taxonomy guided curriculum learning. Experimental results across different LLMs demonstrate improved performance over strong baselines on widely adopted benchmarks.
• 摘要：随着各种指令数据集的出现，一个关键挑战是如何有效地选择和集成这些指令来微调大型语言模型 (LLM)。先前的研究主要侧重于选择单个高质量指令。然而，这些工作忽视了不同类别指令之间的联合交互和依赖关系，导致选择策略不够理想。此外，这些交互模式的性质在很大程度上仍未被探索，更不用说优化与它们相关的指令集了。为了填补这些空白，在本文中，我们：(1) 系统地研究不同类别指令之间的交互和依赖模式，(2) 设法使用基于线性规划的方法优化与交互模式有关的指令集，并使用指令依赖性分类法指导的课程学习优化 SFT 的学习方案。跨不同 LLM 的实验结果表明，在广泛采用的基准上，性能优于强基线。

Title: Native vs Non-Native Language Prompting: A Comparative Analysis

• Authors: Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, Firoj Alam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07054
• Pdf URL: https://arxiv.org/pdf/2409.07054
• Copy Paste: [[2409.07054]] Native vs Non-Native Language Prompting: A Comparative Analysis(https://arxiv.org/abs/2409.07054)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have shown remarkable abilities in different fields, including standard Natural Language Processing (NLP) tasks. To elicit knowledge from LLMs, prompts play a key role, consisting of natural language instructions. Most open and closed source LLMs are trained on available labeled and unlabeled resources--digital content such as text, images, audio, and videos. Hence, these models have better knowledge for high-resourced languages but struggle with low-resourced languages. Since prompts play a crucial role in understanding their capabilities, the language used for prompts remains an important research question. Although there has been significant research in this area, it is still limited, and less has been explored for medium to low-resourced languages. In this study, we investigate different prompting strategies (native vs. non-native) on 11 different NLP tasks associated with 12 different Arabic datasets (9.7K data points). In total, we conducted 197 experiments involving 3 LLMs, 12 datasets, and 3 prompting strategies. Our findings suggest that, on average, the non-native prompt performs the best, followed by mixed and native prompts.
• 摘要：大型语言模型 (LLM) 在不同领域表现出非凡的能力，包括标准自然语言处理 (NLP) 任务。为了从 LLM 中获取知识，提示起着关键作用，它由自然语言指令组成。大多数开源和闭源 LLM 都是在可用的标记和未标记资源（如文本、图像、音频和视频等数字内容）上进行训练的。因此，这些模型对资源丰富的语言有更好的了解，但对资源匮乏的语言却很吃力。由于提示在理解其功能方面起着至关重要的作用，提示所用的语言仍然是一个重要的研究问题。尽管在这一领域已经有大量研究，但研究仍然有限，对中低资源语言的研究较少。在本研究中，我们研究了与 12 个不同的阿拉伯语数据集（9.7K 个数据点）相关的 11 个不同 NLP 任务的不同提示策略（母语与非母语）。总共进行了 197 次实验，涉及 3 个 LLM、12 个数据集和 3 种提示策略。我们的研究结果表明，平均而言，非母语提示表现最好，其次是混合提示和母语提示。

Title: Latent Space Interpretation for Stylistic Analysis and Explainable Authorship Attribution

• Authors: Milad Alshomary, Narutatsu Ri, Marianna Apidianaki, Ajay Patel, Smaranda Muresan, Kathleen McKeown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07072
• Pdf URL: https://arxiv.org/pdf/2409.07072
• Copy Paste: [[2409.07072]] Latent Space Interpretation for Stylistic Analysis and Explainable Authorship Attribution(https://arxiv.org/abs/2409.07072)
• Keywords: llm
• Abstract: Recent state-of-the-art authorship attribution methods learn authorship representations of texts in a latent, non-interpretable space, hindering their usability in real-world applications. Our work proposes a novel approach to interpreting these learned embeddings by identifying representative points in the latent space and utilizing LLMs to generate informative natural language descriptions of the writing style of each point. We evaluate the alignment of our interpretable space with the latent one and find that it achieves the best prediction agreement compared to other baselines. Additionally, we conduct a human evaluation to assess the quality of these style descriptions, validating their utility as explanations for the latent space. Finally, we investigate whether human performance on the challenging AA task improves when aided by our system's explanations, finding an average improvement of around +20% in accuracy.
• 摘要：最近最先进的作者归属方法在潜在的、不可解释的空间中学习文本的作者表征，这阻碍了它们在实际应用中的可用性。我们的工作提出了一种解释这些学习到的嵌入的新方法，即识别潜在空间中的代表点，并利用 LLM 生成每个点的写作风格的信息丰富的自然语言描述。我们评估了可解​​释空间与潜在空间的一致性，发现它与其他基线相比实现了最佳预测一致性。此外，我们进行了人工评估以评估这些风格描述的质量，验证了它们作为潜在空间解释的实用性。最后，我们研究了在系统解释的帮助下，人类在具有挑战性的 AA 任务上的表现是否会有所提高，发现准确率平均提高了约 +20%。

Title: Understanding Knowledge Drift in LLMs through Misinformation

• Authors: Alina Fastowski, Gjergji Kasneci
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07085
• Pdf URL: https://arxiv.org/pdf/2409.07085
• Copy Paste: [[2409.07085]] Understanding Knowledge Drift in LLMs through Misinformation(https://arxiv.org/abs/2409.07085)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have revolutionized numerous applications, making them an integral part of our digital ecosystem. However, their reliability becomes critical, especially when these models are exposed to misinformation. We primarily analyze the susceptibility of state-of-the-art LLMs to factual inaccuracies when they encounter false information in a QnA scenario, an issue that can lead to a phenomenon we refer to as *knowledge drift*, which significantly undermines the trustworthiness of these models. We evaluate the factuality and the uncertainty of the models' responses relying on Entropy, Perplexity, and Token Probability metrics. Our experiments reveal that an LLM's uncertainty can increase up to 56.6% when the question is answered incorrectly due to the exposure to false information. At the same time, repeated exposure to the same false information can decrease the models uncertainty again (-52.8% w.r.t. the answers on the untainted prompts), potentially manipulating the underlying model's beliefs and introducing a drift from its original knowledge. These findings provide insights into LLMs' robustness and vulnerability to adversarial inputs, paving the way for developing more reliable LLM applications across various domains. The code is available at this https URL.
• 摘要：大型语言模型 (LLM) 已经彻底改变了众多应用，使其成为我们数字生态系统不可或缺的一部分。然而，它们的可靠性变得至关重要，尤其是当这些模型接触到错误信息时。我们主要分析最先进的 LLM 在问答场景中遇到虚假信息时对事实不准确的敏感性，这一问题可能导致我们称之为“知识漂移”的现象，从而严重削弱这些模型的可信度。我们根据熵、困惑度和标记概率指标来评估模型响应的真实性和不确定性。我们的实验表明，当由于接触虚假信息而导致问题回答错误时，LLM 的不确定性可能会增加高达 56.6%。同时，反复接触相同的虚假信息可以再次降低模型的不确定性（相对于未受污染提示的答案为 -52.8%），可能会操纵底层模型的信念并引入偏离其原始知识的偏差。这些发现深入了解了 LLM 的稳健性和对抗性输入的脆弱性，为在各个领域开发更可靠的 LLM 应用程序铺平了道路。代码可在此 https URL 上获取。

Title: Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model

• Authors: Daehee Kim, Deokhyung Kang, Sangwon Ryu, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07088
• Pdf URL: https://arxiv.org/pdf/2409.07088
• Copy Paste: [[2409.07088]] Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model(https://arxiv.org/abs/2409.07088)
• Keywords: language model, llm
• Abstract: Knowledge Graph-to-Text (G2T) generation involves verbalizing structured knowledge graphs into natural language text. Recent advancements in Pretrained Language Models (PLMs) have improved G2T performance, but their effectiveness depends on datasets with precise graph-text alignment. However, the scarcity of high-quality, general-domain G2T generation datasets restricts progress in the general-domain G2T generation research. To address this issue, we introduce Wikipedia Ontology-Free Graph-text dataset (WikiOFGraph), a new large-scale G2T dataset generated using a novel method that leverages Large Language Model (LLM) and Data-QuestEval. Our new dataset, which contains 5.85M general-domain graph-text pairs, offers high graph-text consistency without relying on external ontologies. Experimental results demonstrate that PLM fine-tuned on WikiOFGraph outperforms those trained on other datasets across various evaluation metrics. Our method proves to be a scalable and effective solution for generating high-quality G2T data, significantly advancing the field of G2T generation.
• 摘要：知识图谱到文本 (G2T) 生成涉及将结构化知识图谱语言化为自然语言文本。预训练语言模型 (PLM) 的最新进展提高了 G2T 性能，但其有效性取决于具有精确图文对齐的数据集。然而，高质量通用领域 G2T 生成数据集的稀缺限制了通用领域 G2T 生成研究的进展。为了解决这个问题，我们引入了维基百科无本体图文数据集 (WikiOFGraph)，这是一种新的大规模 G2T 数据集，使用一种利用大型语言模型 (LLM) 和 Data-QuestEval 的新方法生成。我们的新数据集包含 585 万个通用领域图文对，无需依赖外部本体即可提供高图文一致性。实验结果表明，在 WikiOFGraph 上微调的 PLM 在各种评估指标上均优于在其他数据集上训练的 PLM。事实证明，我们的方法是生成高质量 G2T 数据的可扩展且有效的解决方案，显著推动了 G2T 生成领域的发展。

Title: Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem

• Authors: Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, Sebastian Möller, Vera Schmitt
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07123
• Pdf URL: https://arxiv.org/pdf/2409.07123
• Copy Paste: [[2409.07123]] Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem(https://arxiv.org/abs/2409.07123)
• Keywords: language model, gpt, llm, chat
• Abstract: Natural language explanations (NLEs) are vital for elucidating the reasoning behind large language model (LLM) decisions. Many techniques have been developed to generate NLEs using LLMs. However, like humans, LLMs might not always produce optimal NLEs on first attempt. Inspired by human learning processes, we introduce Cross-Refine, which employs role modeling by deploying two LLMs as generator and critic, respectively. The generator outputs a first NLE and then refines this initial explanation using feedback and suggestions provided by the critic. Cross-Refine does not require any supervised training data or additional training. We validate Cross-Refine across three NLP tasks using three state-of-the-art open-source LLMs through automatic and human evaluation. We select Self-Refine (Madaan et al., 2023) as the baseline, which only utilizes self-feedback to refine the explanations. Our findings from automatic evaluation and a user study indicate that Cross-Refine outperforms Self-Refine. Meanwhile, Cross-Refine can perform effectively with less powerful LLMs, whereas Self-Refine only yields strong results with ChatGPT. Additionally, we conduct an ablation study to assess the importance of feedback and suggestions. Both of them play an important role in refining explanations. We further evaluate Cross-Refine on a bilingual dataset in English and German.
• 摘要：自然语言解释 (NLE) 对于阐明大型语言模型 (LLM) 决策背后的原因至关重要。已经开发了许多使用 LLM 生成 NLE 的技术。然而，与人类一样，LLM 可能并不总是在第一次尝试时就产生最佳的 NLE。受人类学习过程的启发，我们引入了 Cross-Refine，它通过分别部署两个 LLM 作为生成器和评论家来采用角色建模。生成器输出第一个 NLE，然后使用评论家提供的反馈和建议来改进这个初始解释。Cross-Refine 不需要任何监督训练数据或额外训练。我们通过自动和人工评估，使用三个最先进的开源 LLM 在三个 NLP 任务中验证了 Cross-Refine。我们选择 Self-Refine (Madaan 等人，2023) 作为基线，它仅利用自我反馈来改进解释。我们从自动评估和用户研究中得出的结果表明，Cross-Refine 优于 Self-Refine。同时，Cross-Refine 可以在功能较弱的 LLM 上有效发挥作用，而 Self-Refine 仅在 ChatGPT 上产生强劲效果。此外，我们进行了一项消融研究，以评估反馈和建议的重要性。它们在完善解释方面都发挥着重要作用。我们进一步在英语和德语双语数据集上评估了 Cross-Refine。

Title: Reranking Laws for Language Generation: A Communication-Theoretic Perspective

• Authors: António Farinhas, Haau-Sing Li, André F. T. Martins
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.07131
• Pdf URL: https://arxiv.org/pdf/2409.07131
• Copy Paste: [[2409.07131]] Reranking Laws for Language Generation: A Communication-Theoretic Perspective(https://arxiv.org/abs/2409.07131)
• Keywords: language model, llm
• Abstract: To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free (i.e., yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.
• 摘要：为了确保大型语言模型 (LLM) 的安全使用，必须降低其产生幻觉或生成不可接受答案的倾向。一种简单且经常使用的策略是首先让 LLM 生成多个假设，然后使用重排器选择最佳假设。在本文中，我们将此策略与使用冗余来降低嘈杂通信信道中的错误率进行比较。我们将生成器概念化为通过并行嘈杂信道传输消息的多个描述的发送者。接收者通过对（可能损坏的）描述进行排序并选择最可靠的描述来解码消息。我们提供了该协议渐近无错误的条件（即几乎肯定会产生可接受的答案），即使在重排器不完善（由 Mallows 或 Zipf-Mandelbrot 模型控制）且信道分布具有统计相关性的情况下也是如此。我们使用我们的框架来获得重新排序定律，并使用 LLM 在两个实际任务上进行了实证验证：使用 DeepSeek-Coder 7B 进行文本到代码的生成，以及使用 TowerInstruct 13B 进行医疗数据的机器翻译。

Title: Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models

• Authors: Rui Ye, Rui Ge, Yuchi Fengting, Jingyi Chai, Yanfeng Wang, Siheng Chen
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2409.07136
• Pdf URL: https://arxiv.org/pdf/2409.07136
• Copy Paste: [[2409.07136]] Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models(https://arxiv.org/abs/2409.07136)
• Keywords: language model, llm, prompt
• Abstract: Federated instruction tuning enables multiple clients to collaboratively fine-tune a shared large language model (LLM) that can follow humans' instructions without directly sharing raw data. However, existing literature impractically requires that all the clients readily hold instruction-tuning data (i.e., structured instruction-response pairs), which necessitates massive human annotations since clients' data is usually unstructured text instead. Addressing this, we propose a novel and flexible framework FedIT-U2S, which can automatically transform unstructured corpus into structured data for federated instruction tuning. FedIT-U2S consists two key steps: (1) few-shot instruction-tuning data generation, where each unstructured data piece together with several examples is combined to prompt an LLM in generating an instruction-response pair. To further enhance the flexibility, a retrieval-based example selection technique is proposed, where the examples are automatically selected based on the relatedness between the client's data piece and example pool, bypassing the need of determining examples in advance. (2) A typical federated instruction tuning process based on the generated data. Overall, FedIT-U2S can be applied to diverse scenarios as long as the client holds valuable text corpus, broadening the application scope of federated instruction tuning. We conduct a series of experiments on three domains (medicine, knowledge, and math), showing that our proposed FedIT-U2S can consistently and significantly brings improvement over the base LLM.
• 摘要：联邦指令调优使多个客户端能够协作微调共享的大型语言模型 (LLM)，该模型无需直接共享原始数据即可遵循人类的指令。然而，现有文献不切实际地要求所有客户端都随时掌握指令调优数据（即结构化的指令-响应对），这需要大量的人工注释，因为客户端的数据通常是非结构化的文本。为了解决这个问题，我们提出了一个新颖而灵活的框架 FedIT-U2S，它可以自动将非结构化语料库转换为结构化数据以进行联邦指令调优。FedIT-U2S 包括两个关键步骤：(1) 少量指令调优数据生成，其中每个非结构化数据片段与几个示例组合在一起以提示 LLM 生成指令-响应对。为了进一步增强灵活性，提出了一种基于检索的示例选择技术，其中根据客户端数据片段和示例池之间的相关性自动选择示例，而无需提前确定示例。 （2）基于生成数据的典型联邦指令调优过程。总体而言，只要客户端拥有有价值的文本语料库，FedIT-U2S 就可以应用于各种场景，从而拓宽联邦指令调优的应用范围。我们在三个领域（医学、知识和数学）进行了一系列实验，结果表明，我们提出的 FedIT-U2S 可以持续显著地带来比基础 LLM 更好的性能提升。

Title: A Fine-grained Sentiment Analysis of App Reviews using Large Language Models: An Evaluation Study

• Authors: Faiz Ali Shah, Ahmed Sabir, Rajesh Sharma
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.07162
• Pdf URL: https://arxiv.org/pdf/2409.07162
• Copy Paste: [[2409.07162]] A Fine-grained Sentiment Analysis of App Reviews using Large Language Models: An Evaluation Study(https://arxiv.org/abs/2409.07162)
• Keywords: language model, gpt, llm, chat
• Abstract: Analyzing user reviews for sentiment towards app features can provide valuable insights into users' perceptions of app functionality and their evolving needs. Given the volume of user reviews received daily, an automated mechanism to generate feature-level sentiment summaries of user reviews is needed. Recent advances in Large Language Models (LLMs) such as ChatGPT have shown impressive performance on several new tasks without updating the model's parameters i.e. using zero or a few labeled examples. Despite these advancements, LLMs' capabilities to perform feature-specific sentiment analysis of user reviews remain unexplored. This study compares the performance of state-of-the-art LLMs, including GPT-4, ChatGPT, and LLama-2-chat variants, for extracting app features and associated sentiments under 0-shot, 1-shot, and 5-shot scenarios. Results indicate the best-performing GPT-4 model outperforms rule-based approaches by 23.6% in f1-score with zero-shot feature extraction; 5-shot further improving it by 6%. GPT-4 achieves a 74% f1-score for predicting positive sentiment towards correctly predicted app features, with 5-shot enhancing it by 7%. Our study suggests that LLM models are promising for generating feature-specific sentiment summaries of user reviews.
• 摘要：分析用户评论对应用功能的情绪可以提供有价值的见解，了解用户对应用功能的看法及其不断变化的需求。鉴于每天收到的用户评论数量庞大，需要一种自动机制来生成用户评论的特征级情绪摘要。大型语言模型 (LLM)（例如 ChatGPT）的最新进展在几个新任务上表现出色，而无需更新模型的参数，即使用零个或几个标记示例。尽管取得了这些进步，但 LLM 对用户评论执行特征特定情绪分析的能力仍未得到探索。本研究比较了最先进的 LLM（包括 GPT-4、ChatGPT 和 LLama-2-chat 变体）在 0 次、1 次和 5 次场景下提取应用功能和相关情绪的性能。结果表明，在零次特征提取中，表现最佳的 GPT-4 模型在 f1 得分上比基于规则的方法高出 23.6%；5 次进一步将其提高了 6%。 GPT-4 在预测正确预测的应用功能的积极情绪方面取得了 74% 的 f1 分数，而 5 次测试将其提高了 7%。我们的研究表明，LLM 模型有望生成针对特定功能的用户评论情绪摘要。

Title: Learning Efficient Recursive Numeral Systems via Reinforcement Learning

• Authors: Jonathan D. Thomas, Andrea Silvi, Devdatt Dubhashi, Emil Carlsson, Moa Johansson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07170
• Pdf URL: https://arxiv.org/pdf/2409.07170
• Copy Paste: [[2409.07170]] Learning Efficient Recursive Numeral Systems via Reinforcement Learning(https://arxiv.org/abs/2409.07170)
• Keywords: agent
• Abstract: The emergence of mathematical concepts, such as number systems, is an understudied area in AI for mathematics and reasoning. It has previously been shown Carlsson et al. (2021) that by using reinforcement learning (RL), agents can derive simple approximate and exact-restricted numeral systems. However, it is a major challenge to show how more complex recursive numeral systems, similar to the one utilised in English, could arise via a simple learning mechanism such as RL. Here, we introduce an approach towards deriving a mechanistic explanation of the emergence of recursive number systems where we consider an RL agent which directly optimizes a lexicon under a given meta-grammar. Utilising a slightly modified version of the seminal meta-grammar of Hurford (1975), we demonstrate that our RL agent can effectively modify the lexicon towards Pareto-optimal configurations which are comparable to those observed within human numeral systems.
• 摘要：数学概念（例如数字系统）的出现是数学和推理人工智能中一个研究不足的领域。Carlsson 等人（2021 年）之前曾表明，通过使用强化学习 (RL)，代理可以推导出简单的近似和精确受限的数字系统。然而，要证明更复杂的递归数字系统（类似于英语中使用的数字系统）如何通过简单的学习机制（例如 RL）产生，这是一个重大挑战。在这里，我们介绍了一种推导递归数字系统出现的机械解释的方法，其中我们考虑一个在给定元语法下直接优化词典的 RL 代理。利用 Hurford（1975 年）开创性元语法的略微修改版本，我们证明我们的 RL 代理可以有效地将词典修改为与人类数字系统中观察到的配置相当的帕累托最优配置。

Title: Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs

• Authors: Firoj Alam, Md. Rafiul Biswas, Uzair Shah, Wajdi Zaghouani, Georgios Mikros
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07246
• Pdf URL: https://arxiv.org/pdf/2409.07246
• Copy Paste: [[2409.07246]] Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs(https://arxiv.org/abs/2409.07246)
• Keywords: llm, agent
• Abstract: In the past decade, social media platforms have been used for information dissemination and consumption. While a major portion of the content is posted to promote citizen journalism and public awareness, some content is posted to mislead users. Among different content types such as text, images, and videos, memes (text overlaid on images) are particularly prevalent and can serve as powerful vehicles for propaganda, hate, and humor. In the current literature, there have been efforts to individually detect such content in memes. However, the study of their intersection is very limited. In this study, we explore the intersection between propaganda and hate in memes using a multi-agent LLM-based approach. We extend the propagandistic meme dataset with coarse and fine-grained hate labels. Our finding suggests that there is an association between propaganda and hate in memes. We provide detailed experimental results that can serve as a baseline for future studies. We will make the experimental resources publicly available to the community.
• 摘要：在过去十年中，社交媒体平台已用于信息传播和消费。虽然发布的大部分内容是为了促进公民新闻和公众意识，但有些内容是为了误导用户。在文本、图像和视频等不同类型的内容中，模因（覆盖在图像上的文本）尤为普遍，可以作为宣传、仇恨和幽默的有力载体。在目前的文献中，人们一直在努力单独检测模因中的此类内容。然而，对它们交集的研究非常有限。在本研究中，我们使用基于多智能体 LLM 的方法探索模因中宣传和仇恨之间的交集。我们用粗粒度和细粒度的仇恨标签扩展了宣传模因数据集。我们的发现表明，模因中的宣传和仇恨之间存在关联。我们提供了详细的实验结果，可以作为未来研究的基线。我们将向社区公开实验资源。

Title: MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications

• Authors: Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07314
• Pdf URL: https://arxiv.org/pdf/2409.07314
• Copy Paste: [[2409.07314]] MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications(https://arxiv.org/abs/2409.07314)
• Keywords: language model, llm, hallucination
• Abstract: The rapid development of Large Language Models (LLMs) for healthcare applications has spurred calls for holistic evaluation beyond frequently-cited benchmarks like USMLE, to better reflect real-world performance. While real-world assessments are valuable indicators of utility, they often lag behind the pace of LLM evolution, likely rendering findings obsolete upon deployment. This temporal disconnect necessitates a comprehensive upfront evaluation that can guide model selection for specific clinical applications. We introduce MEDIC, a framework assessing LLMs across five critical dimensions of clinical competence: medical reasoning, ethics and bias, data and language understanding, in-context learning, and clinical safety. MEDIC features a novel cross-examination framework quantifying LLM performance across areas like coverage and hallucination detection, without requiring reference outputs. We apply MEDIC to evaluate LLMs on medical question-answering, safety, summarization, note generation, and other tasks. Our results show performance disparities across model sizes, baseline vs medically finetuned models, and have implications on model selection for applications requiring specific model strengths, such as low hallucination or lower cost of inference. MEDIC's multifaceted evaluation reveals these performance trade-offs, bridging the gap between theoretical capabilities and practical implementation in healthcare settings, ensuring that the most promising models are identified and adapted for diverse healthcare applications.
• 摘要：大型语言模型 (LLM) 在医疗保健领域的快速发展促使人们呼吁进行全面评估，而不仅仅是经常被引用的 USMLE 等基准，以更好地反映现实世界的表现。虽然现实世界的评估是实用性的宝贵指标，但它们往往落后于 LLM 的发展速度，可能会导致部署后的结果过时。这种时间上的脱节需要进行全面的前期评估，以指导特定临床应用的模型选择。我们推出了 MEDIC，这是一个评估 LLM 的框架，涵盖临床能力的五个关键维度：医学推理、道德和偏见、数据和语言理解、情境学习和临床安全。MEDIC 采用了一种新颖的交叉检查框架，可以量化 LLM 在覆盖范围和幻觉检测等领域的表现，而无需参考输出。我们应用 MEDIC 来评估 LLM 在医学问答、安全性、总结、笔记生成和其他任务方面的表现。我们的结果表明，不同模型大小、基线模型与医学微调模型之间存在性能差异，并且对需要特定模型优势（例如低幻觉或较低推理成本）的应用的模型选择有影响。MEDIC 的多方面评估揭示了这些性能权衡，弥合了医疗环境中理论能力与实际实施之间的差距，确保确定最有前景的模型并使其适用于各种医疗应用。

Title: Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation

• Authors: SeongYeub Chu, JongWoo Kim, MunYong Yi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07355
• Pdf URL: https://arxiv.org/pdf/2409.07355
• Copy Paste: [[2409.07355]] Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation(https://arxiv.org/abs/2409.07355)
• Keywords: language model, llm
• Abstract: This study introduces \textbf{InteractEval}, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at \textbf{\url{this https URL}}.
• 摘要：本研究引入了 \textbf{InteractEval}，这是一个将人类专业知识与大型语言模型 (LLM) 相结合的框架，使用 Think-Aloud (TA) 方法生成基于检查表的文本评估的属性。通过将人类的灵活性和推理与 LLM 的一致性相结合，InteractEval 在四个不同维度上的表现优于传统的非 LLM 和基于 LLM 的基线，包括连贯性、流畅性、一致性和相关性。该实验还调查了 TA 方法的有效性，表明它促进了人类和 LLM 的发散思维，从而生成更广泛的相关属性并提高文本评估性能。比较分析表明，人类擅长识别与内部质量相关的属性（连贯性和流畅性），但 LLM 在与外部一致性相关的属性（一致性和相关性）方面表现更好。因此，同时利用人类和 LLM 可以产生最佳评估结果。换句话说，本研究强调了在基于检查表的自动化文本评估框架中有效结合人类和 LLM 的必要性。代码可在 \textbf{\url{此 https URL}} 获得。

Title: Awaking the Slides: A Tuning-free and Knowledge-regulated AI Tutoring System via Language Model Coordination

• Authors: Daniel Zhang-Li, Zheyuan Zhang, Jifan Yu, Joy Lim Jia Yin, Shangqing Tu, Linlu Gong, Haohua Wang, Zhiyuan Liu, Huiqin Liu, Lei Hou, Juanzi Li
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.07372
• Pdf URL: https://arxiv.org/pdf/2409.07372
• Copy Paste: [[2409.07372]] Awaking the Slides: A Tuning-free and Knowledge-regulated AI Tutoring System via Language Model Coordination(https://arxiv.org/abs/2409.07372)
• Keywords: language model
• Abstract: The vast pre-existing slides serve as rich and important materials to carry lecture knowledge. However, effectively leveraging lecture slides to serve students is difficult due to the multi-modal nature of slide content and the heterogeneous teaching actions. We study the problem of discovering effective designs that convert a slide into an interactive lecture. We develop Slide2Lecture, a tuning-free and knowledge-regulated intelligent tutoring system that can (1) effectively convert an input lecture slide into a structured teaching agenda consisting of a set of heterogeneous teaching actions; (2) create and manage an interactive lecture that generates responsive interactions catering to student learning demands while regulating the interactions to follow teaching actions. Slide2Lecture contains a complete pipeline for learners to obtain an interactive classroom experience to learn the slide. For teachers and developers, Slide2Lecture enables customization to cater to personalized demands. The evaluation rated by annotators and students shows that Slide2Lecture is effective in outperforming the remaining implementation. Slide2Lecture's online deployment has made more than 200K interaction with students in the 3K lecture sessions. We open source Slide2Lecture's implementation in https://anonymous.4open.science/r/slide2lecture-4210/.
• 摘要：大量现成的幻灯片是承载讲座知识的丰富而重要的材料。然而，由于幻灯片内容的多模态性和异构的教学行为，有效地利用讲座幻灯片为学生服务是困难的。我们研究发现将幻灯片转换为交互式讲座的有效设计的问题。我们开发了 Slide2Lecture，这是一个无需调整和知识调节的智能辅导系统，它可以 (1) 有效地将输入的讲座幻灯片转换为由一组异构教学行为组成的结构化教学议程；(2) 创建和管理交互式讲座，该讲座生成响应式交互以满足学生的学习需求，同时调节交互以遵循教学行为。Slide2Lecture 包含一个完整的管道，让学习者获得交互式课堂体验来学习幻灯片。对于教师和开发人员来说，Slide2Lecture 可以定制以满足个性化需求。注释者和学生的评估表明，Slide2Lecture 的表现优于其他实现。Slide2Lecture 的在线部署在 3K 讲座课程中与学生进行了超过 200K 次互动。我们在 https://anonymous.4open.science/r/slide2lecture-4210/ 开源了 Slide2Lecture 的实现。

Title: AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge

• Authors: Han Wang, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07394
• Pdf URL: https://arxiv.org/pdf/2409.07394
• Copy Paste: [[2409.07394]] AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge(https://arxiv.org/abs/2409.07394)
• Keywords: language model, llm
• Abstract: Knowledge conflict arises from discrepancies between information in the context of a large language model (LLM) and the knowledge stored in its parameters. This can hurt performance when using standard decoding techniques, which tend to ignore the context. Existing test-time contrastive methods seek to address this by comparing the LLM's output distribution with and without the context and adjust the model according to the contrast between them. However, we find that these methods frequently misjudge the degree of conflict and struggle to handle instances that vary in their amount of conflict, with static methods over-adjusting when conflict is absent. We propose a fine-grained, instance-level approach called AdaCAD, which dynamically infers the weight of adjustment based on the degree of conflict, as measured by the Jensen-Shannon divergence between distributions representing contextual and parametric knowledge. Our experiments across four models on six diverse question-answering (QA) datasets and three summarization tasks demonstrate that our training-free adaptive method consistently outperforms other decoding methods on QA, with average accuracy gains of 14.21% (absolute) over a static contrastive baseline, and improves the factuality of summaries by 5.59 (AlignScore). Furthermore, our analysis shows that while decoding with contrastive baselines hurts performance when conflict is absent, AdaCAD mitigates these losses, making it more applicable to real-world datasets in which some examples have conflict and others do not.
• 摘要：知识冲突源于大型语言模型 (LLM) 上下文中的信息与其参数中存储的知识之间的差异。当使用标准解码技术时，这可能会损害性能，因为标准解码技术往往会忽略上下文。现有的测试时对比方法试图通过比较有上下文和没有上下文的 LLM 输出分布并根据它们之间的对比调整模型来解决此问题。然而，我们发现这些方法经常会误判冲突的程度，并且难以处理冲突程度不同的实例，而静态方法在没有冲突时会过度调整。我们提出了一种名为 AdaCAD 的细粒度实例级方法，它根据冲突程度动态推断调整权重，以表示上下文和参数知识的分布之间的 Jensen-Shannon 散度来衡量。我们对四个模型在六个不同的问答 (QA) 数据集和三个摘要任务上进行的实验表明，我们的无训练自适应方法在 QA 上始终优于其他解码方法，与静态对比基线相比，平均准确率提高了 14.21%（绝对值），摘要的真实性提高了 5.59（AlignScore）。此外，我们的分析表明，虽然使用对比基线解码会在没有冲突的情况下损害性能，但 AdaCAD 可以减轻这些损失，使其更适用于现实世界的数据集，其中一些示例有冲突，而其他示例没有冲突。

Title: Towards Fairer Health Recommendations: finding informative unbiased samples via Word Sense Disambiguation

• Authors: Gavin Butts, Pegah Emdad, Jethro Lee, Shannon Song, Chiman Salavati, Willmar Sosa Diaz, Shiri Dori-Hacohen, Fabricio Murai
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07424
• Pdf URL: https://arxiv.org/pdf/2409.07424
• Copy Paste: [[2409.07424]] Towards Fairer Health Recommendations: finding informative unbiased samples via Word Sense Disambiguation(https://arxiv.org/abs/2409.07424)
• Keywords: gpt, llm, prompt
• Abstract: There have been growing concerns around high-stake applications that rely on models trained with biased data, which consequently produce biased predictions, often harming the most vulnerable. In particular, biased medical data could cause health-related applications and recommender systems to create outputs that jeopardize patient care and widen disparities in health outcomes. A recent framework titled Fairness via AI posits that, instead of attempting to correct model biases, researchers must focus on their root causes by using AI to debias data. Inspired by this framework, we tackle bias detection in medical curricula using NLP models, including LLMs, and evaluate them on a gold standard dataset containing 4,105 excerpts annotated by medical experts for bias from a large corpus. We build on previous work by coauthors which augments the set of negative samples with non-annotated text containing social identifier terms. However, some of these terms, especially those related to race and ethnicity, can carry different meanings (e.g., "white matter of spinal cord"). To address this issue, we propose the use of Word Sense Disambiguation models to refine dataset quality by removing irrelevant sentences. We then evaluate fine-tuned variations of BERT models as well as GPT models with zero- and few-shot prompting. We found LLMs, considered SOTA on many NLP tasks, unsuitable for bias detection, while fine-tuned BERT models generally perform well across all evaluated metrics.
• 摘要：人们越来越担心，高风险应用依赖于使用有偏见数据训练的模型，因此会产生有偏见的预测，往往会伤害最脆弱的群体。特别是，有偏见的医疗数据可能会导致与健康相关的应用程序和推荐系统产生危及患者护理和扩大健康结果差距的输出。最近一个名为“通过人工智能实现公平”的框架认为，研究人员不应该试图纠正模型偏见，而应该通过使用人工智能消除数据偏见来关注其根本原因。受此框架的启发，我们使用 NLP 模型（包括 LLM）来解决医学课程中的偏见检测问题，并在一个黄金标准数据集上对它们进行评估，该数据集包含 4,105 段由医学专家注释的摘录，这些摘录来自一个大型语料库，用于检测偏见。我们在合著者之前的工作基础上，用包含社会标识符术语的未注释文本扩充了负样本集。然而，其中一些术语，尤其是与种族和民族有关的术语，可能具有不同的含义（例如，“脊髓白质”）。为了解决这个问题，我们建议使用词义消歧模型来删除不相关的句子，从而提高数据集的质量。然后，我们评估了 BERT 模型的微调变体以及具有零样本和少样本提示的 GPT 模型。我们发现 LLM 在许多 NLP 任务中被认为是 SOTA，不适合偏差检测，而微调的 BERT 模型通常在所有评估指标上都表现良好。

Title: Agent Workflow Memory

• Authors: Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07429
• Pdf URL: https://arxiv.org/pdf/2409.07429
• Copy Paste: [[2409.07429]] Agent Workflow Memory(https://arxiv.org/abs/2409.07429)
• Keywords: language model, agent
• Abstract: Despite the potential of language model-based agents to solve real-world tasks such as web navigation, current methods still struggle with long-horizon tasks with complex action trajectories. In contrast, humans can flexibly solve complex tasks by learning reusable task workflows from past experiences and using them to guide future actions. To build agents that can similarly benefit from this process, we introduce Agent Workflow Memory (AWM), a method for inducing commonly reused routines, i.e., workflows, and selectively providing workflows to the agent to guide subsequent generations. AWM flexibly applies to both offline and online scenarios, where agents induce workflows from training examples beforehand or from test queries on the fly. We experiment on two major web navigation benchmarks -- Mind2Web and WebArena -- that collectively cover 1000+ tasks from 200+ domains across travel, shopping, and social media, among others. AWM substantially improves the baseline results by 24.6% and 51.1% relative success rate on Mind2Web and WebArena while reducing the number of steps taken to solve WebArena tasks successfully. Furthermore, online AWM robustly generalizes in cross-task, website, and domain evaluations, surpassing baselines from 8.9 to 14.0 absolute points as train-test task distribution gaps widen.
• 摘要：尽管基于语言模型的代理具有解决网络导航等现实任务的潜力，但当前方法仍然难以处理具有复杂动作轨迹的长期任务。相比之下，人类可以通过从过去的经验中学习可重复使用的任务工作流并使用它们来指导未来的行动，灵活地解决复杂任务。为了构建可以从此过程中受益的代理，我们引入了代理工作流内存 (AWM)，这是一种诱导常用重复例程（即工作流）并有选择地向代理提供工作流以指导后续生成的方法。AWM 灵活地应用于离线和在线场景，代理可以从事先的训练示例中或动态测试查询中诱导工作流。我们在两个主要的网络导航基准测试（Mind2Web 和 WebArena）上进行了实验，它们总共涵盖了 200 多个领域的 1000 多项任务，涉及旅游、购物和社交媒体等。 AWM 在 Mind2Web 和 WebArena 上将基准结果大幅提升了 24.6%，相对成功率提升了 51.1%，同时减少了成功解决 WebArena 任务所需的步骤数。此外，在线 AWM 在跨任务、网站和领域评估中表现出色，随着训练测试任务分布差距扩大，绝对分数从 8.9 分超越基准，达到 14.0 分。