2024-08-01

Title: An Application of Large Language Models to Coding Negotiation Transcripts

• Authors: Ray Friedman, Jaewoo Cho, Jeanne Brett, Xuhui Zhan, Ningyu Han, Sriram Kannan, Yingxiang Ma, Jesse Spencer-Smith, Elisabeth Jäckel, Alfred Zerres, Madison Hooper, Katie Babbit, Manish Acharya, Wendi Adair, Soroush Aslani, Tayfun Aykaç, Chris Bauman, Rebecca Bennett, Garrett Brady, Peggy Briggs, Cheryl Dowie, Chase Eck, Igmar Geiger, Frank Jacob, Molly Kern, Sujin Lee, Leigh Anne Liu, Wu Liu, Jeffrey Loewenstein, Anne Lytle, Li Ma, Michel Mann, Alexandra Mislin, Tyree Mitchell, Hannah Martensen née Nagler, Amit Nandkeolyar, Mara Olekalns, Elena Paliakova, Jennifer Parlamis, Jason Pierce, Nancy Pierce, Robin Pinkley, Nathalie Prime, Jimena Ramirez-Marin, Kevin Rockmann, William Ross, Zhaleh Semnani-Azad, Juliana Schroeder, Philip Smith, Elena Stimmer, Roderick Swaab, Leigh Thompson, Cathy Tinsley, Ece Tuncel, Laurie Weingart, Robert Wilken, JingJing Yao, Zhi-Xue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21037
• Pdf URL: https://arxiv.org/pdf/2407.21037
• Copy Paste: [[2407.21037]] An Application of Large Language Models to Coding Negotiation Transcripts(https://arxiv.org/abs/2407.21037)
• Keywords: language model, llm
• Abstract: In recent years, Large Language Models (LLM) have demonstrated impressive capabilities in the field of natural language processing (NLP). This paper explores the application of LLMs in negotiation transcript analysis by the Vanderbilt AI Negotiation Lab. Starting in September 2022, we applied multiple strategies using LLMs from zero shot learning to fine tuning models to in-context learning). The final strategy we developed is explained, along with how to access and use the model. This study provides a sense of both the opportunities and roadblocks for the implementation of LLMs in real life applications and offers a model for how LLMs can be applied to coding in other fields.
• 摘要：近年来，大型语言模型 (LLM) 在自然语言处理 (NLP) 领域展现出了令人印象深刻的能力。本文探讨了范德比尔特大学人工智能谈判实验室在谈判记录分析中对 LLM 的应用。从 2022 年 9 月开始，我们使用 LLM 运用了多种策略，从零样本学习到微调模型再到上下文学习。本文解释了我们开发的最终策略，以及如何访问和使用该模型。这项研究让人们了解了 LLM 在实际应用中实施的机会和障碍，并为 LLM 如何应用于其他领域的编码提供了一个模型。

Title: They Look Like Each Other: Case-based Reasoning for Explainable Depression Detection on Twitter using Large Language Models

• Authors: Mohammad Saeid Mahdavinejad, Peyman Adibi, Amirhassan Monadjemi, Pascal Hitzler
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2407.21041
• Pdf URL: https://arxiv.org/pdf/2407.21041
• Copy Paste: [[2407.21041]] They Look Like Each Other: Case-based Reasoning for Explainable Depression Detection on Twitter using Large Language Models(https://arxiv.org/abs/2407.21041)
• Keywords: language model, prompt
• Abstract: Depression is a common mental health issue that requires prompt diagnosis and treatment. Despite the promise of social media data for depression detection, the opacity of employed deep learning models hinders interpretability and raises bias concerns. We address this challenge by introducing ProtoDep, a novel, explainable framework for Twitter-based depression detection. ProtoDep leverages prototype learning and the generative power of Large Language Models to provide transparent explanations at three levels: (i) symptom-level explanations for each tweet and user, (ii) case-based explanations comparing the user to similar individuals, and (iii) transparent decision-making through classification weights. Evaluated on five benchmark datasets, ProtoDep achieves near state-of-the-art performance while learning meaningful prototypes. This multi-faceted approach offers significant potential to enhance the reliability and transparency of depression detection on social media, ultimately aiding mental health professionals in delivering more informed care.
• 摘要：抑郁症是一种常见的心理健康问题，需要及时诊断和治疗。尽管社交媒体数据有望用于抑郁症检测，但所采用的深度学习模型的不透明性阻碍了可解释性并引发了偏见问题。我们通过引入 ProtoDep 来解决这一挑战，这是一个基于 Twitter 的抑郁症检测的新颖、可解释的框架。ProtoDep 利用原型学习和大型语言模型的生成能力在三个层面提供透明的解释：(i) 每条推文和用户的症状级解释，(ii) 将用户与类似个体进行比较的基于案例的解释，以及 (iii) 通过分类权重进行透明的决策。在五个基准数据集上进行评估后，ProtoDep 在学习有意义的原型的同时实现了接近最先进的性能。这种多方面的方法具有巨大的潜力，可以提高社交媒体上抑郁症检测的可靠性和透明度，最终帮助心理健康专业人员提供更明智的护理。

Title: CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning

• Authors: Yu Feng, Zhen Tian, Yifan Zhu, Zongfu Han, Haoran Luo, Guangwei Zhang, Meina Song
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21043
• Pdf URL: https://arxiv.org/pdf/2407.21043
• Copy Paste: [[2407.21043]] CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning(https://arxiv.org/abs/2407.21043)
• Keywords: prompt
• Abstract: The key challenge of cross-modal domain-incremental learning (DIL) is to enable the learning model to continuously learn from novel data with different feature distributions under the same task without forgetting old ones. However, existing top-performing methods still cause high forgetting rates, by lacking intra-domain knowledge extraction and inter-domain common prompting strategy. In this paper, we propose a simple yet effective framework, CP-Prompt, by training limited parameters to instruct a pre-trained model to learn new domains and avoid forgetting existing feature distributions. CP-Prompt captures intra-domain knowledge by compositionally inserting personalized prompts on multi-head self-attention layers and then learns the inter-domain knowledge with a common prompting strategy. CP-Prompt shows superiority compared with state-of-the-art baselines among three widely evaluated DIL tasks. The source code is available at this https URL.
• 摘要：跨模态域增量学习 (DIL) 的关键挑战是使学习模型能够在同一任务下从具有不同特征分布的新数据中不断学习，而不会忘记旧数据。然而，现有的表现最佳的方法仍然导致高遗忘率，因为缺乏域内知识提取和域间通用提示策略。在本文中，我们提出了一个简单而有效的框架 CP-Prompt，通过训练有限的参数来指导预训练模型学习新领域并避免忘记现有的特征分布。CP-Prompt 通过在多头自注意层上组合插入个性化提示来捕获域内知识，然后使用通用提示策略学习域间知识。在三个广泛评估的 DIL 任务中，CP-Prompt 与最先进的基线相比表现出优越性。源代码可在此 https URL 中找到。

Title: Unlocking the Potential: Benchmarking Large Language Models in Water Engineering and Research

• Authors: Boyan Xu, Liang Wen, Zihao Li, Yuxing Yang, Guanlan Wu, Xiongpeng Tang, Yu Li, Zihao Wu, Qingxian Su, Xueqing Shi, Yue Yang, Rui Tong, How Yong Ng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21045
• Pdf URL: https://arxiv.org/pdf/2407.21045
• Copy Paste: [[2407.21045]] Unlocking the Potential: Benchmarking Large Language Models in Water Engineering and Research(https://arxiv.org/abs/2407.21045)
• Keywords: language model, gpt, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have sparked interest in their potential applications across various fields. This paper embarked on a pivotal inquiry: Can existing LLMs effectively serve as "water expert models" for water engineering and research tasks? This study was the first to evaluate LLMs' contributions across various water engineering and research tasks by establishing a domain-specific benchmark suite, namely, WaterER. Herein, we prepared 983 tasks related to water engineering and research, categorized into "wastewater treatment", "environmental restoration", "drinking water treatment and distribution", "sanitation", "anaerobic digestion" and "contaminants assessment". We evaluated the performance of seven LLMs (i.e., GPT-4, GPT-3.5, Gemini, GLM-4, ERNIE, QWEN and Llama3) on these tasks. We highlighted the strengths of GPT-4 in handling diverse and complex tasks of water engineering and water research, the specialized capabilities of Gemini in academic contexts, Llama3's strongest capacity to answer Chinese water engineering questions and the competitive performance of Chinese-oriented models like GLM-4, ERNIE and QWEN in some water engineering tasks. More specifically, current LLMs excelled particularly in generating precise research gaps for papers on "contaminants and related water quality monitoring and assessment". Additionally, they were more adept at creating appropriate titles for research papers on "treatment processes for wastewaters", "environmental restoration", and "drinking water treatment". Overall, this study pioneered evaluating LLMs in water engineering and research by introducing the WaterER benchmark to assess the trustworthiness of their predictions. This standardized evaluation framework would also drive future advancements in LLM technology by using targeting datasets, propelling these models towards becoming true "water expert".
• 摘要：大型语言模型 (LLM) 的最新进展引发了人们对其在各个领域的潜在应用的兴趣。本文着手进行一项关键调查：现有的 LLM 能否有效地充当水工程和研究任务的“水专家模型”？这项研究首次通过建立特定领域的基准套件 WaterER 来评估 LLM 在各种水工程和研究任务中的贡献。在此，我们准备了 983 项与水工程和研究相关的任务，分为“废水处理”、“环境修复”、“饮用水处理和分配”、“卫生”、“厌氧消化”和“污染物评估”。我们评估了七个 LLM（即 GPT-4、GPT-3.5、Gemini、GLM-4、ERNIE、QWEN 和 Llama3）在这些任务上的表现。我们重点介绍了 GPT-4 在处理水工程和水研究的多样化和复杂任务方面的优势、Gemini 在学术背景下的专业能力、Llama3 在回答中国水工程问题方面最强的能力以及 GLM-4、ERNIE 和 QWEN 等面向中国的模型在某些水工程任务中的竞争性能。更具体地说，目前的 LLM 尤其擅长为“污染物和相关水质监测和评估”论文生成精确的研究差距。此外，他们更擅长为“废水处理工艺”、“环境修复”和“饮用水处理”等研究论文创建合适的标题。总的来说，这项研究开创了评估水工程和研究领域的 LLM 的先河，通过引入 WaterER 基准来评估其预测的可信度。这种标准化的评估框架还将通过使用目标数据集推动 LLM 技术的未来进步，推动这些模型成为真正的“水专家”。

Title: Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines

• Authors: Yuchen Li, Alexandre Kirchmeyer, Aashay Mehta, Yilong Qin, Boris Dadachev, Kishore Papineni, Sanjiv Kumar, Andrej Risteski
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21046
• Pdf URL: https://arxiv.org/pdf/2407.21046
• Copy Paste: [[2407.21046]] Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines(https://arxiv.org/abs/2407.21046)
• Keywords: language model
• Abstract: Autoregressive language models are the currently dominant paradigm for text generation, but they have some fundamental limitations that cannot be remedied by scale-for example inherently sequential and unidirectional generation. While alternate classes of models have been explored, we have limited mathematical understanding of their fundamental power and limitations. In this paper we focus on Generative Masked Language Models (GMLMs), a non-autoregressive paradigm in which we train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model, These models empirically strike a promising speed-quality trade-off as each step can be typically parallelized by decoding the entire sequence in parallel. We develop a mathematical framework for analyzing and improving such models which sheds light on questions of sample complexity and inference speed and quality. Empirically, we adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality compared with autoregressive models. We run careful ablation experiments to give recommendations on key design choices, and make fine-grained observations on the common error modes in connection with our theory. Our mathematical analyses and empirical observations characterize both potentials and limitations of this approach, and can be applied to future works on improving understanding and performance of GMLMs. Our codes are released at this https URL
• 摘要：自回归语言模型是目前文本生成的主流范式，但它们有一些无法通过规模化解决的基本限制，例如固有的顺序和单向生成。虽然已经探索了其他类型的模型，但我们对它们的基本能力和局限性的数学理解有限。在本文中，我们重点介绍生成式掩蔽语言模型 (GMLM)，这是一种非自回归范式，我们训练模型通过掩蔽来拟合数据分布的条件概率，随后将其用作马尔可夫链的输入以从模型中抽取样本，这些模型在经验上实现了良好的速度-质量权衡，因为每个步骤通常都可以通过并行解码整个序列来实现并行化。我们开发了一个数学框架来分析和改进此类模型，从而阐明了样本复杂性和推理速度和质量的问题。从经验上讲，我们调整了 T5 模型以进行迭代细化的并行解码，与自回归模型相比，在机器翻译中实现了 2-3 倍的速度提升，同时质量损失最小。我们进行了仔细的消融实验，就关键的设计选择给出了建议，并对与我们的理论相关的常见错误模式进行了细致的观察。我们的数学分析和经验观察描述了这种方法的潜力和局限性，并且可以应用于未来提高对 GMLM 的理解和性能的工作。我们的代码发布在此 https URL

Title: APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation

• Authors: Yuxuan Hu, Minghuan Tan, Chenwei Zhang, Zixuan Li, Xiaodan Liang, Min Yang, Chengming Li, Xiping Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21048
• Pdf URL: https://arxiv.org/pdf/2407.21048
• Copy Paste: [[2407.21048]] APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation(https://arxiv.org/abs/2407.21048)
• Keywords: llm
• Abstract: Empathetic response generation is designed to comprehend the emotions of others and select the most appropriate strategies to assist them in resolving emotional challenges. Empathy can be categorized into cognitive empathy and affective empathy. The former pertains to the ability to understand and discern the emotional issues and situations of others, while the latter involves the capacity to provide comfort. To enhance one's empathetic abilities, it is essential to develop both these aspects. Therefore, we develop an innovative framework that combines retrieval augmentation and emotional support strategy integration. Our framework starts with the introduction of a comprehensive emotional palette for empathy. We then apply appraisal theory to decompose this palette and create a database of empathetic responses. This database serves as an external resource and enhances the LLM's empathy by integrating semantic retrieval mechanisms. Moreover, our framework places a strong emphasis on the proper articulation of response strategies. By incorporating emotional support strategies, we aim to enrich the model's capabilities in both cognitive and affective empathy, leading to a more nuanced and comprehensive empathetic response. Finally, we extract datasets ED and ET from the empathetic dialogue dataset \textsc{EmpatheticDialogues} and ExTES based on dialogue length. Experiments demonstrate that our framework can enhance the empathy ability of LLMs from both cognitive and affective empathy perspectives. Our code is released at this https URL.
• 摘要：共情反应生成旨在理解他人的情绪，并选择最合适的策略来帮助他们解决情绪挑战。共情可以分为认知共情和情感共情。前者涉及理解和辨别他人情绪问题和情况的能力，而后者涉及提供安慰的能力。为了提高一个人的共情能力，必须发展这两个方面。因此，我们开发了一个创新框架，结合了检索增强和情感支持策略整合。我们的框架首先引入了一个全面的共情情感调色板。然后，我们应用评估理论来分解这个调色板并创建一个共情反应数据库。该数据库作为外部资源，通过集成语义检索机制来增强 LLM 的共情能力。此外，我们的框架非常重视正确表达反应策略。通过整合情感支持策略，我们旨在丰富模型在认知和情感同理心方面的能力，从而实现更细致入微、更全面的同理心反应。最后，我们从共情对话数据集 \textsc{EmpatheticDialogues} 和 ExTES 中提取基于对话长度的数据集 ED 和 ET。实验表明，我们的框架可以从认知和情感同理心的角度增强 LLM 的同理心能力。我们的代码发布在此 https URL 上。

Title: Evaluating Long Range Dependency Handling in Code Generation Models using Multi-Step Key Retrieval

• Authors: Yannick Assogba, Donghao Ren
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21049
• Pdf URL: https://arxiv.org/pdf/2407.21049
• Copy Paste: [[2407.21049]] Evaluating Long Range Dependency Handling in Code Generation Models using Multi-Step Key Retrieval(https://arxiv.org/abs/2407.21049)
• Keywords: language model, prompt
• Abstract: As language models support larger and larger context sizes, evaluating their ability to make effective use of that context becomes increasingly important. We analyze the ability of several code generation models to handle long range dependencies using a suite of multi-step key retrieval tasks in context windows up to 8k tokens in length. The tasks progressively increase in difficulty and allow more nuanced evaluation of model capabilities than tests like the popular needle-in-the-haystack test. We find that performance degrades significantly (up to 2x) when a function references another function that is defined later in the prompt. We also observe that models that use sliding window attention mechanisms have difficulty handling references further than the size of a single window. We perform simple prompt modifications using call graph information to improve multi-step retrieval performance up to 3x. Our analysis highlights different facets of long-context performance and is suggestive of prompt construction strategies for code completion tools
• 摘要：随着语言模型支持越来越大的上下文大小，评估它们有效利用该上下文的能力变得越来越重要。我们使用一系列多步骤密钥检索任务，在长度高达 8k 个标记的上下文窗口中分析了几种代码生成模型处理长距离依赖关系的能力。这些任务的难度逐渐增加，并且比流行的大海捞针测试等测试更能细致地评估模型的能力。我们发现，当一个函数引用提示中稍后定义的另一个函数时，性能会显著下降（最多 2 倍）。我们还观察到，使用滑动窗口注意机制的模型难以处理超过单个窗口大小的引用。我们使用调用图信息执行简单的提示修改，以将多步骤检索性能提高 3 倍。我们的分析突出了长上下文性能的不同方面，并提出了代码完成工具的提示构建策略

Title: Artificial Intelligence in Extracting Diagnostic Data from Dental Records

• Authors: Yao-Shun Chuang, Chun-Teh Lee, Oluwabunmi Tokede, Guo-Hao Lin, Ryan Brandon, Trung Duong Tran, Xiaoqian Jiang, Muhammad F. Walji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21050
• Pdf URL: https://arxiv.org/pdf/2407.21050
• Copy Paste: [[2407.21050]] Artificial Intelligence in Extracting Diagnostic Data from Dental Records(https://arxiv.org/abs/2407.21050)
• Keywords: gpt, llm
• Abstract: This research addresses the issue of missing structured data in dental records by extracting diagnostic information from unstructured text. The updated periodontology classification system's complexity has increased incomplete or missing structured diagnoses. To tackle this, we use advanced AI and NLP methods, leveraging GPT-4 to generate synthetic notes for fine-tuning a RoBERTa model. This significantly enhances the model's ability to understand medical and dental language. We evaluated the model using 120 randomly selected clinical notes from two datasets, demonstrating its improved diagnostic extraction accuracy. The results showed high accuracy in diagnosing periodontal status, stage, and grade, with Site 1 scoring 0.99 and Site 2 scoring 0.98. In the subtype category, Site 2 achieved perfect scores, outperforming Site 1. This method enhances extraction accuracy and broadens its use across dental contexts. The study underscores AI and NLP's transformative impact on healthcare delivery and management. Integrating AI and NLP technologies enhances documentation and simplifies administrative tasks by precisely extracting complex clinical information. This approach effectively addresses challenges in dental diagnostics. Using synthetic training data from LLMs optimizes the training process, improving accuracy and efficiency in identifying periodontal diagnoses from clinical notes. This innovative method holds promise for broader healthcare applications, potentially improving patient care quality.
• 摘要：本研究通过从非结构化文本中提取诊断信息来解决牙科记录中缺少结构化数据的问题。更新后的牙周病学分类系统的复杂性增加了不完整或缺失的结构化诊断。为了解决这个问题，我们使用先进的 AI 和 NLP 方法，利用 GPT-4 生成合成注释以微调 RoBERTa 模型。这大大增强了模型理解医学和牙科语言的能力。我们使用从两个数据集中随机选择的 120 份临床注释对该模型进行了评估，证明了其诊断提取准确性的提高。结果显示，在诊断牙周状况、阶段和等级方面具有很高的准确性，其中站点 1 得分为 0.99，站点 2 得分为 0.98。在亚型类别中，站点 2 获得了满分，表现优于站点 1。这种方法提高了提取准确性，并扩大了其在牙科环境中的使用范围。该研究强调了 AI 和 NLP 对医疗保健提供和管理的变革性影响。通过集成 AI 和 NLP 技术，可以精确提取复杂的临床信息，从而增强文档记录并简化管理任务。这种方法有效地解决了牙科诊断中的挑战。使用 LLM 的合成训练数据可以优化训练过程，提高从临床记录中识别牙周诊断的准确性和效率。这种创新方法有望应用于更广泛的医疗保健领域，并有可能提高患者护理质量。

Title: An Active Inference Strategy for Prompting Reliable Responses from Large Language Models in Medical Practice

• Authors: Roma Shusterman, Allison C. Waters, Shannon O`Neill, Phan Luu, Don M. Tucker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21051
• Pdf URL: https://arxiv.org/pdf/2407.21051
• Copy Paste: [[2407.21051]] An Active Inference Strategy for Prompting Reliable Responses from Large Language Models in Medical Practice(https://arxiv.org/abs/2407.21051)
• Keywords: language model, llm, prompt, agent
• Abstract: Continuing advances in Large Language Models (LLMs) in artificial intelligence offer important capacities in intuitively accessing and using medical knowledge in many contexts, including education and training as well as assessment and treatment. Most of the initial literature on LLMs in medicine has emphasized that LLMs are unsuitable for medical use because they are non-deterministic, may provide incorrect or harmful responses, and cannot be regulated to assure quality control. If these issues could be corrected, optimizing LLM technology could benefit patients and physicians by providing affordable, point-of-care medical knowledge. Our proposed framework refines LLM responses by restricting their primary knowledge base to domain-specific datasets containing validated medical information. Additionally, we introduce an actor-critic LLM prompting protocol based on active inference principles of human cognition, where a Therapist agent initially responds to patient queries, and a Supervisor agent evaluates and adjusts responses to ensure accuracy and reliability. We conducted a validation study where expert cognitive behaviour therapy for insomnia (CBT-I) therapists evaluated responses from the LLM in a blind format. Experienced human CBT-I therapists assessed responses to 100 patient queries, comparing LLM-generated responses with appropriate and inappropriate responses crafted by experienced CBT-I therapists. Results showed that LLM responses received high ratings from the CBT-I therapists, often exceeding those of therapist-generated appropriate responses. This structured approach aims to integrate advanced LLM technology into medical applications, meeting regulatory requirements for establishing the safe and effective use of special purpose validated LLMs in medicine.
• 摘要：人工智能中大型语言模型 (LLM) 的持续发展为直观地访问和使用医学知识提供了重要的能力，这些知识可用于许多环境，包括教育和培训以及评估和治疗。医学领域中关于 LLM 的大部分初始文献都强调 LLM 不适合医疗用途，因为它们是不确定的，可能会提供不正确或有害的响应，并且无法进行监管以确保质量控制。如果这些问题能够得到纠正，优化 LLM 技术可以通过提供负担得起的即时医疗知识使患者和医生受益。我们提出的框架通过将 LLM 响应的主要知识库限制为包含经过验证的医疗信息的领域特定数据集来改进 LLM 响应。此外，我们引入了一种基于人类认知的主动推理原理的演员-评论家 LLM 提示协议，其中治疗师代理最初响应患者的查询，然后主管代理评估和调整响应以确保准确性和可靠性。我们进行了一项验证研究，其中失眠认知行为疗法 (CBT-I) 专家治疗师以盲目形式评估了 LLM 的回答。经验丰富的人类 CBT-I 治疗师评估了 100 个患者问题的回答，将 LLM 生成的回答与经验丰富的 CBT-I 治疗师精心设计的适当和不适当回答进行了比较。结果表明，LLM 回答得到了 CBT-I 治疗师的高度评价，通常超过了治疗师生成的适当回答。这种结构化方法旨在将先进的 LLM 技术集成到医疗应用中，满足监管要求，以确保在医学领域安全有效地使用经过特殊目的验证的 LLM。

Title: Table-Filling via Mean Teacher for Cross-domain Aspect Sentiment Triplet Extraction

• Authors: Kun Peng, Lei Jiang, Qian Li, Haoran Li, Xiaoyan Yu, Li Sun, Shuo Sun, Yanxian Bi, Hao Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21052
• Pdf URL: https://arxiv.org/pdf/2407.21052
• Copy Paste: [[2407.21052]] Table-Filling via Mean Teacher for Cross-domain Aspect Sentiment Triplet Extraction(https://arxiv.org/abs/2407.21052)
• Keywords: language model
• Abstract: Cross-domain Aspect Sentiment Triplet Extraction (ASTE) aims to extract fine-grained sentiment elements from target domain sentences by leveraging the knowledge acquired from the source domain. Due to the absence of labeled data in the target domain, recent studies tend to rely on pre-trained language models to generate large amounts of synthetic data for training purposes. However, these approaches entail additional computational costs associated with the generation process. Different from them, we discover a striking resemblance between table-filling methods in ASTE and two-stage Object Detection (OD) in computer vision, which inspires us to revisit the cross-domain ASTE task and approach it from an OD standpoint. This allows the model to benefit from the OD extraction paradigm and region-level alignment. Building upon this premise, we propose a novel method named \textbf{T}able-\textbf{F}illing via \textbf{M}ean \textbf{T}eacher (TFMT). Specifically, the table-filling methods encode the sentence into a 2D table to detect word relations, while TFMT treats the table as a feature map and utilizes a region consistency to enhance the quality of those generated pseudo labels. Additionally, considering the existence of the domain gap, a cross-domain consistency based on Maximum Mean Discrepancy is designed to alleviate domain shift problems. Our method achieves state-of-the-art performance with minimal parameters and computational costs, making it a strong baseline for cross-domain ASTE.
• 摘要：跨域方面情感三元组提取 (ASTE) 旨在利用从源域获得的知识从目标域句子中提取细粒度情感元素。由于目标域中缺乏标记数据，最近的研究倾向于依赖预训练语言模型来生成大量合成数据用于训练目的。然而，这些方法需要与生成过程相关的额外计算成本。与它们不同，我们发现 ASTE 中的填表方法与计算机视觉中的两阶段对象检测 (OD) 之间有着惊人的相似之处，这启发我们重新审视跨域 ASTE 任务并从 OD 的角度来处理它。这使模型能够从 OD 提取范例和区域级对齐中受益。在此前提下，我们提出了一种名为 \textbf{T}able-\textbf{F}illing via \textbf{M}ean \textbf{T}eacher (TFMT) 的新方法。具体来说，表格填充方法将句子编码为 2D 表格以检测单词关系，而 TFMT 将表格视为特征图并利用区域一致性来提高生成的伪标签的质量。此外，考虑到域间隙的存在，设计了基于最大均值差异的跨域一致性来缓解域转移问题。我们的方法以最少的参数和计算成本实现了最先进的性能，使其成为跨域 ASTE 的强大基线。

Title: Sentiment Reasoning for Healthcare

• Authors: Khai Le-Duc, Khai-Nguyen Nguyen, Bach Phan Tat, Duy Le, Jerry Ngo, Long Vo-Dang, Anh Totti Nguyen, Truong-Son Hy
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.21054
• Pdf URL: https://arxiv.org/pdf/2407.21054
• Copy Paste: [[2407.21054]] Sentiment Reasoning for Healthcare(https://arxiv.org/abs/2407.21054)
• Keywords: language model, llm
• Abstract: Transparency in AI decision-making is crucial in healthcare due to the severe consequences of errors, and this is important for building trust among AI and users in sentiment analysis task. Incorporating reasoning capabilities helps Large Language Models (LLMs) understand human emotions within broader contexts, handle nuanced and ambiguous language, and infer underlying sentiments that may not be explicitly stated. In this work, we introduce a new task - Sentiment Reasoning - for both speech and text modalities, along with our proposed multimodal multitask framework and dataset. Our study showed that rationale-augmented training enhances model performance in sentiment classification across both human transcript and ASR settings. Also, we found that the generated rationales typically exhibit different vocabularies compared to human-generated rationales, but maintain similar semantics. All code, data (English-translated and Vietnamese) and models are published online: this https URL
• 摘要：由于错误会导致严重后果，因此人工智能决策的透明度在医疗保健领域至关重要，这对于在情绪分析任务中建立人工智能和用户之间的信任非常重要。结合推理能力有助于大型语言模型 (LLM) 在更广泛的背景下理解人类情感，处理细微差别和模棱两可的语言，并推断可能未明确说明的潜在情绪。在这项工作中，我们为语音和文本模式引入了一项新任务——情绪推理，以及我们提出的多模态多任务框架和数据集。我们的研究表明，增强理论的训练可提高模型在人类转录和 ASR 设置中的情绪分类性能。此外，我们发现生成的理论通常与人类生成的理论相比表现出不同的词汇，但保持相似的语义。所有代码、数据（英语翻译和越南语）和模型均在线发布：此 https URL

Title: Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applications

• Authors: Cui Long, Yongbin Liu, Chunping Ouyang, Ying Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21055
• Pdf URL: https://arxiv.org/pdf/2407.21055
• Copy Paste: [[2407.21055]] Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applications(https://arxiv.org/abs/2407.21055)
• Keywords: language model, gpt, llm, hallucination, prompt, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have exhibited remarkable proficiency in natural language understanding, prompting extensive exploration of their potential applications across diverse domains. In the medical domain, open-source LLMs have demonstrated moderate efficacy following domain-specific fine-tuning; however, they remain substantially inferior to proprietary models such as GPT-4 and GPT-3.5. These open-source models encounter limitations in the comprehensiveness of domain-specific knowledge and exhibit a propensity for 'hallucinations' during text generation. To mitigate these issues, researchers have implemented the Retrieval-Augmented Generation (RAG) approach, which augments LLMs with background information from external knowledge bases while preserving the model's internal parameters. However, document noise can adversely affect performance, and the application of RAG in the medical field remains in its nascent stages. This study presents the Bailicai framework: a novel integration of retrieval-augmented generation with large language models optimized for the medical domain. The Bailicai framework augments the performance of LLMs in medicine through the implementation of four sub-modules. Experimental results demonstrate that the Bailicai approach surpasses existing medical domain LLMs across multiple medical benchmarks and exceeds the performance of GPT-3.5. Furthermore, the Bailicai method effectively attenuates the prevalent issue of hallucinations in medical applications of LLMs and ameliorates the noise-related challenges associated with traditional RAG techniques when processing irrelevant or pseudo-relevant documents.
• 摘要：大型语言模型 (LLM) 在自然语言理解方面表现出色，这促使人们广泛探索其在不同领域的潜在应用。在医学领域，开源 LLM 在经过领域特定微调后表现出中等效果；然而，它们仍然远远不如 GPT-4 和 GPT-3.5 等专有模型。这些开源模型在领域特定知识的全面性方面受到限制，并且在文本生成过程中表现出“幻觉”倾向。为了缓解这些问题，研究人员实施了检索增强生成 (RAG) 方法，该方法使用来自外部知识库的背景信息增强 LLM，同时保留模型的内部参数。然而，文档噪声会对性能产生不利影响，并且 RAG 在医学领域的应用仍处于起步阶段。本研究提出了 Bailicai 框架：一种针对医学领域优化的检索增强生成与大型语言模型的新集成。百里菜框架通过实现四个子模块来增强医学领域 LLM 的性能。实验结果表明，百里菜方法在多个医学基准上超越了现有的医学领域 LLM，并超过了 GPT-3.5 的性能。此外，百里菜方法有效地减轻了 LLM 在医学应用中普遍存在的幻觉问题，并改善了传统 RAG 技术在处理不相关或伪相关文档时与噪声相关的挑战。

Title: Multi-group Uncertainty Quantification for Long-form Text Generation

• Authors: Terrance Liu, Zhiwei Steven Wu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21057
• Pdf URL: https://arxiv.org/pdf/2407.21057
• Copy Paste: [[2407.21057]] Multi-group Uncertainty Quantification for Long-form Text Generation(https://arxiv.org/abs/2407.21057)
• Keywords: language model, llm, hallucination, prompt
• Abstract: While large language models are rapidly moving towards consumer-facing applications, they are often still prone to factual errors and hallucinations. In order to reduce the potential harms that may come from these errors, it is important for users to know to what extent they can trust an LLM when it makes a factual claim. To this end, we study the problem of uncertainty quantification of factual correctness in long-form natural language generation. Given some output from a large language model, we study both uncertainty at the level of individual claims contained within the output (via calibration) and uncertainty across the entire output itself (via conformal prediction). Moreover, we invoke multicalibration and multivalid conformal prediction to ensure that such uncertainty guarantees are valid both marginally and across distinct groups of prompts. Using the task of biography generation, we demonstrate empirically that having access to and making use of additional group attributes for each prompt improves both overall and group-wise performance. As the problems of calibration, conformal prediction, and their multi-group counterparts have not been extensively explored previously in the context of long-form text generation, we consider these empirical results to form a benchmark for this setting.
• 摘要：虽然大型语言模型正在迅速转向面向消费者的应用程序，但它们通常仍然容易出现事实错误和幻觉。为了减少这些错误可能带来的潜在危害，用户必须知道当 LLM 做出事实声明时，他们可以在多大程度上信任它。为此，我们研究了长篇自然语言生成中事实正确性的不确定性量化问题。给定一个大型语言模型的一些输出，我们研究输出中包含的单个声明级别的不确定性（通过校准）和整个输出本身的不确定性（通过共形预测）。此外，我们调用多重校准和多重有效共形预测来确保这种不确定性保证在边际和不同提示组中都有效。使用传记生成任务，我们通过经验证明，访问和利用每个提示的额外组属性可以提高整体和组级性能。由于校准、共形预测及其多组对应问题以前在长文本生成的背景下尚未得到广泛探索，我们将这些实证结果视为该设置的基准。

Title: Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

• Authors: Muhammad Ali, Swetasudha Panda, Qinlan Shen, Michael Wick, Ari Kobren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21058
• Pdf URL: https://arxiv.org/pdf/2407.21058
• Copy Paste: [[2407.21058]] Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT(https://arxiv.org/abs/2407.21058)
• Keywords: language model
• Abstract: In the current landscape of language model research, larger models, larger datasets and more compute seems to be the only way to advance towards intelligence. While there have been extensive studies of scaling laws and models' scaling behaviors, the effect of scale on a model's social biases and stereotyping tendencies has received less attention. In this study, we explore the influence of model scale and pre-training data on its learnt social biases. We focus on BERT -- an extremely popular language model -- and investigate biases as they show up during language modeling (upstream), as well as during classification applications after fine-tuning (downstream). Our experiments on four architecture sizes of BERT demonstrate that pre-training data substantially influences how upstream biases evolve with model scale. With increasing scale, models pre-trained on large internet scrapes like Common Crawl exhibit higher toxicity, whereas models pre-trained on moderated data sources like Wikipedia show greater gender stereotypes. However, downstream biases generally decrease with increasing model scale, irrespective of the pre-training data. Our results highlight the qualitative role of pre-training data in the biased behavior of language models, an often overlooked aspect in the study of scale. Through a detailed case study of BERT, we shed light on the complex interplay of data and model scale, and investigate how it translates to concrete biases.
• 摘要：在当前的语言模型研究领域，更大的模型、更大的数据集和更多的计算似乎是迈向智能的唯一途径。虽然已经对缩放定律和模型的缩放行为进行了广泛的研究，但规模对模型的社会偏见和刻板印象倾向的影响却受到较少的关注。在这项研究中，我们探讨了模型规模和预训练数据对其学习到的社会偏见的影响。我们专注于 BERT——一种非常流行的语言模型——并研究语言建模过程中（上游）以及微调后的分类应用（下游）中出现的偏见。我们对 BERT 的四种架构大小的实验表明，预训练数据会显著影响上游偏见如何随模型规模而演变。随着规模的增加，在大型互联网抓取（如 Common Crawl）上进行预训练的模型表现出更高的毒性，而在维基百科等经过审核的数据源上进行预训练的模型表现出更大的性别刻板印象。然而，无论预训练数据如何，下游偏见通常会随着模型规模的增加而减少。我们的研究结果强调了预训练数据在语言模型的偏差行为中的定性作用，这是规模研究中经常被忽视的一个方面。通过对 BERT 的详细案例研究，我们阐明了数据和模型规模之间的复杂相互作用，并研究了它如何转化为具体的偏差。

Title: Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks

• Authors: Yunfan Gao, Yun Xiong, Meng Wang, Haofen Wang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.21059
• Pdf URL: https://arxiv.org/pdf/2407.21059
• Copy Paste: [[2407.21059]] Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks(https://arxiv.org/abs/2407.21059)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented Generation (RAG) has markedly enhanced the capabilities of Large Language Models (LLMs) in tackling knowledge-intensive tasks. The increasing demands of application scenarios have driven the evolution of RAG, leading to the integration of advanced retrievers, LLMs and other complementary technologies, which in turn has amplified the intricacy of RAG systems. However, the rapid advancements are outpacing the foundational RAG paradigm, with many methods struggling to be unified under the process of "retrieve-then-generate". In this context, this paper examines the limitations of the existing RAG paradigm and introduces the modular RAG framework. By decomposing complex RAG systems into independent modules and specialized operators, it facilitates a highly reconfigurable framework. Modular RAG transcends the traditional linear architecture, embracing a more advanced design that integrates routing, scheduling, and fusion mechanisms. Drawing on extensive research, this paper further identifies prevalent RAG patterns-linear, conditional, branching, and looping-and offers a comprehensive analysis of their respective implementation nuances. Modular RAG presents innovative opportunities for the conceptualization and deployment of RAG systems. Finally, the paper explores the potential emergence of new operators and paradigms, establishing a solid theoretical foundation and a practical roadmap for the continued evolution and practical deployment of RAG technologies.
• 摘要：检索增强生成 (RAG) 显著增强了大型语言模型 (LLM) 处理知识密集型任务的能力。应用场景的需求不断增长，推动了 RAG 的演进，从而集成了高级检索器、LLM 和其他互补技术，这反过来又增加了 RAG 系统的复杂性。然而，快速的发展正在超越基础 RAG 范式，许多方法在“检索然后生成”的流程下难以统一。在此背景下，本文研究了现有 RAG 范式的局限性，并介绍了模块化 RAG 框架。通过将复杂的 RAG 系统分解为独立的模块和专门的操作符，它有助于实现高度可重构的框架。模块化 RAG 超越了传统的线性架构，采用了一种集成路由、调度和融合机制的更高级设计。本文基于广泛的研究，进一步确定了流行的 RAG 模式（线性、条件、分支和循环），并全面分析了它们各自的实现细节。模块化 RAG 为 RAG 系统的概念化和部署提供了创新机会。最后，本文探讨了新运算符和新范式的潜在出现，为 RAG 技术的持续发展和实际部署奠定了坚实的理论基础和实用路线图。

Title: Using Large Language Models for the Interpretation of Building Regulations

• Authors: Stefan Fuchs, Michael Witbrock, Johannes Dimyadi, Robert Amor
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21060
• Pdf URL: https://arxiv.org/pdf/2407.21060
• Copy Paste: [[2407.21060]] Using Large Language Models for the Interpretation of Building Regulations(https://arxiv.org/abs/2407.21060)
• Keywords: language model, gpt, llm, prompt, chat, chain-of-thought
• Abstract: Compliance checking is an essential part of a construction project. The recent rapid uptake of building information models (BIM) in the construction industry has created more opportunities for automated compliance checking (ACC). BIM enables sharing of digital building design data that can be used for compliance checking with legal requirements, which are conventionally conveyed in natural language and not intended for machine processing. Creating a computable representation of legal requirements suitable for ACC is complex, costly, and time-consuming. Large language models (LLMs) such as the generative pre-trained transformers (GPT), GPT-3.5 and GPT-4, powering OpenAI's ChatGPT, can generate logically coherent text and source code responding to user prompts. This capability could be used to automate the conversion of building regulations into a semantic and computable representation. This paper evaluates the performance of LLMs in translating building regulations into LegalRuleML in a few-shot learning setup. By providing GPT-3.5 with only a few example translations, it can learn the basic structure of the format. Using a system prompt, we further specify the LegalRuleML representation and explore the existence of expert domain knowledge in the model. Such domain knowledge might be ingrained in GPT-3.5 through the broad pre-training but needs to be brought forth by careful contextualisation. Finally, we investigate whether strategies such as chain-of-thought reasoning and self-consistency could apply to this use case. As LLMs become more sophisticated, the increased common sense, logical coherence, and means to domain adaptation can significantly support ACC, leading to more efficient and effective checking processes.
• 摘要：合规性检查是建筑项目的重要组成部分。最近，建筑信息模型 (BIM) 在建筑行业的快速普及为自动化合规性检查 (ACC) 创造了更多机会。BIM 支持共享数字建筑设计数据，这些数据可用于检查法律要求的合规性，而法律要求通常以自然语言传达，不适用于机器处理。创建适合 ACC 的法律要求的可计算表示非常复杂、昂贵且耗时。大型语言模型 (LLM)，例如生成式预训练转换器 (GPT)、GPT-3.5 和 GPT-4，为 OpenAI 的 ChatGPT 提供支持，可以生成逻辑上连贯的文本和源代码以响应用户提示。此功能可用于自动将建筑法规转换为语义和可计算的表示。本文评估了 LLM 在少量学习设置中将建筑法规翻译成 LegalRuleML 的性能。通过为 GPT-3.5 提供少量示例翻译，它可以学习格式的基本结构。使用系统提示，我们进一步指定 LegalRuleML 表示并探索模型中专家领域知识的存在。此类领域知识可能通过广泛的预训练根植于 GPT-3.5 中，但需要通过仔细的语境化才能实现。最后，我们研究了诸如思路链推理和自洽性之类的策略是否可以适用于此用例。随着 LLM 变得越来越复杂，增加的常识、逻辑连贯性和领域适应手段可以显著支持 ACC，从而实现更高效、更有效的检查流程。

Title: LawLLM: Law Large Language Model for the US Legal System

• Authors: Dong Shu, Haoran Zhao, Xukun Liu, David Demeter, Mengnan Du, Yongfeng Zhang
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21065
• Pdf URL: https://arxiv.org/pdf/2407.21065
• Copy Paste: [[2407.21065]] LawLLM: Law Large Language Model for the US Legal System(https://arxiv.org/abs/2407.21065)
• Keywords: language model, llm
• Abstract: In the rapidly evolving field of legal analytics, finding relevant cases and accurately predicting judicial outcomes are challenging because of the complexity of legal language, which often includes specialized terminology, complex syntax, and historical context. Moreover, the subtle distinctions between similar and precedent cases require a deep understanding of legal knowledge. Researchers often conflate these concepts, making it difficult to develop specialized techniques to effectively address these nuanced tasks. In this paper, we introduce the Law Large Language Model (LawLLM), a multi-task model specifically designed for the US legal domain to address these challenges. LawLLM excels at Similar Case Retrieval (SCR), Precedent Case Recommendation (PCR), and Legal Judgment Prediction (LJP). By clearly distinguishing between precedent and similar cases, we provide essential clarity, guiding future research in developing specialized strategies for these tasks. We propose customized data preprocessing techniques for each task that transform raw legal data into a trainable format. Furthermore, we also use techniques such as in-context learning (ICL) and advanced information retrieval methods in LawLLM. The evaluation results demonstrate that LawLLM consistently outperforms existing baselines in both zero-shot and few-shot scenarios, offering unparalleled multi-task capabilities and filling critical gaps in the legal domain.
• 摘要：在快速发展的法律分析领域，由于法律语言的复杂性，查找相关案例并准确预测司法结果具有挑战性，法律语言通常包括专业术语、复杂语法和历史背景。此外，类似案件和先例案件之间的细微差别需要对法律知识有深入的理解。研究人员经常混淆这些概念，因此很难开发专门的技术来有效地解决这些细微的任务。在本文中，我们介绍了法律大型语言模型 (LawLLM)，这是一种专为美国法律领域设计的多任务模型，旨在应对这些挑战。LawLLM 擅长类似案件检索 (SCR)、先例案件推荐 (PCR) 和法律判决预测 (LJP)。通过明确区分先例和类似案件，我们提供了必要的清晰度，指导未来研究为这些任务制定专门的策略。我们为每个任务提出了定制的数据预处理技术，将原始法律数据转换为可训练的格式。此外，我们还在 LawLLM 中使用了上下文学习 (ICL) 和高级信息检索方法等技术。评估结果表明，LawLLM 在零样本和少样本场景中始终优于现有基线，提供了无与伦比的多任务能力并填补了法律领域的关键空白。

Title: ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks

• Authors: Nakamasa Inoue, Shinta Otake, Takumi Hirose, Masanari Ohi, Rei Kawakami
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.21066
• Pdf URL: https://arxiv.org/pdf/2407.21066
• Copy Paste: [[2407.21066]] ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks(https://arxiv.org/abs/2407.21066)
• Keywords: prompt
• Abstract: Self-supervised learning has emerged as a key approach for learning generic representations from speech data. Despite promising results in downstream tasks such as speech recognition, speaker verification, and emotion recognition, a significant number of parameters is required, which makes fine-tuning for each task memory-inefficient. To address this limitation, we introduce ELP-adapter tuning, a novel method for parameter-efficient fine-tuning using three types of adapter, namely encoder adapters (E-adapters), layer adapters (L-adapters), and a prompt adapter (P-adapter). The E-adapters are integrated into transformer-based encoder layers and help to learn fine-grained speech representations that are effective for speech recognition. The L-adapters create paths from each encoder layer to the downstream head and help to extract non-linguistic features from lower encoder layers that are effective for speaker verification and emotion recognition. The P-adapter appends pseudo features to CNN features to further improve effectiveness and efficiency. With these adapters, models can be quickly adapted to various speech processing tasks. Our evaluation across four downstream tasks using five backbone models demonstrated the effectiveness of the proposed method. With the WavLM backbone, its performance was comparable to or better than that of full fine-tuning on all tasks while requiring 90% fewer learnable parameters.
• 摘要：自监督学习已成为从语音数据中学习通用表示的关键方法。尽管在语音识别、说话人验证和情感识别等下游任务中取得了令人鼓舞的结果，但需要大量参数，这使得针对每个任务进行微调的内存效率低下。为了解决这一限制，我们引入了 ELP 适配器调整，这是一种使用三种类型的适配器进行参数高效微调的新方法，即编码器适配器（E 适配器）、层适配器（L 适配器）和提示适配器（P 适配器）。E 适配器集成到基于变压器的编码器层中，有助于学习对语音识别有效的细粒度语音表示。L 适配器从每个编码器层创建到下游头部的路径，并有助于从较低的编码器层中提取对说话人验证和情感识别有效的非语言特征。P 适配器将伪特征附加到 CNN 特征，以进一步提高有效性和效率。借助这些适配器，模型可以快速适应各种语音处理任务。我们使用五个主干模型对四个下游任务进行了评估，证明了所提方法的有效性。使用 WavLM 主干，其性能与所有任务上的完全微调相当或更好，同时所需的可学习参数减少了 90%。

Title: Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models

• Authors: Somshubra Majumdar, Vahid Noroozi, Sean Narenthiran, Aleksander Ficek, Jagadeesh Balam, Boris Ginsburg
• Subjects: cs.CL, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2407.21077
• Pdf URL: https://arxiv.org/pdf/2407.21077
• Copy Paste: [[2407.21077]] Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models(https://arxiv.org/abs/2407.21077)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) rely on instruction samples for alignment, but creating these datasets poses challenges, particularly in expert-dependent tasks like coding, which can be cost-prohibitive. One approach to mitigate these challenges is synthesizing data using another LLM. In this paper, we introduce a scalable method for generating synthetic instructions to enhance the code generation capability of LLMs. The proposed algorithm, Genetic-Instruct, mimics evolutionary processes, utilizing self-instruction to create numerous synthetic samples from a limited number of seeds. Genetic-Instruct is designed for efficient scaling of the generation process. Fine-tuning multiple coding LLMs with the synthetic samples demonstrates a significant improvement in their code generation accuracy compared to the baselines.
• 摘要：大型语言模型 (LLM) 依靠指令样本进行对齐，但创建这些数据集会带来挑战，特别是在编码等依赖专家的任务中，成本可能过高。缓解这些挑战的一种方法是使用另一个 LLM 合成数据。在本文中，我们介绍了一种可扩展的方法来生成合成指令，以增强 LLM 的代码生成能力。所提出的算法 Genetic-Instruct 模仿进化过程，利用自我指令从有限数量的种子中创建大量合成样本。Genetic-Instruct 旨在有效扩展生成过程。使用合成样本对多个编码 LLM 进行微调，与基线相比，它们的代码生成准确性有显著提高。

Title: Accelerating Large Language Model Inference with Self-Supervised Early Exits

• Authors: Florian Valade
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.21082
• Pdf URL: https://arxiv.org/pdf/2407.21082
• Copy Paste: [[2407.21082]] Accelerating Large Language Model Inference with Self-Supervised Early Exits(https://arxiv.org/abs/2407.21082)
• Keywords: language model, llm
• Abstract: This paper presents a novel technique for accelerating inference in large, pre-trained language models (LLMs) by introducing early exits during inference. The computational demands of these models, used across a wide range of applications, can be substantial. By capitalizing on the inherent variability in token complexity, our approach enables selective acceleration of the inference process. Specifically, we propose the integration of early exit ''heads'' atop existing transformer layers, which facilitate conditional terminations based on a confidence metric. These heads are trained in a self-supervised manner using the model's own predictions as training data, thereby eliminating the need for additional annotated data. The confidence metric, established using a calibration set, ensures a desired level of accuracy while enabling early termination when confidence exceeds a predetermined threshold. Notably, our method preserves the original accuracy and reduces computational time on certain tasks, leveraging the existing knowledge of pre-trained LLMs without requiring extensive retraining. This lightweight, modular modification has the potential to greatly enhance the practical usability of LLMs, particularly in applications like real-time language processing in resource-constrained environments.
• 摘要：本文介绍了一种通过在推理过程中引入提前退出来加速大型预训练语言模型 (LLM) 推理的新技术。这些模型用于广泛的应用，其计算需求可能很大。通过利用 token 复杂性的固有可变性，我们的方法可以有选择地加速推理过程。具体来说，我们建议在现有的 Transformer 层之上集成提前退出的“头部”，这有助于基于置信度指标进行条件终止。这些头部以自监督的方式进行训练，使用模型自身的预测作为训练数据，从而无需额外的注释数据。使用校准集建立的置信度指标可确保所需的准确度，同时在置信度超过预定阈值时允许提前终止。值得注意的是，我们的方法保留了原始准确度并减少了某些任务的计算时间，利用了预训练 LLM 的现有知识，而无需进行大量的再训练。这种轻量级、模块化的修改有可能极大地增强 LLM 的实际可用性，特别是在资源受限环境中的实时语言处理等应用中。

Title: Entropy, Thermodynamics and the Geometrization of the Language Model

• Authors: Wenzhe Yang
• Subjects: cs.CL, cond-mat.stat-mech, hep-th, math.DG
• Abstract URL: https://arxiv.org/abs/2407.21092
• Pdf URL: https://arxiv.org/pdf/2407.21092
• Copy Paste: [[2407.21092]] Entropy, Thermodynamics and the Geometrization of the Language Model(https://arxiv.org/abs/2407.21092)
• Keywords: language model, llm
• Abstract: In this paper, we discuss how pure mathematics and theoretical physics can be applied to the study of language models. Using set theory and analysis, we formulate mathematically rigorous definitions of language models, and introduce the concept of the moduli space of distributions for a language model. We formulate a generalized distributional hypothesis using functional analysis and topology. We define the entropy function associated with a language model and show how it allows us to understand many interesting phenomena in languages. We argue that the zero points of the entropy function and the points where the entropy is close to 0 are the key obstacles for an LLM to approximate an intelligent language model, which explains why good LLMs need billions of parameters. Using the entropy function, we formulate a conjecture about AGI. Then, we show how thermodynamics gives us an immediate interpretation to language models. In particular we will define the concepts of partition function, internal energy and free energy for a language model, which offer insights into how language models work. Based on these results, we introduce a general concept of the geometrization of language models and define what is called the Boltzmann manifold. While the current LLMs are the special cases of the Boltzmann manifold.
• 摘要：在本文中，我们讨论了纯数学和理论物理如何应用于语言模型的研究。利用集合论和分析，我们制定了语言模型的数学上严格的定义，并引入了语言模型分布模空间的概念。我们使用泛函分析和拓扑制定了一个广义分布假设。我们定义了与语言模型相关的熵函数，并展示了它如何让我们理解语言中许多有趣的现象。我们认为熵函数的零点和熵接近于 0 的点是 LLM 近似智能语言模型的关键障碍，这解释了为什么好的 LLM 需要数十亿个参数。利用熵函数，我们制定了关于 AGI 的猜想。然后，我们展示了热力学如何为我们提供对语言模型的直接解释。特别是，我们将为语言模型定义配分函数、内部能量和自由能的概念，这些概念为语言模型的工作原理提供了见解。基于这些结果，我们引入了语言模型几何化的一般概念，并定义了所谓的玻尔兹曼流形。而当前的 LLM 是玻尔兹曼流形的特例。

Title: Event-Arguments Extraction Corpus and Modeling using BERT for Arabic

• Authors: Alaa Aljabari, Lina Duaibes, Mustafa Jarrar, Mohammed Khalilia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21153
• Pdf URL: https://arxiv.org/pdf/2407.21153
• Copy Paste: [[2407.21153]] Event-Arguments Extraction Corpus and Modeling using BERT for Arabic(https://arxiv.org/abs/2407.21153)
• Keywords: agent
• Abstract: Event-argument extraction is a challenging task, particularly in Arabic due to sparse linguistic resources. To fill this gap, we introduce the \hadath corpus ($550$k tokens) as an extension of Wojood, enriched with event-argument annotations. We used three types of event arguments: $agent$, $location$, and $date$, which we annotated as relation types. Our inter-annotator agreement evaluation resulted in $82.23\%$ $Kappa$ score and $87.2\%$ $F_1$-score. Additionally, we propose a novel method for event relation extraction using BERT, in which we treat the task as text entailment. This method achieves an $F_1$-score of $94.01\%$. To further evaluate the generalization of our proposed method, we collected and annotated another out-of-domain corpus (about $80$k tokens) called \testNLI and used it as a second test set, on which our approach achieved promising results ($83.59\%$ $F_1$-score). Last but not least, we propose an end-to-end system for event-arguments extraction. This system is implemented as part of SinaTools, and both corpora are publicly available at {\small \url{this https URL}}
• 摘要：事件论元提取是一项具有挑战性的任务，尤其是在阿拉伯语中，因为语言资源稀缺。为了填补这一空白，我们引入了 \hadath 语料库（$550$k 个 token）作为 Wojood 的扩展，并丰富了事件论元注释。我们使用了三种类型的事件论元：$agent$、$location$ 和 $date$，我们将其注释为关系类型。我们的注释者间一致性评估结果为 $82.23\%$ $Kappa$ 分数和 $87.2\%$ $F_1$ 分数。此外，我们提出了一种使用 BERT 进行事件关系提取的新方法，其中我们将任务视为文本蕴涵。该方法的 $F_1$ 分数为 $94.01\%$。为了进一步评估我们提出的方法的泛化能力，我们收集并注释了另一个名为 \testNLI 的域外语料库（约 $80$k 个标记），并将其用作第二个测试集，我们的方法在该语料库上取得了令人满意的结果（$83.59\%$ $F_1$ 得分）。最后但并非最不重要的是，我们提出了一个用于事件参数提取的端到端系统。该系统是作为 SinaTools 的一部分实现的，两个语料库均可在 {\small \url{this https URL}} 上公开获取

Title: Decomposed Prompting to Answer Questions on a Course Discussion Board

• Authors: Brandon Jaipersaud, Paul Zhang, Jimmy Ba, Andrew Petersen, Lisa Zhang, Michael R. Zhang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.21170
• Pdf URL: https://arxiv.org/pdf/2407.21170
• Copy Paste: [[2407.21170]] Decomposed Prompting to Answer Questions on a Course Discussion Board(https://arxiv.org/abs/2407.21170)
• Keywords: language model, gpt, llm, prompt
• Abstract: We propose and evaluate a question-answering system that uses decomposed prompting to classify and answer student questions on a course discussion board. Our system uses a large language model (LLM) to classify questions into one of four types: conceptual, homework, logistics, and not answerable. This enables us to employ a different strategy for answering questions that fall under different types. Using a variant of GPT-3, we achieve $81\%$ classification accuracy. We discuss our system's performance on answering conceptual questions from a machine learning course and various failure modes.
• 摘要：我们提出并评估了一个问答系统，该系统使用分解提示对课程讨论板上的学生问题进行分类和回答。我们的系统使用大型语言模型 (LLM) 将问题分为四种类型之一：概念、家庭作业、物流和不可回答。这使我们能够采用不同的策略来回答属于不同类型的问题。使用 GPT-3 的变体，我们实现了 $81\%$ 的分类准确率。我们讨论了我们的系统在回答机器学习课程中的概念问题方面的表现以及各种故障模式。

Title: Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens

• Authors: Anqi Zhang, Chaofeng Wu
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21248
• Pdf URL: https://arxiv.org/pdf/2407.21248
• Copy Paste: [[2407.21248]] Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens(https://arxiv.org/abs/2407.21248)
• Keywords: language model, llm
• Abstract: While large language models (LLMs) are extensively used, there are raising concerns regarding privacy, security, and copyright due to their opaque training data, which brings the problem of detecting pre-training data on the table. Current solutions to this problem leverage techniques explored in machine learning privacy such as Membership Inference Attacks (MIAs), which heavily depend on LLMs' capability of verbatim memorization. However, this reliance presents challenges, especially given the vast amount of training data and the restricted number of effective training epochs. In this paper, we propose an adaptive pre-training data detection method which alleviates this reliance and effectively amplify the identification. Our method adaptively locates \textit{surprising tokens} of the input. A token is surprising to a LLM if the prediction on the token is "certain but wrong", which refers to low Shannon entropy of the probability distribution and low probability of the ground truth token at the same time. By using the prediction probability of surprising tokens to measure \textit{surprising}, the detection method is achieved based on the simple hypothesis that seeing seen data is less surprising for the model compared with seeing unseen data. The method can be applied without any access to the the pre-training data corpus or additional training like reference models. Our approach exhibits a consistent enhancement compared to existing methods in diverse experiments conducted on various benchmarks and models, achieving a maximum improvement of 29.5\%. We also introduce a new benchmark Dolma-Book developed upon a novel framework, which employs book data collected both before and after model training to provide further evaluation.
• 摘要：虽然大型语言模型 (LLM) 被广泛使用，但由于其不透明的训练数据，人们越来越担心隐私、安全和版权问题，这带来了检测预训练数据的问题。目前，解决这个问题的方法是利用机器学习隐私领域探索的技术，例如成员推理攻击 (MIA)，这在很大程度上依赖于 LLM 的逐字记忆能力。然而，这种依赖带来了挑战，尤其是在训练数据量巨大、有效训练次数有限的情况下。在本文中，我们提出了一种自适应的预训练数据检测方法，可以减轻这种依赖并有效地扩大识别范围。我们的方法可以自适应地定位输入的 \textit{令人惊讶的标记}。如果对标记的预测是“确定但错误的”，则该标记对于 LLM 来说是令人惊讶的，这指的是概率分布的香农熵低和地面真实标记的概率低。通过使用令人惊讶的标记的预测概率来测量 \textit{surprising}，检测方法基于一个简单的假设，即看到看到的数据对模型来说比看到看不见的数据更不令人惊讶。该方法可以在不访问预训练数据语料库或参考模型等额外训练的情况下应用。我们的方法与现有方法相比，在对各种基准和模型进行的各种实验中表现出一致的增强，最大改进为 29.5\%。我们还介绍了一个基于新框架开发的新基准 Dolma-Book，它使用模型训练前后收集的书籍数据来提供进一步的评估。

Title: Model Attribution in Machine-Generated Disinformation: A Domain Generalization Approach with Supervised Contrastive Learning

• Authors: Alimohammad Beigi, Zhen Tan, Nivedh Mudiam, Canyu Chen, Kai Shu, Huan Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21264
• Pdf URL: https://arxiv.org/pdf/2407.21264
• Copy Paste: [[2407.21264]] Model Attribution in Machine-Generated Disinformation: A Domain Generalization Approach with Supervised Contrastive Learning(https://arxiv.org/abs/2407.21264)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Model attribution for machine-generated disinformation poses a significant challenge in understanding its origins and mitigating its spread. This task is especially challenging because modern large language models (LLMs) produce disinformation with human-like quality. Additionally, the diversity in prompting methods used to generate disinformation complicates accurate source attribution. These methods introduce domain-specific features that can mask the fundamental characteristics of the models. In this paper, we introduce the concept of model attribution as a domain generalization problem, where each prompting method represents a unique domain. We argue that an effective attribution model must be invariant to these domain-specific features. It should also be proficient in identifying the originating models across all scenarios, reflecting real-world detection challenges. To address this, we introduce a novel approach based on Supervised Contrastive Learning. This method is designed to enhance the model's robustness to variations in prompts and focuses on distinguishing between different source LLMs. We evaluate our model through rigorous experiments involving three common prompting methods: ``open-ended'', ``rewriting'', and ``paraphrasing'', and three advanced LLMs: ``llama 2'', ``chatgpt'', and ``vicuna''. Our results demonstrate the effectiveness of our approach in model attribution tasks, achieving state-of-the-art performance across diverse and unseen datasets.
• 摘要：机器生成的虚假信息的模型归因对于了解其起源和减轻其传播提出了重大挑战。这项任务尤其具有挑战性，因为现代大型语言模型 (LLM) 产生的虚假信息具有与人类相似的质量。此外，用于生成虚假信息的提示方法的多样性使准确的来源归因变得复杂。这些方法引入了领域特定特征，这些特征可能会掩盖模型的基本特征。在本文中，我们将模型归因的概念引入为领域泛化问题，其中每种提示方法代表一个独特的领域。我们认为，有效的归因模型必须对这些领域特定特征保持不变。它还应该能够熟练地在所有场景中识别原始模型，以反映现实世界的检测挑战。为了解决这个问题，我们引入了一种基于监督对比学习的新方法。该方法旨在增强模型对提示变化的鲁棒性，并侧重于区分不同的源 LLM。我们通过严格的实验来评估我们的模型，实验涉及三种常见的提示方法：“开放式”、“重写”和“释义”，以及三种高级 LLM：“llama 2”、“chatgpt”和“vicuna”。我们的结果证明了我们的方法在模型归因任务中的有效性，在多样化和未见过的数据集上实现了最先进的性能。

Title: Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

• Authors: Zehui Wu, Ziwei Gong, Lin Ai, Pengyuan Shi, Kaan Donbekci, Julia Hirschberg
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21315
• Pdf URL: https://arxiv.org/pdf/2407.21315
• Copy Paste: [[2407.21315]] Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances(https://arxiv.org/abs/2407.21315)
• Keywords: language model, llm, prompt
• Abstract: This paper introduces a novel approach to emotion detection in speech using Large Language Models (LLMs). We address the limitation of LLMs in processing audio inputs by translating speech characteristics into natural language descriptions. Our method integrates these descriptions into text prompts, enabling LLMs to perform multimodal emotion analysis without architectural modifications. We evaluate our approach on two datasets: IEMOCAP and MELD, demonstrating significant improvements in emotion recognition accuracy, particularly for high-quality audio data. Our experiments show that incorporating speech descriptions yields a 2 percentage point increase in weighted F1 score on IEMOCAP (from 70.111\% to 72.596\%). We also compare various LLM architectures and explore the effectiveness of different feature representations. Our findings highlight the potential of this approach in enhancing emotion detection capabilities of LLMs and underscore the importance of audio quality in speech-based emotion recognition tasks. We'll release the source code on Github.
• 摘要：本文介绍了一种使用大型语言模型 (LLM) 进行语音情绪检测的新方法。我们通过将语音特征转换为自然语言描述来解决 LLM 在处理音频输入方面的局限性。我们的方法将这些描述集成到文本提示中，使 LLM 无需修改架构即可执行多模态情绪分析。我们在两个数据集上评估了我们的方法：IEMOCAP 和 MELD，结果显示情绪识别准确率显著提高，尤其是对于高质量音频数据。我们的实验表明，加入语音描述可使 IEMOCAP 上的加权 F1 得分提高 2 个百分点（从 70.111\% 提高到 72.596\%）。我们还比较了各种 LLM 架构，并探索了不同特征表示的有效性。我们的研究结果突出了这种方法在增强 LLM 情绪检测能力方面的潜力，并强调了音频质量在基于语音的情绪识别任务中的重要性。我们将在 Github 上发布源代码。

Title: Performance of Recent Large Language Models for a Low-Resourced Language

• Authors: Ravindu Jayakody, Gihan Dias
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21330
• Pdf URL: https://arxiv.org/pdf/2407.21330
• Copy Paste: [[2407.21330]] Performance of Recent Large Language Models for a Low-Resourced Language(https://arxiv.org/abs/2407.21330)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have shown significant advances in the past year. In addition to new versions of GPT and Llama, several other LLMs have been introduced recently. Some of these are open models available for download and modification. Although multilingual large language models have been available for some time, their performance on low-resourced languages such as Sinhala has been poor. We evaluated four recent LLMs on their performance directly in the Sinhala language, and by translation to and from English. We also evaluated their fine-tunability with a small amount of fine-tuning data. Claude and GPT 4o perform well out-of-the-box and do significantly better than previous versions. Llama and Mistral perform poorly but show some promise of improvement with fine tuning.
• 摘要：大型语言模型 (LLM) 在过去一年中取得了重大进展。除了 GPT 和 Llama 的新版本外，最近还推出了其他几种 LLM。其中一些是可供下载和修改的开放模型。尽管多语言大型语言模型已经问世了一段时间，但它们在僧伽罗语等资源匮乏的语言上的表现一直很差。我们直接评估了四种最近的 LLM 在僧伽罗语中的表现，以及与英语翻译的表现。我们还使用少量微调数据评估了它们的微调性。Claude 和 GPT 4o 开箱即用，表现良好，并且明显优于以前的版本。Llama 和 Mistral 表现不佳，但通过微调显示出一些改进的希望。

Title: Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models

• Authors: Zhengxuan Wu, Yuhao Zhang, Peng Qi, Yumo Xu, Rujun Han, Yian Zhang, Jifan Chen, Bonan Min, Zhiheng Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21417
• Pdf URL: https://arxiv.org/pdf/2407.21417
• Copy Paste: [[2407.21417]] Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models(https://arxiv.org/abs/2407.21417)
• Keywords: language model
• Abstract: Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Self-instruction Tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.
• 摘要：现代语言模型 (LM) 需要遵循人类指令，同时忠实于人类指令；然而，它们往往无法同时实现两者。在这里，我们提供了具体的证据，证明在训练具有这些目标的 LM 时，指令遵循（即遵循开放式指令）和忠实性（即在给定上下文中做出响应）之间存在权衡。例如，在指令遵循数据集上对 LLaMA-7B 进行微调会使其不那么忠实。相反，指令调整后的 Vicuna-7B 在需要上下文基础的任务上进一步优化时，在遵循指令方面表现出下降的性能。一种常见的补救措施是使用数据混合进行多任务学习 (MTL)，但它还远未实现协同结果。我们提出了一种简单而有效的方法，该方法依赖于拒绝采样以进行持续自我指令调整 (ReSet)，其性能明显优于普通 MTL。令人惊讶的是，我们发现少即是多，因为使用高质量但小得多的数据（少三倍）训练 ReSet 会产生更好的结果。我们的研究结果有助于更好地理解 LM 对齐训练中的客观差异。

Title: Cost-Effective Hallucination Detection for LLMs

• Authors: Simon Valentin, Jinmiao Fu, Gianluca Detommaso, Shaoyuan Xu, Giovanni Zappella, Bryan Wang
• Subjects: cs.CL, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.21424
• Pdf URL: https://arxiv.org/pdf/2407.21424
• Copy Paste: [[2407.21424]] Cost-Effective Hallucination Detection for LLMs(https://arxiv.org/abs/2407.21424)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) can be prone to hallucinations - generating unreliable outputs that are unfaithful to their inputs, external facts or internally inconsistent. In this work, we address several challenges for post-hoc hallucination detection in production settings. Our pipeline for hallucination detection entails: first, producing a confidence score representing the likelihood that a generated answer is a hallucination; second, calibrating the score conditional on attributes of the inputs and candidate response; finally, performing detection by thresholding the calibrated score. We benchmark a variety of state-of-the-art scoring methods on different datasets, encompassing question answering, fact checking, and summarization tasks. We employ diverse LLMs to ensure a comprehensive assessment of performance. We show that calibrating individual scoring methods is critical for ensuring risk-aware downstream decision making. Based on findings that no individual score performs best in all situations, we propose a multi-scoring framework, which combines different scores and achieves top performance across all datasets. We further introduce cost-effective multi-scoring, which can match or even outperform more expensive detection methods, while significantly reducing computational overhead.
• 摘要：大型语言模型 (LLM) 容易产生幻觉 - 产生不可靠的输出，这些输出不忠于输入、外部事实或内部不一致。在这项工作中，我们解决了生产环境中事后幻觉检测的几个挑战。我们的幻觉检测流程包括：首先，生成一个置信度分数，表示生成的答案是幻觉的可能性；其次，根据输入和候选响应的属性校准分数；最后，通过对校准分数进行阈值化来执行检测。我们在不同的数据集上对各种最先进的评分方法进行了基准测试，包括问答、事实核查和总结任务。我们采用不同的 LLM 来确保对性能进行全面的评估。我们表明，校准单个评分方法对于确保风险意识的下游决策至关重要。基于没有一个分数在所有情况下都表现最佳的发现，我们提出了一个多评分框架，该框架结合了不同的分数并在所有数据集中实现最佳性能。我们进一步引入了具有成本效益的多重评分，它可以匹配甚至超越更昂贵的检测方法，同时显著降低计算开销。

Title: QuestGen: Effectiveness of Question Generation Methods for Fact-Checking Applications

• Authors: Rivik Setty, Vinay Setty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21441
• Pdf URL: https://arxiv.org/pdf/2407.21441
• Copy Paste: [[2407.21441]] QuestGen: Effectiveness of Question Generation Methods for Fact-Checking Applications(https://arxiv.org/abs/2407.21441)
• Keywords: language model
• Abstract: Verifying fact-checking claims poses a significant challenge, even for humans. Recent approaches have demonstrated that decomposing claims into relevant questions to gather evidence enhances the efficiency of the fact-checking process. In this paper, we provide empirical evidence showing that this question decomposition can be effectively automated. We demonstrate that smaller generative models, fine-tuned for the question generation task using data augmentation from various datasets, outperform large language models by up to 8%. Surprisingly, in some cases, the evidence retrieved using machine-generated questions proves to be significantly more effective for fact-checking than that obtained from human-written questions. We also perform manual evaluation of the decomposed questions to assess the quality of the questions generated.
• 摘要：验证事实核查声明是一项重大挑战，即使对于人类来说也是如此。最近的方法表明，将声明分解为相关问题以收集证据可以提高事实核查过程的效率。在本文中，我们提供了经验证据，表明这种问题分解可以有效地实现自动化。我们证明，使用来自各种数据集的数据增强针对问题生成任务进行微调的小型生成模型比大型语言模型的表现高出 8%。令人惊讶的是，在某些情况下，使用机器生成的问题检索到的证据在事实核查方面比从人工编写的问题中获得的证据更有效。我们还对分解的问题进行手动评估，以评估生成问题的质量。

Title: Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency

• Authors: Taiji Li, Zhi Li, Yin Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21443
• Pdf URL: https://arxiv.org/pdf/2407.21443
• Copy Paste: [[2407.21443]] Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency(https://arxiv.org/abs/2407.21443)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Despite large language models (LLMs) have demonstrated impressive performance in various tasks, they are still suffering from the factual inconsistency problem called hallucinations. For instance, LLMs occasionally generate content that diverges from source article, and prefer to extract information that appears at the beginning and end of the context, especially in long document summarization. Inspired by these findings, we propose to improve the faithfulness of LLMs in summarization by impelling them to process the entire article more fairly and faithfully. We present a novel summary generation strategy, namely SliSum, which exploits the ideas of sliding windows and self-consistency. Specifically, SliSum divides the source article into overlapping windows, and utilizes LLM to generate local summaries for the content in the windows. Finally, SliSum aggregates all local summaries using clustering and majority voting algorithm to produce more faithful summary of entire article. Extensive experiments demonstrate that SliSum significantly improves the faithfulness of diverse LLMs including LLaMA-2, Claude-2 and GPT-3.5 in both short and long text summarization, while maintaining their fluency and informativeness and without additional fine-tuning and resources. We further conduct qualitative and quantitative studies to investigate why SliSum works and impacts of hyperparameters in SliSum on performance.
• 摘要：尽管大型语言模型 (LLM) 在各种任务中表现出色，但它们仍然受到被称为幻觉的事实不一致问题的困扰。例如，LLM 偶尔会生成与源文章不同的内容，并且倾向于提取出现在上下文开头和结尾的信息，尤其是在长文档摘要中。受这些发现的启发，我们建议通过促使 LLM 更公平、更忠实地处理整篇文章来提高其摘要的忠实度。我们提出了一种新颖的摘要生成策略，即 SliSum，它利用滑动窗口和自洽的思想。具体来说，SliSum 将源文章划分为重叠窗口，并利用 LLM 为窗口中的内容生成本地摘要。最后，SliSum 使用聚类和多数投票算法汇总所有本地摘要，以生成更忠实的整篇文章摘要。大量实验表明，SliSum 显著提高了包括 LLaMA-2、Claude-2 和 GPT-3.5 在内的各种 LLM 在短文本和长文本摘要中的忠实度，同时保持了其流畅性和信息量，并且无需额外的微调和资源。我们进一步进行了定性和定量研究，以调查 SliSum 为何有效以及 SliSum 中超参数对性能的影响。

Title: Generative Expressive Conversational Speech Synthesis

• Authors: Rui Liu, Yifan Hu, Ren Yi, Yin Xiang, Haizhou Li
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.21491
• Pdf URL: https://arxiv.org/pdf/2407.21491
• Copy Paste: [[2407.21491]] Generative Expressive Conversational Speech Synthesis(https://arxiv.org/abs/2407.21491)
• Keywords: gpt, agent
• Abstract: Conversational Speech Synthesis (CSS) aims to express a target utterance with the proper speaking style in a user-agent conversation setting. Existing CSS methods employ effective multi-modal context modeling techniques to achieve empathy understanding and expression. However, they often need to design complex network architectures and meticulously optimize the modules within them. In addition, due to the limitations of small-scale datasets containing scripted recording styles, they often fail to simulate real natural conversational styles. To address the above issues, we propose a novel generative expressive CSS system, termed GPT-Talker.We transform the multimodal information of the multi-turn dialogue history into discrete token sequences and seamlessly integrate them to form a comprehensive user-agent dialogue context. Leveraging the power of GPT, we predict the token sequence, that includes both semantic and style knowledge, of response for the agent. After that, the expressive conversational speech is synthesized by the conversation-enriched VITS to deliver feedback to the user.Furthermore, we propose a large-scale Natural CSS Dataset called NCSSD, that includes both naturally recorded conversational speech in improvised styles and dialogues extracted from TV shows. It encompasses both Chinese and English languages, with a total duration of 236 hours.We conducted comprehensive experiments on the reliability of the NCSSD and the effectiveness of our GPT-Talker. Both subjective and objective evaluations demonstrate that our model outperforms other state-of-the-art CSS systems significantly in terms of naturalness and expressiveness. The Code, Dataset, and Pre-trained Model are available at: this https URL.
• 摘要：对话语音合成 (CSS) 旨在在用户代理对话环境中以适当的说话风格表达目标话语。现有的 CSS 方法采用有效的多模态上下文建模技术来实现共情理解和表达。然而，它们通常需要设计复杂的网络架构并精心优化其中的模块。此外，由于包含脚本记录风格的小规模数据集的限制，它们往往无法模拟真实的自然对话风格。为了解决上述问题，我们提出了一种新颖的生成式表达 CSS 系统，称为 GPT-Talker。我们将多轮对话历史的多模态信息转换为离散的 token 序列，并将它们无缝集成以形成全面的用户代理对话上下文。利用 GPT 的强大功能，我们预测代理响应的 token 序列，其中包括语义和风格知识。之后，富有表现力的对话语音由对话丰富的 VITS 合成，以向用户提供反馈。此外，我们提出了一个名为 NCSSD 的大规模自然 CSS 数据集，其中包括即兴风格的自然录制对话语音和从电视节目中提取的对话。它涵盖中文和英文，总时长为 236 小时。我们对 NCSSD 的可靠性和 GPT-Talker 的有效性进行了全面的实验。主观和客观评估都表明，我们的模型在自然性和表现力方面明显优于其他最先进的 CSS 系统。代码、数据集和预训练模型可在以下位置获得：此 https URL。

Title: PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning

• Authors: Min Jae Jung, JooHee Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21571
• Pdf URL: https://arxiv.org/pdf/2407.21571
• Copy Paste: [[2407.21571]] PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning(https://arxiv.org/abs/2407.21571)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.
• 摘要：大型语言模型 (LLM) 在持续学习中遇到了重大挑战，因为灾难性的遗忘会导致新信息覆盖先前获得的知识。这种限制导致了大量环境和经济浪费。在本研究中，我们引入了 PMoE，即专家与非对称 Transformer 的渐进混合，旨在通过利用非对称设计来最大限度地减少遗忘，其中浅层专用于一般知识，深层用于新知识。PMoE 在深层中逐步增加专家，并采用路由器将新知识有效地分配给适当的专家。路由器位于深层附近，利用聚合合并信息的深度特征。这使路由器能够高效运行，将新知识分配给适当的专家，这些专家在深层中逐渐增加。在 TRACE 数据集和一般语言理解数据集上进行的大量实验表明，所提出的 PMoE 优于以前最先进的方法。

Title: TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization Methods

• Authors: Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21630
• Pdf URL: https://arxiv.org/pdf/2407.21630
• Copy Paste: [[2407.21630]] TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization Methods(https://arxiv.org/abs/2407.21630)
• Keywords: language model
• Abstract: Authorship obfuscation aims to disguise the identity of an author within a text by altering the writing style, vocabulary, syntax, and other linguistic features associated with the text author. This alteration needs to balance privacy and utility. While strong obfuscation techniques can effectively hide the author's identity, they often degrade the quality and usefulness of the text for its intended purpose. Conversely, maintaining high utility tends to provide insufficient privacy, making it easier for an adversary to de-anonymize the author. Thus, achieving an optimal trade-off between these two conflicting objectives is crucial. In this paper, we propose TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization, a new unsupervised authorship obfuscation method whose goal is to optimize the privacy-utility trade-off by regenerating the entire text considering its downstream utility. Our approach leverages policy optimization as a fine-tuning paradigm over small language models in order to rewrite texts by preserving author identity and downstream task utility. We show that our approach largely reduce the accuracy of attackers while preserving utility. We make our code and models publicly available.
• 摘要：作者身份混淆旨在通过改变与文本作者相关的写作风格、词汇、语法和其他语言特征来掩盖文本中作者的身份。这种改变需要在隐私和实用性之间取得平衡。虽然强大的混淆技术可以有效地隐藏作者的身份，但它们通常会降低文本的质量和实用性。相反，保持高实用性往往会提供不足的隐私，使对手更容易去匿名化作者。因此，在这两个相互冲突的目标之间实现最佳平衡至关重要。在本文中，我们提出了 TAROT：使用策略优化的面向任务的作者身份混淆，这是一种新的无监督作者身份混淆方法，其目标是通过考虑其下游实用性来重新生成整个文本，从而优化隐私-实用性权衡。我们的方法利用策略优化作为小型语言模型的微调范例，以便通过保留作者身份和下游任务实用性来重写文本。我们表明，我们的方法在保持实用性的同时，大大降低了攻击者的准确性。我们公开了我们的代码和模型。

Title: Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation

• Authors: Xiang Luo, Zhiwen Tang, Jin Wang, Xuejie Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21633
• Pdf URL: https://arxiv.org/pdf/2407.21633
• Copy Paste: [[2407.21633]] Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation(https://arxiv.org/abs/2407.21633)
• Keywords: language model, prompt
• Abstract: Zero-shot dialogue state tracking (DST) seeks to enable dialogue systems to transition to unfamiliar domains without manual annotation or extensive retraining. Prior research has approached this objective by embedding prompts into language models (LMs). Common methodologies include integrating prompts at the input layer or introducing learnable variables at each transformer layer. Nonetheless, each strategy exhibits inherent limitations. Prompts integrated at the input layer risk underutilization, with their impact potentially diminishing across successive transformer layers. Conversely, the addition of learnable variables to each layer can complicate the training process and increase inference latency. To tackle the issues mentioned above, this paper proposes Dual Low-Rank Adaptation (DualLoRA), a plug-and-play architecture designed for zero-shot DST. DualLoRA incorporates two distinct Low-Rank Adaptation (LoRA) components, targeting both dialogue context processing and prompt optimization, to ensure the comprehensive influence of prompts throughout the transformer model layers. This is achieved without incurring additional inference latency, showcasing an efficient integration into existing architectures. Through rigorous evaluation on the MultiWOZ and SGD datasets, DualLoRA demonstrates notable improvements across multiple domains, outperforming traditional baseline methods in zero-shot settings. Our code is accessible at: \url{this https URL}.
• 摘要：零样本对话状态跟踪 (DST) 旨在使对话系统能够过渡到不熟悉的领域，而无需手动注释或大量再训练。先前的研究通过将提示嵌入语言模型 (LM) 来实现这一目标。常用方法包括在输入层集成提示或在每个转换器层引入可学习变量。尽管如此，每种策略都表现出固有的局限性。集成在输入层的提示存在未充分利用的风险，其影响可能会在连续的转换器层中逐渐减弱。相反，在每一层添加可学习变量会使训练过程复杂化并增加推理延迟。为了解决上述问题，本文提出了双低秩自适应 (DualLoRA)，这是一种专为零样本 DST 设计的即插即用架构。DualLoRA 结合了两个不同的低秩自适应 (LoRA) 组件，针对对话上下文处理和提示优化，以确保提示在整个转换器模型层中的全面影响。这是在不产生额外推理延迟的情况下实现的，展示了与现有架构的有效集成。通过对 MultiWOZ 和 SGD 数据集进行严格评估，DualLoRA 在多个领域表现出显著的改进，在零样本设置中优于传统的基线方法。我们的代码可在 \url{此 https URL} 中访问。

Title: Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

• Authors: Shanbo Cheng, Zhichao Huang, Tom Ko, Hang Li, Ningxin Peng, Lu Xu, Qini Zhang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.21646
• Pdf URL: https://arxiv.org/pdf/2407.21646
• Copy Paste: [[2407.21646]] Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent(https://arxiv.org/abs/2407.21646)
• Keywords: llm, agent
• Abstract: In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.
• 摘要：在本文中，我们介绍了跨语言代理——同声传译 (CLASI)，一种高质量且类似于人类的同声语音翻译 (SiST) 系统。受专业人工翻译的启发，我们利用一种新颖的数据驱动读写策略来平衡翻译质量和延迟。为了应对翻译领域内术语的挑战，CLASI 采用多模态检索模块来获取相关信息以增强翻译。在 LLM 的支持下，我们的方法可以通过考虑输入音频、历史背景和检索到的信息来生成错误容忍的翻译。实验结果表明，我们的系统比其他系统好得多。与专业人工翻译保持一致，我们使用更好的人工评估指标——有效信息比例 (VIP) 来评估 CLASI，它衡量可以成功传达给听众的信息量。在现实场景中，由于语音通常不流畅、非正式且不清楚，CLASI 在中译英和英译中方向分别实现了 81.3% 和 78.0% 的 VIP。相比之下，最先进的商业或开源系统仅能达到 35.4% 和 41.6%。在极其困难的数据集上，其他系统的 VIP 低于 13%，而 CLASI 仍能达到 70% 的 VIP。

Title: Defending Jailbreak Attack in VLMs via Cross-modality Information Detector

• Authors: Yue Xu, Xiuyuan Qi, Zhan Qin, Wenjie Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.21659
• Pdf URL: https://arxiv.org/pdf/2407.21659
• Copy Paste: [[2407.21659]] Defending Jailbreak Attack in VLMs via Cross-modality Information Detector(https://arxiv.org/abs/2407.21659)
• Keywords: language model, llm
• Abstract: Vision Language Models (VLMs) extend the capacity of LLMs to comprehensively understand vision information, achieving remarkable performance in many vision-centric tasks. Despite that, recent studies have shown that these models are susceptible to jailbreak attacks, which refer to an exploitative technique where malicious users can break the safety alignment of the target model and generate misleading and harmful answers. This potential threat is caused by both the inherent vulnerabilities of LLM and the larger attack scope introduced by vision input. To enhance the security of VLMs against jailbreak attacks, researchers have developed various defense techniques. However, these methods either require modifications to the model's internal structure or demand significant computational resources during the inference phase. Multimodal information is a double-edged sword. While it increases the risk of attacks, it also provides additional data that can enhance safeguards. Inspired by this, we propose $\underline{\textbf{C}}$ross-modality $\underline{\textbf{I}}$nformation $\underline{\textbf{DE}}$tecto$\underline{\textbf{R}}$ ($\textit{CIDER})$, a plug-and-play jailbreaking detector designed to identify maliciously perturbed image inputs, utilizing the cross-modal similarity between harmful queries and adversarial images. This simple yet effective cross-modality information detector, $\textit{CIDER}$, is independent of the target VLMs and requires less computation cost. Extensive experimental results demonstrate the effectiveness and efficiency of $\textit{CIDER}$, as well as its transferability to both white-box and black-box VLMs.
• 摘要：视觉语言模型 (VLM) 扩展了 LLM 全面理解视觉信息的能力，在许多以视觉为中心的任务中取得了显著的表现。尽管如此，最近的研究表明，这些模型容易受到越狱攻击，越狱攻击是一种利用技术，恶意用户可以破坏目标模型的安全对齐，并生成误导性和有害的答案。这种潜在威胁既是由 LLM 固有的漏洞造成的，也是由视觉输入引入的更大攻击范围造成的。为了增强 VLM 免受越狱攻击的安全性，研究人员开发了各种防御技术。然而，这些方法要么需要修改模型的内部结构，要么在推理阶段需要大量的计算资源。多模态信息是一把双刃剑。虽然它增加了攻击的风险，但它也提供了可以增强安全措施的额外数据。受此启发，我们提出了 $\underline{\textbf{C}}$ross-modality $\underline{\textbf{I}}$nformation $\underline{\textbf{DE}}$tecto$\underline{\textbf{R}}$ ($\textit{CIDER})$，这是一种即插即用的越狱检测器，旨在利用有害查询和对抗图像之间的跨模态相似性来识别恶意扰动的图像输入。这种简单而有效的跨模态信息检测器 $\textit{CIDER}$ 独立于目标 VLM，并且需要较少的计算成本。大量实验结果证明了 $\textit{CIDER}$ 的有效性和效率，以及它对白盒和黑盒 VLM 的可迁移性。

Title: Synth-Empathy: Towards High-Quality Synthetic Empathy Data

• Authors: Hao Liang, Linzhuang Sun, Jingxuan Wei, Xijie Huang, Linkun Sun, Bihui Yu, Conghui He, Wentao Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21669
• Pdf URL: https://arxiv.org/pdf/2407.21669
• Copy Paste: [[2407.21669]] Synth-Empathy: Towards High-Quality Synthetic Empathy Data(https://arxiv.org/abs/2407.21669)
• Keywords: language model, llm
• Abstract: In recent years, with the rapid advancements in large language models (LLMs), achieving excellent empathetic response capabilities has become a crucial prerequisite. Consequently, managing and understanding empathetic datasets have gained increasing significance. However, empathetic data are typically human-labeled, leading to insufficient datasets and wasted human labor. In this work, we present Synth-Empathy, an LLM-based data generation and quality and diversity selection pipeline that automatically generates high-quality empathetic data while discarding low-quality data. With the data generated from a low empathetic model, we are able to further improve empathetic response performance and achieve state-of-the-art (SoTA) results across multiple benchmarks. Moreover, our model achieves SoTA performance on various human evaluation benchmarks, demonstrating its effectiveness and robustness in real-world applications. Furthermore, we show the trade-off between data quantity and quality, providing insights into empathetic data generation and selection.
• 摘要：近年来，随着大型语言模型 (LLM) 的快速发展，实现出色的共情响应能力已成为关键先决条件。因此，管理和理解共情数据集变得越来越重要。然而，共情数据通常是人工标记的，导致数据集不足和人力浪费。在这项工作中，我们提出了 Synth-Empathy，这是一种基于 LLM 的数据生成和质量和多样性选择管道，可自动生成高质量的共情数据并丢弃低质量数据。利用低共情模型生成的数据，我们能够进一步提高共情响应性能并在多个基准测试中实现最先进 (SoTA) 结果。此外，我们的模型在各种人工评估基准测试中实现了 SoTA 性能，证明了其在实际应用中的有效性和稳健性。此外，我们展示了数据数量和质量之间的权衡，为共情数据的生成和选择提供了见解。

Title: Adaptive Retrieval-Augmented Generation for Conversational Systems

• Authors: Xi Wang, Procheta Sen, Ruizhe Li, Emine Yilmaz
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.21712
• Pdf URL: https://arxiv.org/pdf/2407.21712
• Copy Paste: [[2407.21712]] Adaptive Retrieval-Augmented Generation for Conversational Systems(https://arxiv.org/abs/2407.21712)
• Keywords: language model, retrieval augmented generation, retrieval-augmented generation
• Abstract: Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the always need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop RAGate, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying RAGate to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of RAGate in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses and a high generation confidence. This study also identifies the correlation between the generation's confidence level and the relevance of the augmented knowledge.
• 摘要：尽管将大型语言模型集成到对话系统的开发中取得了成功，但许多研究表明检索和增强外部知识对于信息响应的有效性。因此，许多现有研究通常假设对话系统中始终需要检索增强生成 (RAG)，而无需明确控制。这提出了一个关于这种必要性的研究问题。在本研究中，我们建议调查系统响应的每一轮都需要用外部知识进行增强的必要性。具体来说，通过利用人类对自适应增强的二元选择的判断，我们开发了门控模型 RAGate，该模型对对话上下文和相关输入进行建模，以预测对话系统是否需要 RAG 来改进响应。我们对设计和将 RAGate 应用于对话模型进行了广泛的实验，并对不同的对话场景进行了全面的分析。我们的实验结果和分析表明，RAGate 在基于 RAG 的对话系统中有效应用于识别具有高质量响应和高生成置信度的适当 RAG 的系统响应。这项研究还确定了一代人的信心水平和增强知识的相关性之间的相关性。

Title: ShieldGemma: Generative AI Content Moderation Based on Gemma

• Authors: Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, Oscar Wahltinez
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.21772
• Pdf URL: https://arxiv.org/pdf/2407.21772
• Copy Paste: [[2407.21772]] ShieldGemma: Generative AI Content Moderation Based on Gemma(https://arxiv.org/abs/2407.21772)
• Keywords: llm
• Abstract: We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8\% AU-PRC on public benchmarks) and WildCard (+4.3\%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing ShieldGemma, we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.
• 摘要：我们推出了 ShieldGemma，这是一套基于 Gemma2 构建的基于 LLM 的安全内容审核模型。这些模型在用户输入和 LLM 生成的输出中，针对主要伤害类型（色情、危险内容、骚扰、仇恨言论）提供了稳健、最先进的安全风险预测。通过对公共和内部基准进行评估，我们展示了与现有模型（例如 Llama Guard（公共基准上的 AU-PRC 为 +10.8\%）和 WildCard（+4.3\%））相比更出色的性能。此外，我们还提出了一种基于 LLM 的新型数据管理流程，可适应各种安全相关任务及其他任务。我们已展示出主要在合成数据上训练的模型的强大泛化性能。通过发布 ShieldGemma，我们为研究界提供了宝贵的资源，提高了 LLM 安全性，并为开发人员创建了更有效的内容审核解决方案。