2024-06-14

Title: Exploring Fact Memorization and Style Imitation in LLMs Using QLoRA: An Experimental Study and Quality Assessment Methods

• Authors: Eugene Vyborov, Oleksiy Osypenko, Serge Sotnyk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08582
• Pdf URL: https://arxiv.org/pdf/2406.08582
• Copy Paste: [[2406.08582]] Exploring Fact Memorization and Style Imitation in LLMs Using QLoRA: An Experimental Study and Quality Assessment Methods(https://arxiv.org/abs/2406.08582)
• Keywords: llm, prompt
• Abstract: There are various methods for adapting LLMs to different domains. The most common methods are prompting, finetuning, and RAG. In this work, we explore the possibility of adapting a model using one of the PEFT methods - QLoRA. The experiment aims to simulate human responses based on their interviews. The simulation quality is assessed by comparing the quality of the style and the quality of the generated facts.
• 摘要：有多种方法可以将 LLM 适应不同的领域。最常见的方法是提示、微调和 RAG。在这项工作中，我们探索使用 PEFT 方法之一 QLoRA 调整模型的可能性。该实验旨在根据访谈模拟人类的反应。通过比较风格质量和生成事实的质量来评估模拟质量。

Title: CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

• Authors: Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng, Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, Weiran Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.08587
• Pdf URL: https://arxiv.org/pdf/2406.08587
• Copy Paste: [[2406.08587]] CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery(https://arxiv.org/abs/2406.08587)
• Keywords: language model, llm
• Abstract: Computer Science (CS) stands as a testament to the intricacies of human intelligence, profoundly advancing the development of artificial intelligence and modern society. However, the current community of large language models (LLMs) overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first bilingual (Chinese-English) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 5K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. The CS-Bench data and evaluation code are available at this https URL.
• 摘要：计算机科学是人类智能的结晶，对人工智能和现代社会的发展起到了巨大的推动作用。然而，目前的大型语言模型（LLM）研究社区过于注重分析特定基础技能（如数学和代码生成）的基准测试，而忽略了对计算机科学领域的全面评估。为了弥补这一缺陷，我们推出了第一个专门用于评估计算机科学领域 LLM 性能的双语（中英）基准测试 CS-Bench。CS-Bench 包含约 5K 精心挑选的测试样本，涵盖计算机科学 4 个关键领域的 26 个子领域，涵盖各种任务形式以及知识和推理的划分。利用 CS-Bench，我们对 30 多个主流 LLM 进行了全面评估，揭示了 CS 性能与模型规模之间的关系。我们还定量分析了现有 LLM 失败的原因，并指出了改进的方向，包括知识补充和针对 CS 的推理。进一步的跨能力实验表明，LLM 的计算机科学能力与数学和编码能力之间存在高度相关性。此外，专门研究数学和编码的专家 LLM 也在多个 CS 子领域表现出色。展望未来，我们设想 CS-Bench 将成为 LLM 在 CS 领域应用的基石，并为评估 LLM 的多样化推理能力开辟新途径。CS-Bench 数据和评估代码可在此 https URL 上找到。

Title: Language Model Council: Benchmarking Foundation Models on Highly Subjective Tasks by Consensus

• Authors: Justin Zhao, Flor Miriam Plaza-del-Arco, Amanda Cercas Curry
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.08598
• Pdf URL: https://arxiv.org/pdf/2406.08598
• Copy Paste: [[2406.08598]] Language Model Council: Benchmarking Foundation Models on Highly Subjective Tasks by Consensus(https://arxiv.org/abs/2406.08598)
• Keywords: language model, llm, chat
• Abstract: The rapid advancement of Large Language Models (LLMs) necessitates robust and challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how well their responses align with human preferences. However, many tasks such as those related to emotional intelligence, creative writing, or persuasiveness, are highly subjective and often lack majoritarian human agreement. Judges may have irreconcilable disagreements about what constitutes a better response. To address the challenge of ranking LLMs on highly subjective tasks, we propose a novel benchmarking framework, the Language Model Council (LMC). The LMC operates through a democratic process to: 1) formulate a test set through equal participation, 2) administer the test among council members, and 3) evaluate responses as a collective jury. We deploy a council of 20 newest LLMs on an open-ended emotional intelligence task: responding to interpersonal dilemmas. Our results show that the LMC produces rankings that are more separable, robust, and less biased than those from any individual LLM judge, and is more consistent with a human-established leaderboard compared to other benchmarks.
• 摘要：大型语言模型 (LLM) 的快速发展需要强大且具有挑战性的基准。Chatbot Arena 等排行榜根据 LLM 的回答与人类偏好的匹配程度对其进行排名。然而，许多任务（例如与情商、创意写作或说服力相关的任务）都是高度主观的，并且往往缺乏多数人的认同。评委可能对什么是更好的回答存在不可调和的分歧。为了应对在高度主观的任务上对 LLM 进行排名的挑战，我们提出了一个新颖的基准框架，即语言模型委员会 (LMC)。LMC 通过民主程序运作：1) 通过平等参与制定测试集，2) 在委员会成员之间进行测试，3) 作为集体陪审团评估答案。我们部署了一个由 20 名最新 LLM 组成的委员会，负责一项开放式情商任务：应对人际困境。我们的结果表明，LMC 产生的排名比任何单个 LLM 评委的排名更具可分离性、更稳健、更少偏见，并且与其他基准相比，与人类建立的排行榜更加一致。

Title: Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference

• Authors: Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, Shiyu Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.08607
• Pdf URL: https://arxiv.org/pdf/2406.08607
• Copy Paste: [[2406.08607]] Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference(https://arxiv.org/abs/2406.08607)
• Keywords: language model, llm
• Abstract: As Large Language Models (LLMs) demonstrate extensive capability in learning from documents, LLM unlearning becomes an increasingly important research area to address concerns of LLMs in terms of privacy, copyright, etc. A conventional LLM unlearning task typically involves two goals: (1) The target LLM should forget the knowledge in the specified forget documents, and (2) it should retain the other knowledge that the LLM possesses, for which we assume access to a small number of retain documents. To achieve both goals, a mainstream class of LLM unlearning methods introduces an optimization framework with a combination of two objectives - maximizing the prediction loss on the forget documents while minimizing that on the retain documents, which suffers from two challenges, degenerated output and catastrophic forgetting. In this paper, we propose a novel unlearning framework called Unlearning from Logit Difference (ULD), which introduces an assistant LLM that aims to achieve the opposite of the unlearning goals: remembering the forget documents and forgetting the retain knowledge. ULD then derives the unlearned LLM by computing the logit difference between the target and the assistant LLMs. We show that such reversed objectives would naturally resolve both aforementioned challenges while significantly improving the training efficiency. Extensive experiments demonstrate that our method efficiently achieves the intended forgetting while preserving the LLM's overall capabilities, reducing training time by more than threefold. Notably, our method loses 0% of model utility on the ToFU benchmark, whereas baseline methods may sacrifice 17% of utility on average to achieve comparable forget quality. Our code will be publicly available at this https URL.
• 摘要：由于大型语言模型 (LLM) 表现出从文档中学习的强大能力，LLM 反学习成为一个越来越重要的研究领域，以解决 LLM 在隐私、版权等方面的担忧。传统的 LLM 反学习任务通常涉及两个目标：(1) 目标 LLM 应该忘记指定的忘记文档中的知识，(2) 它应该保留 LLM 拥有的其他知识，我们假设可以访问少量保留文档。为了实现这两个目标，一类主流的 LLM 反学习方法引入了一个优化框架，它结合了两个目标——最大化忘记文档的预测损失，同时最小化保留文档的预测损失，这面临两个挑战，即退化输出和灾难性遗忘。在本文中，我们提出了一种称为从 Logit 差异中反学习 (ULD) 的新型反学习框架，它引入了一个辅助 LLM，旨在实现与反学习目标相反的目标：记住忘记的文档并忘记保留的知识。然后，ULD 通过计算目标和辅助 LLM 之间的逻辑差异来导出未学习的 LLM。我们表明，这种逆转的目标将自然地解决上述两个挑战，同时显著提高训练效率。大量实验表明，我们的方法有效地实现了预期的遗忘，同时保留了 LLM 的整体能力，将训练时间缩短了三倍以上。值得注意的是，我们的方法在 ToFU 基准上损失了 0% 的模型效用，而基线方法平均可能会牺牲 17% 的效用来实现相当的遗忘质量。我们的代码将在此 https URL 上公开提供。

Title: Unraveling Code-Mixing Patterns in Migration Discourse: Automated Detection and Analysis of Online Conversations on Reddit

• Authors: Fedor Vitiugin, Sunok Lee, Henna Paakki, Anastasiia Chizhikova, Nitin Sawhney
• Subjects: cs.CL, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2406.08633
• Pdf URL: https://arxiv.org/pdf/2406.08633
• Copy Paste: [[2406.08633]] Unraveling Code-Mixing Patterns in Migration Discourse: Automated Detection and Analysis of Online Conversations on Reddit(https://arxiv.org/abs/2406.08633)
• Keywords: language model
• Abstract: The surge in global migration patterns underscores the imperative of integrating migrants seamlessly into host communities, necessitating inclusive and trustworthy public services. Despite the Nordic countries' robust public sector infrastructure, recent immigrants often encounter barriers to accessing these services, exacerbating social disparities and eroding trust. Addressing digital inequalities and linguistic diversity is paramount in this endeavor. This paper explores the utilization of code-mixing, a communication strategy prevalent among multilingual speakers, in migration-related discourse on social media platforms such as Reddit. We present Ensemble Learning for Multilingual Identification of Code-mixed Texts (ELMICT), a novel approach designed to automatically detect code-mixed messages in migration-related discussions. Leveraging ensemble learning techniques for combining multiple tokenizers' outputs and pre-trained language models, ELMICT demonstrates high performance (with F1 more than 0.95) in identifying code-mixing across various languages and contexts, particularly in cross-lingual zero-shot conditions (with avg. F1 more than 0.70). Moreover, the utilization of ELMICT helps to analyze the prevalence of code-mixing in migration-related threads compared to other thematic categories on Reddit, shedding light on the topics of concern to migrant communities. Our findings reveal insights into the communicative strategies employed by migrants on social media platforms, offering implications for the development of inclusive digital public services and conversational systems. By addressing the research questions posed in this study, we contribute to the understanding of linguistic diversity in migration discourse and pave the way for more effective tools for building trust in multicultural societies.
• 摘要：全球移民模式的激增凸显了将移民无缝融入接收社区的必要性，这需要包容和值得信赖的公共服务。尽管北欧国家拥有强大的公共部门基础设施，但新移民在获取这些服务时经常遇到障碍，加剧了社会差距并削弱了信任。解决数字不平等和语言多样性是这一努力的重中之重。本文探讨了在 Reddit 等社交媒体平台上与移民相关的讨论中使用代码混合（一种在多语言使用者中流行的沟通策略）的情况。我们提出了用于多语言识别代码混合文本的集成学习 (ELMICT)，这是一种旨在自动检测与移民相关的讨论中的代码混合消息的新方法。 ELMICT 利用集成学习技术将多个标记器的输出和预训练语言模型相结合，在识别各种语言和语境中的代码混合方面表现出色（F1 超过 0.95），特别是在跨语言零样本条件下（平均 F1 超过 0.70）。此外，使用 ELMICT 有助于分析与 Reddit 上其他主题类别相比，移民相关主题中代码混合的流行程度，从而揭示移民社区关注的话题。我们的研究结果揭示了移民在社交媒体平台上采用的沟通策略，为包容性数字公共服务和对话系统的发展提供了启示。通过解决本研究中提出的研究问题，我们有助于理解移民话语中的语言多样性，并为在多元文化社会中建立信任的更有效工具铺平道路。

Title: Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning Enhancement in RLHF and Effective-Merged LLMs

• Authors: Chen Zheng, Ke Sun, Xun Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08657
• Pdf URL: https://arxiv.org/pdf/2406.08657
• Copy Paste: [[2406.08657]] Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning Enhancement in RLHF and Effective-Merged LLMs(https://arxiv.org/abs/2406.08657)
• Keywords: language model, gpt, llm
• Abstract: Despite the advances in Large Language Models (LLMs), exemplified by models like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often struggle with generating in-depth and coherent dialogues. This paper presents a novel two-step Coarse-to-Fine Actor model to address the inherent limitations in conversational and analytical capabilities of small-sized LLMs. Our approach begins with the Policy-based Coarse Actor, employing a technique we term "Continuous Maximization". The Coarse Actor establishes an enhanced, knowledge-rich pool adept at aligning with human preference styles in analysis and reasoning. Through the RLHF process, it employs Continuous Maximization, a strategy that dynamically and adaptively extends the output length limit, enabling the generation of more detailed and analytical content. Subsequently, the Fine Actor refines this analytical content, addressing the generation of excessively redundant information from the Coarse Actor. We introduce a "Knowledge Residue Merger" approach, refining the content from the Coarse Actor and merging it with an existing Instruction model to improve quality, correctness, and reduce redundancies. We applied our methodology to the popular Mistral model, creating Mistral-C2F, which has demonstrated exceptional performance across 11 general language tasks and the MT-Bench Dialogue task, outperforming similar-scale models and even larger models with 13B and 30B parameters. Our model has significantly improved conversational and analytical reasoning abilities.
• 摘要：尽管大型语言模型 (LLM) 取得了进展，例如 GPT-4 和 Claude 等模型，但 Llama 和 Mistral 等规模较小的 LLM 往往难以生成深入而连贯的对话。本文提出了一种新颖的两步式“从粗到精”参与者模型，以解决小型 LLM 在对话和分析能力方面的固有局限性。我们的方法从基于策略的粗略参与者开始，采用一种我们称之为“连续最大化”的技术。粗略参与者建立了一个增强的、知识丰富的池，善于与人类在分析和推理中的偏好风格保持一致。通过 RLHF 过程，它采用了连续最大化，这是一种动态和自适应地扩展输出长度限制的策略，从而能够生成更详细和更具分析性的内容。随后，精细参与者完善了这种分析内容，解决了粗略参与者生成过多冗余信息的问题。我们引入了“知识残差合并”方法，精炼来自粗略参与者的内容并将其与现有的指令模型合并，以提高质量、正确性并减少冗余。我们将我们的方法应用于流行的 Mistral 模型，创建了 Mistral-C2F，该模型在 11 个一般语言任务和 MT-Bench 对话任务中表现出色，优于类似规模的模型，甚至具有 13B 和 30B 参数的更大模型。我们的模型显著提高了对话和分析推理能力。

Title: Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification

• Authors: Martin Juan José Bucher, Marco Martini
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.08660
• Pdf URL: https://arxiv.org/pdf/2406.08660
• Copy Paste: [[2406.08660]] Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification(https://arxiv.org/abs/2406.08660)
• Keywords: gpt, llm, prompt, chat
• Abstract: Generative AI offers a simple, prompt-based alternative to fine-tuning smaller BERT-style LLMs for text classification tasks. This promises to eliminate the need for manually labeled training data and task-specific model training. However, it remains an open question whether tools like ChatGPT can deliver on this promise. In this paper, we show that smaller, fine-tuned LLMs (still) consistently and significantly outperform larger, zero-shot prompted models in text classification. We compare three major generative AI models (ChatGPT with GPT-3.5/GPT-4 and Claude Opus) with several fine-tuned LLMs across a diverse set of classification tasks (sentiment, approval/disapproval, emotions, party positions) and text categories (news, tweets, speeches). We find that fine-tuning with application-specific training data achieves superior performance in all cases. To make this approach more accessible to a broader audience, we provide an easy-to-use toolkit alongside this paper. Our toolkit, accompanied by non-technical step-by-step guidance, enables users to select and fine-tune BERT-like LLMs for any classification task with minimal technical and computational effort.
• 摘要：生成式人工智能提供了一种简单的、基于提示的替代方案，可以对较小的 BERT 式 LLM 进行微调，以完成文本分类任务。这有望消除对手动标记的训练数据和特定于任务的模型训练的需求。然而，像 ChatGPT 这样的工具是否能兑现这一承诺仍是一个悬而未决的问题。在本文中，我们表明，在文本分类中，较小的、经过微调的 LLM（仍然）始终如一地显著优于较大的零样本提示模型。我们将三种主要的生成式人工智能模型（带有 GPT-3.5/GPT-4 和 Claude Opus 的 ChatGPT）与几种经过微调的 LLM 进行了比较，这些模型涉及一系列不同的分类任务（情绪、赞成/反对、情绪、党派立场）和文本类别（新闻、推文、演讲）。我们发现，使用特定于应用程序的训练数据进行微调在所有情况下都能实现卓越的性能。为了让这种方法更容易被更广泛的受众接受，我们在本文旁边提供了一个易于使用的工具包。我们的工具包附带非技术性的逐步指导，使用户能够以最少的技术和计算工作量为任何分类任务选择和微调类似 BERT 的 LLM。

Title: HelpSteer2: Open-source dataset for training top-performing reward models

• Authors: Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.08673
• Pdf URL: https://arxiv.org/pdf/2406.08673
• Copy Paste: [[2406.08673]] HelpSteer2: Open-source dataset for training top-performing reward models(https://arxiv.org/abs/2406.08673)
• Keywords: language model, gpt, llm
• Abstract: High-quality preference datasets are essential for training reward models that can effectively guide large language models (LLMs) in generating high-quality responses aligned with human preferences. As LLMs become stronger and better aligned, permissively licensed preference datasets, such as Open Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for reward modeling. Methods that distil preference data from proprietary LLMs such as GPT-4 have restrictions on commercial usage imposed by model providers. To improve upon both generated responses and attribute labeling quality, we release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0). Using a powerful internal base model trained on HelpSteer2, we are able to achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming currently listed open and proprietary models, as of June 12th, 2024. Notably, HelpSteer2 consists of only ten thousand response pairs, an order of magnitude fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly efficient for training reward models. Our extensive experiments demonstrate that reward models trained with HelpSteer2 are effective in aligning LLMs. In particular, we propose SteerLM 2.0, a model alignment approach that can effectively make use of the rich multi-attribute score predicted by our reward models. HelpSteer2 is available at this https URL and code is available at this https URL
• 摘要：高质量的偏好数据集对于训练奖励模型至关重要，这些模型可以有效地指导大型语言模型 (LLM) 生成符合人类偏好的高质量响应。随着 LLM 变得越来越强大和越来越一致，需要更新获得许可的偏好数据集（例如 Open Assistant、HH-RLHF 和 HelpSteer）以保持奖励建模的有效性。从专有 LLM（例如 GPT-4）中提取偏好数据的方法受到模型提供商的商业使用限制。为了提高生成的响应和属性标记质量，我们发布了获得许可的偏好数据集 HelpSteer2（CC-BY-4.0）。使用在 HelpSteer2 上训练的强大的内部基础模型，我们能够在 Reward-Bench 的主要数据集上获得 SOTA 分数（92.0%），截至 2024 年 6 月 12 日，其表现优于目前列出的开放和专有模型。值得注意的是，HelpSteer2 仅包含一万个响应对，比现有的偏好数据集（例如 HH-RLHF）少一个数量级，这使得它非常适合训练奖励模型。我们进行了广泛的实验，表明使用 HelpSteer2 训练的奖励模型在对齐 LLM 方面是有效的。特别是，我们提出了 SteerLM 2.0，这是一种模型对齐方法，可以有效利用我们的奖励模型预测的丰富多属性分数。HelpSteer2 可在此 https URL 上获得，代码可在此 https URL 上获得

Title: Analyzing Large Language Models for Classroom Discussion Assessment

• Authors: Nhat Tran, Benjamin Pierce, Diane Litman, Richard Correnti, Lindsay Clare Matsumura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08680
• Pdf URL: https://arxiv.org/pdf/2406.08680
• Copy Paste: [[2406.08680]] Analyzing Large Language Models for Classroom Discussion Assessment(https://arxiv.org/abs/2406.08680)
• Keywords: language model, llm
• Abstract: Automatically assessing classroom discussion quality is becoming increasingly feasible with the help of new NLP advancements such as large language models (LLMs). In this work, we examine how the assessment performance of 2 LLMs interacts with 3 factors that may affect performance: task formulation, context length, and few-shot examples. We also explore the computational efficiency and predictive consistency of the 2 LLMs. Our results suggest that the 3 aforementioned factors do affect the performance of the tested LLMs and there is a relation between consistency and performance. We recommend a LLM-based assessment approach that has a good balance in terms of predictive performance, computational efficiency, and consistency.
• 摘要：借助大型语言模型 (LLM) 等新的 NLP 进步，自动评估课堂讨论质量变得越来越可行。在这项工作中，我们研究了 2 个 LLM 的评估性能如何与可能影响性能的 3 个因素相互作用：任务制定、上下文长度和少量样本示例。我们还探讨了 2 个 LLM 的计算效率和预测一致性。我们的结果表明，上述 3 个因素确实会影响测试的 LLM 的性能，并且一致性和性能之间存在关系。我们推荐一种基于 LLM 的评估方法，该方法在预测性能、计算效率和一致性方面取得了良好的平衡。

Title: mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

• Authors: Matthieu Futeral, Armel Zebaze, Pedro Ortiz Suarez, Julien Abadji, Rémi Lacroix, Cordelia Schmid, Rachel Bawden, Benoît Sagot
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2406.08707
• Pdf URL: https://arxiv.org/pdf/2406.08707
• Copy Paste: [[2406.08707]] mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus(https://arxiv.org/abs/2406.08707)
• Keywords: language model, llm
• Abstract: Multimodal Large Language Models (mLLMs) are trained on a large amount of text-image data. While most mLLMs are trained on caption-like data only, Alayrac et al. [2022] showed that additionally training them on interleaved sequences of text and images can lead to the emergence of in-context learning capabilities. However, the dataset they used, M3W, is not public and is only in English. There have been attempts to reproduce their results but the released datasets are English-only. In contrast, current multilingual and multimodal datasets are either composed of caption-like only or medium-scale or fully private data. This limits mLLM research for the 7,000 other languages spoken in the world. We therefore introduce mOSCAR, to the best of our knowledge the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We carefully conduct a set of filtering and evaluation steps to make sure mOSCAR is sufficiently safe, diverse and of good quality. We additionally train two types of multilingual model to prove the benefits of mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2) a model train on captioning data only. The model additionally trained on mOSCAR shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, confirming previous findings for English-only mLLMs.
• 摘要：多模态大型语言模型 (mLLM) 是在大量文本图像数据上进行训练的。虽然大多数 mLLM 仅针对类似字幕的数据进行训练，但 Alayrac 等人 [2022] 表示，在交错的文本和图像序列上对它们进行额外训练可以产生上下文学习能力。然而，他们使用的数据集 M3W 并不公开，而且只有英文版。有人尝试重现他们的结果，但发布的数据集只有英文版。相比之下，当前的多语言和多模态数据集要么仅由类似字幕的数据组成，要么由中等规模或完全私有的数据组成。这限制了 mLLM 对世界上其他 7,000 种语言的研究。因此，我们引入了 mOSCAR，据我们所知，这是从网络上抓取的第一个大规模多语言和多模态文档语料库。它涵盖 163 种语言、3.15 亿份文档、2140 亿个标记和 12 亿张图像。我们仔细执行了一系列筛选和评估步骤，以确保 mOSCAR 足够安全、多样化且质量良好。我们还训练了两种多语言模型来证明 mOSCAR 的优势：(1) 在 mOSCAR 子集和字幕数据上训练的模型和 (2) 仅在字幕数据上训练的模型。在 mOSCAR 上额外训练的模型在各种多语言图像文本任务和基准测试中显示出小样本学习性能的大幅提升，证实了之前对仅限英语的 mLLM 的发现。

Title: Enhancing Psychotherapy Counseling: A Data Augmentation Pipeline Leveraging Large Language Models for Counseling Conversations

• Authors: Jun-Woo Kim, Ji-Eun Han, Jun-Seok Koh, Hyeon-Tae Seo, Du-Seong Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08718
• Pdf URL: https://arxiv.org/pdf/2406.08718
• Copy Paste: [[2406.08718]] Enhancing Psychotherapy Counseling: A Data Augmentation Pipeline Leveraging Large Language Models for Counseling Conversations(https://arxiv.org/abs/2406.08718)
• Keywords: language model, llm
• Abstract: We introduce a pipeline that leverages Large Language Models (LLMs) to transform single-turn psychotherapy counseling sessions into multi-turn interactions. While AI-supported online counseling services for individuals with mental disorders exist, they are often constrained by the limited availability of multi-turn training datasets and frequently fail to fully utilize therapists' expertise. Our proposed pipeline effectively addresses these limitations. The pipeline comprises two main steps: 1) Information Extraction and 2) Multi-turn Counseling Generation. Each step is meticulously designed to extract and generate comprehensive multi-turn counseling conversations from the available datasets. Experimental results from both zero-shot and few-shot generation scenarios demonstrate that our approach significantly enhances the ability of LLMs to produce higher quality multi-turn dialogues in the context of mental health counseling. Our pipeline and dataset are publicly available this https URL.
• 摘要：我们引入了一个管道，利用大型语言模型 (LLM) 将单轮心理治疗咨询会话转变为多轮交互。虽然存在针对精神障碍患者的 AI 支持在线咨询服务，但它们往往受到多轮训练数据集有限的限制，并且经常无法充分利用治疗师的专业知识。我们提出的管道有效地解决了这些限制。该管道包括两个主要步骤：1）信息提取和 2）多轮咨询生成。每个步骤都经过精心设计，以从可用数据集中提取和生成全面的多轮咨询对话。零样本和少样本生成场景的实验结果表明，我们的方法显着增强了 LLM 在心理健康咨询背景下产生更高质量多轮对话的能力。我们的管道和数据集可通过此 https URL 公开获取。

Title: Standard Language Ideology in AI-Generated Language

• Authors: Genevieve Smith, Eve Fleisig, Madeline Bossi, Ishita Rustagi, Xavier Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08726
• Pdf URL: https://arxiv.org/pdf/2406.08726
• Copy Paste: [[2406.08726]] Standard Language Ideology in AI-Generated Language(https://arxiv.org/abs/2406.08726)
• Keywords: language model, llm
• Abstract: In this position paper, we explore standard language ideology in language generated by large language models (LLMs). First, we outline how standard language ideology is reflected and reinforced in LLMs. We then present a taxonomy of open problems regarding standard language ideology in AI-generated language with implications for minoritized language communities. We introduce the concept of standard AI-generated language ideology, the process by which AI-generated language regards Standard American English (SAE) as a linguistic default and reinforces a linguistic bias that SAE is the most "appropriate" language. Finally, we discuss tensions that remain, including reflecting on what desirable system behavior looks like, as well as advantages and drawbacks of generative AI tools imitating--or often not--different English language varieties. Throughout, we discuss standard language ideology as a manifestation of existing global power structures in and through AI-generated language before ending with questions to move towards alternative, more emancipatory digital futures.
• 摘要：在本立场文件中，我们探讨了大型语言模型 (LLM) 生成的语言中的标准语言意识形态。首先，我们概述了标准语言意识形态如何在 LLM 中得到体现和强化。然后，我们提出了关于人工智能生成语言中标准语言意识形态的未解决问题的分类，这些问题对少数语言社区具有影响。我们介绍了标准人工智能生成语言意识形态的概念，即人工智能生成的语言将标准美式英语 (SAE) 视为语言默认并强化 SAE 是最“合适”语言的语言偏见的过程。最后，我们讨论了仍然存在的紧张局势，包括反思理想的系统行为是什么样子，以及生成人工智能工具模仿（或通常不模仿）不同英语语言变体的优缺点。在整个过程中，我们讨论了标准语言意识形态作为人工智能生成语言中现有全球权力结构的一种体现，最后提出了一些问题，以迈向替代的、更具解放性的数字未来。

Title: StreamBench: Towards Benchmarking Continuous Improvement of Language Agents

• Authors: Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Hung-yi Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08747
• Pdf URL: https://arxiv.org/pdf/2406.08747
• Copy Paste: [[2406.08747]] StreamBench: Towards Benchmarking Continuous Improvement of Language Agents(https://arxiv.org/abs/2406.08747)
• Keywords: language model, llm, agent
• Abstract: Recent works have shown that large language model (LLM) agents are able to improve themselves from experience, which is an important ability for continuous enhancement post-deployment. However, existing benchmarks primarily evaluate their innate capabilities and do not assess their ability to improve over time. To address this gap, we introduce StreamBench, a pioneering benchmark designed to evaluate the continuous improvement of LLM agents over an input-feedback sequence. StreamBench simulates an online learning environment where LLMs receive a continuous flow of feedback stream and iteratively enhance their performance. In addition, we propose several simple yet effective baselines for improving LLMs on StreamBench, and provide a comprehensive analysis to identify critical components that contribute to successful streaming strategies. Our work serves as a stepping stone towards developing effective online learning strategies for LLMs, paving the way for more adaptive AI systems in streaming scenarios.
• 摘要：最近的研究表明，大型语言模型 (LLM) 代理能够从经验中自我改进，这是部署后持续改进的重要能力。然而，现有的基准主要评估它们的先天能力，而不是评估它们随着时间的推移而改进的能力。为了解决这一差距，我们引入了 StreamBench，这是一个开创性的基准，旨在评估 LLM 代理在输入反馈序列中的持续改进。StreamBench 模拟了一个在线学习环境，其中 LLM 接收连续的反馈流并迭代地提高其性能。此外，我们提出了几个简单但有效的基线来改进 StreamBench 上的 LLM，并提供了全面的分析以确定有助于成功流式传输策略的关键组件。我们的工作是开发有效的 LLM 在线学习策略的垫脚石，为流式传输场景中更具自适应性的 AI 系统铺平了道路。

Title: StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Encoded Structure

• Authors: Bangxin Li, Hengrui Xing, Chao Huang, Jin Qian, Huangqing Xiao, Linfeng Feng, Cong Tian
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2406.08754
• Pdf URL: https://arxiv.org/pdf/2406.08754
• Copy Paste: [[2406.08754]] StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Encoded Structure(https://arxiv.org/abs/2406.08754)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) are widely used in natural language processing but face the risk of jailbreak attacks that maliciously induce them to generate harmful content. Existing jailbreak attacks, including character-level and context-level attacks, mainly focus on the prompt of the plain text without specifically exploring the significant influence of its structure. In this paper, we focus on studying how prompt structure contributes to the jailbreak attack. We introduce a novel structure-level attack method based on tail structures that are rarely used during LLM training, which we refer to as Uncommon Text-Encoded Structure (UTES). We extensively study 12 UTESs templates and 6 obfuscation methods to build an effective automated jailbreak tool named StructuralSleight that contains three escalating attack strategies: Structural Attack, Structural and Character/Context Obfuscation Attack, and Fully Obfuscated Structural Attack. Extensive experiments on existing LLMs show that StructuralSleight significantly outperforms baseline methods. In particular, the attack success rate reaches 94.62\% on GPT-4o, which has not been addressed by state-of-the-art techniques.
• 摘要：大型语言模型（LLM）在自然语言处理中被广泛使用，但面临着被恶意诱导生成有害内容的越狱攻击的风险。现有的越狱攻击，包括字符级和上下文级攻击，主要关注纯文本的提示，而没有专门探究其结构的显著影响。在本文中，我们重点研究提示结构如何有助于越狱攻击。我们介绍了一种基于LLM训练中很少使用的尾部结构的结构级攻击方法，我们将其称为不常见的文本编码结构（UTES）。我们广泛研究了12个UTES模板和6种混淆方法，构建了一个有效的自动越狱工具StructuralSleight，其中包含三种升级攻击策略：结构攻击、结构和字符/上下文混淆攻击以及完全混淆的结构攻击。在现有LLM上进行的大量实验表明，StructuralSleight明显优于基线方法。特别是在 GPT-4o 上，攻击成功率达到 94.62%，而最先进的技术还未能解决这个问题。

Title: Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models

• Authors: Minghao Wu, Thuy-Trang Vu, Lizhen Qu, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08811
• Pdf URL: https://arxiv.org/pdf/2406.08811
• Copy Paste: [[2406.08811]] Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models(https://arxiv.org/abs/2406.08811)
• Keywords: language model, llm, chat
• Abstract: Large language models (LLMs) are typically fine-tuned on diverse and extensive datasets sourced from various origins to develop a comprehensive range of skills, such as writing, reasoning, chatting, coding, and more. Each skill has unique characteristics, and these datasets are often heterogeneous and imbalanced, making the fine-tuning process highly challenging. Balancing the development of each skill while ensuring the model maintains its overall performance requires sophisticated techniques and careful dataset curation. In this work, we propose a general, model-agnostic, reinforcement learning framework, Mixture-of-Skills (MoS), that learns to optimize data usage automatically during the fine-tuning process. This framework ensures the optimal comprehensive skill development of LLMs by dynamically adjusting the focus on different datasets based on their current learning state. To validate the effectiveness of MoS, we conduct extensive experiments using three diverse LLM backbones on two widely used benchmarks and demonstrate that MoS substantially enhances model performance. Building on the success of MoS, we propose MoSpec, an adaptation for task-specific fine-tuning, which harnesses the utilities of various datasets for a specific purpose. Our work underlines the significance of dataset rebalancing and present MoS as a powerful, general solution for optimizing data usage in the fine-tuning of LLMs for various purposes.
• 摘要：大型语言模型 (LLM) 通常在来自不同来源的多样化和广泛的数据集上进行微调，以开发全面的技能，例如写作、推理、聊天、编码等。每项技能都有独特的特点，这些数据集通常是异构的和不平衡的，这使得微调过程极具挑战性。平衡每项技能的发展，同时确保模型保持其整体性能，需要复杂的技术和仔细的数据集管理。在这项工作中，我们提出了一个通用的、与模型无关的强化学习框架，即技能混合 (MoS)，该框架在微调过程中自动学习优化数据使用。该框架通过根据当前学习状态动态调整对不同数据集的关注，确保 LLM 的最佳综合技能发展。为了验证 MoS 的有效性，我们在两个广泛使用的基准上使用三个不同的 LLM 主干进行了广泛的实验，并证明 MoS 显著提高了模型性能。基于 MoS 的成功，我们提出了 MoSpec，这是一种针对特定任务进行微调的改编，它利用各种数据集的效用来实现特定目的。我们的工作强调了数据集重新平衡的重要性，并将 MoS 作为一种强大的通用解决方案，用于优化 LLM 微调中的数据使用，以实现各种目的。

Title: Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination

• Authors: Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, Dan Klein
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2406.08818
• Pdf URL: https://arxiv.org/pdf/2406.08818
• Copy Paste: [[2406.08818]] Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination(https://arxiv.org/abs/2406.08818)
• Keywords: language model, gpt, prompt, chat
• Abstract: We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoken non-"standard" varieties from around the world). We prompted GPT-3.5 Turbo and GPT-4 with text by native speakers of each variety and analyzed the responses via detailed linguistic feature annotation and native speaker evaluation. We find that the models default to "standard" varieties of English; based on evaluation by native speakers, we also find that model responses to non-"standard" varieties consistently exhibit a range of issues: lack of comprehension (10% worse compared to "standard" varieties), stereotyping (16% worse), demeaning content (22% worse), and condescending responses (12% worse). We also find that if these models are asked to imitate the writing style of prompts in non-"standard" varieties, they produce text that exhibits lower comprehension of the input and is especially prone to stereotyping. GPT-4 improves on GPT-3.5 in terms of comprehension, warmth, and friendliness, but it also results in a marked increase in stereotyping (+17%). The results suggest that GPT-3.5 Turbo and GPT-4 exhibit linguistic discrimination in ways that can exacerbate harms for speakers of non-"standard" varieties.
• 摘要：我们对 ChatGPT 所表现出的语言偏见进行了大规模研究，涵盖了十种英语方言（标准美式英语、标准英式英语和来自世界各地的八种广泛使用的非“标准”英语）。我们用每种英语方言的母语人士的文本提示 GPT-3.5 Turbo 和 GPT-4，并通过详细的语言特征注释和母语人士评估分析了这些反应。我们发现模型默认使用“标准”英语方言；根据母语人士的评估，我们还发现模型对非“标准”英语方言的回应始终表现出一系列问题：缺乏理解（与“标准”英语方言相比差 10%）、刻板印象（差 16%）、贬低内容（差 22%）和居高临下的回应（差 12%）。我们还发现，如果要求这些模型模仿非“标准”英语方言提示的写作风格，它们会生成对输入的理解程度较低的文本，并且特别容易产生刻板印象。 GPT-4 在理解力、温暖度和友好度方面比 GPT-3.5 有所改进，但也导致刻板印象明显增加（+17%）。结果表明，GPT-3.5 Turbo 和 GPT-4 表现出语言歧视，可能会加剧非“标准”语言使用者的伤害。

Title: ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

• Authors: Xu Zhang, Xunjian Yin, Xiaojun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08842
• Pdf URL: https://arxiv.org/pdf/2406.08842
• Copy Paste: [[2406.08842]] ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions(https://arxiv.org/abs/2406.08842)
• Keywords: language model, llm
• Abstract: While substantial advancements have been made in developing large language models (LLMs), achieving control over their behavior can be difficult. Direct preference optimization (DPO) assumes the existence of a latent reward function to evaluate the responses of LLMs. This assumption indicates a strict preference ordering of different responses to the same input. However, there always exist contradictions of preference in LLMs according to our experimental observations. In this paper, we construct a graph structure of the preference relationship among different responses with self-annotation to find contradictions in the preference order. We propose ContraSolver, an algorithm that traverses all edges on the preference graph to identify those that might cause contradictions. ContraSolver initializes the graph with a maximum spanning tree and identifies contradictory edges, prioritizing the resolution of low-confidence preferences while preserving high-confidence ones. Experimental results on four different generation tasks show that the performance of different LLMs can be largely improved through our completely unsupervised self-alignment. Furthermore, by analyzing the preference graphs of LLMs with and without self-alignment by ContraSolver, we quantify the reduction in contradictions, suggesting that resolving preference contradictions is crucial for achieving better alignment performance.
• 摘要：尽管在开发大型语言模型 (LLM) 方面取得了重大进展，但控制其行为可能很困难。直接偏好优化 (DPO) 假设存在一个潜在奖励函数来评估 LLM 的响应。该假设表明对同一输入的不同响应具有严格的偏好顺序。然而，根据我们的实验观察，LLM 中总是存在偏好矛盾。在本文中，我们构建了一个具有自注释的不同响应之间的偏好关系图结构，以查找偏好顺序中的矛盾。我们提出了 ContraSolver，这是一种遍历偏好图上的所有边以识别可能导致矛盾的边的算法。ContraSolver 使用最大生成树初始化图并识别矛盾边，优先解决低置信度偏好，同时保留高置信度偏好。在四个不同生成任务上的实验结果表明，通过我们完全无监督的自对齐，可以大大提高不同 LLM 的性能。此外，通过使用 ContraSolver 分析具有和不具有自对齐的 LLM 的偏好图，我们量化了矛盾的减少，这表明解决偏好矛盾对于实现更好的对齐性能至关重要。

Title: An Approach to Build Zero-Shot Slot-Filling System for Industry-Grade Conversational Assistants

• Authors: G P Shrivatsa Bhargav, Sumit Neelam, Udit Sharma, Shajith Ikbal, Dheeraj Sreedhar, Hima Karanam, Sachindra Joshi, Pankaj Dhoolia, Dinesh Garg, Kyle Croutwater, Haode Qi, Eric Wayne, J William Murdock
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.08848
• Pdf URL: https://arxiv.org/pdf/2406.08848
• Copy Paste: [[2406.08848]] An Approach to Build Zero-Shot Slot-Filling System for Industry-Grade Conversational Assistants(https://arxiv.org/abs/2406.08848)
• Keywords: language model, llm
• Abstract: We present an approach to build Large Language Model (LLM) based slot-filling system to perform Dialogue State Tracking in conversational assistants serving across a wide variety of industry-grade applications. Key requirements of this system include: 1) usage of smaller-sized models to meet low latency requirements and to enable convenient and cost-effective cloud and customer premise deployments, and 2) zero-shot capabilities to serve across a wide variety of domains, slot types and conversational scenarios. We adopt a fine-tuning approach where a pre-trained LLM is fine-tuned into a slot-filling model using task specific data. The fine-tuning data is prepared carefully to cover a wide variety of slot-filling task scenarios that the model is expected to face across various domains. We give details of the data preparation and model building process. We also give a detailed analysis of the results of our experimental evaluations. Results show that our prescribed approach for slot-filling model building has resulted in 6.9% relative improvement of F1 metric over the best baseline on a realistic benchmark, while at the same time reducing the latency by 57%. More over, the data we prepared has helped improve F1 on an average by 4.2% relative across various slot-types.
• 摘要：我们提出了一种基于大型语言模型 (LLM) 的槽位填充系统构建方法，用于在各种行业级应用程序中执行对话状态跟踪的对话助手。该系统的关键要求包括：1) 使用较小尺寸的模型来满足低延迟要求并实现方便且经济高效的云和客户端部署，以及 2) 零样本能力以服务于各种领域、槽位类型和对话场景。我们采用微调方法，使用特定于任务的数据将预先训练的 LLM 微调为槽位填充模型。微调数据经过精心准备，以涵盖模型预计在各个领域面临的各种槽位填充任务场景。我们详细介绍了数据准备和模型构建过程。我们还对实验评估的结果进行了详细分析。结果表明，我们规定的槽位填充模型构建方法使 F1 指标比实际基准上的最佳基线相对提高了 6.9%，同时将延迟降低了 57%。此外，我们准备的数据帮助 F1 在不同类型的老虎机中平均提高了 4.2%。

Title: Plan, Generate and Complicate: Improving Low-resource Dialogue State Tracking via Easy-to-Difficult Zero-shot Data Augmentation

• Authors: Ming Gu, Yan Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.08860
• Pdf URL: https://arxiv.org/pdf/2406.08860
• Copy Paste: [[2406.08860]] Plan, Generate and Complicate: Improving Low-resource Dialogue State Tracking via Easy-to-Difficult Zero-shot Data Augmentation(https://arxiv.org/abs/2406.08860)
• Keywords: language model
• Abstract: Data augmentation methods have been a promising direction to improve the performance of small models for low-resource dialogue state tracking. However, traditional methods rely on pre-defined user goals and neglect the importance of data complexity in this task. In this paper, we propose EDZ-DA, an Easy-to-Difficult Zero-shot Data Augmentation framework for low-resource dialogue state tracking that utilizes large language models to automatically catch the relationships of different domains and then generate the dialogue data. We also complicate the dialogues based on the domain relation to enhance the model's capability for co-reference slot tracking. Furthermore, we permute slot values to mitigate the influence of output orders and the problem of incomplete value generation. Experimental results illustrate the superiority of our proposed method compared to previous strong data augmentation baselines on MultiWOZ.
• 摘要：数据增强方法一直是提高低资源对话状态跟踪小型模型性能的一个有前途的方向。然而，传统方法依赖于预定义的用户目标，而忽略了数据复杂性在此任务中的重要性。在本文中，我们提出了 EDZ-DA，一种用于低资源对话状态跟踪的从易到难的零样本数据增强框架，它利用大型语言模型自动捕捉不同领域的关系，然后生成对话数据。我们还根据领域关系使对话复杂化，以增强模型的共指槽跟踪能力。此外，我们排列槽值以减轻输出顺序的影响和不完整值生成的问题。实验结果表明，与 MultiWOZ 上以前的强数据增强基线相比，我们提出的方法具有优越性。

Title: No perspective, no perception!! Perspective-aware Healthcare Answer Summarization

• Authors: Gauri Naik, Sharad Chandakacherla, Shweta Yadav, Md. Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] No perspective, no perception!! Perspective-aware Healthcare Answer Summarization(https://arxiv.org/abs/)
• Keywords: prompt
• Abstract: Healthcare Community Question Answering (CQA) forums offer an accessible platform for individuals seeking information on various healthcare-related topics. People find such platforms suitable for self-disclosure, seeking medical opinions, finding simplified explanations for their medical conditions, and answering others' questions. However, answers on these forums are typically diverse and prone to off-topic discussions. It can be challenging for readers to sift through numerous answers and extract meaningful insights, making answer summarization a crucial task for CQA forums. While several efforts have been made to summarize the community answers, most of them are limited to the open domain and overlook the different perspectives offered by these answers. To address this problem, this paper proposes a novel task of perspective-specific answer summarization. We identify various perspectives, within healthcare-related responses and frame a perspective-driven abstractive summary covering all responses. To achieve this, we annotate 3167 CQA threads with 6193 perspective-aware summaries in our PUMA dataset. Further, we propose PLASMA, a prompt-driven controllable summarization model. To encapsulate the perspective-specific conditions, we design an energy-controlled loss function for the optimization. We also leverage the prefix tuner to learn the intricacies of the health-care perspective summarization. Our evaluation against five baselines suggests the superior performance of PLASMA by a margin of 1.5-21% improvement. We supplement our experiments with ablation and qualitative analysis.
• 摘要：医疗保健社区问答 (CQA) 论坛为寻求各种医疗保健相关主题信息的个人提供了一个可访问的平台。人们发现这样的平台适合自我披露、寻求医疗意见、为自己的病情寻找简化的解释以及回答他人的问题。然而，这些论坛上的答案通常多种多样，容易出现离题讨论。读者很难筛选众多答案并提取有意义的见解，因此答案总结是 CQA 论坛的一项关键任务。虽然已经做出了许多努力来总结社区答案，但其中大多数都局限于开放领域，忽略了这些答案提供的不同观点。为了解决这个问题，本文提出了一种新颖的针对特定观点的答案总结任务。我们在医疗保健相关的回复中识别各种观点，并构建一个涵盖所有回复的观点驱动的抽象总结。为了实现这一点，我们在 PUMA 数据集中使用 6193 个观点感知总结注释了 3167 个 CQA 线程。此外，我们提出了 PLASMA，这是一种提示驱动的可控总结模型。为了封装特定于视角的条件，我们设计了一个能量控制损失函数来进行优化。我们还利用前缀调谐器来学习医疗保健视角总结的复杂性。我们对五个基线的评估表明，PLASMA 的性能优越，提高了 1.5-21%。我们用消融和定性分析来补充我们的实验。

Title: Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models

• Authors: Bowen Ping, Shuo Wang, Hanqing Wang, Xu Han, Yuzhuang Xu, Yukun Yan, Yun Chen, Baobao Chang, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models(https://arxiv.org/abs/)
• Keywords: language model, llm, chat
• Abstract: Fine-tuning is a crucial process for adapting large language models (LLMs) to diverse applications. In certain scenarios, such as multi-tenant serving, deploying multiple LLMs becomes necessary to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs (e.g., WizardMath for math problems). Motivated by the long-tail distribution of singular values in the delta weights, we propose a delta quantization approach using mixed-precision. This method employs higher-bit representation for singular vectors corresponding to larger singular values. We evaluate our approach on various fine-tuned LLMs, including math LLMs, code LLMs, chat LLMs, and even VLMs. Experimental results demonstrate that our approach performs comparably to full fine-tuned LLMs, surpassing both low-rank and low-bit baselines by a considerable margin. Additionally, we show that our method is compatible with various backbone LLMs, such as Llama-2, Llama-3, and Mistral, highlighting its generalizability.
• 摘要：微调是将大型语言模型 (LLM) 适配到各种应用的关键过程。在某些情况下，例如多租户服务，部署多个 LLM 是满足复杂需求的必要条件。最近的研究表明，将微调后的 LLM 分解为基本模型和相应的增量权重，然后使用低秩或低位方法对其进行压缩以降低成本。在这项工作中，我们观察到现有的低秩和低位压缩方法会严重损害特定于任务的微调 LLM（例如，用于数学问题的 WizardMath）的模型性能。受增量权重中奇异值的长尾分布的启发，我们提出了一种使用混合精度的增量量化方法。该方法对与较大奇异值相对应的奇异向量采用更高位表示。我们在各种微调后的 LLM 上评估了我们的方法，包括数学 LLM、代码 LLM、聊天 LLM 甚至 VLM。实验结果表明，我们的方法性能堪比完全微调的 LLM，远超低秩和低位基线。此外，我们还表明，我们的方法与各种骨干 LLM（如 Llama-2、Llama-3 和 Mistral）兼容，凸显了其通用性。

Title: Navigating the Shadows: Unveiling Effective Disturbances for Modern AI Content Detectors

• Authors: Ying Zhou, Ben He, Le Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.08922
• Pdf URL: https://arxiv.org/pdf/2406.08922
• Copy Paste: [[2406.08922]] Navigating the Shadows: Unveiling Effective Disturbances for Modern AI Content Detectors(https://arxiv.org/abs/2406.08922)
• Keywords: language model, gpt, llm, chat
• Abstract: With the launch of ChatGPT, large language models (LLMs) have attracted global attention. In the realm of article writing, LLMs have witnessed extensive utilization, giving rise to concerns related to intellectual property protection, personal privacy, and academic integrity. In response, AI-text detection has emerged to distinguish between human and machine-generated content. However, recent research indicates that these detection systems often lack robustness and struggle to effectively differentiate perturbed texts. Currently, there is a lack of systematic evaluations regarding detection performance in real-world applications, and a comprehensive examination of perturbation techniques and detector robustness is also absent. To bridge this gap, our work simulates real-world scenarios in both informal and professional writing, exploring the out-of-the-box performance of current detectors. Additionally, we have constructed 12 black-box text perturbation methods to assess the robustness of current detection models across various perturbation granularities. Furthermore, through adversarial learning experiments, we investigate the impact of perturbation data augmentation on the robustness of AI-text detectors. We have released our code and data at this https URL.
• 摘要：随着 ChatGPT 的推出，大型语言模型 (LLM) 引起了全球关注。在文章写作领域，LLM 得到了广泛的应用，引发了与知识产权保护、个人隐私和学术诚信相关的担忧。为此，人工智能文本检测应运而生，用于区分人类和机器生成的内容。然而，最近的研究表明，这些检测系统往往缺乏鲁棒性，难以有效区分受干扰的文本。目前，缺乏对实际应用中检测性能的系统评估，也没有对扰动技术和检测器鲁棒性的全面检查。为了弥补这一差距，我们的工作模拟了非正式和专业写作中的真实场景，探索了当前检测器的开箱即用性能。此外，我们构建了 12 种黑盒文本扰动方法来评估当前检测模型在不同扰动粒度下的鲁棒性。此外，通过对抗性学习实验，我们研究了扰动数据增强对人工智能文本检测器鲁棒性的影响。我们已经在这个 https URL 上发布了我们的代码和数据。

Title: Multi-Agent Software Development through Cross-Team Collaboration

• Authors: Zhuoyun Du, Chen Qian, Wei Liu, Zihao Xie, Yifei Wang, Yufan Dang, Weize Chen, Cheng Yang
• Subjects: cs.CL, cs.AI, cs.MA, cs.SE
• Abstract URL: https://arxiv.org/abs/2406.08979
• Pdf URL: https://arxiv.org/pdf/2406.08979
• Copy Paste: [[2406.08979]] Multi-Agent Software Development through Cross-Team Collaboration(https://arxiv.org/abs/2406.08979)
• Keywords: language model, llm, chat, agent
• Abstract: The latest breakthroughs in Large Language Models (LLMs), eg., ChatDev, have catalyzed profound transformations, particularly through multi-agent collaboration for software development. LLM agents can collaborate in teams like humans, and follow the waterfall model to sequentially work on requirements analysis, development, review, testing, and other phases to perform autonomous software generation. However, for an agent team, each phase in a single development process yields only one possible outcome. This results in the completion of only one development chain, thereby losing the opportunity to explore multiple potential decision paths within the solution space. Consequently, this may lead to obtaining suboptimal results. To address this challenge, we introduce Cross-Team Collaboration (CTC), a scalable multi-team framework that enables orchestrated teams to jointly propose various decisions and communicate with their insights in a cross-team collaboration environment for superior content generation. Experimental results in software development reveal a notable increase in quality compared to state-of-the-art baselines, underscoring the efficacy of our framework. The significant improvements in story generation demonstrate the promising generalization ability of our framework across various domains. We anticipate that our work will guide LLM agents towards a cross-team paradigm and contribute to their significant growth in but not limited to software development. The code and data will be available at this https URL.
• 摘要：大型语言模型 (LLM) 的最新突破，例如 ChatDev，已经催化了深刻的变革，特别是通过软件开发的多智能体协作。LLM 智能体可以像人类一样在团队中协作，并遵循瀑布模型按顺序进行需求分析、开发、审查、测试和其他阶段，以执行自主软件生成。然而，对于智能体团队来说，单个开发过程的每个阶段只会产生一种可能的结果。这导致只完成一个开发链，从而失去了探索解决方案空间内多个潜在决策路径的机会。因此，这可能会导致获得次优结果。为了应对这一挑战，我们引入了跨团队协作 (CTC)，这是一个可扩展的多团队框架，使协调的团队能够在跨团队协作环境中共同提出各种决策并交流他们的见解，以生成卓越的内容。软件开发中的实验结果表明，与最先进的基线相比，质量显着提高，凸显了我们框架的有效性。故事生成的显着改进证明了我们的框架在各个领域具有良好的泛化能力。我们期望我们的工作将引导 LLM 代理走向跨团队范式，并促进他们在软件开发领域（但不限于）的显著发展。代码和数据将在此 https URL 上提供。

Title: LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models

• Authors: Xiaohao Yang, He Zhao, Dinh Phung, Wray Buntine, Lan Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09008
• Pdf URL: https://arxiv.org/pdf/2406.09008
• Copy Paste: [[2406.09008]] LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models(https://arxiv.org/abs/2406.09008)
• Keywords: language model, llm
• Abstract: Topic modeling has been a widely used tool for unsupervised text analysis. However, comprehensive evaluations of a topic model remain challenging. Existing evaluation methods are either less comparable across different models (e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic quality or document representation quality) at a time, which is insufficient to reflect the overall model performance. In this paper, we propose WALM (Words Agreement with Language Model), a new evaluation method for topic modeling that comprehensively considers the semantic quality of document representations and topics in a joint manner, leveraging the power of large language models (LLMs). With extensive experiments involving different types of topic models, WALM is shown to align with human judgment and can serve as a complementary evaluation method to the existing ones, bringing a new perspective to topic modeling. Our software package will be available at this https URL, which can be integrated with many widely used topic models.
• 摘要：主题建模已成为无监督文本分析的广泛使用工具。然而，对主题模型进行全面评估仍然具有挑战性。现有的评估方法要么在不同模型之间可比性较差（例如困惑度），要么每次只关注模型的一个特定方面（例如主题质量或文档表示质量），不足以反映整体模型性能。在本文中，我们提出了 WALM（与语言模型的词语一致性），这是一种新的主​​题建模评估方法，它利用大型语言模型 (LLM) 的强大功能，以联合的方式全面考虑文档表示和主题的语义质量。通过涉及不同类型主题模型的大量实验，WALM 被证明与人类判断一致，可以作为现有方法的补充评估方法，为主题建模带来新的视角。我们的软件包将在此 https URL 上提供，可与许多广泛使用的主题模型集成。

Title: Bayesian Statistical Modeling with Predictors from LLMs

• Authors: Michael Franke, Polina Tsvilodub, Fausto Carcassi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09012
• Pdf URL: https://arxiv.org/pdf/2406.09012
• Copy Paste: [[2406.09012]] Bayesian Statistical Modeling with Predictors from LLMs(https://arxiv.org/abs/2406.09012)
• Keywords: language model, llm
• Abstract: State of the art large language models (LLMs) have shown impressive performance on a variety of benchmark tasks and are increasingly used as components in larger applications, where LLM-based predictions serve as proxies for human judgements or decision. This raises questions about the human-likeness of LLM-derived information, alignment with human intuition, and whether LLMs could possibly be considered (parts of) explanatory models of (aspects of) human cognition or language use. To shed more light on these issues, we here investigate the human-likeness of LLMs' predictions for multiple-choice decision tasks from the perspective of Bayesian statistical modeling. Using human data from a forced-choice experiment on pragmatic language use, we find that LLMs do not capture the variance in the human data at the item-level. We suggest different ways of deriving full distributional predictions from LLMs for aggregate, condition-level data, and find that some, but not all ways of obtaining condition-level predictions yield adequate fits to human data. These results suggests that assessment of LLM performance depends strongly on seemingly subtle choices in methodology, and that LLMs are at best predictors of human behavior at the aggregate, condition-level, for which they are, however, not designed to, or usually used to, make predictions in the first place.
• 摘要：最先进的大型语言模型 (LLM) 在各种基准任务上表现出色，并越来越多地用作大型应用程序的组件，其中基于 LLM 的预测充当人类判断或决策的代理。这引发了关于 LLM 得出的信息是否与人类相似、是否与人类直觉一致以及 LLM 是否可以被视为人类认知或语言使用（方面）的解释模型（部分）的问题。为了进一步阐明这些问题，我们在此从贝叶斯统计建模的角度研究了 LLM 对多项选择决策任务的预测是否与人类相似。使用来自实用语言使用的强制选择实验的人类数据，我们发现 LLM 无法捕捉项目级别的人类数据差异。我们提出了从 LLM 中得出聚合条件级数据的完整分布预测的不同方法，并发现一些（但不是所有）获取条件级预测的方法可以充分拟合人类数据。这些结果表明，对 LLM 成绩的评估很大程度上取决于方法论中看似微妙的选择，并且 LLM 充其量只能预测总体条件层面的人类行为，然而，它们的设计目的并非，通常不用于进行预测。

Title: ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models

• Authors: Jing Liu, Ruihao Gong, Mingyang Zhang, Yefei He, Jianfei Cai, Bohan Zhuang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.09041
• Pdf URL: https://arxiv.org/pdf/2406.09041
• Copy Paste: [[2406.09041]] ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models(https://arxiv.org/abs/2406.09041)
• Keywords: language model, llm
• Abstract: The typical process for developing LLMs involves pre-training a general foundation model on massive data, followed by fine-tuning on task-specific data to create specialized experts. Serving these experts poses challenges, as loading all experts onto devices is impractical, and frequent switching between experts in response to user requests incurs substantial I/O costs, increasing latency and expenses. Previous approaches decompose expert weights into pre-trained model weights and residual delta weights, then quantize the delta weights to reduce model size. However, these methods often lead to significant quantization errors at extremely low bitwidths and assume the appropriate model for a user request is known in advance, which is not practical. To address these issues, we introduce ME-Switch, a memory-efficient expert switching framework for LLM serving. ME-Switch uses mixed-precision quantization, selectively quantizing non-salient input channels of delta weights to extremely low bits while keeping salient ones intact, significantly reducing storage demands while maintaining performance. Additionally, we develop a routing method that efficiently directs user queries to the most suitable expert by transforming the model selection problem into a domain classification problem. Extensive experiments show ME-Switch's promising memory efficiency and routing performance. For example, when serving three models from the Mistral-7B family, ME-Switch reduces model size by 1.74x while maintaining nearly lossless performance on instruction, mathematical reasoning, and code generation tasks. Furthermore, ME-Switch can efficiently serve 16 models from the Mistral-7B family on a single NVIDIA A100 GPU.
• 摘要：开发 LLM 的典型过程包括在海量数据上预先训练通用基础模型，然后在特定任务数据上进行微调以创建专业专家。为这些专家提供服务带来了挑战，因为将所有专家加载到设备上是不切实际的，并且根据用户请求频繁切换专家会产生大量 I/O 成本，从而增加延迟和费用。以前的方法将专家权重分解为预训练模型权重和残差增量权重，然后量化增量权重以减小模型大小。然而，这些方法通常会导致极低位宽下的显著量化误差，并且假设预先知道适合用户请求的模型，这并不切实际。为了解决这些问题，我们引入了 ME-Switch，这是一种用于 LLM 服务的内存高效专家切换框架。ME-Switch 使用混合精度量化，选择性地将增量权重的非显著输入通道量化为极低的位，同时保持显著的输入通道不变，从而显着降低存储需求同时保持性能。此外，我们还开发了一种路由方法，通过将模型选择问题转化为领域分类问题，有效地将用户查询定向到最合适的专家。大量实验表明，ME-Switch 具有良好的内存效率和路由性能。例如，在为 Mistral-7B 系列的三个模型提供服务时，ME-Switch 将模型大小缩小了 1.74 倍，同时在指令、数学推理和代码生成任务上保持几乎无损的性能。此外，ME-Switch 可以在单个 NVIDIA A100 GPU 上高效地为 Mistral-7B 系列的 16 个模型提供服务。

Title: Language Models are Crossword Solvers

• Authors: Soumadeep Saha, Sutanoya Chakraborty, Saptarshi Saha, Utpal Garain
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.09043
• Pdf URL: https://arxiv.org/pdf/2406.09043
• Copy Paste: [[2406.09043]] Language Models are Crossword Solvers(https://arxiv.org/abs/2406.09043)
• Keywords: language model, llm
• Abstract: Crosswords are a form of word puzzle that require a solver to demonstrate a high degree of proficiency in natural language understanding, wordplay, reasoning, and world knowledge, along with adherence to character and length constraints. In this paper we tackle the challenge of solving crosswords with Large Language Models (LLMs). We demonstrate that the current generation of state-of-the art (SoTA) language models show significant competence at deciphering cryptic crossword clues, and outperform previously reported SoTA results by a factor of 2-3 in relevant benchmarks. We also develop a search algorithm that builds off this performance to tackle the problem of solving full crossword grids with LLMs for the very first time, achieving an accuracy of 93\% on New York Times crossword puzzles. Contrary to previous work in this area which concluded that LLMs lag human expert performance significantly, our research suggests this gap is a lot narrower.
• 摘要：填字游戏是一种文字谜题，要求解答者在自然语言理解、文字游戏、推理和世界知识方面表现出很高的熟练程度，同时还要遵守字符和长度限制。在本文中，我们解决了使用大型语言模型 (LLM) 解答填字游戏的挑战。我们证明了当前最先进的 (SoTA) 语言模型在解读神秘的填字游戏线索方面表现出了显著的能力，并且在相关基准测试中比之前报告的 SoTA 结果高出 2-3 倍。我们还开发了一种基于此性能的搜索算法，首次解决了使用 LLM 解答完整填字游戏网格的问题，在《纽约时报》填字游戏中实现了 93\% 的准确率。与该领域之前的研究得出的结论相反，该研究认为 LLM 的表现远远落后于人类专家，而我们的研究表明，这一差距要小得多。

Title: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

• Authors: Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09044
• Pdf URL: https://arxiv.org/pdf/2406.09044
• Copy Paste: [[2406.09044]] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning(https://arxiv.org/abs/2406.09044)
• Keywords: language model, llm
• Abstract: Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computation and memory cost. Previous LoRA-based approaches initialize the low-rank matrices with gaussian distribution and zero values, while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might have interference with the well-learned subspace of the pretrained weight matrix. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principle singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principle matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principle matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the finetuning dataset. Extensive experiments on commonsense reasoning, math reasoning and instruction following benchmarks present the superior performance of our method.
• 摘要：大型语言模型 (LLM) 的有效微调旨在以减少计算和内存成本的方式调整 LLM。以前基于 LoRA 的方法使用高斯分布和零值初始化低秩矩阵，同时保持原始权重矩阵不变。然而，在无引导子空间中优化的可训练模型参数可能会干扰预训练权重矩阵的良好学习子空间。在本文中，我们提出了 MiLoRA，这是一种简单而有效的 LLM 微调方法，它只更新权重矩阵的次要奇异分量，同时保持主奇异分量不变。观察到次要矩阵对应于噪声或长尾信息，而主矩阵包含重要知识。MiLoRA 在与主矩阵正交的子空间内初始化低秩矩阵，因此预训练知识有望得到很好的保留。在微调过程中，MiLoRA 充分利用优化程度较低的子空间来学习微调数据集。对常识推理、数学推理和基准指导进行的大量实验证明了我们的方法的卓越性能。

Title: CUDRT: Benchmarking the Detection of Human vs. Large Language Models Generated Texts

• Authors: Zhen Tao, Zhiyu Li, Dinghao Xi, Wei Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.09056
• Pdf URL: https://arxiv.org/pdf/2406.09056
• Copy Paste: [[2406.09056]] CUDRT: Benchmarking the Detection of Human vs. Large Language Models Generated Texts(https://arxiv.org/abs/2406.09056)
• Keywords: language model, llm
• Abstract: The proliferation of large language models (LLMs) has significantly enhanced text generation capabilities across various industries. However, these models' ability to generate human-like text poses substantial challenges in discerning between human and AI authorship. Despite the effectiveness of existing AI-generated text detectors, their development is hindered by the lack of comprehensive, publicly available benchmarks. Current benchmarks are limited to specific scenarios, such as question answering and text polishing, and predominantly focus on English texts, failing to capture the diverse applications and linguistic nuances of LLMs. To address these limitations, this paper constructs a comprehensive bilingual benchmark in both Chinese and English to evaluate mainstream AI-generated text detectors. We categorize LLM text generation into five distinct operations: Create, Update, Delete, Rewrite, and Translate (CUDRT), encompassing all current LLMs activities. We also establish a robust benchmark evaluation framework to support scalable and reproducible experiments. For each CUDRT category, we have developed extensive datasets to thoroughly assess detector performance. By employing the latest mainstream LLMs specific to each language, our datasets provide a thorough evaluation environment. Extensive experimental results offer critical insights for optimizing AI-generated text detectors and suggest future research directions to improve detection accuracy and generalizability across various scenarios.
• 摘要：大型语言模型 (LLM) 的普及显著增强了各行各业的文本生成能力。然而，这些模型生成类似人类文本的能力对辨别人类和人工智能作者构成了巨大挑战。尽管现有的人工智能生成的文本检测器非常有效，但由于缺乏全面、公开的基准，它们的发展受到阻碍。当前的基准仅限于特定场景，例如问答和文本润色，并且主要侧重于英文文本，无法捕捉 LLM 的多种应用和语言细微差别。为了解决这些限制，本文构建了一个全面的中文和英文双语基准来评估主流的人工智能生成的文本检测器。我们将 LLM 文本生成分为五个不同的操作：创建、更新、删除、重写和翻译 (CUDRT)，涵盖了所有当前的 LLM 活动。我们还建立了一个强大的基准评估框架来支持可扩展和可重复的实验。对于每个 CUDRT 类别，我们都开发了大量数据集来全面评估检测器的性能。通过采用针对每种语言的最新主流 LLM，我们的数据集提供了全面的评估环境。大量实验结果为优化 AI 生成的文本检测器提供了重要见解，并提出了未来的研究方向，以提高各种场景的检测准确性和通用性。

Title: Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?

• Authors: Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, Min zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09072
• Pdf URL: https://arxiv.org/pdf/2406.09072
• Copy Paste: [[2406.09072]] Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?(https://arxiv.org/abs/2406.09072)
• Keywords: language model, llm
• Abstract: Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce CoTempQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on CoTempQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs' co-temporal reasoning from a mathematical perspective. We hope that our CoTempQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs. Our code is available at this https URL.
• 摘要：时间推理是大型语言模型 (LLM) 理解世界的基础。当前的时间推理数据集仅限于有关单个或孤立事件的问题，无法反映涉及并发性和复杂时间互连的现实时间特征。在本文中，我们介绍了 CoTempQA，这是一个全面的同时间问答 (QA) 基准，包含四个同时间场景（平等、重叠、期间、混合），有 4,748 个样本，用于评估 LLM 的同时间理解和推理能力。我们进行了广泛的实验，发现当前 LLM 在 CoTempQA 任务上的表现与人类水平的推理之间存在显著差距。即使使用思路链 (CoT) 方法进行了增强，模型仍然始终无法完成我们的任务。在我们的初步探索中，我们发现数学推理在处理同时间事件中起着重要作用，并提出了一种从数学角度增强 LLM 同时间推理的策略。我们希望我们的 CoTempQA 数据集能够推动 LLM 的共时推理能力进一步提升。我们的代码可在此 https URL 上找到。

Title: 3M: Multi-modal Multi-task Multi-teacher Learning for Game Event Detection

• Authors: Thye Shan Ng, Feiqi Cao, Soyeon Caren Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09076
• Pdf URL: https://arxiv.org/pdf/2406.09076
• Copy Paste: [[2406.09076]] 3M: Multi-modal Multi-task Multi-teacher Learning for Game Event Detection(https://arxiv.org/abs/2406.09076)
• Keywords: chat
• Abstract: Esports has rapidly emerged as a global phenomenon with an ever-expanding audience via platforms, like YouTube. Due to the inherent complexity nature of the game, it is challenging for newcomers to comprehend what the event entails. The chaotic nature of online chat, the fast-paced speech of the game commentator, and the game-specific user interface further compound the difficulty for users in comprehending the gameplay. To overcome these challenges, it is crucial to integrate the Multi-Modal (MM) information from the platform and understand the event. The paper introduces a new MM multi-teacher-based game event detection framework, with the ultimate goal of constructing a comprehensive framework that enhances the comprehension of the ongoing game situation. While conventional MM models typically prioritise aligning MM data through concurrent training towards a unified objective, our framework leverages multiple teachers trained independently on different tasks to accomplish the Game Event Detection. The experiment clearly shows the effectiveness of the proposed MM multi-teacher framework.
• 摘要：电子竞技已迅速成为一种全球现象，通过 YouTube 等平台的观众人数不断增加。由于游戏本身的复杂性，新手很难理解事件的含义。在线聊天的混乱性质、游戏评论员的快节奏讲话以及特定于游戏的用户界面进一步增加了用户理解游戏玩法的难度。为了克服这些挑战，整合来自平台的多模态 (MM) 信息并了解事件至关重要。本文介绍了一种新的基于 MM 多教师的游戏事件检测框架，最终目标是构建一个全面的框架，以增强对正在进行的游戏情况的理解。虽然传统的 MM 模型通常优先通过并发训练将 MM 数据对齐到统一目标，但我们的框架利用在不同任务上独立训练的多个教师来完成游戏事件检测。实验清楚地表明了所提出的 MM 多教师框架的有效性。

Title: SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

• Authors: Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, Huajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09098
• Pdf URL: https://arxiv.org/pdf/2406.09098
• Copy Paste: [[2406.09098]] SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models(https://arxiv.org/abs/2406.09098)
• Keywords: language model, llm, prompt
• Abstract: The burgeoning utilization of Large Language Models (LLMs) in scientific research necessitates advanced benchmarks capable of evaluating their understanding and application of scientific knowledge comprehensively. To address this need, we introduce the SciKnowEval benchmark, a novel framework that systematically evaluates LLMs across five progressive levels of scientific knowledge: studying extensively, inquiring earnestly, thinking profoundly, discerning clearly, and practicing assiduously. These levels aim to assess the breadth and depth of scientific knowledge in LLMs, including knowledge coverage, inquiry and exploration capabilities, reflection and reasoning abilities, ethic and safety considerations, as well as practice proficiency. Specifically, we take biology and chemistry as the two instances of SciKnowEval and construct a dataset encompassing 50K multi-level scientific problems and solutions. By leveraging this dataset, we benchmark 20 leading open-source and proprietary LLMs using zero-shot and few-shot prompting strategies. The results reveal that despite achieving state-of-the-art performance, the proprietary LLMs still have considerable room for improvement, particularly in addressing scientific computations and applications. We anticipate that SciKnowEval will establish a comprehensive standard for benchmarking LLMs in science research and discovery, and promote the development of LLMs that integrate scientific knowledge with strong safety awareness. The dataset and code are publicly available at this https URL .
• 摘要：大型语言模型 (LLM) 在科学研究中的广泛应用需要能够全面评估其对科学知识的理解和应用的高级基准。为了满足这一需求，我们引入了 SciKnowEval 基准，这是一个新颖的框架，它系统地评估 LLM 在五个渐进的科学知识水平上的表现：广泛学习、认真探究、深刻思考、清晰辨别和刻苦练习。这些级别旨在评估 LLM 中科学知识的广度和深度，包括知识覆盖范围、探究和探索能力、反思和推理能力、道德和安全考虑以及实践能力。具体来说，我们以生物学和化学作为 SciKnowEval 的两个实例，并构建了一个包含 50K 个多层次科学问题和解决方案的数据集。通过利用该数据集，我们使用零样本和少量样本提示策略对 20 个领先的开源和专有 LLM 进行基准测试。结果表明，尽管专有的 LLM 取得了最先进的性能，但仍有很大的改进空间，特别是在解决科学计算和应用方面。我们预计 SciKnowEval 将为科学研究和发现中的 LLM 基准测试建立一个全面的标准，并促进将科学知识与强大的安全意识相结合的 LLM 的发展。数据集和代码可在此 https URL 上公开获取。

Title: Chain-of-Though (CoT) prompting strategies for medical error detection and correction

• Authors: Zhaolong Wu, Abul Hasan, Jinge Wu, Yunsoo Kim, Jason P.Y. Cheung, Teng Zhang, Honghan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09103
• Pdf URL: https://arxiv.org/pdf/2406.09103
• Copy Paste: [[2406.09103]] Chain-of-Though (CoT) prompting strategies for medical error detection and correction(https://arxiv.org/abs/2406.09103)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: This paper describes our submission to the MEDIQA-CORR 2024 shared task for automatically detecting and correcting medical errors in clinical notes. We report results for three methods of few-shot In-Context Learning (ICL) augmented with Chain-of-Thought (CoT) and reason prompts using a large language model (LLM). In the first method, we manually analyse a subset of train and validation dataset to infer three CoT prompts by examining error types in the clinical notes. In the second method, we utilise the training dataset to prompt the LLM to deduce reasons about their correctness or incorrectness. The constructed CoTs and reasons are then augmented with ICL examples to solve the tasks of error detection, span identification, and error correction. Finally, we combine the two methods using a rule-based ensemble method. Across the three sub-tasks, our ensemble method achieves a ranking of 3rd for both sub-task 1 and 2, while securing 7th place in sub-task 3 among all submissions.
• 摘要：本文介绍了我们向 MEDIQA-CORR 2024 共享任务提交的论文，该论文用于自动检测和纠正临床笔记中的医疗错误。我们报告了使用大型语言模型 (LLM) 增强的三种少量上下文学习 (ICL) 方法的结果，这些方法增强了思路链 (CoT) 和推理提示。在第一种方法中，我们手动分析训练和验证数据集的子集，通过检查临床笔记中的错误类型推断出三个 CoT 提示。在第二种方法中，我们利用训练数据集提示 LLM 推断其正确性或不正确性的原因。然后使用 ICL 示例增强构建的 CoT 和原因，以解决错误检测、跨度识别和错误纠正任务。最后，我们使用基于规则的集成方法将这两种方法结合起来。在三个子任务中，我们的集成方法在子任务 1 和 2 中均排名第 3，而在子任务 3 中在所有提交的作品中排名第 7。

Title: RH-SQL: Refined Schema and Hardness Prompt for Text-to-SQL

• Authors: Jiawen Yi, Guo Chen, Zixiang Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09133
• Pdf URL: https://arxiv.org/pdf/2406.09133
• Copy Paste: [[2406.09133]] RH-SQL: Refined Schema and Hardness Prompt for Text-to-SQL(https://arxiv.org/abs/2406.09133)
• Keywords: language model, prompt
• Abstract: Text-to-SQL is a technology that converts natural language queries into the structured query language SQL. A novel research approach that has recently gained attention focuses on methods based on the complexity of SQL queries, achieving notable performance improvements. However, existing methods entail significant storage and training costs, which hampers their practical application. To address this issue, this paper introduces a method for Text-to-SQL based on Refined Schema and Hardness Prompt. By filtering out low-relevance schema information with a refined schema and identifying query hardness through a Language Model (LM) to form prompts, this method reduces storage and training costs while maintaining performance. It's worth mentioning that this method is applicable to any sequence-to-sequence (seq2seq) LM. Our experiments on the Spider dataset, specifically with large-scale LMs, achieved an exceptional Execution accuracy (EX) of 82.6%, demonstrating the effectiveness and greater suitability of our method for real-world applications.
• 摘要：Text-to-SQL 是一种将自然语言查询转换为结构化查询语言 SQL 的技术。最近，一种新的研究方法引起了人们的关注，该方法基于 SQL 查询的复杂性，取得了显著的性能提升。然而，现有方法需要大量的存储和训练成本，这阻碍了它们的实际应用。针对这一问题，本文介绍了一种基于 Refined Schema 和 Hardness Prompt 的 Text-to-SQL 方法。通过使用 Refined Schema 过滤掉低相关性的 Schema 信息，并通过语言模型 (LM) 识别查询难度以形成提示，该方法在保持性能的同时降低了存储和训练成本。值得一提的是，该方法适用于任何序列到序列 (seq2seq) 语言模型。我们在 Spider 数据集上的实验，特别是使用大规模语言模型，实现了 82.6% 的出色执行准确率 (EX)，证明了我们的方法的有效性和更适合实际应用。

Title: Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

• Authors: Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, Min Lin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.09136
• Pdf URL: https://arxiv.org/pdf/2406.09136
• Copy Paste: [[2406.09136]] Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs(https://arxiv.org/abs/2406.09136)
• Keywords: language model, llm, chain-of-thought, tree-of-thought
• Abstract: The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at this https URL.
• 摘要：思路链 (CoT) 解码的最新发展使得大型语言模型 (LLM) 能够生成明确的逻辑推理路径来解决复杂的问题。然而，研究表明，这些路径并不总是经过深思熟虑和最佳的。思路树 (ToT) 方法采用树搜索来广泛探索推理空间并找到 CoT 解码可能忽略的更好的推理路径。然而，这种深思熟虑是以显著增加推理复杂性为代价的。在这项工作中，我们证明了利用 ToT 构建的搜索树对 LLM 进行微调可以使 CoT 实现类似或更好的性能，从而避免大量的推理负担。这是通过偏好链优化 (CPO) 实现的，其中 LLM 经过微调以使用树搜索过程中的固有偏好信息将 CoT 推理路径的每一步与 ToT 的路径对齐。大量实验结果表明，CPO 显著提高了 LLM 在解决各种复杂问题（包括问答、事实验证和算术推理）方面的表现，证明了其有效性。我们的代码可在此 https URL 上获取。

Title: Investigating the translation capabilities of Large Language Models trained on parallel data only

• Authors: Javier García Gilabert, Carlos Escolano, Aleix Sant Savall, Francesca De Luca Fornaciari, Audrey Mash, Xixian Liao, Maite Melero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09140
• Pdf URL: https://arxiv.org/pdf/2406.09140
• Copy Paste: [[2406.09140]] Investigating the translation capabilities of Large Language Models trained on parallel data only(https://arxiv.org/abs/2406.09140)
• Keywords: language model, llm, prompt
• Abstract: In recent years, Large Language Models (LLMs) have demonstrated exceptional proficiency across a broad spectrum of Natural Language Processing (NLP) tasks, including Machine Translation. However, previous methods predominantly relied on iterative processes such as instruction fine-tuning or continual pre-training, leaving unexplored the challenges of training LLMs solely on parallel data. In this work, we introduce PLUME (Parallel Language Model), a collection of three 2B LLMs featuring varying vocabulary sizes (32k, 128k, and 256k) trained exclusively on Catalan-centric parallel examples. These models perform comparably to previous encoder-decoder architectures on 16 supervised translation directions and 56 zero-shot ones. Utilizing this set of models, we conduct a thorough investigation into the translation capabilities of LLMs, probing their performance, the impact of the different elements of the prompt, and their cross-lingual representation space.
• 摘要：近年来，大型语言模型 (LLM) 在包括机器翻译在内的广泛自然语言处理 (NLP) 任务中表现出色。然而，以前的方法主要依赖于迭代过程，例如指令微调或持续预训练，而仅使用并行数据训练 LLM 所面临的挑战尚未得到探索。在这项工作中，我们引入了 PLUME（并行语言模型），这是三个 2B LLM 的集合，具有不同的词汇量（32k、128k 和 256k），专门针对以加泰罗尼亚语为中心的并行示例进行训练。这些模型在 16 个监督翻译方向和 56 个零样本翻译方向的表现与以前的编码器-解码器架构相当。利用这组模型，我们对 LLM 的翻译能力进行了彻底的研究，探究了它们的性能、提示的不同元素的影响以及它们的跨语言表示空间。

Title: DefAn: Definitive Answer Dataset for LLMs Hallucination Evaluation

• Authors: A B M Ashikur Rahman, Saeed Anwar, Muhammad Usman, Ajmal Mian
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.09155
• Pdf URL: https://arxiv.org/pdf/2406.09155
• Copy Paste: [[2406.09155]] DefAn: Definitive Answer Dataset for LLMs Hallucination Evaluation(https://arxiv.org/abs/2406.09155)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities, revolutionizing the integration of AI in daily life applications. However, they are prone to hallucinations, generating claims that contradict established facts, deviating from prompts, and producing inconsistent responses when the same prompt is presented multiple times. Addressing these issues is challenging due to the lack of comprehensive and easily assessable benchmark datasets. Most existing datasets are small and rely on multiple-choice questions, which are inadequate for evaluating the generative prowess of LLMs. To measure hallucination in LLMs, this paper introduces a comprehensive benchmark dataset comprising over 75,000 prompts across eight domains. These prompts are designed to elicit definitive, concise, and informative answers. The dataset is divided into two segments: one publicly available for testing and assessing LLM performance and a hidden segment for benchmarking various LLMs. In our experiments, we tested six LLMs-GPT-3.5, LLama 2, LLama 3, Gemini, Mixtral, and Zephyr-revealing that overall factual hallucination ranges from 59% to 82% on the public dataset and 57% to 76% in the hidden benchmark. Prompt misalignment hallucination ranges from 6% to 95% in the public dataset and 17% to 94% in the hidden counterpart. Average consistency ranges from 21% to 61% and 22% to 63%, respectively. Domain-wise analysis shows that LLM performance significantly deteriorates when asked for specific numeric information while performing moderately with person, location, and date queries. Our dataset demonstrates its efficacy and serves as a comprehensive benchmark for LLM performance evaluation. Our dataset and LLMs responses are available at \href{this https URL}{this https URL}.
• 摘要：大型语言模型 (LLM) 已展现出卓越的能力，彻底改变了人工智能在日常生活应用中的整合。然而，它们容易产生幻觉，产生与既定事实相矛盾的主张，偏离提示，并在多次出现相同提示时产生不一致的响应。由于缺乏全面且易于评估的基准数据集，解决这些问题具有挑战性。大多数现有数据集都很小，依赖于多项选择题，这不足以评估 LLM 的生成能力。为了测量 LLM 中的幻觉，本文介绍了一个全面的基准数据集，包含八个领域的 75,000 多个提示。这些提示旨在引出明确、简洁和信息丰富的答案。数据集分为两部分：一部分公开用于测试和评估 LLM 性能，另一部分隐藏用于对各种 LLM 进行基准测试。在我们的实验中，我们测试了六款 LLM - GPT-3.5、LLama 2、LLama 3、Gemini、Mixtral 和 Zephyr - 结果显示，在公开数据集上，整体事实幻觉范围为 59% 到 82%，在隐藏基准中为 57% 到 76%。在公开数据集中，提示错位幻觉范围为 6% 到 95%，在隐藏数据集中为 17% 到 94%。平均一致性范围分别为 21% 到 61% 和 22% 到 63%。领域分析表明，当被要求提供特定数字信息时，LLM 性能会显著下降，而在人员、地点和日期查询中则表现中等。我们的数据集证明了它的有效性，并可作为 LLM 性能评估的综合基准。我们的数据集和 LLM 响应可在 \href{this https URL}{this https URL} 上找到。

Title: Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

• Authors: Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, Bryan Perozzi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09170
• Pdf URL: https://arxiv.org/pdf/2406.09170
• Copy Paste: [[2406.09170]] Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning(https://arxiv.org/abs/2406.09170)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: this https URL.
• 摘要：大型语言模型 (LLM) 已经展示了卓越的推理能力，但它们仍然容易出错，特别是在涉及复杂时间逻辑的时间推理任务中。现有研究已经使用各种数据集和基准探索了 LLM 在时间推理方面的表现。然而，这些研究通常依赖于 LLM 在预训练期间可能遇到的真实数据，或者采用匿名化技术，这些技术可能会无意中引入事实不一致。在这项工作中，我们通过引入专门设计用于评估 LLM 在各种场景中时间推理能力的新型合成数据集来解决这些限制。这些数据集中问题类型的多样性使我们能够系统地研究问题结构、大小、问题类型、事实顺序和其他因素对 LLM 性能的影响。我们的研究结果为当前 LLM 在时间推理任务中的优势和劣势提供了宝贵的见解。为了促进该领域的进一步研究，我们将实验中使用的数据集和评估框架开源：此 https URL。

Title: ReadCtrl: Personalizing text generation with readability-controlled instruction learning

• Authors: Hieu Tran, Zonghai Yao, Lingxi Li, Hong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2406.09205
• Pdf URL: https://arxiv.org/pdf/2406.09205
• Copy Paste: [[2406.09205]] ReadCtrl: Personalizing text generation with readability-controlled instruction learning(https://arxiv.org/abs/2406.09205)
• Keywords: language model, gpt, llm
• Abstract: Content generation conditioning on users's readability is an important application for personalization. In an era of large language models (LLMs), readability-controlled text generation based on LLMs has become increasingly important. This paper introduces a novel methodology called "Readability-Controlled Instruction Learning (ReadCtrl)," which aims to instruction-tune LLMs to tailor users' readability levels. Unlike the traditional methods, which primarily focused on categorical readability adjustments typically classified as high, medium, and low or expert and layperson levels with limited success, ReadCtrl introduces a dynamic framework that enables LLMs to generate content at various (near continuous level) complexity levels, thereby enhancing their versatility across different applications. Our results show that the ReadCtrl-Mistral-7B models significantly outperformed strong baseline models such as GPT-4 and Claude-3, with a win rate of 52.1%:35.7% against GPT-4 in human evaluations. Furthermore, Read-Ctrl has shown significant improvements in automatic evaluations, as evidenced by better readability metrics (e.g., FOG, FKGL) and generation quality metrics (e.g., BLEU, SARI, SummaC-Factuality, UniEval-Consistency and Coherence). These results underscore Read-Ctrl's effectiveness and tenacity in producing high-quality, contextually appropriate outputs that closely align with targeted readability levels, marking a significant advancement in personalized content generation using LLMs.
• 摘要：根据用户的可读性生成内容是个性化的重要应用。在大型语言模型 (LLM) 时代，基于 LLM 的可读性控制文本生成变得越来越重要。本文介绍了一种名为“可读性控制指令学习 (ReadCtrl)”的新方法，旨在通过指令调整 LLM 以定制用户的可读性水平。与传统方法不同，传统方法主要侧重于通常分为高、中、低或专家和外行级别的可读性调整，但成功率有限，ReadCtrl 引入了一个动态框架，使 LLM 能够以各种（接近连续级别）的复杂度级别生成内容，从而增强其在不同应用程序中的多功能性。我们的结果表明，ReadCtrl-Mistral-7B 模型明显优于 GPT-4 和 Claude-3 等强大的基线模型，在人工评估中对 GPT-4 的胜率为 52.1%：35.7%。此外，Read-Ctrl 在自动评估方面表现出显著的进步，这体现在更好的可读性指标（例如 FOG、FKGL）和生成质量指标（例如 BLEU、SARI、SummaC-Factuality、UniEval-Consistency 和 Coherence）上。这些结果凸显了 Read-Ctrl 在生成高质量、上下文适当的输出方面的有效性和持久性，这些输出与目标可读性水平紧密相关，标志着使用 LLM 进行个性化内容生成的重大进步。

Title: Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models

• Authors: Christopher Schröder, Gerhard Heyer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.09206
• Pdf URL: https://arxiv.org/pdf/2406.09206
• Copy Paste: [[2406.09206]] Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models(https://arxiv.org/abs/2406.09206)
• Keywords: language model
• Abstract: Active learning is an iterative labeling process that is used to obtain a small labeled subset, despite the absence of labeled data, thereby enabling to train a model for supervised tasks such as text classification. While active learning has made considerable progress in recent years due to improvements provided by pre-trained language models, there is untapped potential in the often neglected unlabeled portion of the data, although it is available in considerably larger quantities than the usually small set of labeled data. Here we investigate how self-training, a semi-supervised approach where a model is used to obtain pseudo-labels from the unlabeled data, can be used to improve the efficiency of active learning for text classification. Starting with an extensive reproduction of four previous self-training approaches, some of which are evaluated for the first time in the context of active learning or natural language processing, we devise HAST, a new and effective self-training strategy, which is evaluated on four text classification benchmarks, on which it outperforms the reproduced self-training approaches and reaches classification results comparable to previous experiments for three out of four datasets, using only 25% of the data.
• 摘要：主动学习是一种迭代标记过程，用于在缺乏标记数据的情况下获取一小部分标记子集，从而能够训练模型以完成文本分类等监督任务。尽管近年来由于预训练语言模型的改进，主动学习取得了长足进步，但经常被忽视的未标记数据部分仍有尚未开发的潜力，尽管此类数据的数量远多于通常较少的标记数据集。本文，我们研究了如何使用自训练（一种半监督方法，其中使用模型从未标记的数据中获取伪标签）来提高文本分类主动学习的效率。从广泛复现之前的四种自训练方法开始，其中一些方法是首次在主动学习或自然语言处理的背景下进行评估，我们设计了一种新的有效的自训练策略 HAST，该策略在四个文本分类基准上进行了评估，其表现优于复现的自训练方法，并且仅使用 25％ 的数据，在四个数据集中的三个上达到了与之前实验相当的分类结果。

Title: Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs

• Authors: Weixuan Wang, Barry Haddow, Wei Peng, Alexandra Birch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09265
• Pdf URL: https://arxiv.org/pdf/2406.09265
• Copy Paste: [[2406.09265]] Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs(https://arxiv.org/abs/2406.09265)
• Keywords: language model, llm
• Abstract: Multilingual large language models (LLMs) have greatly increased the ceiling of performance on non-English tasks. However the mechanisms behind multilingualism in these LLMs are poorly understood. Of particular interest is the degree to which internal representations are shared between languages. Recent work on neuron analysis of LLMs has focused on the monolingual case, and the limited work on the multilingual case has not considered the interaction between tasks and linguistic representations. In our work, we investigate how neuron activation is shared across languages by categorizing neurons into four distinct groups according to their responses across different languages for a particular input: all-shared, partial-shared, specific, and non-activated. This categorization is combined with a study of neuron attribution, i.e. the importance of a neuron w.r.t an output. Our analysis reveals the following insights: (i) the linguistic sharing patterns are strongly affected by the type of task, but neuron behaviour changes across different inputs even for the same task; (ii) all-shared neurons play a key role in generating correct responses; (iii) boosting multilingual alignment by increasing all-shared neurons can enhance accuracy on multilingual tasks. The code is available at this https URL.
• 摘要：多语言大型语言模型 (LLM) 大大提高了非英语任务的性能上限。然而，这些 LLM 中多语言背后的机制尚不清楚。特别令人感兴趣的是不同语言之间共享内部表征的程度。最近对 LLM 神经元分析的研究集中在单语情况，而对多语言情况的有限研究并未考虑任务和语言表征之间的相互作用。在我们的工作中，我们通过根据神经元对特定输入在不同语言中的反应将神经元分为四类不同的组来研究神经元激活在不同语言之间的共享方式：全部共享、部分共享、特定和未激活。这种分类与神经元归因研究相结合，即神经元相对于输出的重要性。我们的分析揭示了以下见解：(i) 语言共享模式受任务类型的强烈影响，但即使对于同一任务，神经元行为也会因不同的输入而发生变化；(ii) 全部共享的神经元在产生正确的反应方面起着关键作用； （三）通过增加所有共享神经元来增强多语言对齐可以提高多语言任务的准确性。代码可在此 https URL 上找到。

Title: Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

• Authors: Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09279
• Pdf URL: https://arxiv.org/pdf/2406.09279
• Copy Paste: [[2406.09279]] Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback(https://arxiv.org/abs/2406.09279)
• Keywords: language model, prompt
• Abstract: Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (this https URL) and evaluating (this https URL) our models, along with the models and datasets themselves (this https URL).
• 摘要：从偏好反馈中学习已成为提高现代语言模型 (LM) 生成质量和性能的重要步骤。尽管基于偏好的学习被广泛使用，但其应用方式却千差万别，使用的数据、学习算法和评估各不相同，因此很难理清每个方面的影响。在这项工作中，我们确定了基于偏好的学习的四个核心方面：偏好数据、学习算法、奖励模型和策略训练提示，系统地研究了这些组件对下游模型性能的影响，并提出了一种强大的偏好反馈学习方法。我们的研究结果表明，所有方面对性能都很重要，更好的偏好数据会带来最大的改进，其次是学习算法的选择、改进的奖励模型的使用，最后是使用额外的未标记提示进行策略训练。值得注意的是，PPO 在数学方面的表现比 DPO 高出 2.5%，在一般领域高出 1.2%。高质量的偏好数据可将指令遵循和真实性提高高达 8%。尽管在扩大奖励模型规模时，数学评估结果的提升高达 5%，但我们意外地发现其他类别的改进幅度很小。我们公开发布了用于训练（此 https URL）和评估（此 https URL）我们模型的代码，以及模型和数据集本身（此 https URL）。

Title: On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models

• Authors: Jinchuan Tian, Yifan Peng, William Chen, Kwanghee Choi, Karen Livescu, Shinji Watanabe
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2406.09282
• Pdf URL: https://arxiv.org/pdf/2406.09282
• Copy Paste: [[2406.09282]] On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models(https://arxiv.org/abs/2406.09282)
• Keywords: language model, llm
• Abstract: The Open Whisper-style Speech Model (OWSM) series was introduced to achieve full transparency in building advanced speech-to-text (S2T) foundation models. To this end, OWSM models are trained on 25 public speech datasets, which are heterogeneous in multiple ways. In this study, we advance the OWSM series by introducing OWSM v3.2, which improves on prior models by investigating and addressing the impacts of this data heterogeneity. Our study begins with a detailed analysis of each dataset, from which we derive two key strategies: data filtering with proxy task to enhance data quality, and the incorporation of punctuation and true-casing using an open large language model (LLM). With all other configurations staying the same, OWSM v3.2 improves performance over the OWSM v3.1 baseline while using 15% less training data.
• 摘要：引入开放式耳语式语音模型 (OWSM) 系列是为了在构建高级语音转文本 (S2T) 基础模型时实现完全透明。为此，OWSM 模型在 25 个公共语音数据集上进行训练，这些数据集在多个方面都是异构的。在本研究中，我们通过引入 OWSM v3.2 来改进 OWSM 系列，它通过调查和解决这种数据异构性的影响来改进以前的模型。我们的研究从对每个数据集的详细分析开始，从中我们得出两个关键策略：使用代理任务进行数据过滤以提高数据质量，以及使用开放式大型语言模型 (LLM) 结合标点符号和真值大小写。在所有其他配置保持不变的情况下，OWSM v3.2 的性能比 OWSM v3.1 基线有所提高，同时使用的训练数据减少了 15%。

Title: Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models

• Authors: Sarah Ball, Frauke Kreuter, Nina Rimsky
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.09289
• Pdf URL: https://arxiv.org/pdf/2406.09289
• Copy Paste: [[2406.09289]] Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models(https://arxiv.org/abs/2406.09289)
• Keywords: language model
• Abstract: Conversational Large Language Models are trained to refuse to answer harmful questions. However, emergent jailbreaking techniques can still elicit unsafe outputs, presenting an ongoing challenge for model alignment. To better understand how different jailbreak types circumvent safeguards, this paper analyses model activations on different jailbreak inputs. We find that it is possible to extract a jailbreak vector from a single class of jailbreaks that works to mitigate jailbreak effectiveness from other classes. This may indicate that different kinds of effective jailbreaks operate via similar internal mechanisms. We investigate a potential common mechanism of harmfulness feature suppression, and provide evidence for its existence by looking at the harmfulness vector component. These findings offer actionable insights for developing more robust jailbreak countermeasures and lay the groundwork for a deeper, mechanistic understanding of jailbreak dynamics in language models.
• 摘要：会话式大型语言模型经过训练，拒绝回答有害问题。然而，新兴的越狱技术仍然可能引发不安全的输出，这对模型对齐提出了持续的挑战。为了更好地了解不同类型的越狱如何规避安全措施，本文分析了不同越狱输入上的模型激活。我们发现可以从一类越狱中提取一个越狱向量，以减轻其他类越狱的有效性。这可能表明不同类型的有效越狱通过类似的内部机制运行。我们研究了一种潜在的常见有害特征抑制机制，并通过查看有害向量组件为其存在提供证据。这些发现为开发更强大的越狱对策提供了可行的见解，并为更深入、更机械地理解语言模型中的越狱动态奠定了基础。

Title: AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

• Authors: Yuhang Wu, Wenmeng Yu, Yean Cheng, Yan Wang, Xiaohan Zhang, Jiazheng Xu, Ming Ding, Yuxiao Dong
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2406.09295
• Pdf URL: https://arxiv.org/pdf/2406.09295
• Copy Paste: [[2406.09295]] AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models(https://arxiv.org/abs/2406.09295)
• Keywords: language model, gpt, prompt
• Abstract: Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, a comprehensive alignment benchmark specifically designed for emerging Chinese VLMs. This benchmark is meticulously curated from real-world scenarios and Chinese Internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we propose CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Finally, we report the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. All evaluation codes and data are available on this https URL.
• 摘要：评估大型视觉语言模型 (VLM) 的对齐能力对于确定它们作为有用助手的有效性至关重要。然而，现有的基准主要关注使用非语言方法的基本能力，例如是非题和多项选择题。在本文中，我们通过引入 AlignMMBench 来解决这一差距，这是一个专为新兴中文 VLM 设计的全面对齐基准。该基准是从现实场景和中文互联网资源中精心挑选出来的，涵盖三个类别的十三个特定任务，包括单轮和多轮对话场景。AlignMMBench 结合了即时重写策略，包含 1,054 幅图像和 4,978 个问答对。为了促进评估流程，我们提出了 CritiqueVLM，这是一个经过规则校准的评估器，其评估能力超过了 GPT-4。最后，我们报告了代表性 VLM 在 AlignMMBench 上的性能，深入了解了不同 VLM 架构的功能和局限性。所有评估代码和数据都可以在这个 https URL 上找到。

Title: Transformers meet Neural Algorithmic Reasoners

• Authors: Wilfried Bounsi, Borja Ibarz, Andrew Dudzik, Jessica B. Hamrick, Larisa Markeeva, Alex Vitvitskyi, Razvan Pascanu, Petar Veličković
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2406.09308
• Pdf URL: https://arxiv.org/pdf/2406.09308
• Copy Paste: [[2406.09308]] Transformers meet Neural Algorithmic Reasoners(https://arxiv.org/abs/2406.09308)
• Keywords: language model
• Abstract: Transformers have revolutionized machine learning with their simple yet effective architecture. Pre-training Transformers on massive text datasets from the Internet has led to unmatched generalization for natural language understanding (NLU) tasks. However, such language models remain fragile when tasked with algorithmic forms of reasoning, where computations must be precise and robust. To address this limitation, we propose a novel approach that combines the Transformer's language understanding with the robustness of graph neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs proved effective as generic solvers for algorithmic tasks, when specified in graph form. To make their embeddings accessible to a Transformer, we propose a hybrid architecture with a two-phase training procedure, allowing the tokens in the language model to cross-attend to the node embeddings from the NAR. We evaluate our resulting TransNAR model on CLRS-Text, the text-based version of the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only models for algorithmic reasoning, both in and out of distribution.
• 摘要：Transformer 以其简单而有效的架构彻底改变了机器学习。在互联网上的大量文本数据集上对 Transformer 进行预训练，为自然语言理解 (NLU) 任务带来了无与伦比的泛化。然而，当执行算法形式的推理时，这种语言模型仍然很脆弱，因为计算必须精确且稳健。为了解决这一限制，我们提出了一种新颖的方法，将 Transformer 的语言理解与基于图神经网络 (GNN) 的神经算法推理器 (NAR) 的稳健性相结合。当以图形形式指定时，此类 NAR 被证明是有效的算法任务通用求解器。为了使 Transformer 可以访问它们的嵌入，我们提出了一种具有两阶段训练过程的混合架构，允许语言模型中的标记交叉关注来自 NAR 的节点嵌入。我们在 CLRS-Text（CLRS-30 基准的基于文本的版本）上评估了我们得到的 TransNAR 模型，并证明了在算法推理方面比仅使用 Transformer 的模型有显著的提升，无论是在分布内还是分布外。

Title: REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space

• Authors: Tomer Ashuach, Martin Tutek, Yonatan Belinkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09325
• Pdf URL: https://arxiv.org/pdf/2406.09325
• Copy Paste: [[2406.09325]] REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space(https://arxiv.org/abs/2406.09325)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) risk inadvertently memorizing and divulging sensitive or personally identifiable information (PII) seen in training data, causing privacy concerns. Current approaches to address this issue involve costly dataset scrubbing, or model filtering through unlearning and model editing, which can be bypassed through extraction attacks. We propose REVS, a novel model editing method for unlearning sensitive information from LLMs. REVS identifies and modifies a small subset of neurons relevant for each piece of sensitive information. By projecting these neurons to the vocabulary space (unembedding), we pinpoint the components driving its generation. We then compute a model edit based on the pseudo-inverse of the unembedding matrix, and apply it to de-promote generation of the targeted sensitive data. To adequately evaluate our method on truly sensitive information, we curate two datasets: an email dataset inherently memorized by GPT-J, and a synthetic social security number dataset that we tune the model to memorize. Compared to other state-of-the-art model editing methods, REVS demonstrates superior performance in both eliminating sensitive information and robustness to extraction attacks, while retaining integrity of the underlying model. The code and a demo notebook are available at this https URL.
• 摘要：大型语言模型 (LLM) 可能会无意中记住并泄露训练数据中的敏感或个人身份信息 (PII)，从而引发隐私问题。目前解决此问题的方法包括昂贵的数据集清理，或通过取消学习和模型编辑进行模型过滤，这些方法可以通过提取攻击来绕过。我们提出了 REVS，这是一种用于从 LLM 中取消学习敏感信息的新型模型编辑方法。REVS 识别并修改与每条敏感信息相关的一小部分神经元。通过将这些神经元投射到词汇空间（取消嵌入），我们可以精确定位驱动其生成的组件。然后，我们根据取消嵌入矩阵的伪逆计算模型编辑，并将其应用于阻止目标敏感数据的生成。为了充分评估我们的方法对真正敏感信息的处理能力，我们整理了两个数据集：一个由 GPT-J 固有记忆的电子邮件数据集，以及一个我们调整模型以记忆的合成社会安全号码数据集。与其他最先进的模型编辑方法相比，REVS 在消除敏感信息和抵御提取攻击方面表现出色，同时保留了底层模型的完整性。代码和演示笔记本可在此 https URL 上找到。

Title: Learning from Natural Language Explanations for Generalizable Entity Matching

• Authors: Somin Wadhwa, Adit Krishnan, Runhui Wang, Byron C. Wallace, Chris Kong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09330
• Pdf URL: https://arxiv.org/pdf/2406.09330
• Copy Paste: [[2406.09330]] Learning from Natural Language Explanations for Generalizable Entity Matching(https://arxiv.org/abs/2406.09330)
• Keywords: llm
• Abstract: Entity matching is the task of linking records from different sources that refer to the same real-world entity. Past work has primarily treated entity linking as a standard supervised learning problem. However, supervised entity matching models often do not generalize well to new data, and collecting exhaustive labeled training data is often cost prohibitive. Further, recent efforts have adopted LLMs for this task in few/zero-shot settings, exploiting their general knowledge. But LLMs are prohibitively expensive for performing inference at scale for real-world entity matching tasks. As an efficient alternative, we re-cast entity matching as a conditional generation task as opposed to binary classification. This enables us to "distill" LLM reasoning into smaller entity matching models via natural language explanations. This approach achieves strong performance, especially on out-of-domain generalization tests (10.85% F-1) where standalone generative methods struggle. We perform ablations that highlight the importance of explanations, both for performance and model robustness.
• 摘要：实体匹配是将来自不同来源的记录链接到同一个现实世界实体的任务。过去的工作主要将实体链接视为标准的监督学习问题。然而，监督实体匹配模型通常不能很好地推广到新数据，而收集详尽的标记训练数据通常成本过高。此外，最近的努力已在少数/零样本设置中采用 LLM 来完成此任务，利用它们的一般知识。但 LLM 对于执行现实世界实体匹配任务的大规模推理而言成本过高。作为一种有效的替代方案，我们将实体匹配重新定义为条件生成任务，而不是二元分类。这使我们能够通过自然语言解释将 LLM 推理“提炼”为较小的实体匹配模型。这种方法实现了强大的性能，尤其是在独立生成方法难以应对的域外泛化测试（10.85% F-1）上。我们执行消融来强调解释的重要性，无论是对于性能还是模型稳健性。

Title: ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models

• Authors: David Anugraha, Genta Indra Winata, Chenyue Li, Patrick Amadeus Irawan, En-Shiun Annie Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2406.09334
• Pdf URL: https://arxiv.org/pdf/2406.09334
• Copy Paste: [[2406.09334]] ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models(https://arxiv.org/abs/2406.09334)
• Keywords: language model
• Abstract: Performance prediction is a method to estimate the performance of multilingual language models (LMs), mitigating computational costs associated with model capacity and data for fine-tuning. Our paper introduces ProxyLM, a scalable framework for predicting LM performance using proxy models in multilingual tasks. These proxy models act as surrogates, approximating the performance of fine-tuned LMs on specific downstream natural language processing (NLP) tasks. By leveraging proxy models, ProxyLM significantly reduces computational overhead on task evaluations, achieving up to a 37.08x speedup compared to traditional methods, even with our smallest proxy models. Additionally, our methodology showcases adaptability to previously unseen languages in pre-trained LMs, outperforming the state-of-the-art performance by 1.89x as measured by root-mean-square-error (RMSE). This framework streamlines model selection, enabling efficient deployment and iterative LM enhancements without extensive computational resources.
• 摘要：性能预测是一种估计多语言语言模型 (LM) 性能的方法，可减轻与模型容量和微调数据相关的计算成本。我们的论文介绍了 ProxyLM，这是一个可扩展的框架，用于使用代理模型在多语言任务中预测 LM 性能。这些代理模型充当替代模型，近似微调 LM 在特定下游自然语言处理 (NLP) 任务上的性能。通过利用代理模型，ProxyLM 显著降低了任务评估的计算开销，与传统方法相比，即使使用我们最小的代理模型，速度也能提高 37.08 倍。此外，我们的方法展示了对预训练 LM 中以前未见过的语言的适应性，以均方根误差 (RMSE) 衡量，其性能比最先进的性能高出 1.89 倍。该框架简化了模型选择，无需大量计算资源即可实现高效部署和迭代 LM 增强。

Title: DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding

• Authors: Suwon Shon, Kwangyoun Kim, Yi-Te Hsu, Prashant Sridhar, Shinji Watanabe, Karen Livescu
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2406.09345
• Pdf URL: https://arxiv.org/pdf/2406.09345
• Copy Paste: [[2406.09345]] DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding(https://arxiv.org/abs/2406.09345)
• Keywords: language model, llm
• Abstract: The integration of pre-trained text-based large language models (LLM) with speech input has enabled instruction-following capabilities for diverse speech tasks. This integration requires the use of a speech encoder, a speech adapter, and an LLM, trained on diverse tasks. We propose the use of discrete speech units (DSU), rather than continuous-valued speech encoder outputs, that are converted to the LLM token embedding space using the speech adapter. We generate DSU using a self-supervised speech encoder followed by k-means clustering. The proposed model shows robust performance on speech inputs from seen/unseen domains and instruction-following capability in spoken question answering. We also explore various types of DSU extracted from different layers of the self-supervised speech encoder, as well as Mel frequency Cepstral Coefficients (MFCC). Our findings suggest that the ASR task and datasets are not crucial in instruction-tuning for spoken question answering tasks.
• 摘要：预训练的基于文本的大型语言模型 (LLM) 与语音输入的集成使各种语音任务的指令跟踪能力成为可能。这种集成需要使用语音编码器、语音适配器和针对各种任务进行训练的 LLM。我们建议使用离散语音单元 (DSU)，而不是连续值的语音编码器输出，这些输出使用语音适配器转换为 LLM 标记嵌入空间。我们使用自监督语音编码器生成 DSU，然后进行 k 均值聚类。所提出的模型在来自可见/不可见域的语音输入和口头问答中的指令跟踪能力方面表现出稳健的性能。我们还探索了从自监督语音编码器的不同层中提取的各种类型的 DSU，以及梅尔频率倒谱系数 (MFCC)。我们的研究结果表明，ASR 任务和数据集对于口头问答任务的指令调整并不重要。