2024-03-21

Title: Automatic Summarization of Doctor-Patient Encounter Dialogues Using Large Language Model through Prompt Tuning

• Authors: Mengxian Lyu, Cheng Peng, Xiaohan Li, Patrick Balian, Jiang Bian, Yonghui Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13089
• Pdf URL: https://arxiv.org/pdf/2403.13089
• Copy Paste: [[2403.13089]] Automatic Summarization of Doctor-Patient Encounter Dialogues Using Large Language Model through Prompt Tuning(https://arxiv.org/abs/2403.13089)
• Keywords: language model, gpt, llm, prompt
• Abstract: Automatic text summarization (ATS) is an emerging technology to assist clinicians in providing continuous and coordinated care. This study presents an approach to summarize doctor-patient dialogues using generative large language models (LLMs). We developed prompt-tuning algorithms to instruct generative LLMs to summarize clinical text. We examined the prompt-tuning strategies, the size of soft prompts, and the few-short learning ability of GatorTronGPT, a generative clinical LLM developed using 277 billion clinical and general English words with up to 20 billion parameters. We compared GatorTronGPT with a previous solution based on fine-tuning of a widely used T5 model, using a clinical benchmark dataset MTS-DIALOG. The experimental results show that the GatorTronGPT- 20B model achieved the best performance on all evaluation metrics. The proposed solution has a low computing cost as the LLM parameters are not updated during prompt-tuning. This study demonstrates the efficiency of generative clinical LLMs for clinical ATS through prompt tuning.
• 摘要：自动文本摘要 (ATS) 是一项新兴技术，可帮助临床医生提供持续、协调的护理。本研究提出了一种使用生成大语言模型（LLM）总结医患对话的方法。我们开发了即时调整算法来指导生成法学硕士总结临床文本。我们检查了 GatorTronGPT 的提示调整策略、软提示的大小以及短学习能力，GatorTronGPT 是一种生成临床法学硕士，使用 2770 亿个临床和通用英语单词以及多达 200 亿个参数开发。我们使用临床基准数据集 MTS-DIALOG，将 GatorTronGPT 与之前基于广泛使用的 T5 模型微调的解决方案进行了比较。实验结果表明，GatorTronGPT-20B模型在所有评估指标上均取得了最佳性能。所提出的解决方案具有较低的计算成本，因为 LLM 参数在提示调整期间不会更新。这项研究证明了通过及时调整，生成临床法学硕士对于临床 ATS 的效率。

Title: Encode Once and Decode in Parallel: Efficient Transformer Decoding

• Authors: Bo-Ru Lu, Nikita Haduong, Chien-Yu Lin, Hao Cheng, Noah A. Smith, Mari Ostendorf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13112
• Pdf URL: https://arxiv.org/pdf/2403.13112
• Copy Paste: [[2403.13112]] Encode Once and Decode in Parallel: Efficient Transformer Decoding(https://arxiv.org/abs/2403.13112)
• Keywords: gpt, prompt
• Abstract: Transformer-based NLP models are powerful but have high computational costs that limit deployment scenarios. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and question-answering tasks where multiple outputs are required of a single input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding, thereby reducing the decoder's memory footprint. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks with comparable or better performance. We release our training/inference code and checkpoints.
• 摘要：基于 Transformer 的 NLP 模型功能强大，但计算成本较高，限制了部署场景。微调编码器-解码器模型在专业领域很流行，并且可以胜过更大、更通用的仅解码器模型，例如 GPT-4。我们引入了一种新的编码器-解码器模型配置，可以提高结构化输出和问答任务的效率，其中单个输入需要多个输出。我们的方法提示解码器 (PiD) 对输入进行一次编码并并行解码输出，通过避免重复输入编码来提高训练和推理效率，从而减少解码器的内存占用。我们实现了计算量的减少，大致与子任务的数量成比例，与用于对话状态跟踪、摘要和问答任务的最先进模型相比，速度提高了 4.6 倍，具有可比或更好的性能。我们发布了训练/推理代码和检查点。

Title: Technical Report: Competition Solution For BetterMixture

• Authors: Shuaijiang Zhao, Xiaoquan Fang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13233
• Pdf URL: https://arxiv.org/pdf/2403.13233
• Copy Paste: [[2403.13233]] Technical Report: Competition Solution For BetterMixture(https://arxiv.org/abs/2403.13233)
• Keywords: language model
• Abstract: In the era of flourishing large-scale models, the challenge of selecting and optimizing datasets from the vast and complex sea of data, to enhance the performance of large language models within the constraints of limited computational resources, has become paramount. This paper details our solution for the BetterMixture challenge, which focuses on the fine-tuning data mixing for large language models. Our approach, which secured third place, incorporates data deduplication, low-level and high-level quality filtering, and diversity selection. The foundation of our solution is Ke-Data-Juicer, an extension of Data-Juicer, demonstrating its robust capabilities in handling and optimizing data for large language models.
• 摘要：在大规模模型蓬勃发展的时代，从浩瀚而复杂的数据海洋中选择和优化数据集，在有限的计算资源的约束下增强大型语言模型的性能已成为至关重要的挑战。本文详细介绍了我们针对 BetterMixture 挑战的解决方案，该解决方案重点关注大型语言模型的数据混合微调。我们的方法获得了第三名，它结合了重复数据删除、低级和高级质量过滤以及多样性选择。我们解决方案的基础是 Ke-Data-Juicer，它是 Data-Juicer 的扩展，展示了其在处理和优化大型语言模型数据方面的强大功能。

Title: SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization

• Authors: Jacob Parnell, Inigo Jauregi Unanue, Massimo Piccardi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13240
• Pdf URL: https://arxiv.org/pdf/2403.13240
• Copy Paste: [[2403.13240]] SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization(https://arxiv.org/abs/2403.13240)
• Keywords: language model
• Abstract: Cross-lingual summarization (XLS) generates summaries in a language different from that of the input documents (e.g., English to Spanish), allowing speakers of the target language to gain a concise view of their content. In the present day, the predominant approach to this task is to take a performing, pretrained multilingual language model (LM) and fine-tune it for XLS on the language pairs of interest. However, the scarcity of fine-tuning samples makes this approach challenging in some cases. For this reason, in this paper we propose revisiting the summarize-and-translate pipeline, where the summarization and translation tasks are performed in a sequence. This approach allows reusing the many, publicly-available resources for monolingual summarization and translation, obtaining a very competitive zero-shot performance. In addition, the proposed pipeline is completely differentiable end-to-end, allowing it to take advantage of few-shot fine-tuning, where available. Experiments over two contemporary and widely adopted XLS datasets (CrossSum and WikiLingua) have shown the remarkable zero-shot performance of the proposed approach, and also its strong few-shot performance compared to an equivalent multilingual LM baseline, that the proposed approach has been able to outperform in many languages with only 10% of the fine-tuning samples.
• 摘要：跨语言摘要 (XLS) 使用与输入文档不同的语言（例如英语到西班牙语）生成摘要，使目标语言的使用者能够获得其内容的简洁视图。目前，完成此任务的主要方法是采用执行的、预训练的多语言语言模型 (LM)，并针对感兴趣的语言对对其进行微调以适用于 XLS。然而，微调样本的稀缺使得这种方法在某些情况下具有挑战性。因此，在本文中，我们建议重新审视总结和翻译管道，其中总结和翻译任务按顺序执行。这种方法允许重复使用许多公开可用的资源进行单语摘要和翻译，从而获得非常有竞争力的零样本性能。此外，所提出的管道是端到端完全可微的，使其能够在可用的情况下利用少样本微调。在两个当代且广泛采用的 XLS 数据集（CrossSum 和 WikiLingua）上进行的实验表明，所提出的方法具有卓越的零样本性能，并且与等效的多语言 LM 基线相比，其强大的少样本性能也表明所提出的方法能够只需 10% 的微调样本即可在许多语言中表现出色。

Title: Instruction Multi-Constraint Molecular Generation Using a Teacher-Student Large Language Model

• Authors: Peng Zhou, Jianmin Wang, Chunyan Li, Zixu Wang, Yiping Liu, Siqi Sun, Jianxin Lin, Longyue Wang, Xiangxiang Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13244
• Pdf URL: https://arxiv.org/pdf/2403.13244
• Copy Paste: [[2403.13244]] Instruction Multi-Constraint Molecular Generation Using a Teacher-Student Large Language Model(https://arxiv.org/abs/2403.13244)
• Keywords: language model, prompt
• Abstract: While various models and computational tools have been proposed for structure and property analysis of molecules, generating molecules that conform to all desired structures and properties remains a challenge. Here, we introduce a multi-constraint molecular generation large language model, TSMMG, which, akin to a student, incorporates knowledge from various small models and tools, namely, the 'teachers'. To train TSMMG, we construct a large set of text-molecule pairs by extracting molecular knowledge from these 'teachers', enabling it to generate novel molecules that conform to the descriptions through various text prompts. We experimentally show that TSMMG remarkably performs in generating molecules meeting complex, natural language-described property requirements across two-, three-, and four-constraint tasks, with an average molecular validity of over 99% and success ratio of 88.08%, 65.27%, and 61.44%, respectively. The model also exhibits adaptability through zero-shot testing, creating molecules that satisfy combinations of properties that have not been encountered. It can comprehend text inputs with various language styles, extending beyond the confines of outlined prompts, as confirmed through empirical validation. Additionally, the knowledge distillation feature of TSMMG contributes to the continuous enhancement of small models, while the innovative approach to dataset construction effectively addresses the issues of data scarcity and quality, which positions TSMMG as a promising tool in the domains of drug discovery and materials science. Code is available at https://github.com/HHW-zhou/TSMMG.
• 摘要：尽管已经提出了各种模型和计算工具用于分子的结构和性质分析，但生成符合所有所需结构和性质的分子仍然是一个挑战。在这里，我们介绍了一种多约束分子生成大语言模型 TSMMG，它类似于学生，融合了来自各种小模型和工具（即“老师”）的知识。为了训练 TSMMG，我们通过从这些“老师”那里提取分子知识来构建大量的文本-分子对，使其能够通过各种文本提示生成符合描述的新颖分子。我们的实验表明，TSMMG 在生成满足复杂的、自然语言描述的属性要求的分子方面表现出色，跨越两个、三个和四个约束任务，平均分子有效性超过 99%，成功率分别为 88.08%、65.27% 、 和 61.44%。该模型还通过零次测试展现了适应性，创建了满足尚未遇到的属性组合的分子。正如通过经验验证所证实的，它可以理解具有各种语言风格的文本输入，超出了概述提示的范围。此外，TSMMG的知识蒸馏功能有助于小模型的不断增强，而数据集构建的创新方法有效解决了数据稀缺和质量问题，这使TSMMG成为药物发现和材料科学领域的一个有前途的工具。代码可在 https://github.com/HHW-zhou/TSMMG 获取。

Title: Facilitating Pornographic Text Detection for Open-Domain Dialogue Systems via Knowledge Distillation of Large Language Models

• Authors: Huachuan Qiu, Shuai Zhang, Hongliang He, Anqi Li, Zhenzhong Lan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13250
• Pdf URL: https://arxiv.org/pdf/2403.13250
• Copy Paste: [[2403.13250]] Facilitating Pornographic Text Detection for Open-Domain Dialogue Systems via Knowledge Distillation of Large Language Models(https://arxiv.org/abs/2403.13250)
• Keywords: language model, gpt, chat
• Abstract: Pornographic content occurring in human-machine interaction dialogues can cause severe side effects for users in open-domain dialogue systems. However, research on detecting pornographic language within human-machine interaction dialogues is an important subject that is rarely studied. To advance in this direction, we introduce CensorChat, a dialogue monitoring dataset aimed at detecting whether the dialogue session contains pornographic content. To this end, we collect real-life human-machine interaction dialogues in the wild and break them down into single utterances and single-turn dialogues, with the last utterance spoken by the chatbot. We propose utilizing knowledge distillation of large language models to annotate the dataset. Specifically, first, the raw dataset is annotated by four open-source large language models, with the majority vote determining the label. Second, we use ChatGPT to update the empty label from the first step. Third, to ensure the quality of the validation and test sets, we utilize GPT-4 for label calibration. If the current label does not match the one generated by GPT-4, we employ a self-criticism strategy to verify its correctness. Finally, to facilitate the detection of pornographic text, we develop a series of text classifiers using a pseudo-labeled dataset. Detailed data analysis demonstrates that leveraging knowledge distillation techniques with large language models provides a practical and cost-efficient method for developing pornographic text detectors.
• 摘要：人机交互对话中出现的色情内容可能会对开放域对话系统中的用户造成严重的副作用。然而，人机交互对话中检测色情语言的研究是一个很少被研究的重要课题。为了朝这个方向前进，我们引入了 CensorChat，这是一个对话监控数据集，旨在检测对话会话是否包含色情内容。为此，我们在野外收集现实生活中的人机交互对话，并将其分解为单句话和单回合对话，最后一句话是由聊天机器人说出的。我们建议利用大型语言模型的知识蒸馏来注释数据集。具体来说，首先，原始数据集由四个开源大型语言模型进行注释，并以多数票决定标签。其次，我们使用 ChatGPT 更新第一步中的空标签。第三，为了确保验证和测试集的质量，我们利用 GPT-4 进行标签校准。如果当前标签与 GPT-4 生成的标签不匹配，我们会采用一种自我批评策略来验证其正确性。最后，为了促进色情文本的检测，我们使用伪标记数据集开发了一系列文本分类器。详细的数据分析表明，利用大型语言模型的知识蒸馏技术为开发色情文本检测器提供了一种实用且经济高效的方法。

Title: Arcee's MergeKit: A Toolkit for Merging Large Language Models

• Authors: Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.13257
• Pdf URL: https://arxiv.org/pdf/2403.13257
• Copy Paste: [[2403.13257]] Arcee's MergeKit: A Toolkit for Merging Large Language Models(https://arxiv.org/abs/2403.13257)
• Keywords: language model, llm
• Abstract: The rapid expansion of the open-source language model landscape presents an opportunity to merge the competencies of these model checkpoints by combining their parameters. Advances in transfer learning, the process of fine-tuning pre-trained models for specific tasks, has resulted in the development of vast amounts of task-specific models, typically specialized in individual tasks and unable to utilize each other's strengths. Model merging facilitates the creation of multitask models without the need for additional training, offering a promising avenue for enhancing model performance and versatility. By preserving the intrinsic capabilities of the original models, model merging addresses complex challenges in AI - including the difficulties of catastrophic forgetting and multi-task learning. To support this expanding area of research, we introduce MergeKit, a comprehensive, open-source library designed to facilitate the application of model merging strategies. MergeKit offers an extensible framework to efficiently merge models on any hardware, providing utility to researchers and practitioners. To date, thousands of models have been merged by the open-source community, leading to the creation of some of the worlds most powerful open-source model checkpoints, as assessed by the Open LLM Leaderboard. The library is accessible at https://github.com/arcee-ai/MergeKit.
• 摘要：开源语言模型领域的快速扩展提供了通过组合参数来合并这些模型检查点能力的机会。迁移学习（针对特定任务对预训练模型进行微调的过程）的进步导致了大量特定于任务的模型的开发，这些模型通常专门针对单个任务并且无法利用彼此的优势。模型合并有助于创建多任务模型，而无需额外的训练，为增强模型性能和多功能性提供了一条有希望的途径。通过保留原始模型的内在功能，模型合并解决了人工智能中的复杂挑战，包括灾难性遗忘和多任务学习的困难。为了支持这一不断扩大的研究领域，我们引入了 MergeKit，这是一个综合性的开源库，旨在促进模型合并策略的应用。 MergeKit 提供了一个可扩展的框架，可以在任何硬件上有效地合并模型，为研究人员和从业者提供实用性。迄今为止，开源社区已经合并了数千个模型，从而创建了一些世界上最强大的开源模型检查点，正如 Open LLM 排行榜所评估的那样。该库可通过 https://github.com/arcee-ai/MergeKit 访问。

Title: LeanReasoner: Boosting Complex Logical Reasoning with Lean

• Authors: Dongwei Jiang, Marcio Fonseca, Shay B. Cohen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13312
• Pdf URL: https://arxiv.org/pdf/2403.13312
• Copy Paste: [[2403.13312]] LeanReasoner: Boosting Complex Logical Reasoning with Lean(https://arxiv.org/abs/2403.13312)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often struggle with complex logical reasoning due to logical inconsistencies and the inherent difficulty of such reasoning. We use Lean, a theorem proving framework, to address these challenges. By formalizing logical reasoning problems into theorems within Lean, we can solve them by proving or disproving the corresponding theorems. This method reduces the risk of logical inconsistencies with the help of Lean's symbolic solver. It also enhances our ability to treat complex reasoning tasks by using Lean's extensive library of theorem proofs. Our method achieves state-of-the-art performance on the FOLIO dataset and achieves performance near this level on ProofWriter. Notably, these results were accomplished by fine-tuning on fewer than 100 in-domain samples for each dataset.
• 摘要：由于逻辑不一致以及此类推理固有的困难，大型语言模型 (LLM) 常常难以应对复杂的逻辑推理。我们使用精益（Lean）这个定理证明框架来应对这些挑战。通过将逻辑推理问题形式化为精益中的定理，我们可以通过证明或反驳相应的定理来解决它们。该方法借助 Lean 的符号求解器降低了逻辑不一致的风险。它还通过使用 Lean 广泛的定理证明库来增强我们处理复杂推理任务的能力。我们的方法在 FOLIO 数据集上实现了最先进的性能，并在 ProofWriter 上实现了接近这一水平的性能。值得注意的是，这些结果是通过对每个数据集的不到 100 个域内样本进行微调来实现的。

Title: Hyacinth6B: A large language model for Traditional Chinese

• Authors: Chih-Wei Song, Yin-Te Tsai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13334
• Pdf URL: https://arxiv.org/pdf/2403.13334
• Copy Paste: [[2403.13334]] Hyacinth6B: A large language model for Traditional Chinese(https://arxiv.org/abs/2403.13334)
• Keywords: language model, llm
• Abstract: This research's primary motivation of this study is to address the high hardware and computational demands typically associated with LLMs.Therefore,our goal is to find a balance between model lightness and performance,striving to maximize performance while using a comparatively lightweight model. Hyacinth6B was developed with this objective in mind,aiming to fully leverage the core capabilities of LLMs without incurring substantial resource costs, effectively pushing the boundaries of smaller model's performance. The training approach involves parameter efficient finetuning using the LoRA method.
• 摘要：本研究的主要动机是解决通常与法学硕士相关的高硬件和计算需求。因此，我们的目标是在模型轻量性和性能之间找到平衡，力求在使用相对轻量级的模型时最大化性能。 Hyacinth6B 的开发就是考虑到这一目标，旨在充分利用法学硕士的核心功能，而不产生大量的资源成本，有效地突破较小模型的性能界限。训练方法涉及使用 LoRA 方法进行参数高效微调。

Title: Computational Models to Study Language Processing in the Human Brain: A Survey

• Authors: Shaonan Wang, Jingyuan Sun, Yunhao Zhang, Nan Lin, Marie-Francine Moens, Chengqing Zong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13368
• Pdf URL: https://arxiv.org/pdf/2403.13368
• Copy Paste: [[2403.13368]] Computational Models to Study Language Processing in the Human Brain: A Survey(https://arxiv.org/abs/2403.13368)
• Keywords: language model
• Abstract: Despite differing from the human language processing mechanism in implementation and algorithms, current language models demonstrate remarkable human-like or surpassing language capabilities. Should computational language models be employed in studying the brain, and if so, when and how? To delve into this topic, this paper reviews efforts in using computational models for brain research, highlighting emerging trends. To ensure a fair comparison, the paper evaluates various computational models using consistent metrics on the same dataset. Our analysis reveals that no single model outperforms others on all datasets, underscoring the need for rich testing datasets and rigid experimental control to draw robust conclusions in studies involving computational models.
• 摘要：尽管在实现和算法上与人类语言处理机制不同，但当前的语言模型表现出了显着的类人类或超越人类的语言能力。是否应该采用计算语言模型来研究大脑？如果是，何时以及如何？为了深入探讨这个主题，本文回顾了使用计算模型进行大脑研究的努力，强调了新兴趋势。为了确保公平比较，本文在同一数据集上使用一致的指标评估各种计算模型。我们的分析表明，没有任何一个模型在所有数据集上都优于其他模型，这强调了在涉及计算模型的研究中需要丰富的测试数据集和严格的实验控制才能得出可靠的结论。

Title: Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting

• Authors: Phillip Richter-Pechanski, Philipp Wiesenbach, Dominic M. Schwab, Christina Kiriakou, Nicolas Geis, Christoph Dieterich, Anette Frank
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.13369
• Pdf URL: https://arxiv.org/pdf/2403.13369
• Copy Paste: [[2403.13369]] Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting(https://arxiv.org/abs/2403.13369)
• Keywords: language model, prompt
• Abstract: Automatic extraction of medical information from clinical documents poses several challenges: high costs of required clinical expertise, limited interpretability of model predictions, restricted computational resources and privacy regulations. Recent advances in domain-adaptation and prompting methods showed promising results with minimal training data using lightweight masked language models, which are suited for well-established interpretability methods. We are first to present a systematic evaluation of these methods in a low-resource setting, by performing multi-class section classification on German doctor's letters. We conduct extensive class-wise evaluations supported by Shapley values, to validate the quality of our small training data set and to ensure the interpretability of model predictions. We demonstrate that a lightweight, domain-adapted pretrained model, prompted with just 20 shots, outperforms a traditional classification model by 30.5% accuracy. Our results serve as a process-oriented guideline for clinical information extraction projects working with low-resource.
• 摘要：从临床文档中自动提取医疗信息带来了几个挑战：所需临床专业知识的成本高昂、模型预测的可解释性有限、计算资源有限和隐私法规。领域适应和提示方法的最新进展显示出使用轻量级掩码语言模型的最少训练数据的有希望的结果，该模型适合完善的可解释性方法。我们首先通过对德国医生的信件进行多类部分分类，在资源匮乏的情况下对这些方法进行系统评估。我们在 Shapley 值的支持下进行广泛的分类评估，以验证小型训练数据集的质量并确保模型预测的可解释性。我们证明了一个轻量级、领域适应的预训练模型，只需 20 个镜头即可提示，其准确率比传统分类模型高出 30.5%。我们的结果可以作为资源匮乏的临床信息提取项目的面向流程的指南。

Title: LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models

• Authors: Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13372
• Pdf URL: https://arxiv.org/pdf/2403.13372
• Copy Paste: [[2403.13372]] LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models(https://arxiv.org/abs/2403.13372)
• Keywords: language model, llm
• Abstract: Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It allows users to flexibly customize the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://github.com/hiyouga/LLaMA-Factory and already received over 13,000 stars and 1,600 forks.
• 摘要：高效的微调对于使大型语言模型 (LLM) 适应下游任务至关重要。然而，在不同的模型上实现这些方法需要付出很大的努力。我们推出了 LlamaFactory，一个集成了一套尖端高效训练方法的统一框架。它允许用户灵活地定制100多个LLM的微调，而无需通过内置的Web UI LlamaBoard进行编码。我们凭经验验证了我们的框架在语言建模和文本生成任务上的效率和有效性。它已在 https://github.com/hiyouga/LLaMA-Factory 发布，并已获得超过 13,000 个 star 和 1,600 个分叉。

Title: An Entropy-based Text Watermarking Detection Method

• Authors: Yijian Lu, Aiwei Liu, Dianzhi Yu, Jingjing Li, Irwin King
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13485
• Pdf URL: https://arxiv.org/pdf/2403.13485
• Copy Paste: [[2403.13485]] An Entropy-based Text Watermarking Detection Method(https://arxiv.org/abs/2403.13485)
• Keywords: language model, llm
• Abstract: Currently, text watermarking algorithms for large language models (LLMs) can embed hidden features to texts generated by LLMs to facilitate subsequent detection, thus alleviating the problem of misuse of LLMs. Although the current text watermarking algorithms perform well in most high-entropy scenarios, its performance in low-entropy scenarios still needs to be improved. In this work, we proposed that the influence of token entropy should be fully considered in the watermark detection process, that is, the weight of each token should be adjusted according to its entropy during watermark detection, rather than setting the weight of all tokens to the same value as in previous methods. Specifically, we proposed an Entropy-based Watermark Detection (EWD) that gives higher-entropy tokens higher weights during watermark detection, so as to better reflect the degree of watermarking. Furthermore, the proposed detection process is training-free and fully automated. %In actual detection, we use a proxy-LLM to calculate the entropy of each token, without the need to use the original LLM. In the experiment, we found that our method can achieve better detection performance in low-entropy scenarios, and our method is also general and can be applied to texts with different entropy distributions. Our code and data will be available online.
• 摘要：目前，大型语言模型（LLM）的文本水印算法可以将隐藏特征嵌入到LLM生成的文本中，以方便后续检测，从而缓解LLM的误用问题。尽管当前的文本水印算法在大多数高熵场景下表现良好，但其在低熵场景下的性能仍有待提高。在这项工作中，我们提出在水印检测过程中应充分考虑令牌熵的影响，即在水印检测过程中应根据每个令牌的熵来调整每个令牌的权重，而不是将所有令牌的权重设置为与之前方法中的值相同。具体来说，我们提出了一种基于熵的水印检测（EWD），在水印检测过程中赋予较高熵的标记更高的权重，从而更好地反映水印的程度。此外，所提出的检测过程无需培训且完全自动化。 %在实际检测中，我们使用proxy-LLM来计算每个token的熵，而不需要使用原始的LLM。在实验中，我们发现我们的方法在低熵场景下可以取得更好的检测性能，而且我们的方法也具有通用性，可以应用于不同熵分布的文本。我们的代码和数据将在线提供。

Title: How Gender Interacts with Political Values: A Case Study on Czech BERT Models

• Authors: Adnan Al Ali, Jindřich Libovický
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.13514
• Pdf URL: https://arxiv.org/pdf/2403.13514
• Copy Paste: [[2403.13514]] How Gender Interacts with Political Values: A Case Study on Czech BERT Models(https://arxiv.org/abs/2403.13514)
• Keywords: language model
• Abstract: Neural language models, which reach state-of-the-art results on most natural language processing tasks, are trained on large text corpora that inevitably contain value-burdened content and often capture undesirable biases, which the models reflect. This case study focuses on the political biases of pre-trained encoders in Czech and compares them with a representative value survey. Because Czech is a gendered language, we also measure how the grammatical gender coincides with responses to men and women in the survey. We introduce a novel method for measuring the model's perceived political values. We find that the models do not assign statement probability following value-driven reasoning, and there is no systematic difference between feminine and masculine sentences. We conclude that BERT-sized models do not manifest systematic alignment with political values and that the biases observed in the models are rather due to superficial imitation of training data patterns than systematic value beliefs encoded in the models.
• 摘要：神经语言模型在大多数自然语言处理任务上都达到了最先进的结果，它是在大型文本语料库上进行训练的，这些文本语料库不可避免地包含价值负担的内容，并且经常捕获模型所反映的不良偏见。本案例研究重点关注捷克预训练编码器的政治偏见，并将其与代表性值调查进行比较。由于捷克语是一种性别语言，我们还衡量了语法性别与调查中男性和女性的回答是否一致。我们引入了一种测量模型感知政治价值的新方法。我们发现模型没有按照价值驱动推理分配陈述概率，并且女性和男性句子之间不存在系统差异。我们的结论是，BERT 规模的模型并未表现出与政治价值观的系统一致性，并且模型中观察到的偏差更多是由于对训练数据模式的肤浅模仿，而不是模型中编码的系统价值信念。

Title: Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation

• Authors: Do June Min, Veronica Perez-Rosas, Kenneth Resnicow, Rada Mihalcea
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.13578
• Pdf URL: https://arxiv.org/pdf/2403.13578
• Copy Paste: [[2403.13578]] Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation(https://arxiv.org/abs/2403.13578)
• Keywords: language model
• Abstract: In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models.
• 摘要：在本文中，我们研究了多奖励强化学习问题，以联合优化自然语言生成的多种文本质量。我们专注于辅导员反思生成的任务，优化生成器以同时提高生成的辅导员响应的流畅性、连贯性和反思质量。我们引入了两种新颖的老虎机方法，DynaOpt 和 C-DynaOpt，它们依赖于将奖励组合成单个值并同时优化它们的广泛策略。具体来说，我们采用非上下文和上下文多臂老虎机在训练期间动态调整多个奖励权重。通过自动和手动评估，我们表明我们提出的技术 DynaOpt 和 C-DynaOpt 优于现有的 naive 和 bandit 基线，展示了它们增强语言模型的潜力。

Title: Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models

• Authors: Adian Liusie, Yassir Fathullah, Mark J. F. Gales
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13590
• Pdf URL: https://arxiv.org/pdf/2403.13590
• Copy Paste: [[2403.13590]] Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models(https://arxiv.org/abs/2403.13590)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities and versatility in NLP tasks, however they sometimes fail to maintain crucial invariances for specific tasks. One example is permutation sensitivity, where LLMs' outputs may significantly vary depending on the order of the input options. While debiasing techniques can mitigate these issues, and yield better performance and reliability, they often come with a high computational cost at inference. This paper addresses this inefficiency at inference time. The aim is to distill the capabilities of a computationally intensive, debiased, teacher model into a more compact student model. We explore two variants of student models: one based on pure distillation, and the other on an error-correction approach for more complex tasks, where the student corrects a single biased decision from the teacher to achieve a debiased output. Our approach is general and can be applied to both black-box and white-box LLMs. Furthermore, we demonstrate that our compact, encoder-only student models can outperform their larger, biased teacher counterparts, achieving better results with significantly fewer parameters.
• 摘要：大型语言模型 (LLM) 在 NLP 任务中展示了令人印象深刻的零样本能力和多功能性，但它们有时无法保持特定任务的关键不变性。一个例子是排列敏感性，其中法学硕士的输出可能会根据输入选项的顺序而显着变化。虽然去偏技术可以缓解这些问题，并产生更好的性能和可靠性，但它们在推理时通常会带来很高的计算成本。本文解决了推理时的低效率问题。目的是将计算密集型、去偏的教师模型的功能提炼为更紧凑的学生模型。我们探索了学生模型的两种变体：一种基于纯粹的蒸馏，另一种基于针对更复杂任务的纠错方法，其中学生纠正教师的单个有偏见的决策以实现无偏见的输出。我们的方法是通用的，可以应用于黑盒和白盒法学硕士。此外，我们证明了我们的紧凑型、仅编码器的学生模型可以胜过更大的、有偏见的教师模型，用更少的参数获得更好的结果。

Title: Llama meets EU: Investigating the European Political Spectrum through the Lens of LLMs

• Authors: Ilias Chalkidis, Stephanie Brandl
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13592
• Pdf URL: https://arxiv.org/pdf/2403.13592
• Copy Paste: [[2403.13592]] Llama meets EU: Investigating the European Political Spectrum through the Lens of LLMs(https://arxiv.org/abs/2403.13592)
• Keywords: language model, llm, chat
• Abstract: Instruction-finetuned Large Language Models inherit clear political leanings that have been shown to influence downstream task performance. We expand this line of research beyond the two-party system in the US and audit Llama Chat in the context of EU politics in various settings to analyze the model's political knowledge and its ability to reason in context. We adapt, i.e., further fine-tune, Llama Chat on speeches of individual euro-parties from debates in the European Parliament to reevaluate its political leaning based on the EUandI questionnaire. Llama Chat shows considerable knowledge of national parties' positions and is capable of reasoning in context. The adapted, party-specific, models are substantially re-aligned towards respective positions which we see as a starting point for using chat-based LLMs as data-driven conversational engines to assist research in political science.
• 摘要：指令微调的大型语言模型继承了明确的政治倾向，这些倾向已被证明会影响下游任务的绩效。我们将这一研究范围扩展到美国的两党制度之外，并在各种环境下的欧盟政治背景下审核 Llama Chat，以分析该模型的政治知识及其在背景下推理的能力。我们根据欧洲议会辩论中各个欧洲政党的演讲对 Llama Chat 进行调整，即进一步微调，以根据 EUandI 调查问卷重新评估其政治倾向。 Llama Chat 显示出对国家政党立场的丰富了解，并且能够结合上下文进行推理。经过调整的、特定于政党的模型基本上针对各自的立场进行了重新调整，我们认为这是使用基于聊天的法学硕士作为数据驱动的对话引擎来协助政治学研究的起点。

Title: Do Not Worry if You Do Not Have Data: Building Pretrained Language Models Using Translationese

• Authors: Meet Doshi, Raj Dabre, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13638
• Pdf URL: https://arxiv.org/pdf/2403.13638
• Copy Paste: [[2403.13638]] Do Not Worry if You Do Not Have Data: Building Pretrained Language Models Using Translationese(https://arxiv.org/abs/2403.13638)
• Keywords: language model
• Abstract: In this paper, we explore the utility of \textit{Translationese} as synthetic data created using machine translation for pre-training language models (LMs). Pre-training requires vast amounts of monolingual data, which is mostly unavailable for languages other than English. Recently, there has been a growing interest in using synthetic data to address this data scarcity. We take the case of English and Indic languages and translate web-crawled monolingual documents (clean) into the target language. Then, we train language models containing 28M and 85M parameters on this translationese data (synthetic). We show that their performance on downstream natural language understanding and generative tasks is only 3.56\% poorer on NLU tasks and 1.51\% on NLG tasks than LMs pre-trained on clean data. Further, we propose the use of lightweight \textit{TinyLMs} pre-trained on clean data to filter synthetic data efficiently which significantly improves the performance of our models. We also find that LMs trained on synthetic data strongly benefit from extended pretraining on a tiny fraction (10\%) of clean data. We release the data we collected and created as a part of this work, \textit{IndicMonoDoc}, the largest collection of monolingual document-level corpora, which we hope will help bridge the gap between English and non-English performance for large language models.
• 摘要：在本文中，我们探讨了 \textit{Translationese} 作为使用机器翻译创建的合成数据用于预训练语言模型（LM）的实用性。预训练需要大量的单语数据，而这对于英语以外的语言来说大多是无法获得的。最近，人们对使用合成数据来解决数据稀缺问题越来越感兴趣。我们以英语和印度语为例，将网络爬取的单语文档（干净的）翻译成目标语言。然后，我们在此翻译数据（合成）上训练包含 28M 和 85M 参数的语言模型。我们表明，与在干净数据上预训练的 LM 相比，它们在下游自然语言理解和生成任务上的性能在 NLU 任务上仅差 3.56%，在 NLG 任务上仅差 1.51%。此外，我们建议使用在干净数据上预训练的轻量级 \textit{TinyLMs} 来有效过滤合成数据，从而显着提高我们模型的性能。我们还发现，在合成数据上训练的 LM 受益于对一小部分 (10\%) 干净数据的扩展预训练。我们发布了作为这项工作的一部分收集和创建的数据，\textit{IndicMonoDoc}，这是最大的单语文档级语料库集合，我们希望它将有助于缩小大型语言模型的英语和非英语性能之间的差距。

Title: Grounding Spatial Relations in Text-Only Language Models

• Authors: Gorka Azkune, Ander Salaberria, Eneko Agirre
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13666
• Pdf URL: https://arxiv.org/pdf/2403.13666
• Copy Paste: [[2403.13666]] Grounding Spatial Relations in Text-Only Language Models(https://arxiv.org/abs/2403.13666)
• Keywords: language model
• Abstract: This paper shows that text-only Language Models (LM) can learn to ground spatial relations like "left of" or "below" if they are provided with explicit location information of objects and they are properly trained to leverage those locations. We perform experiments on a verbalized version of the Visual Spatial Reasoning (VSR) dataset, where images are coupled with textual statements which contain real or fake spatial relations between two objects of the image. We verbalize the images using an off-the-shelf object detector, adding location tokens to every object label to represent their bounding boxes in textual form. Given the small size of VSR, we do not observe any improvement when using locations, but pretraining the LM over a synthetic dataset automatically derived by us improves results significantly when using location tokens. We thus show that locations allow LMs to ground spatial relations, with our text-only LMs outperforming Vision-and-Language Models and setting the new state-of-the-art for the VSR dataset. Our analysis show that our text-only LMs can generalize beyond the relations seen in the synthetic dataset to some extent, learning also more useful information than that encoded in the spatial rules we used to create the synthetic dataset itself.
• 摘要：本文表明，如果为纯文本语言模型（LM）提供了对象的明确位置信息，并且经过适当的训练以利用这些位置，则纯文本语言模型（LM）可以学习基础空间关系，例如“左侧”或“下方”。我们在视觉空间推理（VSR）数据集的语言化版本上进行实验，其中图像与文本语句相结合，其中包含图像的两个对象之间真实或虚假的空间关系。我们使用现成的对象检测器来表达图像，向每个对象标签添加位置标记，以文本形式表示它们的边界框。鉴于 VSR 规模较小，我们在使用位置时没有观察到任何改进，但在我们自动导出的合成数据集上对 LM 进行预训练可以在使用位置标记时显着改善结果。因此，我们表明位置允许 LM 建立空间关系，我们的纯文本 LM 优于视觉和语言模型，并为 VSR 数据集设置了新的最先进技术。我们的分析表明，我们的纯文本 LM 可以在某种程度上泛化到合成数据集中看到的关系之外，还可以学习比我们用来创建合成数据集本身的空间规则中编码的信息更有用的信息。

Title: RoleInteract: Evaluating the Social Interaction of Role-Playing Agents

• Authors: Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, Jingren Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13679
• Pdf URL: https://arxiv.org/pdf/2403.13679
• Copy Paste: [[2403.13679]] RoleInteract: Evaluating the Social Interaction of Role-Playing Agents(https://arxiv.org/abs/2403.13679)
• Keywords: language model, llm, prompt, agent
• Abstract: Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce RoleInteract, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on RoleInteract confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/RoleInteract.
• 摘要：大型语言模型 (LLM) 促进了各种人工智能对话代理的开发，包括模仿不同角色和人类行为的角色扮演对话代理。虽然之前的研究主要集中在增强这些代理的对话能力、特定角色的知识和风格属性上，但在评估他们的社交智力方面存在明显的差距。在本文中，我们介绍了 RoleInteract，这是第一个基准测试，旨在系统地评估角色扮演对话代理在个人和团体层面的社交互动的社交性。该基准测试由多种来源构建而成，涵盖 500 个字符、6,000 多个问题提示和 30,800 条多回合角色扮演话语。我们使用主流的开源和闭源法学硕士对该基准进行综合评估。我们发现，代理人在个人层面上表现出色并不意味着他们在群体层面上也很熟练。此外，个体的行为可能会由于群体内其他主体施加的影响而发生变化。 RoleInteract 的实验结果证实了其作为评估角色扮演对话代理的社交互动的测试平台的重要性。该基准可在 https://github.com/X-PLUG/RoleInteract 上公开访问。

Title: PARAMANU-AYN: An Efficient Novel Generative and Instruction-tuned Language Model for Indian Legal Case Documents

• Authors: Mitodru Niyogi, Arnab Bhattacharya
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.13681
• Pdf URL: https://arxiv.org/pdf/2403.13681
• Copy Paste: [[2403.13681]] PARAMANU-AYN: An Efficient Novel Generative and Instruction-tuned Language Model for Indian Legal Case Documents(https://arxiv.org/abs/2403.13681)
• Keywords: language model, gpt, prompt
• Abstract: In this paper, we present PARAMANU-AYN, a language model based exclusively on case documents of the Supreme Court of India, the Constitution of India, and the Indian Penal Code. The novel Auto Regressive (AR) decoder based model is pretrained from scratch at a context size of 8192. We evaluated our pretrained legal model on perplexity metrics. We also instruction-tuned our pretrained model on a set of 10,763 instructions covering various legal tasks such as legal reasoning, judgement explanation, legal clause generation, legal drafting, legal contract drafting, case summarization, constitutional question-answering, etc. We also evaluated the responses of prompts for instruction-tuned models by GPT-3.5-Turbo on clarity, relevance, completeness, and legal reasoning metrics in a scale of 10. Our model can be run on CPU and achieved 42.46 tokens/sec CPU inference speed. We found that our models, despite not being pretrained on legal books, various legal contracts, and legal documents, were able to learn the domain knowledge required for drafting various legal contracts and legal clauses, and generalize to draft legal contracts and legal clauses with limited instruction tuning. Hence, we conclude that for a strong domain-specialized generative language model (such as legal), very large amounts of data are not required to develop models from scratch. We believe that this work is the first attempt to make a dedicated generative legal language model from scratch for Indian Supreme Court jurisdiction or in legal NLP overall. We plan to release our Paramanu-Ayn model at https://www.bharatgpts.com.
• 摘要：在本文中，我们提出了 PARAMANU-AYN，这是一种完全基于印度最高法院的案例文件、印度宪法和印度刑法典的语言模型。基于新颖的自回归 (AR) 解码器的模型是在上下文大小 8192 下从头开始预训练的。我们根据困惑度指标评估了预训练的法律模型。我们还在一组 10,763 条指令上对我们的预训练模型进行了指令调整，这些指令涵盖了法律推理、判决解释、法律条款生成、法律起草、法律合同起草、案件摘要、宪法问答等各种法律任务。我们还评估了GPT-3.5-Turbo 对指令调整模型的提示响应的清晰度、相关性、完整性和法律推理指标（10 级）。我们的模型可以在 CPU 上运行，并实现了 42.46 个令牌/秒的 CPU 推理速度。我们发现，我们的模型尽管没有接受过法律书籍、各种法律合同和法律文件的预训练，但仍能够学习起草各种法律合同和法律条款所需的领域知识，并在有限的情况下泛化起草法律合同和法律条款。指令调整。因此，我们得出的结论是，对于强大的专业领域生成语言模型（例如法律），不需要大量数据来从头开始开发模型。我们相信这项工作是首次尝试从头开始为印度最高法院管辖权或整个法律 NLP 制作专用的生成法律语言模型。我们计划在 https://www.bharatgpts.com 上发布 Paramanu-Ayn 模型。

Title: EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation

• Authors: Atnafu Lambebo Tonja, Israel Abebe Azime, Tadesse Destaw Belay, Mesay Gemeda Yigezu, Moges Ahmed Mehamed, Abinew Ali Ayele, Ebrahim Chekol Jibril, Michael Melese Woldeyohannis, Olga Kolesnikova, Philipp Slusallek, Dietrich Klakow, Shengwu Xiong, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13737
• Pdf URL: https://arxiv.org/pdf/2403.13737
• Copy Paste: [[2403.13737]] EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation(https://arxiv.org/abs/2403.13737)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have gained popularity recently due to their outstanding performance in various downstream Natural Language Processing (NLP) tasks. However, low-resource languages are still lagging behind current state-of-the-art (SOTA) developments in the field of NLP due to insufficient resources to train LLMs. Ethiopian languages exhibit remarkable linguistic diversity, encompassing a wide array of scripts, and are imbued with profound religious and cultural significance. This paper introduces EthioLLM -- multilingual large language models for five Ethiopian languages (Amharic, Ge'ez, Afan Oromo, Somali, and Tigrinya) and English, and Ethiobenchmark -- a new benchmark dataset for various downstream NLP tasks. We evaluate the performance of these models across five downstream NLP tasks. We open-source our multilingual language models, new benchmark datasets for various downstream tasks, and task-specific fine-tuned language models and discuss the performance of the models. Our dataset and models are available at the https://huggingface.co/EthioNLP repository.
• 摘要：大型语言模型（LLM）最近因其在各种下游自然语言处理（NLP）任务中的出色表现而受到欢迎。然而，由于缺乏足够的资源来培训法学硕士，低资源语言仍然落后于 NLP 领域当前最先进（SOTA）的发展。埃塞俄比亚语言展现出显着的语言多样性，涵盖多种文字，并充满深刻的宗教和文化意义。本文介绍了 EthioLLM——五种埃塞俄比亚语言（阿姆哈拉语、Ge'ez、Afan Oromo、索马里语和提格里尼亚语）和英语的多语言大语言模型，以及 Ethiobenchmark——用于各种下游 NLP 任务的新基准数据集。我们评估这些模型在五个下游 NLP 任务中的性能。我们开源了多语言语言模型、各种下游任务的新基准数据集以及特定于任务的微调语言模型，并讨论了模型的性能。我们的数据集和模型可在 https://huggingface.co/EthioNLP 存储库中获取。

Title: Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement

• Authors: Catherine Arnett, Pamela D. Rivière, Tyler A. Chang, Sean Trott
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13754
• Pdf URL: https://arxiv.org/pdf/2403.13754
• Copy Paste: [[2403.13754]] Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement(https://arxiv.org/abs/2403.13754)
• Keywords: language model
• Abstract: The relationship between language model tokenization and performance is an open area of research. Here, we investigate how different tokenization schemes impact number agreement in Spanish plurals. We find that morphologically-aligned tokenization performs similarly to other tokenization schemes, even when induced artificially for words that would not be tokenized that way during training. We then present exploratory analyses demonstrating that language model embeddings for different plural tokenizations have similar distributions along the embedding space axis that maximally distinguishes singular and plural nouns. Our results suggest that morphologically-aligned tokenization is a viable tokenization approach, and existing models already generalize some morphological patterns to new items. However, our results indicate that morphological tokenization is not strictly required for performance.
• 摘要：语言模型标记化与性能之间的关系是一个开放的研究领域。在这里，我们研究不同的标记化方案如何影响西班牙语复数中的数字一致性。我们发现形态对齐标记化的表现与其他标记化方案类似，即使是在训练期间人为诱导不会以这种方式标记化的单词时也是如此。然后，我们提出探索性分析，证明不同复数标记化的语言模型嵌入沿着嵌入空间轴具有相似的分布，最大限度地区分单数和复数名词。我们的结果表明，形态对齐的标记化是一种可行的标记化方法，并且现有模型已经将一些形态模式推广到新项目。然而，我们的结果表明形态标记化并不是性能的严格要求。

Title: Information-Theoretic Distillation for Reference-less Summarization

• Authors: Jaehun Jung, Ximing Lu, Liwei Jiang, Faeze Brahman, Peter West, Pang Wei Koh, Yejin Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13780
• Pdf URL: https://arxiv.org/pdf/2403.13780
• Copy Paste: [[2403.13780]] Information-Theoretic Distillation for Reference-less Summarization(https://arxiv.org/abs/2403.13780)
• Keywords: language model, gpt, llm, chat
• Abstract: The current winning recipe for automatic summarization is using proprietary large-scale language models (LLMs) such as ChatGPT as is, or imitation learning from them as teacher models. While increasingly ubiquitous dependence on such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek an alternative learning method -- that allows for a more cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization, without relying on either the LLM's capability or human-written references. To achieve this, we first propose a novel formulation of the desiderata of summarization (saliency, faithfulness and brevity) through the lens of mutual information between the original document and the summary. Based on this formulation, we start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization, then self-train the model to optimize for the information-centric measures of ideal summaries. Distilling from the improved teacher, we arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive analysis demonstrates that our approach outperforms in-domain supervised models in human evaluation, let alone state-of-the-art unsupervised methods, and wins over ChatGPT in controllable summarization.
• 摘要：当前自动摘要的制胜秘诀是使用专有的大规模语言模型 (LLM)，例如 ChatGPT，或者将它们作为教师模型进行模仿学习。虽然越来越普遍地依赖这种大规模语言模型很方便，但仍然存在一个重要问题：如果我们要寻求一种替代的学习方法，那么小规模模型是否能够取得有竞争力的结果——这种方法可以实现更具成本效益的学习方法。 ，可控但功能强大的摘要器。我们提出了 InfoSumm，这是一个新颖的框架，可以根据信息论摘要目标提炼出强大的摘要器，而不依赖于法学硕士的能力或人工编写的参考文献。为了实现这一目标，我们首先通过原始文档和摘要之间的相互信息的视角，提出了摘要需求的新颖表述（显着性、忠实性和简洁性）。基于这个公式，我们从 Pythia-2.8B 作为尚不具备摘要能力的教师模型开始，然后自我训练模型以优化理想摘要的以信息为中心的度量。从改进的教师中提炼出来，我们得到了一个紧凑但功能强大的摘要器，只有 568M 参数，其性能可与 ChatGPT 竞争，而无需依赖 ChatGPT 的功能。广泛的分析表明，我们的方法在人类评估方面优于域内监督模型，更不用说最先进的无监督方法，并在可控摘要方面胜过 ChatGPT。

Title: Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts

• Authors: Guangzeng Han, Weisi Liu, Xiaolei Huang, Brian Borsari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.13786
• Pdf URL: https://arxiv.org/pdf/2403.13786
• Copy Paste: [[2403.13786]] Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts(https://arxiv.org/abs/2403.13786)
• Keywords: language model, llm, prompt
• Abstract: Automatic coding patient behaviors is essential to support decision making for psychotherapists during the motivational interviewing (MI), a collaborative communication intervention approach to address psychiatric issues, such as alcohol and drug addiction. While the behavior coding task has rapidly adapted machine learning to predict patient states during the MI sessions, lacking of domain-specific knowledge and overlooking patient-therapist interactions are major challenges in developing and deploying those models in real practice. To encounter those challenges, we introduce the Chain-of-Interaction (CoI) prompting method aiming to contextualize large language models (LLMs) for psychiatric decision support by the dyadic interactions. The CoI prompting approach systematically breaks down the coding task into three key reasoning steps, extract patient engagement, learn therapist question strategies, and integrates dyadic interactions between patients and therapists. This approach enables large language models to leverage the coding scheme, patient state, and domain knowledge for patient behavioral coding. Experiments on real-world datasets can prove the effectiveness and flexibility of our prompting method with multiple state-of-the-art LLMs over existing prompting baselines. We have conducted extensive ablation analysis and demonstrate the critical role of dyadic interactions in applying LLMs for psychotherapy behavior understanding.
• 摘要：自动编码患者行为对于支持心理治疗师在动机访谈 (MI) 中做出决策至关重要，动机访谈是一种协作沟通干预方法，用于解决酒精和毒瘾等精神问题。虽然行为编码任务已经快速适应机器学习来预测 MI 会话期间的患者状态，但缺乏特定领域的知识并忽视患者与治疗师的互动是在实际实践中开发和部署这些模型的主要挑战。为了应对这些挑战，我们引入了交互链（CoI）提示方法，旨在通过二元交互将大语言模型（LLM）置于上下文中以支持精神病学决策。 CoI 提示方法系统地将编码任务分解为三个关键推理步骤，提取患者参与，学习治疗师问题策略，并整合患者和治疗师之间的二元互动。这种方法使大型语言模型能够利用编码方案、患者状态和领域知识进行患者行为编码。对现实世界数据集的实验可以证明我们的提示方法的有效性和灵活性，在现有的提示基线上有多个最先进的法学硕士。我们进行了广泛的消融分析，并证明了二元相互作用在应用法学硕士进行心理治疗行为理解中的关键作用。

Title: Reverse Training to Nurse the Reversal Curse

• Authors: Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, Sainbayar Sukhbaatar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13799
• Pdf URL: https://arxiv.org/pdf/2403.13799
• Copy Paste: [[2403.13799]] Reverse Training to Nurse the Reversal Curse(https://arxiv.org/abs/2403.13799)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.
• 摘要：大型语言模型 (LLM) 有一个令人惊讶的失败：当训练“A 有一个特征 B”时，它们不会概括为“B 是 A 的一个特征”，这被称为“逆转诅咒”。即使使用数万亿个代币进行训练，由于齐普夫定律，这个问题仍然会出现 - 因此即使我们在整个互联网上进行训练。这项工作提出了一种替代训练方案，称为反向训练，其中所有单词都使用两次，使可用标记的数量加倍。通过反转训练字符串，同时保留（即不反转）选定的子字符串（例如实体），LLM 可以在正向和反向方向上进行训练。我们表明，数据匹配的反向训练模型在标准任务上提供了比标准模型更优越的性能，而计算匹配的反向训练模型在逆转任务上提供了远为优越的性能，有助于解决逆转诅咒问题。