2024-05-20

Title: AmazUtah_NLP at SemEval-2024 Task 9: A MultiChoice Question Answering System for Commonsense Defying Reasoning

• Authors: Mina Ghashami, Soumya Smruti Mishra
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.10385
• Pdf URL: https://arxiv.org/pdf/2405.10385
• Copy Paste: [[2405.10385]] AmazUtah_NLP at SemEval-2024 Task 9: A MultiChoice Question Answering System for Commonsense Defying Reasoning(https://arxiv.org/abs/2405.10385)
• Keywords: language model
• Abstract: The SemEval 2024 BRAINTEASER task represents a pioneering venture in Natural Language Processing (NLP) by focusing on lateral thinking, a dimension of cognitive reasoning that is often overlooked in traditional linguistic analyses. This challenge comprises of Sentence Puzzle and Word Puzzle subtasks and aims to test language models' capacity for divergent thinking. In this paper, we present our approach to the BRAINTEASER task. We employ a holistic strategy by leveraging cutting-edge pre-trained models in multiple choice architecture, and diversify the training data with Sentence and Word Puzzle datasets. To gain further improvement, we fine-tuned the model with synthetic humor/jokes dataset and the RiddleSense dataset which helped augmenting the model's lateral thinking abilities. Empirical results show that our approach achieve 92.5\% accuracy in Sentence Puzzle subtask and 80.2\% accuracy in Word Puzzle subtask.
• 摘要：SemEval 2024 BRAINTEASER 任务代表了自然语言处理 (NLP) 领域的一项开创性尝试，它专注于横向思维，这是传统语言分析中经常被忽视的认知推理维度。该挑战包括句子谜题和单词谜题子任务，旨在测试语言模型的发散思维能力。在本文中，我们介绍了 BRAINTEASER 任务的方法。我们通过在多项选择架构中利用尖端的预训练模型来采用整体策略，并通过句子和单词谜题数据集使训练数据多样化。为了获得进一步的改进，我们使用合成幽默/笑话数据集和 RiddleSense 数据集对模型进行了微调，这有助于增强模型的横向思维能力。实证结果表明，我们的方法在句子谜题子任务中达到了 92.5% 的准确率，在单词谜题子任务中达到了 80.2% 的准确率。

Title: Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models

• Authors: Shaz Furniturewala, Surgan Jandial, Abhinav Java, Pragyan Banerjee, Simra Shahid, Sumit Bhatia, Kokil Jaidka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10431
• Pdf URL: https://arxiv.org/pdf/2405.10431
• Copy Paste: [[2405.10431]] Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models(https://arxiv.org/abs/2405.10431)
• Keywords: language model, llm, prompt
• Abstract: Existing debiasing techniques are typically training-based or require access to the model's internals and output distributions, so they are inaccessible to end-users looking to adapt LLM outputs for their particular needs. In this study, we examine whether structured prompting techniques can offer opportunities for fair text generation. We evaluate a comprehensive end-user-focused iterative framework of debiasing that applies System 2 thinking processes for prompts to induce logical, reflective, and critical text generation, with single, multi-step, instruction, and role-based variants. By systematically evaluating many LLMs across many datasets and different prompting strategies, we show that the more complex System 2-based Implicative Prompts significantly improve over other techniques demonstrating lower mean bias in the outputs with competitive performance on the downstream tasks. Our work offers research directions for the design and the potential of end-user-focused evaluative frameworks for LLM use.
• 摘要：现有的去偏技术通常是基于训练的，或者需要访问模型的内部结构和输出分布，因此希望根据其特定需求调整 LLM 输出的最终用户无法访问它们。在这项研究中，我们研究了结构化提示技术是否可以为公平的文本生成提供机会。我们评估了一个以最终用户为中心的综合去偏迭代框架，该框架应用系统 2 思维过程进行提示，以诱导逻辑性、反思性和批判性文本生成，具有单步、多步骤、指令和基于角色的变体。通过跨多个数据集和不同的提示策略系统地评估许多法学硕士，我们表明，更复杂的基于 System 2 的隐式提示比其他技术显着改进，表明输出的平均偏差较低，并且在下游任务上具有竞争性的表现。我们的工作为法学硕士使用的以最终用户为中心的评估框架的设计和潜力提供了研究方向。

Title: Retrieving and Refining: A Hybrid Framework with Large Language Models for Rare Disease Identification

• Authors: Jinge Wu, Hang Dong, Zexi Li, Arijit Patra, Honghan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10440
• Pdf URL: https://arxiv.org/pdf/2405.10440
• Copy Paste: [[2405.10440]] Retrieving and Refining: A Hybrid Framework with Large Language Models for Rare Disease Identification(https://arxiv.org/abs/2405.10440)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: The infrequency and heterogeneity of clinical presentations in rare diseases often lead to underdiagnosis and their exclusion from structured datasets. This necessitates the utilization of unstructured text data for comprehensive analysis. However, the manual identification from clinical reports is an arduous and intrinsically subjective task. This study proposes a novel hybrid approach that synergistically combines a traditional dictionary-based natural language processing (NLP) tool with the powerful capabilities of large language models (LLMs) to enhance the identification of rare diseases from unstructured clinical notes. We comprehensively evaluate various prompting strategies on six large language models (LLMs) of varying sizes and domains (general and medical). This evaluation encompasses zero-shot, few-shot, and retrieval-augmented generation (RAG) techniques to enhance the LLMs' ability to reason about and understand contextual information in patient reports. The results demonstrate effectiveness in rare disease identification, highlighting the potential for identifying underdiagnosed patients from clinical notes.
• 摘要：罕见疾病临床表现的频率低和异质性常常导致诊断不足以及被排除在结构化数据集中。这就需要利用非结构化文本数据进行综合分析。然而，从临床报告中进行手动识别是一项艰巨且本质上主观的任务。这项研究提出了一种新颖的混合方法，将传统的基于字典的自然语言处理（NLP）工具与大语言模型（LLM）的强大功能相结合，以增强从非结构化临床记录中识别罕见疾病的能力。我们全面评估了不同规模和领域（普通和医学）的六种大型语言模型（LLM）的各种提示策略。该评估涵盖零样本、少样本和检索增强生成 (RAG) 技术，以增强法学硕士推理和理解患者报告中的上下文信息的能力。结果证明了罕见病识别的有效性，凸显了从临床记录中识别诊断不足的患者的潜力。

Title: Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation

• Authors: Matthew Raffel, Victor Agostinelli, Lizhong Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.10443
• Pdf URL: https://arxiv.org/pdf/2405.10443
• Copy Paste: [[2405.10443]] Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation(https://arxiv.org/abs/2405.10443)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have achieved state-of-the-art performance in various language processing tasks, motivating their adoption in simultaneous translation. Current fine-tuning methods to adapt LLMs for simultaneous translation focus on prompting optimization strategies using either data augmentation or prompt structure modifications. However, these methods suffer from several issues, such as an unnecessarily expanded training set, computational inefficiency from dumping the KV cache, increased prompt sizes, or restriction to a single decision policy. To eliminate these issues, we propose a new paradigm in fine-tuning LLMs for simultaneous translation, called SimulMask. It utilizes a novel attention mask technique that models simultaneous translation during fine-tuning by masking attention connections under a desired decision policy. Applying the proposed SimulMask on a Falcon LLM for the IWSLT 2017 dataset, we have observed a significant translation quality improvement compared to state-of-the-art prompting optimization strategies on three language pairs when averaged across four different latency regimes while reducing the computational cost.
• 摘要：大型语言模型 (LLM) 在各种语言处理任务中都取得了最先进的性能，这推动了它们在同声翻译中的采用。当前使法学硕士适应同声翻译的微调方法侧重于使用数据增强或提示结构修改来提示优化策略。然而，这些方法存在几个问题，例如不必要地扩展训练集、转储 KV 缓存导致的计算效率低下、提示大小增加或对单个决策策略的限制。为了消除这些问题，我们提出了一种新的范例，用于微调同步翻译法学硕士，称为 SimulMask。它利用一种新颖的注意力屏蔽技术，通过在所需的决策策略下屏蔽注意力连接，在微调过程中对同步翻译进行建模。将所提出的 SimulMask 应用到 IWSLT 2017 数据集的 Falcon LLM 上，我们观察到，与三种语言对上最先进的提示优化策略相比，当在四种不同的延迟情况下进行平均时，翻译质量有了显着的提高，同时降低了计算成本。

Title: Rethinking ChatGPT's Success: Usability and Cognitive Behaviors Enabled by Auto-regressive LLMs' Prompting

• Authors: Xinzhe Li, Ming Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10474
• Pdf URL: https://arxiv.org/pdf/2405.10474
• Copy Paste: [[2405.10474]] Rethinking ChatGPT's Success: Usability and Cognitive Behaviors Enabled by Auto-regressive LLMs' Prompting(https://arxiv.org/abs/2405.10474)
• Keywords: language model, gpt, llm, prompt, chat, agent
• Abstract: Over the last decade, a wide range of training and deployment strategies for Large Language Models (LLMs) have emerged. Among these, the prompting paradigms of Auto-regressive LLMs (AR-LLMs) have catalyzed a significant surge in Artificial Intelligence (AI). This paper aims to emphasize the significance of utilizing free-form modalities (forms of input and output) and verbal free-form contexts as user-directed channels (methods for transforming modalities) for downstream deployment. Specifically, we analyze the structure of modalities within both two types of LLMs and six task-specific channels during deployment. From the perspective of users, our analysis introduces and applies the analytical metrics of task customizability, transparency, and complexity to gauge their usability, highlighting the superior nature of AR-LLMs' prompting paradigms. Moreover, we examine the stimulation of diverse cognitive behaviors in LLMs through the adoption of free-form text and verbal contexts, mirroring human linguistic expressions of such behaviors. We then detail four common cognitive behaviors to underscore how AR-LLMs' prompting successfully imitate human-like behaviors using this free-form modality and channel. Lastly, the potential for improving LLM deployment, both as autonomous agents and within multi-agent systems, is identified via cognitive behavior concepts and principles.
• 摘要：在过去的十年中，出现了各种针对大型语言模型 (LLM) 的培训和部署策略。其中，自回归法学硕士（AR-LLM）的推动范式催化了人工智能（AI）的显着增长。本文旨在强调利用自由形式模态（输入和输出的形式）和言语自由形式上下文作为下游部署的用户导向渠道（转换模态的方法）的重要性。具体来说，我们分析了部署期间两种类型的法学硕士和六个特定于任务的渠道内的模式结构。从用户的角度来看，我们的分析引入并应用了任务可定制性、透明度和复杂性的分析指标来衡量其可用性，突出了 AR-LLM 提示范式的优越性。此外，我们通过采用自由格式的文本和言语环境，反映了此类行为的人类语言表达，研究了法学硕士中不同认知行为的刺激。然后，我们详细介绍了四种常见的认知行为，以强调 AR-LLM 的提示如何使用这种自由形式的模式和渠道成功模仿类人行为。最后，通过认知行为概念和原则确定了改进 LLM 部署（无论是作为自主代理还是在多代理系统内）的潜力。

Title: Language Models can Evaluate Themselves via Probability Discrepancy

• Authors: Tingyu Xia, Bowen Yu, Yuan Wu, Yi Chang, Chang Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10516
• Pdf URL: https://arxiv.org/pdf/2405.10516
• Copy Paste: [[2405.10516]] Language Models can Evaluate Themselves via Probability Discrepancy(https://arxiv.org/abs/2405.10516)
• Keywords: language model, gpt, llm
• Abstract: In this paper, we initiate our discussion by demonstrating how Large Language Models (LLMs), when tasked with responding to queries, display a more even probability distribution in their answers if they are more adept, as opposed to their less skilled counterparts. Expanding on this foundational insight, we propose a new self-evaluation method ProbDiff for assessing the efficacy of various LLMs. This approach obviates the necessity for an additional evaluation model or the dependence on external, proprietary models like GPT-4 for judgment. It uniquely utilizes the LLMs being tested to compute the probability discrepancy between the initial response and its revised versions. A higher discrepancy for a given query between two LLMs indicates a relatively weaker capability. Our findings reveal that ProbDiff achieves results on par with those obtained from evaluations based on GPT-4, spanning a range of scenarios that include natural language generation (NLG) tasks such as translation, summarization, and our proposed Xiaohongshu blog writing task, and benchmarks for LLM evaluation like AlignBench, MT-Bench, and AlpacaEval, across LLMs of varying magnitudes.
• 摘要：在本文中，我们通过演示大型语言模型（LLM）在负责响应查询时如何在其更熟练的情况下在其答案中显示出更均匀的概率分布来开始我们的讨论，而不是技能较差的同行。扩展这一基本见解，我们提出了一种新的自我评估方法 ProbDiff，用于评估各种法学硕士的有效性。这种方法消除了额外评估模型的必要性，也消除了对 GPT-4 等外部专有模型进行判断的依赖。它独特地利用正在测试的法学硕士来计算初始响应与其修订版本之间的概率差异。两个法学硕士之间给定查询的差异越大，表明能力相对较弱。我们的研究结果表明，ProbDiff 取得的结果与基于 GPT-4 的评估获得的结果相当，涵盖了一系列场景，包括自然语言生成 (NLG) 任务，如翻译、摘要和我们提出的小红书博客写作任务以及基准测试用于 LLM 评估，例如 AlignBench、MT-Bench 和 AlpacaEval，涵盖不同程度的 LLM。

Title: Smart Expert System: Large Language Models as Text Classifiers

• Authors: Zhiqiang Wang, Yiran Pang, Yanbin Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10523
• Pdf URL: https://arxiv.org/pdf/2405.10523
• Copy Paste: [[2405.10523]] Smart Expert System: Large Language Models as Text Classifiers(https://arxiv.org/abs/2405.10523)
• Keywords: language model, llm
• Abstract: Text classification is a fundamental task in Natural Language Processing (NLP), and the advent of Large Language Models (LLMs) has revolutionized the field. This paper introduces the Smart Expert System, a novel approach that leverages LLMs as text classifiers. The system simplifies the traditional text classification workflow, eliminating the need for extensive preprocessing and domain expertise. The performance of several LLMs, machine learning (ML) algorithms, and neural network (NN) based structures is evaluated on four datasets. Results demonstrate that certain LLMs surpass traditional methods in sentiment analysis, spam SMS detection and multi-label classification. Furthermore, it is shown that the system's performance can be further enhanced through few-shot or fine-tuning strategies, making the fine-tuned model the top performer across all datasets. Source code and datasets are available in this GitHub repository: this https URL.
• 摘要：文本分类是自然语言处理 (NLP) 中的一项基本任务，大型​​语言模型 (LLM) 的出现彻底改变了该领域。本文介绍了智能专家系统，这是一种利用法学硕士作为文本分类器的新颖方法。该系统简化了传统的文本分类工作流程，无需大量预处理和领域专业知识。多个法学硕士、机器学习 (ML) 算法和基于神经网络 (NN) 的结构的性能在四个数据集上进行了评估。结果表明，某些法学硕士在情感分析、垃圾短信检测和多标签分类方面超越了传统方法。此外，研究表明，系统的性能可以通过少量或微调策略进一步增强，从而使微调后的模型在所有数据集中表现最佳。源代码和数据集可在此 GitHub 存储库中找到：此 https URL。

Title: Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset

• Authors: Jie Zhu, Junhui Li, Yalong Wen, Lifan Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset(https://arxiv.org/abs/)
• Keywords: language model, gpt, llm
• Abstract: In light of recent breakthroughs in large language models (LLMs) that have revolutionized natural language processing (NLP), there is an urgent need for new benchmarks to keep pace with the fast development of LLMs. In this paper, we propose CFLUE, the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions. Specifically, CFLUE provides datasets tailored for both knowledge assessment and application assessment. In knowledge assessment, it consists of 38K+ multiple-choice questions with associated solution explanations. These questions serve dual purposes: answer prediction and question reasoning. In application assessment, CFLUE features 16K+ test instances across distinct groups of NLP tasks such as text classification, machine translation, relation extraction, reading comprehension, and text generation. Upon CFLUE, we conduct a thorough evaluation of representative LLMs. The results reveal that only GPT-4 and GPT-4-turbo achieve an accuracy exceeding 60\% in answer prediction for knowledge assessment, suggesting that there is still substantial room for improvement in current LLMs. In application assessment, although GPT-4 and GPT-4-turbo are the top two performers, their considerable advantage over lightweight LLMs is noticeably diminished. The datasets and scripts associated with CFLUE are openly accessible at this https URL.
• 摘要：鉴于最近大型语言模型（LLM）的突破彻底改变了自然语言处理（NLP），迫切需要新的基准来跟上法学硕士的快速发展。在本文中，我们提出了CFLUE，即中文金融语言理解评估基准，旨在评估法学硕士在各个维度的能力。具体来说，CFLUE 提供了为知识评估和应用评估量身定制的数据集。在知识评估中，它由 38K+ 多项选择题和相关解决方案说明组成。这些问题有双重目的：答案预测和问题推理。在应用评估中，CFLUE 在不同的 NLP 任务组中提供了 16K 多个测试实例，例如文本分类、机器翻译、关系提取、阅读理解和文本生成。在 CFLUE 上，我们对代表性的法学硕士进行了彻底的评估。结果表明，只有 GPT-4 和 GPT-4-turbo 在知识评估的答案预测中达到了超过 60% 的准确率，这表明当前的 LLM 仍有很大的改进空间。在申请评估中，虽然 GPT-4 和 GPT-4-turbo 是表现最好的两个，但它们相对于轻量级 LLM 的巨大优势明显减弱。与 CFLUE 相关的数据集和脚本可通过此 https URL 公开访问。

Title: Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks

• Authors: Anwoy Chatterjee, Eshaan Tanwar, Subhabrata Dutta, Tanmoy Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10548
• Pdf URL: https://arxiv.org/pdf/2405.10548
• Copy Paste: [[2405.10548]] Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks(https://arxiv.org/abs/2405.10548)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities. Automated assistants based on LLMs are gaining popularity; however, adapting them to novel tasks is still challenging. While colossal models excel in zero-shot performance, their computational demands limit widespread use, and smaller language models struggle without context. This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks. Drawing inspiration from biological neurons and the mechanistic interpretation of the Transformer architecture, we explore the potential for information sharing across tasks. We design a cross-task prompting setup with three LLMs and show that LLMs achieve significant performance improvements despite no examples from the target task in the context. Cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zero-shot prompting, and performs comparable to standard in-context learning. The effectiveness of generating pseudo-labels for in-task examples is demonstrated, and our analyses reveal a strong correlation between the effect of cross-task examples and model activation similarities in source and target input tokens. This paper offers a first-of-its-kind exploration of LLMs' ability to solve novel tasks based on contextual signals from different task examples.
• 摘要：大型语言模型 (LLM) 以其卓越的上下文学习 (ICL) 功能改变了 NLP。基于法学硕士的自动化助理越来越受欢迎；然而，让它们适应新任务仍然具有挑战性。虽然巨大的模型在零样本性能方面表现出色，但它们的计算需求限制了广泛使用，而较小的语言模型在没有上下文的情况下举步维艰。本文研究了法学硕士是否可以从预定义任务的标记示例推广到新任务。从生物神经元和 Transformer 架构的机械解释中汲取灵感，我们探索了跨任务信息共享的潜力。我们用三个法学硕士设计了一个跨任务提示设置，并表明尽管上下文中没有目标任务的示例，法学硕士仍实现了显着的性能改进。与零样本提示相比，跨任务提示使 LLaMA-2 7B 的性能显着提高了 107%，LLaMA-2 13B 的性能提高了 18.6%，GPT 3.5 的性能提高了 3.2%，并且性能与标准上下文学习相当。证明了为任务内示例生成伪标签的有效性，并且我们的分析揭示了跨任务示例的效果与源和目标输入标记中的模型激活相似性之间存在很强的相关性。本文对法学硕士根据不同任务示例的上下文信号解决新任务的能力进行了首次探索。

Title: A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models

• Authors: Francesca De Luca Fornaciari, Begoña Altuna, Itziar Gonzalez-Dios, Maite Melero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10579
• Pdf URL: https://arxiv.org/pdf/2405.10579
• Copy Paste: [[2405.10579]] A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models(https://arxiv.org/abs/2405.10579)
• Keywords: language model, llm, prompt
• Abstract: In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and an extensive error analysis.
• 摘要：在这项工作中，我们利用大型语言模型 (LLM) 探索惯用语言处理。我们推出了惯用语言测试套件 IdioTS，这是一个新的困难示例数据集，由语言专家专门设计，用于评估法学硕士在句子级别处理比喻语言的能力。我们提出了一种基于习语检测任务的综合评估方法，其中提示法学硕士检测给定英语句子中的惯用表达。我们对结果进行彻底的自动和手动评估，并进行广泛的错误分析。

Title: RDRec: Rationale Distillation for LLM-based Recommendation

• Authors: Xinfeng Wang, Jin Cui, Yoshimi Suzuki, Fumiyo Fukumoto
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2405.10587
• Pdf URL: https://arxiv.org/pdf/2405.10587
• Copy Paste: [[2405.10587]] RDRec: Rationale Distillation for LLM-based Recommendation(https://arxiv.org/abs/2405.10587)
• Keywords: language model, llm, prompt
• Abstract: Large language model (LLM)-based recommender models that bridge users and items through textual prompts for effective semantic reasoning have gained considerable attention. However, few methods consider the underlying rationales behind interactions, such as user preferences and item attributes, limiting the reasoning capability of LLMs for recommendations. This paper proposes a rationale distillation recommender (RDRec), a compact model designed to learn rationales generated by a larger language model (LM). By leveraging rationales from reviews related to users and items, RDRec remarkably specifies their profiles for recommendations. Experiments show that RDRec achieves state-of-the-art (SOTA) performance in both top-N and sequential recommendations. Our source code is released at this https URL.
• 摘要：基于大语言模型（LLM）的推荐模型通过文本提示连接用户和项目以进行有效的语义推理，已经引起了相当大的关注。然而，很少有方法考虑交互背后的基本原理，例如用户偏好和项目属性，限制了法学硕士的推荐推理能力。本文提出了一种基本原理蒸馏推荐器（RDRec），这是一种紧凑的模型，旨在学习由更大的语言模型（LM）生成的基本原理。通过利用与用户和项目相关的评论的基本原理，RDRec 显着地指定了他们的推荐资料。实验表明，RDRec 在 top-N 推荐和顺序推荐方面均实现了最先进 (SOTA) 的性能。我们的源代码在此 https URL 发布。

Title: Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization

• Authors: Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.10616
• Pdf URL: https://arxiv.org/pdf/2405.10616
• Copy Paste: [[2405.10616]] Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization(https://arxiv.org/abs/2405.10616)
• Keywords: language model, llm
• Abstract: In recent years, large language models (LLMs) have driven advances in natural language processing. Still, their growing scale has increased the computational burden, necessitating a balance between efficiency and performance. Low-rank compression, a promising technique, reduces non-essential parameters by decomposing weight matrices into products of two low-rank matrices. Yet, its application in LLMs has not been extensively studied. The key to low-rank compression lies in low-rank factorization and low-rank dimensions allocation. To address the challenges of low-rank compression in LLMs, we conduct empirical research on the low-rank characteristics of large models. We propose a low-rank compression method suitable for LLMs. This approach involves precise estimation of feature distributions through pooled covariance matrices and a Bayesian optimization strategy for allocating low-rank dimensions. Experiments on the LLaMA-2 models demonstrate that our method outperforms existing strong structured pruning and low-rank compression techniques in maintaining model performance at the same compression ratio.
• 摘要：近年来，大型语言模型（LLM）推动了自然语言处理的进步。尽管如此，它们不断增长的规模增加了计算负担，因此需要在效率和性能之间取得平衡。低秩压缩是一种很有前途的技术，它通过将权重矩阵分解为两个低秩矩阵的乘积来减少非必要参数。然而，其在法学硕士中的应用尚未得到广泛研究。低秩压缩的关键在于低秩分解和低秩维度分配。为了解决法学硕士中低秩压缩的挑战，我们对大型模型的低秩特征进行了实证研究。我们提出了一种适合 LLM 的低秩压缩方法。这种方法涉及通过合并协方差矩阵精确估计特征分布以及分配低秩维度的贝叶斯优化策略。 LLaMA-2 模型上的实验表明，我们的方法在相同压缩比下保持模型性能方面优于现有的强结构化剪枝和低秩压缩技术。

Title: Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction

• Authors: Jiayun Pang, Ivan Vulić
• Subjects: cs.CL, cs.AI, cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2405.10625
• Pdf URL: https://arxiv.org/pdf/2405.10625
• Copy Paste: [[2405.10625]] Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction(https://arxiv.org/abs/2405.10625)
• Keywords: language model
• Abstract: Transformer-based encoder-decoder models have demonstrated impressive results in chemical reaction prediction tasks. However, these models typically rely on pretraining using tens of millions of unlabelled molecules, which can be time-consuming and GPU-intensive. One of the central questions we aim to answer in this work is: Can FlanT5 and ByT5, the encode-decoder models pretrained solely on language data, be effectively specialised for organic reaction prediction through task-specific fine-tuning? We conduct a systematic empirical study on several key issues of the process, including tokenisation, the impact of (SMILES-oriented) pretraining, fine-tuning sample efficiency, and decoding algorithms at inference. Our key findings indicate that although being pretrained only on language tasks, FlanT5 and ByT5 provide a solid foundation to fine-tune for reaction prediction, and thus become `chemistry domain compatible' in the process. This suggests that GPU-intensive and expensive pretraining on a large dataset of unlabelled molecules may be useful yet not essential to leverage the power of language models for chemistry. All our models achieve comparable Top-1 and Top-5 accuracy although some variation across different models does exist. Notably, tokenisation and vocabulary trimming slightly affect final performance but can speed up training and inference; The most efficient greedy decoding strategy is very competitive while only marginal gains can be achieved from more sophisticated decoding algorithms. In summary, we evaluate FlanT5 and ByT5 across several dimensions and benchmark their impact on organic reaction prediction, which may guide more effective use of these state-of-the-art language models for chemistry-related tasks in the future.
• 摘要：基于 Transformer 的编码器-解码器模型在化学反应预测任务中表现出了令人印象深刻的结果。然而，这些模型通常依赖于使用数千万个未标记分子进行预训练，这可能非常耗时且需要大量 GPU。我们在这项工作中要回答的核心问题之一是：FlanT5 和 ByT5（仅在语言数据上预训练的编码解码器模型）能否通过特定于任务的微调有效地专门用于有机反应预测？我们对该过程的几个关键问题进行了系统的实证研究，包括标记化、（面向 SMILES 的）预训练的影响、微调样本效率以及推理时的解码算法。我们的主要发现表明，尽管仅在语言任务上进行了预训练，但 FlanT5 和 ByT5 为反应预测的微调提供了坚实的基础，从而在此过程中成为“化学领域兼容”。这表明，对未标记分子的大型数据集进行 GPU 密集型且昂贵的预训练可能有用，但对于利用化学语言模型的力量而言并不是必需的。尽管不同模型之间确实存在一些差异，但我们所有的模型都达到了可比的 Top-1 和 Top-5 精度。值得注意的是，标记化和词汇修剪会对最终性能产生轻微影响，但可以加快训练和推理速度；最有效的贪婪解码策略非常有竞争力，而更复杂的解码算法只能实现边际收益。总之，我们从多个维度评估了 FlanT5 和 ByT5，并对其对有机反应预测的影响进行了基准测试，这可能会指导未来更有效地使用这些最先进的语言模型来执行化学相关任务。

Title: Dynamic data sampler for cross-language transfer learning in large language models

• Authors: Yudong Li, Yuhao Feng, Wen Zhou, Zhe Zhao, Linlin Shen, Cheng Hou, Xianxu Hou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10626
• Pdf URL: https://arxiv.org/pdf/2405.10626
• Copy Paste: [[2405.10626]] Dynamic data sampler for cross-language transfer learning in large language models(https://arxiv.org/abs/2405.10626)
• Keywords: language model, llm, chat
• Abstract: Large Language Models (LLMs) have gained significant attention in the field of natural language processing (NLP) due to their wide range of applications. However, training LLMs for languages other than English poses significant challenges, due to the difficulty in acquiring large-scale corpus and the requisite computing resources. In this paper, we propose ChatFlow, a cross-language transfer-based LLM, to address these challenges and train large Chinese language models in a cost-effective manner. We employ a mix of Chinese, English, and parallel corpus to continuously train the LLaMA2 model, aiming to align cross-language representations and facilitate the knowledge transfer specifically to the Chinese language model. In addition, we use a dynamic data sampler to progressively transition the model from unsupervised pre-training to supervised fine-tuning. Experimental results demonstrate that our approach accelerates model convergence and achieves superior performance. We evaluate ChatFlow on popular Chinese and English benchmarks, the results indicate that it outperforms other Chinese models post-trained on LLaMA-2-7B.
• 摘要：大型语言模型（LLM）因其广泛的应用而在自然语言处理（NLP）领域获得了极大的关注。然而，由于获取大规模语料库和所需的计算资源的困难，培训英语以外语言的法学硕士面临着巨大的挑战。在本文中，我们提出了 ChatFlow，一种基于跨语言迁移的法学硕士，以应对这些挑战并以经济有效的方式训练大型中文模型。我们采用中文、英文和平行语料库的混合来持续训练 LLaMA2 模型，旨在对齐跨语言表示并促进专门针对中文模型的知识迁移。此外，我们使用动态数据采样器逐步将模型从无监督预训练过渡到有监督微调。实验结果表明，我们的方法加速了模型收敛并实现了卓越的性能。我们在流行的中文和英文基准上评估 ChatFlow，结果表明它优于在 LLaMA-2-7B 上训练后的其他中国模型。

Title: Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges

• Authors: Xiaoming Shi, Zeming Liu, Li Du, Yuxuan Wang, Hongru Wang, Yuhang Guo, Tong Ruan, Jie Xu, Shaoting Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10630
• Pdf URL: https://arxiv.org/pdf/2405.10630
• Copy Paste: [[2405.10630]] Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges(https://arxiv.org/abs/2405.10630)
• Keywords: language model
• Abstract: This paper surveys and organizes research works on medical dialog systems, which is an important yet challenging task. Although these systems have been surveyed in the medical community from an application perspective, a systematic review from a rigorous technical perspective has to date remained noticeably absent. As a result, an overview of the categories, methods, and evaluation of medical dialogue systems remain limited and underspecified, hindering the further improvement of this area. To fill this gap, we investigate an initial pool of 325 papers from well-known computer science, and natural language processing conferences and journals, and make an overview. Recently, large language models have shown strong model capacity on downstream tasks, which also reshaped medical dialog systems' foundation. Despite the alluring practical application value, current medical dialogue systems still suffer from problems. To this end, this paper lists the grand challenges of medical dialog systems, especially of large language models.
• 摘要：本文对医学对话系统的研究工作进行了调查和整理，这是一项重要而又具有挑战性的任务。尽管医学界已经从应用角度对这些系统进行了调查，但迄今为止，仍然明显缺乏从严格的技术角度进行的系统审查。因此，对医学对话系统的类别、方法和评估的概述仍然有限且不明确，阻碍了该领域的进一步改进。为了填补这一空白，我们调查了来自知名计算机科学和自然语言处理会议和期刊的 325 篇论文，并进行了概述。最近，大型语言模型在下游任务上表现出了强大的模型能力，这也重塑了医学对话系统的基础。尽管实际应用价值诱人，但当前的医疗对话系统仍然存在问题。为此，本文列出了医疗对话系统，特别是大型语言模型面临的巨大挑战。

Title: Layer-Condensed KV Cache for Efficient Inference of Large Language Models

• Authors: Haoyi Wu, Kewei Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10637
• Pdf URL: https://arxiv.org/pdf/2405.10637
• Copy Paste: [[2405.10637]] Layer-Condensed KV Cache for Efficient Inference of Large Language Models(https://arxiv.org/abs/2405.10637)
• Keywords: language model
• Abstract: Huge memory consumption has been a major bottleneck for deploying high-throughput large language models in real-world applications. In addition to the large number of parameters, the key-value (KV) cache for the attention mechanism in the transformer architecture consumes a significant amount of memory, especially when the number of layers is large for deep language models. In this paper, we propose a novel method that only computes and caches the KVs of a small number of layers, thus significantly saving memory consumption and improving inference throughput. Our experiments on large language models show that our method achieves up to 26$\times$ higher throughput than standard transformers and competitive performance in language modeling and downstream tasks. In addition, our method is orthogonal to existing transformer memory-saving techniques, so it is straightforward to integrate them with our model, achieving further improvement in inference efficiency. Our code is available at this https URL.
• 摘要：巨大的内存消耗一直是在实际应用中部署高吞吐量大型语言模型的主要瓶颈。除了大量的参数之外，Transformer 架构中注意力机制的键值 (KV) 缓存也会消耗大量内存，尤其是当深度语言模型的层数较大时。在本文中，我们提出了一种新方法，该方法仅计算和缓存少数层的 KV，从而显着节省内存消耗并提高推理吞吐量。我们在大型语言模型上的实验表明，我们的方法比标准 Transformer 实现了高达 26$\times$ 的吞吐量，并且在语言建模和下游任务中具有竞争力的性能。此外，我们的方法与现有的 Transformer 内存节省技术正交，因此可以直接将它们与我们的模型集成，从而进一步提高推理效率。我们的代码可在此 https URL 上找到。

Title: SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation

• Authors: Ziyao Xu, Houfeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10650
• Pdf URL: https://arxiv.org/pdf/2405.10650
• Copy Paste: [[2405.10650]] SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation(https://arxiv.org/abs/2405.10650)
• Keywords: language model, llm
• Abstract: Compositional generalization is an important ability of language models and has many different manifestations. For data-to-text generation, previous research on this ability is limited to a single manifestation called Systematicity and lacks consideration of large language models (LLMs), which cannot fully cover practical application scenarios. In this work, we propose SPOR, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation. SPOR includes four aspects of manifestations (Systematicity, Productivity, Order invariance, and Rule learnability) and allows high-quality evaluation without additional manual annotations based on existing datasets. We demonstrate SPOR on two different datasets and evaluate some existing language models including LLMs. We find that the models are deficient in various aspects of the evaluation and need further improvement. Our work shows the necessity for comprehensive research on different manifestations of compositional generalization in data-to-text generation and provides a framework for evaluation.
• 摘要：组合泛化是语言模型的一项重要能力，有多种不同的表现形式。对于数据到文本的生成，以往对该能力的研究仅限于系统性这一单一表现，缺乏对大语言模型（LLM）的考虑，无法完全覆盖实际应用场景。在这项工作中，我们提出了 SPOR，一种用于数据到文本生成中的组合泛化的全面且实用的评估方法。 SPOR包括四个方面的表现（系统性、生产率、顺序不变性和规则可学习性），并且允许基于现有数据集进行高质量评估，而无需额外的手动注释。我们在两个不同的数据集上演示了 SPOR，并评估了一些现有的语言模型，包括法学硕士。我们发现模型在评估的各个方面都存在缺陷，需要进一步改进。我们的工作表明了对数据到文本生成中组合泛化的不同表现进行综合研究的必要性，并提供了评估框架。

Title: Realistic Evaluation of Toxicity in Large Language Models

• Authors: Tinh Son Luong, Thanh-Thien Le, Linh Ngo Van, Thien Huu Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10659
• Pdf URL: https://arxiv.org/pdf/2405.10659
• Copy Paste: [[2405.10659]] Realistic Evaluation of Toxicity in Large Language Models(https://arxiv.org/abs/2405.10659)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have become integral to our professional workflows and daily lives. Nevertheless, these machine companions of ours have a critical flaw: the huge amount of data which endows them with vast and diverse knowledge, also exposes them to the inevitable toxicity and bias. While most LLMs incorporate defense mechanisms to prevent the generation of harmful content, these safeguards can be easily bypassed with minimal prompt engineering. In this paper, we introduce the new Thoroughly Engineered Toxicity (TET) dataset, comprising manually crafted prompts designed to nullify the protective layers of such models. Through extensive evaluations, we demonstrate the pivotal role of TET in providing a rigorous benchmark for evaluation of toxicity awareness in several popular LLMs: it highlights the toxicity in the LLMs that might remain hidden when using normal prompts, thus revealing subtler issues in their behavior.
• 摘要：大语言模型 (LLM) 已成为我们专业工作流程和日常生活中不可或缺的一部分。然而，我们的这些机器伙伴有一个严重的缺陷：大量的数据赋予它们广泛而多样的知识，但也使它们面临不可避免的毒性和偏见。虽然大多数法学硕士都采用防御机制来防止有害内容的生成，但这些保护措施可以通过最少的即时工程轻松绕过。在本文中，我们介绍了新的彻底工程毒性（TET）数据集，其中包含手动制作的提示，旨在使此类模型的保护层无效。通过广泛的评估，我们证明了 TET 在为几个受欢迎的法学硕士的毒性意识评估提供严格基准方面的关键作用：它突出了法学硕士在使用正常提示时可能隐藏的毒性，从而揭示了他们行为中更微妙的问题。

Title: Revolutionizing Process Mining: A Novel Architecture for ChatGPT Integration and Enhanced User Experience through Optimized Prompt Engineering

• Authors: Mehrdad Agha Mohammad Ali Kermani, Hamid Reza Seddighi, Mehrdad Maghsoudi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10689
• Pdf URL: https://arxiv.org/pdf/2405.10689
• Copy Paste: [[2405.10689]] Revolutionizing Process Mining: A Novel Architecture for ChatGPT Integration and Enhanced User Experience through Optimized Prompt Engineering(https://arxiv.org/abs/2405.10689)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: In the rapidly evolving field of business process management, there is a growing need for analytical tools that can transform complex data into actionable insights. This research introduces a novel approach by integrating Large Language Models (LLMs), such as ChatGPT, into process mining tools, making process analytics more accessible to a wider audience. The study aims to investigate how ChatGPT enhances analytical capabilities, improves user experience, increases accessibility, and optimizes the architectural frameworks of process mining tools. The key innovation of this research lies in developing a tailored prompt engineering strategy for each process mining submodule, ensuring that the AI-generated outputs are accurate and relevant to the context. The integration architecture follows an Extract, Transform, Load (ETL) process, which includes various process mining engine modules and utilizes zero-shot and optimized prompt engineering techniques. ChatGPT is connected via APIs and receives structured outputs from the process mining modules, enabling conversational interactions. To validate the effectiveness of this approach, the researchers used data from 17 companies that employ BehfaLab's Process Mining Tool. The results showed significant improvements in user experience, with an expert panel rating 72% of the results as "Good". This research contributes to the advancement of business process analysis methodologies by combining process mining with artificial intelligence. Future research directions include further optimization of prompt engineering, exploration of integration with other AI technologies, and assessment of scalability across various business environments. This study paves the way for continuous innovation at the intersection of process mining and artificial intelligence, promising to revolutionize the way businesses analyze and optimize their processes.
• 摘要：在快速发展的业务流程管理领域，对能够将复杂数据转化为可操作见解的分析工具的需求不断增长。这项研究引入了一种新颖的方法，将 ChatGPT 等大型语言模型 (LLM) 集成到流程挖掘工具中，使流程分析更容易被更广泛的受众使用。该研究旨在调查 ChatGPT 如何增强分析能力、改善用户体验、提高可访问性并优化流程挖掘工具的架构框架。这项研究的关键创新在于为每个流程挖掘子模块制定量身定制的即时工程策略，确保人工智能生成的输出准确且与上下文相关。集成架构遵循提取、转换、加载（ETL）流程，其中包括各种流程挖掘引擎模块，并利用零样本和优化的即时工程技术。 ChatGPT 通过 API 连接并接收来自流程挖掘模块的结构化输出，从而实现对话交互。为了验证这种方法的有效性，研究人员使用了来自 17 家采用 BehfaLab 流程挖掘工具的公司的数据。结果显示用户体验显着改善，专家小组将 72% 的结果评为“良好”。这项研究通过将流程挖掘与人工智能相结合，促进了业务流程分析方法的进步。未来的研究方向包括进一步优化即时工程、探索与其他人工智能技术的集成以及跨各种业务环境的可扩展性评估。这项研究为流程挖掘和人工智能交叉领域的持续创新铺平了道路，有望彻底改变企业分析和优化流程的方式。

Title: Persian Pronoun Resolution: Leveraging Neural Networks and Language Models

• Authors: Hassan Haji Mohammadi, Alireza Talebpour, Ahmad Mahmoudi Aznaveh, Samaneh Yazdani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10714
• Pdf URL: https://arxiv.org/pdf/2405.10714
• Copy Paste: [[2405.10714]] Persian Pronoun Resolution: Leveraging Neural Networks and Language Models(https://arxiv.org/abs/2405.10714)
• Keywords: language model
• Abstract: Coreference resolution, critical for identifying textual entities referencing the same entity, faces challenges in pronoun resolution, particularly identifying pronoun antecedents. Existing methods often treat pronoun resolution as a separate task from mention detection, potentially missing valuable information. This study proposes the first end-to-end neural network system for Persian pronoun resolution, leveraging pre-trained Transformer models like ParsBERT. Our system jointly optimizes both mention detection and antecedent linking, achieving a 3.37 F1 score improvement over the previous state-of-the-art system (which relied on rule-based and statistical methods) on the Mehr corpus. This significant improvement demonstrates the effectiveness of combining neural networks with linguistic models, potentially marking a significant advancement in Persian pronoun resolution and paving the way for further research in this under-explored area.
• 摘要：共指解析对于识别引用同一实体的文本实体至关重要，但在代词解析方面面临挑战，特别是识别代词先行词。现有的方法通常将代词解析视为与提及检测不同的单独任务，可能会丢失有价值的信息。这项研究提出了第一个用于波斯语代词解析的端到端神经网络系统，利用 ParsBERT 等预先训练的 Transformer 模型。我们的系统联合优化了提及检测和先行词链接，与 Mehr 语料库上之前最先进的系统（依赖于基于规则和统计方法）相比，F1 分数提高了 3.37。这一重大改进证明了神经网络与语言模型相结合的有效性，可能标志着波斯语代词解析方面的重大进步，并为这一尚未探索的领域的进一步研究铺平了道路。

Title: INDUS: Effective and Efficient Language Models for Scientific Applications

• Authors: Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kayleen Bugbee, Mike Little, Elizabeth Fancher, Lauren Sanders, Sylvain Costes, Sergi Blanco-Cuaresma, Kelly Lockhart, Thomas Allen, Felix Grazes, Megan Ansdel, Alberto Accomazzi, Yousef El-Kurdi, Davis Wertheimer, Birgit Pfitzmann, Cesar Berrospi Ramis, Michele Dolfi, Rafael Teixeira de Lima, Panos Vegenas, S. Karthik Mukkavilli, Peter Staar, Sanaz Vahidinia, Ryan McGranaghan, Armin Mehrabian, Tsendgar Lee
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2405.10725
• Pdf URL: https://arxiv.org/pdf/2405.10725
• Copy Paste: [[2405.10725]] INDUS: Effective and Efficient Language Models for Scientific Applications(https://arxiv.org/abs/2405.10725)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this pivotal insight, we developed INDUS, a comprehensive suite of LLMs tailored for the Earth science, biology, physics, heliophysics, planetary sciences and astrophysics domains and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address natural language understanding tasks, (2) a contrastive-learning-based general text embedding model trained using a diverse set of datasets drawn from multiple sources to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation techniques to address applications which have latency or resource constraints. We also created three new scientific benchmark datasets namely, CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. Finally, we show that our models outperform both general-purpose encoders (RoBERTa) and existing domain-specific encoders (SciBERT) on these new tasks as well as existing benchmark tasks in the domains of interest.
• 摘要：在通用领域语料库上训练的大型语言模型 (LLM) 在自然语言处理 (NLP) 任务上显示出显着的结果。然而，之前的研究表明，使用以领域为中心的语料库训练的法学硕士在专门任务上表现更好。受这一关键见解的启发，我们开发了 INDUS，这是一套专为地球科学、生物学、物理学、太阳物理学、行星科学和天体物理学领域量身定制的法学硕士综合套件，并使用从不同数据源提取的精选科学语料库进行培训。该模型套件包括：（1）使用特定领域词汇和语料库训练的编码器模型来解决自然语言理解任务，（2）使用来自多个数据集的不同数据集训练的基于对比学习的通用文本嵌入模型(3) 使用知识蒸馏技术创建这些模型的较小版本，以解决具有延迟或资源限制的应用程序。我们还创建了三个新的科学基准数据集，即 CLIMATE-CHANGE-NER（实体识别）、NASA-QA（提取 QA）和 NASA-IR（IR），以加速这些多学科领域的研究。最后，我们表明，我们的模型在这些新任务以及感兴趣领域的现有基准任务上均优于通用编码器（RoBERTa）和现有的特定领域编码器（SciBERT）。

Title: Feature-Adaptive and Data-Scalable In-Context Learning

• Authors: Jiahao Li, Quan Wang, Licheng Zhang, Guoqing Jin, Zhendong Mao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Feature-Adaptive and Data-Scalable In-Context Learning(https://arxiv.org/abs/)
• Keywords: llm
• Abstract: In-context learning (ICL), which promotes inference with several demonstrations, has become a widespread paradigm to stimulate LLM capabilities for downstream tasks. Due to context length constraints, it cannot be further improved in spite of more training data, and general features directly from LLMs in ICL are not adaptive to the specific downstream task. In this paper, we propose a feature-adaptive and data-scalable in-context learning framework (FADS-ICL), which can leverage task-adaptive features to promote inference on the downstream task, with the supervision of beyond-context samples. Specifically, it first extracts general features of beyond-context samples via the LLM with ICL input form one by one, and introduces a task-specific modulator to perform feature refinement and prediction after fitting a specific downstream task. We conduct extensive experiments on FADS-ICL under varying data settings (4$\sim$128 shots) and LLM scale (0.8$\sim$70B) settings. Experimental results show that FADS-ICL consistently outperforms previous state-of-the-art methods by a significant margin under all settings, verifying the effectiveness and superiority of FADS-ICL. For example, under the 1.5B and 32 shots setting, FADS-ICL can achieve \textbf{+14.3} average accuracy from feature adaptation over vanilla ICL on 10 datasets, with \textbf{+6.2} average accuracy over the previous state-of-the-art method, and the performance can further improve with increasing training data. Code and data are publicly available at \url{this https URL}.
• 摘要：情境学习（ICL）通过多次演示促进推理，已成为激发下游任务的法学硕士能力的广泛范例。由于上下文长度的限制，尽管有更多的训练数据，它仍无法进一步改进，并且直接来自 ICL 中的 LLM 的一般特征不能适应特定的下游任务。在本文中，我们提出了一种特征自适应和数据可扩展的上下文学习框架（FADS-ICL），它可以利用任务自适应特征来促进对下游任务的推理，并监督上下文之外的样本。具体来说，它首先通过LLM与ICL输入形式一一提取超上下文样本的一般特征，并引入特定于任务的调制器在拟合特定下游任务后进行特征细化和预测。我们在不同的数据设置（4$\sim$128 次注射）和 LLM 规模（0.8$\sim$70B）设置下对 FADS-ICL 进行了广泛的实验。实验结果表明，FADS-ICL 在所有设置下均明显优于先前最先进的方法，验证了 FADS-ICL 的有效性和优越性。例如，在 1.5B 和 32 个镜头设置下，FADS-ICL 可以在 10 个数据集上通过特征适应实现 \textbf{+14.3} 平均准确度，而普通 ICL 的平均准确度比之前的状态高出 \textbf{+6.2} -最先进的方法，并且随着训练数据的增加，性能可以进一步提高。代码和数据可在 \url{this https URL} 上公开获取。

Title: ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios

• Authors: Markus Bayer, Christian Reuter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10808
• Pdf URL: https://arxiv.org/pdf/2405.10808
• Copy Paste: [[2405.10808]] ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios(https://arxiv.org/abs/2405.10808)
• Keywords: language model, gpt, llm
• Abstract: Active learning is designed to minimize annotation efforts by prioritizing instances that most enhance learning. However, many active learning strategies struggle with a 'cold start' problem, needing substantial initial data to be effective. This limitation often reduces their utility for pre-trained models, which already perform well in few-shot scenarios. To address this, we introduce ActiveLLM, a novel active learning approach that leverages large language models such as GPT-4, Llama 3, and Mistral Large for selecting instances. We demonstrate that ActiveLLM significantly enhances the classification performance of BERT classifiers in few-shot scenarios, outperforming both traditional active learning methods and the few-shot learning method SetFit. Additionally, ActiveLLM can be extended to non-few-shot scenarios, allowing for iterative selections. In this way, ActiveLLM can even help other active learning strategies to overcome their cold start problem. Our results suggest that ActiveLLM offers a promising solution for improving model performance across various learning setups.
• 摘要：主动学习旨在通过优先考虑最能增强学习的实例来最大限度地减少注释工作。然而，许多主动学习策略都面临着“冷启动”问题，需要大量的初始数据才能发挥作用。这种限制通常会降低它们对于预训练模型的实用性，而这些模型在少数场景中已经表现良好。为了解决这个问题，我们引入了 ActiveLLM，这是一种新颖的主动学习方法，它利用 GPT-4、Llama 3 和 Mistral Large 等大型语言模型来选择实例。我们证明 ActiveLLM 显着增强了 BERT 分类器在少样本场景中的分类性能，优于传统的主动学习方法和少样本学习方法 SetFit。此外，ActiveLLM 可以扩展到非少数场景，从而允许迭代选择。这样，ActiveLLM 甚至可以帮助其他主动学习策略克服其冷启动问题。我们的结果表明，ActiveLLM 为提高各种学习设置中的模型性能提供了一种有前景的解决方案。

Title: ECR-Chain: Advancing Generative Language Models to Better Emotion-Cause Reasoners through Reasoning Chains

• Authors: Zhaopei Huang, Jinming Zhao, Qin Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10860
• Pdf URL: https://arxiv.org/pdf/2405.10860
• Copy Paste: [[2405.10860]] ECR-Chain: Advancing Generative Language Models to Better Emotion-Cause Reasoners through Reasoning Chains(https://arxiv.org/abs/2405.10860)
• Keywords: language model, gpt, prompt, chat
• Abstract: Understanding the process of emotion generation is crucial for analyzing the causes behind emotions. Causal Emotion Entailment (CEE), an emotion-understanding task, aims to identify the causal utterances in a conversation that stimulate the emotions expressed in a target utterance. However, current works in CEE mainly focus on modeling semantic and emotional interactions in conversations, neglecting the exploration of the emotion-generation process. This hinders the models from deeply understanding emotions, restricting their ability to produce explainable predictions. In this work, inspired by the emotion generation process of "stimulus-appraisal-emotion" in the cognitive appraisal theory, we introduce a step-by-step reasoning method, Emotion-Cause Reasoning Chain (ECR-Chain), to infer the stimulus from the target emotional expressions in conversations. Specifically, we first introduce the ECR-Chain to ChatGPT via few-shot prompting, which significantly improves its performance on the CEE task. We further propose an automated construction process to utilize ChatGPT in building an ECR-Chain set, which can enhance the reasoning abilities of smaller models through supervised training and assist the Vicuna-7B model in achieving state-of-the-art CEE performance. Moreover, our methods can enable these generative language models to effectively perform emotion-cause reasoning in an explainable manner. Our code, data and more details are at this https URL.
• 摘要：了解情绪产生的过程对于分析情绪背后的原因至关重要。因果情感蕴涵（CEE）是一项情感理解任务，旨在识别对话中刺激目标话语中表达的情绪的因果话语。然而，当前的CEE工作主要集中在对对话中的语义和情感交互进行建模，忽视了对情感生成过程的探索。这阻碍了模型深入理解情绪，限制了它们产生可解释预测的能力。在这项工作中，受认知评价理论中“刺激-评价-情感”的情感生成过程的启发，我们引入了一种逐步推理方法——情感-原因推理链（ECR-Chain）来推断刺激从对话中的目标情绪表达。具体来说，我们首先通过few-shot提示将ECR-Chain引入ChatGPT，这显着提高了其在CEE任务上的性能。我们进一步提出了一种利用 ChatGPT 构建 ECR-Chain 集的自动化构建过程，它可以通过监督训练增强较小模型的推理能力，并协助 Vicuna-7B 模型实现最先进的 CEE 性能。此外，我们的方法可以使这些生成语言模型以可解释的方式有效地执行情感原因推理。我们的代码、数据和更多详细信息位于此 https URL。

Title: Tailoring Vaccine Messaging with Common-Ground Opinions

• Authors: Rickard Stureborg, Sanxing Chen, Ruoyu Xie, Aayushi Patel, Christopher Li, Chloe Qinyu Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2405.10861
• Pdf URL: https://arxiv.org/pdf/2405.10861
• Copy Paste: [[2405.10861]] Tailoring Vaccine Messaging with Common-Ground Opinions(https://arxiv.org/abs/2405.10861)
• Keywords: gpt, llm, chat
• Abstract: One way to personalize chatbot interactions is by establishing common ground with the intended reader. A domain where establishing mutual understanding could be particularly impactful is vaccine concerns and misinformation. Vaccine interventions are forms of messaging which aim to answer concerns expressed about vaccination. Tailoring responses in this domain is difficult, since opinions often have seemingly little ideological overlap. We define the task of tailoring vaccine interventions to a Common-Ground Opinion (CGO). Tailoring responses to a CGO involves meaningfully improving the answer by relating it to an opinion or belief the reader holds. In this paper we introduce TAILOR-CGO, a dataset for evaluating how well responses are tailored to provided CGOs. We benchmark several major LLMs on this task; finding GPT-4-Turbo performs significantly better than others. We also build automatic evaluation metrics, including an efficient and accurate BERT model that outperforms finetuned LLMs, investigate how to successfully tailor vaccine messaging to CGOs, and provide actionable recommendations from this investigation. Code and model weights: this https URL Dataset: this https URL
• 摘要：个性化聊天机器人交互的一种方法是与目标读者建立共同点。建立相互理解可能特别有影响力的一个领域是疫苗问题和错误信息。疫苗干预措施是一种信息传递形式，旨在回答人们对疫苗接种的担忧。在这个领域调整回应是很困难的，因为观点往往看起来在意识形态上几乎没有重叠。我们定义了根据共同意见（CGO）调整疫苗干预措施的任务。定制对 CGO 的回应涉及通过将其与读者持有的观点或信念联系起来来有意义地改进答案。在本文中，我们介绍了 TAILOR-CGO，这是一个用于评估针对所提供的 CGO 定制响应效果的数据集。我们在这项任务上对几个主要的法学硕士进行了基准测试；发现 GPT-4-Turbo 的性能明显优于其他产品。我们还构建自动评估指标，包括优于微调的法学硕士的高效、准确的 BERT 模型，研究如何成功地为 CGO 定制疫苗消息传递，并从这项调查中提供可行的建议。代码和模型权重：此 https URL 数据集：此 https URL

Title: COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domain

• Authors: Dimitrios P. Panagoulias, Persephone Papatheodosiou, Anastasios P. Palamidas, Mattheos Sanoudos, Evridiki Tsoureli-Nikita, Maria Virvou, George A. Tsihrintzis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10893
• Pdf URL: https://arxiv.org/pdf/2405.10893
• Copy Paste: [[2405.10893]] COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domain(https://arxiv.org/abs/2405.10893)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) constitute a breakthrough state-of-the-art Artificial Intelligence (AI) technology which is rapidly evolving and promises to aid in medical diagnosis either by assisting doctors or by simulating a doctor's workflow in more advanced and complex implementations. In this technical paper, we outline Cognitive Network Evaluation Toolkit for Medical Domains (COGNET-MD), which constitutes a novel benchmark for LLM evaluation in the medical domain. Specifically, we propose a scoring-framework with increased difficulty to assess the ability of LLMs in interpreting medical text. The proposed framework is accompanied with a database of Multiple Choice Quizzes (MCQs). To ensure alignment with current medical trends and enhance safety, usefulness, and applicability, these MCQs have been constructed in collaboration with several associated medical experts in various medical domains and are characterized by varying degrees of difficulty. The current (first) version of the database includes the medical domains of Psychiatry, Dentistry, Pulmonology, Dermatology and Endocrinology, but it will be continuously extended and expanded to include additional medical domains.
• 摘要：大型语言模型 (LLM) 构成了一种突破性的最先进的人工智能 (AI) 技术，该技术正在迅速发展，有望通过协助医生或通过在更先进和复杂的实施中模拟医生的工作流程来帮助医疗诊断。在这篇技术论文中，我们概述了医学领域认知网络评估工具包（COGNET-MD），它构成了医学领域法学硕士评估的新基准。具体来说，我们提出了一个增加难度的评分框架来评估法学硕士解释医学文本的能力。所提出的框架附带了一个多项选择测验（MCQ）数据库。为了确保符合当前的医疗趋势并增强安全性、实用性和适用性，这些 MCQ 是与各个医学领域的几位相关医学专家合作构建的，并具有不同程度的难度。该数据库的当前（第一个）版本包括精神病学、牙科、肺病学、皮肤病学和内分泌学的医学领域，但它将不断扩展和扩展以包括其他医学领域。

Title: A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

• Authors: Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10936
• Pdf URL: https://arxiv.org/pdf/2405.10936
• Copy Paste: [[2405.10936]] A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers(https://arxiv.org/abs/2405.10936)
• Keywords: language model, llm
• Abstract: The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing, attracting global attention in both academia and industry. To mitigate potential discrimination and enhance the overall usability and accessibility for diverse language user groups, it is important for the development of language-fair technology. Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient, where a comprehensive survey to summarize recent approaches, developments, limitations, and potential solutions is desirable. To this end, we provide a survey with multiple perspectives on the utilization of LLMs in the multilingual scenario. We first rethink the transitions between previous and current research on pre-trained language models. Then we introduce several perspectives on the multilingualism of LLMs, including training and inference methods, model security, multi-domain with language culture, and usage of datasets. We also discuss the major challenges that arise in these aspects, along with possible solutions. Besides, we highlight future research directions that aim at further enhancing LLMs with multilingualism. The survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.
• 摘要：大型语言模型（LLM）的快速发展展现了自然语言处理领域卓越的多语言能力，吸引了全球学术界和工业界的关注。为了减少潜在的歧视并提高不同语言用户群体的整体可用性和可访问性，语言公平技术的发展非常重要。尽管法学硕士取得了突破，但对多语言场景的调查仍然不够，需要进行全面的调查来总结最新的方法、发展、局限性和潜在的解决方案。为此，我们对法学硕士在多语言环境中的利用进行了多角度的调查。我们首先重新思考先前和当前预训练语言模型研究之间的转变。然后我们介绍了法学硕士多语言的几个观点，包括训练和推理方法、模型安全性、语言文化的多领域以及数据集的使用。我们还讨论了这些方面出现的主要挑战以及可能的解决方案。此外，我们还强调了未来的研究方向，旨在进一步提高法学硕士的多语言能力。该调查旨在帮助研究界解决多语言问题，全面了解基于法学硕士的多语言自然语言处理的核心概念、关键技术和最新进展。