2024-03-26

Title: X-AMR Annotation Tool

• Authors: Shafiuddin Rehan Ahmed, Jon Z. Cai, Martha Palmer, James H. Martin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15407
• Pdf URL: https://arxiv.org/pdf/2403.15407
• Copy Paste: [[2403.15407]] X-AMR Annotation Tool(https://arxiv.org/abs/2403.15407)
• Keywords: gpt
• Abstract: This paper presents a novel Cross-document Abstract Meaning Representation (X-AMR) annotation tool designed for annotating key corpus-level event semantics. Leveraging machine assistance through the Prodigy Annotation Tool, we enhance the user experience, ensuring ease and efficiency in the annotation process. Through empirical analyses, we demonstrate the effectiveness of our tool in augmenting an existing event corpus, highlighting its advantages when integrated with GPT-4. Code and annotations: https://github.com/ahmeshaf/gpt_coref
• 摘要：本文提出了一种新颖的跨文档抽象含义表示（X-AMR）注释工具，旨在注释关键的语料库级事件语义。通过 Prodigy Annotation Tool 来利用机器辅助，我们增强了用户体验，确保注释过程的轻松和高效。通过实证分析，我们证明了我们的工具在增强现有事件语料库方面的有效性，突出了其与 GPT-4 集成时的优势。代码和注释：https://github.com/ahmeshaf/gpt_coref

Title: Distilling Named Entity Recognition Models for Endangered Species from Large Language Models

• Authors: Jesse Atuhurra, Seiveright Cargill Dujohn, Hidetaka Kamigaito, Hiroyuki Shindo, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15430
• Pdf URL: https://arxiv.org/pdf/2403.15430
• Copy Paste: [[2403.15430]] Distilling Named Entity Recognition Models for Endangered Species from Large Language Models(https://arxiv.org/abs/2403.15430)
• Keywords: language model, gpt, llm
• Abstract: Natural language processing (NLP) practitioners are leveraging large language models (LLM) to create structured datasets from semi-structured and unstructured data sources such as patents, papers, and theses, without having domain-specific knowledge. At the same time, ecological experts are searching for a variety of means to preserve biodiversity. To contribute to these efforts, we focused on endangered species and through in-context learning, we distilled knowledge from GPT-4. In effect, we created datasets for both named entity recognition (NER) and relation extraction (RE) via a two-stage process: 1) we generated synthetic data from GPT-4 of four classes of endangered species, 2) humans verified the factual accuracy of the synthetic data, resulting in gold data. Eventually, our novel dataset contains a total of 3.6K sentences, evenly divided between 1.8K NER and 1.8K RE sentences. The constructed dataset was then used to fine-tune both general BERT and domain-specific BERT variants, completing the knowledge distillation process from GPT-4 to BERT, because GPT-4 is resource intensive. Experiments show that our knowledge transfer approach is effective at creating a NER model suitable for detecting endangered species from texts.
• 摘要：自然语言处理 (NLP) 从业者正在利用大型语言模型 (LLM) 从半结构化和非结构化数据源（例如专利、论文和论文）创建结构化数据集，而无需具备特定领域的知识。与此同时，生态专家正在寻找各种方法来保护生物多样性。为了为这些努力做出贡献，我们重点关注濒危物种，并通过情境学习从 GPT-4 中提取知识。实际上，我们通过两阶段过程创建了命名实体识别 (NER) 和关系提取 (RE) 的数据集：1) 我们从四类濒危物种的 GPT-4 中生成了合成数据，2) 人类验证了事实合成数据的准确性，产生黄金数据。最终，我们的新数据集总共包含 3.6K 个句子，均匀分布在 1.8K NER 和 1.8K RE 句子之间。然后使用构建的数据集来微调通用 BERT 和特定领域的 BERT 变体，完成从 GPT-4 到 BERT 的知识蒸馏过程，因为 GPT-4 是资源密集型的。实验表明，我们的知识转移方法可以有效地创建适合从文本中检测濒危物种的 NER 模型。

Title: ChatPattern: Layout Pattern Customization via Natural Language

• Authors: Zixiao Wang, Yunheng Shen, Xufeng Yao, Wenqian Zhao, Yang Bai, Farzan Farnia, Bei Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15434
• Pdf URL: https://arxiv.org/pdf/2403.15434
• Copy Paste: [[2403.15434]] ChatPattern: Layout Pattern Customization via Natural Language(https://arxiv.org/abs/2403.15434)
• Keywords: llm, chat, agent
• Abstract: Existing works focus on fixed-size layout pattern generation, while the more practical free-size pattern generation receives limited attention. In this paper, we propose ChatPattern, a novel Large-Language-Model (LLM) powered framework for flexible pattern customization. ChatPattern utilizes a two-part system featuring an expert LLM agent and a highly controllable layout pattern generator. The LLM agent can interpret natural language requirements and operate design tools to meet specified needs, while the generator excels in conditional layout generation, pattern modification, and memory-friendly patterns extension. Experiments on challenging pattern generation setting shows the ability of ChatPattern to synthesize high-quality large-scale patterns.
• 摘要：现有的工作重点是固定尺寸的布局图案生成，而更实用的自由尺寸图案生成受到的关注有限。在本文中，我们提出了 ChatPattern，这是一种新颖的大语言模型 (LLM) 支持的框架，用于灵活的模式定制。 ChatPattern 采用由两部分组成的系统，其中包括专业的 LLM 代理和高度可控的布局模式生成器。 LLM 代理可以解释自然语言要求并操作设计工具来满足指定需求，而生成器则擅长条件布局生成、模式修改和内存友好型模式扩展。在具有挑战性的模式生成设置上的实验表明了 ChatPattern 合成高质量大规模模式的能力。

Title: Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression

• Authors: Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, Bo Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15447
• Pdf URL: https://arxiv.org/pdf/2403.15447
• Copy Paste: [[2403.15447]] Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression(https://arxiv.org/abs/2403.15447)
• Keywords: language model, llm
• Abstract: Compressing high-capability Large Language Models (LLMs) has emerged as a favored strategy for resource-efficient inferences. While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected. This study conducts the first, thorough evaluation of three (3) leading LLMs using five (5) SoTA compression techniques across eight (8) trustworthiness dimensions. Our experiments highlight the intricate interplay between compression and trustworthiness, revealing some interesting patterns. We find that quantization is currently a more effective approach than pruning in achieving efficiency and trustworthiness simultaneously. For instance, a 4-bit quantized model retains the trustworthiness of its original counterpart, but model pruning significantly degrades trustworthiness, even at 50% sparsity. Moreover, employing quantization within a moderate bit range could unexpectedly improve certain trustworthiness dimensions such as ethics and fairness. Conversely, extreme quantization to very low bit levels (3 bits) tends to significantly reduce trustworthiness. This increased risk cannot be uncovered by looking at benign performance alone, in turn, mandating comprehensive trustworthiness evaluation in practice. These findings culminate in practical recommendations for simultaneously achieving high utility, efficiency, and trustworthiness in LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
• 摘要：压缩高性能大型语言模型 (LLM) 已成为资源高效推理的首选策略。虽然最先进的 (SoTA) 压缩方法在保持良性任务性能方面取得了令人印象深刻的进步，但压缩在安全性和可信度方面的潜在风险在很大程度上被忽视了。本研究使用五 (5) 种 SoTA 压缩技术，涵盖八 (8) 个可信度维度，对三 (3) 名领先的法学硕士进行了首次全面评估。我们的实验强调了压缩和可信度之间复杂的相互作用，揭示了一些有趣的模式。我们发现，在同时实现效率和可信度方面，量化目前是比剪枝更有效的方法。例如，4 位量化模型保留了其原始对应模型的可信度，但模型剪枝会显着降低可信度，即使稀疏度为 50% 也是如此。此外，在适度的比特范围内采用量化可以出乎意料地提高某些可信度维度，例如道德和公平性。相反，极端量化到非常低的位级别（3 位）往往会显着降低可信度。这种增加的风险不能仅通过观察良性表现来发现，进而要求在实践中进行全面的可信度评估。这些发现最终为法学硕士同时实现高实用性、效率和可信度提出了实用建议。模型和代码可在 https://decoding-comp-trust.github.io/ 获取。

Title: Loops On Retrieval Augmented Generation (LoRAG)

• Authors: Ayush Thakur, Rashmi Vashisth
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.15450
• Pdf URL: https://arxiv.org/pdf/2403.15450
• Copy Paste: [[2403.15450]] Loops On Retrieval Augmented Generation (LoRAG)(https://arxiv.org/abs/2403.15450)
• Keywords: retrieval augmented generation
• Abstract: This paper presents Loops On Retrieval Augmented Generation (LoRAG), a new framework designed to enhance the quality of retrieval-augmented text generation through the incorporation of an iterative loop mechanism. The architecture integrates a generative model, a retrieval mechanism, and a dynamic loop module, allowing for iterative refinement of the generated text through interactions with relevant information retrieved from the input context. Experimental evaluations on benchmark datasets demonstrate that LoRAG surpasses existing state-of-the-art models in terms of BLEU score, ROUGE score, and perplexity, showcasing its effectiveness in achieving both coherence and relevance in generated text. The qualitative assessment further illustrates LoRAG's capability to produce contextually rich and coherent outputs. This research contributes valuable insights into the potential of iterative loops in mitigating challenges in text generation, positioning LoRAG as a promising advancement in the field.
• 摘要：本文提出了检索增强生成循环（LoRAG），这是一个新框架，旨在通过合并迭代循环机制来提高检索增强文本生成的质量。该架构集成了生成模型、检索机制和动态循环模块，允许通过与从输入上下文检索的相关信息交互来迭代细化生成的文本。对基准数据集的实验评估表明，LoRAG 在 BLEU 分数、ROUGE 分数和困惑度方面超越了现有的最先进模型，展示了其在实现生成文本的连贯性和相关性方面的有效性。定性评估进一步说明了 LoRAG 产生上下文丰富且连贯的输出的能力。这项研究为迭代循环在缓解文本生成挑战方面的潜力提供了宝贵的见解，将 LoRAG 定位为该领域的一项有前途的进步。

Title: Towards Enabling FAIR Dataspaces Using Large Language Models

• Authors: Benedikt T. Arnold, Johannes Theissen-Lipp, Diego Collarana, Christoph Lange, Sandra Geisler, Edward Curry, Stefan Decker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15451
• Pdf URL: https://arxiv.org/pdf/2403.15451
• Copy Paste: [[2403.15451]] Towards Enabling FAIR Dataspaces Using Large Language Models(https://arxiv.org/abs/2403.15451)
• Keywords: language model, llm
• Abstract: Dataspaces have recently gained adoption across various sectors, including traditionally less digitized domains such as culture. Leveraging Semantic Web technologies helps to make dataspaces FAIR, but their complexity poses a significant challenge to the adoption of dataspaces and increases their cost. The advent of Large Language Models (LLMs) raises the question of how these models can support the adoption of FAIR dataspaces. In this work, we demonstrate the potential of LLMs in dataspaces with a concrete example. We also derive a research agenda for exploring this emerging field.
• 摘要：数据空间最近在各个领域得到了采用，包括文化等传统上数字化程度较低的领域。利用语义 Web 技术有助于使数据空间变得公平，但其复杂性对数据空间的采用提出了重大挑战，并增加了其成本。大型语言模型 (LLM) 的出现提出了这些模型如何支持 FAIR 数据空间的采用的问题。在这项工作中，我们通过一个具体的例子展示了法学硕士在数据空间中的潜力。我们还制定了探索这一新兴领域的研究议程。

Title: What Are Tools Anyway? A Survey from the Language Model Perspective

• Authors: Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, Graham Neubig
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15452
• Pdf URL: https://arxiv.org/pdf/2403.15452
• Copy Paste: [[2403.15452]] What Are Tools Anyway? A Survey from the Language Model Perspective(https://arxiv.org/abs/2403.15452)
• Keywords: language model
• Abstract: Language models (LMs) are powerful yet mostly for text generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills. However, many works adopt the term "tool" in different ways, raising the question: What is a tool anyway? Subsequently, where and how do tools help LMs? In this survey, we provide a unified definition of tools as external programs used by LMs, and perform a systematic review of LM tooling scenarios and approaches. Grounded on this review, we empirically study the efficiency of various tooling methods by measuring their required compute and performance gains on various benchmarks, and highlight some challenges and potential future research in the field.
• 摘要：语言模型 (LM) 功能强大，但主要用于文本生成任务。工具大大提高了需要复杂技能的任务的性能。然而，许多作品以不同的方式采用“工具”一词，提出了一个问题：到底什么是工具？接下来，工具在哪里以及如何帮助语言模型？在本次调查中，我们对语言模型使用的外部程序工具进行了统一定义，并对语言模型工具场景和方法进行了系统回顾。基于本次审查，我们通过在各种基准上测量所需的计算和性能增益来实证研究各种工具方法的效率，并强调该领域的一些挑战和潜在的未来研究。

Title: Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams

• Authors: Cristiano Mesquita Garcia, Alessandro Lameiras Koerich, Alceu de Souza Britto Jr, Jean Paul Barddal
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15455
• Pdf URL: https://arxiv.org/pdf/2403.15455
• Copy Paste: [[2403.15455]] Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams(https://arxiv.org/abs/2403.15455)
• Keywords: language model
• Abstract: The proliferation of textual data on the Internet presents a unique opportunity for institutions and companies to monitor public opinion about their services and products. Given the rapid generation of such data, the text stream mining setting, which handles sequentially arriving, potentially infinite text streams, is often more suitable than traditional batch learning. While pre-trained language models are commonly employed for their high-quality text vectorization capabilities in streaming contexts, they face challenges adapting to concept drift - the phenomenon where the data distribution changes over time, adversely affecting model performance. Addressing the issue of concept drift, this study explores the efficacy of seven text sampling methods designed to selectively fine-tune language models, thereby mitigating performance degradation. We precisely assess the impact of these methods on fine-tuning the SBERT model using four different loss functions. Our evaluation, focused on Macro F1-score and elapsed time, employs two text stream datasets and an incremental SVM classifier to benchmark performance. Our findings indicate that Softmax loss and Batch All Triplets loss are particularly effective for text stream classification, demonstrating that larger sample sizes generally correlate with improved macro F1-scores. Notably, our proposed WordPieceToken ratio sampling method significantly enhances performance with the identified loss functions, surpassing baseline results.
• 摘要：互联网上文​​本数据的激增为机构和公司监控公众对其服务和产品的舆论提供了独特的机会。鉴于此类数据的快速生成，处理顺序到达的、可能无限的文本流的文本流挖掘设置通常比传统的批量学习更合适。虽然预训练的语言模型因其在流环境中的高质量文本矢量化功能而被广泛采用，但它们面临着适应概念漂移的挑战，即数据分布随时间变化的现象，从而对模型性能产生不利影响。为了解决概念漂移问题，本研究探讨了七种文本采样方法的功效，这些方法旨在选择性地微调语言模型，从而减轻性能下降。我们使用四种不同的损失函数精确评估这些方法对微调 SBERT 模型的影响。我们的评估重点关注宏观 F1 分数和经过时间，采用两个文本流数据集和一个增量 SVM 分类器来衡量性能基准。我们的研究结果表明，Softmax 损失和 Batch All Triplets 损失对于文本流分类特别有效，这表明较大的样本量通常与改进的宏观 F1 分数相关。值得注意的是，我们提出的 WordPieceToken 比率采样方法显着提高了已识别损失函数的性能，超过了基线结果。

Title: Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks

• Authors: Daniel Fesalbon, Arvin De La Cruz, Marvin Mallari, Nelson Rodelas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15458
• Pdf URL: https://arxiv.org/pdf/2403.15458
• Copy Paste: [[2403.15458]] Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks(https://arxiv.org/abs/2403.15458)
• Keywords: language model, gpt, chat
• Abstract: Common problems in playing online mobile and computer games were related to toxic behavior and abusive communication among players. Based on different reports and studies, the study also discusses the impact of online hate speech and toxicity on players' in-game performance and overall well-being. This study investigates the capability of pre-trained language models to classify or detect trash talk or toxic in-game messages The study employs and evaluates the performance of pre-trained BERT and GPT language models in detecting toxicity within in-game chats. Using publicly available APIs, in-game chat data from DOTA 2 game matches were collected, processed, reviewed, and labeled as non-toxic, mild (toxicity), and toxic. The study was able to collect around two thousand in-game chats to train and test BERT (Base-uncased), BERT (Large-uncased), and GPT-3 models. Based on the three models' state-of-the-art performance, this study concludes pre-trained language models' promising potential for addressing online hate speech and in-game insulting trash talk.
• 摘要：玩在线手机和电脑游戏的常见问题与玩家之间的不良行为和辱骂性沟通有关。基于不同的报告和研究，该研究还讨论了在线仇恨言论和毒性对玩家在游戏中的表现和整体福祉的影响。本研究调查了预训练语言模型分类或检测垃圾话或有毒游戏消息的能力。该研究采用并评估了预训练 BERT 和 GPT 语言模型在检测游戏聊天中的毒性方面的性能。使用公开可用的 API，收集、处理、审查 DOTA 2 游戏比赛中的游戏内聊天数据，并将其标记为无毒、轻度（毒性）和有毒。该研究收集了大约 2000 条游戏内聊天记录来训练和测试 BERT（Base-uncased）、BERT（Large-uncased）和 GPT-3 模型。基于这三个模型的最先进性能，本研究得出结论，预训练语言模型在解决在线仇恨言论和游戏中侮辱性垃圾话方面具有巨大的潜力。

Title: LLMs-based Few-Shot Disease Predictions using EHR: A Novel Approach Combining Predictive Agent Reasoning and Critical Agent Instruction

• Authors: Hejie Cui, Zhuocheng Shen, Jieyu Zhang, Hui Shao, Lianhui Qin, Joyce C. Ho, Carl Yang
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.15464
• Pdf URL: https://arxiv.org/pdf/2403.15464
• Copy Paste: [[2403.15464]] LLMs-based Few-Shot Disease Predictions using EHR: A Novel Approach Combining Predictive Agent Reasoning and Critical Agent Instruction(https://arxiv.org/abs/2403.15464)
• Keywords: language model, llm, prompt, agent
• Abstract: Electronic health records (EHRs) contain valuable patient data for health-related prediction tasks, such as disease prediction. Traditional approaches rely on supervised learning methods that require large labeled datasets, which can be expensive and challenging to obtain. In this study, we investigate the feasibility of applying Large Language Models (LLMs) to convert structured patient visit data (e.g., diagnoses, labs, prescriptions) into natural language narratives. We evaluate the zero-shot and few-shot performance of LLMs using various EHR-prediction-oriented prompting strategies. Furthermore, we propose a novel approach that utilizes LLM agents with different roles: a predictor agent that makes predictions and generates reasoning processes and a critic agent that analyzes incorrect predictions and provides guidance for improving the reasoning of the predictor agent. Our results demonstrate that with the proposed approach, LLMs can achieve decent few-shot performance compared to traditional supervised learning methods in EHR-based disease predictions, suggesting its potential for health-oriented applications.
• 摘要：电子健康记录 (EHR) 包含用于健康相关预测任务（例如疾病预测）的宝贵患者数据。传统方法依赖于需要大量标记数据集的监督学习方法，而这些数据集的获取成本高昂且具有挑战性。在本研究中，我们研究了应用大型语言模型 (LLM) 将结构化患者就诊数据（例如诊断、实验室、处方）转换为自然语言叙述的可行性。我们使用各种面向 EHR 预测的提示策略来评估法学硕士的零样本和少样本性能。此外，我们提出了一种利用具有不同角色的 LLM 代理的新颖方法：进行预测并生成推理过程的预测代理和分析不正确的预测并为改进预测代理的推理提供指导的批评代理。我们的结果表明，通过所提出的方法，与传统的监督学习方法相比，在基于 EHR 的疾病预测中，法学硕士可以实现不错的几次表现，这表明其在以健康为导向的应用中的潜力。

Title: Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training

• Authors: James Vo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15470
• Pdf URL: https://arxiv.org/pdf/2403.15470
• Copy Paste: [[2403.15470]] Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training(https://arxiv.org/abs/2403.15470)
• Keywords: language model, llm
• Abstract: The advancement of Large Language Models (LLMs) has significantly transformed the field of natural language processing, although the focus on English-centric models has created a noticeable research gap for specific languages, including Vietnamese. To address this issue, this paper presents vi-mistral-x, an innovative Large Language Model designed expressly for the Vietnamese language. It utilizes a unique method of continual pre-training, based on the Mistral architecture, which incorporates grouped-query attention and sliding window attention techniques. This model, vi-Mistral-X, marks a significant step forward in improving the understanding and generation of the Vietnamese language. It introduces an additional phase of continual pre-training, specifically adapted for Vietnamese, enhancing the model's capability in understanding complex language nuances and generating accurate, context-aware Vietnamese text. Through comprehensive testing on various benchmarks, vi-mistral-x has shown to outperform existing Vietnamese LLMs in several key areas, including text classification, question answering, and text generation. Particularly, in the Vietnamese Multitask Language Understanding (VMLU) benchmark, vi-mistral-x sets a new standard, outperforming other available models significantly. This paper highlights the critical role of continual pre-training in advancing language-specific LLMs and opens new avenues for the development of multilingual models. We aim for vi-mistral-x to not just be an important asset for processing the Vietnamese language but also to encourage more advancements in creating large language models for languages that are less represented.
• 摘要：尽管对以英语为中心的模型的关注为包括越南语在内的特定语言造成了明显的研究空白，但大型语言模型（LLM）的进步极大地改变了自然语言处理领域。为了解决这个问题，本文提出了 vi-mistral-x，一种专为越南语设计的创新大型语言模型。它采用基于 Mistral 架构的独特的持续预训练方法，该方法结合了分组查询注意力和滑动窗口注意力技术。这个模型 vi-Mistral-X 标志着在提高越南语的理解和生成方面向前迈出了重要一步。它引入了专门针对越南语的持续预训练的附加阶段，增强了模型理解复杂语言细微差别并生成准确的上下文感知越南语文本的能力。通过对各种基准的全面测试，vi-mistral-x 在几个关键领域（包括文本分类、问题回答和文本生成）表现优于现有的越南法学硕士。特别是，在越南语多任务语言理解 (VMLU) 基准测试中，vi-mistral-x 树立了新标准，显着优于其他可用模型。本文强调了持续预训练在推进特定语言法学硕士方面的关键作用，并为多语言模型的开发开辟了新途径。我们的目标是 vi-mistral-x 不仅成为处理越南语的重要资产，而且鼓励在为较少代表性的语言创建大型语言模型方面取得更多进展。

Title: Efficient argument classification with compact language models and ChatGPT-4 refinements

• Authors: Marcin Pietron, Rafał Olszowski, Jakub Gomułka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15473
• Pdf URL: https://arxiv.org/pdf/2403.15473
• Copy Paste: [[2403.15473]] Efficient argument classification with compact language models and ChatGPT-4 refinements(https://arxiv.org/abs/2403.15473)
• Keywords: language model, gpt, prompt, chat
• Abstract: Argument mining (AM) is defined as the task of automatically identifying and extracting argumentative components (e.g. premises, claims, etc.) and detecting the existing relations among them (i.e., support, attack, no relations). Deep learning models enable us to analyze arguments more efficiently than traditional methods and extract their semantics. This paper presents comparative studies between a few deep learning-based models in argument mining. The work concentrates on argument classification. The research was done on a wide spectrum of datasets (Args.me, UKP, US2016). The main novelty of this paper is the ensemble model which is based on BERT architecture and ChatGPT-4 as fine tuning model. The presented results show that BERT+ChatGPT-4 outperforms the rest of the models including other Transformer-based and LSTM-based models. The observed improvement is, in most cases, greater than 10The presented analysis can provide crucial insights into how the models for argument classification should be further improved. Additionally, it can help develop a prompt-based algorithm to eliminate argument classification errors.
• 摘要：论证挖掘（AM）被定义为自动识别和提取论证成分（例如前提、主张等）并检测它们之间现有关系（即支持、攻击、无关系）的任务。深度学习模型使我们能够比传统方法更有效地分析论点并提取其语义。本文介绍了一些基于深度学习的论点挖掘模型之间的比较研究。这项工作的重点是论证分类。该研究是在广泛的数据集上完成的（Args.me、UKP、US2016）。本文的主要创新点是基于 BERT 架构和 ChatGPT-4 作为微调模型的集成模型。结果表明，BERT+ChatGPT-4 的性能优于其他模型，包括其他基于 Transformer 和基于 LSTM 的模型。在大多数情况下，观察到的改进大于 10。所提出的分析可以为如何进一步改进参数分类模型提供重要见解。此外，它还可以帮助开发基于提示的算法来消除参数分类错误。

Title: Integrating Supervised Extractive and Generative Language Models for Suicide Risk Evidence Summarization

• Authors: Rika Tanaka, Yusuke Fukazawa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15478
• Pdf URL: https://arxiv.org/pdf/2403.15478
• Copy Paste: [[2403.15478]] Integrating Supervised Extractive and Generative Language Models for Suicide Risk Evidence Summarization(https://arxiv.org/abs/2403.15478)
• Keywords: language model
• Abstract: We propose a method that integrates supervised extractive and generative language models for providing supporting evidence of suicide risk in the CLPsych 2024 shared task. Our approach comprises three steps. Initially, we construct a BERT-based model for estimating sentence-level suicide risk and negative sentiment. Next, we precisely identify high suicide risk sentences by emphasizing elevated probabilities of both suicide risk and negative sentiment. Finally, we integrate generative summaries using the MentaLLaMa framework and extractive summaries from identified high suicide risk sentences and a specialized dictionary of suicidal risk words. SophiaADS, our team, achieved 1st place for highlight extraction and ranked 10th for summary generation, both based on recall and consistency metrics, respectively.
• 摘要：我们提出了一种集成有监督的提取和生成语言模型的方法，用于在 CLPsych 2024 共享任务中提供自杀风险的支持证据。我们的方法包括三个步骤。最初，我们构建了一个基于 BERT 的模型来估计句子级别的自杀风险和负面情绪。接下来，我们通过强调自杀风险和负面情绪的概率升高来精确识别高自杀风险句子。最后，我们使用 MentaLLaMa 框架集成生成摘要，并从已识别的高自杀风险句子和自杀风险单词的专门词典中提取摘要。我们的团队 SophiaADS 在亮点提取方面排名第一，在摘要生成方面排名第十，分别基于召回率和一致性指标。

Title: Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors

• Authors: Alicja Chaszczewicz, Raj Sanjay Shah, Ryan Louie, Bruce A Arnow, Robert Kraut, Diyi Yang
• Subjects: cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15482
• Pdf URL: https://arxiv.org/pdf/2403.15482
• Copy Paste: [[2403.15482]] Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors(https://arxiv.org/abs/2403.15482)
• Keywords: language model
• Abstract: Realistic practice and tailored feedback are key processes for training peer counselors with clinical skills. However, existing mechanisms of providing feedback largely rely on human supervision. Peer counselors often lack mechanisms to receive detailed feedback from experienced mentors, making it difficult for them to support the large number of people with mental health issues who use peer counseling. Our work aims to leverage large language models to provide contextualized and multi-level feedback to empower peer counselors, especially novices, at scale. To achieve this, we co-design with a group of senior psychotherapy supervisors to develop a multi-level feedback taxonomy, and then construct a publicly available dataset with comprehensive feedback annotations of 400 emotional support conversations. We further design a self-improvement method on top of large language models to enhance the automatic generation of feedback. Via qualitative and quantitative evaluation with domain experts, we demonstrate that our method minimizes the risk of potentially harmful and low-quality feedback generation which is desirable in such high-stakes scenarios.
• 摘要：现实的实践和量身定制的反馈是培训同伴辅导员临床技能的关键过程。然而，现有的提供反馈机制很大程度上依赖于人类监督。同伴咨询师往往缺乏从经验丰富的导师那里获得详细反馈的机制，这使得他们很难为大量使用同伴咨询的患有心理健康问题的人提供支持。我们的工作旨在利用大型语言模型提供情境化和多层次的反馈，以大规模地增强同伴辅导员（尤其是新手）的能力。为了实现这一目标，我们与一组高级心理治疗主管共同设计开发多级反馈分类法，然后构建一个公开的数据集，其中包含 400 次情感支持对话的全面反馈注释。我们进一步在大型语言模型之上设计了一种自我改进方法，以增强反馈的自动生成。通过与领域专家的定性和定量评估，我们证明我们的方法最大限度地降低了潜在有害和低质量反馈生成的风险，这在这种高风险场景中是可取的。

Title: RakutenAI-7B: Extending Large Language Models for Japanese

• Authors: Rakuten Group Inc., Aaron Levine, Connie Huang, Chenguang Wang, Eduardo Batista, Ewa Szymanska, Hongyi Ding, Hou Wei Chou, Jean-François Pessiot, Johanes Effendi, Justin Chiu, Kai Torben Ohlhus, Karan Chopra, Keiji Shinzato, Koji Murakami, Lee Xiong, Lei Chen, Maki Kubota, Maksim Tkachenko, Miroku Lee, Naoki Takahashi, Prathyusha Jwalapuram, Ryutaro Tatsushima, Saurabh Jain, Sunil Kumar Yadav, Ting Cai, Wei-Te Chen, Yandi Xia, Yuki Nakayama, Yutaka Higashiyama
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15484
• Pdf URL: https://arxiv.org/pdf/2403.15484
• Copy Paste: [[2403.15484]] RakutenAI-7B: Extending Large Language Models for Japanese(https://arxiv.org/abs/2403.15484)
• Keywords: language model, chat
• Abstract: We introduce RakutenAI-7B, a suite of Japanese-oriented large language models that achieve the best performance on the Japanese LM Harness benchmarks among the open 7B models. Along with the foundation model, we release instruction- and chat-tuned models, RakutenAI-7B-instruct and RakutenAI-7B-chat respectively, under the Apache 2.0 license.
• 摘要：我们推出 RakutenAI-7B，这是一套面向日语的大语言模型，在开放的 7B 模型中，它在日语 LM Harness 基准测试中实现了最佳性能。除了基础模型之外，我们还根据 Apache 2.0 许可证分别发布了指令和聊天调整模型 RakutenAI-7B-instruct 和 RakutenAI-7B-chat。

Title: Sequence-to-Sequence Language Models for Character and Emotion Detection in Dream Narratives

• Authors: Gustave Cortal (ENS Paris Saclay, LISN)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15486
• Pdf URL: https://arxiv.org/pdf/2403.15486
• Copy Paste: [[2403.15486]] Sequence-to-Sequence Language Models for Character and Emotion Detection in Dream Narratives(https://arxiv.org/abs/2403.15486)
• Keywords: language model
• Abstract: The study of dreams has been central to understanding human (un)consciousness, cognition, and culture for centuries. Analyzing dreams quantitatively depends on labor-intensive, manual annotation of dream narratives. We automate this process through a natural language sequence-to-sequence generation framework. This paper presents the first study on character and emotion detection in the English portion of the open DreamBank corpus of dream narratives. Our results show that language models can effectively address this complex task. To get insight into prediction performance, we evaluate the impact of model size, prediction order of characters, and the consideration of proper names and character traits. We compare our approach with a large language model using in-context learning. Our supervised models perform better while having 28 times fewer parameters. Our model and its generated annotations are made publicly available.
• 摘要：几个世纪以来，对梦的研究一直是理解人类（无）意识、认知和文化的核心。定量分析梦境依赖于对梦境叙述的劳动密集型手动注释。我们通过自然语言序列到序列生成框架自动化这个过程。本文首次对开放 DreamBank 梦境叙事语料库的英语部分中的人物和情感检测进行了研究。我们的结果表明语言模型可以有效地解决这一复杂的任务。为了深入了解预测性能，我们评估了模型大小、字符预测顺序以及对专有名称和字符特征的考虑的影响。我们将我们的方法与使用上下文学习的大型语言模型进行比较。我们的监督模型性能更好，同时参数减少了 28 倍。我们的模型及其生成的注释是公开的。

Title: Open Source Conversational LLMs do not know most Spanish words

• Authors: Javier Conde, Miguel González, Nina Melero, Raquel Ferrando, Gonzalo Martínez, Elena Merino-Gómez, José Alberto Hernández, Pedro Reviriego
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15491
• Pdf URL: https://arxiv.org/pdf/2403.15491
• Copy Paste: [[2403.15491]] Open Source Conversational LLMs do not know most Spanish words(https://arxiv.org/abs/2403.15491)
• Keywords: language model, llm, chat
• Abstract: The growing interest in Large Language Models (LLMs) and in particular in conversational models with which users can interact has led to the development of a large number of open-source chat LLMs. These models are evaluated on a wide range of benchmarks to assess their capabilities in answering questions or solving problems on almost any possible topic or to test their ability to reason or interpret texts. Instead, the evaluation of the knowledge that these models have of the languages has received much less attention. For example, the words that they can recognize and use in different languages. In this paper, we evaluate the knowledge that open-source chat LLMs have of Spanish words by testing a sample of words in a reference dictionary. The results show that open-source chat LLMs produce incorrect meanings for an important fraction of the words and are not able to use most of the words correctly to write sentences with context. These results show how Spanish is left behind in the open-source LLM race and highlight the need to push for linguistic fairness in conversational LLMs ensuring that they provide similar performance across languages.
• 摘要：人们对大型语言模型 (LLM)，特别是对用户可以交互的对话模型越来越感兴趣，导致了大量开源聊天 LLM 的开发。这些模型在广泛的基准上进行评估，以评估它们回答问题或解决几乎任何可能主题的问题的能力，或测试它们推理或解释文本的能力。相反，对这些模型所拥有的语言知识的评估却很少受到关注。例如，他们可以识别和使用不同语言的单词。在本文中，我们通过测试参考词典中的单词样本来评估开源聊天法学硕士对西班牙语单词的了解。结果表明，开源聊天法学硕士对重要部分的单词产生了错误的含义，并且无法正确使用大多数单词来编写带有上下文的句子。这些结果显示了西班牙语如何在开源法学硕士竞赛中落后，并强调需要推动对话式法学硕士的语言公平性，确保它们在不同语言之间提供相似的表现。

Title: Enhancing Medical Support in the Arabic Language Through Personalized ChatGPT Assistance

• Authors: Mohamed Issa, Ahmed Abdelwahed
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.15501
• Pdf URL: https://arxiv.org/pdf/2403.15501
• Copy Paste: [[2403.15501]] Enhancing Medical Support in the Arabic Language Through Personalized ChatGPT Assistance(https://arxiv.org/abs/2403.15501)
• Keywords: gpt, prompt, chat
• Abstract: This Paper discusses the growing popularity of online medical diagnosis as an alternative to traditional doctor visits. It highlights the limitations of existing tools and emphasizes the advantages of using ChatGPT, which provides real-time, personalized medical diagnosis at no cost. The paragraph summarizes a research study that evaluated the performance of ChatGPT in Arabic medical diagnosis. The study involved compiling a dataset of disease information and generating multiple messages for each disease using different prompting techniques. ChatGPT's performance was assessed by measuring the similarity between its responses and the actual diseases. The results showed promising performance, with average scores of around 76% for similarity measures. Various prompting techniques were used, and chain prompting demonstrated a relative advantage. The study also recorded an average response time of 6.12 seconds for the ChatGPT API, which is considered acceptable but has room for improvement. While ChatGPT cannot replace human doctors entirely, the findings suggest its potential in emergency cases and addressing general medical inquiries. Overall, the study highlights ChatGPT's viability as a valuable tool in the medical field.
• 摘要：本文讨论了在线医疗诊断作为传统医生就诊的替代方案的日益普及。它强调了现有工具的局限性，并强调了使用 ChatGPT 的优势，ChatGPT 可以免费提供实时、个性化的医疗诊断。该段落总结了一项评估 ChatGPT 在阿拉伯语医学诊断中的性能的研究。该研究涉及编制疾病信息数据集，并使用不同的提示技术为每种疾病生成多条消息。 ChatGPT 的性能是通过测量其反应与实际疾病之间的相似性来评估的。结果显示出良好的性能，相似性度量的平均得分约为 76%。使用了多种提示技术，连锁提示显示出相对优势。该研究还记录了 ChatGPT API 的平均响应时间为 6.12 秒，这被认为是可以接受的，但还有改进的空间。虽然 ChatGPT 无法完全取代人类医生，但研究结果表明它在紧急情况和解决一般医疗询问方面具有潜力。总体而言，该研究强调了 ChatGPT 作为医疗领域有价值的工具的可行性。

Title: Sequential Decision-Making for Inline Text Autocomplete

• Authors: Rohan Chitnis, Shentao Yang, Alborz Geramifard
• Subjects: cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15502
• Pdf URL: https://arxiv.org/pdf/2403.15502
• Copy Paste: [[2403.15502]] Sequential Decision-Making for Inline Text Autocomplete(https://arxiv.org/abs/2403.15502)
• Keywords: language model
• Abstract: Autocomplete suggestions are fundamental to modern text entry systems, with applications in domains such as messaging and email composition. Typically, autocomplete suggestions are generated from a language model with a confidence threshold. However, this threshold does not directly take into account the cognitive load imposed on the user by surfacing suggestions, such as the effort to switch contexts from typing to reading the suggestion, and the time to decide whether to accept the suggestion. In this paper, we study the problem of improving inline autocomplete suggestions in text entry systems via a sequential decision-making formulation, and use reinforcement learning to learn suggestion policies through repeated interactions with a target user over time. This formulation allows us to factor cognitive load into the objective of training an autocomplete model, through a reward function based on text entry speed. We acquired theoretical and experimental evidence that, under certain objectives, the sequential decision-making formulation of the autocomplete problem provides a better suggestion policy than myopic single-step reasoning. However, aligning these objectives with real users requires further exploration. In particular, we hypothesize that the objectives under which sequential decision-making can improve autocomplete systems are not tailored solely to text entry speed, but more broadly to metrics such as user satisfaction and convenience.
• 摘要：自动完成建议是现代文本输入系统的基础，在消息传递和电子邮件撰写等领域都有应用。通常，自动完成建议是从具有置信阈值的语言模型生成的。然而，该阈值没有直接考虑通过呈现建议对用户施加的认知负荷，例如将上下文从打字切换到阅读建议的努力，以及决定是否接受建议的时间。在本文中，我们研究了通过顺序决策制定来改进文本输入系统中的内联自动完成建议的问题，并使用强化学习通过随着时间的推移与目标用户的重复交互来学习建议策略。这个公式允许我们通过基于文本输入速度的奖励函数将认知负荷纳入训练自动完成模型的目标中。我们获得的理论和实验证据表明，在某些目标下，自动完成问题的顺序决策制定提供了比短视单步推理更好的建议策略。然而，将这些目标与真实用户保持一致还需要进一步探索。特别是，我们假设顺序决策可以改进自动完成系统的目标不仅仅针对文本输入速度，而是更广泛地针对用户满意度和便利性等指标。

Title: Evaluating the Performance of LLMs on Technical Language Processing tasks

• Authors: Andrew Kernycky, David Coleman, Christopher Spence, Udayan Das
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.15503
• Pdf URL: https://arxiv.org/pdf/2403.15503
• Copy Paste: [[2403.15503]] Evaluating the Performance of LLMs on Technical Language Processing tasks(https://arxiv.org/abs/2403.15503)
• Keywords: llm, chat
• Abstract: In this paper we present the results of an evaluation study of the perfor-mance of LLMs on Technical Language Processing tasks. Humans are often confronted with tasks in which they have to gather information from dispar-ate sources and require making sense of large bodies of text. These tasks can be significantly complex for humans and often require deep study including rereading portions of a text. Towards simplifying the task of gathering in-formation we evaluated LLMs with chat interfaces for their ability to provide answers to standard questions that a human can be expected to answer based on their reading of a body of text. The body of text under study is Title 47 of the United States Code of Federal Regulations (CFR) which describes regula-tions for commercial telecommunications as governed by the Federal Com-munications Commission (FCC). This has been a body of text of interest be-cause our larger research concerns the issue of making sense of information related to Wireless Spectrum Governance and usage in an automated manner to support Dynamic Spectrum Access. The information concerning this wireless spectrum domain is found in many disparate sources, with Title 47 of the CFR being just one of many. Using a range of LLMs and providing the required CFR text as context we were able to quantify the performance of those LLMs on the specific task of answering the questions below.
• 摘要：在本文中，我们介绍了法学硕士在技术语言处理任务方面的表现评估研究的结果。人类经常面临着必须从不同来源收集信息并理解大量文本的任务。这些任务对于人类来说可能非常复杂，通常需要深入研究，包括重读文本的部分内容。为了简化收集信息的任务，我们评估了具有聊天界面的法学硕士，以评估他们为标准问题提供答案的能力，而人们可以根据对文本正文的阅读来回答这些问题。所研究的正文是美国联邦法规 (CFR) 第 47 章，其中描述了联邦通信委员会 (FCC) 管辖的商业电信法规。这是一段令人感兴趣的文本，因为我们更大的研究涉及理解与无线频谱治理相关的信息以及以自动方式使用以支持动态频谱访问的问题。有关此无线频谱域的信息可以在许多不同的来源中找到，CFR 第 47 章只是其中之一。通过使用一系列法学硕士并提供所需的 CFR 文本作为上下文，我们能够量化这些法学硕士在回答以下问题的特定任务上的表现。

Title: Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentation

• Authors: Kyohoon Jin, Junho Lee, Juhwan Choi, Sangmin Song, Youngbin Kim
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15512
• Pdf URL: https://arxiv.org/pdf/2403.15512
• Copy Paste: [[2403.15512]] Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentation(https://arxiv.org/abs/2403.15512)
• Keywords: language model
• Abstract: Efforts to leverage deep learning models in low-resource regimes have led to numerous augmentation studies. However, the direct application of methods such as mixup and cutout to text data, is limited due to their discrete characteristics. While methods using pretrained language models have exhibited efficiency, they require additional considerations for robustness. Inspired by recent studies on decision boundaries, this paper proposes a decision-boundary-aware data augmentation strategy to enhance robustness using pretrained language models. The proposed technique first focuses on shifting the latent features closer to the decision boundary, followed by reconstruction to generate an ambiguous version with a soft label. Additionally, mid-K sampling is suggested to enhance the diversity of the generated sentences. This paper demonstrates the performance of the proposed augmentation strategy compared to other methods through extensive experiments. Furthermore, the ablation study reveals the effect of soft labels and mid-K sampling and the extensibility of the method with curriculum data augmentation.
• 摘要：在资源匮乏的情况下利用深度学习模型的努力引发了大量的增强研究。然而，由于其离散特性，混合和剪切等方法直接应用于文本数据受到限制。虽然使用预训练语言模型的方法已经表现出效率，但它们需要额外考虑稳健性。受最近关于决策边界的研究的启发，本文提出了一种决策边界感知数据增强策略，以使用预训练的语言模型来增强鲁棒性。所提出的技术首先侧重于将潜在特征移近决策边界，然后进行重建以生成具有软标签的模糊版本。此外，建议使用 mid-K 采样来增强生成句子的多样性。本文通过大量实验展示了所提出的增强策略与其他方法相比的性能。此外，消融研究揭示了软标签和中学抽样的效果以及该方法在课程数据增强方面的可扩展性。

Title: LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers

• Authors: Abdur Rahman Bin Md Faizullah, Ashok Urlana, Rahul Mishra
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.15529
• Pdf URL: https://arxiv.org/pdf/2403.15529
• Copy Paste: [[2403.15529]] LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers(https://arxiv.org/abs/2403.15529)
• Keywords: language model, llm
• Abstract: Examining limitations is a crucial step in the scholarly research reviewing process, revealing aspects where a study might lack decisiveness or require enhancement. This aids readers in considering broader implications for further research. In this article, we present a novel and challenging task of Suggestive Limitation Generation (SLG) for research papers. We compile a dataset called LimGen, encompassing 4068 research papers and their associated limitations from the ACL anthology. We investigate several approaches to harness large language models (LLMs) for producing suggestive limitations, by thoroughly examining the related challenges, practical insights, and potential opportunities. Our LimGen dataset and code can be accessed at https://github.com/armbf/LimGen.
• 摘要：检查局限性是学术研究审查过程中的关键一步，揭示研究可能缺乏决定性或需要改进的方面。这有助于读者考虑进一步研究的更广泛影响。在本文中，我们提出了一项新颖且具有挑战性的研究论文建议限制生成（SLG）任务。我们编译了一个名为 LimGen 的数据集，包含 ACL 选集中的 4068 篇研究论文及其相关限制。我们通过彻底检查相关挑战、实践见解和潜在机会，研究了利用大型语言模型 (LLM) 产生暗示性限制的几种方法。我们的 LimGen 数据集和代码可以在 https://github.com/armbf/LimGen 访问。

Title: AI for Biomedicine in the Era of Large Language Models

• Authors: Zhenyu Bi, Sajib Acharjee Dip, Daniel Hajialigol, Sindhura Kommu, Hanwen Liu, Meng Lu, Xuan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15673
• Pdf URL: https://arxiv.org/pdf/2403.15673
• Copy Paste: [[2403.15673]] AI for Biomedicine in the Era of Large Language Models(https://arxiv.org/abs/2403.15673)
• Keywords: language model, gpt, chat
• Abstract: The capabilities of AI for biomedicine span a wide spectrum, from the atomic level, where it solves partial differential equations for quantum systems, to the molecular level, predicting chemical or protein structures, and further extending to societal predictions like infectious disease outbreaks. Recent advancements in large language models, exemplified by models like ChatGPT, have showcased significant prowess in natural language tasks, such as translating languages, constructing chatbots, and answering questions. When we consider biomedical data, we observe a resemblance to natural language in terms of sequences: biomedical literature and health records presented as text, biological sequences or sequencing data arranged in sequences, or sensor data like brain signals as time series. The question arises: Can we harness the potential of recent large language models to drive biomedical knowledge discoveries? In this survey, we will explore the application of large language models to three crucial categories of biomedical data: 1) textual data, 2) biological sequences, and 3) brain signals. Furthermore, we will delve into large language model challenges in biomedical research, including ensuring trustworthiness, achieving personalization, and adapting to multi-modal data representation
• 摘要：人工智能在生物医学领域的能力涵盖广泛，从原子层面（解决量子系统的偏微分方程）到分子层面（预测化学或蛋白质结构），并进一步扩展到传染病爆发等社会预测。以 ChatGPT 等模型为代表的大型语言模型的最新进展在自然语言任务（例如翻译语言、构建聊天机器人和回答问题）中展示了显着的能力。当我们考虑生物医学数据时，我们观察到在序列方面与自然语言的相似之处：以文本形式呈现的生物医学文献和健康记录、按序列排列的生物序列或测序数据，或者以时间序列形式呈现的大脑信号等传感器数据。问题出现了：我们能否利用最新大型语言模型的潜力来推动生物医学知识发现？在本次调查中，我们将探索大型语言模型在生物医学数据的三个关键类别中的应用：1）文本数据，2）生物序列和3）大脑信号。此外，我们将深入研究生物医学研究中的大型语言模型挑战，包括确保可信性、实现个性化以及适应多模态数据表示

Title: EAGLE: A Domain Generalization Framework for AI-generated Text Detection

• Authors: Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, Huan Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15690
• Pdf URL: https://arxiv.org/pdf/2403.15690
• Copy Paste: [[2403.15690]] EAGLE: A Domain Generalization Framework for AI-generated Text Detection(https://arxiv.org/abs/2403.15690)
• Keywords: language model, gpt, llm
• Abstract: With the advancement in capabilities of Large Language Models (LLMs), one major step in the responsible and safe use of such LLMs is to be able to detect text generated by these models. While supervised AI-generated text detectors perform well on text generated by older LLMs, with the frequent release of new LLMs, building supervised detectors for identifying text from such new models would require new labeled training data, which is infeasible in practice. In this work, we tackle this problem and propose a domain generalization framework for the detection of AI-generated text from unseen target generators. Our proposed framework, EAGLE, leverages the labeled data that is available so far from older language models and learns features invariant across these generators, in order to detect text generated by an unknown target generator. EAGLE learns such domain-invariant features by combining the representational power of self-supervised contrastive learning with domain adversarial training. Through our experiments we demonstrate how EAGLE effectively achieves impressive performance in detecting text generated by unseen target generators, including recent state-of-the-art ones such as GPT-4 and Claude, reaching detection scores of within 4.7% of a fully supervised detector.
• 摘要：随着大型语言模型 (LLM) 功能的进步，负责任且安全地使用此类 LLM 的一个重要步骤是能够检测这些模型生成的文本。虽然受监督的人工智能生成的文本检测器在旧法学硕士生成的文本上表现良好，但随着新法学硕士的频繁发布，构建用于识别此类新模型中的文本的监督检测器将需要新的标记训练数据，这在实践中是不可行的。在这项工作中，我们解决了这个问题，并提出了一个领域泛化框架，用于从看不见的目标生成器中检测人工智能生成的文本。我们提出的框架 EAGLE 利用迄今为止从较旧的语言模型中获得的标记数据，并学习这些生成器之间不变的特征，以便检测未知目标生成器生成的文本。 EAGLE 通过将自监督对比学习的表征能力与领域对抗训练相结合来学习这种领域不变的特征。通过我们的实验，我们展示了 EAGLE 如何有效地在检测由看不见的目标生成器生成的文本方面取得令人印象深刻的性能，包括最近最先进的目标生成器，例如 GPT-4 和 Claude，达到完全监督检测器的 4.7% 以内的检测分数。

Title: FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models

• Authors: Huaiwen Zhang, Yu Chen, Ming Wang, Shi Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15699
• Pdf URL: https://arxiv.org/pdf/2403.15699
• Copy Paste: [[2403.15699]] FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models(https://arxiv.org/abs/2403.15699)
• Keywords: language model, llm
• Abstract: Emotional Support Conversation (ESC) is a typical dialogue that can effec-tively assist the user in mitigating emotional pressures. However, owing to the inherent subjectivity involved in analyzing emotions, current non-artificial methodologies face challenges in effectively appraising the emo-tional support capability. These metrics exhibit a low correlation with human judgments. Concurrently, manual evaluation methods extremely will cause high costs. To solve these problems, we propose a novel model FEEL (Framework for Evaluating Emotional Support Capability with Large Lan-guage Models), employing Large Language Models (LLMs) as evaluators to assess emotional support capabilities. The model meticulously considers var-ious evaluative aspects of ESC to apply a more comprehensive and accurate evaluation method for ESC. Additionally, it employs a probability distribu-tion approach for a more stable result and integrates an ensemble learning strategy, leveraging multiple LLMs with assigned weights to enhance evalua-tion accuracy. To appraise the performance of FEEL, we conduct extensive experiments on existing ESC model dialogues. Experimental results demon-strate our model exhibits a substantial enhancement in alignment with human evaluations compared to the baselines. Our source code is available at https://github.com/Ansisy/FEEL.
• 摘要：情感支持对话（ESC）是一种典型的对话，可以有效帮助用户缓解情绪压力。然而，由于情绪分析固有的主观性，当前的非人工方法在有效评估情绪支持能力方面面临挑战。这些指标与人类判断的相关性较低。同时，人工评估方法会造成极高的成本。为了解决这些问题，我们提出了一种新的模型FEEL（使用大型语言模型评估情感支持能力的框架），采用大型语言模型（LLM）作为评估器来评估情感支持能力。该模型精心考虑了ESC的各个评价方面，应用了更加全面、准确的ESC评价方法。此外，它采用概率分布方法来获得更稳定的结果，并集成集成学习策略，利用多个具有指定权重的法学硕士来提高评估准确性。为了评估 FEEL 的性能，我们对现有的 ESC 模型对话进行了广泛的实验。实验结果表明，与基线相比，我们的模型与人类评估的一致性有了显着增强。我们的源代码可在 https://github.com/Ansisy/FEEL 获取。

Title: EDDA: A Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detection

• Authors: Daijun Ding, Li Dong, Zhichao Huang, Guangning Xu, Xu Huang, Bo Liu, Liwen Jing, Bowen Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15715
• Pdf URL: https://arxiv.org/pdf/2403.15715
• Copy Paste: [[2403.15715]] EDDA: A Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detection(https://arxiv.org/abs/2403.15715)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Stance detection aims to determine the attitude expressed in text towards a given target. Zero-shot stance detection (ZSSD) has emerged to classify stances towards unseen targets during inference. Recent data augmentation techniques for ZSSD increase transferable knowledge between targets through text or target augmentation. However, these methods exhibit limitations. Target augmentation lacks logical connections between generated targets and source text, while text augmentation relies solely on training data, resulting in insufficient generalization. To address these issues, we propose an encoder-decoder data augmentation (EDDA) framework. The encoder leverages large language models and chain-of-thought prompting to summarize texts into target-specific if-then rationales, establishing logical relationships. The decoder generates new samples based on these expressions using a semantic correlation word replacement strategy to increase syntactic diversity. We also analyze the generated expressions to develop a rationale-enhanced network that fully utilizes the augmented data. Experiments on benchmark datasets demonstrate our approach substantially improves over state-of-the-art ZSSD techniques. The proposed EDDA framework increases semantic relevance and syntactic variety in augmented texts while enabling interpretable rationale-based learning.
• 摘要：立场检测旨在确定文本中表达的对给定目标的态度。零镜头姿态检测（ZSSD）的出现可以在推理过程中对看不见的目标的姿态进行分类。最近的 ZSSD 数据增强技术通过文本或目标增强增加了目标之间的可转移知识。然而，这些方法存在局限性。目标增强在生成的目标和源文本之间缺乏逻辑联系，而文本增强仅依赖于训练数据，导致泛化不足。为了解决这些问题，我们提出了一种编码器-解码器数据增强（EDDA）框架。编码器利用大型语言模型和思维链提示将文本总结为特定目标的 if-then 基本原理，建立逻辑关系。解码器使用语义相关词替换策略基于这些表达生成新样本以增加句法多样性。我们还分析生成的表达式，以开发充分利用增强数据的基本原理增强网络。对基准数据集的实验表明，我们的方法比最先进的 ZSSD 技术有了显着的改进。所提出的 EDDA 框架增加了增强文本中的语义相关性和句法多样性，同时实现了可解释的基于原理的学习。

Title: Towards a \textbf{RAG}-based Summarization Agent for the Electron-Ion Collider

• Authors: Karthik Suresh, Neeltje Kackar, Luke Schleck, Cristiano Fanelli
• Subjects: cs.CL, cs.AI, hep-ex, physics.ins-det
• Abstract URL: https://arxiv.org/abs/2403.15729
• Pdf URL: https://arxiv.org/pdf/2403.15729
• Copy Paste: [[2403.15729]] Towards a \textbf{RAG}-based Summarization Agent for the Electron-Ion Collider(https://arxiv.org/abs/2403.15729)
• Keywords: language model, llm, prompt, retrieval augmented generation, agent
• Abstract: The complexity and sheer volume of information encompassing documents, papers, data, and other resources from large-scale experiments demand significant time and effort to navigate, making the task of accessing and utilizing these varied forms of information daunting, particularly for new collaborators and early-career scientists. To tackle this issue, a Retrieval Augmented Generation (RAG)--based Summarization AI for EIC (RAGS4EIC) is under development. This AI-Agent not only condenses information but also effectively references relevant responses, offering substantial advantages for collaborators. Our project involves a two-step approach: first, querying a comprehensive vector database containing all pertinent experiment information; second, utilizing a Large Language Model (LLM) to generate concise summaries enriched with citations based on user queries and retrieved data. We describe the evaluation methods that use RAG assessments (RAGAs) scoring mechanisms to assess the effectiveness of responses. Furthermore, we describe the concept of prompt template-based instruction-tuning which provides flexibility and accuracy in summarization. Importantly, the implementation relies on LangChain, which serves as the foundation of our entire workflow. This integration ensures efficiency and scalability, facilitating smooth deployment and accessibility for various user groups within the Electron Ion Collider (EIC) community. This innovative AI-driven framework not only simplifies the understanding of vast datasets but also encourages collaborative participation, thereby empowering researchers. As a demonstration, a web application has been developed to explain each stage of the RAG Agent development in detail.
• 摘要：包含来自大规模实验的文档、论文、数据和其他资源的信息复杂且数量庞大，需要大量的时间和精力来导航，使得访问和利用这些不同形式的信息的任务令人畏惧，特别是对于新合作者和早期合作者来说。 -职业科学家。为了解决这个问题，基于检索增强生成 (RAG) 的 EIC 摘要人工智能 (RAGS4EIC) 正在开发中。该AI-Agent不仅可以浓缩信息，还可以有效引用相关响应，为协作者提供巨大的优势。我们的项目涉及两步方法：首先，查询包含所有相关实验信息的综合向量数据库；其次，利用大型语言模型（LLM）根据用户查询和检索的数据生成包含丰富引文的简明摘要。我们描述了使用 RAG 评估 (RAGA) 评分机制来评估响应有效性的评估方法。此外，我们描述了基于提示模板的指令调整的概念，它提供了总结的灵活性和准确性。重要的是，这个实现依赖于LangChain，它是我们整个工作流程的基础。这种集成确保了效率和可扩展性，促进电子离子对撞机 (EIC) 社区内各个用户组的顺利部署和可访问性。这种创新的人工智能驱动框架不仅简化了对大量数据集的理解，还鼓励协作参与，从而为研究人员提供支持。作为演示，我们开发了一个 Web 应用程序来详细解释 RAG 代理开发的每个阶段。

Title: LLMs Instruct LLMs:An Extraction and Editing Method

• Authors: Xin Zhang, Tianjie Ju, Huijia Liang, Ying Fu, Qin Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15736
• Pdf URL: https://arxiv.org/pdf/2403.15736
• Copy Paste: [[2403.15736]] LLMs Instruct LLMs:An Extraction and Editing Method(https://arxiv.org/abs/2403.15736)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The interest in updating Large Language Models (LLMs) without retraining from scratch is substantial, yet it comes with some challenges.This is especially true for situations demanding complex reasoning with limited samples, a scenario we refer to as the Paucity-Constrained Complex Reasoning Adaptation for LLMs (PCRA-LLM).Traditional methods like Low-Rank Adaptation (LoRA) and Retrieval-Augmented Generation (RAG) are inadequate for this critical issue, particularly evident in our exploration of a specific medical context that epitomize the PCRA-LLM's distinct needs.To address the issue, we propose a Sequential Fusion method to incorporate knowledge from complex context into LLMs. This method employs a two-stage framework: initially, it leverages general LLMs to construct knowledge graphs (KGs) for extracting knowledge from complex texts; subsequently, it updates the domain LLMs through knowledge edit. According to our method, the domain LLM achieved a 71.69\% accuracy in question answering tasks. Subsequently, we broadened our assessment to a novel dataset we developed in the economics and management field, where our method realized a 75\% accuracy. These outcomes underline the efficacy and adaptability of our approach for PCRA-LLM across various domains.
• 摘要：人们对在不从头开始重新训练的情况下更新大型语言模型 (LLM) 的兴趣很大，但它也带来了一些挑战。对于需要使用有限样本进行复杂推理的情况尤其如此，我们将这种情况称为“缺乏约束复杂推理适应”低阶适应 (LoRA) 和检索增强生成 (RAG) 等传统方法不足以解决这个关键问题，在我们对体现 PCRA-LLM 独特医学背景的探索中尤其明显为了解决这个问题，我们提出了一种顺序融合方法，将复杂上下文中的知识融入到法学硕士中。该方法采用两阶段框架：首先，利用通用法学硕士构建知识图谱（KG）以从复杂文本中提取知识；随后，它通过知识编辑更新领域法学硕士。根据我们的方法，领域法学硕士在问答任务中达到了 71.69% 的准确率。随后，我们将评估范围扩大到我们在经济和管理领域开发的新数据集，我们的方法实现了 75% 的准确率。这些结果强调了我们的 PCRA-LLM 方法在各个领域的有效性和适应性。

Title: Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoning

• Authors: Zhouhang Xie, Bodhisattwa Prasad Majumder, Mengjie Zhao, Yoshinori Maeda, Keiichi Yamada, Hiromi Wakaki, Julian McAuley
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15737
• Pdf URL: https://arxiv.org/pdf/2403.15737
• Copy Paste: [[2403.15737]] Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoning(https://arxiv.org/abs/2403.15737)
• Keywords: language model
• Abstract: We consider the task of building a dialogue system that can motivate users to adopt positive lifestyle changes: Motivational Interviewing. Addressing such a task requires a system that can infer \textit{how} to motivate a user effectively. We propose DIIT, a framework that is capable of learning and applying conversation strategies in the form of natural language inductive rules from expert demonstrations. Automatic and human evaluation on instruction-following large language models show natural language strategy descriptions discovered by DIIR can improve active listening skills, reduce unsolicited advice, and promote more collaborative and less authoritative responses, outperforming various demonstration utilization methods.
• 摘要：我们考虑建立一个对话系统来激励用户采取积极的生活方式改变的任务：动机访谈。解决此类任务需要一个能够推断\textit{如何}有效激励用户的系统。我们提出了 DIIT，这是一个能够从专家演示中以自然语言归纳规则的形式学习和应用对话策略的框架。对遵循指令的大型语言模型的自动和人工评估表明，DIIR 发现的自然语言策略描述可以提高主动听力技能，减少主动建议，并促进更多协作和更少权威的响应，优于各种演示利用方法。

Title: Ghost Sentence: A Tool for Everyday Users to Copyright Data from Large Language Models

• Authors: Shuai Zhao, Linchao Zhu, Ruijie Quan, Yi Yang
• Subjects: cs.CL, cs.CR, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15740
• Pdf URL: https://arxiv.org/pdf/2403.15740
• Copy Paste: [[2403.15740]] Ghost Sentence: A Tool for Everyday Users to Copyright Data from Large Language Models(https://arxiv.org/abs/2403.15740)
• Keywords: language model, llm
• Abstract: Web user data plays a central role in the ecosystem of pre-trained large language models (LLMs) and their fine-tuned variants. Billions of data are crawled from the web and fed to LLMs. How can \textit{\textbf{everyday web users}} confirm if LLMs misuse their data without permission? In this work, we suggest that users repeatedly insert personal passphrases into their documents, enabling LLMs to memorize them. These concealed passphrases in user documents, referred to as \textit{ghost sentences}, once they are identified in the generated content of LLMs, users can be sure that their data is used for training. To explore the effectiveness and usage of this copyrighting tool, we define the \textit{user training data identification} task with ghost sentences. Multiple datasets from various sources at different scales are created and tested with LLMs of different sizes. For evaluation, we introduce a last $k$ words verification manner along with two metrics: document and user identification accuracy. In the specific case of instruction tuning of a 3B LLaMA model, 11 out of 16 users with ghost sentences identify their data within the generation content. These 16 users contribute 383 examples to $\sim$1.8M training documents. For continuing pre-training of a 1.1B TinyLlama model, 61 out of 64 users with ghost sentences identify their data within the LLM output. These 64 users contribute 1156 examples to $\sim$10M training documents.
• 摘要：Web 用户数据在预训练大型语言模型 (LLM) 及其微调变体的生态系统中发挥着核心作用。从网络上抓取的数十亿数据并提供给法学硕士。 \textit{\textbf{日常网络用户}}如何确认法学硕士是否未经许可滥用其数据？在这项工作中，我们建议用户在其文档中重复插入个人密码，以便法学硕士能够记住它们。这些隐藏在用户文档中的密码短语，被称为\textit{鬼句子}，一旦在LLM生成的内容中被识别出来，用户就可以确定他们的数据用于训练。为了探索这个版权工具的有效性和用途，我们用幽灵句子定义了 \textit{用户训练数据识别} 任务。创建来自不同来源、不同规模的多个数据集，并使用不同规模的法学硕士进行测试。为了进行评估，我们引入了最后 $k$ 个单词验证方式以及两个指标：文档和用户识别准确性。在 3B LLaMA 模型的指令调整的特定情况下，16 个有鬼句子的用户中有 11 个在生成内容中识别出他们的数据。这 16 位用户为 $\sim$1.8M 的培训文档贡献了 383 个示例。为了继续对 1.1B TinyLlama 模型进行预训练，64 个使用鬼句子的用户中有 61 个在 LLM 输出中识别出了他们的数据。这 64 位用户为 $\sim$10M 的培训文档贡献了 1156 个示例。

Title: Understanding Emergent Abilities of Language Models from the Loss Perspective

• Authors: Zhengxiao Du, Aohan Zeng, Yuxiao Dong, Jie Tang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15796
• Pdf URL: https://arxiv.org/pdf/2403.15796
• Copy Paste: [[2403.15796]] Understanding Emergent Abilities of Language Models from the Loss Perspective(https://arxiv.org/abs/2403.15796)
• Keywords: language model
• Abstract: Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.
• 摘要：最近的研究对语言模型中的涌现能力是大型模型所独有的信念提出了质疑。这种怀疑源于两个观察结果：1）较小的模型也可以在新兴能力上表现出高性能；2）对用于衡量这些能力的不连续指标存在疑问。在本文中，我们建议研究预训练损失镜头中的涌现能力，而不是模型大小或训练计算。我们证明，具有相同预训练损失但不同模型和数据大小的模型在各种下游任务上产生相同的性能。我们还发现，当模型的预训练损失低于特定阈值时，无论指标的连续性如何，模型都会在某些任务上表现出新兴能力。在达到这个阈值之前，其性能仍停留在随机猜测的水平。这激励我们重新定义涌现能力，即那些在预训练损失较低的模型中表现出来的能力，强调这些能力不能仅仅通过推断具有较高预训练损失的模型的性能趋势来预测。

Title: Computational Sentence-level Metrics Predicting Human Sentence Comprehension

• Authors: Kun Sun, Rong Wang
• Subjects: cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.15822
• Pdf URL: https://arxiv.org/pdf/2403.15822
• Copy Paste: [[2403.15822]] Computational Sentence-level Metrics Predicting Human Sentence Comprehension(https://arxiv.org/abs/2403.15822)
• Keywords: language model, llm
• Abstract: The majority of research in computational psycholinguistics has concentrated on the processing of words. This study introduces innovative methods for computing sentence-level metrics using multilingual large language models. The metrics developed sentence surprisal and sentence relevance and then are tested and compared to validate whether they can predict how humans comprehend sentences as a whole across languages. These metrics offer significant interpretability and achieve high accuracy in predicting human sentence reading speeds. Our results indicate that these computational sentence-level metrics are exceptionally effective at predicting and elucidating the processing difficulties encountered by readers in comprehending sentences as a whole across a variety of languages. Their impressive performance and generalization capabilities provide a promising avenue for future research in integrating LLMs and cognitive science.
• 摘要：计算心理语言学的大部分研究都集中在单词的处理上。本研究介绍了使用多语言大语言模型计算句子级度量的创新方法。这些指标开发了句子惊讶度和句子相关性，然后进行测试和比较，以验证它们是否可以预测人类如何跨语言理解整个句子。这些指标提供了显着的可解释性，并在预测人类句子阅读速度方面实现了高精度。我们的结果表明，这些计算句子级指标在预测和阐明读者在理解各种语言的整个句子时遇到的处理困难方面非常有效。他们令人印象深刻的表现和泛化能力为未来整合法学硕士和认知科学的研究提供了一个有前途的途径。

Title: Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

• Authors: Lukas Vöge, Vincent Gurgul, Stefan Lessmann
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15886
• Pdf URL: https://arxiv.org/pdf/2403.15886
• Copy Paste: [[2403.15886]] Leveraging Zero-Shot Prompting for Efficient Language Model Distillation(https://arxiv.org/abs/2403.15886)
• Keywords: language model, llm, prompt
• Abstract: This paper introduces a novel approach for efficiently distilling LLMs into smaller, application-specific models, significantly reducing operational costs and manual labor. Addressing the challenge of deploying computationally intensive LLMs in specific applications or edge devices, this technique utilizes LLMs' reasoning capabilities to generate labels and natural language rationales for unlabeled data. Our approach enhances both finetuning and distillation by employing a multi-task training framework where student models mimic these rationales alongside teacher predictions. Key contributions include the employment of zero-shot prompting to elicit teacher model rationales, reducing the necessity for handcrafted few-shot examples and lowering the overall token count required, which directly translates to cost savings given the pay-per-token billing model of major tech companies' LLM APIs. Additionally, the paper investigates the impact of explanation properties on distillation efficiency, demonstrating that minimal performance loss occurs even when rationale augmentation is not applied across the entire dataset, facilitating further reductions of tokens. This research marks a step toward the efficient training of task-specific models with minimal human intervention, offering substantial cost-savings while maintaining, or even enhancing, performance.
• 摘要：本文介绍了一种新颖的方法，可以有效地将法学硕士转化为更小的、特定于应用程序的模型，从而显着降低运营成本和体力劳动。为了解决在特定应用程序或边缘设备中部署计算密集型 LLM 的挑战，该技术利用 LLM 的推理功能为未标记数据生成标签和自然语言原理。我们的方法通过采用多任务训练框架来增强微调和蒸馏，其中学生模型模仿这些基本原理以及教师的预测。主要贡献包括使用零样本提示来引出教师模型的基本原理，减少手工制作少样本示例的必要性，并降低所需的总体令牌数量，考虑到主要的按令牌付费计费模型，这直接转化为成本节约。科技公司的 LLM API。此外，本文还研究了解释属性对蒸馏效率的影响，证明即使没有在整个数据集中应用基本原理增强，也会发生最小的性能损失，从而促进进一步减少标记。这项研究标志着朝着以最少的人为干预有效训练特定任务模型的方向迈出了一步，在保持甚至增强性能的同时节省了大量成本。

Title: LlamBERT: Large-scale low-cost data annotation in NLP

• Authors: Bálint Csanády, Lajos Muzsai, Péter Vedres, Zoltán Nádasdy, András Lukács
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15938
• Pdf URL: https://arxiv.org/pdf/2403.15938
• Copy Paste: [[2403.15938]] LlamBERT: Large-scale low-cost data annotation in NLP(https://arxiv.org/abs/2403.15938)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs), such as GPT-4 and Llama 2, show remarkable proficiency in a wide range of natural language processing (NLP) tasks. Despite their effectiveness, the high costs associated with their use pose a challenge. We present LlamBERT, a hybrid approach that leverages LLMs to annotate a small subset of large, unlabeled databases and uses the results for fine-tuning transformer encoders like BERT and RoBERTa. This strategy is evaluated on two diverse datasets: the IMDb review dataset and the UMLS Meta-Thesaurus. Our results indicate that the LlamBERT approach slightly compromises on accuracy while offering much greater cost-effectiveness.
• 摘要：大型语言模型 (LLM)，例如 GPT-4 和 Llama 2，在各种自然语言处理 (NLP) 任务中表现出卓越的熟练程度。尽管它们有效，但与其使用相关的高成本构成了挑战。我们提出了 LlamBERT，这是一种混合方法，利用 LLM 来注释大型未标记数据库的一小部分，并使用结果来微调 BERT 和 RoBERTa 等 Transformer 编码器。该策略在两个不同的数据集上进行评估：IMDb 评论数据集和 UMLS 元同义词库。我们的结果表明，LlamBERT 方法在准确性方面略有妥协，但提供了更高的成本效益。

Title: CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering

• Authors: Hongbin Na
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16008
• Pdf URL: https://arxiv.org/pdf/2403.16008
• Copy Paste: [[2403.16008]] CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering(https://arxiv.org/abs/2403.16008)
• Keywords: language model, llm, prompt
• Abstract: The recent advancements in artificial intelligence highlight the potential of language models in psychological health support. While models trained on data from mental health service platform have achieved preliminary success, challenges persist in areas such as data scarcity, quality, and ensuring a solid foundation in psychological techniques. To address these challenges, this study introduces a novel approach to enhance the precision and efficacy of psychological support through large language models. Specifically, we design a specific prompt derived from principles of Cognitive Behavioral Therapy (CBT) and have generated the CBT QA dataset, specifically for Chinese psychological health Q&A based on CBT structured intervention strategies. Unlike previous methods, our dataset emphasizes professional and structured response. Utilizing this dataset, we fine-tuned the large language model, giving birth to CBT-LLM, the large-scale language model specifically designed for Cognitive Behavioral Therapy techniques. Empirical evaluations demonstrate that CBT-LLM excels in generating structured, professional, and highly relevant responses in psychological health support tasks, showcasing its practicality and quality. The model is available on Hugging Face: https://huggingface.co/Hongbin37/CBT-LLM.
• 摘要：人工智能的最新进展凸显了语言模型在心理健康支持方面的潜力。虽然基于心理健康服务平台数据训练的模型取得了初步成功，但在数据稀缺、质量和确保心理技术基础等方面仍然存在挑战。为了应对这些挑战，本研究引入了一种新方法，通过大型语言模型来提高心理支持的准确性和有效性。具体来说，我们根据认知行为疗法（CBT）原理设计了一个特定的提示，并生成了CBT QA数据集，专门针对基于CBT结构化干预策略的中国心理健康问答。与以前的方法不同，我们的数据集强调专业和结构化的响应。利用这个数据集，我们对大型语言模型进行了微调，诞生了 CBT-LLM，这是专门为认知行为治疗技术设计的大型语言模型。实证评估表明，CBT-LLM在心理健康支持任务中擅长生成结构化、专业且高度相关的响应，展示了其实用性和质量。该模型可在 Hugging Face 上找到：https://huggingface.co/hongbin37/CBT-LLM。

Title: Monotonic Paraphrasing Improves Generalization of Language Model Prompting

• Authors: Qin Liu, Fei Wang, Nan Xu, Tianyi Yan, Tao Meng, Muhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16038
• Pdf URL: https://arxiv.org/pdf/2403.16038
• Copy Paste: [[2403.16038]] Monotonic Paraphrasing Improves Generalization of Language Model Prompting(https://arxiv.org/abs/2403.16038)
• Keywords: language model, llm, prompt
• Abstract: Performance of large language models (LLMs) may vary with different prompts or instructions of even the same task. One commonly recognized factor for this phenomenon is the model's familiarity with the given prompt or instruction, which is typically estimated by its perplexity. However, finding the prompt with the lowest perplexity is challenging, given the enormous space of possible prompting phrases. In this paper, we propose monotonic paraphrasing (MonoPara), an end-to-end decoding strategy that paraphrases given prompts or instructions into their lower perplexity counterparts based on an ensemble of a paraphrase LM for prompt (or instruction) rewriting, and a target LM (i.e. the prompt or instruction executor) that constrains the generation for lower perplexity. The ensemble decoding process can efficiently paraphrase the original prompt without altering its semantic meaning, while monotonically decreasing the perplexity of each generation as calculated by the target LM. We explore in detail both greedy and search-based decoding as two alternative decoding schemes of MonoPara. Notably, MonoPara does not require any training and can monotonically lower the perplexity of the paraphrased prompt or instruction, leading to improved performance of zero-shot LM prompting as evaluated on a wide selection of tasks. In addition, MonoPara is also shown to effectively improve LMs' generalization on perturbed and unseen task instructions.
• 摘要：即使同一任务的不同提示或指令，大语言模型 (LLM) 的性能也可能会有所不同。这种现象的一个普遍认可的因素是模型对给定提示或指令的熟悉程度，这通常是通过其困惑度来估计的。然而，考虑到可能的提示短语的空间巨大，找到困惑度最低的提示是具有挑战性的。在本文中，我们提出了单调释义（MonoPara），这是一种端到端解码策略，基于用于提示（或指令）重写的释义 LM 的集合，将给定的提示或指令释义为其较低困惑度的对应物，以及一个目标LM（即提示或指令执行器）限制生成以降低困惑度。集成解码过程可以有效地解释原始提示而不改变其语义，同时单调减少目标 LM 计算的每一代的困惑度。我们详细探讨了贪婪解码和基于搜索的解码作为 MonoPara 的两种替代解码方案。值得注意的是，MonoPara 不需要任何训练，并且可以单调地降低释义提示或指令的复杂性，从而提高在广泛的任务选择上评估的零样本 LM 提示的性能。此外，MonoPara 还被证明可以有效提高 LM 对受干扰和未见过的任务指令的泛化能力。

Title: Qibo: A Large Language Model for Traditional Chinese Medicine

• Authors: Heyi Zhang, Xin Wang, Zhaopeng Meng, Yongzhe Jia, Dawei Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.16056
• Pdf URL: https://arxiv.org/pdf/2403.16056
• Copy Paste: [[2403.16056]] Qibo: A Large Language Model for Traditional Chinese Medicine(https://arxiv.org/abs/2403.16056)
• Keywords: language model, llm
• Abstract: In the field of Artificial Intelligence, Large Language Models (LLMs) have demonstrated significant advances in user intent understanding and response in a number of specialized domains, including medicine, law, and finance. However, in the unique domain of traditional Chinese medicine (TCM), the performance enhancement of LLMs is challenged by the essential differences between its theories and modern medicine, as well as the lack of specialized corpus resources. In this paper, we aim to construct and organize a professional corpus in the field of TCM, to endow the large model with professional knowledge that is characteristic of TCM theory, and to successfully develop the Qibo model based on LLaMA, which is the first LLM in the field of TCM to undergo a complete training process from pre-training to Supervised Fine-Tuning (SFT). Furthermore, we develop the Qibo-benchmark, a specialized tool for evaluating the performance of LLMs, which is a specialized tool for evaluating the performance of LLMs in the TCM domain. This tool will provide an important basis for quantifying and comparing the understanding and application capabilities of different models in the field of traditional Chinese medicine, and provide guidance for future research directions and practical applications of intelligent assistants for traditional Chinese medicine. Finally, we conducted sufficient experiments to prove that Qibo has good performance in the field of traditional Chinese medicine.
• 摘要：在人工智能领域，大型语言模型 (LLM) 在医学、法律和金融等多个专业领域的用户意图理解和响应方面取得了显着进步。然而，在中医这一独特领域，法学硕士的成绩提升面临着其理论与现代医学的本质差异以及专业语料库资源缺乏的挑战。本文旨在构建和整理中医领域的专业语料库，赋予大模型以中医理论特色的专业知识，并成功开发了基于LLaMA的启博模型，这是第一个法学硕士在TCM领域经历从预训练到监督微调（SFT）的完整训练过程。此外，我们还开发了Qibo-benchmark，一个专门用于评估LLM表现的工具，这是一个专门用于评估TCM领域LLM表现的工具。该工具将为量化和比较中医领域不同模型的理解和应用能力提供重要依据，并为中医智能助手未来的研究方向和实际应用提供指导。最后，我们进行了充分的实验，证明奇博在中药领域具有良好的表现。

Title: Argument Quality Assessment in the Age of Instruction-Following Large Language Models

• Authors: Henning Wachsmuth, Gabriella Lapesa, Elena Cabrio, Anne Lauscher, Joonsuk Park, Eva Maria Vecchi, Serena Villata, Timon Ziegenbein
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16084
• Pdf URL: https://arxiv.org/pdf/2403.16084
• Copy Paste: [[2403.16084]] Argument Quality Assessment in the Age of Instruction-Following Large Language Models(https://arxiv.org/abs/2403.16084)
• Keywords: language model, llm
• Abstract: The computational treatment of arguments on controversial issues has been subject to extensive NLP research, due to its envisioned impact on opinion formation, decision making, writing education, and the like. A critical task in any such application is the assessment of an argument's quality - but it is also particularly challenging. In this position paper, we start from a brief survey of argument quality research, where we identify the diversity of quality notions and the subjectiveness of their perception as the main hurdles towards substantial progress on argument quality assessment. We argue that the capabilities of instruction-following large language models (LLMs) to leverage knowledge across contexts enable a much more reliable assessment. Rather than just fine-tuning LLMs towards leaderboard chasing on assessment tasks, they need to be instructed systematically with argumentation theories and scenarios as well as with ways to solve argument-related problems. We discuss the real-world opportunities and ethical issues emerging thereby.
• 摘要：由于其对意见形成、决策、写作教育等的预期影响，对有争议问题的争论的计算处理已经受到广泛的 NLP 研究的影响。任何此类应用中的一项关键任务是评估论证的质量 - 但它也特别具有挑战性。在这篇立场文件中，我们从对论证质量研究的简要调查开始，我们确定质量概念的多样性及其感知的主观性是论证质量评估取得实质性进展的主要障碍。我们认为，遵循指令的大型语言模型 (LLM) 能够跨上下文利用知识，从而实现更可靠的评估。法学硕士不仅仅需要对评估任务的排行榜进行微调，还需要系统地指导他们论证理论和场景以及解决论证相关问题的方法。我们讨论现实世界的机会和由此出现的道德问题。

Title: A Survey on Lexical Ambiguity Detection and Word Sense Disambiguation

• Authors: Miuru Abeysiriwardana, Deshan Sumanathilaka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16129
• Pdf URL: https://arxiv.org/pdf/2403.16129
• Copy Paste: [[2403.16129]] A Survey on Lexical Ambiguity Detection and Word Sense Disambiguation(https://arxiv.org/abs/2403.16129)
• Keywords: language model
• Abstract: This paper explores techniques that focus on understanding and resolving ambiguity in language within the field of natural language processing (NLP), highlighting the complexity of linguistic phenomena such as polysemy and homonymy and their implications for computational models. Focusing extensively on Word Sense Disambiguation (WSD), it outlines diverse approaches ranging from deep learning techniques to leveraging lexical resources and knowledge graphs like WordNet. The paper introduces cutting-edge methodologies like word sense extension (WSE) and neuromyotonic approaches, enhancing disambiguation accuracy by predicting new word senses. It examines specific applications in biomedical disambiguation and language specific optimisation and discusses the significance of cognitive metaphors in discourse analysis. The research identifies persistent challenges in the field, such as the scarcity of sense annotated corpora and the complexity of informal clinical texts. It concludes by suggesting future directions, including using large language models, visual WSD, and multilingual WSD systems, emphasising the ongoing evolution in addressing lexical complexities in NLP. This thinking perspective highlights the advancement in this field to enable computers to understand language more accurately.
• 摘要：本文探讨了自然语言处理（NLP）领域内专注于理解和解决语言歧义的技术，强调了多义词和同音词等语言现象的复杂性及其对计算模型的影响。它广泛关注词义消歧 (WSD)，概述了从深度学习技术到利用词汇资源和知识图（例如 WordNet）的各种方法。本文介绍了词义扩展 (WSE) 和神经肌强直方法等前沿方法，通过预测新词义来提高消歧准确性。它研究了生物医学消歧和语言特定优化中的具体应用，并讨论了认知隐喻在话语分析中的重要性。该研究确定了该领域持续存在的挑战，例如语义注释语料库的稀缺和非正式临床文本的复杂性。最后提出了未来的方向，包括使用大型语言模型、视觉 WSD 和多语言 WSD 系统，强调了 NLP 中词汇复杂性的持续发展。这种思维视角凸显了该领域的进步，使计算机能够更准确地理解语言。

Title: A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

• Authors: Masahiro Kaneko, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16139
• Pdf URL: https://arxiv.org/pdf/2403.16139
• Copy Paste: [[2403.16139]] A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish(https://arxiv.org/abs/2403.16139)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are trained on massive web-crawled corpora. This poses risks of leakage, including personal information, copyrighted texts, and benchmark datasets. Such leakage leads to undermining human trust in AI due to potential unauthorized generation of content or overestimation of performance. We establish the following three criteria concerning the leakage issues: (1) leakage rate: the proportion of leaked data in training data, (2) output rate: the ease of generating leaked data, and (3) detection rate: the detection performance of leaked versus non-leaked data. Despite the leakage rate being the origin of data leakage issues, it is not understood how it affects the output rate and detection rate. In this paper, we conduct an experimental survey to elucidate the relationship between the leakage rate and both the output rate and detection rate for personal information, copyrighted texts, and benchmark data. Additionally, we propose a self-detection approach that uses few-shot learning in which LLMs detect whether instances are present or absent in their training data, in contrast to previous methods that do not employ explicit learning. To explore the ease of generating leaked information, we create a dataset of prompts designed to elicit personal information, copyrighted text, and benchmarks from LLMs. Our experiments reveal that LLMs produce leaked information in most cases despite less such data in their training set. This indicates even small amounts of leaked data can greatly affect outputs. Our self-detection method showed superior performance compared to existing detection methods.
• 摘要：大型语言模型 (LLM) 在大量网络爬取语料库上进行训练。这会带来泄露风险，包括个人信息、受版权保护的文本和基准数据集。由于潜在的未经授权的内容生成或对性能的高估，这种泄露会破坏人类对人工智能的信任。我们针对泄漏问题建立了以下三个标准：（1）泄漏率：泄漏数据在训练数据中的比例，（2）输出率：生成泄漏数据的难易程度，（3）检测率：泄漏数据的检测性能。泄露数据与非泄露数据。尽管泄漏率是数据泄漏问题的根源，但尚不清楚它如何影响输出率和检测率。在本文中，我们进行了一项实验调查，以阐明泄漏率与个人信息、版权文本和基准数据的输出率和检测率之间的关系。此外，我们提出了一种使用少样本学习的自我检测方法，其中法学硕士检测其训练数据中是否存在实例，这与之前不采用显式学习的方法形成鲜明对比。为了探索生成泄露信息的难易程度，我们创建了一个提示数据集，旨在引出个人信息、受版权保护的文本和法学硕士的基准。我们的实验表明，法学硕士在大多数情况下都会产生泄露信息，尽管他们的训练集中此类数据较少。这表明即使是少量的泄露数据也会极大地影响输出。与现有的检测方法相比，我们的自我检测方法表现出优越的性能。

Title: Korean Bio-Medical Corpus (KBMC) for Medical Named Entity Recognition

• Authors: Sungjoo Byun, Jiseung Hong, Sumin Park, Dongjun Jang, Jean Seo, Minseok Kim, Chaeyoung Oh, Hyopil Shin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16158
• Pdf URL: https://arxiv.org/pdf/2403.16158
• Copy Paste: [[2403.16158]] Korean Bio-Medical Corpus (KBMC) for Medical Named Entity Recognition(https://arxiv.org/abs/2403.16158)
• Keywords: gpt, chat
• Abstract: Named Entity Recognition (NER) plays a pivotal role in medical Natural Language Processing (NLP). Yet, there has not been an open-source medical NER dataset specifically for the Korean language. To address this, we utilized ChatGPT to assist in constructing the KBMC (Korean Bio-Medical Corpus), which we are now presenting to the public. With the KBMC dataset, we noticed an impressive 20% increase in medical NER performance compared to models trained on general Korean NER datasets. This research underscores the significant benefits and importance of using specialized tools and datasets, like ChatGPT, to enhance language processing in specialized fields such as healthcare.
• 摘要：命名实体识别（NER）在医学自然语言处理（NLP）中发挥着关键作用。然而，还没有专门针对韩语的开源医学 NER 数据集。为了解决这个问题，我们利用 ChatGPT 来协助构建 KBMC（韩国生物医学语料库），现在我们将其向公众展示。通过 KBMC 数据集，我们注意到与在一般韩国 NER 数据集上训练的模型相比，医学 NER 性能显着提高了 20%。这项研究强调了使用 ChatGPT 等专用工具和数据集来增强医疗保健等专业领域的语言处理的显着好处和重要性。

Title: ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

• Authors: Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, Yvette Graham
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16187
• Pdf URL: https://arxiv.org/pdf/2403.16187
• Copy Paste: [[2403.16187]] ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models(https://arxiv.org/abs/2403.16187)
• Keywords: language model
• Abstract: Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in the era of large language models. Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular and representative method. However, it is implemented with a fixed intrinsic rank that might not be the ideal setting for the downstream tasks. Recognizing the need for more flexible downstream task adaptation, we extend the methodology of LoRA to an innovative approach we call allocating low-rank adaptation (ALoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. First, we propose a novel method, AB-LoRA, that can effectively estimate the importance score of each LoRA rank. Second, guided by AB-LoRA, we gradually prune abundant and negatively impacting LoRA ranks and allocate the pruned LoRA budgets to important Transformer modules needing higher ranks. We have conducted experiments on various tasks, and the experimental results demonstrate that our ALoRA method can outperform the recent baselines with comparable tunable parameters.
• 摘要：参数高效微调（PEFT）因其在大型语言模型时代的有效性和效率而被广泛研究。低秩适应（LoRA）作为一种流行且具有代表性的方法表现出了值得称赞的性能。然而，它是通过固定的内在等级来实现的，这可能不是下游任务的理想设置。认识到需要更灵活的下游任务适应，我们将 LoRA 的方法扩展到一种创新方法，我们称之为分配低秩适应 (ALoRA)，该方法可以在适应过程中动态调整内在等级。首先，我们提出了一种新方法 AB-LoRA，它可以有效地估计每个 LoRA 等级的重要性得分。其次，在 AB-LoRA 的指导下，我们逐渐修剪大量且有负面影响的 LoRA 排名，并将修剪后的 LoRA 预算分配给需要更高排名的重要 Transformer 模块。我们对各种任务进行了实验，实验结果表明，我们的 ALoRA 方法可以在具有可比可调参数的情况下优于最近的基线。

Title: SQL-Encoder: Improving NL2SQL In-Context Learning Through a Context-Aware Encoder

• Authors: Mohammadreza Pourreza, Davood Rafiei, Yuxi Feng, Raymond Li, Zhenan Fan, Weiwei Zhang
• Subjects: cs.CL, cs.DB, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.16204
• Pdf URL: https://arxiv.org/pdf/2403.16204
• Copy Paste: [[2403.16204]] SQL-Encoder: Improving NL2SQL In-Context Learning Through a Context-Aware Encoder(https://arxiv.org/abs/2403.16204)
• Keywords: gpt
• Abstract: Detecting structural similarity between queries is essential for selecting examples in in-context learning models. However, assessing structural similarity based solely on the natural language expressions of queries, without considering SQL queries, presents a significant challenge. This paper explores the significance of this similarity metric and proposes a model for accurately estimating it. To achieve this, we leverage a dataset comprising 170k question pairs, meticulously curated to train a similarity prediction model. Our comprehensive evaluation demonstrates that the proposed model adeptly captures the structural similarity between questions, as evidenced by improvements in Kendall-Tau distance and precision@k metrics. Notably, our model outperforms strong competitive embedding models from OpenAI and Cohere. Furthermore, compared to these competitive models, our proposed encoder enhances the downstream performance of NL2SQL models in 1-shot in-context learning scenarios by 1-2\% for GPT-3.5-turbo, 4-8\% for CodeLlama-7B, and 2-3\% for CodeLlama-13B.
• 摘要：检测查询之间的结构相似性对于在上下文学习模型中选择示例至关重要。然而，仅根据查询的自然语言表达而不考虑 SQL 查询来评估结构相似性提出了重大挑战。本文探讨了这种相似性度量的重要性，并提出了一种准确估计它的模型。为了实现这一目标，我们利用包含 17 万个问题对的数据集，精心策划来训练相似性预测模型。我们的综合评估表明，所提出的模型巧妙地捕捉了问题之间的结构相似性，Kendall-Tau 距离和 precision@k 指标的改进证明了这一点。值得注意的是，我们的模型优于 OpenAI 和 Cohere 的强大竞争嵌入模型。此外，与这些竞争模型相比，我们提出的编码器将 GPT-3.5-turbo 的 NL2SQL 模型在 1-shot 上下文学习场景中的下游性能提高了 1-2\%，对于 CodeLlama-7B 提高了 4-8\%， CodeLlama-13B 为 2-3%。

Title: Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling

• Authors: Yida Mu, Chun Dong, Kalina Bontcheva, Xingyi Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16248
• Pdf URL: https://arxiv.org/pdf/2403.16248
• Copy Paste: [[2403.16248]] Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling(https://arxiv.org/abs/2403.16248)
• Keywords: language model, llm, prompt
• Abstract: Topic modelling, as a well-established unsupervised technique, has found extensive use in automatically detecting significant topics within a corpus of documents. However, classic topic modelling approaches (e.g., LDA) have certain drawbacks, such as the lack of semantic understanding and the presence of overlapping topics. In this work, we investigate the untapped potential of large language models (LLMs) as an alternative for uncovering the underlying topics within extensive text corpora. To this end, we introduce a framework that prompts LLMs to generate topics from a given set of documents and establish evaluation protocols to assess the clustering efficacy of LLMs. Our findings indicate that LLMs with appropriate prompts can stand out as a viable alternative, capable of generating relevant topic titles and adhering to human guidelines to refine and merge topics. Through in-depth experiments and evaluation, we summarise the advantages and constraints of employing LLMs in topic extraction.
• 摘要：主题建模作为一种成熟的无监督技术，已广泛用于自动检测文档语料库中的重要主题。然而，经典的主题建模方法（例如LDA）具有某些缺点，例如缺乏语义理解和存在重叠主题。在这项工作中，我们研究了大型语言模型（LLM）尚未开发的潜力，作为揭示广泛文本语料库中潜在主题的替代方案。为此，我们引入了一个框架，促使法学硕士从一组给定的文档中生成主题，并建立评估协议来评估法学硕士的聚类功效。我们的研究结果表明，具有适当提示的法学硕士可以作为可行的替代方案脱颖而出，能够生成相关的主题标题并遵循人类指南来完善和合并主题。通过深入的实验和评估，我们总结了采用LLM进行主题提取的优势和限制。

Title: LexDrafter: Terminology Drafting for Legislative Documents using Retrieval Augmented Generation

• Authors: Ashish Chouhan, Michael Gertz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16295
• Pdf URL: https://arxiv.org/pdf/2403.16295
• Copy Paste: [[2403.16295]] LexDrafter: Terminology Drafting for Legislative Documents using Retrieval Augmented Generation(https://arxiv.org/abs/2403.16295)
• Keywords: retrieval augmented generation
• Abstract: With the increase in legislative documents at the EU, the number of new terms and their definitions is increasing as well. As per the Joint Practical Guide of the European Parliament, the Council and the Commission, terms used in legal documents shall be consistent, and identical concepts shall be expressed without departing from their meaning in ordinary, legal, or technical language. Thus, while drafting a new legislative document, having a framework that provides insights about existing definitions and helps define new terms based on a document's context will support such harmonized legal definitions across different regulations and thus avoid ambiguities. In this paper, we present LexDrafter, a framework that assists in drafting Definitions articles for legislative documents using retrieval augmented generation (RAG) and existing term definitions present in different legislative documents. For this, definition elements are built by extracting definitions from existing documents. Using definition elements and RAG, a Definitions article can be suggested on demand for a legislative document that is being drafted. We demonstrate and evaluate the functionality of LexDrafter using a collection of EU documents from the energy domain. The code for LexDrafter framework is available at https://github.com/achouhan93/LexDrafter.
• 摘要：随着欧盟立法文件的增加，新术语及其定义的数量也在增加。根据欧洲议会、理事会和委员会的联合实用指南，法律文件中使用的术语应保持一致，相同的概念应在不偏离普通、法律或技术语言含义的情况下表达。因此，在起草新的立法文件时，拥有一个提供对现有定义的见解并帮助根据文件上下文定义新术语的框架将支持跨不同法规的统一法律定义，从而避免歧义。在本文中，我们提出了 LexDrafter，这是一个框架，可帮助使用检索增强生成 (RAG) 和不同立法文件中存在的现有术语定义起草立法文件的定义文章。为此，定义元素是通过从现有文档中提取定义来构建的。使用定义元素和 RAG，可以根据正在起草的立法文件的需要建议定义文章。我们使用能源领域的欧盟文件集合来演示和评估 LexDrafter 的功能。 LexDrafter 框架的代码可在 https://github.com/achouhan93/LexDrafter 获取。

Title: Enhanced Facet Generation with LLM Editing

• Authors: Joosung Lee, Jinhong Kim
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.16345
• Pdf URL: https://arxiv.org/pdf/2403.16345
• Copy Paste: [[2403.16345]] Enhanced Facet Generation with LLM Editing(https://arxiv.org/abs/2403.16345)
• Keywords: language model, llm
• Abstract: In information retrieval, facet identification of a user query is an important task. If a search service can recognize the facets of a user's query, it has the potential to offer users a much broader range of search results. Previous studies can enhance facet prediction by leveraging retrieved documents and related queries obtained through a search engine. However, there are challenges in extending it to other applications when a search engine operates as part of the model. First, search engines are constantly updated. Therefore, additional information may change during training and test, which may reduce performance. The second challenge is that public search engines cannot search for internal documents. Therefore, a separate search system needs to be built to incorporate documents from private domains within the company. We propose two strategies that focus on a framework that can predict facets by taking only queries as input without a search engine. The first strategy is multi-task learning to predict SERP. By leveraging SERP as a target instead of a source, the proposed model deeply understands queries without relying on external modules. The second strategy is to enhance the facets by combining Large Language Model (LLM) and the small model. Overall performance improves when small model and LLM are combined rather than facet generation individually.
• 摘要：在信息检索中，用户查询的方面识别是一项重要任务。如果搜索服务可以识别用户查询的各个方面，它就有可能为用户提供更广泛的搜索结果。先前的研究可以通过利用检索到的文档和通过搜索引擎获得的相关查询来增强方面预测。然而，当搜索引擎作为模型的一部分运行时，将其扩展到其他应用程序会遇到挑战。首先，搜索引擎不断更新。因此，附加信息可能会在训练和测试期间发生变化，这可能会降低性能。第二个挑战是公共搜索引擎无法搜索内部文档。因此，需要建立一个单独的搜索系统来合并来自公司内部私人领域的文档。我们提出了两种策略，重点关注一个框架，该框架可以通过仅将查询作为输入而不使用搜索引擎来预测方面。第一个策略是通过多任务学习来预测 SERP。通过利用 SERP 作为目标而不是源，所提出的模型无需依赖外部模块即可深入理解查询。第二个策略是通过结合大型语言模型（LLM）和小型模型来增强方面。当小模型和 LLM 相结合而不是单独生成构面时，整体性能会得到提高。

Title: Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases

• Authors: Wenhao Huang, Qianyu He, Zhixu Li, Jiaqing Liang, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16396
• Pdf URL: https://arxiv.org/pdf/2403.16396
• Copy Paste: [[2403.16396]] Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases(https://arxiv.org/abs/2403.16396)
• Keywords: language model
• Abstract: Definition bias is a negative phenomenon that can mislead models. Definition bias in information extraction appears not only across datasets from different domains but also within datasets sharing the same domain. We identify two types of definition bias in IE: bias among information extraction datasets and bias between information extraction datasets and instruction tuning datasets. To systematically investigate definition bias, we conduct three probing experiments to quantitatively analyze it and discover the limitations of unified information extraction and large language models in solving definition bias. To mitigate definition bias in information extraction, we propose a multi-stage framework consisting of definition bias measurement, bias-aware fine-tuning, and task-specific bias mitigation. Experimental results demonstrate the effectiveness of our framework in addressing definition bias. Resources of this paper can be found at https://github.com/EZ-hwh/definition-bias
• 摘要：定义偏差是一种负面现象，可能会误导模型。信息提取中的定义偏差不仅出现在不同域的数据集之间，而且还出现在共享同一域的数据集中。我们在 IE 中识别出两种类型的定义偏差：信息提取数据集之间的偏差以及信息提取数据集和指令调整数据集之间的偏差。为了系统地研究定义偏差，我们进行了三个探测实验来对其进行定量分析，并发现统一信息提取和大型语言模型在解决定义偏差方面的局限性。为了减轻信息提取中的定义偏差，我们提出了一个多阶段框架，包括定义偏差测量、偏差感知微调和特定于任务的偏差缓解。实验结果证明了我们的框架在解决定义偏差方面的有效性。本文的资源可以在 https://github.com/EZ-hwh/definition-bias 找到

Title: $\textit{LinkPrompt}$: Natural and Universal Adversarial Attacks on Prompt-based Language Models

• Authors: Yue Xu, Wenjie Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.16432
• Pdf URL: https://arxiv.org/pdf/2403.16432
• Copy Paste: [[2403.16432]] $\textit{LinkPrompt}$: Natural and Universal Adversarial Attacks on Prompt-based Language Models(https://arxiv.org/abs/2403.16432)
• Keywords: language model, gpt, llm, prompt
• Abstract: Prompt-based learning is a new language model training paradigm that adapts the Pre-trained Language Models (PLMs) to downstream tasks, which revitalizes the performance benchmarks across various natural language processing (NLP) tasks. Instead of using a fixed prompt template to fine-tune the model, some research demonstrates the effectiveness of searching for the prompt via optimization. Such prompt optimization process of prompt-based learning on PLMs also gives insight into generating adversarial prompts to mislead the model, raising concerns about the adversarial vulnerability of this paradigm. Recent studies have shown that universal adversarial triggers (UATs) can be generated to alter not only the predictions of the target PLMs but also the prediction of corresponding Prompt-based Fine-tuning Models (PFMs) under the prompt-based learning paradigm. However, UATs found in previous works are often unreadable tokens or characters and can be easily distinguished from natural texts with adaptive defenses. In this work, we consider the naturalness of the UATs and develop $\textit{LinkPrompt}$, an adversarial attack algorithm to generate UATs by a gradient-based beam search algorithm that not only effectively attacks the target PLMs and PFMs but also maintains the naturalness among the trigger tokens. Extensive results demonstrate the effectiveness of $\textit{LinkPrompt}$, as well as the transferability of UATs generated by \textit{LinkPrompt} to open-sourced Large Language Model (LLM) Llama2 and API-accessed LLM GPT-3.5-turbo.
• 摘要：基于提示的学习是一种新的语言模型训练范例，可将预训练语言模型 (PLM) 应用于下游任务，从而重振各种自然语言处理 (NLP) 任务的性能基准。一些研究证明了通过优化搜索提示的有效性，而不是使用固定的提示模板来微调模型。 PLM 上基于提示的学习的这种提示优化过程还提供了生成对抗性提示来误导模型的见解，引发了人们对该范式的对抗性脆弱性的担忧。最近的研究表明，可以生成通用对抗性触发器（UAT），不仅可以改变目标 PLM 的预测，还可以改变基于提示的学习范式下相应的基于提示的微调模型（PFM）的预测。然而，在之前的作品中发现的UAT通常是不可读的标记或字符，并且可以通过自适应防御轻松地与自然文本区分开来。在这项工作中，我们考虑了UAT的自然性，并开发了$\textit{LinkPrompt}$，这是一种对抗性攻击算法，通过基于梯度的波束搜索算法生成UAT，该算法不仅有效地攻击目标PLM和PFM，而且还保持了触发标记之间的自然性。大量结果证明了 $\textit{LinkPrompt}$ 的有效性，以及 \textit{LinkPrompt} 生成的 UAT 到开源大型语言模型 (LLM) Llama2 和 API 访问的 LLM GPT-3.5-turbo 的可转移性。

Title: InstUPR : Instruction-based Unsupervised Passage Reranking with Large Language Models

• Authors: Chao-Wei Huang, Yun-Nung Chen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.16435
• Pdf URL: https://arxiv.org/pdf/2403.16435
• Copy Paste: [[2403.16435]] InstUPR : Instruction-based Unsupervised Passage Reranking with Large Language Models(https://arxiv.org/abs/2403.16435)
• Keywords: language model, llm
• Abstract: This paper introduces InstUPR, an unsupervised passage reranking method based on large language models (LLMs). Different from existing approaches that rely on extensive training with query-document pairs or retrieval-specific instructions, our method leverages the instruction-following capabilities of instruction-tuned LLMs for passage reranking without any additional fine-tuning. To achieve this, we introduce a soft score aggregation technique and employ pairwise reranking for unsupervised passage reranking. Experiments on the BEIR benchmark demonstrate that InstUPR outperforms unsupervised baselines as well as an instruction-tuned reranker, highlighting its effectiveness and superiority. Source code to reproduce all experiments is open-sourced at https://github.com/MiuLab/InstUPR
• 摘要：本文介绍了 InstUPR，一种基于大型语言模型 (LLM) 的无监督段落重排序方法。与依赖于查询文档对或检索特定指令的广泛训练的现有方法不同，我们的方法利用指令调整的 LLM 的指令跟踪功能来进行段落重新排序，而无需任何额外的微调。为了实现这一目标，我们引入了软分数聚合技术，并采用成对重新排名来进行无监督的段落重新排名。 BEIR 基准测试表明，InstUPR 的性能优于无监督基线以及指令调整的重排序器，凸显了其有效性和优越性。重现所有实验的源代码在 https://github.com/MiuLab/InstUPR 上开源

Title: If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions

• Authors: Reza Esfandiarpoor, Cristina Menghini, Stephen H. Bach
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.16442
• Pdf URL: https://arxiv.org/pdf/2403.16442
• Copy Paste: [[2403.16442]] If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions(https://arxiv.org/abs/2403.16442)
• Keywords: language model
• Abstract: Recent works often assume that Vision-Language Model (VLM) representations are based on visual attributes like shape. However, it is unclear to what extent VLMs prioritize this information to represent concepts. We propose Extract and Explore (EX2), a novel approach to characterize important textual features for VLMs. EX2 uses reinforcement learning to align a large language model with VLM preferences and generates descriptions that incorporate the important features for the VLM. Then, we inspect the descriptions to identify the features that contribute to VLM representations. We find that spurious descriptions have a major role in VLM representations despite providing no helpful information, e.g., Click to enlarge photo of CONCEPT. More importantly, among informative descriptions, VLMs rely significantly on non-visual attributes like habitat to represent visual concepts. Also, our analysis reveals that different VLMs prioritize different attributes in their representations. Overall, we show that VLMs do not simply match images to scene descriptions and that non-visual or even spurious descriptions significantly influence their representations.
• 摘要：最近的工作通常假设视觉语言模型（VLM）表示基于形状等视觉属性。然而，尚不清楚 VLM 在多大程度上优先考虑这些信息来表示概念。我们提出了 Extract and Explore (EX2)，这是一种表征 VLM 重要文本特征的新颖方法。 EX2 使用强化学习将大型语言模型与 VLM 偏好保持一致，并生成包含 VLM 重要功能的描述。然后，我们检查描述以识别有助于 VLM 表示的特征。我们发现虚假描述在 VLM 表示中起着重要作用，尽管没有提供有用的信息，例如，单击放大概念的照片。更重要的是，在信息描述中，VLM 很大程度上依赖于栖息地等非视觉属性来表示视觉概念。此外，我们的分析表明，不同的 VLM 在其表示中优先考虑不同的属性。总体而言，我们表明 VLM 并不简单地将图像与场景描述进行匹配，非视觉甚至虚假描述会显着影响其表示。

Title: CodeS: Natural Language to Code Repository via Multi-Layer Sketch

• Authors: Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang, Lizhen Cui
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.16443
• Pdf URL: https://arxiv.org/pdf/2403.16443
• Copy Paste: [[2403.16443]] CodeS: Natural Language to Code Repository via Multi-Layer Sketch(https://arxiv.org/abs/2403.16443)
• Keywords: language model, llm
• Abstract: The impressive performance of large language models (LLMs) on code-related tasks has shown the potential of fully automated software development. In light of this, we introduce a new software engineering task, namely Natural Language to code Repository (NL2Repo). This task aims to generate an entire code repository from its natural language requirements. To address this task, we propose a simple yet effective framework CodeS, which decomposes NL2Repo into multiple sub-tasks by a multi-layer sketch. Specifically, CodeS includes three modules: RepoSketcher, FileSketcher, and SketchFiller. RepoSketcher first generates a repository's directory structure for given requirements; FileSketcher then generates a file sketch for each file in the generated structure; SketchFiller finally fills in the details for each function in the generated file sketch. To rigorously assess CodeS on the NL2Repo task, we carry out evaluations through both automated benchmarking and manual feedback analysis. For benchmark-based evaluation, we craft a repository-oriented benchmark, SketchEval, and design an evaluation metric, SketchBLEU. For feedback-based evaluation, we develop a VSCode plugin for CodeS and engage 30 participants in conducting empirical studies. Extensive experiments prove the effectiveness and practicality of CodeS on the NL2Repo task.
• 摘要：大型语言模型 (LLM) 在代码相关任务上的令人印象深刻的性能显示了完全自动化软件开发的潜力。鉴于此，我们引入了一个新的软件工程任务，即自然语言代码存储库（NL2Repo）。此任务旨在根据其自然语言需求生成整个代码存储库。为了解决这个任务，我们提出了一个简单而有效的框架CodeS，它通过多层草图将NL2Repo分解为多个子任务。具体来说，CodeS包括三个模块：RepoSketcher、FileSketcher和SketchFiller。 RepoSketcher 首先根据给定的需求生成存储库的目录结构；然后，FileSketcher 为生成的结构中的每个文件生成文件草图； SketchFiller 最终在生成的文件 sketch 中填写每个函数的详细信息。为了严格评估 CodeS 在 NL2Repo 任务上的表现，我们通过自动基准测试和手动反馈分析进行评估。对于基于基准的评估，我们制定了面向存储库的基准 SketchEval，并设计了评估指标 SketchBLEU。对于基于反馈的评估，我们为 CodeS 开发了 VSCode 插件，并邀请 30 名参与者进行实证研究。大量的实验证明了CodeS在NL2Repo任务上的有效性和实用性。

Title: KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models

• Authors: Dongjun Jang, Sungjoo Byun, Hyemi Jo, Hyopil Shin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16444
• Pdf URL: https://arxiv.org/pdf/2403.16444
• Copy Paste: [[2403.16444]] KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models(https://arxiv.org/abs/2403.16444)
• Keywords: language model, gpt, llm, chat
• Abstract: Instruction Tuning on Large Language Models is an essential process for model to function well and achieve high performance in specific tasks. Accordingly, in mainstream languages such as English, instruction-based datasets are being constructed and made publicly available. In the case of Korean, publicly available models and datasets all rely on using the output of ChatGPT or translating datasets built in English. In this paper, We introduce \textit{KIT-19} as an instruction dataset for the development of LLM in Korean. \textit{KIT-19} is a dataset created in an instruction format, comprising 19 existing open-source datasets for Korean NLP tasks. In this paper, we train a Korean Pretrained LLM using \textit{KIT-19} to demonstrate its effectiveness. The experimental results show that the model trained on \textit{KIT-19} significantly outperforms existing Korean LLMs. Based on the its quality and empirical results, this paper proposes that \textit{KIT-19} has the potential to make a substantial contribution to the future improvement of Korean LLMs' performance.
• 摘要：大型语言模型的指令调优是模型在特定任务中良好运行并获得高性能的重要过程。因此，在英语等主流语言中，正在构建基于指令的数据集并公开提供。就韩语而言，公开可用的模型和数据集都依赖于使用 ChatGPT 的输出或翻译用英语构建的数据集。在本文中，我们引入 \textit{KIT-19} 作为韩语 LLM 开发的指令数据集。 \textit{KIT-19} 是一个以指令格式创建的数据集，包含 19 个现有的韩国 NLP 任务开源数据集。在本文中，我们使用 \textit{KIT-19} 训练韩国预训练法学硕士以证明其有效性。实验结果表明，在 \textit{KIT-19} 上训练的模型显着优于现有的韩国法学硕士。基于其质量和实证结果，本文提出 \textit{KIT-19} 有潜力为韩国法学硕士未来的表现做出重大贡献。

Title: Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm

• Authors: Lei Liu, Xiaoyan Yang, Fangzhou Li, Chenfei Chi, Yue Shen, Shiwei Lyu Ming Zhang, Xiaowei Ma, Xiangguo Lyu, Liya Ma, Zhiqiang Zhang, Wei Xue, Yiran Huang, Jinjie Gu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16446
• Pdf URL: https://arxiv.org/pdf/2403.16446
• Copy Paste: [[2403.16446]] Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm(https://arxiv.org/abs/2403.16446)
• Keywords: language model, llm, hallucination, agent
• Abstract: Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis, owing to their unprecedented performance in modelling natural language. Ensuring the safe and reliable clinical applications, the evaluation of LLMs indeed becomes critical for better mitigating the potential risks, e.g., hallucinations. However, current evaluation methods heavily rely on labor-intensive human participation to achieve human-preferred judgements. To overcome this challenge, we propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services, e.g., disease diagnosis and treatment. The evaluation paradigm contains three basic elements: metric, data, and algorithm. Specifically, inspired by professional clinical practice pathways, we formulate a LLM-specific clinical pathway (LCP) to define the clinical capabilities that a doctor agent should possess. Then, Standardized Patients (SPs) from the medical education are introduced as the guideline for collecting medical data for evaluation, which can well ensure the completeness of the evaluation procedure. Leveraging these steps, we develop a multi-agent framework to simulate the interactive environment between SPs and a doctor agent, which is equipped with a Retrieval-Augmented Evaluation (RAE) to determine whether the behaviors of a doctor agent are in accordance with LCP. The above paradigm can be extended to any similar clinical scenarios to automatically evaluate the LLMs' medical capabilities. Applying such paradigm, we construct an evaluation benchmark in the field of urology, including a LCP, a SPs dataset, and an automated RAE. Extensive experiments are conducted to demonstrate the effectiveness of the proposed approach, providing more insights for LLMs' safe and reliable deployments in clinical practice.
• 摘要：由于大型语言模型 (LLM) 在自然语言建模方面具有前所未有的性能，因此在提高医学诊断的临床效率方面越来越受到人们的关注。为了确保临床应用的安全可靠，法学硕士的评估对于更好地减轻幻觉等潜在风险确实变得至关重要。然而，当前的评估方法严重依赖劳动密集型的人类参与来实现人类偏好的判断。为了克服这一挑战，我们提出了一种自动评估范式，旨在评估法学硕士提供临床服务（例如疾病诊断和治疗）的能力。评估范式包含三个基本要素：指标、数据和算法。具体来说，受专业临床实践路径的启发，我们制定了法学硕士特定的临床路径（LCP）来定义医生代理人应具备的临床能力。然后，引入医学教育中的标准化患者（SP）作为收集评估医学数据的指南，可以很好地保证评估程序的完整性。利用这些步骤，我们开发了一个多代理框架来模拟 SP 和医生代理之间的交互环境，该框架配备了检索增强评估（RAE）来确定医生代理的行为是否符合 LCP。上述范例可以扩展到任何类似的临床场景，以自动评估法学硕士的医疗能力。应用这种范式，我们构建了泌尿外科领域的评估基准，包括 LCP、SPs 数据集和自动化 RAE。进行了大量的实验来证明所提出方法的有效性，为法学硕士在临床实践中安全可靠的部署提供更多见解。

Title: LARA: Linguistic-Adaptive Retrieval-Augmented LLMs for Multi-Turn Intent Classification

• Authors: Liu Junhua, Tan Yong Keat, Fu Bin
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.16504
• Pdf URL: https://arxiv.org/pdf/2403.16504
• Copy Paste: [[2403.16504]] LARA: Linguistic-Adaptive Retrieval-Augmented LLMs for Multi-Turn Intent Classification(https://arxiv.org/abs/2403.16504)
• Keywords: language model, llm, chat
• Abstract: Following the significant achievements of large language models (LLMs), researchers have employed in-context learning for text classification tasks. However, these studies focused on monolingual, single-turn classification tasks. In this paper, we introduce LARA (Linguistic-Adaptive Retrieval-Augmented Language Models), designed to enhance accuracy in multi-turn classification tasks across six languages, accommodating numerous intents in chatbot interactions. Multi-turn intent classification is notably challenging due to the complexity and evolving nature of conversational contexts. LARA tackles these issues by combining a fine-tuned smaller model with a retrieval-augmented mechanism, integrated within the architecture of LLMs. This integration allows LARA to dynamically utilize past dialogues and relevant intents, thereby improving the understanding of the context. Furthermore, our adaptive retrieval techniques bolster the cross-lingual capabilities of LLMs without extensive retraining and fine-tune. Comprehensive experiments demonstrate that LARA achieves state-of-the-art performance on multi-turn intent classification tasks, enhancing the average accuracy by 3.67% compared to existing methods.
• 摘要：继大型语言模型（LLM）取得重大成就之后，研究人员将上下文学习用于文本分类任务。然而，这些研究侧重于单语言、单轮分类任务。在本文中，我们介绍了 LARA（语言自适应检索增强语言模型），旨在提高六种语言的多轮分类任务的准确性，适应聊天机器人交互中的众多意图。由于对话上下文的复杂性和不断变化的性质，多轮意图分类尤其具有挑战性。 LARA 通过将微调的较小模型与检索增强机制相结合来解决这些问题，并将其集成到法学硕士的架构中。这种集成使 LARA 能够动态地利用过去的对话和相关意图，从而提高对上下文的理解。此外，我们的自适应检索技术增强了法学硕士的跨语言能力，无需进行大量的再培训和微调。综合实验表明，LARA 在多轮意图分类任务上实现了最先进的性能，与现有方法相比，平均准确率提高了 3.67%。

Title: LLMs Are Few-Shot In-Context Low-Resource Language Learners

• Authors: Samuel Cahyawijaya, Holy Lovenia, Pascale Fung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.16512
• Pdf URL: https://arxiv.org/pdf/2403.16512
• Copy Paste: [[2403.16512]] LLMs Are Few-Shot In-Context Low-Resource Language Learners(https://arxiv.org/abs/2403.16512)
• Keywords: language model, llm
• Abstract: In-context learning (ICL) empowers large language models (LLMs) to perform diverse tasks in underrepresented languages using only short in-context information, offering a crucial avenue for narrowing the gap between high-resource and low-resource languages. Nonetheless, there is only a handful of works explored ICL for low-resource languages with most of them focusing on relatively high-resource languages, such as French and Spanish. In this work, we extensively study ICL and its cross-lingual variation (X-ICL) on 25 low-resource and 7 relatively higher-resource languages. Our study not only assesses the effectiveness of ICL with LLMs in low-resource languages but also identifies the shortcomings of in-context label alignment, and introduces a more effective alternative: query alignment. Moreover, we provide valuable insights into various facets of ICL for low-resource languages. Our study concludes the significance of few-shot in-context information on enhancing the low-resource understanding quality of LLMs through semantically relevant information by closing the language gap in the target language and aligning the semantics between the targeted low-resource and the high-resource language that the model is proficient in. Our work highlights the importance of advancing ICL research, particularly for low-resource languages.
• 摘要：上下文学习 (ICL) 使大型语言模型 (LLM) 能够仅使用简短的上下文信息以代表性不足的语言执行各种任务，为缩小高资源语言和低资源语言之间的差距提供了重要途径。尽管如此，只有少数作品针对低资源语言探索了 ICL，其中大多数集中于相对高资源语言，例如法语和西班牙语。在这项工作中，我们广泛研究了 25 种低资源语言和 7 种资源相对较高的语言的 ICL 及其跨语言变体 (X-ICL)。我们的研究不仅评估了 ICL 与 LLM 在低资源语言中的有效性，而且还确定了上下文标签对齐的缺点，并引入了更有效的替代方案：查询对齐。此外，我们还针对低资源语言的 ICL 的各个方面提供了宝贵的见解。我们的研究总结了少量上下文信息对于通过语义相关信息提高法学硕士的低资源理解质量的重要性，通过缩小目标语言中的语言差距并调整目标低资源和高资源之间的语义。我们的工作强调了推进 ICL 研究的重要性，特别是对于低资源语言。

Title: Efficient Information Extraction in Few-Shot Relation Classification through Contrastive Representation Learning

• Authors: Philipp Borchert, Jochen De Weerdt, Marie-Francine Moens
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.16543
• Pdf URL: https://arxiv.org/pdf/2403.16543
• Copy Paste: [[2403.16543]] Efficient Information Extraction in Few-Shot Relation Classification through Contrastive Representation Learning(https://arxiv.org/abs/2403.16543)
• Keywords: prompt
• Abstract: Differentiating relationships between entity pairs with limited labeled instances poses a significant challenge in few-shot relation classification. Representations of textual data extract rich information spanning the domain, entities, and relations. In this paper, we introduce a novel approach to enhance information extraction combining multiple sentence representations and contrastive learning. While representations in relation classification are commonly extracted using entity marker tokens, we argue that substantial information within the internal model representations remains untapped. To address this, we propose aligning multiple sentence representations, such as the [CLS] token, the [MASK] token used in prompting, and entity marker tokens. Our method employs contrastive learning to extract complementary discriminative information from these individual representations. This is particularly relevant in low-resource settings where information is scarce. Leveraging multiple sentence representations is especially effective in distilling discriminative information for relation classification when additional information, like relation descriptions, are not available. We validate the adaptability of our approach, maintaining robust performance in scenarios that include relation descriptions, and showcasing its flexibility to adapt to different resource constraints.
• 摘要：区分具有有限标记实例的实体对之间的关​​系对少样本关系分类提出了重大挑战。文本数据的表示提取跨越领域、实体和关系的丰富信息。在本文中，我们介绍了一种结合多句子表示和对比学习来增强信息提取的新方法。虽然关系分类中的表示通常使用实体标记标记来提取，但我们认为内部模型表示中的大量信息仍未开发。为了解决这个问题，我们建议对齐多个句子表示，例如 [CLS] 标记、提示中使用的 [MASK] 标记和实体标记标记。我们的方法采用对比学习从这些个体表征中提取互补的判别信息。这在信息匮乏的资源匮乏环境中尤其重要。当附加信息（如关系描述）不可用时，利用多个句子表示对于提取关系分类的判别信息特别有效。我们验证了我们的方法的适应性，在包含关系描述的场景中保持稳健的性能，并展示其适应不同资源限制的灵活性。

Title: NSINA: A News Corpus for Sinhala

• Authors: Hansi Hettiarachchi, Damith Premasiri, Lasitha Uyangodage, Tharindu Ranasinghe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.16571
• Pdf URL: https://arxiv.org/pdf/2403.16571
• Copy Paste: [[2403.16571]] NSINA: A News Corpus for Sinhala(https://arxiv.org/abs/2403.16571)
• Keywords: language model, llm
• Abstract: The introduction of large language models (LLMs) has advanced natural language processing (NLP), but their effectiveness is largely dependent on pre-training resources. This is especially evident in low-resource languages, such as Sinhala, which face two primary challenges: the lack of substantial training data and limited benchmarking datasets. In response, this study introduces NSINA, a comprehensive news corpus of over 500,000 articles from popular Sinhala news websites, along with three NLP tasks: news media identification, news category prediction, and news headline generation. The release of NSINA aims to provide a solution to challenges in adapting LLMs to Sinhala, offering valuable resources and benchmarks for improving NLP in the Sinhala language. NSINA is the largest news corpus for Sinhala, available up to date.
• 摘要：大语言模型（LLM）的引入具有先进的自然语言处理（NLP），但其有效性很大程度上依赖于预训练资源。这在僧伽罗语等资源匮乏的语言中尤其明显，这些语言面临两个主要挑战：缺乏大量的训练数据和有限的基准数据集。为此，本研究引入了 NSINA，这是一个包含来自热门僧伽罗新闻网站的超过 500,000 篇文章的综合新闻语料库，以及三个 NLP 任务：新闻媒体识别、新闻类别预测和新闻标题生成。 NSINA 的发布旨在为法学硕士适应僧伽罗语的挑战提供解决方案，为改进僧伽罗语 NLP 提供宝贵的资源和基准。 NSINA 是迄今为止最大的僧伽罗语新闻语料库。

Title: Can Large Language Models (or Humans) Distill Text?

• Authors: Nicolas Audinet de Pieuchon, Adel Daoud, Connor Thomas Jerzak, Moa Johansson, Richard Johansson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16584
• Pdf URL: https://arxiv.org/pdf/2403.16584
• Copy Paste: [[2403.16584]] Can Large Language Models (or Humans) Distill Text?(https://arxiv.org/abs/2403.16584)
• Keywords: language model, llm
• Abstract: We investigate the potential of large language models (LLMs) to distill text: to remove the textual traces of an undesired forbidden variable. We employ a range of LLMs with varying architectures and training approaches to distill text by identifying and removing information about the target variable while preserving other relevant signals. Our findings shed light on the strengths and limitations of LLMs in addressing the distillation and provide insights into the strategies for leveraging these models in computational social science investigations involving text data. In particular, we show that in the strong test of removing sentiment, the statistical association between the processed text and sentiment is still clearly detectable to machine learning classifiers post-LLM-distillation. Furthermore, we find that human annotators also struggle to distill sentiment while preserving other semantic content. This suggests there may be limited separability between concept variables in some text contexts, highlighting limitations of methods relying on text-level transformations and also raising questions about the robustness of distillation methods that achieve statistical independence in representation space if this is difficult for human coders operating on raw text to attain.
• 摘要：我们研究大型语言模型（LLM）提取文本的潜力：删除不需要的禁止变量的文本痕迹。我们采用一系列具有不同架构和训练方法的法学硕士，通过识别和删除有关目标变量的信息，同时保留其他相关信号来提取文本。我们的研究结果揭示了法学硕士在解决蒸馏方面的优势和局限性，并为在涉及文本数据的计算社会科学研究中利用这些模型的策略提供了见解。特别是，我们表明，在消除情感的强烈测试中，经过 LLM 蒸馏后的机器学习分类器仍然可以清楚地检测到处理后的文本和情感之间的统计关联。此外，我们发现人类注释者在保留其他语义内容的同时也很难提取情感。这表明在某些文本上下文中概念变量之间可能存在有限的可分离性，突出了依赖于文本级转换的方法的局限性，并且还提出了关于在表示空间中实现统计独立性的蒸馏方法的稳健性的问题，如果这对于人类编码员操作来说是困难的在原始文本上实现。

Title: TrustAI at SemEval-2024 Task 8: A Comprehensive Analysis of Multi-domain Machine Generated Text Detection Techniques

• Authors: Ashok Urlana, Aditya Saibewar, Bala Mallikarjunarao Garlapati, Charaka Vinayak Kumar, Ajeet Kumar Singh, Srinivasa Rao Chalamala
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16592
• Pdf URL: https://arxiv.org/pdf/2403.16592
• Copy Paste: [[2403.16592]] TrustAI at SemEval-2024 Task 8: A Comprehensive Analysis of Multi-domain Machine Generated Text Detection Techniques(https://arxiv.org/abs/2403.16592)
• Keywords: language model, llm
• Abstract: The Large Language Models (LLMs) exhibit remarkable ability to generate fluent content across a wide spectrum of user queries. However, this capability has raised concerns regarding misinformation and personal information leakage. In this paper, we present our methods for the SemEval2024 Task8, aiming to detect machine-generated text across various domains in both mono-lingual and multi-lingual contexts. Our study comprehensively analyzes various methods to detect machine-generated text, including statistical, neural, and pre-trained model approaches. We also detail our experimental setup and perform a in-depth error analysis to evaluate the effectiveness of these methods. Our methods obtain an accuracy of 86.9\% on the test set of subtask-A mono and 83.7\% for subtask-B. Furthermore, we also highlight the challenges and essential factors for consideration in future studies.
• 摘要：大型语言模型 (LLM) 表现出在广泛的用户查询中生成流畅内容的卓越能力。然而，这种功能引起了人们对错误信息和个人信息泄露的担忧。在本文中，我们介绍了 SemEval2024 Task8 的方法，旨在检测单语言和多语言上下文中各个领域的机器生成文本。我们的研究全面分析了检测机器生成文本的各种方法，包括统计、神经和预训练模型方法。我们还详细介绍了我们的实验设置并进行了深入的误差分析以评估这些方法的有效性。我们的方法在子任务 A mono 的测试集上获得了 86.9% 的准确率，在子任务 B 的测试集上获得了 83.7% 的准确率。此外，我们还强调了未来研究中需要考虑的挑战和基本因素。

Title: Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units

• Authors: Biswesh Mohapatra, Seemab Hassan, Laurent Romary, Justine Cassell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16609
• Pdf URL: https://arxiv.org/pdf/2403.16609
• Copy Paste: [[2403.16609]] Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units(https://arxiv.org/abs/2403.16609)
• Keywords: language model, agent
• Abstract: Successful conversations often rest on common understanding, where all parties are on the same page about the information being shared. This process, known as conversational grounding, is crucial for building trustworthy dialog systems that can accurately keep track of and recall the shared information. The proficiencies of an agent in grounding the conveyed information significantly contribute to building a reliable dialog system. Despite recent advancements in dialog systems, there exists a noticeable deficit in their grounding capabilities. Traum provided a framework for conversational grounding introducing Grounding Acts and Grounding Units, but substantial progress, especially in the realm of Large Language Models, remains lacking. To bridge this gap, we present the annotation of two dialog corpora employing Grounding Acts, Grounding Units, and a measure of their degree of grounding. We discuss our key findings during the annotation and also provide a baseline model to test the performance of current Language Models in categorizing the grounding acts of the dialogs. Our work aims to provide a useful resource for further research in making conversations with machines better understood and more reliable in natural day-to-day collaborative dialogs.
• 摘要：成功的对话通常依赖于共同的理解，各方对于所共享的信息达成共识。这个过程被称为对话基础，对于构建可以准确跟踪和回忆共享信息的值得信赖的对话系统至关重要。代理在基础所传达的信息方面的熟练程度对于构建可靠的对话系统有很大贡献。尽管对话系统最近取得了进步，但其基础能力仍存在明显缺陷。 Traum 为会话基础提供了一个框架，引入了基础行为和基础单元，但仍然缺乏实质性进展，特别是在大型语言模型领域。为了弥补这一差距，我们提出了使用基础行为、基础单元及其基础程度度量的两个对话语料库的注释。我们在注释过程中讨论了我们的主要发现，并提供了一个基线模型来测试当前语言模型在对对话的基础行为进行分类方面的性能。我们的工作旨在为进一步研究提供有用的资源，使与机器的对话在自然的日常协作对话中更容易理解和更可靠。

Title: Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts

• Authors: Rabindra Lamsal, Maria Rodriguez Read, Shanika Karunasekera
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16614
• Pdf URL: https://arxiv.org/pdf/2403.16614
• Copy Paste: [[2403.16614]] Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts(https://arxiv.org/abs/2403.16614)
• Keywords: language model
• Abstract: Tasks such as semantic search and clustering on crisis-related social media texts enhance our comprehension of crisis discourse, aiding decision-making and targeted interventions. Pre-trained language models have advanced performance in crisis informatics, but their contextual embeddings lack semantic meaningfulness. Although the CrisisTransformers family includes a sentence encoder to address the semanticity issue, it remains monolingual, processing only English texts. Furthermore, employing separate models for different languages leads to embeddings in distinct vector spaces, introducing challenges when comparing semantic similarities between multi-lingual texts. Therefore, we propose multi-lingual sentence encoders (CT-XLMR-SE and CT-mBERT-SE) that embed crisis-related social media texts for over 50 languages, such that texts with similar meanings are in close proximity within the same vector space, irrespective of language diversity. Results in sentence encoding and sentence matching tasks are promising, suggesting these models could serve as robust baselines when embedding multi-lingual crisis-related social media texts. The models are publicly available at: https://huggingface.co/crisistransformers.
• 摘要：与危机相关的社交媒体文本的语义搜索和聚类等任务增强了我们对危机话语的理解，帮助决策和有针对性的干预。预训练的语言模型在危机信息学方面具有先进的性能，但其上下文嵌入缺乏语义意义。尽管 CrisisTransformers 系列包含一个句子编码器来解决语义问题，但它仍然是单语言的，仅处理英语文本。此外，对不同语言采用单独的模型会导致嵌入到不同的向量空间中，从而在比较多语言文本之间的语义相似性时带来挑战。因此，我们提出了多语言句子编码器（CT-XLMR-SE 和 CT-mBERT-SE），嵌入 50 多种语言的与危机相关的社交媒体文本，使得具有相似含义的文本在同一向量空间内非常接近，与语言多样性无关。句子编码和句子匹配任务的结果很有希望，这表明这些模型在嵌入多语言危机相关社交媒体文本时可以作为可靠的基线。这些模型可在以下网址公开获取：https://huggingface.co/crisistransformers。

Title: Grammatical vs Spelling Error Correction: An Investigation into the Responsiveness of Transformer-based Language Models using BART and MarianMT

• Authors: Rohit Raju, Peeta Basa Pati, SA Gandheesh, Gayatri Sanjana Sannala, Suriya KS
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16655
• Pdf URL: https://arxiv.org/pdf/2403.16655
• Copy Paste: [[2403.16655]] Grammatical vs Spelling Error Correction: An Investigation into the Responsiveness of Transformer-based Language Models using BART and MarianMT(https://arxiv.org/abs/2403.16655)
• Keywords: language model
• Abstract: Text continues to remain a relevant form of representation for information. Text documents are created either in digital native platforms or through the conversion of other media files such as images and speech. While the digital native text is invariably obtained through physical or virtual keyboards, technologies such as OCR and speech recognition are utilized to transform the images and speech signals into text content. All these variety of mechanisms of text generation also introduce errors into the captured text. This project aims at analyzing different kinds of error that occurs in text documents. The work employs two of the advanced deep neural network-based language models, namely, BART and MarianMT, to rectify the anomalies present in the text. Transfer learning of these models with available dataset is performed to finetune their capacity for error correction. A comparative study is conducted to investigate the effectiveness of these models in handling each of the defined error categories. It is observed that while both models can bring down the erroneous sentences by 20+%, BART can handle spelling errors far better (24.6%) than grammatical errors (8.8%).
• 摘要：文本仍然是信息的相关表示形式。文本文档可以在数字原生平台中创建，也可以通过图像和语音等其他媒体文件的转换来创建。虽然数字原生文本总是通过物理或虚拟键盘获得，但利用 OCR 和语音识别等技术将图像和语音信号转换为文本内容。所有这些不同的文本生成机制也会在捕获的文本中引入错误。该项目旨在分析文本文档中发生的不同类型的错误。该工作采用了两种先进的基于深度神经网络的语言模型，即 BART 和 MarianMT，来纠正文本中存在的异常情况。使用可用数据集对这些模型进行迁移学习，以微调其纠错能力。进行了比较研究，以调查这些模型在处理每个定义的错误类别方面的有效性。据观察，虽然两种模型都能将错误句子减少 20% 以上，但 BART 处理拼写错误 (24.6%) 的能力远远好于语法错误 (8.8%)。

Title: RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict

• Authors: Yirong Zeng, Xiao Ding, Yi Zhao, Xiangyu Li, Jie Zhang, Chao Yao, Ting Liu, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16662
• Pdf URL: https://arxiv.org/pdf/2403.16662
• Copy Paste: [[2403.16662]] RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict(https://arxiv.org/abs/2403.16662)
• Keywords: language model
• Abstract: Fact-checking is the task of verifying the factuality of a given claim by examining the available evidence. High-quality evidence plays a vital role in enhancing fact-checking systems and facilitating the generation of explanations that are understandable to humans. However, the provision of both sufficient and relevant evidence for explainable fact-checking systems poses a challenge. To tackle this challenge, we propose a method based on a Large Language Model to automatically retrieve and summarize evidence from the Web. Furthermore, we construct RU22Fact, a novel multilingual explainable fact-checking dataset on the Russia-Ukraine conflict in 2022 of 16K samples, each containing real-world claims, optimized evidence, and referenced explanation. To establish a baseline for our dataset, we also develop an end-to-end explainable fact-checking system to verify claims and generate explanations. Experimental results demonstrate the prospect of optimized evidence in increasing fact-checking performance and also indicate the possibility of further progress in the end-to-end claim verification and explanation generation tasks.
• 摘要：事实核查是通过检查现有证据来验证给定主张的真实性的任务。高质量的证据在增强事实核查系统和促进生成人类可以理解的解释方面发挥着至关重要的作用。然而，为可解释的事实核查系统提供充分且相关的证据提出了挑战。为了应对这一挑战，我们提出了一种基于大型语言模型的方法，可以自动检索和总结网络证据。此外，我们构建了 RU22Fact，这是一个关于 2022 年俄罗斯-乌克兰冲突的新型多语言可解释事实核查数据集，包含 16K 个样本，每个样本都包含真实世界的主张、优化的证据和参考解释。为了为我们的数据集建立基线，我们还开发了一个端到端可解释的事实检查系统来验证声明并生成解释。实验结果证明了优化证据在提高事实核查性能方面的前景，也表明了在端到端声明验证和解释生成任务方面取得进一步进展的可能性。

Title: ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search

• Authors: Zehan Li, Jianfei Zhang, Chuantao Yin, Yuanxin Ouyang, Wenge Rong
• Subjects: cs.CL, cs.IR, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.16702
• Pdf URL: https://arxiv.org/pdf/2403.16702
• Copy Paste: [[2403.16702]] ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search(https://arxiv.org/abs/2403.16702)
• Keywords: language model
• Abstract: Retrieval-based code question answering seeks to match user queries in natural language to relevant code snippets. Previous approaches typically rely on pretraining models using crafted bi-modal and uni-modal datasets to align text and code representations. In this paper, we introduce ProCQA, a large-scale programming question answering dataset extracted from the StackOverflow community, offering naturally structured mixed-modal QA pairs. To validate its effectiveness, we propose a modality-agnostic contrastive pre-training approach to improve the alignment of text and code representations of current code language models. Compared to previous models that primarily employ bimodal and unimodal pairs extracted from CodeSearchNet for pre-training, our model exhibits significant performance improvements across a wide range of code retrieval benchmarks.
• 摘要：基于检索的代码问答旨在将自然语言中的用户查询与相关代码片段进行匹配。以前的方法通常依赖于使用精心设计的双模态和单模态数据集来对齐文本和代码表示的预训练模型。在本文中，我们介绍了 ProCQA，这是一个从 StackOverflow 社区提取的大规模编程问答数据集，提供自然结构化的混合模式 QA 对。为了验证其有效性，我们提出了一种与模态无关的对比预训练方法，以改善当前代码语言模型的文本和代码表示的对齐。与之前主要使用从 CodeSearchNet 中提取的双峰和单峰对进行预训练的模型相比，我们的模型在各种代码检索基准测试中都表现出了显着的性能改进。

Title: Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback

• Authors: Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Xuanhua Shi, Hai Jin
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.16792
• Pdf URL: https://arxiv.org/pdf/2403.16792
• Copy Paste: [[2403.16792]] Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback(https://arxiv.org/abs/2403.16792)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large language models (LLMs) have shown remarkable progress in automated code generation. Yet, incorporating LLM-based code generation into real-life software projects poses challenges, as the generated code may contain errors in API usage, class, data structure, or missing project-specific information. As much of this project-specific context cannot fit into the prompts of LLMs, we must find ways to allow the model to explore the project-level code context. To this end, this paper puts forward a novel approach, termed ProCoder, which iteratively refines the project-level code context for precise code generation, guided by the compiler feedback. In particular, ProCoder first leverages compiler techniques to identify a mismatch between the generated code and the project's context. It then iteratively aligns and fixes the identified errors using information extracted from the code repository. We integrate ProCoder with two representative LLMs, i.e., GPT-3.5-Turbo and Code Llama (13B), and apply it to Python code generation. Experimental results show that ProCoder significantly improves the vanilla LLMs by over 80% in generating code dependent on project context, and consistently outperforms the existing retrieval-based code generation baselines.
• 摘要：大型语言模型（LLM）在自动代码生成方面取得了显着的进步。然而，将基于 LLM 的代码生成合并到现实软件项目中会带来挑战，因为生成的代码可能包含 API 使用、类、数据结构方面的错误或缺少项目特定信息。由于许多特定于项目的上下文无法适应法学硕士的提示，因此我们必须找到允许模型探索项目级代码上下文的方法。为此，本文提出了一种名为 ProCoder 的新颖方法，该方法在编译器反馈的指导下迭代地细化项目级代码上下文，以实现精确的代码生成。特别是，ProCoder 首先利用编译器技术来识别生成的代码与项目上下文之间的不匹配。然后，它使用从代码存储库中提取的信息迭代地对齐和修复已识别的错误。我们将 ProCoder 与两个代表性的 LLM，即 GPT-3.5-Turbo 和 Code Llama (13B) 集成，并将其应用于 Python 代码生成。实验结果表明，ProCoder 在根据项目上下文生成代码方面将普通法学硕士显着提高了 80% 以上，并且始终优于现有的基于检索的代码生成基线。

Title: An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing

• Authors: Ziwei Chai, Guoyin Wang, Jing Su, Tianjie Zhang, Xuanwen Huang, Xuwu Wang, Jingjing Xu, Jianbo Yuan, Hongxia Yang, Fei Wu, Yang Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.16854
• Pdf URL: https://arxiv.org/pdf/2403.16854
• Copy Paste: [[2403.16854]] An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing(https://arxiv.org/abs/2403.16854)
• Keywords: llm
• Abstract: We present Expert-Token-Routing, a unified generalist framework that facilitates seamless integration of multiple expert LLMs. Our framework represents expert LLMs as special expert tokens within the vocabulary of a meta LLM. The meta LLM can route to an expert LLM like generating new tokens. Expert-Token-Routing not only supports learning the implicit expertise of expert LLMs from existing instruction dataset but also allows for dynamic extension of new expert LLMs in a plug-and-play manner. It also conceals the detailed collaboration process from the user's perspective, facilitating interaction as though it were a singular LLM. Our framework outperforms various existing multi-LLM collaboration paradigms across benchmarks that incorporate six diverse expert domains, demonstrating effectiveness and robustness in building generalist LLM system via synergizing multiple expert LLMs.
• 摘要：我们推出 Expert-Token-Routing，这是一个统一的通才框架，有助于多个专家 LLM 的无缝集成。我们的框架将专家 LLM 表示为元 LLM 词汇表中的特殊专家标记。元 LLM 可以路由到专家 LLM，就像生成新令牌一样。专家令牌路由不仅支持从现有指令数据集中学习专家LLM的隐式专业知识，而且还允许以即插即用的方式动态扩展新的专家LLM。它还从用户的角度隐藏了详细的协作过程，促进了交互，就好像它是一个单一的法学硕士一样。我们的框架在包含六个不同专家领域的基准中优于各种现有的多法学硕士协作范例，展示了通过协同多个专家法学硕士构建通才法学硕士系统的有效性和稳健性。

Title: Encoding of lexical tone in self-supervised models of spoken language

• Authors: Gaofei Shen, Michaela Watkins, Afra Alishahi, Arianna Bisazza, Grzegorz Chrupała
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.16865
• Pdf URL: https://arxiv.org/pdf/2403.16865
• Copy Paste: [[2403.16865]] Encoding of lexical tone in self-supervised models of spoken language(https://arxiv.org/abs/2403.16865)
• Keywords: language model
• Abstract: Interpretability research has shown that self-supervised Spoken Language Models (SLMs) encode a wide variety of features in human speech from the acoustic, phonetic, phonological, syntactic and semantic levels, to speaker characteristics. The bulk of prior research on representations of phonology has focused on segmental features such as phonemes; the encoding of suprasegmental phonology (such as tone and stress patterns) in SLMs is not yet well understood. Tone is a suprasegmental feature that is present in more than half of the world's languages. This paper aims to analyze the tone encoding capabilities of SLMs, using Mandarin and Vietnamese as case studies. We show that SLMs encode lexical tone to a significant degree even when they are trained on data from non-tonal languages. We further find that SLMs behave similarly to native and non-native human participants in tone and consonant perception studies, but they do not follow the same developmental trajectory.
• 摘要：可解释性研究表明，自监督口语模型 (SLM) 编码人类语音中的各种特征，从声学、语音、音系、句法和语义级别到说话者特征。先前关于音韵学表征的大部分研究都集中在音素等分段特征上。 SLM 中超音段音系（例如声调和重音模式）的编码尚未得到很好的理解。声调是一种超语段特征，存在于世界上一半以上的语言中。本文旨在以普通话和越南语作为案例研究来分析 SLM 的音调编码能力。我们表明，即使 SLM 接受非声调语言的数据训练，它们也会在很大程度上对词汇声调进行编码。我们进一步发现，在声调和辅音感知研究中，SLM 的行为与本地和非本地人类参与者相似，但它们并不遵循相同的发展轨迹。

Title: New Intent Discovery with Attracting and Dispersing Prototype

• Authors: Shun Zhang, Jian Yang, Jiaqi Bai, Chaoran Yan, Tongliang Li, Zhao Yan, Zhoujun Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16913
• Pdf URL: https://arxiv.org/pdf/2403.16913
• Copy Paste: [[2403.16913]] New Intent Discovery with Attracting and Dispersing Prototype(https://arxiv.org/abs/2403.16913)
• Keywords: language model
• Abstract: New Intent Discovery (NID) aims to recognize known and infer new intent categories with the help of limited labeled and large-scale unlabeled data. The task is addressed as a feature-clustering problem and recent studies augment instance representation. However, existing methods fail to capture cluster-friendly representations, since they show less capability to effectively control and coordinate within-cluster and between-cluster distances. Tailored to the NID problem, we propose a Robust and Adaptive Prototypical learning (RAP) framework for globally distinct decision boundaries for both known and new intent categories. Specifically, a robust prototypical attracting learning (RPAL) method is designed to compel instances to gravitate toward their corresponding prototype, achieving greater within-cluster compactness. To attain larger between-cluster separation, another adaptive prototypical dispersing learning (APDL) method is devised to maximize the between-cluster distance from the prototype-to-prototype perspective. Experimental results evaluated on three challenging benchmarks (CLINC, BANKING, and StackOverflow) of our method with better cluster-friendly representation demonstrate that RAP brings in substantial improvements over the current state-of-the-art methods (even large language model) by a large margin (average +5.5% improvement).
• 摘要：新意图发现（NID）旨在借助有限的标记和大规模未标记数据来识别已知并推断新的意图类别。该任务被视为特征聚类问题，最近的研究增强了实例表示。然而，现有方法无法捕获集群友好的表示，因为它们有效控制和协调集群内和集群间距离的能力较差。针对 NID 问题，我们提出了一个鲁棒自适应原型学习 (RAP) 框架，用于已知和新意图类别的全局不同决策边界。具体来说，设计了一种鲁棒的原型吸引学习（RPAL）方法来迫使实例倾向于其相应的原型，从而实现更大的簇内紧凑性。为了获得更大的簇间分离，设计了另一种自适应原型分散学习（APDL）方法，以从原型到原型的角度最大化簇间距离。在我们的方法的三个具有挑战性的基准（CLINC、BANKING 和 StackOverflow）上评估的实验结果具有更好的集群友好表示，表明 RAP 比当前最先进的方法（甚至大型语言模型）带来了实质性改进大幅提升（平均提高 5.5%）。

Title: SPACE-IDEAS: A Dataset for Salient Information Detection in Space Innovation

• Authors: Andrés García-Silva, Cristian Berrío, José Manuel Gómez-Pérez
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2403.16941
• Pdf URL: https://arxiv.org/pdf/2403.16941
• Copy Paste: [[2403.16941]] SPACE-IDEAS: A Dataset for Salient Information Detection in Space Innovation(https://arxiv.org/abs/2403.16941)
• Keywords: language model
• Abstract: Detecting salient parts in text using natural language processing has been widely used to mitigate the effects of information overflow. Nevertheless, most of the datasets available for this task are derived mainly from academic publications. We introduce SPACE-IDEAS, a dataset for salient information detection from innovation ideas related to the Space domain. The text in SPACE-IDEAS varies greatly and includes informal, technical, academic and business-oriented writing styles. In addition to a manually annotated dataset we release an extended version that is annotated using a large generative language model. We train different sentence and sequential sentence classifiers, and show that the automatically annotated dataset can be leveraged using multitask learning to train better classifiers.
• 摘要：使用自然语言处理检测文本中的显着部分已被广泛用于减轻信息溢出的影响。然而，可用于此任务的大多数数据集主要来自学术出版物。我们介绍 SPACE-IDEAS，这是一个用于从与空间领域相关的创新想法中检测显着信息的数据集。 SPACE-IDEAS 中的文本差异很大，包括非正式、技术、学术和商业导向的写作风格。除了手动注释的数据集之外，我们还发布了使用大型生成语言模型注释的扩展版本。我们训练了不同的句子和顺序句子分类器，并表明可以使用多任务学习来利用自动注释的数据集来训练更好的分类器。

Title: Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

• Authors: Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen, Nigel Collier
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.16950
• Pdf URL: https://arxiv.org/pdf/2403.16950
• Copy Paste: [[2403.16950]] Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators(https://arxiv.org/abs/2403.16950)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human judgement, revealing that existing calibration methods aimed at mitigating biases are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PAIRS), an uncertainty-guided search method that employs LLMs to conduct pairwise comparisons and efficiently ranks candidate texts. PAIRS achieves state-of-the-art performance on representative evaluation tasks and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the transitivity of LLMs and demonstrate how PAIRS benefits from calibration.
• 摘要：大型语言模型 (LLM) 作为自动评估器在评估生成的自然语言的质量方面表现出了良好的能力。然而，法学硕士在评估中仍然存在偏见，并且常常难以生成与人类评估一致的连贯评估。在这项工作中，我们首先对LLM评估者与人类判断之间的偏差进行了系统研究，揭示了现有旨在减轻偏差的校准方法不足以有效地调整LLM评估者。受 RLHF 中偏好数据使用的启发，我们将评估制定为排名问题，并引入配对偏好搜索（PAIRS），这是一种不确定性引导的搜索方法，利用 LLM 进行成对比较并有效地对候选文本进行排名。 PAIRS 在代表性评估任务上实现了最先进的性能，并比直接评分有了显着改进。此外，我们深入探讨了成对偏好在量化法学硕士传递性中的作用，并展示了配对如何从校准中受益。

Title: Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

• Authors: Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, Xipeng Qiu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.16952
• Pdf URL: https://arxiv.org/pdf/2403.16952
• Copy Paste: [[2403.16952]] Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance(https://arxiv.org/abs/2403.16952)
• Keywords: language model
• Abstract: Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules
• 摘要：大型语言模型的预训练数据由多个领域（例如网络文本、学术论文、代码）组成，其混合比例对结果模型的能力至关重要。虽然现有的努力依赖于启发式或定性策略来调整比例，但我们发现关于函数形式的混合比例的模型性能的定量可预测性，我们将其称为数据混合定律。在样本混合物上拟合此类函数可以在实际运行之前揭示模型在未见过的混合物上的性能，从而指导选择理想的数据混合物。此外，我们建议嵌套使用训练步骤的缩放法则、模型大小和我们的数据混合法则，以便能够预测在各种混合下的海量数据上训练的大型模型的性能，仅需要小规模的训练。此外，实验结果验证了我们的方法有效地优化了在 RedPajama 中训练 100B 令牌的 1B 模型的训练混合物，其性能与在默认混合物上多训练 48% 的步骤的性能相当。将数据混合定律的应用扩展到持续训练，可以准确预测关键混合比例，避免灾难性遗忘，并展望动态数据计划的潜力

Title: Evaluating Shortest Edit Script Methods for Contextual Lemmatization

• Authors: Olia Toporkov, Rodrigo Agerri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16968
• Pdf URL: https://arxiv.org/pdf/2403.16968
• Copy Paste: [[2403.16968]] Evaluating Shortest Edit Script Methods for Contextual Lemmatization(https://arxiv.org/abs/2403.16968)
• Keywords: language model
• Abstract: Modern contextual lemmatizers often rely on automatically induced Shortest Edit Scripts (SES), namely, the number of edit operations to transform a word form into its lemma. In fact, different methods of computing SES have been proposed as an integral component in the architecture of several state-of-the-art contextual lemmatizers currently available. However, previous work has not investigated the direct impact of SES in the final lemmatization performance. In this paper we address this issue by focusing on lemmatization as a token classification task where the only input that the model receives is the word-label pairs in context, where the labels correspond to previously induced SES. Thus, by modifying in our lemmatization system only the SES labels that the model needs to learn, we may then objectively conclude which SES representation produces the best lemmatization results. We experiment with seven languages of different morphological complexity, namely, English, Spanish, Basque, Russian, Czech, Turkish and Polish, using multilingual and language-specific pre-trained masked language encoder-only models as a backbone to build our lemmatizers. Comprehensive experimental results, both in- and out-of-domain, indicate that computing the casing and edit operations separately is beneficial overall, but much more clearly for languages with high-inflected morphology. Notably, multilingual pre-trained language models consistently outperform their language-specific counterparts in every evaluation setting.
• 摘要：现代上下文词形还原器通常依赖于自动诱导的最短编辑脚本（SES），即将单词形式转换为其词条的编辑操作数量。事实上，计算 SES 的不同方法已经被提出作为当前可用的几种最先进的上下文词形还原器体系结构中的组成部分。然而，之前的工作并没有研究SES对最终词形还原性能的直接影响。在本文中，我们通过将词形还原作为标记分类任务来解决这个问题，其中模型接收的唯一输入是上下文中的单词标签对，其中标签对应于先前引入的 SES。因此，通过在我们的词形还原系统中仅修改模型需要学习的 SES 标签，我们就可以客观地得出哪种 SES 表示产生最佳词形还原结果的结论。我们对七种不同形态复杂性的语言进行了实验，即英语、西班牙语、巴斯克语、俄语、捷克语、土耳其语和波兰语，使用多语言和特定于语言的预训练掩码语言编码器模型作为构建我们的词形还原器的骨干。域内和域外的综合实验结果表明，单独计算大小写和编辑操作总体上是有益的，但对于具有高屈折形态的语言来说更明显。值得注意的是，多语言预训练语言模型在每个评估环境中始终优于特定语言的模型。

Title: A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course

• Authors: Will Yeadon, Alex Peach, Craig P. Testrow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.16977
• Pdf URL: https://arxiv.org/pdf/2403.16977
• Copy Paste: [[2403.16977]] A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course(https://arxiv.org/abs/2403.16977)
• Keywords: gpt, prompt, chat
• Abstract: This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed $n = 300$ data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = $2.482 \times 10^{-10}$). Prompt engineering significantly improved scores for both GPT-4 (p = $1.661 \times 10^{-4}$) and GPT-3.5 (p = $4.967 \times 10^{-9}$). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from `Definitely AI' to `Definitely Human'. They accurately identified the authorship, with 92.1% of the work categorized as 'Definitely Human' being human-authored. Simplifying this to a binary `AI' or `Human' categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students' work, it often remains detectable by human evaluators.
• 摘要：本研究评估了 ChatGPT 变体、GPT-3.5 和 GPT-4 的性能，无论有没有即时工程，都针对仅学生作业以及使用 Python 的大学级物理编码作业中包含学生和 GPT-4 贡献的混合类别。语言。将 50 份学生提交的内容与 50 份人工智能生成的不同类别的提交内容进行比较，并由三个独立标记进行盲目标记，我们收集了 $n = 300$ 数据点。学生的平均得分为 91.9% (SE:0.4)，超过了表现最高的 AI 提交类别 GPT-4，其得分为 81.1% (SE:0.8) - 具有统计显着性差异 (p = $2.482 \times 10^{-10 }$）。即时工程显着提高了 GPT-4 (p = $1.661 \times 10^{-4}$) 和 GPT-3.5 (p = $4.967 \times 10^{-9}$) 的分数。此外，盲评员的任务是按照从“绝对是人工智能”到“绝对是人类”的四点李克特量表来猜测提交内容的作者身份。他们准确地确定了作者身份，被归类为“绝对是人类”的作品中有 92.1% 是人类创作的。将其简化为“人工智能”或“人类”二元分类，平均准确率为 85.3%。这些发现表明，虽然人工智能生成的作业非常接近大学生作业的质量，但它通常仍然可以被人类评估者检测到。

Title: Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows

• Authors: Shujian Zhang, Lemeng Wu, Chengyue Gong, Xingchao Liu
• Subjects: cs.CL, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.16995
• Pdf URL: https://arxiv.org/pdf/2403.16995
• Copy Paste: [[2403.16995]] Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows(https://arxiv.org/abs/2403.16995)
• Keywords: language model
• Abstract: Recent works have demonstrated success in controlling sentence attributes ($e.g.$, sentiment) and structure ($e.g.$, syntactic structure) based on the diffusion language model. A key component that drives theimpressive performance for generating high-quality samples from noise is iteratively denoise for thousands of steps. While beneficial, the complexity of starting from the noise and the learning steps has limited its implementation to many NLP real-world applications. This paper proposes Language Rectified Flow ({\ours}). Our method is based on the reformulation of the standard probabilistic flow models. Language rectified flow learns (neural) ordinary differential equation models to transport between the source distribution and the target distribution, hence providing a unified and effective solution to generative modeling and domain transfer. From the source distribution, our language rectified flow yields fast simulation and effectively decreases the inference time. Experiments on three challenging fine-grained control tasks and multiple high-quality text editing show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
• 摘要：最近的工作证明了基于扩散语言模型在控制句子属性（$例如$，情感）和结构（$例如$，句法结构）方面取得了成功。驱动从噪声生成高质量样本的令人印象深刻的性能的关键组件是数千步的迭代降噪。虽然有益，但从噪声和学习步骤开始的复杂性限制了其在许多 NLP 实际应用中的实现。本文提出了语言校正流（{\ours}）。我们的方法基于标准概率流模型的重新表述。语言校正流学习（神经）常微分方程模型在源分布和目标分布之间传输，从而为生成建模和域转移提供统一且有效的解决方案。从源分布来看，我们的语言修正流程可以实现快速模拟并有效减少推理时间。对三个具有挑战性的细粒度控制任务和多个高质量文本编辑的实验表明，我们的方法始终优于其基线。大量的实验和消融研究表明，我们的方法对于许多 NLP 任务来说是通用的、有效的且有益的。