2024-03-12

Title: Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification

• Authors: Zhijian Li, Stefan Larson, Kevin Leach
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05640
• Pdf URL: https://arxiv.org/pdf/2403.05640
• Copy Paste: [[2403.05640]] Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification(https://arxiv.org/abs/2403.05640)
• Keywords: gpt, chat
• Abstract: Intent classifiers must be able to distinguish when a user's utterance does not belong to any supported intent to avoid producing incorrect and unrelated system responses. Although out-of-scope (OOS) detection for intent classifiers has been studied, previous work has not yet studied changes in classifier performance against hard-negative out-of-scope utterances (i.e., inputs that share common features with in-scope data, but are actually out-of-scope). We present an automated technique to generate hard-negative OOS data using ChatGPT. We use our technique to build five new hard-negative OOS datasets, and evaluate each against three benchmark intent classifiers. We show that classifiers struggle to correctly identify hard-negative OOS utterances more than general OOS utterances. Finally, we show that incorporating hard-negative OOS data for training improves model robustness when detecting hard-negative OOS data and general OOS data. Our technique, datasets, and evaluation address an important void in the field, offering a straightforward and inexpensive way to collect hard-negative OOS data and improve intent classifiers' robustness.
• 摘要：意图分类器必须能够区分用户的话语何时不属于任何支持的意图，以避免产生不正确和不相关的系统响应。尽管已经研究了意图分类器的范围外（OOS）检测，但之前的工作尚未研究针对硬负范围外话语（即与范围内数据共享共同特征的输入）分类器性能的变化，但实际上超出了范围）。我们提出了一种使用 ChatGPT 生成硬负 OOS 数据的自动化技术。我们使用我们的技术构建了五个新的硬负 OOS 数据集，并根据三个基准意图分类器对每个数据集进行了评估。我们表明，与一般的 OOS 话语相比，分类器更难以正确识别硬否定 OOS 话语。最后，我们表明，在检测硬负 OOS 数据和一般 OOS 数据时，结合硬负 OOS 数据进行训练可以提高模型的鲁棒性。我们的技术、数据集和评估解决了该领域的一个重要空白，提供了一种简单且廉价的方法来收集硬阴性 OOS 数据并提高意图分类器的稳健性。

Title: PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design

• Authors: Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, Tim Kraska
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05676
• Pdf URL: https://arxiv.org/pdf/2403.05676
• Copy Paste: [[2403.05676]] PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design(https://arxiv.org/abs/2403.05676)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) can enhance the generation quality of large language models (LLMs) by incorporating external token databases. However, retrievals from large databases can constitute a substantial portion of the overall generation time, particularly when retrievals are periodically performed to align the retrieved content with the latest states of generation. In this paper, we introduce PipeRAG, a novel algorithm-system co-design approach to reduce generation latency and enhance generation quality. PipeRAG integrates (1) pipeline parallelism to enable concurrent retrieval and generation processes, (2) flexible retrieval intervals to maximize the efficiency of pipeline parallelism, and (3) a performance model to automatically balance retrieval quality and latency based on the generation states and underlying hardware. Our evaluation shows that, by combining the three aforementioned methods, PipeRAG achieves up to 2.6$\times$ speedup in end-to-end generation latency while improving generation quality. These promising results showcase the effectiveness of co-designing algorithms with underlying systems, paving the way for the adoption of PipeRAG in future RAG systems.
• 摘要：检索增强生成（RAG）可以通过合并外部标记数据库来提高大型语言模型（LLM）的生成质量。然而，从大型数据库的检索可能占整个生成时间的很大一部分，特别是当定期执行检索以使检索到的内容与生成的最新状态对齐时。在本文中，我们介绍了 PipeRAG，这是一种新颖的算法系统协同设计方法，可减少生成延迟并提高生成质量。 PipeRAG 集成了 (1) 管道并行性，以实现并发检索和生成过程；(2) 灵活的检索间隔，以最大限度地提高管道并行性的效率；以及 (3) 性能模型，可根据生成状态和底层自动平衡检索质量和延迟硬件。我们的评估表明，通过结合上述三种方法，PipeRAG 在端到端生成延迟方面实现了高达 2.6$\times$ 的加速，同时提高了生成质量。这些有希望的结果展示了与底层系统共同设计算法的有效性，为未来 RAG 系统中采用 PipeRAG 铺平了道路。

Title: SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes

• Authors: Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, Sunipa Dev
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05696
• Pdf URL: https://arxiv.org/pdf/2403.05696
• Copy Paste: [[2403.05696]] SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes(https://arxiv.org/abs/2403.05696)
• Keywords: llm
• Abstract: While generative multilingual models are rapidly being deployed, their safety and fairness evaluations are largely limited to resources collected in English. This is especially problematic for evaluations targeting inherently socio-cultural phenomena such as stereotyping, where it is important to build multi-lingual resources that reflect the stereotypes prevalent in respective language communities. However, gathering these resources, at scale, in varied languages and regions pose a significant challenge as it requires broad socio-cultural knowledge and can also be prohibitively expensive. To overcome this critical gap, we employ a recently introduced approach that couples LLM generations for scale with culturally situated validations for reliability, and build SeeGULL Multilingual, a global-scale multilingual dataset of social stereotypes, containing over 25K stereotypes, spanning 20 languages, with human annotations across 23 regions, and demonstrate its utility in identifying gaps in model evaluations. Content warning: Stereotypes shared in this paper can be offensive.
• 摘要：虽然生成式多语言模型正在迅速部署，但其安全性和公平性评估在很大程度上仅限于以英语收集的资源。对于针对固有社会文化现象（例如陈规定型观念）的评估来说，这一点尤其成问题，因为在这种情况下，建立反映各个语言社区中普遍存在的陈规定型观念的多语言资源非常重要。然而，以不同语言和地区大规模收集这些资源构成了重大挑战，因为它需要广泛的社会文化知识，而且成本也可能高得令人望而却步。为了克服这一关键差距，我们采用了最近推出的一种方法，将法学硕士世代的规模与文化背景的可靠性验证结合起来，并构建了 SeeGULL Multilingual，这是一个全球规模的社会刻板印象多语言数据集，包含超过 25K 刻板印象，涵盖 20 种语言，跨 23 个区域的人工注释，并展示了其在识别模型评估差距方面的实用性。内容警告：本文中分享的刻板印象可能会令人反感。

Title: A Benchmark of Domain-Adapted Large Language Models for Generating Brief Hospital Course Summaries

• Authors: Asad Aali, Dave Van Veen, Yamin Ishraq Arefeen, Jason Hom, Christian Bluethgen, Eduardo Pontes Reis, Sergios Gatidis, Namuun Clifford, Joseph Daws, Arash S. Tehrani, Jangwon Kim, Akshay S. Chaudhari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05720
• Pdf URL: https://arxiv.org/pdf/2403.05720
• Copy Paste: [[2403.05720]] A Benchmark of Domain-Adapted Large Language Models for Generating Brief Hospital Course Summaries(https://arxiv.org/abs/2403.05720)
• Keywords: language model, gpt, llm, prompt
• Abstract: Brief hospital course (BHC) summaries are common clinical documents generated by summarizing clinical notes. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as BHC synthesis have not been shown. To enable the adaptation of LLMs for BHC synthesis, we introduce a novel benchmark consisting of a pre-processed dataset extracted from MIMIC-IV notes, encapsulating clinical note, and brief hospital course (BHC) pairs. We assess the performance of two general-purpose LLMs and three healthcare-adapted LLMs to improve BHC synthesis from clinical notes. Using clinical notes as input for generating BHCs, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to three open-source LLMs (Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5, GPT-4). We quantitatively evaluate the performance of these LLMs across varying context-length inputs using conventional natural language similarity metrics. We further perform a qualitative study where five diverse clinicians blindly compare clinician-written BHCs and two LLM-generated BHCs for 30 samples across metrics of comprehensiveness, conciseness, factual correctness, and fluency. Overall, we present a new benchmark and pre-processed dataset for using LLMs in BHC synthesis from clinical notes. We observe high-quality summarization performance for both in-context proprietary and fine-tuned open-source LLMs using both quantitative metrics and a qualitative clinical reader study. We propose our work as a benchmark to motivate future works to adapt and assess the performance of LLMs in BHC synthesis.
• 摘要：简短的医院课程 (BHC) 摘要是通过总结临床记录生成的常见临床文档。虽然大型语言模型 (LLM) 在自动化现实世界任务方面展现了卓越的能力，但它们在 BHC 合成等医疗保健应用中的能力尚未得到展示。为了使法学硕士能够适应 BHC 合成，我们引入了一种新颖的基准，其中包含从 MIMIC-IV 笔记中提取的预处理数据集、封装的临床笔记和简短的医院课程 (BHC) 对。我们评估了两个通用法学硕士和三个医疗保健适应法学硕士的表现，以根据临床记录改进 BHC 合成。使用临床记录作为生成 BHC 的输入，我们将基于提示（使用上下文学习）和基于微调的适应策略应用于三个开源 LLM（Clinical-T5-Large、Llama2-13B、FLAN-UL2）和两个专有的法学硕士（GPT-3.5、GPT-4）。我们使用传统的自然语言相似性指标定量评估这些法学硕士在不同上下文长度输入上的表现。我们进一步进行了一项定性研究，其中五位不同的临床医生对 30 个样本的临床医生编写的 BHC 和两个法学硕士生成的 BHC 进行盲目比较，涵盖全面性、简洁性、事实正确性和流畅性等指标。总的来说，我们提出了一个新的基准和预处理数据集，用于根据临床记录在 BHC 合成中使用法学硕士。我们使用定量指标和定性临床读者研究来观察上下文中专有和微调的开源法学硕士的高质量总结性能。我们建议我们的工作作为基准，以激励未来的工作适应和评估法学硕士在 BHC 合成中的表现。

Title: Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text

• Authors: Sara Abdali, Richard Anarfi, CJ Barberan, Jia He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05750
• Pdf URL: https://arxiv.org/pdf/2403.05750
• Copy Paste: [[2403.05750]] Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text(https://arxiv.org/abs/2403.05750)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized the field of Natural Language Generation (NLG) by demonstrating an impressive ability to generate human-like text. However, their widespread usage introduces challenges that necessitate thoughtful examination, ethical scrutiny, and responsible practices. In this study, we delve into these challenges, explore existing strategies for mitigating them, with a particular emphasis on identifying AI-generated text as the ultimate solution. Additionally, we assess the feasibility of detection from a theoretical perspective and propose novel research directions to address the current limitations in this domain.
• 摘要：大型语言模型 (LLM) 展示了生成类人文本的令人印象深刻的能力，彻底改变了自然语言生成 (NLG) 领域。然而，它们的广泛使用带来了挑战，需要深思熟虑的检查、道德审查和负责任的做法。在这项研究中，我们深入研究这些挑战，探索缓解这些挑战的现有策略，特别强调将人工智能生成的文本确定为最终解决方案。此外，我们从理论角度评估了检测的可行性，并提出了新的研究方向来解决该领域当前的局限性。

Title: FLAP: Flow Adhering Planning with Constrained Decoding in LLMs

• Authors: Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, Arshit Gupta
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05766
• Pdf URL: https://arxiv.org/pdf/2403.05766
• Copy Paste: [[2403.05766]] FLAP: Flow Adhering Planning with Constrained Decoding in LLMs(https://arxiv.org/abs/2403.05766)
• Keywords: llm, prompt, agent
• Abstract: Planning is a crucial task for agents in task oriented dialogs (TODs). Human agents typically resolve user issues by following predefined workflows, decomposing workflow steps into actionable items, and performing actions by executing APIs in order; all of which require reasoning and planning. With the recent advances in LLMs, there have been increasing attempts to use LLMs for task planning and API usage. However, the faithfulness of the plans to predefined workflows and API dependencies, is not guaranteed with LLMs because of their bias towards pretraining data. Moreover, in real life, workflows are custom-defined and prone to change, hence, quickly adapting agents to the changes is desirable. In this paper, we study faithful planning in TODs to resolve user intents by following predefined flows and preserving API dependencies. We propose a constrained decoding algorithm based on lookahead heuristic for faithful planning. Our algorithm alleviates the need for finetuning LLMs using domain specific data, outperforms other decoding and prompting-based baselines, and applying our algorithm on smaller LLMs (7B) we achieve comparable performance to larger LLMs (30B-40B).
• 摘要：对于面向任务的对话 (TOD) 中的代理来说，规划是一项至关重要的任务。人工代理通常通过遵循预定义的工作流程、将工作流程步骤分解为可操作的项目以及通过按顺序执行 API 来执行操作来解决用户问题；所有这些都需要推理和计划。随着 LLM 的最新进展，越来越多的人尝试使用 LLM 进行任务规划和 API 使用。然而，LLM 无法保证计划对预定义工作流程和 API 依赖关系的忠实性，因为他们偏向于预训练数据。此外，在现实生活中，工作流程是自定义的并且容易发生变化，因此，需要快速使代理适应变化。在本文中，我们研究了 TOD 中的忠实规划，通过遵循预定义的流程并保留 API 依赖关系来解决用户意图。我们提出了一种基于前瞻启发式的约束解码算法，以实现忠实的规划。我们的算法减轻了使用特定领域数据微调 LLM 的需要，优于其他基于解码和提示的基线，并且将我们的算法应用于较小的 LLM (7B)，我们实现了与较大的 LLM (30B-40B) 相当的性能。

Title: On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization

• Authors: Lorenzo Jaime Yu Flores, Arman Cohan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05788
• Pdf URL: https://arxiv.org/pdf/2403.05788
• Copy Paste: [[2403.05788]] On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization(https://arxiv.org/abs/2403.05788)
• Keywords: hallucination
• Abstract: Text summarization and simplification are among the most widely used applications of AI. However, models developed for such tasks are often prone to hallucination, which can result from training on unaligned data. One efficient approach to address this issue is Loss Truncation (LT) (Kang and Hashimoto, 2020), an approach to modify the standard log loss to adaptively remove noisy examples during training. However, we find that LT alone yields a considerable number of hallucinated entities on various datasets. We study the behavior of the underlying losses between factual and non-factual examples, to understand and refine the performance of LT. We demonstrate that LT's performance is limited when the underlying assumption that noisy targets have higher NLL loss is not satisfied, and find that word-level NLL among entities provides better signal for distinguishing factuality. We then leverage this to propose a fine-grained NLL loss and fine-grained data cleaning strategies, and observe improvements in hallucination reduction across some datasets. Our work is available at https://https://github.com/yale-nlp/fine-grained-lt.
• 摘要：文本摘要和简化是人工智能最广泛使用的应用之一。然而，为此类任务开发的模型通常容易产生幻觉，这可能是由于对未对齐数据的训练造成的。解决此问题的一种有效方法是损失截断 (LT)（Kang 和 Hashimoto，2020），这是一种修改标准对数损失以在训练期间自适应删除噪声示例的方法。然而，我们发现仅 LT 在各种数据集上就产生了大量的幻觉实体。我们研究事实和非事实示例之间潜在损失的行为，以理解和完善 LT 的性能。我们证明，当噪声目标具有较高 NLL 损失的基本假设不满足时，LT 的性能受到限制，并发现实体之间的字级 NLL 为区分事实提供了更好的信号。然后，我们利用这一点提出细粒度的 NLL 损失和细粒度的数据清理策略，并观察某些数据集在减少幻觉方面的改进。我们的工作可在 https://https://github.com/yale-nlp/fine-grained-lt 获取。

Title: ItD: Large Language Models Can Teach Themselves Induction through Deduction

• Authors: Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, Kang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05789
• Pdf URL: https://arxiv.org/pdf/2403.05789
• Copy Paste: [[2403.05789]] ItD: Large Language Models Can Teach Themselves Induction through Deduction(https://arxiv.org/abs/2403.05789)
• Keywords: language model, llm
• Abstract: Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt ``post processes'' paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search & refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://anonymous.4open.science/r/ItD-E844.
• 摘要：尽管大型语言模型（LLM）在广泛的自然语言处理任务中表现出了令人印象深刻的性能，但研究人员发现它们的归纳能力仍然有限。最近的工作主要采用“后处理”范式来提高法学硕士在归纳方面的性能（例如，假设搜索和细化方法），但它们的性能仍然受到法学硕士固有的归纳能力的限制。在本文中，我们提出了一个新颖的框架，即通过演绎归纳（ItD），使法学硕士能够通过演绎自学归纳。 ItD 框架由两个主要组件组成：用于生成归纳数据的演绎数据生成模块和用于优化 LLM 微调和解码的朴素贝叶斯归纳模块。我们的实证结果展示了 ItD 在​​两个归纳基准上的有效性，与之前最先进的技术相比，分别实现了 36% 和 10% 的相对性能提升。我们的消融研究验证了 ItD 两个关键模块的有效性。我们还验证了 ItD 在​​不同法学硕士和扣除者中的有效性。本文的数据和代码可以在https://anonymous.4open.science/r/ItD-E844找到。

Title: ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes

• Authors: Zhichao Yang, Avijit Mitra, Sunjae Kwon, Hong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05795
• Pdf URL: https://arxiv.org/pdf/2403.05795
• Copy Paste: [[2403.05795]] ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes(https://arxiv.org/abs/2403.05795)
• Keywords: language model, gpt
• Abstract: The advancement of natural language processing (NLP) systems in healthcare hinges on language model ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient's medical history. However, most earlier clinical language models were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a superior performance in modeling clinical language across extended text lengths compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and accuracy, outperforming existing clinical language models and general domain large models like GPT-4 in longitudinal clinical notes information extraction tasks.
• 摘要：自然语言处理 (NLP) 系统在医疗保健领域的进步取决于语言模型解释临床记录中包含的复杂信息的能力。此过程通常需要整合患者病史中不同时间点的信息。然而，大多数早期的临床语言模型都是经过预训练的，上下文长度仅限于大约一份临床文档。在这项研究中，我们引入了 ClinicalMamba，这是 Mamba 语言模型的专门版本，它在大量纵向临床笔记的语料库上进行了预训练，以满足医学领域独特的语言特征和信息处理需求。与 Mamba 和 Clinical Llama 相比，ClinicalMamba 拥有 1.3 亿和 28 亿个参数，在跨扩展文本长度的临床语言建模方面表现出卓越的性能。通过少量学习，ClinicalMamba 在速度和准确性方面达到了显着的基准，在纵向临床记录信息提取任务中优于现有的临床语言模型和 GPT-4 等通用领域大型模型。

Title: Algorithmic progress in language models

• Authors: Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, Jaime Sevilla
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05812
• Pdf URL: https://arxiv.org/pdf/2403.05812
• Copy Paste: [[2403.05812]] Algorithmic progress in language models(https://arxiv.org/abs/2403.05812)
• Keywords: language model
• Abstract: We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.
• 摘要：我们研究了自深度学习出现以来预训练语言模型算法的改进速度。使用 Wikitext 和 Penn Treebank 2012 年至 2023 年期间超过 200 个语言模型评估的数据集，我们发现达到设定性能阈值所需的计算量大约每 8 个月减少一半，95% 的置信区间约为 5 至 14 个月，比摩尔定律的硬件增益快得多。我们估计增强的缩放定律，这使我们能够量化算法进展并确定缩放模型与训练算法创新的相对贡献。尽管算法进步很快，变压器等新架构不断发展，但我们的分析表明，计算量的增加对这段时间内整体性能的提高做出了更大的贡献。尽管受到嘈杂的基准数据的限制，我们的分析量化了语言建模的快速进展，揭示了计算和算法的相对贡献。

Title: MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs

• Authors: Yerin Hwang, Yongil Kim, Yunah Jang, Jeesoo Bang, Hyunkyung Bae, Kyomin Jung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05814
• Pdf URL: https://arxiv.org/pdf/2403.05814
• Copy Paste: [[2403.05814]] MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs(https://arxiv.org/abs/2403.05814)
• Keywords: language model, llm
• Abstract: Despite advancements in on-topic dialogue systems, effectively managing topic shifts within dialogues remains a persistent challenge, largely attributed to the limited availability of training datasets. To address this issue, we propose Multi-Passage to Dialogue (MP2D), a data generation framework that automatically creates conversational question-answering datasets with natural topic transitions. By leveraging the relationships between entities in a knowledge graph, MP2D maps the flow of topics within a dialogue, effectively mirroring the dynamics of human conversation. It retrieves relevant passages corresponding to the topics and transforms them into dialogues through the passage-to-dialogue method. Through quantitative and qualitative experiments, we demonstrate MP2D's efficacy in generating dialogue with natural topic shifts. Furthermore, this study introduces a novel benchmark for topic shift dialogues, TS-WikiDialog. Utilizing the dataset, we demonstrate that even Large Language Models (LLMs) struggle to handle topic shifts in dialogue effectively, and we showcase the performance improvements of models trained on datasets generated by MP2D across diverse topic shift dialogue tasks.
• 摘要：尽管主题对话系统取得了进步，但有效管理对话中的主题转移仍然是一个持续存在的挑战，这在很大程度上归因于训练数据集的可用性有限。为了解决这个问题，我们提出了 Multi-Passage to Dialogue (MP2D)，这是一种数据生成框架，可以自动创建具有自然主题转换的对话问答数据集。通过利用知识图中实体之间的关系，MP2D 映射对话中的主题流，有效地反映了人类对话的动态。它检索与主题相对应的相关段落，并通过段落到对话的方法将其转换为对话。通过定量和定性实验，我们证明了 MP2D 在生成具有自然主题转移的对话方面的功效。此外，本研究引入了主题转移对话的新颖基准 TS-WikiDialog。利用该数据集，我们证明即使是大型语言模型 (LLM) 也很难有效地处理对话中的主题转换，并且我们展示了在 MP2D 生成的数据集上训练的模型在不同主题转换对话任务中的性能改进。

Title: Reverse That Number! Decoding Order Matters in Arithmetic Learning

• Authors: Daniel Zhang-Li, Nianyi Lin, Jifan Yu, Zheyuan Zhang, Zijun Yao, Xiaokang Zhang, Lei Hou, Jing Zhang, Juanzi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05845
• Pdf URL: https://arxiv.org/pdf/2403.05845
• Copy Paste: [[2403.05845]] Reverse That Number! Decoding Order Matters in Arithmetic Learning(https://arxiv.org/abs/2403.05845)
• Keywords: language model, llm
• Abstract: Recent advancements in pretraining have demonstrated that modern Large Language Models (LLMs) possess the capability to effectively learn arithmetic operations. However, despite acknowledging the significance of digit order in arithmetic computation, current methodologies predominantly rely on sequential, step-by-step approaches for teaching LLMs arithmetic, resulting in a conclusion where obtaining better performance involves fine-grained step-by-step. Diverging from this conventional path, our work introduces a novel strategy that not only reevaluates the digit order by prioritizing output from the least significant digit but also incorporates a step-by-step methodology to substantially reduce complexity. We have developed and applied this method in a comprehensive set of experiments. Compared to the previous state-of-the-art (SOTA) method, our findings reveal an overall improvement of in accuracy while requiring only a third of the tokens typically used during training. For the purpose of facilitating replication and further research, we have made our code and dataset publicly available at \url{https://anonymous.4open.science/r/RAIT-9FB7/}.
• 摘要：预训练方面的最新进展表明，现代大型语言模型 (LLM) 具有有效学习算术运算的能力。然而，尽管承认数字顺序在算术计算中的重要性，但当前的方法主要依赖于顺序的、逐步的方法来教授法学硕士算术，从而得出这样的结论：获得更好的性能需要细粒度的逐步进行。与这种传统路径不同，我们的工作引入了一种新颖的策略，该策略不仅通过优先考虑最低有效数字的输出来重新评估数字顺序，而且还采用了逐步方法来大幅降低复杂性。我们已经在一系列综合实验中开发并应用了这种方法。与之前最先进的 (SOTA) 方法相比，我们的研究结果表明，准确性整体提高，同时只需要训练期间通常使用的令牌的三分之一。为了促进复制和进一步研究，我们已在 \url{https://anonymous.4open.science/r/RAIT-9FB7/} 上公开提供我们的代码和数据集。

Title: KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques

• Authors: Rui Yang, Haoran Liu, Qingcheng Zeng, Yu He Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Caverlee, Yutaka Matsuo, Irene Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05881
• Pdf URL: https://arxiv.org/pdf/2403.05881
• Copy Paste: [[2403.05881]] KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques(https://arxiv.org/abs/2403.05881)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have significantly advanced healthcare innovation on generation capabilities. However, their application in real clinical settings is challenging due to potential deviations from medical facts and inherent biases. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) with ranking and re-ranking techniques, aiming to improve free-text question-answering (QA) in the medical domain. Specifically, upon receiving a question, we initially retrieve triplets from a medical KG to gather factual information. Subsequently, we innovatively apply ranking methods to refine the ordering of these triplets, aiming to yield more precise answers. To the best of our knowledge, KG-Rank is the first application of ranking models combined with KG in medical QA specifically for generating long answers. Evaluation of four selected medical QA datasets shows that KG-Rank achieves an improvement of over 18% in the ROUGE-L score. Moreover, we extend KG-Rank to open domains, where it realizes a 14% improvement in ROUGE-L, showing the effectiveness and potential of KG-Rank.
• 摘要：大型语言模型 (LLM) 在生成能力方面显着推进了医疗保健创新。然而，由于可能偏离医学事实和固有偏见，它们在实际临床环境中的应用具有挑战性。在这项工作中，我们开发了一个增强的 LLM 框架 KG-Rank，它利用具有排名和重新排名技术的医学知识图 (KG)，旨在提高医学领域的自由文本问答 (QA)。具体来说，收到问题后，我们首先从医学知识库中检索三元组以收集事实信息。随后，我们创新地应用排序方法来细化这些三元组的排序，旨在产生更精确的答案。据我们所知，KG-Rank 是排序模型与 KG 相结合在医学 QA 中的第一个应用，专门用于生成长答案。对四个选定的医学 QA 数据集的评估表明，KG-Rank 在 ROUGE-L 分数上实现了超过 18% 的改进。此外，我们将 KG-Rank 扩展到开放域，实现了 ROUGE-L 14% 的改进，显示了 KG-Rank 的有效性和潜力。

Title: High Throughput Phenotyping of Physician Notes with Large Language and Hybrid NLP Models

• Authors: Syed I. Munzir, Daniel B. Hier, Michael D. Carrithers
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05920
• Pdf URL: https://arxiv.org/pdf/2403.05920
• Copy Paste: [[2403.05920]] High Throughput Phenotyping of Physician Notes with Large Language and Hybrid NLP Models(https://arxiv.org/abs/2403.05920)
• Keywords: language model
• Abstract: Deep phenotyping is the detailed description of patient signs and symptoms using concepts from an ontology. The deep phenotyping of the numerous physician notes in electronic health records requires high throughput methods. Over the past thirty years, progress toward making high throughput phenotyping feasible. In this study, we demonstrate that a large language model and a hybrid NLP model (combining word vectors with a machine learning classifier) can perform high throughput phenotyping on physician notes with high accuracy. Large language models will likely emerge as the preferred method for high throughput deep phenotyping of physician notes.
• 摘要：深度表型分析是使用本体概念对患者体征和症状的详细描述。对电子健康记录中大量医生记录的深入表型分析需要高通量方法。在过去的三十年中，在使高通量表型分析变得可行方面取得了进展。在这项研究中，我们证明大型语言模型和混合 NLP 模型（将词向量与机器学习分类器相结合）可以以高精度对医生笔记进行高通量表型分析。大型语言模型可能会成为医生笔记高通量深度表型分析的首选方法。

Title: Thread Detection and Response Generation using Transformers with Prompt Optimisation

• Authors: Kevin Joshua T, Arnav Agarwal, Shriya Sanjay, Yash Sarda, John Sahaya Rani Alex, Saurav Gupta, Sushant Kumar, Vishwanath Kamath
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05931
• Pdf URL: https://arxiv.org/pdf/2403.05931
• Copy Paste: [[2403.05931]] Thread Detection and Response Generation using Transformers with Prompt Optimisation(https://arxiv.org/abs/2403.05931)
• Keywords: language model, llm, prompt
• Abstract: Conversational systems are crucial for human-computer interaction, managing complex dialogues by identifying threads and prioritising responses. This is especially vital in multi-party conversations, where precise identification of threads and strategic response prioritisation ensure efficient dialogue management. To address these challenges an end-to-end model that identifies threads and prioritises their response generation based on the importance was developed, involving a systematic decomposition of the problem into discrete components - thread detection, prioritisation, and performance optimisation which was meticulously analysed and optimised. These refined components seamlessly integrate into a unified framework, in conversational systems. Llama2 7b is used due to its high level of generalisation but the system can be updated with any open source Large Language Model(LLM). The computational capabilities of the Llama2 model was augmented by using fine tuning methods and strategic prompting techniques to optimise the model's performance, reducing computational time and increasing the accuracy of the model. The model achieves up to 10x speed improvement, while generating more coherent results compared to existing models.
• 摘要：对话系统对于人机交互至关重要，通过识别线程和优先响应来管理复杂的对话。这在多方对话中尤其重要，其中线程的精确识别和战略响应优先级可确保有效的对话管理。为了应对这些挑战，开发了一种端到端模型，该模型可识别线程并根据重要性对其响应生成进行优先级排序，其中涉及将问题系统地分解为离散组件 - 线程检测、优先级排序和性能优化，并对其进行了仔细分析和分析。优化。这些精致的组件无缝集成到会话系统中的统一框架中。使用 Llama2 7b 是因为其高度通用化，但该系统可以使用任何开源大型语言模型 (LLM) 进行更新。通过使用微调方法和策略提示技术来优化模型性能，减少计算时间并提高模型的准确性，增强了 Llama2 模型的计算能力。与现有模型相比，该模型的速度提高了 10 倍，同时生成了更加连贯的结果。

Title: Calibrating Large Language Models Using Their Generations Only

• Authors: Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, Seong Joon Oh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05973
• Pdf URL: https://arxiv.org/pdf/2403.05973
• Copy Paste: [[2403.05973]] Calibrating Large Language Models Using Their Generations Only(https://arxiv.org/abs/2403.05973)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model's confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs - especially when the only interface to the models is their generated text - remains a challenge. We propose APRICOT (auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM's confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or adjusting the given answer based on the confidence. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.
• 摘要：随着大型语言模型 (LLM) 越来越多地部署在面向用户的应用程序中，通过准确量化模型对其预测的置信度来建立信任和维护安全性变得更加重要。然而，找到校准法学硕士的有效方法——尤其是当模型的唯一接口是其生成的文本时——仍然是一个挑战。我们提出 APRICOT（置信度目标的辅助预测）：一种设置置信度目标并训练附加模型的方法，该模型仅根据文本输入和输出来预测法学硕士的置信度。这种方法有几个优点：它在概念上很简单，不需要访问超出其输出的目标模型，不干扰语言生成，并且具有多种潜在用途，例如通过语言表达预测的置信度或调整给定的置信度。根据置信度回答。我们展示了我们的方法如何在闭卷问答中的白盒和黑盒 LLM 的校准误差方面具有竞争力，以检测不正确的 LLM 答案。

Title: Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages

• Authors: Christopher Toukmaji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06018
• Pdf URL: https://arxiv.org/pdf/2403.06018
• Copy Paste: [[2403.06018]] Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages(https://arxiv.org/abs/2403.06018)
• Keywords: language model, prompt
• Abstract: Large pre-trained language models (PLMs) are at the forefront of advances in Natural Language Processing. One widespread use case of PLMs is "prompting" - or in-context learning - where a user provides a description of a task and some completed examples of the task to a PLM as context before prompting the PLM to perform the task on a new example. Only the largest, most capable PLMs are able to perform in-context learning effectively, and these models are typically trained with a predominantly English corpus, leaving all other languages behind. The data limitations in most languages preclude the training of language-specific PLMs capable of prompting. Albeit the surge in work of prompting settings, it is still unclear how PLMs should be adapted cross-lingually specifically for prompting. We evaluate the possible methods to adapt LLaMa, a 7B parameter open-source PLM mainly trained in English, for prompting in low-resource languages, namely for Kinyarwanda, Hausa, and Luganda. We consider three methods: few-shot prompting (prompt), language-adaptive fine-tuning (LAFT), and neural machine translation (translate), and evaluate on abstractive summarization, multi-class topic classification, and named-entity recognition. Although LAFT carries the greatest compute cost and intuitively should lead to the best results, our experiments exhibit that LAFT is only occasionally the optimal choice for adapting PLMs for prompting. Rather, the translate and prompt settings are a compute-efficient and cost-effective method of few-shot prompting for the selected low-resource languages. We find that the results are task and language dependent but find that the prompting method is the best on average across all tasks and languages. Results show that the prompt setting performs better than both translating and LAFT with statistical significance for all shots when aggregated across all tasks and languages.
• 摘要：大型预训练语言模型 (PLM) 处于自然语言处理进步的最前沿。 PLM 的一个广泛使用案例是“提示”（或上下文学习），其中用户在提示 PLM 对新示例执行任务之前，向 PLM 提供任务描述和一些已完成的任务示例作为上下文。只有最大、最有能力的 PLM 才能有效地执行上下文学习，并且这些模型通常使用主要是英语的语料库进行训练，而将所有其他语言抛在后面。大多数语言的数据限制妨碍了能够提示的特定语言 PLM 的训练。尽管提示设置的工作量激增，但仍不清楚 PLM 应如何跨语言专门针对提示进行调整。我们评估了采用 LLaMa（一种主要用英语训练的 7B 参数开源 PLM）的可能方法，以用低资源语言（即基尼亚卢旺达语、豪萨语和卢干达语）进行提示。我们考虑三种方法：少镜头提示（prompt）、语言自适应微调（LAFT）和神经机器翻译（translate），并对抽象摘要、多类主题分类和命名实体识别进行评估。尽管 LAFT 的计算成本最高，并且直观上应该会产生最佳结果，但我们的实验表明，LAFT 只是偶尔成为采用 PLM 进行提示的最佳选择。相反，翻译和提示设置是针对所选低资源语言的少量提示的计算效率高且经济高效的方法。我们发现结果取决于任务和语言，但发现提示方法在所有任务和语言中平均而言是最好的。结果表明，提示设置的效果优于翻译和 LAFT，当跨所有任务和语言进行汇总时，所有镜头均具有统计显着性。

Title: Ensemble Language Models for Multilingual Sentiment Analysis

• Authors: Md Arid Hasan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06060
• Pdf URL: https://arxiv.org/pdf/2403.06060
• Copy Paste: [[2403.06060]] Ensemble Language Models for Multilingual Sentiment Analysis(https://arxiv.org/abs/2403.06060)
• Keywords: language model
• Abstract: The rapid advancement of social media enables us to analyze user opinions. In recent times, sentiment analysis has shown a prominent research gap in understanding human sentiment based on the content shared on social media. Although sentiment analysis for commonly spoken languages has advanced significantly, low-resource languages like Arabic continue to get little research due to resource limitations. In this study, we explore sentiment analysis on tweet texts from SemEval-17 and the Arabic Sentiment Tweet dataset. Moreover, We investigated four pretrained language models and proposed two ensemble language models. Our findings include monolingual models exhibiting superior performance and ensemble models outperforming the baseline while the majority voting ensemble outperforms the English language.
• 摘要：社交媒体的快速发展使我们能够分析用户的意见。近年来，情感分析显示出在基于社交媒体共享内容理解人类情感方面存在显着的研究差距。尽管常用语言的情感分析取得了显着进展，但由于资源限制，阿拉伯语等资源匮乏的语言仍然很少得到研究。在本研究中，我们探索了对 SemEval-17 和阿拉伯情绪推文数据集的推文文本进行情感分析。此外，我们研究了四种预训练的语言模型，并提出了两种集成语言模型。我们的研究结果包括单语模型表现出卓越的性能，而集成模型的性能优于基线，而多数投票集成模型的性能优于英语。

Title: Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue

• Authors: Jian Wang, Dongding Lin, Wenjie Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06063
• Pdf URL: https://arxiv.org/pdf/2403.06063
• Copy Paste: [[2403.06063]] Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue(https://arxiv.org/abs/2403.06063)
• Keywords: prompt
• Abstract: Target-oriented proactive dialogue systems aim to lead conversations from a dialogue context toward a pre-determined target, such as making recommendations on designated items or introducing new specific topics. To this end, it is critical for such dialogue systems to plan reasonable actions to drive the conversation proactively, and meanwhile, to plan appropriate topics to move the conversation forward to the target topic smoothly. In this work, we mainly focus on effective dialogue planning for target-oriented dialogue generation. Inspired by decision-making theories in cognitive science, we propose a novel target-constrained bidirectional planning (TRIP) approach, which plans an appropriate dialogue path by looking ahead and looking back. By formulating the planning as a generation task, our TRIP bidirectionally generates a dialogue path consisting of a sequence of pairs using two Transformer decoders. They are expected to supervise each other and converge on consistent actions and topics by minimizing the decision gap and contrastive generation of targets. Moreover, we propose a target-constrained decoding algorithm with a bidirectional agreement to better control the planning process. Subsequently, we adopt the planned dialogue paths to guide dialogue generation in a pipeline manner, where we explore two variants: prompt-based generation and plan-controlled generation. Extensive experiments are conducted on two challenging dialogue datasets, which are re-purposed for exploring target-oriented dialogue. Our automatic and human evaluations demonstrate that the proposed methods significantly outperform various baseline models.
• 摘要：面向目标的主动对话系统旨在将对话从对话上下文引导向预定目标，例如对指定项目提出建议或引入新的特定主题。为此，此类对话系统必须规划合理的行动以主动推动对话，同时规划适当的主题以将对话顺利推进到目标主题。在这项工作中，我们主要关注针对目标的对话生成的有效对话规划。受认知科学决策理论的启发，我们提出了一种新颖的目标约束双向规划（TRIP）方法，该方法通过向前看和向后看来规划适当的对话路径。通过将规划制定为生成任务，我们的 TRIP 使用两个 Transformer 解码器双向生成由一系列 对组成的对话路径。他们应该相互监督，并通过最小化决策差距和目标的对比生成来集中一致的行动和主题。此外，我们提出了一种具有双向协议的目标约束解码算法，以更好地控制规划过程。随后，我们采用规划的对话路径以管道方式指导对话生成，其中我们探索了两种变体：基于提示的生成和计划控制的生成。在两个具有挑战性的对话数据集上进行了广泛的实验，这些数据集被重新用于探索面向目标的对话。我们的自动和人工评估表明，所提出的方法显着优于各种基线模型。

Title: Can LLM Substitute Human Labeling? A Case Study of Fine-grained Chinese Address Entity Recognition Dataset for UAV Delivery

• Authors: Yuxuan Yao, Sichun Luo, Haohan Zhao, Guanzhi Deng, Linqi Song
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.06097
• Pdf URL: https://arxiv.org/pdf/2403.06097
• Copy Paste: [[2403.06097]] Can LLM Substitute Human Labeling? A Case Study of Fine-grained Chinese Address Entity Recognition Dataset for UAV Delivery(https://arxiv.org/abs/2403.06097)
• Keywords: llm
• Abstract: We present CNER-UAV, a fine-grained \textbf{C}hinese \textbf{N}ame \textbf{E}ntity \textbf{R}ecognition dataset specifically designed for the task of address resolution in \textbf{U}nmanned \textbf{A}erial \textbf{V}ehicle delivery systems. The dataset encompasses a diverse range of five categories, enabling comprehensive training and evaluation of NER models. To construct this dataset, we sourced the data from a real-world UAV delivery system and conducted a rigorous data cleaning and desensitization process to ensure privacy and data integrity. The resulting dataset, consisting of around 12,000 annotated samples, underwent human experts and \textbf{L}arge \textbf{L}anguage \textbf{M}odel annotation. We evaluated classical NER models on our dataset and provided in-depth analysis. The dataset and models are publicly available at \url{https://github.com/zhhvvv/CNER-UAV}.
• 摘要：我们提出了 CNER-UAV，一个细粒度的 \textbf{C}hinese \textbf{N}ame \textbf{E}ntity \textbf{R} 识别数据集，专为 \textbf{U}nmanned 中的地址解析任务而设计\textbf{A}空中\textbf{V}车辆运输系统。该数据集包含五个类别，可实现 NER 模型的全面训练和评估。为了构建该数据集，我们从现实世界的无人机交付系统中获取数据，并进行了严格的数据清理和脱敏过程，以确保隐私和数据完整性。生成的数据集由大约 12,000 个带注释的样本组成，经过人类专家和 \textbf{L}arge \textbf{L}anguage \textbf{M}odel 注释。我们在数据集上评估了经典的 NER 模型并提供了深入的分析。数据集和模型可在 \url{https://github.com/zhhvvv/CNER-UAV} 上公开获取。

Title: Large Language Models on Fine-grained Emotion Detection Dataset with Data Augmentation and Transfer Learning

• Authors: Kaipeng Wang, Zhi Jing, Yongye Su, Yikun Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06108
• Pdf URL: https://arxiv.org/pdf/2403.06108
• Copy Paste: [[2403.06108]] Large Language Models on Fine-grained Emotion Detection Dataset with Data Augmentation and Transfer Learning(https://arxiv.org/abs/2403.06108)
• Keywords: language model
• Abstract: This paper delves into enhancing the classification performance on the GoEmotions dataset, a large, manually annotated dataset for emotion detection in text. The primary goal of this paper is to address the challenges of detecting subtle emotions in text, a complex issue in Natural Language Processing (NLP) with significant practical applications. The findings offer valuable insights into addressing the challenges of emotion detection in text and suggest directions for future research, including the potential for a survey paper that synthesizes methods and performances across various datasets in this domain.
• 摘要：本文深入研究了如何增强 GoEmotions 数据集的分类性能，GoEmotions 数据集是一个用于文本情感检测的大型手动注释数据集。本文的主要目标是解决检测文本中微妙情感的挑战，这是自然语言处理（NLP）中具有重要实际应用的复杂问题。这些发现为解决文本中情绪检测的挑战提供了宝贵的见解，并为未来的研究提出了方向，包括综合该领域各种数据集的方法和性能的调查论文的潜力。

Title: FMPAF: How Do Fed Chairs Affect the Financial Market? A Fine-grained Monetary Policy Analysis Framework on Their Language

• Authors: Yayue Deng, Mohan Xu, Yao Tang
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2403.06115
• Pdf URL: https://arxiv.org/pdf/2403.06115
• Copy Paste: [[2403.06115]] FMPAF: How Do Fed Chairs Affect the Financial Market? A Fine-grained Monetary Policy Analysis Framework on Their Language(https://arxiv.org/abs/2403.06115)
• Keywords: language model, llm
• Abstract: The effectiveness of central bank communication is a crucial aspect of monetary policy transmission. While recent research has examined the influence of policy communication by the chairs of the Federal Reserve on various financial variables, much of the literature relies on rule-based or dictionary-based methods in parsing the language of the chairs, leaving nuanced information about policy stance contained in nonverbal emotion out of the analysis. In the current study, we propose the Fine-Grained Monetary Policy Analysis Framework (FMPAF), a novel approach that integrates large language models (LLMs) with regression analysis to provide a comprehensive analysis of the impact of the press-conference communications of chairs of the Federal Reserve on financial markets. We conduct extensive comparisons of model performance under different levels of granularity, modalities, and communication scenarios. Based on our preferred specification, a one-unit increase in the sentiment score is associated with an increase of the price of S\&P 500 Exchange-Traded Fund by approximately 500 basis points, a 15-basis-point decrease in the policy interest rate, while not leading to a significant response in exchange rates.
• 摘要：央行沟通的有效性是货币政策传导的一个重要方面。虽然最近的研究考察了美联储主席的政策沟通对各种金融变量的影响，但许多文献依赖于基于规则或基于字典的方法来解析主席的语言，留下了有关政策立场的细微信息分析出来的非语言情感所包含的内容。在当前的研究中，我们提出了细粒度货币政策分析框架（FMPAF），这是一种将大语言模型（LLM）与回归分析相结合的新颖方法，可以对各国主席新闻发布会沟通的影响进行全面分析。美联储对金融市场的影响。我们对不同粒度、模式和通信场景下的模型性能进行了广泛的比较。根据我们的首选规范，情绪评分每增加一个单位，标普 500 交易所交易基金价格将上涨约 500 个基点，政策利率将下降 15 个基点，同时不会导致汇率出现重大反应。

Title: Fine-grainedly Synthesize Streaming Data Based On Large Language Models With Graph Structure Understanding For Data Sparsity

• Authors: Xin Zhang, Linhai Zhang, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06139
• Pdf URL: https://arxiv.org/pdf/2403.06139
• Copy Paste: [[2403.06139]] Fine-grainedly Synthesize Streaming Data Based On Large Language Models With Graph Structure Understanding For Data Sparsity(https://arxiv.org/abs/2403.06139)
• Keywords: language model, llm
• Abstract: Due to the sparsity of user data, sentiment analysis on user reviews in e-commerce platforms often suffers from poor performance, especially when faced with extremely sparse user data or long-tail labels. Recently, the emergence of LLMs has introduced new solutions to such problems by leveraging graph structures to generate supplementary user profiles. However, previous approaches have not fully utilized the graph understanding capabilities of LLMs and have struggled to adapt to complex streaming data environments. In this work, we propose a fine-grained streaming data synthesis framework that categorizes sparse users into three categories: Mid-tail, Long-tail, and Extreme. Specifically, we design LLMs to comprehensively understand three key graph elements in streaming data, including Local-global Graph Understanding, Second-Order Relationship Extraction, and Product Attribute Understanding, which enables the generation of high-quality synthetic data to effectively address sparsity across different categories. Experimental results on three real datasets demonstrate significant performance improvements, with synthesized data contributing to MSE reductions of 45.85%, 3.16%, and 62.21%, respectively.
• 摘要：由于用户数据的稀疏性，电商平台中用户评论的情感分析往往表现不佳，尤其是在面对极其稀疏的用户数据或长尾标签时。最近，法学硕士的出现通过利用图结构生成补充用户配置文件为此类问题引入了新的解决方案。然而，之前的方法并没有充分利用LLM的图理解能力，并且很难适应复杂的流数据环境。在这项工作中，我们提出了一个细粒度的流数据合成框架，将稀疏用户分为三类：中尾、长尾和极端。具体来说，我们设计的法学硕士全面理解流数据中的三个关键图元素，包括局部全局图理解、二阶关系提取和产品属性理解，从而能够生成高质量的合成数据，从而有效解决不同数据之间的稀疏性问题。类别。三个真实数据集的实验结果表明性能显着提高，合成数据使 MSE 分别降低了 45.85%、3.16% 和 62.21%。

Title: Can Large Language Models Automatically Score Proficiency of Written Essays?

• Authors: Watheq Mansour, Salam Albatarni, Sohaila Eltanbouly, Tamer Elsayed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06149
• Pdf URL: https://arxiv.org/pdf/2403.06149
• Copy Paste: [[2403.06149]] Can Large Language Models Automatically Score Proficiency of Written Essays?(https://arxiv.org/abs/2403.06149)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Although several methods were proposed to address the problem of automated essay scoring (AES) in the last 50 years, there is still much to desire in terms of effectiveness. Large Language Models (LLMs) are transformer-based models that demonstrate extraordinary capabilities on various tasks. In this paper, we test the ability of LLMs, given their powerful linguistic knowledge, to analyze and effectively score written essays. We experimented with two popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do this task and, if so, how their performance is positioned among the state-of-the-art (SOTA) models across two levels, holistically and per individual writing trait. We utilized prompt-engineering tactics in designing four different prompts to bring their maximum potential to this task. Our experiments conducted on the ASAP dataset revealed several interesting observations. First, choosing the right prompt depends highly on the model and nature of the task. Second, the two LLMs exhibited comparable average performance in AES, with a slight advantage for ChatGPT. Finally, despite the performance gap between the two LLMs and SOTA models in terms of predictions, they provide feedback to enhance the quality of the essays, which can potentially help both teachers and students.
• 摘要：尽管在过去 50 年中提出了多种方法来解决自动作文评分 (AES) 问题，但在有效性方面仍然存在很大差距。大型语言模型 (LLM) 是基于 Transformer 的模型，在各种任务上展示出非凡的能力。在本文中，我们测试了法学硕士凭借其强大的语言知识分析和有效评分书面论文的能力。我们尝试了两种流行的法学硕士，即 ChatGPT 和 Llama。我们的目标是检查这些模型是否可以完成这项任务，如果可以，那么它们的性能在两个层面上（整体上和个人写作特征）在最先进（SOTA）模型中的定位如何。我们利用提示工程策略设计了四种不同的提示，以最大限度地发挥其潜力来完成这项任务。我们在 ASAP 数据集上进行的实验揭示了一些有趣的观察结果。首先，选择正确的提示在很大程度上取决于任务的模型和性质。其次，两位法学硕士在 AES 中表现出相当的平均性能，而 ChatGPT 略有优势。最后，尽管两个 LLM 和 SOTA 模型在预测方面存在性能差距，但它们提供反馈以提高论文质量，这可能对教师和学生都有帮助。

Title: Are You Being Tracked? Discover the Power of Zero-Shot Trajectory Tracing with LLMs!

• Authors: Huanqi Yang, Sijie Ji, Rucheng Wu, Weitao Xu
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06201
• Pdf URL: https://arxiv.org/pdf/2403.06201
• Copy Paste: [[2403.06201]] Are You Being Tracked? Discover the Power of Zero-Shot Trajectory Tracing with LLMs!(https://arxiv.org/abs/2403.06201)
• Keywords: language model, llm, prompt
• Abstract: There is a burgeoning discussion around the capabilities of Large Language Models (LLMs) in acting as fundamental components that can be seamlessly incorporated into Artificial Intelligence of Things (AIoT) to interpret complex trajectories. This study introduces LLMTrack, a model that illustrates how LLMs can be leveraged for Zero-Shot Trajectory Recognition by employing a novel single-prompt technique that combines role-play and think step-by-step methodologies with unprocessed Inertial Measurement Unit (IMU) data. We evaluate the model using real-world datasets designed to challenge it with distinct trajectories characterized by indoor and outdoor scenarios. In both test scenarios, LLMTrack not only meets but exceeds the performance benchmarks set by traditional machine learning approaches and even contemporary state-of-the-art deep learning models, all without the requirement of training on specialized datasets. The results of our research suggest that, with strategically designed prompts, LLMs can tap into their extensive knowledge base and are well-equipped to analyze raw sensor data with remarkable effectiveness.
• 摘要：围绕大型语言模型 (LLM) 作为基本组件的能力，人们正在迅速展开讨论，这些模型可以无缝地融入到物联网人工智能 (AIoT) 中以解释复杂的轨迹。本研究介绍了 LLMTrack，该模型说明了如何通过采用新颖的单提示技术来利用 LLM 进行零射击轨迹识别，该技术将角色扮演和分步思考方法与未处理的惯性测量单元 (IMU) 数据相结合。我们使用真实世界的数据集来评估模型，这些数据集旨在通过以室内和室外场景为特征的不同轨迹来挑战模型。在这两个测试场景中，LLMTrack 不仅满足甚至超过了传统机器学习方法甚至当代最先进的深度学习模型设定的性能基准，而且都不需要在专门的数据集上进行训练。我们的研究结果表明，通过战略性设计的提示，法学硕士可以利用其广泛的知识库，并有能力以显着的效率分析原始传感器数据。

Title: Identifying and interpreting non-aligned human conceptual representations using language modeling

• Authors: Wanqian Bao, Uri Hasson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06204
• Pdf URL: https://arxiv.org/pdf/2403.06204
• Copy Paste: [[2403.06204]] Identifying and interpreting non-aligned human conceptual representations using language modeling(https://arxiv.org/abs/2403.06204)
• Keywords: language model
• Abstract: The question of whether people's experience in the world shapes conceptual representation and lexical semantics is longstanding. Word-association, feature-listing and similarity rating tasks aim to address this question but require a subjective interpretation of the latent dimensions identified. In this study, we introduce a supervised representational-alignment method that (i) determines whether two groups of individuals share the same basis of a certain category, and (ii) explains in what respects they differ. In applying this method, we show that congenital blindness induces conceptual reorganization in both a-modal and sensory-related verbal domains, and we identify the associated semantic shifts. We first apply supervised feature-pruning to a language model (GloVe) to optimize prediction accuracy of human similarity judgments from word embeddings. Pruning identifies one subset of retained GloVe features that optimizes prediction of judgments made by sighted individuals and another subset that optimizes judgments made by blind. A linear probing analysis then interprets the latent semantics of these feature-subsets by learning a mapping from the retained GloVe features to 65 interpretable semantic dimensions. We applied this approach to seven semantic domains, including verbs related to motion, sight, touch, and amodal verbs related to knowledge acquisition. We find that blind individuals more strongly associate social and cognitive meanings to verbs related to motion or those communicating non-speech vocal utterances (e.g., whimper, moan). Conversely, for amodal verbs, they demonstrate much sparser information. Finally, for some verbs, representations of blind and sighted are highly similar. The study presents a formal approach for studying interindividual differences in word meaning, and the first demonstration of how blindness impacts conceptual representation of everyday verbs.
• 摘要：人们在世界上的经历是否塑造概念表征和词汇语义的问题由来已久。单词关联、特征列表和相似性评级任务旨在解决这个问题，但需要对所识别的潜在维度进行主观解释。在本研究中，我们引入了一种监督表征对齐方法，该方法（i）确定两组个体是否具有相同的特定类别基础，以及（ii）解释他们在哪些方面存在差异。在应用这种方法时，我们表明先天性失明会导致非模态和感觉相关言语领域的概念重组，并且我们识别出相关的语义转变。我们首先将监督特征修剪应用于语言模型（GloVe），以优化词嵌入的人类相似性判断的预测准确性。修剪识别了保留的 GloVe 特征的一个子集，该子集优化了对有视力的人做出的判断的预测，以及另一个优化了盲人做出的判断的子集。然后，线性探测分析通过学习从保留的 GloVe 特征到 65 个可解释语义维度的映射来解释这些特征子集的潜在语义。我们将这种方法应用于七个语义领域，包括与运动、视觉、触觉相关的动词以及与知识获取相关的非模态动词。我们发现盲人更强烈地将社会和认知意义与与运动相关的动词或那些交流非言语声音的动词（例如，呜咽、呻吟）联系起来。相反，对于非情态动词，它们展示的信息要稀疏得多。最后，对于某些动词，盲人和有视力的表示非常相似。该研究提出了一种研究个体间词义差异的正式方法，并首次证明了失明如何影响日常动词的概念表征。

Title: Personalized LoRA for Human-Centered Text Understanding

• Authors: You Zhang, Jin Wang, Liang-Chih Yu, Dan Xu, Xuejie Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06208
• Pdf URL: https://arxiv.org/pdf/2403.06208
• Copy Paste: [[2403.06208]] Personalized LoRA for Human-Centered Text Understanding(https://arxiv.org/abs/2403.06208)
• Keywords: language model
• Abstract: Effectively and efficiently adapting a pre-trained language model (PLM) for human-centered text understanding (HCTU) is challenging since user tokens are million-level in most personalized applications and do not have concrete explicit semantics. A standard and parameter-efficient approach (e.g., LoRA) necessitates memorizing numerous suits of adapters for each user. In this work, we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically deploying in PLMs. Moreover, a personalized dropout and a mutual information maximizing strategies are adopted and hence the proposed PLoRA can be well adapted to few/zero-shot learning scenarios for the cold-start issue. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios for the HCTU task, even though it has fewer trainable parameters. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.
• 摘要：有效且高效地采用预训练语言模型 (PLM) 来实现以人为中心的文本理解 (HCTU) 具有挑战性，因为在大多数个性化应用程序中，用户令牌都是百万级的，并且没有具体的显式语义。标准且参数高效的方法（例如 LoRA）需要为每个用户记住大量适配器。在这项工作中，我们为 HCTU 任务引入了带有即插即用（PnP）框架的个性化 LoRA（PLoRA）。 PLoRA 是有效的、参数高效的并且可以在 PLM 中动态部署。此外，采用了个性化 dropout 和互信息最大化策略，因此所提出的 PLoRA 可以很好地适应冷启动问题的少样本/零样本学习场景。在四个基准数据集上进行的实验表明，所提出的方法在 HCTU 任务的全/少/零样本学习场景中优于现有方法，尽管它的可训练参数较少。为了重现性，本文的代码可从以下网址获取：https://github.com/yoyo-yun/PLoRA。

Title: Editing Conceptual Knowledge for Large Language Models

• Authors: Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06259
• Pdf URL: https://arxiv.org/pdf/2403.06259
• Copy Paste: [[2403.06259]] Editing Conceptual Knowledge for Large Language Models(https://arxiv.org/abs/2403.06259)
• Keywords: language model, llm
• Abstract: Recently, there has been a growing interest in knowledge editing for Large Language Models (LLMs). Current approaches and evaluations merely explore the instance-level editing, while whether LLMs possess the capability to modify concepts remains unclear. This paper pioneers the investigation of editing conceptual knowledge for LLMs, by constructing a novel benchmark dataset ConceptEdit and establishing a suite of new metrics for evaluation. The experimental results reveal that, although existing editing methods can efficiently modify concept-level definition to some extent, they also have the potential to distort the related instantial knowledge in LLMs, leading to poor performance. We anticipate this can inspire further progress in better understanding LLMs. Our project homepage is available at https://zjunlp.github.io/project/ConceptEdit.
• 摘要：最近，人们对大型语言模型（LLM）的知识编辑越来越感兴趣。目前的方法和评估仅探索实例级编辑，而法学硕士是否具有修改概念的能力仍不清楚。本文通过构建一个新颖的基准数据集 ConceptEdit 并建立一套新的评估指标，开创了法学硕士编辑概念知识的研究。实验结果表明，虽然现有的编辑方法可以在一定程度上有效地修改概念级定义，但它们也有可能扭曲法学硕士中的相关即时知识，导致性能不佳。我们预计这可以激发人们更好地理解法学硕士的进一步进展。我们的项目主页位于 https://zjunlp.github.io/project/ConceptEdit。

Title: Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance

• Authors: Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, Reut Tsarfaty
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06265
• Pdf URL: https://arxiv.org/pdf/2403.06265
• Copy Paste: [[2403.06265]] Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance(https://arxiv.org/abs/2403.06265)
• Keywords: language model
• Abstract: Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear. In this paper, we argue for the theoretical importance of compression, that can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. We also demonstrate the empirical importance of compression for downstream success of pre-trained language models. We control the compression ability of several BPE tokenizers by varying the amount of documents available during their training: from 1 million documents to a character-based tokenizer equivalent to no training data at all. We then pre-train English language models based on those tokenizers and fine-tune them over several tasks. We show that there is a correlation between tokenizers' compression and models' downstream performance, suggesting that compression is a reliable intrinsic indicator of tokenization quality. These correlations are more pronounced for generation tasks (over classification) or for smaller models (over large ones). We replicated a representative part of our experiments on Turkish and found similar results, confirming that our results hold for languages with typological characteristics dissimilar to English. We conclude that building better compressing tokenizers is a fruitful avenue for further research and for improving overall model performance.
• 摘要：尽管它是最常见的标记化算法 BPE 的基石，但压缩在标记化过程中的重要性仍不清楚。在本文中，我们论证了压缩的理论重要性，可以将其视为 0-gram 语言模型，其中为所有标记分配相同的概率。我们还证明了压缩对于预训练语言模型下游成功的经验重要性。我们通过改变训练期间可用文档的数量来控制多个 BPE 分词器的压缩能力：从 100 万个文档到相当于根本没有训练数据的基于字符的分词器。然后，我们根据这些分词器预训练英语语言模型，并在多个任务中对它们进行微调。我们表明，分词器的压缩和模型的下游性能之间存在相关性，这表明压缩是分词质量的可靠内在指标。这些相关性对于生成任务（超过分类）或较小的模型（超过大模型）更为明显。我们在土耳其语上复制了实验的代表性部分，并发现了类似的结果，证实我们的结果适用于类型特征与英语不同的语言。我们的结论是，构建更好的压缩分词器是进一步研究和提高整体模型性能的有效途径。

Title: LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition

• Authors: Xiaomeng Zhu, Robert Frank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06301
• Pdf URL: https://arxiv.org/pdf/2403.06301
• Copy Paste: [[2403.06301]] LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition(https://arxiv.org/abs/2403.06301)
• Keywords: language model
• Abstract: Discourse Entity (DE) recognition is the task of identifying novel and known entities introduced within a text. While previous work has found that large language models have basic, if imperfect, DE recognition abilities (Schuster and Linzen, 2022), it remains largely unassessed which of the fundamental semantic properties that govern the introduction and subsequent reference to DEs they have knowledge of. We propose the Linguistically-Informed Evaluation for Discourse Entity Recognition (LIEDER) dataset that allows for a detailed examination of language models' knowledge of four crucial semantic properties: existence, uniqueness, plurality, and novelty. We find evidence that state-of-the-art large language models exhibit sensitivity to all of these properties except novelty, which demonstrates that they have yet to reach human-level language understanding abilities.
• 摘要：话语实体（DE）识别是识别文本中引入的新颖和已知实体的任务。虽然之前的工作发现大型语言模型具有基本的（尽管不完美）DE 识别能力（Schuster 和 Linzen，2022），但在很大程度上仍未评估哪些基本语义属性控制着它们所了解的 DE 的引入和后续引用。我们提出了话语实体识别的语言信息评估（LIEDER）数据集，该数据集允许详细检查语言模型对四个关键语义属性的了解：存在性、唯一性、多样性和新颖性。我们发现证据表明，最先进的大型语言模型对除新颖性之外的所有这些属性都表现出敏感性，这表明它们尚未达到人类水平的语言理解能力。

Title: From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification

• Authors: Fei Wang, Chao Shang, Sarthak Jain, Shuai Wang, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, Dan Roth
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06326
• Pdf URL: https://arxiv.org/pdf/2403.06326
• Copy Paste: [[2403.06326]] From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification(https://arxiv.org/abs/2403.06326)
• Keywords: language model, prompt
• Abstract: User alignment is crucial for adapting general-purpose language models (LMs) to downstream tasks, but human annotations are often not available for all types of instructions, especially those with customized constraints. We observe that user instructions typically contain constraints. While assessing response quality in terms of the whole instruction is often costly, efficiently evaluating the satisfaction rate of constraints is feasible. We investigate common constraints in NLP tasks, categorize them into three classes based on the types of their arguments, and propose a unified framework, ACT (Aligning to ConsTraints), to automatically produce supervision signals for user alignment with constraints. Specifically, ACT uses constraint verifiers, which are typically easy to implement in practice, to compute constraint satisfaction rate (CSR) of each response. It samples multiple responses for each prompt and collect preference labels based on their CSR automatically. Subsequently, ACT adapts the LM to the target task through a ranking-based learning process. Experiments on fine-grained entity typing, abstractive summarization, and temporal question answering show that ACT is able to enhance LMs' capability to adhere to different classes of constraints, thereby improving task performance. Further experiments show that the constraint-following capabilities are transferable.
• 摘要：用户对齐对于使通用语言模型 (LM) 适应下游任务至关重要，但人工注释通常不适用于所有类型的指令，尤其是那些具有自定义约束的指令。我们观察到用户指令通常包含约束。虽然根据整个指令评估响应质量通常成本高昂，但有效评估约束的满足率是可行的。我们研究了 NLP 任务中的常见约束，根据参数类型将它们分为三类，并提出了一个统一的框架 ACT（Aligning to ConsTraints），以自动生成监督信号，以便用户与约束对齐。具体来说，ACT 使用约束验证器（通常在实践中很容易实现）来计算每个响应的约束满足率 (CSR)。它对每个提示的多个响应进行采样，并根据其 CSR 自动收集偏好标签。随后，ACT 通过基于排名的学习过程使 LM 适应目标任务。细粒度实体类型、抽象摘要和时间问答的实验表明，ACT 能够增强 LM 遵守不同类别约束的能力，从而提高任务性能。进一步的实验表明，约束跟随能力是可转移的。

Title: IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

• Authors: Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, Mitesh M. Khapra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06350
• Pdf URL: https://arxiv.org/pdf/2403.06350
• Copy Paste: [[2403.06350]] IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages(https://arxiv.org/abs/2403.06350)
• Keywords: llm, prompt
• Abstract: Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. Recognizing the importance of both data quality and quantity, our approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. We build a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. For instruction-fine tuning, we amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. Additionally, we address toxicity alignment by generating toxic prompts for multiple scenarios and then generate non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. We hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages. The data and other artifacts created as part of this work are released with permissive licenses.
• 摘要：尽管英语法学硕士取得了相当大的进步，但由于缺乏定制资源，为其他语言构建可比模型的进展受到阻碍。我们的工作旨在通过引入专为印度法学硕士开发而设计的广泛资源套件来弥合这一鸿沟，涵盖 22 种语言，总共包含 251B 个令牌和 7480 万个指令响应对。认识到数据质量和数量的重要性，我们的方法结合了精心策划的手动验证数据、未经验证但有价值的数据和合成数据。我们构建了一个干净的开源管道，用于管理来自不同来源（包括网站、PDF 和视频）的预训练数据，并结合爬行、清理、标记和重复数据删除的最佳实践。为了进行指令微调，我们合并了现有的印度语数据集，将英语数据集翻译/音译为印度语言，并利用 LLaMa2 和 Mixtral 模型创建基于印度维基百科和 Wikihow 文章的对话。此外，我们通过为多种场景生成有毒提示来解决毒性对齐问题，然后通过将这些有毒提示输入对齐的 LLaMa2 模型来生成无毒响应。我们希望作为这项工作的一部分发布的数据集、工具和资源不仅能够推动印度语法学硕士的研究和开发，还能建立一个开源蓝图，将这种努力扩展到其他语言。作为这项工作的一部分创建的数据和其他工件是通过许可发布的。

Title: Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

• Authors: Michael Andersland
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06354
• Pdf URL: https://arxiv.org/pdf/2403.06354
• Copy Paste: [[2403.06354]] Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages(https://arxiv.org/abs/2403.06354)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) like GPT-4 and LLaMA have shown incredible proficiency at natural language processing tasks and have even begun to excel at tasks across other modalities such as vision and audio. Despite their success, LLMs often struggle to perform well on low-resource languages because there is so little training data available. This shortcoming is especially prevalent with open source models. In this work, we explore training LLaMA-2 to speak Amharic, a language which is spoken by over 50 million people world wide, but has orders of magnitude less data available than languages like English. We employ methods previously used for training LLMs on other languages with data scarcity, and use open source translation models to perform data augmentation and grow our dataset from millions of tokens to billions. We further enhance the capabilities of our model by connecting an image encoder and training on a translated visual instruction tuning dataset in the same manner as LLaVA, resulting in a multimodal Amharic LLM that can understand images along with text. We introduce an Amharic version of a popular benchmarking dataset to evaluate our work. Our models and dataset are open sourced and available on GitHub.
• 摘要：像 GPT-4 和 LLaMA 这样的大型语言模型 (LLM) 在自然语言处理任务方面表现出了令人难以置信的熟练程度，甚至开始擅长跨其他模态（例如视觉和音频）的任务。尽管法学硕士取得了成功，但由于可用的培训数据太少，法学硕士通常很难在资源匮乏的语言上表现良好。这个缺点在开源模型中尤其普遍。在这项工作中，我们探索训练 LLaMA-2 讲阿姆哈拉语，这种语言在全世界有超过 5000 万人使用，但可用数据比英语等语言少几个数量级。我们采用以前用于在数据稀缺的其他语言上训练法学硕士的方法，并使用开源翻译模型来执行数据增强，并将我们的数据集从数百万个令牌增长到数十亿个令牌。我们通过连接图像编码器并以与 LLaVA 相同的方式对翻译后的视觉指令调整数据集进行训练，进一步增强了模型的功能，从而形成了可以理解图像和文本的多模式阿姆哈拉语 LLM。我们引入了流行基准测试数据集的阿姆哈拉语版本来评估我们的工作。我们的模型和数据集是开源的，可在 GitHub 上获取。

Title: 'One size doesn't fit all': Learning how many Examples to use for In-Context Learning for Improved Text Classification

• Authors: Manish Chandra, Debasis Ganguly, Yiwen Li, Iadh Ounis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06402
• Pdf URL: https://arxiv.org/pdf/2403.06402
• Copy Paste: [[2403.06402]] 'One size doesn't fit all': Learning how many Examples to use for In-Context Learning for Improved Text Classification(https://arxiv.org/abs/2403.06402)
• Keywords: prompt
• Abstract: Predictive models in natural language processing (NLP) have evolved from training models from scratch to fine-tuning pre-trained models with labelled data. An extreme form of this fine-tuning involves in-context learning (ICL), where the output of a pre-trained generative model (frozen decoder parameters) is controlled only with variations in the input strings (called instructions or prompts). An important component of ICL is the use of a small number of labelled data instances as examples in the prompt. While existing work uses a static number of examples during inference for each data instance, in this paper we propose a novel methodology of dynamically adapting the number of examples as per the data. This is analogous to the use of a variable-sized neighborhood in k-nearest neighbors (k-NN) classifier. In our proposed workflow of adaptive ICL (AICL), the number of demonstrations to employ during the inference on a particular data instance is predicted by the Softmax posteriors of a classifier. The parameters of this classifier are fitted on the optimal number of examples in ICL required to correctly infer the label of each instance in the training set with the hypothesis that a test instance that is similar to a training instance should use the same (or a closely matching) number of few-shot examples. Our experiments show that our AICL method results in improvement in text classification task on several standard datasets.
• 摘要：自然语言处理（NLP）中的预测模型已经从从头开始的训练模型发展到使用标记数据微调预训练模型。这种微调的一种极端形式涉及上下文学习（ICL），其中预训练生成模型（冻结解码器参数）的输出仅通过输入字符串（称为指令或提示）的变化进行控制。 ICL 的一个重要组成部分是在提示中使用少量标记数据实例作为示例。虽然现有工作在每个数据实例的推理过程中使用静态数量的示例，但在本文中，我们提出了一种根据数据动态调整示例数量的新颖方法。这类似于在 k 最近邻 (k-NN) 分类器中使用可变大小的邻域。在我们提出的自适应 ICL (AICL) 工作流程中，在对特定数据实例进行推理期间要使用的演示数量由分类器的 Softmax 后验预测。该分类器的参数适合 ICL 中正确推断训练集中每个实例的标签所需的最佳示例数量，假设与训练实例相似的测试实例应该使用相同的（或紧密的）匹配）少量样本的数量。我们的实验表明，我们的 AICL 方法可以改善几个标准数据集上的文本分类任务。

Title: CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean

• Authors: Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06412
• Pdf URL: https://arxiv.org/pdf/2403.06412
• Copy Paste: [[2403.06412]] CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean(https://arxiv.org/abs/2403.06412)
• Keywords: language model, llm
• Abstract: Despite the rapid development of large language models (LLMs) for the Korean language, there remains an obvious lack of benchmark datasets that test the requisite Korean cultural and linguistic knowledge. Because many existing Korean benchmark datasets are derived from the English counterparts through translation, they often overlook the different cultural contexts. For the few benchmark datasets that are sourced from Korean data capturing cultural knowledge, only narrow tasks such as bias and hate speech detection are offered. To address this gap, we introduce a benchmark of Cultural and Linguistic Intelligence in Korean (CLIcK), a dataset comprising 1,995 QA pairs. CLIcK sources its data from official Korean exams and textbooks, partitioning the questions into eleven categories under the two main categories of language and culture. For each instance in CLIcK, we provide fine-grained annotation of which cultural and linguistic knowledge is required to answer the question correctly. Using CLIcK, we test 13 language models to assess their performance. Our evaluation uncovers insights into their performances across the categories, as well as the diverse factors affecting their comprehension. CLIcK offers the first large-scale comprehensive Korean-centric analysis of LLMs' proficiency in Korean culture and language.
• 摘要：尽管韩语的大型语言模型（LLM）发展迅速，但仍然明显缺乏测试必要的韩国文化和语言知识的基准数据集。由于许多现有的韩语基准数据集是通过翻译从英语对应数据集衍生而来的，因此它们往往忽略了不同的文化背景。对于来自捕获文化知识的韩国数据的少数基准数据集，仅提供诸如偏见和仇恨言论检测之类的狭隘任务。为了解决这一差距，我们引入了韩语文化和语言智能基准 (CLIcK)，该数据集包含 1,995 个 QA 对。 CLIcK 的数据来自官方韩国语考试和教科书，将问题分为语言和文化两大类下的 11 类。对于 CLIcK 中的每个实例，我们都提供了正确回答问题所需的文化和语言知识的细粒度注释。我们使用 CLIcK 测试了 13 种语言模型来评估它们的性能。我们的评估揭示了他们在各个类别中的表现以及影响他们理解的各种因素。 CLIcK 首次对法学硕士对韩国文化和语言的熟练程度进行了大规模、以韩国为中心的综合分析。

Title: Evolving Knowledge Distillation with Large Language Models and Active Learning

• Authors: Chengyuan Liu, Yangyang Kang, Fubang Zhao, Kun Kuang, Zhuoren Jiang, Changlong Sun, Fei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06414
• Pdf URL: https://arxiv.org/pdf/2403.06414
• Copy Paste: [[2403.06414]] Evolving Knowledge Distillation with Large Language Models and Active Learning(https://arxiv.org/abs/2403.06414)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks. However, their computational costs are prohibitively high. To address this issue, previous research has attempted to distill the knowledge of LLMs into smaller models by generating annotated data. Nonetheless, these works have mainly focused on the direct use of LLMs for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. In this paper, we propose EvoKD: Evolving Knowledge Distillation, which leverages the concept of active learning to interactively enhance the process of data generation using large language models, simultaneously improving the task capabilities of small domain model (student model). Different from previous work, we actively analyze the student model's weaknesses, and then synthesize labeled samples based on the analysis. In addition, we provide iterative feedback to the LLMs regarding the student model's performance to continuously construct diversified and challenging samples. Experiments and analysis on different NLP tasks, namely, text classification and named entity recognition show the effectiveness of EvoKD.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务中表现出了卓越的能力。然而，它们的计算成本非常高。为了解决这个问题，之前的研究试图通过生成带注释的数据将法学硕士的知识提炼成更小的模型。尽管如此，这些工作主要集中在直接使用法学硕士进行文本生成和标记，而没有充分挖掘其理解目标任务和获取有价值知识的潜力。在本文中，我们提出了EvoKD：Evolving Knowledge Distillation，它利用主动学习的概念，交互式地增强使用大型语言模型的数据生成过程，同时提高小领域模型（学生模型）的任务能力。与之前的工作不同，我们积极分析学生模型的弱点，然后根据分析合成标记样本。此外，我们还向法学硕士提供有关学生模型表现的迭代反馈，以不断构建多样化且具有挑战性的样本。对不同 NLP 任务（即文本分类和命名实体识别）的实验和分析表明了 EvoKD 的有效性。

Title: Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models

• Authors: Weihang Su, Changyue Wang, Qingyao Ai, Yiran HU, Zhijing Wu, Yujia Zhou, Yiqun Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06448
• Pdf URL: https://arxiv.org/pdf/2403.06448
• Copy Paste: [[2403.06448]] Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models(https://arxiv.org/abs/2403.06448)
• Keywords: language model, llm, hallucination
• Abstract: Hallucinations in large language models (LLMs) refer to the phenomenon of LLMs producing responses that are coherent yet factually inaccurate. This issue undermines the effectiveness of LLMs in practical applications, necessitating research into detecting and mitigating hallucinations of LLMs. Previous studies have mainly concentrated on post-processing techniques for hallucination detection, which tend to be computationally intensive and limited in effectiveness due to their separation from the LLM's inference process. To overcome these limitations, we introduce MIND, an unsupervised training framework that leverages the internal states of LLMs for real-time hallucination detection without requiring manual annotations. Additionally, we present HELM, a new benchmark for evaluating hallucination detection across multiple LLMs, featuring diverse LLM outputs and the internal states of LLMs during their inference process. Our experiments demonstrate that MIND outperforms existing state-of-the-art methods in hallucination detection.
• 摘要：大语言模型 (LLM) 中的幻觉是指 LLM 产生连贯但实际上不准确的响应的现象。这个问题破坏了法学硕士在实际应用中的有效性，因此需要研究检测和减轻法学硕士的幻觉。先前的研究主要集中在幻觉检测的后处理技术上，由于它们与法学硕士的推理过程分离，因此往往需要大量计算，并且有效性有限。为了克服这些限制，我们引入了 MIND，这是一种无监督训练框架，它利用 LLM 的内部状态进行实时幻觉检测，而无需手动注释。此外，我们还推出了 HELM，这是一个用于评估多个 LLM 幻觉检测的新基准，具有不同的 LLM 输出以及 LLM 在推理过程中的内部状态。我们的实验表明，MIND 在幻觉检测方面优于现有最先进的方法。

Title: How to Understand Named Entities: Using Common Sense for News Captioning

• Authors: Ning Xu, Yanhui Wang, Tingting Zhang, Hongshuo Tian, Mohan Kankanhalli, An-An Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06520
• Pdf URL: https://arxiv.org/pdf/2403.06520
• Copy Paste: [[2403.06520]] How to Understand Named Entities: Using Common Sense for News Captioning(https://arxiv.org/abs/2403.06520)
• Keywords: agent
• Abstract: News captioning aims to describe an image with its news article body as input. It greatly relies on a set of detected named entities, including real-world people, organizations, and places. This paper exploits commonsense knowledge to understand named entities for news captioning. By ``understand'', we mean correlating the news content with common sense in the wild, which helps an agent to 1) distinguish semantically similar named entities and 2) describe named entities using words outside of training corpora. Our approach consists of three modules: (a) Filter Module aims to clarify the common sense concerning a named entity from two aspects: what does it mean? and what is it related to?, which divide the common sense into explanatory knowledge and relevant knowledge, respectively. (b) Distinguish Module aggregates explanatory knowledge from node-degree, dependency, and distinguish three aspects to distinguish semantically similar named entities. (c) Enrich Module attaches relevant knowledge to named entities to enrich the entity description by commonsense information (e.g., identity and social position). Finally, the probability distributions from both modules are integrated to generate the news captions. Extensive experiments on two challenging datasets (i.e., GoodNews and NYTimes) demonstrate the superiority of our method. Ablation studies and visualization further validate its effectiveness in understanding named entities.
• 摘要：新闻字幕旨在以新闻文章正文作为输入来描述图像。它很大程度上依赖于一组检测到的命名实体，包括现实世界的人员、组织和地点。本文利用常识知识来理解新闻字幕的命名实体。我们所说的“理解”是指将新闻内容与野外常识相关联，这有助于代理 1）区分语义上相似的命名实体，2）使用训练语料库之外的单词描述命名实体。我们的方法由三个模块组成：（a）过滤模块旨在从两个方面澄清有关命名实体的常识：它意味着什么？它与什么有关？分别将常识分为解释性知识和相关知识。 (b)区分模块从节点度、依赖度和区分三个方面聚合解释性知识来区分语义相似的命名实体。 (c)丰富模块将相关知识附加到命名实体，以通过常识信息（例如身份和社会地位）丰富实体描述。最后，整合两个模块的概率分布以生成新闻标题。对两个具有挑战性的数据集（即 GoodNews 和 NYTimes）的广泛实验证明了我们方法的优越性。消融研究和可视化进一步验证了其在理解命名实体方面的有效性。

Title: On the Consideration of AI Openness: Can Good Intent Be Abused?

• Authors: Yeeun Kim, Eunkyung Choi, Hyunjun Kim, Hongseok Oh, Hyunseo Shin, Wonseok Hwang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06537
• Pdf URL: https://arxiv.org/pdf/2403.06537
• Copy Paste: [[2403.06537]] On the Consideration of AI Openness: Can Good Intent Be Abused?(https://arxiv.org/abs/2403.06537)
• Keywords: llm
• Abstract: Openness is critical for the advancement of science. In particular, recent rapid progress in AI has been made possible only by various open-source models, datasets, and libraries. However, this openness also means that technologies can be freely used for socially harmful purposes. Can open-source models or datasets be used for malicious purposes? If so, how easy is it to adapt technology for such goals? Here, we conduct a case study in the legal domain, a realm where individual decisions can have profound social consequences. To this end, we build EVE, a dataset consisting of 200 examples of questions and corresponding answers about criminal activities based on 200 Korean precedents. We found that a widely accepted open-source LLM, which initially refuses to answer unethical questions, can be easily tuned with EVE to provide unethical and informative answers about criminal activities. This implies that although open-source technologies contribute to scientific progress, some care must be taken to mitigate possible malicious use cases. Warning: This paper contains contents that some may find unethical.
• 摘要：开放对于科学的进步至关重要。特别是，人工智能最近的快速进展是通过各种开源模型、数据集和库才得以实现的。然而，这种开放性也意味着技术可以自由地用于对社会有害的目的。开源模型或数据集是否可以用于恶意目的？如果是这样，为了实现这些目标而采用技术有多容易？在这里，我们在法律领域进行了案例研究，在这个领域中，个人的决定可能会产生深远的社会影响。为此，我们构建了 EVE，这是一个数据集，包含 200 个基于 200 个韩国先例的犯罪活动问题示例和相应答案。我们发现，一个被广泛接受的开源法学硕士最初拒绝回答不道德的问题，但可以通过 EVE 轻松调整，以提供有关犯罪活动的不道德和信息丰富的答案。这意味着，尽管开源技术有助于科学进步，但必须注意减少可能的恶意使用情况。警告：本文包含某些人可能认为不道德的内容。

Title: AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models

• Authors: Yuting Wei, Yuanxing Xu, Xinru Wei, Simin Yang, Yangfu Zhu, Yuqing Li, Di Liu, Bin Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06574
• Pdf URL: https://arxiv.org/pdf/2403.06574
• Copy Paste: [[2403.06574]] AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models(https://arxiv.org/abs/2403.06574)
• Keywords: language model, llm
• Abstract: Given the importance of ancient Chinese in capturing the essence of rich historical and cultural heritage, the rapid advancements in Large Language Models (LLMs) necessitate benchmarks that can effectively evaluate their understanding of ancient contexts. To meet this need, we present AC-EVAL, an innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese. AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension: general historical knowledge, short text understanding, and long text comprehension. The benchmark comprises 13 tasks, spanning historical facts, geography, social customs, art, philosophy, classical poetry and prose, providing a comprehensive assessment framework. Our extensive evaluation of top-performing LLMs, tailored for both English and Chinese, reveals a substantial potential for enhancing ancient text comprehension. By highlighting the strengths and weaknesses of LLMs, AC-EVAL aims to promote their development and application forward in the realms of ancient Chinese language education and scholarly research. The AC-EVAL data and evaluation code are available at https://github.com/yuting-wei/AC-EVAL.
• 摘要：鉴于古代汉语在捕捉丰富历史和文化遗产精髓方面的重要性，大型语言模型（LLM）的快速发展需要能够有效评估其对古代语境理解的基准。为了满足这一需求，我们推出了 AC-EVAL，这是一个创新基准，旨在评估法学硕士在古代汉语背景下的先进知识和推理能力。 AC-EVAL 分为三个难度级别，反映了语言理解的不同方面：一般历史知识、短文本理解和长文本理解。该基准包括13个任务，涵盖史实、地理、社会风俗、艺术、哲学、古典诗词、散文等领域，提供了一个全面的评估框架。我们对针对英语和中文的顶尖法学硕士进行了广泛的评估，揭示了增强古代文本理解的巨大潜力。 AC-EVAL旨在通过突出法学硕士的优势和劣势，促进其在古汉语教育和学术研究领域的发展和应用。 AC-EVAL数据和评估代码可在https://github.com/yuting-wei/AC-EVAL获取。

Title: Academically intelligent LLMs are not necessarily socially intelligent

• Authors: Ruoxi Xu, Hongyu Lin, Xianpei Han, Le Sun, Yingfei Sun
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.06591
• Pdf URL: https://arxiv.org/pdf/2403.06591
• Copy Paste: [[2403.06591]] Academically intelligent LLMs are not necessarily socially intelligent(https://arxiv.org/abs/2403.06591)
• Keywords: language model, llm, agent
• Abstract: The academic intelligence of large language models (LLMs) has made remarkable progress in recent times, but their social intelligence performance remains unclear. Inspired by established human social intelligence frameworks, particularly Daniel Goleman's social intelligence theory, we have developed a standardized social intelligence test based on real-world social scenarios to comprehensively assess the social intelligence of LLMs, termed as the Situational Evaluation of Social Intelligence (SESI). We conducted an extensive evaluation with 13 recent popular and state-of-art LLM agents on SESI. The results indicate the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors. Moreover, there exists a relatively low correlation between the social intelligence and academic intelligence exhibited by LLMs, suggesting that social intelligence is distinct from academic intelligence for LLMs. Additionally, while it is observed that LLMs can't ``understand'' what social intelligence is, their social intelligence, similar to that of humans, is influenced by social factors.
• 摘要：近年来，大语言模型（LLM）的学术智能取得了显着进步，但其社交智能表现仍不清楚。受人类既定社会智能框架，特别是丹尼尔·戈尔曼（Daniel Goleman）社会智能理论的启发，我们开发了基于现实社会场景的标准化社会智能测试，以全面评估法学硕士的社交智能，称为社会智能情境评估（SESI） 。我们对 SESI 上最近流行的 13 名最先进的法学硕士代理人进行了广泛的评估。结果表明，法学硕士的社交智力仍有很大的提升空间，表面上的友善是导致错误的主要原因。此外，法学硕士的社交智力和学术智力之间存在相对较低的相关性，这表明法学硕士的社交智力与学术智力不同。此外，虽然据观察，法学硕士无法“理解”什么是社交智力，但他们的社交智力与人类相似，受到社会因素的影响。

Title: Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds

• Authors: Jiageng WU, Xian Wu, Jie Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06609
• Pdf URL: https://arxiv.org/pdf/2403.06609
• Copy Paste: [[2403.06609]] Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds(https://arxiv.org/abs/2403.06609)
• Keywords: language model, gpt, llm, hallucination, chat
• Abstract: Clinical reasoning refers to the cognitive process that physicians employ in evaluating and managing patients. This process typically involves suggesting necessary examinations, diagnosing patients' diseases, and deciding on appropriate therapies, etc. Accurate clinical reasoning requires extensive medical knowledge and rich clinical experience, setting a high bar for physicians. This is particularly challenging in developing countries due to the overwhelming number of patients and limited physician resources, contributing significantly to global health inequity and necessitating automated clinical reasoning approaches. Recently, the emergence of large language models (LLMs) such as ChatGPT and GPT-4 have demonstrated their potential in clinical reasoning. However, these LLMs are prone to hallucination problems, and the reasoning process of LLMs may not align with the clinical decision path of physicians. In this study, we introduce a novel framework, In-Context Padding (ICP), designed to enhance LLMs with medical knowledge. Specifically, we infer critical clinical reasoning elements (referred to as knowledge seeds) and use these as anchors to guide the generation process of LLMs. Experiments on two clinical question datasets demonstrate that ICP significantly improves the clinical reasoning ability of LLMs.
• 摘要：临床推理是指医生在评估和管理患者时采用的认知过程。这个过程通常涉及建议必要的检查、诊断患者的疾病以及决定适当的治疗方法等。准确的临床推理需要广泛的医学知识和丰富的临床经验，这对医生提出了很高的要求。这在发展中国家尤其具有挑战性，因为患者数量巨大且医生资源有限，这极大地加剧了全球健康不平等，并需要自动化的临床推理方法。最近，ChatGPT和GPT-4等大型语言模型（LLM）的出现证明了它们在临床推理中的潜力。然而，这些法学硕士很容易出现幻觉问题，而且法学硕士的推理过程可能与医生的临床决策路径不一致。在这项研究中，我们引入了一种新颖的框架，即上下文填充（ICP），旨在增强法学硕士的医学知识。具体来说，我们推断关键的临床推理元素（称为知识种子），并使用它们作为锚点来指导法学硕士的生成过程。对两个临床问题数据集的实验表明，ICP 显着提高了法学硕士的临床推理能力。

Title: MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway Encoding

• Authors: Jiageng Wu, Xian Wu, Yefeng Zheng, Jie Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06611
• Pdf URL: https://arxiv.org/pdf/2403.06611
• Copy Paste: [[2403.06611]] MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway Encoding(https://arxiv.org/abs/2403.06611)
• Keywords: language model, llm, hallucination
• Abstract: With appropriate data selection and training techniques, Large Language Models (LLMs) have demonstrated exceptional success in various medical examinations and multiple-choice questions. However, the application of LLMs in medical dialogue generation-a task more closely aligned with actual medical practice-has been less explored. This gap is attributed to the insufficient medical knowledge of LLMs, which leads to inaccuracies and hallucinated information in the generated medical responses. In this work, we introduce the Medical dialogue with Knowledge enhancement and clinical Pathway encoding (MedKP) framework, which integrates an external knowledge enhancement module through a medical knowledge graph and an internal clinical pathway encoding via medical entities and physician actions. Evaluated with comprehensive metrics, our experiments on two large-scale, real-world online medical consultation datasets (MedDG and KaMed) demonstrate that MedKP surpasses multiple baselines and mitigates the incidence of hallucinations, achieving a new state-of-the-art. Extensive ablation studies further reveal the effectiveness of each component of MedKP. This enhancement advances the development of reliable, automated medical consultation responses using LLMs, thereby broadening the potential accessibility of precise and real-time medical assistance.
• 摘要：通过适当的数据选择和训练技术，大型语言模型（LLM）在各种医学检查和多项选择题中表现出了非凡的成功。然而，法学硕士在医学对话生成中的应用（一项与实际医疗实践更紧密结合的任务）的探索较少。这种差距归因于法学硕士的医学知识不足，导致生成的医疗反应中的信息不准确和产生幻觉。在这项工作中，我们引入了知识增强和临床路径编码的医学对话（MedKP）框架，该框架通过医学知识图集成了外部知识增强模块，并通过医疗实体和医生行为进行了内部临床路径编码。通过综合指标评估，我们在两个大型真实在线医疗咨询数据集（MedDG 和 KaMed）上进行的实验表明，MedKP 超越了多个基线并减轻了幻觉的发生率，达到了新的最先进水平。广泛的消融研究进一步揭示了 MedKP 各个组成部分的有效性。这一增强功能促进了使用法学硕士的可靠、自动化医疗咨询响应的开发，从而扩大了精确和实时医疗援助的潜在可及性。

Title: ACT-MNMT Auto-Constriction Turning for Multilingual Neural Machine Translation

• Authors: Shaojie Dai, Xin Liu, Ping Luo, Yue Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06745
• Pdf URL: https://arxiv.org/pdf/2403.06745
• Copy Paste: [[2403.06745]] ACT-MNMT Auto-Constriction Turning for Multilingual Neural Machine Translation(https://arxiv.org/abs/2403.06745)
• Keywords: language model, llm, prompt
• Abstract: Large language model (LLM) has achieved promising performance in multilingual machine translation tasks through zero/few-shot prompts or prompt-tuning. However, due to the mixture of multilingual data during the pre-training of LLM, the LLM-based translation models face the off-target issue in both prompt-based methods, including a series of phenomena, namely instruction misunderstanding, translation with wrong language and over-generation. For this issue, this paper introduces an \textbf{\underline{A}}uto-\textbf{\underline{C}}onstriction \textbf{\underline{T}}urning mechanism for \textbf{\underline{M}}ultilingual \textbf{\underline{N}}eural \textbf{\underline{M}}achine \textbf{\underline{T}}ranslation (\model), which is a novel supervised fine-tuning mechanism and orthogonal to the traditional prompt-based methods. In this method, \model automatically constructs a constrained template in the target side by adding trigger tokens ahead of the ground truth. Furthermore, trigger tokens can be arranged and combined freely to represent different task semantics, and they can be iteratively updated to maximize the label likelihood. Experiments are performed on WMT test sets with multiple metrics, and the experimental results demonstrate that \model achieves substantially improved performance across multiple translation directions and reduce the off-target phenomena in the translation.
• 摘要：大语言模型（LLM）通过零/少样本提示或提示调整，在多语言机器翻译任务中取得了可喜的性能。然而，由于LLM预训练时多语言数据的混合，基于LLM的翻译模型在两种基于提示的方法中都面临着脱靶问题，包括指令误解、翻译错误等一系列现象。和超一代。针对这个问题，本文介绍了一种针对 \textbf{\underline{M}} 的 \textbf{\underline{A}}uto-\textbf{\underline{C}}限制 \textbf{\underline{T}} 机制多语言\textbf{\underline{N}}eural \textbf{\underline{M}}achine \textbf{\underline{T}}翻译（\model），这是一种新颖的监督微调机制，与传统的基于提示的方法。在这种方法中，\model 通过在真实值之前添加触发标记，自动在目标端构建约束模板。此外，触发令牌可以自由排列和组合以表示不同的任务语义，并且可以迭代更新它们以最大化标签可能性。在具有多个指标的WMT测试集上进行实验，实验结果表明模型在多个翻译方向上实现了性能的大幅提升，并减少了翻译中的脱靶现象。

Title: ALaRM: Align Language Models via Hierarchical Rewards Modeling

• Authors: Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06754
• Pdf URL: https://arxiv.org/pdf/2403.06754
• Copy Paste: [[2403.06754]] ALaRM: Align Language Models via Hierarchical Rewards Modeling(https://arxiv.org/abs/2403.06754)
• Keywords: language model, gpt, llm
• Abstract: We introduce ALaRM, the first framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF), which is designed to enhance the alignment of large language models (LLMs) with human preferences. The framework addresses the limitations of current alignment approaches, which often struggle with the inconsistency and sparsity of human supervision signals, by integrating holistic rewards with aspect-specific rewards. This integration enables more precise and consistent guidance of language models towards desired outcomes, particularly in complex and open text generation tasks. By employing a methodology that filters and combines multiple rewards based on their consistency, the framework provides a reliable mechanism for improving model alignment. We validate our approach through applications in long-form question answering and machine translation tasks, employing gpt-3.5-turbo for pairwise comparisons, and demonstrate improvements over existing baselines. Our work underscores the effectiveness of hierarchical rewards modeling in refining LLM training processes for better human preference alignment. We release our code at https://ALaRM-fdu.github.io.
• 摘要：我们推出了 ALaRM，这是第一个根据人类反馈（RLHF）对强化学习中的分层奖励进行建模的框架，旨在增强大型语言模型（LLM）与人类偏好的一致性。该框架通过将整体奖励与特定方面的奖励相结合，解决了当前对齐方法的局限性，这些方法经常与人类监督信号的不一致和稀疏性作斗争。这种集成可以更精确、一致地指导语言模型实现预期结果，特别是在复杂和开放的文本生成任务中。通过采用一种根据一致性来过滤和组合多个奖励的方法，该框架提供了一种可靠的机制来改进模型一致性。我们通过在长格式问答和机器翻译任务中的应用来验证我们的方法，使用 gpt-3.5-turbo 进行成对比较，并展示对现有基线的改进。我们的工作强调了分层奖励模型在完善法学硕士培训流程以更好地调整人类偏好方面的有效性。我们在 https://ALaRM-fdu.github.io 发布了我们的代码。

Title: ConspEmoLLM: Conspiracy Theory Detection Using an Emotion-Based Large Language Model

• Authors: Zhiwei Liu, Boyang Liu, Paul Thompson, Kailai Yang, Raghav Jain, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06765
• Pdf URL: https://arxiv.org/pdf/2403.06765
• Copy Paste: [[2403.06765]] ConspEmoLLM: Conspiracy Theory Detection Using an Emotion-Based Large Language Model(https://arxiv.org/abs/2403.06765)
• Keywords: language model, gpt, llm, chat
• Abstract: The internet has brought both benefits and harms to society. A prime example of the latter is misinformation, including conspiracy theories, which flood the web. Recent advances in natural language processing, particularly the emergence of large language models (LLMs), have improved the prospects of accurate misinformation detection. However, most LLM-based approaches to conspiracy theory detection focus only on binary classification and fail to account for the important relationship between misinformation and affective features (i.e., sentiment and emotions). Driven by a comprehensive analysis of conspiracy text that reveals its distinctive affective features, we propose ConspEmoLLM, the first open-source LLM that integrates affective information and is able to perform diverse tasks relating to conspiracy theories. These tasks include not only conspiracy theory detection, but also classification of theory type and detection of related discussion (e.g., opinions towards theories). ConspEmoLLM is fine-tuned based on an emotion-oriented LLM using our novel ConDID dataset, which includes five tasks to support LLM instruction tuning and evaluation. We demonstrate that when applied to these tasks, ConspEmoLLM largely outperforms several open-source general domain LLMs and ChatGPT, as well as an LLM that has been fine-tuned using ConDID, but which does not use affective features. This project will be released on https://github.com/lzw108/ConspEmoLLM/.
• 摘要：互联网给社会带来了好处，也带来了危害。后者的一个典型例子是错误信息，包括充斥网络的阴谋论。自然语言处理的最新进展，特别是大型语言模型（LLM）的出现，改善了准确错误信息检测的前景。然而，大多数基于法学硕士的阴谋论检测方法仅关注二元分类，而未能考虑错误信息和情感特征（即情绪和情绪）之间的重要关系。在对阴谋文本的全面分析揭示其独特的情感特征的推动下，我们提出了ConspEmoLLM，这是第一个集成情感信息并能够执行与阴谋论相关的各种任务的开源法学硕士。这些任务不仅包括阴谋论检测，还包括理论类型的分类和相关讨论的检测（例如，对理论的看法）。 ConspEmoLLM 使用我们新颖的 ConDID 数据集基于面向情感的 LLM 进行微调，其中包括支持 LLM 指令调整和评估的五个任务。我们证明，当应用于这些任务时，ConspEmoLLM 在很大程度上优于几个开源通用领域 LLM 和 ChatGPT，以及使用 ConDID 进行微调但不使用情感特征的 LLM。该项目将在 https://github.com/lzw108/ConspEmoLLM/ 上发布。

Title: Strength Lies in Differences! Towards Effective Non-collaborative Dialogues via Tailored Strategy Planning

• Authors: Tong Zhang, Chen Huang, Yang Deng, Hongru Liang, Jia Liu, Zujie Wen, Wenqiang Lei, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06769
• Pdf URL: https://arxiv.org/pdf/2403.06769
• Copy Paste: [[2403.06769]] Strength Lies in Differences! Towards Effective Non-collaborative Dialogues via Tailored Strategy Planning(https://arxiv.org/abs/2403.06769)
• Keywords: agent
• Abstract: We investigate non-collaborative dialogue agents that must engage in tailored strategic planning for diverse users to secure a favorable agreement. This poses challenges for existing dialogue agents due to two main reasons: their inability to integrate user-specific characteristics into their strategic planning and their training paradigm's failure to produce strategic planners that can generalize to diverse users. To address these challenges, we propose TRIP to enhance the capability in tailored strategic planning, incorporating a user-aware strategic planning module and a population-based training paradigm. Through experiments on benchmark non-collaborative dialogue tasks, we demonstrate the effectiveness of TRIP in catering to diverse users.
• 摘要：我们研究非协作对话代理，它们必须为不同的用户进行量身定制的战略规划，以确保达成有利的协议。这给现有的对话代理带来了挑战，原因有两个：他们无法将用户特定的特征整合到他们的战略规划中，以及他们的训练范式无法产生可以推广到不同用户的战略规划者。为了应对这些挑战，我们建议 TRIP 增强定制战略规划的能力，结合用户意识的战略规划模块和基于人群的培训范例。通过基准非协作对话任务的实验，我们证明了 TRIP 在满足不同用户需求方面的有效性。

Title: The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework

• Authors: Zhuo Chen, Yin Fang, Yichi Zhang, Lingbing Guo, Jiaoyan Chen, Huajun Chen, Wen Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06832
• Pdf URL: https://arxiv.org/pdf/2403.06832
• Copy Paste: [[2403.06832]] The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework(https://arxiv.org/abs/2403.06832)
• Keywords: language model, llm, hallucination
• Abstract: The advancement of Multi-modal Pre-training highlights the necessity for a robust Multi-Modal Knowledge Graph (MMKG) representation learning framework. This framework is crucial for integrating structured knowledge into multi-modal Large Language Models (LLMs) at scale, aiming to alleviate issues like knowledge misconceptions and multi-modal hallucinations. In this work, to evaluate models' ability to accurately embed entities within MMKGs, we focus on two widely researched tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking for the robust integration of multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets (three for MKGC and seven for MEMA), demonstrating its robustness and versatility. Besides, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Our code and data are available at: https://github.com/zjukg/SNAG.
• 摘要：多模态预训练的进步凸显了强大的多模态知识图（MMKG）表示学习框架的必要性。该框架对于将结构化知识大规模集成到多模态大语言模型（LLM）中至关重要，旨在缓解知识误解和多模态幻觉等问题。在这项工作中，为了评估模型在 MMKG 中准确嵌入实体的能力，我们重点关注两个广泛研究的任务：多模态知识图完成（MKGC）和多模态实体对齐（MMEA）。在此基础上，我们提出了一种新颖的 SNAG 方法，该方法利用基于 Transformer 的架构，配备模态级噪声掩蔽，用于知识图谱中多模态实体特征的稳健集成。通过结合 MKGC 和 MMEA 的特定训练目标，我们的方法在总共 10 个数据集（三个用于 MKGC，七个用于 MEMA）上实现了 SOTA 性能，展示了其稳健性和多功能性。此外，SNAG不仅可以作为独立模型运行，还可以增强其他现有方法，提供稳定的性能改进。我们的代码和数据可在：https://github.com/zjukg/SNAG 获取。

Title: RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback

• Authors: Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, Tianyu Du
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06840
• Pdf URL: https://arxiv.org/pdf/2403.06840
• Copy Paste: [[2403.06840]] RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback(https://arxiv.org/abs/2403.06840)
• Keywords: language model, gpt, llm, hallucination, retrieval-augmented generation
• Abstract: Large language models (LLMs) demonstrate exceptional performance in numerous tasks but still heavily rely on knowledge stored in their parameters. Moreover, updating this knowledge incurs high training costs. Retrieval-augmented generation (RAG) methods address this issue by integrating external knowledge. The model can answer questions it couldn't previously by retrieving knowledge relevant to the query. This approach improves performance in certain scenarios for specific tasks. However, if irrelevant texts are retrieved, it may impair model performance. In this paper, we propose Retrieval Augmented Iterative Self-Feedback (RA-ISF), a framework that iteratively decomposes tasks and processes them in three submodules to enhance the model's problem-solving capabilities. Experiments show that our method outperforms existing benchmarks, performing well on models like GPT3.5, Llama2, significantly enhancing factual reasoning capabilities and reducing hallucinations.
• 摘要：大型语言模型 (LLM) 在许多任务中表现出卓越的性能，但仍然严重依赖于存储在其参数中的知识。此外，更新这些知识会带来高昂的培训成本。检索增强生成（RAG）方法通过整合外部知识来解决这个问题。该模型可以通过检索与查询相关的知识来回答以前无法回答的问题。这种方法可以提高特定任务在某些场景下的性能。但是，如果检索到不相关的文本，可能会损害模型性能。在本文中，我们提出了检索增强迭代自反馈（RA-ISF），这是一个迭代分解任务并在三个子模块中处理它们的框架，以增强模型解决问题的能力。实验表明，我们的方法优于现有基准，在 GPT3.5、Llama2 等模型上表现良好，显着增强了事实推理能力并减少了幻觉。

Title: Development of a Reliable and Accessible Caregiving Language Model (CaLM)

• Authors: Bambang Parmanto, Bayu Aryoyudanta, Wilbert Soekinto, I Made Agus Setiawan, Yuhan Wang, Haomin Hu, Andi Saptono, Yong K. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06857
• Pdf URL: https://arxiv.org/pdf/2403.06857
• Copy Paste: [[2403.06857]] Development of a Reliable and Accessible Caregiving Language Model (CaLM)(https://arxiv.org/abs/2403.06857)
• Keywords: language model, gpt, retrieval augmented generation
• Abstract: Unlike professional caregivers, family caregivers often assume this role without formal preparation or training. Because of this, there is an urgent need to enhance the capacity of family caregivers to provide quality care. Large language models can potentially be used as a foundation technology for supporting caregivers as educational tools or as adjunct to care. This study aimed to develop a reliable Caregiving Language Model (CaLM) by using FMs and a caregiving knowledge base, develop an accessible CaLM using a small FM that requires fewer computing resources, and evaluate the performance of the model compared to a large FM. We developed CaLM using the Retrieval Augmented Generation (RAG) framework combined with FM fine-tuning for improving the quality of FM answers by grounding the model on a caregiving knowledge base. We used two small FMs as candidates for the FM of CaLM (LLaMA-2 and Falcon with 7B parameters) and larger FM GPT-3.5 as a benchmark. We developed the caregiving knowledge base by gathering various types of documents from the Internet. In this study, we focused on caregivers of individuals with Alzheimer's Disease Related Dementias. We evaluated the models' performance using the benchmark metrics commonly used in evaluating language models and their reliability to provide accurate references with the answers. The RAG framework improved the performance of all FMs used in this study across all measures. As expected, the large FM performed better than small FMs across all metrics. The most interesting result is that small fine-tuned FMs with RAG performed significantly better than GPT 3.5 across all metrics. The fine-tuned LLaMA-2 small FM performed better than GPT 3.5 (even with RAG) in returning references with the answers. The study shows that reliable and accessible CaLM can be developed by using small FMs with a knowledge base specific to the caregiving domain.
• 摘要：与专业护理人员不同，家庭护理人员通常在没有正式准备或培训的情况下承担这一角色。因此，迫切需要提高家庭护理人员提供优质护理的能力。大型语言模型有可能用作支持护理人员的基础技术，作为教育工具或护理的辅助手段。本研究旨在通过使用 FM 和护理知识库开发可靠的护理语言模型 (CaLM)，使用需要较少计算资源的小型 FM 开发可访问的 CaLM，并评估该模型与大型 FM 相比的性能。我们使用检索增强生成 (RAG) 框架结合 FM 微调开发了 CaLM，通过将模型建立在护理知识库上来提高 FM 答案的质量。我们使用两个小型 FM 作为 CaLM FM 的候选者（LLaMA-2 和具有 7B 参数的 Falcon），并使用较大的 FM GPT-3.5 作为基准。我们通过从互联网上收集各种类型的文档来开发护理知识库。在这项研究中，我们重点关注患有阿尔茨海默病相关痴呆症的患者的护理人员。我们使用评估语言模型及其可靠性时常用的基准指标来评估模型的性能，以提供准确的参考答案。 RAG 框架提高了本研究中使用的所有 FM 的所有指标的性能。正如预期的那样，大型 FM 在所有指标上的表现均优于小型 FM。最有趣的结果是，带有 RAG 的小型微调 FM 在所有指标上的表现均明显优于 GPT 3.5。经过微调的 LLaMA-2 小型 FM 在返回带有答案的参考文献方面表现优于 GPT 3.5（即使使用 RAG）。研究表明，可以通过使用具有特定于护理领域的知识库的小型 FM 来开发可靠且易于使用的 CaLM。

Title: Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents

• Authors: Nishchal Prasad, Mohand Boughanem, Taoufiq Dkaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06872
• Pdf URL: https://arxiv.org/pdf/2403.06872
• Copy Paste: [[2403.06872]] Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents(https://arxiv.org/abs/2403.06872)
• Keywords: language model, gpt, llm
• Abstract: Legal judgment prediction suffers from the problem of long case documents exceeding tens of thousands of words, in general, and having a non-uniform structure. Predicting judgments from such documents becomes a challenging task, more so on documents with no structural annotation. We explore the classification of these large legal documents and their lack of structural information with a deep-learning-based hierarchical framework which we call MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment prediction. Specifically, we divide a document into parts to extract their embeddings from the last four layers of a custom fine-tuned Large Language Model, and try to approximate their structure through unsupervised clustering. Which we use in another set of transformer encoder layers to learn the inter-chunk representations. We analyze the adaptability of Large Language Models (LLMs) with multi-billion parameters (GPT-Neo, and GPT-J) with the hierarchical framework of MESc and compare them with their standalone performance on legal texts. We also study their intra-domain(legal) transfer learning capability and the impact of combining embeddings from their last layers in MESc. We test these methods and their effectiveness with extensive experiments and ablation studies on legal documents from India, the European Union, and the United States with the ILDC dataset and a subset of the LexGLUE dataset. Our approach achieves a minimum total performance gain of approximately 2 points over previous state-of-the-art methods.
• 摘要：法律判决预测存在案件文书一般超过数万字、结构不统一的问题。从此类文档中预测判断成为一项具有挑战性的任务，对于没有结构注释的文档更是如此。我们利用基于深度学习的分层框架（我们称之为 MESc）来探索这些大型法律文件的分类及其缺乏结构信息的情况； “基于多级编码器的有监督聚类”；用于判断预测。具体来说，我们将文档分成几个部分，从自定义微调大型语言模型的最后四层中提取它们的嵌入，并尝试通过无监督聚类来近似它们的结构。我们在另一组变压器编码器层中使用它来学习块间表示。我们分析了具有数十亿参数的大型语言模型（LLM）（GPT-Neo 和 GPT-J）与 MESc 分层框架的适应性，并将它们与它们在法律文本上的独立性能进行比较。我们还研究了它们的域内（法律）迁移学习能力以及在 MESc 中结合最后一层嵌入的影响。我们使用 ILDC 数据集和 LexGLUE 数据集的子集，对印度、欧盟和美国的法律文件进行了广泛的实验和消融研究，测试了这些方法及其有效性。与以前最先进的方法相比，我们的方法实现了约 2 个点的最小总性能增益。

Title: MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning

• Authors: Yichuan Li, Xiyao Ma, Sixing Lu, Kyumin Lee, Xiaohu Liu, Chenlei Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06914
• Pdf URL: https://arxiv.org/pdf/2403.06914
• Copy Paste: [[2403.06914]] MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning(https://arxiv.org/abs/2403.06914)
• Keywords: language model, gpt, llm
• Abstract: Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations leads to a quadratic increase in the computational overhead of the self-attention mechanism. Existing solutions attempt to distill lengthy demonstrations into compact vectors. However, they often require task-specific retraining or compromise LLM's in-context learning performance. To mitigate these challenges, we present Meta dEmonstratioN Distillation (MEND), where a language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. We exploit the knowledge distillation to enhance alignment between MEND and LLM, achieving both efficiency and effectiveness simultaneously. MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning. Comprehensive evaluations across seven diverse ICL task partitions using decoder-only (GPT-2) and encoder-decoder (T5) attest to MEND's prowess. It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. This innovation promises enhanced scalability and efficiency for the practical deployment of large language models
• 摘要：大型语言模型 (LLM) 已经展示了令人印象深刻的上下文学习 (ICL) 功能，其中 LLM 对给定的测试输入以及一些输入输出对（演示）进行预测。然而，包含演示会导致自注意力机制的计算开销成倍增加。现有的解决方案试图将冗长的演示提炼成紧凑的向量。然而，它们通常需要针对特定​​任务的再培训或损害法学硕士的情境学习表现。为了缓解这些挑战，我们提出了元演示蒸馏（MEND），其中语言模型学习将任何冗长的演示蒸馏为向量，而无需针对新的下游任务进行重新训练。我们利用知识蒸馏来增强 MEND 和 LLM 之间的一致性，同时实现效率和效果。 MEND 通过两阶段训练过程赋予了蒸馏演示的元知识，其中包括元蒸馏预训练和微调。使用仅解码器 (GPT-2) 和编码器-解码器 (T5) 对七个不同的 ICL 任务分区进行的综合评估证明了 MEND 的实力。它不仅匹配甚至优于 Vanilla ICL 以及其他最先进的蒸馏模型，同时显着降低了计算需求。这项创新有望增强大型语言模型实际部署的可扩展性和效率

Title: ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis

• Authors: Yanming Liu, Xinyue Peng, Tianyu Du, Jianwei Yin, Weihao Liu, Xuhong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06932
• Pdf URL: https://arxiv.org/pdf/2403.06932
• Copy Paste: [[2403.06932]] ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis(https://arxiv.org/abs/2403.06932)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) have achieved commendable accomplishments in various natural language processing tasks. However, LLMs still encounter significant challenges when dealing with complex scenarios involving multiple entities. These challenges arise from the presence of implicit relationships that demand multi-step reasoning. In this paper, we propose a novel approach ERA-CoT, which aids LLMs in understanding context by capturing relationships between entities and supports the reasoning of diverse tasks through Chain-of-Thoughts (CoT). Experimental results show that ERA-CoT demonstrates the superior performance of our proposed method compared to current CoT prompting methods, achieving a significant improvement of an average of 5.1\% on GPT3.5 compared to previous SOTA baselines. Our analysis indicates that ERA-CoT increases the LLM's understanding of entity relationships, significantly improves the accuracy of question answering, and enhances the reasoning ability of LLMs.
• 摘要：大型语言模型（LLM）在各种自然语言处理任务中取得了令人称赞的成就。然而，法学硕士在处理涉及多个实体的复杂场景时仍然面临重大挑战。这些挑战源于需要多步骤推理的隐含关系的存在。在本文中，我们提出了一种新颖的方法 ERA-CoT，它通过捕获实体之间的关系来帮助法学硕士理解上下文，并通过思想链（CoT）支持不同任务的推理。实验结果表明，ERA-CoT 展示了我们提出的方法与当前 CoT 提示方法相比的优越性能，与之前的 SOTA 基线相比，在 GPT3.5 上实现了平均 5.1% 的显着改进。我们的分析表明，ERA-CoT 增加了法学硕士对实体关系的理解，显着提高了回答问题的准确性，并增强了法学硕士的推理能力。

Title: Naming, Describing, and Quantifying Visual Objects in Humans and LLMs

• Authors: Alberto Testoni, Juell Sprott, Sandro Pezzelle
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06935
• Pdf URL: https://arxiv.org/pdf/2403.06935
• Copy Paste: [[2403.06935]] Naming, Describing, and Quantifying Visual Objects in Humans and LLMs(https://arxiv.org/abs/2403.06935)
• Keywords: language model, llm
• Abstract: While human speakers use a variety of different expressions when describing the same object in an image, giving rise to a distribution of plausible labels driven by pragmatic constraints, the extent to which current Vision \& Language Large Language Models (VLLMs) can mimic this crucial feature of language use is an open question. This applies to common, everyday objects, but it is particularly interesting for uncommon or novel objects for which a category label may be lacking or fuzzy. Furthermore, humans show clear production preferences for highly context-sensitive expressions, such as the quantifiers `few' or `most'. In our work, we evaluate VLLMs (FROMAGe, BLIP-2, LLaVA) on three categories (nouns, attributes, and quantifiers) where humans show great subjective variability concerning the distribution over plausible labels, using datasets and resources mostly under-explored in previous work. Our results reveal mixed evidence on the ability of VLLMs to capture human naming preferences, with all models failing in tasks that require high-level reasoning such as assigning quantifiers.
• 摘要：虽然人类说话者在描述图像中的同一对象时使用各种不同的表达方式，从而产生由语用约束驱动的合理标签的分布，但当前的视觉和语言大型语言模型（VLLM）可以在多大程度上模仿这一关键特征语言使用的特征是一个悬而未决的问题。这适用于常见的日常物品，但对于类别标签可能缺乏或模糊的不常见或新颖的物品尤其有趣。此外，人类对高度上下文敏感的表达方式表现出明显的生产偏好，例如量词“很少”或“大多数”。在我们的工作中，我们使用以前未充分探索的数据集和资源，在三个类别（名词、属性和量词）上评估 VLLM（FROMAGe、BLIP-2、LLaVA），其中人类在合理标签的分布方面表现出巨大的主观变异性。工作。我们的结果揭示了 VLLM 捕获人类命名偏好能力的混合证据，所有模型都无法完成需要高级推理的任务，例如分配量词。

Title: Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

• Authors: Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06965
• Pdf URL: https://arxiv.org/pdf/2403.06965
• Copy Paste: [[2403.06965]] Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena(https://arxiv.org/abs/2403.06965)
• Keywords: language model, gpt, llm
• Abstract: Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact that ``sneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.
• 摘要：论证结构构造 (ASC) 是研究最深入的构造组之一，为展示构造语法 (CxG) 的实用性提供了独特的机会。例如，引起运动的结构（CMC，“她打喷嚏了卡布奇诺的泡沫”）表明结构必须具有意义，否则在这种情况下“打喷嚏”引起运动的事实就无法解释。我们假设，即使对于最先进的大型语言模型（LLM）来说，这仍然具有挑战性，为此我们设计了一个基于用原型运动动词替换动词的测试。为了能够在统计上显着的规模上执行此测试，在缺乏足够的 CxG 语料库的情况下，我们开发了一种新的 NLP 辅助收集语言注释文本的管道。我们展示了如何使用依赖解析和 GPT-3.5 来显着降低注释成本，从而能够大规模注释罕见现象。然后，我们使用新收集的语料库评估 GPT、Gemini、Llama2 和 Mistral 模型对 CMC 的理解。我们发现所有模型都很难理解 CMC 添加到句子中的运动成分。

Title: MRL Parsing Without Tears: The Case of Hebrew

• Authors: Shaltiel Shmidman, Avi Shmidman, Moshe Koppel, Reut Tsarfaty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06970
• Pdf URL: https://arxiv.org/pdf/2403.06970
• Copy Paste: [[2403.06970]] MRL Parsing Without Tears: The Case of Hebrew(https://arxiv.org/abs/2403.06970)
• Keywords: llm
• Abstract: Syntactic parsing remains a critical tool for relation extraction and information extraction, especially in resource-scarce languages where LLMs are lacking. Yet in morphologically rich languages (MRLs), where parsers need to identify multiple lexical units in each token, existing systems suffer in latency and setup complexity. Some use a pipeline to peel away the layers: first segmentation, then morphology tagging, and then syntax parsing; however, errors in earlier layers are then propagated forward. Others use a joint architecture to evaluate all permutations at once; while this improves accuracy, it is notoriously slow. In contrast, and taking Hebrew as a test case, we present a new "flipped pipeline": decisions are made directly on the whole-token units by expert classifiers, each one dedicated to one specific task. The classifiers are independent of one another, and only at the end do we synthesize their predictions. This blazingly fast approach sets a new SOTA in Hebrew POS tagging and dependency parsing, while also reaching near-SOTA performance on other Hebrew NLP tasks. Because our architecture does not rely on any language-specific resources, it can serve as a model to develop similar parsers for other MRLs.
• 摘要：句法解析仍然是关系提取和信息提取的关键工具，特别是在缺乏法学硕士的资源稀缺语言中。然而，在形态丰富的语言（MRL）中，解析器需要识别每个标记中的多个词汇单元，现有系统会受到延迟和设置复杂性的影响。有些使用管道来剥离层：首先分段，然后形态标记，然后语法解析；然而，较早层中的错误会向前传播。其他人使用联合架构来一次评估所有排列；虽然这提高了准确性，但速度却很慢。相比之下，以希伯来语作为测试用例，我们提出了一种新的“翻转管道”：由专家分类器直接对整个令牌单元做出决策，每个分类器专用于一项特定任务。分类器是相互独立的，只有在最后我们才会综合它们的预测。这种极快的方法在希伯来语词性标注和依存解析中树立了新的 SOTA，同时在其他希伯来语 NLP 任务上也达到了接近 SOTA 的性能。因为我们的架构不依赖于任何特定于语言的资源，所以它可以作为为其他 MRL 开发类似解析器的模型。