2024-03-25

Title: The opportunities and risks of large language models in mental health

• Authors: Hannah R. Lawrence, Renee A. Schneider, Susan B. Rubin, Maja J. Mataric, Daniel J. McDuff, Megan Jones Bell
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14814
• Pdf URL: https://arxiv.org/pdf/2403.14814
• Copy Paste: [[2403.14814]] The opportunities and risks of large language models in mental health(https://arxiv.org/abs/2403.14814)
• Keywords: language model, llm
• Abstract: Global rates of mental health concerns are rising and there is increasing realization that existing models of mental healthcare will not adequately expand to meet the demand. With the emergence of large language models (LLMs) has come great optimism regarding their promise to create novel, large-scale solutions to support mental health. Despite their nascence, LLMs have already been applied to mental health-related tasks. In this review, we summarize the extant literature on efforts to use LLMs to provide mental health education, assessment, and intervention and highlight key opportunities for positive impact in each area. We then highlight risks associated with LLMs application to mental health and encourage adoption of strategies to mitigate these risks. The urgent need for mental health support must be balanced with responsible development, testing, and deployment of mental health LLMs. Especially critical is ensuring that mental health LLMs are fine-tuned for mental health, enhance mental health equity, adhere to ethical standards, and that people, including those with lived experience with mental health concerns, are involved in all stages from development through deployment. Prioritizing these efforts will minimize potential harms to mental health and maximize the likelihood that LLMs will positively impact mental health globally.
• 摘要：全球心理健康问题的发生率正在上升，人们越来越认识到现有的心理保健模式将无法充分扩展以满足需求。随着大型语言模型（LLM）的出现，人们对它们创造新颖的、大规模的解决方案来支持心理健康的承诺感到非常乐观。尽管法学硕士刚刚起步，但它已经应用于与心理健康相关的任务。在这篇综述中，我们总结了关于利用法学硕士提供心理健康教育、评估和干预的现有文献，并强调了在每个领域产生积极影响的关键机会。然后，我们强调与法学硕士申请心理健康相关的风险，并鼓励采取策略来减轻这些风险。对心理健康支持的迫切需求必须与负责任的心理健康法学硕士的开发、测试和部署相平衡。尤其重要的是，确保心理健康法学硕士针对心理健康进行微调，增强心理健康公平性，遵守道德标准，并确保人们，包括那些有心理健康问题生活经验的人，参与从开发到部署的所有阶段。优先考虑这些努力将最大限度地减少对心理健康的潜在危害，并最大限度地提高法学硕士对全球心理健康产生积极影响的可能性。

Title: TAMS: Translation-Assisted Morphological Segmentation

• Authors: Enora Rice, Ali Marashian, Luke Gessler, Alexis Palmer, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14840
• Pdf URL: https://arxiv.org/pdf/2403.14840
• Copy Paste: [[2403.14840]] TAMS: Translation-Assisted Morphological Segmentation(https://arxiv.org/abs/2403.14840)
• Keywords: language model
• Abstract: Canonical morphological segmentation is the process of analyzing words into the standard (aka underlying) forms of their constituent morphemes. This is a core task in language documentation, and NLP systems have the potential to dramatically speed up this process. But in typical language documentation settings, training data for canonical morpheme segmentation is scarce, making it difficult to train high quality models. However, translation data is often much more abundant, and, in this work, we present a method that attempts to leverage this data in the canonical segmentation task. We propose a character-level sequence-to-sequence model that incorporates representations of translations obtained from pretrained high-resource monolingual language models as an additional signal. Our model outperforms the baseline in a super-low resource setting but yields mixed results on training splits with more data. While further work is needed to make translations useful in higher-resource settings, our model shows promise in severely resource-constrained settings.
• 摘要：规范形态分割是将单词分析为其构成语素的标准（也称为基础）形式的过程。这是语言文档的核心任务，NLP 系统有潜力显着加快这一过程。但在典型的语言文档环境中，规范语素分割的训练数据很少，因此很难训练高质量的模型。然而，翻译数据通常要丰富得多，在这项工作中，我们提出了一种尝试在规范分割任务中利用这些数据的方法。我们提出了一种字符级序列到序列模型，该模型将从预训练的高资源单语语言模型获得的翻译表示作为附加信号。我们的模型在超低资源设置中优于基线，但在使用更多数据进行训练分割时产生混合结果。虽然需要进一步的工作来使翻译在资源较多的环境中发挥作用，但我们的模型在资源严重受限的环境中显示出了希望。

Title: Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models

• Authors: Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, Anna A. Ivanova
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14859
• Pdf URL: https://arxiv.org/pdf/2403.14859
• Copy Paste: [[2403.14859]] Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models(https://arxiv.org/abs/2403.14859)
• Keywords: language model, llm, prompt
• Abstract: Instruction-tuned LLMs can respond to explicit queries formulated as prompts, which greatly facilitates interaction with human users. However, prompt-based approaches might not always be able to tap into the wealth of implicit knowledge acquired by LLMs during pre-training. This paper presents a comprehensive study of ways to evaluate semantic plausibility in LLMs. We compare base and instruction-tuned LLM performance on an English sentence plausibility task via (a) explicit prompting and (b) implicit estimation via direct readout of the probabilities models assign to strings. Experiment 1 shows that, across model architectures and plausibility datasets, (i) log likelihood ($\textit{LL}$) scores are the most reliable indicator of sentence plausibility, with zero-shot prompting yielding inconsistent and typically poor results; (ii) $\textit{LL}$-based performance is still inferior to human performance; (iii) instruction-tuned models have worse $\textit{LL}$-based performance than base models. In Experiment 2, we show that $\textit{LL}$ scores across models are modulated by context in the expected way, showing high performance on three metrics of context-sensitive plausibility and providing a direct match to explicit human plausibility judgments. Overall, $\textit{LL}$ estimates remain a more reliable measure of plausibility in LLMs than direct prompting.
• 摘要：经过指令调整的法学硕士可以响应以提示形式表达的显式查询，这极大地促进了与人类用户的交互。然而，基于提示的方法可能并不总是能够利用法学硕士在预培训期间获得的大量隐性知识。本文对法学硕士语义合理性的评估方法进行了全面研究。我们通过（a）显式提示和（b）通过直接读出分配给字符串的概率模型进行隐式估计，比较英语句子合理性任务上的基础法学硕士和指令调整法学硕士的表现。实验 1 表明，在模型架构和合理性数据集上，(i) 对数似然 ($\textit{LL}$) 分数是句子合理性最可靠的指标，零样本提示会产生不一致且通常很差的结果； (ii) 基于 $\textit{LL}$ 的性能仍然不如人类的性能； (iii) 指令调整模型基于 $\textit{LL}$ 的性能比基本模型差。在实验 2 中，我们表明，跨模型的 $\textit{LL}$ 分数以预期的方式受上下文调节，在上下文敏感的合理性的三个指标上显示出高性能，并提供与明确的人类合理性判断的直接匹配。总体而言，$\textit{LL}$ 估计仍然是比直接提示更可靠的法学硕士合理性衡量标准。

Title: AutoRE: Document-Level Relation Extraction with Large Language Models

• Authors: Xue Lilong, Zhang Dan, Dong Yuxiao, Tang Jie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14888
• Pdf URL: https://arxiv.org/pdf/2403.14888
• Copy Paste: [[2403.14888]] AutoRE: Document-Level Relation Extraction with Large Language Models(https://arxiv.org/abs/2403.14888)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated exceptional abilities in comprehending and generating text, motivating numerous researchers to utilize them for Information Extraction (IE) purposes, including Relation Extraction (RE). Nonetheless, most existing methods are predominantly designed for Sentence-level Relation Extraction (SentRE) tasks, which typically encompass a restricted set of relations and triplet facts within a single sentence. Furthermore, certain approaches resort to treating relations as candidate choices integrated into prompt templates, leading to inefficient processing and suboptimal performance when tackling Document-Level Relation Extraction (DocRE) tasks, which entail handling multiple relations and triplet facts distributed across a given document, posing distinct challenges. To overcome these limitations, we introduce AutoRE, an end-to-end DocRE model that adopts a novel RE extraction paradigm named RHF (Relation-Head-Facts). Unlike existing approaches, AutoRE does not rely on the assumption of known relation options, making it more reflective of real-world scenarios. Additionally, we have developed an easily extensible RE framework using a Parameters Efficient Fine Tuning (PEFT) algorithm (QLoRA). Our experiments on the RE-DocRED dataset showcase AutoRE's best performance, achieving state-of-the-art results, surpassing TAG by 10.03% and 9.03% respectively on the dev and test set.
• 摘要：大型语言模型 (LLM) 在理解和生成文本方面表现出了卓越的能力，促使众多研究人员将其用于信息提取 (IE) 目的，包括关系提取 (RE)。尽管如此，大多数现有方法主要是为句子级关系提取（SentRE）任务而设计的，该任务通常在单个句子中包含一组有限的关系和三元组事实。此外，某些方法将关系视为集成到提示模板中的候选选择，导致处理文档级关系提取（DocRE）任务时处理效率低下和性能不佳，这需要处理分布在给定文档中的多个关系和三元组事实，提出独特的挑战。为了克服这些限制，我们引入了 AutoRE，这是一种端到端的 DocRE 模型，它采用了一种名为 RHF（Relation-Head-Facts）的新型 RE 提取范例。与现有方法不同，AutoRE 不依赖于已知关系选项的假设，使其更能反映现实世界的场景。此外，我们还使用参数高效微调 (PEFT) 算法 (QLoRA) 开发了一个易于扩展的 RE 框架。我们在 RE-DocRED 数据集上的实验展示了 AutoRE 的最佳性能，达到了最先进的结果，在开发集和测试集上分别超过 TAG 10.03% 和 9.03%。

Title: Stance Reasoner: Zero-Shot Stance Detection on Social Media with Explicit Reasoning

• Authors: Maksym Taranukhin, Vered Shwartz, Evangelos Milios
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14895
• Pdf URL: https://arxiv.org/pdf/2403.14895
• Copy Paste: [[2403.14895]] Stance Reasoner: Zero-Shot Stance Detection on Social Media with Explicit Reasoning(https://arxiv.org/abs/2403.14895)
• Keywords: language model, chain-of-thought
• Abstract: Social media platforms are rich sources of opinionated content. Stance detection allows the automatic extraction of users' opinions on various topics from such content. We focus on zero-shot stance detection, where the model's success relies on (a) having knowledge about the target topic; and (b) learning general reasoning strategies that can be employed for new topics. We present Stance Reasoner, an approach to zero-shot stance detection on social media that leverages explicit reasoning over background knowledge to guide the model's inference about the document's stance on a target. Specifically, our method uses a pre-trained language model as a source of world knowledge, with the chain-of-thought in-context learning approach to generate intermediate reasoning steps. Stance Reasoner outperforms the current state-of-the-art models on 3 Twitter datasets, including fully supervised models. It can better generalize across targets, while at the same time providing explicit and interpretable explanations for its predictions.
• 摘要：社交媒体平台是固执己见内容的丰富来源。立场检测允许从此类内容中自动提取用户对各种主题的意见。我们专注于零样本姿态检测，其中模型的成功依赖于（a）了解目标主题； (b) 学习可用于新主题的一般推理策略。我们提出了 Stance Reasoner，这是一种在社交媒体上进行零样本立场检测的方法，它利用背景知识的显式推理来指导模型推断文档对目标的立场。具体来说，我们的方法使用预先训练的语言模型作为世界知识的来源，并使用上下文中的思想链学习方法来生成中间推理步骤。 Stance Reasoner 在 3 个 Twitter 数据集上的性能优于当前最先进的模型，包括完全监督的模型。它可以更好地跨目标进行概括，同时为其预测提供明确且可解释的解释。

Title: Hierarchical Skip Decoding for Efficient Autoregressive Text Generation

• Authors: Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Wensheng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14919
• Pdf URL: https://arxiv.org/pdf/2403.14919
• Copy Paste: [[2403.14919]] Hierarchical Skip Decoding for Efficient Autoregressive Text Generation(https://arxiv.org/abs/2403.14919)
• Keywords: language model
• Abstract: Autoregressive decoding strategy is a commonly used method for text generation tasks with pre-trained language models, while early-exiting is an effective approach to speedup the inference stage. In this work, we propose a novel decoding strategy named Hierarchical Skip Decoding (HSD) for efficient autoregressive text generation. Different from existing methods that require additional trainable components, HSD is a plug-and-play method applicable to autoregressive text generation models, it adaptively skips decoding layers in a hierarchical manner based on the current sequence length, thereby reducing computational workload and allocating computation resources. Comprehensive experiments on five text generation datasets with pre-trained language models demonstrate HSD's advantages in balancing efficiency and text quality. With almost half of the layers skipped, HSD can sustain 90% of the text quality compared to vanilla autoregressive decoding, outperforming the competitive approaches.
• 摘要：自回归解码策略是预训练语言模型文本生成任务的常用方法，而提前退出是加速推理阶段的有效方法。在这项工作中，我们提出了一种名为分层跳过解码（HSD）的新颖解码策略，用于高效的自回归文本生成。与需要额外可训练组件的现有方法不同，HSD是一种适用于自回归文本生成模型的即插即用方法，它根据当前序列长度以分层方式自适应地跳过解码层，从而减少计算工作量并分配计算资源。使用预训练语言模型对五个文本生成数据集进行综合实验，证明了 HSD 在平衡效率和文本质量方面的优势。与普通自回归解码相比，HSD 跳过了近一半的层，可以维持 90% 的文本质量，优于竞争方法。

Title: Attention-Driven Reasoning: Unlocking the Potential of Large Language Models

• Authors: Bingli Liao, Danilo Vasconcellos Vargas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14932
• Pdf URL: https://arxiv.org/pdf/2403.14932
• Copy Paste: [[2403.14932]] Attention-Driven Reasoning: Unlocking the Potential of Large Language Models(https://arxiv.org/abs/2403.14932)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities, but their reasoning abilities and underlying mechanisms remain poorly understood. We present a novel approach to enhance LLMs' reasoning through attention mechanism optimization, without additional training data. We identify inefficiencies in the attention distribution caused by non-semantic tokens and propose an algorithm to re-balance the skewed distribution, enabling the model to abstract more nuanced knowledge. Our experiments demonstrate significantly improved reasoning capabilities, particularly for non-STEM questions. We provide insights into the role of attention patterns in LLMs' reasoning and propose a method to enhance these abilities, paving the way for more powerful and versatile language models.
• 摘要：大型语言模型（LLM）已经显示出非凡的能力，但它们的推理能力和底层机制仍然知之甚少。我们提出了一种新颖的方法，通过注意力机制优化来增强法学硕士的推理能力，无需额外的训练数据。我们发现了非语义标记导致的注意力分配效率低下，并提出了一种重新平衡倾斜分布的算法，使模型能够抽象出更细致的知识。我们的实验证明推理能力显着提高，尤其是对于非 STEM 问题。我们深入探讨了注意力模式在法学硕士推理中的作用，并提出了一种增强这些能力的方法，为更强大、更通用的语言模型铺平了道路。

Title: On Zero-Shot Counterspeech Generation by LLMs

• Authors: Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, Animesh Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14938
• Pdf URL: https://arxiv.org/pdf/2403.14938
• Copy Paste: [[2403.14938]] On Zero-Shot Counterspeech Generation by LLMs(https://arxiv.org/abs/2403.14938)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: With the emergence of numerous Large Language Models (LLM), the usage of such models in various Natural Language Processing (NLP) applications is increasing extensively. Counterspeech generation is one such key task where efforts are made to develop generative models by fine-tuning LLMs with hatespeech - counterspeech pairs, but none of these attempts explores the intrinsic properties of large language models in zero-shot settings. In this work, we present a comprehensive analysis of the performances of four LLMs namely GPT-2, DialoGPT, ChatGPT and FlanT5 in zero-shot settings for counterspeech generation, which is the first of its kind. For GPT-2 and DialoGPT, we further investigate the deviation in performance with respect to the sizes (small, medium, large) of the models. On the other hand, we propose three different prompting strategies for generating different types of counterspeech and analyse the impact of such strategies on the performance of the models. Our analysis shows that there is an improvement in generation quality for two datasets (17%), however the toxicity increase (25%) with increase in model size. Considering type of model, GPT-2 and FlanT5 models are significantly better in terms of counterspeech quality but also have high toxicity as compared to DialoGPT. ChatGPT are much better at generating counter speech than other models across all metrics. In terms of prompting, we find that our proposed strategies help in improving counter speech generation across all the models.
• 摘要：随着大量大型语言模型（LLM）的出现，此类模型在各种自然语言处理（NLP）应用中的使用正在广泛增加。反言语生成就是这样一项关键任务，人们努力通过使用仇恨言语-反言语对微调法学硕士来开发生成模型，但这些尝试都没有探索零样本设置中大型语言模型的内在属性。在这项工作中，我们对四种 LLM（即 GPT-2、DialoGPT、ChatGPT 和 FlanT5）在用于反语音生成的零样本设置中的性能进行了全面分析，这在同类研究中尚属首次。对于 GPT-2 和 DialoGPT，我们进一步研究了模型尺寸（小、中、大）方面的性能偏差。另一方面，我们提出了三种不同的提示策略来生成不同类型的反言语，并分析了这些策略对模型性能的影响。我们的分析表明，两个数据集的生成质量有所提高 (17%)，但毒性随着模型大小的增加而增加 (25%)。考虑到模型类型，GPT-2 和 FlanT5 模型在反语音质量方面明显更好，但与 DialoGPT 相比也具有较高的毒性。 ChatGPT 在所有指标上都比其他模型更擅长生成反驳言论。在提示方面，我们发现我们提出的策略有助于改善所有模型的反言语生成。

Title: KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

• Authors: Xindi Luo, Zequn Sun, Jing Zhao, Zhe Zhao, Wei Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14950
• Pdf URL: https://arxiv.org/pdf/2403.14950
• Copy Paste: [[2403.14950]] KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation(https://arxiv.org/abs/2403.14950)
• Keywords: language model, llm, prompt
• Abstract: Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that \modelname can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.
• 摘要：参数高效微调（PEFT）是使大型语言模型（LLM）适应下游任务的关键技术。在本文中，我们研究利用知识图嵌入来提高 PEFT 的有效性。我们提出了一种称为 KnowLA 的知识适应方法。它将适应层插入到 LLM 中，以集成输入文本中出现的实体的嵌入。适配层结合 LoRA 对指令数据进行训练。使用两个流行的法学硕士和三个知识图在六个基准上进行的实验证明了 KnowLA 的有效性和稳健性。我们证明 \modelname 可以帮助激活 LLM 中的相关参数化知识来回答问题，而无需更改其参数或输入提示。

Title: Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation

• Authors: Zhenrui Yue, Huimin Zeng, Yimeng Lu, Lanyu Shang, Yang Zhang, Dong Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14952
• Pdf URL: https://arxiv.org/pdf/2403.14952
• Copy Paste: [[2403.14952]] Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation(https://arxiv.org/abs/2403.14952)
• Keywords: language model, llm
• Abstract: The proliferation of online misinformation has posed significant threats to public interest. While numerous online users actively participate in the combat against misinformation, many of such responses can be characterized by the lack of politeness and supporting facts. As a solution, text generation approaches are proposed to automatically produce counter-misinformation responses. Nevertheless, existing methods are often trained end-to-end without leveraging external knowledge, resulting in subpar text quality and excessively repetitive responses. In this paper, we propose retrieval augmented response generation for online misinformation (RARG), which collects supporting evidence from scientific sources and generates counter-misinformation responses based on the evidences. In particular, our RARG consists of two stages: (1) evidence collection, where we design a retrieval pipeline to retrieve and rerank evidence documents using a database comprising over 1M academic articles; (2) response generation, in which we align large language models (LLMs) to generate evidence-based responses via reinforcement learning from human feedback (RLHF). We propose a reward function to maximize the utilization of the retrieved evidence while maintaining the quality of the generated text, which yields polite and factual responses that clearly refutes misinformation. To demonstrate the effectiveness of our method, we study the case of COVID-19 and perform extensive experiments with both in- and cross-domain datasets, where RARG consistently outperforms baselines by generating high-quality counter-misinformation responses.
• 摘要：网络错误信息的泛滥对公共利益构成了重大威胁。尽管许多在线用户积极参与打击虚假信息的斗争，但许多此类回应的特点是缺乏礼貌和支持事实。作为一种解决方案，提出了文本生成方法来自动生成反错误信息响应。然而，现有方法通常是在不利用外部知识的情况下进行端到端训练，导致文本质量不佳和过度重复的响应。在本文中，我们提出针对在线错误信息的检索增强响应生成（RARG），它从科学来源收集支持证据，并根据证据生成反错误信息响应。具体来说，我们的 RARG 包括两个阶段：（1）证据收集，我们设计一个检索管道，使用包含超过 100 万篇学术文章的数据库来检索和重新排列证据文档； (2) 响应生成，其中我们调整大型语言模型 (LLM)，通过人类反馈的强化学习 (RLHF) 生成基于证据的响应。我们提出了一个奖励函数，以最大限度地利用检索到的证据，同时保持生成文本的质量，从而产生礼貌和事实的响应，明确驳斥错误信息。为了证明我们方法的有效性，我们研究了 COVID-19 的案例，并使用域内和跨域数据集进行了广泛的实验，其中 RARG 通过生成高质量的反错误信息响应，始终优于基线。

Title: MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts

• Authors: Md Nishat Raihan, Dhiman Goswami, Al Nahian Bin Emran, Sadiya Sayara Chowdhury Puspo, Amrita Ganguly, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14982
• Pdf URL: https://arxiv.org/pdf/2403.14982
• Copy Paste: [[2403.14982]] MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts(https://arxiv.org/abs/2403.14982)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Our paper presents team MasonTigers submission to the SemEval-2024 Task 9 - which provides a dataset of puzzles for testing natural language understanding. We employ large language models (LLMs) to solve this task through several prompting techniques. Zero-shot and few-shot prompting generate reasonably good results when tested with proprietary LLMs, compared to the open-source models. We obtain further improved results with chain-of-thought prompting, an iterative prompting method that breaks down the reasoning process step-by-step. We obtain our best results by utilizing an ensemble of chain-of-thought prompts, placing 2nd in the word puzzle subtask and 13th in the sentence puzzle subtask. The strong performance of prompted LLMs demonstrates their capability for complex reasoning when provided with a decomposition of the thought process. Our work sheds light on how step-wise explanatory prompts can unlock more of the knowledge encoded in the parameters of large models.
• 摘要：我们的论文介绍了 MasonTigers 团队向 SemEval-2024 任务 9 提交的内容，该任务提供了用于测试自然语言理解的谜题数据集。我们采用大型语言模型（LLM）通过多种提示技术来解决此任务。与开源模型相比，在使用专有法学硕士进行测试时，零样本和少样本提示会产生相当好的结果。我们通过思想链提示获得了进一步改进的结果，这是一种逐步分解推理过程的迭代提示方法。我们通过利用一系列思维链提示获得了最好的结果，在单词谜题子任务中排名第二，在句子谜题子任务中排名第 13。提示法学硕士的出色表现证明了他们在提供思维过程分解时进行复杂推理的能力。我们的工作揭示了逐步解释提示如何解锁大型模型参数中编码的更多知识。

Title: Risk and Response in Large Language Models: Evaluating Key Threat Categories

• Authors: Bahareh Harandizadeh, Abel Salinas, Fred Morstatter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14988
• Pdf URL: https://arxiv.org/pdf/2403.14988
• Copy Paste: [[2403.14988]] Risk and Response in Large Language Models: Evaluating Key Threat Categories(https://arxiv.org/abs/2403.14988)
• Keywords: language model, llm
• Abstract: This paper explores the pressing issue of risk assessment in Large Language Models (LLMs) as they become increasingly prevalent in various applications. Focusing on how reward models, which are designed to fine-tune pretrained LLMs to align with human values, perceive and categorize different types of risks, we delve into the challenges posed by the subjective nature of preference-based training data. By utilizing the Anthropic Red-team dataset, we analyze major risk categories, including Information Hazards, Malicious Uses, and Discrimination/Hateful content. Our findings indicate that LLMs tend to consider Information Hazards less harmful, a finding confirmed by a specially developed regression model. Additionally, our analysis shows that LLMs respond less stringently to Information Hazards compared to other risks. The study further reveals a significant vulnerability of LLMs to jailbreaking attacks in Information Hazard scenarios, highlighting a critical security concern in LLM risk assessment and emphasizing the need for improved AI safety measures.
• 摘要：本文探讨了大型语言模型 (LLM) 中风险评估的紧迫问题，因为它们在各种应用中变得越来越普遍。我们专注于奖励模型（旨在微调预训练的法学硕士以符合人类价值观）如何感知和分类不同类型的风险，深入研究基于偏好的训练数据的主观性质所带来的挑战。通过利用 Anthropic 红队数据集，我们分析主要风险类别，包括信息危害、恶意使用和歧视/仇恨内容。我们的研究结果表明，法学硕士倾向于认为信息危害的危害较小，这一发现得到了专门开发的回归模型的证实。此外，我们的分析表明，与其他风险相比，法学硕士对信息危害的反应不太严格。该研究进一步揭示了法学硕士在信息危害场景中易受越狱攻击的重大漏洞，强调了法学硕士风险评估中的一个关键安全问题，并强调需要改进人工智能安全措施。

Title: MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection

• Authors: Sadiya Sayara Chowdhury Puspo, Md Nishat Raihan, Dhiman Goswami, Al Nahian Bin Emran, Amrita Ganguly, Ozlem Uzuner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14989
• Pdf URL: https://arxiv.org/pdf/2403.14989
• Copy Paste: [[2403.14989]] MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection(https://arxiv.org/abs/2403.14989)
• Keywords: prompt
• Abstract: This paper presents the MasonTigers entry to the SemEval-2024 Task 8 - Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. The task encompasses Binary Human-Written vs. Machine-Generated Text Classification (Track A), Multi-Way Machine-Generated Text Classification (Track B), and Human-Machine Mixed Text Detection (Track C). Our best performing approaches utilize mainly the ensemble of discriminator transformer models along with sentence transformer and statistical machine learning approaches in specific cases. Moreover, zero-shot prompting and fine-tuning of FLAN-T5 are used for Track A and B.
• 摘要：本文介绍了 MasonTigers 进入 SemEval-2024 任务 8 - 多生成器、多域和多语言黑盒机器生成文本检测的情况。该任务包括二进制人类书写与机器生成的文本分类（轨道 A）、多路机器生成的文本分类（轨道 B）和人机混合文本检测（轨道 C）。我们性能最佳的方法主要利用判别器变换器模型的集合以及特定情况下的句子变换器和统计机器学习方法。此外，A、B轨均采用FLAN-T5的零镜头提示和微调。

Title: ESG Classification by Implicit Rule Learning via GPT-4

• Authors: Hyo Jeong Yun, Chanyoung Kim, Moonjeong Hahm, Kyuri Kim, Guijin Son
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15040
• Pdf URL: https://arxiv.org/pdf/2403.15040
• Copy Paste: [[2403.15040]] ESG Classification by Implicit Rule Learning via GPT-4(https://arxiv.org/abs/2403.15040)
• Keywords: language model, gpt, prompt, chain-of-thought
• Abstract: Environmental, social, and governance (ESG) factors are widely adopted as higher investment return indicators. Accordingly, ongoing efforts are being made to automate ESG evaluation with language models to extract signals from massive web text easily. However, recent approaches suffer from a lack of training data, as rating agencies keep their evaluation metrics confidential. This paper investigates whether state-of-the-art language models like GPT-4 can be guided to align with unknown ESG evaluation criteria through strategies such as prompting, chain-of-thought reasoning, and dynamic in-context learning. We demonstrate the efficacy of these approaches by ranking 2nd in the Shared-Task ML-ESG-3 Impact Type track for Korean without updating the model on the provided training data. We also explore how adjusting prompts impacts the ability of language models to address financial tasks leveraging smaller models with openly available weights. We observe longer general pre-training to correlate with enhanced performance in financial downstream tasks. Our findings showcase the potential of language models to navigate complex, subjective evaluation guidelines despite lacking explicit training examples, revealing opportunities for training-free solutions for financial downstream tasks.
• 摘要：环境、社会和治理（ESG）因素被广泛采用作为更高的投资回报指标。因此，人们正在不断努力利用语言模型实现 ESG 评估自动化，以便轻松地从大量网络文本中提取信号。然而，最近的方法缺乏培训数据，因为评级机构对其评估指标保密。本文研究了是否可以通过提示、思维链推理和动态上下文学习等策略来引导 GPT-4 等最先进的语言模型与未知的 ESG 评估标准保持一致。我们通过在韩语的共享任务 ML-ESG-3 影响类型跟踪中排名第二来证明这些方法的有效性，而无需根据提供的训练数据更新模型。我们还探讨了调整提示如何影响语言模型利用具有公开可用权重的较小模型来解决财务任务的能力。我们观察到更长的一般预培训与金融下游任务绩效的提高相关。我们的研究结果展示了语言模型在复杂的主观评估指南中导航的潜力，尽管缺乏明确的训练示例，揭示了针对金融下游任务的免训练解决方案的机会。

Title: LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

• Authors: Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15042
• Pdf URL: https://arxiv.org/pdf/2403.15042
• Copy Paste: [[2403.15042]] LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement(https://arxiv.org/abs/2403.15042)
• Keywords: language model, llm
• Abstract: Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a LLaMA2-7B student model.
• 摘要：预训练的大型语言模型 (LLM) 目前是解决绝大多数自然语言处理任务的最先进技术。虽然许多实际应用程序仍然需要微调才能达到令人满意的性能水平，但其中许多应用程序处于低数据状态，这使得微调具有挑战性。为了解决这个问题，我们提出了LLM2LLM，这是一种有针对性的迭代数据增强策略，它使用教师LLM通过增强可用于对特定任务进行微调的附加数据来增强小型种子数据集。 LLM2LLM (1) 在初始种子数据上微调基线学生 LLM，(2) 评估并提取模型出错的数据点，以及 (3) 使用教师 LLM 基于这些不正确的数据点生成合成数据，然后将其添加回训练数据中。这种方法放大了法学硕士在训练期间错误预测的数据点的信号，并将它们重新整合到数据集中，以专注于法学硕士更具挑战性的示例。我们的结果表明，LLM2LLM 显着增强了 LLM 在低数据情况下的性能，优于传统的微调和其他数据增强基线。 LLM2LLM 减少了对劳动密集型数据管理的依赖，并为更具可扩展性和性能的 LLM 解决方案铺平了道路，使我们能够处理数据受限的领域和任务。与使用 LLaMA2-7B 学生在低数据状态下进行常规微调相比，我们在 GSM8K 数据集上实现了高达 24.2% 的改进，在 CaseHOLD 上实现了 32.6%，在 SNIPS 上实现了 32.0%，在 TREC 上实现了 52.6%，在 SST-2 上实现了 39.8%模型。

Title: CHisIEC: An Information Extraction Corpus for Ancient Chinese History

• Authors: Xuemei Tang, Zekun Deng, Qi Su, Hao Yang, Jun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15088
• Pdf URL: https://arxiv.org/pdf/2403.15088
• Copy Paste: [[2403.15088]] CHisIEC: An Information Extraction Corpus for Ancient Chinese History(https://arxiv.org/abs/2403.15088)
• Keywords: language model, llm
• Abstract: Natural Language Processing (NLP) plays a pivotal role in the realm of Digital Humanities (DH) and serves as the cornerstone for advancing the structural analysis of historical and cultural heritage texts. This is particularly true for the domains of named entity recognition (NER) and relation extraction (RE). In our commitment to expediting ancient history and culture, we present the ``Chinese Historical Information Extraction Corpus''(CHisIEC). CHisIEC is a meticulously curated dataset designed to develop and evaluate NER and RE tasks, offering a resource to facilitate research in the field. Spanning a remarkable historical timeline encompassing data from 13 dynasties spanning over 1830 years, CHisIEC epitomizes the extensive temporal range and text heterogeneity inherent in Chinese historical documents. The dataset encompasses four distinct entity types and twelve relation types, resulting in a meticulously labeled dataset comprising 14,194 entities and 8,609 relations. To establish the robustness and versatility of our dataset, we have undertaken comprehensive experimentation involving models of various sizes and paradigms. Additionally, we have evaluated the capabilities of Large Language Models (LLMs) in the context of tasks related to ancient Chinese history. The dataset and code are available at \url{https://github.com/tangxuemei1995/CHisIEC}.
• 摘要：自然语言处理（NLP）在数字人文（DH）领域发挥着关键作用，是推进历史和文化遗产文本结构分析的基石。对于命名实体识别 (NER) 和关系提取 (RE) 领域尤其如此。为了加速古代历史和文化的发展，我们推出了“中国历史信息提取语料库”（CHisIEC）。 CHisIEC 是一个精心策划的数据集，旨在开发和评估 NER 和 RE 任务，为促进该领域的研究提供资源。 CHisIEC 跨越了一段非凡的历史时间线，涵盖了 1830 多年的 13 个朝代的数据，集中体现了中国历史文献所固有的广泛的时间范围和文本异质性。该数据集包含四种不同的实体类型和十二种关系类型，从而形成一个包含 14,194 个实体和 8,609 个关系的精心标记的数据集。为了建立我们的数据集的稳健性和多功能性，我们对各种规模和范式的模型进行了全面的实验。此外，我们还评估了大型语言模型（LLM）在与中国古代历史相关的任务中的能力。数据集和代码可在 \url{https://github.com/tangxuemei1995/CHisIEC} 获取。

Title: Text clustering with LLM embeddings

• Authors: Alina Petukhova, Joao P. Matos-Carvalho, Nuno Fachada
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15112
• Pdf URL: https://arxiv.org/pdf/2403.15112
• Copy Paste: [[2403.15112]] Text clustering with LLM embeddings(https://arxiv.org/abs/2403.15112)
• Keywords: language model, llm
• Abstract: Text clustering is an important approach for organising the growing amount of digital content, helping to structure and find hidden patterns in uncategorised data. In this research, we investigated how different textual embeddings - particularly those used in large language models (LLMs) - and clustering algorithms affect how text datasets are clustered. A series of experiments were conducted to assess how embeddings influence clustering results, the role played by dimensionality reduction through summarisation, and embedding size adjustment. Results reveal that LLM embeddings excel at capturing the nuances of structured language, while BERT leads the lightweight options in performance. In addition, we find that increasing embedding dimensionality and summarisation techniques do not uniformly improve clustering efficiency, suggesting that these strategies require careful analysis to use in real-life models. These results highlight a complex balance between the need for nuanced text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by incorporating embeddings from LLMs, thereby paving the way for improved methodologies and opening new avenues for future research in various types of textual analysis.
• 摘要：文本聚类是组织不断增长的数字内容的重要方法，有助于构建和查找未分类数据中的隐藏模式。在这项研究中，我们研究了不同的文本嵌入（尤其是大型语言模型 (LLM) 中使用的文本嵌入）和聚类算法如何影响文本数据集的聚类方式。进行了一系列实验来评估嵌入如何影响聚类结果、通过摘要降维所发挥的作用以及嵌入大小调整。结果表明，LLM 嵌入擅长捕捉结构化语言的细微差别，而 BERT 在性能方面领先于轻量级选项。此外，我们发现增加嵌入维数和汇总技术并不能统一提高聚类效率，这表明这些策略需要仔细分析才能在现实模型中使用。这些结果凸显了文本聚类应用中细致入微的文本表示需求和计算可行性之间的复杂平衡。这项研究通过结合法学硕士的嵌入来扩展传统的文本聚类框架，从而为改进方法铺平道路，并为未来各种类型的文本分析研究开辟新途径。

Title: Language Models in Dialogue: Conversational Maxims for Human-AI Interactions

• Authors: Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth M. Daly, David Piorkowski, John T. Richards
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.15115
• Pdf URL: https://arxiv.org/pdf/2403.15115
• Copy Paste: [[2403.15115]] Language Models in Dialogue: Conversational Maxims for Human-AI Interactions(https://arxiv.org/abs/2403.15115)
• Keywords: language model, llm, agent
• Abstract: Modern language models, while sophisticated, exhibit some inherent shortcomings, particularly in conversational settings. We claim that many of the observed shortcomings can be attributed to violation of one or more conversational principles. By drawing upon extensive research from both the social science and AI communities, we propose a set of maxims -- quantity, quality, relevance, manner, benevolence, and transparency -- for describing effective human-AI conversation. We first justify the applicability of the first four maxims (from Grice) in the context of human-AI interactions. We then argue that two new maxims, benevolence (concerning the generation of, and engagement with, harmful content) and transparency (concerning recognition of one's knowledge boundaries, operational constraints, and intents), are necessary for addressing behavior unique to modern human-AI interactions. The proposed maxims offer prescriptive guidance on how to assess conversational quality between humans and LLM-driven conversational agents, informing both their evaluation and improved design.
• 摘要：现代语言模型虽然复杂，但也表现出一些固有的缺点，特别是在对话环境中。我们声称，许多观察到的缺陷可归因于违反一项或多项对话原则。通过借鉴社会科学和人工智能界的广泛研究，我们提出了一套格言——数量、质量、相关性、方式、仁慈和透明度——来描述有效的人类与人工智能对话。我们首先证明前四个格言（来自 Grice）在人类与人工智能交互的背景下的适用性。然后，我们认为两条新的格言：仁慈（关于有害内容的产生和参与）和透明度（关于认识一个人的知识边界、操作限制和意图）对于解决现代人类人工智能特有的行为是必要的互动。所提出的准则为如何评估人类和法学硕士驱动的对话代理之间的对话质量提供了规范性指导，为他们的评估和改进设计提供了信息。

Title: Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study

• Authors: Tim van Dam, Frank van der Heijden, Philippe de Bekker, Berend Nieuwschepen, Marc Otten, Maliheh Izadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15185
• Pdf URL: https://arxiv.org/pdf/2403.15185
• Copy Paste: [[2403.15185]] Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study(https://arxiv.org/abs/2403.15185)
• Keywords: language model, gpt, llm
• Abstract: Language model-based code completion models have quickly grown in use, helping thousands of developers write code in many different programming languages. However, research on code completion models typically focuses on imperative languages such as Python and JavaScript, which results in a lack of representation for functional programming languages. Consequently, these models often perform poorly on functional languages such as Haskell. To investigate whether this can be alleviated, we evaluate the performance of two language models for code, CodeGPT and UniXcoder, on the functional programming language Haskell. We fine-tune and evaluate the models on Haskell functions sourced from a publicly accessible Haskell dataset on HuggingFace. Additionally, we manually evaluate the models using our novel translated HumanEval dataset. Our automatic evaluation shows that knowledge of imperative programming languages in the pre-training of LLMs may not transfer well to functional languages, but that code completion on functional languages is feasible. Consequently, this shows the need for more high-quality Haskell datasets. A manual evaluation on HumanEval-Haskell indicates CodeGPT frequently generates empty predictions and extra comments, while UniXcoder more often produces incomplete or incorrect predictions. Finally, we release HumanEval-Haskell, along with the fine-tuned models and all code required to reproduce our experiments on GitHub (https://github.com/AISE-TUDelft/HaskellCCEval).
• 摘要：基于语言模型的代码完成模型的使用迅速增长，帮助成千上万的开发人员用许多不同的编程语言编写代码。然而，代码补全模型的研究通常集中在Python、JavaScript等命令式语言上，这导致缺乏对函数式编程语言的表征。因此，这些模型在 Haskell 等函数式语言上通常表现不佳。为了研究是否可以缓解这个问题，我们在函数式编程语言 Haskell 上评估了两种代码语言模型 CodeGPT 和 UniXcoder 的性能。我们对源自 HuggingFace 上可公开访问的 Haskell 数据集的 Haskell 函数模型进行微调和评估。此外，我们使用我们新颖的翻译 HumanEval 数据集手动评估模型。我们的自动评估表明，法学硕士预训练中的命令式编程语言知识可能无法很好地迁移到函数式语言，但函数式语言上的代码完成是可行的。因此，这表明需要更多高质量的 Haskell 数据集。对 HumanEval-Haskell 的手动评估表明 CodeGPT 经常生成空预测和额外注释，而 UniXcoder 更经常生成不完整或不正确的预测。最后，我们在 GitHub (https://github.com/AISE-TUDelft/HaskellCCEval) 上发布了 HumanEval-Haskell，以及微调模型和重现实验所需的所有代码。

Title: Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach

• Authors: Kun Sun, Rong Wang, Haitao Liu, Anders Søgaard
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15250
• Pdf URL: https://arxiv.org/pdf/2403.15250
• Copy Paste: [[2403.15250]] Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach(https://arxiv.org/abs/2403.15250)
• Keywords: llm
• Abstract: Amidst the rapid evolution of LLMs, the significance of evaluation in comprehending and propelling these models forward is increasingly paramount. Evaluations have revealed that factors such as scaling, training types, architectures and other factors profoundly impact the performance of LLMs. However, the extent and nature of these impacts continue to be subjects of debate because most assessments have been restricted to a limited number of models and data points. Clarifying the effects of these factors on performance scores can be more effectively achieved through a statistical lens. Our study embarks on a thorough re-examination of these LLMs, targeting the inadequacies in current evaluation methods. With the advent of a uniform evaluation framework, our research leverages an expansive dataset of evaluation results, introducing a comprehensive statistical methodology. This includes the application of ANOVA, Tukey HSD tests, GAMM, and clustering technique, offering a robust and transparent approach to deciphering LLM performance data. Contrary to prevailing findings, our results challenge assumptions about emergent abilities and the influence of given training types and architectures in LLMs. These findings furnish new perspectives on the characteristics, intrinsic nature, and developmental trajectories of LLMs. By providing straightforward and reliable methods to scrutinize and reassess LLM performance data, this study contributes a nuanced perspective on LLM efficiency and potentials.
• 摘要：在法学硕士的快速发展中，评估对于理解和推动这些模型向前发展的重要性变得越来越重要。评估显示，规模、培训类型、架构和其他因素等因素深刻影响着法学硕士的表现。然而，这些影响的程度和性质仍然是争论的主题，因为大多数评估仅限于有限数量的模型和数据点。通过统计视角可以更有效地阐明这些因素对绩效分数的影响。我们的研究开始对这些法学硕士进行彻底的重新审查，针对当前评估方法的不足之处。随着统一评估框架的出现，我们的研究利用了广泛的评估结果数据集，引入了全面的统计方法。这包括方差分析、Tukey HSD 测试、GAMM 和聚类技术的应用，提供了一种稳健且透明的方法来解读 LLM 性能数据。与普遍的研究结果相反，我们的结果挑战了关于新兴能力的假设以及法学硕士特定培训类型和架构的影响。这些发现为理解法学硕士的特征、内在本质和发展轨迹提供了新的视角。通过提供简单可靠的方法来审查和重新评估法学硕士表现数据，本研究为法学硕士的效率和潜力提供了细致入微的视角。

Title: Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models

• Authors: Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Shengping Liu, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15268
• Pdf URL: https://arxiv.org/pdf/2403.15268
• Copy Paste: [[2403.15268]] Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models(https://arxiv.org/abs/2403.15268)
• Keywords: language model, llm, long context
• Abstract: Retrieval-Augmented-Generation and Gener-ation-Augmented-Generation have been proposed to enhance the knowledge required for question answering over Large Language Models (LLMs). However, the former depends on external resources, and both require incorporating the explicit documents into the context, which results in longer contexts that lead to more resource consumption. Recent works indicate that LLMs have modeled rich knowledge, albeit not effectively triggered or activated. Inspired by this, we propose a novel knowledge-augmented framework, Imagination-Augmented-Generation (IAG), which simulates the human capacity to compensate for knowledge deficits while answering questions solely through imagination, without relying on external resources. Guided by IAG, we propose an imagine richer context method for question answering (IMcQA), which obtains richer context through the following two modules: explicit imagination by generating a short dummy document with long context compress and implicit imagination with HyperNetwork for generating adapter weights. Experimental results on three datasets demonstrate that IMcQA exhibits significant advantages in both open-domain and closed-book settings, as well as in both in-distribution performance and out-of-distribution generalizations. Our code will be available at https://github.com/Xnhyacinth/IAG.
• 摘要：检索增强生成和生成增强生成已被提出，以增强大型语言模型（LLM）回答问题所需的知识。然而，前者依赖于外部资源，并且两者都需要将显式文档合并到上下文中，这会导致上下文更长，从而导致更多的资源消耗。最近的研究表明，法学硕士已经模拟了丰富的知识，尽管没有有效地触发或激活。受此启发，我们提出了一种新颖的知识增强框架——想象力增强生成（IAG），它模拟人类补偿知识缺陷的能力，同时仅通过想象力回答问题，而不依赖外部资源。在IAG的指导下，我们提出了一种想象更丰富的上下文问答方法（IMcQA），它通过以下两个模块获得更丰富的上下文：通过生成具有长上下文压缩的短虚拟文档的显式想象和使用HyperNetwork生成适配器权重的隐式想象。三个数据集上的实验结果表明，IMcQA 在开放域和闭卷设置以及分布内性能和分布外泛化方面都表现出显着的优势。我们的代码将在 https://github.com/Xnhyacinth/IAG 上提供。

Title: Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs

• Authors: Xiaobin Zhang, Liangjun Zang, Qianwen Liu, Shuchong Wei, Songlin Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15273
• Pdf URL: https://arxiv.org/pdf/2403.15273
• Copy Paste: [[2403.15273]] Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs(https://arxiv.org/abs/2403.15273)
• Keywords: language model, llm, prompt
• Abstract: Event temporal relation (TempRel) is a primary subject of the event relation extraction task. However, the inherent ambiguity of TempRel increases the difficulty of the task. With the rise of prompt engineering, it is important to design effective prompt templates and verbalizers to extract relevant knowledge. The traditional manually designed templates struggle to extract precise temporal knowledge. This paper introduces a novel retrieval-augmented TempRel extraction approach, leveraging knowledge retrieved from large language models (LLMs) to enhance prompt templates and verbalizers. Our method capitalizes on the diverse capabilities of various LLMs to generate a wide array of ideas for template and verbalizer design. Our proposed method fully exploits the potential of LLMs for generation tasks and contributes more knowledge to our design. Empirical evaluations across three widely recognized datasets demonstrate the efficacy of our method in improving the performance of event temporal relation extraction tasks.
• 摘要：事件时间关系（TempRel）是事件关系提取任务的主要主题。然而，TempRel 固有的模糊性增加了任务的难度。随着提示工程的兴起，设计有效的提示模板和语言器来提取相关知识变得非常重要。传统的手动设计的模板很难提取精确的时间知识。本文介绍了一种新颖的检索增强 TempRel 提取方法，利用从大型语言模型 (LLM) 检索的知识来增强提示模板和语言器。我们的方法利用了不同法学硕士的不同能力，为模板和语言器设计产生了广泛的想法。我们提出的方法充分利用了法学硕士在生成任务方面的潜力，并为我们的设计贡献了更多知识。对三个广泛认可的数据集的实证评估证明了我们的方法在提高事件时间关系提取任务的性能方面的有效性。

Title: CoLLEGe: Concept Embedding Generation for Large Language Models

• Authors: Ryan Teehan, Brenden Lake, Mengye Ren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15362
• Pdf URL: https://arxiv.org/pdf/2403.15362
• Copy Paste: [[2403.15362]] CoLLEGe: Concept Embedding Generation for Large Language Models(https://arxiv.org/abs/2403.15362)
• Keywords: language model, prompt
• Abstract: Current language models are unable to quickly learn new concepts on the fly, often requiring a more involved finetuning process to learn robustly. Prompting in-context is not robust to context distractions, and often fails to confer much information about the new concepts. Classic methods for few-shot word learning in NLP, relying on global word vectors, are less applicable to large language models. In this paper, we introduce a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning. CoLLEGe is a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions. Our primary meta-learning objective is simply to facilitate a language model to make next word predictions in forthcoming sentences, making it compatible with language model pretraining. We design a series of tasks to test new concept learning in challenging real-world scenarios, including new word acquisition, definition inference, and verbal reasoning, and demonstrate that our method succeeds in each setting without task-specific training.
• 摘要：当前的语言模型无法快速快速学习新概念，通常需要更复杂的微调过程才能稳健地学习。上下文提示对于上下文干扰的影响并不强，并且通常无法提供有关新概念的大量信息。 NLP 中少样本单词学习的经典方法依赖于全局词向量，不太适用于大型语言模型。在本文中，我们介绍了一种名为 CoLLEGe（带有语言嵌入生成的概念学习）的新颖方法，以实现小样本概念学习的现代化。 CoLLEGe 是一个元学习框架，能够使用少量例句或定义为新概念生成灵活的嵌入。我们的主要元学习目标只是促进语言模型在即将到来的句子中进行下一个单词预测，使其与语言模型预训练兼容。我们设计了一系列任务来测试具有挑战性的现实场景中的新概念学习，包括新词习得、定义推理和言语推理，并证明我们的方法在每个设置中都取得了成功，无需进行特定任务的训练。