2024-03-27

Title: Attribute First, then Generate: Locally-attributable Grounded Text Generation

• Authors: Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, Ido Dagan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17104
• Pdf URL: https://arxiv.org/pdf/2403.17104
• Copy Paste: [[2403.17104]] Attribute First, then Generate: Locally-attributable Grounded Text Generation(https://arxiv.org/abs/2403.17104)
• Keywords: language model, llm, hallucination
• Abstract: Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named ``Attribute First, then Generate'', breaks down the conventional end-to-end generation process into three intuitive steps: content selection, sentence planning, and sequential sentence generation. By initially identifying relevant source segments (``select first'') and then conditioning the generation process on them (``then generate''), we ensure these segments also act as the output's fine-grained attributions (``select'' becomes ``attribute''). Tested on Multi-document Summarization and Long-form Question-answering, our method not only yields more concise citations than the baselines but also maintains - and in some cases enhances - both generation quality and attribution accuracy. Furthermore, it significantly reduces the time required for fact verification by human assessors.
• 摘要：最近解决大语言模型（LLM）中的幻觉的努力主要集中在归因文本生成上，它通过引用支持来源来补充生成的文本，以进行生成后的事实检查和更正。然而，这些引文通常指向整个文档或段落，给用户带来大量验证工作的负担。在本文中，我们介绍了一种本地归因文本生成方法，优先考虑简洁的归因。我们的方法名为“先属性，然后生成”，将传统的端到端生成过程分解为三个直观的步骤：内容选择、句子规划和顺序句子生成。通过最初识别相关的源片段（“首先选择”），然后调整它们的生成过程（“然后生成”），我们确保这些片段也充当输出的细粒度属性（“选择”）成为“属性”）。经过多文档摘要和长格式问答测试，我们的方法不仅产生比基线更简洁的引文，而且还保持（在某些情况下增强）生成质量和归因准确性。此外，它还大大减少了人类评估员进行事实验证所需的时间。

Title: The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition

• Authors: Georgios Chochlakis, Alexandros Potamianos, Kristina Lerman, Shrikanth Narayanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17125
• Pdf URL: https://arxiv.org/pdf/2403.17125
• Copy Paste: [[2403.17125]] The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition(https://arxiv.org/abs/2403.17125)
• Keywords: language model, llm
• Abstract: In-context Learning (ICL) has emerged as a powerful paradigm for performing natural language tasks with Large Language Models (LLM) without updating the models' parameters, in contrast to the traditional gradient-based finetuning. The promise of ICL is that the LLM can adapt to perform the present task at a competitive or state-of-the-art level at a fraction of the cost. The ability of LLMs to perform tasks in this few-shot manner relies on their background knowledge of the task (or task priors). However, recent work has found that, unlike traditional learning, LLMs are unable to fully integrate information from demonstrations that contrast task priors. This can lead to performance saturation at suboptimal levels, especially for subjective tasks such as emotion recognition, where the mapping from text to emotions can differ widely due to variability in human annotations. In this work, we design experiments and propose measurements to explicitly quantify the consistency of proxies of LLM priors and their pull on the posteriors. We show that LLMs have strong yet inconsistent priors in emotion recognition that ossify their predictions. We also find that the larger the model, the stronger these effects become. Our results suggest that caution is needed when using ICL with larger LLMs for affect-centered tasks outside their pre-training domain and when interpreting ICL results.
• 摘要：与传统的基于梯度的微调相比，上下文学习 (ICL) 已成为使用大型语言模型 (LLM) 执行自然语言任务而无需更新模型参数的强大范例。 ICL 的承诺是，法学硕士可以适应以具有竞争力或最先进的水平以一小部分成本执行当前任务。法学硕士以这种少量方式执行任务的能力依赖于他们对任务（或任务先验）的背景知识。然而，最近的研究发现，与传统学习不同，法学硕士无法完全整合来自对比任务先验的演示的信息。这可能会导致性能饱和在次优水平，特别是对于情感识别等主观任务，其中由于人类注释的可变性，从文本到情感的映射可能存在很大差异。在这项工作中，我们设计了实验并提出了测量方法，以明确量化法学硕士先验代理的一致性及其对后验的影响。我们表明，法学硕士在情感识别方面具有强大但不一致的先验，这使他们的预测变得僵化。我们还发现模型越大，这些效应就越强。我们的结果表明，在将 ICL 与较大的法学硕士一起用于预训练领域之外的以情感为中心的任务以及解释 ICL 结果时需要谨慎。

Title: MetaAligner: Conditional Weak-to-Strong Correction for Generalizable Multi-Objective Alignment of Language Models

• Authors: Kailai Yang, Zhiwei Liu, Qianqian Xie, Tianlin Zhang, Nirui Song, Jimin Huang, Ziyan Kuang, Sophia Ananiadou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17141
• Pdf URL: https://arxiv.org/pdf/2403.17141
• Copy Paste: [[2403.17141]] MetaAligner: Conditional Weak-to-Strong Correction for Generalizable Multi-Objective Alignment of Language Models(https://arxiv.org/abs/2403.17141)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) aim to tackle heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are parameter-adherent to the policy model, leading to two key limitations: (1) the high-cost repetition of their alignment algorithms for each new target model; (2) they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), a model that performs conditional weak-to-strong correction for weak responses to approach strong responses. MetaAligner is the first policy-agnostic and generalizable method for multi-objective preference alignment, which enables plug-and-play alignment by decoupling parameter updates from the policy models and facilitates zero-shot preference alignment for unseen objectives via in-context learning. Experimental results show that MetaAligner achieves significant and balanced improvements in multi-objective alignments on 11 policy models with up to 63x more parameters, and outperforms previous alignment methods with down to 22.27x less computational resources. The model also accurately aligns with unseen objectives, marking the first step towards generalizable multi-objective preference alignment.
• 摘要：大型语言模型（LLM）的最新进展旨在通过多目标偏好调整来解决不同的人类期望和价值观。然而，现有方法与策略模型的参数一致，导致两个关键限制：（1）针对每个新目标模型重复对齐算法的成本很高； (2) 由于其静态对齐目标，它们无法扩展到看不见的目标。在这项工作中，我们提出了元目标对齐器（MetaAligner），这是一种对弱响应进行条件弱到强校正以接近强响应的模型。 MetaAligner 是第一个与策略无关且可推广的多目标偏好对齐方法，它通过将参数更新与策略模型解耦来实现即插即用对齐，并通过上下文学习促进未见目标的零样本偏好对齐。实验结果表明，MetaAligner 在 11 个策​​略模型上实现了多目标对齐的显着且均衡的改进，参数增加了 63 倍，并且优于之前的对齐方法，计算资源减少了 22.27 倍。该模型还准确地与看不见的目标对齐，标志着迈向可推广的多目标偏好对齐的第一步。

Title: Outcome-Constrained Large Language Models for Countering Hate Speech

• Authors: Lingzi Hong, Pengcheng Luo, Eduardo Blanco, Xiaoying Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17146
• Pdf URL: https://arxiv.org/pdf/2403.17146
• Copy Paste: [[2403.17146]] Outcome-Constrained Large Language Models for Countering Hate Speech(https://arxiv.org/abs/2403.17146)
• Keywords: language model, llm, prompt
• Abstract: Counterspeech that challenges or responds to hate speech has been seen as an alternative to mitigate the negative impact of hate speech and foster productive online communications. Research endeavors have been directed to using language models for the automatic generation of counterspeech to assist efforts in combating online hate. Existing research focuses on the generation of counterspeech with certain linguistic attributes, such as being polite, informative, and intent-driven. However, it remains unclear what impact the counterspeech might have in an online environment. We first explore methods that utilize large language models (LLM) to generate counterspeech constrained by potential conversation outcomes. We build two conversation outcome classifiers that predict the incivility level and the hater reentry behavior following replies to hate with Reddit data, then propose four methods to incorporate the desired outcomes, i.e., low conversation incivility and non-hateful hater reentry, into the text generation process, including Prompt with Instructions, Prompt and Select, LLM finetune, and LLM transformer reinforcement learning (TRL). Evaluation results show effective strategies to generate outcome-constrained counterspeech and the linguistic characteristics of texts generated by different methods.
• 摘要：挑战或回应仇恨言论的反言论被视为减轻仇恨言论负面影响和促进富有成效的在线交流的替代方案。研究工作的重点是使用语言模型自动生成反驳言论，以协助打击网络仇恨。现有的研究重点是生成具有某些语言属性的反言语，例如礼貌、信息丰富和意图驱动。然而，目前尚不清楚反言论在网络环境中可能产生什么影响。我们首先探索利用大语言模型（LLM）来生成受潜在对话结果限制的反言语的方法。我们构建了两个对话结果分类器，用于预测使用 Reddit 数据回复仇恨后的不文明程度和仇恨者再进入行为，然后提出四种方法将期望的结果（即低对话不文明性和非仇恨性仇恨者再进入）纳入文本生成中流程，包括提示提示、提示和选择、LLM 微调和 LLM 变压器强化学习 (TRL)。评估结果显示了生成结果受限的反语音的有效策略以及不同方法生成的文本的语言特征。

Title: Reflecting the Male Gaze: Quantifying Female Objectification in 19th and 20th Century Novels

• Authors: Kexin Luo, Yue Mao, Bei Zhang, Sophie Hao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17158
• Pdf URL: https://arxiv.org/pdf/2403.17158
• Copy Paste: [[2403.17158]] Reflecting the Male Gaze: Quantifying Female Objectification in 19th and 20th Century Novels(https://arxiv.org/abs/2403.17158)
• Keywords: agent
• Abstract: Inspired by the concept of the male gaze (Mulvey, 1975) in literature and media studies, this paper proposes a framework for analyzing gender bias in terms of female objectification: the extent to which a text portrays female individuals as objects of visual pleasure. Our framework measures female objectification along two axes. First, we compute an agency bias score that indicates whether male entities are more likely to appear in the text as grammatical agents than female entities. Next, by analyzing the word embedding space induced by a text (Caliskan et al., 2017), we compute an appearance bias score that indicates whether female entities are more closely associated with appearance-related words than male entities. Applying our framework to 19th and 20th century novels reveals evidence of female objectification in literature: we find that novels written from a male perspective systematically objectify female characters, while novels written from a female perspective do not exhibit statistically significant objectification of any gender.
• 摘要：受文学和媒体研究中男性凝视概念（Mulvey，1975）的启发，本文提出了一个分析女性客体化方面的性别偏见的框架：文本将女性个体描绘为视觉愉悦对象的程度。我们的框架沿着两个轴衡量女性物化。首先，我们计算代理偏差分数，该分数表明男性实体是否比女性实体更有可能作为语法代理出现在文本中。接下来，通过分析文本引起的词嵌入空间（Caliskan et al., 2017），我们计算了一个外观偏差分数，该分数表明女性实体是否比男性实体与外观相关单词的关联更紧密。将我们的框架应用到 19 世纪和 20 世纪的小说中，揭示了文学中女性客体化的证据：我们发现从男性角度写的小说系统地客体化了女性角色，而从女性角度写的小说并没有表现出统计上显着的任何性别的客体化。

Title: GPT-4 Understands Discourse at Least as Well as Humans Do

• Authors: Thomas Shultz, Jamie Wise, Ardavan Salehi Nobandegani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17196
• Pdf URL: https://arxiv.org/pdf/2403.17196
• Copy Paste: [[2403.17196]] GPT-4 Understands Discourse at Least as Well as Humans Do(https://arxiv.org/abs/2403.17196)
• Keywords: gpt
• Abstract: We test whether a leading AI system GPT-4 understands discourse as well as humans do, using a standardized test of discourse comprehension. Participants are presented with brief stories and then answer eight yes/no questions probing their comprehension of the story. The questions are formatted to assess the separate impacts of directness (stated vs. implied) and salience (main idea vs. details). GPT-4 performs slightly, but not statistically significantly, better than humans given the very high level of human performance. Both GPT-4 and humans exhibit a strong ability to make inferences about information that is not explicitly stated in a story, a critical test of understanding.
• 摘要：我们使用标准化的话语理解测试来测试领先的人工智能系统 GPT-4 是否能像人类一样理解话语。参与者会看到简短的故事，然后回答八个是/否问题，探究他们对故事的理解。这些问题的格式旨在评估直接性（明示与暗示）和显着性（主要思想与细节）的单独影响。鉴于人类的表现水平非常高，GPT-4 的表现略好于人类，但在统计上并不显着。 GPT-4 和人类都表现出强大的能力，可以对故事中未明确陈述的信息进行推断，这是对理解力的关键考验。

Title: Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model

• Authors: Braja Gopal Patra, Lauren A. Lepow, Praneet Kasi Reddy Jagadeesh Kumar, Veer Vekaria, Mohit Manoj Sharma, Prakash Adekkanattu, Brian Fennessy, Gavin Hynes, Isotta Landi, Jorge A. Sanchez-Ruiz, Euijung Ryu, Joanna M. Biernacka, Girish N. Nadkarni, Ardesheer Talati, Myrna Weissman, Mark Olfson, J. John Mann, Alexander W. Charney, Jyotishman Pathak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17199
• Pdf URL: https://arxiv.org/pdf/2403.17199
• Copy Paste: [[2403.17199]] Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model(https://arxiv.org/abs/2403.17199)
• Keywords: language model, llm
• Abstract: Background: Social support (SS) and social isolation (SI) are social determinants of health (SDOH) associated with psychiatric outcomes. In electronic health records (EHRs), individual-level SS/SI is typically documented as narrative clinical notes rather than structured coded data. Natural language processing (NLP) algorithms can automate the otherwise labor-intensive process of data extraction. Data and Methods: Psychiatric encounter notes from Mount Sinai Health System (MSHS, n=300) and Weill Cornell Medicine (WCM, n=225) were annotated and established a gold standard corpus. A rule-based system (RBS) involving lexicons and a large language model (LLM) using FLAN-T5-XL were developed to identify mentions of SS and SI and their subcategories (e.g., social network, instrumental support, and loneliness). Results: For extracting SS/SI, the RBS obtained higher macro-averaged f-scores than the LLM at both MSHS (0.89 vs. 0.65) and WCM (0.85 vs. 0.82). For extracting subcategories, the RBS also outperformed the LLM at both MSHS (0.90 vs. 0.62) and WCM (0.82 vs. 0.81). Discussion and Conclusion: Unexpectedly, the RBS outperformed the LLMs across all metrics. Intensive review demonstrates that this finding is due to the divergent approach taken by the RBS and LLM. The RBS were designed and refined to follow the same specific rules as the gold standard annotations. Conversely, the LLM were more inclusive with categorization and conformed to common English-language understanding. Both approaches offer advantages and are made available open-source for future testing.
• 摘要：背景：社会支持（SS）和社会隔离（SI）是与精神病结果相关的健康社会决定因素（SDOH）。在电子健康记录 (EHR) 中，个人级别的 SS/SI 通常记录为叙述性临床记录，而不是结构化编码数据。自然语言处理 (NLP) 算法可以自动化原本劳动密集型的数据提取过程。数据和方法：对西奈山卫生系统 (MSHS，n=300) 和威尔康奈尔医学 (WCM，n=225) 的精神病治疗笔记进行了注释并建立了金标准语料库。使用 FLAN-T5-XL 开发了一个涉及词典和大型语言模型 (LLM) 的基于规则的系统 (RBS)，用于识别 SS 和 SI 及其子类别（例如社交网络、工具支持和孤独）的提及。结果：对于提取 SS/SI，RBS 在 MSHS（0.89 vs. 0.65）和 WCM（0.85 vs. 0.82）方面均获得了比 LLM 更高的宏观平均 f 分数。在提取子类别方面，RBS 在 MSHS（0.90 vs. 0.62）和 WCM（0.82 vs. 0.81）方面也优于 LLM。讨论和结论：出乎意料的是，苏格兰皇家商学院在所有指标上都优于法学硕士。深入审查表明，这一发现是由于苏格兰皇家银行和法学硕士采取的不同方法造成的。 RBS 的设计和改进遵循与黄金标准注释相同的特定规则。相反，法学硕士在分类上更具包容性，并且符合通用的英语语言理解。这两种方法都有优势，并且可以开源以供将来测试。

Title: Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis

• Authors: Na Li, Thomas Bailleux, Zied Bouraoui, Steven Schockaert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17216
• Pdf URL: https://arxiv.org/pdf/2403.17216
• Copy Paste: [[2403.17216]] Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis(https://arxiv.org/abs/2403.17216)
• Keywords: language model
• Abstract: We consider the problem of finding plausible knowledge that is missing from a given ontology, as a generalisation of the well-studied taxonomy expansion task. One line of work treats this task as a Natural Language Inference (NLI) problem, thus relying on the knowledge captured by language models to identify the missing knowledge. Another line of work uses concept embeddings to identify what different concepts have in common, taking inspiration from cognitive models for category based induction. These two approaches are intuitively complementary, but their effectiveness has not yet been compared. In this paper, we introduce a benchmark for evaluating ontology completion methods and thoroughly analyse the strengths and weaknesses of both approaches. We find that both approaches are indeed complementary, with hybrid strategies achieving the best overall results. We also find that the task is highly challenging for Large Language Models, even after fine-tuning.
• 摘要：我们将寻找给定本体中缺失的合理知识的问题视为经过充分研究的分类扩展任务的概括。其中一项工作将此任务视为自然语言推理（NLI）问题，从而依靠语言模型捕获的知识来识别缺失的知识。另一项工作使用概念嵌入来识别不同概念的共同点，从基于类别的归纳的认知模型中获取灵感。这两种方法直观上是互补的，但其有效性尚未进行比较。在本文中，我们介绍了评估本体完成方法的基准，并彻底分析了两种方法的优缺点。我们发现这两种方法确实是互补的，混合策略可以获得最佳的总体结果。我们还发现，即使在微调之后，这项任务对于大型语言模型来说也极具挑战性。

Title: The Role of $n$-gram Smoothing in the Age of Neural Networks

• Authors: Luca Malagutti, Andrius Buinovskij, Anej Svete, Clara Meister, Afra Amini, Ryan Cotterell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17240
• Pdf URL: https://arxiv.org/pdf/2403.17240
• Copy Paste: [[2403.17240]] The Role of $n$-gram Smoothing in the Age of Neural Networks(https://arxiv.org/abs/2403.17240)
• Keywords: language model
• Abstract: For nearly three decades, language models derived from the $n$-gram assumption held the state of the art on the task. The key to their success lay in the application of various smoothing techniques that served to combat overfitting. However, when neural language models toppled $n$-gram models as the best performers, $n$-gram smoothing techniques became less relevant. Indeed, it would hardly be an understatement to suggest that the line of inquiry into $n$-gram smoothing techniques became dormant. This paper re-opens the role classical $n$-gram smoothing techniques may play in the age of neural language models. First, we draw a formal equivalence between label smoothing, a popular regularization technique for neural language models, and add-$\lambda$ smoothing. Second, we derive a generalized framework for converting \emph{any} $n$-gram smoothing technique into a regularizer compatible with neural language models. Our empirical results find that our novel regularizers are comparable to and, indeed, sometimes outperform label smoothing on language modeling and machine translation.
• 摘要：近三十年来，源自 $n$-gram 假设的语言模型一直保持着该任务的最先进水平。他们成功的关键在于应用各种平滑技术来对抗过度拟合。然而，当神经语言模型取代 $n$-gram 模型成为表现最佳的模型时，$n$-gram 平滑技术就变得不那么重要了。事实上，毫不夸张地说，对 $n$-gram 平滑技术的研究已经处于休眠状态。本文重新阐述了经典的 $n$-gram 平滑技术在神经语言模型时代可能发挥的作用。首先，我们在标签平滑（一种流行的神经语言模型正则化技术）和 add-$\lambda$ 平滑之间建立了形式上的等价关系。其次，我们推导了一个通用框架，用于将 \emph{any} $n$-gram 平滑技术转换为与神经语言模型兼容的正则化器。我们的实证结果发现，我们的新型正则化器在语言建模和机器翻译方面与标签平滑相当，甚至有时优于标签平滑。

Title: A Hybrid Approach To Aspect Based Sentiment Analysis Using Transfer Learning

• Authors: Gaurav Negi, Rajdeep Sarkar, Omnia Zayed, Paul Buitelaar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17254
• Pdf URL: https://arxiv.org/pdf/2403.17254
• Copy Paste: [[2403.17254]] A Hybrid Approach To Aspect Based Sentiment Analysis Using Transfer Learning(https://arxiv.org/abs/2403.17254)
• Keywords: language model, llm
• Abstract: Aspect-Based Sentiment Analysis (ABSA) aims to identify terms or multiword expressions (MWEs) on which sentiments are expressed and the sentiment polarities associated with them. The development of supervised models has been at the forefront of research in this area. However, training these models requires the availability of manually annotated datasets which is both expensive and time-consuming. Furthermore, the available annotated datasets are tailored to a specific domain, language, and text type. In this work, we address this notable challenge in current state-of-the-art ABSA research. We propose a hybrid approach for Aspect Based Sentiment Analysis using transfer learning. The approach focuses on generating weakly-supervised annotations by exploiting the strengths of both large language models (LLM) and traditional syntactic dependencies. We utilise syntactic dependency structures of sentences to complement the annotations generated by LLMs, as they may overlook domain-specific aspect terms. Extensive experimentation on multiple datasets is performed to demonstrate the efficacy of our hybrid method for the tasks of aspect term extraction and aspect sentiment classification. Keywords: Aspect Based Sentiment Analysis, Syntactic Parsing, large language model (LLM)
• 摘要：基于方面的情感分析 (ABSA) 旨在识别表达情感的术语或多词表达 (MWE) 以及与之相关的情感极性。监督模型的发展一直处于该领域研究的前沿。然而，训练这些模型需要手动注释的数据集，这既昂贵又耗时。此外，可用的带注释的数据集是针对特定领域、语言和文本类型定制的。在这项工作中，我们解决了当前最先进的 ABSA 研究中的这一显着挑战。我们提出了一种使用迁移学习进行基于方面的情感分析的混合方法。该方法侧重于通过利用大型语言模型（LLM）和传统句法依赖性的优势来生成弱监督注释。我们利用句子的句法依存结构来补充法学硕士生成的注释，因为它们可能会忽略特定领域的方面术语。对多个数据集进行了广泛的实验，以证明我们的混合方法对于方面术语提取和方面情感分类任务的有效性。关键词：基于方面的情感分析、句法分析、大语言模型（LLM）

Title: Automate Knowledge Concept Tagging on Math Questions with LLMs

• Authors: Hang Li, Tianlong Xu, Jiliang Tang, Qingsong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17281
• Pdf URL: https://arxiv.org/pdf/2403.17281
• Copy Paste: [[2403.17281]] Automate Knowledge Concept Tagging on Math Questions with LLMs(https://arxiv.org/abs/2403.17281)
• Keywords: language model, llm
• Abstract: Knowledge concept tagging for questions plays a crucial role in contemporary intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been conducted manually with help from pedagogical experts, as the task requires not only a strong semantic understanding of both question stems and knowledge definitions but also deep insights into connecting question-solving logic with corresponding knowledge concepts. In this paper, we explore automating the tagging task using Large Language Models (LLMs), in response to the inability of prior manual methods to meet the rapidly growing demand for concept tagging in questions posed by advanced educational applications. Moreover, the zero/few-shot learning capability of LLMs makes them well-suited for application in educational scenarios, which often face challenges in collecting large-scale, expertise-annotated datasets. By conducting extensive experiments with a variety of representative LLMs, we demonstrate that LLMs are a promising tool for concept tagging in math questions. Furthermore, through case studies examining the results from different LLMs, we draw some empirical conclusions about the key factors for success in applying LLMs to the automatic concept tagging task.
• 摘要：问题的知识概念标签在当代智能教育应用中发挥着至关重要的作用，包括学习进度诊断、练习题推荐、课程内容组织等。传统上，这些注释是在教学专家的帮助下手动进行的，因为这项任务不仅需要对问题主干和知识定义有很强的语义理解，还需要深入了解将问题解决逻辑与相应的知识概念联系起来。在本文中，我们探索使用大型语言模型（LLM）自动化标记任务，以应对先前的手动方法无法满足高级教育应用程序提出的问题中概念标记快速增长的需求。此外，法学硕士的零/少样本学习能力使其非常适合在教育场景中应用，而教育场景在收集大规模、有专业知识注释的数据集方面经常面临挑战。通过对各种具有代表性的法学硕士进行广泛的实验，我们证明法学硕士是数学问题中概念标记的有前途的工具。此外，通过案例研究检查不同法学硕士的结果，我们得出了一些关于成功将法学硕士应用于自动概念标记任务的关键因素的实证结论。

Title: InternLM2 Technical Report

• Authors: Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, et al. (31 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17297
• Pdf URL: https://arxiv.org/pdf/2403.17297
• Copy Paste: [[2403.17297]] InternLM2 Technical Report(https://arxiv.org/abs/2403.17297)
• Keywords: language model, gpt, llm, chat
• Abstract: The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.
• 摘要：ChatGPT 和 GPT-4 等大型语言模型 (LLM) 的发展引发了关于通用人工智能 (AGI) 出现的讨论。然而，在开源模型中复制这些进步一直具有挑战性。本文介绍了 InternLM2，这是一个开源法学硕士，通过创新的预训练和优化技术，它在 6 个维度和 30 个基准的综合评估、长上下文建模和开放式主观评估方面优于其前辈。 InternLM2的预训练过程非常细致，突出了文本、代码、长上下文数据等多种数据类型的准备。 InternLM2 有效地捕获了长期依赖关系，最初在 4k token 上进行训练，然后在预训练和微调阶段升级到 32k token，在 200k 的“大海捞针”测试中表现出了出色的性能。InternLM2 进一步对齐使用监督微调 (SFT) 和新颖的人类反馈条件在线强化学习 (COOL RLHF) 策略来解决人类偏好冲突和奖励黑客行为。通过发布不同训练阶段和模型大小的 InternLM2 模型，我们为社区提供了见解进入模型的演变。

Title: Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs

• Authors: Linyang He, Peili Chen, Ercong Nie, Yuanning Li, Jonathan R. Brennan
• Subjects: cs.CL, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.17299
• Pdf URL: https://arxiv.org/pdf/2403.17299
• Copy Paste: [[2403.17299]] Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs(https://arxiv.org/abs/2403.17299)
• Keywords: language model, gpt
• Abstract: Inspired by cognitive neuroscience studies, we introduce a novel `decoding probing' method that uses minimal pairs benchmark (BLiMP) to probe internal linguistic characteristics in neural language models layer by layer. By treating the language model as the `brain' and its representations as `neural activations', we decode grammaticality labels of minimal pairs from the intermediate layers' representations. This approach reveals: 1) Self-supervised language models capture abstract linguistic structures in intermediate layers that GloVe and RNN language models cannot learn. 2) Information about syntactic grammaticality is robustly captured through the first third layers of GPT-2 and also distributed in later layers. As sentence complexity increases, more layers are required for learning grammatical capabilities. 3) Morphological and semantics/syntax interface-related features are harder to capture than syntax. 4) For Transformer-based models, both embeddings and attentions capture grammatical features but show distinct patterns. Different attention heads exhibit similar tendencies toward various linguistic phenomena, but with varied contributions.
• 摘要：受认知神经科学研究的启发，我们引入了一种新颖的“解码探测”方法，该方法使用最小对基准（BLiMP）来逐层探测神经语言模型中的内部语言特征。通过将语言模型视为“大脑”并将其表示视为“神经激活”，我们从中间层的表示中解码最小对的语法标签。这种方法揭示了：1）自监督语言模型捕获了 GloVe 和 RNN 语言模型无法学习的中间层中的抽象语言结构。 2）关于句法语法性的信息通过 GPT-2 的前第三层被可靠地捕获，并且也分布在后面的层中。随着句子复杂性的增加，需要更多的层次来学习语法能力。 3）形态和语义/语法接口相关的特征比语法更难捕获。 4）对于基于 Transformer 的模型，嵌入和注意力都捕获语法特征，但显示出不同的模式。不同的注意力头对各种语言现象表现出相似的倾向，但贡献也不同。

Title: JMultiWOZ: A Large-Scale Japanese Multi-Domain Task-Oriented Dialogue Dataset

• Authors: Atsumoto Ohashi, Ryu Hirai, Shinya Iizuka, Ryuichiro Higashinaka
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17319
• Pdf URL: https://arxiv.org/pdf/2403.17319
• Copy Paste: [[2403.17319]] JMultiWOZ: A Large-Scale Japanese Multi-Domain Task-Oriented Dialogue Dataset(https://arxiv.org/abs/2403.17319)
• Keywords: language model, llm
• Abstract: Dialogue datasets are crucial for deep learning-based task-oriented dialogue system research. While numerous English language multi-domain task-oriented dialogue datasets have been developed and contributed to significant advancements in task-oriented dialogue systems, such a dataset does not exist in Japanese, and research in this area is limited compared to that in English. In this study, towards the advancement of research and development of task-oriented dialogue systems in Japanese, we constructed JMultiWOZ, the first Japanese language large-scale multi-domain task-oriented dialogue dataset. Using JMultiWOZ, we evaluated the dialogue state tracking and response generation capabilities of the state-of-the-art methods on the existing major English benchmark dataset MultiWOZ2.2 and the latest large language model (LLM)-based methods. Our evaluation results demonstrated that JMultiWOZ provides a benchmark that is on par with MultiWOZ2.2. In addition, through evaluation experiments of interactive dialogues with the models and human participants, we identified limitations in the task completion capabilities of LLMs in Japanese.
• 摘要：对话数据集对于基于深度学习的任务导向对话系统研究至关重要。虽然已经开发了许多英语语言多领域面向任务的对话数据集，并为面向任务的对话系统的重大进步做出了贡献，但日语中不存在这样的数据集，并且与英语相比，该领域的研究有限。在本研究中，为了推进日语任务导向对话系统的研究和开发，我们构建了第一个日语大规模多领域任务导向对话数据集JMultiWOZ。使用 JMultiWOZ，我们在现有主要英语基准数据集 MultiWOZ2.2 和最新的基于大语言模型 (LLM) 的方法上评估了最先进方法的对话状态跟踪和响应生成能力。我们的评估结果表明，JMultiWOZ 提供了与 MultiWOZ2.2 相当的基准。此外，通过与模型和人类参与者的互动对话的评估实验，我们发现了日语法学硕士任务完成能力的局限性。

Title: Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models

• Authors: Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17359
• Pdf URL: https://arxiv.org/pdf/2403.17359
• Copy Paste: [[2403.17359]] Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models(https://arxiv.org/abs/2403.17359)
• Keywords: language model, hallucination, prompt
• Abstract: We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions. Methodologically, we propose three types of domain-adaptable `Plug-and-Play' actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score (MRFS) to verify and resolve conflicts in the answers. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.
• 摘要：我们提出了一个用于多模式和检索增强问答（QA）的行动链（CoA）框架。与文献相比，CoA 克服了当前 QA 应用程序的两个主要挑战：（i）与实时或领域事实不一致的不忠实幻觉；（ii）对组合信息的推理性能较弱。我们的主要贡献是一种新颖的推理检索机制，通过系统的提示和预先设计的动作将复杂的问题分解为推理链。在方法上，我们提出了三种类型的领域适应性“即插即用”操作，用于从异构源检索实时信息。我们还提出了多参考信念评分（MRFS）来验证和解决答案中的冲突。根据经验，我们利用公共基准和 Web3 案例研究来证明 CoA 相对于其他方法的能力。

Title: Extracting Biomedical Entities from Noisy Audio Transcripts

• Authors: Nima Ebadi, Kellen Morgan, Adrian Tan, Billy Linares, Sheri Osborn, Emma Majors, Jeremy Davis, Anthony Rios
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17363
• Pdf URL: https://arxiv.org/pdf/2403.17363
• Copy Paste: [[2403.17363]] Extracting Biomedical Entities from Noisy Audio Transcripts(https://arxiv.org/abs/2403.17363)
• Keywords: gpt
• Abstract: Automatic Speech Recognition (ASR) technology is fundamental in transcribing spoken language into text, with considerable applications in the clinical realm, including streamlining medical transcription and integrating with Electronic Health Record (EHR) systems. Nevertheless, challenges persist, especially when transcriptions contain noise, leading to significant drops in performance when Natural Language Processing (NLP) models are applied. Named Entity Recognition (NER), an essential clinical task, is particularly affected by such noise, often termed the ASR-NLP gap. Prior works have primarily studied ASR's efficiency in clean recordings, leaving a research gap concerning the performance in noisy environments. This paper introduces a novel dataset, BioASR-NER, designed to bridge the ASR-NLP gap in the biomedical domain, focusing on extracting adverse drug reactions and mentions of entities from the Brief Test of Adult Cognition by Telephone (BTACT) exam. Our dataset offers a comprehensive collection of almost 2,000 clean and noisy recordings. In addressing the noise challenge, we present an innovative transcript-cleaning method using GPT4, investigating both zero-shot and few-shot methodologies. Our study further delves into an error analysis, shedding light on the types of errors in transcription software, corrections by GPT4, and the challenges GPT4 faces. This paper aims to foster improved understanding and potential solutions for the ASR-NLP gap, ultimately supporting enhanced healthcare documentation practices.
• 摘要：自动语音识别 (ASR) 技术是将口语转录为文本的基础技术，在临床领域具有大量应用，包括简化医疗转录和与电子健康记录 (EHR) 系统集成。然而，挑战仍然存在，特别是当转录包含噪声时，导致应用自然语言处理（NLP）模型时性能显着下降。命名实体识别 (NER) 是一项重要的临床任务，尤其受到这种噪声的影响，通常被称为 ASR-NLP 差距。之前的工作主要研究了 ASR 在干净录音中的效率，在噪声环境中的性能方面留下了研究空白。本文介绍了一个新颖的数据集 BioASR-NER，旨在弥合生物医学领域的 ASR-NLP 差距，重点是从成人电话认知简短测试 (BTACT) 考试中提取药物不良反应和提及的实体。我们的数据集提供了近 2,000 个干净和嘈杂录音的全面集合。为了解决噪声挑战，我们提出了一种使用 GPT4 的创新转录本清理方法，研究了零样本和少样本方法。我们的研究进一步深入研究了错误分析，揭示了转录软件中的错误类型、GPT4 的更正以及 GPT4 面临的挑战。本文旨在促进对 ASR-NLP 差距的更好理解和潜在解决方案，最终支持增强的医疗保健文档实践。

Title: ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?

• Authors: Fan Huang, Haewoon Kwak, Kunwoo Park, Jisun An
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17368
• Pdf URL: https://arxiv.org/pdf/2403.17368
• Copy Paste: [[2403.17368]] ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?(https://arxiv.org/abs/2403.17368)
• Keywords: language model, gpt, prompt, chat
• Abstract: As AI becomes more integral in our lives, the need for transparency and responsibility grows. While natural language explanations (NLEs) are vital for clarifying the reasoning behind AI decisions, evaluating them through human judgments is complex and resource-intensive due to subjectivity and the need for fine-grained ratings. This study explores the alignment between ChatGPT and human assessments across multiple scales (i.e., binary, ternary, and 7-Likert scale). We sample 300 data instances from three NLE datasets and collect 900 human annotations for both informativeness and clarity scores as the text quality measurement. We further conduct paired comparison experiments under different ranges of subjectivity scores, where the baseline comes from 8,346 human annotations. Our results show that ChatGPT aligns better with humans in more coarse-grained scales. Also, paired comparisons and dynamic prompting (i.e., providing semantically similar examples in the prompt) improve the alignment. This research advances our understanding of large language models' capabilities to assess the text explanation quality in different configurations for responsible AI development.
• 摘要：随着人工智能在我们的生活中变得越来越不可或缺，对透明度和责任的需求也不断增长。虽然自然语言解释 (NLE) 对于阐明人工智能决策背后的推理至关重要，但由于主观性和细粒度评级的需要，通过人类判断对其进行评估非常复杂且需要大量资源。本研究探讨了 ChatGPT 与多个尺度（即二元、三元和 7-Likert 尺度）的人类评估之间的一致性。我们从三个 NLE 数据集中采样了 300 个数据实例，并收集了 900 个人工注释，以获取信息量和清晰度分数作为文本质量衡量标准。我们进一步在不同主观评分范围下进行配对比较实验，其中基线来自 8,346 个人类注释。我们的结果表明，ChatGPT 在更粗粒度的尺度上与人类更加一致。此外，配对比较和动态提示（即在提示中提供语义相似的示例）可以改善一致性。这项研究增进了我们对大型语言模型能力的理解，以评估不同配置下的文本解释质量，以实现负责任的人工智能开发。

Title: ELLEN: Extremely Lightly Supervised Learning For Efficient Named Entity Recognition

• Authors: Haris Riaz, Razvan-Gabriel Dumitru, Mihai Surdeanu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17385
• Pdf URL: https://arxiv.org/pdf/2403.17385
• Copy Paste: [[2403.17385]] ELLEN: Extremely Lightly Supervised Learning For Efficient Named Entity Recognition(https://arxiv.org/abs/2403.17385)
• Keywords: language model, gpt
• Abstract: In this work, we revisit the problem of semi-supervised named entity recognition (NER) focusing on extremely light supervision, consisting of a lexicon containing only 10 examples per class. We introduce ELLEN, a simple, fully modular, neuro-symbolic method that blends fine-tuned language models with linguistic rules. These rules include insights such as ''One Sense Per Discourse'', using a Masked Language Model as an unsupervised NER, leveraging part-of-speech tags to identify and eliminate unlabeled entities as false negatives, and other intuitions about classifier confidence scores in local and global context. ELLEN achieves very strong performance on the CoNLL-2003 dataset when using the minimal supervision from the lexicon above. It also outperforms most existing (and considerably more complex) semi-supervised NER methods under the same supervision settings commonly used in the literature (i.e., 5% of the training data). Further, we evaluate our CoNLL-2003 model in a zero-shot scenario on WNUT-17 where we find that it outperforms GPT-3.5 and achieves comparable performance to GPT-4. In a zero-shot setting, ELLEN also achieves over 75% of the performance of a strong, fully supervised model trained on gold data. Our code is available at: https://github.com/hriaz17/ELLEN.
• 摘要：在这项工作中，我们重新审视半监督命名实体识别 (NER) 问题，重点关注极轻监督，由每类仅包含 10 个示例的词典组成。我们介绍 ELLEN，一种简单、完全模块化的神经符号方法，它将微调的语言模型与语言规则相结合。这些规则包括诸如“每个话语一个意义”之类的见解，使用掩码语言模型作为无监督的命名实体识别，利用词性标签来识别和消除作为漏报的未标记实体，以及关于分类器置信度得分的其他直觉。本地和全球背景。当使用上述词典中的最小监督时，ELLEN 在 CoNLL-2003 数据集上取得了非常出色的性能。在文献中常用的相同监督设置（即 5% 的训练数据）下，它还优于大多数现有（并且更加复杂）的半监督 NER 方法。此外，我们在 WNUT-17 上的零样本场景中评估了我们的 CoNLL-2003 模型，发现它的性能优于 GPT-3.5，并实现了与 GPT-4 相当的性能。在零样本设置中，ELLEN 的性能也达到了基于黄金数据训练的强大、完全监督模型的 75% 以上。我们的代码位于：https://github.com/hriaz17/ELLEN。

Title: PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models

• Authors: Jinyi Li, Yihuai Lan, Lei Wang, Hao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17411
• Pdf URL: https://arxiv.org/pdf/2403.17411
• Copy Paste: [[2403.17411]] PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models(https://arxiv.org/abs/2403.17411)
• Keywords: language model, llm, prompt
• Abstract: Prompt compression is an innovative method for efficiently condensing input prompts while preserving essential information. To facilitate quick-start services, user-friendly interfaces, and compatibility with common datasets and metrics, we present the Prompt Compression Toolkit (PCToolkit). This toolkit is a unified plug-and-play solution for compressing prompts in Large Language Models (LLMs), featuring cutting-edge prompt compressors, diverse datasets, and metrics for comprehensive performance evaluation. PCToolkit boasts a modular design, allowing for easy integration of new datasets and metrics through portable and user-friendly interfaces. In this paper, we outline the key components and functionalities of PCToolkit. We conducted evaluations of the compressors within PCToolkit across various natural language tasks, including reconstruction, summarization, mathematical problem-solving, question answering, few-shot learning, synthetic tasks, code completion, boolean expressions, multiple choice questions, and lies recognition.
• 摘要：提示压缩是一种创新方法，可有效压缩输入提示，同时保留基本信息。为了促进快速启动服务、用户友好的界面以及与常见数据集和指标的兼容性，我们推出了提示压缩工具包 (PCToolkit)。该工具包是一个统一的即插即用解决方案，用于压缩大型语言模型 (LLM) 中的提示，具有尖端的提示压缩器、多样化的数据集和用于综合性能评估的指标。 PCToolkit 采用模块化设计，可以通过便携式且用户友好的界面轻松集成新的数据集和指标。在本文中，我们概述了 PCToolkit 的关键组件和功能。我们对 PCToolkit 中的压缩器进行了跨各种自然语言任务的评估，包括重建、总结、数学问题解决、问答、小样本学习、合成任务、代码完成、布尔表达式、多项选择题和谎言识别。

Title: LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction

• Authors: Yixuan Wang, Baoxin Wang, Yijun Liu, Dayong Wu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17413
• Pdf URL: https://arxiv.org/pdf/2403.17413
• Copy Paste: [[2403.17413]] LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction(https://arxiv.org/abs/2403.17413)
• Keywords: gpt, chat
• Abstract: Over-correction is a critical problem in Chinese grammatical error correction (CGEC) task. Recent work using model ensemble methods based on voting can effectively mitigate over-correction and improve the precision of the GEC system. However, these methods still require the output of several GEC systems and inevitably lead to reduced error recall. In this light, we propose the LM-Combiner, a rewriting model that can directly modify the over-correction of GEC system outputs without a model ensemble. Specifically, we train the model on an over-correction dataset constructed through the proposed K-fold cross inference method, which allows it to directly generate filtered sentences by combining the original and the over-corrected text. In the inference stage, we directly take the original sentences and the output results of other systems as input and then obtain the filtered sentences through LM-Combiner. Experiments on the FCGEC dataset show that our proposed method effectively alleviates the over-correction of the original system (+18.2 Precision) while ensuring the error recall remains unchanged. Besides, we find that LM-Combiner still has a good rewriting performance even with small parameters and few training data, and thus can cost-effectively mitigate the over-correction of black-box GEC systems (e.g., ChatGPT).
• 摘要：过度纠正是中文语法错误纠正（CGEC）任务中的一个关键问题。最近的工作使用基于投票的模型集成方法可以有效减轻过度校正并提高 GEC 系统的精度。然而，这些方法仍然需要多个 GEC 系统的输出，并且不可避免地导致错误召回率降低。有鉴于此，我们提出了 LM-Combiner，这是一种重写模型，可以直接修改 GEC 系统输出的过度校正，而无需模型集成。具体来说，我们在通过所提出的 K 折交叉推理方法构建的过度校正数据集上训练模型，该方法允许模型通过结合原始文本和过度校正文本直接生成过滤后的句子。在推理阶段，我们直接将原始句子和其他系统的输出结果作为输入，然后通过LM-Combiner获得过滤后的句子。 FCGEC数据集上的实验表明，我们提出的方法有效缓解了原始系统的过度校正（+18.2精度），同时保证错误召回率保持不变。此外，我们发现即使参数较小和训练数据较少，LM-Combiner 仍然具有良好的重写性能，因此可以经济有效地减轻黑盒 GEC 系统（例如 ChatGPT）的过度校正。

Title: Robust and Scalable Model Editing for Large Language Models

• Authors: Yingfa Chen, Zhengyan Zhang, Xu Han, Chaojun Xiao, Zhiyuan Liu, Chen Chen, Kuai Li, Tao Yang, Maosong Sun
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17431
• Pdf URL: https://arxiv.org/pdf/2403.17431
• Copy Paste: [[2403.17431]] Robust and Scalable Model Editing for Large Language Models(https://arxiv.org/abs/2403.17431)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context. In many scenarios, a desirable behavior is that LLMs give precedence to contextual knowledge when it conflicts with the parametric knowledge, and fall back to using their parametric knowledge when the context is irrelevant. This enables updating and correcting the model's knowledge by in-context editing instead of retraining. Previous works have shown that LLMs are inclined to ignore contextual knowledge and fail to reliably fall back to parametric knowledge when presented with irrelevant context. In this work, we discover that, with proper prompting methods, instruction-finetuned LLMs can be highly controllable by contextual knowledge and robust to irrelevant context. Utilizing this feature, we propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. To better evaluate the robustness of model editors, we collect a new dataset, that contains irrelevant questions that are more challenging than the ones in existing datasets. Empirical results show that our method outperforms current state-of-the-art methods by a large margin. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs (and vice versa). The source code can be found at https://github.com/thunlp/EREN.
• 摘要：大型语言模型 (LLM) 可以使用参数知识（模型权重中编码的知识）或上下文知识（上下文中呈现的知识）进行预测。在许多情况下，理想的行为是法学硕士在上下文知识与参数知识冲突时优先考虑上下文知识，并在上下文不相关时回退到使用参数知识。这使得可以通过上下文编辑而不是重新训练来更新和纠正模型的知识。之前的研究表明，法学硕士倾向于忽略上下文知识，并且在出现不相关上下文时无法可靠地回归到参数知识。在这项工作中，我们发现，通过适当的提示方法，指令微调的法学硕士可以通过上下文知识高度可控，并且对不相关的上下文具有鲁棒性。利用这一特性，我们提出了EREN（Edit models by READing Notes）来提高LLM编辑的可扩展性和鲁棒性。为了更好地评估模型编辑器的稳健性，我们收集了一个新的数据集，其中包含比现有数据集中的问题更具挑战性的不相关问题。实证结果表明，我们的方法大大优于当前最先进的方法。与现有技术不同，它可以整合来自多个编辑的知识，并正确响应语法相似但语义不相关的输入（反之亦然）。源代码可以在 https://github.com/thunlp/EREN 找到。

Title: DGoT: Dynamic Graph of Thoughts for Scientific Abstract Generation

• Authors: Xinyu Ning, Yutong Zhao, Yitong Liu, Hongwen Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17491
• Pdf URL: https://arxiv.org/pdf/2403.17491
• Copy Paste: [[2403.17491]] DGoT: Dynamic Graph of Thoughts for Scientific Abstract Generation(https://arxiv.org/abs/2403.17491)
• Keywords: language model, llm, hallucination, prompt
• Abstract: The method of training language models based on domain datasets has obtained significant achievements in the task of generating scientific paper abstracts. However, such models face problems of generalization and expensive training costs. The use of large language models (LLMs) to solve the task of generating paper abstracts saves the cost of model training. However, due to the hallucination problem of LLM, it is often necessary to improve the reliability of the results through multi-round query prompt approach such as Graph of Thoughts (GoT), which also brings additional reasoning costs. In this paper, we propose a Dynamic Graph of Thought (DGoT). It not only inherits the advantages of the existing GoT prompt approach, but also dynamically adjust the graph structure according to data characteristics while reducing model reasoning cost. Experimental results show that our method's cost-effectiveness in abstract generation tasks is only 43.7% to 56.4% of other multi-round query prompt approaches. Our code is available at https://github.com/JayceNing/DGoT.
• 摘要：基于领域数据集训练语言模型的方法在生成科学论文摘要的任务中取得了显着的成果。然而，此类模型面临泛化和昂贵的培训成本问题。使用大型语言模型（LLM）来解决生成论文摘要的任务，节省了模型训练的成本。但由于LLM的幻觉问题，往往需要通过Graph of Thoughts（GoT）等多轮查询提示方式来提高结果的可靠性，这也带来了额外的推理成本。在本文中，我们提出了动态思维图（DGoT）。它不仅继承了现有GoT提示方法的优点，而且可以根据数据特征动态调整图结构，同时降低模型推理成本。实验结果表明，我们的方法在抽象生成任务中的成本效益仅为其他多轮查询提示方法的43.7%至56.4%。我们的代码可在 https://github.com/JayceNing/DGoT 获取。

Title: Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies

• Authors: Philipp Sadler, Sherzod Hakimov, David Schlangen
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17497
• Pdf URL: https://arxiv.org/pdf/2403.17497
• Copy Paste: [[2403.17497]] Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies(https://arxiv.org/abs/2403.17497)
• Keywords: agent
• Abstract: In collaborative goal-oriented settings, the participants are not only interested in achieving a successful outcome, but do also implicitly negotiate the effort they put into the interaction (by adapting to each other). In this work, we propose a challenging interactive reference game that requires two players to coordinate on vision and language observations. The learning signal in this game is a score (given after playing) that takes into account the achieved goal and the players' assumed efforts during the interaction. We show that a standard Proximal Policy Optimization (PPO) setup achieves a high success rate when bootstrapped with heuristic partner behaviors that implement insights from the analysis of human-human interactions. And we find that a pairing of neural partners indeed reduces the measured joint effort when playing together repeatedly. However, we observe that in comparison to a reasonable heuristic pairing there is still room for improvement -- which invites further research in the direction of cost-sharing in collaborative interactions.
• 摘要：在以目标为导向的协作环境中，参与者不仅对取得成功的结果感兴趣，而且还隐含地协商他们在交互中投入的努力（通过相互适应）。在这项工作中，我们提出了一个具有挑战性的交互式参考游戏，需要两个玩家协调视觉和语言观察。该游戏中的学习信号是一个分数（玩后给出），该分数考虑了已实现的目标和玩家在交互过程中假设的努力。我们表明，标准的近端策略优化（PPO）设置在利用启发式合作伙伴行为进行引导时可以实现很高的成功率，这些行为可以实现人与人交互分析中的见解。我们发现，神经伙伴配对确实会减少反复一起玩耍时测量到的共同努力。然而，我们观察到，与合理的启发式配对相比，仍有改进的空间——这需要在协作交互中的成本分摊方向进行进一步的研究。

Title: Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications

• Authors: Chihiro Yano, Akihiko Fukuchi, Shoko Fukasawa, Hideyuki Tachibana, Yotaro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17528
• Pdf URL: https://arxiv.org/pdf/2403.17528
• Copy Paste: [[2403.17528]] Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications(https://arxiv.org/abs/2403.17528)
• Keywords: language model
• Abstract: Prior work on multilingual sentence embedding has demonstrated that the efficient use of natural language inference (NLI) data to build high-performance models can outperform conventional methods. However, the potential benefits from the recent ``exponential'' growth of language models with billions of parameters have not yet been fully explored. In this paper, we introduce Multilingual Sentence T5 (m-ST5), as a larger model of NLI-based multilingual sentence embedding, by extending Sentence T5, an existing monolingual model. By employing the low-rank adaptation (LoRA) technique, we have achieved a successful scaling of the model's size to 5.7 billion parameters. We conducted experiments to evaluate the performance of sentence embedding and verified that the method outperforms the NLI-based prior approach. Furthermore, we also have confirmed a positive correlation between the size of the model and its performance. It was particularly noteworthy that languages with fewer resources or those with less linguistic similarity to English benefited more from the parameter increase. Our model is available at https://huggingface.co/pkshatech/m-ST5.
• 摘要：先前关于多语言句子嵌入的工作已经证明，有效使用自然语言推理（NLI）数据来构建高性能模型可以优于传统方法。然而，具有数十亿参数的语言模型最近“指数”增长的潜在好处尚未得到充分探索。在本文中，我们通过扩展现有的单语言模型 Sentence T5，引入了多语言句子 T5 (m-ST5)，作为基于 NLI 的多语言句子嵌入的更大模型。通过采用低秩适应 (LoRA) 技术，我们成功地将模型大小扩展到 57 亿个参数。我们进行了实验来评估句子嵌入的性能，并验证该方法优于基于 NLI 的先验方法。此外，我们还证实了模型的大小与其性能之间存在正相关关系。特别值得注意的是，资源较少或与英语语言相似度较低的语言从参数增加中受益更多。我们的模型可在 https://huggingface.co/pkshatech/m-ST5 上找到。

Title: ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler

• Authors: Paramita Mirza, Viju Sudhi, Soumya Ranjan Sahoo, Sinchana Ramakanth Bhat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17536
• Pdf URL: https://arxiv.org/pdf/2403.17536
• Copy Paste: [[2403.17536]] ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler(https://arxiv.org/abs/2403.17536)
• Keywords: language model, gpt, llm, prompt
• Abstract: State-of-the-art intent classification (IC) and slot filling (SF) methods often rely on data-intensive deep learning models, limiting their practicality for industry applications. Large language models on the other hand, particularly instruction-tuned models (Instruct-LLMs), exhibit remarkable zero-shot performance across various natural language tasks. This study evaluates Instruct-LLMs on popular benchmark datasets for IC and SF, emphasizing their capacity to learn from fewer examples. We introduce ILLUMINER, an approach framing IC and SF as language generation tasks for Instruct-LLMs, with a more efficient SF-prompting method compared to prior work. A comprehensive comparison with multiple baselines shows that our approach, using the FLAN-T5 11B model, outperforms the state-of-the-art joint IC+SF method and in-context learning with GPT3.5 (175B), particularly in slot filling by 11.1--32.2 percentage points. Additionally, our in-depth ablation study demonstrates that parameter-efficient fine-tuning requires less than 6% of training data to yield comparable performance with traditional full-weight fine-tuning.
• 摘要：最先进的意图分类（IC）和槽填充（SF）方法通常依赖于数据密集型深度学习模型，限制了它们在行业应用中的实用性。另一方面，大型语言模型，特别是指令调整模型（Instruct-LLM），在各种自然语言任务中表现出卓越的零样本性能。这项研究在 IC 和 SF 的流行基准数据集上评估了 Instruct-LLM，强调他们从更少的例子中学习的能力。我们引入了 ILLUMINER，一种将 IC 和 SF 作为 Instruct-LLM 的语言生成任务的方法，与之前的工作相比，具有更有效的 SF 提示方法。与多个基线的全面比较表明，我们使用 FLAN-T5 11B 模型的方法优于最先进的联合 IC+SF 方法和 GPT3.5 (175B) 的上下文学习，特别是在槽填充方面11.1--32.2 个百分点。此外，我们深入的消融研究表明，参数高效的微调只需不到 6% 的训练数据即可产生与传统全权重微调相当的性能。

Title: Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction

• Authors: Masamune Kobayashi, Masato Mita, Mamoru Komachi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17540
• Pdf URL: https://arxiv.org/pdf/2403.17540
• Copy Paste: [[2403.17540]] Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction(https://arxiv.org/abs/2403.17540)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have been reported to outperform existing automatic evaluation metrics in some tasks, such as text summarization and machine translation. However, there has been a lack of research on LLMs as evaluators in grammatical error correction (GEC). In this study, we investigate the performance of LLMs in GEC evaluation by employing prompts designed to incorporate various evaluation criteria inspired by previous research. Our extensive experimental results demonstrate that GPT-4 achieved Kendall's rank correlation of 0.662 with human judgments, surpassing all existing methods. Furthermore, in recent GEC evaluations, we have underscored the significance of the LLMs scale and particularly emphasized the importance of fluency among evaluation criteria.
• 摘要：据报道，大型语言模型 (LLM) 在某些任务中的表现优于现有的自动评估指标，例如文本摘要和机器翻译。然而，目前缺乏对法学硕士作为语法错误纠正（GEC）评估者的研究。在这项研究中，我们通过采用旨在纳入受先前研究启发的各种评估标准的提示来调查法学硕士在 GEC 评估中的表现。我们广泛的实验结果表明，GPT-4 与人类判断的 Kendall 等级相关性达到 0.662，超越了所有现有方法。此外，在最近的GEC评估中，我们强调了LLM量表的重要性，特别强调了评估标准之间的流畅性的重要性。

Title: Naive Bayes-based Context Extension for Large Language Models

• Authors: Jianlin Su, Murtadha Ahmed, Wenbo, Luo Ao, Mingren Zhu, Yunfeng Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17552
• Pdf URL: https://arxiv.org/pdf/2403.17552
• Copy Paste: [[2403.17552]] Naive Bayes-based Context Extension for Large Language Models(https://arxiv.org/abs/2403.17552)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown promising in-context learning abilities. However, conventional In-Context Learning (ICL) approaches are often impeded by length limitations of transformer architecture, which pose challenges when attempting to effectively integrate supervision from a substantial number of demonstration examples. In this paper, we introduce a novel framework, called Naive Bayes-based Context Extension (NBCE), to enable existing LLMs to perform ICL with an increased number of demonstrations by significantly expanding their context size. Importantly, this expansion does not require fine-tuning or dependence on particular model architectures, all the while preserving linear efficiency. NBCE initially splits the context into equal-sized windows fitting the target LLM's maximum length. Then, it introduces a voting mechanism to select the most relevant window, regarded as the posterior context. Finally, it employs Bayes' theorem to generate the test task. Our experimental results demonstrate that NBCE substantially enhances performance, particularly as the number of demonstration examples increases, consistently outperforming alternative methods. The NBCE code will be made publicly accessible. The code NBCE is available at: https://github.com/amurtadha/NBCE-master
• 摘要：大型语言模型（LLM）已显示出有前途的上下文学习能力。然而，传统的上下文学习（ICL）方法常常受到变压器架构长度限制的阻碍，这在尝试有效地集成大量演示示例的监督时提出了挑战。在本文中，我们介绍了一种称为基于朴素贝叶斯的上下文扩展（NBCE）的新颖框架，使现有的法学硕士能够通过显着扩展其上下文大小来执行 ICL，并增加演示数量。重要的是，这种扩展不需要微调或依赖于特定的模型架构，同时保持线性效率。 NBCE 最初将上下文分割成大小相等的窗口，以适合目标 LLM 的最大长度。然后，它引入了投票机制来选择最相关的窗口，将其视为后验上下文。最后，它利用贝叶斯定理生成测试任务。我们的实验结果表明，NBCE 显着提高了性能，特别是随着演示示例数量的增加，始终优于替代方法。 NBCE 代码将公开。代码 NBCE 位于：https://github.com/amurtadha/NBCE-master

Title: RuBia: A Russian Language Bias Detection Dataset

• Authors: Veronika Grigoreva, Anastasiia Ivanova, Ilseyar Alimova, Ekaterina Artemova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17553
• Pdf URL: https://arxiv.org/pdf/2403.17553
• Copy Paste: [[2403.17553]] RuBia: A Russian Language Bias Detection Dataset(https://arxiv.org/abs/2403.17553)
• Keywords: language model, llm
• Abstract: Warning: this work contains upsetting or disturbing content. Large language models (LLMs) tend to learn the social and cultural biases present in the raw pre-training data. To test if an LLM's behavior is fair, functional datasets are employed, and due to their purpose, these datasets are highly language and culture-specific. In this paper, we address a gap in the scope of multilingual bias evaluation by presenting a bias detection dataset specifically designed for the Russian language, dubbed as RuBia. The RuBia dataset is divided into 4 domains: gender, nationality, socio-economic status, and diverse, each of the domains is further divided into multiple fine-grained subdomains. Every example in the dataset consists of two sentences with the first reinforcing a potentially harmful stereotype or trope and the second contradicting it. These sentence pairs were first written by volunteers and then validated by native-speaking crowdsourcing workers. Overall, there are nearly 2,000 unique sentence pairs spread over 19 subdomains in RuBia. To illustrate the dataset's purpose, we conduct a diagnostic evaluation of state-of-the-art or near-state-of-the-art LLMs and discuss the LLMs' predisposition to social biases.
• 摘要：警告：本作品包含令人不安或令人不安的内容。大型语言模型（LLM）倾向于学习原始预训练数据中存在的社会和文化偏见。为了测试法学硕士的行为是否公平，使用了功能数据集，并且由于其目的，这些数据集具有高度的语言和文化特定性。在本文中，我们通过提出专门为俄语设计的偏差检测数据集（称为 Rubia）来解决多语言偏差评估范围内的空白。 Rubia数据集分为4个域：性别、国籍、社会经济地位和多样性，每个域又分为多个细粒度的子域。数据集中的每个示例都由两个句子组成，第一个句子强化了潜在有害的刻板印象或比喻，第二个句子与之相矛盾。这些句子对首先由志愿者编写，然后由母语众包工作人员验证。总体而言，RuBia 中有近 2,000 个独特的句子对分布在 19 个子域中。为了说明数据集的目的，我们对最先进或接近最先进的法学硕士进行了诊断评估，并讨论了法学硕士对社会偏见的倾向。

Title: m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt

• Authors: Jian Yang, Hongcheng Guo, Yuwei Yin, Jiaqi Bai, Bing Wang, Jiaheng Liu, Xinnian Liang, Linzheng Cahi, Liqun Yang, Zhoujun Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17556
• Pdf URL: https://arxiv.org/pdf/2403.17556
• Copy Paste: [[2403.17556]] m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt(https://arxiv.org/abs/2403.17556)
• Keywords: prompt
• Abstract: Multilingual translation supports multiple translation directions by projecting all languages in a shared space, but the translation quality is undermined by the difference between languages in the text-only modality, especially when the number of languages is large. To bridge this gap, we introduce visual context as the universal language-independent representation to facilitate multilingual translation. In this paper, we propose a framework to leverage the multimodal prompt to guide the Multimodal Multilingual neural Machine Translation (m3P), which aligns the representations of different languages with the same meaning and generates the conditional vision-language memory for translation. We construct a multilingual multimodal instruction dataset (InstrMulti102) to support 102 languages. Our method aims to minimize the representation distance of different languages by regarding the image as a central language. Experimental results show that m3P outperforms previous text-only baselines and multilingual multimodal methods by a large margin. Furthermore, the probing experiments validate the effectiveness of our method in enhancing translation under the low-resource and massively multilingual scenario.
• 摘要：多语言翻译通过将所有语言投影在共享空间中来支持多个翻译方向，但纯文本模式下的语言之间的差异会降低翻译质量，尤其是当语言数量较多时。为了弥合这一差距，我们引入视觉上下文作为独立于语言的通用表示，以促进多语言翻译。在本文中，我们提出了一个利用多模态提示来指导多模态多语言神经机器翻译（m3P）的框架，该框架将具有相同含义的不同语言的表示对齐，并生成用于翻译的条件视觉语言记忆。我们构建了一个多语言多模式指令数据集（InstrMulti102）来支持 102 种语言。我们的方法旨在通过将图像视为中心语言来最小化不同语言的表示距离。实验结果表明，m3P 大幅优于之前的纯文本基线和多语言多模态方法。此外，探索实验验证了我们的方法在资源匮乏和大规模多语言场景下增强翻译的有效性。

Title: Towards a Zero-Data, Controllable, Adaptive Dialog System

• Authors: Dirk Väth, Lindsey Vanderlyn, Ngoc Thang Vu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17582
• Pdf URL: https://arxiv.org/pdf/2403.17582
• Copy Paste: [[2403.17582]] Towards a Zero-Data, Controllable, Adaptive Dialog System(https://arxiv.org/abs/2403.17582)
• Keywords: language model, agent
• Abstract: Conversational Tree Search (V\"ath et al., 2023) is a recent approach to controllable dialog systems, where domain experts shape the behavior of a Reinforcement Learning agent through a dialog tree. The agent learns to efficiently navigate this tree, while adapting to information needs, e.g., domain familiarity, of different users. However, the need for additional training data hinders deployment in new domains. To address this, we explore approaches to generate this data directly from dialog trees. We improve the original approach, and show that agents trained on synthetic data can achieve comparable dialog success to models trained on human data, both when using a commercial Large Language Model for generation, or when using a smaller open-source model, running on a single GPU. We further demonstrate the scalability of our approach by collecting and testing on two new datasets: ONBOARD, a new domain helping foreign residents moving to a new city, and the medical domain DIAGNOSE, a subset of Wikipedia articles related to scalp and head symptoms. Finally, we perform human testing, where no statistically significant differences were found in either objective or subjective measures between models trained on human and generated data.
• 摘要：对话树搜索（V\"ath et al., 2023）是可控对话系统的一种最新方法，其中领域专家通过对话树塑造强化学习代理的行为。代理学习有效地导航这棵树，同时适应不同用户的信息需求，例如领域熟悉度。然而，对额外训练数据的需求阻碍了在新领域的部署。为了解决这个问题，我们探索了直接从对话树生成这些数据的方法。我们改进了原始方法，并且表明，无论是使用商业大型语言模型进行生成，还是使用在单个 GPU 上运行的较小开源模型时，基于合成数据训练的代理都可以取得与基于人类数据训练的模型相当的对话成功率。通过收集和测试两个新数据集来提高我们方法的可扩展性：ONBOARD，一个帮助外国居民搬到新城市的新域；以及医疗域 DIAGNOSE，与头皮和头部症状相关的维基百科文章的子集。最后，我们进行了人体测试，在人类数据和生成数据上训练的模型之间，无论是客观还是主观测量，都没有发现统计上的显着差异。

Title: "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling

• Authors: Christopher Bagdon, Prathamesh Karmalker, Harsha Gurulingappa, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17612
• Pdf URL: https://arxiv.org/pdf/2403.17612
• Copy Paste: [[2403.17612]] "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling(https://arxiv.org/abs/2403.17612)
• Keywords: language model
• Abstract: Labeling corpora constitutes a bottleneck to create models for new tasks or domains. Large language models mitigate the issue with automatic corpus labeling methods, particularly for categorical annotations. Some NLP tasks such as emotion intensity prediction, however, require text regression, but there is no work on automating annotations for continuous label assignments. Regression is considered more challenging than classification: The fact that humans perform worse when tasked to choose values from a rating scale lead to comparative annotation methods, including best-worst scaling. This raises the question if large language model-based annotation methods show similar patterns, namely that they perform worse on rating scale annotation tasks than on comparative annotation tasks. To study this, we automate emotion intensity predictions and compare direct rating scale predictions, pairwise comparisons and best-worst scaling. We find that the latter shows the highest reliability. A transformer regressor fine-tuned on these data performs nearly on par with a model trained on the original manual annotations.
• 摘要：标记语料库构成了为新任务或领域创建模型的瓶颈。大型语言模型通过自动语料库标记方法缓解了这个问题，特别是对于分类注释。然而，一些 NLP 任务（例如情绪强度预测）需要文本回归，但还没有针对连续标签分配的自动化注释的工作。回归被认为比分类更具挑战性：事实上，当人类被要求从评级量表中选择值时表现更差，这一事实导致了比较注释方法，包括最佳-最差缩放。这就提出了一个问题：基于大型语言模型的注释方法是否表现出类似的模式，即它们在评级量表注释任务上的表现比在比较注释任务上的表现更差。为了研究这一点，我们自动进行情绪强度预测，并比较直接评分量表预测、成对比较和最佳-最差缩放。我们发现后者显示出最高的可靠性。对这些数据进行微调的变压器回归器的性能几乎与根据原始手动注释训练的模型相当。

Title: DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition

• Authors: Yi-Cheng Wang, Hsin-Wei Wang, Bi-Cheng Yan, Chi-Han Lin, Berlin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17645
• Pdf URL: https://arxiv.org/pdf/2403.17645
• Copy Paste: [[2403.17645]] DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition(https://arxiv.org/abs/2403.17645)
• Keywords: language model
• Abstract: End-to-end automatic speech recognition (E2E ASR) systems often suffer from mistranscription of domain-specific phrases, such as named entities, sometimes leading to catastrophic failures in downstream tasks. A family of fast and lightweight named entity correction (NEC) models for ASR have recently been proposed, which normally build on phonetic-level edit distance algorithms and have shown impressive NEC performance. However, as the named entity (NE) list grows, the problems of phonetic confusion in the NE list are exacerbated; for example, homophone ambiguities increase substantially. In view of this, we proposed a novel Description Augmented Named entity CorrEctoR (dubbed DANCER), which leverages entity descriptions to provide additional information to facilitate mitigation of phonetic confusion for NEC on ASR transcription. To this end, an efficient entity description augmented masked language model (EDA-MLM) comprised of a dense retrieval model is introduced, enabling MLM to adapt swiftly to domain-specific entities for the NEC task. A series of experiments conducted on the AISHELL-1 and Homophone datasets confirm the effectiveness of our modeling approach. DANCER outperforms a strong baseline, the phonetic edit-distance-based NEC model (PED-NEC), by a character error rate (CER) reduction of about 7% relatively on AISHELL-1 for named entities. More notably, when tested on Homophone that contain named entities of high phonetic confusion, DANCER offers a more pronounced CER reduction of 46% relatively over PED-NEC for named entities.
• 摘要：端到端自动语音识别 (E2E ASR) 系统经常遭受特定领域短语（例如命名实体）的错误转录，有时会导致下游任务发生灾难性故障。最近提出了一系列快速、轻量级的 ASR 命名实体校正 (NEC) 模型，这些模型通常建立在语音级编辑距离算法的基础上，并显示出令人印象深刻的 NEC 性能。然而，随着命名实体（NE）列表的增长，NE列表中的语音混乱问题日益加剧；例如，同音异义词的歧义大幅增加。鉴于此，我们提出了一种新颖的描述增强命名实体 CorrEctoR（称为 DANCER），它利用实体描述来提供附加信息，以帮助缓解 NEC 在 ASR 转录上的语音混乱。为此，引入了由密集检索模型组成的高效实体描述增强掩码语言模型（EDA-MLM），使MLM能够快速适应NEC任务的特定领域实体。在 AISHELL-1 和 Homophone 数据集上进行的一系列实验证实了我们建模方法的有效性。 DANCER 的性能优于强大的基线，即基于语音编辑距离的 NEC 模型 (PED-NEC)，相对于命名实体的 AISHELL-1，字符错误率 (CER) 降低了约 7%。更值得注意的是，当对包含高度语音混乱的命名实体的同音字进行测试时，DANCER 相对于命名实体的 PED-NEC 而言，CER 降低了 46%。

Title: Language Models for Text Classification: Is In-Context Learning Enough?

• Authors: Aleksandra Edwards, Jose Camacho-Collados
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17661
• Pdf URL: https://arxiv.org/pdf/2403.17661
• Copy Paste: [[2403.17661]] Language Models for Text Classification: Is In-Context Learning Enough?(https://arxiv.org/abs/2403.17661)
• Keywords: language model, prompt
• Abstract: Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches based on fine-tuning is the ability to understand instructions written in natural language (prompts), which helps them generalise better to different tasks and domains without the need for specific training data. This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances. However, existing research is limited in scale and lacks understanding of how text generation models combined with prompting techniques compare to more established methods for text classification such as fine-tuning masked language models. In this paper, we address this research gap by performing a large-scale evaluation study for 16 text classification datasets covering binary, multiclass, and multilabel problems. In particular, we compare zero- and few-shot approaches of large language models to fine-tuning smaller language models. We also analyse the results by prompt, classification type, domain, and number of labels. In general, the results show how fine-tuning smaller and more efficient language models can still outperform few-shot approaches of larger language models, which have room for improvement when it comes to text classification.
• 摘要：最近的基础语言模型在零样本和少样本设置的许多 NLP 任务中显示出了最先进的性能。与基于微调的更标准方法相比，这些模型的优点是能够理解以自然语言（提示）编写的指令，这有助于它们更好地推广到不同的任务和领域，而无需特定的训练数据。这使得它们适合解决带有有限数量注释实例的领域的文本分类问题。然而，现有的研究规模有限，并且缺乏对文本生成模型与提示技术相结合如何与更成熟的文本分类方法（例如微调掩码语言模型）进行比较的理解。在本文中，我们通过对涵盖二元、多类和多标签问题的 16 个文本分类数据集进行大规模评估研究来解决这一研究空白。特别是，我们将大型语言模型的零样本和少样本方法与微调较小的语言模型进行了比较。我们还按提示、分类类型、域和标签数量分析结果。总的来说，结果表明，微调更小、更高效的语言模型仍然可以优于大型语言模型的小样本方法，而后者在文本分类方面还有改进的空间。

Title: Enhanced Short Text Modeling: Leveraging Large Language Models for Topic Refinement

• Authors: Shuyu Chang, Rui Wang, Peng Ren, Haiping Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17706
• Pdf URL: https://arxiv.org/pdf/2403.17706
• Copy Paste: [[2403.17706]] Enhanced Short Text Modeling: Leveraging Large Language Models for Topic Refinement(https://arxiv.org/abs/2403.17706)
• Keywords: language model, llm, prompt
• Abstract: Crafting effective topic models for brief texts, like tweets and news headlines, is essential for capturing the swift shifts in social dynamics. Traditional topic models, however, often fall short in accurately representing the semantic intricacies of short texts due to their brevity and lack of contextual data. In our study, we harness the advanced capabilities of Large Language Models (LLMs) to introduce a novel approach termed "Topic Refinement". This approach does not directly involve itself in the initial modeling of topics but focuses on improving topics after they have been mined. By employing prompt engineering, we direct LLMs to eliminate off-topic words within a given topic, ensuring that only contextually relevant words are preserved or substituted with ones that fit better semantically. This method emulates human-like scrutiny and improvement of topics, thereby elevating the semantic quality of the topics generated by various models. Our comprehensive evaluation across three unique datasets has shown that our topic refinement approach significantly enhances the semantic coherence of topics.
• 摘要：为推文和新闻标题等简短文本创建有效的主题模型对于捕捉社会动态的快速变化至关重要。然而，由于短文本简短且缺乏上下文数据，传统主题模型通常无法准确表示短文本的复杂语义。在我们的研究中，我们利用大型语言模型 (LLM) 的先进功能引入了一种称为“主题细化”的新颖方法。这种方法并不直接参与主题的初始建模，而是专注于在挖掘主题后对其进行改进。通过采用即时工程，我们指导法学硕士消除给定主题中偏离主题的单词，确保只保留上下文相关的单词或用更适合语义的单词替换。该方法模拟人类对主题的审查和改进，从而提高各种模型生成的主题的语义质量。我们对三个独特数据集的综合评估表明，我们的主题细化方法显着增强了主题的语义一致性。

Title: Continual Few-shot Event Detection via Hierarchical Augmentation Networks

• Authors: Chenlong Zhang, Pengfei Cao, Yubo Chen, Kang Liu, Zhiqiang Zhang, Mengshu Sun, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17733
• Pdf URL: https://arxiv.org/pdf/2403.17733
• Copy Paste: [[2403.17733]] Continual Few-shot Event Detection via Hierarchical Augmentation Networks(https://arxiv.org/abs/2403.17733)
• Keywords: gpt, chat
• Abstract: Traditional continual event detection relies on abundant labeled data for training, which is often impractical to obtain in real-world applications. In this paper, we introduce continual few-shot event detection (CFED), a more commonly encountered scenario when a substantial number of labeled samples are not accessible. The CFED task is challenging as it involves memorizing previous event types and learning new event types with few-shot samples. To mitigate these challenges, we propose a memory-based framework: Hierarchical Augmentation Networks (HANet). To memorize previous event types with limited memory, we incorporate prototypical augmentation into the memory set. For the issue of learning new event types in few-shot scenarios, we propose a contrastive augmentation module for token representations. Despite comparing with previous state-of-the-art methods, we also conduct comparisons with ChatGPT. Experiment results demonstrate that our method significantly outperforms all of these methods in multiple continual few-shot event detection tasks.
• 摘要：传统的连续事件检测依赖于丰富的标记数据进行训练，这在实际应用中通常是不切实际的。在本文中，我们介绍了连续少样本事件检测（CFED），这是当大量标记样本无法访问时更常见的情况。 CFED 任务具有挑战性，因为它涉及记住以前的事件类型并通过少量样本学习新的事件类型。为了缓解这些挑战，我们提出了一个基于内存的框架：分层增强网络（HANet）。为了用有限的记忆来记住以前的事件类型，我们将原型增强纳入记忆集中。对于在少数场景中学习新事件类型的问题，我们提出了一种用于标记表示的对比增强模块。尽管与以前最先进的方法进行比较，我们也与 ChatGPT 进行比较。实验结果表明，我们的方法在多个连续的少样本事件检测任务中显着优于所有这些方法。

Title: Can multiple-choice questions really be useful in detecting the abilities of LLMs?

• Authors: Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, Noa Garcia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17752
• Pdf URL: https://arxiv.org/pdf/2403.17752
• Copy Paste: [[2403.17752]] Can multiple-choice questions really be useful in detecting the abilities of LLMs?(https://arxiv.org/abs/2403.17752)
• Keywords: language model, llm
• Abstract: Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) due to their simplicity and efficiency. However, there are concerns about whether MCQs can truly measure LLM's capabilities, particularly in knowledge-intensive scenarios where long-form generation (LFG) answers are required. The misalignment between the task and the evaluation method demands a thoughtful analysis of MCQ's efficacy, which we undertake in this paper by evaluating nine LLMs on four question-answering (QA) datasets in two languages: Chinese and English. We identify a significant issue: LLMs exhibit an order sensitivity in bilingual MCQs, favoring answers located at specific positions, i.e., the first position. We further quantify the gap between MCQs and long-form generation questions (LFGQs) by comparing their direct outputs, token logits, and embeddings. Our results reveal a relatively low correlation between answers from MCQs and LFGQs for identical questions. Additionally, we propose two methods to quantify the consistency and confidence of LLMs' output, which can be generalized to other QA evaluation benchmarks. Notably, our analysis challenges the idea that the higher the consistency, the greater the accuracy. We also find MCQs to be less reliable than LFGQs in terms of expected calibration error. Finally, the misalignment between MCQs and LFGQs is not only reflected in the evaluation performance but also in the embedding space. Our code and models can be accessed at https://github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs.
• 摘要：多项选择题（MCQ）由于其简单性和高效性而被广泛用于大型语言模型（LLM）的评估。然而，人们担心MCQ是否能够真正衡量LLM的能力，特别是在需要长格式生成（LFG）答案的知识密集型场景中。任务和评估方法之间的不一致需要对 MCQ 的功效进行深思熟虑的分析，我们在本文中通过使用中文和英语两种语言的四个问答（QA）数据集评估九名法学硕士。我们发现了一个重要问题：法学硕士在双语 MCQ 中表现出顺序敏感性，偏向位于特定位置（即第一个位置）的答案。我们通过比较 MCQ 和长格式生成问题 (LFGQ) 的直接输出、token logits 和嵌入，进一步量化它们之间的差距。我们的结果表明，对于相同问题，MCQ 和 LFGQ 的答案之间的相关性相对较低。此外，我们提出了两种方法来量化法学硕士输出的一致性和置信度，这可以推广到其他质量保证评估基准。值得注意的是，我们的分析挑战了一致性越高、准确性越高的观点。我们还发现，就预期校准误差而言，MCQ 的可靠性不如 LFGQ。最后，MCQ 和 LFGQ 之间的不一致不仅体现在评估性能上，还体现在嵌入空间上。我们的代码和模型可以在 https://github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs 访问。

Title: Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

• Authors: Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, Lori Levin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17760
• Pdf URL: https://arxiv.org/pdf/2403.17760
• Copy Paste: [[2403.17760]] Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons(https://arxiv.org/abs/2403.17760)
• Keywords: language model, gpt, llm
• Abstract: In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM's understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don't adequately represent their meaning or capture the lexical properties of phrasal heads.
• 摘要：在本文中，我们做出的贡献可以从两个角度来理解：从 NLP 的角度来看，我们为 NLI 引入了一个具有大量词汇重叠的小型挑战数据集，这最大限度地减少了模型仅根据标记区别来识别蕴涵的可能性，并显示GPT-4 和 Llama 2 因强烈偏见而失败。然后，我们创建进一步具有挑战性的子任务，以努力解释这种失败。从计算语言学的角度来看，我们识别了一组具有三类形容词的结构，这些形容词无法通过表面特征来区分。这使我们能够以各种方式探究 LLM 对这些结构的理解，我们发现它们在多种方面都无法区分它们，这表明它们没有充分地表达它们的含义或捕获短语中心词的词汇属性。

Title: Graph Language Model (GLM): A new graph-based approach to detect social instabilities

• Authors: Wallyson Lemes de Oliveira, Vahid Shamsaddini, Ali Ghofrani, Rahul Singh Inda, Jithendra Sai Veeramaneni, Étienne Voutaz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17816
• Pdf URL: https://arxiv.org/pdf/2403.17816
• Copy Paste: [[2403.17816]] Graph Language Model (GLM): A new graph-based approach to detect social instabilities(https://arxiv.org/abs/2403.17816)
• Keywords: language model
• Abstract: This scientific report presents a novel methodology for the early prediction of important political events using News datasets. The methodology leverages natural language processing, graph theory, clique analysis, and semantic relationships to uncover hidden predictive signals within the data. Initially, we designed a preliminary version of the method and tested it on a few events. This analysis revealed limitations in the initial research phase. We then enhanced the model in two key ways: first, we added a filtration step to only consider politically relevant news before further processing; second, we adjusted the input features to make the alert system more sensitive to significant spikes in the data. After finalizing the improved methodology, we tested it on eleven events including US protests, the Ukraine war, and French protests. Results demonstrate the superiority of our approach compared to baseline methods. Through targeted refinements, our model can now provide earlier and more accurate predictions of major political events based on subtle patterns in news data.
• 摘要：这份科学报告提出了一种使用新闻数据集早期预测重要政治事件的新颖方法。该方法利用自然语言处理、图论、派分析和语义关系来揭示数据中隐藏的预测信号。最初，我们设计了该方法的初步版本，并在一些事件上进行了测试。该分析揭示了初始研究阶段的局限性。然后，我们通过两个关键方式增强了模型：首先，我们添加了过滤步骤，以便在进一步处理之前仅考虑政治相关新闻；其次，我们调整了输入功能，使警报系统对数据中的显着峰值更加敏感。在最终确定改进的方法后，我们在美国抗议、乌克兰战争和法国抗议等十一个事件中对其进行了测试。结果证明了我们的方法与基线方法相比的优越性。通过有针对性的改进，我们的模型现在可以根据新闻数据中的微妙模式对重大政治事件进行更早、更准确的预测。

Title: ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

• Authors: Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.17848
• Pdf URL: https://arxiv.org/pdf/2403.17848
• Copy Paste: [[2403.17848]] ArabicaQA: A Comprehensive Dataset for Arabic Question Answering(https://arxiv.org/abs/2403.17848)
• Keywords: language model, llm
• Abstract: In this paper, we address the significant gap in Arabic natural language processing (NLP) resources by introducing ArabicaQA, the first large-scale dataset for machine reading comprehension and open-domain question answering in Arabic. This comprehensive dataset, consisting of 89,095 answerable and 3,701 unanswerable questions created by crowdworkers to look similar to answerable ones, along with additional labels of open-domain questions marks a crucial advancement in Arabic NLP resources. We also present AraDPR, the first dense passage retrieval model trained on the Arabic Wikipedia corpus, specifically designed to tackle the unique challenges of Arabic text retrieval. Furthermore, our study includes extensive benchmarking of large language models (LLMs) for Arabic question answering, critically evaluating their performance in the Arabic language context. In conclusion, ArabicaQA, AraDPR, and the benchmarking of LLMs in Arabic question answering offer significant advancements in the field of Arabic NLP. The dataset and code are publicly accessible for further research https://github.com/DataScienceUIBK/ArabicaQA.
• 摘要：在本文中，我们通过引入ArabicaQA（第一个用于阿拉伯语机器阅读理解和开放域问答的大型数据集）来解决阿拉伯语自然语言处理（NLP）资源的巨大差距。这个综合数据集由众包工作者创建的 89,095 个可回答问题和 3,701 个不可回答问题组成，看起来与可回答问题类似，以及开放域问题的附加标签，标志着阿拉伯语 NLP 资源的重大进步。我们还推出了 AraDPR，这是第一个在阿拉伯语维基百科语料库上训练的密集段落检索模型，专门用于解决阿拉伯语文本检索的独特挑战。此外，我们的研究还包括对阿拉伯语问答的大型语言模型 (LLM) 进行广泛的基准测试，批判性地评估它们在阿拉伯语环境中的表现。总之，ArabicaQA、AraDPR 和法学硕士在阿拉伯语问答领域的基准测试为阿拉伯语 NLP 领域带来了重大进步。数据集和代码可公开访问以供进一步研究 https://github.com/DataScienceUIBK/ArabicaQA。

Title: Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs

• Authors: David R. Mortensen, Valentina Izrailevitch, Yunze Xiao, Hinrich Schütze, Leonie Weissweiler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17856
• Pdf URL: https://arxiv.org/pdf/2403.17856
• Copy Paste: [[2403.17856]] Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs(https://arxiv.org/abs/2403.17856)
• Keywords: language model, gpt, llm
• Abstract: Lexical-syntactic flexibility, in the form of conversion (or zero-derivation) is a hallmark of English morphology. In conversion, a word with one part of speech is placed in a non-prototypical context, where it is coerced to behave as if it had a different part of speech. However, while this process affects a large part of the English lexicon, little work has been done to establish the degree to which language models capture this type of generalization. This paper reports the first study on the behavior of large language models with reference to conversion. We design a task for testing lexical-syntactic flexibility -- the degree to which models can generalize over words in a construction with a non-prototypical part of speech. This task is situated within a natural language inference paradigm. We test the abilities of five language models -- two proprietary models (GPT-3.5 and GPT-4), three open-source models (Mistral 7B, Falcon 40B, and Llama 2 70B). We find that GPT-4 performs best on the task, followed by GPT-3.5, but that the open source language models are also able to perform it and that the 7B parameter Mistral displays as little difference between its baseline performance on the natural language inference task and the non-prototypical syntactic category task, as the massive GPT-4.
• 摘要：转换（或零派生）形式的词汇句法灵活性是英语形态的标志。在转换过程中，具有一种词性的单词被放置在非原型上下文中，在那里它被迫表现得好像它具有不同的词性一样。然而，虽然这个过程影响了英语词典的很大一部分，但在确定语言模型捕获这种类型的概括的程度方面却几乎没有做任何工作。本文报告了有关转换的大型语言模型行为的首次研究。我们设计了一个测试词汇句法灵活性的任务——模型对具有非原型词性的结构中的单词进行泛化的程度。该任务位于自然语言推理范式内。我们测试了五种语言模型的能力——两种专有模型（GPT-3.5 和 GPT-4）、三种开源模型（Mistral 7B、Falcon 40B 和 Llama 2 70B）。我们发现 GPT-4 在该任务上表现最好，其次是 GPT-3.5，但开源语言模型也能够执行该任务，并且 7B 参数 Mistral 在自然语言推理方面的基线性能之间显示出很小的差异任务和非原型句法类别任务，如大规模 GPT-4。

Title: ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages

• Authors: Bhawna Piryani, Jamshid Mozafari, Adam Jatowt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17859
• Pdf URL: https://arxiv.org/pdf/2403.17859
• Copy Paste: [[2403.17859]] ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages(https://arxiv.org/abs/2403.17859)
• Keywords: language model
• Abstract: Question answering (QA) and Machine Reading Comprehension (MRC) tasks have significantly advanced in recent years due to the rapid development of deep learning techniques and, more recently, large language models. At the same time, many benchmark datasets have become available for QA and MRC tasks. However, most existing large-scale benchmark datasets have been created predominantly using synchronous document collections like Wikipedia or the Web. Archival document collections, such as historical newspapers, contain valuable information from the past that is still not widely used to train large language models. To further contribute to advancing QA and MRC tasks and to overcome the limitation of previous datasets, we introduce ChroniclingAmericaQA, a large-scale dataset with 485K question-answer pairs created based on the historical newspaper collection Chronicling America. Our dataset is constructed from a subset of the Chronicling America newspaper collection spanning 120 years. One of the significant challenges for utilizing digitized historical newspaper collections is the low quality of OCR text. Therefore, to enable realistic testing of QA models, our dataset can be used in three different ways: answering questions from raw and noisy content, answering questions from cleaner, corrected version of the content, as well as answering questions from scanned images of newspaper pages. This and the fact that ChroniclingAmericaQA spans the longest time period among available QA datasets make it quite a unique and useful resource.
• 摘要：近年来，由于深度学习技术和大型语言模型的快速发展，问答（QA）和机器阅读理解（MRC）任务取得了显着进步。与此同时，许多基准数据集已可用于 QA 和 MRC 任务。然而，大多数现有的大规模基准数据集主要是使用维基百科或网络等同步文档集合创建的。档案文档集（例如历史报纸）包含过去的宝贵信息，但尚未广泛用于训练大型语言模型。为了进一步推进 QA 和 MRC 任务并克服以前数据集的限制，我们引入了 ChroniclingAmericaQA，这是一个基于历史报纸集 Chronicling America 创建的包含 485K 问答对的大型数据集。我们的数据集是根据 120 年的 Chronicling America 报纸收藏的子集构建的。利用数字化历史报纸馆藏的重大挑战之一是 OCR 文本质量低下。因此，为了能够对 QA 模型进行实际测试，我们的数据集可以通过三种不同的方式使用：回答原始和嘈杂内容的问题，回答更干净、更正的内容版本的问题，以及回答报纸页面扫描图像的问题。这一事实以及 ChroniclingAmericaQA 跨越可用 QA 数据集中最长的时间段的事实使其成为一个非常独特且有用的资源。

Title: Exploring LLMs as a Source of Targeted Synthetic Textual Data to Minimize High Confidence Misclassifications

• Authors: Philip Lippmann, Matthijs Spaan, Jie Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17860
• Pdf URL: https://arxiv.org/pdf/2403.17860
• Copy Paste: [[2403.17860]] Exploring LLMs as a Source of Targeted Synthetic Textual Data to Minimize High Confidence Misclassifications(https://arxiv.org/abs/2403.17860)
• Keywords: language model, llm
• Abstract: Natural Language Processing (NLP) models optimized for predictive performance often make high confidence errors and suffer from vulnerability to adversarial and out-of-distribution data. Existing work has mainly focused on mitigation of such errors using either humans or an automated approach. In this study, we explore the usage of large language models (LLMs) for data augmentation as a potential solution to the issue of NLP models making wrong predictions with high confidence during classification tasks. We compare the effectiveness of synthetic data generated by LLMs with that of human data obtained via the same procedure. For mitigation, humans or LLMs provide natural language characterizations of high confidence misclassifications to generate synthetic data, which are then used to extend the training set. We conduct an extensive evaluation of our approach on three classification tasks and demonstrate its effectiveness in reducing the number of high confidence misclassifications present in the model, all while maintaining the same level of accuracy. Moreover, we find that the cost gap between humans and LLMs surpasses an order of magnitude, as LLMs attain human-like performance while being more scalable.
• 摘要：针对预测性能进行优化的自然语言处理 (NLP) 模型经常会出现高置信度错误，并且容易受到对抗性数据和分布外数据的影响。现有的工作主要集中在使用人工或自动化方法来减轻此类错误。在本研究中，我们探索使用大型语言模型 (LLM) 进行数据增强，作为解决 NLP 模型在分类任务期间高置信度地做出错误预测问题的潜在解决方案。我们将法学硕士生成的合成数据的有效性与通过相同程序获得的人类数据的有效性进行了比较。为了缓解这种情况，人类或法学硕士提供高置信度错误分类的自然语言特征来生成合成数据，然后将其用于扩展训练集。我们对三个分类任务的方法进行了广泛的评估，并证明了其在减少模型中存在的高置信度错误分类数量方面的有效性，同时保持了相同的准确性水平。此外，我们发现人类和法学硕士之间的成本差距超过了一个数量级，因为法学硕士获得了类似人类的性能，同时更具可扩展性。

Title: The Unreasonable Ineffectiveness of the Deeper Layers

• Authors: Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.17887
• Pdf URL: https://arxiv.org/pdf/2403.17887
• Copy Paste: [[2403.17887]] The Unreasonable Ineffectiveness of the Deeper Layers(https://arxiv.org/abs/2403.17887)
• Keywords: llm
• Abstract: We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that layer pruning methods can complement other PEFT strategies to further reduce computational resources of finetuning on the one hand, and can improve the memory and latency of inference on the other hand. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge.
• 摘要：我们根据经验研究了一种针对流行的开放权重预训练 LLM 系列的简单层修剪策略，发现在删除大部分（最多一半）层之前，不同问答基准上的性能下降最小。为了修剪这些模型，我们通过考虑层之间的相似性来确定要修剪的最佳层块；然后，为了“治愈”损坏，我们进行少量微调。特别是，我们使用参数高效微调 (PEFT) 方法，特别是量化和低阶适配器 (QLoRA)，这样我们的每个实验都可以在单个 A100 GPU 上执行。从实践的角度来看，这些结果表明层剪枝方法可以补充其他 PEFT 策略，一方面进一步减少微调的计算资源，另一方面可以改善推理的内存和延迟。从科学的角度来看，这些 LLM 对删除层的鲁棒性意味着当前的预训练方法没有正确利用网络深层中的参数，或者浅层在存储知识方面发挥着关键作用。