2024-05-13

Title: A Mixture-of-Experts Approach to Few-Shot Task Transfer in Open-Ended Text Worlds

• Authors: Christopher Z. Cui, Xiangyu Peng, Mark O. Riedl
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.06059
• Pdf URL: https://arxiv.org/pdf/2405.06059
• Copy Paste: [[2405.06059]] A Mixture-of-Experts Approach to Few-Shot Task Transfer in Open-Ended Text Worlds(https://arxiv.org/abs/2405.06059)
• Keywords: agent
• Abstract: Open-ended worlds are those in which there are no pre-specified goals or environmental reward signal. As a consequence, an agent must know how to perform a multitude of tasks. However, when a new task is presented to an agent, we expect it to be able to reuse some of what it knows from previous tasks to rapidly learn that new task. We introduce a novel technique whereby policies for different a priori known tasks are combined into a Mixture-of-Experts model with an attention mechanism across a mix of frozen and unfrozen experts. The model learns when to attend to frozen task-specific experts when appropriate and learns new experts to handle novel situations. We work in an open-ended text-based environment in which the agent is tasked with behaving like different types of character roles and must rapidly learn behaviors associated with new character role types. We show that our agent both obtains more rewards in the zero-shot setting, and discovers these rewards with greater sample efficiency in the few-shot learning settings.
• 摘要：开放式世界是指没有预先指定的目标或环境奖励信号的世界。因此，代理必须知道如何执行多项任务。然而，当向智能体呈现新任务时，我们希望它能够重用从以前的任务中学到的一些知识来快速学习新任务。我们引入了一种新技术，将不同先验已知任务的策略组合成专家混合模型，该模型具有跨冻结和未冻结专家混合的注意力机制。该模型学习何时在适当的时候关注冻结的特定任务专家，并学习新的专家来处理新情况。我们在基于文本的开放式环境中工作，其中代理的任务是表现得像不同类型的角色角色，并且必须快速学习与新角色角色类型相关的行为。我们表明，我们的代理既在零样本设置中获得了更多奖励，又在少样本学习设置中以更高的样本效率发现了这些奖励。

Title: HMT: Hierarchical Memory Transformer for Long Context Language Processing

• Authors: Zifan He, Zongyue Qin, Neha Prakriya, Yizhou Sun, Jason Cong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.06067
• Pdf URL: https://arxiv.org/pdf/2405.06067
• Copy Paste: [[2405.06067]] HMT: Hierarchical Memory Transformer for Long Context Language Processing(https://arxiv.org/abs/2405.06067)
• Keywords: language model, llm, long context
• Abstract: Transformer-based large language models (LLM) have been widely used in language processing applications. However, most of them restrict the context window that permits the model to attend to every token in the inputs. Previous works in recurrent models can memorize past tokens to enable unlimited context and maintain effectiveness. However, they have "flat" memory architectures, which have limitations in selecting and filtering information. Since humans are good at learning and self-adjustment, we speculate that imitating brain memory hierarchy is beneficial for model memorization. We propose the Hierarchical Memory Transformer (HMT), a novel framework that enables and improves models' long-context processing ability by imitating human memorization behavior. Leveraging memory-augmented segment-level recurrence, we organize the memory hierarchy by preserving tokens from early input token segments, passing memory embeddings along the sequence, and recalling relevant information from history. Evaluating general language modeling (Wikitext-103, PG-19) and question-answering tasks (PubMedQA), we show that HMT steadily improves the long-context processing ability of context-constrained and long-context models. With an additional 0.5% - 2% of parameters, HMT can easily plug in and augment future LLMs to handle long context effectively. Our code is open-sourced on Github: this https URL.
• 摘要：基于 Transformer 的大语言模型（LLM）已广泛应用于语言处理应用中。然而，它们中的大多数限制了允许模型处理输入中的每个标记的上下文窗口。以前的循环模型工作可以记住过去的标记，以实现无限的上下文并保持有效性。然而，它们具有“扁平”内存架构，在选择和过滤信息方面存在局限性。由于人类善于学习和自我调整，我们推测模仿大脑记忆层次结构有利于模型记忆。我们提出了分层记忆变换器（HMT），这是一种新颖的框架，可以通过模仿人类记忆行为来启用和提高模型的长上下文处理能力。利用记忆增强的段级递归，我们通过保留早期输入标记段中的标记、沿序列传递记忆嵌入以及从历史记录中调用相关信息来组织记忆层次结构。通过评估通用语言模型（Wikitext-103、PG-19）和问答任务（PubMedQA），我们表明 HMT 稳步提高了上下文约束和长上下文模型的长上下文处理能力。通过额外的 0.5% - 2% 参数，HMT 可以轻松插入和增强未来的 LLM，以有效处理长上下文。我们的代码在 Github 上开源：此 https URL。

Title: Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?

• Authors: Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06105
• Pdf URL: https://arxiv.org/pdf/2405.06105
• Copy Paste: [[2405.06105]] Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?(https://arxiv.org/abs/2405.06105)
• Keywords: language model, llm
• Abstract: Recent studies have shown that Large Language Models (LLMs) have the potential to process extremely long text. Many works only evaluate LLMs' long-text processing ability on the language modeling task, with perplexity (PPL) as the evaluation metric. However, in our study, we find that there is no correlation between PPL and LLMs' long-text understanding ability. Besides, PPL may only reflect the model's ability to model local information instead of catching long-range dependency. Therefore, only using PPL to prove the model could process long text is inappropriate. The local focus feature of PPL could also explain some existing phenomena, such as the great extrapolation ability of the position method ALiBi. When evaluating a model's ability in long text, we might pay more attention to PPL's limitation and avoid overly relying on it.
• 摘要：最近的研究表明，大型语言模型（LLM）具有处理极长文本的潜力。许多作品仅评估LLM在语言建模任务上的长文本处理能力，以困惑度（PPL）作为评估指标。然而，在我们的研究中，我们发现PPL和LLM的长文本理解能力之间没有相关性。此外，PPL可能只反映模型对本地信息进行建模的能力，而不是捕捉远程依赖关系。因此，仅仅用PPL来证明模型可以处理长文本是不合适的。 PPL的局部焦点特征也可以解释一些现有现象，例如位置方法ALiBi强大的外推能力。在评估模型在长文本中的能力时，我们可能会更多地关注 PPL 的局限性，并避免过度依赖它。

Title: Reddit-Impacts: A Named Entity Recognition Dataset for Analyzing Clinical and Social Effects of Substance Use Derived from Social Media

• Authors: Yao Ge, Sudeshna Das, Karen O'Connor, Mohammed Ali Al-Garadi, Graciela Gonzalez-Hernandez, Abeed Sarker
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.06145
• Pdf URL: https://arxiv.org/pdf/2405.06145
• Copy Paste: [[2405.06145]] Reddit-Impacts: A Named Entity Recognition Dataset for Analyzing Clinical and Social Effects of Substance Use Derived from Social Media(https://arxiv.org/abs/2405.06145)
• Keywords: gpt
• Abstract: Substance use disorders (SUDs) are a growing concern globally, necessitating enhanced understanding of the problem and its trends through data-driven research. Social media are unique and important sources of information about SUDs, particularly since the data in such sources are often generated by people with lived experiences. In this paper, we introduce Reddit-Impacts, a challenging Named Entity Recognition (NER) dataset curated from subreddits dedicated to discussions on prescription and illicit opioids, as well as medications for opioid use disorder. The dataset specifically concentrates on the lesser-studied, yet critically important, aspects of substance use--its clinical and social impacts. We collected data from chosen subreddits using the publicly available Application Programming Interface for Reddit. We manually annotated text spans representing clinical and social impacts reported by people who also reported personal nonmedical use of substances including but not limited to opioids, stimulants and benzodiazepines. Our objective is to create a resource that can enable the development of systems that can automatically detect clinical and social impacts of substance use from text-based social media data. The successful development of such systems may enable us to better understand how nonmedical use of substances affects individual health and societal dynamics, aiding the development of effective public health strategies. In addition to creating the annotated data set, we applied several machine learning models to establish baseline performances. Specifically, we experimented with transformer models like BERT, and RoBERTa, one few-shot learning model DANN by leveraging the full training dataset, and GPT-3.5 by using one-shot learning, for automatic NER of clinical and social impacts. The dataset has been made available through the 2024 SMM4H shared tasks.
• 摘要：药物滥用障碍（SUD）在全球范围内日益受到关注，需要通过数据驱动的研究来加深对这一问题及其趋势的了解。社交媒体是有关 SUD 的独特而重要的信息来源，特别是因为此类来源中的数据通常是由有生活经验的人生成的。在本文中，我们介绍了 Reddit-Impacts，这是一个具有挑战性的命名实体识别 (NER) 数据集，由 Reddit 子版块精心策划，专门用于讨论处方和非法阿片类药物以及阿片类药物使用障碍的药物。该数据集特别关注物质使用的研究较少但至关重要的方面——其临床和社会影响。我们使用公开的 Reddit 应用程序编程接口从选定的 Reddit 子版块收集数据。我们手动注释了代表临床和社会影响的文本范围，这些影响由那些还报告个人非医疗使用物质（包括但不限于阿片类药物、兴奋剂和苯二氮卓类药物）的人报告。我们的目标是创建一种资源，支持开发能够从基于文本的社交媒体数据自动检测药物使用的临床和社会影响的系统。此类系统的成功开发可以使我们更好地了解物质的非医疗使用如何影响个人健康和社会动态，从而有助于制定有效的公共卫生战略。除了创建带注释的数据集之外，我们还应用了多种机器学习模型来建立基准性能。具体来说，我们试验了 BERT 和 RoBERTa 等 Transformer 模型，通过利用完整的训练数据集进行了几次学习模型 DANN，并通过使用一次性学习进行了 GPT-3.5，以实现临床和社会影响的自动 NER。该数据集已通过 2024 SMM4H 共享任务提供。

Title: A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models

• Authors: Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, Qing Li
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models(https://arxiv.org/abs/)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: As one of the most advanced techniques in AI, Retrieval-Augmented Generation (RAG) techniques can offer reliable and up-to-date external knowledge, providing huge convenience for numerous tasks. Particularly in the era of AI-generated content (AIGC), the powerful capacity of retrieval in RAG in providing additional knowledge enables retrieval-augmented generation to assist existing generative AI in producing high-quality outputs. Recently, large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation, while still facing inherent limitations, such as hallucinations and out-of-date internal knowledge. Given the powerful abilities of RAG in providing the latest and helpful auxiliary information, retrieval-augmented large language models have emerged to harness external and authoritative knowledge bases, rather than solely relying on the model's internal knowledge, to augment the generation quality of LLMs. In this survey, we comprehensively review existing research studies in retrieval-augmented large language models (RA-LLMs), covering three primary technical perspectives: architectures, training strategies, and applications. As the preliminary knowledge, we briefly introduce the foundations and recent advances of LLMs. Then, to illustrate the practical significance of RAG for LLMs, we categorize mainstream relevant work by application areas, detailing specifically the challenges of each and the corresponding capabilities of RA-LLMs. Finally, to deliver deeper insights, we discuss current limitations and several promising directions for future research.
• 摘要：作为人工智能领域最先进的技术之一，检索增强生成（RAG）技术可以提供可靠且最新的外部知识，为众多任务提供巨大便利。特别是在人工智能生成内容（AIGC）时代，RAG 强大的检索能力可以提供额外的知识，使得检索增强生成能够协助现有的生成式人工智能产生高质量的输出。最近，大型语言模型（LLM）在语言理解和生成方面表现出了革命性的能力，但仍然面临着固有的局限性，例如幻觉和过时的内部知识。鉴于RAG在提供最新且有用的辅助信息方面的强大能力，检索增强的大型语言模型已经出现，以利用外部和权威的知识库，而不是仅仅依靠模型的内部知识来提高LLM的生成质量。在本次调查中，我们全面回顾了检索增强大语言模型（RA-LLM）的现有研究，涵盖三个主要技术视角：架构、训练策略和应用。作为预备知识，我们简要介绍了LLM的基础和最新进展。然后，为了说明RAG对LLM的实际意义，我们按应用领域对主流相关工作进行了分类，具体详细说明了每个领域的挑战以及RA-LLM的相应能力。最后，为了提供更深入的见解，我们讨论了当前的局限性和未来研究的几个有希望的方向。

Title: SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora

• Authors: Faisal Qarah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.06239
• Pdf URL: https://arxiv.org/pdf/2405.06239
• Copy Paste: [[2405.06239]] SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora(https://arxiv.org/abs/2405.06239)
• Keywords: language model
• Abstract: In this paper, we introduce SaudiBERT, a monodialect Arabic language model pretrained exclusively on Saudi dialectal text. To demonstrate the model's effectiveness, we compared SaudiBERT with six different multidialect Arabic language models across 11 evaluation datasets, which are divided into two groups: sentiment analysis and text classification. SaudiBERT achieved average F1-scores of 86.15\% and 87.86\% in these groups respectively, significantly outperforming all other comparative models. Additionally, we present two novel Saudi dialectal corpora: the Saudi Tweets Mega Corpus (STMC), which contains over 141 million tweets in Saudi dialect, and the Saudi Forums Corpus (SFC), which includes 15.2 GB of text collected from five Saudi online forums. Both corpora are used in pretraining the proposed model, and they are the largest Saudi dialectal corpora ever reported in the literature. The results confirm the effectiveness of SaudiBERT in understanding and analyzing Arabic text expressed in Saudi dialect, achieving state-of-the-art results in most tasks and surpassing other language models included in the study. SaudiBERT model is publicly available on \url{this https URL}.
• 摘要：在本文中，我们介绍了SaudiBERT，这是一种专门针对沙特方言文本进行预训练的单一方言阿拉伯语言模型。为了证明该模型的有效性，我们在 11 个评估数据集中将沙特伯特与 6 种不同的多方言阿拉伯语言模型进行了比较，这些数据集分为两组：情感分析和文本分类。在这些组中，SaudiBERT 的平均 F1 分数分别为 86.15% 和 87.86%，显着优于所有其他比较模型。此外，我们还提供了两个新颖的沙特方言语料库：沙特推文巨型语料库 (STMC)，其中包含超过 1.41 亿条沙特方言推文，以及沙特论坛语料库 (SFC)，其中包含从五个沙特在线论坛收集的 15.2 GB 文本。这两个语料库都用于预训练所提出的模型，它们是文献中报道的最大的沙特方言语料库。结果证实了SaudiBERT在理解和分析用沙特方言表达的阿拉伯文本方面的有效性，在大多数任务中取得了最先进的结果，并超越了研究中包含的其他语言模型。 SaudiBERT 模型可在 \url{此 https URL} 上公开获取。

Title: Automatic Generation of Model and Data Cards: A Step Towards Responsible AI

• Authors: Jiarui Liu, Wenkai Li, Zhijing Jin, Mona Diab
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06258
• Pdf URL: https://arxiv.org/pdf/2405.06258
• Copy Paste: [[2405.06258]] Automatic Generation of Model and Data Cards: A Step Towards Responsible AI(https://arxiv.org/abs/2405.06258)
• Keywords: language model, llm
• Abstract: In an era of model and data proliferation in machine learning/AI especially marked by the rapid advancement of open-sourced technologies, there arises a critical need for standardized consistent documentation. Our work addresses the information incompleteness in current human-generated model and data cards. We propose an automated generation approach using Large Language Models (LLMs). Our key contributions include the establishment of CardBench, a comprehensive dataset aggregated from over 4.8k model cards and 1.4k data cards, coupled with the development of the CardGen pipeline comprising a two-step retrieval process. Our approach exhibits enhanced completeness, objectivity, and faithfulness in generated model and data cards, a significant step in responsible AI documentation practices ensuring better accountability and traceability.
• 摘要：在机器学习/人工智能领域模型和数据激增的时代，尤其以开源技术的快速发展为标志，迫切需要标准化、一致的文档。我们的工作解决了当前人类生成的模型和数据卡中信息不完整的问题。我们提出了一种使用大型语言模型（LLM）的自动生成方法。我们的主要贡献包括建立 CardBench（一个由超过 4800 个模型卡和 14000 个数据卡聚合而成的综合数据集），以及包含两步检索过程的 CardGen 管道的开发。我们的方法在生成的模型和数据卡中表现出增强的完整性、客观性和忠实性，这是负责任的人工智能文档实践的重要一步，确保更好的问责性和可追溯性。

Title: Pruning as a Domain-specific LLM Extractor

• Authors: Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, Haifeng Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06275
• Pdf URL: https://arxiv.org/pdf/2405.06275
• Copy Paste: [[2405.06275]] Pruning as a Domain-specific LLM Extractor(https://arxiv.org/abs/2405.06275)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have exhibited remarkable proficiency across a wide array of NLP tasks. However, the escalation in model size also engenders substantial deployment costs. While few efforts have explored model pruning techniques to reduce the size of LLMs, they mainly center on general or task-specific weights. This leads to suboptimal performance due to lacking specificity on the target domain or generality on different tasks when applied to domain-specific challenges. This work introduces an innovative unstructured dual-pruning methodology, D-Pruner, for domain-specific compression on LLM. It extracts a compressed, domain-specific, and task-agnostic LLM by identifying LLM weights that are pivotal for general capabilities, like linguistic capability and multi-task solving, and domain-specific knowledge. More specifically, we first assess general weight importance by quantifying the error incurred upon their removal with the help of an open-domain calibration dataset. Then, we utilize this general weight importance to refine the training loss, so that it preserves generality when fitting into a specific domain. Moreover, by efficiently approximating weight importance with the refined training loss on a domain-specific calibration dataset, we obtain a pruned model emphasizing generality and specificity. Our comprehensive experiments across various tasks in healthcare and legal domains show the effectiveness of D-Pruner in domain-specific compression. Our code is available at this https URL.
• 摘要：大型语言模型 (LLM) 在广泛的 NLP 任务中表现出了卓越的熟练程度。然而，模型大小的增加也会产生大量的部署成本。虽然很少有人探索模型修剪技术来减少法学硕士的规模，但它们主要集中在一般或特定任务的权重上。当应用于特定领域的挑战时，由于目标领域缺乏特异性或不同任务的通用性，这会导致性能不佳。这项工作引入了一种创新的非结构化双剪枝方法 D-Pruner，用于 LLM 的特定领域压缩。它通过识别对于一般能力（如语言能力和多任务解决以及特定领域知识）至关重要的法学硕士权重，提取压缩的、特定领域和与任务无关的法学硕士。更具体地说，我们首先通过在开放域校准数据集的帮助下量化去除权重时产生的误差来评估一般权重的重要性。然后，我们利用这个一般权重重要性来细化训练损失，以便它在拟合特定领域时保留通用性。此外，通过在特定领域的校准数据集上使用精细的训练损失来有效地近似权重重要性，我们获得了强调通用性和特异性的修剪模型。我们在医疗保健和法律领域的各种任务中进行的综合实验表明了 D-Pruner 在特定领域压缩中的有效性。我们的代码可以在这个 https URL 上找到。

Title: Correlation Dimension of Natural Language in a Statistical Manifold

• Authors: Xin Du, Kumiko Tanaka-Ishii
• Subjects: cs.CL, cond-mat.stat-mech, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.06321
• Pdf URL: https://arxiv.org/pdf/2405.06321
• Copy Paste: [[2405.06321]] Correlation Dimension of Natural Language in a Statistical Manifold(https://arxiv.org/abs/2405.06321)
• Keywords: language model
• Abstract: The correlation dimension of natural language is measured by applying the Grassberger-Procaccia algorithm to high-dimensional sequences produced by a large-scale language model. This method, previously studied only in a Euclidean space, is reformulated in a statistical manifold via the Fisher-Rao distance. Language exhibits a multifractal, with global self-similarity and a universal dimension around 6.5, which is smaller than those of simple discrete random sequences and larger than that of a Barabási-Albert process. Long memory is the key to producing self-similarity. Our method is applicable to any probabilistic model of real-world discrete sequences, and we show an application to music data.
• 摘要：自然语言的相关维数是通过将 Grassberger-Procaccia 算法应用于大规模语言模型产生的高维序列来测量的。该方法以前仅在欧几里得空间中研究，现在通过 Fisher-Rao 距离在统计流形中重新表述。语言表现出多重分形，具有全局自相似性，通用维数约为 6.5，小于简单离散随机序列的维数，大于 Barabási-Albert 过程的维数。长记忆是产生自相似性的关键。我们的方法适用于现实世界离散序列的任何概率模型，并且我们展示了在音乐数据中的应用。

Title: LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play

• Authors: Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung-yi Lee, Shao-Hua Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.06373
• Pdf URL: https://arxiv.org/pdf/2405.06373
• Copy Paste: [[2405.06373]] LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play(https://arxiv.org/abs/2405.06373)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown exceptional proficiency in natural language processing but often fall short of generating creative and original responses to open-ended questions. To enhance LLM creativity, our key insight is to emulate the human process of inducing collective creativity through engaging discussions with participants from diverse backgrounds and perspectives. To this end, we propose LLM Discussion, a three-phase discussion framework that facilitates vigorous and diverging idea exchanges and ensures convergence to creative answers. Moreover, we adopt a role-playing technique by assigning distinct roles to LLMs to combat the homogeneity of LLMs. We evaluate the efficacy of the proposed framework with the Alternative Uses Test, Similarities Test, Instances Test, and Scientific Creativity Test through both LLM evaluation and human study. Our proposed framework outperforms single-LLM approaches and existing multi-LLM frameworks across various creativity metrics.
• 摘要：大型语言模型 (LLM) 在自然语言处理方面表现出了卓越的能力，但往往无法对开放式问题产生创造性和原创性的回答。为了增强法学硕士的创造力，我们的关键见解是通过与来自不同背景和观点的参与者进行讨论来模仿人类诱导集体创造力的过程。为此，我们提出了法学硕士讨论，这是一个三阶段的讨论框架，可以促进活跃和多样化的思想交流，并确保汇聚到创造性的答案。此外，我们采用角色扮演技术，为法学硕士分配不同的角色，以对抗法学硕士的同质性。我们通过法学硕士评估和人类研究，通过替代用途测试、相似性测试、实例测试和科学创造力测试来评估所提出框架的有效性。我们提出的框架在各种创造力指标上都优于单一法学硕士方法和现有的多法学硕士框架。

Title: Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL

• Authors: Ning Cheng, Zhaohui Yan, Ziming Wang, Zhijie Li, Jiaming Yu, Zilong Zheng, Kewei Tu, Jinan Xu, Wenjuan Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06410
• Pdf URL: https://arxiv.org/pdf/2405.06410
• Copy Paste: [[2405.06410]] Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL(https://arxiv.org/abs/2405.06410)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) play a crucial role in capturing structured semantics to enhance language understanding, improve interpretability, and reduce bias. Nevertheless, an ongoing controversy exists over the extent to which LLMs can grasp structured semantics. To assess this, we propose using Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to extract structured semantics. In our assessment, we employ the prompting approach, which leads to the creation of our few-shot SRL parser, called PromptSRL. PromptSRL enables LLMs to map natural languages to explicit semantic structures, which provides an interpretable window into the properties of LLMs. We find interesting potential: LLMs can indeed capture semantic structures, and scaling-up doesn't always mirror potential. Additionally, limitations of LLMs are observed in C-arguments, etc. Lastly, we are surprised to discover that significant overlap in the errors is made by both LLMs and untrained humans, accounting for almost 30% of all errors.
• 摘要：大型语言模型 (LLM) 在捕获结构化语义以增强语言理解、提高可解释性和减少偏差方面发挥着至关重要的作用。尽管如此，关于法学硕士能够在多大程度上掌握结构化语义仍然存在争议。为了评估这一点，我们建议使用语义角色标签（SRL）作为探索法学硕士提取结构化语义的能力的基本任务。在我们的评估中，我们采用了提示方法，这导致了我们的小样本 SRL 解析器的创建，称为 PromptSRL。 PromptSRL 使法学硕士能够将自然语言映射到显式语义结构，这为了解法学硕士的属性提供了一个可解释的窗口。我们发现了有趣的潜力：法学硕士确实可以捕获语义结构，而扩大规模并不总是反映潜力。此外，LLM 的局限性还体现在 C 参数等方面。最后，我们惊讶地发现，LLM 和未经训练的人员都犯了明显的错误，几乎占所有错误的 30%。

Title: Can Large Language Models Replicate ITS Feedback on Open-Ended Math Questions?

• Authors: Hunter McNichols, Jaewook Lee, Stephen Fancsali, Steve Ritter, Andrew Lan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06414
• Pdf URL: https://arxiv.org/pdf/2405.06414
• Copy Paste: [[2405.06414]] Can Large Language Models Replicate ITS Feedback on Open-Ended Math Questions?(https://arxiv.org/abs/2405.06414)
• Keywords: language model, llm
• Abstract: Intelligent Tutoring Systems (ITSs) often contain an automated feedback component, which provides a predefined feedback message to students when they detect a predefined error. To such a feedback component, we often resort to template-based approaches. These approaches require significant effort from human experts to detect a limited number of possible student errors and provide corresponding feedback. This limitation is exemplified in open-ended math questions, where there can be a large number of different incorrect errors. In our work, we examine the capabilities of large language models (LLMs) to generate feedback for open-ended math questions, similar to that of an established ITS that uses a template-based approach. We fine-tune both open-source and proprietary LLMs on real student responses and corresponding ITS-provided feedback. We measure the quality of the generated feedback using text similarity metrics. We find that open-source and proprietary models both show promise in replicating the feedback they see during training, but do not generalize well to previously unseen student errors. These results suggest that despite being able to learn the formatting of feedback, LLMs are not able to fully understand mathematical errors made by students.
• 摘要：智能辅导系统 (ITS) 通常包含一个自动反馈组件，当学生检测到预定义错误时，该组件会向学生提供预定义的反馈消息。对于这样的反馈组件，我们经常采用基于模板的方法。这些方法需要人类专家付出巨大的努力来检测有限数量的可能的学生错误并提供相应的反馈。这种限制在开放式数学问题中得到了体现，其中可能存在大量不同的不正确错误。在我们的工作中，我们研究了大型语言模型 (LLM) 为开放式数学问题生成反馈的能力，类似于使用基于模板的方法的已建立的 ITS。我们根据学生的真实反应和 ITS 提供的相应反馈对开源和专有法学硕士进行微调。我们使用文本相似性指标来衡量生成的反馈的质量。我们发现开源模型和专有模型都有望复制他们在训练期间看到的反馈，但不能很好地推广到以前未见过的学生错误。这些结果表明，尽管法学硕士能够学习反馈的格式，但他们无法完全理解学生所犯的数学错误。

Title: Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation

• Authors: JoonHo Lee, Jae Oh Woo, Juree Seok, Parisa Hassanzadeh, Wooseok Jang, JuYoun Son, Sima Didari, Baruch Gutow, Heng Hao, Hankyu Moon, Wenjun Hu, Yeong-Dae Kwon, Taehee Lee, Seungjai Min
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.06424
• Pdf URL: https://arxiv.org/pdf/2405.06424
• Copy Paste: [[2405.06424]] Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation(https://arxiv.org/abs/2405.06424)
• Keywords: language model
• Abstract: Assessing response quality to instructions in language models is vital but challenging due to the complexity of human language across different contexts. This complexity often results in ambiguous or inconsistent interpretations, making accurate assessment difficult. To address this issue, we propose a novel Uncertainty-aware Reward Model (URM) that introduces a robust uncertainty estimation for the quality of paired responses based on Bayesian approximation. Trained with preference datasets, our uncertainty-enabled proxy not only scores rewards for responses but also evaluates their inherent uncertainty. Empirical results demonstrate significant benefits of incorporating the proposed proxy into language model training. Our method boosts the instruction following capability of language models by refining data curation for training and improving policy optimization objectives, thereby surpassing existing methods by a large margin on benchmarks such as Vicuna and MT-bench. These findings highlight that our proposed approach substantially advances language model training and paves a new way of harnessing uncertainty within language models.
• 摘要：由于人类语言在不同上下文中的复杂性，评估对语言模型中的指令的响应质量至关重要，但也具有挑战性。这种复杂性常常会导致含糊或不一致的解释，从而使准确的评估变得困难。为了解决这个问题，我们提出了一种新颖的不确定性感知奖励模型（URM），该模型引入了基于贝叶斯近似的配对响应质量的鲁棒不确定性估计。经过偏好数据集的训练，我们的不确定性代理不仅可以对响应进行奖励，还可以评估其固有的不确定性。实证结果表明，将所提出的代理纳入语言模型训练具有显着的好处。我们的方法通过细化训练数据管理和改进策略优化目标来提高语言模型的指令跟踪能力，从而在 Vicuna 和 MT-bench 等基准上大幅超越现有方法。这些发现强调，我们提出的方法极大地推进了语言模型训练，并为利用语言模型中的不确定性铺平了新的途径。

Title: E2TP: Element to Tuple Prompting Improves Aspect Sentiment Tuple Prediction

• Authors: Mohammad Ghiasvand Mohammadkhani, Niloofar Ranjbar, Saeedeh Momtazi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06454
• Pdf URL: https://arxiv.org/pdf/2405.06454
• Copy Paste: [[2405.06454]] E2TP: Element to Tuple Prompting Improves Aspect Sentiment Tuple Prediction(https://arxiv.org/abs/2405.06454)
• Keywords: prompt
• Abstract: Generative approaches have significantly influenced Aspect-Based Sentiment Analysis (ABSA), garnering considerable attention. However, existing studies often predict target text components monolithically, neglecting the benefits of utilizing single elements for tuple prediction. In this paper, we introduce Element to Tuple Prompting (E2TP), employing a two-step architecture. The former step focuses on predicting single elements, while the latter step completes the process by mapping these predicted elements to their corresponding tuples. E2TP is inspired by human problem-solving, breaking down tasks into manageable parts, using the first step's output as a guide in the second step. Within this strategy, three types of paradigms, namely E2TP($diet$), E2TP($f_1$), and E2TP($f_2$), are designed to facilitate the training process. Beyond in-domain task-specific experiments, our paper addresses cross-domain scenarios, demonstrating the effectiveness and generalizability of the approach. By conducting a comprehensive analysis on various benchmarks, we show that E2TP achieves new state-of-the-art results in nearly all cases.
• 摘要：生成方法对基于方面的情感分析（ABSA）产生了重大影响，引起了相当多的关注。然而，现有的研究通常整体预测目标文本组件，忽略了利用单个元素进行元组预测的好处。在本文中，我们引入了元素到元组提示（E2TP），采用两步架构。前一步侧重于预测单个元素，而后一步通过将这些预测元素映射到其相应的元组来完成该过程。 E2TP 的灵感来自人类解决问题的过程，将任务分解为可管理的部分，并使用第一步的输出作为第二步的指导。在该策略中，设计了三种类型的范例，即 E2TP($diet$)、E2TP($f_1$) 和 E2TP($f_2$) 来促进培训过程。除了域内特定任务的实验之外，我们的论文还解决了跨域场景，证明了该方法的有效性和通用性。通过对各种基准进行全面分析，我们表明 E2TP 在几乎所有情况下都取得了新的最先进的结果。

Title: Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts

• Authors: Wenyu Huang, Guancheng Zhou, Mirella Lapata, Pavlos Vougiouklis, Sebastien Montella, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06524
• Pdf URL: https://arxiv.org/pdf/2405.06524
• Copy Paste: [[2405.06524]] Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts(https://arxiv.org/abs/2405.06524)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Although Large Language Models (LLMs) are effective in performing various NLP tasks, they still struggle to handle tasks that require extensive, real-world knowledge, especially when dealing with long-tail facts (facts related to long-tail entities). This limitation highlights the need to supplement LLMs with non-parametric knowledge. To address this issue, we analysed the effects of different types of non-parametric knowledge, including textual passage and knowledge graphs (KGs). Since LLMs have probably seen the majority of factual question-answering datasets already, to facilitate our analysis, we proposed a fully automatic pipeline for creating a benchmark that requires knowledge of long-tail facts for answering the involved questions. Using this pipeline, we introduce the LTGen benchmark. We evaluate state-of-the-art LLMs in different knowledge settings using the proposed benchmark. Our experiments show that LLMs alone struggle with answering these questions, especially when the long-tail level is high or rich knowledge is required. Nonetheless, the performance of the same models improved significantly when they were prompted with non-parametric knowledge. We observed that, in most cases, prompting LLMs with KG triples surpasses passage-based prompting using a state-of-the-art retriever. In addition, while prompting LLMs with both KG triples and documents does not consistently improve knowledge coverage, it can dramatically reduce hallucinations in the generated content.
• 摘要：尽管大型语言模型（LLM）可以有效地执行各种 NLP 任务，但它们仍然难以处理需要广泛的现实世界知识的任务，特别是在处理长尾事实（与长尾实体相关的事实）时。这一限制凸显了法学硕士需要用非参数知识来补充。为了解决这个问题，我们分析了不同类型的非参数知识的影响，包括文本段落和知识图（KG）。由于法学硕士可能已经看到了大多数事实问答数据集，为了便于我们的分析，我们提出了一个全自动管道来创建一个基准，该基准需要了解长尾事实来回答所涉及的问题。使用这个管道，我们引入了 LTGen 基准测试。我们使用建议的基准评估不同知识环境中最先进的法学硕士。我们的实验表明，仅法学硕士很难回答这些问题，尤其是当长尾水平较高或需要丰富的知识时。尽管如此，当使用非参数知识提示时，相同模型的性能显着提高。我们观察到，在大多数情况下，用知识图谱三元组提示法学硕士优于使用最先进的检索器基于段落的提示。此外，虽然用知识图谱三元组和文档提示法学硕士并不能持续提高知识覆盖率，但它可以显着减少生成内容中的幻觉。

Title: Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrieval

• Authors: Mengjia Niu, Hao Li, Jie Shi, Hamed Haddadi, Fan Mo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.06545
• Pdf URL: https://arxiv.org/pdf/2405.06545
• Copy Paste: [[2405.06545]] Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrieval(https://arxiv.org/abs/2405.06545)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across various domains, although their susceptibility to hallucination poses significant challenges for their deployment in critical areas such as healthcare. To address this issue, retrieving relevant facts from knowledge graphs (KGs) is considered a promising method. Existing KG-augmented approaches tend to be resource-intensive, requiring multiple rounds of retrieval and verification for each factoid, which impedes their application in real-world scenarios. In this study, we propose Self-Refinement-Enhanced Knowledge Graph Retrieval (Re-KGR) to augment the factuality of LLMs' responses with less retrieval efforts in the medical field. Our approach leverages the attribution of next-token predictive probability distributions across different tokens, and various model layers to primarily identify tokens with a high potential for hallucination, reducing verification rounds by refining knowledge triples associated with these tokens. Moreover, we rectify inaccurate content using retrieved knowledge in the post-processing stage, which improves the truthfulness of generated responses. Experimental results on a medical dataset demonstrate that our approach can enhance the factual capability of LLMs across various foundational models as evidenced by the highest scores on truthfulness.
• 摘要：大型语言模型（LLM）在各个领域都表现出了卓越的能力，尽管它们对幻觉的敏感性对其在医疗保健等关键领域的部署构成了重大挑战。为了解决这个问题，从知识图（KG）中检索相关事实被认为是一种有前途的方法。现有的知识图谱增强方法往往是资源密集型的，需要对每个事实进行多轮检索和验证，这阻碍了它们在现实场景中的应用。在这项研究中，我们提出了自我完善增强型知识图检索（Re-KGR），以减少医学领域的检索工作，从而增强法学硕士回答的真实性。我们的方法利用不同代币和各种模型层之间的下一个代币预测概率分布的归因来主要识别具有较高幻觉潜力的代币，通过细化与这些代币相关的知识三元组来减少验证轮次。此外，我们在后处理阶段使用检索到的知识来纠正不准确的内容，从而提高了生成响应的真实性。医学数据集上的实验结果表明，我们的方法可以增强法学硕士跨各种基础模型的事实能力，真实性最高分就证明了这一点。

Title: What Can Natural Language Processing Do for Peer Review?

• Authors: Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06563
• Pdf URL: https://arxiv.org/pdf/2405.06563
• Copy Paste: [[2405.06563]] What Can Natural Language Processing Do for Peer Review?(https://arxiv.org/abs/2405.06563)
• Keywords: language model, llm
• Abstract: The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.
• 摘要：每年发表的科学文章数量都在快速增长。对它们进行质量控制对于科学家来说至关重要，最终对于公众利益也至关重要。在现代科学中，这个过程主要委托给同行评审——这是一种分布式程序，其中每份提交的内容都由该领域的几位独立专家进行评估。同行评审被广泛使用，但它很困难、耗时且容易出错。由于同行评审涉及的工件（手稿、评论、讨论）主要是基于文本的，因此自然语言处理在改进评审方面具有巨大的潜力。随着大型语言模型（LLM）的出现使得 NLP 能够协助许多新任务，关于机器辅助同行评审的讨论正在加快步伐。然而，到底哪里需要帮助，NLP 可以在哪里提供帮助，以及它应该站在哪里？我们论文的目标是为未来 NLP 同行评审协助工作奠定基础。我们将同行评审作为一个一般过程进行讨论，以人工智能会议上的评审为例。我们详细介绍了从稿件提交到准备上镜的修订过程的每一步，并讨论了 NLP 协助的相关挑战和机遇，并通过现有工作进行了说明。然后，我们转向整个 NLP 同行评审的重大挑战，包括数据获取和许可、操作化和实验以及道德问题。为了帮助巩固社区的努力，我们创建了一个配套存储库，用于聚合与同行评审相关的关键数据集。最后，我们向科学界、NLP 和 AI 研究人员、政策制定者和资助机构发出详细的行动呼吁，帮助推动 NLP 研究的同行评审。我们希望我们的工作将有助于为人工智能时代、NLP 社区内外的机器辅助科学质量控制研究制定议程。

Title: Linearizing Large Language Models

• Authors: Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, Thomas Kollar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.06640
• Pdf URL: https://arxiv.org/pdf/2405.06640
• Copy Paste: [[2405.06640]] Linearizing Large Language Models(https://arxiv.org/abs/2405.06640)
• Keywords: language model, llm
• Abstract: Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers inference cost. However, their original formulation suffers from poor scaling and underperforms compute-matched transformers. Recent linear models such as RWKV and Mamba have attempted to address these shortcomings by proposing novel time-mixing and gating architectures, but pre-training large language models requires significant data and compute investments. Thus, the search for subquadratic architectures is limited by the availability of compute and quality pre-training datasets. As a cost-effective alternative to pre-training linear transformers, we propose Scalable UPtraining for Recurrent Attention (SUPRA). We present a method to uptrain existing large pre-trained transformers into Recurrent Neural Networks (RNNs) with a modest compute budget. This allows us to leverage the strong pre-training data and performance of existing transformer LLMs, while requiring 5% of the training cost. We find that our linearization technique leads to competitive performance on standard benchmarks, but we identify persistent in-context learning and long-context modeling shortfalls for even the largest linear models. Our code and models can be found at this https URL.
• 摘要：线性变换器已成为 softmax 注意力的次二次时间替代品，并且由于其固定大小的循环状态降低了推理成本而引起了极大的兴趣。然而，他们的原始公式存在扩展性差且性能低于计算匹配变压器的问题。最近的线性模型（例如 RWKV 和 Mamba）试图通过提出新颖的时间混合和门控架构来解决这些缺点，但预训练大型语言模型需要大量的数据和计算投资。因此，对次二次架构的搜索受到计算和质量预训练数据集的可用性的限制。作为预训练线性变压器的一种经济高效的替代方案，我们提出了可扩展的循环注意力 UPtraining (SUPRA)。我们提出了一种方法，以适度的计算预算将现有的大型预训练 Transformer 升级为循环神经网络（RNN）。这使我们能够利用现有 Transformer LLM 强大的预训练数据和性能，同时需要 5% 的培训成本。我们发现，我们的线性化技术可以在标准基准上带来具有竞争力的性能，但我们也发现，即使是最大的线性模型，持续的上下文学习和长上下文建模也存在缺陷。我们的代码和模型可以在此 https URL 中找到。