2024-05-09

Title: PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models

• Authors: Arpit Aggarwal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.04585
• Pdf URL: https://arxiv.org/pdf/2405.04585
• Copy Paste: [[2405.04585]] PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models(https://arxiv.org/abs/2405.04585)
• Keywords: language model
• Abstract: There are several improvements proposed over the baseline Absolute Positional Encoding (APE) method used in original transformer. In this study, we aim to investigate the implications of inadequately representing positional encoding in higher dimensions on crucial aspects of the attention mechanism, the model's capacity to learn relative positional information, and the convergence of models, all stemming from the choice of sinusoidal basis functions. Through a combination of theoretical insights and empirical analyses, we elucidate how these challenges extend beyond APEs and may adversely affect the performance of Relative Positional Encoding (RPE) methods, such as Rotatory Positional Encoding (RoPE). Subsequently, we introduce an innovative solution termed Orthogonal Polynomial Based Positional Encoding (PoPE) to address some of the limitations associated with existing methods. The PoPE method encodes positional information by leveraging Orthogonal Legendre polynomials. Legendre polynomials as basis functions offers several desirable properties for positional encoding, including improved correlation structure, non-periodicity, orthogonality, and distinct functional forms among polynomials of varying orders. Our experimental findings demonstrate that transformer models incorporating PoPE outperform baseline transformer models on the $Multi30k$ English-to-German translation task, thus establishing a new performance benchmark. Furthermore, PoPE-based transformers exhibit significantly accelerated convergence rates. Additionally, we will present novel theoretical perspectives on position encoding based on the superior performance of PoPE.
• 摘要：相对于原始 Transformer 中使用的基线绝对位置编码 (APE) 方法，提出了一些改进。在本研究中，我们的目的是研究在更高维度上不充分表示位置编码对注意机制、模型学习相对位置信息的能力以及模型收敛的关键方面的影响，所有这些都源于正弦基函数的选择。通过理论见解和实证分析的结合，我们阐明了这些挑战如何超出 APE 范围，并可能对相对位置编码 (RPE) 方法（例如旋转位置编码 (RoPE)）的性能产生不利影响。随后，我们引入了一种称为基于正交多项式的位置编码（PoPE）的创新解决方案，以解决与现有方法相关的一些限制。 PoPE 方法利用正交勒让德多项式对位置信息进行编码。作为基函数的勒让德多项式为位置编码提供了几个理想的属性，包括改进的相关结构、非周期性、正交性以及不同阶多项式之间的不同函数形式。我们的实验结果表明，在 $Multi30k$ 英德翻译任务中，包含 PoPE 的 Transformer 模型优于基线 Transformer 模型，从而建立了新的性能基准。此外，基于 PoPE 的变压器表现出显着加快的收敛速度。此外，我们将基于 PoPE 的优越性能提出关于位置编码的新颖理论观点。

Title: Language Modeling Using Tensor Trains

• Authors: Zhan Su, Yuqin Zhou, Fengran Mo, Jakob Grue Simonsen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2405.04590
• Pdf URL: https://arxiv.org/pdf/2405.04590
• Copy Paste: [[2405.04590]] Language Modeling Using Tensor Trains(https://arxiv.org/abs/2405.04590)
• Keywords: language model
• Abstract: We propose a novel tensor network language model based on the simplest tensor network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM). TTLM represents sentences in an exponential space constructed by the tensor product of words, but computing the probabilities of sentences in a low-dimensional fashion. We demonstrate that the architectures of Second-order RNNs, Recurrent Arithmetic Circuits (RACs), and Multiplicative Integration RNNs are, essentially, special cases of TTLM. Experimental evaluations on real language modeling tasks show that the proposed variants of TTLM (i.e., TTLM-Large and TTLM-Tiny) outperform the vanilla Recurrent Neural Networks (RNNs) with low-scale of hidden units. (The code is available at this https URL.)
• 摘要：我们提出了一种基于最简单的张量网络（即张量序列）的新型张量网络语言模型，称为“张量序列语言模型”（TTLM）。 TTLM 在由单词张量积构建的指数空间中表示句子，但以低维方式计算句子的概率。我们证明二阶 RNN、循环算术电路 (RAC) 和乘法积分 RNN 的架构本质上是 TTLM 的特殊情况。对真实语言建模任务的实验评估表明，所提出的 TTLM 变体（即 TTLM-Large 和 TTLM-Tiny）优于具有低规模隐藏单元的普通循环神经网络（RNN）。 （该代码可从此 https URL 获取。）

Title: Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense

• Authors: Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, Rada Mihalcea
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.04655
• Pdf URL: https://arxiv.org/pdf/2405.04655
• Copy Paste: [[2405.04655]] Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense(https://arxiv.org/abs/2405.04655)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs' general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks. Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally aware language models.
• 摘要：大型语言模型 (LLM) 通过大量基准评估展示了丰富的常识性理解。然而，他们对文化常识的理解在很大程度上仍未得到检验。在本文中，我们在文化常识任务的背景下对几位最先进的法学硕士的能力和局限性进行了全面检查。使用几个一般和文化常识基准，我们发现（1）法学硕士在对不同文化的特定文化常识知识进行测试时，表现存在显着差异； （2）法学硕士的一般常识能力受文化背景的影响； (3) 用于查询法学硕士的语言会影响其在文化相关任务上的表现。我们的研究指出了法学硕士文化理解中固有的偏见，并提供了有助于开发文化意识语言模型的见解。

Title: Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

• Authors: Emre Can Acikgoz, Mete Erdogan, Deniz Yuret
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.04685
• Pdf URL: https://arxiv.org/pdf/2405.04685
• Copy Paste: [[2405.04685]] Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking(https://arxiv.org/abs/2405.04685)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are becoming crucial across various fields, emphasizing the urgency for high-quality models in underrepresented languages. This study explores the unique challenges faced by low-resource languages, such as data scarcity, model selection, evaluation, and computational limitations, with a special focus on Turkish. We conduct an in-depth analysis to evaluate the impact of training strategies, model choices, and data availability on the performance of LLMs designed for underrepresented languages. Our approach includes two methodologies: (i) adapting existing LLMs originally pretrained in English to understand Turkish, and (ii) developing a model from the ground up using Turkish pretraining data, both supplemented with supervised fine-tuning on a novel Turkish instruction-tuning dataset aimed at enhancing reasoning capabilities. The relative performance of these methods is evaluated through the creation of a new leaderboard for Turkish LLMs, featuring benchmarks that assess different reasoning and knowledge skills. Furthermore, we conducted experiments on data and model scaling, both during pretraining and fine-tuning, simultaneously emphasizing the capacity for knowledge transfer across languages and addressing the challenges of catastrophic forgetting encountered during fine-tuning on a different language. Our goal is to offer a detailed guide for advancing the LLM framework in low-resource linguistic contexts, thereby making natural language processing (NLP) benefits more globally accessible.
• 摘要：大型语言模型 (LLM) 在各个领域变得至关重要，这凸显了在代表性不足的语言中建立高质量模型的紧迫性。本研究探讨了低资源语言面临的独特挑战，例如数据稀缺、模型选择、评估和计算限制，特别关注土耳其语。我们进行深入分析，以评估培训策略、模型选择和数据可用性对针对代表性不足的语言设计的法学硕士表现的影响。我们的方法包括两种方法：（i）调整最初用英语预训练的现有法学硕士以理解土耳其语，以及（ii）使用土耳其语预训练数据从头开始开发模型，两者都辅以新颖的土耳其语指令调整的监督微调旨在增强推理能力的数据集。这些方法的相对表现是通过为土耳其法学硕士创建一个新的排行榜来评估的，该排行榜以评估不同推理和知识技能的基准为特色。此外，我们在预训练和微调过程中进行了数据和模型扩展实验，同时强调跨语言知识转移的能力，并解决不同语言微调过程中遇到的灾难性遗忘的挑战。我们的目标是为在资源匮乏的语言环境中推进法学硕士框架提供详细的指南，从而使自然语言处理（NLP）的优势在全球范围内更容易获得。

Title: BiasKG: Adversarial Knowledge Graphs to Induce Bias in Large Language Models

• Authors: Chu Fei Luo, Ahmad Ghawanmeh, Xiaodan Zhu, Faiza Khan Khattak
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.04756
• Pdf URL: https://arxiv.org/pdf/2405.04756
• Copy Paste: [[2405.04756]] BiasKG: Adversarial Knowledge Graphs to Induce Bias in Large Language Models(https://arxiv.org/abs/2405.04756)
• Keywords: language model, llm
• Abstract: Modern large language models (LLMs) have a significant amount of world knowledge, which enables strong performance in commonsense reasoning and knowledge-intensive tasks when harnessed properly. The language model can also learn social biases, which has a significant potential for societal harm. There have been many mitigation strategies proposed for LLM safety, but it is unclear how effective they are for eliminating social biases. In this work, we propose a new methodology for attacking language models with knowledge graph augmented generation. We refactor natural language stereotypes into a knowledge graph, and use adversarial attacking strategies to induce biased responses from several open- and closed-source language models. We find our method increases bias in all models, even those trained with safety guardrails. This demonstrates the need for further research in AI safety, and further work in this new adversarial space.
• 摘要：现代大语言模型 (LLM) 拥有大量的世界知识，如果利用得当，可以在常识推理和知识密集型任务中表现出色。语言模型还可以学习社会偏见，这很有可能造成社会危害。针对法学硕士的安全性，已经提出了许多缓解策略，但尚不清楚它们对于消除社会偏见的效果如何。在这项工作中，我们提出了一种利用知识图增强生成来攻击语言模型的新方法。我们将自然语言刻板印象重构为知识图谱，并使用对抗性攻击策略来诱导几种开源和闭源语言模型的有偏见的反应。我们发现我们的方法增加了所有模型的偏差，甚至是那些接受过安全护栏训练的模型。这表明需要进一步研究人工智能安全，并在这个新的对抗空间中开展进一步的工作。

Title: Empathy Through Multimodality in Conversational Interfaces

• Authors: Mahyar Abbasian, Iman Azimi, Mohammad Feli, Amir M. Rahmani, Ramesh Jain
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.04777
• Pdf URL: https://arxiv.org/pdf/2405.04777
• Copy Paste: [[2405.04777]] Empathy Through Multimodality in Conversational Interfaces(https://arxiv.org/abs/2405.04777)
• Keywords: language model, llm, prompt, agent
• Abstract: Agents represent one of the most emerging applications of Large Language Models (LLMs) and Generative AI, with their effectiveness hinging on multimodal capabilities to navigate complex user environments. Conversational Health Agents (CHAs), a prime example of this, are redefining healthcare by offering nuanced support that transcends textual analysis to incorporate emotional intelligence. This paper introduces an LLM-based CHA engineered for rich, multimodal dialogue-especially in the realm of mental health support. It adeptly interprets and responds to users' emotional states by analyzing multimodal cues, thus delivering contextually aware and empathetically resonant verbal responses. Our implementation leverages the versatile openCHA framework, and our comprehensive evaluation involves neutral prompts expressed in diverse emotional tones: sadness, anger, and joy. We evaluate the consistency and repeatability of the planning capability of the proposed CHA. Furthermore, human evaluators critique the CHA's empathic delivery, with findings revealing a striking concordance between the CHA's outputs and evaluators' assessments. These results affirm the indispensable role of vocal (soon multimodal) emotion recognition in strengthening the empathetic connection built by CHAs, cementing their place at the forefront of interactive, compassionate digital health solutions.
• 摘要：代理是大型语言模型 (LLM) 和生成式 AI 最新兴的应用之一，其有效性取决于驾驭复杂用户环境的多模态能力。对话式健康代理 (CHA) 就是一个典型的例子，它通过提供超越文本分析、融入情商的细致入微的支持，重新定义了医疗保健。本文介绍了一种基于 LLM 的 CHA，专为丰富的多模态对话而设计，尤其是在心理健康支持领域。它通过分析多模态线索，熟练地解释和响应用户的情绪状态，从而提供情境感知和共情共鸣的口头回应。我们的实施利用了多功能的 openCHA 框架，我们的综合评估涉及以不同情绪基调表达的中性提示：悲伤、愤怒和喜悦。我们评估了拟议 CHA 的规划能力的一致性和可重复性。此外，人类评估员对 CHA 的共情交付进行了批评，结果显示 CHA 的输出与评估员的评估之间存在惊人的一致性。这些结果肯定了声音（即将推出的多模式）情绪识别在加强 CHA 建立的共情联系方面不可或缺的作用，巩固了他们在交互式、富有同情心的数字健康解决方案领域的领先地位。

Title: CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization

• Authors: Zheyan Qu, Lu Yin, Zitong Yu, Wenbo Wang, Xing zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.04781
• Pdf URL: https://arxiv.org/pdf/2405.04781
• Copy Paste: [[2405.04781]] CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization(https://arxiv.org/abs/2405.04781)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large language models (LLMs) have demonstrated astonishing capabilities in natural language processing (NLP) tasks, sparking interest in their application to professional domains with higher specialized requirements. However, restricted access to closed-source LLMs via APIs and the difficulty in collecting massive high-quality datasets pose obstacles to the development of large language models in education fields of various courses. Given these challenges, we propose CourseGPT-zh, a course-oriented education LLM that supports customization and low-cost deployment. To address the comprehensiveness and diversity requirements of course-specific corpora, we design a high-quality question-answering corpus distillation framework incorporating prompt optimization, which effectively mines textbook knowledge and enhances its diversity. Moreover, considering the alignment of LLM responses with user needs, a novel method for discrete prompt optimization based on LLM-as-Judge is introduced. During optimization, this framework leverages the LLM's ability to reflect on and exploit error feedback and patterns, allowing for prompts that meet user needs and preferences while saving response length. Lastly, we obtain CourseGPT-zh based on the open-source LLM using parameter-efficient fine-tuning. Experimental results show that our discrete prompt optimization framework effectively improves the response quality of ChatGPT, and CourseGPT-zh exhibits strong professional capabilities in specialized knowledge question-answering, significantly outperforming comparable open-source models.
• 摘要：大型语言模型（LLM）在自然语言处理（NLP）任务中表现出了惊人的能力，激发了人们对其应用于具有更高专业要求的专业领域的兴趣。然而，通过API访问闭源LLM的限制以及收集海量高质量数据集的困难，给各类课程教育领域的大型语言模型的开发带来了障碍。鉴于这些挑战，我们提出了CourseGPT-zh，这是一种支持定制和低成本部署的面向课程的教育法学硕士。针对课程语料的全面性和多样性要求，我们设计了一个结合即时优化的高质量问答语料蒸馏框架，有效挖掘课本知识并增强其多样性。此外，考虑到LLM响应与用户需求的一致性，引入了一种基于LLM-as-Judge的离散提示优化的新方法。在优化过程中，该框架利用了法学硕士反思和利用错误反馈和模式的能力，允许提供满足用户需求和偏好的提示，同时节省响应长度。最后，我们通过参数高效的微调，获得了基于开源LLM的CourseGPT-zh。实验结果表明，我们的离散提示优化框架有效提高了ChatGPT的响应质量，CourseGPT-zh在专业知识问答方面表现出强大的专业能力，显着优于同类开源模型。

Title: Zero-shot LLM-guided Counterfactual Generation for Text

• Authors: Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, Huan Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.04793
• Pdf URL: https://arxiv.org/pdf/2405.04793
• Copy Paste: [[2405.04793]] Zero-shot LLM-guided Counterfactual Generation for Text(https://arxiv.org/abs/2405.04793)
• Keywords: language model, llm
• Abstract: Counterfactual examples are frequently used for model development and evaluation in many natural language processing (NLP) tasks. Although methods for automated counterfactual generation have been explored, such methods depend on models such as pre-trained language models that are then fine-tuned on auxiliary, often task-specific datasets. Collecting and annotating such datasets for counterfactual generation is labor intensive and therefore, infeasible in practice. Therefore, in this work, we focus on a novel problem setting: \textit{zero-shot counterfactual generation}. To this end, we propose a structured way to utilize large language models (LLMs) as general purpose counterfactual example generators. We hypothesize that the instruction-following and textual understanding capabilities of recent LLMs can be effectively leveraged for generating high quality counterfactuals in a zero-shot manner, without requiring any training or fine-tuning. Through comprehensive experiments on various downstream tasks in natural language processing (NLP), we demonstrate the efficacy of LLMs as zero-shot counterfactual generators in evaluating and explaining black-box NLP models.
• 摘要：反事实示例经常用于许多自然语言处理 (NLP) 任务中的模型开发和评估。尽管已经探索了自动反事实生成的方法，但此类方法依赖于诸如预先训练的语言模型之类的模型，然后在辅助的、通常特定于任务的数据集上进行微调。收集和注释此类用于反事实生成的数据集是劳动密集型的，因此在实践中不可行。因此，在这项工作中，我们专注于一个新颖的问题设置：\textit{零样本反事实生成}。为此，我们提出了一种利用大型语言模型（LLM）作为通用反事实示例生成器的结构化方法。我们假设，最近法学硕士的指令跟踪和文本理解能力可以有效地利用，以零样本的方式生成高质量的反事实，而不需要任何培训或微调。通过对自然语言处理（NLP）中各种下游任务的综合实验，我们证明了法学硕士作为零样本反事实生成器在评估和解释黑盒 NLP 模型方面的功效。

Title: ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation

• Authors: Ana Brassard, Benjamin Heinzerling, Keito Kudo, Keisuke Sakaguchi, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.04818
• Pdf URL: https://arxiv.org/pdf/2405.04818
• Copy Paste: [[2405.04818]] ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation(https://arxiv.org/abs/2405.04818)
• Keywords: language model, gpt, llm
• Abstract: Evaluating free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to gain insights into how LLMs evaluate explanations. We observed that replacing one of the human ratings sometimes maintained, but more often lowered the inter-annotator agreement across different settings and quality aspects, suggesting that their judgments are not always consistent with human raters. We further quantified this difference by comparing the correlation between LLM-generated ratings with majority-voted human ratings across different quality aspects. With the best system, Spearman's rank correlation ranged between 0.53 to 0.95, averaging 0.72 across aspects, indicating moderately high but imperfect alignment. Finally, we considered the alternative of using an LLM as an additional rater when human raters are scarce, and measured the correlation between majority-voted labels with a limited human pool and LLMs as an additional rater, compared to the original gold labels. While GPT-4 improved the outcome when there were only two human raters, in all other observed cases, LLMs were neutral to detrimental when there were three or more human raters. We publicly release the dataset to support future improvements in LLM-in-the-loop evaluation here: this https URL.
• 摘要：评估自由文本解释是一项多方面、主观且劳动密集型的任务。大型语言模型 (LLM) 因其一致性、可扩展性和成本效益的潜力而成为一种有吸引力的替代方案。在这项工作中，我们提出了 ACORN，一个包含 3,500 个自由文本解释和方面质量评级的新数据集，并使用它来深入了解法学硕士如何评估解释。我们观察到，替换其中一项人工评分有时会维持不变，但更常见的是降低不同设置和质量方面的注释者间一致性，这表明他们的判断并不总是与人工评分者一致。我们通过比较法学硕士生成的评级与不同质量方面的多数投票的人类评级之间的相关性，进一步量化了这种差异。在最好的系统中，Spearman 的等级相关性在 0.53 到 0.95 之间，各个方面的平均值为 0.72，表明一致性较高但不完美。最后，我们考虑了当人类评估员稀缺时使用法学硕士作为额外评估员的替代方案，并与原始黄金标签相比，测量了有限人力库的多数投票标签与法学硕士作为额外评估员之间的相关性。虽然 GPT-4 在只有两名人类评估者时改善了结果，但在所有其他观察到的案例中，当有三名或更多人类评估者时，法学硕士是中性甚至有害的。我们在这里公开发布数据集，以支持 LLM 循环评估的未来改进：此 https URL。

Title: DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature

• Authors: Dawei Li, Shu Yang, Zhen Tan, Jae Young Baik, Sunkwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran, Ying Ding, Huan Liu, Li Shen, Tianlong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.04819
• Pdf URL: https://arxiv.org/pdf/2405.04819
• Copy Paste: [[2405.04819]] DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature(https://arxiv.org/abs/2405.04819)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have achieved promising performances across various applications. Nonetheless, the ongoing challenge of integrating long-tail knowledge continues to impede the seamless adoption of LLMs in specialized domains. In this work, we introduce DALK, a.k.a. Dynamic Co-Augmentation of LLMs and KG, to address this limitation and demonstrate its ability on studying Alzheimer's Disease (AD), a specialized sub-field in biomedicine and a global health priority. With a synergized framework of LLM and KG mutually enhancing each other, we first leverage LLM to construct an evolving AD-specific knowledge graph (KG) sourced from AD-related scientific literature, and then we utilize a coarse-to-fine sampling method with a novel self-aware knowledge retrieval approach to select appropriate knowledge from the KG to augment LLM inference capabilities. The experimental results, conducted on our constructed AD question answering (ADQA) benchmark, underscore the efficacy of DALK. Additionally, we perform a series of detailed analyses that can offer valuable insights and guidelines for the emerging topic of mutually enhancing KG and LLM. We will release the code and data at this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展在各种应用程序中取得了令人鼓舞的性能。尽管如此，整合长尾知识的持续挑战仍然阻碍了法学硕士在专业领域的无缝采用。在这项工作中，我们引入了 DALK（又名 LLM 和 KG 的动态联合增强）来解决这一限制，并展示其研究阿尔茨海默病 (AD) 的能力，阿尔茨海默病是生物医学的一个专业子领域，也是全球健康的优先事项。通过LLM和KG相互增强的协同框架，我们首先利用LLM构建一个源自AD相关科学文献的不断发展的AD特定知识图谱（KG），然后利用从粗到细的采样方法一种新颖的自我意识知识检索方法，用于从 KG 中选择适当的知识来增强 LLM 推理能力。在我们构建的 AD 问答 (ADQA) 基准上进行的实验结果强调了 DALK 的功效。此外，我们还进行了一系列详细的分析，可以为 KG 和 LLM 相互增强的新兴主题提供有价值的见解和指南。我们将在此 https URL 发布代码和数据。

Title: APrompt4EM: Augmented Prompt Tuning for Generalized Entity Matching

• Authors: Yikuan Xia, Jiazun Chen, Xinchi Li, Jun Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.04820
• Pdf URL: https://arxiv.org/pdf/2405.04820
• Copy Paste: [[2405.04820]] APrompt4EM: Augmented Prompt Tuning for Generalized Entity Matching(https://arxiv.org/abs/2405.04820)
• Keywords: language model, llm, prompt
• Abstract: Generalized Entity Matching (GEM), which aims at judging whether two records represented in different formats refer to the same real-world entity, is an essential task in data management. The prompt tuning paradigm for pre-trained language models (PLMs), including the recent PromptEM model, effectively addresses the challenges of low-resource GEM in practical applications, offering a robust solution when labeled data is scarce. However, existing prompt tuning models for GEM face the challenges of prompt design and information gap. This paper introduces an augmented prompt tuning framework for the challenges, which consists of two main improvements. The first is an augmented contextualized soft token-based prompt tuning method that extracts a guiding soft token benefit for the PLMs' prompt tuning, and the second is a cost-effective information augmentation strategy leveraging large language models (LLMs). Our approach performs well on the low-resource GEM challenges. Extensive experiments show promising advancements of our basic model without information augmentation over existing methods based on moderate-size PLMs (average 5.24%+), and our model with information augmentation achieves comparable performance compared with fine-tuned LLMs, using less than 14% of the API fee.
• 摘要：广义实体匹配（GEM）旨在判断以不同格式表示的两条记录是否指代同一个现实世界实体，是数据管理中的一项重要任务。预训练语言模型（PLM）的即时调优范例，包括最近的 PromptEM 模型，有效解决了实际应用中资源匮乏的 GEM 挑战，在标记数据稀缺时提供了强大的解决方案。然而，现有的GEM即时调优模型面临着即时设计和信息鸿沟的挑战。本文介绍了针对挑战的增强提示调整框架，其中包括两个主要改进。第一个是基于增强上下文软令牌的提示调整方法，该方法为 PLM 的提示调整提取指导性软令牌优势，第二个是利用大型语言模型 (LLM) 的经济高效的信息增强策略。我们的方法在低资源 GEM 挑战中表现良好。大量实验表明，与基于中等规模 PLM 的现有方法（平均 5.24%+）相比，我们的没有信息增强的基本模型取得了有希望的进步，并且我们的信息增强模型与微调的 LLM 相比，使用了不到 14% 的时间，实现了与微调的 LLM 相当的性能。 API 费用。

Title: ChuXin: 1.6B Technical Report

• Authors: Xiaomin Zhuang, Yufan Jiang, Qiaozhi He, Zhihua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] ChuXin: 1.6B Technical Report(https://arxiv.org/abs/)
• Keywords: language model
• Abstract: In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weights and architecture, we have made everything needed to train a model available, including the training data, the training process, and the evaluation code. Our goal is to empower and strengthen the open research community, fostering transparency and enabling a new wave of innovation in the field of language modeling. Furthermore, we extend the context length to 1M tokens through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. The weights for both models are available at Hugging Face to download and use.
• 摘要：在这份报告中，我们展示了 ChuXin，一个完全开源的语言模型，拥有 16 亿个参数。与大多数仅开源模型权重和架构的作品不同，我们已经提供了训练模型所需的一切，包括训练数据、训练过程和评估代码。我们的目标是增强和加强开放研究社区的能力，提高透明度并推动语言建模领域的新一波创新浪潮。此外，我们通过轻量级持续预训练将上下文长度扩展到 1M 个 token，并展示了强大的大海捞针检索性能。两种模型的权重均可在 Hugging Face 上下载和使用。

Title: Logical Negation Augmenting and Debiasing for Prompt-based Methods

• Authors: Yitian Li, Jidong Tian, Hao He, Yaohui Jin
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2405.04872
• Pdf URL: https://arxiv.org/pdf/2405.04872
• Copy Paste: [[2405.04872]] Logical Negation Augmenting and Debiasing for Prompt-based Methods(https://arxiv.org/abs/2405.04872)
• Keywords: prompt
• Abstract: Prompt-based methods have gained increasing attention on NLP and shown validity on many downstream tasks. Many works have focused on mining these methods' potential for knowledge extraction, but few explore their ability to make logical reasoning. In this work, we focus on the effectiveness of the prompt-based methods on first-order logical reasoning and find that the bottleneck lies in logical negation. Based on our analysis, logical negation tends to result in spurious correlations to negative answers, while propositions without logical negation correlate to positive answers. To solve the problem, we propose a simple but effective method, Negation Augmenting and Negation Debiasing (NAND), which introduces negative propositions to prompt-based methods without updating parameters. Specifically, these negative propositions can counteract spurious correlations by providing "not" for all instances so that models cannot make decisions only by whether expressions contain a logical negation. Experiments on three datasets show that NAND not only solves the problem of calibrating logical negation but also significantly enhances prompt-based methods of logical reasoning without model retraining.
• 摘要：基于提示的方法在 NLP 领域受到越来越多的关注，并在许多下游任务中显示出有效性。许多工作都集中于挖掘这些方法在知识提取方面的潜力，但很少探讨它们进行逻辑推理的能力。在这项工作中，我们关注基于提示的方法在一阶逻辑推理上的有效性，发现瓶颈在于逻辑否定。根据我们的分析，逻辑否定往往会导致与否定答案的虚假相关，而没有逻辑否定的命题则与肯定答案相关。为了解决这个问题，我们提出了一种简单但有效的方法，否定增强和否定去偏（NAND），它将否定命题引入基于提示的方法而不更新参数。具体来说，这些否定命题可以通过为所有实例提供“not”来抵消虚假相关性，以便模型不能仅根据表达式是否包含逻辑否定来做出决策。在三个数据集上的实验表明，NAND 不仅解决了校准逻辑否定的问题，而且还显着增强了基于提示的逻辑推理方法，而无需模型重新训练。

Title: P-ICL: Point In-Context Learning for Named Entity Recognition with Large Language Models

• Authors: Guochao Jiang, Zepeng Ding, Yuchen Shi, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.04960
• Pdf URL: https://arxiv.org/pdf/2405.04960
• Copy Paste: [[2405.04960]] P-ICL: Point In-Context Learning for Named Entity Recognition with Large Language Models(https://arxiv.org/abs/2405.04960)
• Keywords: language model, llm, prompt
• Abstract: In recent years, the rise of large language models (LLMs) has made it possible to directly achieve named entity recognition (NER) without any demonstration samples or only using a few samples through in-context learning (ICL). However, standard ICL only helps LLMs understand task instructions, format and input-label mapping, but neglects the particularity of the NER task itself. In this paper, we propose a new prompting framework P-ICL to better achieve NER with LLMs, in which some point entities are leveraged as the auxiliary information to recognize each entity type. With such significant information, the LLM can achieve entity classification more precisely. To obtain optimal point entities for prompting LLMs, we also proposed a point entity selection method based on K-Means clustering. Our extensive experiments on some representative NER benchmarks verify the effectiveness of our proposed strategies in P-ICL and point entity selection.
• 摘要：近年来，大型语言模型（LLM）的兴起使得无需任何演示样本或仅使用少量样本通过上下文学习（ICL）直接实现命名实体识别（NER）成为可能。然而，标准 ICL 只能帮助 LLM 理解任务指令、格式和输入标签映射，而忽略了 NER 任务本身的特殊性。在本文中，我们提出了一种新的提示框架 P-ICL，以更好地利用 LLM 实现 NER，其中利用一些点实体作为辅助信息来识别每种实体类型。有了这些重要的信息，法学硕士可以更精确地实现实体分类。为了获得促进LLM的最佳点实体，我们还提出了一种基于K-Means聚类的点实体选择方法。我们对一些代表性的 NER 基准进行了广泛的实验，验证了我们提出的 P-ICL 和点实体选择策略的有效性。

Title: ADELIE: Aligning Large Language Models on Information Extraction

• Authors: Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05008
• Pdf URL: https://arxiv.org/pdf/2405.05008
• Copy Paste: [[2405.05008]] ADELIE: Aligning Large Language Models on Information Extraction(https://arxiv.org/abs/2405.05008)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) usually fall short on information extraction (IE) tasks and struggle to follow the complex instructions of IE tasks. This primarily arises from LLMs not being aligned with humans, as mainstream alignment datasets typically do not include IE data. In this paper, we introduce ADELIE (Aligning large language moDELs on Information Extraction), an aligned LLM that effectively solves various IE tasks, including closed IE, open IE, and on-demand IE. We first collect and construct a high-quality alignment corpus IEInstruct for IE. Then we train ADELIE_SFT using instruction tuning on IEInstruct. We further train ADELIE_SFT with direct preference optimization (DPO) objective, resulting in ADELIE_DPO. Extensive experiments on various held-out IE datasets demonstrate that our models (ADELIE_SFT and ADELIE_DPO) achieve state-of-the-art (SoTA) performance among open-source models. We further explore the general capabilities of ADELIE, and experimental results reveal that their general capabilities do not exhibit a noticeable decline. We will release the code, data, and models to facilitate further research.
• 摘要：大型语言模型 (LLM) 通常在信息提取 (IE) 任务上表现不佳，并且难以遵循 IE 任务的复杂指令。这主要是因为 LLM 与人类不一致，因为主流对齐数据集通常不包含 IE 数据。在本文中，我们介绍了 ADELIE（在信息提取上对齐大型语言模型），这是一种对齐的 LLM，可以有效地解决各种 IE 任务，包括封闭 IE、开放 IE 和按需 IE。我们首先收集并构建一个高质量的 IE 对齐语料库 IEInstruct。然后我们使用对 IEInstruct 的指令调整来训练 ADELIE_SFT。我们进一步使用直接偏好优化 (DPO) 目标训练 ADELIE_SFT，得到 ADELIE_DPO。在各种保留的 IE 数据集上进行的大量实验表明，我们的模型（ADELIE_SFT 和 ADELIE_DPO）在开源模型中实现了最先进的 (SoTA) 性能。我们进一步探索了 ADELIE 的通用能力，实验结果表明其通用能力没有出现明显下降。我们将发布代码、数据和模型，以方便进一步研究。

Title: Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources

• Authors: Lasse Hyldig Hansen, Nikolaj Andersen, Jack Gallifant, Liam G. McCoy, James K Stone, Nura Izath, Marcela Aguirre-Jerez, Danielle S Bitterman, Judy Gichoya, Leo Anthony Celi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05049
• Pdf URL: https://arxiv.org/pdf/2405.05049
• Copy Paste: [[2405.05049]] Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources(https://arxiv.org/abs/2405.05049)
• Keywords: language model, gpt, llm
• Abstract: Background Advancements in Large Language Models (LLMs) hold transformative potential in healthcare, however, recent work has raised concern about the tendency of these models to produce outputs that display racial or gender biases. Although training data is a likely source of such biases, exploration of disease and demographic associations in text data at scale has been limited. Methods We conducted a large-scale textual analysis using a dataset comprising diverse web sources, including Arxiv, Wikipedia, and Common Crawl. The study analyzed the context in which various diseases are discussed alongside markers of race and gender. Given that LLMs are pre-trained on similar datasets, this approach allowed us to examine the potential biases that LLMs may learn and internalize. We compared these findings with actual demographic disease prevalence as well as GPT-4 outputs in order to evaluate the extent of bias representation. Results Our findings indicate that demographic terms are disproportionately associated with specific disease concepts in online texts. gender terms are prominently associated with disease concepts, while racial terms are much less frequently associated. We find widespread disparities in the associations of specific racial and gender terms with the 18 diseases analyzed. Most prominently, we see an overall significant overrepresentation of Black race mentions in comparison to population proportions. Conclusions Our results highlight the need for critical examination and transparent reporting of biases in LLM pretraining datasets. Our study suggests the need to develop mitigation strategies to counteract the influence of biased training data in LLMs, particularly in sensitive domains such as healthcare.
• 摘要：背景大型语言模型（LLM）的进步在医疗保健领域具有变革潜力，然而，最近的工作引起了人们对这些模型产生显示种族或性别偏见的输出的趋势的担忧。尽管训练数据可能是此类偏差的来源，但对文本数据中疾病和人口统计学关联的大规模探索仍然有限。方法 我们使用包含不同网络资源（包括 Arxiv、维基百科和 Common Crawl）的数据集进行了大规模文本分析。该研究分析了各种疾病与种族和性别标记一起讨论的背景。鉴于法学硕士是在类似的数据集上进行预训练的，这种方法使我们能够检查法学硕士可能学习和内化的潜在偏差。我们将这些发现与实际人口疾病患病率以及 GPT-4 输出进行比较，以评估偏差代表性的程度。结果我们的研究结果表明，人口统计学术语与在线文本中的特定疾病概念相关性不成比例。性别术语与疾病概念显着相关，而种族术语的相关性则要少得多。我们发现特定种族和性别术语与所分析的 18 种疾病之间的关联存在广泛差异。最突出的是，与人口比例相比，我们发现黑人种族的提及总体上显着过高。结论 我们的结果强调了对法学硕士预训练数据集中的偏差进行严格检查和透明报告的必要性。我们的研究表明，需要制定缓解策略，以抵消法学硕士培训数据有偏见的影响，特别是在医疗保健等敏感领域。

Title: Conversational Topic Recommendation in Counseling and Psychotherapy with Decision Transformer and Large Language Models

• Authors: Aylin Gunal, Baihan Lin, Djallel Bouneffouf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05060
• Pdf URL: https://arxiv.org/pdf/2405.05060
• Copy Paste: [[2405.05060]] Conversational Topic Recommendation in Counseling and Psychotherapy with Decision Transformer and Large Language Models(https://arxiv.org/abs/2405.05060)
• Keywords: language model, llm
• Abstract: Given the increasing demand for mental health assistance, artificial intelligence (AI), particularly large language models (LLMs), may be valuable for integration into automated clinical support systems. In this work, we leverage a decision transformer architecture for topic recommendation in counseling conversations between patients and mental health professionals. The architecture is utilized for offline reinforcement learning, and we extract states (dialogue turn embeddings), actions (conversation topics), and rewards (scores measuring the alignment between patient and therapist) from previous turns within a conversation to train a decision transformer model. We demonstrate an improvement over baseline reinforcement learning methods, and propose a novel system of utilizing our model's output as synthetic labels for fine-tuning a large language model for the same task. Although our implementation based on LLaMA-2 7B has mixed results, future work can undoubtedly build on the design.
• 摘要：鉴于对心理健康援助的需求不断增长，人工智能 (AI)，尤其是大型语言模型 (LLM)，可能有助于集成到自动化临床支持系统中。在这项工作中，我们利用决策转换器架构在患者和心理健康专业人员之间的咨询对话中进行主题推荐。该架构用于离线强化学习，我们从对话中的前几轮中提取状态（对话轮次嵌入）、动作（对话主题）和奖励（衡量患者和治疗师之间一致性的分数）来训练决策转换器模型。我们展示了比基线强化学习方法的改进，并提出了一种新颖的系统，利用我们模型的输出作为合成标签来微调用于相同任务的大型语言模型。虽然我们基于 LLaMA-2 7B 的实现结果好坏参半，但未来的工作无疑可以在此设计的基础上进行。

Title: QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs

• Authors: Weijia Zhang, Vaishali Pal, Jia-Hong Huang, Evangelos Kanoulas, Maarten de Rijke
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.05109
• Pdf URL: https://arxiv.org/pdf/2405.05109
• Copy Paste: [[2405.05109]] QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs(https://arxiv.org/abs/2405.05109)
• Keywords: language model, llm
• Abstract: Table summarization is a crucial task aimed at condensing information from tabular data into concise and comprehensible textual summaries. However, existing approaches often fall short of adequately meeting users' information and quality requirements and tend to overlook the complexities of real-world queries. In this paper, we propose a novel method to address these limitations by introducing query-focused multi-table summarization. Our approach, which comprises a table serialization module, a summarization controller, and a large language model (LLM), utilizes textual queries and multiple tables to generate query-dependent table summaries tailored to users' information needs. To facilitate research in this area, we present a comprehensive dataset specifically tailored for this task, consisting of 4909 query-summary pairs, each associated with multiple tables. Through extensive experiments using our curated dataset, we demonstrate the effectiveness of our proposed method compared to baseline approaches. Our findings offer insights into the challenges of complex table reasoning for precise summarization, contributing to the advancement of research in query-focused multi-table summarization.
• 摘要：表格摘要是一项关键任务，旨在将表格数据中的信息压缩为简洁且易于理解的文本摘要。然而，现有的方法往往无法充分满足用户的信息和质量要求，并且往往忽视现实世界查询的复杂性。在本文中，我们提出了一种新方法，通过引入以查询为中心的多表汇总来解决这些限制。我们的方法包括表序列化模块、摘要控制器和大型语言模型 (LLM)，利用文本查询和多个表来生成适合用户信息需求的依赖于查询的表摘要。为了促进这一领域的研究，我们提供了专门为此任务定制的综合数据集，其中包含 4909 个查询-摘要对，每个查询-摘要对与多个表相关联。通过使用我们策划的数据集进行大量实验，我们证明了我们提出的方法与基线方法相比的有效性。我们的研究结果提供了对精确摘要的复杂表推理挑战的见解，有助于以查询为中心的多表摘要研究的进步。

Title: XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples

• Authors: Peiqin Lin, André F. T. Martins, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05116
• Pdf URL: https://arxiv.org/pdf/2405.05116
• Copy Paste: [[2405.05116]] XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples(https://arxiv.org/abs/2405.05116)
• Keywords: language model
• Abstract: Recent studies have shown that leveraging off-the-shelf or fine-tuned retrievers, capable of retrieving high-quality in-context examples, significantly improves in-context learning of English. However, adapting these methods to other languages, especially low-resource ones, presents challenges due to the scarcity of available cross-lingual retrievers and annotated data. In this paper, we introduce XAMPLER: Cross-Lingual Example Retrieval, a method tailored to tackle the challenge of cross-lingual in-context learning using only annotated English data. XAMPLER first trains a retriever with positive/negative English samples, which are constructed based on the predictions of the multilingual large language model for in-context learning. Then, the trained retriever is directly employed to retrieve English examples as few-shot examples for in-context learning of target languages. Experiments on the massively multilingual text classification benchmark of SIB200 with 176 languages demonstrate that XAMPLER substantially improves the in-context learning performance across languages. Our code is available at this https URL.
• 摘要：最近的研究表明，利用现成的或经过微调的检索器能够检索高质量的上下文示例，可以显着提高英语的上下文学习。然而，由于可用的跨语言检索器和注释数据的稀缺，将这些方法应用于其他语言，尤其是资源匮乏的语言，会带来挑战。在本文中，我们介绍了 XAMPLER：跨语言示例检索，这是一种专门针对仅使用带注释的英语数据来解决跨语言上下文学习挑战的方法。 XAMPLER 首先使用正/负英语样本训练检索器，这些样本是根据用于上下文学习的多语言大语言模型的预测构建的。然后，直接使用经过训练的检索器来检索英语示例，作为目标语言的上下文学习的小样本示例。在 SIB200 176 种语言的大规模多语言文本分类基准上进行的实验表明，XAMPLER 显着提高了跨语言的上下文学习性能。我们的代码可以在这个 https URL 上找到。

Title: Encoder-Decoder Framework for Interactive Free Verses with Generation with Controllable High-Quality Rhyming

• Authors: Tommaso Pasini, Alejo López-Ávila, Husam Quteineh, Gerasimos Lampouras, Jinhua Du, Yubing Wang, Ze Li, Yusen Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05176
• Pdf URL: https://arxiv.org/pdf/2405.05176
• Copy Paste: [[2405.05176]] Encoder-Decoder Framework for Interactive Free Verses with Generation with Controllable High-Quality Rhyming(https://arxiv.org/abs/2405.05176)
• Keywords: language model
• Abstract: Composing poetry or lyrics involves several creative factors, but a challenging aspect of generation is the adherence to a more or less strict metric and rhyming pattern. To address this challenge specifically, previous work on the task has mainly focused on reverse language modeling, which brings the critical selection of each rhyming word to the forefront of each verse. On the other hand, reversing the word order requires that models be trained from scratch with this task-specific goal and cannot take advantage of transfer learning from a Pretrained Language Model (PLM). We propose a novel fine-tuning approach that prepends the rhyming word at the start of each lyric, which allows the critical rhyming decision to be made before the model commits to the content of the lyric (as during reverse language modeling), but maintains compatibility with the word order of regular PLMs as the lyric itself is still generated in left-to-right order. We conducted extensive experiments to compare this fine-tuning against the current state-of-the-art strategies for rhyming, finding that our approach generates more readable text and better rhyming capabilities. Furthermore, we furnish a high-quality dataset in English and 12 other languages, analyse the approach's feasibility in a multilingual context, provide extensive experimental results shedding light on good and bad practices for lyrics generation, and propose metrics to compare methods in the future.
• 摘要：创作诗歌或歌词涉及多种创作因素，但创作的一个具有挑战性的方面是遵守或多或少严格的韵律和押韵模式。为了具体解决这一挑战，之前的任务工作主要集中在反向语言建模上，它将每个押韵词的关键选择带到每节经文的最前面。另一方面，颠倒词序需要从头开始训练模型以实现特定于任务的目标，并且无法利用预训练语言模型 (PLM) 的迁移学习。我们提出了一种新颖的微调方法，在每个歌词的开头添加押韵词，这允许在模型提交歌词内容之前做出关键的押韵决定（如在反向语言建模期间），但保持兼容性与常规 PLM 的词序相同，因为歌词本身仍然按从左到右的顺序生成。我们进行了广泛的实验，将这种微调与当前最先进的押韵策略进行比较，发现我们的方法生成了更易读的文本和更好的押韵能力。此外，我们提供了英语和其他 12 种语言的高质量数据集，分析了该方法在多语言环境中的可行性，提供了广泛的实验结果，揭示了歌词生成的好坏做法，并提出了未来比较方法的指标。

Title: MIDGARD: Self-Consistency Using Minimum Description Length for Structured Commonsense Reasoning

• Authors: Inderjeet Nair, Lu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.05189
• Pdf URL: https://arxiv.org/pdf/2405.05189
• Copy Paste: [[2405.05189]] MIDGARD: Self-Consistency Using Minimum Description Length for Structured Commonsense Reasoning(https://arxiv.org/abs/2405.05189)
• Keywords: language model, llm, prompt
• Abstract: We study the task of conducting structured reasoning as generating a reasoning graph from natural language input using large language models (LLMs). Previous approaches have explored various prompting schemes, yet they suffer from error propagation due to the autoregressive nature and single-pass-based decoding, which lack error correction capability. Additionally, relying solely on a single sample may result in the omission of true nodes and edges. To counter this, we draw inspiration from self-consistency (SC), which involves sampling a diverse set of reasoning chains and taking the majority vote as the final answer. To tackle the substantial challenge of applying SC on generated graphs, we propose MIDGARD (MInimum Description length Guided Aggregation of Reasoning in Directed acyclic graph) that leverages Minimum Description Length (MDL)-based formulation to identify consistent properties among the different graph samples generated by an LLM. This formulation helps reject properties that appear in only a few samples, which are likely to be erroneous, while enabling the inclusion of missing elements without compromising precision. Our method demonstrates superior performance than comparisons across various structured reasoning tasks, including argument structure extraction, explanation graph generation, inferring dependency relations among actions for everyday tasks, and semantic graph generation from natural texts.
• 摘要：我们研究进行结构化推理的任务，即使用大型语言模型（LLM）从自然语言输入生成推理图。以前的方法已经探索了各种提示方案，但由于自回归性质和基于单通道的解码缺乏纠错能力，它们会遭受错误传播。此外，仅依赖单个样本可能会导致遗漏真实的节点和边。为了解决这个问题，我们从自我一致性（SC）中汲取灵感，它涉及对一组不同的推理链进行采样，并以多数票作为最终答案。为了解决在生成的图上应用 SC 的重大挑战，我们提出了 MIDGARD（有向无环图中的最小描述长度引导推理聚合），它利用基于最小描述长度 (MDL) 的公式来识别由生成的不同图样本之间的一致属性。法学硕士。该公式有助于拒绝仅出现在少数样本中的属性，这些属性可能是错误的，同时能够在不影响精度的情况下包含缺失的元素。我们的方法表现出比各种结构化推理任务的比较优越的性能，包括参数结构提取、解释图生成、推断日常任务的动作之间的依赖关系以及从自然文本生成语义图。

Title: CARE-SD: Classifier-based analysis for recognizing and eliminating stigmatizing and doubt marker labels in electronic health records: model development and validation

• Authors: Drew Walker, Annie Thorne, Sudeshna Das, Jennifer Love, Hannah LF Cooper, Melvin Livingston III, Abeed Sarker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05204
• Pdf URL: https://arxiv.org/pdf/2405.05204
• Copy Paste: [[2405.05204]] CARE-SD: Classifier-based analysis for recognizing and eliminating stigmatizing and doubt marker labels in electronic health records: model development and validation(https://arxiv.org/abs/2405.05204)
• Keywords: gpt
• Abstract: Objective: To detect and classify features of stigmatizing and biased language in intensive care electronic health records (EHRs) using natural language processing techniques. Materials and Methods: We first created a lexicon and regular expression lists from literature-driven stem words for linguistic features of stigmatizing patient labels, doubt markers, and scare quotes within EHRs. The lexicon was further extended using Word2Vec and GPT 3.5, and refined through human evaluation. These lexicons were used to search for matches across 18 million sentences from the de-identified Medical Information Mart for Intensive Care-III (MIMIC-III) dataset. For each linguistic bias feature, 1000 sentence matches were sampled, labeled by expert clinical and public health annotators, and used to supervised learning classifiers. Results: Lexicon development from expanded literature stem-word lists resulted in a doubt marker lexicon containing 58 expressions, and a stigmatizing labels lexicon containing 127 expressions. Classifiers for doubt markers and stigmatizing labels had the highest performance, with macro F1-scores of .84 and .79, positive-label recall and precision values ranging from .71 to .86, and accuracies aligning closely with human annotator agreement (.87). Discussion: This study demonstrated the feasibility of supervised classifiers in automatically identifying stigmatizing labels and doubt markers in medical text, and identified trends in stigmatizing language use in an EHR setting. Additional labeled data may help improve lower scare quote model performance. Conclusions: Classifiers developed in this study showed high model performance and can be applied to identify patterns and target interventions to reduce stigmatizing labels and doubt markers in healthcare systems.
• 摘要：目的：使用自然语言处理技术检测重症监护电子健康记录（EHR）中的污名化和偏见语言特征并进行分类。材料和方法：我们首先根据文献驱动的词干词创建了一个词典和正则表达式列表，用于 EHR 中污名化患者标签、疑问标记和恐吓引用的语言特征。使用 Word2Vec 和 GPT 3.5 进一步扩展了词典，并通过人工评估进行了完善。这些词典用于从去识别化的重症监护医疗信息市场 III (MIMIC-III) 数据集中搜索 1800 万个句子的匹配项。对于每个语言偏差特征，我们对 1000 个句子匹配进行了采样，由临床和公共卫生专家注释者进行标记，并用于监督学习分类器。结果：根据扩展的文献干词列表开发的词典产生了包含 58 个表达的疑问标记词典和包含 127 个表达的污名化标签词典。怀疑标记和污名化标签的分类器具有最高的性能，宏观 F1 分数为 0.84 和 0.79，正标签召回率和精度值范围为 0.71 到 0.86，准确度与人类注释者协议密切相关 (0.87 ）。讨论：这项研究证明了监督分类器在自动识别医学文本中的污名化标签和疑问标记方面的可行性，并确定了电子病历环境中污名化语言使用的趋势。额外的标记数据可能有助于提高较低的恐慌报价模型性能。结论：本研究开发的分类器显示出较高的模型性能，可用于识别模式和目标干预措施，以减少医疗保健系统中的污名化标签和怀疑标记。

Title: LLMs with Personalities in Multi-issue Negotiation Games

• Authors: Sean Noh, Ho-Chun Herbert Chang
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2405.05248
• Pdf URL: https://arxiv.org/pdf/2405.05248
• Copy Paste: [[2405.05248]] LLMs with Personalities in Multi-issue Negotiation Games(https://arxiv.org/abs/2405.05248)
• Keywords: language model, llm, agent
• Abstract: Powered by large language models (LLMs), AI agents have become capable of many human tasks. Using the most canonical definitions of the Big Five personality, we measure the ability of LLMs to negotiate within a game-theoretical framework, as well as methodological challenges to measuring notions of fairness and risk. Simulations (n=1,500) for both single-issue and multi-issue negotiation reveal increase in domain complexity with asymmetric issue valuations improve agreement rates but decrease surplus from aggressive negotiation. Through gradient-boosted regression and Shapley explainers, we find high openness, conscientiousness, and neuroticism are associated with fair tendencies; low agreeableness and low openness are associated with rational tendencies. Low conscientiousness is associated with high toxicity. These results indicate that LLMs may have built-in guardrails that default to fair behavior, but can be "jail broken" to exploit agreeable opponents. We also offer pragmatic insight in how negotiation bots can be designed, and a framework of assessing negotiation behavior based on game theory and computational social science.
• 摘要：在大型语言模型 (LLM) 的支持下，人工智能代理已经能够执行许多人类任务。我们使用大五人格最规范的定义，衡量法学硕士在博弈论框架内谈判的能力，以及衡量公平和风险概念的方法挑战。单问题和多问题谈判的模拟（n=1,500）表明，问题估值不对称会导致领域复杂性增加，从而提高协议率，但会减少激进谈判带来的盈余。通过梯度增强回归和沙普利解释器，我们发现高度开放性、责任心和神经质与公平倾向相关；低宜人性和低开放性与理性倾向相关。低责任感与高毒性相关。这些结果表明，法学硕士可能具有默认公平行为的内置护栏，但可能会“越狱”以利用令人愉快的对手。我们还提供了关于如何设计谈判机器人的实用见解，以及基于博弈论和计算社会科学的评估谈判行为的框架。

Title: Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge

• Authors: Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, Paul Denny
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2405.05253
• Pdf URL: https://arxiv.org/pdf/2405.05253
• Copy Paste: [[2405.05253]] Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge(https://arxiv.org/abs/2405.05253)
• Keywords: language model, gpt, llm, chat
• Abstract: Large language models (LLMs) have shown great potential for the automatic generation of feedback in a wide range of computing contexts. However, concerns have been voiced around the privacy and ethical implications of sending student work to proprietary models. This has sparked considerable interest in the use of open source LLMs in education, but the quality of the feedback that such open models can produce remains understudied. This is a concern as providing flawed or misleading generated feedback could be detrimental to student learning. Inspired by recent work that has utilised very powerful LLMs, such as GPT-4, to evaluate the outputs produced by less powerful models, we conduct an automated analysis of the quality of the feedback produced by several open source models using a dataset from an introductory programming course. First, we investigate the viability of employing GPT-4 as an automated evaluator by comparing its evaluations with those of a human expert. We observe that GPT-4 demonstrates a bias toward positively rating feedback while exhibiting moderate agreement with human raters, showcasing its potential as a feedback evaluator. Second, we explore the quality of feedback generated by several leading open-source LLMs by using GPT-4 to evaluate the feedback. We find that some models offer competitive performance with popular proprietary LLMs, such as ChatGPT, indicating opportunities for their responsible use in educational settings.
• 摘要：大型语言模型（LLM）在广泛的计算环境中显示出自动生成反馈的巨大潜力。然而，人们对将学生作品发送到专有模型的隐私和道德影响表示担忧。这引发了人们对在教育中使用开源法学硕士的极大兴趣，但这种开放模型所能产生的反馈质量仍然没有得到充分研究。这是一个令人担忧的问题，因为提供有缺陷或误导性的生成反馈可能不利于学生的学习。受到最近利用非常强大的 LLM（例如 GPT-4）来评估功能较弱的模型产生的输出的工作的启发，我们使用来自介绍性模型的数据集对几个开源模型产生的反馈质量进行了自动分析。编程课程。首先，我们通过将 GPT-4 的评估与人类专家的评估进行比较来研究使用 GPT-4 作为自动评估器的可行性。我们观察到，GPT-4 表现出对积极评价反馈的偏见，同时与人类评分者表现出适度的一致性，展示了其作为反馈评估器的潜力。其次，我们通过使用 GPT-4 来评估反馈，探讨了几个领先的开源 LLM 生成的反馈的质量。我们发现，一些模型与流行的专有法学硕士（例如 ChatGPT）相比提供了具有竞争力的性能，这表明它们有机会在教育环境中负责任地使用。

Title: You Only Cache Once: Decoder-Decoder Architectures for Language Models

• Authors: Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.05254
• Pdf URL: https://arxiv.org/pdf/2405.05254
• Copy Paste: [[2405.05254]] You Only Cache Once: Decoder-Decoder Architectures for Language Models(https://arxiv.org/abs/2405.05254)
• Keywords: language model
• Abstract: We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e., a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes. Code is available at this https URL.
• 摘要：我们为大型语言模型引入了一种解码器-解码器架构 YOCO，它只缓存一次键值对。它由两个组件组成，即堆叠在自解码器上的交叉解码器。自解码器有效地对全局键值（KV）缓存进行编码，交叉解码器通过交叉注意力重用这些缓存。整个模型的行为就像一个仅解码器的 Transformer，尽管 YOCO 仅缓存一次。该设计大大降低了 GPU 内存需求，但保留了全局注意力能力。此外，计算流程使预填充能够提前退出，而不改变最终输出，从而显着加快预填充阶段。实验结果表明，与 Transformer 相比，YOCO 在扩大模型大小和训练令牌数量的各种设置中都取得了良好的性能。我们还将 YOCO 扩展到 1M 上下文长度，并具有近乎完美的针检索精度。分析结果表明，YOCO 在上下文长度和模型大小方面将推理内存、预填充延迟和吞吐量提高了几个数量级。代码可从此 https URL 获取。