2024-04-03

Title: Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers

• Authors: Philip Kenneweg, Alexander Schulz, Sarah Schröder, Barbara Hammer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01317
• Pdf URL: https://arxiv.org/pdf/2404.01317
• Copy Paste: [[2404.01317]] Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers(https://arxiv.org/abs/2404.01317)
• Keywords: language model
• Abstract: Pretraining language models on large text corpora is a common practice in natural language processing. Fine-tuning of these models is then performed to achieve the best results on a variety of tasks. In this paper, we investigate the problem of catastrophic forgetting in transformer neural networks and question the common practice of fine-tuning with a flat learning rate for the entire network in this context. We perform a hyperparameter optimization process to find learning rate distributions that are better than a flat learning rate. We combine the learning rate distributions thus found and show that they generalize to better performance with respect to the problem of catastrophic forgetting. We validate these learning rate distributions with a variety of NLP benchmarks from the GLUE dataset.
• 摘要：在大型文本语料库上预训练语言模型是自然语言处理中的常见做法。然后对这些模型进行微调，以在各种任务上获得最佳结果。在本文中，我们研究了变压器神经网络中的灾难性遗忘问题，并对在此背景下对整个网络进行平坦学习率微调的常见做法提出了质疑。我们执行超参数优化过程来找到比平坦学习率更好的学习率分布。我们结合了由此发现的学习率分布，并表明它们对于灾难性遗忘问题可以泛化为更好的性能。我们使用 GLUE 数据集中的各种 NLP 基准验证这些学习率分布。

Title: A Review of Multi-Modal Large Language and Vision Models

• Authors: Kilian Carolan, Laura Fennelly, Alan F. Smeaton
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01322
• Pdf URL: https://arxiv.org/pdf/2404.01322
• Copy Paste: [[2404.01322]] A Review of Multi-Modal Large Language and Vision Models(https://arxiv.org/abs/2404.01322)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have recently emerged as a focal point of research and application, driven by their unprecedented ability to understand and generate text with human-like quality. Even more recently, LLMs have been extended into multi-modal large language models (MM-LLMs) which extends their capabilities to deal with image, video and audio information, in addition to text. This opens up applications like text-to-video generation, image captioning, text-to-speech, and more and is achieved either by retro-fitting an LLM with multi-modal capabilities, or building a MM-LLM from scratch. This paper provides an extensive review of the current state of those LLMs with multi-modal capabilities as well as the very recent MM-LLMs. It covers the historical development of LLMs especially the advances enabled by transformer-based architectures like OpenAI's GPT series and Google's BERT, as well as the role of attention mechanisms in enhancing model performance. The paper includes coverage of the major and most important of the LLMs and MM-LLMs and also covers the techniques of model tuning, including fine-tuning and prompt engineering, which tailor pre-trained models to specific tasks or domains. Ethical considerations and challenges, such as data bias and model misuse, are also analysed to underscore the importance of responsible AI development and deployment. Finally, we discuss the implications of open-source versus proprietary models in AI research. Through this review, we provide insights into the transformative potential of MM-LLMs in various applications.
• 摘要：大型语言模型 (LLM) 最近已成为研究和应用的焦点，其驱动力是其前所未有的理解和生成具有类人质量的文本的能力。最近，LLM 已扩展到多模态大语言模型 (MM-LLM)，除了文本之外，还扩展了其处理图像、视频和音频信息的能力。这开辟了文本到视频生成、图像字幕、文本到语音等应用程序，并且可以通过改造具有多模态功能的法学硕士或从头开始构建 MM-LLM 来实现。本文对具有多模式能力的法学硕士以及最近的 MM-LLM 的现状进行了广泛的回顾。它涵盖了法学硕士的历史发展，特别是 OpenAI 的 GPT 系列和 Google 的 BERT 等基于 Transformer 的架构带来的进步，以及注意力机制在增强模型性能方面的作用。该论文涵盖了 LLM 和 MM-LLM 的主要和最重要的内容，还涵盖了模型调整技术，包括微调和即时工程，这些技术根据特定任务或领域定制预训练模型。还分析了数据偏差和模型滥用等道德考虑因素和挑战，以强调负责任的人工智能开发和部署的重要性。最后，我们讨论人工智能研究中开源模型与专有模型的影响。通过这次审查，我们深入了解了 MM-LLM 在各种应用中的变革潜力。

Title: Entertainment chatbot for the digital inclusion of elderly people without abstraction capabilities

• Authors: Silvia García-Méndez, Francisco de Arriba-Pérez, Francisco J. González-Castaño, José A. Regueiro-Janeiro, Felipe Gil-Castiñeira
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01327
• Pdf URL: https://arxiv.org/pdf/2404.01327
• Copy Paste: [[2404.01327]] Entertainment chatbot for the digital inclusion of elderly people without abstraction capabilities(https://arxiv.org/abs/2404.01327)
• Keywords: chat
• Abstract: Current language processing technologies allow the creation of conversational chatbot platforms. Even though artificial intelligence is still too immature to support satisfactory user experience in many mass market domains, conversational interfaces have found their way into ad hoc applications such as call centres and online shopping assistants. However, they have not been applied so far to social inclusion of elderly people, who are particularly vulnerable to the digital divide. Many of them relieve their loneliness with traditional media such as TV and radio, which are known to create a feeling of companionship. In this paper we present the EBER chatbot, designed to reduce the digital gap for the elderly. EBER reads news in the background and adapts its responses to the user's mood. Its novelty lies in the concept of "intelligent radio", according to which, instead of simplifying a digital information system to make it accessible to the elderly, a traditional channel they find familiar -- background news -- is augmented with interactions via voice dialogues. We make it possible by combining Artificial Intelligence Modelling Language, automatic Natural Language Generation and Sentiment Analysis. The system allows accessing digital content of interest by combining words extracted from user answers to chatbot questions with keywords extracted from the news items. This approach permits defining metrics of the abstraction capabilities of the users depending on a spatial representation of the word space. To prove the suitability of the proposed solution we present results of real experiments conducted with elderly people that provided valuable insights. Our approach was considered satisfactory during the tests and improved the information search capabilities of the participants.
• 摘要：当前的语言处理技术允许创建对话式聊天机器人平台。尽管人工智能还不够成熟，无法在许多大众市场领域提供令人满意的用户体验，但会话界面已经进入呼叫中心和在线购物助理等临时应用程序。然而，迄今为止，它们尚未应用于老年人的社会融入，因为老年人特别容易受到数字鸿沟的影响。他们中的许多人通过电视和广播等传统媒体来缓解孤独感，众所周知，这些媒体可以创造一种陪伴感。在本文中，我们介绍了 EBER 聊天机器人，旨在缩小老年人的数字鸿沟。 EBER 在后台读取新闻并根据用户的情绪调整其响应。它的新颖之处在于“智能广播”的概念，根据该概念，不是简化数字信息系统以方便老年人使用，而是通过语音对话增强了他们熟悉的传统频道（背景新闻） 。我们通过结合人工智能建模语言、自动自然语言生成和情感分析使之成为可能。该系统允许通过将从用户对聊天机器人问题的回答中提取的单词与从新闻条目中提取的关键字相结合来访问感兴趣的数字内容。这种方法允许根据词空间的空间表示来定义用户的抽象能力的度量。为了证明所提出的解决方案的适用性，我们展示了对老年人进行的真实实验的结果，这些结果提供了有价值的见解。我们的方法在测试期间被认为是令人满意的，并且提高了参与者的信息搜索能力。

Title: LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model

• Authors: Musashi Hinck, Matthew L. Olson, David Cobbley, Shao-Yen Tseng, Vasudev Lal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01331
• Pdf URL: https://arxiv.org/pdf/2404.01331
• Copy Paste: [[2404.01331]] LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model(https://arxiv.org/abs/2404.01331)
• Keywords: language model, llm
• Abstract: We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs). Of particular interest is the 2B parameter Gemma model, which provides opportunities to construct capable small-scale MMFMs. In line with findings from other papers in this space, we test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models. Closer analysis of performance shows mixed effects; skipping pretraining tends to reduce performance, larger vision models sometimes improve performance, and increasing language model size has inconsistent effects. We publicly release training recipes, code and weights for our models for the LLaVA-Gemma models.
• 摘要：我们使用流行的 LLaVA 框架和最近发布的 Gemma 系列大语言模型 (LLM) 来训练一套多模态基础模型 (MMFM)。特别令人感兴趣的是 2B 参数 Gemma 模型，它提供了构建小型 MMFM 的机会。根据该领域其他论文的发现，我们测试了消除三个设计特征的效果：预训练连接器、利用更强大的图像主干以及增加语言主干的大小。由此产生的模型（我们称之为 LLaVA-Gemma）在一系列评估中表现出中等的性能，但未能超越当前同等大小的 SOTA 模型。对性能的更仔细分析表明效果参差不齐；跳过预训练往往会降低性能，较大的视觉模型有时会提高性能，而增加语言模型大小会产生不一致的效果。我们公开发布了 LLaVA-Gemma 模型的训练方案、代码和权重。

Title: Wait, It's All Token Noise? Always Has Been: Interpreting LLM Behavior Using Shapley Value

• Authors: Behnam Mohammadi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01332
• Pdf URL: https://arxiv.org/pdf/2404.01332
• Copy Paste: [[2404.01332]] Wait, It's All Token Noise? Always Has Been: Interpreting LLM Behavior Using Shapley Value(https://arxiv.org/abs/2404.01332)
• Keywords: language model, llm, prompt
• Abstract: The emergence of large language models (LLMs) has opened up exciting possibilities for simulating human behavior and cognitive processes, with potential applications in various domains, including marketing research and consumer behavior analysis. However, the validity of utilizing LLMs as stand-ins for human subjects remains uncertain due to glaring divergences that suggest fundamentally different underlying processes at play and the sensitivity of LLM responses to prompt variations. This paper presents a novel approach based on Shapley values from cooperative game theory to interpret LLM behavior and quantify the relative contribution of each prompt component to the model's output. Through two applications-a discrete choice experiment and an investigation of cognitive biases-we demonstrate how the Shapley value method can uncover what we term "token noise" effects, a phenomenon where LLM decisions are disproportionately influenced by tokens providing minimal informative content. This phenomenon raises concerns about the robustness and generalizability of insights obtained from LLMs in the context of human behavior simulation. Our model-agnostic approach extends its utility to proprietary LLMs, providing a valuable tool for marketers and researchers to strategically optimize prompts and mitigate apparent cognitive biases. Our findings underscore the need for a more nuanced understanding of the factors driving LLM responses before relying on them as substitutes for human subjects in research settings. We emphasize the importance of researchers reporting results conditioned on specific prompt templates and exercising caution when drawing parallels between human behavior and LLMs.
• 摘要：大语言模型 (LLM) 的出现为模拟人类行为和认知过程开辟了令人兴奋的可能性，在营销研究和消费者行为分析等各个领域都有潜在的应用。然而，利用法学硕士作为人类受试者替代品的有效性仍然不确定，因为明显的分歧表明根本不同的潜在过程正在发挥作用，以及法学硕士对即时变化的反应的敏感性。本文提出了一种基于合作博弈论中的 Shapley 值的新颖方法，用于解释 LLM 行为并量化每个提示组件对模型输出的相对贡献。通过两个应用程序——离散选择实验和认知偏差调查——我们展示了沙普利值方法如何揭示我们所说的“令牌噪声”效应，在这种现象中，LLM 的决策不成比例地受到提供最少信息内容的令牌的影响。这种现象引起了人们对法学硕士在人类行为模拟背景下获得的见解的鲁棒性和普遍性的担忧。我们与模型无关的方法将其实用性扩展到专有的法学硕士，为营销人员和研究人员提供了一个有价值的工具，可以战略性地优化提示并减轻明显的认知偏差。我们的研究结果强调，在依赖法学硕士作为研究环境中人类受试者的替代品之前，需要更细致地了解驱动法学硕士反应的因素。我们强调研究人员根据特定提示模板报告结果的重要性，并在将人类行为与法学硕士进行比较时保持谨慎。

Title: Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation

• Authors: Yuji Naraki, Ryosuke Yamaki, Yoshikazu Ikeda, Takafumi Horie, Hiroki Naganuma
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01334
• Pdf URL: https://arxiv.org/pdf/2404.01334
• Copy Paste: [[2404.01334]] Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation(https://arxiv.org/abs/2404.01334)
• Keywords: language model, llm
• Abstract: In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is recognized as a critical technology, employed across a wide array of applications. Traditional methodologies for annotating datasets for NER models are challenged by high costs and variations in dataset quality. This research introduces a novel hybrid annotation approach that synergizes human effort with the capabilities of Large Language Models (LLMs). This approach not only aims to ameliorate the noise inherent in manual annotations, such as omissions, thereby enhancing the performance of NER models, but also achieves this in a cost-effective manner. Additionally, by employing a label mixing strategy, it addresses the issue of class imbalance encountered in LLM-based annotations. Through an analysis across multiple datasets, this method has been consistently shown to provide superior performance compared to traditional annotation methods, even under constrained budget conditions. This study illuminates the potential of leveraging LLMs to improve dataset quality, introduces a novel technique to mitigate class imbalances, and demonstrates the feasibility of achieving high-performance NER in a cost-effective way.
• 摘要：在自然语言处理 (NLP) 领域，命名实体识别 (NER) 被认为是一项关键技术，广泛应用于各种应用中。为 NER 模型注释数据集的传统方法受到高成本和数据集质量变化的挑战。这项研究引入了一种新颖的混合注释方法，该方法将人类的努力与大型语言模型（LLM）的功能相结合。这种方法不仅旨在改善手动注释中固有的噪声（例如遗漏），从而提高 NER 模型的性能，而且还以经济高效的方式实现这一目标。此外，通过采用标签混合策略，它解决了基于 LLM 的注释中遇到的类不平衡问题。通过对多个数据集的分析，即使在预算有限的情况下，与传统注释方法相比，该方法始终能够提供卓越的性能。这项研究阐明了利用法学硕士提高数据集质量的潜力，引入了一种减轻类别不平衡的新技术，并证明了以经济高效的方式实现高性能 NER 的可行性。

Title: Humane Speech Synthesis through Zero-Shot Emotion and Disfluency Generation

• Authors: Rohan Chaudhury, Mihir Godbole, Aakash Garg, Jinsil Hwaryoung Seo
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.01339
• Pdf URL: https://arxiv.org/pdf/2404.01339
• Copy Paste: [[2404.01339]] Humane Speech Synthesis through Zero-Shot Emotion and Disfluency Generation(https://arxiv.org/abs/2404.01339)
• Keywords: language model
• Abstract: Contemporary conversational systems often present a significant limitation: their responses lack the emotional depth and disfluent characteristic of human interactions. This absence becomes particularly noticeable when users seek more personalized and empathetic interactions. Consequently, this makes them seem mechanical and less relatable to human users. Recognizing this gap, we embarked on a journey to humanize machine communication, to ensure AI systems not only comprehend but also resonate. To address this shortcoming, we have designed an innovative speech synthesis pipeline. Within this framework, a cutting-edge language model introduces both human-like emotion and disfluencies in a zero-shot setting. These intricacies are seamlessly integrated into the generated text by the language model during text generation, allowing the system to mirror human speech patterns better, promoting more intuitive and natural user interactions. These generated elements are then adeptly transformed into corresponding speech patterns and emotive sounds using a rule-based approach during the text-to-speech phase. Based on our experiments, our novel system produces synthesized speech that's almost indistinguishable from genuine human communication, making each interaction feel more personal and authentic.
• 摘要：当代对话系统通常存在显着的局限性：它们的反应缺乏人类互动的情感深度和不连贯特征。当用户寻求更加个性化和同理心的互动时，这种缺失变得尤其明显。因此，这使得它们看起来很机械，与人类用户不太相关。认识到这一差距，我们踏上了人性化机器通信的旅程，以确保人工智能系统不仅能够理解，而且能够产生共鸣。为了解决这个缺点，我们设计了一种创新的语音合成管道。在此框架内，尖端的语言模型在零样本环境中引入了类人情感和不流畅性。这些错综复杂的内容在文本生成过程中通过语言模型无缝集成到生成的文本中，使系统能够更好地反映人类语音模式，从而促进更直观和自然的用户交互。然后，在文本转语音阶段，使用基于规则的方法将这些生成的元素巧妙地转换为相应的语音模式和情感声音。根据我们的实验，我们的新颖系统生成的合成语音几乎与真实的人类交流没有区别，使每次交互都感觉更加个性化和真实。

Title: DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

• Authors: Lirui Zhao, Yue Yang, Kaipeng Zhang, Wenqi Shao, Yuxin Zhang, Yu Qiao, Ping Luo, Rongrong Ji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01342
• Pdf URL: https://arxiv.org/pdf/2404.01342
• Copy Paste: [[2404.01342]] DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model(https://arxiv.org/abs/2404.01342)
• Keywords: language model, llm, agent
• Abstract: Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research. For example, the Civitai community, a platform for T2I innovation, currently hosts an impressive array of 74,492 distinct models. However, this diversity presents a formidable challenge in selecting the most appropriate model and parameters, a process that typically requires numerous trials. Drawing inspiration from the tool usage research of large language models (LLMs), we introduce DiffAgent, an LLM agent designed to screen the accurate selection in seconds via API calls. DiffAgent leverages a novel two-stage training framework, SFTA, enabling it to accurately align T2I API responses with user input in accordance with human preferences. To train and evaluate DiffAgent's capabilities, we present DABench, a comprehensive dataset encompassing an extensive range of T2I APIs from the community. Our evaluations reveal that DiffAgent not only excels in identifying the appropriate T2I API but also underscores the effectiveness of the SFTA training framework. Codes are available at https://github.com/OpenGVLab/DiffAgent.
• 摘要：文本到图像（T2I）生成模型引起了极大的关注，并在学术研究内外得到了广泛的应用。例如，Civitai 社区是一个 T2I 创新平台，目前拥有 74,492 个不同模型，令人印象深刻。然而，这种多样性给选择最合适的模型和参数带来了巨大的挑战，这个过程通常需要进行多次试验。从大型语言模型 (LLM) 的工具使用研究中汲取灵感，我们推出了 DiffAgent，这是一种 LLM 代理，旨在通过 API 调用在几秒钟内筛选出准确的选择。 DiffAgent 利用新颖的两阶段训练框架 SFTA，使其能够根据人类偏好准确地将 T2I API 响应与用户输入对齐。为了训练和评估 DiffAgent 的功能，我们推出了 DABench，这是一个包含来自社区的广泛 T2I API 的综合数据集。我们的评估表明，DiffAgent 不仅在识别合适的 T2I API 方面表现出色，而且还强调了 SFTA 培训框架的有效性。代码可在 https://github.com/OpenGVLab/DiffAgent 获取。

Title: CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs

• Authors: Jingzhe Shi, Jialuo Li, Qinwei Ma, Zaiwen Yang, Huan Ma, Lei Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01343
• Pdf URL: https://arxiv.org/pdf/2404.01343
• Copy Paste: [[2404.01343]] CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs(https://arxiv.org/abs/2404.01343)
• Keywords: language model, gpt, llm, chat, agent
• Abstract: Businesses and software platforms are increasingly turning to Large Language Models (LLMs) such as GPT-3.5, GPT-4, GLM-3, and LLaMa-2 for chat assistance with file access or as reasoning agents for customer service. However, current LLM-based customer service models have limited integration with customer profiles and lack the operational capabilities necessary for effective service. Moreover, existing API integrations emphasize diversity over the precision and error avoidance essential in real-world customer service scenarios. To address these issues, we propose an LLM agent named CHOPS (CHat with custOmer Profile in existing System), designed to: (1) efficiently utilize existing databases or systems for accessing user information or interacting with these systems following existing guidelines; (2) provide accurate and reasonable responses or carry out required operations in the system while avoiding harmful operations; and (3) leverage a combination of small and large LLMs to achieve satisfying performance at a reasonable inference cost. We introduce a practical dataset, the CPHOS-dataset, which includes a database, guiding files, and QA pairs collected from CPHOS, an online platform that facilitates the organization of simulated Physics Olympiads for high school teachers and students. We have conducted extensive experiments to validate the performance of our proposed CHOPS architecture using the CPHOS-dataset, with the aim of demonstrating how LLMs can enhance or serve as alternatives to human customer service. Our code and dataset will be open-sourced soon.
• 摘要：企业和软件平台越来越多地转向大型语言模型 (LLM)，例如 GPT-3.5、GPT-4、GLM-3 和 LLaMa-2，以提供文件访问聊天帮助或作为客户服务的推理代理。然而，当前基于法学硕士的客户服务模型与客户档案的整合有限，并且缺乏有效服务所需的运营能力。此外，现有的 API 集成强调多样性，而不是现实客户服务场景中至关重要的精度和避免错误。为了解决这些问题，我们提出了一种名为 CHOPS（现有系统中与客户资料的 CHat）的 LLM 代理，旨在：（1）有效利用现有数据库或系统来访问用户信息或遵循现有准则与这些系统进行交互； （二）在系统中提供准确、合理的响应或进行所需的操作，同时避免有害操作； (3) 利用小型和大型法学硕士的组合，以合理的推理成本实现令人满意的性能。我们介绍了一个实用的数据集，即 CPHOS 数据集，其中包括从 CPHOS 收集的数据库、指导文件和 QA 对，CPHOS 是一个在线平台，有助于为高中教师和学生组织模拟物理奥林匹克竞赛。我们使用 CPHOS 数据集进行了广泛的实验来验证我们提出的 CHOPS 架构的性能，目的是展示法学硕士如何增强或替代人类客户服务。我们的代码和数据集将很快开源。

Title: Fairness in Large Language Models: A Taxonomic Survey

• Authors: Zhibo Chu, Zichong Wang, Wenbin Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01349
• Pdf URL: https://arxiv.org/pdf/2404.01349
• Copy Paste: [[2404.01349]] Fairness in Large Language Models: A Taxonomic Survey(https://arxiv.org/abs/2404.01349)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable success across various domains. However, despite their promising performance in numerous real-world applications, most of these algorithms lack fairness considerations. Consequently, they may lead to discriminatory outcomes against certain communities, particularly marginalized populations, prompting extensive study in fair LLMs. On the other hand, fairness in LLMs, in contrast to fairness in traditional machine learning, entails exclusive backgrounds, taxonomies, and fulfillment techniques. To this end, this survey presents a comprehensive overview of recent advances in the existing literature concerning fair LLMs. Specifically, a brief introduction to LLMs is provided, followed by an analysis of factors contributing to bias in LLMs. Additionally, the concept of fairness in LLMs is discussed categorically, summarizing metrics for evaluating bias in LLMs and existing algorithms for promoting fairness. Furthermore, resources for evaluating bias in LLMs, including toolkits and datasets, are summarized. Finally, existing research challenges and open questions are discussed.
• 摘要：大型语言模型 (LLM) 在各个领域都取得了显着的成功。然而，尽管它们在许多实际应用中表现良好，但大多数算法缺乏公平性考虑。因此，它们可能会导致针对某些社区，特别是边缘化人群的歧视性结果，从而促使公平的法学硕士进行广泛的研究。另一方面，法学硕士的公平性与传统机器学习的公平性相反，需要独特的背景、分类法和实现技术。为此，本次调查全面概述了有关公平法学硕士的现有文献的最新进展。具体来说，对法学硕士进行了简要介绍，然后分析了导致法学硕士偏见的因素。此外，还对法学硕士公平性的概念进行了分类讨论，总结了评估法学硕士偏见的指标以及促进公平性的现有算法。此外，还总结了用于评估法学硕士偏差的资源，包括工具包和数据集。最后，讨论了现有的研究挑战和悬而未决的问题。

Title: LLM Attributor: Interactive Visual Attribution for LLM Generation

• Authors: Seongmin Lee, Zijie J. Wang, Aishwarya Chakravarthy, Alec Helbling, ShengYun Peng, Mansi Phute, Duen Horng Chau, Minsuk Kahng
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01361
• Pdf URL: https://arxiv.org/pdf/2404.01361
• Copy Paste: [[2404.01361]] LLM Attributor: Interactive Visual Attribution for LLM Generation(https://arxiv.org/abs/2404.01361)
• Keywords: language model, llm
• Abstract: While large language models (LLMs) have shown remarkable capability to generate convincing text across diverse domains, concerns around its potential risks have highlighted the importance of understanding the rationale behind text generation. We present LLM Attributor, a Python library that provides interactive visualizations for training data attribution of an LLM's text generation. Our library offers a new way to quickly attribute an LLM's text generation to training data points to inspect model behaviors, enhance its trustworthiness, and compare model-generated text with user-provided text. We describe the visual and interactive design of our tool and highlight usage scenarios for LLaMA2 models fine-tuned with two different datasets: online articles about recent disasters and finance-related question-answer pairs. Thanks to LLM Attributor's broad support for computational notebooks, users can easily integrate it into their workflow to interactively visualize attributions of their models. For easier access and extensibility, we open-source LLM Attributor at https://github.com/poloclub/ LLM-Attribution. The video demo is available at https://youtu.be/mIG2MDQKQxM.
• 摘要：虽然大型语言模型 (LLM) 已显示出在不同领域生成令人信服的文本的卓越能力，但对其潜在风险的担忧凸显了理解文本生成背后原理的重要性。我们推出了 LLM Attributor，这是一个 Python 库，它为 LLM 文本生成的训练数据归因提供交互式可视化。我们的库提供了一种新方法，可以快速将法学硕士的文本生成归因于训练数据点，以检查模型行为、增强其可信度，并将模型生成的文本与用户提供的文本进行比较。我们描述了工具的视觉和交互设计，并重点介绍了使用两个不同数据集进行微调的 LLaMA2 模型的使用场景：有关近期灾难的在线文章和与金融相关的问答对。由于 LLM Attributor 对计算笔记本的广泛支持，用户可以轻松地将其集成到他们的工作流程中，以交互方式可视化模型的归因。为了更方便地访问和扩展，我们在 https://github.com/poloclub/LLM-Attribution 上开源了 LLM Attributor。视频演示可在 https://youtu.be/mIG2MDQKQxM 上获取。

Title: Developing Safe and Responsible Large Language Models -- A Comprehensive Framework

• Authors: Shaina Raza, Oluwanifemi Bamgbose, Shardul Ghuge, Fatemeh Tavakoli, Deepak John Reji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01399
• Pdf URL: https://arxiv.org/pdf/2404.01399
• Copy Paste: [[2404.01399]] Developing Safe and Responsible Large Language Models -- A Comprehensive Framework(https://arxiv.org/abs/2404.01399)
• Keywords: language model, llm
• Abstract: Given the growing concerns around the safety and risks of Large Language Models (LLMs), it is essential to develop methods for mitigating these issues. We introduce Safe and Responsible Large Language Model (SR$_{\text{LLM}}$) , a model designed to enhance the safety of language generation using LLMs. Our approach incorporates a comprehensive LLM safety risk taxonomy and utilizes a dataset annotated by experts that align with this taxonomy. SR$_{\text{LLM}}$ is designed to identify potentially unsafe content and produce benign variations. It employs instruction-based and parameter-efficient fine-tuning methods, making the model not only effective in enhancing safety but also resource-efficient and straightforward to adjust. Through our testing on five benchmark datasets and two proprietary datasets, we observed notable reductions in the generation of unsafe content. Moreover, following the implementation of safety measures, there was a significant improvement in the production of safe content. We detail our fine-tuning processes and how we benchmark safety for SR$_{\text{LLM}}$ with the community engagement and promote the responsible advancement of LLMs. All the data and code are available anonymous at https://github.com/shainarazavi/Safe-Responsible-LLM .
• 摘要：鉴于人们对大型语言模型 (LLM) 的安全性和风险的担忧日益增加，开发缓解这些问题的方法至关重要。我们引入安全且负责任的大型语言模型（SR$_{\text{LLM}}$），该模型旨在增强使用 LLM 的语言生成的安全性。我们的方法结合了全面的法学硕士安全风险分类法，并利用由符合该分类法的专家注释的数据集。 SR$_{\text{LLM}}$ 旨在识别潜在不安全内容并产生良性变化。它采用基于指令和参数高效的微调方法，使模型不仅可以有效增强安全性，而且资源高效且易于调整。通过对五个基准数据集和两个专有数据集的测试，我们观察到不安全内容的生成显着减少。而且，随着安全措施的落实，安全内容的生产也有了显着的提升。我们详细介绍了我们的微调流程以及如何通过社区参与来衡量 SR$_{\text{LLM}}$ 的安全性，并促进法学硕士负责任的进步。所有数据和代码均可在 https://github.com/shainarazavi/Safe-Responsible-LLM 上匿名获取。

Title: Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs

• Authors: Zheng Zhang, Fan Yang, Ziyan Jiang, Zheng Chen, Zhengyang Zhao, Chengyuan Ma, Liang Zhao, Yang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01430
• Pdf URL: https://arxiv.org/pdf/2404.01430
• Copy Paste: [[2404.01430]] Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs(https://arxiv.org/abs/2404.01430)
• Keywords: language model, llm, long context, prompt
• Abstract: Recent advances in large language models (LLMs) have enhanced their ability to process long input contexts. This development is particularly crucial for tasks that involve retrieving knowledge from an external datastore, which can result in long inputs. However, recent studies show a positional bias in LLMs, demonstrating varying performance depending on the location of useful information within the input sequence. In this study, we conduct extensive experiments to investigate the root causes of positional bias. Our findings indicate that the primary contributor to LLM positional bias stems from the inherent positional preferences of different models. We demonstrate that merely employing prompt-based solutions is inadequate for overcoming the positional preferences. To address this positional bias issue of a pre-trained LLM, we developed a Position-Aware Parameter Efficient Fine-Tuning (PAPEFT) approach which is composed of a data augmentation technique and a parameter efficient adapter, enhancing a uniform attention distribution across the input context. Our experiments demonstrate that the proposed approach effectively reduces positional bias, improving LLMs' effectiveness in handling long context sequences for various tasks that require externally retrieved knowledge.
• 摘要：大型语言模型 (LLM) 的最新进展增强了它们处理长输入上下文的能力。这种开发对于涉及从外部数据存储检索知识的任务尤其重要，这可能会导致较长的输入。然而，最近的研究表明法学硕士存在位置偏差，表现出不同的性能取决于输入序列中有用信息的位置。在这项研究中，我们进行了大量的实验来调查位置偏差的根本原因。我们的研究结果表明，法学硕士位置偏差的主要原因源于不同模型固有的位置偏好。我们证明，仅仅采用基于提示的解决方案不足以克服位置偏好。为了解决预训练 LLM 的位置偏差问题，我们开发了一种位置感知参数高效微调（PAPEFT）方法，该方法由数据增强技术和参数高效适配器组成，增强了输入中的均匀注意力分布语境。我们的实验表明，所提出的方法有效地减少了位置偏差，提高了法学硕士在处理需要外部检索知识的各种任务的长上下文序列时的有效性。

Title: Unveiling Divergent Inductive Biases of LLMs on Temporal Data

• Authors: Sindhu Kishore, Hangfeng He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01453
• Pdf URL: https://arxiv.org/pdf/2404.01453
• Copy Paste: [[2404.01453]] Unveiling Divergent Inductive Biases of LLMs on Temporal Data(https://arxiv.org/abs/2404.01453)
• Keywords: language model, gpt, llm, prompt
• Abstract: Unraveling the intricate details of events in natural language necessitates a subtle understanding of temporal dynamics. Despite the adeptness of Large Language Models (LLMs) in discerning patterns and relationships from data, their inherent comprehension of temporal dynamics remains a formidable challenge. This research meticulously explores these intrinsic challenges within LLMs, with a specific emphasis on evaluating the performance of GPT-3.5 and GPT-4 models in the analysis of temporal data. Employing two distinct prompt types, namely Question Answering (QA) format and Textual Entailment (TE) format, our analysis probes into both implicit and explicit events. The findings underscore noteworthy trends, revealing disparities in the performance of GPT-3.5 and GPT-4. Notably, biases toward specific temporal relationships come to light, with GPT-3.5 demonstrating a preference for "AFTER'' in the QA format for both implicit and explicit events, while GPT-4 leans towards "BEFORE''. Furthermore, a consistent pattern surfaces wherein GPT-3.5 tends towards "TRUE'', and GPT-4 exhibits a preference for "FALSE'' in the TE format for both implicit and explicit events. This persistent discrepancy between GPT-3.5 and GPT-4 in handling temporal data highlights the intricate nature of inductive bias in LLMs, suggesting that the evolution of these models may not merely mitigate bias but may introduce new layers of complexity.
• 摘要：用自然语言揭示事件的复杂细节需要对时间动态有微妙的理解。尽管大型语言模型（LLM）擅长从数据中辨别模式和关系，但它们对时间动态的固有理解仍然是一个巨大的挑战。这项研究细致地探讨了法学硕士内的这些内在挑战，特别强调评估 GPT-3.5 和 GPT-4 模型在时态数据分析中的性能。我们的分析采用两种不同的提示类型，即问答（QA）格式和文本蕴涵（TE）格式，探讨隐式和显式事件。研究结果强调了值得注意的趋势，揭示了 GPT-3.5 和 GPT-4 性能的差异。值得注意的是，对特定时间关系的偏见暴露出来，GPT-3.5 在 QA 格式中对隐式和显式事件表现出偏好“之后”，而 GPT-4 则倾向于“之前”。此外，一致的模式表面，其中 GPT-3.5 倾向于“TRUE”，而 GPT-4 在 TE 格式中对于隐式和显式事件都表现出对“FALSE”的偏好。 GPT-3.5 和 GPT-4 在处理时态数据方面的持续差异凸显了法学硕士中归纳偏差的复杂性，表明这些模型的演变可能不仅会减轻偏差，而且可能会引入新的复杂性层。

Title: Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs

• Authors: Pengda Wang, Zilin Xiao, Hanjie Chen, Frederick L. Oswald
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.01461
• Pdf URL: https://arxiv.org/pdf/2404.01461
• Copy Paste: [[2404.01461]] Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs(https://arxiv.org/abs/2404.01461)
• Keywords: language model, llm, prompt
• Abstract: Although large language models (LLMs) have demonstrated remarkable proficiency in understanding text and generating human-like text, they may exhibit biases acquired from training data in doing so. Specifically, LLMs may be susceptible to a common cognitive trap in human decision-making called the representativeness heuristic. This is a concept in psychology that refers to judging the likelihood of an event based on how closely it resembles a well-known prototype or typical example versus considering broader facts or statistical evidence. This work investigates the impact of the representativeness heuristic on LLM reasoning. We created REHEAT (Representativeness Heuristic AI Testing), a dataset containing a series of problems spanning six common types of representativeness heuristics. Experiments reveal that four LLMs applied to REHEAT all exhibited representativeness heuristic biases. We further identify that the model's reasoning steps are often incorrectly based on a stereotype rather than the problem's description. Interestingly, the performance improves when adding a hint in the prompt to remind the model of using its knowledge. This suggests the uniqueness of the representativeness heuristic compared to traditional biases. It can occur even when LLMs possess the correct knowledge while failing in a cognitive trap. This highlights the importance of future research focusing on the representativeness heuristic in model reasoning and decision-making and on developing solutions to address it.
• 摘要：尽管大型语言模型 (LLM) 在理解文本和生成类人文本方面表现出了卓越的能力，但它们在这样做时可能会表现出从训练数据中获得的偏差。具体来说，法学硕士可能容易受到人类决策中常见的认知陷阱（称为代表性启发法）的影响。这是心理学中的一个概念，指的是根据事件与众所周知的原型或典型例子的相似程度来判断事件的可能性，而不是考虑更广泛的事实或统计证据。这项工作研究了代表性启发式对法学硕士推理的影响。我们创建了 REHEAT（代表性启发式人工智能测试），这是一个包含一系列问题的数据集，涵盖六种常见类型的代表性启发式。实验表明，应用于 REHEAT 的四名法学硕士均表现出代表性启发式偏差。我们进一步发现模型的推理步骤通常错误地基于刻板印象而不是问题的描述。有趣的是，当在提示中添加提示以提醒模型使用其知识时，性能会提高。这表明与传统偏差相比代表性启发法的独特性。即使法学硕士拥有正确的知识，但陷入认知陷阱，这种情况也可能发生。这凸显了未来研究重点关注模型推理和决策中的代表性启发式以及开发解决方案的重要性。

Title: A Study on Scaling Up Multilingual News Framing Analysis

• Authors: Syeda Sabrina Akter, Antonios Anastasopoulos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01481
• Pdf URL: https://arxiv.org/pdf/2404.01481
• Copy Paste: [[2404.01481]] A Study on Scaling Up Multilingual News Framing Analysis(https://arxiv.org/abs/2404.01481)
• Keywords: language model, llm
• Abstract: Media framing is the study of strategically selecting and presenting specific aspects of political issues to shape public opinion. Despite its relevance to almost all societies around the world, research has been limited due to the lack of available datasets and other resources. This study explores the possibility of dataset creation through crowdsourcing, utilizing non-expert annotators to develop training corpora. We first extend framing analysis beyond English news to a multilingual context (12 typologically diverse languages) through automatic translation. We also present a novel benchmark in Bengali and Portuguese on the immigration and same-sex marriage domains. Additionally, we show that a system trained on our crowd-sourced dataset, combined with other existing ones, leads to a 5.32 percentage point increase from the baseline, showing that crowdsourcing is a viable option. Last, we study the performance of large language models (LLMs) for this task, finding that task-specific fine-tuning is a better approach than employing bigger non-specialized models.
• 摘要：媒体框架是一门研究战略性选择和呈现政治问题的具体方面以塑造公众舆论的学科。尽管它与世界各地几乎所有社会都有相关性，但由于缺乏可用的数据集和其他资源，研究受到限制。本研究探讨了通过众包创建数据集的可能性，利用非专家注释者来开发训练语料库。我们首先通过自动翻译将框架分析从英语新闻扩展到多语言环境（12 种不同类型的语言）。我们还以孟加拉语和葡萄牙语提出了关于移民和同性婚姻领域的新颖基准。此外，我们还表明，在我们的众包数据集上训练的系统与其他现有数据集相结合，比基线提高了 5.32 个百分点，这表明众包是一个可行的选择。最后，我们研究了大型语言模型（LLM）在该任务中的性能，发现针对特定任务的微调是比采用更大的非专业模型更好的方法。

Title: Set-Aligning Framework for Auto-Regressive Event Temporal Graph Generation

• Authors: Xingwei Tan, Yuxiang Zhou, Gabriele Pergola, Yulan He
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.01532
• Pdf URL: https://arxiv.org/pdf/2404.01532
• Copy Paste: [[2404.01532]] Set-Aligning Framework for Auto-Regressive Event Temporal Graph Generation(https://arxiv.org/abs/2404.01532)
• Keywords: language model, llm
• Abstract: Event temporal graphs have been shown as convenient and effective representations of complex temporal relations between events in text. Recent studies, which employ pre-trained language models to auto-regressively generate linearised graphs for constructing event temporal graphs, have shown promising results. However, these methods have often led to suboptimal graph generation as the linearised graphs exhibit set characteristics which are instead treated sequentially by language models. This discrepancy stems from the conventional text generation objectives, leading to erroneous penalisation of correct predictions caused by the misalignment of elements in target sequences. To address these challenges, we reframe the task as a conditional set generation problem, proposing a Set-aligning Framework tailored for the effective utilisation of Large Language Models (LLMs). The framework incorporates data augmentations and set-property regularisations designed to alleviate text generation loss penalties associated with the linearised graph edge sequences, thus encouraging the generation of more relation edges. Experimental results show that our framework surpasses existing baselines for event temporal graph generation. Furthermore, under zero-shot settings, the structural knowledge introduced through our framework notably improves model generalisation, particularly when the training examples available are limited.
• 摘要：事件时间图已被证明是文本中事件之间复杂时间关系的方便且有效的表示。最近的研究采用预先训练的语言模型自动回归生成线性图来构建事件时间图，已经显示出有希望的结果。然而，这些方法通常会导致图生成不理想，因为线性化图表现出集合特征，而这些特征是由语言模型顺序处理的。这种差异源于传统的文本生成目标，导致由于目标序列中的元素未对齐而导致对正确预测的错误惩罚。为了应对这些挑战，我们将任务重新定义为条件集生成问题，提出了一个专为有效利用大型语言模型（LLM）而定制的集对齐框架。该框架结合了数据增强和集合属性正则化，旨在减轻与线性化图边缘序列相关的文本生成损失惩罚，从而鼓励生成更多关系边缘。实验结果表明，我们的框架超越了事件时间图生成的现有基线。此外，在零样本设置下，通过我们的框架引入的结构知识显着提高了模型泛化能力，特别是当可用的训练示例有限时。

Title: Laying Anchors: Semantically Priming Numerals in Language Modeling

• Authors: Mandar Sharma, Rutuja Murlidhar Taware, Pravesh Koirala, Nikhil Muralidhar, Naren Ramakrishnan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01536
• Pdf URL: https://arxiv.org/pdf/2404.01536
• Copy Paste: [[2404.01536]] Laying Anchors: Semantically Priming Numerals in Language Modeling(https://arxiv.org/abs/2404.01536)
• Keywords: language model
• Abstract: Off-the-shelf pre-trained language models have become the de facto standard in NLP pipelines for a multitude of downstream tasks. However, the inability of these models to properly encode numerals limits their performance on tasks requiring numeric comprehension. We introduce strategies to semantically prime numerals in any corpus by generating anchors governed by the distribution of numerals in said corpus, thereby enabling mathematically grounded representations of these numeral tokens. We establish the superiority of our proposed techniques through evaluation on a range of numeracy tasks for both in-domain (seen) and out-domain (unseen) numerals. Further, we expand our empirical evaluations to numerals ranging from 1 to 10 billion, a significantly broader range compared to previous studies of the same nature, and we demonstrate significant improvements in the mathematical grounding of our learned embeddings.
• 摘要：现成的预训练语言模型已成为众多下游任务的 NLP 管道的事实上的标准。然而，这些模型无法正确编码数字，限制了它们在需要数字理解的任务上的性能。我们通过生成由所述语料库中数字分布控制的锚点来引入对任何语料库中的语义质数数字的策略，从而实现这些数字标记的数学基础表示。我们通过对域内（看到的）和域外（看不见的）数字的一系列计算任务进行评估来确定我们提出的技术的优越性。此外，我们将实证评估扩展到从 1 到 100 亿的数字，与之前相同性质的研究相比，范围要大得多，并且我们展示了学习嵌入的数学基础的显着改进。

Title: Octopus: On-device language model for function calling of software APIs

• Authors: Wei Chen, Zhiyuan Li, Mingyuan Ma
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.01549
• Pdf URL: https://arxiv.org/pdf/2404.01549
• Copy Paste: [[2404.01549]] Octopus: On-device language model for function calling of software APIs(https://arxiv.org/abs/2404.01549)
• Keywords: language model, gpt, llm
• Abstract: In the rapidly evolving domain of artificial intelligence, Large Language Models (LLMs) play a crucial role due to their advanced text processing and generation abilities. This study introduces a new strategy aimed at harnessing on-device LLMs in invoking software APIs. We meticulously compile a dataset derived from software API documentation and apply fine-tuning to LLMs with capacities of 2B, 3B and 7B parameters, specifically to enhance their proficiency in software API interactions. Our approach concentrates on refining the models' grasp of API structures and syntax, significantly enhancing the accuracy of API function calls. Additionally, we propose \textit{conditional masking} techniques to ensure outputs in the desired formats and reduce error rates while maintaining inference speeds. We also propose a novel benchmark designed to evaluate the effectiveness of LLMs in API interactions, establishing a foundation for subsequent research. Octopus, the fine-tuned model, is proved to have better performance than GPT-4 for the software APIs calling. This research aims to advance automated software development and API integration, representing substantial progress in aligning LLM capabilities with the demands of practical software engineering applications.
• 摘要：在快速发展的人工智能领域，大型语言模型 (LLM) 因其先进的文本处理和生成能力而发挥着至关重要的作用。本研究引入了一种新策略，旨在利用设备上的 LLM 来调用软件 API。我们精心编译源自软件 API 文档的数据集，并对具有 2B、3B 和 7B 参数能力的法学硕士进行微调，专门提高他们在软件 API 交互方面的熟练程度。我们的方法专注于完善模型对 API 结构和语法的掌握，显着提高 API 函数调用的准确性。此外，我们提出 \textit{条件屏蔽} 技术来确保以所需格式输出并降低错误率，同时保持推理速度。我们还提出了一个新颖的基准，旨在评估法学硕士在 API 交互中的有效性，为后续研究奠定基础。事实证明，经过微调的模型 Octopus 在软件 API 调用方面比 GPT-4 具有更好的性能。这项研究旨在推进自动化软件开发和 API 集成，代表着法学硕士能力与实际软件工程应用需求相结合方面取得了实质性进展。

Title: Evaluating Large Language Models Using Contrast Sets: An Experimental Approach

• Authors: Manish Sanwal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01569
• Pdf URL: https://arxiv.org/pdf/2404.01569
• Copy Paste: [[2404.01569]] Evaluating Large Language Models Using Contrast Sets: An Experimental Approach(https://arxiv.org/abs/2404.01569)
• Keywords: language model
• Abstract: In the domain of Natural Language Inference (NLI), especially in tasks involving the classification of multiple input texts, the Cross-Entropy Loss metric is widely employed as a standard for error measurement. However, this metric falls short in effectively evaluating a model's capacity to understand language entailments. In this study, we introduce an innovative technique for generating a contrast set for the Stanford Natural Language Inference (SNLI) dataset. Our strategy involves the automated substitution of verbs, adverbs, and adjectives with their synonyms to preserve the original meaning of sentences. This method aims to assess whether a model's performance is based on genuine language comprehension or simply on pattern recognition. We conducted our analysis using the ELECTRA-small model. The model achieved an accuracy of 89.9% on the conventional SNLI dataset but showed a reduced accuracy of 72.5% on our contrast set, indicating a substantial 17% decline. This outcome led us to conduct a detailed examination of the model's learning behaviors. Following this, we improved the model's resilience by fine-tuning it with a contrast-enhanced training dataset specifically designed for SNLI, which increased its accuracy to 85.5% on the contrast sets. Our findings highlight the importance of incorporating diverse linguistic expressions into datasets for NLI tasks. We hope that our research will encourage the creation of more inclusive datasets, thereby contributing to the development of NLI models that are both more sophisticated and effective.
• 摘要：在自然语言推理（NLI）领域，特别是在涉及多个输入文本分类的任务中，交叉熵损失度量被广泛用作错误测量的标准。然而，该指标无法有效评估模型理解语言蕴含的能力。在这项研究中，我们介绍了一种为斯坦福自然语言推理（SNLI）数据集生成对比集的创新技术。我们的策略涉及用同义词自动替换动词、副词和形容词，以保留句子的原始含义。该方法旨在评估模型的性能是基于真正的语言理解还是仅仅基于模式识别。我们使用 ELECTRA-small 模型进行分析。该模型在传统 SNLI 数据集上的准确率达到 89.9%，但在我们的对比集上显示准确率降低了 72.5%，表明大幅下降了 17%。这一结果促使我们对模型的学习行为进行了详细检查。随后，我们通过使用专为 SNLI 设计的对比度增强训练数据集进行微调，提高了模型的弹性，这将其在对比集上的准确率提高到了 85.5%。我们的研究结果强调了将不同语言表达纳入 NLI 任务数据集中的重要性。我们希望我们的研究能够鼓励创建更具包容性的数据集，从而有助于开发更复杂、更有效的 NLI 模型。

Title: BERT-Enhanced Retrieval Tool for Homework Plagiarism Detection System

• Authors: Jiarong Xian, Jibao Yuan, Peiwei Zheng, Dexian Chen
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.01582
• Pdf URL: https://arxiv.org/pdf/2404.01582
• Copy Paste: [[2404.01582]] BERT-Enhanced Retrieval Tool for Homework Plagiarism Detection System(https://arxiv.org/abs/2404.01582)
• Keywords: gpt
• Abstract: Text plagiarism detection task is a common natural language processing task that aims to detect whether a given text contains plagiarism or copying from other texts. In existing research, detection of high level plagiarism is still a challenge due to the lack of high quality datasets. In this paper, we propose a plagiarized text data generation method based on GPT-3.5, which produces 32,927 pairs of text plagiarism detection datasets covering a wide range of plagiarism methods, bridging the gap in this part of research. Meanwhile, we propose a plagiarism identification method based on Faiss with BERT with high efficiency and high accuracy. Our experiments show that the performance of this model outperforms other models in several metrics, including 98.86\%, 98.90%, 98.86%, and 0.9888 for Accuracy, Precision, Recall, and F1 Score, respectively. At the end, we also provide a user-friendly demo platform that allows users to upload a text library and intuitively participate in the plagiarism analysis.
• 摘要：文本抄袭检测任务是一种常见的自然语言处理任务，旨在检测给定文本是否包含抄袭或抄袭其他文本。在现有的研究中，由于缺乏高质量的数据集，高水平抄袭的检测仍然是一个挑战。在本文中，我们提出了一种基于GPT-3.5的抄袭文本数据生成方法，该方法产生了32,927对文本抄袭检测数据集，涵盖了广泛的抄袭方法，弥补了这部分研究的空白。同时，我们提出了一种基于Faiss with BERT的高效率、高精度的抄袭识别方法。我们的实验表明，该模型的性能在多个指标上优于其他模型，包括准确度、精确度、召回率和 F1 分数分别为 98.86%、98.90%、98.86% 和 0.9888。最后，我们还提供了一个人性化的演示平台，允许用户上传文本库并直观地参与抄袭分析。

Title: Hallucination Diversity-Aware Active Learning for Text Summarization

• Authors: Yu Xia, Xu Liu, Tong Yu, Sungchul Kim, Ryan A. Rossi, Anup Rao, Tung Mai, Shuai Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01588
• Pdf URL: https://arxiv.org/pdf/2404.01588
• Copy Paste: [[2404.01588]] Hallucination Diversity-Aware Active Learning for Text Summarization(https://arxiv.org/abs/2404.01588)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have shown propensity to generate hallucinated outputs, i.e., texts that are factually incorrect or unsupported. Existing methods for alleviating hallucinations typically require costly human annotations to identify and correct hallucinations in LLM outputs. Moreover, most of these methods focus on a specific type of hallucination, e.g., entity or token errors, which limits their effectiveness in addressing various types of hallucinations exhibited in LLM outputs. To our best knowledge, in this paper we propose the first active learning framework to alleviate LLM hallucinations, reducing costly human annotations of hallucination needed. By measuring fine-grained hallucinations from errors in semantic frame, discourse and content verifiability in text summarization, we propose HAllucination Diversity-Aware Sampling (HADAS) to select diverse hallucinations for annotations in active learning for LLM finetuning. Extensive experiments on three datasets and different backbone models demonstrate advantages of our method in effectively and efficiently mitigating LLM hallucinations.
• 摘要：大型语言模型 (LLM) 已显示出生成幻觉输出的倾向，即实际上不正确或不受支持的文本。现有的减轻幻觉的方法通常需要昂贵的人工注释来识别和纠正法学硕士输出中的幻觉。此外，这些方法大多数都专注于特定类型的幻觉，例如实体或令牌错误，这限制了它们在解决法学硕士输出中表现出的各种类型幻觉的有效性。据我们所知，在本文中，我们提出了第一个主动学习框架来减轻法学硕士的幻觉，减少所需的昂贵的人类幻觉注释。通过测量文本摘要中语义框架、话语和内容可验证性中的错误引起的细粒度幻觉，我们提出了 HAllucination 多样性感知采样（HADAS），以在 LLM 微调的主动学习中选择不同的幻觉进行注释。对三个数据集和不同骨干模型的广泛实验证明了我们的方法在有效且高效地缓解 LLM 幻觉方面的优势。

Title: Classifying Cancer Stage with Open-Source Clinical Large Language Models

• Authors: Chia-Hsuan Chang, Mary M. Lucas, Grace Lu-Yao, Christopher C. Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01589
• Pdf URL: https://arxiv.org/pdf/2404.01589
• Copy Paste: [[2404.01589]] Classifying Cancer Stage with Open-Source Clinical Large Language Models(https://arxiv.org/abs/2404.01589)
• Keywords: language model, llm, prompt
• Abstract: Cancer stage classification is important for making treatment and care management plans for oncology patients. Information on staging is often included in unstructured form in clinical, pathology, radiology and other free-text reports in the electronic health record system, requiring extensive work to parse and obtain. To facilitate the extraction of this information, previous NLP approaches rely on labeled training datasets, which are labor-intensive to prepare. In this study, we demonstrate that without any labeled training data, open-source clinical large language models (LLMs) can extract pathologic tumor-node-metastasis (pTNM) staging information from real-world pathology reports. Our experiments compare LLMs and a BERT-based model fine-tuned using the labeled data. Our findings suggest that while LLMs still exhibit subpar performance in Tumor (T) classification, with the appropriate adoption of prompting strategies, they can achieve comparable performance on Metastasis (M) classification and improved performance on Node (N) classification.
• 摘要：癌症分期分类对于制定肿瘤患者的治疗和护理管理计划非常重要。分期信息通常以非结构化形式包含在电子健康记录系统中的临床、病理学、放射学和其他自由文本报告中，需要大量的工作来解析和获取。为了促进这些信息的提取，以前的 NLP 方法依赖于标记的训练数据集，而这些数据集的准备工作是劳动密集型的。在这项研究中，我们证明，在没有任何标记的训练数据的情况下，开源临床大语言模型（LLM）可以从真实世界的病理报告中提取病理肿瘤-淋巴结-转移（pTNM）分期信息。我们的实验比较了 LLM 和使用标记数据进行微调的基于 BERT 的模型。我们的研究结果表明，虽然法学硕士在肿瘤 (T) 分类中的表现仍然不佳，但通过适当采用提示策略，他们可以在转移 (M) 分类上实现可比的表现，并在节点 (N) 分类上提高表现。

Title: Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Game

• Authors: Silin Du, Xiaowei Zhang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.01602
• Pdf URL: https://arxiv.org/pdf/2404.01602
• Copy Paste: [[2404.01602]] Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Game(https://arxiv.org/abs/2404.01602)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have exhibited memorable strategic behaviors in social deductive games. However, the significance of opinion leadership exhibited by LLM-based agents has been overlooked, which is crucial for practical applications in multi-agent and human-AI interaction settings. Opinion leaders are individuals who have a noticeable impact on the beliefs and behaviors of others within a social group. In this work, we employ the Werewolf game as a simulation platform to assess the opinion leadership of LLMs. The game features the role of the Sheriff, tasked with summarizing arguments and recommending decision options, and therefore serves as a credible proxy for an opinion leader. We develop a framework integrating the Sheriff role and devise two novel metrics for evaluation based on the critical characteristics of opinion leaders. The first metric measures the reliability of the opinion leader, and the second assesses the influence of the opinion leader on other players' decisions. We conduct extensive experiments to evaluate LLMs of different scales. In addition, we collect a Werewolf question-answering dataset (WWQA) to assess and enhance LLM's grasp of the game rules, and we also incorporate human participants for further analysis. The results suggest that the Werewolf game is a suitable test bed to evaluate the opinion leadership of LLMs and few LLMs possess the capacity for opinion leadership.
• 摘要：大语言模型（LLM）在社交演绎游戏中表现出了令人难忘的策略行为。然而，基于法学硕士的智能体所表现出的意见领导力的重要性却被忽视了，而这对于多智能体和人机交互环境中的实际应用至关重要。意见领袖是对社会群体中其他人的信仰和行为有显着影响的个人。在这项工作中，我们采用狼人游戏作为模拟平台来评估法学硕士的意见领导力。该游戏以警长的角色为特色，其任务是总结论点并推荐决策选项，因此可以作为意见领袖的可靠代理人。我们开发了一个整合警长角色的框架，并根据意见领袖的关键特征设计了两个新颖的评估指标。第一个指标衡量意见领袖的可靠性，第二个指标评估意见领袖对其他参与者决策的影响。我们进行了大量的实验来评估不同规模的法学硕士。此外，我们还收集了狼人问答数据集（WWQA）来评估和增强LLM对游戏规则的掌握，并且我们还纳入了人类参与者进行进一步分析。结果表明，狼人游戏是评估法学硕士意见领导力的合适测试平台，而很少有法学硕士具备意见领导能力。

Title: Transforming LLMs into Cross-modal and Cross-lingual RetrievalSystems

• Authors: Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego
• Subjects: cs.CL, cs.IR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.01616
• Pdf URL: https://arxiv.org/pdf/2404.01616
• Copy Paste: [[2404.01616]] Transforming LLMs into Cross-modal and Cross-lingual RetrievalSystems(https://arxiv.org/abs/2404.01616)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require speech data during LLM pre-training and can exploit LLM's multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data.
• 摘要：大型语言模型 (LLM) 是在纯文本数据上进行训练的，这些数据远远超出了具有配对语音和文本数据的语言。与此同时，基于双编码器（DE）的检索系统将查询和文档投影到同一嵌入空间中，并证明了它们在检索和双文本挖掘方面的成功。为了匹配多种语言的语音和文本，我们建议使用 LLM 来初始化多模态 DE 检索系统。与传统方法不同，我们的系统在LLM预训练期间不需要语音数据，并且可以利用LLM的多语言文本理解能力来匹配检索训练期间未见过的语言的语音和文本。我们基于 LLM 的多模态检索系统能够匹配 102 种语言的语音和文本，尽管只接受了 21 种语言的训练。我们的系统优于之前经过所有 102 种语言明确训练的系统。这些语言的 Recall@1 平均绝对提高了 10%。此外，我们的模型演示了跨语言语音和文本匹配，并通过现成的机器翻译数据进一步增强了这一点。

Title: NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps

• Authors: Kristina Gligoric, Myra Cheng, Lucia Zheng, Esin Durmus, Dan Jurafsky
• Subjects: cs.CL, cs.CY, cs.HC, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.01651
• Pdf URL: https://arxiv.org/pdf/2404.01651
• Copy Paste: [[2404.01651]] NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps(https://arxiv.org/abs/2404.01651)
• Keywords: language model, prompt
• Abstract: The use of words to convey speaker's intent is traditionally distinguished from the `mention' of words for quoting what someone said, or pointing out properties of a word. Here we show that computationally modeling this use-mention distinction is crucial for dealing with counterspeech online. Counterspeech that refutes problematic content often mentions harmful language but is not harmful itself (e.g., calling a vaccine dangerous is not the same as expressing disapproval of someone for calling vaccines dangerous). We show that even recent language models fail at distinguishing use from mention, and that this failure propagates to two key downstream tasks: misinformation and hate speech detection, resulting in censorship of counterspeech. We introduce prompting mitigations that teach the use-mention distinction, and show they reduce these errors. Our work highlights the importance of the use-mention distinction for NLP and CSS and offers ways to address it.
• 摘要：传统上，使用词语来传达说话者的意图与“提及”词语（用于引用某人所说的话或指出词语的属性）是有区别的。在这里，我们表明，对这种使用提及的区别进行计算建模对于处理在线反言论至关重要。反驳有问题内容的反言论通常会提到有害语言，但本身并无害（例如，称疫苗危险并不等同于表达不赞成某人称疫苗危险）。我们表明，即使是最近的语言模型也无法区分使用和提及，并且这种失败会传播到两个关键的下游任务：错误信息和仇恨言论检测，从而导致对反言论的审查。我们引入了提示性缓解措施来教导使用提及的区别，并表明它们可以减少这些错误。我们的工作强调了 NLP 和 CSS 中使用提及区别的重要性，并提供了解决该问题的方法。

Title: Release of Pre-Trained Models for the Japanese Language

• Authors: Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, Koh Mitsuda
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.01657
• Pdf URL: https://arxiv.org/pdf/2404.01657
• Copy Paste: [[2404.01657]] Release of Pre-Trained Models for the Japanese Language(https://arxiv.org/abs/2404.01657)
• Keywords: gpt
• Abstract: AI democratization aims to create a world in which the average person can utilize AI techniques. To achieve this goal, numerous research institutes have attempted to make their results accessible to the public. In particular, large pre-trained models trained on large-scale data have shown unprecedented potential, and their release has had a significant impact. However, most of the released models specialize in the English language, and thus, AI democratization in non-English-speaking communities is lagging significantly. To reduce this gap in AI access, we released Generative Pre-trained Transformer (GPT), Contrastive Language and Image Pre-training (CLIP), Stable Diffusion, and Hidden-unit Bidirectional Encoder Representations from Transformers (HuBERT) pre-trained in Japanese. By providing these models, users can freely interface with AI that aligns with Japanese cultural values and ensures the identity of Japanese culture, thus enhancing the democratization of AI. Additionally, experiments showed that pre-trained models specialized for Japanese can efficiently achieve high performance in Japanese tasks.
• 摘要：人工智能民主化旨在创造一个普通人可以利用人工智能技术的世界。为了实现这一目标，许多研究机构试图向公众公开他们的研究结果。特别是在大规模数据上训练的大型预训练模型显示出前所未有的潜力，它们的发布产生了重大影响。然而，大多数已发布的模型都专门针对英语，因此非英语社区的人工智能民主化明显滞后。为了缩小人工智能访问方面的差距，我们发布了日语预训练的生成式预训练 Transformer (GPT)、对比语言和图像预训练 (CLIP)、稳定扩散和来自 Transformers 的隐藏单元双向编码器表示 (HuBERT) 。通过提供这些模型，用户可以自由地与符合日本文化价值观的人工智能进行交互，并确保日本文化的身份，从而增强人工智能的民主化。此外，实验表明专门针对日语的预训练模型可以在日语任务中有效地实现高性能。

Title: CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models

• Authors: Xuechen Liang, Meiling Tao, Tianyu Shi, Yiting Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01663
• Pdf URL: https://arxiv.org/pdf/2404.01663
• Copy Paste: [[2404.01663]] CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models(https://arxiv.org/abs/2404.01663)
• Keywords: language model, gpt, llm, agent
• Abstract: Open large language models (LLMs) have significantly advanced the field of natural language processing, showcasing impressive performance across various tasks.Despite the significant advancements in LLMs, their effective operation still relies heavily on human input to accurately guide the dialogue flow, with agent tuning being a crucial optimization technique that involves human adjustments to the model for better response to such guidance.Addressing this dependency, our work introduces the TinyAgent model, trained on a meticulously curated high-quality dataset. We also present the Collaborative Multi-Agent Tuning (CMAT) framework, an innovative system designed to augment language agent capabilities through adaptive weight updates based on environmental feedback. This framework fosters collaborative learning and real-time adaptation among multiple intelligent agents, enhancing their context-awareness and long-term memory. In this research, we propose a new communication agent framework that integrates multi-agent systems with environmental feedback mechanisms, offering a scalable method to explore cooperative behaviors. Notably, our TinyAgent-7B model exhibits performance on par with GPT-3.5, despite having fewer parameters, signifying a substantial improvement in the efficiency and effectiveness of LLMs.
• 摘要：开放大语言模型 (LLM) 显着推进了自然语言处理领域的发展，在各种任务中展示了令人印象深刻的性能。尽管 LLM 取得了显着进步，但其有效操作仍然在很大程度上依赖于人类输入来准确指导对话流，并通过代理调整这是一项关键的优化技术，涉及对模型的人工调整，以便更好地响应此类指导。为了解决这种依赖性，我们的工作引入了 TinyAgent 模型，该模型在精心策划的高质量数据集上进行了训练。我们还提出了协作多智能体调整（CMAT）框架，这是一个创新系统，旨在通过基于环境反馈的自适应权重更新来增强语言智能体的能力。该框架促进多个智能体之间的协作学习和实时适应，增强它们的情境意识和长期记忆。在这项研究中，我们提出了一种新的通信代理框架，它将多代理系统与环境反馈机制集成在一起，提供了一种可扩展的方法来探索合作行为。值得注意的是，我们的 TinyAgent-7B 模型尽管参数较少，但表现出与 GPT-3.5 相当的性能，这意味着 LLM 的效率和有效性有了显着提高。

Title: METAL: Towards Multilingual Meta-Evaluation

• Authors: Rishav Hada, Varun Gumma, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01667
• Pdf URL: https://arxiv.org/pdf/2404.01667
• Copy Paste: [[2404.01667]] METAL: Towards Multilingual Meta-Evaluation(https://arxiv.org/abs/2404.01667)
• Keywords: language model, gpt, llm
• Abstract: With the rising human-like precision of Large Language Models (LLMs) in numerous tasks, their utilization in a variety of real-world applications is becoming more prevalent. Several studies have shown that LLMs excel on many standard NLP benchmarks. However, it is challenging to evaluate LLMs due to test dataset contamination and the limitations of traditional metrics. Since human evaluations are difficult to collect, there is a growing interest in the community to use LLMs themselves as reference-free evaluators for subjective metrics. However, past work has shown that LLM-based evaluators can exhibit bias and have poor alignment with human judgments. In this study, we propose a framework for an end-to-end assessment of LLMs as evaluators in multilingual scenarios. We create a carefully curated dataset, covering 10 languages containing native speaker judgments for the task of summarization. This dataset is created specifically to evaluate LLM-based evaluators, which we refer to as meta-evaluation (METAL). We compare the performance of LLM-based evaluators created using GPT-3.5-Turbo, GPT-4, and PaLM2. Our results indicate that LLM-based evaluators based on GPT-4 perform the best across languages, while GPT-3.5-Turbo performs poorly. Additionally, we perform an analysis of the reasoning provided by LLM-based evaluators and find that it often does not match the reasoning provided by human judges.
• 摘要：随着大型语言模型 (LLM) 在众多任务中的类人精度不断提高，它们在各种现实应用中的使用变得越来越普遍。多项研究表明，法学硕士在许多标准 NLP 基准测试中表现出色。然而，由于测试数据集污染和传统指标的局限性，评估法学硕士具有挑战性。由于人类评估很难收集，因此社区越来越有兴趣使用法学硕士本身作为主观指标的无参考评估器。然而，过去的工作表明，基于法学硕士的评估者可能会表现出偏见，并且与人类判断的一致性较差。在这项研究中，我们提出了一个框架，用于对法学硕士作为多语言场景中的评估者进行端到端评估。我们创建了一个精心策划的数据集，涵盖 10 种语言，其中包含母语人士对摘要任务的判断。该数据集是专门为评估基于 LLM 的评估器而创建的，我们将其称为元评估 (METAL)。我们比较了使用 GPT-3.5-Turbo、GPT-4 和 PaLM2 创建的基于 LLM 的评估器的性能。我们的结果表明，基于 GPT-4 的 LLM 评估器在跨语言中表现最佳，而 GPT-3.5-Turbo 表现较差。此外，我们对基于法学硕士的评估者提供的推理进行了分析，发现它通常与人类法官提供的推理不匹配。

Title: On the Role of Summary Content Units in Text Summarization Evaluation

• Authors: Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01701
• Pdf URL: https://arxiv.org/pdf/2404.01701
• Copy Paste: [[2404.01701]] On the Role of Summary Content Units in Text Summarization Evaluation(https://arxiv.org/abs/2404.01701)
• Keywords: language model
• Abstract: At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.
• 摘要：文本摘要金字塔评估方法的核心在于人工书写的摘要内容单元 (SCU)。这些 SCU 是简洁的句子，将摘要分解为小事实。此类 SCU 可用于判断候选摘要的质量，可能通过自然语言推理 (NLI) 系统部分自动化。有趣的是，为了完全自动化金字塔评估，Zhang 和 Bansal（2021）表明，SCU 可以通过自动生成的语义角色三元组（STU）来近似。然而，目前有几个问题缺乏答案，特别是： i) 是否有其他可以提供优势的近似 SCU 的方法？ ii) 在什么条件下，SCU（或其近似值）提供最大价值？在这项工作中，我们研究了两种近似 SCU 的新颖策略：分别从 AMR 含义表示 (SMU) 和大型语言模型 (SGU) 生成 SCU 近似值。我们发现，虽然 STU 和 SMU 具有竞争力，但 SGU 实现了最佳近似质量。我们还通过简单的句子分解基线 (SSU) 表明，SCU（及其近似值）在对简短摘要进行排名时提供最大价值，但在对系统或较长摘要进行排名时可能没有多大帮助。

Title: Self-Improvement Programming for Temporal Knowledge Graph Question Answering

• Authors: Zhuo Chen, Zhao Zhang, Zixuan Li, Fei Wang, Yutao Zeng, Xiaolong Jin, Yongjun Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01720
• Pdf URL: https://arxiv.org/pdf/2404.01720
• Copy Paste: [[2404.01720]] Self-Improvement Programming for Temporal Knowledge Graph Question Answering(https://arxiv.org/abs/2404.01720)
• Keywords: language model, llm
• Abstract: Temporal Knowledge Graph Question Answering (TKGQA) aims to answer questions with temporal intent over Temporal Knowledge Graphs (TKGs). The core challenge of this task lies in understanding the complex semantic information regarding multiple types of time constraints (e.g., before, first) in questions. Existing end-to-end methods implicitly model the time constraints by learning time-aware embeddings of questions and candidate answers, which is far from understanding the question comprehensively. Motivated by semantic-parsing-based approaches that explicitly model constraints in questions by generating logical forms with symbolic operators, we design fundamental temporal operators for time constraints and introduce a novel self-improvement Programming method for TKGQA (Prog-TQA). Specifically, Prog-TQA leverages the in-context learning ability of Large Language Models (LLMs) to understand the combinatory time constraints in the questions and generate corresponding program drafts with a few examples given. Then, it aligns these drafts to TKGs with the linking module and subsequently executes them to generate the answers. To enhance the ability to understand questions, Prog-TQA is further equipped with a self-improvement strategy to effectively bootstrap LLMs using high-quality self-generated drafts. Extensive experiments demonstrate the superiority of the proposed Prog-TQA on MultiTQ and CronQuestions datasets, especially in the Hits@1 metric.
• 摘要：时态知识图问答（TKGQA）旨在通过时态知识图（TKG）回答具有时间意图的问题。该任务的核心挑战在于理解问题中有关多种类型时间限制（例如，之前、首先）的复杂语义信息。现有的端到端方法通过学习问题和候选答案的时间感知嵌入来隐式建模时间约束，这远远不能全面理解问题。基于语义解析的方法通过使用符号运算符生成逻辑形式来显式建模问题中的约束，我们为时间约束设计了基本的时间运算符，并引入了一种新颖的 TKGQA (Prog-TQA) 自我改进编程方法。具体来说，Prog-TQA 利用大型语言模型（LLM）的上下文学习能力来理解问题中的组合时间限制，并通过给出的一些示例生成相应的程序草稿。然后，它使用链接模块将这些草稿与 TKG 对齐，并随后执行它们以生成答案。为了增强理解问题的能力，Prog-TQA进一步配备了自我改进策略，使用高质量的自行生成的草稿来有效引导LLM。大量实验证明了所提出的 Prog-TQA 在 MultiTQ 和 CronQuestions 数据集上的优越性，特别是在 Hits@1 指标上。

Title: Sentence-level Media Bias Analysis with Event Relation Graph

• Authors: Yuanyuan Lei, Ruihong Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01722
• Pdf URL: https://arxiv.org/pdf/2404.01722
• Copy Paste: [[2404.01722]] Sentence-level Media Bias Analysis with Event Relation Graph(https://arxiv.org/abs/2404.01722)
• Keywords: language model
• Abstract: Media outlets are becoming more partisan and polarized nowadays. In this paper, we identify media bias at the sentence level, and pinpoint bias sentences that intend to sway readers' opinions. As bias sentences are often expressed in a neutral and factual way, considering broader context outside a sentence can help reveal the bias. In particular, we observe that events in a bias sentence need to be understood in associations with other events in the document. Therefore, we propose to construct an event relation graph to explicitly reason about event-event relations for sentence-level bias identification. The designed event relation graph consists of events as nodes and four common types of event relations: coreference, temporal, causal, and subevent relations. Then, we incorporate event relation graph for bias sentences identification in two steps: an event-aware language model is built to inject the events and event relations knowledge into the basic language model via soft labels; further, a relation-aware graph attention network is designed to update sentence embedding with events and event relations information based on hard labels. Experiments on two benchmark datasets demonstrate that our approach with the aid of event relation graph improves both precision and recall of bias sentence identification.
• 摘要：如今，媒体机构变得更加党派和两极分化。在本文中，我们在句子层面识别媒体偏见，并查明意图影响读者观点的偏见句子。由于偏见句子通常以中立和事实的方式表达，因此考虑句子之外更广泛的上下文可以帮助揭示偏见。特别是，我们观察到偏见句子中的事件需要与文档中的其他事件相关联来理解。因此，我们建议构建一个事件关系图来明确推理事件与事件之间的关系，以进行句子级偏差识别。设计的事件关系图由作为节点的事件和四种常见类型的事件关系组成：共指关系、时间关系、因果关系和子事件关系。然后，我们通过两个步骤结合事件关系图来识别偏见句子：建立事件感知语言模型，通过软标签将事件和事件关系知识注入到基本语言模型中；此外，设计了一个关系感知图注意网络，以基于硬标签更新事件和事件关系信息的句子嵌入。在两个基准数据集上的实验表明，我们借助事件关系图的方法提高了偏见句子识别的精度和召回率。

Title: Octopus v2: On-device language model for super agent

• Authors: Wei Chen, Zhiyuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01744
• Pdf URL: https://arxiv.org/pdf/2404.01744
• Copy Paste: [[2404.01744]] Octopus v2: On-device language model for super agent(https://arxiv.org/abs/2404.01744)
• Keywords: language model, gpt, agent
• Abstract: Language models have shown effectiveness in a variety of software applications, particularly in tasks related to automatic workflow. These models possess the crucial ability to call functions, which is essential in creating AI agents. Despite the high performance of large-scale language models in cloud environments, they are often associated with concerns over privacy and cost. Current on-device models for function calling face issues with latency and accuracy. Our research presents a new method that empowers an on-device model with 2 billion parameters to surpass the performance of GPT-4 in both accuracy and latency, and decrease the context length by 95\%. When compared to Llama-7B with a RAG-based function calling mechanism, our method enhances latency by 35-fold. This method reduces the latency to levels deemed suitable for deployment across a variety of edge devices in production environments, aligning with the performance requisites for real-world applications.
• 摘要：语言模型已在各种软件应用程序中显示出有效性，特别是在与自动工作流程相关的任务中。这些模型拥有调用函数的关键能力，这对于创建人工智能代理至关重要。尽管云环境中的大规模语言模型具有高性能，但它们通常与隐私和成本问题相关。当前用于函数调用的设备上模型面临延迟和准确性问题。我们的研究提出了一种新方法，使具有 20 亿个参数的设备上模型能够在准确性和延迟方面超越 GPT-4 的性能，并将上下文长度减少 95%。与具有基于 RAG 的函数调用机制的 Llama-7B 相比，我们的方法将延迟提高了 35 倍。此方法将延迟降低到适合在生产环境中的各种边缘设备上部署的水平，从而符合实际应用程序的性能要求。

Title: M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

• Authors: Gaurish Thakkar, Sherzod Hakimov, Marko Tadić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01753
• Pdf URL: https://arxiv.org/pdf/2404.01753
• Copy Paste: [[2404.01753]] M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets(https://arxiv.org/abs/2404.01753)
• Keywords: language model
• Abstract: In recent years, multimodal natural language processing, aimed at learning from diverse data types, has garnered significant attention. However, there needs to be more clarity when it comes to analysing multimodal tasks in multi-lingual contexts. While prior studies on sentiment analysis of tweets have predominantly focused on the English language, this paper addresses this gap by transforming an existing textual Twitter sentiment dataset into a multimodal format through a straightforward curation process. Our work opens up new avenues for sentiment-related research within the research community. Additionally, we conduct baseline experiments utilising this augmented dataset and report the findings. Notably, our evaluations reveal that when comparing unimodal and multimodal configurations, using a sentiment-tuned large language model as a text encoder performs exceptionally well.
• 摘要：近年来，旨在从不同数据类型中学习的多模式自然语言处理引起了人们的广泛关注。然而，在分析多语言环境中的多模式任务时，需要更加清晰。虽然之前关于推文情感分析的研究主要集中在英语上，但本文通过简单的管理过程将现有的文本 Twitter 情感数据集转换为多模式格式，从而解决了这一差距。我们的工作为研究界内情感相关研究开辟了新途径。此外，我们利用这个增强的数据集进行基线实验并报告结果。值得注意的是，我们的评估表明，在比较单模态和多模态配置时，使用情感调整的大型语言模型作为文本编码器表现得非常好。

Title: Class-Incremental Few-Shot Event Detection

• Authors: Kailin Zhao, Xiaolong Jin, Long Bai, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01767
• Pdf URL: https://arxiv.org/pdf/2404.01767
• Copy Paste: [[2404.01767]] Class-Incremental Few-Shot Event Detection(https://arxiv.org/abs/2404.01767)
• Keywords: prompt
• Abstract: Event detection is one of the fundamental tasks in information extraction and knowledge graph. However, a realistic event detection system often needs to deal with new event classes constantly. These new classes usually have only a few labeled instances as it is time-consuming and labor-intensive to annotate a large number of unlabeled instances. Therefore, this paper proposes a new task, called class-incremental few-shot event detection. Nevertheless, this task faces two problems, i.e., old knowledge forgetting and new class overfitting. To solve these problems, this paper further presents a novel knowledge distillation and prompt learning based method, called Prompt-KD. Specifically, to handle the forgetting problem about old knowledge, Prompt-KD develops an attention based multi-teacher knowledge distillation framework, where the ancestor teacher model pre-trained on base classes is reused in all learning sessions, and the father teacher model derives the current student model via adaptation. On the other hand, in order to cope with the few-shot learning scenario and alleviate the corresponding new class overfitting problem, Prompt-KD is also equipped with a prompt learning mechanism. Extensive experiments on two benchmark datasets, i.e., FewEvent and MAVEN, demonstrate the superior performance of Prompt-KD.
• 摘要：事件检测是信息提取和知识图谱的基本任务之一。然而，现实的事件检测系统通常需要不断处理新的事件类。这些新类通常只有几个标记实例，因为注释大量未标记实例非常耗时且费力。因此，本文提出了一个新任务，称为类增量少样本事件检测。然而，该任务面临两个问题，即旧知识遗忘和新类过度拟合。为了解决这些问题，本文进一步提出了一种新颖的基于知识蒸馏和即时学习的方法，称为Prompt-KD。具体来说，为了解决旧知识的遗忘问题，Prompt-KD开发了一种基于注意力的多教师知识蒸馏框架，其中在基类上预训练的祖先教师模型在所有学习过程中被重用，而父亲教师模型推导出通过适应当前的学生模型。另一方面，为了应对few-shot学习场景，缓解相应的新类过拟合问题，Prompt-KD还配备了提示学习机制。在两个基准数据集 FewEvent 和 MAVEN 上进行的大量实验证明了 Prompt-KD 的优越性能。

Title: Auditing Large Language Models for Enhanced Text-Based Stereotype Detection and Probing-Based Bias Evaluation

• Authors: Zekun Wu, Sahan Bulathwela, Maria Perez-Ortiz, Adriano Soares Koshiyama
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01768
• Pdf URL: https://arxiv.org/pdf/2404.01768
• Copy Paste: [[2404.01768]] Auditing Large Language Models for Enhanced Text-Based Stereotype Detection and Probing-Based Bias Evaluation(https://arxiv.org/abs/2404.01768)
• Keywords: language model, gpt, llm, prompt
• Abstract: Recent advancements in Large Language Models (LLMs) have significantly increased their presence in human-facing Artificial Intelligence (AI) applications. However, LLMs could reproduce and even exacerbate stereotypical outputs from training data. This work introduces the Multi-Grain Stereotype (MGS) dataset, encompassing 51,867 instances across gender, race, profession, religion, and stereotypical text, collected by fusing multiple previously publicly available stereotype detection datasets. We explore different machine learning approaches aimed at establishing baselines for stereotype detection, and fine-tune several language models of various architectures and model sizes, presenting in this work a series of stereotypes classifier models for English text trained on MGS. To understand whether our stereotype detectors capture relevant features (aligning with human common sense) we utilise a variety of explanainable AI tools, including SHAP, LIME, and BertViz, and analyse a series of example cases discussing the results. Finally, we develop a series of stereotype elicitation prompts and evaluate the presence of stereotypes in text generation tasks with popular LLMs, using one of our best performing previously presented stereotypes detectors. Our experiments yielded several key findings: i) Training stereotype detectors in a multi-dimension setting yields better results than training multiple single-dimension classifiers.ii) The integrated MGS Dataset enhances both the in-dataset and cross-dataset generalisation ability of stereotype detectors compared to using the datasets separately. iii) There is a reduction in stereotypes in the content generated by GPT Family LLMs with newer versions.
• 摘要：大型语言模型 (LLM) 的最新进展显着增加了它们在面向人类的人工智能 (AI) 应用中的存在。然而，法学硕士可能会重现甚至加剧训练数据的刻板输出。这项工作引入了 Multi-Grain Stereotype (MGS) 数据集，包含跨性别、种族、职业、宗教和刻板文本的 51,867 个实例，这些实例是通过融合多个先前公开的刻板印象检测数据集收集的。我们探索了不同的机器学习方法，旨在建立刻板印象检测的基线，并对不同架构和模型大小的几种语言模型进行微调，在这项工作中提出了一系列在 MGS 上训练的英语文本的刻板印象分类器模型。为了了解我们的刻板印象检测器是否捕获了相关特征（符合人类常识），我们利用了各种可解释的人工智能工具，包括 SHAP、LIME 和 BertViz，并分析了一系列讨论结果的示例案例。最后，我们开发了一系列刻板印象引发提示，并使用我们之前提出的性能最佳的刻板印象检测器之一，使用流行的法学硕士来评估文本生成任务中刻板印象的存在。我们的实验得出了几个关键发现：i）在多维设置中训练刻板型检测器比训练多个单维分类器产生更好的结果。ii）集成的 MGS 数据集增强了刻板型检测器的数据集内和跨数据集泛化能力与单独使用数据集相比。 iii) 新版本的 GPT Family LLM 生成的内容中的刻板印象有所减少。

Title: Generative AI-Based Text Generation Methods Using Pre-Trained GPT-2 Model

• Authors: Rohit Pandey, Hetvi Waghela, Sneha Rakshit, Aparna Rangari, Anjali Singh, Rahul Kumar, Ratnadeep Ghosal, Jaydip Sen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01786
• Pdf URL: https://arxiv.org/pdf/2404.01786
• Copy Paste: [[2404.01786]] Generative AI-Based Text Generation Methods Using Pre-Trained GPT-2 Model(https://arxiv.org/abs/2404.01786)
• Keywords: gpt
• Abstract: This work delved into the realm of automatic text generation, exploring a variety of techniques ranging from traditional deterministic approaches to more modern stochastic methods. Through analysis of greedy search, beam search, top-k sampling, top-p sampling, contrastive searching, and locally typical searching, this work has provided valuable insights into the strengths, weaknesses, and potential applications of each method. Each text-generating method is evaluated using several standard metrics and a comparative study has been made on the performance of the approaches. Finally, some future directions of research in the field of automatic text generation are also identified.
• 摘要：这项工作深入研究了自动文本生成领域，探索了从传统的确定性方法到更现代的随机方法的各种技术。通过对贪婪搜索、波束搜索、top-k 采样、top-p 采样、对比搜索和局部典型搜索的分析，这项工作为每种方法的优点、缺点和潜在应用提供了宝贵的见解。每种文本生成方法都使用几个标准指标进行评估，并对这些方法的性能进行了比较研究。最后，还确定了自动文本生成领域的一些未来研究方向。

Title: PATCH -- Psychometrics-AssisTed benCHmarking of Large Language Models: A Case Study of Mathematics Proficiency

• Authors: Qixiang Fang, Daniel L. Oberski, Dong Nguyen
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.01799
• Pdf URL: https://arxiv.org/pdf/2404.01799
• Copy Paste: [[2404.01799]] PATCH -- Psychometrics-AssisTed benCHmarking of Large Language Models: A Case Study of Mathematics Proficiency(https://arxiv.org/abs/2404.01799)
• Keywords: language model, gpt, llm
• Abstract: Many existing benchmarks of large (multimodal) language models (LLMs) focus on measuring LLMs' academic proficiency, often with also an interest in comparing model performance with human test takers. While these benchmarks have proven key to the development of LLMs, they suffer from several limitations, including questionable measurement quality (e.g., Do they measure what they are supposed to in a reliable way?), lack of quality assessment on the item level (e.g., Are some items more important or difficult than others?) and unclear human population reference (e.g., To whom can the model be compared?). In response to these challenges, we propose leveraging knowledge from psychometrics - a field dedicated to the measurement of latent variables like academic proficiency - into LLM benchmarking. We make three primary contributions. First, we introduce PATCH: a novel framework for Psychometrics-AssisTed benCHmarking of LLMs. PATCH addresses the aforementioned limitations, presenting a new direction for LLM benchmark research. Second, we implement PATCH by measuring GPT-4 and Gemini-Pro-Vision's proficiency in 8th grade mathematics against 56 human populations. We show that adopting a psychometrics-based approach yields evaluation outcomes that diverge from those based on existing benchmarking practices. Third, we release 4 datasets to support measuring and comparing LLM proficiency in grade school mathematics and science against human populations.
• 摘要：许多现有的大型（多模式）语言模型（LLM）基准侧重于衡量 LLM 的学术熟练程度，通常也有兴趣将模型表现与人类考生进行比较。虽然这些基准已被证明对法学硕士的发展至关重要，但它们也受到一些限制，包括测量质量值得怀疑（例如，它们是否以可靠的方式测量了应有的内容？）、缺乏项目级别的质量评估（例如，项目级别的质量评估）。 ，某些项目是否比其他项目更重要或更困难？）以及不清楚的人群参考（例如，模型可以与谁进行比较？）。为了应对这些挑战，我们建议将心理测量学的知识（一个致力于测量学术能力等潜在变量的领域）的知识运用到法学硕士基准测试中。我们做出了三项主要贡献。首先，我们介绍 PATCH：一种用于法学硕士心理测量学辅助基准测试的新颖框架。 PATCH解决了上述局限性，为LLM基准研究提供了新方向。其次，我们通过测量 GPT-4 和 Gemini-Pro-Vision 对 56 个人的 8 年级数学熟练程度来实施 PATCH。我们表明，采用基于心理测量的方法产生的评估结果与基于现有基准测试实践的评估结果不同。第三，我们发布了 4 个数据集，以支持衡量和比较法学硕士在小学数学和科学方面的熟练程度与人类的水平。

Title: IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces

• Authors: Fajri Koto, Rahmad Mahendra, Nurul Aisyah, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01854
• Pdf URL: https://arxiv.org/pdf/2404.01854
• Copy Paste: [[2404.01854]] IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces(https://arxiv.org/abs/2404.01854)
• Keywords: language model, gpt
• Abstract: Although commonsense reasoning is greatly shaped by cultural and geographical factors, previous studies on language models have predominantly centered on English cultures, potentially resulting in an Anglocentric bias. In this paper, we introduce IndoCulture, aimed at understanding the influence of geographical factors on language model reasoning ability, with a specific emphasis on the diverse cultures found within eleven Indonesian provinces. In contrast to prior works that relied on templates (Yin et al., 2022) and online scrapping (Fung et al., 2024), we created IndoCulture by asking local people to manually develop the context and plausible options based on predefined topics. Evaluations of 23 language models reveal several insights: (1) even the best open-source model struggles with an accuracy of 53.2%, (2) models often provide more accurate predictions for specific provinces, such as Bali and West Java, and (3) the inclusion of location contexts enhances performance, especially in larger models like GPT-4, emphasizing the significance of geographical context in commonsense reasoning.
• 摘要：尽管常识推理在很大程度上受到文化和地理因素的影响，但之前对语言模型的研究主要集中在英语文化上，这可能导致以英语为中心的偏见。在本文中，我们介绍了 IndoCulture，旨在了解地理因素对语言模型推理能力的影响，特别强调印度尼西亚 11 个省份的多元文化。与之前依赖模板（Yin 等人，2022）和在线抓取（Fung 等人，2024）的作品相比，我们通过要求当地人根据预定义主题手动开发上下文和合理选项来创建 IndoCulture。对 23 种语言模型的评估揭示了一些见解：(1) 即使是最好的开源模型，准确率也只有 53.2%，(2) 模型通常能为特定省份提供更准确的预测，例如巴厘岛和西爪哇省，以及 (3 ）包含位置上下文可以提高性能，特别是在 GPT-4 这样的大型模型中，强调地理上下文在常识推理中的重要性。

Title: Poro 34B and the Blessing of Multilinguality

• Authors: Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, Väinö Hatanpää, Peter Sarlin, Sampo Pyysalo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01856
• Pdf URL: https://arxiv.org/pdf/2404.01856
• Copy Paste: [[2404.01856]] Poro 34B and the Blessing of Multilinguality(https://arxiv.org/abs/2404.01856)
• Keywords: language model
• Abstract: The pretraining of state-of-the-art large language models now requires trillions of words of text, which is orders of magnitude more than available for the vast majority of languages. While including text in more than one language is an obvious way to acquire more pretraining data, multilinguality is often seen as a curse, and most model training efforts continue to focus near-exclusively on individual large languages. We believe that multilinguality can be a blessing and that it should be possible to substantially improve over the capabilities of monolingual models for small languages through multilingual training. In this study, we introduce Poro 34B, a 34 billion parameter model trained for 1 trillion tokens of Finnish, English, and programming languages, and demonstrate that a multilingual training approach can produce a model that not only substantially advances over the capabilities of existing models for Finnish, but also excels in translation and is competitive in its class in generating English and programming languages. We release the model parameters, scripts, and data under open licenses at https://huggingface.co/LumiOpen/Poro-34B.
• 摘要：最先进的大型语言模型的预训练现在需要数万亿个单词的文本，这比绝大多数语言的可用数量级要多几个数量级。虽然包含多种语言的文本是获取更多预训练数据的一种明显方法，但多语言通常被视为一种诅咒，并且大多数模型训练工作仍然几乎完全专注于个别大语言。我们相信多语言是一种福祉，并且应该可以通过多语言训练大幅提高小语言单语言模型的能力。在这项研究中，我们介绍了 Poro 34B，这是一个针对芬兰语、英语和编程语言的 1 万亿个标记进行训练的 340 亿个参数模型，并证明多语言训练方法可以生成一个模型，该模型不仅可以大幅提高现有模型的能力芬兰语，但在翻译方面也很出色，并且在生成英语和编程语言方面在同类中具有竞争力。我们在开放许可下发布模型参数、脚本和数据：https://huggingface.co/LumiOpen/Poro-34B。

Title: Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey

• Authors: Philipp Mondorf, Barbara Plank
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01869
• Pdf URL: https://arxiv.org/pdf/2404.01869
• Copy Paste: [[2404.01869]] Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey(https://arxiv.org/abs/2404.01869)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have recently shown impressive performance on tasks involving reasoning, leading to a lively debate on whether these models possess reasoning capabilities similar to humans. However, despite these successes, the depth of LLMs' reasoning abilities remains uncertain. This uncertainty partly stems from the predominant focus on task performance, measured through shallow accuracy metrics, rather than a thorough investigation of the models' reasoning behavior. This paper seeks to address this gap by providing a comprehensive review of studies that go beyond task accuracy, offering deeper insights into the models' reasoning processes. Furthermore, we survey prevalent methodologies to evaluate the reasoning behavior of LLMs, emphasizing current trends and efforts towards more nuanced reasoning analyses. Our review suggests that LLMs tend to rely on surface-level patterns and correlations in their training data, rather than on genuine reasoning abilities. Additionally, we identify the need for further research that delineates the key differences between human and LLM-based reasoning. Through this survey, we aim to shed light on the complex reasoning processes within LLMs.
• 摘要：大型语言模型（LLM）最近在涉及推理的任务上表现出了令人印象深刻的性能，引发了关于这些模型是否具有类似于人类的推理能力的激烈争论。然而，尽管取得了这些成功，法学硕士的推理能力的深度仍然不确定。这种不确定性部分源于对任务绩效的主要关注，通过浅层准确性指标来衡量，而不是对模型推理行为的彻底调查。本文旨在通过对超越任务准确性的研究进行全面回顾，提供对模型推理过程的更深入见解来弥补这一差距。此外，我们调查了评估法学硕士推理行为的流行方法，强调当前的趋势和为更细致的推理分析所做的努力。我们的审查表明，法学硕士倾向于依赖训练数据中的表面模式和相关性，而不是真正的推理能力。此外，我们确定需要进一步研究来描述人类推理和基于法学硕士的推理之间的主要区别。通过这项调查，我们的目的是揭示法学硕士内复杂的推理过程。

Title: Activation Steering for Robust Type Prediction in CodeLLMs

• Authors: Francesca Lucchetti, Arjun Guha
• Subjects: cs.CL, cs.LG, cs.PL
• Abstract URL: https://arxiv.org/abs/2404.01903
• Pdf URL: https://arxiv.org/pdf/2404.01903
• Copy Paste: [[2404.01903]] Activation Steering for Robust Type Prediction in CodeLLMs(https://arxiv.org/abs/2404.01903)
• Keywords: llm
• Abstract: Contemporary LLMs pretrained on code are capable of succeeding at a wide variety of programming tasks. However, their performance is very sensitive to syntactic features, such as the names of variables and types, the structure of code, and presence of type hints. We contribute an inference-time technique to make CodeLLMs more robust to syntactic distractors that are semantically irrelevant. Our methodology relies on activation steering, which involves editing internal model activations to steer the model towards the correct prediction. We contribute a novel way to construct steering vectors by taking inspiration from mutation testing, which constructs minimal semantics-breaking code edits. In contrast, we construct steering vectors from semantics-preserving code edits. We apply our approach to the task of type prediction for the gradually typed languages Python and TypeScript. This approach corrects up to 90% of type mispredictions. Finally, we show that steering vectors calculated from Python activations reliably correct type mispredictions in TypeScript, and vice versa. This result suggests that LLMs may be learning to transfer knowledge of types across programming languages.
• 摘要：当代法学硕士接受过代码预训练，能够成功完成各种编程任务。然而，它们的性能对语法特征非常敏感，例如变量和类型的名称、代码的结构以及类型提示的存在。我们贡献了一种推理时间技术，使 CodeLLM 对语义无关的句法干扰项更加稳健。我们的方法依赖于激活引导，这涉及编辑内部模型激活以引导模型做出正确的预测。我们从突变测试中汲取灵感，贡献了一种构建引导向量的新颖方法，该方法构建了最少的破坏语义的代码编辑。相反，我们从保留语义的代码编辑中构建转向向量。我们将我们的方法应用于逐渐类型语言 Python 和 TypeScript 的类型预测任务。这种方法可以纠正高达 90% 的类型错误预测。最后，我们证明了根据 Python 激活计算出的转向向量能够可靠地纠正 TypeScript 中的类型错误预测，反之亦然。这一结果表明法学硕士可能正在学习跨编程语言迁移类型知识。

Title: Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack

• Authors: Ying Zhou, Ben He, Le Sun
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01907
• Pdf URL: https://arxiv.org/pdf/2404.01907
• Copy Paste: [[2404.01907]] Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack(https://arxiv.org/abs/2404.01907)
• Keywords: language model, llm
• Abstract: With the development of large language models (LLMs), detecting whether text is generated by a machine becomes increasingly challenging in the face of malicious use cases like the spread of false information, protection of intellectual property, and prevention of academic plagiarism. While well-trained text detectors have demonstrated promising performance on unseen test data, recent research suggests that these detectors have vulnerabilities when dealing with adversarial attacks such as paraphrasing. In this paper, we propose a framework for a broader class of adversarial attacks, designed to perform minor perturbations in machine-generated content to evade detection. We consider two attack settings: white-box and black-box, and employ adversarial learning in dynamic scenarios to assess the potential enhancement of the current detection model's robustness against such attacks. The empirical results reveal that the current detection models can be compromised in as little as 10 seconds, leading to the misclassification of machine-generated text as human-written content. Furthermore, we explore the prospect of improving the model's robustness over iterative adversarial learning. Although some improvements in model robustness are observed, practical applications still face significant challenges. These findings shed light on the future development of AI-text detectors, emphasizing the need for more accurate and robust detection methods.
• 摘要：随着大型语言模型（LLM）的发展，面对虚假信息传播、知识产权保护和防止学术抄袭等恶意用例，检测文本是否由机器生成变得越来越具有挑战性。虽然训练有素的文本检测器在未见过的测试数据上表现出了良好的性能，但最近的研究表明，这些检测器在处理释义等对抗性攻击时存在漏洞。在本文中，我们提出了一个针对更广泛类别的对抗性攻击的框架，旨在对机器生成的内容进行微小的扰动以逃避检测。我们考虑两种攻击设置：白盒和黑盒，并在动态场景中采用对抗性学习来评估当前检测模型针对此类攻击的鲁棒性的潜在增强。实证结果表明，当前的检测模型可能会在短短 10 秒内受到损害，导致机器生成的文本被错误分类为人类编写的内容。此外，我们探索了提高模型在迭代对抗性学习中的鲁棒性的前景。尽管观察到模型稳健性有所改善，但实际应用仍然面临重大挑战。这些发现揭示了人工智能文本检测器的未来发展，强调需要更准确和更强大的检测方法。

Title: A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution

• Authors: Bowen Ding, Qingkai Min, Shengkun Ma, Yingjie Li, Linyi Yang, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01921
• Pdf URL: https://arxiv.org/pdf/2404.01921
• Copy Paste: [[2404.01921]] A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution(https://arxiv.org/abs/2404.01921)
• Keywords: language model, llm
• Abstract: Based on Pre-trained Language Models (PLMs), event coreference resolution (ECR) systems have demonstrated outstanding performance in clustering coreferential events across documents. However, the existing system exhibits an excessive reliance on the `triggers lexical matching' spurious pattern in the input mention pair text. We formalize the decision-making process of the baseline ECR system using a Structural Causal Model (SCM), aiming to identify spurious and causal associations (i.e., rationales) within the ECR task. Leveraging the debiasing capability of counterfactual data augmentation, we develop a rationale-centric counterfactual data augmentation method with LLM-in-the-loop. This method is specialized for pairwise input in the ECR system, where we conduct direct interventions on triggers and context to mitigate the spurious association while emphasizing the causation. Our approach achieves state-of-the-art performance on three popular cross-document ECR benchmarks and demonstrates robustness in out-of-domain scenarios.
• 摘要：基于预先训练的语言模型 (PLM)，事件共指解析 (ECR) 系统在跨文档的共指事件聚类方面表现出了出色的性能。然而，现有系统过度依赖输入提及对文本中的“触发词汇匹配”虚假模式。我们使用结构因果模型（SCM）将基线 ECR 系统的决策过程形式化，旨在识别 ECR 任务中的虚假和因果关联（即基本原理）。利用反事实数据增强的去偏差能力，我们开发了一种基于 LLM-in-the-loop 的以理论为中心的反事实数据增强方法。该方法专门用于 ECR 系统中的成对输入，我们对触发器和上下文进行直接干预，以减轻虚假关联，同时强调因果关系。我们的方法在三个流行的跨文档 ECR 基准上实现了最先进的性能，并在域外场景中展示了鲁棒性。

Title: SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation

• Authors: Shasha Guo, Lizi Liao, Jing Zhang, Yanling Wang, Cuiping Li, Hong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01923
• Pdf URL: https://arxiv.org/pdf/2404.01923
• Copy Paste: [[2404.01923]] SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation(https://arxiv.org/abs/2404.01923)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Knowledge base question generation (KBQG) aims to generate natural language questions from a set of triplet facts extracted from KB. Existing methods have significantly boosted the performance of KBQG via pre-trained language models (PLMs) thanks to the richly endowed semantic knowledge. With the advance of pre-training techniques, large language models (LLMs) (e.g., GPT-3.5) undoubtedly possess much more semantic knowledge. Therefore, how to effectively organize and exploit the abundant knowledge for KBQG becomes the focus of our study. In this work, we propose SGSH--a simple and effective framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG. The framework incorporates "skeleton heuristics", which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions, encompassing essential elements like the question phrase and the auxiliary verb.More specifically, we devise an automatic data construction strategy leveraging ChatGPT to construct a skeleton training dataset, based on which we employ a soft prompting approach to train a BART model dedicated to generating the skeleton associated with each input. Subsequently, skeleton heuristics are encoded into the prompt to incentivize GPT-3.5 to generate desired questions. Extensive experiments demonstrate that SGSH derives the new state-of-the-art performance on the KBQG tasks.
• 摘要：知识库问题生成（KBQG）旨在根据从 KB 中提取的一组三元组事实生成自然语言问题。由于丰富的语义知识，现有方法通过预先训练的语言模型（PLM）显着提高了 KBQG 的性能。随着预训练技术的进步，大型语言模型（LLM）（例如 GPT-3.5）无疑拥有更多的语义知识。因此，如何有效地组织和利用KBQG丰富的知识成为我们研究的重点。在这项工作中，我们提出了 SGSH——一个简单而有效的框架，通过骨架启发式来刺激 GPT-3.5 以增强 KBQG。该框架采用了“骨架启发式”，它提供了与每个输入相关的更细粒度的指导，以刺激法学硕士生成最佳问题，包括问题短语和助动词等基本元素。更具体地说，我们设计了一种自动数据构建策略，利用ChatGPT 构建骨架训练数据集，在此基础上我们采用软提示方法来训练 BART 模型，专用于生成与每个输入相关的骨架。随后，骨架启发式被编码到提示中，以激励 GPT-3.5 生成所需的问题。大量实验表明，SGSH 在 KBQG 任务上获得了最先进的性能。

Title: Towards Better Understanding of Cybercrime: The Role of Fine-Tuned LLMs in Translation

• Authors: Veronica Valeros, Anna Širokova, Carlos Catania, Sebastian Garcia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01940
• Pdf URL: https://arxiv.org/pdf/2404.01940
• Copy Paste: [[2404.01940]] Towards Better Understanding of Cybercrime: The Role of Fine-Tuned LLMs in Translation(https://arxiv.org/abs/2404.01940)
• Keywords: language model, llm, chat
• Abstract: Understanding cybercrime communications is paramount for cybersecurity defence. This often involves translating communications into English for processing, interpreting, and generating timely intelligence. The problem is that translation is hard. Human translation is slow, expensive, and scarce. Machine translation is inaccurate and biased. We propose using fine-tuned Large Language Models (LLM) to generate translations that can accurately capture the nuances of cybercrime language. We apply our technique to public chats from the NoName057(16) Russian-speaking hacktivist group. Our results show that our fine-tuned LLM model is better, faster, more accurate, and able to capture nuances of the language. Our method shows it is possible to achieve high-fidelity translations and significantly reduce costs by a factor ranging from 430 to 23,000 compared to a human translator.
• 摘要：了解网络犯罪通信对于网络安全防御至关重要。这通常涉及将通信内容翻译成英语以进行处理、解释和生成及时的情报。问题是翻译很难。人工翻译速度慢、成本高且稀缺。机器翻译不准确且有偏见。我们建议使用经过微调的大型语言模型（LLM）来生成可以准确捕捉网络犯罪语言细微差别的翻译。我们将我们的技术应用于 NoName057(16) 俄语黑客活动组织的公共聊天。我们的结果表明，经过微调的 LLM 模型更好、更快、更准确，并且能够捕捉语言的细微差别。我们的方法表明，与人工翻译相比，可以实现高保真翻译并显着降低成本 430 至 23,000 倍。

Title: HyperCLOVA X Technical Report

• Authors: Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, Donghyun Kwak, Hanock Kwak, Se Jung Kwon, Bado Lee, Dongsoo Lee, Gichang Lee, Jooho Lee, Baeseong Park, Seongjin Shin, Joonsang Yu, Seolki Baek, Sumin Byeon, Eungsup Cho, Dooseok Choe, Jeesung Han, Youngkyun Jin, Hyein Jun, Jaeseung Jung, Chanwoong Kim, Jinhong Kim, Jinuk Kim, Dokyeong Lee, Dongwook Park, Jeong Min Sohn, Sujung Han, Jiae Heo, Sungju Hong, Mina Jeon, Hyunhoon Jung, Jungeun Jung, Wangkyo Jung, Chungjoon Kim, Hyeri Kim, Jonghyun Kim, Min Young Kim, Soeun Lee, Joonhee Park, Jieun Shin, Sojin Yang, Jungsoon Yoon, Hwaran Lee, Sanghwan Bae, Jeehwan Cha, Donghoon Ham, Youngki Hong, Yunki Hong, Myunggeun Ji, Yeguk Jin, Chansong Jo, Shinyoung Joo, Seunghwan Jung, et al. (316 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01954
• Pdf URL: https://arxiv.org/pdf/2404.01954
• Copy Paste: [[2404.01954]] HyperCLOVA X Technical Report(https://arxiv.org/abs/2404.01954)
• Keywords: language model, llm, chat
• Abstract: We introduce HyperCLOVA X, a family of large language models (LLMs) tailored to the Korean language and culture, along with competitive capabilities in English, math, and coding. HyperCLOVA X was trained on a balanced mix of Korean, English, and code data, followed by instruction-tuning with high-quality human-annotated datasets while abiding by strict safety guidelines reflecting our commitment to responsible AI. The model is evaluated across various benchmarks, including comprehensive reasoning, knowledge, commonsense, factuality, coding, math, chatting, instruction-following, and harmlessness, in both Korean and English. HyperCLOVA X exhibits strong reasoning capabilities in Korean backed by a deep understanding of the language and cultural nuances. Further analysis of the inherent bilingual nature and its extension to multilingualism highlights the model's cross-lingual proficiency and strong generalization ability to untargeted languages, including machine translation between several language pairs and cross-lingual inference tasks. We believe that HyperCLOVA X can provide helpful guidance for regions or countries in developing their sovereign LLMs.
• 摘要：我们推出 HyperCLOVA X，这是一个针对韩国语言和文化量身定制的大型语言模型 (LLM) 系列，以及英语、数学和编码方面的竞争能力。 HyperCLOVA X 接受了韩语、英语和代码数据的均衡组合训练，然后使用高质量的人工注释数据集进行指令调整，同时遵守严格的安全准则，反映了我们对负责任的 AI 的承诺。该模型通过韩语和英语的各种基准进行评估，包括综合推理、知识、常识、事实性、编码、数学、聊天、遵循指令和无害性。 HyperCLOVA X 凭借对语言和文化细微差别的深刻理解，展现出强大的韩语推理能力。对固有双语性质及其向多语言的扩展的进一步分析凸显了该模型的跨语言熟练程度和对非目标语言的强大泛化能力，包括多个语言对之间的机器翻译和跨语言推理任务。我们相信HyperCLOVA X可以为地区或国家发展其主权法学硕士提供有益的指导。

Title: Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4

• Authors: Dan Schumacher, Anthony Rios
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01961
• Pdf URL: https://arxiv.org/pdf/2404.01961
• Copy Paste: [[2404.01961]] Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4(https://arxiv.org/abs/2404.01961)
• Keywords: gpt, prompt, chain-of-thought
• Abstract: In this paper, we present our system for the SemEval Task 5, The Legal Argument Reasoning Task in Civil Procedure Challenge. Legal argument reasoning is an essential skill that all law students must master. Moreover, it is important to develop natural language processing solutions that can reason about a question given terse domain-specific contextual information. Our system explores a prompt-based solution using GPT4 to reason over legal arguments. We also evaluate an ensemble of prompting strategies, including chain-of-thought reasoning and in-context learning. Overall, our system results in a Macro F1 of .8095 on the validation dataset and .7315 (5th out of 21 teams) on the final test set. Code for this project is available at https://github.com/danschumac1/CivilPromptReasoningGPT4.
• 摘要：在本文中，我们介绍了 SemEval 任务 5（民事诉讼挑战中的法律论证推理任务）的系统。法律论证推理是所有法学院学生必须掌握的一项基本技能。此外，开发可以在给定简洁的特定领域上下文信息的情况下推理问题的自然语言处理解决方案也很重要。我们的系统探索了一种基于提示的解决方案，使用 GPT4 对法律争论进行推理。我们还评估了一系列提示策略，包括思维链推理和情境学习。总体而言，我们的系统在验证数据集上的 Macro F1 为 0.8095，在最终测试集上的 Macro F1 为 0.7315（21 个团队中的第五名）。该项目的代码可在 https://github.com/danschumac1/CivilPromptReasoningGPT4 上找到。

Title: Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models

• Authors: Stephan Linzbach, Dimitar Dimitrov, Laura Kallmeyer, Kilian Evang, Hajira Jabeen, Stefan Dietze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01992
• Pdf URL: https://arxiv.org/pdf/2404.01992
• Copy Paste: [[2404.01992]] Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models(https://arxiv.org/abs/2404.01992)
• Keywords: language model, prompt
• Abstract: Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge. One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects or objects. Typically, designing these prompts is a tedious task because small differences in syntax or semantics can have a substantial impact on knowledge retrieval performance. Simultaneously, evaluating the impact of either prompt syntax or information is challenging due to their interdependence. We designed CONPARE-LAMA - a dedicated probe, consisting of 34 million distinct prompts that facilitate comparison across minimal paraphrases. These paraphrases follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations. CONPARE-LAMA enables insights into the independent impact of either syntactical form or semantic information of paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge retrieval experiments using our probe reveal that prompts following clausal syntax have several desirable properties in comparison to appositive syntax: i) they are more useful when querying PLMs with a combination of supplementary information, ii) knowledge is more consistently recalled across different combinations of supplementary information, and iii) they decrease response uncertainty when retrieving known facts. In addition, range information can boost knowledge retrieval performance more than domain information, even though domain information is more reliably helpful across syntactic forms.
• 摘要：众所周知，预训练语言模型 (PLM) 包含各种知识。推断关系知识的一种方法是使用完形填空式提示，其中模型的任务是预测缺失的主题或对象。通常，设计这些提示是一项乏味的任务，因为语法或语义上的微小差异可能会对知识检索性能产生重大影响。同时，由于提示语法或信息的相互依赖性，评估它们的影响也具有挑战性。我们设计了 CONPARE-LAMA - 一个专用探针，由 3400 万个不同的提示组成，有助于跨最小释义进行比较。这些释义遵循统一的元模板，可以跨任意关系控制语法和语义的变化。 CONPARE-LAMA 能够深入了解释义的句法形式或语义信息对 PLM 知识检索性能的独立影响。使用我们的探针进行的广泛知识检索实验表明，与同位语法相比，分句语法的提示具有几个理想的属性：i）在使用补充信息组合查询 PLM 时它们更有用，ii）跨不同的组合可以更一致地回忆知识补充信息，以及 iii) 在检索已知事实时减少响应的不确定性。此外，范围信息比领域信息更能提高知识检索性能，尽管领域信息在跨句法形式中更可靠地提供帮助。

Title: Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts

• Authors: Zhuo Chen, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Kewei Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02022
• Pdf URL: https://arxiv.org/pdf/2404.02022
• Copy Paste: [[2404.02022]] Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts(https://arxiv.org/abs/2404.02022)
• Keywords: language model, long context, retrieval augmented generation
• Abstract: In the era of large language models, applying techniques such as Retrieval Augmented Generation can better address Open-Domain Question-Answering problems. Due to constraints including model sizes and computing resources, the length of context is often limited, and it becomes challenging to empower the model to cover overlong contexts while answering questions from open domains. This paper proposes a general and convenient method to covering longer contexts in Open-Domain Question-Answering tasks. It leverages a small encoder language model that effectively encodes contexts, and the encoding applies cross-attention with origin inputs. With our method, the origin language models can cover several times longer contexts while keeping the computing requirements close to the baseline. Our experiments demonstrate that after fine-tuning, there is improved performance across two held-in datasets, four held-out datasets, and also in two In Context Learning settings.
• 摘要：在大语言模型时代，应用检索增强生成等技术可以更好地解决开放域问答问题。由于模型大小和计算资源等限制，上下文的长度通常受到限制，使得模型能够覆盖超长的上下文，同时回答开放领域的问题变得具有挑战性。本文提出了一种通用且方便的方法来覆盖开放域问答任务中的较长上下文。它利用一个小型编码器语言模型来有效地编码上下文，并且编码应用与原始输入的交叉注意力。使用我们的方法，原始语言模型可以覆盖数倍长的上下文，同时保持计算要求接近基线。我们的实验表明，经过微调后，两个保留数据集、四个保留数据集以及两个上下文学习设置的性能都有所提高。

Title: MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages

• Authors: Daryna Dementieva, Nikolay Babakov, Alexander Panchenko
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02037
• Pdf URL: https://arxiv.org/pdf/2404.02037
• Copy Paste: [[2404.02037]] MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages(https://arxiv.org/abs/2404.02037)
• Keywords: language model, llm
• Abstract: Text detoxification is a textual style transfer (TST) task where a text is paraphrased from a toxic surface form, e.g. featuring rude words, to the neutral register. Recently, text detoxification methods found their applications in various task such as detoxification of Large Language Models (LLMs) (Leong et al., 2023; He et al., 2024; Tang et al., 2023) and toxic speech combating in social networks (Deng et al., 2023; Mun et al., 2023; Agarwal et al., 2023). All these applications are extremely important to ensure safe communication in modern digital worlds. However, the previous approaches for parallel text detoxification corpora collection -- ParaDetox (Logacheva et al., 2022) and APPADIA (Atwell et al., 2022) -- were explored only in monolingual setup. In this work, we aim to extend ParaDetox pipeline to multiple languages presenting MultiParaDetox to automate parallel detoxification corpus collection for potentially any language. Then, we experiment with different text detoxification models -- from unsupervised baselines to LLMs and fine-tuned models on the presented parallel corpora -- showing the great benefit of parallel corpus presence to obtain state-of-the-art text detoxification models for any language.
• 摘要：文本解毒是一项文本风格迁移（TST）任务，其中文本是从有毒的表面形式（例如，以粗鲁的言语为特色，以中立的方式进行。最近，文本去毒方法在各种任务中得到了应用，例如大型语言模型（LLM）的去毒（Leong et al., 2023; He et al., 2024; Tang et al., 2023）和社交网络中的有毒言论打击（Deng 等人，2023 年；Mun 等人，2023 年；Agarwal 等人，2023 年）。所有这些应用对于确保现代数字世界的安全通信都极其重要。然而，之前的并行文本解毒语料库收集方法——ParaDetox（Logacheva 等人，2022）和 APPADIA（Atwell 等人，2022）——仅在单语言设置中进行了探索。在这项工作中，我们的目标是将 ParaDetox 管道扩展到多种语言，呈现 MultiParaDetox，以自动对任何语言进行并行解毒语料库收集。然后，我们尝试了不同的文本去毒模型——从无监督基线到法学硕士和在所提出的平行语料库上的微调模型——展示了平行语料库的存在对于获得最先进的文本去毒模型的巨大好处。语言。

Title: Ukrainian Texts Classification: Exploration of Cross-lingual Knowledge Transfer Approaches

• Authors: Daryna Dementieva, Valeriia Khylenko, Georg Groh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02043
• Pdf URL: https://arxiv.org/pdf/2404.02043
• Copy Paste: [[2404.02043]] Ukrainian Texts Classification: Exploration of Cross-lingual Knowledge Transfer Approaches(https://arxiv.org/abs/2404.02043)
• Keywords: llm
• Abstract: Despite the extensive amount of labeled datasets in the NLP text classification field, the persistent imbalance in data availability across various languages remains evident. Ukrainian, in particular, stands as a language that still can benefit from the continued refinement of cross-lingual methodologies. Due to our knowledge, there is a tremendous lack of Ukrainian corpora for typical text classification tasks. In this work, we leverage the state-of-the-art advances in NLP, exploring cross-lingual knowledge transfer methods avoiding manual data curation: large multilingual encoders and translation systems, LLMs, and language adapters. We test the approaches on three text classification tasks -- toxicity classification, formality classification, and natural language inference -- providing the "recipe" for the optimal setups.
• 摘要：尽管 NLP 文本分类领域有大量标记数据集，但各种语言之间数据可用性的持续不平衡仍然很明显。尤其是乌克兰语，它仍然可以从跨语言方法的不断完善中受益。据我们所知，用于典型文本分类任务的乌克兰语语料库非常缺乏。在这项工作中，我们利用 NLP 领域最先进的技术，探索跨语言知识转移方法，避免手动数据管理：大型多语言编码器和翻译系统、法学硕士和语言适配器。我们在三个文本分类任务（毒性分类、形式分类和自然语言推理）上测试了这些方法，为最佳设置提供了“秘诀”。

Title: Deconstructing In-Context Learning: Understanding Prompts via Corruption

• Authors: Namrata Shivagunde, Vladislav Lialin, Sherin Muckatira, Anna Rumshisky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02054
• Pdf URL: https://arxiv.org/pdf/2404.02054
• Copy Paste: [[2404.02054]] Deconstructing In-Context Learning: Understanding Prompts via Corruption(https://arxiv.org/abs/2404.02054)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: The ability of large language models (LLMs) to "learn in context" based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI assistants are known to be robust to minor prompt modifications, mostly due to alignment techniques that use human feedback. In contrast, the underlying pre-trained LLMs they use as a backbone are known to be brittle in this respect. Building high-quality backbone models remains a core challenge, and a common approach to assessing their quality is to conduct few-shot evaluation. Such evaluation is notorious for being highly sensitive to minor prompt modifications, as well as the choice of specific in-context examples. Prior work has examined how modifying different elements of the prompt can affect model performance. However, these earlier studies tended to concentrate on a limited number of specific prompt attributes and often produced contradictory results. Additionally, previous research either focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging. In the present study, we decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration. We investigate the effects of structural and semantic corruptions of these elements on model performance. We study models ranging from 1.5B to 70B in size, using ten datasets covering classification and generation tasks. We find that repeating text within the prompt boosts model performance, and bigger models ($\geq$30B) are more sensitive to the semantics of the prompt. Finally, we observe that adding task and inline instructions to the demonstrations enhances model performance even when the instructions are semantically corrupted.
• 摘要：大型语言模型 (LLM) 根据提供的提示“在上下文中学习”的能力导致了其使用的爆炸性增长，最终导致了 ChatGPT、Claude 和 Bard 等人工智能助手的激增。众所周知，这些人工智能助手对微小的即时修改具有鲁棒性，这主要是由于使用人类反馈的对齐技术。相比之下，他们用作骨干的底层预训练法学硕士在这方面是脆弱的。构建高质量的骨干模型仍然是一个核心挑战，评估其质量的常见方法是进行小样本评估。这种评估因对微小的即时修改以及特定上下文示例的选择高度敏感而臭名昭著。之前的工作已经研究了修改提示的不同元素如何影响模型性能。然而，这些早期的研究往往集中在有限数量的特定提示属性上，并且经常产生相互矛盾的结果。此外，之前的研究要么侧重于参数少于 150 亿个的模型，要么专门检查 GPT-3 或 PaLM 等黑盒模型，这使得复制具有挑战性。在本研究中，我们将整个提示分解为四个部分：任务描述、演示输入、标签以及为每个演示提供的内联指令。我们研究这些元素的结构和语义损坏对模型性能的影响。我们使用涵盖分类和生成任务的 10 个数据集来研究大小从 1.5B 到 70B 的模型。我们发现在提示中重复文本可以提高模型性能，并且较大的模型 ($\geq$30B) 对提示的语义更加敏感。最后，我们观察到，即使指令在语义上被破坏，向演示添加任务和内联指令也可以增强模型性能。

Title: Long-context LLMs Struggle with Long In-context Learning

• Authors: Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, Wenhu Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02060
• Pdf URL: https://arxiv.org/pdf/2404.02060
• Copy Paste: [[2404.02060]] Long-context LLMs Struggle with Long In-context Learning(https://arxiv.org/abs/2404.02060)
• Keywords: language model, gpt, llm, long context
• Abstract: Large Language Models (LLMs) have made significant strides in handling long sequences exceeding 32K tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their abilities in more nuanced, real-world scenarios. This study introduces a specialized benchmark (LIConBench) focusing on long in-context learning within the realm of extreme-label classification. We meticulously selected six datasets with a label range spanning 28 to 174 classes covering different input (few-shot demonstration) length from 2K to 50K. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct prediction. We evaluate 13 long-context LLMs on our benchmarks. We find that the long-context LLMs perform relatively well under the token length of 20K and the performance benefits from utilizing the long context window. However, after the context window exceeds 20K, most LLMs except GPT-4 will dip dramatically. This suggests a notable gap in current LLM capabilities for processing and understanding long, context-rich sequences. Further analysis revealed a tendency among models to favor predictions for labels presented towards the end at the sequence. Their ability to reason over multiple pieces in the long sequence is yet to be improved. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LIConBench could serve as a more realistic evaluation for the future long context LLMs.
• 摘要：大型语言模型 (LLM) 在处理超过 32K 标记的长序列方面取得了重大进展。然而，他们的绩效评估很大程度上局限于复杂度和综合任务等指标，这些指标可能无法完全捕捉他们在更细致的现实场景中的能力。这项研究引入了一个专门的基准（LIConBench），专注于极端标签分类领域内的长期上下文学习。我们精心选择了 6 个数据集，其标签范围涵盖 28 到 174 个类别，涵盖从 2K 到 50K 的不同输入（少量演示）长度。我们的基准要求法学硕士理解整个输入，识别大量标签空间，从而做出正确的预测。我们根据基准评估了 13 个长背景法学硕士。我们发现长上下文 LLM 在 20K 的 token 长度下表现相对较好，并且性能受益于利用长上下文窗口。然而，上下文窗口超过 20K 后，除 GPT-4 之外的大多数 LLM 都会急剧下降。这表明当前法学硕士在处理和理解长的、上下文丰富的序列方面的能力存在显着差距。进一步的分析揭示了模型倾向于预测序列末尾出现的标签的趋势。他们对长序列中的多个片段进行推理的能力还有待提高。我们的研究表明，对于现有的法学硕士来说，长上下文理解和推理仍然是一项具有挑战性的任务。我们相信 LIConBench 可以为未来的长期背景法学硕士提供更现实的评估。

Title: CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems

• Authors: Sara Rosenthal, Avirup Sil, Radu Florian, Salim Roukos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02103
• Pdf URL: https://arxiv.org/pdf/2404.02103
• Copy Paste: [[2404.02103]] CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems(https://arxiv.org/abs/2404.02103)
• Keywords: language model, hallucination, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) has become a popular application for large language models. It is preferable that successful RAG systems provide accurate answers that are supported by being grounded in a passage without any hallucinations. While considerable work is required for building a full RAG pipeline, being able to benchmark performance is also necessary. We present ClapNQ, a benchmark Long-form Question Answering dataset for the full RAG pipeline. ClapNQ includes long answers with grounded gold passages from Natural Questions (NQ) and a corpus to perform either retrieval, generation, or the full RAG pipeline. The ClapNQ answers are concise, 3x smaller than the full passage, and cohesive, with multiple pieces of the passage that are not contiguous. RAG models must adapt to these properties to be successful at ClapNQ. We present baseline experiments and analysis for ClapNQ that highlight areas where there is still significant room for improvement in grounded RAG. CLAPNQ is publicly available at https://github.com/primeqa/clapnq
• 摘要：检索增强生成（RAG）已成为大型语言模型的流行应用。成功的 RAG 系统最好能够提供准确的答案，并通过在没有任何幻觉的情况下扎根于段落来支持这些答案。虽然构建完整的 RAG 管道需要大量工作，但能够对性能进行基准测试也是必要的。我们推出了 ClapNQ，这是一个适用于完整 RAG 流程的基准长格式问答数据集。 ClapNQ 包含来自自然问题 (NQ) 的长答案和接地黄金段落，以及用于执行检索、生成或完整 RAG 管道的语料库。 ClapNQ 的答案简洁，比完整段落小 3 倍，并且具有凝聚力，段落的多个部分不连续。 RAG 模型必须适应这些特性才能在 ClapNQ 取得成功。我们提出了 ClapNQ 的基线实验和分析，强调了接地 RAG 仍有很大改进空间的领域。 CLAPNQ 可在 https://github.com/primeqa/clapnq 上公开获取

Title: GINopic: Topic Modeling with Graph Isomorphism Network

• Authors: Suman Adhya, Debarshi Kumar Sanyal
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02115
• Pdf URL: https://arxiv.org/pdf/2404.02115
• Copy Paste: [[2404.02115]] GINopic: Topic Modeling with Graph Isomorphism Network(https://arxiv.org/abs/2404.02115)
• Keywords: language model
• Abstract: Topic modeling is a widely used approach for analyzing and exploring large document collections. Recent research efforts have incorporated pre-trained contextualized language models, such as BERT embeddings, into topic modeling. However, they often neglect the intrinsic informational value conveyed by mutual dependencies between words. In this study, we introduce GINopic, a topic modeling framework based on graph isomorphism networks to capture the correlation between words. By conducting intrinsic (quantitative as well as qualitative) and extrinsic evaluations on diverse benchmark datasets, we demonstrate the effectiveness of GINopic compared to existing topic models and highlight its potential for advancing topic modeling.
• 摘要：主题建模是一种广泛使用的分析和探索大型文档集合的方法。最近的研究工作已将预训练的情境化语言模型（例如 BERT 嵌入）纳入主题建模中。然而，他们常常忽视单词之间相互依赖所传达的内在信息价值。在本研究中，我们引入了 GINopic，一个基于图同构网络的主题建模框架，用于捕获单词之间的相关性。通过对不同的基准数据集进行内在（定量和定性）和外在评估，我们证明了 GINopic 与现有主题模型相比的有效性，并强调了其推进主题建模的潜力。

Title: Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models

• Authors: Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Otero Ornelas, Andrew Lan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02124
• Pdf URL: https://arxiv.org/pdf/2404.02124
• Copy Paste: [[2404.02124]] Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models(https://arxiv.org/abs/2404.02124)
• Keywords: language model, llm
• Abstract: Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and are a reliable format in assessments and practices. One of the most important aspects of MCQs is the distractors, i.e., incorrect options that are designed to target common errors or misconceptions among real students. To date, the task of crafting high-quality distractors largely remains a labor and time-intensive process for teachers and learning content designers, which has limited scalability. In this work, we study the task of automated distractor generation in the domain of math MCQs and explore a wide variety of large language model (LLM)-based approaches, from in-context learning to fine-tuning. We conduct extensive experiments using a real-world math MCQ dataset and find that although LLMs can generate some mathematically valid distractors, they are less adept at anticipating common errors or misconceptions among real students.
• 摘要：多项选择题 (MCQ) 在几乎所有级别的教育中都很普遍，因为它们易于管理、评分，并且是评估和实践中可靠的格式。 MCQ 最重要的方面之一是干扰因素，即针对真实学生中常见错误或误解而设计的不正确选项。迄今为止，对于教师和学习内容设计者来说，制作高质量干扰项的任务在很大程度上仍然是一个劳动和时间密集的过程，其可扩展性有限。在这项工作中，我们研究了数学 MCQ 领域中自动干扰项生成的任务，并探索了各种基于大语言模型 (LLM) 的方法，从上下文学习到微调。我们使用现实世界的数学 MCQ 数据集进行了广泛的实验，发现虽然法学硕士可以产生一些数学上有效的干扰因素，但他们不太擅长预测真实学生中的常见错误或误解。

Title: FLawN-T5: An Empirical Examination of Effective Instruction-Tuning Data Mixtures for Legal Reasoning

• Authors: Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M. Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, Christopher Manning
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02127
• Pdf URL: https://arxiv.org/pdf/2404.02127
• Copy Paste: [[2404.02127]] FLawN-T5: An Empirical Examination of Effective Instruction-Tuning Data Mixtures for Legal Reasoning(https://arxiv.org/abs/2404.02127)
• Keywords: language model, llm
• Abstract: Instruction tuning is an important step in making language models useful for direct user interaction. However, many legal tasks remain out of reach for most open LLMs and there do not yet exist any large scale instruction datasets for the domain. This critically limits research in this application area. In this work, we curate LawInstruct, a large legal instruction dataset, covering 17 jurisdictions, 24 languages and a total of 12M examples. We present evidence that domain-specific pretraining and instruction tuning improve performance on LegalBench, including improving Flan-T5 XL by 8 points or 16\% over the baseline. However, the effect does not generalize across all tasks, training regimes, model sizes, and other factors. LawInstruct is a resource for accelerating the development of models with stronger information processing and decision making capabilities in the legal domain.
• 摘要：指令调优是使语言模型可用于直接用户交互的重要一步。然而，对于大多数开放式法学硕士来说，许多法律任务仍然遥不可及，而且该领域尚不存在任何大规模的教学数据集。这严重限制了该应用领域的研究。在这项工作中，我们整理了 LawInstruct，这是一个大型法律说明数据集，涵盖 17 个司法管辖区、24 种语言和总共 1200 万个示例。我们提供的证据表明，特定领域的预训练和指令调整可提高 LegalBench 上的性能，包括将 Flan-T5 XL 比基线提高 8 个点或 16%。然而，这种效果并不能推广到所有任务、训练制度、模型大小和其他因素。 LawInstruct是一个加速开发法律领域具有更强信息处理和决策能力的模型的资源。