2024-07-08

Title: Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

• Authors: Sudha Rao, Weijia Xu, Michael Xu, Jorge Leandro, Ken Lobb, Gabriel DesGarennes, Chris Brockett, Bill Dolan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03460
• Pdf URL: https://arxiv.org/pdf/2407.03460
• Copy Paste: [[2407.03460]] Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft(https://arxiv.org/abs/2407.03460)
• Keywords: language model, gpt, llm
• Abstract: The use of generative AI in video game development is on the rise, and as the conversational and other capabilities of large language models continue to improve, we expect LLM-driven non-player characters (NPCs) to become widely deployed. In this paper, we seek to understand how human players collaborate with LLM-driven NPCs to accomplish in-game goals. We design a minigame within Minecraft where a player works with two GPT4-driven NPCs to complete a quest. We perform a user study in which 28 Minecraft players play this minigame and share their feedback. On analyzing the game logs and recordings, we find that several patterns of collaborative behavior emerge from the NPCs and the human players. We also report on the current limitations of language-only models that do not have rich game-state or visual understanding. We believe that this preliminary study and analysis will inform future game developers on how to better exploit these rapidly improving generative AI models for collaborative roles in games.
• 摘要：生成式 AI 在视频游戏开发中的应用正在不断增加，随着大型语言模型的对话和其他功能的不断改进，我们预计 LLM 驱动的非玩家角色 (NPC) 将得到广泛部署。在本文中，我们试图了解人类玩家如何与 LLM 驱动的 NPC 合作完成游戏中的目标。我们在 Minecraft 中设计了一个迷你游戏，玩家与两个 GPT4 驱动的 NPC 合作完成任务。我们进行了一项用户研究，其中 28 名 Minecraft 玩家玩这个迷你游戏并分享他们的反馈。在分析游戏日志和录音时，我们发现 NPC 和人类玩家出现了几种协作行为模式。我们还报告了目前没有丰富游戏状态或视觉理解的纯语言模型的局限性。我们相信这项初步研究和分析将为未来的游戏开发者提供信息，让他们了解如何更好地利用这些快速改进的生成式 AI 模型来实现游戏中的协作角色。

Title: Improving LLM Abilities in Idiomatic Translation

• Authors: Sundesh Donthi, Maximilian Spencer, Om Patel, Joon Doh, Eid Rodan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03518
• Pdf URL: https://arxiv.org/pdf/2407.03518
• Copy Paste: [[2407.03518]] Improving LLM Abilities in Idiomatic Translation(https://arxiv.org/abs/2407.03518)
• Keywords: language model, gpt, llm
• Abstract: For large language models (LLMs) like NLLB and GPT, translating idioms remains a challenge. Our goal is to enhance translation fidelity by improving LLM processing of idiomatic language while preserving the original linguistic style. This has a significant social impact, as it preserves cultural nuances and ensures translated texts retain their intent and emotional resonance, fostering better cross-cultural communication. Previous work has utilized knowledge bases like IdiomKB by providing the LLM with the meaning of an idiom to use in translation. Although this method yielded better results than a direct translation, it is still limited in its ability to preserve idiomatic writing style across languages. In this research, we expand upon the knowledge base to find corresponding idioms in the target language. Our research performs translations using two methods: The first method employs the SentenceTransformers model to semantically generate cosine similarity scores between the meanings of the original and target language idioms, selecting the best idiom (Cosine Similarity method). The second method uses an LLM to find a corresponding idiom in the target language for use in the translation (LLM-generated idiom method). As a baseline, we performed a direct translation without providing additional information. Human evaluations on the English -> Chinese, and Chinese -> English show the Cosine Similarity Lookup method out-performed others in all GPT4o translations. To further build upon IdiomKB, we developed a low-resource Urdu dataset containing Urdu idioms and their translations. Despite dataset limitations, the Cosine Similarity Lookup method shows promise, potentially overcoming language barriers and enabling the exploration of diverse literary works in Chinese and Urdu. For access to the code and replication of our experiments, please visit (this https URL).
• 摘要：对于 NLLB 和 GPT 等大型语言模型 (LLM)，翻译习语仍然是一项挑战。我们的目标是通过改进 LLM 对惯用语言的处理来提高翻译保真度，同时保留原始语言风格。这具有重大的社会影响，因为它保留了文化细微差别并确保翻译文本保留其意图和情感共鸣，从而促进更好的跨文化交流。之前的研究利用了 IdiomKB 等知识库，为 LLM 提供用于翻译的习语含义。虽然这种方法比直接翻译产生了更好的结果，但它在跨语言保留惯用写作风格的能力方面仍然有限。在本研究中，我们扩展了知识库以查找目标语言中的对应习语。我们的研究使用两种方法进行翻译：第一种方法采用 SentenceTransformers 模型在语义上生成原始语言和目标语言习语含义之间的余弦相似度得分，从而选择最佳习语（余弦相似度方法）。第二种方法是使用 LLM 查找目标语言中相应的成语以用于翻译（LLM 生成的成语方法）。作为基准，我们进行了直接翻译，而不提供其他信息。对英语 -> 中文和中文 -> 英语的人工评估表明，余弦相似度查找方法在所有 GPT4o 翻译中的表现均优于其他方法。为了进一步构建 IdiomKB，我们开发了一个低资源乌尔都语数据集，其中包含乌尔都语成语及其翻译。尽管数据集存在限制，但余弦相似度查找方法显示出良好的前景，有可能克服语言障碍并实现对中文和乌尔都语中各种文学作品的探索。如需访问代码和复制我们的实验，请访问（此 https URL）。

Title: UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorization

• Authors: Md Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth, Tran Cao Son, Eduardo Blanco, Steven R. Corman, Chitta Baral
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03525
• Pdf URL: https://arxiv.org/pdf/2407.03525
• Copy Paste: [[2407.03525]] UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorization(https://arxiv.org/abs/2407.03525)
• Keywords: language model, llm
• Abstract: This paper introduces UnSeenTimeQA, a novel time-sensitive question-answering (TSQA) benchmark that diverges from traditional TSQA benchmarks by avoiding factual and web-searchable queries. We present a series of time-sensitive event scenarios decoupled from real-world factual information. It requires large language models (LLMs) to engage in genuine temporal reasoning, disassociating from the knowledge acquired during the pre-training phase. Our evaluation of six open-source LLMs (ranging from 2B to 70B in size) and three closed-source LLMs reveal that the questions from the UnSeenTimeQA present substantial challenges. This indicates the models' difficulties in handling complex temporal reasoning scenarios. Additionally, we present several analyses shedding light on the models' performance in answering time-sensitive questions.
• 摘要：本文介绍了 UnSeenTimeQA，这是一种新颖的时间敏感型问答 (TSQA) 基准，它通过避免事实和可网络搜索的查询而与传统 TSQA 基准有所不同。我们提出了一系列与现实世界事实信息分离的时间敏感型事件场景。它需要大型语言模型 (LLM) 进行真正的时间推理，与预训练阶段获得的知识脱钩。我们对六个开源 LLM（大小从 2B 到 70B 不等）和三个闭源 LLM 的评估表明，UnSeenTimeQA 的问题带来了巨大的挑战。这表明模型在处理复杂的时间推理场景时存在困难。此外，我们还提出了几项分析，阐明了模型在回答时间敏感型问题方面的表现。

Title: Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias

• Authors: Jayanta Sadhu, Maneesha Rani Saha, Rifat Shahriyar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03536
• Pdf URL: https://arxiv.org/pdf/2407.03536
• Copy Paste: [[2407.03536]] Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias(https://arxiv.org/abs/2407.03536)
• Keywords: language model, llm
• Abstract: The rapid growth of Large Language Models (LLMs) has put forward the study of biases as a crucial field. It is important to assess the influence of different types of biases embedded in LLMs to ensure fair use in sensitive fields. Although there have been extensive works on bias assessment in English, such efforts are rare and scarce for a major language like Bangla. In this work, we examine two types of social biases in LLM generated outputs for Bangla language. Our main contributions in this work are: (1) bias studies on two different social biases for Bangla (2) a curated dataset for bias measurement benchmarking (3) two different probing techniques for bias detection in the context of Bangla. This is the first work of such kind involving bias assessment of LLMs for Bangla to the best of our knowledge. All our code and resources are publicly available for the progress of bias related research in Bangla NLP.
• 摘要：大型语言模型 (LLM) 的快速发展使偏见研究成为一个重要领域。评估 LLM 中嵌入的不同类型偏见的影响非常重要，以确保在敏感领域得到公平使用。尽管已经有大量关于英语偏见评估的研究，但对于孟加拉语这样的主要语言来说，这样的努力却很少见。在这项工作中，我们研究了 LLM 为孟加拉语生成的输出中的两种社会偏见。我们在这项工作中的主要贡献是：(1) 对孟加拉语的两种不同社会偏见进行偏见研究 (2) 用于偏见测量基准测试的精选数据集 (3) 在孟加拉语环境中检测偏见的两种不同探测技术。据我们所知，这是第一项涉及孟加拉语 LLM 偏见评估的此类研究。我们所有的代码和资源都是公开的，用于推动孟加拉语 NLP 中偏见相关的研究。

Title: Core: Robust Factual Precision Scoring with Informative Sub-Claim Identification

• Authors: Zhengping Jiang, Jingyu Zhang, Nathaniel Weir, Seth Ebner, Miriam Wanner, Kate Sanders, Daniel Khashabi, Anqi Liu, Benjamin Van Durme
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03572
• Pdf URL: https://arxiv.org/pdf/2407.03572
• Copy Paste: [[2407.03572]] Core: Robust Factual Precision Scoring with Informative Sub-Claim Identification(https://arxiv.org/abs/2407.03572)
• Keywords: language model, llm, hallucination
• Abstract: Hallucinations -- the generation of untrue claims -- pose a challenge to the application of large language models (LLMs) [1] thereby motivating the development of metrics to evaluate factual precision. We observe that popular metrics using the Decompose-Then-Verify framework, such as FActScore [2], can be manipulated by adding obvious or repetitive claims to artificially inflate scores. We expand the FActScore dataset to design and analyze factual precision metrics, demonstrating that models can be trained to achieve high scores under existing metrics through exploiting the issues we identify. This motivates our new customizable plug-and-play subclaim selection component called Core, which filters down individual subclaims according to their uniqueness and informativeness. Metrics augmented by Core are substantially more robust as shown in head-to-head comparisons. We release an evaluation framework supporting the modular use of Core (this https URL) and various decomposition strategies, and we suggest its adoption by the LLM community. [1] Hong et al., "The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models", arXiv:2404.05904v2 [cs.CL]. [2] Min et al., "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation", arXiv:2305.14251v2 [cs.CL].
• 摘要：幻觉——产生不真实的声明——对大型语言模型 (LLM) [1] 的应用构成了挑战，从而推动了评估事实精度的指标的发展。我们观察到，使用“分解然后验证”框架的流行指标（例如 FActScore [2]）可以通过添加明显或重复的声明来人为地提高分数。我们扩展了 FActScore 数据集来设计和分析事实精度指标，表明可以通过利用我们发现的问题来训练模型以在现有指标下获得高分。这促使我们开发了新的可定制即插即用子声明选择组件 Core，该组件根据各个子声明的独特性和信息量对其进行筛选。通过 Core 增强的指标在头对头比较中显示出了显著的稳健性。我们发布了一个支持模块化使用 Core（此 https URL）和各种分解策略的评估框架，并建议 LLM 社区采用它。 [1] Hong 等人，“幻觉排行榜——一项在大型语言模型中测量幻觉的公开努力”，arXiv:2404.05904v2 [cs.CL]。[2] Min 等人，“FActScore：长文本生成中事实精度的细粒度原子评估”，arXiv:2305.14251v2 [cs.CL]。

Title: Integrating Randomness in Large Language Models: A Linear Congruential Generator Approach for Generating Clinically Relevant Content

• Authors: Andrew Bouras
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03582
• Pdf URL: https://arxiv.org/pdf/2407.03582
• Copy Paste: [[2407.03582]] Integrating Randomness in Large Language Models: A Linear Congruential Generator Approach for Generating Clinically Relevant Content(https://arxiv.org/abs/2407.03582)
• Keywords: language model, gpt, prompt
• Abstract: Generating diverse, high-quality outputs from language models is crucial for applications in education and content creation. Achieving true randomness and avoiding repetition remains a significant challenge. This study uses the Linear Congruential Generator method for systematic fact selection, combined with AI-powered content generation. We ensured unique combinations of gastrointestinal physiology and pathology facts across multiple rounds, integrating these facts into prompts for GPT-4o to create clinically relevant, vignette-style outputs. Over 14 rounds, 98 unique outputs were generated, demonstrating LCG's effectiveness in producing diverse and high-quality content. This method addresses key issues of randomness and repetition, enhancing the quality and efficiency of language model-generated content for various applications.
• 摘要：从语言模型生成多样化、高质量的输出对于教育和内容创作中的应用至关重要。实现真正的随机性和避免重复仍然是一项重大挑战。本研究使用线性一致性生成器方法进行系统性事实选择，并结合人工智能驱动的内容生成。我们确保了多轮胃肠道生理学和病理学事实的独特组合，将这些事实整合到 GPT-4o 的提示中，以创建与临床相关的小插图式输出。在 14 轮中，生成了 98 个独特的输出，证明了 LCG 在生成多样化和高质量内容方面的有效性。该方法解决了随机性和重复性的关键问题，提高了语言模型为各种应用生成内容的质量和效率。

Title: Zero-shot Persuasive Chatbots with LLM-Generated Strategies and Information Retrieval

• Authors: Kazuaki Furumai, Roberto Legaspi, Julio Vizcarra, Yudai Yamazaki, Yasutaka Nishimura, Sina J. Semnani, Kazushi Ikeda, Weiyan Shi, Monica S. Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03585
• Pdf URL: https://arxiv.org/pdf/2407.03585
• Copy Paste: [[2407.03585]] Zero-shot Persuasive Chatbots with LLM-Generated Strategies and Information Retrieval(https://arxiv.org/abs/2407.03585)
• Keywords: language model, llm, chat
• Abstract: Persuasion plays a pivotal role in a wide range of applications from health intervention to the promotion of social good. Persuasive chatbots can accelerate the positive effects of persuasion in such applications. Existing methods rely on fine-tuning persuasive chatbots with task-specific training data which is costly, if not infeasible, to collect. To address this issue, we propose a method to leverage the generalizability and inherent persuasive abilities of large language models (LLMs) in creating effective and truthful persuasive chatbot for any given domain in a zero-shot manner. Unlike previous studies which used pre-defined persuasion strategies, our method first uses an LLM to generate responses, then extracts the strategies used on the fly, and replaces any unsubstantiated claims in the response with retrieved facts supporting the strategies. We applied our chatbot, PersuaBot, to three significantly different domains needing persuasion skills: donation solicitation, recommendations, and health intervention. Our experiments on simulated and human conversations show that our zero-shot approach is more persuasive than prior work, while achieving factual accuracy surpassing state-of-the-art knowledge-oriented chatbots. Our study demonstrated that when persuasive chatbots are employed responsibly for social good, it is an enabler of positive individual and social change.
• 摘要：说服在从健康干预到促进社会公益等广泛应用中发挥着关键作用。说服聊天机器人可以加速说服在这些应用中的积极影响。现有方法依赖于使用特定于任务的训练数据对说服聊天机器人进行微调，而这些数据的收集成本很高，甚至不可行。为了解决这个问题，我们提出了一种方法，利用大型语言模型 (LLM) 的通用性和固有说服能力，以零样本方式为任何给定领域创建有效且真实的说服聊天机器人。与以前使用预定义说服策略的研究不同，我们的方法首先使用 LLM 生成响应，然后动态提取使用的策略，并用检索到的支持策略的事实替换响应中任何未经证实的声明。我们将聊天机器人 PersuaBot 应用于三个需要说服技能的截然不同的领域：捐款征集、推荐和健康干预。我们对模拟和人类对话的实验表明，我们的零样本方法比以前的研究更有说服力，同时实现了超越最先进的知识型聊天机器人的事实准确性。我们的研究表明，当说服性聊天机器人被负责任地用于社会公益时，它将成为个人和社会积极变革的推动者。

Title: Contrastive Chain-of-Thought Prompting

• Authors: Grant Kruttschnitt, Jay Shim, Alyssa Ma, Daniel Kim, Benjamin Chek, Athul Anand, Kevin Zhu, Sean O'Brien
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2311.09277
• Pdf URL: https://arxiv.org/pdf/2311.09277
• Copy Paste: [[2311.09277]] Contrastive Chain-of-Thought Prompting(https://arxiv.org/abs/2311.09277)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Rapidly increasing model scales coupled with steering methods such as chain-of-thought prompting have led to drastic improvements in language model reasoning. At the same time, models struggle with compositional generalization and are far from human performance on many reasoning-based benchmarks. Leveraging the success of chain-of-thought prompting, and also taking inspiration from context-aware decoding (CAD), we explore input-based contrasting methods to further encourage the type of reasoning induced by chain-of-thought prompting. While work remains to stabilize these results across datasets and models, the improvements we find warrant further investigation into input-based steering methods for context-aware reasoning.
• 摘要：模型规模的快速增长，加上思路链提示等引导方法，已导致语言模型推理得到大幅改进。与此同时，模型在组合泛化方面存在困难，并且在许多基于推理的基准上与人类的表现相差甚远。利用思路链提示的成功，并从上下文感知解码 (CAD) 中汲取灵感，我们探索基于输入的对比方法，以进一步鼓励由思路链提示引发的推理类型。虽然仍需努力在数据集和模型之间稳定这些结果，但我们发现的改进值得进一步研究基于输入的上下文感知推理引导方法。

Title: Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations

• Authors: Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, Lifu Huang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.03604
• Pdf URL: https://arxiv.org/pdf/2407.03604
• Copy Paste: [[2407.03604]] Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations(https://arxiv.org/abs/2407.03604)
• Keywords: language model
• Abstract: Recent advancements in Vision-Language Models (VLMs) have led to the development of Vision-Language Generalists (VLGs) capable of understanding and generating interleaved images and text. Despite these advances, VLGs still struggle to follow user instructions for interleaved text and image generation. To address this issue, we introduce LeafInstruct, the first open-sourced interleaved instruction tuning data with over 30,000 high-quality instances across more than 10 domains. Due to the extensive size of existing VLGs, we opt for parameter-efficient tuning. However, we observe that VLGs tuned with a standard LoRA typically exhibit inferior performance in interleaved text-image generation. We attribute this problem to modality interference and the lack of modality-specialized adaptation design. Hence, we propose Lateralization LoRA, a novel modality-specialized adaptation method inspired by the concept of brain lateralization. Lateralization LoRA employs a hybrid approach, combining the traditional linear LoRA and a Convolutional LoRA for generating text and images, enabling the generation of high-quality text and images by leveraging modality-specific structures and parameter sets. We perform instruction tuning of the VLG (i.e., EMU2) using Lateralization LoRA on the LeafInstruct dataset. Extensive experiments demonstrate that EMU2 tuned with Lateralization LoRA achieve state-of-the-art performance, significantly surpassing baseline models in complex interleaved tasks.
• 摘要：视觉语言模型 (VLM) 的最新进展促成了视觉语言通才 (VLG) 的发展，它们能够理解和生成交错的图像和文本。尽管取得了这些进展，但 VLG 仍然难以遵循用户对交错文本和图像生成的指令。为了解决这个问题，我们推出了 LeafInstruct，这是第一个开源交错指令调优数据，包含 10 多个领域的 30,000 多个高质量实例。由于现有 VLG 的规模庞大，我们选择了参数高效的调优。然而，我们观察到使用标准 LoRA 调优的 VLG 在交错文本图像生成中通常表现较差。我们将此问题归因于模态干扰和缺乏针对模态的专门适应设计。因此，我们提出了 Lateralization LoRA，这是一种受大脑侧化概念启发的新型针对模态的专门适应方法。 Lateralization LoRA 采用混合方法，将传统的线性 LoRA 与卷积 LoRA 相结合以生成文本和图像，从而利用特定于模态的结构和参数集生成高质量的文本和图像。我们使用 Lateralization LoRA 在 LeafInstruct 数据集上对 VLG（即 EMU2）进行指令调整。大量实验表明，使用 Lateralization LoRA 调整的 EMU2 实现了最先进的性能，在复杂的交错任务中显著超越了基线模型。

Title: Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models

• Authors: Chang-Sheng Kao, Yun-Nung Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03615
• Pdf URL: https://arxiv.org/pdf/2407.03615
• Copy Paste: [[2407.03615]] Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models(https://arxiv.org/abs/2407.03615)
• Keywords: language model, llm
• Abstract: Recent advancements in dialogue systems have highlighted the significance of integrating multimodal responses, which enable conveying ideas through diverse modalities rather than solely relying on text-based interactions. This enrichment not only improves overall communicative efficacy but also enhances the quality of conversational experiences. However, existing methods for dialogue-to-image retrieval face limitations due to the constraints of pre-trained vision language models (VLMs) in comprehending complex dialogues accurately. To address this, we present a novel approach leveraging the robust reasoning capabilities of large language models (LLMs) to generate precise dialogue-associated visual descriptors, facilitating seamless connection with images. Extensive experiments conducted on benchmark data validate the effectiveness of our proposed approach in deriving concise and accurate visual descriptors, leading to significant enhancements in dialogue-to-image retrieval performance. Furthermore, our findings demonstrate the method's generalizability across diverse visual cues, various LLMs, and different datasets, underscoring its practicality and potential impact in real-world applications.
• 摘要：对话系统的最新进展凸显了整合多模态响应的重要性，这使得人们能够通过多种模态传达思想，而不仅仅是依靠基于文本的交互。这种丰富不仅提高了整体交流效率，而且提高了对话体验的质量。然而，现有的对话到图像检索方法面临局限性，因为预训练的视觉语言模型 (VLM) 在准确理解复杂对话方面受到限制。为了解决这个问题，我们提出了一种新方法，利用大型语言模型 (LLM) 的强大推理能力来生成精确的对话相关视觉描述符，促进与图像的无缝连接。在基准数据上进行的大量实验验证了我们提出的方法在得出简洁准确的视觉描述符方面的有效性，从而显著提高了对话到图像检索的性能。此外，我们的研究结果证明了该方法在各种视觉线索、各种 LLM 和不同数据集中的通用性，强调了其实用性和在实际应用中的潜在影响。

Title: The Mysterious Case of Neuron 1512: Injectable Realignment Architectures Reveal Internal Characteristics of Meta's Llama 2 Model

• Authors: Brenden Smith, Dallin Baker, Clayton Chase, Myles Barney, Kaden Parker, Makenna Allred, Peter Hu, Alex Evans, Nancy Fulda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03621
• Pdf URL: https://arxiv.org/pdf/2407.03621
• Copy Paste: [[2407.03621]] The Mysterious Case of Neuron 1512: Injectable Realignment Architectures Reveal Internal Characteristics of Meta's Llama 2 Model(https://arxiv.org/abs/2407.03621)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have an unrivaled and invaluable ability to "align" their output to a diverse range of human preferences, by mirroring them in the text they generate. The internal characteristics of such models, however, remain largely opaque. This work presents the Injectable Realignment Model (IRM) as a novel approach to language model interpretability and explainability. Inspired by earlier work on Neural Programming Interfaces, we construct and train a small network -- the IRM -- to induce emotion-based alignments within a 7B parameter LLM architecture. The IRM outputs are injected via layerwise addition at various points during the LLM's forward pass, thus modulating its behavior without changing the weights of the original model. This isolates the alignment behavior from the complex mechanisms of the transformer model. Analysis of the trained IRM's outputs reveals a curious pattern. Across more than 24 training runs and multiple alignment datasets, patterns of IRM activations align themselves in striations associated with a neuron's index within each transformer layer, rather than being associated with the layers themselves. Further, a single neuron index (1512) is strongly correlated with all tested alignments. This result, although initially counterintuitive, is directly attributable to design choices present within almost all commercially available transformer architectures, and highlights a potential weak point in Meta's pretrained Llama 2 models. It also demonstrates the value of the IRM architecture for language model analysis and interpretability. Our code and datasets are available at this https URL
• 摘要：大型语言模型 (LLM) 具有无可比拟且无价的能力，可以通过在生成的文本中反映人类的各种偏好，将其输出与人类的各种偏好“对齐”。然而，此类模型的内部特征在很大程度上仍然不透明。这项工作提出了可注入重新对齐模型 (IRM)，作为一种语言模型可解释性和可解释性的新方法。受早期神经编程接口工作的启发，我们构建并训练了一个小型网络 - IRM - 以在 7B 参数 LLM 架构中诱导基于情感的对齐。IRM 输出通过逐层添加在 LLM 的前向传递过程中的各个点注入，从而在不改变原始模型权重的情况下调节其行为。这将对齐行为与转换器模型的复杂机制隔离开来。对训练后的 IRM 输出的分析揭示了一个有趣的模式。在超过 24 次训练运行和多个对齐数据集中，IRM 激活模式与每个 Transformer 层内神经元索引相关的条纹对齐，而不是与层本身相关联。此外，单个神经元索引 (1512) 与所有测试对齐都密切相关。这个结果虽然最初违反直觉，但直接归因于几乎所有市售 Transformer 架构中存在的设计选择，并凸显了 Meta 预训练的 Llama 2 模型中的潜在弱点。它还展示了 IRM 架构对于语言模型分析和可解释性的价值。我们的代码和数据集可在此 https URL 上找到

Title: Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks

• Authors: Dharunish Yugeswardeenoo, Kevin Zhu, Sean O'Brien
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03624
• Pdf URL: https://arxiv.org/pdf/2407.03624
• Copy Paste: [[2407.03624]] Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks(https://arxiv.org/abs/2407.03624)
• Keywords: gpt, llm, prompt, chain-of-thought
• Abstract: Although LLMs have the potential to transform many fields, they still underperform humans in reasoning tasks. Existing methods induce the model to produce step-by-step calculations, but this research explores the question: Does making the LLM analyze the question improve its performance? We propose a novel prompting strategy called Question Analysis Prompting (QAP), in which the model is prompted to explain the question in $n$ words before solving. The value of $n$ influences the length of response generated by the model. QAP is evaluated on GPT 3.5 Turbo and GPT 4 Turbo on arithmetic datasets GSM8K, AQuA, and SAT and commonsense dataset StrategyQA. QAP is compared with other state-of-the-art prompts including Chain-of-Thought (CoT), Plan and Solve Prompting (PS+) and Take A Deep Breath (TADB). QAP outperforms all state-of-the-art prompts on AQuA and SAT datasets on both GPT3.5 and GPT4. QAP consistently ranks among the top-2 prompts on 75\% of the tests. A key factor of QAP performance can be attributed to response length, where detailed responses are beneficial when answering harder questions, but can negatively affect easy questions.
• 摘要：尽管 LLM 有可能改变许多领域，但它们在推理任务方面的表现仍然不及人类。现有方法诱导模型进行分步计算，但本研究探讨了这样一个问题：让 LLM 分析问题是否会提高其性能？我们提出了一种称为问题分析提示 (QAP) 的新型提示策略，在该策略中，模型在解决问题之前被提示用 $n$ 个字解释问题。$n$ 的值会影响模型生成的响应长度。在算术数据集 GSM8K、AQuA 和 SAT 以及常识数据集 StrategyQA 上，在 GPT 3.5 Turbo 和 GPT 4 Turbo 上对 QAP 进行了评估。QAP 与其他最先进的提示进行了比较，包括思维链 (CoT)、计划和解决提示 (PS+) 和深呼吸 (TADB)。在 GPT3.5 和 GPT4 上，QAP 的表现均优于 AQuA 和 SAT 数据集上所有最先进的提示。 QAP 在 75% 的测试中始终位居前 2 名。QAP 性能的一个关键因素可以归因于响应长度，其中详细的响应在回答较难的问题时很有用，但会对简单问题产生负面影响。

Title: DSLR: Document Refinement with Sentence-Level Re-ranking and Reconstruction to Enhance Retrieval-Augmented Generation

• Authors: Taeho Hwang, Soyeong Jeong, Sukmin Cho, SeungYoon Han, Jong C. Park
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03627
• Pdf URL: https://arxiv.org/pdf/2407.03627
• Copy Paste: [[2407.03627]] DSLR: Document Refinement with Sentence-Level Re-ranking and Reconstruction to Enhance Retrieval-Augmented Generation(https://arxiv.org/abs/2407.03627)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Recent advancements in Large Language Models (LLMs) have significantly improved their performance across various Natural Language Processing (NLP) tasks. However, LLMs still struggle with generating non-factual responses due to limitations in their parametric memory. Retrieval-Augmented Generation (RAG) systems address this issue by incorporating external knowledge with a retrieval module. Despite their successes, however, current RAG systems face challenges with retrieval failures and the limited ability of LLMs to filter out irrelevant information. Therefore, in this work, we propose \textit{\textbf{DSLR}} (\textbf{D}ocument Refinement with \textbf{S}entence-\textbf{L}evel \textbf{R}e-ranking and Reconstruction), an unsupervised framework that decomposes retrieved documents into sentences, filters out irrelevant sentences, and reconstructs them again into coherent passages. We experimentally validate \textit{DSLR} on multiple open-domain QA datasets and the results demonstrate that \textit{DSLR} significantly enhances the RAG performance over conventional fixed-size passage. Furthermore, our \textit{DSLR} enhances performance in specific, yet realistic scenarios without the need for additional training, providing an effective and efficient solution for refining retrieved documents in RAG systems.
• 摘要：大型语言模型 (LLM) 的最新进展显著提高了它们在各种自然语言处理 (NLP) 任务中的表现。然而，由于参数记忆的限制，LLM 仍然难以生成非事实响应。检索增强生成 (RAG) 系统通过将外部知识与检索模块相结合来解决此问题。然而，尽管取得了成功，当前的 RAG 系统仍面临着检索失败和 LLM 过滤掉不相关信息的能力有限的挑战。因此，在这项工作中，我们提出了 \textit{\textbf{DSLR}}（\textbf{D}ocument Refinement with \textbf{S}entence-\textbf{L}evel \textbf{R}e-ranking and Reconstruction），这是一个无监督框架，它将检索到的文档分解为句子，过滤掉不相关的句子，然后将它们重新重构为连贯的段落。我们在多个开放域 QA 数据集上通过实验验证了 \textit{DSLR}，结果表明 \textit{DSLR} 显著提高了 RAG 性能，优于传统的固定大小段落。此外，我们的 \textit{DSLR} 无需额外训练即可在特定但现实的场景中提高性能，为在 RAG 系统中细化检索到的文档提供了有效且高效的解决方案。

Title: Differentiating between human-written and AI-generated texts using linguistic features automatically extracted from an online computational tool

• Authors: Georgios P. Georgiou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03646
• Pdf URL: https://arxiv.org/pdf/2407.03646
• Copy Paste: [[2407.03646]] Differentiating between human-written and AI-generated texts using linguistic features automatically extracted from an online computational tool(https://arxiv.org/abs/2407.03646)
• Keywords: gpt, prompt, chat
• Abstract: While extensive research has focused on ChatGPT in recent years, very few studies have systematically quantified and compared linguistic features between human-written and Artificial Intelligence (AI)-generated language. This study aims to investigate how various linguistic components are represented in both types of texts, assessing the ability of AI to emulate human writing. Using human-authored essays as a benchmark, we prompted ChatGPT to generate essays of equivalent length. These texts were analyzed using Open Brain AI, an online computational tool, to extract measures of phonological, morphological, syntactic, and lexical constituents. Despite AI-generated texts appearing to mimic human speech, the results revealed significant differences across multiple linguistic features such as consonants, word stress, nouns, verbs, pronouns, direct objects, prepositional modifiers, and use of difficult words among others. These findings underscore the importance of integrating automated tools for efficient language assessment, reducing time and effort in data analysis. Moreover, they emphasize the necessity for enhanced training methodologies to improve the capacity of AI for producing more human-like text.
• 摘要：虽然近年来对 ChatGPT 的研究非常广泛，但很少有研究系统地量化和比较人类书写和人工智能 (AI) 生成的语言之间的语言特征。本研究旨在调查各种语言成分在这两种文本中的体现方式，以评估人工智能模仿人类写作的能力。以人类撰写的文章为基准，我们促使 ChatGPT 生成长度相当的文章。我们使用在线计算工具 Open Brain AI 分析这些文本，以提取语音、形态、句法和词汇成分的度量。尽管人工智能生成的文本似乎模仿了人类的语音，但结果显示，辅音、单词重音、名词、动词、代词、直接宾语、介词修饰语和难词的使用等多种语言特征存在显著差异。这些发现强调了整合自动化工具以进行高效语言评估、减少数据分析时间和精力的重要性。此外，他们强调加强训练方法的必要性，以提高人工智能生成更像人类的文本的能力。

Title: Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction

• Authors: Amanda Dsouza, Christopher Glaze, Changho Shin, Frederic Sala
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03651
• Pdf URL: https://arxiv.org/pdf/2407.03651
• Copy Paste: [[2407.03651]] Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction(https://arxiv.org/abs/2407.03651)
• Keywords: language model, gpt, long context
• Abstract: Large language models are prominently used in real-world applications, often tasked with reasoning over large volumes of documents. An exciting development in this space is models boasting extended context capabilities, with some accommodating over 2 million tokens. Such long context model capabilities remain uncertain in production systems, motivating the need to benchmark their performance on real world use cases. We address this challenge by proposing SWiM, an evaluation framework that addresses the limitations of standard tests. Testing the framework on eight long context models, we find that even strong models such as GPT-4 and Claude 3 Opus degrade in performance when information is present in the middle of the context window (lost-in-the-middle effect). Next, in addition to our benchmark, we propose medoid voting, a simple, but effective training-free approach that helps alleviate this effect, by generating responses a few times, each time randomly permuting documents in the context, and selecting the medoid answer. We evaluate medoid voting on single document QA tasks, achieving up to a 24% lift in accuracy.
• 摘要：大型语言模型在实际应用中被广泛使用，通常用于对大量文档进行推理。该领域一个令人兴奋的发展是具有扩展上下文功能的模型，其中一些模型可容纳超过 200 万个标记。这种长上下文模型功能在生产系统中仍不确定，这促使我们需要在实际用例上对其性能进行基准测试。我们通过提出 SWiM 来应对这一挑战，SWiM 是一个解决标准测试局限性的评估框架。在八个长上下文模型上测试该框架后，我们发现，即使是 GPT-4 和 Claude 3 Opus 等强大的模型，当信息出现在上下文窗口中间时，性能也会下降（中间丢失效应）。接下来，除了我们的基准之外，我们还提出了 medoid 投票，这是一种简单但有效的无需训练的方法，有助于缓解这种影响，通过生成几次响应，每次随机排列上下文中的文档，并选择 medoid 答案。我们在单个文档 QA 任务上评估了 medoid 投票，准确率提高了 24%。

Title: GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels

• Authors: Jianhao Yan, Pingchuan Yan, Yulong Chen, Judy Li, Xianchao Zhu, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03658
• Pdf URL: https://arxiv.org/pdf/2407.03658
• Copy Paste: [[2407.03658]] GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels(https://arxiv.org/abs/2407.03658)
• Keywords: language model, gpt, llm
• Abstract: This study comprehensively evaluates the translation quality of Large Language Models (LLMs), specifically GPT-4, against human translators of varying expertise levels across multiple language pairs and domains. Through carefully designed annotation rounds, we find that GPT-4 performs comparably to junior translators in terms of total errors made but lags behind medium and senior translators. We also observe the imbalanced performance across different languages and domains, with GPT-4's translation capability gradually weakening from resource-rich to resource-poor directions. In addition, we qualitatively study the translation given by GPT-4 and human translators, and find that GPT-4 translator suffers from literal translations, but human translators sometimes overthink the background information. To our knowledge, this study is the first to evaluate LLMs against human translators and analyze the systematic differences between their outputs, providing valuable insights into the current state of LLM-based translation and its potential limitations.
• 摘要：本研究全面评估了大型语言模型 (LLM)，特别是 GPT-4，与不同专业水平的人工翻译在多个语言对和领域的翻译质量。通过精心设计的注释轮次，我们发现 GPT-4 在总错误率方面的表现与初级翻译相当，但落后于中级和高级翻译。我们还观察到不同语言和领域的表现不平衡，GPT-4 的翻译能力从资源丰富的方向逐渐减弱到资源贫乏的方向。此外，我们定性研究了 GPT-4 和人工翻译给出的翻译，发现 GPT-4 翻译存在直译问题，但人工翻译有时会过度考虑背景信息。据我们所知，这项研究是第一个将 LLM 与人工翻译进行比较并分析其输出之间的系统性差异的研究，为基于 LLM 的翻译的现状及其潜在局限性提供了有价值的见解。

Title: Improving Self Consistency in LLMs through Probabilistic Tokenization

• Authors: Ashutosh Sathe, Divyanshu Aggarwal, Sunayana Sitaram
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03678
• Pdf URL: https://arxiv.org/pdf/2407.03678
• Copy Paste: [[2407.03678]] Improving Self Consistency in LLMs through Probabilistic Tokenization(https://arxiv.org/abs/2407.03678)
• Keywords: language model, llm
• Abstract: Prior research has demonstrated noticeable performance gains through the use of probabilistic tokenizations, an approach that involves employing multiple tokenizations of the same input string during the training phase of a language model. Despite these promising findings, modern large language models (LLMs) have yet to be trained using probabilistic tokenizations. Interestingly, while the tokenizers of these contemporary LLMs have the capability to generate multiple tokenizations, this property remains underutilized. In this work, we propose a novel method to leverage the multiple tokenization capabilities of modern LLM tokenizers, aiming to enhance the self-consistency of LLMs in reasoning tasks. Our experiments indicate that when utilizing probabilistic tokenizations, LLMs generate logically diverse reasoning paths, moving beyond mere surface-level linguistic diversity.We carefully study probabilistic tokenization and offer insights to explain the self consistency improvements it brings through extensive experimentation on 5 LLM families and 4 reasoning benchmarks.
• 摘要：先前的研究表明，通过使用概率标记化，性能得到了显著提升。概率标记化是一种在语言模型的训练阶段对同一输入字符串进行多次标记化的方法。尽管有这些令人鼓舞的发现，但现代大型语言模型 (LLM) 尚未使用概率标记化进行训练。有趣的是，虽然这些当代 LLM 的标记器能够生成多个标记化，但这一特性仍未得到充分利用。在这项工作中，我们提出了一种新方法来利用现代 LLM 标记器的多重标记化功能，旨在增强 LLM 在推理任务中的自洽性。我们的实验表明，当使用概率标记化时，LLM 会生成逻辑上不同的推理路径，而不仅仅是表面级别的语言多样性。我们仔细研究了概率标记化，并通过对 5 个 LLM 系列和 4 个推理基准进行大量实验，提供了解释其带来的自洽性改进的见解。

Title: STOC-TOT: Stochastic Tree-of-Thought with Constrained Decoding for Complex Reasoning in Multi-Hop Question Answering

• Authors: Zhenyu Bi, Daniel Hajialigol, Zhongkai Sun, Jie Hao, Xuan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03687
• Pdf URL: https://arxiv.org/pdf/2407.03687
• Copy Paste: [[2407.03687]] STOC-TOT: Stochastic Tree-of-Thought with Constrained Decoding for Complex Reasoning in Multi-Hop Question Answering(https://arxiv.org/abs/2407.03687)
• Keywords: language model, hallucination, prompt, chain-of-thought, tree-of-thought
• Abstract: Multi-hop question answering (MHQA) requires a model to retrieve and integrate information from multiple passages to answer a complex question. Recent systems leverage the power of large language models and integrate evidence retrieval with reasoning prompts (e.g., chain-of-thought reasoning) for the MHQA task. However, the complexities in the question types (bridge v.s. comparison questions) and the reasoning types (sequential v.s. parallel reasonings) require more novel and fine-grained prompting methods to enhance the performance of MHQA under the zero-shot setting. In this paper, we propose STOC-TOT, a stochastic tree-of-thought reasoning prompting method with constrained decoding for MHQA and conduct a detailed comparison with other reasoning prompts on different question types and reasoning types. Specifically, we construct a tree-like reasoning structure by prompting the model to break down the original question into smaller sub-questions to form different reasoning paths. In addition, we prompt the model to provide a probability estimation for each reasoning path at each reasoning step. At answer time, we conduct constrained decoding on the model to generate more grounded answers and reduce hallucination. Experiments comparing STOC-TOT with two MHQA datasets and five large language models showed that our framework outperforms other reasoning prompts by a significant margin.
• 摘要：多跳问答 (MHQA) 需要模型从多个段落中检索和整合信息来回答复杂问题。最近的系统利用大型语言模型的强大功能，将证据检索与推理提示（如思路链推理）相结合以完成 MHQA 任务。然而，问题类型（桥梁与比较问题）和推理类型（顺序与并行推理）的复杂性需要更多新颖、更细粒度的提示方法来提升零样本设置下 MHQA 的性能。在本文中，我们提出了一种用于 MHQA 的带约束解码的随机思路树推理提示方法 STOC-TOT，并在不同类型的问题类型和推理类型上与其他推理提示进行了详细的比较。具体而言，我们通过提示模型将原始问题分解为更小的子问题以形成不同的推理路径来构建树状推理结构。此外，我们提示模型在每个推理步骤中为每个推理路径提供概率估计。在回答问题时，我们对模型进行约束解码，以生成更扎实的答案并减少幻觉。将 STOC-TOT 与两个 MHQA 数据集和五个大型语言模型进行比较的实验表明，我们的框架比其他推理提示的表现要好得多。

Title: Text2TimeSeries: Enhancing Financial Forecasting through Time Series Prediction Updates with Event-Driven Insights from Large Language Models

• Authors: Litton Jose Kurisinkel, Pruthwik Mishra, Yue Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03689
• Pdf URL: https://arxiv.org/pdf/2407.03689
• Copy Paste: [[2407.03689]] Text2TimeSeries: Enhancing Financial Forecasting through Time Series Prediction Updates with Event-Driven Insights from Large Language Models(https://arxiv.org/abs/2407.03689)
• Keywords: language model
• Abstract: Time series models, typically trained on numerical data, are designed to forecast future values. These models often rely on weighted averaging techniques over time intervals. However, real-world time series data is seldom isolated and is frequently influenced by non-numeric factors. For instance, stock price fluctuations are impacted by daily random events in the broader world, with each event exerting a unique influence on price signals. Previously, forecasts in financial markets have been approached in two main ways: either as time-series problems over price sequence or sentiment analysis tasks. The sentiment analysis tasks aim to determine whether news events will have a positive or negative impact on stock prices, often categorizing them into discrete labels. Recognizing the need for a more comprehensive approach to accurately model time series prediction, we propose a collaborative modeling framework that incorporates textual information about relevant events for predictions. Specifically, we leverage the intuition of large language models about future changes to update real number time series predictions. We evaluated the effectiveness of our approach on financial market data.
• 摘要：时间序列模型通常基于数值数据进行训练，旨在预测未来值。这些模型通常依赖于时间间隔内的加权平均技术。然而，现实世界的时间序列数据很少是孤立的，而且经常受到非数字因素的影响。例如，股票价格波动受到更广阔世界中每日随机事件的影响，每个事件都会对价格信号产生独特的影响。以前，金融市场的预测主要以两种方式进行：要么是价格序列的时间序列问题，要么是情绪分析任务。情绪分析任务旨在确定新闻事件对股价是产生积极影响还是消极影响，通常将其归类为离散标签。认识到需要一种更全面的方法来准确地对时间序列预测进行建模，我们提出了一个协作建模框架，该框架结合了有关相关事件的文本信息以进行预测。具体来说，我们利用大型语言模型对未来变化的直觉来更新实数时间序列预测。我们评估了我们的方法对金融市场数据的有效性。

Title: Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques

• Authors: Anar Yeginbergen, Maite Oronoz, Rodrigo Agerri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03748
• Pdf URL: https://arxiv.org/pdf/2407.03748
• Copy Paste: [[2407.03748]] Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques(https://arxiv.org/abs/2407.03748)
• Keywords: language model, prompt
• Abstract: Recent research on sequence labelling has been exploring different strategies to mitigate the lack of manually annotated data for the large majority of the world languages. Among others, the most successful approaches have been based on (i) the cross-lingual transfer capabilities of multilingual pre-trained language models (model-transfer), (ii) data translation and label projection (data-transfer) and (iii), prompt-based learning by reusing the mask objective to exploit the few-shot capabilities of pre-trained language models (few-shot). Previous work seems to conclude that model-transfer outperforms data-transfer methods and that few-shot techniques based on prompting are superior to updating the model's weights via fine-tuning. In this paper, we empirically demonstrate that, for Argument Mining, a sequence labelling task which requires the detection of long and complex discourse structures, previous insights on cross-lingual transfer or few-shot learning do not apply. Contrary to previous work, we show that for Argument Mining data transfer obtains better results than model-transfer and that fine-tuning outperforms few-shot methods. Regarding the former, the domain of the dataset used for data-transfer seems to be a deciding factor, while, for few-shot, the type of task (length and complexity of the sequence spans) and sampling method prove to be crucial.
• 摘要：序列标记的最新研究一直在探索不同的策略，以缓解世界上大多数语言缺乏手动注释数据的问题。其中，最成功的方法基于 (i) 多语言预训练语言模型的跨语言迁移能力 (模型迁移)、(ii) 数据翻译和标签投影 (数据传输) 和 (iii) 基于提示的学习，通过重用掩码目标来利用预训练语言模型的少样本能力 (少样本)。先前的研究似乎得出结论，模型迁移优于数据传输方法，而基于提示的少样本技术优于通过微调更新模型的权重。在本文中，我们通过实证证明，对于论证挖掘这种需要检测长而复杂的话语结构的序列标记任务，先前关于跨语言迁移或少样本学习的见解并不适用。与之前的研究相反，我们表明，对于论证挖掘，数据传输比模型传输获得更好的结果，而微调优于少样本方法。对于前者，用于数据传输的数据集的域似乎是一个决定性因素，而对于少样本，任务类型（序列跨度的长度和复杂性）和采样方法被证明是至关重要的。

Title: Convolutional vs Large Language Models for Software Log Classification in Edge-Deployable Cellular Network Testing

• Authors: Achintha Ihalage, Sayed M. Taheri, Faris Muhammad, Hamed Al-Raweshidy
• Subjects: cs.CL, cs.AI, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2407.03759
• Pdf URL: https://arxiv.org/pdf/2407.03759
• Copy Paste: [[2407.03759]] Convolutional vs Large Language Models for Software Log Classification in Edge-Deployable Cellular Network Testing(https://arxiv.org/abs/2407.03759)
• Keywords: language model, llm
• Abstract: Software logs generated by sophisticated network emulators in the telecommunications industry, such as VIAVI TM500, are extremely complex, often comprising tens of thousands of text lines with minimal resemblance to natural language. Only specialised expert engineers can decipher such logs and troubleshoot defects in test runs. While AI offers a promising solution for automating defect triage, potentially leading to massive revenue savings for companies, state-of-the-art large language models (LLMs) suffer from significant drawbacks in this specialised domain. These include a constrained context window, limited applicability to text beyond natural language, and high inference costs. To address these limitations, we propose a compact convolutional neural network (CNN) architecture that offers a context window spanning up to 200,000 characters and achieves over 96% accuracy (F1>0.9) in classifying multifaceted software logs into various layers in the telecommunications protocol stack. Specifically, the proposed model is capable of identifying defects in test runs and triaging them to the relevant department, formerly a manual engineering process that required expert knowledge. We evaluate several LLMs; LLaMA2-7B, Mixtral 8x7B, Flan-T5, BERT and BigBird, and experimentally demonstrate their shortcomings in our specialized application. Despite being lightweight, our CNN significantly outperforms LLM-based approaches in telecommunications log classification while minimizing the cost of production. Our defect triaging AI model is deployable on edge devices without dedicated hardware and widely applicable across software logs in various industries.
• 摘要：电信行业中复杂的网络模拟器（例如 VIAVI TM500）生成的软件日志极其复杂，通常包含数万行文本，与自然语言的相似性极低。只有专业的专家工程师才能解读此类日志并在测试运行中排除缺陷。虽然人工智能为自动缺陷分类提供了一种有前途的解决方案，可能为公司节省大量收入，但最先进的大型语言模型 (LLM) 在这一专业领域存在重大缺陷。这些缺陷包括上下文窗口受限、对自然语言以外的文本的适用性有限以及推理成本高。为了解决这些限制，我们提出了一种紧凑的卷积神经网络 (CNN) 架构，它提供了一个多达 200,000 个字符的上下文窗口，并且在将多方面的软件日志分类到电信协议堆栈中的各个层时实现了超过 96% 的准确率（F1>0.9）。具体来说，所提出的模型能够识别测试运行中的缺陷并将其分类到相关部门，而这以前是一个需要专业知识的手动工程过程。我们评估了几个 LLM：LLaMA2-7B、Mixtral 8x7B、Flan-T5、BERT 和 BigBird，并通过实验证明了它们在我们的专业应用中的缺点。尽管重量轻，但我们的 CNN 在电信日志分类方面的表现明显优于基于 LLM 的方法，同时最大限度地降低了生产成本。我们的缺陷分类 AI 模型可以在没有专用硬件的边缘设备上部署，并且广泛应用于各个行业的软件日志。

Title: Functional Faithfulness in the Wild: Circuit Discovery with Differentiable Computation Graph Pruning

• Authors: Lei Yu, Jingcheng Niu, Zining Zhu, Gerald Penn
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03779
• Pdf URL: https://arxiv.org/pdf/2407.03779
• Copy Paste: [[2407.03779]] Functional Faithfulness in the Wild: Circuit Discovery with Differentiable Computation Graph Pruning(https://arxiv.org/abs/2407.03779)
• Keywords: language model
• Abstract: In this paper, we introduce a comprehensive reformulation of the task known as Circuit Discovery, along with DiscoGP, a novel and effective algorithm based on differentiable masking for discovering circuits. Circuit discovery is the task of interpreting the computational mechanisms of language models (LMs) by dissecting their functions and capabilities into sparse subnetworks (circuits). We identified two major limitations in existing circuit discovery efforts: (1) a dichotomy between weight-based and connection-edge-based approaches forces researchers to choose between pruning connections or weights, thereby limiting the scope of mechanistic interpretation of LMs; (2) algorithms based on activation patching tend to identify circuits that are neither functionally faithful nor complete. The performance of these identified circuits is substantially reduced, often resulting in near-random performance in isolation. Furthermore, the complement of the circuit -- i.e., the original LM with the identified circuit removed -- still retains adequate performance, indicating that essential components of a complete circuits are missed by existing methods. DiscoGP successfully addresses the two aforementioned issues and demonstrates state-of-the-art faithfulness, completeness, and sparsity. The effectiveness of the algorithm and its novel structure open up new avenues of gathering new insights into the internal workings of generative AI.
• 摘要：在本文中，我们介绍了一种称为“电路发现”任务的全面重构，以及一种基于可微分掩码的用于发现电路的新型有效算法 DiscoGP。电路发现是通过将语言模型 (LM) 的功能和能力分解为稀疏子网络 (电路) 来解释语言模型 (LM) 的计算机制的任务。我们发现了现有电路发现工作中的两个主要限制：(1) 基于权重和基于连接边缘的方法之间的二分法迫使研究人员在修剪连接或权重之间做出选择，从而限制了 LM 的机械解释范围；(2) 基于激活修补的算法往往会识别功能不忠实也不完整的电路。这些已识别电路的性能大幅降低，通常导致孤立时的性能接近随机。此外，电路的补集（即删除了已识别电路的原始 LM）仍然保留了足够的性能，这表明现有方法遗漏了完整电路的基本组件。 DiscoGP 成功解决了上述两个问题，并展示了最先进的忠实性、完整性和稀疏性。该算法的有效性及其新颖的结构开辟了收集有关生成式 AI 内部运作的新见解的新途径。

Title: M$\mathbf5$ -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

• Authors: Florian Schneider, Sunayana Sitaram
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] M$\mathbf5$ -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks(https://arxiv.org/abs/)
• Keywords: language model, gpt, llm, chat
• Abstract: Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and $41$ languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.
• 摘要：自 ChatGPT 发布以来，自然语言处理领域经历了快速发展，尤其是在大型语言模型 (LLM) 及其多模态对应物大型多模态模型 (LMM) 方面。尽管 LLM 具有令人印象深刻的功能，但它们在不同语言和文化背景下的性能差异往往很大，这一点在各种纯文本基准测试中都得到了证实。然而，目前的研究缺乏针对多模态视觉语言设置的此类基准测试。这项工作通过引入 M5 填补了这一空白，M5 是第一个旨在评估多语言和多文化背景下的多种视觉语言任务上的 LMM 的综合基准测试。M5 包括八个数据集，涵盖五项任务和 41 种语言，重点关注代表性不足的语言和文化多样化的图像。此外，我们引入了两个新数据集 M5-VGR 和 M5-VLOD，包括一个新的 Visio-Linguistic 离群值检测任务，其中所有评估的开源模型都未能显著超越随机基线。通过广泛的评估和分析，我们强调了高资源语言和低资源语言之间存在显著的任务无关性能差异。此外，我们还表明，在多语言环境中，较大的模型并不一定比较小的模型表现更好。

Title: Cognitive Modeling with Scaffolded LLMs: A Case Study of Referential Expression Generation

• Authors: Polina Tsvilodub, Michael Franke, Fausto Carcassi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03805
• Pdf URL: https://arxiv.org/pdf/2407.03805
• Copy Paste: [[2407.03805]] Cognitive Modeling with Scaffolded LLMs: A Case Study of Referential Expression Generation(https://arxiv.org/abs/2407.03805)
• Keywords: gpt, llm
• Abstract: To what extent can LLMs be used as part of a cognitive model of language generation? In this paper, we approach this question by exploring a neuro-symbolic implementation of an algorithmic cognitive model of referential expression generation by Dale & Reiter (1995). The symbolic task analysis implements the generation as an iterative procedure that scaffolds symbolic and gpt-3.5-turbo-based modules. We compare this implementation to an ablated model and a one-shot LLM-only baseline on the A3DS dataset (Tsvilodub & Franke, 2023). We find that our hybrid approach is cognitively plausible and performs well in complex contexts, while allowing for more open-ended modeling of language generation in a larger domain.
• 摘要：LLM 在多大程度上可以用作语言生成认知模型的一部分？在本文中，我们通过探索 Dale & Reiter (1995) 提出的指称表达生成算法认知模型的神经符号实现来解决这个问题。符号任务分析将生成实现为一个迭代过程，为符号和基于 GPT-3.5-turbo 的模块提供支持。我们将此实现与 A3DS 数据集上的消融模型和一次性 LLM 基线进行了比较 (Tsvilodub & Franke, 2023)。我们发现我们的混合方法在认知上是合理的，并且在复杂的环境中表现良好，同时允许在更大的领域中对语言生成进行更开放的建模。

Title: ConText at WASSA 2024 Empathy and Personality Shared Task: History-Dependent Embedding Utterance Representations for Empathy and Emotion Prediction in Conversations

• Authors: Patrícia Pereira, Helena Moniz, Joao Paulo Carvalho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03818
• Pdf URL: https://arxiv.org/pdf/2407.03818
• Copy Paste: [[2407.03818]] ConText at WASSA 2024 Empathy and Personality Shared Task: History-Dependent Embedding Utterance Representations for Empathy and Emotion Prediction in Conversations(https://arxiv.org/abs/2407.03818)
• Keywords: language model, agent
• Abstract: Empathy and emotion prediction are key components in the development of effective and empathetic agents, amongst several other applications. The WASSA shared task on empathy and emotion prediction in interactions presents an opportunity to benchmark approaches to these tasks. Appropriately selecting and representing the historical context is crucial in the modelling of empathy and emotion in conversations. In our submissions, we model empathy, emotion polarity and emotion intensity of each utterance in a conversation by feeding the utterance to be classified together with its conversational context, i.e., a certain number of previous conversational turns, as input to an encoder Pre-trained Language Model, to which we append a regression head for prediction. We also model perceived counterparty empathy of each interlocutor by feeding all utterances from the conversation and a token identifying the interlocutor for which we are predicting the empathy. Our system officially ranked $1^{st}$ at the CONV-turn track and $2^{nd}$ at the CONV-dialog track.
• 摘要：同理心和情绪预测是开发有效和有同理心的代理的关键组成部分，还有其他一些应用。WASSA 在交互中关于同理心和情绪预测的共享任务为对这些任务的方法进行了基准测试提供了机会。适当地选择和表示历史背景对于在对话中模拟同理心和情绪至关重要。在我们的提交中，我们通过将要分类的话语及其对话背景（即一定数量的先前对话轮次）作为输入输入到编码器预训练语言模型中，来模拟对话中每句话的同理心、情绪极性和情绪强度，我们在该模型上附加了一个回归头以进行预测。我们还通过输入对话中的所有话语和一个识别我们预测同理心的对话者的标记来模拟每个对话者感知到的对方同理心。我们的系统在 CONV-turn 轨道上正式排名第一，在 CONV-dialog 轨道上排名第二。

Title: On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation

• Authors: John Mendonça, Alon Lavie, Isabel Trancoso
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03841
• Pdf URL: https://arxiv.org/pdf/2407.03841
• Copy Paste: [[2407.03841]] On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation(https://arxiv.org/abs/2407.03841)
• Keywords: language model, gpt, llm, chat
• Abstract: Large Language Models (LLMs) have showcased remarkable capabilities in various Natural Language Processing tasks. For automatic open-domain dialogue evaluation in particular, LLMs have been seamlessly integrated into evaluation frameworks, and together with human evaluation, compose the backbone of most evaluations. However, existing evaluation benchmarks often rely on outdated datasets and evaluate aspects like Fluency and Relevance, which fail to adequately capture the capabilities and limitations of state-of-the-art chatbot models. This paper critically examines current evaluation benchmarks, highlighting that the use of older response generators and quality aspects fail to accurately reflect modern chatbot capabilities. A small annotation experiment on a recent LLM-generated dataset (SODA) reveals that LLM evaluators such as GPT-4 struggle to detect actual deficiencies in dialogues generated by current LLM chatbots.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理任务中展现出了卓越的能力。特别是对于自动开放域对话评估，LLM 已无缝集成到评估框架中，并与人工评估一起构成了大多数评估的支柱。然而，现有的评估基准通常依赖于过时的数据集并评估流畅度和相关性等方面，这些方面无法充分捕捉到最先进的聊天机器人模型的能力和局限性。本文批判性地审查了当前的评估基准，强调使用较旧的响应生成器和质量方面无法准确反映现代聊天机器人的能力。在最近的 LLM 生成的数据集 (SODA) 上进行的一项小型注释实验表明，GPT-4 等 LLM 评估器很难检测到当前 LLM 聊天机器人生成的对话中的实际缺陷。

Title: HYBRINFOX at CheckThat! 2024 -- Task 1: Enhancing Language Models with Structured Information for Check-Worthiness Estimation

• Authors: Géraud Faye, Morgane Casanova, Benjamin Icard, Julien Chanson, Guillaume Gadek, Guillaume Gravier, Paul Égré
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] HYBRINFOX at CheckThat! 2024 -- Task 1: Enhancing Language Models with Structured Information for Check-Worthiness Estimation(https://arxiv.org/abs/)
• Keywords: language model
• Abstract: This paper summarizes the experiments and results of the HYBRINFOX team for the CheckThat! 2024 - Task 1 competition. We propose an approach enriching Language Models such as RoBERTa with embeddings produced by triples (subject ; predicate ; object) extracted from the text sentences. Our analysis of the developmental data shows that this method improves the performance of Language Models alone. On the evaluation data, its best performance was in English, where it achieved an F1 score of 71.1 and ranked 12th out of 27 candidates. On the other languages (Dutch and Arabic), it obtained more mixed results. Future research tracks are identified toward adapting this processing pipeline to more recent Large Language Models.
• 摘要：本文总结了 HYBRINFOX 团队在 CheckThat! 2024 - Task 1 竞赛中的实验和结果。我们提出了一种方法，通过从文本句子中提取的三元组（主语；谓语；宾语）生成的嵌入来丰富 RoBERTa 等语言模型。我们对开发数据的分析表明，这种方法单独提高了语言模型的性能。在评估数据上，其最佳表现是在英语中，F1 得分为 71.1，在 27 个候选中排名第 12。在其他语言（荷兰语和阿拉伯语）上，它获得了更多混合结果。未来的研究方向是使该处理流程适应更新的大型语言模型。

Title: Anthropocentric bias and the possibility of artificial cognition

• Authors: Raphaël Millière, Charles Rathkopf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03859
• Pdf URL: https://arxiv.org/pdf/2407.03859
• Copy Paste: [[2407.03859]] Anthropocentric bias and the possibility of artificial cognition(https://arxiv.org/abs/2407.03859)
• Keywords: language model, llm
• Abstract: Evaluating the cognitive capacities of large language models (LLMs) requires overcoming not only anthropomorphic but also anthropocentric biases. This article identifies two types of anthropocentric bias that have been neglected: overlooking how auxiliary factors can impede LLM performance despite competence (Type-I), and dismissing LLM mechanistic strategies that differ from those of humans as not genuinely competent (Type-II). Mitigating these biases necessitates an empirically-driven, iterative approach to mapping cognitive tasks to LLM-specific capacities and mechanisms, which can be done by supplementing carefully designed behavioral experiments with mechanistic studies.
• 摘要：评估大型语言模型 (LLM) 的认知能力不仅需要克服拟人化偏见，还需要克服人类中心主义偏见。本文确定了两种被忽视的人类中心主义偏见：忽视辅助因素如何阻碍 LLM 的表现（尽管有能力）（类型 I），以及认为与人类不同的 LLM 机械策略不是真正的能力（类型 II）。减轻这些偏见需要一种以经验为导向的迭代方法，将认知任务映射到 LLM 特定的能力和机制，这可以通过用机械研究补充精心设计的行为实验来实现。

Title: Planning with Large Language Models for Conversational Agents

• Authors: Zhigen Li, Jianxiang Peng, Yanmeng Wang, Tianhao Shen, Minghui Zhang, Linxi Su, Shang Wu, Yihang Wu, Yuqian Wang, Ye Wang, Wei Hu, Jianfeng Li, Shaojun Wang, Jing Xiao, Deyi Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03884
• Pdf URL: https://arxiv.org/pdf/2407.03884
• Copy Paste: [[2407.03884]] Planning with Large Language Models for Conversational Agents(https://arxiv.org/abs/2407.03884)
• Keywords: language model, llm, agent
• Abstract: Controllability and proactivity are crucial properties of autonomous conversational agents (CAs). Controllability requires the CAs to follow the standard operating procedures (SOPs), such as verifying identity before activating credit cards. Proactivity requires the CAs to guide the conversation towards the goal during user uncooperation, such as persuasive dialogue. Existing research cannot be unified with controllability, proactivity, and low manual annotation. To bridge this gap, we propose a new framework for planning-based conversational agents (PCA) powered by large language models (LLMs), which only requires humans to define tasks and goals for the LLMs. Before conversation, LLM plans the core and necessary SOP for dialogue offline. During the conversation, LLM plans the best action path online referring to the SOP, and generates responses to achieve process controllability. Subsequently, we propose a semi-automatic dialogue data creation framework and curate a high-quality dialogue dataset (PCA-D). Meanwhile, we develop multiple variants and evaluation metrics for PCA, e.g., planning with Monte Carlo Tree Search (PCA-M), which searches for the optimal dialogue action while satisfying SOP constraints and achieving the proactive of the dialogue. Experiment results show that LLMs finetuned on PCA-D can significantly improve the performance and generalize to unseen domains. PCA-M outperforms other CoT and ToT baselines in terms of conversation controllability, proactivity, task success rate, and overall logical coherence, and is applicable in industry dialogue scenarios. The dataset and codes are available at XXXX.
• 摘要：可控性和主动性是自主对话代理（CA）的关键属性。可控性要求CA遵循标准操作程序（SOP），例如在激活信用卡之前验证身份。主动性要求CA在用户不合作的情况下引导对话朝着目标前进，例如说服性对话。现有研究无法将可控性、主动性和低手动注释统一起来。为了弥补这一差距，我们提出了一种由大型语言模型（LLM）驱动的基于规划的对话代理（PCA）新框架，该框架仅需要人类为LLM定义任务和目标。在对话之前，LLM离线规划对话的核心和必要SOP。在对话过程中，LLM参考SOP在线规划最佳行动路径，并生成响应以实现过程可控性。随后，我们提出了一个半自动对话数据创建框架并整理了一个高质量的对话数据集（PCA-D）。同时，我们为 PCA 开发了多种变体和评估指标，例如使用蒙特卡洛树搜索 (PCA-M) 进行规划，它在满足 SOP 约束并实现对话的主动性的同时搜索最佳对话动作。实验结果表明，在 PCA-D 上微调的 LLM 可以显著提高性能并推广到未知领域。PCA-M 在对话可控性、主动性、任务成功率和整体逻辑连贯性方面优于其他 CoT 和 ToT 基线，适用于行业对话场景。数据集和代码可在 XXXX 上找到。

Title: TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models

• Authors: Jiahuan Cao, Dezhi Peng, Peirong Zhang, Yongxin Shi, Yang Liu, Kai Ding, Lianwen Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03937
• Pdf URL: https://arxiv.org/pdf/2407.03937
• Copy Paste: [[2407.03937]] TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models(https://arxiv.org/abs/2407.03937)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose \textbf{TongGu} (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu's superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset will be public available.
• 摘要：古汉语是通向中国古代丰富遗产和智慧的大门，但其复杂性对大多数不具备专业知识的现代人来说构成了巨大的理解障碍。虽然大型语言模型 (LLM) 在自然语言处理 (NLP) 方面表现出色，但它们在古汉语理解 (CCU) 方面却举步维艰，尤其是在数据要求高和知识密集型任务中。为了解决这一困境，我们提出了 \textbf{TongGu}（意为理解古今），这是第一个特定于 CCU 的 LLM，由三个核心贡献支撑。首先，我们构建了一个两阶段指令调优数据集 ACCN-INS，该数据集源自丰富的古汉语语料库，旨在释放 LLM 的全部 CCU 潜力。其次，我们提出了冗余感知调优 (RAT) 来防止灾难性遗忘，使 TongGu 能够在保留基础知识的同时获得新功能。第三，我们提出了一种基于知识基础的 CCU 检索增强生成 (CCU-RAG) 技术来减少幻觉。在 24 个不同的 CCU 任务中进行的大量实验验证了 TongGu 的卓越能力，凸显了 RAT 和 CCU-RAG 的有效性。该模型和数据集将公开发布。

Title: A framework for annotating and modelling intentions behind metaphor use

• Authors: Gianluca Michelli, Xiaoyu Tong, Ekaterina Shutova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03952
• Pdf URL: https://arxiv.org/pdf/2407.03952
• Copy Paste: [[2407.03952]] A framework for annotating and modelling intentions behind metaphor use(https://arxiv.org/abs/2407.03952)
• Keywords: language model, llm
• Abstract: Metaphors are part of everyday language and shape the way in which we conceptualize the world. Moreover, they play a multifaceted role in communication, making their understanding and generation a challenging task for language models (LMs). While there has been extensive work in the literature linking metaphor to the fulfilment of individual intentions, no comprehensive taxonomy of such intentions, suitable for natural language processing (NLP) applications, is available to present day. In this paper, we propose a novel taxonomy of intentions commonly attributed to metaphor, which comprises 9 categories. We also release the first dataset annotated for intentions behind metaphor use. Finally, we use this dataset to test the capability of large language models (LLMs) in inferring the intentions behind metaphor use, in zero- and in-context few-shot settings. Our experiments show that this is still a challenge for LLMs.
• 摘要：隐喻是日常语言的一部分，它塑造了我们概念化世界的方式。此外，隐喻在交流中发挥着多方面的作用，因此对语言模型 (LM) 来说，理解和生成隐喻是一项具有挑战性的任务。尽管文献中有大量研究将隐喻与个人意图的实现联系起来，但目前还没有适用于自然语言处理 (NLP) 应用的此类意图的综合分类法。在本文中，我们提出了一种通常归因于隐喻的意图的新分类法，该分类法包含 9 个类别。我们还发布了第一个注释了隐喻使用背后意图的数据集。最后，我们使用该数据集测试大型语言模型 (LLM) 在零样本和上下文小样本设置中推断隐喻使用背后意图的能力。我们的实验表明，这对 LLM 来说仍然是一个挑战。

Title: Meta-prompting Optimized Retrieval-augmented Generation

• Authors: João Rodrigues, António Branco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Meta-prompting Optimized Retrieval-augmented Generation(https://arxiv.org/abs/)
• Keywords: language model, prompt, retrieval-augmented generation
• Abstract: Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. The excessive volume of retrieved content, the possible dispersion of its parts, or their out of focus range may happen nevertheless to eventually have a detrimental rather than an incremental effect. To mitigate this issue and improve retrieval-augmented generation, we propose a method to refine the retrieved content before it is included in the prompt by resorting to meta-prompting optimization. Put to empirical test with the demanding multi-hop question answering task from the StrategyQA dataset, the evaluation results indicate that this method outperforms a similar retrieval-augmented system but without this method by over 30%.
• 摘要：检索增强生成借助从外部来源检索的内容，以利用大型语言模型在下游任务中的性能。然而，检索内容的数量过多、其各部分可能分散或它们超出焦点范围，最终可能会产生有害影响，而不是增量影响。为了缓解这个问题并改进检索增强生成，我们提出了一种方法，通过借助元提示优化来优化检索到的内容，然后再将其包含在提示中。使用来自 StrategyQA 数据集的苛刻多跳问答任务进行实证测试，评估结果表明，该方法比没有使用此方法的类似检索增强系统的性能高出 30% 以上。

Title: Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge

• Authors: Young-Jun Lee, Dokyong Lee, Junyoung Youn, Kyeongjin Oh, Byungsoo Ko, Jonghwan Hyeon, Ho-Jin Choi
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.03958
• Pdf URL: https://arxiv.org/pdf/2407.03958
• Copy Paste: [[2407.03958]] Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge(https://arxiv.org/abs/2407.03958)
• Keywords: gpt, chat
• Abstract: Humans share a wide variety of images related to their personal experiences within conversations via instant messaging tools. However, existing works focus on (1) image-sharing behavior in singular sessions, leading to limited long-term social interaction, and (2) a lack of personalized image-sharing behavior. In this work, we introduce Stark, a large-scale long-term multi-modal conversation dataset that covers a wide range of social personas in a multi-modality format, time intervals, and images. To construct Stark automatically, we propose a novel multi-modal contextualization framework, Mcu, that generates long-term multi-modal dialogue distilled from ChatGPT and our proposed Plan-and-Execute image aligner. Using our Stark, we train a multi-modal conversation model, Ultron 7B, which demonstrates impressive visual imagination ability. Furthermore, we demonstrate the effectiveness of our dataset in human evaluation. We make our source code and dataset publicly available.
• 摘要：人们通过即时通讯工具在对话中分享各种与个人经历相关的图像。然而，现有的研究主要关注 (1) 单一会话中的图像共享行为，导致长期社交互动有限，以及 (2) 缺乏个性化的图像共享行为。在这项工作中，我们引入了 Stark，这是一个大规模长期多模态对话数据集，涵盖了多模态格式、时间间隔和图像的各种社交角色。为了自动构建 Stark，我们提出了一个新颖的多模态语境化框架 Mcu，它可以生成从 ChatGPT 和我们提出的计划和执行图像对齐器中提炼出来的长期多模态对话。使用我们的 Stark，我们训练了一个多模态对话模型 Ultron 7B，它展示了令人印象深刻的视觉想象能力。此外，我们证明了我们的数据集在人工评估中的有效性。我们公开了我们的源代码和数据集。

Title: LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

• Authors: LLM-jp: Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, Hiroshi Kataoka, Satoru Katsumata, Daisuke Kawahara, Seiya Kawano, Atsushi Keyaki, Keisuke Kiryu, Hirokazu Kiyomaru, Takashi Kodama, Takahiro Kubo, Yohei Kuga, Ryoma Kumon, Shuhei Kurita, Sadao Kurohashi, Conglong Li, Taiki Maekawa, Hiroshi Matsuda, Yusuke Miyao, Kentaro Mizuki, Sakae Mizuki, Yugo Murawaki, Ryo Nakamura, Taishi Nakamura, Kouta Nakayama, Tomoka Nakazato, Takuro Niitsuma, Jiro Nishitoba, Yusuke Oda, Hayato Ogawa, Takumi Okamoto, Naoaki Okazaki, Yohei Oseki, Shintaro Ozaki, Koki Ryu, Rafal Rzepka, Keisuke Sakaguchi, Shota Sasaki, Satoshi Sekine, Kohei Suda, Saku Sugawara, Issa Sugiura, Hiroaki Sugiyama, Hisami Suzuki, Jun Suzuki, Toyotaro Suzumura, Kensuke Tachibana, Yu Takagi, Kyosuke Takami, Koichi Takeda, Masashi Takeshita, Masahiro Tanaka, Kenjiro Taura, Arseny Tolmachev, Nobuhiro Ueda, Zhen Wan, Shuntaro Yada, Sakiko Yahata, Yuya Yamamoto, Yusuke Yamauchi, Hitomi Yanaka, Rio Yokota, Koichiro Yoshino
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03963
• Pdf URL: https://arxiv.org/pdf/2407.03963
• Copy Paste: [[2407.03963]] LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs(https://arxiv.org/abs/2407.03963)
• Keywords: language model, llm
• Abstract: This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit this https URL.
• 摘要：本文介绍了 LLM-jp，这是一个跨组织研究和开发日语大型语言模型 (LLM) 的项目。LLM-jp 旨在开发开源且强大的日语 LLM，截至本文撰写时，来自学术界和工业界的 1,500 多名参与者正在为此共同努力。本文介绍了 LLM-jp 成立的背景、其活动摘要以及 LLM-jp 开发的 LLM 的技术报告。有关最新活动，请访问此 https URL。

Title: Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models

• Authors: Fuxiang Zhang, Junyou Li, Yi-Chen Li, Zongzhang Zhang, Yang Yu, Deheng Ye
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03964
• Pdf URL: https://arxiv.org/pdf/2407.03964
• Copy Paste: [[2407.03964]] Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models(https://arxiv.org/abs/2407.03964)
• Keywords: language model, llm, prompt
• Abstract: Low sample efficiency is an enduring challenge of reinforcement learning (RL). With the advent of versatile large language models (LLMs), recent works impart common-sense knowledge to accelerate policy learning for RL processes. However, we note that such guidance is often tailored for one specific task but loses generalizability. In this paper, we introduce a framework that harnesses LLMs to extract background knowledge of an environment, which contains general understandings of the entire environment, making various downstream RL tasks benefit from one-time knowledge representation. We ground LLMs by feeding a few pre-collected experiences and requesting them to delineate background knowledge of the environment. Afterward, we represent the output knowledge as potential functions for potential-based reward shaping, which has a good property for maintaining policy optimality from task rewards. We instantiate three variants to prompt LLMs for background knowledge, including writing code, annotating preferences, and assigning goals. Our experiments show that these methods achieve significant sample efficiency improvements in a spectrum of downstream tasks from Minigrid and Crafter domains.
• 摘要：样本效率低是强化学习 (RL) 面临的一个长期挑战。随着多功能大型语言模型 (LLM) 的出现，最近的研究传递了常识性知识，以加速 RL 过程的策略学习。然而，我们注意到，这种指导通常是针对一项特定任务量身定制的，但失去了普遍性。在本文中，我们介绍了一个框架，该框架利用 LLM 来提取环境的背景知识，其中包含对整个环境的一般理解，使各种下游 RL 任务受益于一次性知识表示。我们通过提供一些预先收集的经验并要求它们描述环境的背景知识来奠定 LLM 的基础。之后，我们将输出知识表示为基于潜力的奖励塑造的潜在函数，这具有保持任务奖励的策略最优性的良好特性。我们实例化了三种变体来提示 LLM 的背景知识，包括编写代码、注释偏好和分配目标。我们的实验表明，这些方法在 Minigrid 和 Crafter 域的一系列下游任务中实现了显着的样本效率改进。

Title: LLM Roleplay: Simulating Human-Chatbot Interaction

• Authors: Hovhannes Tamoyan, Hendrik Schuff, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03974
• Pdf URL: https://arxiv.org/pdf/2407.03974
• Copy Paste: [[2407.03974]] LLM Roleplay: Simulating Human-Chatbot Interaction(https://arxiv.org/abs/2407.03974)
• Keywords: language model, llm, chat
• Abstract: The development of chatbots requires collecting a large number of human-chatbot dialogues to reflect the breadth of users' sociodemographic backgrounds and conversational goals. However, the resource requirements to conduct the respective user studies can be prohibitively high and often only allow for a narrow analysis of specific dialogue goals and participant demographics. In this paper, we propose LLM-Roleplay: a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. LLM-Roleplay can be applied to generate dialogues with any type of chatbot and uses large language models (LLMs) to play the role of textually described personas. To validate our method we collect natural human-chatbot dialogues from different sociodemographic groups and conduct a human evaluation to compare real human-chatbot dialogues with our generated dialogues. We compare the abilities of state-of-the-art LLMs in embodying personas and holding a conversation and find that our method can simulate human-chatbot dialogues with a high indistinguishability rate.
• 摘要：聊天机器人的开发需要收集大量的人机对话，以反映用户社会人口背景和对话目标的广度。然而，进行相应的用户研究所需的资源可能高得令人望而却步，而且通常只能对特定的对话目标和参与者的人口统计数据进行狭隘的分析。在本文中，我们提出了 LLM-Roleplay：一种以目标为导向、基于角色的方法，用于自动生成模拟人机交互的多样化多轮对话。LLM-Roleplay 可用于生成与任何类型的聊天机器人的对话，并使用大型语言模型 (LLM) 来扮演文本描述的角色。为了验证我们的方法，我们从不同的社会人口群体中收集了自然的人机对话，并进行了人工评估，以将真实的人机对话与我们生成的对话进行比较。我们比较了最先进的 LLM 在体现角色和进行对话方面的能力，发现我们的方法可以模拟具有高不可区分率的人机对话。

Title: Benchmarking Complex Instruction-Following with Multiple Constraints Composition

• Authors: Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, Minlie Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03978
• Pdf URL: https://arxiv.org/pdf/2407.03978
• Copy Paste: [[2407.03978]] Benchmarking Complex Instruction-Following with Multiple Constraints Composition(https://arxiv.org/abs/2407.03978)
• Keywords: language model, llm
• Abstract: Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to deal with complex human instructions in real-world scenarios. Therefore, how to evaluate the ability of complex instruction-following of LLMs has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints, which is an indispensable constituent in complex instructions. To this end, we propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints. We propose a hierarchical taxonomy for complex instructions, including 4 constraint types, 19 constraint dimensions, and 4 composition types, and manually collect a high-quality dataset accordingly. To make the evaluation reliable, we augment LLM-based evaluators with rules to effectively verify whether generated texts can satisfy each constraint and composition. Furthermore, we obtain the final evaluation score based on the dependency structure determined by different composition types. ComplexBench identifies significant deficiencies in existing LLMs when dealing with complex instructions with multiple constraints composition.
• 摘要：指令跟随是大型语言模型（LLM）的基本能力之一。随着LLM能力的不断提升，它们在现实场景中越来越多地应用于处理复杂的人类指令。因此，如何评估LLM的复杂指令跟随能力成为一个关键的研究问题。现有的基准测试主要侧重于对人类指令中不同类型的约束进行建模，而忽略了不同约束的组合，而组合是复杂指令中不可或缺的组成部分。为此，我们提出了ComplexBench，这是一个全面评估LLM遵循由多种约束组成的复杂指令的能力的基准测试。我们提出了一种复杂指令的分层分类法，包括4种约束类型、19种约束维度和4种组合类型，并据此人工收集了高质量的数据集。为了使评估可靠，我们为基于LLM的评估器添加了规则，以有效地验证生成的文本是否能够满足每个约束和组合。此外，我们根据不同组合类型确定的依赖结构获得最终的评估分数。 ComplexBench 发现现有 LLM 在处理具有多重约束组合的复杂指令时存在重大缺陷。

Title: A Survey on Natural Language Counterfactual Generation

• Authors: Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng, Yuhong Feng, Zhiqi Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03993
• Pdf URL: https://arxiv.org/pdf/2407.03993
• Copy Paste: [[2407.03993]] A Survey on Natural Language Counterfactual Generation(https://arxiv.org/abs/2407.03993)
• Keywords: language model
• Abstract: Natural Language Counterfactual generation aims to minimally modify a given text such that the modified text will be classified into a different class. The generated counterfactuals provide insight into the reasoning behind a model's predictions by highlighting which words significantly influence the outcomes. Additionally, they can be used to detect model fairness issues or augment the training data to enhance the model's robustness. A substantial amount of research has been conducted to generate counterfactuals for various NLP tasks, employing different models and methodologies. With the rapid growth of studies in this field, a systematic review is crucial to guide future researchers and developers. To bridge this gap, this survey comprehensively overview textual counterfactual generation methods, particularly including those based on Large Language Models. We propose a new taxonomy that categorizes the generation methods into four groups and systematically summarize the metrics for evaluating the generation quality. Finally, we discuss ongoing research challenges and outline promising directions for future work.
• 摘要：自然语言反事实生成旨在对给定文本进行最小程度的修改，以便将修改后的文本归类到不同的类别中。生成的反事实通过突出显示哪些词对结果有显著影响，提供了对模型预测背后原因的洞察。此外，它们还可用于检测模型公平性问题或增强训练数据以增强模型的鲁棒性。已经进行了大量研究，采用不同的模型和方法为各种 NLP 任务生成反事实。随着该领域研究的快速发展，系统综述对于指导未来的研究人员和开发人员至关重要。为了弥补这一差距，本调查全面概述了文本反事实生成方法，特别是包括基于大型语言模型的方法。我们提出了一种新的分类法，将生成方法分为四类，并系统地总结了评估生成质量的指标。最后，我们讨论了正在进行的研究挑战并概述了未来工作的有希望的方向。

Title: Unlocking the Potential of Model Merging for Low-Resource Languages

• Authors: Mingxu Tao, Chen Zhang, Quzhe Huang, Tianyao Ma, Songfang Huang, Dongyan Zhao, Yansong Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03994
• Pdf URL: https://arxiv.org/pdf/2407.03994
• Copy Paste: [[2407.03994]] Unlocking the Potential of Model Merging for Low-Resource Languages(https://arxiv.org/abs/2407.03994)
• Keywords: language model, llm
• Abstract: Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach struggles with limited data in the context of low-resource languages, failing to balance language modeling and task-solving capabilities. We thus propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. We use model merging to develop task-solving LLMs for low-resource languages without SFT data in the target languages. Our experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data. Observing performance saturation in model merging with more training tokens, we further analyze the merging process and introduce a slack variable to the model merging algorithm to mitigate the loss of important parameters, thereby enhancing performance. We hope that model merging can benefit more human languages suffering from data scarcity with its higher data efficiency.
• 摘要：将大型语言模型 (LLM) 适配到新语言通常需要持续的预训练 (CT)，然后进行监督微调 (SFT)。然而，这种先进行 CT，然后进行 SFT 的方法在资源匮乏的语言环境中数据有限，无法平衡语言建模和任务解决能力。因此，我们提出模型合并作为资源匮乏语言的替代方案，将具有不同功能的模型组合成一个模型，而无需额外训练。我们使用模型合并为资源匮乏的语言开发任务解决 LLM，而无需目标语言中的 SFT 数据。我们基于 Llama-2-7B 的实验表明，模型合并有效地赋予资源匮乏语言的 LLM 任务解决能力，在数据极其稀缺的场景中表现优于先进行 CT，然后进行 SFT。观察到使用更多训练标记进行模型合并时的性能饱和，我们进一步分析合并过程，并在模型合并算法中引入松弛变量，以减轻重要参数的损失，从而提高性能。我们希望模型合并能够以其更高的数据效率使更多遭受数据稀缺困扰的人类语言受益。

Title: LLMAEL: Large Language Models are Good Context Augmenters for Entity Linking

• Authors: Amy Xin, Yunjia Qi, Zijun Yao, Fangwei Zhu, Kaisheng Zeng, Xu Bin, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04020
• Pdf URL: https://arxiv.org/pdf/2407.04020
• Copy Paste: [[2407.04020]] LLMAEL: Large Language Models are Good Context Augmenters for Entity Linking(https://arxiv.org/abs/2407.04020)
• Keywords: language model, llm
• Abstract: Entity Linking (EL) models are well-trained at mapping mentions to their corresponding entities according to a given context. However, EL models struggle to disambiguate long-tail entities due to their limited training data. Meanwhile, large language models (LLMs) are more robust at interpreting uncommon mentions. Yet, due to a lack of specialized training, LLMs suffer at generating correct entity IDs. Furthermore, training an LLM to perform EL is cost-intensive. Building upon these insights, we introduce LLM-Augmented Entity Linking LLMAEL, a plug-and-play approach to enhance entity linking through LLM data augmentation. We leverage LLMs as knowledgeable context augmenters, generating mention-centered descriptions as additional input, while preserving traditional EL models for task specific processing. Experiments on 6 standard datasets show that the vanilla LLMAEL outperforms baseline EL models in most cases, while the fine-tuned LLMAEL set the new state-of-the-art results across all 6 benchmarks.
• 摘要：实体链接 (EL) 模型经过良好训练，可根据给定上下文将提及映射到其对应的实体。然而，由于训练数据有限，EL 模型很难消除长尾实体的歧义。同时，大型语言模型 (LLM) 在解释不常见的提及方面更为稳健。然而，由于缺乏专门的训练，LLM 在生成正确的实体 ID 方面表现不佳。此外，训练 LLM 执行 EL 的成本很高。基于这些见解，我们引入了 LLM 增强实体链接 LLMAEL，这是一种通过 LLM 数据增强来增强实体链接的即插即用方法。我们利用 LLM 作为知识丰富的上下文增强器，生成以提及为中心的描述作为附加输入，同时保留传统的 EL 模型以进行特定任务的处理。在 6 个标准数据集上进行的实验表明，原始 LLMAEL 在大多数情况下都优于基线 EL 模型，而经过微调的 LLMAEL 在所有 6 个基准中都创下了新的最先进结果。

Title: Systematic Task Exploration with LLMs: A Study in Citation Text Generation

• Authors: Furkan Şahinuç, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04046
• Pdf URL: https://arxiv.org/pdf/2407.04046
• Copy Paste: [[2407.04046]] Systematic Task Exploration with LLMs: A Study in Citation Text Generation(https://arxiv.org/abs/2407.04046)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) bring unprecedented flexibility in defining and executing complex, creative natural language generation (NLG) tasks. Yet, this flexibility brings new challenges, as it introduces new degrees of freedom in formulating the task inputs and instructions and in evaluating model performance. To facilitate the exploration of creative NLG tasks, we propose a three-component research framework that consists of systematic input manipulation, reference data, and output measurement. We use this framework to explore citation text generation -- a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric and has not yet been tackled within the LLM paradigm. Our results highlight the importance of systematically investigating both task instruction and input configuration when prompting LLMs, and reveal non-trivial relationships between different evaluation metrics used for citation text generation. Additional human generation and human evaluation experiments provide new qualitative insights into the task to guide future research in citation text generation. We make our code and data publicly available.
• 摘要：大型语言模型 (LLM) 在定义和执行复杂、富有创意的自然语言生成 (NLG) 任务方面带来了前所未有的灵活性。然而，这种灵活性也带来了新的挑战，因为它在制定任务输入和指令以及评估模型性能方面引入了新的自由度。为了促进对创造性 NLG 任务的探索，我们提出了一个由系统输入操作、参考数据和输出测量组成的三部分研究框架。我们使用这个框架来探索引文文本生成——这是一项流行的学术 NLP 任务，在任务定义和评估指标上缺乏共识，并且尚未在 LLM 范式中得到解决。我们的结果强调了在提示 LLM 时系统地调查任务指令和输入配置的重要性，并揭示了用于引文文本生成的不同评估指标之间的非平凡关系。额外的人工生成和人工评估实验为该任务提供了新的定性见解，以指导未来对引文文本生成的研究。我们公开了我们的代码和数据。

Title: Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis

• Authors: Cong-Thanh Do, Shuhei Imai, Rama Doddipatla, Thomas Hain
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.04047
• Pdf URL: https://arxiv.org/pdf/2407.04047
• Copy Paste: [[2407.04047]] Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis(https://arxiv.org/abs/2407.04047)
• Keywords: prompt
• Abstract: This paper investigates the use of unsupervised text-to-speech synthesis (TTS) as a data augmentation method to improve accented speech recognition. TTS systems are trained with a small amount of accented speech training data and their pseudo-labels rather than manual transcriptions, and hence unsupervised. This approach enables the use of accented speech data without manual transcriptions to perform data augmentation for accented speech recognition. Synthetic accented speech data, generated from text prompts by using the TTS systems, are then combined with available non-accented speech data to train automatic speech recognition (ASR) systems. ASR experiments are performed in a self-supervised learning framework using a Wav2vec2.0 model which was pre-trained on large amount of unsupervised accented speech data. The accented speech data for training the unsupervised TTS are read speech, selected from L2-ARCTIC and British Isles corpora, while spontaneous conversational speech from the Edinburgh international accents of English corpus are used as the evaluation data. Experimental results show that Wav2vec2.0 models which are fine-tuned to downstream ASR task with synthetic accented speech data, generated by the unsupervised TTS, yield up to 6.1% relative word error rate reductions compared to a Wav2vec2.0 baseline which is fine-tuned with the non-accented speech data from Librispeech corpus.
• 摘要：本文研究了使用无监督文本转语音合成 (TTS) 作为数据增强方法来改进口音语音识别。TTS 系统使用少量口音语音训练数据及其伪标签进行训练，而不是手动转录，因此是无监督的。这种方法允许使用没有手动转录的口音语音数据来执行口音语音识别的数据增强。然后，将使用 TTS 系统从文本提示生成的合成口音语音数据与可用的非口音语音数据相结合，以训练自动语音识别 (ASR) 系统。ASR 实验是在自监督学习框架中使用 Wav2vec2.0 模型进行的，该模型已在大量无监督口音语音数据上进行了预训练。用于训练无监督 TTS 的口音语音数据是从 L2-ARCTIC 和不列颠群岛语料库中选择的朗读语音，而来自爱丁堡国际英语口音语料库的自发对话语音则用作评估数据。实验结果表明，与使用来自 Librispeech 语料库的非重音语音数据进行微调的 Wav2vec2.0 基线相比，使用无监督 TTS 生成的合成重音语音数据针对下游 ASR 任务进行微调的 Wav2vec2.0 模型可将相对词错误率降低高达 6.1%。

Title: Deep Content Understanding Toward Entity and Aspect Target Sentiment Analysis on Foundation Models

• Authors: Vorakit Vorakitphan, Milos Basic, Guilhaume Leroy Meline
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04050
• Pdf URL: https://arxiv.org/pdf/2407.04050
• Copy Paste: [[2407.04050]] Deep Content Understanding Toward Entity and Aspect Target Sentiment Analysis on Foundation Models(https://arxiv.org/abs/2407.04050)
• Keywords: language model
• Abstract: Introducing Entity-Aspect Sentiment Triplet Extraction (EASTE), a novel Aspect-Based Sentiment Analysis (ABSA) task which extends Target-Aspect-Sentiment Detection (TASD) by separating aspect categories (e.g., food#quality) into pre-defined entities (e.g., meal, drink) and aspects (e.g., taste, freshness) which add a fine-gainer level of complexity, yet help exposing true sentiment of chained aspect to its entity. We explore the task of EASTE solving capabilities of language models based on transformers architecture from our proposed unified-loss approach via token classification task using BERT architecture to text generative models such as Flan-T5, Flan-Ul2 to Llama2, Llama3 and Mixtral employing different alignment techniques such as zero/few-shot learning, Parameter Efficient Fine Tuning (PEFT) such as Low-Rank Adaptation (LoRA). The model performances are evaluated on the SamEval-2016 benchmark dataset representing the fair comparison to existing works. Our research not only aims to achieve high performance on the EASTE task but also investigates the impact of model size, type, and adaptation techniques on task performance. Ultimately, we provide detailed insights and achieving state-of-the-art results in complex sentiment analysis.
• 摘要：引入实体-方面情绪三元组提取 (EASTE)，这是一种新颖的基于方面的情绪分析 (ABSA) 任务，它通过将方面类别（例如，食物#质量）分离为预定义实体（例如，餐点、饮料）和方面（例如，味道、新鲜度）来扩展目标-方面-情绪检测 (TASD)，这增加了精细增益级别的复杂性，但有助于将链式方面的真实情绪暴露给其实体。我们探索了基于 Transformer 架构的语言模型的 EASTE 解决能力的任务，从我们提出的统一损失方法通过使用 BERT 架构的 token 分类任务到文本生成模型，例如 Flan-T5、Flan-Ul2 到 Llama2、Llama3 和 Mixtral，采用不同的对齐技术，例如零/小样本学习、参数有效微调 (PEFT)，例如低秩自适应 (LoRA)。在 SamEval-2016 基准数据集上对模型性能进行了评估，与现有工作进行了公平比较。我们的研究不仅旨在在 EASTE 任务上取得高性能，还研究了模型大小、类型和适应技术对任务性能的影响。最终，我们提供详细的见解并在复杂情绪分析中取得最先进的结果。

Title: Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM

• Authors: Peiran Yao, Kostyantyn Guzhva, Denilson Barbosa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04067
• Pdf URL: https://arxiv.org/pdf/2407.04067
• Copy Paste: [[2407.04067]] Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM(https://arxiv.org/abs/2407.04067)
• Keywords: language model, llm, prompt
• Abstract: Symbolic sentence meaning representations, such as AMR (Abstract Meaning Representation) provide expressive and structured semantic graphs that act as intermediates that simplify downstream NLP tasks. However, the instruction-following capability of large language models (LLMs) offers a shortcut to effectively solve NLP tasks, questioning the utility of semantic graphs. Meanwhile, recent work has also shown the difficulty of using meaning representations merely as a helpful auxiliary for LLMs. We revisit the position of semantic graphs in syntactic simplification, the task of simplifying sentence structures while preserving their meaning, which requires semantic understanding, and evaluate it on a new complex and natural dataset. The AMR-based method that we propose, AMRS$^3$, demonstrates that state-of-the-art meaning representations can lead to easy-to-implement simplification methods with competitive performance and unique advantages in cost, interpretability, and generalization. With AMRS$^3$ as an anchor, we discover that syntactic simplification is a task where semantic graphs are helpful in LLM prompting. We propose AMRCoC prompting that guides LLMs to emulate graph algorithms for explicit symbolic reasoning on AMR graphs, and show its potential for improving LLM on semantic-centered tasks like syntactic simplification.
• 摘要：符号句子意义表示，例如 AMR（抽象意义表示）提供了富有表现力和结构化的语义图，可作为简化下游 NLP 任务的中间件。然而，大型语言模型 (LLM) 的指令跟踪能力为有效解决 NLP 任务提供了一条捷径，这质疑了语义图的实用性。同时，最近的研究也表明，仅将意义表示用作 LLM 的有用辅助手段存在困难。我们重新审视了语义图在句法简化中的位置，句法简化是简化句子结构同时保留其意义的任务，这需要语义理解，并在一个新的复杂而自然的数据集上对其进行评估。我们提出的基于 AMR 的方法 AMRS$^3$ 表明，最先进的意义表示可以带来易于实施的简化方法，具有竞争性的性能和成本、可解释性和泛化方面的独特优势。以 AMRS$^3$ 为锚点，我们发现句法简化是一项语义图有助于 LLM 提示的任务。我们提出了 AMRCoC 提示，引导 LLM 模拟图形算法，以便在 AMR 图上进行显式符号推理，并展示其在句法简化等以语义为中心的任务上改进 LLM 的潜力。

Title: A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

• Authors: Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04069
• Pdf URL: https://arxiv.org/pdf/2407.04069
• Copy Paste: [[2407.04069]] A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations(https://arxiv.org/abs/2407.04069)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.
• 摘要：大型语言模型 (LLM) 最近因其在各个领域执行各种任务的出色能力而受到广泛关注。然而，在将这些模型部署到实际应用中之前，对其进行全面评估至关重要，以确保它们产生可靠的性能。尽管评估 LLM 在社区中的重要性已得到充分认可，但评估过程的复杂性导致评估设置各异，从而导致结果和解释不一致。为了解决这个问题，我们系统地回顾了导致 LLM 评估各个步骤中出现这些不一致和不可靠评估的主要挑战和限制。基于我们的批判性审查，我们提出了我们的观点和建议，以确保 LLM 评估是可重复的、可靠的和稳健的。

Title: DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoning

• Authors: Chengpeng Li, Guanting Dong, Mingfeng Xue, Ru Peng, Xiang Wang, Dayiheng Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04078
• Pdf URL: https://arxiv.org/pdf/2407.04078
• Copy Paste: [[2407.04078]] DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoning(https://arxiv.org/abs/2407.04078)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have made impressive progress in handling simple math problems, yet they still struggle with more challenging and complex mathematical tasks. In this paper, we introduce a series of LLMs that employs the Decomposition of thought with code assistance and self-correction for mathematical reasoning, dubbed as DotaMath. DotaMath models tackle complex mathematical tasks by decomposing them into simpler logical subtasks, leveraging code to solve these subtasks, obtaining fine-grained feedback from the code interpreter, and engaging in self-reflection and correction. By annotating diverse interactive tool-use trajectories and employing query evolution on GSM8K and MATH datasets, we generate an instruction fine-tuning dataset called DotaMathQA with 574K query-response pairs. We train a series of base LLMs using imitation learning on DotaMathQA, resulting in DotaMath models that achieve remarkable performance compared to open-source LLMs across various in-domain and out-of-domain benchmarks. Notably, DotaMath-deepseek-7B showcases an outstanding performance of 64.8% on the competitive MATH dataset and 86.7% on GSM8K. Besides, DotaMath-deepseek-7B maintains strong competitiveness on a series of in-domain and out-of-domain benchmarks (Avg. 80.1%). Looking forward, we anticipate that the DotaMath paradigm will open new pathways for addressing intricate mathematical problems. Our code is publicly available at this https URL.
• 摘要：大型语言模型 (LLM) 在处理简单数学问题方面取得了令人瞩目的进展，但它们在处理更具挑战性和复杂性的数学任务时仍然举步维艰。在本文中，我们介绍了一系列采用思维分解、代码辅助和自我修正进行数学推理的 LLM，称为 DotaMath。DotaMath 模型通过将复杂的数学任务分解为更简单的逻辑子任务、利用代码解决这些子任务、从代码解释器获取细粒度反馈以及进行自我反思和修正来解决这些数学任务。通过注释各种交互式工具使用轨迹并在 GSM8K 和 MATH 数据集上使用查询演化，我们生成了一个名为 DotaMathQA 的指令微调数据集，其中包含 574K 个查询-响应对。我们使用 DotaMathQA 上的模仿学习训练了一系列基础 LLM，从而产生了 DotaMath 模型，与开源 LLM 相比，这些模型在各种领域内和领域外基准测试中都取得了卓越的性能。值得注意的是，DotaMath-deepseek-7B 在竞争性 MATH 数据集上表现出色，准确率为 64.8%，在 GSM8K 上表现出色，准确率为 86.7%。此外，DotaMath-deepseek-7B 在一系列域内和域外基准测试中保持强劲竞争力（平均 80.1%）。展望未来，我们预计 DotaMath 范式将为解决复杂的数学问题开辟新途径。我们的代码在此 https URL 上公开提供。

Title: Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversations

• Authors: Hao Yang, Hongyuan Lu, Xinhua Zeng, Yang Liu, Xiang Zhang, Haoran Yang, Yumeng Zhang, Yiran Wei, Wai Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04093
• Pdf URL: https://arxiv.org/pdf/2407.04093
• Copy Paste: [[2407.04093]] Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversations(https://arxiv.org/abs/2407.04093)
• Keywords: language model, llm, chat
• Abstract: In the rapidly evolving field of natural language processing, dialogue systems primarily employ a single-step dialogue paradigm. Although this paradigm is efficient, it lacks the depth and fluidity of human interactions and does not appear natural. We introduce a novel \textbf{Step}-by-Step Dialogue Paradigm (Stephanie), designed to mimic the ongoing dynamic nature of human conversations. By employing a dual learning strategy and a further-split post-editing method, we generated and utilized a high-quality step-by-step dialogue dataset to fine-tune existing large language models, enabling them to perform step-by-step dialogues. We thoroughly present Stephanie. Tailored automatic and human evaluations are conducted to assess its effectiveness compared to the traditional single-step dialogue paradigm. We will release code, Stephanie datasets, and Stephanie LLMs to facilitate the future of chatbot eras.
• 摘要：在快速发展的自然语言处理领域，对话系统主要采用单步对话范式。虽然这种范式很有效，但它缺乏人类互动的深度和流动性，而且看起来不自然。我们引入了一种新颖的 \textbf{Step}-by-Step 对话范式 (Stephanie)，旨在模仿人类对话的持续动态性质。通过采用双重学习策略和进一步拆分的后期编辑方法，我们生成并利用了高质量的分步对话数据集来微调现有的大型语言模型，使它们能够执行分步对话。我们详细介绍了 Stephanie。进行了量身定制的自动和人工评估，以评估其与传统单步对话范式相比的有效性。我们将发布代码、Stephanie 数据集和 Stephanie LLM，以促进聊天机器人时代的未来。

Title: Can Pre-trained Language Models Understand Chinese Humor?

• Authors: Yuyan Chen, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Bang Liu, Yunwen Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04105
• Pdf URL: https://arxiv.org/pdf/2407.04105
• Copy Paste: [[2407.04105]] Can Pre-trained Language Models Understand Chinese Humor?(https://arxiv.org/abs/2407.04105)
• Keywords: language model
• Abstract: Humor understanding is an important and challenging research in natural language processing. As the popularity of pre-trained language models (PLMs), some recent work makes preliminary attempts to adopt PLMs for humor recognition and generation. However, these simple attempts do not substantially answer the question: {\em whether PLMs are capable of humor understanding?} This paper is the first work that systematically investigates the humor understanding ability of PLMs. For this purpose, a comprehensive framework with three evaluation steps and four evaluation tasks is designed. We also construct a comprehensive Chinese humor dataset, which can fully meet all the data requirements of the proposed evaluation framework. Our empirical study on the Chinese humor dataset yields some valuable observations, which are of great guiding value for future optimization of PLMs in humor understanding and generation.
• 摘要：幽默理解是自然语言处理中一项重要且具有挑战性的研究。随着预训练语言模型 (PLM) 的流行，最近的一些研究对采用 PLM 进行幽默识别和生成进行了初步尝试。然而，这些简单的尝试并没有从根本上回答这个问题：{\em PLM 是否能够理解幽默？} 本文是第一篇系统地研究 PLM 幽默理解能力的研究。为此，我们设计了一个包含三个评估步骤和四个评估任务的综合框架。我们还构建了一个全面的中文幽默数据集，它可以完全满足所提出的评估框架的所有数据要求。我们对中文幽默数据集的实证研究得出了一些有价值的观察结果，这些观察结果对未来优化 PLM 在幽默理解和生成方面的性能具有重要的指导价值。

Title: MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization

• Authors: Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04118
• Pdf URL: https://arxiv.org/pdf/2407.04118
• Copy Paste: [[2407.04118]] MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization(https://arxiv.org/abs/2407.04118)
• Keywords: language model, llm, prompt
• Abstract: Prompt engineering, as an efficient and effective way to leverage Large Language Models (LLM), has drawn a lot of attention from the research community. The existing research primarily emphasizes the importance of adapting prompts to specific tasks, rather than specific LLMs. However, a good prompt is not solely defined by its wording, but also binds to the nature of the LLM in question. In this work, we first quantitatively demonstrate that different prompts should be adapted to different LLMs to enhance their capabilities across various downstream tasks in NLP. Then we novelly propose a model-adaptive prompt optimizer (MAPO) method that optimizes the original prompts for each specific LLM in downstream tasks. Extensive experiments indicate that the proposed method can effectively refine prompts for an LLM, leading to significant improvements over various downstream tasks.
• 摘要：提示工程作为一种高效且有效的利用大型语言模型 (LLM) 的方法，引起了研究界的广泛关注。现有研究主要强调将提示适应特定任务而不是特定的 LLM。然而，好的提示不仅仅由其措辞来定义，还与所讨论的 LLM 的性质有关。在这项工作中，我们首先定量地证明不同的提示应该适应不同的 LLM，以增强它们在 NLP 中各种下游任务中的能力。然后，我们新颖地提出了一种模型自适应提示优化器 (MAPO) 方法，该方法优化下游任务中每个特定 LLM 的原始提示。大量实验表明，所提出的方法可以有效地细化 LLM 的提示，从而显著改善各种下游任务。

Title: Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

• Authors: Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04121
• Pdf URL: https://arxiv.org/pdf/2407.04121
• Copy Paste: [[2407.04121]] Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models(https://arxiv.org/abs/2407.04121)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, including question answering and dialogue systems. However, a major drawback of LLMs is the issue of hallucination, where they generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. In this paper, we propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers. RelD is trained on the constructed RelQA, a bilingual question-answering dialogue dataset along with answers generated by LLMs and a comprehensive set of metrics. Our experimental results demonstrate that the proposed RelD successfully detects hallucination in the answers generated by diverse LLMs. Moreover, it performs well in distinguishing hallucination in LLMs' generated answers from both in-distribution and out-of-distribution datasets. Additionally, we also conduct a thorough analysis of the types of hallucinations that occur and present valuable insights. This research significantly contributes to the detection of reliable answers generated by LLMs and holds noteworthy implications for mitigating hallucination in the future work.
• 摘要：大型语言模型 (LLM) 已广泛应用于各种自然语言处理任务，包括问答和对话系统。然而，LLM 的一个主要缺点是幻觉问题，它们会生成与输入源不符或不一致的内容，从而导致严重后果。在本文中，我们提出了一个名为 RelD 的强大鉴别器，以有效检测 LLM 生成的答案中的幻觉。RelD 在构建的 RelQA（一个双语问答对话数据集）以及 LLM 生成的答案和一组全面的指标上进行训练。我们的实验结果表明，所提出的 RelD 成功地检测到了由各种 LLM 生成的答案中的幻觉。此外，它在区分分布内和分布外数据集中 LLM 生成的答案中的幻觉方面表现良好。此外，我们还对发生的幻觉类型进行了彻底的分析，并提出了有价值的见解。这项研究对检测 LLM 产生的可靠答案做出了重大贡献，并对于未来工作中减轻幻觉具有重要意义。

Title: Query-Guided Self-Supervised Summarization of Nursing Notes

• Authors: Ya Gao, Hans Moen, Saila Koivusalo, Miika Koskinen, Pekka Marttinen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04125
• Pdf URL: https://arxiv.org/pdf/2407.04125
• Copy Paste: [[2407.04125]] Query-Guided Self-Supervised Summarization of Nursing Notes(https://arxiv.org/abs/2407.04125)
• Keywords: language model, llm
• Abstract: Nursing notes, an important component of Electronic Health Records (EHRs), keep track of the progression of a patient's health status during a care episode. Distilling the key information in nursing notes through text summarization techniques can improve clinicians' efficiency in understanding patients' conditions when reviewing nursing notes. However, existing abstractive summarization methods in the clinical setting have often overlooked nursing notes and require the creation of reference summaries for supervision signals, which is time-consuming. In this work, we introduce QGSumm, a query-guided self-supervised domain adaptation framework for nursing note summarization. Using patient-related clinical queries as guidance, our approach generates high-quality, patient-centered summaries without relying on reference summaries for training. Through automatic and manual evaluation by an expert clinician, we demonstrate the strengths of our approach compared to the state-of-the-art Large Language Models (LLMs) in both zero-shot and few-shot settings. Ultimately, our approach provides a new perspective on conditional text summarization, tailored to the specific interests of clinical personnel.
• 摘要：护理记录是电子健康记录 (EHR) 的重要组成部分，可跟踪患者在护理过程中的健康状况进展。通过文本摘要技术提炼护理记录中的关键信息可以提高临床医生在审查护理记录时了解患者病情的效率。然而，现有的临床抽象摘要方法经常忽略护理记录，需要为监督信号创建参考摘要，这非常耗时。在这项工作中，我们引入了 QGSumm，这是一个用于护理记录摘要的查询引导自监督领域自适应框架。使用与患者相关的临床查询作为指导，我们的方法可以生成高质量、以患者为中心的摘要，而无需依赖参考摘要进行训练。通过专家临床医生的自动和手动评估，我们展示了我们的方法与最先进的大型语言模型 (LLM) 在零样本和小样本设置中相比的优势。最终，我们的方法为条件文本摘要提供了一个新的视角，可以根据临床人员的特定兴趣进行定制。

Title: Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4

• Authors: Sachin Yadav, Tejaswi Choppa, Dominik Schlechtweg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04130
• Pdf URL: https://arxiv.org/pdf/2407.04130
• Copy Paste: [[2407.04130]] Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4(https://arxiv.org/abs/2407.04130)
• Keywords: gpt, llm, prompt
• Abstract: This paper explores using GPT-3.5 and GPT-4 to automate the data annotation process with automatic prompting techniques. The main aim of this paper is to reuse human annotation guidelines along with some annotated data to design automatic prompts for LLMs, focusing on the semantic proximity annotation task. Automatic prompts are compared to customized prompts. We further implement the prompting strategies into an open-source text annotation tool, enabling easy online use via the OpenAI API. Our study reveals the crucial role of accurate prompt design and suggests that prompting GPT-4 with human-like instructions is not straightforwardly possible for the semantic proximity task. We show that small modifications to the human guidelines already improve the performance, suggesting possible ways for future research.
• 摘要：本文探讨了如何使用 GPT-3.5 和 GPT-4 通过自动提示技术实现数据注释过程的自动化。本文的主要目的是重用人工注释指南以及一些注释数据来设计 LLM 的自动提示，重点关注语义接近度注释任务。将自动提示与自定义提示进行了比较。我们进一步将提示策略实现到开源文本注释工具中，以便通过 OpenAI API 轻松在线使用。我们的研究揭示了准确提示设计的关键作用，并表明对于语义接近度任务，使用类似人类的指令提示 GPT-4 并非直接可能。我们表明，对人类指南进行微小修改已经可以提高性能，这为未来的研究提供了可能的方法。

Title: Securing Multi-turn Conversational Language Models Against Distributed Backdoor Triggers

• Authors: Terry Tong, Jiashu Xu, Qin Liu, Muhao Chen
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04151
• Pdf URL: https://arxiv.org/pdf/2407.04151
• Copy Paste: [[2407.04151]] Securing Multi-turn Conversational Language Models Against Distributed Backdoor Triggers(https://arxiv.org/abs/2407.04151)
• Keywords: language model, llm
• Abstract: The security of multi-turn conversational large language models (LLMs) is understudied despite it being one of the most popular LLM utilization. Specifically, LLMs are vulnerable to data poisoning backdoor attacks, where an adversary manipulates the training data to cause the model to output malicious responses to predefined triggers. Specific to the multi-turn dialogue setting, LLMs are at the risk of even more harmful and stealthy backdoor attacks where the backdoor triggers may span across multiple utterances, giving lee-way to context-driven attacks. In this paper, we explore a novel distributed backdoor trigger attack that serves to be an extra tool in an adversary's toolbox that can interface with other single-turn attack strategies in a plug and play manner. Results on two representative defense mechanisms indicate that distributed backdoor triggers are robust against existing defense strategies which are designed for single-turn user-model interactions, motivating us to propose a new defense strategy for the multi-turn dialogue setting that is more challenging. To this end, we also explore a novel contrastive decoding based defense that is able to mitigate the backdoor with a low computational tradeoff.
• 摘要：尽管多轮对话大型语言模型 (LLM) 是最流行的 LLM 用途之一，但其安全性研究不足。具体来说，LLM 容易受到数据中毒后门攻击，攻击者可以操纵训练数据，使模型对预定义的触发器输出恶意响应。具体到多轮对话设置，LLM 面临着更具危害性和隐蔽性的后门攻击的风险，其中后门触发器可能跨越多个话语，为上下文驱动的攻击提供了余地。在本文中，我们探索了一种新颖的分布式后门触发器攻击，它可以作为攻击者工具箱中的额外工具，可以以即插即用的方式与其他单轮攻击策略交互。两种代表性防御机制的结果表明，分布式后门触发器对现有的为单轮用户模型交互而设计的防御策略具有很强的鲁棒性，这促使我们提出一种更具挑战性的多轮对话设置的新防御策略。为此，我们还探索了一种新颖的基于对比解码的防御方法，该防御方法能够以较低的计算代价缓解后门攻击。

Title: Defense Against Syntactic Textual Backdoor Attacks with Token Substitution

• Authors: Xinglin Li, Xianwen He, Yao Li, Minhao Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04179
• Pdf URL: https://arxiv.org/pdf/2407.04179
• Copy Paste: [[2407.04179]] Defense Against Syntactic Textual Backdoor Attacks with Token Substitution(https://arxiv.org/abs/2407.04179)
• Keywords: language model, llm
• Abstract: Textual backdoor attacks present a substantial security risk to Large Language Models (LLM). It embeds carefully chosen triggers into a victim model at the training stage, and makes the model erroneously predict inputs containing the same triggers as a certain class. Prior backdoor defense methods primarily target special token-based triggers, leaving syntax-based triggers insufficiently addressed. To fill this gap, this paper proposes a novel online defense algorithm that effectively counters syntax-based as well as special token-based backdoor attacks. The algorithm replaces semantically meaningful words in sentences with entirely different ones but preserves the syntactic templates or special tokens, and then compares the predicted labels before and after the substitution to determine whether a sentence contains triggers. Experimental results confirm the algorithm's performance against these two types of triggers, offering a comprehensive defense strategy for model integrity.
• 摘要：文本后门攻击对大型语言模型 (LLM) 构成了重大的安全风险。它在训练阶段将精心选择的触发器嵌入到受害者模型中，并使模型错误地预测包含与某一类相同的触发器的输入。现有的后门防御方法主要针对基于特殊标记的触发器，而对基于语法的触发器关注不够。为了填补这一空白，本文提出了一种新颖的在线防御算法，可以有效地对抗基于语法和特殊标记的后门攻击。该算法将句子中语义上有意义的单词替换为完全不同的单词，但保留句法模板或特殊标记，然后比较替换前后的预测标签以确定句子是否包含触发器。实验结果证实了该算法对这两类触发器的性能，为模型完整性提供了全面的防御策略。

Title: Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms

• Authors: Joshua Ashkinaze, Ruijia Guan, Laura Kurek, Eytan Adar, Ceren Budak, Eric Gilbert
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.04183
• Pdf URL: https://arxiv.org/pdf/2407.04183
• Copy Paste: [[2407.04183]] Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms(https://arxiv.org/abs/2407.04183)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are trained on broad corpora and then used in communities with specialized norms. Is providing LLMs with community rules enough for models to follow these norms? We evaluate LLMs' capacity to detect (Task 1) and correct (Task 2) biased Wikipedia edits according to Wikipedia's Neutral Point of View (NPOV) policy. LLMs struggled with bias detection, achieving only 64% accuracy on a balanced dataset. Models exhibited contrasting biases (some under- and others over-predicted bias), suggesting distinct priors about neutrality. LLMs performed better at generation, removing 79% of words removed by Wikipedia editors. However, LLMs made additional changes beyond Wikipedia editors' simpler neutralizations, resulting in high-recall but low-precision editing. Interestingly, crowdworkers rated AI rewrites as more neutral (70%) and fluent (61%) than Wikipedia-editor rewrites. Qualitative analysis found LLMs sometimes applied NPOV more comprehensively than Wikipedia editors but often made extraneous non-NPOV-related changes (such as grammar). LLMs may apply rules in ways that resonate with the public but diverge from community experts. While potentially effective for generation, LLMs may reduce editor agency and increase moderation workload (e.g., verifying additions). Even when rules are easy to articulate, having LLMs apply them like community members may still be difficult.
• 摘要：大型语言模型 (LLM) 在广泛的语料库上进行训练，然后在具有专门规范的社区中使用。为 LLM 提供社区规则是否足以使模型遵循这些规范？我们根据维基百科的中立观点 (NPOV) 政策评估 LLM 检测（任务 1）和纠正（任务 2）有偏见的维基百科编辑的能力。LLM 在偏见检测方面表现不佳，在平衡数据集上的准确率仅为 64%。模型表现出对比鲜明的偏见（一些模型低估了偏见，另一些模型高估了偏见），表明对中立性存在不同的先验。LLM 在生成方面表现更好，删除了维基百科编辑删除的 79% 的单词。然而，除了维基百科编辑更简单的中和之外，LLM 还进行了其他更改，导致编辑的召回率高但准确率低。有趣的是，众包工作者认为 AI 重写比维基百科编辑重写更中立（70%）和流畅（61%）。定性分析发现，LLM 有时比维基百科编辑更全面地应用 NPOV，但经常做出与 NPOV 无关的无关更改（例如语法）。LLM 可能以引起公众共鸣但与社区专家不同的方式应用规则。虽然 LLM 可能对生成有效，但它可能会减少编辑代理并增加审核工作量（例如，验证添加的内容）。即使规则很容易表达，让 LLM 像社区成员一样应用它们可能仍然很困难。

Title: HAF-RM: A Hybrid Alignment Framework for Reward Model Training

• Authors: Shujun Liu, Xiaoyu Shen, Yuhang Lai, Siyuan Wang, Shengbin Yue, Zengfeng Huang, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04185
• Pdf URL: https://arxiv.org/pdf/2407.04185
• Copy Paste: [[2407.04185]] HAF-RM: A Hybrid Alignment Framework for Reward Model Training(https://arxiv.org/abs/2407.04185)
• Keywords: language model, llm
• Abstract: The reward model has become increasingly important in alignment, assessment, and data construction for large language models (LLMs). Most existing researchers focus on enhancing reward models through data improvements, following the conventional training framework for reward models that directly optimizes the predicted rewards. In this paper, we propose a hybrid alignment framework HaF-RM for reward model training by introducing an additional constraint on token-level policy probabilities in addition to the reward score. It can simultaneously supervise the internal preference model at the token level and optimize the mapping layer of the reward model at the sequence level. Theoretical justifications and experiment results on five datasets show the validity and effectiveness of our proposed hybrid framework for training a high-quality reward model. By decoupling the reward modeling procedure and incorporating hybrid supervision, our HaF-RM framework offers a principled and effective approach to enhancing the performance and alignment of reward models, a critical component in the responsible development of powerful language models. We release our code at this https URL.
• 摘要：奖励模型在大型语言模型 (LLM) 的对齐、评估和数据构建中变得越来越重要。大多数现有研究人员专注于通过数据改进来增强奖励模型，遵循直接优化预测奖励的传统奖励模型训练框架。在本文中，我们提出了一种用于奖励模型训练的混合对齐框架 HaF-RM，除了奖励分数之外，还引入了对 token 级策略概率的额外约束。它可以同时在 token 级别监督内部偏好模型，并在序列级别优化奖励模型的映射层。理论依据和在五个数据集上的实验结果表明，我们提出的混合框架在训练高质量奖励模型方面的有效性和有效性。通过解耦奖励建模过程并结合混合监督，我们的 HaF-RM 框架提供了一种原则性和有效的方法来增强奖励模型的性能和对齐，这是负责任地开发强大语言模型的关键组成部分。我们在此 https URL 上发布了我们的代码。

Title: BiosERC: Integrating Biography Speakers Supported by LLMs for ERC Tasks

• Authors: Jieying Xue, Minh Phuong Nguyen, Blake Matheny, Le Minh Nguyen
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.04279
• Pdf URL: https://arxiv.org/pdf/2407.04279
• Copy Paste: [[2407.04279]] BiosERC: Integrating Biography Speakers Supported by LLMs for ERC Tasks(https://arxiv.org/abs/2407.04279)
• Keywords: language model, llm
• Abstract: In the Emotion Recognition in Conversation task, recent investigations have utilized attention mechanisms exploring relationships among utterances from intra- and inter-speakers for modeling emotional interaction between them. However, attributes such as speaker personality traits remain unexplored and present challenges in terms of their applicability to other tasks or compatibility with diverse model architectures. Therefore, this work introduces a novel framework named BiosERC, which investigates speaker characteristics in a conversation. By employing Large Language Models (LLMs), we extract the "biographical information" of the speaker within a conversation as supplementary knowledge injected into the model to classify emotional labels for each utterance. Our proposed method achieved state-of-the-art (SOTA) results on three famous benchmark datasets: IEMOCAP, MELD, and EmoryNLP, demonstrating the effectiveness and generalization of our model and showcasing its potential for adaptation to various conversation analysis tasks. Our source code is available at this https URL.
• 摘要：在对话中的情绪识别任务中，最近的研究利用注意力机制来探索说话者内部和说话者之间的话语关系，以建模他们之间的情感互动。然而，说话者个性特征等属性仍未得到探索，并且在它们适用于其他任务或与各种模型架构的兼容性方面存在挑战。因此，这项工作引入了一个名为 BiosERC 的新框架，该框架研究对话中的说话者特征。通过使用大型语言模型 (LLM)，我们提取对话中说话者的“传记信息”作为补充知识注入模型，以对每个话语的情绪标签进行分类。我们提出的方法在三个著名的基准数据集上取得了最先进的 (SOTA) 结果：IEMOCAP、MELD 和 EmoryNLP，证明了我们模型的有效性和泛化能力，并展示了其适应各种对话分析任务的潜力。我们的源代码可在此 https URL 上找到。

Title: Crafting Large Language Models for Enhanced Interpretability

• Authors: Chung-En Sun, Tuomas Oikarinen, Tsui-Wei Weng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04307
• Pdf URL: https://arxiv.org/pdf/2407.04307
• Copy Paste: [[2407.04307]] Crafting Large Language Models for Enhanced Interpretability(https://arxiv.org/abs/2407.04307)
• Keywords: language model, llm
• Abstract: We introduce the Concept Bottleneck Large Language Model (CB-LLM), a pioneering approach to creating inherently interpretable Large Language Models (LLMs). Unlike traditional black-box LLMs that rely on post-hoc interpretation methods with limited neuron function insights, CB-LLM sets a new standard with its built-in interpretability, scalability, and ability to provide clear, accurate explanations. This innovation not only advances transparency in language models but also enhances their effectiveness. Our unique Automatic Concept Correction (ACC) strategy successfully narrows the performance gap with conventional black-box LLMs, positioning CB-LLM as a model that combines the high accuracy of traditional LLMs with the added benefit of clear interpretability -- a feature markedly absent in existing LLMs.
• 摘要：我们引入了概念瓶颈大型语言模型 (CB-LLM)，这是一种创建固有可解释大型语言模型 (LLM) 的开创性方法。与依赖事后解释方法且对神经元功能了解有限的传统黑盒 LLM 不同，CB-LLM 以其内置的可解释性、可扩展性和提供清晰、准确解释的能力树立了新标准。这项创新不仅提高了语言模型的透明度，还提高了其有效性。我们独特的自动概念校正 (ACC) 策略成功缩小了与传统黑盒 LLM 的性能差距，将 CB-LLM 定位为一种结合了传统 LLM 的高精度和清晰可解释性的额外优势的模型——这是现有 LLM 明显缺乏的功能。

Title: Romanization Encoding For Multilingual ASR

• Authors: Wen Ding, Fei Jia, Hainan Xu, Yu Xi, Junjie Lai, Boris Ginsburg
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.04368
• Pdf URL: https://arxiv.org/pdf/2407.04368
• Copy Paste: [[2407.04368]] Romanization Encoding For Multilingual ASR(https://arxiv.org/abs/2407.04368)
• Keywords: language model
• Abstract: We introduce romanization encoding for script-heavy languages to optimize multilingual and code-switching Automatic Speech Recognition (ASR) systems. By adopting romanization encoding alongside a balanced concatenated tokenizer within a FastConformer-RNNT framework equipped with a Roman2Char module, we significantly reduce vocabulary and output dimensions, enabling larger training batches and reduced memory consumption. Our method decouples acoustic modeling and language modeling, enhancing the flexibility and adaptability of the system. In our study, applying this method to Mandarin-English ASR resulted in a remarkable 63.51% vocabulary reduction and notable performance gains of 13.72% and 15.03% on SEAME code-switching benchmarks. Ablation studies on Mandarin-Korean and Mandarin-Japanese highlight our method's strong capability to address the complexities of other script-heavy languages, paving the way for more versatile and effective multilingual ASR systems.
• 摘要：我们为脚本繁多的语言引入了罗马化编码，以优化多语言和代码转换自动语音识别 (ASR) 系统。通过在配备 Roman2Char 模块的 FastConformer-RNNT 框架中采用罗马化编码和平衡级联标记器，我们显著减少了词汇量和输出维度，从而实现了更大的训练批次并减少了内存消耗。我们的方法将声学建模和语言建模分离，增强了系统的灵活性和适应性。在我们的研究中，将这种方法应用于普通话-英语 ASR，在 SEAME 代码转换基准上显著减少了 63.51% 的词汇量，性能显著提高了 13.72% 和 15.03%。对普通话-韩语和普通话-日语的消融研究突出了我们的方法解决其他脚本繁多语言复杂性的强大能力，为更通用、更有效的多语言 ASR 系统铺平了道路。

Title: From 'Showgirls' to 'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs

• Authors: Marion Bartl, Susan Leavy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04434
• Pdf URL: https://arxiv.org/pdf/2407.04434
• Copy Paste: [[2407.04434]] From 'Showgirls' to 'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs(https://arxiv.org/abs/2407.04434)
• Keywords: language model, llm
• Abstract: Gender bias is not only prevalent in Large Language Models (LLMs) and their training data, but also firmly ingrained into the structural aspects of language itself. Therefore, adapting linguistic structures within LLM training data to promote gender-inclusivity can make gender representations within the model more inclusive. The focus of our work are gender-exclusive affixes in English, such as in 'show-girl' or 'man-cave', which can perpetuate gender stereotypes and binary conceptions of gender. We use an LLM training dataset to compile a catalogue of 692 gender-exclusive terms along with gender-neutral variants and from this, develop a gender-inclusive fine-tuning dataset, the 'Tiny Heap'. Fine-tuning three different LLMs with this dataset, we observe an overall reduction in gender-stereotyping tendencies across the models. Our approach provides a practical method for enhancing gender inclusivity in LLM training data and contributes to incorporating queer-feminist linguistic activism in bias mitigation research in NLP.
• 摘要：性别偏见不仅在大型语言模型 (LLM) 及其训练数据中普遍存在，而且在语言本身的结构方面也根深蒂固。因此，调整 LLM 训练数据中的语言结构以促进性别包容性可以使模型中的性别表示更具包容性。我们工作的重点是英语中的性别专属词缀，例如“show-girl”或“man-cave”，这些词缀可以延续性别刻板印象和性别二元概念。我们使用 LLM 训练数据集编制了一份包含 692 个性别专属术语以及性别中立变体的目录，并据此开发了一个性别包容的微调数据集“Tiny Heap”。使用该数据集对三个不同的 LLM 进行微调，我们观察到模型中的性别刻板印象倾向总体有所减少。我们的方法为增强 LLM 训练数据中的性别包容性提供了一种实用的方法，并有助于将酷儿女权主义语言行动主义纳入 NLP 中的偏见缓解研究中。

Title: Generalists vs. Specialists: Evaluating Large Language Models for Urdu

• Authors: Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais Athar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04459
• Pdf URL: https://arxiv.org/pdf/2407.04459
• Copy Paste: [[2407.04459]] Generalists vs. Specialists: Evaluating Large Language Models for Urdu(https://arxiv.org/abs/2407.04459)
• Keywords: language model, gpt, llm
• Abstract: In this paper, we compare general-purpose pretrained models, GPT-4-Turbo and Llama-3-8b-Instruct with special-purpose models fine-tuned on specific tasks, XLM-Roberta-large, mT5-large, and Llama-3-8b-Instruct. We focus on seven classification and six generation tasks to evaluate the performance of these models on Urdu language. Urdu has 70 million native speakers, yet it remains underrepresented in Natural Language Processing (NLP). Despite the frequent advancements in Large Language Models (LLMs), their performance in low-resource languages, including Urdu, still needs to be explored. We also conduct a human evaluation for the generation tasks and compare the results with the evaluations performed by GPT-4-Turbo and Llama-3-8b-Instruct. We find that special-purpose models consistently outperform general-purpose models across various tasks. We also find that the evaluation done by GPT-4-Turbo for generation tasks aligns more closely with human evaluation compared to the evaluation by Llama-3-8b-Instruct. This paper contributes to the NLP community by providing insights into the effectiveness of general and specific-purpose LLMs for low-resource languages.
• 摘要：在本文中，我们将通用预训练模型 GPT-4-Turbo 和 Llama-3-8b-Instruct 与针对特定任务进行微调的专用模型 XLM-Roberta-large、mT5-large 和 Llama-3-8b-Instruct 进行了比较。我们专注于七个分类任务和六个生成任务，以评估这些模型在乌尔都语上的表现。乌尔都语有 7000 万母语使用者，但在自然语言处理 (NLP) 中仍然代表性不足。尽管大型语言模型 (LLM) 不断取得进步，但它们在包括乌尔都语在内的低资源语言中的表现仍有待探索。我们还对生成任务进行了人工评估，并将结果与​​ GPT-4-Turbo 和 Llama-3-8b-Instruct 执行的评估进行了比较。我们发现专用模型在各种任务中的表现始终优于通用模型。我们还发现，与 Llama-3-8b-Instruct 的评估相比，GPT-4-Turbo 对生成任务的评估与人工评估更为接近。本文通过深入了解通用和专用 LLM 对低资源语言的有效性，为 NLP 社区做出了贡献。

Title: Using LLMs to label medical papers according to the CIViC evidence model

• Authors: Markus Hisch, Xing David Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04466
• Pdf URL: https://arxiv.org/pdf/2407.04466
• Copy Paste: [[2407.04466]] Using LLMs to label medical papers according to the CIViC evidence model(https://arxiv.org/abs/2407.04466)
• Keywords: language model, gpt, llm, prompt
• Abstract: We introduce the sequence classification problem CIViC Evidence to the field of medical NLP. CIViC Evidence denotes the multi-label classification problem of assigning labels of clinical evidence to abstracts of scientific papers which have examined various combinations of genomic variants, cancer types, and treatment approaches. We approach CIViC Evidence using different language models: We fine-tune pretrained checkpoints of BERT and RoBERTa on the CIViC Evidence dataset and challenge their performance with models of the same architecture which have been pretrained on domain-specific text. In this context, we find that BiomedBERT and BioLinkBERT can outperform BERT on CIViC Evidence (+0.8% and +0.9% absolute improvement in class-support weighted F1 score). All transformer-based models show a clear performance edge when compared to a logistic regression trained on bigram tf-idf scores (+1.5 - 2.7% improved F1 score). We compare the aforementioned BERT-like models to OpenAI's GPT-4 in a few-shot setting (on a small subset of our original test dataset), demonstrating that, without additional prompt-engineering or fine-tuning, GPT-4 performs worse on CIViC Evidence than our six fine-tuned models (66.1% weighted F1 score compared to 71.8% for the best fine-tuned model). However, performance gets reasonably close to the benchmark of a logistic regression model trained on bigram tf-idf scores (67.7% weighted F1 score).
• 摘要：我们将序列分类问题 CIViC Evidence 引入医学 NLP 领域。CIViC Evidence 表示将临床证据标签分配给科学论文摘要的多标签分类问题，这些论文摘要研究了各种基因组变异、癌症类型和治疗方法的组合。我们使用不同的语言模型来处理 CIViC Evidence：我们在 CIViC Evidence 数据集上微调了 BERT 和 RoBERTa 的预训练检查点，并使用在特定领域文本上预训练过的相同架构的模型来测试它们的性能。在这种情况下，我们发现 BiomedBERT 和 BioLinkBERT 在 CIViC Evidence 上的表现可以胜过 BERT（类支持加权 F1 分数的绝对提高 +0.8% 和 +0.9%）。与在二元 tf-idf 分数上训练的逻辑回归相比，所有基于 Transformer 的模型都表现出明显的性能优势（F1 分数提高了 +1.5 - 2.7%）。我们在少样本设置中（在我们原始测试数据集的一小部分上）将上述类似 BERT 的模型与 OpenAI 的 GPT-4 进行了比较，结果表明，在没有额外的提示工程或微调的情况下，GPT-4 在 CIViC Evidence 上的表现比我们的六个微调模型更差（66.1% 的加权 F1 得分，而最佳微调模型为 71.8%）。然而，性能相当接近在二元 tf-idf 得分上训练的逻辑回归模型的基准（67.7% 的加权 F1 得分）。

Title: Leveraging Graph Structures to Detect Hallucinations in Large Language Models

• Authors: Noa Nonkes, Sergei Agaronian, Evangelos Kanoulas, Roxana Petcu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04485
• Pdf URL: https://arxiv.org/pdf/2407.04485
• Copy Paste: [[2407.04485]] Leveraging Graph Structures to Detect Hallucinations in Large Language Models(https://arxiv.org/abs/2407.04485)
• Keywords: language model, hallucination
• Abstract: Large language models are extensively applied across a wide range of tasks, such as customer support, content creation, educational tutoring, and providing financial guidance. However, a well-known drawback is their predisposition to generate hallucinations. This damages the trustworthiness of the information these models provide, impacting decision-making and user confidence. We propose a method to detect hallucinations by looking at the structure of the latent space and finding associations within hallucinated and non-hallucinated generations. We create a graph structure that connects generations that lie closely in the embedding space. Moreover, we employ a Graph Attention Network which utilizes message passing to aggregate information from neighboring nodes and assigns varying degrees of importance to each neighbor based on their relevance. Our findings show that 1) there exists a structure in the latent space that differentiates between hallucinated and non-hallucinated generations, 2) Graph Attention Networks can learn this structure and generalize it to unseen generations, and 3) the robustness of our method is enhanced when incorporating contrastive learning. When evaluated against evidence-based benchmarks, our model performs similarly without access to search-based methods.
• 摘要：大型语言模型广泛应用于各种任务，例如客户支持、内容创建、教育辅导和提供财务指导。然而，一个众所周知的缺点是它们容易产生幻觉。这损害了这些模型提供的信息的可信度，影响了决策和用户信心。我们提出了一种检测幻觉的方法，通过查看潜在空间的结构并在幻觉和非幻觉代中寻找关联。我们创建了一个图形结构，将嵌入空间中紧密相连的代连接起来。此外，我们采用了图形注意力网络，它利用消息传递来聚合来自邻近节点的信息，并根据它们的相关性为每个邻居分配不同程度的重要性。我们的研究结果表明：1）潜在空间中存在一种结构，可以区分幻觉和非幻觉的代际；2）图注意力网络可以学习这种结构并将其推广到未见过的代际；3）结合对比学习可以增强我们方法的稳健性。当根据基于证据的基准进行评估时，我们的模型在不使用基于搜索的方法的情况下表现类似。

Title: GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning

• Authors: Aleksander Ficek, Jiaqi Zeng, Oleksii Kuchaiev
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04528
• Pdf URL: https://arxiv.org/pdf/2407.04528
• Copy Paste: [[2407.04528]] GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/2407.04528)
• Keywords: language model, gpt, retrieval-augmented generation
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) and Retrieval-Augmented Generation (RAG) have become popular methods for adapting large language models while minimizing compute requirements. In this paper, we apply PEFT methods (P-tuning, Adapters, and LoRA) to a modified Retrieval-Enhanced Transformer (RETRO) and a baseline GPT model across several sizes, ranging from 823 million to 48 billion parameters. We show that RETRO models outperform GPT models in zero-shot settings due to their unique pre-training process but GPT models have higher performance potential with PEFT. Additionally, our study indicates that 8B parameter models strike an optimal balance between cost and performance and P-tuning lags behind other PEFT techniques. We further provide a comparative analysis of between applying PEFT to an Instruction-tuned RETRO model and base RETRO model. This work presents the first comprehensive comparison of various PEFT methods integrated with RAG, applied to both GPT and RETRO models, highlighting their relative performance.
• 摘要：参数高效微调 (PEFT) 和检索增强生成 (RAG) 已成为适应大型语言模型同时最小化计算要求的流行方法。在本文中，我们将 PEFT 方法（P 调优、适配器和 LoRA）应用于改进的检索增强转换器 (RETRO) 和基线 GPT 模型，这些模型的参数大小从 8.23 亿到 480 亿不等。我们表明，RETRO 模型由于其独特的预训练过程，在零样本设置中的表现优于 GPT 模型，但 GPT 模型在 PEFT 方面具有更高的性能潜力。此外，我们的研究表明，8B 参数模型在成本和性能之间达到了最佳平衡，而 P 调优落后于其他 PEFT 技术。我们进一步提供了将 PEFT 应用于指令调优的 RETRO 模型和基本 RETRO 模型的比较分析。这项工作首次全面比较了与 RAG 集成的各种 PEFT 方法，应用于 GPT 和 RETRO 模型，突出了它们的相对性能。

Title: PoPreRo: A New Dataset for Popularity Prediction of Romanian Reddit Posts

• Authors: Ana-Cristina Rogoz, Maria Ilinca Nechita, Radu Tudor Ionescu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04541
• Pdf URL: https://arxiv.org/pdf/2407.04541
• Copy Paste: [[2407.04541]] PoPreRo: A New Dataset for Popularity Prediction of Romanian Reddit Posts(https://arxiv.org/abs/2407.04541)
• Keywords: language model, prompt
• Abstract: We introduce PoPreRo, the first dataset for Popularity Prediction of Romanian posts collected from Reddit. The PoPreRo dataset includes a varied compilation of post samples from five distinct subreddits of Romania, totaling 28,107 data samples. Along with our novel dataset, we introduce a set of competitive models to be used as baselines for future research. Interestingly, the top-scoring model achieves an accuracy of 61.35% and a macro F1 score of 60.60% on the test set, indicating that the popularity prediction task on PoPreRo is very challenging. Further investigations based on few-shot prompting the Falcon-7B Large Language Model also point in the same direction. We thus believe that PoPreRo is a valuable resource that can be used to evaluate models on predicting the popularity of social media posts in Romanian. We release our dataset at this https URL.
• 摘要：我们推出了 PoPreRo，这是从 Reddit 收集的第一个罗马尼亚语帖子流行度预测数据集。PoPreRo 数据集包括来自罗马尼亚五个不同子版块的帖子样本，总计 28,107 个数据样本。除了我们的新数据集外，我们还引入了一组竞争模型，用作未来研究的基线。有趣的是，得分最高的模型在测试集上的准确率为 61.35%，宏观 F1 得分为 60.60%，这表明 PoPreRo 上的流行度预测任务非常具有挑战性。基于 Falcon-7B 大型语言模型的少样本提示的进一步调查也指向了同一方向。因此，我们认为 PoPreRo 是一种宝贵的资源，可用于评估预测罗马尼亚语社交媒体帖子流行度的模型。我们在此 https URL 上发布了我们的数据集。

Title: Spontaneous Reward Hacking in Iterative Self-Refinement

• Authors: Jane Pan, He He, Samuel R. Bowman, Shi Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04549
• Pdf URL: https://arxiv.org/pdf/2407.04549
• Copy Paste: [[2407.04549]] Spontaneous Reward Hacking in Iterative Self-Refinement(https://arxiv.org/abs/2407.04549)
• Keywords: language model
• Abstract: Language models are capable of iteratively improving their outputs based on natural language feedback, thus enabling in-context optimization of user preference. In place of human users, a second language model can be used as an evaluator, providing feedback along with numerical ratings which the generator attempts to optimize. However, because the evaluator is an imperfect proxy of user preference, this optimization can lead to reward hacking, where the evaluator's ratings improve while the generation quality remains stagnant or even decreases as judged by actual user preference. The concern of reward hacking is heightened in iterative self-refinement where the generator and the evaluator use the same underlying language model, in which case the optimization pressure can drive them to exploit shared vulnerabilities. Using an essay editing task, we show that iterative self-refinement leads to deviation between the language model evaluator and human judgment, demonstrating that reward hacking can occur spontaneously in-context with the use of iterative self-refinement. In addition, we study conditions under which reward hacking occurs and observe two factors that affect reward hacking severity: model size and context sharing between the generator and the evaluator.
• 摘要：语言模型能够基于自然语言反馈迭代改进其输出，从而实现用户偏好的上下文优化。可以使用第二个语言模型代替人类用户作为评估器，提供反馈以及生成器试图优化的数字评分。但是，由于评估器不是用户偏好的完美代理，因此这种优化可能会导致奖励黑客攻击，即评估器的评分提高，而生成质量却保持不变甚至根据实际用户偏好判断下降。在迭代自我改进中，生成器和评估器使用相同的底层语言模型，在这种情况下，优化压力会驱使他们利用共同的漏洞，这加剧了对奖励黑客攻击的担忧。使用论文编辑任务，我们表明迭代自我改进会导致语言模型评估器和人类判断之间的偏差，表明奖励黑客攻击可以在使用迭代自我改进的情况下自发地在上下文中发生。此外，我们研究了奖励黑客发生的条件，并观察到影响奖励黑客严重程度的两个因素：模型大小以及生成器和评估器之间的上下文共享。

Title: Testing learning hypotheses using neural networks by manipulating learning data

• Authors: Cara Su-Yi Leong, Tal Linzen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04593
• Pdf URL: https://arxiv.org/pdf/2407.04593
• Copy Paste: [[2407.04593]] Testing learning hypotheses using neural networks by manipulating learning data(https://arxiv.org/abs/2407.04593)
• Keywords: language model
• Abstract: Although passivization is productive in English, it is not completely general -- some exceptions exist (e.g. *One hour was lasted by the meeting). How do English speakers learn these exceptions to an otherwise general pattern? Using neural network language models as theories of acquisition, we explore the sources of indirect evidence that a learner can leverage to learn whether a verb can passivize. We first characterize English speakers' judgments of exceptions to the passive, confirming that speakers find some verbs more passivizable than others. We then show that a neural network language model can learn restrictions to the passive that are similar to those displayed by humans, suggesting that evidence for these exceptions is available in the linguistic input. We test the causal role of two hypotheses for how the language model learns these restrictions by training models on modified training corpora, which we create by altering the existing training corpora to remove features of the input implicated by each hypothesis. We find that while the frequency with which a verb appears in the passive significantly affects its passivizability, the semantics of the verb does not. This study highlight the utility of altering a language model's training data for answering questions where complete control over a learner's input is vital.
• 摘要：尽管被动语态在英语中很常见，但它并不完全普遍——存在一些例外（例如 *会议持续了一个小时）。英语使用者如何学习这些普遍模式的例外？使用神经网络语言模型作为习得理论，我们探索学习者可以利用的间接证据来源，以了解动词是否可以被动化。我们首先描述英语使用者对被动语态例外的判断，证实说话者发现某些动词比其他动词更容易被动化。然后我们表明神经网络语言模型可以学习与人类相似的被动语态限制，这表明这些例外的证据可以在语言输入中找到。我们通过在修改后的训练语料库上训练模型来测试语言模型如何学习这些限制的两个假设的因果作用，我们通过改变现有的训练语料库来创建这些语料库，以删除每个假设所涉及的输入特征。我们发现，虽然动词在被动语态中出现的频率会显著影响其被动性，但动词的语义却不会。这项研究强调了改变语言模型的训练数据对于回答问题的实用性，因为完全控制学习者的输入至关重要。

Title: ARM: Efficient Guided Decoding with Autoregressive Reward Models

• Authors: Sergey Troshin, Vlad Niculae, Antske Fokkens
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.04615
• Pdf URL: https://arxiv.org/pdf/2407.04615
• Copy Paste: [[2407.04615]] ARM: Efficient Guided Decoding with Autoregressive Reward Models(https://arxiv.org/abs/2407.04615)
• Keywords: language model
• Abstract: Language models trained on large amounts of data require careful tuning to be safely deployed in real world. We revisit the guided decoding paradigm, where the goal is to augment the logits of the base language model using the scores from a task-specific reward model. We propose a simple but efficient parameterization of the autoregressive reward model enabling fast and effective guided decoding. On detoxification and sentiment control tasks, we show that our efficient parameterization performs on par with RAD, a strong but less efficient guided decoding approach.
• 摘要：在大量数据上训练的语言模型需要仔细调整才能安全地部署到现实世界中。我们重新审视了引导解码范式，其目标是使用特定于任务的奖励模型的分数来增强基础语言模型的逻辑。我们提出了一种简单但有效的自回归奖励模型参数化方法，可实现快速有效的引导解码。在排毒和情绪控制任务中，我们表明我们的高效参数化与 RAD 相当，后者是一种强大但效率较低的引导解码方法。

Title: Entity Decomposition with Filtering: A Zero-Shot Clinical Named Entity Recognition Framework

• Authors: Reza Averly, Xia Ning
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04629
• Pdf URL: https://arxiv.org/pdf/2407.04629
• Copy Paste: [[2407.04629]] Entity Decomposition with Filtering: A Zero-Shot Clinical Named Entity Recognition Framework(https://arxiv.org/abs/2407.04629)
• Keywords: language model, llm
• Abstract: Clinical named entity recognition (NER) aims to retrieve important entities within clinical narratives. Recent works have demonstrated that large language models (LLMs) can achieve strong performance in this task. While previous works focus on proprietary LLMs, we investigate how open NER LLMs, trained specifically for entity recognition, perform in clinical NER. In this paper, we aim to improve them through a novel framework, entity decomposition with filtering, or EDF. Our key idea is to decompose the entity recognition task into several retrievals of sub-entity types. We also introduce a filtering mechanism to remove incorrect entities. Our experimental results demonstrate the efficacy of our framework across all metrics, models, datasets, and entity types. Our analysis reveals that entity decomposition can recognize previously missed entities with substantial improvement. We further provide a comprehensive evaluation of our framework and an in-depth error analysis to pave future works.
• 摘要：临床命名实体识别 (NER) 旨在检索临床叙述中的重要实体。最近的研究表明，大型语言模型 (LLM) 可以在这项任务中取得出色的表现。虽然以前的研究主要关注专有 LLM，但我们研究了专门为实体识别而训练的开放式 NER LLM 在临床 NER 中的表现。在本文中，我们旨在通过一个新颖的框架（带过滤的实体分解或 EDF）来改进它们。我们的主要思想是将实体识别任务分解为几个子实体类型的检索。我们还引入了一种过滤机制来删除不正确的实体。我们的实验结果证明了我们的框架在所有指标、模型、数据集和实体类型中的有效性。我们的分析表明，实体分解可以显著提高识别以前遗漏的实体的能力。我们进一步对我们的框架进行了全面的评估和深入的错误分析，为未来的工作铺平了道路。

Title: ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models

• Authors: Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, Kai Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.04693
• Pdf URL: https://arxiv.org/pdf/2407.04693
• Copy Paste: [[2407.04693]] ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models(https://arxiv.org/abs/2407.04693)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Large language models (LLMs) exhibit hallucinations in long-form question-answering tasks across various domains and wide applications. Current hallucination detection and mitigation datasets are limited in domains and sizes, which struggle to scale due to prohibitive labor costs and insufficient reliability of existing hallucination annotators. To facilitate the scalable oversight of LLM hallucinations, this paper introduces an iterative self-training framework that simultaneously and progressively scales up the hallucination annotation dataset and improves the accuracy of the hallucination annotator. Based on the Expectation Maximization (EM) algorithm, in each iteration, the framework first applies a hallucination annotation pipeline to annotate a scaled dataset and then trains a more accurate hallucination annotator on the dataset. This new hallucination annotator is adopted in the hallucination annotation pipeline used for the next iteration. Extensive experimental results demonstrate that the finally obtained hallucination annotator with only 7B parameters surpasses the performance of GPT-4 and obtains new state-of-the-art hallucination detection results on HaluEval and HalluQA by zero-shot inference. Such an annotator can not only evaluate the hallucination levels of various LLMs on the large-scale dataset but also help to mitigate the hallucination of LLMs generations, with the Natural Language Inference (NLI) metric increasing from 25% to 37% on HaluEval.
• 摘要：大型语言模型 (LLM) 在各个领域和广泛应用中的长篇问答任务中表现出幻觉。当前的幻觉检测和缓解数据集受限于领域和大小，由于劳动力成本过高和现有幻觉注释器的可靠性不足而难以扩展。为了促进对 LLM 幻觉的可扩展监督，本文介绍了一个迭代自训练框架，该框架可以同时逐步扩大幻觉注释数据集并提高幻觉注释器的准确性。基于期望最大化 (EM) 算法，在每次迭代中，该框架首先应用幻觉注释管道来注释缩放后的数据集，然后在该数据集上训练更准确的幻觉注释器。新的幻觉注释器将在下一次迭代使用的幻觉注释管道中采用。大量实验结果表明，最终得到的仅使用 7B 参数的幻觉标注器超越了 GPT-4 的性能，并通过零样本推理在 HaluEval 和 HalluQA 上获得了新的最佳幻觉检测结果。该标注器不仅可以评估大规模数据集上各种 LLM 的幻觉水平，而且有助于缓解 LLM 生成的幻觉，其中自然语言推理 (NLI) 指标在 HaluEval 上从 25% 提升到 37%。

Title: Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs

• Authors: Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Jeremy Scheurer, Mikita Balesni, Marius Hobbhahn, Alexander Meinke, Owain Evans
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.04694
• Pdf URL: https://arxiv.org/pdf/2407.04694
• Copy Paste: [[2407.04694]] Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs(https://arxiv.org/abs/2407.04694)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness. To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the $\textbf{Situational Awareness Dataset (SAD)}$, a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge. We evaluate 16 LLMs on SAD, including both base (pretrained) and chat models. While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU). Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks. The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control. Code and latest results available at this https URL .
• 摘要：ChatGPT 等人工智能助手经过训练，可以通过说“我是一个大型语言模型”来回应用户。这引发了一些问题。这些模型是否知道它们是 LLM，并可靠地根据这些知识采取行动？它们是否了解当前的情况，例如被部署到公众中？我们将模型对自身及其环境的了解称为情境意识。为了量化 LLM 中的情境意识，我们引入了一系列基于问答和指令遵循的行为测试。这些测试构成了 $\textbf{情境意识数据集 (SAD)}$，这是一个包含 7 个任务类别和超过 13,000 个问题的基准。该基准测试了许多能力，包括 LLM 的能力：(i) 识别自己生成的文本、(ii) 预测自己的行为、(iii) 确定提示是来自内部评估还是现实世界部署，以及 (iv) 遵循依赖于自我知识的指令。我们在 SAD 上评估了 16 个 LLM，包括基础（预训练）和聊天模型。虽然所有模型的表现都比偶然性要好，但即使是得分最高的模型（Claude 3 Opus）在某些任务上也远远落后于人类基线。我们还观察到，SAD 上的表现只能部分通过常识指标（例如 MMLU）来预测。经过微调以充当 AI 助手的聊天模型在 SAD 上的表现优于其相应的基础模型，但在常识任务上则不然。SAD 的目的是通过将 LLM 中的态势感知分解为定量能力来促进对态势感知的科学理解。态势感知很重要，因为它可以增强模型的自主规划和行动能力。虽然这对自动化有潜在的好处，但也带来了与 AI 安全和控制相关的新风险。代码和最新结果可在此 https URL 上获得。