2024-03-11

Title: QASE Enhanced PLMs: Improved Control in Text Generation for MRC

• Authors: Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04771
• Pdf URL: https://arxiv.org/pdf/2403.04771
• Copy Paste: [[2403.04771]] QASE Enhanced PLMs: Improved Control in Text Generation for MRC(https://arxiv.org/abs/2403.04771)
• Keywords: language model, gpt, llm
• Abstract: To address the challenges of out-of-control generation in generative models for machine reading comprehension (MRC), we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning of pre-trained generative language models (PLMs), QASE enables these PLMs to match SOTA extractive methods and outperform leading LLMs like GPT-4 in MRC tasks, without significant increases in computational costs.
• 摘要：为了解决机器阅读理解（MRC）生成模型中失控生成的挑战，我们引入了问题参与跨度提取（QASE）模块。 QASE 在预训练生成语言模型 (PLM) 的微调过程中进行集成，使这些 PLM 能够匹配 SOTA 提取方法，并在 MRC 任务中优于 GPT-4 等领先的 LLM，而不会显着增加计算成本。

Title: MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining

• Authors: Yanchao Tan, Hang Lv, Xinyi Huang, Jiawei Zhang, Shiping Wang, Carl Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04780
• Pdf URL: https://arxiv.org/pdf/2403.04780
• Copy Paste: [[2403.04780]] MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining(https://arxiv.org/abs/2403.04780)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: Graphs with abundant attributes are essential in modeling interconnected entities and improving predictions in various real-world applications. Traditional Graph Neural Networks (GNNs), which are commonly used for modeling attributed graphs, need to be re-trained every time when applied to different graph tasks and datasets. Although the emergence of Large Language Models (LLMs) has introduced a new paradigm in natural language processing, the generative potential of LLMs in graph mining remains largely under-explored. To this end, we propose a novel framework MuseGraph, which seamlessly integrates the strengths of GNNs and LLMs and facilitates a more effective and generic approach for graph mining across different tasks and datasets. Specifically, we first introduce a compact graph description via the proposed adaptive input generation to encapsulate key information from the graph under the constraints of language token limitations. Then, we propose a diverse instruction generation mechanism, which distills the reasoning capabilities from LLMs (e.g., GPT-4) to create task-specific Chain-of-Thought-based instruction packages for different graph tasks. Finally, we propose a graph-aware instruction tuning with a dynamic instruction package allocation strategy across tasks and datasets, ensuring the effectiveness and generalization of the training process. Our experimental results demonstrate significant improvements in different graph tasks, showcasing the potential of our MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while keeping the generation powers of LLMs.
• 摘要：具有丰富属性的图对于互连实体建模和改进各种现实应用中的预测至关重要。传统的图神经网络（GNN）通常用于建模属性图，每次应用于不同的图任务和数据集时都需要重新训练。尽管大型语言模型（LLM）的出现引入了自然语言处理的新范式，但 LLM 在图挖掘中的生成潜力在很大程度上仍未得到充分开发。为此，我们提出了一种新颖的框架 MuseGraph，它无缝地集成了 GNN 和 LLM 的优势，并为跨不同任务和数据集的图挖掘提供了更有效和通用的方法。具体来说，我们首先通过所提出的自适应输入生成引入紧凑的图描述，以在语言标记限制的约束下封装图中的关键信息。然后，我们提出了一种多样化的指令生成机制，该机制从 LLM（例如 GPT-4）中提炼推理能力，为不同的图形任务创建特定于任务的基于思想链的指令包。最后，我们提出了一种图形感知指令调优，具有跨任务和数据集的动态指令包分配策略，确保训练过程的有效性和泛化性。我们的实验结果证明了不同图形任务的显着改进，展示了我们的 MuseGraph 在提高面向图形的下游任务的准确性方面的潜力，同时保持了 LLM 的生成能力。

Title: Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data

• Authors: Jun-En Ding, Phan Nguyen Minh Thao, Wen-Chih Peng, Jian-Zhe Wang, Chun-Cheng Chug, Min-Chen Hsieh, Yun-Chien Tseng, Ling Chen, Dongsheng Luo, Chi-Te Wang, Pei-fu Chen, Feng Liu, Fang-Ming Hung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04785
• Pdf URL: https://arxiv.org/pdf/2403.04785
• Copy Paste: [[2403.04785]] Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data(https://arxiv.org/abs/2403.04785)
• Keywords: language model, llm
• Abstract: Chronic diseases such as diabetes are the leading causes of morbidity and mortality worldwide. Numerous research studies have been attempted with various deep learning models in diagnosis. However, most previous studies had certain limitations, including using publicly available datasets (e.g. MIMIC), and imbalanced data. In this study, we collected five-year electronic health records (EHRs) from the Taiwan hospital database, including 1,420,596 clinical notes, 387,392 laboratory test results, and more than 1,505 laboratory test items, focusing on research pre-training large language models. We proposed a novel Large Language Multimodal Models (LLMMs) framework incorporating multimodal data from clinical notes and laboratory test results for the prediction of chronic disease risk. Our method combined a text embedding encoder and multi-head attention layer to learn laboratory test values, utilizing a deep neural network (DNN) module to merge blood features with chronic disease semantics into a latent space. In our experiments, we observe that clinicalBERT and PubMed-BERT, when combined with attention fusion, can achieve an accuracy of 73% in multiclass chronic diseases and diabetes prediction. By transforming laboratory test values into textual descriptions and employing the Flan T-5 model, we achieved a 76% Area Under the ROC Curve (AUROC), demonstrating the effectiveness of leveraging numerical text data for training and inference in language models. This approach significantly improves the accuracy of early-stage diabetes prediction.
• 摘要：糖尿病等慢性病是全世界发病和死亡的主要原因。人们在诊断中尝试了各种深度学习模型的大量研究。然而，之前的大多数研究都有一定的局限性，包括使用公开的数据集（例如 MIMIC）和不平衡的数据。在这项研究中，我们从台湾医院数据库中收集了五年的电子健康记录（EHR），包括1,420,596条临床记录、387,392条实验室检查结果和超过1,505个实验室检查项目，重点研究预训练大语言模型。我们提出了一种新颖的大语言多模态模型（LLMM）框架，该框架结合了来自临床记录和实验室测试结果的多模态数据，用于预测慢性病风险。我们的方法结合了文本嵌入编码器和多头注意力层来学习实验室测试值，利用深度神经网络（DNN）模块将血液特征与慢性疾病语义合并到潜在空间中。在我们的实验中，我们观察到，clinicalBERT 和 PubMed-BERT 与注意力融合相结合，在多类慢性病和糖尿病预测中可以达到 73% 的准确率。通过将实验室测试值转换为文本描述并采用 Flan T-5 模型，我们实现了 76% 的 ROC 曲线下面积 (AUROC)，证明了利用数字文本数据进行语言模型训练和推理的有效性。这种方法显着提高了早期糖尿病预测的准确性。

Title: Ever-Evolving Memory by Blending and Refining the Past

• Authors: Seo Hyun Kim, Keummin Ka, Yohan Jo, Seung-won Hwang, Dongha Lee, Jinyoung Yeo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04787
• Pdf URL: https://arxiv.org/pdf/2403.04787
• Copy Paste: [[2403.04787]] Ever-Evolving Memory by Blending and Refining the Past(https://arxiv.org/abs/2403.04787)
• Keywords: chat
• Abstract: For a human-like chatbot, constructing a long-term memory is crucial. A naive approach for making a memory could be simply listing the summarized dialogue. However, this can lead to problems when the speaker's status change over time and contradictory information gets accumulated. It is important that the memory stays organized to lower the confusion for the response generator. In this paper, we propose a novel memory scheme for long-term conversation, CREEM. Unlike existing approaches that construct memory based solely on current sessions, our proposed model blending past memories during memory formation. Additionally, we introduce refining process to handle redundant or outdated information. This innovative approach seeks for overall improvement and coherence of chatbot responses by ensuring a more informed and dynamically evolving long-term memory.
• 摘要：对于类人聊天机器人来说，构建长期记忆至关重要。一种幼稚的记忆方法可能是简单地列出总结的对话。然而，当说话者的状态随着时间的推移而变化并且矛盾的信息积累时，这可能会导致问题。重要的是，记忆保持井井有条，以减少响应生成器的混乱。在本文中，我们提出了一种用于长期对话的新颖记忆方案 CREEM。与仅基于当前会话构建记忆的现有方法不同，我们提出的模型在记忆形成过程中混合了过去的记忆。此外，我们引入精炼流程来处理冗余或过时的信息。这种创新方法通过确保更明智和动态发展的长期记忆来寻求聊天机器人响应的整体改进和一致性。

Title: Online Training of Large Language Models: Learn while chatting

• Authors: Juhao Liang, Ziwei Wang, Zhuoheng Ma, Jianquan Li, Zhiyi Zhang, Xiangbo Wu, Benyou Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04790
• Pdf URL: https://arxiv.org/pdf/2403.04790
• Copy Paste: [[2403.04790]] Online Training of Large Language Models: Learn while chatting(https://arxiv.org/abs/2403.04790)
• Keywords: language model, llm, chat, agent
• Abstract: Large Language Models(LLMs) have dramatically revolutionized the field of Natural Language Processing(NLP), offering remarkable capabilities that have garnered widespread usage. However, existing interaction paradigms between LLMs and users are constrained by either inflexibility, limitations in customization, or a lack of persistent learning. This inflexibility is particularly evident as users, especially those without programming skills, have restricted avenues to enhance or personalize the model. Existing frameworks further complicate the model training and deployment process due to their computational inefficiencies and lack of user-friendly interfaces. To overcome these challenges, this paper introduces a novel interaction paradigm-'Online Training using External Interactions'-that merges the benefits of persistent, real-time model updates with the flexibility for individual customization through external interactions such as AI agents or online/offline knowledge bases.
• 摘要：大型语言模型 (LLM) 极大地改变了自然语言处理 (NLP) 领域，提供了广泛使用的卓越功能。然而，法学硕士和用户之间现有的交互范式要么缺乏灵活性，要么定制化有限，要么缺乏持续学习。当用户（尤其是那些没有编程技能的用户）增强或个性化模型的途径受到限制时，这种不灵活性尤其明显。由于计算效率低下且缺乏用户友好的界面，现有框架使模型训练和部署过程进一步复杂化。为了克服这些挑战，本文引入了一种新颖的交互范式——“使用外部交互的在线训练”——它将持久、实时模型更新的优点与通过外部交互（例如人工智能代理或在线/离线）进行个性化定制的灵活性结合在一起知识库。

Title: LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset

• Authors: Ahmed Izzidien, Holli Sargeant, Felix Steffek
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04791
• Pdf URL: https://arxiv.org/pdf/2403.04791
• Copy Paste: [[2403.04791]] LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset(https://arxiv.org/abs/2403.04791)
• Keywords: language model, llm, prompt
• Abstract: To undertake computational research of the law, efficiently identifying datasets of court decisions that relate to a specific legal issue is a crucial yet challenging endeavour. This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We introduce a comparative analysis of two computational methods: (1) a traditional natural language processing-based approach leveraging expert-generated keywords and logical operators and (2) an innovative application of the Claude 2 large language model to classify cases based on content-specific prompts. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. Despite iterative refinement, the search logic based on keywords fails to capture nuances in legal language. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span. The paper marks a pioneering step in employing advanced natural language processing to tackle core legal research tasks, demonstrating how these technologies can bridge systemic gaps and enhance the accessibility of legal information. We share the extracted dataset metrics to support further research on summary judgments.
• 摘要：为了进行法律的计算研究，有效识别与特​​定法律问题相关的法院判决数据集是一项至关重要但具有挑战性的工作。这项研究解决了与大型法律语料库合作的文献中关于如何将案件（在我们的案件简易判决中）与英国法院判决的大型语料库分离出来的空白。我们介绍了两种计算方法的比较分析：(1) 基于传统自然语言处理的方法，利用专家生成的关键字和逻辑运算符；(2) Claude 2 大语言模型的创新应用，根据内容对案例进行分类具体提示。我们使用包含 356,011 个英国法院判决的剑桥法律语料库，并确定大型语言模型的加权 F1 分数为 0.94，而关键字的加权 F1 分数为 0.78。尽管进行了迭代细化，基于关键字的搜索逻辑仍无法捕捉法律语言中的细微差别。我们识别并提取了 3,102 个简易判决案件，使我们能够绘制它们在一段时间内在英国各个法院的分布情况。该论文标志着采用先进的自然语言处理来解决核心法律研究任务的开创性一步，展示了这些技术如何弥合系统性差距并提高法律信息的可访问性。我们共享提取的数据集指标以支持对总结判断的进一步研究。

Title: Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?

• Authors: Yotam Intrator, Matan Halfon, Roman Goldenberg, Reut Tsarfaty, Matan Eyal, Ehud Rivlin, Yossi Matias, Natalia Aizenberg
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04792
• Pdf URL: https://arxiv.org/pdf/2403.04792
• Copy Paste: [[2403.04792]] Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?(https://arxiv.org/abs/2403.04792)
• Keywords: language model, llm
• Abstract: Large language models hold significant promise in multilingual applications. However, inherent biases stemming from predominantly English-centric pre-training have led to the widespread practice of pre-translation, i.e., translating non-English inputs to English before inference, leading to complexity and information loss. This study re-evaluates the need for pre-translation in the context of PaLM2 models (Anil et al., 2023), which have been established as highly performant in multilingual tasks. We offer a comprehensive investigation across 108 languages and 6 diverse benchmarks, including open-end generative tasks, which were excluded from previous similar studies. Our findings challenge the pre-translation paradigm established in prior research, highlighting the advantages of direct inference in PaLM2. Specifically, PaLM2-L consistently outperforms pre-translation in 94 out of 108 languages. These findings pave the way for more efficient and effective multilingual applications, alleviating the limitations associated with pre-translation and unlocking linguistic authenticity.
• 摘要：大型语言模型在多语言应用中具有重大前景。然而，主要以英语为中心的预训练所产生的固有偏差导致了预翻译的广泛实践，即在推理之前将非英语输入翻译成英语，从而导致复杂性和信息丢失。这项研究重新评估了 PaLM2 模型（Anil 等人，2023）背景下预翻译的需求，该模型已被证明在多语言任务中具有高性能。我们提供了针对 108 种语言和 6 个不同基准的全面调查，包括开放式生成任务，这些任务被排除在之前的类似研究之外。我们的研究结果挑战了先前研究中建立的预翻译范式，强调了 PaLM2 中直接推理的优势。具体来说，PaLM2-L 在 108 种语言中的 94 种中始终优于预翻译。这些发现为更高效和有效的多语言应用铺平了道路，减轻了与预翻译相关的限制并释放了语言的真实性。

Title: Large Language Models in Fire Engineering: An Examination of Technical Questions Against Domain Knowledge

• Authors: Haley Hostetter, M.Z. Naser, Xinyan Huang, John Gales
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04795
• Pdf URL: https://arxiv.org/pdf/2403.04795
• Copy Paste: [[2403.04795]] Large Language Models in Fire Engineering: An Examination of Technical Questions Against Domain Knowledge(https://arxiv.org/abs/2403.04795)
• Keywords: language model, gpt, chat
• Abstract: This communication presents preliminary findings from comparing two recent chatbots, OpenAI's ChatGPT and Google's Bard, in the context of fire engineering by evaluating their responses in handling fire safety related queries. A diverse range of fire engineering questions and scenarios were created and examined, including structural fire design, fire prevention strategies, evacuation, building code compliance, and fire suppression systems (some of which resemble those commonly present in the Fire Protection exam (FPE)). The results reveal some key differences in the performance of the chatbots, with ChatGPT demonstrating a relatively superior performance. Then, this communication highlights the potential for chatbot technology to revolutionize fire engineering practices by providing instant access to critical information while outlining areas for further improvement and research. Evidently, and when it matures, this technology will likely be elemental to our engineers' practice and education.
• 摘要：本次交流通过评估最近的两个聊天机器人（OpenAI 的 ChatGPT 和 Google 的 Bard）在消防工程背景下在处理消防安全相关查询时的响应，提出了比较的初步结果。创建并检查了各种消防工程问题和场景，包括结构消防设计、防火策略、疏散、建筑规范合规性和灭火系统（其中一些类似于消防考试（FPE）中常见的问题和场景） 。结果揭示了聊天机器人性能的一些关键差异，其中 ChatGPT 表现出相对优越的性能。然后，本次交流强调了聊天机器人技术通过提供对关键信息的即时访问，同时概述了需要进一步改进和研究的领域，从而彻底改变消防工程实践的潜力。显然，当这项技术成熟时，它可能会成为我们工程师的实践和教育的基础。

Title: Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

• Authors: Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04797
• Pdf URL: https://arxiv.org/pdf/2403.04797
• Copy Paste: [[2403.04797]] Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding(https://arxiv.org/abs/2403.04797)
• Keywords: language model, llm, long context
• Abstract: This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.
• 摘要：本文旨在克服大型语言模型（LLM）的“迷失在中间”的挑战。虽然最近的进展已成功使法学硕士能够使用多达 400 万个标记执行稳定的语言建模，但大多数法学硕士在识别位于上下文中间的相关信息方面所面临的持续困难尚未得到充分解决。为了解决这个问题，本文引入了多尺度位置编码（Ms-PoE），这是一种简单而有效的即插即用方法，可以增强 LLM 处理位于上下文中间的相关信息的能力，而无需微调或引入任何额外的开销。 Ms-PoE 利用位置指数重新缩放来缓解 RoPE 引入的长期衰减效应，同时精心地将不同的缩放比例分配给不同的注意力头，以保留在预训练步骤中学到的基本知识，形成多尺度上下文融合短距离到长距离。对各种法学硕士进行的广泛实验证明了我们方法的有效性。值得注意的是，与原始法学硕士相比，Ms-PoE 在零滚动基准测试中的平均准确度提高了 3.8。代码可在 https://github.com/VITA-Group/Ms-PoE 获取。

Title: JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models

• Authors: Arefa, Mohammed Abbas Ansari, Chandni Saxena, Tanvir Ahmad
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04798
• Pdf URL: https://arxiv.org/pdf/2403.04798
• Copy Paste: [[2403.04798]] JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models(https://arxiv.org/abs/2403.04798)
• Keywords: gpt
• Abstract: This paper presents our system development for SemEval-2024 Task 3: "The Competition of Multimodal Emotion Cause Analysis in Conversations". Effectively capturing emotions in human conversations requires integrating multiple modalities such as text, audio, and video. However, the complexities of these diverse modalities pose challenges for developing an efficient multimodal emotion cause analysis (ECA) system. Our proposed approach addresses these challenges by a two-step framework. We adopt two different approaches in our implementation. In Approach 1, we employ instruction-tuning with two separate Llama 2 models for emotion and cause prediction. In Approach 2, we use GPT-4V for conversation-level video description and employ in-context learning with annotated conversation using GPT 3.5. Our system wins rank 4, and system ablation experiments demonstrate that our proposed solutions achieve significant performance gains. All the experimental codes are available on Github.
• 摘要：本文介绍了我们针对 SemEval-2024 任务 3：“对话中多模态情感原因分析的竞争”的系统开发。有效捕捉人类对话中的情感需要集成文本、音频和视频等多种模式。然而，这些不同模式的复杂性给开发高效的多模式情绪原因分析（ECA）系统带来了挑战。我们提出的方法通过两步框架解决这些挑战。我们在实施中采用两种不同的方法。在方法 1 中，我们采用两个独立的 Llama 2 模型的指令调整来进行情绪和原因预测。在方法 2 中，我们使用 GPT-4V 进行对话级视频描述，并使用 GPT 3.5 通过带注释的对话进行上下文学习。我们的系统获得了第四名，系统消融实验表明我们提出的解决方案实现了显着的性能提升。所有实验代码均可在Github上获取。

Title: Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

• Authors: Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, Santu Rana
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04801
• Pdf URL: https://arxiv.org/pdf/2403.04801
• Copy Paste: [[2403.04801]] Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs(https://arxiv.org/abs/2403.04801)
• Keywords: llm, prompt, agent
• Abstract: In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs. We use an iterative rejection-sampling optimization process to find instruction-based prompts with two main characteristics: (1) minimal overlap with the training data to avoid presenting the solution directly to the model, and (2) maximal overlap between the victim model's output and the training data, aiming to induce the victim to spit out training data. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that (1) instruction-tuned models can expose pre-training data as much as their base-models, if not more so, (2) contexts other than the original training data can lead to leakage, and (3) using instructions proposed by other LLMs can open a new avenue of automated attacks that we should further study and explore. The code can be found at https://github.com/Alymostafa/Instruction_based_attack .
• 摘要：在本文中，我们介绍了一种黑盒提示优化方法，与直接用训练数据提示目标模型所揭示的内容相比，该方法使用攻击者 LLM 代理来揭示受害者代理中更高水平的记忆，这是主要的法学硕士量化记忆的方法。我们使用迭代拒绝采样优化过程来查找具有两个主要特征的基于指令的提示：(1) 与训练数据的最小重叠，以避免将解决方案直接呈现给模型，以及 (2) 受害者模型输出之间的最大重叠以及训练数据，旨在诱导受害者吐出训练数据。我们观察到，与基线前缀后缀测量相比，基于指令的提示生成的输出与训练数据的重叠度高 23.7%。我们的研究结果表明，（1）指令调整模型可以暴露与其基本模型一样多的预训练数据，甚至更多，（2）原始训练数据以外的上下文可能导致泄漏，（3）使用其他法学硕士提出的指令可以开辟一条新的自动化攻击途径，我们应该进一步研究和探索。代码可以在 https://github.com/Alymostafa/Instruction_based_attack 找到。

Title: Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

• Authors: Linyuan Gong, Sida Wang, Mostafa Elhoushi, Alvin Cheung
• Subjects: cs.CL, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.04814
• Pdf URL: https://arxiv.org/pdf/2403.04814
• Copy Paste: [[2403.04814]] Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks(https://arxiv.org/abs/2403.04814)
• Keywords: language model, llm, prompt
• Abstract: We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses on syntax-aware completions of program structures such as code blocks and conditional expressions, and includes 17,720 examples from multiple programming languages, sourced from recent code submissions after April 2022 to minimize data contamination. SAFIM provides a robust framework with various prompt designs and novel syntax-aware post-processing techniques, facilitating accurate and fair comparisons across LLMs. Our comprehensive evaluation of 15 LLMs shows that FIM pretraining not only enhances FIM proficiency but also improves Left-to-Right (L2R) inference using LLMs. Our findings challenge conventional beliefs and suggest that pretraining methods and data quality have more impact than model size. SAFIM thus serves as a foundational platform for future research in effective pretraining strategies for code LLMs. The evaluation toolkit and dataset are available at https://github.com/gonglinyuan/safim, and the leaderboard is available at https://safimbenchmark.com.
• 摘要：我们引入了语法感知填充中间（SAFIM），这是一个新的基准，用于在代码填充中间（FIM）任务上评估大型语言模型（LLM）。该基准测试重点关注代码块和条件表达式等程序结构的语法感知补全，包括来自多种编程语言的 17,720 个示例，这些示例源自 2022 年 4 月后最近提交的代码，以最大限度地减少数据污染。 SAFIM 提供了一个强大的框架，具有各种提示设计和新颖的语法感知后处理技术，促进法学硕士之间的准确和公平比较。我们对 15 名法学硕士的综合评估表明，FIM 预训练不仅提高了 FIM 熟练程度，而且还提高了法学硕士的从左到右 (L2R) 推理能力。我们的研究结果挑战了传统观念，并表明预训练方法和数据质量比模型大小具有更大的影响。因此，SAFIM 可以作为未来研究代码法学硕士有效预训练策略的基础平台。评估工具包和数据集可在 https://github.com/gonglinyuan/safim 获取，排行榜可在 https://safimbenchmark.com 获取。

Title: Evaluating Biases in Context-Dependent Health Questions

• Authors: Sharon Levy, Tahilin Sanchez Karver, William D. Adler, Michelle R. Kaufman, Mark Dredze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04858
• Pdf URL: https://arxiv.org/pdf/2403.04858
• Copy Paste: [[2403.04858]] Evaluating Biases in Context-Dependent Health Questions(https://arxiv.org/abs/2403.04858)
• Keywords: language model, chat
• Abstract: Chat-based large language models have the opportunity to empower individuals lacking high-quality healthcare access to receive personalized information across a variety of topics. However, users may ask underspecified questions that require additional context for a model to correctly answer. We study how large language model biases are exhibited through these contextual questions in the healthcare domain. To accomplish this, we curate a dataset of sexual and reproductive healthcare questions that are dependent on age, sex, and location attributes. We compare models' outputs with and without demographic context to determine group alignment among our contextual questions. Our experiments reveal biases in each of these attributes, where young adult female users are favored.
• 摘要：基于聊天的大语言模型有机会使缺乏高质量医疗保健访问权限的个人能够接收各种主题的个性化信息。但是，用户可能会提出未指定的问题，这些问题需要额外的上下文才能使模型正确回答。我们研究医疗保健领域的这些上下文问题如何表现出巨大的语言模型偏差。为了实现这一目标，我们整理了一个取决于年龄、性别和位置属性的性和生殖保健问题的数据集。我们比较有和没有人口统计背景的模型输出，以确定我们的背景问题之间的群体一致性。我们的实验揭示了这些属性中的每一个都存在偏差，其中年轻的成年女性用户受到青睐。

Title: Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text

• Authors: Frances A. Laureano De Leon, Harish Tayyar Madabushi, Mark Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04872
• Pdf URL: https://arxiv.org/pdf/2403.04872
• Copy Paste: [[2403.04872]] Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text(https://arxiv.org/abs/2403.04872)
• Keywords: language model
• Abstract: Code-switching is a prevalent linguistic phenomenon in which multilingual individuals seamlessly alternate between languages. Despite its widespread use online and recent research trends in this area, research in code-switching presents unique challenges, primarily stemming from the scarcity of labelled data and available resources. In this study we investigate how pre-trained Language Models handle code-switched text in three dimensions: a) the ability of PLMs to detect code-switched text, b) variations in the structural information that PLMs utilise to capture code-switched text, and c) the consistency of semantic information representation in code-switched text. To conduct a systematic and controlled evaluation of the language models in question, we create a novel dataset of well-formed naturalistic code-switched text along with parallel translations into the source languages. Our findings reveal that pre-trained language models are effective in generalising to code-switched text, shedding light on the abilities of these models to generalise representations to CS corpora. We release all our code and data including the novel corpus at https://github.com/francesita/code-mixed-probes.
• 摘要：语码转换是一种普遍的语言现象，多语言个体在语言之间无缝切换。尽管语码转换在网上得到了广泛的应用，而且该领域的最新研究趋势也很明显，但语码转换的研究仍然面临着独特的挑战，这主要源于标记数据和可用资源的稀缺。在本研究中，我们研究预训练的语言模型如何在三个维度上处理代码转换文本：a) PLM 检测代码转换文本的能力，b) PLM 用于捕获代码转换文本的结构信息的变化， c) 语码转换文本中语义信息表示的一致性。为了对相关语言模型进行系统和受控的评估，我们创建了一个新颖的数据集，其中包含格式良好的自然语码转换文本以及源语言的并行翻译。我们的研究结果表明，预先训练的语言模型可以有效地推广到代码转换文本，揭示了这些模型将表示推广到 CS 语料库的能力。我们在 https://github.com/francesita/code-mixed-probes 上发布了所有代码和数据，包括小说语料库。

Title: Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering

• Authors: Ojas Gramopadhye, Saeel Sandeep Nachane, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, Sachindra Joshi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04890
• Pdf URL: https://arxiv.org/pdf/2403.04890
• Copy Paste: [[2403.04890]] Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering(https://arxiv.org/abs/2403.04890)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language models (LLMs) have demonstrated significant potential in transforming healthcare by automating tasks such as clinical documentation, information retrieval, and decision support. In this aspect, carefully engineered prompts have emerged as a powerful tool for using LLMs for medical scenarios, e.g., patient clinical scenarios. In this paper, we propose a modified version of the MedQA-USMLE dataset, which is subjective, to mimic real-life clinical scenarios. We explore the Chain of Thought (CoT) reasoning based on subjective response generation for the modified MedQA-USMLE dataset with appropriate LM-driven forward reasoning for correct responses to the medical questions. Keeping in mind the importance of response verification in the medical setting, we utilize a reward training mechanism whereby the language model also provides an appropriate verified response for a particular response to a clinical question. In this regard, we also include human-in-the-loop for different evaluation aspects. We develop better in-contrast learning strategies by modifying the 5-shot-codex-CoT-prompt from arXiv:2207.08143 for the subjective MedQA dataset and developing our incremental-reasoning prompt. Our evaluations show that the incremental reasoning prompt performs better than the modified codex prompt in certain scenarios. We also show that greedy decoding with the incremental reasoning method performs better than other strategies, such as prompt chaining and eliminative reasoning.
• 摘要：大型语言模型 (LLM) 通过自动化临床文档、信息检索和决策支持等任务，在医疗保健转型方面展现出了巨大的潜力。在这方面，精心设计的提示已成为将法学硕士用于医疗场景（例如患者临床场景）的强大工具。在本文中，我们提出了 MedQA-USMLE 数据集的修改版本，该数据集是主观的，以模仿现实生活中的临床场景。我们探索基于修改后的 MedQA-USMLE 数据集的主观响应生成的思想链 (CoT) 推理，并使用适当的 LM 驱动的前向推理来正确回答医学问题。考虑到响应验证在医疗环境中的重要性，我们利用奖励训练机制，通过该机制，语言模型还为临床问题的特定响应提供适当的验证响应。在这方面，我们还纳入了人机参与的不同评估方面。我们通过针对主观 MedQA 数据集修改 arXiv:2207.08143 中的 5-shot-codex-CoT-prompt 并开发我们的增量推理提示来开发更好的对比学习策略。我们的评估表明，在某些情况下，增量推理提示比修改后的 Codex 提示表现得更好。我们还表明，使用增量推理方法的贪婪解码比其他策略（例如提示链接和消除推理）表现更好。

Title: ConstitutionalExperts: Training a Mixture of Principle-based Prompts

• Authors: Savvas Petridis, Ben Wedin, Ann Yuan, James Wexler, Nithum Thain
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04894
• Pdf URL: https://arxiv.org/pdf/2403.04894
• Copy Paste: [[2403.04894]] ConstitutionalExperts: Training a Mixture of Principle-based Prompts(https://arxiv.org/abs/2403.04894)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are highly capable at a variety of tasks given the right prompt, but writing one is still a difficult and tedious process. In this work, we introduce ConstitutionalExperts, a method for learning a prompt consisting of constitutional principles (i.e. rules), given a training dataset. Unlike prior methods that optimize the prompt as a single entity, our method incrementally improves the prompt by surgically editing individual principles. We also show that we can improve overall performance by learning unique prompts for different semantic regions of the training data and using a mixture-of-experts (MoE) architecture to route inputs at inference time. We compare our method to other state of the art prompt-optimization techniques across six benchmark datasets. We also investigate whether MoE improves these other techniques. Our results suggest that ConstitutionalExperts outperforms other prompt optimization techniques by 10.9% (F1) and that mixture-of-experts improves all techniques, suggesting its broad applicability.
• 摘要：在适当的提示下，大型语言模型（LLM）非常有能力完成各种任务，但编写一个模型仍然是一个困难而乏味的过程。在这项工作中，我们引入了ConstitutionalExperts，这是一种在给定训练数据集的情况下学习由宪法原则（即规则）组成的提示的方法。与之前将提示优化为单个实体的方法不同，我们的方法通过外科手术式编辑各个原则来逐步改进提示。我们还表明，我们可以通过学习训练数据不同语义区域的独特提示并使用专家混合 (MoE) 架构在推理时路由输入来提高整体性能。我们将我们的方法与六个基准数据集上的其他最先进的提示优化技术进行比较。我们还调查了 MoE 是否改进了这些其他技术。我们的结果表明，ConstitutionalExperts 的性能优于其他即时优化技术 10.9% (F1)，并且专家混合改进了所有技术，表明其广泛的适用性。

Title: Electrocardiogram Instruction Tuning for Report Generation

• Authors: Zhongwei Wan, Che Liu, Xin Wang, Chaofan Tao, Hui Shen, Zhenwu Peng, Jie Fu, Rossella Arcucci, Huaxiu Yao, Mi Zhang
• Subjects: cs.CL, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.04945
• Pdf URL: https://arxiv.org/pdf/2403.04945
• Copy Paste: [[2403.04945]] Electrocardiogram Instruction Tuning for Report Generation(https://arxiv.org/abs/2403.04945)
• Keywords: llm
• Abstract: Electrocardiogram (ECG) serves as the primary non-invasive diagnostic tool for cardiac conditions monitoring, are crucial in assisting clinicians. Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation, which is not only time-consuming but also requires clinical expertise. To automate ECG report generation and ensure its versatility, we propose the Multimodal ECG Instruction Tuning (MEIT) framework, the \textit{first} attempt to tackle ECG report generation with LLMs and multimodal instructions. To facilitate future research, we establish a benchmark to evaluate MEIT with various LLMs backbones across two large-scale ECG datasets. Our approach uniquely aligns the representations of the ECG signal and the report, and we conduct extensive experiments to benchmark MEIT with nine open source LLMs, using more than 800,000 ECG reports. MEIT's results underscore the superior performance of instruction-tuned LLMs, showcasing their proficiency in quality report generation, zero-shot capabilities, and resilience to signal perturbation. These findings emphasize the efficacy of our MEIT framework and its potential for real-world clinical application.
• 摘要：心电图 (ECG) 是监测心脏状况的主要无创诊断工具，对于协助临床医生至关重要。最近的研究集中在使用心电图数据对心脏疾病进行分类，但忽略了心电图报告的生成，这不仅耗时，而且需要临床专业知识。为了自动化心电图报告生成并确保其多功能性，我们提出了多模态心电图指令调整（MEIT）框架，\textit{first}尝试使用法学硕士和多模态指令来解决心电图报告生成问题。为了促进未来的研究，我们建立了一个基准来评估跨两个大型心电图数据集的各种法学硕士骨干的 MEIT。我们的方法以独特的方式将心电图信号和报告的表示方式结合起来，并且我们使用超过 800,000 份心电图报告，通过九个开源法学硕士进行了广泛的实验，以对 MEIT 进行基准测试。 MEIT 的结果强调了指令调整的法学硕士的卓越性能，展示了他们在质量报告生成、零样本能力和信号扰动恢复能力方面的熟练程度。这些发现强调了我们的 MEIT 框架的有效性及其在现实世界临床应用中的潜力。

Title: An In-depth Evaluation of GPT-4 in Sentence Simplification with Error-based Human Assessment

• Authors: Xuanxin Wu, Yuki Arase
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04963
• Pdf URL: https://arxiv.org/pdf/2403.04963
• Copy Paste: [[2403.04963]] An In-depth Evaluation of GPT-4 in Sentence Simplification with Error-based Human Assessment(https://arxiv.org/abs/2403.04963)
• Keywords: language model, gpt, llm
• Abstract: Sentence simplification, which rewrites a sentence to be easier to read and understand, is a promising technique to help people with various reading difficulties. With the rise of advanced large language models (LLMs), evaluating their performance in sentence simplification has become imperative. Recent studies have used both automatic metrics and human evaluations to assess the simplification abilities of LLMs. However, the suitability of existing evaluation methodologies for LLMs remains in question. First, the suitability of current automatic metrics on LLMs' simplification evaluation is still uncertain. Second, current human evaluation approaches in sentence simplification often fall into two extremes: they are either too superficial, failing to offer a clear understanding of the models' performance, or overly detailed, making the annotation process complex and prone to inconsistency, which in turn affects the evaluation's reliability. To address these problems, this study provides in-depth insights into LLMs' performance while ensuring the reliability of the evaluation. We design an error-based human annotation framework to assess the GPT-4's simplification capabilities. Results show that GPT-4 generally generates fewer erroneous simplification outputs compared to the current state-of-the-art. However, LLMs have their limitations, as seen in GPT-4's struggles with lexical paraphrasing. Furthermore, we conduct meta-evaluations on widely used automatic metrics using our human annotations. We find that while these metrics are effective for significant quality differences, they lack sufficient sensitivity to assess the overall high-quality simplification by GPT-4.
• 摘要：句子简化，即重写句子，使其更易于阅读和理解，是一种很有前途的技术，可以帮助有各种阅读困难的人。随着高级大语言模型（LLM）的兴起，评估其在句子简化方面的表现已势在必行。最近的研究使用自动指标和人工评估来评估法学硕士的简化能力。然而，现有的法学硕士评估方法是否适用仍然存在疑问。首先，当前自动指标对法学硕士简化评估的适用性仍不确定。其次，当前句子简化中的人类评估方法往往陷入两个极端：要么过于肤浅，无法清晰地理解模型的性能，要么过于详细，使得标注过程复杂且容易出现不一致，进而导致影响评估的可靠性。为了解决这些问题，本研究深入了解法学硕士的表现，同时确保评估的可靠性。我们设计了一个基于错误的人工注释框架来评估 GPT-4 的简化能力。结果表明，与当前最先进的技术相比，GPT-4 通常生成更少的错误简化输出。然而，LLM 也有其局限性，正如 GPT-4 在词汇释义方面的挣扎所示。此外，我们使用人工注释对广泛使用的自动指标进行元评估。我们发现，虽然这些指标对于显着的质量差异有效，但它们缺乏足够的敏感性来评估 GPT-4 的整体高质量简化。

Title: DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation

• Authors: Jiapeng Wang, Chengyu Wang, Tingfeng Cao, Jun Huang, Lianwen Jin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04997
• Pdf URL: https://arxiv.org/pdf/2403.04997
• Copy Paste: [[2403.04997]] DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation(https://arxiv.org/abs/2403.04997)
• Keywords: language model, llm, prompt, chat
• Abstract: We present DiffChat, a novel method to align Large Language Models (LLMs) to "chat" with prompt-as-input Text-to-Image Synthesis (TIS) models (e.g., Stable Diffusion) for interactive image creation. Given a raw prompt/image and a user-specified instruction, DiffChat can effectively make appropriate modifications and generate the target prompt, which can be leveraged to create the target image of high quality. To achieve this, we first collect an instruction-following prompt engineering dataset named InstructPE for the supervised training of DiffChat. Next, we propose a reinforcement learning framework with the feedback of three core criteria for image creation, i.e., aesthetics, user preference, and content integrity. It involves an action-space dynamic modification technique to obtain more relevant positive samples and harder negative samples during the off-policy sampling. Content integrity is also introduced into the value estimation function for further improvement of produced images. Our method can exhibit superior performance than baseline models and strong competitors based on both automatic and human evaluations, which fully demonstrates its effectiveness.
• 摘要：我们提出了 DiffChat，这是一种将大型语言模型 (LLM) 与提示输入文本到图像合成 (TIS) 模型（例如稳定扩散）进行“聊天”的新颖方法，用于交互式图像创建。给定原始提示/图像和用户指定的指令，DiffChat 可以有效地进行适当的修改并生成目标提示，从而可以用来创建高质量的目标图像。为了实现这一目标，我们首先收集一个名为 InstructPE 的指令跟踪提示工程数据集，用于 DiffChat 的监督训练。接下来，我们提出了一个强化学习框架，其中包含图像创建的三个核心标准的反馈，即美观、用户偏好和内容完整性。它涉及动作空间动态修改技术，以在离策略采样期间获得更多相关的正样本和更难的负样本。内容完整性也被引入价值评估功能中，以进一步改进生成的图像。基于自动和人工评估，我们的方法可以表现出比基线模型和强大竞争对手更优越的性能，这充分证明了其有效性。

Title: Can't Remember Details in Long Documents? You Need Some R&R

• Authors: Devanshu Agrawal, Shang Gao, Martin Gajek
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05004
• Pdf URL: https://arxiv.org/pdf/2403.05004
• Copy Paste: [[2403.05004]] Can't Remember Details in Long Documents? You Need Some R&R(https://arxiv.org/abs/2403.05004)
• Keywords: language model, gpt, llm, prompt
• Abstract: Long-context large language models (LLMs) hold promise for tasks such as question-answering (QA) over long documents, but they tend to miss important information in the middle of context documents (arXiv:2307.03172v3). Here, we introduce $\textit{R&R}$ -- a combination of two novel prompt-based methods called $\textit{reprompting}$ and $\textit{in-context retrieval}$ (ICR) -- to alleviate this effect in document-based QA. In reprompting, we repeat the prompt instructions periodically throughout the context document to remind the LLM of its original task. In ICR, rather than instructing the LLM to answer the question directly, we instruct it to retrieve the top $k$ passage numbers most relevant to the given question, which are then used as an abbreviated context in a second QA prompt. We test R&R with GPT-4 Turbo and Claude-2.1 on documents up to 80k tokens in length and observe a 16-point boost in QA accuracy on average. Our further analysis suggests that R&R improves performance on long document-based QA because it reduces the distance between relevant context and the instructions. Finally, we show that compared to short-context chunkwise methods, R&R enables the use of larger chunks that cost fewer LLM calls and output tokens, while minimizing the drop in accuracy.
• 摘要：长上下文大语言模型 (LLM) 有望完成长文档问答 (QA) 等任务，但它们往往会错过上下文文档中间的重要信息 (arXiv:2307.03172v3)。在这里，我们引入 $\textit{R&R}$ - 两种新颖的基于提示的方法的组合，称为 $\textit{reprompting}$ 和 $\textit{in-context检索}$ (ICR) - 来减轻这种影响在基于文档的 QA 中。在重新提示中，我们在整个上下文文档中定期重复提示指令，以提醒法学硕士其原始任务。在 ICR 中，我们不是指示 LLM 直接回答问题，而是指示它检索与给定问题最相关的前 $k$ 段落编号，然后将其用作第二个 QA 提示中的缩写上下文。我们使用 GPT-4 Turbo 和 Claude-2.1 在长达 80k token 的文档上测试 R&R，并观察到 ​​QA 准确度平均提高了 16 个百分点。我们的进一步分析表明，R&R 提高了基于长文档的 QA 的性能，因为它缩短了相关上下文和指令之间的距离。最后，我们表明，与短上下文分块方法相比，R&R 可以使用更大的块，从而减少 LLM 调用和输出令牌的成本，同时最大限度地减少准确性的下降。

Title: Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

• Authors: Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, Maarten Sap
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05020
• Pdf URL: https://arxiv.org/pdf/2403.05020
• Copy Paste: [[2403.05020]] Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs(https://arxiv.org/abs/2403.05020)
• Keywords: language model, llm, agent
• Abstract: Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena with LLM-based agents. However, most work has used an omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that humans have. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that interlocutors simulated omnisciently are much more successful at accomplishing social goals compared to non-omniscient agents, despite the latter being the more realistic setting. Furthermore, we demonstrate that learning from omniscient simulations improves the apparent naturalness of interactions but scarcely enhances goal achievement in cooperative scenarios. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.
• 摘要：大语言模型 (LLM) 的最新进展实现了更丰富的社会模拟，允许使用基于 LLM 的代理来研究各种社会现象。然而，大多数工作都对这些模拟使用了全知的视角（例如，单个法学硕士来生成所有对话者），这从根本上与人类所具有的非全知的、信息不对称的交互相矛盾。为了检查这些差异，我们开发了一个评估框架来模拟在各种环境（全知、非全知）下与法学硕士的社交互动。我们的实验表明，与非全知代理人相比，全知模拟的对话者在实现社会目标方面要成功得多，尽管后者是更现实的环境。此外，我们证明，从全知模拟中学习可以提高交互的明显自然性，但几乎不能提高合作场景中的目标实现。我们的研究结果表明，解决信息不对称问题仍然是法学硕士代理人面临的根本挑战。

Title: Are Human Conversations Special? A Large Language Model Perspective

• Authors: Toshish Jawale, Chaitanya Animesh, Sekhar Vallath, Kartik Talamadupula, Larry Heck
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05045
• Pdf URL: https://arxiv.org/pdf/2403.05045
• Copy Paste: [[2403.05045]] Are Human Conversations Special? A Large Language Model Perspective(https://arxiv.org/abs/2403.05045)
• Keywords: language model, llm
• Abstract: This study analyzes changes in the attention mechanisms of large language models (LLMs) when used to understand natural conversations between humans (human-human). We analyze three use cases of LLMs: interactions over web content, code, and mathematical texts. By analyzing attention distance, dispersion, and interdependency across these domains, we highlight the unique challenges posed by conversational data. Notably, conversations require nuanced handling of long-term contextual relationships and exhibit higher complexity through their attention patterns. Our findings reveal that while language models exhibit domain-specific attention behaviors, there is a significant gap in their ability to specialize in human conversations. Through detailed attention entropy analysis and t-SNE visualizations, we demonstrate the need for models trained with a diverse array of high-quality conversational data to enhance understanding and generation of human-like dialogue. This research highlights the importance of domain specialization in language models and suggests pathways for future advancement in modeling human conversational nuances.
• 摘要：本研究分析了大型语言模型（LLM）在用于理解人类（人与人）之间的自然对话时注意力机制的变化。我们分析了法学硕士的三个用例：网络内容、代码和数学文本的交互。通过分析这些领域的注意力距离、分散性和相互依赖性，我们强调了会话数据带来的独特挑战。值得注意的是，对话需要对长期上下文关系进行细致入微的处理，并通过其注意力模式表现出更高的复杂性。我们的研究结果表明，虽然语言模型表现出特定领域的注意力行为，但它们专门研究人类对话的能力存在显着差距。通过详细的注意力熵分析和 t-SNE 可视化，我们证明了需要使用各种高质量对话数据来训练模型，以增强理解和生成类人对话。这项研究强调了语言模型领域专业化的重要性，并为人类对话细微差别建模的未来发展提出了途径。

Title: Can we obtain significant success in RST discourse parsing by using Large Language Models?

• Authors: Aru Maekawa, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05065
• Pdf URL: https://arxiv.org/pdf/2403.05065
• Copy Paste: [[2403.05065]] Can we obtain significant success in RST discourse parsing by using Large Language Models?(https://arxiv.org/abs/2403.05065)
• Keywords: language model, llm, prompt
• Abstract: Recently, decoder-only pre-trained large language models (LLMs), with several tens of billion parameters, have significantly impacted a wide range of natural language processing (NLP) tasks. While encoder-only or encoder-decoder pre-trained language models have already proved to be effective in discourse parsing, the extent to which LLMs can perform this task remains an open research question. Therefore, this paper explores how beneficial such LLMs are for Rhetorical Structure Theory (RST) discourse parsing. Here, the parsing process for both fundamental top-down and bottom-up strategies is converted into prompts, which LLMs can work with. We employ Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned. Experimental results on three benchmark datasets, RST-DT, Instr-DT, and the GUM corpus, demonstrate that Llama 2 with 70 billion parameters in the bottom-up strategy obtained state-of-the-art (SOTA) results with significant differences. Furthermore, our parsers demonstrated generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.
• 摘要：最近，仅解码器预训练的大型语言模型（LLM）具有数百亿个参数，对广泛的自然语言处理（NLP）任务产生了重大影响。虽然纯编码器或编码器-解码器预训练语言模型已被证明在语篇解析中有效，但法学硕士可以在多大程度上执行此任务仍然是一个悬而未决的研究问题。因此，本文探讨了此类法学硕士对于修辞结构理论（RST）语篇解析的益处。在这里，基本自上而下和自下而上策略的解析过程都被转换为法学硕士可以使用的提示。我们使用 Llama 2 并使用 QLoRA 对其进行微调，其中可调整的参数较少。在 RST-DT、Instr-DT 和 GUM 语料库三个基准数据集上的实验结果表明，采用自下而上策略的 700 亿个参数的 Llama 2 获得了具有显着差异的最先进（SOTA）结果。此外，我们的解析器在 RST-DT 上进行评估时表现出了普遍性，表明尽管使用 GUM 语料库进行了训练，但它仍获得了与使用 RST-DT 训练的现有解析器相似的性能。

Title: Rule-driven News Captioning

• Authors: Ning Xu, Tingting Zhang, Hongshuo Tian, Yongdong Zhang, An-An Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05101
• Pdf URL: https://arxiv.org/pdf/2403.05101
• Copy Paste: [[2403.05101]] Rule-driven News Captioning(https://arxiv.org/abs/2403.05101)
• Keywords: agent
• Abstract: News captioning task aims to generate sentences by describing named entities or concrete events for an image with its news article. Existing methods have achieved remarkable results by relying on the large-scale pre-trained models, which primarily focus on the correlations between the input news content and the output predictions. However, the news captioning requires adhering to some fundamental rules of news reporting, such as accurately describing the individuals and actions associated with the event. In this paper, we propose the rule-driven news captioning method, which can generate image descriptions following designated rule signal. Specifically, we first design the news-aware semantic rule for the descriptions. This rule incorporates the primary action depicted in the image (e.g., "performing") and the roles played by named entities involved in the action (e.g., "Agent" and "Place"). Second, we inject this semantic rule into the large-scale pre-trained model, BART, with the prefix-tuning strategy, where multiple encoder layers are embedded with news-aware semantic rule. Finally, we can effectively guide BART to generate news sentences that comply with the designated rule. Extensive experiments on two widely used datasets (i.e., GoodNews and NYTimes800k) demonstrate the effectiveness of our method.
• 摘要：新闻字幕任务旨在通过描述带有新闻文章的图像的命名实体或具体事件来生成句子。现有方法依靠大规模预训练模型取得了显着的效果，这些模型主要关注输入新闻内容和输出预测之间的相关性。然而，新闻标题需要遵守新闻报道的一些基本规则，例如准确描述与事件相关的个人和行为。在本文中，我们提出了规则驱动的新闻字幕方法，该方法可以按照指定的规则信号生成图像描述。具体来说，我们首先为描述设计新闻感知语义规则。该规则包含图像中描绘的主要动作（例如“表演”）以及参与该动作的命名实体所扮演的角色（例如“代理”和“地点”）。其次，我们通过前缀调整策略将此语义规则注入到大规模预训练模型 BART 中，其中多个编码器层嵌入了新闻感知语义规则。最后，我们可以有效引导BART生成符合指定规则的新闻句子。对两个广泛使用的数据集（即 GoodNews 和 NYTimes800k）的大量实验证明了我们方法的有效性。

Title: ChatUIE: Exploring Chat-based Unified Information Extraction using Large Language Models

• Authors: Jun Xu, Mengshu Sun, Zhiqiang Zhang, Jun Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05132
• Pdf URL: https://arxiv.org/pdf/2403.05132
• Copy Paste: [[2403.05132]] ChatUIE: Exploring Chat-based Unified Information Extraction using Large Language Models(https://arxiv.org/abs/2403.05132)
• Keywords: language model, prompt, chat
• Abstract: Recent advancements in large language models have shown impressive performance in general chat. However, their domain-specific capabilities, particularly in information extraction, have certain limitations. Extracting structured information from natural language that deviates from known schemas or instructions has proven challenging for previous prompt-based methods. This motivated us to explore domain-specific modeling in chat-based language models as a solution for extracting structured information from natural language. In this paper, we present ChatUIE, an innovative unified information extraction framework built upon ChatGLM. Simultaneously, reinforcement learning is employed to improve and align various tasks that involve confusing and limited samples. Furthermore, we integrate generation constraints to address the issue of generating elements that are not present in the input. Our experimental results demonstrate that ChatUIE can significantly improve the performance of information extraction with a slight decrease in chatting ability.
• 摘要：大型语言模型的最新进展在一般聊天中表现出了令人印象深刻的性能。然而，它们的特定领域能力，特别是在信息提取方面，具有一定的局限性。事实证明，从偏离已知模式或指令的自然语言中提取结构化信息对于以前基于提示的方法来说具有挑战性。这促使我们探索基于聊天的语言模型中的特定领域建模，作为从自然语言中提取结构化信息的解决方案。在本文中，我们提出了 ChatUIE，一种基于 ChatGLM 构建的创新统一信息提取框架。同时，强化学习用于改进和调整涉及混乱和有限样本的各种任务。此外，我们集成生成约束来解决生成输入中不存在的元素的问题。我们的实验结果表明，ChatUIE 可以显着提高信息提取的性能，同时聊天能力略有下降。

Title: Towards a Psychology of Machines: Large Language Models Predict Human Memory

• Authors: Markus Huff, Elanur Ulakçı
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05152
• Pdf URL: https://arxiv.org/pdf/2403.05152
• Copy Paste: [[2403.05152]] Towards a Psychology of Machines: Large Language Models Predict Human Memory(https://arxiv.org/abs/2403.05152)
• Keywords: language model, gpt, llm, chat
• Abstract: Large language models (LLMs) are demonstrating remarkable capabilities across various tasks despite lacking a foundation in human cognition. This raises the question: can these models, beyond simply mimicking human language patterns, offer insights into the mechanisms underlying human cognition? This study explores the ability of ChatGPT to predict human performance in a language-based memory task. Building upon theories of text comprehension, we hypothesize that recognizing ambiguous sentences (e.g., "Because Bill drinks wine is never kept in the house") is facilitated by preceding them with contextually relevant information. Participants, both human and ChatGPT, were presented with pairs of sentences. The second sentence was always a garden-path sentence designed to be inherently ambiguous, while the first sentence either provided a fitting (e.g., "Bill has chronic alcoholism") or an unfitting context (e.g., "Bill likes to play golf"). We measured both human's and ChatGPT's ratings of sentence relatedness, ChatGPT's memorability ratings for the garden-path sentences, and humans' spontaneous memory for the garden-path sentences. The results revealed a striking alignment between ChatGPT's assessments and human performance. Sentences deemed more related and assessed as being more memorable by ChatGPT were indeed better remembered by humans, even though ChatGPT's internal mechanisms likely differ significantly from human cognition. This finding, which was confirmed with a robustness check employing synonyms, underscores the potential of generative AI models to predict human performance accurately. We discuss the broader implications of these findings for leveraging LLMs in the development of psychological theories and for gaining a deeper understanding of human cognition.
• 摘要：尽管缺乏人类认知的基础，大型语言模型（LLM）在各种任务中展示了卓越的能力。这就提出了一个问题：这些模型除了简单地模仿人类语言模式之外，还能提供对人类认知机制的见解吗？本研究探讨了 ChatGPT 在基于语言的记忆任务中预测人类表现的能力。基于文本理解理论，我们假设通过在句子前面添加上下文相关信息可以促进识别歧义句子（例如，“因为比尔喝葡萄酒从来没有留在房子里”）。参与者（包括人类和 ChatGPT）都会看到成对的句子。第二句话始终是一个花园小径句子，设计本质上是模棱两可的，而第一句话要么提供了合适的上下文（例如，“比尔有慢性酒精中毒”），要么提供了不合适的上下文（例如，“比尔喜欢打高尔夫球”）。我们测量了人类和 ChatGPT 对句子相关性的评分、ChatGPT 对花园小径句子的记忆性评分以及人类对花园小径句子的自发​​记忆。结果显示 ChatGPT 的评估与人类表现之间存在惊人的一致性。尽管 ChatGPT 的内部机制可能与人类认知存在显着差异，但 ChatGPT 认为更相关且更容易记住的句子确实更容易被人类记住。这一发现通过使用同义词的稳健性检查得到了证实，强调了生成式人工智能模型准确预测人类表现的潜力。我们讨论了这些发现对于利用法学硕士发展心理学理论和加深对人类认知的理解的更广泛的影响。

Title: Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge

• Authors: Xin Zhao, Naoki Yoshinaga, Daisuke Oba
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05189
• Pdf URL: https://arxiv.org/pdf/2403.05189
• Copy Paste: [[2403.05189]] Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge(https://arxiv.org/abs/2403.05189)
• Keywords: language model
• Abstract: Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire and represent factual knowledge. Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT). We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways in which ML-LMs acquire specific facts. We finally identified three patterns of acquiring and representing facts in ML-LMs: language-independent, cross-lingual shared and transferred, and devised methods for differentiating them. Our findings highlight the challenge of maintaining consistent factual knowledge across languages, underscoring the need for better fact representation learning in ML-LMs.
• 摘要：获取低资源语言的语言模型（LM）的事实知识提出了严峻的挑战，因此诉诸多语言 LM（ML-LM）中的跨语言迁移。在本研究中，我们询问 ML-LM 如何获取和表示事实知识。使用多语言事实知识探测数据集 mLAMA，我们首先对 ML-LM（特别是多语言 BERT）进行了神经元研究。然后，我们将事实的根源追溯到知识源（维基百科），以确定 ML-LM 获取特定事实的方式。我们最终确定了 ML-LM 中获取和表示事实的三种模式：独立于语言、跨语言共享和转移，并设计了区分它们的方法。我们的研究结果强调了保持跨语言一致的事实知识的挑战，强调了在 ML-LM 中更好的事实表示学习的必要性。

Title: SocialPET: Socially Informed Pattern Exploiting Training for Few-Shot Stance Detection in Social Media

• Authors: Parisa Jamadi Khiabani, Arkaitz Zubiaga
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.05216
• Pdf URL: https://arxiv.org/pdf/2403.05216
• Copy Paste: [[2403.05216]] SocialPET: Socially Informed Pattern Exploiting Training for Few-Shot Stance Detection in Social Media(https://arxiv.org/abs/2403.05216)
• Keywords: language model
• Abstract: Stance detection, as the task of determining the viewpoint of a social media post towards a target as 'favor' or 'against', has been understudied in the challenging yet realistic scenario where there is limited labeled data for a certain target. Our work advances research in few-shot stance detection by introducing SocialPET, a socially informed approach to leveraging language models for the task. Our proposed approach builds on the Pattern Exploiting Training (PET) technique, which addresses classification tasks as cloze questions through the use of language models. To enhance the approach with social awareness, we exploit the social network structure surrounding social media posts. We prove the effectiveness of SocialPET on two stance datasets, Multi-target and P-Stance, outperforming competitive stance detection models as well as the base model, PET, where the labeled instances for the target under study is as few as 100. When we delve into the results, we observe that SocialPET is comparatively strong in identifying instances of the `against' class, where baseline models underperform.
• 摘要：立场检测是确定社交媒体帖子对目标的观点是“支持”还是“反对”的任务，在具有挑战性但现实的场景中尚未得到充分研究，其中特定目标的标记数据有限。我们的工作通过引入 SocialPET 来推进少镜头姿势检测的研究，SocialPET 是一种利用语言模型完成任务的社交知情方法。我们提出的方法建立在模式利用训练（PET）技术的基础上，该技术通过使用语言模型将分类任务作为完形填空问题来解决。为了增强这种方法的社会意识，我们利用了围绕社交媒体帖子的社交网络结构。我们证明了 SocialPET 在两个姿态数据集（多目标和 P-姿态）上的有效性，优于竞争性姿态检测模型以及基本模型 PET，其中所研究目标的标记实例只有 100 个。深入研究结果，我们发现 SocialPET 在识别“反对”类实例方面相对较强，而基线模型在这些实例中表现不佳。

Title: Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering

• Authors: Hongda Sun, Yuxuan Liu, Chengwei Wu, Haiyu Yan, Cheng Tai, Xin Gao, Shuo Shang, Rui Yan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.05217
• Pdf URL: https://arxiv.org/pdf/2403.05217
• Copy Paste: [[2403.05217]] Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering(https://arxiv.org/abs/2403.05217)
• Keywords: language model, llm, prompt
• Abstract: Open-domain question answering (ODQA) has emerged as a pivotal research spotlight in information systems. Existing methods follow two main paradigms to collect evidence: (1) The \textit{retrieve-then-read} paradigm retrieves pertinent documents from an external corpus; and (2) the \textit{generate-then-read} paradigm employs large language models (LLMs) to generate relevant documents. However, neither can fully address multifaceted requirements for evidence. To this end, we propose LLMQA, a generalized framework that formulates the ODQA process into three basic steps: query expansion, document selection, and answer generation, combining the superiority of both retrieval-based and generation-based evidence. Since LLMs exhibit their excellent capabilities to accomplish various tasks, we instruct LLMs to play multiple roles as generators, rerankers, and evaluators within our framework, integrating them to collaborate in the ODQA process. Furthermore, we introduce a novel prompt optimization algorithm to refine role-playing prompts and steer LLMs to produce higher-quality evidence and answers. Extensive experimental results on widely used benchmarks (NQ, WebQ, and TriviaQA) demonstrate that LLMQA achieves the best performance in terms of both answer accuracy and evidence quality, showcasing its potential for advancing ODQA research and applications.
• 摘要：开放域问答（ODQA）已成为信息系统中的一个关键研究热点。现有的方法遵循两个主要范式来收集证据：（1）\textit{retrieve-then-read}范式从外部语料库中检索相关文档； (2) \textit{generate-then-read} 范式采用大型语言模型 (LLM) 来生成相关文档。然而，两者都不能完全满足证据的多方面要求。为此，我们提出了 LLMQA，这是一个通用框架，它将 ODQA 过程制定为三个基本步骤：查询扩展、文档选择和答案生成，结合了基于检索和基于生成的证据的优势。由于法学硕士展示了完成各种任务的出色能力，因此我们指示法学硕士在我们的框架内扮演生成者、重新排序者和评估者等多重角色，将他们整合到 ODQA 流程中进行协作。此外，我们引入了一种新颖的提示优化算法来完善角色扮演提示并引导法学硕士产生更高质量的证据和答案。在广泛使用的基准（NQ、WebQ 和 TriviaQA）上进行的大量实验结果表明，LLMMA 在答案准确性和证据质量方面均实现了最佳性能，展示了其推进 ODQA 研究和应用的潜力。

Title: Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity

• Authors: Sho Hoshino, Akihiko Kato, Soichiro Murakami, Peinan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05257
• Pdf URL: https://arxiv.org/pdf/2403.05257
• Copy Paste: [[2403.05257]] Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity(https://arxiv.org/abs/2403.05257)
• Keywords: language model
• Abstract: Learning better sentence embeddings leads to improved performance for natural language understanding tasks including semantic textual similarity (STS) and natural language inference (NLI). As prior studies leverage large-scale labeled NLI datasets for fine-tuning masked language models to yield sentence embeddings, task performance for languages other than English is often left behind. In this study, we directly compared two data augmentation techniques as potential solutions for monolingual STS: (a) cross-lingual transfer that exploits English resources alone as training data to yield non-English sentence embeddings as zero-shot inference, and (b) machine translation that coverts English data into pseudo non-English training data in advance. In our experiments on monolingual STS in Japanese and Korean, we find that the two data techniques yield performance on par. Rather, we find a superiority of the Wikipedia domain over the NLI domain for these languages, in contrast to prior studies that focused on NLI as training data. Combining our findings, we demonstrate that the cross-lingual transfer of Wikipedia data exhibits improved performance, and that native Wikipedia data can further improve performance for monolingual STS.
• 摘要：学习更好的句子嵌入可以提高自然语言理解任务的性能，包括语义文本相似性（STS）和自然语言推理（NLI）。由于先前的研究利用大规模标记的 NLI 数据集来微调掩码语言模型以生成句子嵌入，因此英语以外的语言的任务性能通常会落后。在本研究中，我们直接比较了两种数据增强技术作为单语言 STS 的潜在解决方案：(a) 跨语言迁移，仅利用英语资源作为训练数据，以产生非英语句子嵌入作为零样本推理，以及 (b)机器翻译，提前将英语数据转换为伪非英语训练数据。在我们对日语和韩语单语 STS 进行的实验中，我们发现这两种数据技术的性能相当。相反，我们发现对于这些语言来说，维基百科域优于 NLI 域，这与之前专注于 NLI 作为训练数据的研究形成鲜明对比。结合我们的研究结果，我们证明维基百科数据的跨语言传输表现出改进的性能，并且原生维基百科数据可以进一步提高单语言 STS 的性能。

Title: ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

• Authors: Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, Steven Euijong Whang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05266
• Pdf URL: https://arxiv.org/pdf/2403.05266
• Copy Paste: [[2403.05266]] ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models(https://arxiv.org/abs/2403.05266)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: Large language models (LLMs) have achieved unprecedented performance in various applications, yet their evaluation remains a critical issue. Existing hallucination benchmarks are either static or lack adjustable complexity for thorough analysis. We contend that utilizing existing relational databases is a promising approach for constructing benchmarks due to their accurate knowledge description via functional dependencies. We propose ERBench to automatically convert any relational database into a benchmark based on the entity-relationship (ER) model. Our key idea is to construct questions using the database schema, records, and functional dependencies such that they can be automatically verified. In addition, we use foreign key constraints to join relations and construct multihop questions, which can be arbitrarily complex and used to debug the intermediate answers of LLMs. Finally, ERBench supports continuous evaluation, multimodal questions, and various prompt engineering techniques. In our experiments, we construct an LLM benchmark using databases of multiple domains and make an extensive comparison of contemporary LLMs. We observe that better LLMs like GPT-4 can handle a larger variety of question types, but are by no means perfect. Also, correct answers do not necessarily imply correct rationales, which is an important evaluation that ERBench does better than other benchmarks for various question types. Code is available at https: //github.com/DILAB-KAIST/ERBench.
• 摘要：大型语言模型（LLM）在各种应用中取得了前所未有的性能，但它们的评估仍然是一个关键问题。现有的幻觉基准要么是静态的，要么缺乏可调节的复杂性以进行彻底分析。我们认为，利用现有的关系数据库是构建基准的一种有前途的方法，因为它们通过函数依赖关系进行了准确的知识描述。我们建议 ERBench 自动将任何关系数据库转换为基于实体关系（ER）模型的基准。我们的关键思想是使用数据库模式、记录和功能依赖关系构建问题，以便可以自动验证它们。此外，我们使用外键约束来连接关系并构造多跳问题，这些问题可以任意复杂，并用于调试法学硕士的中间答案。最后，ERBench 支持连续评估、多模式问题和各种即时工程技术。在我们的实验中，我们使用多个领域的数据库构建了法学硕士基准，并对当代法学硕士进行了广泛的比较。我们观察到，像 GPT-4 这样更好的法学硕士可以处理更多种类的问题类型，但绝不是完美的。此外，正确的答案并不一定意味着正确的理由，这是 ERBench 在各种问题类型上比其他基准测试做得更好的一个重要评估。代码可在 https://github.com/DILAB-KAIST/ERBench 获取。

Title: Deep Prompt Multi-task Network for Abuse Language Detection

• Authors: Jian Zhu, Yuping Ruan, Jingfei Chang, Cheng Luo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05268
• Pdf URL: https://arxiv.org/pdf/2403.05268
• Copy Paste: [[2403.05268]] Deep Prompt Multi-task Network for Abuse Language Detection(https://arxiv.org/abs/2403.05268)
• Keywords: language model, prompt
• Abstract: The detection of abusive language remains a long-standing challenge with the extensive use of social networks. The detection task of abusive language suffers from limited accuracy. We argue that the existing detection methods utilize the fine-tuning technique of the pre-trained language models (PLMs) to handle downstream tasks. Hence, these methods fail to stimulate the general knowledge of the PLMs. To address the problem, we propose a novel Deep Prompt Multi-task Network (DPMN) for abuse language detection. Specifically, DPMN first attempts to design two forms of deep prompt tuning and light prompt tuning for the PLMs. The effects of different prompt lengths, tuning strategies, and prompt initialization methods on detecting abusive language are studied. In addition, we propose a Task Head based on Bi-LSTM and FFN, which can be used as a short text classifier. Eventually, DPMN utilizes multi-task learning to improve detection metrics further. The multi-task network has the function of transferring effective knowledge. The proposed DPMN is evaluated against eight typical methods on three public datasets: OLID, SOLID, and AbuseAnalyzer. The experimental results show that our DPMN outperforms the state-of-the-art methods.
• 摘要：随着社交网络的广泛使用，辱骂性语言的检测仍然是一个长期存在的挑战。辱骂性语言的检测任务的准确性有限。我们认为现有的检测方法利用预训练语言模型（PLM）的微调技术来处理下游任务。因此，这些方法无法激发 PLM 的一般知识。为了解决这个问题，我们提出了一种用于滥用语言检测的新型深度提示多任务网络（DPMN）。具体来说，DPMN首先尝试为PLM设计深度提示调优和轻提示调优两种形式。研究了不同提示长度、调整策略和提示初始化方法对检测辱骂性语言的影响。此外，我们提出了一个基于Bi-LSTM和FFN的Task Head，它可以用作短文本分类器。最终，DPMN 利用多任务学习来进一步改进检测指标。多任务网络具有传递有效知识的功能。所提出的 DPMN 在三个公共数据集（OLID、SOLID 和 AbuseAnalyzer）上针对八种典型方法进行了评估。实验结果表明我们的 DPMN 优于最先进的方法。

Title: ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications

• Authors: Sotaro Takeshita, Tommaso Green, Ines Reinig, Kai Eckert, Simone Paolo Ponzetto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05303
• Pdf URL: https://arxiv.org/pdf/2403.05303
• Copy Paste: [[2403.05303]] ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications(https://arxiv.org/abs/2403.05303)
• Keywords: language model, llm
• Abstract: Extensive efforts in the past have been directed toward the development of summarization datasets. However, a predominant number of these resources have been (semi)-automatically generated, typically through web data crawling, resulting in subpar resources for training and evaluating summarization systems, a quality compromise that is arguably due to the substantial costs associated with generating ground-truth summaries, particularly for diverse languages and specialized domains. To address this issue, we present ACLSum, a novel summarization dataset carefully crafted and evaluated by domain experts. In contrast to previous datasets, ACLSum facilitates multi-aspect summarization of scientific papers, covering challenges, approaches, and outcomes in depth. Through extensive experiments, we evaluate the quality of our resource and the performance of models based on pretrained language models and state-of-the-art large language models (LLMs). Additionally, we explore the effectiveness of extractive versus abstractive summarization within the scholarly domain on the basis of automatically discovered aspects. Our results corroborate previous findings in the general domain and indicate the general superiority of end-to-end aspect-based summarization. Our data is released at https://github.com/sobamchan/aclsum.
• 摘要：过去的大量努力都是针对摘要数据集的开发。然而，这些资源中的大部分是（半）自动生成的，通常是通过网络数据爬行生成的，导致用于培训和评估摘要系统的资源低于标准，这是一种质量妥协，可以说是由于与生成地面相关的大量成本造成的。真相摘要，特别是针对不同语言和专业领域。为了解决这个问题，我们提出了 ACLSum，这是一个由领域专家精心制作和评估的新型摘要数据集。与以前的数据集相比，ACLSum 有助于对科学论文进行多方面的总结，深入涵盖挑战、方法和结果。通过广泛的实验，我们评估了我们的资源质量以及基于预训练语言模型和最先进的大型语言模型（LLM）的模型的性能。此外，我们基于自动发现的方面，探讨了学术领域内提取与抽象概括的有效性。我们的结果证实了之前在通用领域的发现，并表明了端到端基于方面的摘要的总体优越性。我们的数据发布于 https://github.com/sobamchan/aclsum。

Title: RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation

• Authors: Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, Yitao Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05313
• Pdf URL: https://arxiv.org/pdf/2403.05313
• Copy Paste: [[2403.05313]] RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation(https://arxiv.org/abs/2403.05313)
• Keywords: language model, gpt, hallucination
• Abstract: We explore how iterative revising a chain of thoughts with the help of information retrieval significantly improves large language models' reasoning and generation ability in long-horizon generation tasks, while hugely mitigating hallucination. In particular, the proposed method -- *retrieval-augmented thoughts* (RAT) -- revises each thought step one by one with retrieved information relevant to the task query, the current and the past thought steps, after the initial zero-shot CoT is generated. Applying RAT to GPT-3.5, GPT-4, and CodeLLaMA-7b substantially improves their performances on various long-horizon generation tasks; on average of relatively increasing rating scores by 13.63% on code generation, 16.96% on mathematical reasoning, 19.2% on creative writing, and 42.78% on embodied task planning. The demo page can be found at https://craftjarvis.github.io/RAT
• 摘要：我们探索在信息检索的帮助下迭代修改思想链如何显着提高大型语言模型在长视野生成任务中的推理和生成能力，同时极大地减轻幻觉。特别是，所提出的方法——*检索增强思想*（RAT）——在初始零样本 CoT 之后，使用与任务查询相关的检索信息、当前和过去的思想步骤逐一修改每个思想步骤被生成。将 RAT 应用于 GPT-3.5、GPT-4 和 CodeLLaMA-7b 显着提高了它们在各种长视野生成任务上的性能；代码生成方面的评分平均相对提高了 13.63%，数学推理方面平均提高了 16.96%，创意写作方面平均提高了 19.2%，具体任务规划方面平均提高了 42.78%。演示页面可以在 https://craftjarvis.github.io/RAT 找到

Title: ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues

• Authors: Yiding Liu, Jingjing Wang, Jiaming Luo, Tao Zeng, Guodong Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05326
• Pdf URL: https://arxiv.org/pdf/2403.05326
• Copy Paste: [[2403.05326]] ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues(https://arxiv.org/abs/2403.05326)
• Keywords: language model, llm, hallucination, chat
• Abstract: Aspect Sentiment Understanding (ASU) in interactive scenarios (e.g., Question-Answering and Dialogue) has attracted ever-more interest in recent years and achieved important progresses. However, existing studies on interactive ASU largely ignore the coreference issue for opinion targets (i.e., aspects), while this phenomenon is ubiquitous in interactive scenarios especially dialogues, limiting the ASU performance. Recently, large language models (LLMs) shows the powerful ability to integrate various NLP tasks with the chat paradigm. In this way, this paper proposes a new Chat-based Aspect Sentiment Understanding (ChatASU) task, aiming to explore LLMs' ability in understanding aspect sentiments in dialogue scenarios. Particularly, this ChatASU task introduces a sub-task, i.e., Aspect Chain Reasoning (ACR) task, to address the aspect coreference issue. On this basis, we propose a Trusted Self-reflexion Approach (TSA) with ChatGLM as backbone to ChatASU. Specifically, this TSA treats the ACR task as an auxiliary task to boost the performance of the primary ASU task, and further integrates trusted learning into reflexion mechanisms to alleviate the LLMs-intrinsic factual hallucination problem in TSA. Furthermore, a high-quality ChatASU dataset is annotated to evaluate TSA, and extensive experiments show that our proposed TSA can significantly outperform several state-of-the-art baselines, justifying the effectiveness of TSA to ChatASU and the importance of considering the coreference and hallucination issues in ChatASU.
• 摘要：交互场景（例如问答和对话）中的方面情感理解（ASU）近年来引起了越来越多的兴趣并取得了重要进展。然而，现有的交互式ASU研究很大程度上忽略了观点目标（即方面）的共指问题，而这种现象在交互场景尤其是对话中普遍存在，限制了ASU的性能。最近，大型语言模型（LLM）显示出将各种 NLP 任务与聊天范式集成的强大能力。为此，本文提出了一种新的基于聊天的方面情感理解（ChatASU）任务，旨在探索法学硕士在对话场景中理解方面情感的能力。特别地，这个ChatASU任务引入了一个子任务，即方面链推理（ACR）任务，来解决方面共指问题。在此基础上，我们提出了一种以 ChatGLM 作为 ChatASU 骨干的可信自我反思方法 (TSA)。具体来说，该TSA将ACR任务视为辅助任务，以提高主要ASU任务的性能，并进一步将可信学习融入反射机制中，以缓解TSA中LLM固有的事实幻觉问题。此外，高质量的 ChatASU 数据集被注释来评估 TSA，并且大量实验表明，我们提出的 TSA 可以显着优于几个最先进的基线，证明了 TSA 对 ChatASU 的有效性以及考虑共指和ChatASU 中的幻觉问题。

Title: Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings

• Authors: Wei Zhou, Heike Adel, Hendrik Schuff, Ngoc Thang Vu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05338
• Pdf URL: https://arxiv.org/pdf/2403.05338
• Copy Paste: [[2403.05338]] Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings(https://arxiv.org/abs/2403.05338)
• Keywords: language model, prompt
• Abstract: Attribution scores indicate the importance of different input parts and can, thus, explain model behaviour. Currently, prompt-based models are gaining popularity, i.a., due to their easier adaptability in low-resource settings. However, the quality of attribution scores extracted from prompt-based models has not been investigated yet. In this work, we address this topic by analyzing attribution scores extracted from prompt-based models w.r.t. plausibility and faithfulness and comparing them with attribution scores extracted from fine-tuned models and large language models. In contrast to previous work, we introduce training size as another dimension into the analysis. We find that using the prompting paradigm (with either encoder-based or decoder-based models) yields more plausible explanations than fine-tuning the models in low-resource settings and Shapley Value Sampling consistently outperforms attention and Integrated Gradients in terms of leading to more plausible and faithful explanations.
• 摘要：归因分数表明不同输入部分的重要性，因此可以解释模型行为。目前，基于提示的模型越来越受欢迎，因为它们在资源匮乏的环境中更容易适应。然而，从基于提示的模型中提取的归因分数的质量尚未得到研究。在这项工作中，我们通过分析从基于提示的模型中提取的归因分数来解决这个主题。合理性和忠实性，并将它们与从微调模型和大型语言模型中提取的归因分数进行比较。与之前的工作相比，我们将训练规模作为另一个维度引入到分析中。我们发现，使用提示范式（基于编码器或基于解码器的模型）比在资源匮乏的环境中微调模型能产生更合理的解释，并且 Shapley 值采样在导致更多结果方面始终优于注意力和集成梯度。合理且忠实的解释。

Title: Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs

• Authors: Arijit Nag, Animesh Mukherjee, Niloy Ganguly, Soumen Chakrabarti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05434
• Pdf URL: https://arxiv.org/pdf/2403.05434
• Copy Paste: [[2403.05434]] Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs(https://arxiv.org/abs/2403.05434)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) exhibit impressive zero/few-shot inference and generation quality for high-resource languages(HRLs). A few of them have been trained in low-resource languages (LRLs) and give decent performance. Owing to the prohibitive costs of training LLMs, they are usually used as a network service, with the client charged by the count of input and output tokens. The number of tokens strongly depends on the script and language, as well as the LLM's sub-word vocabulary. We show that LRLs are at a pricing disadvantage, because the well-known LLMs produce more tokens for LRLs than HRLs. This is because most currently popular LLMs are optimized for HRL vocabularies. Our objective is to level the playing field: reduce the cost of processing LRLs in contemporary LLMs while ensuring that predictive and generative qualities are not compromised. As means to reduce the number of tokens processed by the LLM, we consider code-mixing, translation, and transliteration of LRLs to HRLs. We perform an extensive study using the IndicXTREME dataset, covering 15 Indian languages, while using GPT-4 (one of the costliest LLM services released so far) as a commercial LLM. We observe and analyze interesting patterns involving token count, cost,and quality across a multitude of languages and tasks. We show that choosing the best policy to interact with the LLM can reduce cost by 90% while giving better or comparable performance, compared to communicating with the LLM in the original LRL.
• 摘要：大型语言模型 (LLM) 表现出令人印象深刻的零/少样本推理和高资源语言 (HRL) 的生成质量。其中一些已经接受过低资源语言（LRL）的培训并提供了不错的性能。由于培训 LLM 的成本高昂，它们通常被用作网络服务，客户按输入和输出代币的数量收费。标记的数量很大程度上取决于脚本和语言，以及法学硕士的子词词汇。我们表明 LRL 处于定价劣势，因为著名的 LLM 为 LRL 生产的代币比 HRL 更多。这是因为目前大多数流行的法学硕士都针对 HRL 词汇进行了优化。我们的目标是创造公平的竞争环境：降低当代法学硕士处理 LRL 的成本，同时确保预测和生成质量不受影响。作为减少 LLM 处理的标记数量的方法，我们考虑了 LRL 到 HRL 的代码混合、翻译和音译。我们使用 IndicXTREME 数据集进行了广泛的研究，涵盖 15 种印度语言，同时使用 GPT-4（迄今为止发布的最昂贵的 LLM 服务之一）作为商业 LLM。我们观察并分析多种语言和任务中涉及令牌计数、成本和质量的有趣模式。我们表明，与在原始 LRL 中与 LLM 通信相比，选择最佳策略与 LLM 交互可以降低 90% 的成本，同时提供更好或相当的性能。

Title: Will GPT-4 Run DOOM?

• Authors: Adrian de Wynter
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05468
• Pdf URL: https://arxiv.org/pdf/2403.05468
• Copy Paste: [[2403.05468]] Will GPT-4 Run DOOM?(https://arxiv.org/abs/2403.05468)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: We show that GPT-4's reasoning and planning capabilities extend to the 1993 first-person shooter Doom. This large language model (LLM) is able to run and play the game with only a few instructions, plus a textual description--generated by the model itself from screenshots--about the state of the game being observed. We find that GPT-4 can play the game to a passable degree: it is able to manipulate doors, combat enemies, and perform pathing. More complex prompting strategies involving multiple model calls provide better results. While further work is required to enable the LLM to play the game as well as its classical, reinforcement learning-based counterparts, we note that GPT-4 required no training, leaning instead on its own reasoning and observational capabilities. We hope our work pushes the boundaries on intelligent, LLM-based agents in video games. We conclude by discussing the ethical implications of our work.
• 摘要：我们展示了 GPT-4 的推理和规划能力扩展到 1993 年的第一人称射击游戏《毁灭战士》。这种大型语言模型（LLM）只需几条指令即可运行和玩游戏，再加上模型本身根据屏幕截图生成的关于所观察到的游戏状态的文本描述。我们发现 GPT-4 可以达到还过得去的程度：它能够操纵门、与敌人战斗并执行路径。涉及多个模型调用的更复杂的提示策略可提供更好的结果。虽然需要进一步的工作来使 LLM 能够像基于强化学习的经典游戏一样玩游戏，但我们注意到 GPT-4 不需要训练，而是依靠其自身的推理和观察能力。我们希望我们的工作能够突破视频游戏中基于法学硕士的智能代理的界限。最后，我们讨论了我们工作的道德影响。

Title: To Err Is Human, but Llamas Can Learn It Too

• Authors: Agnes Luhtaru, Taido Purason, Martin Vainikko, Maksym Del, Mark Fishel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05493
• Pdf URL: https://arxiv.org/pdf/2403.05493
• Copy Paste: [[2403.05493]] To Err Is Human, but Llamas Can Learn It Too(https://arxiv.org/abs/2403.05493)
• Keywords: language model, gpt, prompt
• Abstract: This study explores enhancing grammatical error correction (GEC) through artificial error generation (AEG) using language models (LMs). Specifically, we fine-tune Llama 2-based LMs for error generation and find that this approach yields synthetic errors akin to human errors. Next, we train GEC Llama models with the help of these artificial errors and outperform previous state-of-the-art error correction models, with gains ranging between 0.8 and 6 F0.5 points across all tested languages (German, Ukrainian, and Estonian). Moreover, we demonstrate that generating errors by fine-tuning smaller sequence-to-sequence models and prompting large commercial LMs (GPT-3.5 and GPT-4) also results in synthetic errors beneficially affecting error generation models.
• 摘要：本研究探索通过使用语言模型 (LM) 的人工错误生成 (AEG) 来增强语法错误纠正 (GEC)。具体来说，我们对基于 Llama 2 的 LM 进行微调以生成错误，并发现这种方法会产生类似于人为错误的合成错误。接下来，我们借助这些人为错误训练 GEC Llama 模型，并超越了之前最先进的纠错模型，在所有测试语言（德语、乌克兰语和爱沙尼亚语）中获得了 0.8 到 6 F0.5 点的增益）。此外，我们证明，通过微调较小的序列到序列模型并提示大型商业 LM（GPT-3.5 和 GPT-4）来生成错误也会导致合成错误，从而有益地影响错误生成模型。

Title: Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

• Authors: James Chua, Edward Rees, Hunar Batra, Samuel R. Bowman, Julian Michael, Ethan Perez, Miles Turpin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05518
• Pdf URL: https://arxiv.org/pdf/2403.05518
• Copy Paste: [[2403.05518]] Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought(https://arxiv.org/abs/2403.05518)
• Keywords: language model, gpt, prompt, chain-of-thought
• Abstract: While chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning, it can systematically misrepresent the factors influencing models' behavior--for example, rationalizing answers in line with a user's opinion without mentioning this bias. To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features. We construct a suite testing nine forms of biased reasoning on seven question-answering tasks, and find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks. Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%. As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where supervision for ground truth reasoning is unavailable.
• 摘要：虽然思想链提示 (CoT) 有潜力提高语言模型推理的可解释性，但它可能会系统地歪曲影响模型行为的因素，例如，根据用户的意见合理化答案，而不提及这种偏见。为了缓解这种有偏见的推理问题，我们引入了偏见增强一致性训练（BCT），这是一种无监督的微调方案，可以训练模型在有或没有偏见特征的提示中提供一致的推理。我们构建了一个套件，在七个问答任务上测试九种形式的偏见推理，发现将 BCT 应用到具有一种偏见的 GPT-3.5-Turbo 可以将保留任务的偏见推理率降低 86%。此外，该模型还可以推广到其他形式的偏见，将针对保留偏见的偏见推理平均减少 37%。由于 BCT 能够泛化到保留的偏见并且不需要黄金标签，因此这种方法可能有望减少来自未知偏见的偏见推理以及无法对地面真相推理进行监督的任务。

Title: Authorship Attribution in Bangla Literature (AABL) via Transfer Learning using ULMFiT

• Authors: Aisha Khatun, Anisur Rahman, Md Saiful Islam, Hemayet Ahmed Chowdhury, Ayesha Tasnim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05519
• Pdf URL: https://arxiv.org/pdf/2403.05519
• Copy Paste: [[2403.05519]] Authorship Attribution in Bangla Literature (AABL) via Transfer Learning using ULMFiT(https://arxiv.org/abs/2403.05519)
• Keywords: language model
• Abstract: Authorship Attribution is the task of creating an appropriate characterization of text that captures the authors' writing style to identify the original author of a given piece of text. With increased anonymity on the internet, this task has become increasingly crucial in various security and plagiarism detection fields. Despite significant advancements in other languages such as English, Spanish, and Chinese, Bangla lacks comprehensive research in this field due to its complex linguistic feature and sentence structure. Moreover, existing systems are not scalable when the number of author increases, and the performance drops for small number of samples per author. In this paper, we propose the use of Average-Stochastic Gradient Descent Weight-Dropped Long Short-Term Memory (AWD-LSTM) architecture and an effective transfer learning approach that addresses the problem of complex linguistic features extraction and scalability for authorship attribution in Bangla Literature (AABL). We analyze the effect of different tokenization, such as word, sub-word, and character level tokenization, and demonstrate the effectiveness of these tokenizations in the proposed model. Moreover, we introduce the publicly available Bangla Authorship Attribution Dataset of 16 authors (BAAD16) containing 17,966 sample texts and 13.4+ million words to solve the standard dataset scarcity problem and release six variations of pre-trained language models for use in any Bangla NLP downstream task. For evaluation, we used our developed BAAD16 dataset as well as other publicly available datasets. Empirically, our proposed model outperformed state-of-the-art models and achieved 99.8% accuracy in the BAAD16 dataset. Furthermore, we showed that the proposed system scales much better even with an increasing number of authors, and performance remains steady despite few training samples.
• 摘要：作者归属是创建适当的文本特征的任务，捕捉作者的写作风格，以识别给定文本的原始作者。随着互联网上匿名性的增加，这项任务在各种安全和抄袭检测领域变得越来越重要。尽管英语、西班牙语和汉语等其他语言取得了显着进步，但孟加拉语由于其复杂的语言特征和句子结构，在该领域缺乏全面的研究。此外，当作者数量增加时，现有系统不可扩展，并且对于每个作者的样本数量较少，性能会下降。在本文中，我们提出使用平均随机梯度下降权重下降长短期记忆（AWD-LSTM）架构和有效的迁移学习方法来解决孟加拉语作者归属的复杂语言特征提取和可扩展性问题文学（AABL）。我们分析了不同标记化的效果，例如单词、子词和字符级标记化，并在所提出的模型中证明了这些标记化的有效性。此外，我们引入了公开的 16 位作者的孟加拉作者归属数据集 (BAAD16)，其中包含 17,966 个样本文本和 1340 万多个单词，以解决标准数据集稀缺问题，并发布预训练语言模型的六种变体，以供任何孟加拉 NLP 下游使用任务。为了进行评估，我们使用了我们开发的 BAAD16 数据集以及其他公开可用的数据集。根据经验，我们提出的模型优于最先进的模型，在 BAAD16 数据集中实现了 99.8% 的准确率。此外，我们表明，即使作者数量不断增加，所提出的系统也能更好地扩展，并且尽管训练样本很少，但性能仍然保持稳定。

Title: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

• Authors: Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, et al. (619 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05530
• Pdf URL: https://arxiv.org/pdf/2403.05530
• Copy Paste: [[2403.05530]] Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context(https://arxiv.org/abs/2403.05530)
• Keywords: language model, gpt
• Abstract: In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
• 摘要：在本报告中，我们介绍了 Gemini 系列的最新模型 Gemini 1.5 Pro，这是一种计算效率高的多模式专家混合模型，能够从数百万个上下文标记中调用细粒度信息并进行推理，包括多个长文件以及视频和音频的时间。 Gemini 1.5 Pro 在跨模态的长上下文检索任务上实现了近乎完美的召回，提高了长文档 QA、长视频 QA 和长上下文 ASR 的最新水平，并匹配或超越 Gemini 1.0 Ultra 的状态在一系列广泛的基准测试中具有最先进的性能。研究 Gemini 1.5 Pro 长上下文能力的局限性，我们发现下一个标记预测和近乎完美的检索 (>99%) 方面持续改进，最多可达至少 10M 个标记，这是对 Claude 2.1（200k）等现有模型的一代飞跃。 ）和 GPT-4 Turbo (128k)。最后，我们强调大型语言模型在前沿的令人惊讶的新功能；当给定卡拉芒语（一种全球使用人数不足 200 人的语言）的语法手册时，模型可以学习将英语翻译成卡拉芒语，其水平与学习相同内容的人相似。

Title: Bayesian Preference Elicitation with Language Models

• Authors: Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, Belinda Z. Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05534
• Pdf URL: https://arxiv.org/pdf/2403.05534
• Copy Paste: [[2403.05534]] Bayesian Preference Elicitation with Language Models(https://arxiv.org/abs/2403.05534)
• Keywords: language model
• Abstract: Aligning AI systems to users' interests requires understanding and incorporating humans' complex values and preferences. Recently, language models (LMs) have been used to gather information about the preferences of human users. This preference data can be used to fine-tune or guide other LMs and/or AI systems. However, LMs have been shown to struggle with crucial aspects of preference learning: quantifying uncertainty, modeling human mental states, and asking informative questions. These challenges have been addressed in other areas of machine learning, such as Bayesian Optimal Experimental Design (BOED), which focus on designing informative queries within a well-defined feature space. But these methods, in turn, are difficult to scale and apply to real-world problems where simply identifying the relevant features can be difficult. We introduce OPEN (Optimal Preference Elicitation with Natural language) a framework that uses BOED to guide the choice of informative questions and an LM to extract features and translate abstract BOED queries into natural language questions. By combining the flexibility of LMs with the rigor of BOED, OPEN can optimize the informativity of queries while remaining adaptable to real-world domains. In user studies, we find that OPEN outperforms existing LM- and BOED-based methods for preference elicitation.
• 摘要：让人工智能系统与用户的兴趣保持一致需要理解并融入人类复杂的价值观和偏好。最近，语言模型（LM）已被用来收集有关人类用户偏好的信息。该偏好数据可用于微调或指导其他语言模型和/或人工智能系统。然而，LM 已被证明在偏好学习的关键方面遇到了困难：量化不确定性、模拟人类心理状态以及提出信息丰富的问题​​。这些挑战已在机器学习的其他领域得到解决，例如贝叶斯最优实验设计（BOED），其重点是在明确定义的特征空间内设计信息丰富的查询。但这些方法反过来又难以扩展并应用于现实世界的问题，在现实世界中，简单地识别相关特征可能很困难。我们引入了 OPEN（自然语言最优偏好诱导）框架，该框架使用 BOED 来指导信息性问题的选择，并使用 LM 来提取特征并将抽象的 BOED 查询翻译为自然语言问题。通过将 LM 的灵活性与 BOED 的严格性相结合，OPEN 可以优化查询的信息量，同时保持对现实世界领域的适应性。在用户研究中，我们发现 OPEN 优于现有的基于 LM 和 BOED 的偏好诱导方法。