2024-05-29

Title: HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs

• Authors: Jocelyn Shen, Joel Mire, Hae Won Park, Cynthia Breazeal, Maarten Sap
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17633
• Pdf URL: https://arxiv.org/pdf/2405.17633
• Copy Paste: [[2405.17633]] HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs(https://arxiv.org/abs/2405.17633)
• Keywords: llm, prompt
• Abstract: Empathy serves as a cornerstone in enabling prosocial behaviors, and can be evoked through sharing of personal experiences in stories. While empathy is influenced by narrative content, intuitively, people respond to the way a story is told as well, through narrative style. Yet the relationship between empathy and narrative style is not fully understood. In this work, we empirically examine and quantify this relationship between style and empathy using LLMs and large-scale crowdsourcing studies. We introduce a novel, theory-based taxonomy, HEART (Human Empathy and Narrative Taxonomy) that delineates elements of narrative style that can lead to empathy with the narrator of a story. We establish the performance of LLMs in extracting narrative elements from HEART, showing that prompting with our taxonomy leads to reasonable, human-level annotations beyond what prior lexicon-based methods can do. To show empirical use of our taxonomy, we collect a dataset of empathy judgments of stories via a large-scale crowdsourcing study with N=2,624 participants. We show that narrative elements extracted via LLMs, in particular, vividness of emotions and plot volume, can elucidate the pathways by which narrative style cultivates empathy towards personal stories. Our work suggests that such models can be used for narrative analyses that lead to human-centered social and behavioral insights.
• 摘要：同理心是促成亲社会行为的基石，可通过在故事中分享个人经历来激发同理心。虽然同理心受叙事内容的影响，但直觉上，人们也会通过叙事风格对故事的讲述方式做出反应。然而，同理心和叙事风格之间的关系尚未完全了解。在这项工作中，我们使用 LLM 和大规模众包研究，对风格和同理心之间的关系进行了实证检验和量化。我们引入了一种新颖的基于理论的分类法 HEART（人类同理心和叙事分类法），它描述了可以引起对故事叙述者产生同理心的叙事风格元素。我们确定了 LLM 在从 HEART 中提取叙事元素方面的表现，表明使用我们的分类法进行提示可以产生合理的、人类级别的注释，超出了以前基于词典的方法所能达到的效果。为了展示我们分类法的实证应用，我们通过一项大规模众包研究收集了故事共情判断数据集，参与者人数为 N=2,624。我们表明，通过 LLM 提取的叙事元素（尤其是情感生动性和情节量）可以阐明叙事风格培养对个人故事的共情的途径。我们的工作表明，此类模型可用于叙事分析，从而获得以人为本的社会和行为洞察。

Title: CLAIM Your Data: Enhancing Imputation Accuracy with Contextual Large Language Models

• Authors: Ahatsham Hayat, Mohammad Rashedul Hasan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.17712
• Pdf URL: https://arxiv.org/pdf/2405.17712
• Copy Paste: [[2405.17712]] CLAIM Your Data: Enhancing Imputation Accuracy with Contextual Large Language Models(https://arxiv.org/abs/2405.17712)
• Keywords: language model, llm
• Abstract: This paper introduces the Contextual Language model for Accurate Imputation Method (CLAIM), a novel strategy that capitalizes on the expansive knowledge and reasoning capabilities of pre-trained large language models (LLMs) to address missing data challenges in tabular datasets. Unlike traditional imputation methods, which predominantly rely on numerical estimations, CLAIM utilizes contextually relevant natural language descriptors to fill missing values. This approach transforms datasets into natural language contextualized formats that are inherently more aligned with LLMs' capabilities, thereby facilitating the dual use of LLMs: first, to generate missing value descriptors, and then, to fine-tune the LLM on the enriched dataset for improved performance in downstream tasks. Our evaluations across diverse datasets and missingness patterns reveal CLAIM's superior performance over existing imputation techniques. Furthermore, our investigation into the effectiveness of context-specific versus generic descriptors for missing data highlights the importance of contextual accuracy in enhancing LLM performance for data imputation. The results underscore CLAIM's potential to markedly improve the reliability and quality of data analysis and machine learning models, offering a more nuanced and effective solution for handling missing data.
• 摘要：本文介绍了用于准确插补方法的上下文语言模型 (CLAIM)，这是一种新颖的策略，它利用预先训练的大型语言模型 (LLM) 的广泛知识和推理能力来解决表格数据集中的缺失数据挑战。与主要依赖数值估计的传统插补方法不同，CLAIM 使用上下文相关的自然语言描述符来填充缺失值。这种方法将数据集转换为自然语言上下文格式，本质上更符合 LLM 的功能，从而促进了 LLM 的双重用途：首先，生成缺失值描述符，然后在丰富的数据集上微调 LLM，以提高下游任务的性能。我们对不同数据集和缺失模式的评估表明，CLAIM 的性能优于现有的插补技术。此外，我们对上下文特定描述符与通用描述符对缺失数据的有效性的研究突出了上下文准确性在提高 LLM 数据插补性能方面的重要性。结果强调了 CLAIM 显著提高数据分析和机器学习模型的可靠性和质量的潜力，为处理缺失数据提供更细致、更有效的解决方案。

Title: C$^{3}$Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models

• Authors: Jiahuan Cao, Yongxin Shi, Dezhi Peng, Yang Liu, Lianwen Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] C$^{3}$Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: Classical Chinese Understanding (CCU) holds significant value in preserving and exploration of the outstanding traditional Chinese culture. Recently, researchers have attempted to leverage the potential of Large Language Models (LLMs) for CCU by capitalizing on their remarkable comprehension and semantic capabilities. However, no comprehensive benchmark is available to assess the CCU capabilities of LLMs. To fill this gap, this paper introduces C$^{3}$bench, a Comprehensive Classical Chinese understanding benchmark, which comprises 50,000 text pairs for five primary CCU tasks, including classification, retrieval, named entity recognition, punctuation, and translation. Furthermore, the data in C$^{3}$bench originates from ten different domains, covering most of the categories in classical Chinese. Leveraging the proposed C$^{3}$bench, we extensively evaluate the quantitative performance of 15 representative LLMs on all five CCU tasks. Our results not only establish a public leaderboard of LLMs' CCU capabilities but also gain some findings. Specifically, existing LLMs are struggle with CCU tasks and still inferior to supervised models. Additionally, the results indicate that CCU is a task that requires special attention. We believe this study could provide a standard benchmark, comprehensive baselines, and valuable insights for the future advancement of LLM-based CCU research. The evaluation pipeline and dataset are available at \url{this https URL}.
• 摘要：古汉语理解 (CCU) 对保护和发掘中国优秀传统文化具有重要价值。最近，研究人员试图利用大型语言模型 (LLM) 卓越的理解和语义能力，充分发挥其在 CCU 中的潜力。然而，目前还没有全面的基准来评估 LLM 的 CCU 能力。为了填补这一空白，本文引入了 C$^{3}$bench，这是一个全面的古汉语理解基准，包含 50,000 个文本对，用于五个主要 CCU 任务，包括分类、检索、命名实体识别、标点符号和翻译。此外，C$^{3}$bench 中的数据来自十个不同的领域，涵盖了古汉语的大部分类别。利用所提出的 C$^{3}$bench，我们对 15 个代表性 LLM 在所有五个 CCU 任务上的定量性能进行了广泛的评估。我们的结果不仅建立了 LLM CCU 能力的公共排行榜，而且还获得了一些发现。具体而言，现有的 LLM 在 CCU 任务上表现不佳，并且仍然不如监督模型。此外，结果表明 CCU 是一项需要特别关注的任务。我们相信这项研究可以为未来基于 LLM 的 CCU 研究的发展提供标准基准、全面基线和宝贵见解。评估流程和数据集可在 \url{此 https URL} 获得。

Title: MobileConvRec: A Conversational Dataset for Mobile Apps Recommendations

• Authors: Srijata Maji, Moghis Fereidouni, Vinaik Chhetri, Umar Farooq, A.B. Siddique
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17740
• Pdf URL: https://arxiv.org/pdf/2405.17740
• Copy Paste: [[2405.17740]] MobileConvRec: A Conversational Dataset for Mobile Apps Recommendations(https://arxiv.org/abs/2405.17740)
• Keywords: language model
• Abstract: Existing recommendation systems have focused on two paradigms: 1- historical user-item interaction-based recommendations and 2- conversational recommendations. Conversational recommendation systems facilitate natural language dialogues between users and the system, allowing the system to solicit users' explicit needs while enabling users to inquire about recommendations and provide feedback. Due to substantial advancements in natural language processing, conversational recommendation systems have gained prominence. Existing conversational recommendation datasets have greatly facilitated research in their respective domains. Despite the exponential growth in mobile users and apps in recent years, research in conversational mobile app recommender systems has faced substantial constraints. This limitation can primarily be attributed to the lack of high-quality benchmark datasets specifically tailored for mobile apps. To facilitate research for conversational mobile app recommendations, we introduce MobileConvRec. MobileConvRec simulates conversations by leveraging real user interactions with mobile apps on the Google Play store, originally captured in large-scale mobile app recommendation dataset MobileRec. The proposed conversational recommendation dataset synergizes sequential user-item interactions, which reflect implicit user preferences, with comprehensive multi-turn conversations to effectively grasp explicit user needs. MobileConvRec consists of over 12K multi-turn recommendation-related conversations spanning 45 app categories. Moreover, MobileConvRec presents rich metadata for each app such as permissions data, security and privacy-related information, and binary executables of apps, among others. We demonstrate that MobileConvRec can serve as an excellent testbed for conversational mobile app recommendation through a comparative study of several pre-trained large language models.
• 摘要：现有的推荐系统主要集中在两种模式上：1-基于历史用户-项目交互的推荐和2-对话式推荐。对话式推荐系统促进用户与系统之间的自然语言对话，使系统能够征求用户的明确需求，同时使用户能够查询推荐并提供反馈。由于自然语言处理的重大进步，对话式推荐系统已获得广泛关注。现有的对话式推荐数据集极大地促进了各自领域的研究。尽管近年来移动用户和应用程序呈指数级增长，但对话式移动应用推荐系统的研究仍面临巨大的制约。这一限制主要归因于缺乏专门针对移动应用的高质量基准数据集。为了促进对话式移动应用推荐的研究，我们引入了 MobileConvRec。MobileConvRec 通过利用 Google Play 商店中真实的用户与移动应用的交互来模拟对话，这些交互最初是在大型移动应用推荐数据集 MobileRec 中捕获的。所提出的对话式推荐数据集将反映隐性用户偏好的连续用户-项目交互与全面的多轮对话相结合，以有效掌握明确的用户需求。MobileConvRec 包含 12K 多个多轮推荐相关对话，涵盖 45 个应用类别。此外，MobileConvRec 为每个应用提供丰富的元数据，例如权限数据、安全和隐私相关信息以及应用的二进制可执行文件等。我们通过对几个预先训练的大型语言模型的比较研究，证明了 MobileConvRec 可以作为对话式移动应用推荐的绝佳试验台。

Title: ORLM: Training Large Language Models for Optimization Modeling

• Authors: Zhengyang Tang, Chenyu Huang, Xin Zheng, Shixi Hu, Zizhuo Wang, Dongdong Ge, Benyou Wang
• Subjects: cs.CL, cs.AI, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.17743
• Pdf URL: https://arxiv.org/pdf/2405.17743
• Copy Paste: [[2405.17743]] ORLM: Training Large Language Models for Optimization Modeling(https://arxiv.org/abs/2405.17743)
• Keywords: language model, llm, prompt, agent
• Abstract: Large Language Models (LLMs) have emerged as powerful tools for complex Operations Research (OR) in automating optimization modeling. However, current methodologies heavily rely on prompt engineering (e.g., multi-agent cooperation) with proprietary LLMs, raising data privacy concerns that could be prohibitive in industry applications. To tackle this issue, we propose training open-source LLMs for optimization modeling. We identify four critical requirements for the training dataset of OR LLMs, design and implement OR-Instruct, a semi-automated process for creating synthetic data tailored to specific requirements. We also introduce the IndustryOR benchmark, the first industrial benchmark for testing LLMs on solving real-world OR problems. We apply the data from OR-Instruct to various open-source LLMs of 7b size (termed as ORLMs), resulting in a significantly improved capability for optimization modeling. Our best-performing ORLM achieves state-of-the-art performance on the NL4OPT, MAMO, and IndustryOR benchmarks. Our code and data will be available at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 已成为复杂运筹学 (OR) 实现优化建模自动化的强大工具。然而，当前的方法严重依赖于专有 LLM 的即时工程（例如，多智能体合作），这引发了数据隐私问题，这可能会阻碍行业应用。为了解决这个问题，我们建议训练开源 LLM 进行优化建模。我们确定了 OR LLM 训练数据集的四个关键要求，设计并实施了 OR-Instruct，这是一种根据特定要求定制的半自动化合成数据创建过程。我们还推出了 IndustryOR 基准，这是第一个用于测试 LLM 解决现实世界 OR 问题的工业基准。我们将 OR-Instruct 中的数据应用于各种 7b 大小的开源 LLM（称为 ORLM），从而显著提高了优化建模的能力。我们表现最佳的 ORLM 在 NL4OPT、MAMO 和 IndustryOR 基准上实现了最先进的性能。我们的代码和数据将在 \url{this https URL} 上提供。

Title: XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference

• Authors: Shengnan Wang, Youhui Bai, Lin Zhang, Pingyi Zhou, Shixiong Zhao, Gong Zhang, Sen Wang, Renhai Chen, Hua Xu, Hongwei Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.17755
• Pdf URL: https://arxiv.org/pdf/2405.17755
• Copy Paste: [[2405.17755]] XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference(https://arxiv.org/abs/2405.17755)
• Keywords: language model, llm
• Abstract: Length generalization failure problem, namely the large language model (LLM) fails to generalize to texts longer than its maximum training length, greatly restricts the application of LLM in the scenarios with streaming long inputs. To address this problem, the existing methods either require substantial costs or introduce precision loss. In this paper, we empirically find that the accuracy of the LLM's prediction is highly correlated to its certainty. Based on this, we propose an efficient training free framework, named XL3M (it means extra-long large language model), which enables the LLMs trained on short sequences to reason extremely long sequence without any further training or fine-tuning. Under the XL3M framework, the input context will be firstly decomposed into multiple short sub-contexts, where each sub-context contains an independent segment and a common ``question'' which is a few tokens from the end of the original context. Then XL3M gives a method to measure the relevance between each segment and the ``question'', and constructs a concise key context by splicing all the relevant segments in chronological order. The key context is further used instead of the original context to complete the inference task. Evaluations on comprehensive benchmarks show the superiority of XL3M. Using our framework, a Llama2-7B model is able to reason 20M long sequences on an 8-card Huawei Ascend 910B NPU machine with 64GB memory per card.
• 摘要：长度泛化失败问题，即大型语言模型 (LLM) 无法泛化到超过其最大训练长度的文本，极大地限制了 LLM 在流式长输入场景中的应用。为了解决这个问题，现有的方法要么需要大量的成本，要么引入精度损失。在本文中，我们通过经验发现 LLM 预测的准确率与其确定性高度相关。基于此，我们提出了一个高效的免训练框架 XL3M（超长大型语言模型），它使得在短序列上训练的 LLM 能够推理极长的序列，而无需任何进一步的训练或微调。在 XL3M 框架下，输入上下文首先被分解为多个短子上下文，其中每个子上下文包含一个独立的段和一个公共 “问题”，该 “问题” 距离原始上下文末尾有几个标记。然后XL3M给出了一种衡量每个片段与“问题”之间相关性的方法，并通过按时间顺序拼接所有相关片段来构建一个简洁的关键上下文。进一步使用关键上下文代替原始上下文完成推理任务。综合基准测试的评估显示了XL3M的优越性。使用我们的框架，Llama2-7B模型能够在8卡华为Ascend 910B NPU机器上推理20M长的序列，每卡64GB内存。

Title: On the Sequence Evaluation based on Stochastic Processes

• Authors: Tianhao Zhang, Zhexiao Lin, Zhecheng Sheng, Chen Jiang, Dongyeop Kang
• Subjects: cs.CL, cs.AI, math.ST
• Abstract URL: https://arxiv.org/abs/2405.17764
• Pdf URL: https://arxiv.org/pdf/2405.17764
• Copy Paste: [[2405.17764]] On the Sequence Evaluation based on Stochastic Processes(https://arxiv.org/abs/2405.17764)
• Keywords: language model
• Abstract: Modeling and analyzing long sequences of text is an essential task for Natural Language Processing. Success in capturing long text dynamics using neural language models will facilitate many downstream tasks such as coherence evaluation, text generation, machine translation and so on. This paper presents a novel approach to model sequences through a stochastic process. We introduce a likelihood-based training objective for the text encoder and design a more thorough measurement (score) for long text evaluation compared to the previous approach. The proposed training objective effectively preserves the sequence coherence, while the new score comprehensively captures both temporal and spatial dependencies. Theoretical properties of our new score show its advantages in sequence evaluation. Experimental results show superior performance in various sequence evaluation tasks, including global and local discrimination within and between documents of different lengths. We also demonstrate the encoder achieves competitive results on discriminating human and AI written text.
• 摘要：对长文本序列进行建模和分析是自然语言处理的一项基本任务。使用神经语言模型成功捕捉长文本动态将促进许多下游任务，如连贯性评估、文本生成、机器翻译等。本文提出了一种通过随机过程对序列进行建模的新方法。我们为文本编码器引入了一个基于可能性的训练目标，并设计了一个比以前的方法更彻底的长文本评估测量（分数）。提出的训练目标有效地保持了序列的连贯性，而新的分数全面捕捉了时间和空间依赖性。我们的新分数的理论特性显示了它在序列评估中的优势。实验结果显示在各种序列评估任务中的优异性能，包括不同长度的文档内和文档之间的全局和局部区分。我们还证明了编码器在区分人类和人工智能书面文本方面取得了有竞争力的结果。

Title: Detection-Correction Structure via General Language Model for Grammatical Error Correction

• Authors: Wei Li, Houfeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17804
• Pdf URL: https://arxiv.org/pdf/2405.17804
• Copy Paste: [[2405.17804]] Detection-Correction Structure via General Language Model for Grammatical Error Correction(https://arxiv.org/abs/2405.17804)
• Keywords: language model, llm
• Abstract: Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits, which can be decoupled into two components: detection and correction. However, previous works have predominantly focused on direct correction, with no prior efforts to integrate both into a single model. Moreover, the exploration of the detection-correction paradigm by large language models (LLMs) remains underdeveloped. This paper introduces an integrated detection-correction structure, named DeCoGLM, based on the General Language Model (GLM). The detection phase employs a fault-tolerant detection template, while the correction phase leverages autoregressive mask infilling for localized error correction. Through the strategic organization of input tokens and modification of attention masks, we facilitate multi-task learning within a single model. Our model demonstrates competitive performance against the state-of-the-art models on English and Chinese GEC datasets. Further experiments present the effectiveness of the detection-correction structure in LLMs, suggesting a promising direction for GEC.
• 摘要：语法错误纠正 (GEC) 是一项致力于以最少的编辑来纠正文本的任务，可以将其分解为两个部分：检测和纠正。然而，以前的研究主要集中在直接纠正上，之前没有将两者整合到单个模型中的努力。此外，大型语言模型 (LLM) 对检测-纠正范式的探索仍未得到充分发展。本文介绍了一种基于通用语言模型 (GLM) 的集成检测-纠正结构 DeCoGLM。检测阶段采用容错检测模板，而纠正阶段利用自回归掩码填充进行局部错误纠正。通过对输入标记进行战略性组织和对注意力掩码进行修改，我们促进了单个模型内的多任务学习。我们的模型在英语和中文 GEC 数据集上表现出与最先进模型相媲美的性能。进一步的实验展示了 LLM 中检测-纠正结构的有效性，为 GEC 指明了一个有希望的方向。

Title: Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action

• Authors: Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.17822
• Pdf URL: https://arxiv.org/pdf/2405.17822
• Copy Paste: [[2405.17822]] Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action(https://arxiv.org/abs/2405.17822)
• Keywords: language model, hallucination, prompt
• Abstract: We present a Conversational Chain-of-Action (Conv-CoA) framework for Open-domain Conversational Question Answering (OCQA). Compared with literature, Conv-CoA addresses three major challenges: (i) unfaithful hallucination that is inconsistent with real-time or domain facts, (ii) weak reasoning performance in conversational scenarios, and (iii) unsatisfying performance in conversational information retrieval. Our key contribution is a dynamic reasoning-retrieval mechanism that extracts the intent of the question and decomposes it into a reasoning chain to be solved via systematic prompting, pre-designed actions, updating the Contextual Knowledge Set (CKS), and a novel Hopfield-based retriever. Methodologically, we propose a resource-efficiency Hopfield retriever to enhance the efficiency and accuracy of conversational information retrieval within our actions. Additionally, we propose a conversational-multi-reference faith score (Conv-MRFS) to verify and resolve conflicts between retrieved knowledge and answers in conversations. Empirically, we conduct comparisons between our framework and 23 state-of-the-art methods across five different research directions and two public benchmarks. These comparisons demonstrate that our Conv-CoA outperforms other methods in both the accuracy and efficiency dimensions.
• 摘要：我们提出了一个用于开放域对话式问答 (OCQA) 的对话式行动链 (Conv-CoA) 框架。与文献相比，Conv-CoA 解决了三大挑战：(i) 与实时或领域事实不一致的不真实幻觉，(ii) 对话场景中的推理性能较弱，以及 (iii) 对话信息检索中的表现不令人满意。我们的主要贡献是一种动态推理检索机制，它提取问题的意图并将其分解为一个推理链，通过系统提示、预先设计的动作、更新上下文知识集 (CKS) 和一个基于 Hopfield 的新型检索器来解决。从方法论上讲，我们提出了一种资源高效的 Hopfield 检索器，以提高我们行动中对话信息检索的效率和准确性。此外，我们提出了一个对话多参考信念分数 (Conv-MRFS) 来验证和解决对话中检索到的知识和答案之间的冲突。从实证角度来看，我们对我们的框架与 5 个不同研究方向和 2 个公共基准中的 23 种最先进方法进行了比较。这些比较表明，我们的 Conv-CoA 在准确率和效率方面均优于其他方法。

Title: More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs

• Authors: Chengyuan Liu, Shihang Wang, Yangyang Kang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, Fei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17830
• Pdf URL: https://arxiv.org/pdf/2405.17830
• Copy Paste: [[2405.17830]] More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs(https://arxiv.org/abs/2405.17830)
• Keywords: language model, llm
• Abstract: The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.
• 摘要：大型语言模型 (LLM) 在特定领域任务上进行微调后，在一般任务上的表现会下降，这种现象被称为灾难性遗忘 (CF)。然而，本文提出了超越 CF 的领域特定 LLM 实际应用的另一个挑战，称为一般能力集成 (GCI)，这需要在单个实例中集成一般能力和领域知识。GCI 的目标不仅仅是保留以前获得的一般能力和新的领域知识，而是以有凝聚力的方式协调和利用这两组技能，以提高在特定领域任务上的表现。以法律领域为例，我们精心设计了三组不缺乏实用性的训练和测试任务，并构建了相应的数据集。为了更好地将一般能力融入特定领域场景，我们引入了 ALoRA，它在 LoRA 上使用多头注意力模块，促进从前一个 token 到当前 token 的直接信息传递。这种增强允许表示根据注意力在领域特定知识和一般能力之间动态切换。针对所提出的任务进行了大量的实验。结果证明了我们的设置的重要性以及我们方法的有效性。

Title: Benchmark Underestimates the Readiness of Multi-lingual Dialogue Agents

• Authors: Andrew H. Lee, Sina J. Semnani, Galo Castillo-López, Gäel de Chalendar, Monojit Choudhury, Ashna Dua, Kapil Rajesh Kavitha, Sungkyun Kim, Prashant Kodali, Ponnurangam Kumaraguru, Alexis Lombard, Mehrad Moradshahi, Gihyun Park, Nasredine Semmar, Jiwon Seo, Tianhao Shen, Manish Shrivastava, Deyi Xiong, Monica S. Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Benchmark Underestimates the Readiness of Multi-lingual Dialogue Agents(https://arxiv.org/abs/)
• Keywords: gpt, prompt, agent
• Abstract: Creating multilingual task-oriented dialogue (TOD) agents is challenging due to the high cost of training data acquisition. Following the research trend of improving training data efficiency, we show for the first time, that in-context learning is sufficient to tackle multilingual TOD. To handle the challenging dialogue state tracking (DST) subtask, we break it down to simpler steps that are more compatible with in-context learning where only a handful of few-shot examples are used. We test our approach on the multilingual TOD dataset X-RiSAWOZ, which has 12 domains in Chinese, English, French, Korean, Hindi, and code-mixed Hindi-English. Our turn-by-turn DST accuracy on the 6 languages range from 55.6% to 80.3%, seemingly worse than the SOTA results from fine-tuned models that achieve from 60.7% to 82.8%; our BLEU scores in the response generation (RG) subtask are also significantly lower than SOTA. However, after manual evaluation of the validation set, we find that by correcting gold label errors and improving dataset annotation schema, GPT-4 with our prompts can achieve (1) 89.6%-96.8% accuracy in DST, and (2) more than 99% correct response generation across different languages. This leads us to conclude that current automatic metrics heavily underestimate the effectiveness of in-context learning.
• 摘要：由于训练数据获取成本高昂，创建多语言任务导向对话 (TOD) 代理具有挑战性。顺应提高训练数据效率的研究趋势，我们首次表明，情境学习足以解决多语言 TOD。为了处理具有挑战性的对话状态跟踪 (DST) 子任务，我们将其分解为更简单的步骤，这些步骤与仅使用少量样本的情境学习更兼容。我们在多语言 TOD 数据集 X-RiSAWOZ 上测试了我们的方法，该数据集包含中文、英语、法语、韩语、印地语和混合代码的印地语-英语 12 个域。我们对 6 种语言的逐向 DST 准确率范围为 55.6% 到 80.3%，似乎比微调模型的 SOTA 结果更差，后者实现了 60.7% 到 82.8%；我们在响应生成 (RG) 子任务中的 BLEU 分数也明显低于 SOTA。然而，在对验证集进行手动评估后，我们发现，通过纠正金标签错误并改进数据集注释模式，使用我们的提示的 GPT-4 可以实现 (1) DST 中 89.6%-96.8% 的准确率，以及 (2) 跨不同语言的 99% 以上的正确响应生成。这让我们得出结论，当前的自动指标严重低估了上下文学习的有效性。

Title: Arithmetic Reasoning with LLM: Prolog Generation & Permutation

• Authors: Xiaocheng Yang, Bingsen Chen, Yik-Cheung Tam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.17893
• Pdf URL: https://arxiv.org/pdf/2405.17893
• Copy Paste: [[2405.17893]] Arithmetic Reasoning with LLM: Prolog Generation & Permutation(https://arxiv.org/abs/2405.17893)
• Keywords: language model, llm
• Abstract: Instructing large language models (LLMs) to solve elementary school math problems has shown great success using Chain of Thought (CoT). However, the CoT approach relies on an LLM to generate a sequence of arithmetic calculations which can be prone to cascaded calculation errors. We hypothesize that an LLM should focus on extracting predicates and generating symbolic formulas from the math problem description so that the underlying calculation can be done via an external code interpreter. We investigate using LLM to generate Prolog programs to solve mathematical questions. Experimental results show that our Prolog-based arithmetic problem-solving outperforms CoT generation in the GSM8K benchmark across three distinct LLMs. In addition, given the insensitive ordering of predicates and symbolic formulas in Prolog, we propose to permute the ground truth predicates for more robust LLM training via data augmentation.
• 摘要：使用思路链 (CoT) 指导大型语言模型 (LLM) 解决小学数学问题已取得巨大成功。然而，CoT 方法依赖于 LLM 来生成一系列算术计算，而这些计算容易出现级联计算错误。我们假设 LLM 应该专注于从数学问题描述中提取谓词并生成符号公式，以便可以通过外部代码解释器完成底层计算。我们研究使用 LLM 生成 Prolog 程序来解决数学问题。实验结果表明，在三个不同的 LLM 中，我们基于 Prolog 的算术问题求解在 GSM8K 基准测试中优于 CoT 生成。此外，考虑到 Prolog 中谓词和符号公式的排序不敏感，我们建议通过数据增强对基本事实谓词进行排列，以实现更稳健的 LLM 训练。

Title: Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models

• Authors: Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17915
• Pdf URL: https://arxiv.org/pdf/2405.17915
• Copy Paste: [[2405.17915]] Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models(https://arxiv.org/abs/2405.17915)
• Keywords: language model, llm, long context
• Abstract: Long-context modeling capabilities are important for large language models (LLMs) in various applications. However, directly training LLMs with long context windows is insufficient to enhance this capability since some training samples do not exhibit strong semantic dependencies across long contexts. In this study, we propose a data mining framework \textbf{ProLong} that can assign each training sample with a long dependency score, which can be used to rank and filter samples that are more advantageous for enhancing long-context modeling abilities in LLM training. Specifically, we first use delta perplexity scores to measure the \textit{Dependency Strength} between text segments in a given document. Then we refine this metric based on the \textit{Dependency Distance} of these segments to incorporate spatial relationships across long-contexts. Final results are calibrated with a \textit{Dependency Specificity} metric to prevent trivial dependencies introduced by repetitive patterns. Moreover, a random sampling approach is proposed to optimize the computational efficiency of ProLong. Comprehensive experiments on multiple benchmarks indicate that ProLong effectively identifies documents that carry long dependencies and LLMs trained on these documents exhibit significantly enhanced long-context modeling capabilities.
• 摘要：长上下文建模能力对于各种应用中的大型语言模型 (LLM) 非常重要。然而，直接使用长上下文窗口训练 LLM 不足以增强这种能力，因为一些训练样本在长上下文中没有表现出很强的语义依赖性。在本研究中，我们提出了一个数据挖掘框架 \textbf{ProLong}，它可以为每个训练样本分配一个长依赖性分数，该分数可用于对 LLM 训练中更有利于增强长上下文建模能力的样本进行排序和筛选。具体来说，我们首先使用 delta 困惑度分数来测量给定文档中文本段之间的 \textit{依赖强度}。然后，我们根据这些段的 \textit{依赖距离} 改进该指标，以纳入长上下文中的空间关系。最终结果通过 \textit{依赖特异性} 指标进行校准，以防止重复模式引入的琐碎依赖关系。此外，还提出了一种随机抽样方法来优化 ProLong 的计算效率。在多个基准上进行的综合实验表明，ProLong 可以有效识别带有长依赖关系的文档，并且在这些文档上训练的 LLM 表现出显著增强的长上下文建模能力。

Title: Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment

• Authors: Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, Chang Zhou
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.17931
• Pdf URL: https://arxiv.org/pdf/2405.17931
• Copy Paste: [[2405.17931]] Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment(https://arxiv.org/abs/2405.17931)
• Keywords: language model, llm
• Abstract: Effectively aligning Large Language Models (LLMs) with human-centric values while preventing the degradation of abilities acquired through Pre-training and Supervised Fine-tuning (SFT) poses a central challenge in Reinforcement Learning from Human Feedback (RLHF). In this paper, we first discover that interpolating RLHF and SFT model parameters can adjust the trade-off between human preference and basic capabilities, thereby reducing the alignment tax at the cost of alignment reward. Inspired by this, we propose integrating the RL policy and SFT models at each optimization step in RLHF to continuously regulate the training direction, introducing the Online Merging Optimizer. Specifically, we merge gradients with the parameter differences between SFT and pretrained models, effectively steering the gradient towards maximizing rewards in the direction of SFT optimization. We demonstrate that our optimizer works well with different LLM families, such as Qwen and LLaMA, across various model sizes ranging from 1.8B to 8B, various RLHF algorithms like DPO and KTO, and existing model merging methods. It significantly enhances alignment reward while mitigating alignment tax, achieving higher overall performance across 14 benchmarks.
• 摘要：有效地将大型语言模型 (LLM) 与以人为中心的价值观对齐，同时防止通过预训练和监督微调 (SFT) 获得的能力退化，这是强化学习从人类反馈 (RLHF) 中面临的一个核心挑战。在本文中，我们首先发现插入 RLHF 和 SFT 模型参数可以调整人类偏好和基本能力之间的权衡，从而以对齐奖励为代价降低对齐税。受此启发，我们建议在 RLHF 的每个优化步骤中集成 RL 策略和 SFT 模型以不断调节训练方向，并引入在线合并优化器。具体来说，我们将梯度与 SFT 和预训练模型之间的参数差异合并，有效地将梯度引导到最大化 SFT 优化方向上的奖励。我们证明我们的优化器可以很好地与不同的 LLM 系列（例如 Qwen 和 LLaMA）配合使用，适用于从 1.8B 到 8B 的各种模型大小、各种 RLHF 算法（如 DPO 和 KTO）以及现有的模型合并方法。它显著提高了对齐奖励，同时减轻了对齐税，在 14 个基准测试中实现了更高的整体性能。

Title: Tool Learning with Large Language Models: A Survey

• Authors: Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, Ji-Rong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.17935
• Pdf URL: https://arxiv.org/pdf/2405.17935
• Copy Paste: [[2405.17935]] Tool Learning with Large Language Models: A Survey(https://arxiv.org/abs/2405.17935)
• Keywords: language model, llm
• Abstract: Recently, tool learning with large language models (LLMs) has emerged as a promising paradigm for augmenting the capabilities of LLMs to tackle highly complex problems. Despite growing attention and rapid advancements in this field, the existing literature remains fragmented and lacks systematic organization, posing barriers to entry for newcomers. This gap motivates us to conduct a comprehensive survey of existing works on tool learning with LLMs. In this survey, we focus on reviewing existing literature from the two primary aspects (1) why tool learning is beneficial and (2) how tool learning is implemented, enabling a comprehensive understanding of tool learning with LLMs. We first explore the "why" by reviewing both the benefits of tool integration and the inherent benefits of the tool learning paradigm from six specific aspects. In terms of "how", we systematically review the literature according to a taxonomy of four key stages in the tool learning workflow: task planning, tool selection, tool calling, and response generation. Additionally, we provide a detailed summary of existing benchmarks and evaluation methods, categorizing them according to their relevance to different stages. Finally, we discuss current challenges and outline potential future directions, aiming to inspire both researchers and industrial developers to further explore this emerging and promising area.
• 摘要：最近，使用大型语言模型 (LLM) 的工具学习已成为一种有前途的范例，可增强 LLM 解决高度复杂问题的能力。尽管该领域受到越来越多的关注和快速发展，但现有文献仍然支离破碎，缺乏系统组织，对新手构成了进入障碍。这一差距促使我们对现有的 LLM 工具学习研究进行全面调查。在本次调查中，我们重点从两个主要方面回顾现有文献 (1) 工具学习为何有益以及 (2) 工具学习如何实施，从而全面了解 LLM 工具学习。我们首先通过从六个具体方面回顾工具集成的好处和工具学习范式的固有好处来探索“为什么”。在“如何”方面，我们根据工具学习工作流程中的四个关键阶段的分类系统地回顾文献：任务规划、工具选择、工具调用和响应生成。此外，我们还提供了现有基准和评估方法的详细摘要，并根据它们与不同阶段的相关性对它们进行分类。最后，我们讨论了当前的挑战并概述了未来的潜在方向，旨在激励研究人员和工业开发人员进一步探索这一新兴且有前景的领域。

Title: Knowledge Circuits in Pretrained Transformers

• Authors: Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.CV, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.17969
• Pdf URL: https://arxiv.org/pdf/2405.17969
• Copy Paste: [[2405.17969]] Knowledge Circuits in Pretrained Transformers(https://arxiv.org/abs/2405.17969)
• Keywords: language model, gpt, hallucination
• Abstract: The remarkable capabilities of modern large language models are rooted in their vast repositories of knowledge encoded within their parameters, enabling them to perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investigation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. In this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and TinyLLAMA, has allowed us to observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within the model. Moreover, we evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, we utilize knowledge circuits to analyze and interpret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuit holds potential for advancing our understanding of Transformers and guiding the improved design of knowledge editing. Code and data are available in this https URL.
• 摘要：现代大型语言模型的卓越能力源于其参数中编码的大量知识库，使它们能够感知世界并进行推理。这些模型如何存储知识的内部工作原理长期以来一直是研究人员强烈关注和研究的主题。迄今为止，大多数研究都集中在这些模型中的孤立组件上，例如多层感知器和注意力头。在本文中，我们深入研究语言模型的计算图，以揭示有助于表达特定知识的知识回路。使用 GPT2 和 TinyLLAMA 进行的实验使我们能够观察到某些信息头、关系头和多层感知器如何在模型中协作编码知识。此外，我们评估了当前知识编辑技术对这些知识回路的影响，从而更深入地了解这些编辑方法的功能和约束。最后，我们利用知识回路来分析和解释语言模型行为，例如幻觉和情境学习。我们相信知识回路有潜力增进我们对 Transformer 的理解并指导知识编辑的改进设计。代码和数据可在此 https URL 中获取。

Title: Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations

• Authors: Yi-Pei Chen, Noriki Nishida, Hideki Nakayama, Yuji Matsumoto
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.17974
• Pdf URL: https://arxiv.org/pdf/2405.17974
• Copy Paste: [[2405.17974]] Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations(https://arxiv.org/abs/2405.17974)
• Keywords: language model, llm, agent
• Abstract: Enhancing user engagement through personalization in conversational agents has gained significance, especially with the advent of large language models that generate fluent responses. Personalized dialogue generation, however, is multifaceted and varies in its definition -- ranging from instilling a persona in the agent to capturing users' explicit and implicit cues. This paper seeks to systemically survey the recent landscape of personalized dialogue generation, including the datasets employed, methodologies developed, and evaluation metrics applied. Covering 22 datasets, we highlight benchmark datasets and newer ones enriched with additional features. We further analyze 17 seminal works from top conferences between 2021-2023 and identify five distinct types of problems. We also shed light on recent progress by LLMs in personalized dialogue generation. Our evaluation section offers a comprehensive summary of assessment facets and metrics utilized in these works. In conclusion, we discuss prevailing challenges and envision prospect directions for future research in personalized dialogue generation.
• 摘要：通过对话代理的个性化来增强用户参与度已变得非常重要，尤其是随着能够生成流畅响应的大型语言模型的出现。然而，个性化对话生成是多方面的，其定义各不相同——从在代理中灌输角色到捕捉用户的显性和隐性线索。本文旨在系统地调查个性化对话生成的最新情况，包括所使用的数据集、开发的方法和应用的评估指标。我们涵盖 22 个数据集，重点介绍基准数据集和添加了附加特征的新数据集。我们进一步分析了 2021-2023 年间顶级会议上的 17 篇开创性作品，并确定了五种不同类型的问题。我们还阐明了 LLM 在个性化对话生成方面的最新进展。我们的评估部分全面总结了这些作品中使用的评估方面和指标。最后，我们讨论了当前面临的挑战，并展望了个性化对话生成未来研究的前景方向。

Title: Aligning to Thousands of Preferences via System Message Generalization

• Authors: Seongyun Lee, Sue Hyun Park, Seungone Kim, Minjoon Seo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17977
• Pdf URL: https://arxiv.org/pdf/2405.17977
• Copy Paste: [[2405.17977]] Aligning to Thousands of Preferences via System Message Generalization(https://arxiv.org/abs/2405.17977)
• Keywords: language model, gpt, llm, prompt
• Abstract: Although humans inherently have diverse values, current large language model (LLM) alignment methods often assume that aligning LLMs with the general public's preferences is optimal. A major challenge in adopting a more individualized approach to LLM alignment is its lack of scalability, as it involves repeatedly acquiring preference data and training new reward models and LLMs for each individual's preferences. To address these challenges, we propose a new paradigm where users specify what they value most within the system message, steering the LLM's generation behavior to better align with the user's intentions. However, a naive application of such an approach is non-trivial since LLMs are typically trained on a uniform system message (e.g., "You are a helpful assistant") which limits their ability to generalize to diverse, unseen system messages. To improve this generalization, we create the Multifaceted Collection, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions. Using this dataset, we train a 7B LLM called Janus and test it on 921 prompts from 5 benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, and Self-Instruct) by adding various unseen system messages that reflect user preferences. Janus achieves tie+win rate of 75.2%, 72.4%, and 66.4% against Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4, respectively. Unexpectedly, on three benchmarks focused on response helpfulness (AlpacaEval 2.0, MT-Bench, Arena Hard Auto v0.1), Janus also outperforms LLaMA 3 8B Instruct by a +4.0%, +0.1%, +3.0% margin, underscoring that training with a vast array of system messages could also enhance alignment to the general public's preference as well. Our code, dataset, benchmark, and models are available at this https URL.
• 摘要：尽管人类天生就具有多样化的价值观，但当前的大型语言模型 (LLM) 对齐方法通常假设将 LLM 与公众的偏好对齐是最佳选择。采用更加个性化的 LLM 对齐方法的主要挑战是其缺乏可扩展性，因为它涉及反复获取偏好数据并针对每个人的偏好训练新的奖励模型和 LLM。为了应对这些挑战，我们提出了一种新范式，其中用户在系统消息中指定他们最看重的内容，从而引导 LLM 的生成行为更好地与用户的意图保持一致。然而，这种方法的简单应用并不简单，因为 LLM 通常是在统一的系统消息（例如，“你是一个乐于助人的助手”）上进行训练的，这限制了它们推广到各种看不见的系统消息的能力。为了改进这种泛化，我们创建了多面集合，这是一个偏好数据集，除了通用的有用性和无害性之外，还有 192k 种值组合，涵盖 65k 条用户指令。利用该数据集，我们训练了一个名为 Janus 的 7B LLM，并通过添加各种反映用户偏好的未见系统消息，在 5 个基准测试（AlpacaEval 2.0、FLASK、Koala、MT-Bench 和 Self-Instruct）的 921 个提示上对其进行了测试。Janus 在与 Mistral 7B Instruct v0.2、GPT-3.5 Turbo 和 GPT-4 的比赛中分别取得了 75.2%、72.4% 和 66.4% 的平局+胜率。出乎意料的是，在三个关注响应有用性的基准测试（AlpacaEval 2.0、MT-Bench、Arena Hard Auto v0.1）中，Janus 的表现也比 LLaMA 3 8B Instruct 好 4.0%、0.1%、3.0%，这强调了使用大量系统消息进行训练也可以增强与公众偏好的一致性。我们的代码、数据集、基准测试和模型可在此 https URL 上找到。

Title: Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering

• Authors: Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, Balaji Vasan Srinivasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.17980
• Pdf URL: https://arxiv.org/pdf/2405.17980
• Copy Paste: [[2405.17980]] Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering(https://arxiv.org/abs/2405.17980)
• Keywords: language model, gpt, llm, prompt
• Abstract: With the enhancement in the field of generative artificial intelligence (AI), contextual question answering has become extremely relevant. Attributing model generations to the input source document is essential to ensure trustworthiness and reliability. We observe that when large language models (LLMs) are used for contextual question answering, the output answer often consists of text copied verbatim from the input prompt which is linked together with "glue text" generated by the LLM. Motivated by this, we propose that LLMs have an inherent awareness from where the text was copied, likely captured in the hidden states of the LLM. We introduce a novel method for attribution in contextual question answering, leveraging the hidden state representations of LLMs. Our approach bypasses the need for extensive model retraining and retrieval model overhead, offering granular attributions and preserving the quality of generated answers. Our experimental results demonstrate that our method performs on par or better than GPT-4 at identifying verbatim copied segments in LLM generations and in attributing these segments to their source. Importantly, our method shows robust performance across various LLM architectures, highlighting its broad applicability. Additionally, we present Verifiability-granular, an attribution dataset which has token level annotations for LLM generations in the contextual question answering setup.
• 摘要：随着生成式人工智能 (AI) 领域的增强，情境问答变得极为重要。将模型生成归因于输入源文档对于确保可信度和可靠性至关重要。我们观察到，当使用大型语言模型 (LLM) 进行情境问答时，输出答案通常由从输入提示逐字复制的文本组成，并与 LLM 生成的“粘合文本”链接在一起。受此启发，我们提出 LLM 具有固有的意识，可以知道文本是从哪里复制的，很可能在 LLM 的隐藏状态中捕获。我们介绍了一种用于情境问答归因的新方法，利用 LLM 的隐藏状态表示。我们的方法绕过了对大量模型再训练和检索模型开销的需求，提供了细粒度的归因并保持了生成的答案的质量。我们的实验结果表明，我们的方法在识别 LLM 生成中的逐字复制片段并将这些片段归因于其来源方面的表现与 GPT-4 相当或更好。重要的是，我们的方法在各种 LLM 架构中都表现出了强大的性能，凸显了其广泛的适用性。此外，我们还提供了可验证性粒度的归因数据集，该数据集具有上下文问答设置中 LLM 生成的标记级注释。

Title: fMRI predictors based on language models of increasing complexity recover brain left lateralization

• Authors: Laurent Bonnasse-Gahot, Christophe Pallier
• Subjects: cs.CL, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2405.17992
• Pdf URL: https://arxiv.org/pdf/2405.17992
• Copy Paste: [[2405.17992]] fMRI predictors based on language models of increasing complexity recover brain left lateralization(https://arxiv.org/abs/2405.17992)
• Keywords: language model
• Abstract: Over the past decade, studies of naturalistic language processing where participants are scanned while listening to continuous text have flourished. Using word embeddings at first, then large language models, researchers have created encoding models to analyze the brain signals. Presenting these models with the same text as the participants allows to identify brain areas where there is a significant correlation between the functional magnetic resonance imaging (fMRI) time series and the ones predicted by the models' artificial neurons. One intriguing finding from these studies is that they have revealed highly symmetric bilateral activation patterns, somewhat at odds with the well-known left lateralization of language processing. Here, we report analyses of an fMRI dataset where we manipulate the complexity of large language models, testing 28 pretrained models from 8 different families, ranging from 124M to 14.2B parameters. First, we observe that the performance of models in predicting brain responses follows a scaling law, where the fit with brain activity increases linearly with the logarithm of the number of parameters of the model (and its performance on natural language processing tasks). Second, we show that a left-right asymmetry gradually appears as model size increases, and that the difference in left-right brain correlations also follows a scaling law. Whereas the smallest models show no asymmetry, larger models fit better and better left hemispheric activations than right hemispheric ones. This finding reconciles computational analyses of brain activity using large language models with the classic observation from aphasic patients showing left hemisphere dominance for language.
• 摘要：在过去十年中，自然语言处理研究蓬勃发展，参与者在听连续文本的同时接受扫描。研究人员首先使用词嵌入，然后使用大型语言模型，创建了编码模型来分析大脑信号。向这些模型展示与参与者相同的文本，可以识别出功能性磁共振成像 (fMRI) 时间序列与模型的人工神经元预测的时间序列之间存在显著相关性的大脑区域。这些研究的一个有趣发现是，它们揭示了高度对称的双侧激活模式，这与众所周知的语言处理左侧化有些不一致。在这里，我们报告了对 fMRI 数据集的分析，其中我们操纵大型语言模型的复杂性，测试了来自 8 个不同系列的 28 个预训练模型，参数范围从 124M 到 14.2B。首先，我们观察到模型在预测大脑反应方面的表现遵循缩放定律，其中与大脑活动的拟合度随着模型参数数量的对数（及其在自然语言处理任务中的表现）线性增加。其次，我们表明，随着模型尺寸的增加，左右脑不对称现象逐渐出现，左右脑相关性的差异也遵循缩放定律。最小的模型没有不对称现象，而较大的模型对左半球激活的拟合效果越来越好，而对右半球激活的拟合效果则越来越好。这一发现将使用大型语言模型对大脑活动的计算分析与失语症患者的经典观察结果相一致，后者显示左半球在语言方面占主导地位。

Title: Exploring Context Window of Large Language Models via Decomposed Positional Vectors

• Authors: Zican Dong, Junyi Li, Xin Men, Wayne Xin Zhao, Bingbing Wang, Zhen Tian, Weipeng Chen, Ji-Rong Wen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.18009
• Pdf URL: https://arxiv.org/pdf/2405.18009
• Copy Paste: [[2405.18009]] Exploring Context Window of Large Language Models via Decomposed Positional Vectors(https://arxiv.org/abs/2405.18009)
• Keywords: language model, llm
• Abstract: Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.
• 摘要：基于 Transformer 的大型语言模型 (LLM) 通常具有有限的上下文窗口，当处理超出上下文窗口长度的文本时，性能会显著下降。已经有大量研究提出了扩展上下文窗口和实现 LLM 长度外推的研究，但对这些方法的深入解释仍然不足。在本研究中，我们探索上下文窗口内外的位置信息以揭示 LLM 的潜在机制。通过使用基于均值的分解方法，我们将位置向量从 LLM 的隐藏状态中分离出来，并分析它们的形成和对注意力的影响。此外，当文本超出上下文窗口时，我们在直接外推和上下文窗口扩展两种情况下分析了位置向量的变化。基于我们的研究结果，我们设计了两种无需训练的上下文窗口扩展方法，位置向量替换和注意力窗口扩展。实验结果表明我们的方法可以有效地延长上下文窗口长度。

Title: TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models

• Authors: Jaewoo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, Gunhee Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18027
• Pdf URL: https://arxiv.org/pdf/2405.18027
• Copy Paste: [[2405.18027]] TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models(https://arxiv.org/abs/2405.18027)
• Keywords: language model, gpt, llm, hallucination, agent
• Abstract: While Large Language Models (LLMs) can serve as agents to simulate human behaviors (i.e., role-playing agents), we emphasize the importance of point-in-time role-playing. This situates characters at specific moments in the narrative progression for three main reasons: (i) enhancing users' narrative immersion, (ii) avoiding spoilers, and (iii) fostering engagement in fandom role-playing. To accurately represent characters at specific time points, agents must avoid character hallucination, where they display knowledge that contradicts their characters' identities and historical timelines. We introduce TimeChara, a new benchmark designed to evaluate point-in-time character hallucination in role-playing LLMs. Comprising 10,895 instances generated through an automated pipeline, this benchmark reveals significant hallucination issues in current state-of-the-art LLMs (e.g., GPT-4o). To counter this challenge, we propose Narrative-Experts, a method that decomposes the reasoning steps and utilizes narrative experts to reduce point-in-time character hallucinations effectively. Still, our findings with TimeChara highlight the ongoing challenges of point-in-time character hallucination, calling for further study.
• 摘要：虽然大型语言模型 (LLM) 可以充当模拟人类行为的代理（即角色扮演代理），但我们强调了即时角色扮演的重要性。这将角色置于叙事进程中的特定时刻，主要有三个原因：(i) 增强用户的叙事沉浸感，(ii) 避免剧透，以及 (iii) 促进粉丝角色扮演的参与度。为了准确地表示特定时间点的角色，代理必须避免角色幻觉，即他们所显示的知识与其角色的身份和历史时间线相矛盾。我们推出了 TimeChara，这是一个新的基准，旨在评估角色扮演 LLM 中即时角色幻觉。该基准包含通过自动化管道生成的 10,895 个实例，揭示了当前最先进的 LLM（例如 GPT-4o）中存在严重的幻觉问题。为了应对这一挑战，我们提出了 Narrative-Experts，这种方法可以分解推理步骤并利用叙事专家来有效减少时间点人物幻觉。不过，我们对 TimeChara 的研究结果凸显了时间点人物幻觉的持续挑战，需要进一步研究。

Title: Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints

• Authors: Aryo Pradipta Gema, Chaeeun Lee, Pasquale Minervini, Luke Daines, T. Ian Simpson, Beatrice Alex
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.18028
• Pdf URL: https://arxiv.org/pdf/2405.18028
• Copy Paste: [[2405.18028]] Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints(https://arxiv.org/abs/2405.18028)
• Keywords: language model, gpt, llm, prompt
• Abstract: The MEDIQA-CORR 2024 shared task aims to assess the ability of Large Language Models (LLMs) to identify and correct medical errors in clinical notes. In this study, we evaluate the capability of general LLMs, specifically GPT-3.5 and GPT-4, to identify and correct medical errors with multiple prompting strategies. Recognising the limitation of LLMs in generating accurate corrections only via prompting strategies, we propose incorporating error-span predictions from a smaller, fine-tuned model in two ways: 1) by presenting it as a hint in the prompt and 2) by framing it as multiple-choice questions from which the LLM can choose the best correction. We found that our proposed prompting strategies significantly improve the LLM's ability to generate corrections. Our best-performing solution with 8-shot + CoT + hints ranked sixth in the shared task leaderboard. Additionally, our comprehensive analyses show the impact of the location of the error sentence, the prompted role, and the position of the multiple-choice option on the accuracy of the LLM. This prompts further questions about the readiness of LLM to be implemented in real-world clinical settings.
• 摘要：MEDIQA-CORR 2024 共享任务旨在评估大型语言模型 (LLM) 识别和纠正临床笔记中的医疗错误的能力。在本研究中，我们评估了通用 LLM（特别是 GPT-3.5 和 GPT-4）使用多种提示策略识别和纠正医疗错误的能力。认识到 LLM 仅通过提示策略生成准确更正的局限性，我们建议以两种方式结合较小、经过微调的模型中的错误跨度预测：1) 通过在提示中将其呈现为提示；2) 将其设计为多项选择题，LLM 可以从中选择最佳更正。我们发现我们提出的提示策略显著提高了 LLM 生成更正的能力。我们表现最佳的解决方案是 8 次 + CoT + 提示，在共享任务排行榜上排名第六。此外，我们的综合分析显示了错误句子的位置、提示角色和多项选择选项的位置对 LLM 准确性的影响。这引发了人们对 LLM 是否已准备好在现实临床环境中实施的进一步质疑。

Title: Instruction Tuning with Retrieval-based Examples Ranking for Aspect-based Sentiment Analysis

• Authors: Guangmin Zheng, Jin Wang, Liang-Chih Yu, Xuejie Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18035
• Pdf URL: https://arxiv.org/pdf/2405.18035
• Copy Paste: [[2405.18035]] Instruction Tuning with Retrieval-based Examples Ranking for Aspect-based Sentiment Analysis(https://arxiv.org/abs/2405.18035)
• Keywords: language model, prompt
• Abstract: Aspect-based sentiment analysis (ABSA) identifies sentiment information related to specific aspects and provides deeper market insights to businesses and organizations. With the emergence of large language models (LMs), recent studies have proposed using fixed examples for instruction tuning to reformulate ABSA as a generation task. However, the performance is sensitive to the selection of in-context examples; several retrieval methods are based on surface similarity and are independent of the LM generative objective. This study proposes an instruction learning method with retrieval-based example ranking for ABSA tasks. For each target sample, an LM was applied as a scorer to estimate the likelihood of the output given the input and a candidate example as the prompt, and training examples were labeled as positive or negative by ranking the scores. An alternating training schema is proposed to train both the retriever and LM. Instructional prompts can be constructed using high-quality examples. The LM is used for both scoring and inference, improving the generation efficiency without incurring additional computational costs or training difficulties. Extensive experiments on three ABSA subtasks verified the effectiveness of the proposed method, demonstrating its superiority over various strong baseline models. Code and data are released at https://anonymous.4open.science/r/IT-RER-ABSA-181F.
• 摘要：基于方面的情绪分析 (ABSA) 可识别与特定方面相关的情绪信息，并为企业和组织提供更深入的市场洞察。随着大型语言模型 (LM) 的出现，最近的研究提出使用固定示例进行指令调整，以将 ABSA 重新表述为生成任务。然而，性能对上下文示例的选择很敏感；几种检索方法基于表面相似性，与 LM 生成目标无关。本研究提出了一种基于检索的示例排名的 ABSA 任务指令学习方法。对于每个目标样本，LM 被用作评分器来估计给定输入和候选示例作为提示的输出的可能性，并且通过对分数进行排名将训练示例标记为正面或负面。提出了一种交替训练方案来训练检索器和 LM。可以使用高质量示例构建指令提示。LM 用于评分和推理，提高了生成效率，而不会产生额外的计算成本或训练困难。在三个 ABSA 子任务上进行的大量实验验证了所提方法的有效性，证明了其优于各种强基线模型。代码和数据发布于 https://anonymous.4open.science/r/IT-RER-ABSA-181F。

Title: ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator

• Authors: Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, Lei Sha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18111
• Pdf URL: https://arxiv.org/pdf/2405.18111
• Copy Paste: [[2405.18111]] ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator(https://arxiv.org/abs/2405.18111)
• Keywords: language model, llm, hallucination, retrieval-augmented generation, agent
• Abstract: Large language model (LLM) has proven to benefit a lot from retrieval augmentation in alleviating hallucinations confronted with knowledge-intensive questions. Retrieval-augmented generation (RAG) adopts IR-based techniques utilizing semantic-relevant documents as the generator's input context and realizes external knowledge injection. However, on today's Internet which is flooded with content generated by LLMs, there are too many "related yet useless" documents or even fake knowledge fabricated by LLMs, which will introduce extra noise to the generator and distract it from giving correct results. To this end, we regard the training of the RAG generator model as a multi-agent adversarial-defensive system, guiding the generator to have a better taste of whether a specific document helps answer the question through the Adversarial Tuning in a Multi-agent (ATM) system to strengthen the generator's robustness in an RAG pipeline. After rounds of multi-agent iterative tuning, we find that the ATM Generator can eventually discriminate useful documents amongst LLM fabrications and achieve better performance than strong baselines.
• 摘要：大型语言模型（LLM）已被证明可以从检索增强中获益匪浅，有助于缓解面对知识密集型问题时的幻觉。检索增强生成（RAG）采用基于 IR 的技术，利用语义相关文档作为生成器的输入上下文，实现外部知识注入。然而，在当今充斥着 LLM 生成内容的互联网上，有太多“相关但无用”的文档甚至 LLM 编造的虚假知识，这会给生成器带来额外的噪音，使其无法给出正确的结果。为此，我们将 RAG 生成器模型的训练视为一个多智能体对抗防御系统，通过多智能体系统中的对抗调优（ATM）引导生成器更好地了解特定文档是否有助于回答问题，以增强生成器在 RAG 管道中的鲁棒性。经过多轮多智能体迭代调整，我们发现 ATM 生成器最终可以在 LLM 制作中区分出有用的文档，并取得比强基线更好的性能。

Title: Facilitating Multi-Role and Multi-Behavior Collaboration of Large Language Models for Online Job Seeking and Recruiting

• Authors: Hongda Sun, Hongzhan Lin, Haiyu Yan, Chen Zhu, Yang Song, Xin Gao, Shuo Shang, Rui Yan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.18113
• Pdf URL: https://arxiv.org/pdf/2405.18113
• Copy Paste: [[2405.18113]] Facilitating Multi-Role and Multi-Behavior Collaboration of Large Language Models for Online Job Seeking and Recruiting(https://arxiv.org/abs/2405.18113)
• Keywords: language model, llm, prompt, agent
• Abstract: The emergence of online recruitment services has revolutionized the traditional landscape of job seeking and recruitment, necessitating the development of high-quality industrial applications to improve person-job fitting. Existing methods generally rely on modeling the latent semantics of resumes and job descriptions and learning a matching function between them. Inspired by the powerful role-playing capabilities of Large Language Models (LLMs), we propose to introduce a mock interview process between LLM-played interviewers and candidates. The mock interview conversations can provide additional evidence for candidate evaluation, thereby augmenting traditional person-job fitting based solely on resumes and job descriptions. However, characterizing these two roles in online recruitment still presents several challenges, such as developing the skills to raise interview questions, formulating appropriate answers, and evaluating two-sided fitness. To this end, we propose MockLLM, a novel applicable framework that divides the person-job matching process into two modules: mock interview generation and two-sided evaluation in handshake protocol, jointly enhancing their performance through collaborative behaviors between interviewers and candidates. We design a role-playing framework as a multi-role and multi-behavior paradigm to enable a single LLM agent to effectively behave with multiple functions for both parties. Moreover, we propose reflection memory generation and dynamic prompt modification techniques to refine the behaviors of both sides, enabling continuous optimization of the augmented additional evidence. Extensive experimental results show that MockLLM can achieve the best performance on person-job matching accompanied by high mock interview quality, envisioning its emerging application in real online recruitment in the future.
• 摘要：在线招聘服务的出现彻底改变了传统的求职和招聘格局，需要开发高质量的工业应用来改善人与工作的匹配度。现有的方法通常依赖于对简历和职位描述的潜在语义进行建模，并学习它们之间的匹配函数。受大型语言模型 (LLM) 强大的角色扮演功能的启发，我们建议引入 LLM 扮演的面试官和候选人之间的模拟面试流程。模拟面试对话可以为候选人评估提供额外的证据，从而增强传统的仅基于简历和职位描述的人与工作的匹配度。然而，在在线招聘中描述这两个角色仍然存在一些挑战，例如培养提出面试问题、制定适当答案和评估双边适应度的技能。为此，我们提出了 MockLLM，这是一个新颖的适用框架，它将人与工作的匹配过程分为两个模块：握手协议中的模拟面试生成和双边评估，通过面试官和候选人之间的协作行为共同提高他们的绩效。我们设计了一个角色扮演框架，作为多角色和多行为范式，使单个 LLM 代理能够有效地为双方执行多种功能。此外，我们提出了反射内存生成和动态提示修改技术来细化双方的行为，从而能够不断优化增强的附加证据。大量实验结果表明，MockLLM 可以在人机匹配方面实现最佳性能，同时具有较高的模拟面试质量，并有望在未来的实际在线招聘中得到应用。

Title: IAPT: Instruction-Aware Prompt Tuning for Large Language Models

• Authors: Wei Zhu, Aaron Xuxiang Tian, Congrui Yin, Yuan Ni, Xiaoling Wang, Guotong Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18203
• Pdf URL: https://arxiv.org/pdf/2405.18203
• Copy Paste: [[2405.18203]] IAPT: Instruction-Aware Prompt Tuning for Large Language Models(https://arxiv.org/abs/2405.18203)
• Keywords: language model, llm, prompt
• Abstract: Soft prompt tuning is a widely studied parameter-efficient fine-tuning method. However, it has a clear drawback: many soft tokens must be inserted into the input sequences to guarantee downstream performance. As a result, soft prompt tuning is less considered than Low-rank adaptation (LoRA) in the large language modeling (LLM) era. In this work, we propose a novel prompt tuning method, Instruction-Aware Prompt Tuning (IAPT), that requires only four soft tokens. First, we install a parameter-efficient soft prompt generator at each Transformer layer to generate idiosyncratic soft prompts for each input instruction. The generated soft prompts can be seen as a semantic summary of the input instructions and can effectively guide the output generation. Second, the soft prompt generators are modules with a bottleneck architecture consisting of a self-attention pooling operation, two linear projections, and an activation function. Pilot experiments show that prompt generators at different Transformer layers require different activation functions. Thus, we propose to learn the idiosyncratic activation functions for prompt generators automatically with the help of rational functions. We have conducted experiments on various tasks, and the experimental results demonstrate that (a) our IAPT method can outperform the recent baselines with comparable tunable parameters. (b) Our IAPT method is more efficient than LoRA under the single-backbone multi-tenant setting.
• 摘要：软提示调优是一种被广泛研究的参数高效的微调方法。然而，它有一个明显的缺点：必须在输入序列中插入许多软标记才能保证下游性能。因此，在大型语言建模 (LLM) 时代，软提示调优比低秩自适应 (LoRA) 受到的关注较少。在这项工作中，我们提出了一种新颖的提示调优方法，即指令感知提示调优 (IAPT)，它只需要四个软标记。首先，我们在每个 Transformer 层安装一个参数高效的软提示生成器，为每个输入指令生成独特的软提示。生成的软提示可以看作是输入指令的语义摘要，可以有效地指导输出生成。其次，软提示生成器是具有瓶颈架构的模块，由自注意力池操作、两个线性投影和一个激活函数组成。初步实验表明，不同 Transformer 层的提示生成器需要不同的激活函数。因此，我们提出借助有理函数自动学习提示生成器的独特激活函数。我们已经对各种任务进行了实验，实验结果表明：（a）我们的 IAPT 方法可以胜过具有可比可调参数的最新基线。（b）我们的 IAPT 方法在单主干多租户设置下比 LoRA 更有效。

Title: Active Use of Latent Constituency Representation in both Humans and Large Language Models

• Authors: Wei Liu, Ming Xiang, Nai Ding
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.18241
• Pdf URL: https://arxiv.org/pdf/2405.18241
• Copy Paste: [[2405.18241]] Active Use of Latent Constituency Representation in both Humans and Large Language Models(https://arxiv.org/abs/2405.18241)
• Keywords: language model, gpt, llm, chat
• Abstract: Understanding how sentences are internally represented in the human brain, as well as in large language models (LLMs) such as ChatGPT, is a major challenge for cognitive science. Classic linguistic theories propose that the brain represents a sentence by parsing it into hierarchically organized constituents. In contrast, LLMs do not explicitly parse linguistic constituents and their latent representations remains poorly explained. Here, we demonstrate that humans and LLMs construct similar latent representations of hierarchical linguistic constituents by analyzing their behaviors during a novel one-shot learning task, in which they infer which words should be deleted from a sentence. Both humans and LLMs tend to delete a constituent, instead of a nonconstituent word string. In contrast, a naive sequence processing model that has access to word properties and ordinal positions does not show this property. Based on the word deletion behaviors, we can reconstruct the latent constituency tree representation of a sentence for both humans and LLMs. These results demonstrate that a latent tree-structured constituency representation can emerge in both the human brain and LLMs.
• 摘要：了解句子在人类大脑以及 ChatGPT 等大型语言模型 (LLM) 中的内部表示方式是认知科学面临的一大挑战。经典语言理论认为，大脑通过将句子解析为分层组织的成分来表示句子。相比之下，LLM 不会明确解析语言成分，其潜在表示仍未得到很好的解释。在这里，我们通过分析人类和 LLM 在一项新颖的一次性学习任务中的行为来证明人类和 LLM 构建了类似的分层语言成分潜在表示，在这个任务中，他们推断哪些单词应该从句子中删除。人类和 LLM 都倾向于删除成分，而不是非成分字符串。相比之下，可以访问单词属性和序数位置的简单序列处理模型不会显示此属性。基于单词删除行为，我们可以为人类和 LLM 重建句子的潜在成分树表示。这些结果表明，潜在的树结构成分表征可以在人类大脑和 LLM 中出现。

Title: Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning

• Authors: Renzhi Wang, Piji Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18292
• Pdf URL: https://arxiv.org/pdf/2405.18292
• Copy Paste: [[2405.18292]] Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning(https://arxiv.org/abs/2405.18292)
• Keywords: language model, llm
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of Large Language Models (LLMs) to various downstream applications. However, the effectiveness of the PEFT diminishes notably when downstream tasks require accurate learning of factual knowledge. In this paper, we adopt a semantic perspective to investigate this phenomenon, uncovering the reasons behind PEFT's limitations in knowledge learning task. Our findings reveal that: (1) PEFT presents a notable risk of pushing the model away from the intended knowledge target; (2) multiple knowledge interfere with each other, and such interference suppresses the learning and expression of knowledge features. Based on these insights, we introduce a data filtering strategy to exclude data that is detrimental to knowledge learning and a re-weighted learning strategy to make the model attentive to semantic distance during knowledge learning. Experimental results demonstrate the effectiveness of the proposed method on open-source large language model, further validate the semantic challenge in PEFT, thus paving the way for future research.
• 摘要：参数高效微调 (PEFT) 方法能够使大型语言模型 (LLM) 高效适应各种下游应用。然而，当下游任务需要准确学习事实知识时，PEFT 的有效性会显著降低。在本文中，我们采用语义视角来研究这一现象，揭示 PEFT 在知识学习任务中局限性的原因。我们的研究结果表明：（1）PEFT 存在将模型推离预期知识目标的显著风险；（2）多种知识相互干扰，这种干扰抑制了知识特征的学习和表达。基于这些见解，我们引入了一种数据过滤策略来排除不利于知识学习的数据，并引入了一种重新加权学习策略，使模型在知识学习过程中注意语义距离。实验结果证明了所提方法在开源大型语言模型上的有效性，进一步验证了 PEFT 中的语义挑战，为未来的研究铺平了道路。

Title: The Battle of LLMs: A Comparative Study in Conversational QA Tasks

• Authors: Aryan Rangapur, Aman Rangapur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.18344
• Pdf URL: https://arxiv.org/pdf/2405.18344
• Copy Paste: [[2405.18344]] The Battle of LLMs: A Comparative Study in Conversational QA Tasks(https://arxiv.org/abs/2405.18344)
• Keywords: language model, gpt, llm, chat
• Abstract: Large language models have gained considerable interest for their impressive performance on various tasks. Within this domain, ChatGPT and GPT-4, developed by OpenAI, and the Gemini, developed by Google, have emerged as particularly popular among early adopters. Additionally, Mixtral by Mistral AI and Claude by Anthropic are newly released, further expanding the landscape of advanced language models. These models are viewed as disruptive technologies with applications spanning customer service, education, healthcare, and finance. More recently, Mistral has entered the scene, captivating users with its unique ability to generate creative content. Understanding the perspectives of these users is crucial, as they can offer valuable insights into the potential strengths, weaknesses, and overall success or failure of these technologies in various domains. This research delves into the responses generated by ChatGPT, GPT-4, Gemini, Mixtral and Claude across different Conversational QA corpora. Evaluation scores were meticulously computed and subsequently compared to ascertain the overall performance of these models. Our study pinpointed instances where these models provided inaccurate answers to questions, offering insights into potential areas where they might be susceptible to errors. In essence, this research provides a comprehensive comparison and evaluation of these state of-the-art language models, shedding light on their capabilities while also highlighting potential areas for improvement
• 摘要：大型语言模型因其在各种任务上的出色表现而引起了人们的极大兴趣。在这个领域，OpenAI 开发的 ChatGPT 和 GPT-4 以及 Google 开发的 Gemini 在早期采用者中特别受欢迎。此外，Mistral AI 的 Mixtral 和 Anthropic 的 Claude 也新近发布，进一步扩大了高级语言模型的领域。这些模型被视为颠覆性技术，应用范围涵盖客户服务、教育、医疗保健和金融。最近，Mistral 进入了这一领域，以其独特的创意内容生成能力吸引了用户。了解这些用户的观点至关重要，因为他们可以提供有价值的见解，了解这些技术在各个领域的潜在优势、劣势以及整体成败。这项研究深入研究了 ChatGPT、GPT-4、Gemini、Mixtral 和 Claude 在不同的对话式问答语料库中生成的响应。评估分数经过精心计算，随后进行比较，以确定这些模型的整体性能。我们的研究指出了这些模型对问题提供不准确答案的情况，并深入了解了它们可能容易出错的潜在领域。本质上，这项研究对这些最先进的语言模型进行了全面的比较和评估，揭示了它们的能力，同时也突出了潜在的改进领域

Title: Faithful Logical Reasoning via Symbolic Chain-of-Thought

• Authors: Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, Wynne Hsu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18357
• Pdf URL: https://arxiv.org/pdf/2405.18357
• Copy Paste: [[2405.18357]] Faithful Logical Reasoning via Symbolic Chain-of-Thought(https://arxiv.org/abs/2405.18357)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rigid deducing rules. To strengthen the logical reasoning capability of LLMs, we propose a novel Symbolic Chain-of-Thought, namely SymbCoT, a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting. Technically, building upon an LLM, SymbCoT 1) first translates the natural language context into the symbolic format, and then 2) derives a step-by-step plan to solve the problem with symbolic logical rules, 3) followed by a verifier to check the translation and reasoning chain. Via thorough evaluations on 5 standard datasets with both First-Order Logic and Constraint Optimization symbolic expressions, SymbCoT shows striking improvements over the CoT method consistently, meanwhile refreshing the current state-of-the-art performances. We further demonstrate that our system advances in more faithful, flexible, and explainable logical reasoning. To our knowledge, this is the first to combine symbolic expressions and rules into CoT for logical reasoning with LLMs. Code is open at this https URL.
• 摘要：虽然最近的思维链 (CoT) 技术利用心智理论增强了大型语言模型 (LLM) 的推理能力，但它在处理严重依赖符号表达式和严格推理规则的逻辑推理方面可能仍存在困难。为了增强 LLM 的逻辑推理能力，我们提出了一种新的符号思维链，即 SymbCoT，这是一个完全基于 LLM 的框架，它将符号表达式和逻辑规则与 CoT 提示相结合。从技术上讲，在 LLM 的基础上，SymbCoT 1) 首先将自然语言上下文转换为符号格式，然后 2) 得出使用符号逻辑规则解决问题的分步计划，3) 然后由验证器检查翻译和推理链。通过对 5 个标准数据集进行全面评估，包括一阶逻辑和约束优化符号表达式，SymbCoT 始终显示出比 CoT 方法显著的改进，同时刷新了当前最先进的性能。我们进一步证明，我们的系统在更忠实、更灵活、更易于解释的逻辑推理方面取得了进步。据我们所知，这是首次将符号表达式和规则结合到 CoT 中，用于 LLM 逻辑推理。代码在此 https URL 中开放。

Title: MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning

• Authors: Somnath Kumar, Yash Gadhia, Tanuja Ganu, Akshay Nambi
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.18358
• Pdf URL: https://arxiv.org/pdf/2405.18358
• Copy Paste: [[2405.18358]] MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning(https://arxiv.org/abs/2405.18358)
• Keywords: language model, llm, agent
• Abstract: Recent advancements in Multi-modal Large Language Models (MLLMs) have significantly improved their performance in tasks combining vision and language. However, challenges persist in detailed multi-modal understanding, comprehension of complex tasks, and reasoning over multi-modal information. This paper introduces MMCTAgent, a novel multi-modal critical thinking agent framework designed to address the inherent limitations of current MLLMs in complex visual reasoning tasks. Inspired by human cognitive processes and critical thinking, MMCTAgent iteratively analyzes multi-modal information, decomposes queries, plans strategies, and dynamically evolves its reasoning. Additionally, MMCTAgent incorporates critical thinking elements such as verification of final answers and self-reflection through a novel approach that defines a vision-based critic and identifies task-specific evaluation criteria, thereby enhancing its decision-making abilities. Through rigorous evaluations across various image and video understanding benchmarks, we demonstrate that MMCTAgent (with and without the critic) outperforms both foundational MLLMs and other tool-augmented pipelines.
• 摘要：多模态大型语言模型 (MLLM) 的最新进展显著提高了它们在结合视觉和语言的任务中的表现。然而，在详细的多模态理解、复杂任务的理解和多模态信息推理方面仍然存在挑战。本文介绍了 MMCTAgent，这是一种新型的多模态批判性思维代理框架，旨在解决当前 MLLM 在复杂视觉推理任务中的固有局限性。受人类认知过程和批判性思维的启发，MMCTAgent 迭代分析多模态信息、分解查询、规划策略并动态发展其推理。此外，MMCTAgent 通过一种新颖的方法结合了批判性思维元素，例如最终答案的验证和自我反思，该方法定义了基于视觉的批评者并确定了特定于任务的评估标准，从而增强了其决策能力。通过对各种图像和视频理解基准的严格评估，我们证明 MMCTAgent（有和没有批评者）的表现优于基础 MLLM 和其他工具增强管道。

Title: Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs

• Authors: Somnath Kumar, Vaibhav Balloli, Mercy Ranjit, Kabir Ahuja, Tanuja Ganu, Sunayana Sitaram, Kalika Bali, Akshay Nambi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.18359
• Pdf URL: https://arxiv.org/pdf/2405.18359
• Copy Paste: [[2405.18359]] Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs(https://arxiv.org/abs/2405.18359)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: Large language models (LLMs) are at the forefront of transforming numerous domains globally. However, their inclusivity and effectiveness remain limited for non-Latin scripts and low-resource languages. This paper tackles the imperative challenge of enhancing the multilingual performance of LLMs without extensive training or fine-tuning. Through systematic investigation and evaluation of diverse languages using popular question-answering (QA) datasets, we present novel techniques that unlock the true potential of LLMs in a polyglot landscape. Our approach encompasses three key strategies that yield significant improvements in multilingual proficiency. First, by meticulously optimizing prompts tailored for polyglot LLMs, we unlock their latent capabilities, resulting in substantial performance boosts across languages. Second, we introduce a new hybrid approach that synergizes LLM Retrieval Augmented Generation (RAG) with multilingual embeddings and achieves improved multilingual task performance. Finally, we introduce a novel learning approach that dynamically selects the optimal prompt strategy, LLM model, and embedding model per query at run-time. This dynamic adaptation maximizes the efficacy of LLMs across languages, outperforming best static and random strategies. Additionally, our approach adapts configurations in both offline and online settings, and can seamlessly adapt to new languages and datasets, leading to substantial advancements in multilingual understanding and generation across diverse languages.
• 摘要：大型语言模型 (LLM) 正处于全球众多领域变革的前沿。然而，对于非拉丁文字和资源匮乏的语言，它们的包容性和有效性仍然有限。本文解决了在无需大量训练或微调的情况下提高 LLM 多语言性能的迫切挑战。通过使用流行的问答 (QA) 数据集对多种语言进行系统调查和评估，我们提出了新颖的技术，以释放 LLM 在多语言环境中的真正潜力。我们的方法包含三种关键策略，可显着提高多语言能力。首先，通过精心优化针对多语言 LLM 量身定制的提示，我们释放了它们的潜在能力，从而显着提高了跨语言的性能。其次，我们引入了一种新的混合方法，将 LLM 检索增强生成 (RAG) 与多语言嵌入协同起来，并实现了改进的多语言任务性能。最后，我们介绍了一种新颖的学习方法，可在运行时动态选择每个查询的最佳提示策略、LLM 模型和嵌入模型。这种动态适应性可最大限度地提高 LLM 在各种语言中的效率，优于最佳静态和随机策略。此外，我们的方法可适应离线和在线环境中的配置，并可无缝适应新语言和数据集，从而显著提高跨多种语言的多语言理解和生成能力。

Title: PromptWizard: Task-Aware Agent-driven Prompt Optimization Framework

• Authors: Eshaan Agarwal, Vivek Dani, Tanuja Ganu, Akshay Nambi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.18369
• Pdf URL: https://arxiv.org/pdf/2405.18369
• Copy Paste: [[2405.18369]] PromptWizard: Task-Aware Agent-driven Prompt Optimization Framework(https://arxiv.org/abs/2405.18369)
• Keywords: language model, llm, prompt, agent
• Abstract: Large language models (LLMs) have revolutionized AI across diverse domains, showcasing remarkable capabilities. Central to their success is the concept of prompting, which guides model output generation. However, manual prompt engineering is labor-intensive and domain-specific, necessitating automated solutions. This paper introduces PromptWizard, a novel framework leveraging LLMs to iteratively synthesize and refine prompts tailored to specific tasks. Unlike existing approaches, PromptWizard optimizes both prompt instructions and in-context examples, maximizing model performance. The framework iteratively refines prompts by mutating instructions and incorporating negative examples to deepen understanding and ensure diversity. It further enhances both instructions and examples with the aid of a critic, synthesizing new instructions and examples enriched with detailed reasoning steps for optimal performance. PromptWizard offers several key features and capabilities, including computational efficiency compared to state-of-the-art approaches, adaptability to scenarios with varying amounts of training data, and effectiveness with smaller LLMs. Rigorous evaluation across 35 tasks on 8 datasets demonstrates PromptWizard's superiority over existing prompt strategies, showcasing its efficacy and scalability in prompt optimization.
• 摘要：大型语言模型 (LLM) 已经彻底改变了不同领域的人工智能，展示了非凡的能力。它们成功的核心是提示的概念，它指导模型输出的生成。然而，手动提示工程是劳动密集型的和特定于领域的，需要自动化解决方案。本文介绍了 PromptWizard，这是一个利用 LLM 迭代合成和优化针对特定任务的提示的新框架。与现有方法不同，PromptWizard 优化了提示指令和上下文示例，最大限度地提高了模型性能。该框架通过改变指令和合并反面例子来迭代优化提示，以加深理解并确保多样性。它在评论家的帮助下进一步增强了指令和示例，合成了新的指令和示例，并丰富了详细的推理步骤，以实现最佳性能。PromptWizard 提供了几个关键特性和能力，包括与最先进方法相比的计算效率、对具有不同数量训练数据的场景的适应性以及对较小 LLM 的有效性。对 8 个数据集上的 35 个任务进行的严格评估证明了 PromptWizard 优于现有的提示策略，展示了其在提示优化方面的有效性和可扩展性。

Title: Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning

• Authors: Phakphum Artkaew
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18375
• Pdf URL: https://arxiv.org/pdf/2405.18375
• Copy Paste: [[2405.18375]] Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning(https://arxiv.org/abs/2405.18375)
• Keywords: language model, gpt
• Abstract: Commonsense reasoning is one of the important aspect of natural language understanding, with several benchmarks developed to evaluate it. However, only a few of these benchmarks are available in languages other than English. Developing parallel benchmarks facilitates cross-lingual evaluation, enabling a better understanding of different languages. This research introduces a collection of Winograd Schemas in Thai, a novel dataset designed to evaluate commonsense reasoning capabilities in the context of the Thai language. Through a methodology involving native speakers, professional translators, and thorough validation, the schemas aim to closely reflect Thai language nuances, idioms, and cultural references while maintaining ambiguity and commonsense challenges. We evaluate the performance of popular large language models on this benchmark, revealing their strengths, limitations, and providing insights into the current state-of-the-art. Results indicate that while models like GPT-4 and Claude-3-Opus achieve high accuracy in English, their performance significantly drops in Thai, highlighting the need for further advancements in multilingual commonsense reasoning.
• 摘要：常识推理是自然语言理解的重要方面之一，已开发了多个基准来评估它。然而，这些基准中只有少数可用于英语以外的语言。开发并行基准有助于跨语言评估，从而更好地理解不同的语言。这项研究介绍了泰语中的一组 Winograd Schemas，这是一个新颖的数据集，旨在评估泰语背景下的常识推理能力。通过涉及母语人士、专业翻译人员和彻底验证的方法，这些模式旨在紧密反映泰语的细微差别、习语和文化参考，同时保持歧义和常识挑战。我们在这个基准上评估了流行的大型语言模型的性能，揭示了它们的优势和局限性，并提供了对当前最先进技术的见解。结果表明，虽然 GPT-4 和 Claude-3-Opus 等模型在英语中实现了高精度，但它们在泰语中的表现却显著下降，这凸显了在多语言常识推理方面需要进一步改进。

Title: Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass

• Authors: Ethan Shen, Alan Fan, Sarah M Pratt, Jae Sung Park, Matthew Wallingford, Sham M. Kakade, Ari Holtzman, Ranjay Krishna, Ali Farhadi, Aditya Kusupati
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.18400
• Pdf URL: https://arxiv.org/pdf/2405.18400
• Copy Paste: [[2405.18400]] Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass(https://arxiv.org/abs/2405.18400)
• Keywords: language model
• Abstract: Many applications today provide users with multiple auto-complete drafts as they type, including GitHub's code completion, Gmail's smart compose, and Apple's messaging auto-suggestions. Under the hood, language models support this by running an autoregressive inference pass to provide a draft. Consequently, providing $k$ drafts to the user requires running an expensive language model $k$ times. To alleviate the computation cost of running $k$ inference passes, we propose Superposed Decoding, a new decoding algorithm that generates $k$ drafts at the computation cost of one autoregressive inference pass. We achieve this by feeding a superposition of the $k$ most recent token embeddings from the drafts as input to the next decoding step of the language model. At every inference step we combine the $k$ drafts with the top-$k$ tokens to get $k^2$ new drafts and cache the $k$ most likely options, using an n-gram interpolation with minimal compute overhead to filter out incoherent generations. Our experiments show that $k$ drafts from Superposed Decoding are at least as coherent and factual as Nucleus Sampling and Greedy Decoding respectively, while being at least $2.44\times$ faster for $k\ge3$. In a compute-normalized setting, user evaluations demonstrably favor text generated by Superposed Decoding over Nucleus Sampling. Code and more examples open-sourced at this https URL.
• 摘要：如今，许多应用程序都会在用户输入时为他们提供多个自动完成的草稿，包括 GitHub 的代码完成、Gmail 的智能撰写和 Apple 的消息自动建议。在底层，语言模型通过运行自回归推理过程来提供草稿，从而支持这一点。因此，向用户提供 $k$ 个草稿需要运行昂贵的语言模型 $k$ 次。为了减轻运行 $k$ 次推理过程的计算成本，我们提出了叠加解码，这是一种新的解码算法，它以一次自回归推理过程的计算成本生成 $k$ 个草稿。我们通过将草稿中 $k$ 个最新的标记嵌入的叠加作为语言模型的下一个解码步骤的输入来实现这一点。在每个推理步骤中，我们将 $k$ 个草稿与前 $k$ 个标记相结合以获得 $k^2$ 个新草稿并缓存 $k$ 个最可能的选项，使用计算开销最小的 n-gram 插值来过滤掉不连贯的生成。我们的实验表明，叠加解码的 $k$ 草稿至少与核采样和贪婪解码一样连贯和真实，而对于 $k\ge3$，速度至少快 $2.44\times$。在计算标准化设置中，用户评价明显偏向叠加解码生成的文本，而不是核采样。代码和更多示例在此 https URL 上开源。

Title: Don't Forget to Connect! Improving RAG with Graph-based Reranking

• Authors: Jialin Dong, Bahare Fatemi, Bryan Perozzi, Lin F. Yang, Anton Tsitsulin
• Subjects: cs.CL, cs.AI, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Don't Forget to Connect! Improving RAG with Graph-based Reranking(https://arxiv.org/abs/)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) has greatly improved the performance of Large Language Model (LLM) responses by grounding generation with context from existing documents. These systems work well when documents are clearly relevant to a question context. But what about when a document has partial information, or less obvious connections to the context? And how should we reason about connections between documents? In this work, we seek to answer these two core questions about RAG generation. We introduce G-RAG, a reranker based on graph neural networks (GNNs) between the retriever and reader in RAG. Our method combines both connections between documents and semantic information (via Abstract Meaning Representation graphs) to provide a context-informed ranker for RAG. G-RAG outperforms state-of-the-art approaches while having smaller computational footprint. Additionally, we assess the performance of PaLM 2 as a reranker and find it to significantly underperform G-RAG. This result emphasizes the importance of reranking for RAG even when using Large Language Models.
• 摘要：检索增强生成 (RAG) 通过将生成与现有文档的上下文相结合，极大地提高了大型语言模型 (LLM) 响应的性能。当文档与问题上下文明显相关时，这些系统效果很好。但是，当文档包含部分信息或与上下文的联系不太明显时，情况会怎样呢？我们应该如何推断文档之间的联系？在这项工作中，我们试图回答有关 RAG 生成的这两个核心问题。我们引入了 G-RAG，这是一种基于 RAG 中检索器和阅读器之间的图神经网络 (GNN) 的重新排序器。我们的方法结合了文档之间的联系和语义信息（通过抽象含义表示图），为 RAG 提供基于上下文的排序器。G-RAG 的性能优于最先进的方法，同时计算占用空间更小。此外，我们评估了 PaLM 2 作为重新排序器的性能，发现它的性能明显低于 G-RAG。这一结果强调了即使在使用大型语言模型时重新排序对 RAG 的重要性。

Title: Notes on Applicability of GPT-4 to Document Understanding

• Authors: Łukasz Borchmann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.18433
• Pdf URL: https://arxiv.org/pdf/2405.18433
• Copy Paste: [[2405.18433]] Notes on Applicability of GPT-4 to Document Understanding(https://arxiv.org/abs/2405.18433)
• Keywords: gpt
• Abstract: We perform a missing, reproducible evaluation of all publicly available GPT-4 family models concerning the Document Understanding field, where it is frequently required to comprehend text spacial arrangement and visual clues in addition to textual semantics. Benchmark results indicate that though it is hard to achieve satisfactory results with text-only models, GPT-4 Vision Turbo performs well when one provides both text recognized by an external OCR engine and document images on the input. Evaluation is followed by analyses that suggest possible contamination of textual GPT-4 models and indicate the significant performance drop for lengthy documents.
• 摘要：我们对文档理解领域的所有公开可用的 GPT-4 系列模型进行了缺失的可重复评估，在这个领域，除了文本语义之外，还经常需要理解文本的空间排列和视觉线索。基准测试结果表明，尽管使用纯文本模型很难获得令人满意的结果，但当输入中同时提供外部 OCR 引擎识别的文本和文档图像时，GPT-4 Vision Turbo 表现良好。评估之后是分析，这些分析表明文本 GPT-4 模型可能受到污染，并表明长文档的性能显著下降。