2024-04-11

Title: Less is More for Improving Automatic Evaluation of Factual Consistency

• Authors: Tong Wang, Ninad Kulkarni, Yanjun Qi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06579
• Pdf URL: https://arxiv.org/pdf/2404.06579
• Copy Paste: [[2404.06579]] Less is More for Improving Automatic Evaluation of Factual Consistency(https://arxiv.org/abs/2404.06579)
• Keywords: language model, gpt, chat
• Abstract: Assessing the factual consistency of automatically generated texts in relation to source context is crucial for developing reliable natural language generation applications. Recent literature proposes AlignScore which uses a unified alignment model to evaluate factual consistency and substantially outperforms previous methods across many benchmark tasks. In this paper, we take a closer look of datasets used in AlignScore and uncover an unexpected finding: utilizing a smaller number of data points can actually improve performance. We process the original AlignScore training dataset to remove noise, augment with robustness-enhanced samples, and utilize a subset comprising 10\% of the data to train an improved factual consistency evaluation model, we call LIM-RA (Less Is More for Robust AlignScore). LIM-RA demonstrates superior performance, consistently outperforming AlignScore and other strong baselines like ChatGPT across four benchmarks (two utilizing traditional natural language generation datasets and two focused on large language model outputs). Our experiments show that LIM-RA achieves the highest score on 24 of the 33 test datasets, while staying competitive on the rest, establishing the new state-of-the-art benchmarks.
• 摘要：评估自动生成的文本相对于源上下文的事实一致性对于开发可靠的自然语言生成应用程序至关重要。最近的文献提出了 AlignScore，它使用统一的对齐模型来评估事实一致性，并且在许多基准任务中大大优于以前的方法。在本文中，我们仔细研究了 AlignScore 中使用的数据集，并发现了一个意外的发现：利用较少数量的数据点实际上可以提高性能。我们处理原始 AlignScore 训练数据集以消除噪声，使用鲁棒性增强的样本进行扩充，并利用包含 10% 数据的子集来训练改进的事实一致性评估模型，我们称之为 LIM-RA（Less Is More for Robust AlignScore） ）。 LIM-RA 在四个基准测试中表现出卓越的性能，始终优于 AlignScore 和 ChatGPT 等其他强大基准（两个基准测试使用传统的自然语言生成数据集，两个基准测试专注于大型语言模型输出）。我们的实验表明，LIM-RA 在 33 个测试数据集中的 24 个上取得了最高分，同时在其余数据集上保持竞争力，建立了新的最先进基准。

Title: FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations

• Authors: Jane Dwivedi-Yu, Raaz Dwivedi, Timo Schick
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06619
• Pdf URL: https://arxiv.org/pdf/2404.06619
• Copy Paste: [[2404.06619]] FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations(https://arxiv.org/abs/2404.06619)
• Keywords: language model
• Abstract: The accurate evaluation of differential treatment in language models to specific groups is critical to ensuring a positive and safe user experience. An ideal evaluation should have the properties of being robust, extendable to new groups or attributes, and being able to capture biases that appear in typical usage (rather than just extreme, rare cases). Relatedly, bias evaluation should surface not only egregious biases but also ones that are subtle and commonplace, such as a likelihood for talking about appearances with regard to women. We present FairPair, an evaluation framework for assessing differential treatment that occurs during ordinary usage. FairPair operates through counterfactual pairs, but crucially, the paired continuations are grounded in the same demographic group, which ensures equivalent comparison. Additionally, unlike prior work, our method factors in the inherent variability that comes from the generation process itself by measuring the sampling variability. We present an evaluation of several commonly used generative models and a qualitative analysis that indicates a preference for discussing family and hobbies with regard to women.
• 摘要：准确评估语言模型中针对特定群体的差异化处理对于确保积极且安全的用户体验至关重要。理想的评估应该具有鲁棒性、可扩展到新的组或属性，并且能够捕获典型用法中出现的偏差（而不仅仅是极端、罕见的情况）。与此相关的是，偏见评估不仅应该揭露严重的偏见，还应该揭露那些微妙和普遍的偏见，例如谈论女性外表的可能性。我们提出 FairPair，一个评估框架，用于评估日常使用过程中发生的差别待遇。 FairPair 通过反事实对进行操作，但最重要的是，配对的延续基于相同的人口统计群体，这确保了等效比较。此外，与之前的工作不同，我们的方法通过测量采样变异性来考虑生成过程本身的固有变异性。我们对几种常用的生成模型进行了评估，并进行了定性分析，表明女性更倾向于讨论家庭和爱好。

Title: What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models

• Authors: Jeongrok Yu, Seong Ug Kim, Jacob Choi, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06621
• Pdf URL: https://arxiv.org/pdf/2404.06621
• Copy Paste: [[2404.06621]] What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models(https://arxiv.org/abs/2404.06621)
• Keywords: language model
• Abstract: Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based Masked Language Models (MLMs) and their impact on many NLP tasks, a systematic evaluation of bias in these models is needed more than ever. While many studies have evaluated gender bias in English MLMs, only a few works have been conducted for the task in other languages. This paper proposes a multilingual approach to estimate gender bias in MLMs from 5 languages: Chinese, English, German, Portuguese, and Spanish. Unlike previous work, our approach does not depend on parallel corpora coupled with English to detect gender bias in other languages using multilingual lexicons. Moreover, a novel model-based method is presented to generate sentence pairs for a more robust analysis of gender bias, compared to the traditional lexicon-based method. For each language, both the lexicon-based and model-based methods are applied to create two datasets respectively, which are used to evaluate gender bias in an MLM specifically trained for that language using one existing and 3 new scoring metrics. Our results show that the previous approach is data-sensitive and not stable as it does not remove contextual dependencies irrelevant to gender. In fact, the results often flip when different scoring metrics are used on the same dataset, suggesting that gender bias should be studied on a large dataset using multiple evaluation metrics for best practice.
• 摘要：偏见是一种偏袒一方而反对另一方的不成比例的偏见。由于基于 Transformer 的屏蔽语言模型 (MLM) 的成功及其对许多 NLP 任务的影响，比以往任何时候都更需要对这些模型中的偏差进行系统评估。虽然许多研究评估了英语传销中的性别偏见，但只有少数研究用其他语言进行过这项任务。本文提出了一种多语言方法来估计 5 种语言的传销中的性别偏见：中文、英语、德语、葡萄牙语和西班牙语。与之前的工作不同，我们的方法不依赖于平行语料库和英语来使用多语言词典检测其他语言中的性别偏见。此外，与传统的基于词典的方法相比，提出了一种基于模型的新颖方法来生成句子对，以便对性别偏见进行更稳健的分析。对于每种语言，基于词典和基于模型的方法分别用于创建两个数据集，这些数据集用于使用一个现有的和 3 个新的评分指标来评估专门针对该语言训练的 MLM 中的性别偏见。我们的结果表明，以前的方法是数据敏感的并且不稳定，因为它没有消除与性别无关的上下文依赖性。事实上，当在同一数据集上使用不同的评分指标时，结果经常会发生翻转，这表明应使用多个评估指标在大型数据集上研究性别偏见，以实现最佳实践。

Title: Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

• Authors: Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06644
• Pdf URL: https://arxiv.org/pdf/2404.06644
• Copy Paste: [[2404.06644]] Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?(https://arxiv.org/abs/2404.06644)
• Keywords: language model, llm
• Abstract: Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Additionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages. In response to this necessity, we introduce Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc., aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various educational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such as human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use of original, non-translated data tailored for Persian speakers, ensuring the framework is free from translation challenges and errors while encompassing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked an evaluation framework that combined all of these features into a single comprehensive benchmark. Furthermore, we evaluate a wide range of existing LLMs that support the Persian language, with statistical analyses and interpretations of their outputs.
• 摘要：由于其生成性质，评估大型语言模型 (LLM) 具有挑战性，需要精确的评估方法。此外，非英语LLM的评估落后于英语，导致许多语言的LLM缺乏或薄弱。为了满足这一需求，我们推出了 Khayyam Challenge（也称为 PersianMMLU），这是一个精心策划的题库，包含 20,192 道四选题，这些题目来自波斯语考试中提取的 38 项不同任务，涵盖广泛的主题、复杂性和年龄。海亚姆挑战赛的主要目标是促进对支持波斯语的法学硕士的严格评估。 Khayyam挑战赛的显着特点是（i）全面覆盖各种主题，包括文学理解、数学、科学、逻辑、智力测试等，旨在评估法学硕士的语言理解、推理和信息检索等不同方面涵盖从小学到高中的各个教育阶段 (ii) 包含丰富的元数据，例如人类响应率、难度级别和描述性答案 (iii) 利用新数据来避免现有框架中普遍存在的数据污染问题(iv) 使用为波斯语使用者量身定制的原始非翻译数据，确保该框架不存在翻译挑战和错误，同时涵盖文化细微差别 (v) 其固有的可扩展性，可用于未来的数据更新和评估，无需特殊的人力。以前的工作缺乏将所有这些功能组合成一个综合基准的评估框架。此外，我们还评估了各种支持波斯语的现有法学硕士，并对它们的输出进行了统计分析和解释。

Title: RULER: What's the Real Context Size of Your Long-Context Language Models?

• Authors: Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Boris Ginsburg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06654
• Pdf URL: https://arxiv.org/pdf/2404.06654
• Copy Paste: [[2404.06654]] RULER: What's the Real Context Size of Your Long-Context Language Models?(https://arxiv.org/abs/2404.06654)
• Keywords: language model, gpt
• Abstract: The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate ten long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only four models (GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.
• 摘要：大海捞针（NIAH）测试检查从长干扰文本（“大海捞针”）中检索一条信息（“针”）的能力，已被广泛用于评估长上下文语言模型（LM）。然而，这种简单的基于检索的测试仅表明长上下文理解的表面形式。为了对长上下文 LM 提供更全面的评估，我们创建了一个新的综合基准标尺，该标尺具有灵活的配置，可定制序列长度和任务复杂性。 RULER 对普通 NIAH 测试进行了扩展，涵盖了不同类型和数量的针的变化。此外，RULER 引入了新的任务类别多跳跟踪和聚合来测试从上下文搜索之外的行为。我们在 RULER 中评估了 10 个长上下文 LM，其中包含 13 个代表性任务。尽管在普通 NIAH 测试中实现了近乎完美的准确性，但随着上下文长度的增加，所有模型都表现出大幅性能下降。虽然这些模型都声称上下文大小为 32K 令牌或更大，但只有四种模型（GPT-4、Command-R、Yi-34B 和 Mixtral）可以在 32K 长度下保持令人满意的性能。我们对支持 200K 上下文长度的 Yi-34B 的分析表明，随着输入长度和任务复杂性的增加，还有很大的改进空间。我们开源 RULER 来促进长上下文 LM 的综合评估。

Title: CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge

• Authors: Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06664
• Pdf URL: https://arxiv.org/pdf/2404.06664
• Copy Paste: [[2404.06664]] CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge(https://arxiv.org/abs/2404.06664)
• Keywords: language model, llm
• Abstract: Frontier large language models (LLMs) are developed by researchers and practitioners with skewed cultural backgrounds and on datasets with skewed sources. However, LLMs' (lack of) multicultural knowledge cannot be effectively assessed with current methods for developing benchmarks. Existing multicultural evaluations primarily rely on expensive and restricted human annotations or potentially outdated internet resources. Thus, they struggle to capture the intricacy, dynamics, and diversity of cultural norms. LLM-generated benchmarks are promising, yet risk propagating the same biases they are meant to measure. To synergize the creativity and expert cultural knowledge of human annotators and the scalability and standardizability of LLM-based automation, we introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build truly challenging evaluation dataset for assessing the multicultural knowledge of LLMs, while improving annotators' capabilities and experiences. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions, that modern LLMs fail at, in a gamified manner. Importantly, the increased level of AI assistance (e.g., LLM-generated revision hints) empowers users to create more difficult questions with enhanced perceived creativity of themselves, shedding light on the promises of involving heavier AI assistance in modern evaluation dataset creation procedures. Through a series of 1-hour workshop sessions, we gather CULTURALBENCH-V0.1, a compact yet high-quality evaluation dataset with users' red-teaming attempts, that different families of modern LLMs perform with accuracy ranging from 37.7% to 72.2%, revealing a notable gap in LLMs' multicultural proficiency.
• 摘要：前沿大语言模型（LLM）是由具有不同文化背景的研究人员和从业者在具有不同来源的数据集上开发的。然而，法学硕士（缺乏）多元文化知识无法通过当前制定基准的方法进行有效评估。现有的多元文化评估主要依赖于昂贵且受限的人工注释或可能过时的互联网资源。因此，他们努力捕捉文化规范的复杂性、动态性和多样性。 LLM 生成的基准很有前途，但也存在传播其旨在衡量的相同偏差的风险。为了协同人类注释者的创造力和专业文化知识以及基于法学硕士的自动化的可扩展性和标准化，我们引入了CulturalTeaming，这是一个交互式红队系统，利用人类与人工智能的协作来构建真正具有挑战性的评估数据集，用于评估法学硕士，同时提高注释者的能力和经验。我们的研究表明，CulturalTeaming 的各种人工智能辅助模式支持注释者以游戏化的方式提出现代法学硕士无法解决的文化问题。重要的是，人工智能辅助水平的提高（例如，法学硕士生成的修订提示）使用户能够通过增强的自身感知创造力来创建更困难的问题，从而揭示了在现代评估数据集创建程序中涉及更重的人工智能辅助的承诺。通过一系列时长一小时的研讨会，我们收集了 CULTURALBENCH-V0.1，这是一个紧凑但高质量的评估数据集，其中包含用户的红队尝试，现代法学硕士的不同系列的准确率在 37.7% 到 72.2% 之间，揭示了法学硕士多元文化能力的显着差距。

Title: Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology

• Authors: Shashi Kant Gupta, Aditya Basu, Bradley Taylor, Anai Kothari, Hrituraj Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06680
• Pdf URL: https://arxiv.org/pdf/2404.06680
• Copy Paste: [[2404.06680]] Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology(https://arxiv.org/abs/2404.06680)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieving information from EHR systems is essential for answering specific questions about patient journeys and improving the delivery of clinical care. Despite this fact, most EHR systems still rely on keyword-based searches. With the advent of generative large language models (LLMs), retrieving information can lead to better search and summarization capabilities. Such retrievers can also feed Retrieval-augmented generation (RAG) pipelines to answer any query. However, the task of retrieving information from EHR real-world clinical data contained within EHR systems in order to solve several downstream use cases is challenging due to the difficulty in creating query-document support pairs. We provide a blueprint for creating such datasets in an affordable manner using large language models. Our method results in a retriever that is 30-50 F-1 points better than propriety counterparts such as Ada and Mistral for oncology data elements. We further compare our model, called Onco-Retriever, against fine-tuned PubMedBERT model as well. We conduct an extensive manual evaluation on real-world EHR data along with latency analysis of the different models and provide a path forward for healthcare organizations to build domain-specific retrievers.
• 摘要：从 EHR 系统检索信息对于回答有关患者旅程的具体问题和改善临床护理的提供至关重要。尽管如此，大多数 EHR 系统仍然依赖基于关键字的搜索。随着生成式大语言模型 (LLM) 的出现，检索信息可以带来更好的搜索和摘要能力。此类检索器还可以提供检索增强生成（RAG）管道来回答任何查询。然而，由于创建查询文档支持对的困难，从 EHR 系统中包含的 EHR 现实世界临床数据中检索信息以解决多个下游用例的任务具有挑战性。我们提供了使用大型语言模型以经济实惠的方式创建此类数据集的蓝图。对于肿瘤学数据元素，我们的方法生成的检索器比 Ada 和 Mistral 等专有检索器好 30-50 F-1 点。我们进一步将我们的模型（称为 Onco-Retriever）与微调的 PubMedBERT 模型进行比较。我们对现实世界的 EHR 数据进行广泛的手动评估，并对不同模型进行延迟分析，并为医疗保健组织构建特定领域的检索器提供了一条前进的道路。

Title: CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers

• Authors: Longwei Zou, Qingyang Wang, Han Zhao, Jiangang Kong, Yi Yang, Yangdong Deng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06709
• Pdf URL: https://arxiv.org/pdf/2404.06709
• Copy Paste: [[2404.06709]] CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers(https://arxiv.org/abs/2404.06709)
• Keywords: language model
• Abstract: The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on the LLaMA-33B model, while maintaining a close level of performance.
• 摘要：快速增长的大规模语言模型在几乎所有自然语言处理任务上都提供了前所未有的性能。然而，大型语言模型的有效性依赖于呈指数增长的参数数量。巨大的计算复杂性会导致较高的推理延迟，从而对用户体验产生负面影响。现有的提高推理效率的方法，例如张量并行和量化，目标是减少每层计算延迟，但却忽略了由于层数而导致的累积延迟。然而，最近通过层删除来减少累积延迟的工作导致性能显着下降。受相邻层之间输入相似性的启发，我们建议识别准独立层，这些层可以同时计算以显着减少推理延迟。我们还引入了一种旁路技术来减轻信息丢失的影响。所提出的方法在 LLaMA 模型上的实证实验证实，准独立层并发计算 (CQIL) 可以在 LLaMA-33B 模型上减少高达 48.3% 的延迟，同时保持接近的性能水平。

Title: MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education

• Authors: Murong Yue, Wijdane Mifdal, Yixuan Zhang, Jennifer Suh, Ziyu Yao
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06711
• Pdf URL: https://arxiv.org/pdf/2404.06711
• Copy Paste: [[2404.06711]] MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education(https://arxiv.org/abs/2404.06711)
• Keywords: language model, llm
• Abstract: Mathematical modeling (MM) is considered a fundamental skill for students in STEM disciplines. Practicing the MM skill is often the most effective when students can engage in group discussion and collaborative problem-solving. However, due to unevenly distributed teachers and educational resources needed to monitor such group activities, students do not always receive equal opportunities for this practice. Excitingly, large language models (LLMs) have recently demonstrated strong capability in both modeling mathematical problems and simulating characters with different traits and properties. Drawing inspiration from the advancement of LLMs, in this work, we present MATHVC, the very first LLM-powered virtual classroom containing multiple LLM-simulated student characters, with whom a human student can practice their MM skill. To encourage each LLM character's behaviors to be aligned with their specified math-relevant properties (termed "characteristics alignment") and the overall conversational procedure to be close to an authentic student MM discussion (termed "conversational procedural alignment"), we proposed three innovations: integrating MM domain knowledge into the simulation, defining a symbolic schema as the ground for character simulation, and designing a meta planner at the platform level to drive the conversational procedure. Through experiments and ablation studies, we confirmed the effectiveness of our simulation approach and showed the promise for MATHVC to benefit real-life students in the future.
• 摘要：数学建模 (MM) 被认为是 STEM 学科学生的一项基本技能。当学生能够参与小组讨论和协作解决问题时，练习 MM 技能通常是最有效的。然而，由于监督此类团体活动所需的教师和教育资源分布不均，学生并不总是能获得平等的机会进行这种实践。令人兴奋的是，大型语言模型（LLM）最近在数学问题建模和模拟具有不同特征和属性的角色方面表现出了强大的能力。从法学硕士的进步中汲取灵感，在这项工作中，我们推出了 MATHVC，这是第一个由法学硕士支持的虚拟教室，其中包含多个法学硕士模拟的学生角色，人类学生可以与他们一起练习他们的 MM 技能。为了鼓励每个 LLM 角色的行为与其指定的数学相关属性保持一致（称为“特征对齐”），并使整体对话过程接近真实的学生 MM 讨论（称为“对话程序对齐”），我们提出了三项创新：将 MM 领域知识集成到模拟中，定义符号模式作为角色模拟的基础，并在平台级别设计元规划器来驱动对话过程。通过实验和消融研究，我们证实了模拟方法的有效性，并表明 MATHVC 有望在未来使现实生活中的学生受益。

Title: Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness

• Authors: Xincan Feng, Akifumi Yoshimoto
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.06714
• Pdf URL: https://arxiv.org/pdf/2404.06714
• Copy Paste: [[2404.06714]] Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness(https://arxiv.org/abs/2404.06714)
• Keywords: language model, llm
• Abstract: Recent advancements in Natural Language Processing (NLP) have seen Large-scale Language Models (LLMs) excel at producing high-quality text for various purposes. Notably, in Text-To-Speech (TTS) systems, the integration of BERT for semantic token generation has underscored the importance of semantic content in producing coherent speech outputs. Despite this, the specific utility of LLMs in enhancing TTS synthesis remains considerably limited. This research introduces an innovative approach, Llama-VITS, which enhances TTS synthesis by enriching the semantic content of text using LLM. Llama-VITS integrates semantic embeddings from Llama2 with the VITS model, a leading end-to-end TTS framework. By leveraging Llama2 for the primary speech synthesis process, our experiments demonstrate that Llama-VITS matches the naturalness of the original VITS (ORI-VITS) and those incorporate BERT (BERT-VITS), on the LJSpeech dataset, a substantial collection of neutral, clear speech. Moreover, our method significantly enhances emotive expressiveness on the EmoV_DB_bea_sem dataset, a curated selection of emotionally consistent speech from the EmoV_DB dataset, highlighting its potential to generate emotive speech.
• 摘要：自然语言处理 (NLP) 的最新进展表明，大规模语言模型 (LLM) 擅长为各种目的生成高质量文本。值得注意的是，在文本转语音 (TTS) 系统中，将 BERT 集成到语义标记生成中强调了语义内容在生成连贯语音输出中的重要性。尽管如此，法学硕士在增强 TTS 合成方面的具体效用仍然相当有限。本研究引入了一种创新方法 Llama-VITS，该方法通过使用 LLM 丰富文本的语义内容来增强 TTS 合成。 Llama-VITS 将 Llama2 的语义嵌入与领先的端到端 TTS 框架 VITS 模型集成。通过利用 Llama2 进行主要语音合成过程，我们的实验证明 Llama-VITS 与原始 VITS (ORI-VITS) 的自然度相匹配，并且在 LJSpeech 数据集上融入了 BERT (BERT-VITS)，LJSpeech 数据集是中性、清晰的言语。此外，我们的方法显着增强了 EmoV_DB_bea_sem 数据集的情感表达能力，该数据集是从 EmoV_DB 数据集中精心挑选的情感一致的语音，突出了其生成情感语音的潜力。

Title: Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking

• Authors: Xiaokang Zhang, Zijun Yao, Jing Zhang, Kaifeng Yun, Jifan Yu, Juanzi Li, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06742
• Pdf URL: https://arxiv.org/pdf/2404.06742
• Copy Paste: [[2404.06742]] Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking(https://arxiv.org/abs/2404.06742)
• Keywords: language model, llm
• Abstract: Detecting non-factual content is a longstanding goal to increase the trustworthiness of large language models (LLMs) generations. Current factuality probes, trained using humanannotated labels, exhibit limited transferability to out-of-distribution content, while online selfconsistency checking imposes extensive computation burden due to the necessity of generating multiple outputs. This paper proposes PINOSE, which trains a probing model on offline self-consistency checking results, thereby circumventing the need for human-annotated data and achieving transferability across diverse data distributions. As the consistency check process is offline, PINOSE reduces the computational burden of generating multiple responses by online consistency verification. Additionally, it examines various aspects of internal states prior to response decoding, contributing to more effective detection of factual inaccuracies. Experiment results on both factuality detection and question answering benchmarks show that PINOSE achieves surpassing results than existing factuality detection methods. Our code and datasets are publicly available on this anonymized repository.
• 摘要：检测非事实内容是提高大型语言模型 (LLM) 一代的可信度的长期目标。当前的事实性探针是使用人工注释标签进行训练的，对未发布内容的可转移性有限，而在线自我一致性检查由于需要生成多个输出而造成了大量的计算负担。本文提出了PINOSE，它在离线自一致性检查结果上训练探测模型，从而避免了对人工注释数据的需求，并实现了跨不同数据分布的可转移性。由于一致性检查过程是离线的，PINOSE通过在线一致性验证减少了生成多个响应的计算负担。此外，它还在响应解码之前检查内部状态的各个方面，有助于更有效地检测事实不准确之处。事实性检测和问答基准的实验结果表明，PINOSE 取得了优于现有事实性检测方法的结果。我们的代码和数据集在这个匿名存储库中公开可用。

Title: Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems

• Authors: Zhengyuan Liu, Stella Xin Yin, Geyu Lin, Nancy F. Chen
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06762
• Pdf URL: https://arxiv.org/pdf/2404.06762
• Copy Paste: [[2404.06762]] Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems(https://arxiv.org/abs/2404.06762)
• Keywords: language model, llm
• Abstract: Intelligent Tutoring Systems (ITSs) can provide personalized and self-paced learning experience. The emergence of large language models (LLMs) further enables better human-machine interaction, and facilitates the development of conversational ITSs in various disciplines such as math and language learning. In dialogic teaching, recognizing and adapting to individual characteristics can significantly enhance student engagement and learning efficiency. However, characterizing and simulating student's persona remain challenging in training and evaluating conversational ITSs. In this work, we propose a framework to construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario. We further enhance the framework with multi-aspect validation, and conduct extensive analysis from both teacher and student perspectives. Our experimental results show that state-of-the-art LLMs can produce diverse student responses according to the given language ability and personality traits, and trigger teacher's adaptive scaffolding strategies.
• 摘要：智能辅导系统（ITS）可以提供个性化和自定进度的学习体验。大语言模型（LLM）的出现进一步实现了更好的人机交互，并促进了数学和语言学习等各个学科中会话式ITS的发展。在对话教学中，认识并适应个体特征可以显着提高学生的参与度和学习效率。然而，在训练和评估会话式智能交通系统中，表征和模拟学生的性格仍然具有挑战性。在这项工作中，我们提出了一个框架，通过完善和整合认知和非认知方面来构建不同学生群体的概况，并利用法学硕士在语言学习场景中进行人格意识的学生模拟。我们通过多方面验证进一步增强了框架，并从教师和学生的角度进行了广泛的分析。我们的实验结果表明，最先进的法学硕士可以根据给定的语言能力和个性特征产生不同的学生反应，并触发教师的适应性脚手架策略。

Title: Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation

• Authors: Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng, Sirui Wang, Xunliang Cai, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06809
• Pdf URL: https://arxiv.org/pdf/2404.06809
• Copy Paste: [[2404.06809]] Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation(https://arxiv.org/abs/2404.06809)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: The rapid development of large language models has led to the widespread adoption of Retrieval-Augmented Generation (RAG), which integrates external knowledge to alleviate knowledge bottlenecks and mitigate hallucinations. However, the existing RAG paradigm inevitably suffers from the impact of flawed information introduced during the retrieval phrase, thereby diminishing the reliability and correctness of the generated outcomes. In this paper, we propose Credibility-aware Generation (CAG), a universally applicable framework designed to mitigate the impact of flawed information in RAG. At its core, CAG aims to equip models with the ability to discern and process information based on its credibility. To this end, we propose an innovative data transformation framework that generates data based on credibility, thereby effectively endowing models with the capability of CAG. Furthermore, to accurately evaluate the models' capabilities of CAG, we construct a comprehensive benchmark covering three critical real-world scenarios. Experimental results demonstrate that our model can effectively understand and utilize credibility for generation, significantly outperform other models with retrieval augmentation, and exhibit resilience against the disruption caused by noisy documents, thereby maintaining robust performance. Moreover, our model supports customized credibility, offering a wide range of potential applications.
• 摘要：大型语言模型的快速发展导致了检索增强生成（RAG）的广泛采用，它整合外部知识来缓解知识瓶颈并减轻幻觉。然而，现有的RAG范式不可避免地受到检索阶段引入的有缺陷信息的影响，从而降低了生成结果的可靠性和正确性。在本文中，我们提出了可信度感知生成（CAG），这是一个普遍适用的框架，旨在减轻 RAG 中缺陷信息的影响。 CAG 的核心目标是让模型具备基于可信度识别和处理信息的能力。为此，我们提出了一种创新的数据转换框架，基于可信度生成数据，从而有效地赋予模型CAG的能力。此外，为了准确评估 CAG 模型的能力，我们构建了涵盖三个关键现实场景的综合基准。实验结果表明，我们的模型可以有效地理解和利用生成的可信度，在检索增强方面显着优于其他模型，并且表现出对噪声文档造成的破坏的弹性，从而保持稳健的性能。此外，我们的模型支持定制可信度，提供广泛的潜在应用。

Title: Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

• Authors: Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, Daniel Hershcovich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06833
• Pdf URL: https://arxiv.org/pdf/2404.06833
• Copy Paste: [[2404.06833]] Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge(https://arxiv.org/abs/2404.06833)
• Keywords: language model, llm
• Abstract: Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain, a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.
• 摘要：最近的研究强调了大型语言模型（LLM）中存在文化偏见，但往往缺乏强有力的方法来全面剖析这些现象。我们的工作旨在通过深入研究食品领域来弥合这一差距，食品领域是人类生活中普遍相关且具有文化多样性的一个方面。我们介绍 FmLAMA，这是一个以食品相关文化事实和食品实践变化为中心的多语言数据集。我们分析各种架构和配置的法学硕士，评估其在单语言和多语言环境中的表现。通过利用六种不同语言的模板，我们研究了法学硕士如何与特定语言和文化知识互动。我们的研究结果表明，(1) 法学硕士对美国流行的食品知识表现出明显的偏见； （2）融入相关文化背景可显着提高法学硕士获取文化知识的能力； (3) 法学硕士在捕捉文化细微差别方面的功效高度依赖于探究语言、特定模型架构和相关文化背景之间的相互作用。这项研究强调了将文化理解融入法学硕士的复杂性，并强调了文化多样性数据集对于减少偏见和提高不同文化领域模型性能的重要性。

Title: Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?

• Authors: Miriam Anschütz, Edoardo Mosca, Georg Groh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06838
• Pdf URL: https://arxiv.org/pdf/2404.06838
• Copy Paste: [[2404.06838]] Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?(https://arxiv.org/abs/2404.06838)
• Keywords: gpt, llm, prompt
• Abstract: Text simplification seeks to improve readability while retaining the original content and meaning. Our study investigates whether pre-trained classifiers also maintain such coherence by comparing their predictions on both original and simplified inputs. We conduct experiments using 11 pre-trained models, including BERT and OpenAI's GPT 3.5, across six datasets spanning three languages. Additionally, we conduct a detailed analysis of the correlation between prediction change rates and simplification types/strengths. Our findings reveal alarming inconsistencies across all languages and models. If not promptly addressed, simplified inputs can be easily exploited to craft zero-iteration model-agnostic adversarial attacks with success rates of up to 50%
• 摘要：文本简化旨在提高可读性，同时保留原始内容和含义。我们的研究通过比较预训练的分类器对原始输入和简化输入的预测来调查它们是否也保持这种一致性。我们使用 11 个预训练模型（包括 BERT 和 OpenAI 的 GPT 3.5）在跨越三种语言的 6 个数据集上进行实验。此外，我们对预测变化率和简化类型/强度之间的相关性进行了详细分析。我们的发现揭示了所有语言和模型之间令人震惊的不一致。如果不及时解决，简化的输入很容易被利用来制造零迭代的与模型无关的对抗性攻击，成功率高达 50%

Title: Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation

• Authors: Thomas Merth, Qichen Fu, Mohammad Rastegari, Mahyar Najibi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06910
• Pdf URL: https://arxiv.org/pdf/2404.06910
• Copy Paste: [[2404.06910]] Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation(https://arxiv.org/abs/2404.06910)
• Keywords: language model, llm, long context, prompt, retrieval-augmented generation
• Abstract: Despite the successes of large language models (LLMs), they exhibit significant drawbacks, particularly when processing long contexts. Their inference cost scales quadratically with respect to sequence length, making it expensive for deployment in some real-world text processing applications, such as retrieval-augmented generation (RAG). Additionally, LLMs also exhibit the "distraction phenomenon," where irrelevant context in the prompt degrades output quality. To address these drawbacks, we propose a novel RAG prompting methodology, superposition prompting, which can be directly applied to pre-trained transformer-based LLMs without the need for fine-tuning. At a high level, superposition prompting allows the LLM to process input documents in parallel prompt paths, discarding paths once they are deemed irrelevant. We demonstrate the capability of our method to simultaneously enhance time efficiency across a variety of question-answering benchmarks using multiple pre-trained LLMs. Furthermore, our technique significantly improves accuracy when the retrieved context is large relative the context the model was trained on. For example, our approach facilitates an 93x reduction in compute time while improving accuracy by 43\% on the NaturalQuestions-Open dataset with the MPT-7B instruction-tuned model over naive RAG.
• 摘要：尽管大型语言模型 (LLM) 取得了成功，但它们也表现出明显的缺点，特别是在处理长上下文时。它们的推理成本与序列长度呈二次方关系，这使得在某些现实世界的文本处理应用程序（例如检索增强生成（RAG））中部署成本高昂。此外，法学硕士还表现出“分心现象”，即提示中不相关的上下文会降低输出质量。为了解决这些缺点，我们提出了一种新颖的 RAG 提示方法，即叠加提示，它可以直接应用于预训练的基于 Transformer 的 LLM，无需进行微调。在高层次上，叠加提示允许法学硕士以并行提示路径处理输入文档，一旦认为路径不相关就丢弃它们。我们证明了我们的方法能够使用多个预训练的法学硕士同时提高各种问答基准的时间效率。此外，当检索到的上下文相对于模型训练的上下文较大时，我们的技术显着提高了准确性。例如，我们的方法使计算时间减少了 93 倍，同时使用 MPT-7B 指令调整模型在 NaturalQuestions-Open 数据集上比朴素的 RAG 提高了 43% 的准确性。

Title: GraSAME: Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanism

• Authors: Shuzhou Yuan, Michael Färber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06911
• Pdf URL: https://arxiv.org/pdf/2404.06911
• Copy Paste: [[2404.06911]] GraSAME: Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanism(https://arxiv.org/abs/2404.06911)
• Keywords: language model
• Abstract: Pretrained Language Models (PLMs) benefit from external knowledge stored in graph structures for various downstream tasks. However, bridging the modality gap between graph structures and text remains a significant challenge. Traditional methods like linearizing graphs for PLMs lose vital graph connectivity, whereas Graph Neural Networks (GNNs) require cumbersome processes for integration into PLMs. In this work, we propose a novel graph-guided self-attention mechanism, GraSAME. GraSAME seamlessly incorporates token-level structural information into PLMs without necessitating additional alignment or concatenation efforts. As an end-to-end, lightweight multimodal module, GraSAME follows a multi-task learning strategy and effectively bridges the gap between graph and textual modalities, facilitating dynamic interactions between GNNs and PLMs. Our experiments on the graph-to-text generation task demonstrate that GraSAME outperforms baseline models and achieves results comparable to state-of-the-art (SOTA) models on WebNLG datasets. Furthermore, compared to SOTA models, GraSAME eliminates the need for extra pre-training tasks to adjust graph inputs and reduces the number of trainable parameters by over 100 million.
• 摘要：预训练语言模型 (PLM) 受益于存储在各种下游任务的图形结构中的外部知识。然而，弥合图形结构和文本之间的模态差距仍然是一个重大挑战。 PLM 的线性化图等传统方法失去了重要的图连接性，而图神经网络 (GNN) 需要繁琐的过程才能集成到 PLM 中。在这项工作中，我们提出了一种新颖的图引导自注意力机制，GraSAME。 GraSAME 将令牌级结构信息无缝合并到 PLM 中，无需额外的对齐或串联工作。作为一个端到端的轻量级多模态模块，GraSAME 遵循多任务学习策略，有效地弥合了图形和文本模态之间的差距，促进 GNN 和 PLM 之间的动态交互。我们在图形到文本生成任务上的实验表明，GraSAME 的性能优于基线模型，并在 WebNLG 数据集上取得了与最先进 (SOTA) 模型相当的结果。此外，与 SOTA 模型相比，GraSAME 无需额外的预训练任务来调整图输入，并将可训练参数的数量减少了超过 1 亿个。

Title: GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications

• Authors: Shishir G. Patil, Tianjun Zhang, Vivian Fang, Noppapon C., Roy Huang, Aaron Hao, Martin Casado, Joseph E. Gonzalez, Raluca Ada Popa, Ion Stoica
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06921
• Pdf URL: https://arxiv.org/pdf/2404.06921
• Copy Paste: [[2404.06921]] GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications(https://arxiv.org/abs/2404.06921)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) are evolving beyond their classical role of providing information within dialogue systems to actively engaging with tools and performing actions on real-world applications and services. Today, humans verify the correctness and appropriateness of the LLM-generated outputs (e.g., code, functions, or actions) before putting them into real-world execution. This poses significant challenges as code comprehension is well known to be notoriously difficult. In this paper, we study how humans can efficiently collaborate with, delegate to, and supervise autonomous LLMs in the future. We argue that in many cases, "post-facto validation" - verifying the correctness of a proposed action after seeing the output - is much easier than the aforementioned "pre-facto validation" setting. The core concept behind enabling a post-facto validation system is the integration of an intuitive undo feature, and establishing a damage confinement for the LLM-generated actions as effective strategies to mitigate the associated risks. Using this, a human can now either revert the effect of an LLM-generated output or be confident that the potential risk is bounded. We believe this is critical to unlock the potential for LLM agents to interact with applications and services with limited (post-facto) human involvement. We describe the design and implementation of our open-source runtime for executing LLM actions, Gorilla Execution Engine (GoEX), and present open research questions towards realizing the goal of LLMs and applications interacting with each other with minimal human supervision. We release GoEX at https://github.com/ShishirPatil/gorilla/.
• 摘要：大型语言模型 (LLM) 正在不断发展，超越了在对话系统中提供信息的传统角色，积极使用工具并对现实世界的应用程序和服务执行操作。如今，人们在将 LLM 生成的输出（例如代码、函数或操作）投入实际执行之前验证它们的正确性和适当性。这带来了重大挑战，因为众所周知，代码理解非常困难。在本文中，我们研究了人类未来如何有效地与自主法学硕士合作、委托和监督。我们认为，在许多情况下，“事后验证”（在看到输出后验证提议的操作的正确性）比前面提到的“事前验证”设置要容易得多。启用事后验证系统背后的核心概念是集成直观的撤消功能，并为 LLM 生成的操作建立损害限制，作为减轻相关风险的有效策略。利用这一点，人们现在可以恢复法学硕士生成的输出的影响，或者确信潜在风险是有限的。我们认为，这对于释放 LLM 代理在有限（事后）人工参与的情况下与应用程序和服务交互的潜力至关重要。我们描述了用于执行 LLM 操作的开源运行时、Gorilla 执行引擎 (GoEX) 的设计和实现，并提出了开放研究问题，以实现 LLM 和应用程序在最少的人工监督下相互交互的目标。我们在 https://github.com/ShishirPatil/gorilla/ 发布了 GoEX。

Title: MetaCheckGPT -- A Multi-task Hallucination Detection Using LLM Uncertainty and Meta-models

• Authors: Rahul Mehta, Andrew Hoblitzell, Jack O'Keefe, Hyeju Jang, Vasudeva Varma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06948
• Pdf URL: https://arxiv.org/pdf/2404.06948
• Copy Paste: [[2404.06948]] MetaCheckGPT -- A Multi-task Hallucination Detection Using LLM Uncertainty and Meta-models(https://arxiv.org/abs/2404.06948)
• Keywords: language model, gpt, llm, hallucination
• Abstract: This paper presents our winning solution for the SemEval-2024 Task 6 competition. We propose a meta-regressor framework of large language models (LLMs) for model evaluation and integration that achieves the highest scores on the leader board. Our approach leverages uncertainty signals present in a diverse basket of LLMs to detect hallucinations more robustly.
• 摘要：本文介绍了我们在 SemEval-2024 Task 6 竞赛中获胜的解决方案。我们提出了一个用于模型评估和集成的大型语言模型（LLM）的元回归框架，该框架在排行榜上取得了最高分。我们的方法利用各种法学硕士中存在的不确定性信号来更可靠地检测幻觉。

Title: Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy

• Authors: Yijin Liu, Fandong Meng, Jie Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06954
• Pdf URL: https://arxiv.org/pdf/2404.06954
• Copy Paste: [[2404.06954]] Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy(https://arxiv.org/abs/2404.06954)
• Keywords: language model, llm
• Abstract: Recently, dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations through elaborate heuristics or additional predictors. However, in the decoding process of existing approaches, different samples are assigned different computational budgets, which cannot guarantee a stable and precise acceleration effect. Furthermore, existing approaches generally skip multiple contiguous layers at the bottom or top of the layers, leading to a drastic change in the model's layer-wise representations, and thus a consequent performance degeneration. Therefore, we propose a Unified Layer Skipping strategy, which selects the number of layers to skip computation based solely on the target speedup ratio, and then skips the corresponding number of intermediate layer computations in a balanced manner. Since the Unified Layer Skipping strategy is independent of input samples, it naturally supports popular acceleration techniques such as batch decoding and KV caching, thus demonstrating more practicality for real-world applications. Experimental results on two common tasks, i.e., machine translation and text summarization, indicate that given a target speedup ratio, the Unified Layer Skipping strategy significantly enhances both the inference performance and the actual model throughput over existing dynamic approaches.
• 摘要：最近，动态计算方法通过精心设计的启发式方法或附加预测器跳过多层计算，显示出大型语言模型 (LLM) 的显着加速。然而，现有方法的解码过程中，不同的样本被分配不同的计算预算，无法保证稳定且精确的加速效果。此外，现有方法通常会跳过层底部或顶部的多个连续层，导致模型的逐层表示发生巨大变化，从而导致性能下降。因此，我们提出了一种统一的跳层策略，仅根据目标加速比选择跳过计算的层数，然后以平衡的方式跳过相应数量的中间层计算。由于Unified Layer Skipping策略与输入样本无关，它自然支持批量解码和KV缓存等流行的加速技术，从而为实际应用展示了更多的实用性。机器翻译和文本摘要这两个常见任务的实验结果表明，在给定目标加速比的情况下，统一跳层策略比现有动态方法显着提高了推理性能和实际模型吞吐量。

Title: XNLIeu: a dataset for cross-lingual NLI in Basque

• Authors: Maite Heredia, Julen Etxaniz, Muitze Zulaika, Xabier Saralegi, Jeremy Barnes, Aitor Soroa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06996
• Pdf URL: https://arxiv.org/pdf/2404.06996
• Copy Paste: [[2404.06996]] XNLIeu: a dataset for cross-lingual NLI in Basque(https://arxiv.org/abs/2404.06996)
• Keywords: llm
• Abstract: XNLI is a popular Natural Language Inference (NLI) benchmark widely used to evaluate cross-lingual Natural Language Understanding (NLU) capabilities across languages. In this paper, we expand XNLI to include Basque, a low-resource language that can greatly benefit from transfer-learning approaches. The new dataset, dubbed XNLIeu, has been developed by first machine-translating the English XNLI corpus into Basque, followed by a manual post-edition step. We have conducted a series of experiments using mono- and multilingual LLMs to assess a) the effect of professional post-edition on the MT system; b) the best cross-lingual strategy for NLI in Basque; and c) whether the choice of the best cross-lingual strategy is influenced by the fact that the dataset is built by translation. The results show that post-edition is necessary and that the translate-train cross-lingual strategy obtains better results overall, although the gain is lower when tested in a dataset that has been built natively from scratch. Our code and datasets are publicly available under open licenses.
• 摘要：XNLI 是一种流行的自然语言推理 (NLI) 基准，广泛用于评估跨语言的跨语言自然语言理解 (NLU) 能力。在本文中，我们将 XNLI 扩展为包括巴斯克语，这是一种资源匮乏的语言，可以从迁移学习方法中受益匪浅。这个名为 XNLIeu 的新数据集是通过首先将英语 XNLI 语料库机器翻译成巴斯克语，然后进行手动后期编辑步骤来开发的。我们使用单语和多语种法学硕士进行了一系列实验，以评估 a) 专业译后编辑对 MT 系统的影响； b) 巴斯克语 NLI 的最佳跨语言策略； c) 最佳跨语言策略的选择是否受到数据集是通过翻译构建这一事实的影响。结果表明，后期编辑是必要的，并且翻译训练跨语言策略总体上获得了更好的结果，尽管在从头开始本地构建的数据集中进行测试时增益较低。我们的代码和数据集在开放许可下公开可用。

Title: Event Grounded Criminal Court View Generation withCooperative (Large) Language Models

• Authors: Linan Yue, Qi Liu, Lili Zhao, Li Wang, Weibo Gao, Yanqing An
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07001
• Pdf URL: https://arxiv.org/pdf/2404.07001
• Copy Paste: [[2404.07001]] Event Grounded Criminal Court View Generation withCooperative (Large) Language Models(https://arxiv.org/abs/2404.07001)
• Keywords: language model, llm
• Abstract: With the development of legal intelligence, Criminal Court View Generation has attracted much attention as a crucial task of legal intelligence, which aims to generate concise and coherent texts that summarize case facts and provide explanations for verdicts. Existing researches explore the key information in case facts to yield the court views. Most of them employ a coarse-grained approach that partitions the facts into broad segments (e.g., verdict-related sentences) to make predictions. However, this approach fails to capture the complex details present in the case facts, such as various criminal elements and legal events. To this end, in this paper, we propose an Event Grounded Generation (EGG) method for criminal court view generation with cooperative (Large) Language Models, which introduces the fine-grained event information into the generation. Specifically, we first design a LLMs-based extraction method that can extract events in case facts without massive annotated events. Then, we incorporate the extracted events into court view generation by merging case facts and events. Besides, considering the computational burden posed by the use of LLMs in the extraction phase of EGG, we propose a LLMs-free EGG method that can eliminate the requirement for event extraction using LLMs in the inference phase. Extensive experimental results on a real-world dataset clearly validate the effectiveness of our proposed method.
• 摘要：随着法律情报的发展，刑事法庭视图生成作为法律情报的一项重要任务而备受关注，其目的是生成简洁、连贯的文本来总结案件事实并为判决提供解释。现有研究探索案件事实中的关键信息以得出法院观点。他们中的大多数采用粗粒度的方法，将事实划分为广泛的部分（例如，与判决相关的句子）来做出预测。然而，这种方法无法捕捉案件事实中存在的复杂细节，例如各种犯罪要素和法律事件。为此，在本文中，我们提出了一种基于事件生成（EGG）的方法，用于使用合作（大）语言模型生成刑事法庭视图，该方法将细粒度的事件信息引入到生成中。具体来说，我们首先设计了一种基于LLM的提取方法，该方法可以提取案例事实中的事件，而无需大量注释事件。然后，我们通过合并案件事实和事件，将提取的事件纳入法庭视图生成中。此外，考虑到在 EGG 提取阶段使用 LLM 带来的计算负担，我们提出了一种无 LLMs 的 EGG 方法，可以消除在推理阶段使用 LLM 进行事件提取的要求。在真实世界数据集上的大量实验结果清楚地验证了我们提出的方法的有效性。

Title: LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models

• Authors: Igor Tufanov, Karen Hambardzumyan, Javier Ferrando, Elena Voita
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07004
• Pdf URL: https://arxiv.org/pdf/2404.07004
• Copy Paste: [[2404.07004]] LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models(https://arxiv.org/abs/2404.07004)
• Keywords: language model
• Abstract: We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previously existing tools that focus on isolated parts of the decision-making process, our framework is designed to make the entire prediction process transparent, and allows tracing back model behavior from the top-layer representation to very fine-grained parts of the model. Specifically, it (1) shows the important part of the whole input-to-output information flow, (2) allows attributing any changes done by a model block to individual attention heads and feed-forward neurons, (3) allows interpreting the functions of those heads or neurons. A crucial part of this pipeline is showing the importance of specific model components at each step. As a result, we are able to look at the roles of model components only in cases where they are important for a prediction. Since knowing which components should be inspected is key for analyzing large models where the number of these components is extremely high, we believe our tool will greatly support the interpretability community both in research settings and in practical applications.
• 摘要：我们推出了 LM Transparency Tool (LM-TT)，这是一个开源交互式工具包，用于分析基于 Transformer 的语言模型的内部工作原理。与以前专注于决策过程的孤立部分的现有工具不同，我们的框架旨在使整个预测过程透明，并允许从顶层表示追溯到模型的非常细粒度的部分。 。具体来说，它（1）显示了整个输入到输出信息流的重要部分，（2）允许将模型块所做的任何更改归因于各个注意头和前馈神经元，（3）允许解释功能这些头或神经元。该流程的一个关键部分是显示每个步骤中特定模型组件的重要性。因此，我们只能在模型组件对预测很重要的情况下查看其角色。由于知道应该检查哪些组件是分析组件数量非常多的大型模型的关键，因此我们相信我们的工具将在研究环境和实际应用中极大地支持可解释性社区。

Title: A Mathematical Theory for Learning Semantic Languages by Abstract Learners

• Authors: Kuo-Yu Liao, Cheng-Shang Chang, Y.-W. Peter Hong
• Subjects: cs.CL, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07009
• Pdf URL: https://arxiv.org/pdf/2404.07009
• Copy Paste: [[2404.07009]] A Mathematical Theory for Learning Semantic Languages by Abstract Learners(https://arxiv.org/abs/2404.07009)
• Keywords: language model, llm
• Abstract: Recent advances in Large Language Models (LLMs) have demonstrated the emergence of capabilities (learned skills) when the number of system parameters and the size of training data surpass certain thresholds. The exact mechanisms behind such phenomena are not fully understood and remain a topic of active research. Inspired by the skill-text bipartite graph model presented in [1] for modeling semantic language, we develop a mathematical theory to explain the emergence of learned skills, taking the learning (or training) process into account. Our approach models the learning process for skills in the skill-text bipartite graph as an iterative decoding process in Low-Density Parity Check (LDPC) codes and Irregular Repetition Slotted ALOHA (IRSA). Using density evolution analysis, we demonstrate the emergence of learned skills when the ratio of the size of training texts to the number of skills exceeds a certain threshold. Our analysis also yields a scaling law for testing errors relative to the size of training texts. Upon completion of the training, we propose a method for semantic compression and discuss its application in semantic communication.
• 摘要：大型语言模型 (LLM) 的最新进展表明，当系统参数的数量和训练数据的大小超过一定阈值时，能力（学习的技能）就会出现。这种现象背后的确切机制尚未完全了解，仍然是活跃研究的主题。受[1]中提出的用于建模语义语言的技能-文本二部图模型的启发，我们开发了一种数学理论来解释所学技能的出现，并将学习（或训练）过程考虑在内。我们的方法将技能文本二分图中的技能学习过程建模为低密度奇偶校验 (LDPC) 代码和不规则重复时隙 ALOHA (IRSA) 中的迭代解码过程。使用密度演化分析，我们证明了当训练文本的大小与技能数量的比率超过一定阈值时，学习技能的出现。我们的分析还得出了相对于训练文本大小的测试误差的缩放法则。训练完成后，我们提出了一种语义压缩方法，并讨论了其在语义通信中的应用。

Title: Improving Language Model Reasoning with Self-motivated Learning

• Authors: Yunlong Feng, Yang Xu, Libo Qin, Yasheng Wang, Wanxiang Che
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07017
• Pdf URL: https://arxiv.org/pdf/2404.07017
• Copy Paste: [[2404.07017]] Improving Language Model Reasoning with Self-motivated Learning(https://arxiv.org/abs/2404.07017)
• Keywords: language model
• Abstract: Large-scale high-quality training data is important for improving the performance of models. After trained with data that has rationales (reasoning steps), models gain reasoning capability. However, the dataset with high-quality rationales is relatively scarce due to the high annotation cost. To address this issue, we propose \textit{Self-motivated Learning} framework. The framework motivates the model itself to automatically generate rationales on existing datasets. Based on the inherent rank from correctness across multiple rationales, the model learns to generate better rationales, leading to higher reasoning capability. Specifically, we train a reward model with the rank to evaluate the quality of rationales, and improve the performance of reasoning through reinforcement learning. Experiment results of Llama2 7B on multiple reasoning datasets show that our method significantly improves the reasoning ability of models, even outperforming text-davinci-002 in some datasets.
• 摘要：大规模高质量训练数据对于提高模型性能具有重要意义。在使用具有基本原理（推理步骤）的数据进行训练后，模型获得推理能力。然而，由于标注成本较高，具有高质量原理的数据集相对稀缺。为了解决这个问题，我们提出了 \textit{自我激励学习} 框架。该框架激励模型本身自动生成现有数据集的基本原理。基于多个理由的正确性的固有排名，该模型学习生成更好的理由，从而获得更高的推理能力。具体来说，我们训练一个带有排名的奖励模型来评估推理的质量，并通过强化学习提高推理的性能。 Llama2 7B在多个推理数据集上的实验结果表明，我们的方法显着提高了模型的推理能力，甚至在某些数据集上优于text-davinci-002。

Title: Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation

• Authors: Elisa Sanchez-Bayona, Rodrigo Agerri
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07053
• Pdf URL: https://arxiv.org/pdf/2404.07053
• Copy Paste: [[2404.07053]] Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation(https://arxiv.org/abs/2404.07053)
• Keywords: language model
• Abstract: Metaphors, although occasionally unperceived, are ubiquitous in our everyday language. Thus, it is crucial for Language Models to be able to grasp the underlying meaning of this kind of figurative language. In this work, we present Meta4XNLI, a novel parallel dataset for the tasks of metaphor detection and interpretation that contains metaphor annotations in both Spanish and English. We investigate language models' metaphor identification and understanding abilities through a series of monolingual and cross-lingual experiments by leveraging our proposed corpus. In order to comprehend how these non-literal expressions affect models' performance, we look over the results and perform an error analysis. Additionally, parallel data offers many potential opportunities to investigate metaphor transferability between these languages and the impact of translation on the development of multilingual annotated resources.
• 摘要：隐喻虽然有时未被察觉，但在我们的日常语言中却无处不在。因此，语言模型能否掌握这种比喻语言的深层含义至关重要。在这项工作中，我们提出了 Meta4XNLI，这是一个用于隐喻检测和解释任务的新型并行数据集，其中包含西班牙语和英语的隐喻注释。我们利用我们提出的语料库，通过一系列单语言和跨语言实验来研究语言模型的隐喻识别和理解能力。为了理解这些非文字表达式如何影响模型的性能，我们查看结果并执行错误分析。此外，并行数据提供了许多潜在的机会来研究这些语言之间的隐喻可转移性以及翻译对多语言注释资源开发的影响。

Title: Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study

• Authors: Alessandro Stolfo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07060
• Pdf URL: https://arxiv.org/pdf/2404.07060
• Copy Paste: [[2404.07060]] Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study(https://arxiv.org/abs/2404.07060)
• Keywords: language model, llm, hallucination
• Abstract: We present an empirical study of groundedness in long-form question answering (LFQA) by retrieval-augmented large language models (LLMs). In particular, we evaluate whether every generated sentence is grounded in the retrieved documents or the model's pre-training data. Across 3 datasets and 4 model families, our findings reveal that a significant fraction of generated sentences are consistently ungrounded, even when those sentences contain correct ground-truth answers. Additionally, we examine the impacts of factors such as model size, decoding strategy, and instruction tuning on groundedness. Our results show that while larger models tend to ground their outputs more effectively, a significant portion of correct answers remains compromised by hallucinations. This study provides novel insights into the groundedness challenges in LFQA and underscores the necessity for more robust mechanisms in LLMs to mitigate the generation of ungrounded content.
• 摘要：我们通过检索增强大语言模型（LLM）对长格式问答（LFQA）的基础性进行了实证研究。特别是，我们评估每个生成的句子是否基于检索到的文档或模型的预训练数据。在 3 个数据集和 4 个模型系列中，我们的研究结果表明，生成的句子中有很大一部分始终没有根据，即使这些句子包含正确的真实答案。此外，我们还研究了模型大小、解码策略和指令调整等因素对基础性的影响。我们的结果表明，虽然较大的模型往往更有效地确定其输出，但很大一部分正确答案仍然受到幻觉的影响。这项研究为 LFQA 中的接地挑战提供了新颖的见解，并强调了法学硕士中需要更强大的机制来减少不接地内容的产生。

Title: Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?

• Authors: Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, Yongfeng Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07066
• Pdf URL: https://arxiv.org/pdf/2404.07066
• Copy Paste: [[2404.07066]] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?(https://arxiv.org/abs/2404.07066)
• Keywords: language model
• Abstract: This paper studies the phenomenon that different concepts are learned in different layers of large language models, i.e. more difficult concepts are fully acquired with deeper layers. We define the difficulty of concepts by the level of abstraction, and here it is crudely categorized by factual, emotional, and inferential. Each category contains a spectrum of tasks, arranged from simple to complex. For example, within the factual dimension, tasks range from lie detection to categorizing mathematical problems. We employ a probing technique to extract representations from different layers of the model and apply these to classification tasks. Our findings reveal that models tend to efficiently classify simpler tasks, indicating that these concepts are learned in shallower layers. Conversely, more complex tasks may only be discernible at deeper layers, if at all. This paper explores the implications of these findings for our understanding of model learning processes and internal representations. Our implementation is available at \url{https://github.com/Luckfort/CD}.
• 摘要：本文研究了在大型语言模型的不同层中学习不同概念的现象，即更难的概念随着层数的加深而被完全习得。我们通过抽象程度来定义概念的难度，这里粗略地按事实、情感和推理进行分类。每个类别都包含一系列任务，从简单到复杂排列。例如，在事实维度内，任务范围从测谎到数学问题分类。我们采用探测技术从模型的不同层提取表示并将其应用于分类任务。我们的研究结果表明，模型倾向于有效地对更简单的任务进行分类，这表明这些概念是在较浅的层中学习的。相反，更复杂的任务可能只能在更深层次上才能辨别（如果有的话）。本文探讨了这些发现对我们理解模型学习过程和内部表征的影响。我们的实现可在 \url{https://github.com/Luckfort/CD} 获取。

Title: Dynamic Generation of Personalities with Large Language Models

• Authors: Jianzhi Liu, Hexiang Gu, Tianyu Zheng, Liuyu Xiang, Huijia Wu, Jie Fu, Zhaofeng He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07084
• Pdf URL: https://arxiv.org/pdf/2404.07084
• Copy Paste: [[2404.07084]] Dynamic Generation of Personalities with Large Language Models(https://arxiv.org/abs/2404.07084)
• Keywords: language model, gpt, llm, prompt
• Abstract: In the realm of mimicking human deliberation, large language models (LLMs) show promising performance, thereby amplifying the importance of this research area. Deliberation is influenced by both logic and personality. However, previous studies predominantly focused on the logic of LLMs, neglecting the exploration of personality aspects. In this work, we introduce Dynamic Personality Generation (DPG), a dynamic personality generation method based on Hypernetworks. Initially, we embed the Big Five personality theory into GPT-4 to form a personality assessment machine, enabling it to evaluate characters' personality traits from dialogues automatically. We propose a new metric to assess personality generation capability based on this evaluation method. Then, we use this personality assessment machine to evaluate dialogues in script data, resulting in a personality-dialogue dataset. Finally, we fine-tune DPG on the personality-dialogue dataset. Experiments prove that DPG's personality generation capability is stronger after fine-tuning on this dataset than traditional fine-tuning methods, surpassing prompt-based GPT-4.
• 摘要：在模仿人类思考领域，大型语言模型（LLM）显示出良好的性能，从而放大了该研究领域的重要性。深思熟虑受到逻辑和个性的影响。然而，以往的研究主要集中在法学硕士的逻辑上，忽视了对人格方面的探索。在这项工作中，我们介绍了动态个性生成（DPG），一种基于超网络的动态个性生成方法。最初，我们将大五人格理论嵌入到GPT-4中，形成人格评估机，使其能够从对话中自动评估人物的人格特质。基于这种评估方法，我们提出了一种评估人格生成能力的新指标。然后，我们使用这个性格评估机来评估脚本数据中的对话，从而产生性格对话数据集。最后，我们在个性对话数据集上微调 DPG。实验证明，在该数据集上进行微调后，DPG 的个性生成能力比传统微调方法更强，超越了基于提示的 GPT-4。

Title: Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

• Authors: Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Suhang Wang, Yu Meng, Jiawei Han
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07103
• Pdf URL: https://arxiv.org/pdf/2404.07103
• Copy Paste: [[2404.07103]] Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs(https://arxiv.org/abs/2404.07103)
• Keywords: language model, llm, hallucination, chain-of-thought
• Abstract: Large language models (LLMs), while exhibiting exceptional performance, suffer from hallucinations, especially on knowledge-intensive tasks. Existing works propose to augment LLMs with individual text units retrieved from external knowledge corpora to alleviate the issue. However, in many domains, texts are interconnected (e.g., academic papers in a bibliographic graph are linked by citations and co-authorships) which form a (text-attributed) graph. The knowledge in such graphs is encoded not only in single texts/nodes but also in their associated connections. To facilitate the research of augmenting LLMs with graphs, we manually construct a Graph Reasoning Benchmark dataset called GRBench, containing 1,740 questions that can be answered with the knowledge from 10 domain graphs. Then, we propose a simple and effective framework called Graph Chain-of-thought (Graph-CoT) to augment LLMs with graphs by encouraging LLMs to reason on the graph iteratively. Each Graph-CoT iteration consists of three sub-steps: LLM reasoning, LLM-graph interaction, and graph execution. We conduct systematic experiments with three LLM backbones on GRBench, where Graph-CoT outperforms the baselines consistently. The code is available at https://github.com/PeterGriffinJin/Graph-CoT.
• 摘要：大型语言模型（LLM）虽然表现出卓越的性能，但也存在幻觉，尤其是在知识密集型任务上。现有的工作建议用从外部知识语料库检索到的单个文本单元来增强法学硕士，以缓解这个问题。然而，在许多领域，文本是相互关联的（例如，书目图中的学术论文通过引文和共同作者链接），形成（文本属性）图。此类图中的知识不仅编码在单个文本/节点中，而且还编码在它们的关联连接中。为了促进用图增强法学硕士的研究，我们手动构建了一个名为 GRBench 的图推理基准数据集，其中包含 1,740 个问题，可以用 10 个领域图的知识来回答。然后，我们提出了一个简单而有效的框架，称为图思想链（Graph-CoT），通过鼓励法学硕士在图上迭代推理，用图来增强法学硕士。每个 Graph-CoT 迭代都包含三个子步骤：LLM 推理、LLM-图交互和图执行。我们在 GRBench 上使用三个 LLM 主干进行系统实验，其中 Graph-CoT 始终优于基线。代码可在 https://github.com/PeterGriffinJin/Graph-CoT 获取。

Title: From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications

• Authors: Yongqiang Ma, Lizhi Qin, Jiawei Liu, Yangyang Kang, Yue Zhang, Wei Lu, Xiaozhong Liu, Qikai Cheng
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.07108
• Pdf URL: https://arxiv.org/pdf/2404.07108
• Copy Paste: [[2404.07108]] From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications(https://arxiv.org/abs/2404.07108)
• Keywords: language model, gpt, llm
• Abstract: Evaluating large language models (LLMs) is fundamental, particularly in the context of practical applications. Conventional evaluation methods, typically designed primarily for LLM development, yield numerical scores that ignore the user experience. Therefore, our study shifts the focus from model-centered to human-centered evaluation in the context of AI-powered writing assistance applications. Our proposed metric, termed ``Revision Distance,'' utilizes LLMs to suggest revision edits that mimic the human writing process. It is determined by counting the revision edits generated by LLMs. Benefiting from the generated revision edit details, our metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. Our results show that for the easy-writing task, ``Revision Distance'' is consistent with established metrics (ROUGE, Bert-score, and GPT-score), but offers more insightful, detailed feedback and better distinguishes between texts. Moreover, in the context of challenging academic writing tasks, our metric still delivers reliable evaluations where other metrics tend to struggle. Furthermore, our metric also holds significant potential for scenarios lacking reference texts.
• 摘要：评估大型语言模型 (LLM) 至关重要，尤其是在实际应用中。传统的评估方法通常主要为法学硕士开发而设计，产生的数字分数忽略了用户体验。因此，在人工智能驱动的写作辅助应用程序中，我们的研究重点从以模型为中心的评估转向以人为中心的评估。我们提出的指标称为“修订距离”，利用法学硕士来建议模仿人类写作过程的修订编辑。它是通过计算法学硕士生成的修订编辑来确定的。受益于生成的修订编辑详细信息，我们的指标可以以人类可理解的方式提供不言自明的文本评估结果，超越上下文无关的分数。我们的结果表明，对于简单的写作任务，“修订距离”与既定指标（ROUGE、Bert 分数和 GPT 分数）一致，但提供了更有洞察力、更详细的反馈并更好地区分文本。此外，在具有挑战性的学术写作任务的背景下，我们的指标仍然可以提供可靠的评估，而其他指标往往难以做到这一点。此外，我们的指标对于缺乏参考文本的场景也具有巨大的潜力。

Title: Continuous Language Model Interpolation for Dynamic and Controllable Text Generation

• Authors: Sara Kangaslahti, David Alvarez-Melis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07117
• Pdf URL: https://arxiv.org/pdf/2404.07117
• Copy Paste: [[2404.07117]] Continuous Language Model Interpolation for Dynamic and Controllable Text Generation(https://arxiv.org/abs/2404.07117)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) have gained popularity for a variety of use cases, making them adaptable and controllable has become increasingly important, especially for user-facing applications. While the existing literature on LLM adaptation primarily focuses on finding a model (or models) that optimizes a single predefined objective, here we focus on the challenging case where the model must dynamically adapt to diverse -- and often changing -- user preferences. For this, we leverage adaptation methods based on linear weight interpolation, casting them as continuous multi-domain interpolators that produce models with specific prescribed generation characteristics on-the-fly. Specifically, we use low-rank updates to fine-tune a base model to various different domains, yielding a set of anchor models with distinct generation profiles. Then, we use the weight updates of these anchor models to parametrize the entire (infinite) class of models contained within their convex hull. We empirically show that varying the interpolation weights yields predictable and consistent change in the model outputs with respect to all of the controlled attributes. We find that there is little entanglement between most attributes and identify and discuss the pairs of attributes for which this is not the case. Our results suggest that linearly interpolating between the weights of fine-tuned models facilitates predictable, fine-grained control of model outputs with respect to multiple stylistic characteristics simultaneously.
• 摘要：随着大型语言模型 (LLM) 在各种用例中越来越受欢迎，使其具有适应性和可控性变得越来越重要，特别是对于面向用户的应用程序。虽然关于 LLM 适应的现有文献主要侧重于寻找一个（或多个）模型来优化单个预定义目标，但这里我们关注具有挑战性的案例，其中模型必须动态适应不同的（并且经常变化的）用户偏好。为此，我们利用基于线性权重插值的自适应方法，将它们转换为连续的多域插值器，即时生成具有特定指定生成特征的模型。具体来说，我们使用低秩更新将基础模型微调到各种不同的领域，产生一组具有不同生成配置文件的锚模型。然后，我们使用这些锚模型的权重更新来参数化其凸包中包含的整个（无限）模型类。我们凭经验表明，改变插值权重会导致模型输出相对于所有受控属性产生可预测且一致的变化。我们发现大多数属性之间几乎没有纠缠，并识别和讨论并非如此的属性对。我们的结果表明，在微调模型的权重之间进行线性插值有助于同时针对多种风格特征对模型输出进行可预测的、细粒度的控制。

Title: Towards Robustness of Text-to-Visualization Translation against Lexical and Phrasal Variability

• Authors: Jinwei Lu, Yuanfeng Song, Haodi Zhang, Chen Zhang, Raymond Chi-Wing Wong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07135
• Pdf URL: https://arxiv.org/pdf/2404.07135
• Copy Paste: [[2404.07135]] Towards Robustness of Text-to-Visualization Translation against Lexical and Phrasal Variability(https://arxiv.org/abs/2404.07135)
• Keywords: retrieval-augmented generation
• Abstract: Text-to-Vis is an emerging task in the natural language processing (NLP) area that aims to automatically generate data visualizations from natural language questions (NLQs). Despite their progress, existing text-to-vis models often heavily rely on lexical matching between words in the questions and tokens in data schemas. This overreliance on lexical matching may lead to a diminished level of model robustness against input variations. In this study, we thoroughly examine the robustness of current text-to-vis models, an area that has not previously been explored. In particular, we construct the first robustness dataset nvBench-Rob, which contains diverse lexical and phrasal variations based on the original text-to-vis benchmark nvBench. Then, we found that the performance of existing text-to-vis models on this new dataset dramatically drops, implying that these methods exhibit inadequate robustness overall. Finally, we propose a novel framework based on Retrieval-Augmented Generation (RAG) technique, named GRED, specifically designed to address input perturbations in these two variants. The framework consists of three parts: NLQ-Retrieval Generator, Visualization Query-Retrieval Retuner and Annotation-based Debugger, which are used to tackle the challenges posed by natural language variants, programming style differences and data schema variants, respectively. Extensive experimental evaluations show that, compared to the state-of-the-art model RGVisNet in the Text-to-Vis field, RGDR performs better in terms of model robustness, with a 32% increase in accuracy on the proposed nvBench-Rob dataset.
• 摘要：Text-to-Vis 是自然语言处理 (NLP) 领域的一项新兴任务，旨在从自然语言问题 (NLQ) 自动生成数据可视化。尽管取得了进步，现有的文本到可见模型通常严重依赖问题中的单词与数据模式中的标记之间的词汇匹配。这种对词汇匹配的过度依赖可能会导致模型针对输入变化的鲁棒性降低。在这项研究中，我们彻底检查了当前文本到可见模型的稳健性，这是一个以前从未探索过的领域。特别是，我们构建了第一个鲁棒性数据集 nvBench-Rob，其中包含基于原始文本到可见基准 nvBench 的各种词汇和短语变体。然后，我们发现现有的文本到可见模型在这个新数据集上的性能急剧下降，这意味着这些方法总体上表现出不足的稳健性。最后，我们提出了一种基于检索增强生成（RAG）技术的新颖框架，名为 GRED，专门设计用于解决这两种变体中的输入扰动。该框架由三部分组成：NLQ-Retrieval Generator、Visualization Query-Retrieval Retuner 和 Annotation-based Debugger，分别用于解决自然语言变体、编程风格差异和数据模式变体带来的挑战。大量的实验评估表明，与 Text-to-Vis 领域最先进的模型 RGVisNet 相比，RGDR 在模型鲁棒性方面表现更好，在所提出的 nvBench-Rob 数据集上准确率提高了 32% 。

Title: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

• Authors: Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.07143
• Pdf URL: https://arxiv.org/pdf/2404.07143
• Copy Paste: [[2404.07143]] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention(https://arxiv.org/abs/2404.07143)
• Keywords: language model, llm
• Abstract: This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
• 摘要：这项工作引入了一种有效的方法，可以将基于 Transformer 的大型语言模型 (LLM) 扩展到具有有限内存和计算的无限长输入。我们提出的方法的一个关键组成部分是一种称为“无限注意力”的新注意力技术。 Infini-attention 将压缩记忆融入到普通的注意力机制中，并在单个 Transformer 块中构建了屏蔽局部注意力和长期线性注意力机制。我们通过 1B 和 8B LLM 证明了我们的方法在长上下文语言建模基准、1M 序列长度密钥上下文块检索和 500K 长度书籍摘要任务上的有效性。我们的方法引入了最小的有界内存参数，并支持 LLM 的快速流式推理。