2024-04-23

Title: FlowMind: Automatic Workflow Generation with LLMs

• Authors: Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, Manuela Veloso
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13050
• Pdf URL: https://arxiv.org/pdf/2404.13050
• Copy Paste: [[2404.13050]] FlowMind: Automatic Workflow Generation with LLMs(https://arxiv.org/abs/2404.13050)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: The rapidly evolving field of Robotic Process Automation (RPA) has made significant strides in automating repetitive processes, yet its effectiveness diminishes in scenarios requiring spontaneous or unpredictable tasks demanded by users. This paper introduces a novel approach, FlowMind, leveraging the capabilities of Large Language Models (LLMs) such as Generative Pretrained Transformer (GPT), to address this limitation and create an automatic workflow generation system. In FlowMind, we propose a generic prompt recipe for a lecture that helps ground LLM reasoning with reliable Application Programming Interfaces (APIs). With this, FlowMind not only mitigates the common issue of hallucinations in LLMs, but also eliminates direct interaction between LLMs and proprietary data or code, thus ensuring the integrity and confidentiality of information - a cornerstone in financial services. FlowMind further simplifies user interaction by presenting high-level descriptions of auto-generated workflows, enabling users to inspect and provide feedback effectively. We also introduce NCEN-QA, a new dataset in finance for benchmarking question-answering tasks from N-CEN reports on funds. We used NCEN-QA to evaluate the performance of workflows generated by FlowMind against baseline and ablation variants of FlowMind. We demonstrate the success of FlowMind, the importance of each component in the proposed lecture recipe, and the effectiveness of user interaction and feedback in FlowMind.
• 摘要：快速发展的机器人流程自动化 (RPA) 领域在自动化重复流程方面取得了重大进展，但在需要用户要求的自发或不可预测任务的场景中，其有效性会降低。本文介绍了一种新颖的方法 FlowMind，利用大型语言模型 (LLM)（例如生成预训练变换器 (GPT)）的功能来解决此限制并创建自动工作流生成系统。在 FlowMind 中，我们提出了一个通用的讲座提示方案，通过可靠的应用程序编程接口 (API) 帮助奠定 LLM 推理的基础。这样，FlowMind 不仅可以缓解法学硕士中常见的幻觉问题，还可以消除法学硕士与专有数据或代码之间的直接交互，从而确保信息的完整性和机密性 - 这是金融服务的基石。 FlowMind 通过呈现自动生成的工作流程的高级描述来进一步简化用户交互，使用户能够有效地检查和提供反馈。我们还引入了 NCEN-QA，这是一个新的金融数据集，用于对 N-CEN 基金报告中的问答任务进行基准测试。我们使用 NCEN-QA 对照 FlowMind 的基线和消融变体来评估 FlowMind 生成的工作流程的性能。我们展示了 FlowMind 的成功、拟议讲座方案中每个组件的重要性以及 FlowMind 中用户交互和反馈的有效性。

Title: "Hey..! This medicine made me sick": Sentiment Analysis of User-Generated Drug Reviews using Machine Learning Techniques

• Authors: Abhiram B. Nair, Abhinand K., Anamika U., Denil Tom Jaison, Ajitha V., V. S. Anoop
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13057
• Pdf URL: https://arxiv.org/pdf/2404.13057
• Copy Paste: [[2404.13057]] "Hey..! This medicine made me sick": Sentiment Analysis of User-Generated Drug Reviews using Machine Learning Techniques(https://arxiv.org/abs/2404.13057)
• Keywords: language model
• Abstract: Sentiment analysis has become increasingly important in healthcare, especially in the biomedical and pharmaceutical fields. The data generated by the general public on the effectiveness, side effects, and adverse drug reactions are goldmines for different agencies and medicine producers to understand the concerns and reactions of people. Despite the challenge of obtaining datasets on drug-related problems, sentiment analysis on this topic would be a significant boon to the field. This project proposes a drug review classification system that classifies user reviews on a particular drug into different classes, such as positive, negative, and neutral. This approach uses a dataset that is collected from publicly available sources containing drug reviews, such as drugs.com. The collected data is manually labeled and verified manually to ensure that the labels are correct. Three pre-trained language models, such as BERT, SciBERT, and BioBERT, are used to obtain embeddings, which were later used as features to different machine learning classifiers such as decision trees, support vector machines, random forests, and also deep learning algorithms such as recurrent neural networks. The performance of these classifiers is quantified using precision, recall, and f1-score, and the results show that the proposed approaches are useful in analyzing the sentiments of people on different drugs.
• 摘要：情感分析在医疗保健领域变得越来越重要，特别是在生物医学和制药领域。公众产生的关于有效性、副作用和药物不良反应的数据是不同机构和药品生产商了解人们的担忧和反应的金矿。尽管获取毒品相关问题的数据集存在挑战，但对该主题的情绪分析将对该领域带来重大好处。该项目提出了一个药物评论分类系统，将用户对特定药物的评论分为不同的类别，例如正面、负面和中立。此方法使用从包含药物评论的公开来源（例如 drug.com）收集的数据集。对采集到的数据进行人工标注并人工验证，确保标注正确。使用 BERT、SciBERT 和 BioBERT 等三种预训练语言模型来获得嵌入，随后将其用作不同机器学习分类器（例如决策树、支持向量机、随机森林和深度学习算法）的特征例如循环神经网络。这些分类器的性能通过精度、召回率和 f1 分数进行量化，结果表明所提出的方法可用于分析人们对不同药物的情绪。

Title: Intellecta Cognitiva: A Comprehensive Dataset for Advancing Academic Knowledge and Machine Reasoning

• Authors: Ajmal PS, Ditto PS, Jithin VG
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13065
• Pdf URL: https://arxiv.org/pdf/2404.13065
• Copy Paste: [[2404.13065]] Intellecta Cognitiva: A Comprehensive Dataset for Advancing Academic Knowledge and Machine Reasoning(https://arxiv.org/abs/2404.13065)
• Keywords: language model
• Abstract: Intellecta dataset emerges as an innovative synthetic dataset, engineered to enhance the cognitive processing capabilities of contemporary language models. With a composition of 11.53 billion tokens, integrating 8.01 billion tokens of synthetic data with 3.52 billion tokens of rich textbook data, Intellecta is crafted to foster advanced reasoning and comprehensive educational narrative generation. Leveraging the Mixtral-8x7B-Instruct-v0.1 model, the dataset facilitates the generation of complex thought processes and detailed, textbook-style explanations, thus enabling language models to engage in both critical thinking and profound educational discourse. This hybrid dataset stands as a testament to the potential of synthetic data in pushing the boundaries of AI, offering a repository that is not only vast and varied but also refined to align with ethical standards and intellectual rigor.
• 摘要：Intellecta 数据集是一个创新的综合数据集，旨在增强当代语言模型的认知处理能力。 Intellecta 由 115.3 亿个代币组成，整合了 80.1 亿个合成数据代币和 35.2 亿个丰富教科书数据代币，旨在促进高级推理和全面的教育叙事生成。利用 Mixtral-8x7B-Instruct-v0.1 模型，该数据集有助于生成复杂的思维过程和详细的教科书式解释，从而使语言模型能够参与批判性思维和深刻的教育话语。这个混合数据集证明了合成数据在突破人工智能边界方面的潜力，提供的存储库不仅庞大且多样化，而且经过完善以符合道德标准和智力严谨性。

Title: Leveraging Large Language Model as Simulated Patients for Clinical Education

• Authors: Yaneng Li, Cheng Zeng, Jialun Zhong, Ruoyu Zhang, Minhao Zhang, Lei Zou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13066
• Pdf URL: https://arxiv.org/pdf/2404.13066
• Copy Paste: [[2404.13066]] Leveraging Large Language Model as Simulated Patients for Clinical Education(https://arxiv.org/abs/2404.13066)
• Keywords: language model, llm, chat
• Abstract: Simulated Patients (SPs) play a crucial role in clinical medical education by providing realistic scenarios for student practice. However, the high cost of training and hiring qualified SPs, along with the heavy workload and potential risks they face in consistently portraying actual patients, limit students' access to this type of clinical training. Consequently, the integration of computer program-based simulated patients has emerged as a valuable educational tool in recent years. With the rapid development of Large Language Models (LLMs), their exceptional capabilities in conversational artificial intelligence and role-playing have been demonstrated, making them a feasible option for implementing Virtual Simulated Patient (VSP). In this paper, we present an integrated model-agnostic framework called CureFun that harnesses the potential of LLMs in clinical medical education. This framework facilitates natural conversations between students and simulated patients, evaluates their dialogue, and provides suggestions to enhance students' clinical inquiry skills. Through comprehensive evaluations, our approach demonstrates more authentic and professional SP-scenario dialogue flows compared to other LLM-based chatbots, thus proving its proficiency in simulating patients. Additionally, leveraging CureFun's evaluation ability, we assess several medical LLMs and discuss the possibilities and limitations of using LLMs as virtual doctors from the perspective of their diagnostic abilities.
• 摘要：模拟患者 (SP) 通过为学生实践提供真实场景，在临床医学教育中发挥着至关重要的作用。然而，培训和雇用合格 SP 的成本高昂，加上他们在持续描绘真实患者时面临的繁重工作量和潜在风险，限制了学生接受此类临床培训。因此，基于计算机程序的模拟患者的集成近年来已成为一种有价值的教育工具。随着大型语言模型（LLM）的快速发展，其在对话式人工智能和角色扮演方面的卓越能力已得到证明，使其成为实施虚拟模拟患者（VSP）的可行选择。在本文中，我们提出了一个名为 CureFun 的集成模型不可知框架，它利用了法学硕士在临床医学教育中的潜力。该框架促进学生和模拟患者之间的自然对话，评估他们的对话，并提供提高学生临床探究技能的建议。通过综合评估，与其他基于LLM的聊天机器人相比，我们的方法展示了更真实、更专业的SP场景对话流程，从而证明了其模拟患者的熟练程度。此外，利用CureFun的评估能力，我们对几位医学法学硕士进行了评估，并从其诊断能力的角度讨论了使用法学硕士作为虚拟医生的可能性和局限性。

Title: Evidence from counterfactual tasks supports emergent analogical reasoning in large language models

• Authors: Taylor Webb, Keith J. Holyoak, Hongjing Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13070
• Pdf URL: https://arxiv.org/pdf/2404.13070
• Copy Paste: [[2404.13070]] Evidence from counterfactual tasks supports emergent analogical reasoning in large language models(https://arxiv.org/abs/2404.13070)
• Keywords: language model
• Abstract: We recently reported evidence that large language models are capable of solving a wide range of text-based analogy problems in a zero-shot manner, indicating the presence of an emergent capacity for analogical reasoning. Two recent commentaries have challenged these results, citing evidence from so-called `counterfactual' tasks in which the standard sequence of the alphabet is arbitrarily permuted so as to decrease similarity with materials that may have been present in the language model's training data. Here, we reply to these critiques, clarifying some misunderstandings about the test materials used in our original work, and presenting evidence that language models are also capable of generalizing to these new counterfactual task variants.
• 摘要：我们最近报告的证据表明，大型语言模型能够以零样本的方式解决各种基于文本的类比问题，这表明类比推理能力的出现。最近的两篇评论对这些结果提出了质疑，引用了所谓“反事实”任务的证据，在这些任务中，字母表的标准序列被任意排列，以减少与语言模型训练数据中可能存在的材料的相似性。在这里，我们回应这些批评，澄清对我们原始工作中使用的测试材料的一些误解，并提供证据表明语言模型也能够泛化到这些新的反事实任务变体。

Title: Modeling Emotions and Ethics with Large Language Models

• Authors: Edward Y. Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13071
• Pdf URL: https://arxiv.org/pdf/2404.13071
• Copy Paste: [[2404.13071]] Modeling Emotions and Ethics with Large Language Models(https://arxiv.org/abs/2404.13071)
• Keywords: language model, llm
• Abstract: This paper explores the integration of human-like emotions and ethical considerations into Large Language Models (LLMs). We first model eight fundamental human emotions, presented as opposing pairs, and employ collaborative LLMs to reinterpret and express these emotions across a spectrum of intensity. Our focus extends to embedding a latent ethical dimension within LLMs, guided by a novel self-supervised learning algorithm with human feedback (SSHF). This approach enables LLMs to perform self-evaluations and adjustments concerning ethical guidelines, enhancing their capability to generate content that is not only emotionally resonant but also ethically aligned. The methodologies and case studies presented herein illustrate the potential of LLMs to transcend mere text and image generation, venturing into the realms of empathetic interaction and principled decision-making, thereby setting a new precedent in the development of emotionally aware and ethically conscious AI systems.
• 摘要：本文探讨了将类人情感和伦理考虑整合到大型语言模型（LLM）中。我们首先对八种基本的人类情感进行建模，以对立的形式呈现，并利用合作法学硕士来重新解释和表达不同强度的这些情感。我们的重点扩展到在法学硕士中嵌入潜在的道德维度，以带有人类反馈的新型自我监督学习算法（SSHF）为指导。这种方法使法学硕士能够对道德准则进行自我评估和调整，从而增强他们生成不仅在情感上引起共鸣而且在道德上一致的内容的能力。本文介绍的方法和案例研究说明了法学硕士超越单纯的文本和图像生成的潜力，涉足同理心互动和原则性决策领域，从而在情感意识和道德意识人工智能系统的发展中树立了新的先例。

Title: Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey

• Authors: Amogh Mannekote
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13074
• Pdf URL: https://arxiv.org/pdf/2404.13074
• Copy Paste: [[2404.13074]] Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey(https://arxiv.org/abs/2404.13074)
• Keywords: language model, llm
• Abstract: Compositional generalization is the ability of a model to generalize to complex, previously unseen types of combinations of entities from just having seen the primitives. This type of generalization is particularly relevant to the semantic parsing community for applications such as task-oriented dialogue, text-to-SQL parsing, and information retrieval, as they can harbor infinite complexity. Despite the success of large language models (LLMs) in a wide range of NLP tasks, unlocking perfect compositional generalization still remains one of the few last unsolved frontiers. The past few years has seen a surge of interest in works that explore the limitations of, methods to improve, and evaluation metrics for compositional generalization capabilities of LLMs for semantic parsing tasks. In this work, we present a literature survey geared at synthesizing recent advances in analysis, methods, and evaluation schemes to offer a starting point for both practitioners and researchers in this area.
• 摘要：组合泛化是模型从刚刚看到的基元中泛化到复杂的、以前未见过的实体组合类型的能力。这种类型的泛化与面向任务的对话、文本到 SQL 解析和信息检索等应用程序的语义解析社区特别相关，因为它们可能具有无限的复杂性。尽管大型语言模型（LLM）在广泛的 NLP 任务中取得了成功，但解锁完美的组合泛化仍然是最后几个未解决的前沿领域之一。在过去的几年里，人们对探索语义解析任务的法学硕士的组合泛化能力的局限性、改进方法和评估指标的研究兴趣激增。在这项工作中，我们提出了一项文献调查，旨在综合分析、方法和评估方案的最新进展，为该领域的从业者和研究人员提供一个起点。

Title: LLM Evaluators Recognize and Favor Their Own Generations

• Authors: Arjun Panickssery, Samuel R. Bowman, Shi Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13076
• Pdf URL: https://arxiv.org/pdf/2404.13076
• Copy Paste: [[2404.13076]] LLM Evaluators Recognize and Favor Their Own Generations(https://arxiv.org/abs/2404.13076)
• Keywords: language model, gpt, llm
• Abstract: Self-evaluation using large language models (LLMs) has proven valuable not only in benchmarking but also methods like reward modeling, constitutional AI, and self-refinement. But new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality. But do LLMs actually recognize their own outputs when they give those texts higher scores, or is it just a coincidence? In this paper, we investigate if self-recognition capability contributes to self-preference. We discover that, out of the box, LLMs such as GPT-4 and Llama 2 have non-trivial accuracy at distinguishing themselves from other LLMs and humans. By fine-tuning LLMs, we discover a linear correlation between self-recognition capability and the strength of self-preference bias; using controlled experiments, we show that the causal explanation resists straightforward confounders. We discuss how self-recognition can interfere with unbiased evaluations and AI safety more generally.
• 摘要：事实证明，使用大型语言模型 (LLM) 进行自我评估不仅在基准测试方面很有价值，而且在奖励建模、宪法人工智能和自我完善等方法方面也很有价值。但由于同一个法学硕士同时充当评估者和被评估者，因此引入了新的偏见。其中一种偏见是自我偏好，法学硕士评估者对自己的输出评分高于其他人的输出，而人类注释者则认为它们具有相同的质量。但是，当法学硕士给这些文本更高的分数时，他们是否真的认识到自己的输出，或者这只是巧合？在本文中，我们研究了自我认知能力是否有助于自我偏好。我们发现，像 GPT-4 和 Llama 2 这样的 LLM 开箱即用，在区分自己与其他 LLM 和人类方面具有非凡的准确性。通过微调法学硕士，我们发现自我识别能力与自我偏好偏差强度之间存在线性相关性；通过对照实验，我们表明因果解释可以抵抗简单的混杂因素。我们更广泛地讨论自我识别如何干扰公正的评估和人工智能安全。

Title: Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning

• Authors: Yilin Gao, Sai Kumar Arava, Yancheng Li, James W. Snyder Jr
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13077
• Pdf URL: https://arxiv.org/pdf/2404.13077
• Copy Paste: [[2404.13077]] Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning(https://arxiv.org/abs/2404.13077)
• Keywords: language model, gpt, llm, prompt
• Abstract: Artificial intelligence (AI) is widely deployed to solve problems related to marketing attribution and budget optimization. However, AI models can be quite complex, and it can be difficult to understand model workings and insights without extensive implementation teams. In principle, recently developed large language models (LLMs), like GPT-4, can be deployed to provide marketing insights, reducing the time and effort required to make critical decisions. In practice, there are substantial challenges that need to be overcome to reliably use such models. We focus on domain-specific question-answering, SQL generation needed for data retrieval, and tabular analysis and show how a combination of semantic search, prompt engineering, and fine-tuning can be applied to dramatically improve the ability of LLMs to execute these tasks accurately. We compare both proprietary models, like GPT-4, and open-source models, like Llama-2-70b, as well as various embedding methods. These models are tested on sample use cases specific to marketing mix modeling and attribution.
• 摘要：人工智能（AI）被广泛部署来解决与营销归因和预算优化相关的问题。然而，人工智能模型可能非常复杂，如果没有广泛的实施团队，就很难理解模型的工作原理和见解。原则上，可以部署最近开发的大型语言模型 (LLM)（例如 GPT-4）来提供营销见解，从而减少做出关键决策所需的时间和精力。在实践中，要可靠地使用此类模型需要克服巨大的挑战。我们专注于特定领域的问答、数据检索所需的 SQL 生成以及表格分析，并展示如何应用语义搜索、提示工程和微调的组合来显着提高法学硕士执行这些任务的能力准确。我们比较了专有模型（如 GPT-4）和开源模型（如 Llama-2-70b）以及各种嵌入方法。这些模型在特定于营销组合建模和归因的示例用例上进行了测试。

Title: Relational Graph Convolutional Networks for Sentiment Analysis

• Authors: Asal Khosravi, Zahed Rahmati, Ali Vefghi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13079
• Pdf URL: https://arxiv.org/pdf/2404.13079
• Copy Paste: [[2404.13079]] Relational Graph Convolutional Networks for Sentiment Analysis(https://arxiv.org/abs/2404.13079)
• Keywords: language model
• Abstract: With the growth of textual data across online platforms, sentiment analysis has become crucial for extracting insights from user-generated content. While traditional approaches and deep learning models have shown promise, they cannot often capture complex relationships between entities. In this paper, we propose leveraging Relational Graph Convolutional Networks (RGCNs) for sentiment analysis, which offer interpretability and flexibility by capturing dependencies between data points represented as nodes in a graph. We demonstrate the effectiveness of our approach by using pre-trained language models such as BERT and RoBERTa with RGCN architecture on product reviews from Amazon and Digikala datasets and evaluating the results. Our experiments highlight the effectiveness of RGCNs in capturing relational information for sentiment analysis tasks.
• 摘要：随着在线平台上文本数据的增长，情感分析对于从用户生成的内容中提取见解变得至关重要。虽然传统方法和深度学习模型已显示出前景，但它们通常无法捕获实体之间的复杂关系。在本文中，我们建议利用关系图卷积网络（RGCN）进行情感分析，它通过捕获图中节点表示的数据点之间的依赖关系来提供可解释性和灵活性。我们通过使用预先训练的语言模型（例如带有 RGCN 架构的 BERT 和 RoBERTa）对 Amazon 和 Digikala 数据集的产品评论进行评估并评估结果，证明了我们方法的有效性。我们的实验强调了 RGCN 在捕获情感分析任务的关系信息方面的有效性。

Title: SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

• Authors: Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, Jinwoo Shin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13081
• Pdf URL: https://arxiv.org/pdf/2404.13081
• Copy Paste: [[2404.13081]] SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs(https://arxiv.org/abs/2404.13081)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising way to improve QA with LLMs, the existing methods often require additional fine-tuning which becomes infeasible with recent LLMs. Augmenting retrieved passages via prompting has the potential to address this limitation, but this direction has been limitedly explored. To this end, we design a simple yet effective framework to enhance open-domain QA (ODQA) with LLMs, based on the summarized retrieval (SuRe). SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval that could be viewed as an explicit rationale extracted from the retrieved passages. Specifically, SuRe first constructs summaries of the retrieved passages for each of the multiple answer candidates. Then, SuRe confirms the most plausible answer from the candidate set by evaluating the validity and ranking of the generated summaries. Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches. SuRe also can be integrated with a broad range of retrieval methods and LLMs. Finally, the generated summaries from SuRe show additional advantages to measure the importance of retrieved passages and serve as more preferred rationales by models and humans.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理任务中取得了显着进步，包括问答 (QA) 任务。虽然将新信息与相关段落的检索相结合是提高法学硕士质量保证的一种有前途的方法，但现有方法通常需要额外的微调，这对于最近的法学硕士来说是不可行的。通过提示增强检索到的段落有可能解决这一限制，但这一方向的探索有限。为此，我们设计了一个简单而有效的框架，以基于汇总检索（SuRe）的法学硕士增强开放域质量保证（ODQA）。 SuRe 帮助法学硕士预测给定问题的更准确答案，这些答案得到了总结性检索的充分支持，可以将其视为从检索到的段落中提取的明确基本原理。具体来说，SuRe 首先为每个多个候选答案构建检索到的段落的摘要。然后，SuRe 通过评估生成的摘要的有效性和排名来确认候选集中最合理的答案。各种 ODQA 基准的实验结果证明了 SuRe 的优越性，与标准提示方法相比，精确匹配 (EM) 提高了 4.6%，F1 分数提高了 4.0%。 SuRe 还可以与广泛的检索方法和法学硕士集成。最后，SuRe 生成的摘要显示出额外的优势，可以衡量检索到的段落的重要性，并作为模型和人类更喜欢的理由。

Title: TREACLE: Thrifty Reasoning via Context-Aware LLM and Prompt Selection

• Authors: Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, Jiasi Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13082
• Pdf URL: https://arxiv.org/pdf/2404.13082
• Copy Paste: [[2404.13082]] TREACLE: Thrifty Reasoning via Context-Aware LLM and Prompt Selection(https://arxiv.org/abs/2404.13082)
• Keywords: language model, llm, prompt
• Abstract: Recent successes in natural language processing have led to the proliferation of large language models (LLMs) by multiple providers. Each LLM offering has different inference accuracy, monetary cost, and latency, and their accuracy further depends on the exact wording of the question (i.e., the specific prompt). At the same time, users often have a limit on monetary budget and latency to answer all their questions, and they do not know which LLMs to choose for each question to meet their accuracy and long-term budget requirements. To navigate this rich design space, we propose TREACLE (Thrifty Reasoning via Context-Aware LLM and Prompt Selection), a reinforcement learning policy that jointly selects the model and prompting scheme while respecting the user's monetary cost and latency constraints. TREACLE uses the problem context, including question text embeddings (reflecting the type or difficulty of a query) and the response history (reflecting the consistency of previous responses) to make smart decisions. Our evaluations on standard reasoning datasets (GSM8K, CSQA, and LLC ) with various LLMs and prompts show that TREACLE enables cost savings of up to 85% compared to baselines while maintaining high accuracy. Importantly, it provides the user with the ability to gracefully trade off accuracy for cost.
• 摘要：最近自然语言处理领域的成功导致了多个提供商的大型语言模型 (LLM) 的激增。每个 LLM 课程都有不同的推理准确性、货币成本和延迟，其准确性进一步取决于问题的确切措辞（即具体提示）。与此同时，用户在回答所有问题时通常会受到金钱预算和延迟的限制，并且他们不知道为每个问题选择哪些法学硕士才能满足其准确性和长期预算要求。为了驾驭这个丰富的设计空间，我们提出了 TREACLE（通过上下文感知 LLM 和提示选择进行节俭推理），这是一种强化学习策略，可以联合选择模型和提示方案，同时尊重用户的货币成本和延迟限制。 TRREACLE 使用问题上下文，包括问题文本嵌入（反映查询的类型或难度）和响应历史记录（反映先前响应的一致性）来做出明智的决策。我们对具有各种 LLM 和提示的标准推理数据集（GSM8K、CSQA 和 LLC）进行的评估表明，与基线相比，TREACLE 可以节省高达 85% 的成本，同时保持高精度。重要的是，它使用户能够优雅地权衡准确性和成本。

Title: Demystifying Legalese: An Automated Approach for Summarizing and Analyzing Overlaps in Privacy Policies and Terms of Service

• Authors: Shikha Soneji, Mitchell Hoesing, Sujay Koujalgi, Jonathan Dodge
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13087
• Pdf URL: https://arxiv.org/pdf/2404.13087
• Copy Paste: [[2404.13087]] Demystifying Legalese: An Automated Approach for Summarizing and Analyzing Overlaps in Privacy Policies and Terms of Service(https://arxiv.org/abs/2404.13087)
• Keywords: language model
• Abstract: The complexities of legalese in terms and policy documents can bind individuals to contracts they do not fully comprehend, potentially leading to uninformed data sharing. Our work seeks to alleviate this issue by developing language models that provide automated, accessible summaries and scores for such documents, aiming to enhance user understanding and facilitate informed decisions. We compared transformer-based and conventional models during training on our dataset, and RoBERTa performed better overall with a remarkable 0.74 F1-score. Leveraging our best-performing model, RoBERTa, we highlighted redundancies and potential guideline violations by identifying overlaps in GDPR-required documents, underscoring the necessity for stricter GDPR compliance.
• 摘要：条款和政策文件中法律术语的复杂性可能会将个人束缚在他们不完全理解的合同上，从而可能导致不知情的数据共享。我们的工作旨在通过开发语言模型来缓解这一问题，这些模型为此类文档提供自动化、可访问的摘要和分数，旨在增强用户理解并促进明智的决策。我们在数据集训练期间比较了基于 Transformer 的模型和传统模型，RoBERTa 总体表现更好，F1 得分高达 0.74。利用我们性能最佳的模型 RoBERTa，我们通过识别 GDPR 要求的文件中的重叠部分来强调冗余和潜在的违反准则的情况，强调更严格的 GDPR 合规性的必要性。

Title: Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

• Authors: Avinash Anand, Mohit Gupta, Kritarth Prasad, Navya Singla, Sanjana Sanjeev, Jatin Kumar, Adarsh Raj Shivam, Rajiv Ratn Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13099
• Pdf URL: https://arxiv.org/pdf/2404.13099
• Copy Paste: [[2404.13099]] Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks(https://arxiv.org/abs/2404.13099)
• Keywords: language model, llm
• Abstract: The rapid progress in the field of natural language processing (NLP) systems and the expansion of large language models (LLMs) have opened up numerous opportunities in the field of education and instructional methods. These advancements offer the potential for tailored learning experiences and immediate feedback, all delivered through accessible and cost-effective services. One notable application area for this technological advancement is in the realm of solving mathematical problems. Mathematical problem-solving not only requires the ability to decipher complex problem statements but also the skill to perform precise arithmetic calculations at each step of the problem-solving process. However, the evaluation of the arithmetic capabilities of large language models remains an area that has received relatively little attention. In response, we introduce an extensive mathematics dataset called "MathQuest" sourced from the 11th and 12th standard Mathematics NCERT textbooks. This dataset encompasses mathematical challenges of varying complexity and covers a wide range of mathematical concepts. Utilizing this dataset, we conduct fine-tuning experiments with three prominent LLMs: LLaMA-2, WizardMath, and MAmmoTH. These fine-tuned models serve as benchmarks for evaluating their performance on our dataset. Our experiments reveal that among the three models, MAmmoTH-13B emerges as the most proficient, achieving the highest level of competence in solving the presented mathematical problems. Consequently, MAmmoTH-13B establishes itself as a robust and dependable benchmark for addressing NCERT mathematics problems.
• 摘要：自然语言处理（NLP）系统领域的快速进步和大型语言模型（LLM）的扩展为教育和教学方法领域开辟了众多机会。这些进步提供了定制学习体验和即时反馈的潜力，所有这些都通过易于访问且具有成本效益的服务提供。这项技术进步的一个值得注意的应用领域是解决数学问题。解决数学问题不仅需要破译复杂问题陈述的能力，还需要在解决问题过程的每一步进行精确算术计算的能力。然而，大型语言模型的算术能力评估仍然是一个相对较少受到关注的领域。作为回应，我们引入了一个名为“MathQuest”的广泛数学数据集，该数据集源自第 11 版和第 12 版标准数学 NCERT 教科书。该数据集包含不同复杂程度的数学挑战，并涵盖广泛的数学概念。利用该数据集，我们对三个著名的法学硕士进行了微调实验：LLaMA-2、WizardMath 和 MAmmoTH。这些经过微调的模型可作为评估其在我们的数据集上的性能的基准。我们的实验表明，在这三个模型中，MAmmoTH-13B 是最熟练的，在解决所提出的数学问题方面达到了最高水平的能力。因此，MAmmoTH-13B 成为解决 NCERT 数学问题的稳健且可靠的基准。

Title: Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging

• Authors: Chia-Hsuan Chang, Mary M. Lucas, Yeawon Lee, Christopher C. Yang, Grace Lu-Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13149
• Pdf URL: https://arxiv.org/pdf/2404.13149
• Copy Paste: [[2404.13149]] Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging(https://arxiv.org/abs/2404.13149)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Advances in large language models (LLMs) have encouraged their adoption in the healthcare domain where vital clinical information is often contained in unstructured notes. Cancer staging status is available in clinical reports, but it requires natural language processing to extract the status from the unstructured text. With the advance in clinical-oriented LLMs, it is promising to extract such status without extensive efforts in training the algorithms. Prompting approaches of the pre-trained LLMs that elicit a model's reasoning process, such as chain-of-thought, may help to improve the trustworthiness of the generated responses. Using self-consistency further improves model performance, but often results in inconsistent generations across the multiple reasoning paths. In this study, we propose an ensemble reasoning approach with the aim of improving the consistency of the model generations. Using an open access clinical large language model to determine the pathologic cancer stage from real-world pathology reports, we show that the ensemble reasoning approach is able to improve both the consistency and performance of the LLM in determining cancer stage, thereby demonstrating the potential to use these models in clinical or other domains where reliability and trustworthiness are critical.
• 摘要：大语言模型 (LLM) 的进步鼓励了它们在医疗保健领域的采用，其中重要的临床信息通常包含在非结构化笔记中。临床报告中提供了癌症分期状态，但需要自然语言处理才能从非结构化文本中提取状态。随着面向临床的法学硕士的进步，无需在算法训练上进行大量努力即可获得这种状态。预先训练的法学硕士的提示方法可以引发模型的推理过程，例如思想链，可能有助于提高生成的响应的可信度。使用自一致性可以进一步提高模型性能，但通常会导致多个推理路径上的生成不一致。在本研究中，我们提出了一种集成推理方法，旨在提高模型生成的一致性。使用开放获取的临床大语言模型从真实世界的病理报告中确定病理癌症分期，我们表明集成推理方法能够提高法学硕士在确定癌症分期方面的一致性和性能，从而证明了在可靠性和可信度至关重要的临床或其他领域使用这些模型。

Title: Heterogeneous Subgraph Transformer for Fake News Detection

• Authors: Yuchen Zhang, Xiaoxiao Ma, Jia Wu, Jian Yang, Hao Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13192
• Pdf URL: https://arxiv.org/pdf/2404.13192
• Copy Paste: [[2404.13192]] Heterogeneous Subgraph Transformer for Fake News Detection(https://arxiv.org/abs/2404.13192)
• Keywords: language model
• Abstract: Fake news is pervasive on social media, inflicting substantial harm on public discourse and societal well-being. We investigate the explicit structural information and textual features of news pieces by constructing a heterogeneous graph concerning the relations among news topics, entities, and content. Through our study, we reveal that fake news can be effectively detected in terms of the atypical heterogeneous subgraphs centered on them, which encapsulate the essential semantics and intricate relations between news elements. However, suffering from the heterogeneity, exploring such heterogeneous subgraphs remains an open problem. To bridge the gap, this work proposes a heterogeneous subgraph transformer (HeteroSGT) to exploit subgraphs in our constructed heterogeneous graph. In HeteroSGT, we first employ a pre-trained language model to derive both word-level and sentence-level semantics. Then the random walk with restart (RWR) is applied to extract subgraphs centered on each news, which are further fed to our proposed subgraph Transformer to quantify the authenticity. Extensive experiments on five real-world datasets demonstrate the superior performance of HeteroSGT over five baselines. Further case and ablation studies validate our motivation and demonstrate that performance improvement stems from our specially designed components.
• 摘要：假新闻在社交媒体上普遍存在，对公众言论和社会福祉造成了重大损害。我们通过构建有关新闻主题、实体和内容之间关系的异构图来研究新闻片段的显式结构信息和文本特征。通过我们的研究，我们发现假新闻可以通过以其为中心的非典型异构子图来有效地检测，这些子图封装了新闻元素之间的基本语义和复杂关系。然而，由于异构性，探索这种异构子图仍然是一个悬而未决的问题。为了弥补这一差距，这项工作提出了一种异构子图转换器（HeteroSGT）来利用我们构建的异构图中的子图。在 HeteroSGT 中，我们首先采用预训练的语言模型来推导单词级和句子级语义。然后应用重新启动随机游走（RWR）来提取以每个新闻为中心的子图，这些子图进一步馈送到我们提出的子图 Transformer 以量化真实性。对五个真实世界数据集的广泛实验证明了 HeteroSGT 相对于五个基线的卓越性能。进一步的案例和消融研究验证了我们的动机，并证明性能的提高源于我们专门设计的组件。

Title: ISQA: Informative Factuality Feedback for Scientific Summarization

• Authors: Zekai Li, Yanxia Qin, Qian Liu, Min-Yen Kan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13246
• Pdf URL: https://arxiv.org/pdf/2404.13246
• Copy Paste: [[2404.13246]] ISQA: Informative Factuality Feedback for Scientific Summarization(https://arxiv.org/abs/2404.13246)
• Keywords: llm, agent
• Abstract: We propose Iterative Facuality Refining on Informative Scientific Question-Answering (ISQA) feedback\footnote{Code is available at \url{https://github.com/lizekai-richard/isqa}}, a method following human learning theories that employs model-generated feedback consisting of both positive and negative information. Through iterative refining of summaries, it probes for the underlying rationale of statements to enhance the factuality of scientific summarization. ISQA does this in a fine-grained manner by asking a summarization agent to reinforce validated statements in positive feedback and fix incorrect ones in negative feedback. Our findings demonstrate that the ISQA feedback mechanism significantly improves the factuality of various open-source LLMs on the summarization task, as evaluated across multiple scientific datasets.
• 摘要：我们提出了对信息科学问答（ISQA）反馈的迭代Facuality Refining\footnote{代码可在\url{https://github.com/lizekai-richard/isqa}}获得，这是一种遵循人类学习理论并采用模型的方法- 生成包含正面和负面信息的反馈。通过对摘要的迭代提炼，探究陈述的基本原理，以增强科学摘要的真实性。 ISQA 以细粒度的方式做到这一点，要求摘要代理强化正面反馈中经过验证的陈述，并修复负面反馈中的错误陈述。我们的研究结果表明，ISQA 反馈机制显着提高了各种开源法学硕士在总结任务上的真实性，正如在多个科学数据集上进行评估的那样。

Title: Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge

• Authors: Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, Gábor Bella
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13292
• Pdf URL: https://arxiv.org/pdf/2404.13292
• Copy Paste: [[2404.13292]] Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge(https://arxiv.org/abs/2404.13292)
• Keywords: language model
• Abstract: The popular subword tokenizers of current language models, such as Byte-Pair Encoding (BPE), are known not to respect morpheme boundaries, which affects the downstream performance of the models. While many improved tokenization algorithms have been proposed, their evaluation and cross-comparison is still an open problem. As a solution, we propose a combined intrinsic-extrinsic evaluation framework for subword tokenization. Intrinsic evaluation is based on our new UniMorph Labeller tool that classifies subword tokenization as either morphological or alien. Extrinsic evaluation, in turn, is performed via the Out-of-Vocabulary Generalization Challenge 1.0 benchmark, which consists of three newly specified downstream text classification tasks. Our empirical findings show that the accuracy of UniMorph Labeller is 98%, and that, in all language models studied (including ALBERT, BERT, RoBERTa, and DeBERTa), alien tokenization leads to poorer generalizations compared to morphological tokenization for semantic compositionality of word meanings.
• 摘要：众所周知，当前语言模型的流行子词标记器（例如字节对编码（BPE））不尊重语素边界，这会影响模型的下游性能。尽管已经提出了许多改进的标记化算法，但它们的评估和交叉比较仍然是一个悬而未决的问题。作为解决方案，我们提出了一个用于子词标记化的内在-外在组合评估框架。内在评估基于我们新的 UniMorph Labeller 工具，该工具将子词标记化分类为形态或外来。反过来，外部评估是通过词汇外泛化挑战 1.0 基准进行的，该基准由三个新指定的下游文本分类任务组成。我们的实证结果表明，UniMorph Labeller 的准确率为 98%，并且在所有研究的语言模型（包括 ALBERT、BERT、RoBERTa 和 DeBERTa）中，与词义语义组合性的形态标记化相比，外来标记化会导致较差的泛化能力。

Title: Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions

• Authors: Soumyadeep Roy, Aparup Khatua, Fatemeh Ghoochani, Uwe Hadler, Wolfgang Nejdl, Niloy Ganguly
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13307
• Pdf URL: https://arxiv.org/pdf/2404.13307
• Copy Paste: [[2404.13307]] Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions(https://arxiv.org/abs/2404.13307)
• Keywords: gpt, llm
• Abstract: GPT-4 demonstrates high accuracy in medical QA tasks, leading with an accuracy of 86.70%, followed by Med-PaLM 2 at 86.50%. However, around 14% of errors remain. Additionally, current works use GPT-4 to only predict the correct option without providing any explanation and thus do not provide any insight into the thinking process and reasoning used by GPT-4 or other LLMs. Therefore, we introduce a new domain-specific error taxonomy derived from collaboration with medical students. Our GPT-4 USMLE Error (G4UE) dataset comprises 4153 GPT-4 correct responses and 919 incorrect responses to the United States Medical Licensing Examination (USMLE) respectively. These responses are quite long (258 words on average), containing detailed explanations from GPT-4 justifying the selected option. We then launch a large-scale annotation study using the Potato annotation platform and recruit 44 medical experts through Prolific, a well-known crowdsourcing platform. We annotated 300 out of these 919 incorrect data points at a granular level for different classes and created a multi-label span to identify the reasons behind the error. In our annotated dataset, a substantial portion of GPT-4's incorrect responses is categorized as a "Reasonable response by GPT-4," by annotators. This sheds light on the challenge of discerning explanations that may lead to incorrect options, even among trained medical professionals. We also provide medical concepts and medical semantic predications extracted using the SemRep tool for every data point. We believe that it will aid in evaluating the ability of LLMs to answer complex medical questions. We make the resources available at https://github.com/roysoumya/usmle-gpt4-error-taxonomy .
• 摘要：GPT-4 在医疗 QA 任务中表现出较高的准确率，领先，准确率为 86.70%，其次是 Med-PaLM 2，准确率为 86.50%。然而，大约 14% 的错误仍然存​​在。此外，当前的工作使用 GPT-4 仅预测正确的选项，而不提供任何解释，因此无法深入了解 GPT-4 或其他法学硕士使用的思维过程和推理。因此，我们引入了一种新的特定领域错误分类法，该分类法是与医学生合作得出的。我们的 GPT-4 USMLE 错误 (G4UE) 数据集分别包含对美国医疗执照考试 (USMLE) 的 4153 个 GPT-4 正确答案和 919 个错误答案。这些回复相当长（平均 258 个字），包含 GPT-4 的详细解释，证明所选选项的合理性。然后，我们利用 Potato 标注平台开展了大规模标注研究，并通过知名众包平台 Prolific 招募了 44 名医学专家。我们对不同类别的 919 个错误数据点中的 300 个进行了细粒度注释，并创建了多标签范围来识别错误背后的原因。在我们带注释的数据集中，GPT-4 错误响应的很大一部分被注释者归类为“GPT-4 的合理响应”。这揭示了即使在训练有素的医疗专业人员中，辨别解释也可能导致错误选择的挑战。我们还提供使用 SemRep 工具针对每个数据点提取的医学概念和医学语义预测。我们相信，它将有助于评估法学硕士回答复杂医学问题的能力。我们在 https://github.com/roysoumya/usmle-gpt4-error-taxonomy 提供资源。

Title: UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions

• Authors: Ana-Cristina Rogoz, Radu Tudor Ionescu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13343
• Pdf URL: https://arxiv.org/pdf/2404.13343
• Copy Paste: [[2404.13343]] UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions(https://arxiv.org/abs/2404.13343)
• Keywords: language model, llm
• Abstract: This work explores a novel data augmentation method based on Large Language Models (LLMs) for predicting item difficulty and response time of retired USMLE Multiple-Choice Questions (MCQs) in the BEA 2024 Shared Task. Our approach is based on augmenting the dataset with answers from zero-shot LLMs (Falcon, Meditron, Mistral) and employing transformer-based models based on six alternative feature combinations. The results suggest that predicting the difficulty of questions is more challenging. Notably, our top performing methods consistently include the question text, and benefit from the variability of LLM answers, highlighting the potential of LLMs for improving automated assessment in medical licensing exams. We make our code available https://github.com/ana-rogoz/BEA-2024.
• 摘要：这项工作探索了一种基于大型语言模型 (LLM) 的新颖数据增强方法，用于预测 BEA 2024 共享任务中已退役的 USMLE 多项选择题 (MCQ) 的项目难度和响应时间。我们的方法基于使用零样本法学硕士（Falcon、Meditron、Mistral）的答案来增强数据集，并采用基于六种替代特征组合的基于变压器的模型。结果表明，预测问题的难度更具挑战性。值得注意的是，我们表现最好的方法始终包括问题文本，并受益于法学硕士答案的可变性，突显了法学硕士在改进医疗许可考试自动化评估方面的潜力。我们提供代码 https://github.com/ana-rogoz/BEA-2024。

Title: Retrieval-Augmented Generation-based Relation Extraction

• Authors: Sefika Efeoglu, Adrian Paschke
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13397
• Pdf URL: https://arxiv.org/pdf/2404.13397
• Copy Paste: [[2404.13397]] Retrieval-Augmented Generation-based Relation Extraction(https://arxiv.org/abs/2404.13397)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Information Extraction (IE) is a transformative process that converts unstructured text data into a structured format by employing entity and relation extraction (RE) methodologies. The identification of the relation between a pair of entities plays a crucial role within this framework. Despite the existence of various techniques for relation extraction, their efficacy heavily relies on access to labeled data and substantial computational resources. In addressing these challenges, Large Language Models (LLMs) emerge as promising solutions; however, they might return hallucinating responses due to their own training data. To overcome these limitations, Retrieved-Augmented Generation-based Relation Extraction (RAG4RE) in this work is proposed, offering a pathway to enhance the performance of relation extraction tasks. This work evaluated the effectiveness of our RAG4RE approach utilizing different LLMs. Through the utilization of established benchmarks, such as TACRED, TACREV, Re-TACRED, and SemEval RE datasets, our aim is to comprehensively evaluate the efficacy of our RAG4RE approach. In particularly, we leverage prominent LLMs including Flan T5, Llama2, and Mistral in our investigation. The results of our study demonstrate that our RAG4RE approach surpasses performance of traditional RE approaches based solely on LLMs, particularly evident in the TACRED dataset and its variations. Furthermore, our approach exhibits remarkable performance compared to previous RE methodologies across both TACRED and TACREV datasets, underscoring its efficacy and potential for advancing RE tasks in natural language processing.
• 摘要：信息提取 (IE) 是一个转换过程，它通过采用实体和关系提取 (RE) 方法将非结构化文本数据转换为结构化格式。识别一对实体之间的关系在此框架中起着至关重要的作用。尽管存在各种关系提取技术，但它们的功效在很大程度上依赖于对标记数据和大量计算资源的访问。在应对这些挑战时，大型语言模型（LLM）成为有前景的解决方案；然而，他们可能会由于自己的训练数据而返回幻觉反应。为了克服这些限制，本文提出了基于检索增强生成的关系提取（RAG4RE），为增强关系提取任务的性能提供了一条途径。这项工作利用不同的法学硕士评估了我们的 RAG4RE 方法的有效性。通过利用已建立的基准，例如 TACRED、TACREV、Re-TACRED 和 SemEval RE 数据集，我们的目标是全面评估 RAG4RE 方法的有效性。特别是，我们在调查中利用了著名的法学硕士，包括 Flan T5、Llama2 和 Mistral。我们的研究结果表明，我们的 RAG4RE 方法超越了仅基于 LLM 的传统 RE 方法的性能，在 TACRED 数据集及其变体中尤其明显。此外，与之前的 RE 方法相比，我们的方法在 TACRED 和 TACREV 数据集上表现出了卓越的性能，强调了其在自然语言处理中推进 RE 任务的功效和潜力。

Title: "A good pun is its own reword": Can Large Language Models Understand Puns?

• Authors: Zhijun Xu, Siyu Yuan, Lingjie Chen, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13599
• Pdf URL: https://arxiv.org/pdf/2404.13599
• Copy Paste: [[2404.13599]] "A good pun is its own reword": Can Large Language Models Understand Puns?(https://arxiv.org/abs/2404.13599)
• Keywords: language model, llm
• Abstract: Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the "lazy pun generation" pattern and identify the primary challenges LLMs encounter in understanding puns.
• 摘要：双关语因其独特的结构和明确的定义，有助于对语言幽默的全面分析，在学术研究中发挥着至关重要的作用。然而，大语言模型（LLM）中对双关语的理解尚未得到彻底检验，限制了它们在创意写作和幽默创作中的使用。在本文中，我们利用三项流行的任务，即双关语识别、解释和生成来系统地评估法学硕士在双关语理解方面的能力。除了采用先前研究中的自动评估指标之外，我们还引入了更适合法学硕士情境学习范式的新评估方法和指标。这些新指标对法学硕士理解双关语的能力提供了更严格的评估，并且比以前的指标更符合人类认知。我们的研究结果揭示了“懒惰双关语生成”模式，并确定了法学硕士在理解双关语时遇到的主要挑战。

Title: NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding

• Authors: Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13627
• Pdf URL: https://arxiv.org/pdf/2404.13627
• Copy Paste: [[2404.13627]] NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding(https://arxiv.org/abs/2404.13627)
• Keywords: language model, llm, chain-of-thought, agent
• Abstract: Large Language Models (LLMs) have sparked substantial interest and debate concerning their potential emergence of Theory of Mind (ToM) ability. Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations, which lacks evaluation of machine ToM ability in real-world human interaction scenarios. This poses a pressing demand to develop new real-world scenario benchmarks. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states (i.e., desires, beliefs, and intentions). Our benchmark builds upon the Belief-Desire-Intention (BDI) agent modeling theory and conducts the necessary empirical experiments to evaluate large language models. Our findings demonstrate that NegotiationToM is challenging for state-of-the-art LLMs, as they consistently perform significantly worse than humans, even when employing the chain-of-thought (CoT) method.
• 摘要：大型语言模型 (LLM) 引发了人们对其心智理论 (ToM) 能力潜在出现的极大兴趣和争论。心智理论评估目前侧重于使用机器生成的数据或容易出现捷径和虚假相关性的游戏设置来测试模型，缺乏对真实人类交互场景中机器 ToM 能力的评估。这就迫切需要开发新的现实场景基准。我们引入了 NegotiationToM，这是一个新的基准，旨在围绕多维心理状态（即愿望、信念和意图）在现实世界谈判中对机器 ToM 进行压力测试。我们的基准建立在信念-欲望-意图（BDI）代理建模理论的基础上，并进行必要的实证实验来评估大型语言模型。我们的研究结果表明，NegotiationToM 对于最先进的法学硕士来说具有挑战性，因为即使采用思想链 (CoT) 方法，他们的表现始终明显低于人类。

Title: Trojan Detection in Large Language Models: Insights from The Trojan Detection Challenge

• Authors: Narek Maloyan, Ekansh Verma, Bulat Nutfullin, Bislan Ashinov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13660
• Pdf URL: https://arxiv.org/pdf/2404.13660
• Copy Paste: [[2404.13660]] Trojan Detection in Large Language Models: Insights from The Trojan Detection Challenge(https://arxiv.org/abs/2404.13660)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in various domains, but their vulnerability to trojan or backdoor attacks poses significant security risks. This paper explores the challenges and insights gained from the Trojan Detection Competition 2023 (TDC2023), which focused on identifying and evaluating trojan attacks on LLMs. We investigate the difficulty of distinguishing between intended and unintended triggers, as well as the feasibility of reverse engineering trojans in real-world scenarios. Our comparative analysis of various trojan detection methods reveals that achieving high Recall scores is significantly more challenging than obtaining high Reverse-Engineering Attack Success Rate (REASR) scores. The top-performing methods in the competition achieved Recall scores around 0.16, comparable to a simple baseline of randomly sampling sentences from a distribution similar to the given training prefixes. This finding raises questions about the detectability and recoverability of trojans inserted into the model, given only the harmful targets. Despite the inability to fully solve the problem, the competition has led to interesting observations about the viability of trojan detection and improved techniques for optimizing LLM input prompts. The phenomenon of unintended triggers and the difficulty in distinguishing them from intended triggers highlights the need for further research into the robustness and interpretability of LLMs. The TDC2023 has provided valuable insights into the challenges and opportunities associated with trojan detection in LLMs, laying the groundwork for future research in this area to ensure their safety and reliability in real-world applications.
• 摘要：大型语言模型 (LLM) 在各个领域都展示了卓越的功能，但它们容易受到木马或后门攻击，从而带来重大的安全风险。本文探讨了 2023 年特洛伊木马检测竞赛 (TDC2023) 所面临的挑战和见解，该竞赛的重点是识别和评估对法学硕士的特洛伊木马攻击。我们研究了区分有意触发和无意触发的难度，以及逆向工程木马在现实场景中的可行性。我们对各种木马检测方法的比较分析表明，获得高召回分数比获得高逆向工程攻击成功率 (REASR) 分数更具挑战性。竞赛中表现最好的方法获得了 0.16 左右的召回分数，与从类似于给定训练前缀的分布中随机采样句子的简单基线相当。这一发现引发了关于插入模型的木马的可检测性和可恢复性的问题（仅考虑有害目标）。尽管无法完全解决问题，但竞赛还是引发了人们对木马检测可行性的有趣观察，并改进了优化 LLM 输入提示的技术。非有意触发的现象以及区分它们与有意触发的困难凸显了对法学硕士的稳健性和可解释性进行进一步研究的必要性。 TDC2023 为法学硕士中与木马检测相关的挑战和机遇提供了宝贵的见解，为该领域的未来研究奠定了基础，以确保其在实际应用中的安全性和可靠性。

Title: How to Encode Domain Information in Relation Classification

• Authors: Elisa Bassignana, Viggo Unmack Gascou, Frida Nøhr Laustsen, Gustav Kristensen, Marie Haahr Petersen, Rob van der Goot, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13760
• Pdf URL: https://arxiv.org/pdf/2404.13760
• Copy Paste: [[2404.13760]] How to Encode Domain Information in Relation Classification(https://arxiv.org/abs/2404.13760)
• Keywords: language model
• Abstract: Current language models require a lot of training data to obtain high performance. For Relation Classification (RC), many datasets are domain-specific, so combining datasets to obtain better performance is non-trivial. We explore a multi-domain training setup for RC, and attempt to improve performance by encoding domain information. Our proposed models improve > 2 Macro-F1 against the baseline setup, and our analysis reveals that not all the labels benefit the same: The classes which occupy a similar space across domains (i.e., their interpretation is close across them, for example "physical") benefit the least, while domain-dependent relations (e.g., "part-of'') improve the most when encoding domain information.
• 摘要：当前的语言模型需要大量的训练数据才能获得高性能。对于关系分类 (RC)，许多数据集都是特定于领域的，因此组合数据集以获得更好的性能并非易事。我们探索了 RC 的多域训练设置，并尝试通过编码域信息来提高性能。我们提出的模型相对于基线设置改进了> 2 Macro-F1，并且我们的分析表明并非所有标签都受益相同：跨域占据相似空间的类（即，它们的解释在它们之间接近，例如“物理”）在编码领域信息时，“”）受益最少，而领域相关关系（例如“部分”）改善最多。

Title: Using Adaptive Empathetic Responses for Teaching English

• Authors: Li Siyan, Teresa Shao, Zhou Yu, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13764
• Pdf URL: https://arxiv.org/pdf/2404.13764
• Copy Paste: [[2404.13764]] Using Adaptive Empathetic Responses for Teaching English(https://arxiv.org/abs/2404.13764)
• Keywords: gpt, prompt, chat
• Abstract: Existing English-teaching chatbots rarely incorporate empathy explicitly in their feedback, but empathetic feedback could help keep students engaged and reduce learner anxiety. Toward this end, we propose the task of negative emotion detection via audio, for recognizing empathetic feedback opportunities in language learning. We then build the first spoken English-teaching chatbot with adaptive, empathetic feedback. This feedback is synthesized through automatic prompt optimization of ChatGPT and is evaluated with English learners. We demonstrate the effectiveness of our system through a preliminary user study.
• 摘要：现有的英语教学聊天机器人很少在反馈中明确地融入同理心，但同理心反馈可以帮助保持学生的参与度并减少学习者的焦虑。为此，我们提出了通过音频进行负面情绪检测的任务，以识别语言学习中的同理心反馈机会。然后，我们构建了第一个具有自适应、同理心反馈的英语口语教学聊天机器人。该反馈是通过 ChatGPT 自动提示优化合成的，并由英语学习者进行评估。我们通过初步的用户研究证明了我们系统的有效性。

Title: Automated Text Mining of Experimental Methodologies from Biomedical Literature

• Authors: Ziqing Guo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13779
• Pdf URL: https://arxiv.org/pdf/2404.13779
• Copy Paste: [[2404.13779]] Automated Text Mining of Experimental Methodologies from Biomedical Literature(https://arxiv.org/abs/2404.13779)
• Keywords: language model
• Abstract: Biomedical literature is a rapidly expanding field of science and technology. Classification of biomedical texts is an essential part of biomedicine research, especially in the field of biology. This work proposes the fine-tuned DistilBERT, a methodology-specific, pre-trained generative classification language model for mining biomedicine texts. The model has proven its effectiveness in linguistic understanding capabilities and has reduced the size of BERT models by 40\% but by 60\% faster. The main objective of this project is to improve the model and assess the performance of the model compared to the non-fine-tuned model. We used DistilBert as a support model and pre-trained on a corpus of 32,000 abstracts and complete text articles; our results were impressive and surpassed those of traditional literature classification methods by using RNN or LSTM. Our aim is to integrate this highly specialised and specific model into different research industries.
• 摘要：生物医学文献是一个迅速扩展的科学技术领域。生物医学文本的分类是生物医学研究的重要组成部分，尤其是在生物学领域。这项工作提出了经过微调的 DistilBERT，这是一种特定于方法的、预先训练的生成分类语言模型，用于挖掘生物医学文本。该模型已经证明了其在语言理解能力方面的有效性，并将 BERT 模型的大小减少了 40%，但速度提高了 60%。该项目的主要目标是改进模型并评估模型与非微调模型相比的性能。我们使用 DistilBert 作为支持模型，并在包含 32,000 篇摘要和完整文本文章的语料库上进行了预训练；我们的结果令人印象深刻，超越了使用 RNN 或 LSTM 的传统文献分类方法。我们的目标是将这种高度专业化和具体的模型整合到不同的研究行业中。

Title: Evaluating Retrieval Quality in Retrieval-Augmented Generation

• Authors: Alireza Salemi, Hamed Zamani
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.13781
• Pdf URL: https://arxiv.org/pdf/2404.13781
• Copy Paste: [[2404.13781]] Evaluating Retrieval Quality in Retrieval-Augmented Generation(https://arxiv.org/abs/2404.13781)
• Keywords: language model, retrieval-augmented generation
• Abstract: Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's $\tau$ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.
• 摘要：评估检索增强生成（RAG）提出了挑战，特别是对于这些系统中的检索模型。传统的端到端评估方法的计算成本很高。此外，基于查询文档相关性标签的检索模型性能评估显示与 RAG 系统的下游性能有很小的相关性。我们提出了一种新颖的评估方法 eRAG，其中检索列表中的每个文档都由 RAG 系统内的大型语言模型单独使用。然后根据下游任务的真实标签评估为每个文档生成的输出。以这种方式，每个文档的下游性能充当其相关性标签。我们采用各种下游任务指标来获取文档级注释，并使用基于集合的指标或排名指标来聚合它们。对各种数据集的大量实验表明，与基线方法相比，eRAG 与下游 RAG 性能实现了更高的相关性，Kendall 的 $\tau$ 相关性提高了 0.168 到 0.494。此外，eRAG 还具有显着的计算优势，可提高运行时间，并且与端到端评估相比，消耗的 GPU 内存最多可减少 50 倍。

Title: From LLM to NMT: Advancing Low-Resource Machine Translation with Claude

• Authors: Maxim Enis, Mark Hopkins
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13813
• Pdf URL: https://arxiv.org/pdf/2404.13813
• Copy Paste: [[2404.13813]] From LLM to NMT: Advancing Low-Resource Machine Translation with Claude(https://arxiv.org/abs/2404.13813)
• Keywords: language model, llm
• Abstract: We show that Claude 3 Opus, a large language model (LLM) released by Anthropic in March 2024, exhibits stronger machine translation competence than other LLMs. Though we find evidence of data contamination with Claude on FLORES-200, we curate new benchmarks that corroborate the effectiveness of Claude for low-resource machine translation into English. We find that Claude has remarkable \textit{resource efficiency} -- the degree to which the quality of the translation model depends on a language pair's resource level. Finally, we show that advancements in LLM translation can be compressed into traditional neural machine translation (NMT) models. Using Claude to generate synthetic data, we demonstrate that knowledge distillation advances the state-of-the-art in Yoruba-English translation, meeting or surpassing strong baselines like NLLB-54B and Google Translate.
• 摘要：我们发现 Anthropic 于 2024 年 3 月发布的大型语言模型（LLM）Claude 3 Opus 表现出比其他 LLM 更强的机器翻译能力。尽管我们在 FLORES-200 上发现了 Claude 数据污染的证据，但我们策划了新的基准来证实 Claude 在低资源机器翻译成英语方面的有效性。我们发现 Claude 具有显着的\textit{资源效率}——翻译模型的质量取决于语言对的资源水平的程度。最后，我们表明 LLM 翻译的进步可以压缩到传统的神经机器翻译（NMT）模型中。使用 Claude 生成合成数据，我们证明知识蒸馏提高了约鲁巴语-英语翻译的最先进水平，达到或超越了 NLLB-54B 和谷歌翻译等强大的基线。

Title: Understanding the role of FFNs in driving multilingual behaviour in LLMs

• Authors: Sunit Bhattacharya, Ondřej Bojar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13855
• Pdf URL: https://arxiv.org/pdf/2404.13855
• Copy Paste: [[2404.13855]] Understanding the role of FFNs in driving multilingual behaviour in LLMs(https://arxiv.org/abs/2404.13855)
• Keywords: language model, llm
• Abstract: Multilingualism in Large Language Models (LLMs) is an yet under-explored area. In this paper, we conduct an in-depth analysis of the multilingual capabilities of a family of a Large Language Model, examining its architecture, activation patterns, and processing mechanisms across languages. We introduce novel metrics to probe the model's multilingual behaviour at different layers and shed light on the impact of architectural choices on multilingual processing. Our findings reveal different patterns of multilinugal processing in the sublayers of Feed-Forward Networks of the models. Furthermore, we uncover the phenomenon of "over-layerization" in certain model configurations, where increasing layer depth without corresponding adjustments to other parameters may degrade model performance. Through comparisons within and across languages, we demonstrate the interplay between model architecture, layer depth, and multilingual processing capabilities of LLMs trained on multiple languages.
• 摘要：大语言模型（LLM）中的多语言是一个尚未充分探索的领域。在本文中，我们对大型语言模型家族的多语言能力进行了深入分析，检查其架构、激活模式和跨语言的处理机制。我们引入了新颖的指标来探究模型在不同层的多语言行为，并阐明架构选择对多语言处理的影响。我们的研究结果揭示了模型前馈网络子层中多语言处理的不同模式。此外，我们发现了某些模型配置中的“过度分层”现象，其中增加层深度而不对其他参数进行相应调整可能会降低模型性能。通过语言内部和跨语言的比较，我们展示了模型架构、层深度和接受多种语言训练的法学硕士的多语言处理能力之间的相互作用。

Title: Context-Enhanced Language Models for Generating Multi-Paper Citations

• Authors: Avinash Anand, Kritarth Prasad, Ujjwal Goel, Mohit Gupta, Naman Lal, Astha Verma, Rajiv Ratn Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13865
• Pdf URL: https://arxiv.org/pdf/2404.13865
• Copy Paste: [[2404.13865]] Context-Enhanced Language Models for Generating Multi-Paper Citations(https://arxiv.org/abs/2404.13865)
• Keywords: language model, llm, prompt
• Abstract: Citation text plays a pivotal role in elucidating the connection between scientific documents, demanding an in-depth comprehension of the cited paper. Constructing citations is often time-consuming, requiring researchers to delve into extensive literature and grapple with articulating relevant content. To address this challenge, the field of citation text generation (CTG) has emerged. However, while earlier methods have primarily centered on creating single-sentence citations, practical scenarios frequently necessitate citing multiple papers within a single paragraph. To bridge this gap, we propose a method that leverages Large Language Models (LLMs) to generate multi-citation sentences. Our approach involves a single source paper and a collection of target papers, culminating in a coherent paragraph containing multi-sentence citation text. Furthermore, we introduce a curated dataset named MCG-S2ORC, composed of English-language academic research papers in Computer Science, showcasing multiple citation instances. In our experiments, we evaluate three LLMs LLaMA, Alpaca, and Vicuna to ascertain the most effective model for this endeavor. Additionally, we exhibit enhanced performance by integrating knowledge graphs from target papers into the prompts for generating citation text. This research underscores the potential of harnessing LLMs for citation generation, opening a compelling avenue for exploring the intricate connections between scientific documents.
• 摘要：引文文本在阐明科学文献之间的联系方面发挥着关键作用，要求深入理解被引用的论文。构建引文通常非常耗时，需要研究人员深入研究大量文献并努力阐明相关内容。为了应对这一挑战，引文文本生成（CTG）领域应运而生。然而，虽然早期的方法主要集中于创建单句引用，但实际场景经常​​需要在单个段落中引用多篇论文。为了弥补这一差距，我们提出了一种利用大型语言模型（LLM）生成多重引用句子的方法。我们的方法涉及单个源论文和目标论文的集合，最终形成包含多句引用文本的连贯段落。此外，我们还引入了一个名为 MCG-S2ORC 的精选数据集，由计算机科学领域的英语学术研究论文组成，展示了多个引用实例。在我们的实验中，我们评估了三位法学硕士 LLaMA、Alpaca 和 Vicuna，以确定最有效的模型。此外，我们通过将目标论文的知识图集成到生成引文文本的提示中来展示增强的性能。这项研究强调了利用法学硕士进行引文生成的潜力，为探索科学文献之间复杂的联系开辟了一条引人注目的途径。

Title: VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models

• Authors: Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, Nanyun Peng
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13874
• Pdf URL: https://arxiv.org/pdf/2404.13874
• Copy Paste: [[2404.13874]] VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models(https://arxiv.org/abs/2404.13874)
• Keywords: language model, llm, hallucination
• Abstract: Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs, undermining their reliability. A comprehensive quantitative evaluation is necessary to identify and understand the extent of hallucinations in these models. However, existing benchmarks are often limited in scope, focusing mainly on object hallucinations. Furthermore, current evaluation methods struggle to effectively address the subtle semantic distinctions between model outputs and reference data, as well as the balance between hallucination and informativeness. To address these issues, we introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases. Moreover, we propose an large language model (LLM)-based two-stage evaluation framework that generalizes the popular CHAIR metric and incorporates both faithfulness and coverage into the evaluation. Experiments on 10 established LVLMs demonstrate that our evaluation metric is more comprehensive and better correlated with humans than existing work when evaluating on our challenging human annotated benchmark dataset. Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.
• 摘要：大视觉语言模型 (LVLM) 存在幻觉问题，其中模型会生成听起来合理但实际上不正确的输出，从而损害了其可靠性。为了识别和理解这些模型中幻觉的程度，需要进行全面的定量评估。然而，现有的基准通常范围有限，主要集中在物体幻觉上。此外，当前的评估方法难以有效解决模型输出和参考数据之间微妙的语义区别，以及幻觉和信息性之间的平衡。为了解决这些问题，我们引入了一个涵盖对象、属性和关系的多维基准，并根据关联偏差选择具有挑战性的图像。此外，我们提出了一种基于大型语言模型（LLM）的两阶段评估框架，该框架概括了流行的 CHAIR 指标，并将忠实度和覆盖率纳入评估中。对 10 个已建立的 LVLM 进行的实验表明，在评估我们具有挑战性的人类注释基准数据集时，我们的评估指标比现有工作更全面，并且与人类的相关性更好。我们的工作还强调了模型输出的忠实度和覆盖范围之间的关键平衡，并鼓励未来的工作解决 LVLM 中的幻觉，同时保持输出的信息丰富。

Title: Towards Better Text-to-Image Generation Alignment via Attention Modulation

• Authors: Yihang Wu, Xiao Cao, Kaixin Li, Zitan Chen, Haonan Wang, Lei Meng, Zhiyong Huang
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.13899
• Pdf URL: https://arxiv.org/pdf/2404.13899
• Copy Paste: [[2404.13899]] Towards Better Text-to-Image Generation Alignment via Attention Modulation(https://arxiv.org/abs/2404.13899)
• Keywords: prompt
• Abstract: In text-to-image generation tasks, the advancements of diffusion models have facilitated the fidelity of generated results. However, these models encounter challenges when processing text prompts containing multiple entities and attributes. The uneven distribution of attention results in the issues of entity leakage and attribute misalignment. Training from scratch to address this issue requires numerous labeled data and is resource-consuming. Motivated by this, we propose an attribution-focusing mechanism, a training-free phase-wise mechanism by modulation of attention for diffusion model. One of our core ideas is to guide the model to concentrate on the corresponding syntactic components of the prompt at distinct timesteps. To achieve this, we incorporate a temperature control mechanism within the early phases of the self-attention modules to mitigate entity leakage issues. An object-focused masking scheme and a phase-wise dynamic weight control mechanism are integrated into the cross-attention modules, enabling the model to discern the affiliation of semantic information between entities more effectively. The experimental results in various alignment scenarios demonstrate that our model attain better image-text alignment with minimal additional computational cost.
• 摘要：在文本到图像的生成任务中，扩散模型的进步促进了生成结果的保真度。然而，这些模型在处理包含多个实体和属性的文本提示时遇到了挑战。注意力分布不均导致实体泄漏和属性错位的问题。从头开始训练以解决这个问题需要大量标记数据并且耗费资源。受此启发，我们提出了一种归因聚焦机制，即一种通过调节注意力来为扩散模型实现的无需训练的分阶段机制。我们的核心思想之一是引导模型在不同的时间步骤上集中注意力于提示的相应句法成分。为了实现这一点，我们在自注意力模块的早期阶段加入了温度控制机制来缓解实体泄漏问题。以对象为中心的掩蔽方案和分阶段动态权重控制机制被集成到交叉注意力模块中，使模型能够更有效地辨别实体之间的语义信息从属关系。各种对齐场景中的实验结果表明，我们的模型以最小的额外计算成本实现了更好的图像-文本对齐。

Title: Generating Attractive and Authentic Copywriting from Customer Reviews

• Authors: Yu-Xiang Lin, Wei-Yun Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13906
• Pdf URL: https://arxiv.org/pdf/2404.13906
• Copy Paste: [[2404.13906]] Generating Attractive and Authentic Copywriting from Customer Reviews(https://arxiv.org/abs/2404.13906)
• Keywords: language model, gpt, llm, chat
• Abstract: The goal of product copywriting is to capture the interest of potential buyers by emphasizing the features of products through text descriptions. As e-commerce platforms offer a wide range of services, it's becoming essential to dynamically adjust the styles of these auto-generated descriptions. Typical approaches to copywriting generation often rely solely on specified product attributes, which may result in dull and repetitive content. To tackle this issue, we propose to generate copywriting based on customer reviews, as they provide firsthand practical experiences with products, offering a richer source of information than just product attributes. We have developed a sequence-to-sequence framework, enhanced with reinforcement learning, to produce copywriting that is attractive, authentic, and rich in information. Our framework outperforms all existing baseline and zero-shot large language models, including LLaMA-2-chat-7B and GPT-3.5, in terms of both attractiveness and faithfulness. Furthermore, this work features the use of LLMs for aspect-based summaries collection and argument allure assessment. Experiments demonstrate the effectiveness of using LLMs for marketing domain corpus construction. The code and the dataset is publicly available at: https://github.com/YuXiangLin1234/Copywriting-Generation.
• 摘要：产品文案的目标是通过文字描述强调产品的特点来吸引潜在买家的兴趣。随着电子商务平台提供广泛的服务，动态调整这些自动生成的描述的样式变得至关重要。典型的文案生成方法通常仅依赖于指定的产品属性，这可能会导致内容枯燥且重复。为了解决这个问题，我们建议根据客户评论生成文案，因为它们提供了产品的第一手实践经验，提供了比产品属性更丰富的信息来源。我们开发了一个序列到序列的框架，并通过强化学习进行增强，以生成有吸引力、真实且信息丰富的文案。我们的框架在吸引力和忠实度方面优于所有现有的基线和零样本大型语言模型，包括 LLaMA-2-chat-7B 和 GPT-3.5。此外，这项工作的特点是使用法学硕士进行基于方面的摘要收集和论点吸引力评估。实验证明了使用法学硕士进行营销领域语料库构建的有效性。代码和数据集可公开获取：https://github.com/YuXiangLin1234/Copywriting-Generation。

Title: Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models

• Authors: Yukyung Lee, Soonwon Ka, Bokyung Son, Pilsung Kang, Jaewook Kang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.13919
• Pdf URL: https://arxiv.org/pdf/2404.13919
• Copy Paste: [[2404.13919]] Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models(https://arxiv.org/abs/2404.13919)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have significantly impacted the writing process, enabling collaborative content creation and enhancing productivity. However, generating high-quality, user-aligned text remains challenging. In this paper, we propose Writing Path, a framework that uses explicit outlines to guide LLMs in generating goal-oriented, high-quality pieces of writing. Our approach draws inspiration from structured writing planning and reasoning paths, focusing on capturing and reflecting user intentions throughout the writing process. We construct a diverse dataset from unstructured blog posts to benchmark writing performance and introduce a comprehensive evaluation framework assessing the quality of outlines and generated texts. Our evaluations with GPT-3.5-turbo, GPT-4, and HyperCLOVA X demonstrate that the Writing Path approach significantly enhances text quality according to both LLMs and human evaluations. This study highlights the potential of integrating writing-specific techniques into LLMs to enhance their ability to meet the diverse writing needs of users.
• 摘要：大型语言模型 (LLM) 极大地影响了写作过程，支持协作内容创建并提高生产力。然而，生成高质量、用户对齐的文本仍然具有挑战性。在本文中，我们提出了写作路径，这是一个使用明确的大纲来指导法学硕士生成以目标为导向的高质量写作的框架。我们的方法从结构化写作规划和推理路径中汲取灵感，专注于在整个写作过程中捕捉和反映用户意图。我们从非结构化博客文章构建了一个多样化的数据集来衡量写作表现，并引入了一个全面的评估框架来评估大纲和生成文本的质量。我们对 GPT-3.5-turbo、GPT-4 和 HyperCLOVA X 的评估表明，根据法学硕士和人工评估，写作路径方法显着提高了文本质量。这项研究强调了将写作特定技术融入法学硕士的潜力，以增强其满足用户多样化写作需求的能力。

Title: MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit

• Authors: Boning Zhang, Chengxi Li, Kai Fan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13925
• Pdf URL: https://arxiv.org/pdf/2404.13925
• Copy Paste: [[2404.13925]] MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit(https://arxiv.org/abs/2404.13925)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have been explored in a variety of reasoning tasks including solving of mathematical problems. Each math dataset typically includes its own specially designed evaluation script, which, while suitable for its intended use, lacks generalizability across different datasets. Consequently, updates and adaptations to these evaluation tools tend to occur without being systematically reported, leading to inconsistencies and obstacles to fair comparison across studies. To bridge this gap, we introduce a comprehensive mathematical evaluation toolkit that not only utilizes a python computer algebra system (CAS) for its numerical accuracy, but also integrates an optional LLM, known for its considerable natural language processing capabilities. To validate the effectiveness of our toolkit, we manually annotated two distinct datasets. Our experiments demonstrate that the toolkit yields more robust evaluation results compared to prior works, even without an LLM. Furthermore, when an LLM is incorporated, there is a notable enhancement. The code for our method will be made available at \url{https://github.com/MARIO-Math-Reasoning/math_evaluation}.
• 摘要：大型语言模型 (LLM) 已在各种推理任务中进行了探索，包括解决数学问题。每个数学数据集通常都包含自己专门设计的评估脚本，该脚本虽然适合其预期用途，但缺乏跨不同数据集的通用性。因此，这些评估工具的更新和调整往往没有得到系统报告，从而导致研究之间的不一致和公平比较的障碍。为了弥补这一差距，我们引入了一个全面的数学评估工具包，该工具包不仅利用 Python 计算机代数系统 (CAS) 来提高数值准确性，而且还集成了可选的法学硕士，该工具以其强大的自然语言处理能力而闻名。为了验证我们的工具包的有效性，我们手动注释了两个不同的数据集。我们的实验表明，即使没有法学硕士，该工具包也能比之前的工作产生更稳健的评估结果。此外，当加入法学硕士后，会有显着的增强。我们的方法的代码将在 \url{https://github.com/MARIO-Math-Reasoning/math_evaluation} 中提供。

Title: A User-Centric Benchmark for Evaluating Large Language Models

• Authors: Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, Jian-Yun Nie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13940
• Pdf URL: https://arxiv.org/pdf/2404.13940
• Copy Paste: [[2404.13940]] A User-Centric Benchmark for Evaluating Large Language Models(https://arxiv.org/abs/2404.13940)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are essential tools to collaborate with users on different tasks. Evaluating their performance to serve users' needs in real-world scenarios is important. While many benchmarks have been created, they mainly focus on specific predefined model abilities. Few have covered the intended utilization of LLMs by real users. To address this oversight, we propose benchmarking LLMs from a user perspective in both dataset construction and evaluation designs. We first collect 1863 real-world use cases with 15 LLMs from a user study with 712 participants from 23 countries. These self-reported cases form the User Reported Scenarios(URS) dataset with a categorization of 7 user intents. Secondly, on this authentic multi-cultural dataset, we benchmark 10 LLM services on their efficacy in satisfying user needs. Thirdly, we show that our benchmark scores align well with user-reported experience in LLM interactions across diverse intents, both of which emphasize the overlook of subjective scenarios. In conclusion, our study proposes to benchmark LLMs from a user-centric perspective, aiming to facilitate evaluations that better reflect real user needs. The benchmark dataset and code are available at https://github.com/Alice1998/URS.
• 摘要：大型语言模型 (LLM) 是与用户协作完成不同任务的重要工具。评估它们在现实场景中满足用户需求的性能非常重要。虽然已经创建了许多基准，但它们主要关注特定的预定义模型能力。很少有人涵盖了真实用户对法学硕士的预期用途。为了解决这种监督问题，我们建议从用户的角度在数据集构建和评估设计中对法学硕士进行基准测试。我们首先从一项对来自 23 个国家的 712 名参与者进行的用户研究中收集了 15 名法学硕士的 1863 个现实世界用例。这些自我报告的案例形成了用户报告场景 (URS) 数据集，其中包含 7 个用户意图的分类。其次，在这个真实的多元文化数据集上，我们对 10 个法学硕士服务满足用户需求的功效进行了基准测试。第三，我们表明，我们的基准分数与用户报告的跨不同意图的法学硕士互动体验非常吻合，两者都强调了对主观场景的忽视。总之，我们的研究建议从以用户为中心的角度对法学硕士进行基准测试，旨在促进更好地反映用户真实需求的评估。基准数据集和代码可在 https://github.com/Alice1998/URS 上获取。

Title: Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations

• Authors: Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, Jong C. Park
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13948
• Pdf URL: https://arxiv.org/pdf/2404.13948
• Copy Paste: [[2404.13948]] Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations(https://arxiv.org/abs/2404.13948)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The robustness of recent Large Language Models (LLMs) has become increasingly crucial as their applicability expands across various domains and real-world applications. Retrieval-Augmented Generation (RAG) is a promising solution for addressing the limitations of LLMs, yet existing studies on the robustness of RAG often overlook the interconnected relationships between RAG components or the potential threats prevalent in real-world databases, such as minor textual errors. In this work, we investigate two underexplored aspects when assessing the robustness of RAG: 1) vulnerability to noisy documents through low-level perturbations and 2) a holistic evaluation of RAG robustness. Furthermore, we introduce a novel attack method, the Genetic Attack on RAG (\textit{GARAG}), which targets these aspects. Specifically, GARAG is designed to reveal vulnerabilities within each component and test the overall system functionality against noisy documents. We validate RAG robustness by applying our \textit{GARAG} to standard QA datasets, incorporating diverse retrievers and LLMs. The experimental results show that GARAG consistently achieves high attack success rates. Also, it significantly devastates the performance of each component and their synergy, highlighting the substantial risk that minor textual inaccuracies pose in disrupting RAG systems in the real world.
• 摘要：随着大型语言模型 (LLM) 的适用性扩展到各个领域和实际应用，其稳健性变得越来越重要。检索增强生成（RAG）是解决法学硕士局限性的一种有前途的解决方案，但现有的关于 RAG 稳健性的研究经常忽视 RAG 组件之间的相互关联或现实世界数据库中普遍存在的潜在威胁，例如微小的文本错误。在这项工作中，我们在评估 RAG 的稳健性时研究了两个尚未充分探索的方面：1）通过低级扰动对噪声文档的脆弱性；2）RAG 稳健性的整体评估。此外，我们引入了一种新的攻击方法，即针对这些方面的 RAG 遗传攻击（\textit{GARAG}）。具体来说，GARAG 旨在揭示每个组件内的漏洞，并针对嘈杂的文档测试整体系统功能。我们通过将 \textit{GARAG} 应用于标准 QA 数据集，结合不同的检索器和 LLM 来验证 RAG 的稳健性。实验结果表明，GARAG 始终获得较高的攻击成功率。此外，它还严重破坏了每个组件的性能及其协同作用，突显了现实世界中微小的文本不准确会带来破坏 RAG 系统的巨大风险。

Title: How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO

• Authors: Man Tik Ng, Hui Tung Tse, Jen-tse Huang, Jingjing Li, Wenxuan Wang, Michael R. Lyu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13957
• Pdf URL: https://arxiv.org/pdf/2404.13957
• Copy Paste: [[2404.13957]] How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO(https://arxiv.org/abs/2404.13957)
• Keywords: language model, gpt, llm, chat
• Abstract: The role-play ability of Large Language Models (LLMs) has emerged as a popular research direction. However, existing studies focus on imitating well-known public figures or fictional characters, overlooking the potential for simulating ordinary individuals. Such an oversight limits the potential for advancements in digital human clones and non-player characters in video games. To bridge this gap, we introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. Notably, our framework focuses on emulating average individuals rather than historical or fictional figures, presenting a unique advantage to apply the Turing Test. We evaluated three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models, alongside the online application GPTs from OpenAI. Our results demonstrate that GPT-4 more effectively deceives human evaluators, and GPTs achieves a leading success rate of 48.3%. Furthermore, we investigated whether LLMs could discern between human-generated and machine-generated texts. While GPT-4 can identify differences, it could not determine which texts were human-produced. Our code and results of reproducing the role-playing LLMs are made publicly available via https://github.com/CUHK-ARISE/ECHO.
• 摘要：大型语言模型（LLM）的角色扮演能力已成为热门研究方向。然而，现有的研究侧重于模仿知名公众人物或虚构人物，忽视了模拟普通个体的潜力。这种疏忽限制了电子游戏中数字克隆人和非玩家角色的进步潜力。为了弥补这一差距，我们引入了 ECHO，这是一个受图灵测试启发的评估框架。该框架让目标个体的熟人参与进来，以区分人类和机器生成的响应。值得注意的是，我们的框架侧重于模拟普通个人而不是历史或虚构人物，为应用图灵测试提供了独特的优势。我们使用 ECHO 评估了三个角色扮演法学硕士，其中 GPT-3.5 和 GPT-4 作为基础模型，以及来自 OpenAI 的在线应用程序 GPT。我们的结果表明，GPT-4 能够更有效地欺骗人类评估者，GPT 的成功率领先，达到 48.3%。此外，我们还研究了法学硕士是否能够区分人类生成的文本和机器生成的文本。虽然 GPT-4 可以识别差异，但它无法确定哪些文本是人类生成的。我们的代码和复制角色扮演法学硕士的结果通过 https://github.com/CUHK-ARISE/ECHO 公开发布。

Title: Protecting Your LLMs with Information Bottleneck

• Authors: Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, Jiang Bian
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13968
• Pdf URL: https://arxiv.org/pdf/2404.13968
• Copy Paste: [[2404.13968]] Protecting Your LLMs with Information Bottleneck(https://arxiv.org/abs/2404.13968)
• Keywords: language model, llm, prompt
• Abstract: The advent of large language models (LLMs) has revolutionized the field of natural language processing, yet they might be attacked to produce harmful content. Despite efforts to ethically align LLMs, these are often fragile and can be circumvented by jailbreaking attacks through optimized or manual adversarial prompts. To address this, we introduce the Information Bottleneck Protector (IBProtector), a defense mechanism grounded in the information bottleneck principle, and we modify the objective to avoid trivial solutions. The IBProtector selectively compresses and perturbs prompts, facilitated by a lightweight and trainable extractor, preserving only essential information for the target LLMs to respond with the expected answer. Moreover, we further consider a situation where the gradient is not visible to be compatible with any LLM. Our empirical evaluations show that IBProtector outperforms current defense methods in mitigating jailbreak attempts, without overly affecting response quality or inference speed. Its effectiveness and adaptability across various attack methods and target LLMs underscore the potential of IBProtector as a novel, transferable defense that bolsters the security of LLMs without requiring modifications to the underlying models.
• 摘要：大型语言模型 (LLM) 的出现彻底改变了自然语言处理领域，但它们可能会受到攻击而产生有害内容。尽管努力使法学硕士在道德上保持一致，但这些通常很脆弱，可以通过优化或手动对抗性提示通过越狱攻击来规避。为了解决这个问题，我们引入了信息瓶颈保护器（IBProtector），这是一种基于信息瓶颈原理的防御机制，并且我们修改了目标以避免琐碎的解决方案。 IBProtector 有选择地压缩和扰乱提示，由轻量级且可训练的提取器促进，仅保留目标 LLM 响应预期答案的基本信息。此外，我们进一步考虑梯度不可见的情况与任何LLM兼容。我们的实证评估表明，IBProtector 在减少越狱尝试方面优于当前的防御方法，而不会过度影响响应质量或推理速度。它在各种攻击方法和目标 LLM 中的有效性和适应性强调了 IBProtector 作为一种新颖的、可转移的防御的潜力，可以增强 LLM 的安全性，而无需修改底层模型。

Title: Information Re-Organization Improves Reasoning in Large Language Models

• Authors: Xiaoxia Cheng, Zeqi Tan, Weiming Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13985
• Pdf URL: https://arxiv.org/pdf/2404.13985
• Copy Paste: [[2404.13985]] Information Re-Organization Improves Reasoning in Large Language Models(https://arxiv.org/abs/2404.13985)
• Keywords: language model, gpt, llm
• Abstract: Improving the reasoning capabilities of large language models (LLMs) has attracted considerable interest. Recent approaches primarily focus on improving the reasoning process to yield a more precise final answer. However, in scenarios involving contextually aware reasoning, these methods neglect the importance of first identifying logical relationships from the context before proceeding with the reasoning. This oversight could lead to a superficial understanding and interaction with the context, potentially undermining the quality and reliability of the reasoning outcomes. In this paper, we propose an information re-organization (InfoRE) method before proceeding with the reasoning to enhance the reasoning ability of LLMs. We first perform a re-organization processing of the contextual content, e.g., documents or paragraphs, to obtain logical relationships. Then, we utilize the re-organized information in the reasoning process. This enables LLMs to deeply understand the contextual content by clearly perceiving these logical relationships. To demonstrate the effectiveness of our approach in improving the reasoning ability, we conduct experiments using Llama2-70B, GPT-3.5, and GPT-4 on various contextually aware multi-hop reasoning tasks. Using only a zero-shot setting, our method achieves an average improvement of 3\% across all tasks, highlighting its potential to improve the reasoning performance of LLMs. Our source code is available at https://github.com/hustcxx/InfoRE.
• 摘要：提高大型语言模型（LLM）的推理能力引起了人们极大的兴趣。最近的方法主要集中在改进推理过程以产生更精确的最终答案。然而，在涉及上下文感知推理的场景中，这些方法忽略了在进行推理之前首先从上下文中识别逻辑关系的重要性。这种疏忽可能会导致对上下文的肤浅理解和交互，从而可能损害推理结果的质量和可靠性。在本文中，我们在进行推理之前提出了一种信息重组（InfoRE）方法，以增强法学硕士的推理能力。我们首先对上下文内容（例如文档或段落）进行重新组织处理，以获得逻辑关系。然后，我们在推理过程中利用重新组织的信息。这使得法学硕士能够通过清楚地感知这些逻辑关系来深入理解上下文内容。为了证明我们的方法在提高推理能力方面的有效性，我们使用 Llama2-70B、GPT-3.5 和 GPT-4 在各种上下文感知多跳推理任务上进行了实验。仅使用零样本设置，我们的方法在所有任务中平均提高了 3%，凸显了其提高 LLM 推理性能的潜力。我们的源代码可在 https://github.com/hustcxx/InfoRE 获取。

Title: LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation

• Authors: Keheng Wang, Feiyu Duan, Peiguang Li, Sirui Wang, Xunliang Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14043
• Pdf URL: https://arxiv.org/pdf/2404.14043
• Copy Paste: [[2404.14043]] LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation(https://arxiv.org/abs/2404.14043)
• Keywords: llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) demonstrates great value in alleviating outdated knowledge or hallucination by supplying LLMs with updated and relevant knowledge. However, there are still several difficulties for RAG in understanding complex multi-hop query and retrieving relevant documents, which require LLMs to perform reasoning and retrieve step by step. Inspired by human's reasoning process in which they gradually search for the required information, it is natural to ask whether the LLMs could notice the missing information in each reasoning step. In this work, we first experimentally verified the ability of LLMs to extract information as well as to know the missing. Based on the above discovery, we propose a Missing Information Guided Retrieve-Extraction-Solving paradigm (MIGRES), where we leverage the identification of missing information to generate a targeted query that steers the subsequent knowledge retrieval. Besides, we design a sentence-level re-ranking filtering approach to filter the irrelevant content out from document, along with the information extraction capability of LLMs to extract useful information from cleaned-up documents, which in turn to bolster the overall efficacy of RAG. Extensive experiments conducted on multiple public datasets reveal the superiority of the proposed MIGRES method, and analytical experiments demonstrate the effectiveness of our proposed modules.
• 摘要：检索增强生成（RAG）通过为法学硕士提供更新的相关知识，在减轻过时的知识或幻觉方面展现出巨大的价值。然而，RAG在理解复杂的多跳查询和检索相关文档方面仍然存在一些困难，这需要LLM逐步进行推理和检索。受人类逐渐搜索所需信息的推理过程的启发，法学硕士很自然地会问法学硕士是否能够注意到每个推理步骤中缺失的信息。在这项工作中，我们首先通过实验验证了法学硕士提取信息以及了解缺失信息的能力。基于上述发现，我们提出了一种缺失信息引导检索-提取-解决范式（MIGRES），其中我们利用缺失信息的识别来生成有针对性的查询，以指导后续的知识检索。此外，我们设计了一种句子级重排序过滤方法来过滤掉文档中不相关的内容，并结合LLM的信息提取能力来从清理后的文档中提取有用的信息，从而增强RAG的整体效率。在多个公共数据集上进行的广泛实验揭示了所提出的 MIGRES 方法的优越性，分析实验证明了我们提出的模块的有效性。

Title: Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?

• Authors: Dawei Zhu, Pinzhen Chen, Miaoran Zhang, Barry Haddow, Xiaoyu Shen, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14122
• Pdf URL: https://arxiv.org/pdf/2404.14122
• Copy Paste: [[2404.14122]] Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?(https://arxiv.org/abs/2404.14122)
• Keywords: language model, llm
• Abstract: Traditionally, success in multilingual machine translation can be attributed to three key factors in training data: large volume, diverse translation directions, and high quality. In the current practice of fine-tuning large language models (LLMs) for translation, we revisit the importance of all these factors. We find that LLMs display strong translation capability after being fine-tuned on as few as 32 training instances, and that fine-tuning on a single translation direction effectively enables LLMs to translate in multiple directions. However, the choice of direction is critical: fine-tuning LLMs with English on the target side can lead to task misinterpretation, which hinders translations into non-English languages. A similar problem arises when noise is introduced into the target side of parallel data, especially when the target language is well-represented in the LLM's pre-training. In contrast, noise in an under-represented language has a less pronounced effect. Our findings suggest that attaining successful alignment hinges on teaching the model to maintain a "superficial" focus, thereby avoiding the learning of erroneous biases beyond translation.
• 摘要：传统上，多语言机器翻译的成功可归因于训练数据的三个关键因素：海量、翻译方向多样和高质量。在当前对翻译大型语言模型（LLM）进行微调的实践中，我们重新审视所有这些因素的重要性。我们发现LLM在仅仅32个训练实例上进行微调后就显示出强大的翻译能力，并且在单个翻译方向上的微调有效地使LLM能够在多个方向上进行翻译。然而，方向的选择至关重要：在目标端用英语对法学硕士进行微调可能会导致任务误解，从而阻碍翻译成非英语语言。当并行数据的目标端引入噪声时，就会出现类似的问题，特别是当目标语言在 LLM 的预训练中得到很好的体现时。相比之下，代表性不足的语言中的噪音影响则不太明显。我们的研究结果表明，实现成功的对齐取决于教导模型保持“肤浅”的焦点，从而避免学习翻译之外的错误偏见。

Title: SemEval-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-Generated Text Detection

• Authors: Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Chenxi Whitehouse, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14183
• Pdf URL: https://arxiv.org/pdf/2404.14183
• Copy Paste: [[2404.14183]] SemEval-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-Generated Text Detection(https://arxiv.org/abs/2404.14183)
• Keywords: llm
• Abstract: We present the results and the main findings of SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection. The task featured three subtasks. Subtask A is a binary classification task determining whether a text is written by a human or generated by a machine. This subtask has two tracks: a monolingual track focused solely on English texts and a multilingual track. Subtask B is to detect the exact source of a text, discerning whether it is written by a human or generated by a specific LLM. Subtask C aims to identify the changing point within a text, at which the authorship transitions from human to machine. The task attracted a large number of participants: subtask A monolingual (126), subtask A multilingual (59), subtask B (70), and subtask C (30). In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For all subtasks, the best systems used LLMs.
• 摘要：我们展示了 SemEval-2024 任务 8：多生成器、多域和多语言机器生成文本检测的结果和主要发现。该任务包含三个子任务。子任务 A 是一个二元分类任务，确定文本是由人编写还是由机器生成。该子任务有两个轨道：一个仅专注于英语文本的单语言轨道和一个多语言轨道。子任务 B 是检测文本的确切来源，辨别它是由人类编写还是由特定的法学硕士生成。子任务 C 旨在识别文本中的变化点，在该变化点上作者身份从人类转变为机器。该任务吸引了大量参与者：子任务 A 单语（126）、子任务 A 多语（59）、子任务 B（70）和子任务 C（30）。在本文中，我们提出了任务，分析了结果，并讨论了系统提交及其使用的方法。对于所有子任务，最好的系统都使用法学硕士。

Title: EnzChemRED, a rich enzyme chemistry relation extraction dataset

• Authors: Po-Ting Lai, Elisabeth Coudert, Lucila Aimo, Kristian Axelsen, Lionel Breuza, Edouard de Castro, Marc Feuermann, Anne Morgat, Lucille Pourcel, Ivo Pedruzzi, Sylvain Poux, Nicole Redaschi, Catherine Rivoire, Anastasia Sveshnikova, Chih-Hsuan Wei, Robert Leaman, Ling Luo, Zhiyong Lu, Alan Bridge
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14209
• Pdf URL: https://arxiv.org/pdf/2404.14209
• Copy Paste: [[2404.14209]] EnzChemRED, a rich enzyme chemistry relation extraction dataset(https://arxiv.org/abs/2404.14209)
• Keywords: language model
• Abstract: Expert curation is essential to capture knowledge of enzyme functions from the scientific literature in FAIR open knowledgebases but cannot keep pace with the rate of new discoveries and new publications. In this work we present EnzChemRED, for Enzyme Chemistry Relation Extraction Dataset, a new training and benchmarking dataset to support the development of Natural Language Processing (NLP) methods such as (large) language models that can assist enzyme curation. EnzChemRED consists of 1,210 expert curated PubMed abstracts in which enzymes and the chemical reactions they catalyze are annotated using identifiers from the UniProt Knowledgebase (UniProtKB) and the ontology of Chemical Entities of Biological Interest (ChEBI). We show that fine-tuning pre-trained language models with EnzChemRED can significantly boost their ability to identify mentions of proteins and chemicals in text (Named Entity Recognition, or NER) and to extract the chemical conversions in which they participate (Relation Extraction, or RE), with average F1 score of 86.30% for NER, 86.66% for RE for chemical conversion pairs, and 83.79% for RE for chemical conversion pairs and linked enzymes. We combine the best performing methods after fine-tuning using EnzChemRED to create an end-to-end pipeline for knowledge extraction from text and apply this to abstracts at PubMed scale to create a draft map of enzyme functions in literature to guide curation efforts in UniProtKB and the reaction knowledgebase Rhea. The EnzChemRED corpus is freely available at https://ftp.expasy.org/databases/rhea/nlp/.
• 摘要：专家管理对于从 FAIR 开放知识库的科学文献中获取酶功能知识至关重要，但无法跟上新发现和新出版物的速度。在这项工作中，我们提出了 EnzChemRED（酶化学关系提取数据集），这是一个新的训练和基准测试数据集，用于支持自然语言处理（NLP）方法的开发，例如可以协助酶管理的（大型）语言模型。 EnzChemRED 由 1,210 篇专家策划的 PubMed 摘要组成，其中酶及其催化的化学反应使用来自 UniProt 知识库 (UniProtKB) 和生物重要化学实体 (ChEBI) 本体的标识符进行注释。我们表明，使用 EnzChemRED 微调预训练的语言模型可以显着提高其识别文本中提及的蛋白质和化学物质（命名实体识别，或 NER）以及提取它们参与的化学转换（关系提取，或RE），NER 的平均 F1 得分为 86.30%，化学转化对的 RE 的平均 F1 得分为 86.66%，化学转化对和连接酶的 RE 的平均 F1 得分为 83.79%。我们使用 EnzChemRED 微调后结合最佳性能方法，创建一个用于从文本中提取知识的端到端管道，并将其应用于 PubMed 规模的摘要，以创建文献中酶功能的草图，以指导 UniProtKB 的管理工作和反应知识库 Rhea。 EnzChemRED 语料库可在 https://ftp.expasy.org/databases/rhea/nlp/ 上免费获取。

Title: Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction

• Authors: Zheye Deng, Chunkit Chan, Weiqi Wang, Yuxi Sun, Wei Fan, Tianshi Zheng, Yauwai Yim, Yangqiu Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14215
• Pdf URL: https://arxiv.org/pdf/2404.14215
• Copy Paste: [[2404.14215]] Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction(https://arxiv.org/abs/2404.14215)
• Keywords: language model, llm
• Abstract: The task of condensing large chunks of textual information into concise and structured tables has gained attention recently due to the emergence of Large Language Models (LLMs) and their potential benefit for downstream tasks, such as text summarization and text mining. Previous approaches often generate tables that directly replicate information from the text, limiting their applicability in broader contexts, as text-to-table generation in real-life scenarios necessitates information extraction, reasoning, and integration. However, there is a lack of both datasets and methodologies towards this task. In this paper, we introduce LiveSum, a new benchmark dataset created for generating summary tables of competitions based on real-time commentary texts. We evaluate the performances of state-of-the-art LLMs on this task in both fine-tuning and zero-shot settings, and additionally propose a novel pipeline called $T^3$(Text-Tuple-Table) to improve their performances. Extensive experimental results demonstrate that LLMs still struggle with this task even after fine-tuning, while our approach can offer substantial performance gains without explicit training. Further analyses demonstrate that our method exhibits strong generalization abilities, surpassing previous approaches on several other text-to-table datasets. Our code and data can be found at https://github.com/HKUST-KnowComp/LiveSum-TTT.
• 摘要：由于大型语言模型 (LLM) 的出现及其对文本摘要和文本挖掘等下游任务的潜在益处，将大量文本信息压缩为简洁且结构化的表格的任务最近引起了人们的关注。以前的方法通常会生成直接复制文本信息的表格，这限制了它们在更广泛环境中的适用性，因为在现实生活中，文本到表格的生成需要信息提取、推理和集成。然而，这项任务缺乏数据集和方法。在本文中，我们介绍了 LiveSum，这是一个新的基准数据集，用于根据实时评论文本生成比赛摘要表。我们在微调和零样本设置中评估了最先进的 LLM 在该任务上的表现，并另外提出了一种名为 $T^3$(Text-Tuple-Table) 的新管道来提高它们的性能。大量的实验结果表明，即使经过微调，LLM 仍然难以完成这项任务，而我们的方法可以在没有明确训练的情况下提供显着的性能提升。进一步的分析表明，我们的方法表现出强大的泛化能力，在其他几个文本到表格数据集上超越了以前的方法。我们的代码和数据可以在 https://github.com/HKUST-KnowComp/LiveSum-TTT 找到。

Title: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

• Authors: Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, et al. (31 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14219
• Pdf URL: https://arxiv.org/pdf/2404.14219
• Copy Paste: [[2404.14219]] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone(https://arxiv.org/abs/2404.14219)
• Keywords: language model, gpt, chat
• Abstract: We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).
• 摘要：我们引入了 phi-3-mini，这是一个在 3.3 万亿个代币上训练的 38 亿参数语言模型，根据学术基准和内部测试衡量，其整体性能可与 Mixtral 8x7B 和 GPT-3.5 等模型相媲美（例如 phi -3-mini 在 MMLU 上达到了 69%，在 MT-bench 上达到了 8.38），尽管它足够小，可以部署在手机上。创新完全在于我们的训练数据集，这是用于 phi-2 的数据集的放大版本，由经过严格过滤的网络数据和合成数据组成。该模型还进一步调整了稳健性、安全性和聊天格式。我们还提供了一些针对 4.8T 代币训练的 7B 和 14B 模型的初始参数缩放结果，称为 phi-3-small 和 phi-3-medium，两者都比 phi-3-mini 能力更强（例如，分别为 75%） MMLU 上为 78%，MT 基准上为 8.7 和 8.9）。

Title: What do Transformers Know about Government?

• Authors: Jue Hou, Anisia Katinskaia, Lari Kotilainen, Sathianpong Trangcasanchai, Anh-Duc Vu, Roman Yangarber
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14270
• Pdf URL: https://arxiv.org/pdf/2404.14270
• Copy Paste: [[2404.14270]] What do Transformers Know about Government?(https://arxiv.org/abs/2404.14270)
• Keywords: language model
• Abstract: This paper investigates what insights about linguistic features and what knowledge about the structure of natural language can be obtained from the encodings in transformer language models.In particular, we explore how BERT encodes the government relation between constituents in a sentence. We use several probing classifiers, and data from two morphologically rich languages. Our experiments show that information about government is encoded across all transformer layers, but predominantly in the early layers of the model. We find that, for both languages, a small number of attention heads encode enough information about the government relations to enable us to train a classifier capable of discovering new, previously unknown types of government, never seen in the training data. Currently, data is lacking for the research community working on grammatical constructions, and government in particular. We release the Government Bank -- a dataset defining the government relations for thousands of lemmas in the languages in our experiments.
• 摘要：本文研究了从 Transformer 语言模型的编码中可以获得哪些关于语言特征的见解以及哪些关于自然语言结构的知识。特别是，我们探讨了 BERT 如何编码句子中成分之间的政府关系。我们使用几个探测分类器以及来自两种形态丰富的语言的数据。我们的实验表明，有关政府的信息在所有变压器层中进行编码，但主要在模型的早期层中。我们发现，对于这两种语言，少量的注意力头编码了足够的有关政府关系的信息，使我们能够训练一个分类器，该分类器能够发现新的、以前未知的政府类型，而这些类型在训练数据中从未见过。目前，从事语法结构研究的研究界，尤其是政府，缺乏数据。我们发布了政府银行——一个数据集，定义了我们实验中语言中数千个引理的政府关系。

Title: A Survey on Efficient Inference for Large Language Models

• Authors: Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, Yu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14294
• Pdf URL: https://arxiv.org/pdf/2404.14294
• Copy Paste: [[2404.14294]] A Survey on Efficient Inference for Large Language Models(https://arxiv.org/abs/2404.14294)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. Efforts within the field have been directed towards developing techniques aimed at enhancing the efficiency of LLM inference. This paper presents a comprehensive survey of the existing literature on efficient LLM inference. We start by analyzing the primary causes of the inefficient LLM inference, i.e., the large model size, the quadratic-complexity attention operation, and the auto-regressive decoding approach. Then, we introduce a comprehensive taxonomy that organizes the current literature into data-level, model-level, and system-level optimization. Moreover, the paper includes comparative experiments on representative methods within critical sub-fields to provide quantitative insights. Last but not least, we provide some knowledge summary and discuss future research directions.
• 摘要：大型语言模型（LLM）因其在各种任务中的出色表现而受到广泛关注。然而，LLM 推理的大量计算和内存要求对资源受限场景中的部署提出了挑战。该领域内的努力致力于开发旨在提高法学硕士推理效率的技术。本文对有关高效 LLM 推理的现有文献进行了全面调查。我们首先分析 LLM 推理效率低下的主要原因，即模型尺寸大、二次复杂度注意操作和自回归解码方法。然后，我们引入了一个全面的分类法，将当前文献组织为数据级、模型级和系统级优化。此外，本文还对关键子领域内的代表性方法进行了比较实验，以提供定量见解。最后但并非最不重要的一点是，我们提供了一些知识总结并讨论了未来的研究方向。

Title: Marking: Visual Grading with Highlighting Errors and Annotating Missing Bits

• Authors: Shashank Sonkar, Naiming Liu, Debshila B. Mallick, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14301
• Pdf URL: https://arxiv.org/pdf/2404.14301
• Copy Paste: [[2404.14301]] Marking: Visual Grading with Highlighting Errors and Annotating Missing Bits(https://arxiv.org/abs/2404.14301)
• Keywords: language model
• Abstract: In this paper, we introduce "Marking", a novel grading task that enhances automated grading systems by performing an in-depth analysis of student responses and providing students with visual highlights. Unlike traditional systems that provide binary scores, "marking" identifies and categorizes segments of the student response as correct, incorrect, or irrelevant and detects omissions from gold answers. We introduce a new dataset meticulously curated by Subject Matter Experts specifically for this task. We frame "Marking" as an extension of the Natural Language Inference (NLI) task, which is extensively explored in the field of Natural Language Processing. The gold answer and the student response play the roles of premise and hypothesis in NLI, respectively. We subsequently train language models to identify entailment, contradiction, and neutrality from student response, akin to NLI, and with the added dimension of identifying omissions from gold answers. Our experimental setup involves the use of transformer models, specifically BERT and RoBERTa, and an intelligent training step using the e-SNLI dataset. We present extensive baseline results highlighting the complexity of the "Marking" task, which sets a clear trajectory for the upcoming study. Our work not only opens up new avenues for research in AI-powered educational assessment tools, but also provides a valuable benchmark for the AI in education community to engage with and improve upon in the future. The code and dataset can be found at https://github.com/luffycodes/marking.
• 摘要：在本文中，我们介绍了“标记”，这是一种新颖的评分任务，它通过对学生的反应进行深入分析并为学生提供视觉亮点来增强自动评分系统。与提供二进制分数的传统系统不同，“标记”将学生回答的各个部分识别为正确、不正确或不相关，并进行分类，并检测黄金答案中的遗漏。我们引入了由主题专家专门为此任务精心策划的新数据集。我们将“标记”定义为自然语言推理（NLI）任务的扩展，该任务在自然语言处理领域得到了广泛的探索。黄金答案和学生反应分别在 NLI 中扮演前提和假设的角色。随后，我们训练语言模型来识别学生反应中的蕴涵、矛盾和中立性，类似于 NLI，并增加了识别黄金答案中遗漏的维度。我们的实验设置涉及使用 Transformer 模型，特别是 BERT 和 RoBERTa，以及使用 e-SNLI 数据集的智能训练步骤。我们提供了广泛的基线结果，强调了“标记”任务的复杂性，这为即将进行的研究设定了明确的轨迹。我们的工作不仅为人工智能驱动的教育评估工具的研究开辟了新的途径，而且为教育界的人工智能在未来的参与和改进提供了宝贵的基准。代码和数据集可以在 https://github.com/luffycodes/marking 找到。

Title: Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels

• Authors: Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, Noah D. Goodman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14313
• Pdf URL: https://arxiv.org/pdf/2404.14313
• Copy Paste: [[2404.14313]] Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels(https://arxiv.org/abs/2404.14313)
• Keywords: language model, prompt
• Abstract: When prompting a language model (LM), users frequently expect the model to adhere to a set of behavioral principles across diverse tasks, such as producing insightful content while avoiding harmful or biased language. Instilling such principles into a model can be resource-intensive and technically challenging, generally requiring human preference labels or examples. We introduce SAMI, a method for teaching a pretrained LM to follow behavioral principles that does not require any preference labels or demonstrations. SAMI is an iterative algorithm that finetunes a pretrained LM to increase the conditional mutual information between constitutions and self-generated responses given queries from a datasest. On single-turn dialogue and summarization, a SAMI-trained mistral-7b outperforms the initial pretrained model, with win rates between 66% and 77%. Strikingly, it also surpasses an instruction-finetuned baseline (mistral-7b-instruct) with win rates between 55% and 57% on single-turn dialogue. SAMI requires a "principle writer" model; to avoid dependence on stronger models, we further evaluate aligning a strong pretrained model (mixtral-8x7b) using constitutions written by a weak instruction-finetuned model (mistral-7b-instruct). The SAMI-trained mixtral-8x7b outperforms both the initial model and the instruction-finetuned model, achieving a 65% win rate on summarization. Our results indicate that a pretrained LM can learn to follow constitutions without using preference labels, demonstrations, or human oversight.
• 摘要：在提示语言模型 (LM) 时，用户经常期望模型在不同的任务中遵守一组行为原则，例如生成有洞察力的内容，同时避免有害或有偏见的语言。将这些原则灌输到模型中可能会占用大量资源，并且在技术上具有挑战性，通常需要人类偏好标签或示例。我们引入了 SAMI，这是一种教导预先训练的 LM 遵循行为原则的方法，不需要任何偏好标签或演示。 SAMI 是一种迭代算法，可对预训练的 LM 进行微调，以增加构成与给定数据集查询的自生成响应之间的条件互信息。在单轮对话和摘要方面，经过 SAMI 训练的 mistra-7b 优于最初的预训练模型，胜率在 66% 到 77% 之间。引人注目的是，它还超越了指令微调基线 (mistral-7b-instruct)，单轮对话的胜率在 55% 到 57% 之间。 SAMI 需要一个“原则编写者”模型；为了避免依赖于更强的模型，我们进一步评估使用弱指令微调模型（mistral-7b-instruct）编写的构造来对齐强预训练模型（mixtral-8x7b）。经过 SAMI 训练的 mixtral-8x7b 的性能优于初始模型和指令微调模型，在摘要方面实现了 65% 的胜率。我们的结果表明，经过预训练的 LM 可以学习遵守宪法，而无需使用偏好标签、演示或人工监督。

Title: Automated Long Answer Grading with RiceChem Dataset

• Authors: Shashank Sonkar, Kangqi Ni, Lesa Tran Lu, Kristi Kincaid, John S. Hutchinson, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14316
• Pdf URL: https://arxiv.org/pdf/2404.14316
• Copy Paste: [[2404.14316]] Automated Long Answer Grading with RiceChem Dataset(https://arxiv.org/abs/2404.14316)
• Keywords: language model, gpt, llm
• Abstract: We introduce a new area of study in the field of educational Natural Language Processing: Automated Long Answer Grading (ALAG). Distinguishing itself from Automated Short Answer Grading (ASAG) and Automated Essay Grading (AEG), ALAG presents unique challenges due to the complexity and multifaceted nature of fact-based long answers. To study ALAG, we introduce RiceChem, a dataset derived from a college chemistry course, featuring real student responses to long-answer questions with an average word count notably higher than typical ASAG datasets. We propose a novel approach to ALAG by formulating it as a rubric entailment problem, employing natural language inference models to verify whether each criterion, represented by a rubric item, is addressed in the student's response. This formulation enables the effective use of MNLI for transfer learning, significantly improving the performance of models on the RiceChem dataset. We demonstrate the importance of rubric-based formulation in ALAG, showcasing its superiority over traditional score-based approaches in capturing the nuances of student responses. We also investigate the performance of models in cold start scenarios, providing valuable insights into the practical deployment considerations in educational settings. Lastly, we benchmark state-of-the-art open-sourced Large Language Models (LLMs) on RiceChem and compare their results to GPT models, highlighting the increased complexity of ALAG compared to ASAG. Despite leveraging the benefits of a rubric-based approach and transfer learning from MNLI, the lower performance of LLMs on RiceChem underscores the significant difficulty posed by the ALAG task. With this work, we offer a fresh perspective on grading long, fact-based answers and introduce a new dataset to stimulate further research in this important area. Code: \url{https://github.com/luffycodes/Automated-Long-Answer-Grading}.
• 摘要：我们引入了教育自然语言处理领域的一个新研究领域：自动长答案评分（ALAG）。与自动短答案评分 (ASAG) 和自动作文评分 (AEG) 不同，ALAG 由于基于事实的长答案的复杂性和多方面性而提出了独特的挑战。为了研究 ALAG，我们引入了 RiceChem，这是一个源自大学化学课程的数据集，其中包含学生对长答案问题的真实回答，平均字数明显高于典型的 ASAG 数据集。我们提出了一种新颖的 ALAG 方法，将其表述为一个标题蕴涵问题，采用自然语言推理模型来验证学生的回答中是否解决了由标题项目表示的每个标准。该公式可以有效地使用 MNLI 进行迁移学习，显着提高 RiceChem 数据集上模型的性能。我们展示了 ALAG 中基于评分标准的制定的重要性，展示了其在捕捉学生反应的细微差别方面比传统的基于分数的方法的优越性。我们还研究了模型在冷启动场景中的性能，为教育环境中的实际部署注意事项提供了宝贵的见解。最后，我们在 RiceChem 上对最先进的开源大型语言模型 (LLM) 进行基准测试，并将其结果与 GPT 模型进行比较，突出显示 ALAG 与 ASAG 相比增加的复杂性。尽管利用了基于规则的方法和 MNLI 迁移学习的优势，但 RiceChem 上法学硕士的较低表现凸显了 ALAG 任务带来的巨大困难。通过这项工作，我们为对基于事实的长答案进行评分提供了全新的视角，并引入了新的数据集来刺激这一重要领域的进一步研究。代码：\url{https://github.com/luffycodes/Automated-Long-Answer-Grading}。

Title: Calc-CMU at SemEval-2024 Task 7: Pre-Calc -- Learning to Use the Calculator Improves Numeracy in Language Models

• Authors: Vishruth Veerendranath, Vishwa Shah, Kshitish Ghate
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14355
• Pdf URL: https://arxiv.org/pdf/2404.14355
• Copy Paste: [[2404.14355]] Calc-CMU at SemEval-2024 Task 7: Pre-Calc -- Learning to Use the Calculator Improves Numeracy in Language Models(https://arxiv.org/abs/2404.14355)
• Keywords: language model
• Abstract: Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to be helpful to improve mathematical reasoning in large pretrained decoder-only language models, this remains unexplored for smaller language models with encoders. In this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning to use the calculator for both encoder-only and encoder-decoder architectures, formulated as a discriminative and generative task respectively. We pre-train BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves performance on downstream tasks that require numerical understanding. Our code and data are available at https://github.com/calc-cmu/pre-calc.
• 摘要：语言的定量和数值理解是教育和金融等许多领域的一项重要任务，但对于语言模型来说仍然是一项具有挑战性的任务。虽然工具和计算器的使用已被证明有助于改善大型预训练仅解码器语言模型的数学推理，但对于带有编码器的小型语言模型来说，这仍未得到探索。在本文中，我们提出了 Pre-Calc，一个简单的预微调目标，学习如何将计算器用于仅编码器和编码器-解码器架构，分别表示为判别任务和生成任务。我们在 MAWPS、SVAMP 和 AsDiv-A 数据集上预训练 BERT 和 RoBERTa 以用于判别计算器，并预训练 Flan-T5 以用于生成计算器，这提高了需要数值理解的下游任务的性能。我们的代码和数据可在 https://github.com/calc-cmu/pre-calc 获取。

Title: Better Synthetic Data by Retrieving and Transforming Existing Datasets

• Authors: Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14361
• Pdf URL: https://arxiv.org/pdf/2404.14361
• Copy Paste: [[2404.14361]] Better Synthetic Data by Retrieving and Transforming Existing Datasets(https://arxiv.org/abs/2404.14361)
• Keywords: language model, prompt
• Abstract: Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, \textit{DataTune}, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49\% and improves over existing methods that use synthetic or retrieved training data by 34\%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community: https://github.com/neulab/prompt2model.
• 摘要：尽管大型语言模型最近取得了进展，但构建可靠且可部署的 NLP 模型通常需要丰富、高质量的训练数据。然而，许多用例无法获得特定于任务的数据，并且手动管理特定于任务的数据是劳动密集型的。最近的工作研究了使用大型语言模型的提示驱动的合成数据生成，但这些生成的数据集往往缺乏复杂性和多样性。为了解决这些限制，我们引入了一种方法 \textit{DataTune}，以更好地利用现有的、公开可用的数据集来改进自动数据集生成。 DataTune 执行数据集转换，从而将公开可用的数据集重新调整为直接符合目标任务特定要求的格式。在 BIG-Bench 基准测试中的一组不同的基于语言的任务中，我们发现通过 DataTune 微调语言模型比几次提示基线提高了 49%，比使用合成或检索训练数据的现有方法提高了 34% \％。我们发现数据集转换显着增加了许多任务中生成数据的多样性和难度。我们将 DataTune 集成到开源存储库中，以使社区可以访问此方法：https://github.com/neulab/prompt2model。

Title: Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph

• Authors: Xiaochen Kev Gao, Feng Yao, Kewen Zhao, Beilei He, Animesh Kumar, Vish Krishnan, Jingbo Shang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14372
• Pdf URL: https://arxiv.org/pdf/2404.14372
• Copy Paste: [[2404.14372]] Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph(https://arxiv.org/abs/2404.14372)
• Keywords: language model, llm, prompt
• Abstract: Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval pre-diction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. Specifically, we first extend the embedding-based state-of-the-art (SOTA) by scaling up its backbone model with various sizes of open-source LLMs, then explore prompt-based methods to harness proprietary LLMs' potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. Hence, we propose a novel Fine-grained cLAim depeNdency (FLAN) Graph through meticulous patent data analyses, capturing the inherent dependencies across segments of the patent text. As it is model-agnostic, we apply cost-effective graph models to our FLAN Graph to obtain representations for approval prediction. Extensive experiments and detailed analyses prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly. We hope that our observations and analyses in this paper can bring more attention to this challenging task and prompt further research into the limitations of LLMs. Our source code and dataset can be obtained from this http URL
• 摘要：由于大型语言模型 (LLM) 的成功，模型缩放正在成为许多语言任务的默认选择。然而，在简单的定制方法表现出色的特定场景中，它可能会出现不足。在本文中，我们深入研究了专利批准预测任务，并揭示了利用专利数据中的内在依赖性，简单的特定领域图方法优于扩大模型。具体来说，我们首先通过使用各种规模的开源 LLM 扩展其骨干模型来扩展基于嵌入的最先进技术 (SOTA)，然后探索基于提示的方法来利用专有 LLM 的潜力，但发现最佳结果接近随机猜测，凸显了模型扩展的无效性。因此，我们通过细致的专利数据分析提出了一种新颖的细粒度权利要求依赖性（FLAN）图，捕获专利文本各段之间的固有依赖性。由于它与模型无关，因此我们将具有成本效益的图模型应用于 FLAN 图，以获得批准预测的表示。大量的实验和详细的分析证明，通过各种图形模型合并 FLAN Graph 的性能始终显着优于所有 LLM 基线。我们希望本文的观察和分析能够引起人们对这一具有挑战性的任务的更多关注，并促进对法学硕士局限性的进一步研究。我们的源代码和数据集可以从此http URL获取

Title: A Survey on Self-Evolution of Large Language Models

• Authors: Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, Jingren Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14387
• Pdf URL: https://arxiv.org/pdf/2404.14387
• Copy Paste: [[2404.14387]] A Survey on Self-Evolution of Large Language Models(https://arxiv.org/abs/2404.14387)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have significantly advanced in various fields and intelligent agent applications. However, current LLMs that learn from human or external model supervision are costly and may face performance ceilings as task complexity and diversity increase. To address this issue, self-evolution approaches that enable LLM to autonomously acquire, refine, and learn from experiences generated by the model itself are rapidly growing. This new training paradigm inspired by the human experiential learning process offers the potential to scale LLMs towards superintelligence. In this work, we present a comprehensive survey of self-evolution approaches in LLMs. We first propose a conceptual framework for self-evolution and outline the evolving process as iterative cycles composed of four phases: experience acquisition, experience refinement, updating, and evaluation. Second, we categorize the evolution objectives of LLMs and LLM-based agents; then, we summarize the literature and provide taxonomy and insights for each module. Lastly, we pinpoint existing challenges and propose future directions to improve self-evolution frameworks, equipping researchers with critical insights to fast-track the development of self-evolving LLMs.
• 摘要：大型语言模型（LLM）在各个领域和智能代理应用中取得了显着的进步。然而，目前从人类或外部模型监督中学习的法学硕士成本高昂，并且随着任务复杂性和多样性的增加可能面临性能天花板。为了解决这个问题，使法学硕士能够自主获取、完善模型本身生成的经验并从中学习的自我进化方法正在迅速发展。这种受人类体验式学习过程启发的新培训范式提供了将法学硕士扩展到超级智能的潜力。在这项工作中，我们对法学硕士的自我进化方法进行了全面的调查。我们首先提出了自我进化的概念框架，并将进化过程概述为由四个阶段组成的迭代循环：经验获取、经验细化、更新和评估。其次，我们对LLM和基于LLM的代理的进化目标进行分类；然后，我们总结文献并为每个模块提供分类和见解。最后，我们指出了现有的挑战，并提出了改进自我进化框架的未来方向，为研究人员提供了快速发展自我进化法学硕士的重要见解。

Title: PARAMANU-GANITA: Language Model with Mathematical Capabilities

• Authors: Mitodru Niyogi, Arnab Bhattacharya
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14395
• Pdf URL: https://arxiv.org/pdf/2404.14395
• Copy Paste: [[2404.14395]] PARAMANU-GANITA: Language Model with Mathematical Capabilities(https://arxiv.org/abs/2404.14395)
• Keywords: language model, llm
• Abstract: In this paper, we present Paramanu-Ganita, a 208 million parameter novel Auto Regressive (AR) decoder based language model on mathematics. The model is pretrained from scratch at context size of 4096 on our curated mixed mathematical corpus. We evaluate our model on both perplexity metric and GSM8k mathematical benchmark. Paramanu-Ganita despite being 35 times smaller than 7B LLMs, outperformed generalist LLMs such as LLaMa-1 7B by 28.4% points, LLaMa-2 7B by 27.6% points, Falcon 7B by 32.6% points, PaLM 8B by 35.3% points, and math specialised LLMs such as Minerva 8B by 23.2% points, and LLEMMA-7B by 3.0% points in GSM8k test accuracy metric respectively. Paramanu-Ganita also outperformed giant LLMs like PaLM 62B by 6.4% points, Falcon 40B by 19.8% points, LLaMa-1 33B by 3.8% points and Vicuna 13B by 11.8% points respectively. The large significant margin improvement in performance of our math model over the existing LLMs signifies that reasoning capabilities of language model are just not restricted to LLMs with humongous number of parameters. Paramanu-Ganita took 146 hours of A100 training whereas math specialised LLM, LLEMMA 7B, was trained for 23,000 A100 hours of training equivalent. Thus, our approach of pretraining powerful domain specialised language models from scratch for domain adaptation is much more cost-effective than performing continual training of LLMs for domain adaptation. Hence, we conclude that for strong mathematical reasoning abilities of language model, we do not need giant LLMs and immense computing power to our end. In the end, we want to point out that we have only trained Paramanu-Ganita only on a part of our entire mathematical corpus and yet to explore the full potential of our model.
• 摘要：在本文中，我们提出了 Paramanu-Ganita，一种基于数学语言模型的 2.08 亿参数新颖的自回归 (AR) 解码器。该模型是在我们精心策划的混合数学语料库上以 4096 的上下文大小从头开始预训练的。我们根据困惑度指标和 GSM8k 数学基准评估我们的模型。 Paramanu-Ganita 尽管比 7B LLM 小 35 倍，但其表现却比 LLaMa-1 7B 等多面手 LLM 好 28.4%、LLaMa-2 7B 27.6%、Falcon 7B 32.6%、PaLM 8B 35.3% 和数学专业法学硕士（例如 Minerva 8B）在 GSM8k 测试准确度指标中分别提高了 23.2% 分和 LLEMMA-7B 3.0% 分。 Paramanu-Ganita 的表现也比 PaLM 62B 等大型法学硕士高出 6.4%，Falcon 40B 高出 19.8%，LLaMa-1 33B 高出 3.8%，Vicuna 13B 高出 11.8%。与现有的法学硕士相比，我们的数学模型的性能有了显着的提高，这意味着语言模型的推理能力不仅限于参数数量巨大的法学硕士。 Paramanu-Ganita 接受了 146 小时的 A100 培训，而数学专业法学硕士 (LLEMMA 7B) 接受了 23,000 小时的 A100 同等培训。因此，我们从头开始预训练强大的领域专用语言模型以进行领域适应的方法比对法学硕士进行持续训练以进行领域适应更具成本效益。因此，我们得出的结论是，对于语言模型强大的数学推理能力，我们不需要巨大的法学硕士和巨大的计算能力。最后，我们想指出，我们仅在整个数学语料库的一部分上训练了 Paramanu-Ganita，尚未探索我们模型的全部潜力。

Title: RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?

• Authors: Adrian de Wynter, Ishaan Watts, Nektar Ege Altıntoprak, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Lena Baur, Samantha Claudet, Pavel Gajdusek, Can Gören, Qilong Gu, Anna Kaminska, Tomasz Kaminski, Ruby Kuo, Akiko Kyuba, Jongho Lee, Kartik Mathur, Petter Merok, Ivana Milovanović, Nani Paananen, Vesa-Matti Paananen, Anna Pavlenko, Bruno Pereira Vidal, Luciano Strika, Yueh Tsao, Davide Turcato, Oleksandr Vakhno, Judit Velcsov, Anna Vickers, Stéphanie Visser, Herdyan Widarmanto, Andrey Zaikin, Si-Qing Chen
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14397
• Pdf URL: https://arxiv.org/pdf/2404.14397
• Copy Paste: [[2404.14397]] RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?(https://arxiv.org/abs/2404.14397)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) and small language models (SLMs) are being adopted at remarkable speed, although their safety still remains a serious concern. With the advent of multilingual S/LLMs, the question now becomes a matter of scale: can we expand multilingual safety evaluations of these models with the same velocity at which they are deployed? To this end we introduce RTP-LX, a human-transcreated and human-annotated corpus of toxic prompts and outputs in 28 languages. RTP-LX follows participatory design practices, and a portion of the corpus is especially designed to detect culturally-specific toxic language. We evaluate seven S/LLMs on their ability to detect toxic content in a culturally-sensitive, multilingual scenario. We find that, although they typically score acceptably in terms of accuracy, they have low agreement with human judges when judging holistically the toxicity of a prompt, and have difficulty discerning harm in context-dependent scenarios, particularly with subtle-yet-harmful content (e.g. microagressions, bias). We release of this dataset to contribute to further reduce harmful uses of these models and improve their safe deployment.
• 摘要：大语言模型（LLM）和小语言模型（SLM）正在以惊人的速度被采用，尽管它们的安全性仍然是一个严重的问题。随着多语言 S/LLM 的出现，现在的问题变成了规模问题：我们能否以与部署这些模型相同的速度扩展对这些模型的多语言安全评估？为此，我们引入了 RTP-LX，这是一个由人工创译和人工注释的有毒提示和输出语料库，具有 28 种语言版本。 RTP-LX 遵循参与式设计实践，语料库的一部分专门用于检测特定文化的有毒语言。我们评估了七名 S/LLM 在文化敏感、多语言场景中检测有毒成分的能力。我们发现，虽然它们通常在准确性方面得分可以接受，但在整体判断提示的毒性时，它们与人类法官的一致性较低，并且难以在依赖于上下文的场景中辨别危害，特别是对于微妙但有害的内容（例如微攻击、偏见）。我们发布此数据集是为了进一步减少这些模型的有害使用并改善其安全部署。

Title: SpaceByte: Towards Deleting Tokenization from Large Language Modeling

• Authors: Kevin Slagle
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14408
• Pdf URL: https://arxiv.org/pdf/2404.14408
• Copy Paste: [[2404.14408]] SpaceByte: Towards Deleting Tokenization from Large Language Modeling(https://arxiv.org/abs/2404.14408)
• Keywords: language model
• Abstract: Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and roughly matches the performance of tokenized Transformer architectures.
• 摘要：标记化被广泛用于大型语言模型，因为它可以显著提高性能。然而，标记化有几个缺点，例如性能偏差、增加对抗性弱点、降低字符级建模性能以及增加建模复杂性。为了在不牺牲性能的情况下解决这些缺点，我们提出了 SpaceByte，这是一种新颖的字节级解码器架构，可缩小字节级和子字自回归语言建模之间的性能差距。SpaceByte 由字节级 Transformer 模型组成，但在层中间插入了额外的较大 Transformer 块。我们发现，仅在某些字节（例如通常表示单词边界的空格字符）之后应用这些较大的块可以显著提高性能。我们的实验表明，对于固定的训练和推理计算预算，SpaceByte 的表现优于其他字节级架构，并且大致与标记化 Transformer 架构的性能相匹配。