2024-04-01

Title: AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving

• Authors: Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19708
• Pdf URL: https://arxiv.org/pdf/2403.19708
• Copy Paste: [[2403.19708]] AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving(https://arxiv.org/abs/2403.19708)
• Keywords: language model, llm, prompt
• Abstract: Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines for executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes AttentionStore, a new attention mechanism that enables the reuse of KV caches (i.e., attention reuse) across multi-turn conversations, significantly reducing the repetitive computation overheads. AttentionStore maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache access overheads from slow mediums, AttentionStore employs layer-wise pre-loading and asynchronous saving schemes to overlap the KV cache access with the GPU computation. To ensure that the KV caches to be accessed are placed in the fastest hierarchy, AttentionStore employs scheduler-aware fetching and eviction schemes to consciously place the KV caches in different layers based on the hints from the inference job scheduler. To avoid the invalidation of the saved KV caches incurred by context window overflow, AttentionStore enables the saved KV caches to remain valid via decoupling the positional encoding and effectively truncating the KV caches. Extensive experimental results demonstrate that AttentionStore significantly decreases the time to the first token (TTFT) by up to 88%, improves the prompt prefilling throughput by 8.2$\times$ for multi-turn conversations, and reduces the end-to-end inference cost by up to 56%. For long sequence inference, AttentionStore reduces the TTFT by up to 95% and improves the prompt prefilling throughput by 22$\times$.
• 摘要：通过多轮对话与人类交互是大型语言模型（LLM）的基本特征。然而，现有用于执行多轮会话的LLM服务引擎​​由于需要重复计算历史令牌的键值（KV）缓存而效率低下，从而产生高昂的服务成本。为了解决这个问题，本文提出了AttentionStore，一种新的注意力机制，可以在多轮对话中重用KV缓存（即注意力重用），从而显着减少重复计算开销。 AttentionStore维护一个分层的KV缓存系统，利用经济高效的内存/存储介质来保存所有请求的KV缓存。为了减少慢速介质的 KV 缓存访问开销，AttentionStore 采用分层预加载和异步保存方案，将 KV 缓存访问与 GPU 计算重叠。为了确保要访问的 KV 缓存放置在最快的层次结构中，AttentionStore 采用调度程序感知的获取和驱逐方案，根据推理作业调度程序的提示有意识地将 KV 缓存放置在不同的层中。为了避免上下文窗口溢出导致已保存的 KV 缓存失效，AttentionStore 通过解耦位置编码并有效截断 KV 缓存，使已保存的 KV 缓存保持有效。大量实验结果表明，AttentionStore 显着缩短了第一个令牌 (TTFT) 的时间高达 88%，将多轮对话的提示预填充吞吐量提高了 8.2$\times$，并降低了端到端推理成本高达 56%。对于长序列推理，AttentionStore 将 TTFT 降低高达 95%，并将提示预填充吞吐量提高 22$\times$。

Title: STRUM-LLM: Attributed and Structured Contrastive Summarization

• Authors: Beliz Gunel, James B. Wendt, Jing Xie, Yichao Zhou, Nguyen Vo, Zachary Fisher, Sandeep Tata
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19710
• Pdf URL: https://arxiv.org/pdf/2403.19710
• Copy Paste: [[2403.19710]] STRUM-LLM: Attributed and Structured Contrastive Summarization(https://arxiv.org/abs/2403.19710)
• Keywords: llm
• Abstract: Users often struggle with decision-making between two options (A vs B), as it usually requires time-consuming research across multiple web pages. We propose STRUM-LLM that addresses this challenge by generating attributed, structured, and helpful contrastive summaries that highlight key differences between the two options. STRUM-LLM identifies helpful contrast: the specific attributes along which the two options differ significantly and which are most likely to influence the user's decision. Our technique is domain-agnostic, and does not require any human-labeled data or fixed attribute list as supervision. STRUM-LLM attributes all extractions back to the input sources along with textual evidence, and it does not have a limit on the length of input sources that it can process. STRUM-LLM Distilled has 100x more throughput than the models with comparable performance while being 10x smaller. In this paper, we provide extensive evaluations for our method and lay out future directions for our currently deployed system.
• 摘要：用户经常在两个选项（A 与 B）之间做出决策时遇到困难，因为这通常需要跨多个网页进行耗时的研究。我们提出 STRUM-LLM，通过生成归因的、结构化的和有用的对比摘要来应对这一挑战，突出显示两个选项之间的关键差异。 STRUM-LLM 识别出有用的对比：两个选项显着不同且最有可能影响用户决策的特定属性。我们的技术与领域无关，并且不需要任何人工标记数据或固定属性列表作为监督。 STRUM-LLM 将所有提取内容与文本证据一起归因于输入源，并且它对其可以处理的输入源的长度没有限制。 STRUM-LLM Distilled 的吞吐量比具有同等性能的模型高出 100 倍，同时尺寸缩小 10 倍。在本文中，我们对我们的方法进行了广泛的评估，并为我们当前部署的系统制定了未来的方向。

Title: Capability-aware Prompt Reformulation Learning for Text-to-Image Generation

• Authors: Jingtao Zhan, Qingyao Ai, Yiqun Liu, Jia Chen, Shaoping Ma
• Subjects: cs.CL, cs.AI, cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.19716
• Pdf URL: https://arxiv.org/pdf/2403.19716
• Copy Paste: [[2403.19716]] Capability-aware Prompt Reformulation Learning for Text-to-Image Generation(https://arxiv.org/abs/2403.19716)
• Keywords: prompt
• Abstract: Text-to-image generation systems have emerged as revolutionary tools in the realm of artistic creation, offering unprecedented ease in transforming textual prompts into visual art. However, the efficacy of these systems is intricately linked to the quality of user-provided prompts, which often poses a challenge to users unfamiliar with prompt crafting. This paper addresses this challenge by leveraging user reformulation data from interaction logs to develop an automatic prompt reformulation model. Our in-depth analysis of these logs reveals that user prompt reformulation is heavily dependent on the individual user's capability, resulting in significant variance in the quality of reformulation pairs. To effectively use this data for training, we introduce the Capability-aware Prompt Reformulation (CAPR) framework. CAPR innovatively integrates user capability into the reformulation process through two key components: the Conditional Reformulation Model (CRM) and Configurable Capability Features (CCF). CRM reformulates prompts according to a specified user capability, as represented by CCF. The CCF, in turn, offers the flexibility to tune and guide the CRM's behavior. This enables CAPR to effectively learn diverse reformulation strategies across various user capacities and to simulate high-capability user reformulation during inference. Extensive experiments on standard text-to-image generation benchmarks showcase CAPR's superior performance over existing baselines and its remarkable robustness on unseen systems. Furthermore, comprehensive analyses validate the effectiveness of different components. CAPR can facilitate user-friendly interaction with text-to-image systems and make advanced artistic creation more achievable for a broader range of users.
• 摘要：文本到图像生成系统已成为艺术创作领域的革命性工具，为将文本提示转化为视觉艺术提供了前所未有的便利。然而，这些系统的功效与用户提供的提示的质量密切相关，这通常会给不熟悉提示制作的用户带来挑战。本文通过利用交互日志中的用户重新表述数据来开发自动提示重新表述模型来解决这一挑战。我们对这些日志的深入分析表明，用户提示重新制定在很大程度上取决于单个用户的能力，从而导致重新制定对的质量存在显着差异。为了有效地使用这些数据进行训练，我们引入了能力感知提示重组（CAPR）框架。 CAPR 通过两个关键组件创新地将用户能力集成到重构过程中：条件重构模型 (CRM) 和可配置能力特征 (CCF)。 CRM 根据指定的用户能力重新制定提示，如 CCF 所示。反过来，CCF 提供了调整和指导 CRM 行为的灵活性。这使得 CAPR 能够有效地学习跨不同用户能力的不同重构策略，并在推理过程中模拟高能力用户重构。对标准文本到图像生成基准的大量实验展示了 CAPR 相对于现有基准的卓越性能及其在未见过的系统上的卓越鲁棒性。此外，综合分析验证了不同组件的有效性。 CAPR 可以促进与文本到图像系统的用户友好交互，并使更广泛的用户更容易实现先进的艺术创作。

Title: HGT: Leveraging Heterogeneous Graph-enhanced Large Language Models for Few-shot Complex Table Understanding

• Authors: Rihui Jin, Yu Li, Guilin Qi, Nan Hu, Yuan-Fang Li, Jiaoyan Chen, Jianan Wang, Yongrui Chen, Dehai Min
• Subjects: cs.CL, cs.AI, cs.DB, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.19723
• Pdf URL: https://arxiv.org/pdf/2403.19723
• Copy Paste: [[2403.19723]] HGT: Leveraging Heterogeneous Graph-enhanced Large Language Models for Few-shot Complex Table Understanding(https://arxiv.org/abs/2403.19723)
• Keywords: language model, llm, prompt
• Abstract: Table understanding (TU) has achieved promising advancements, but it faces the challenges of the scarcity of manually labeled tables and the presence of complex table structures.To address these challenges, we propose HGT, a framework with a heterogeneous graph (HG)-enhanced large language model (LLM) to tackle few-shot TU tasks.It leverages the LLM by aligning the table semantics with the LLM's parametric knowledge through soft prompts and instruction turning and deals with complex tables by a multi-task pre-training scheme involving three novel multi-granularity self-supervised HG pre-training objectives.We empirically demonstrate the effectiveness of HGT, showing that it outperforms the SOTA for few-shot complex TU on several benchmarks.
• 摘要：表理解（TU）已经取得了有希望的进步，但它面临着手动标记表的稀缺和复杂表结构的存在的挑战。为了解决这些挑战，我们提出了 HGT，一种具有异构图（HG）增强的框架它利用 LLM，通过软提示和指令转换将表语义与 LLM 的参数化知识对齐，并通过涉及三个的多任务预训练方案处理复杂的表新颖的多粒度自监督 HG 预训练目标。我们凭经验证明了 HGT 的有效性，表明它在几个基准上优于少样本复杂 TU 的 SOTA。

Title: MUGC: Machine Generated versus User Generated Content Detection

• Authors: Yaqi Xie, Anjali Rawal, Yujing Cen, Dixuan Zhao, Sunil K Narang, Shanu Sushmita
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19725
• Pdf URL: https://arxiv.org/pdf/2403.19725
• Copy Paste: [[2403.19725]] MUGC: Machine Generated versus User Generated Content Detection(https://arxiv.org/abs/2403.19725)
• Keywords: language model, llm
• Abstract: As advanced modern systems like deep neural networks (DNNs) and generative AI continue to enhance their capabilities in producing convincing and realistic content, the need to distinguish between user-generated and machine generated content is becoming increasingly evident. In this research, we undertake a comparative evaluation of eight traditional machine-learning algorithms to distinguish between machine-generated and human-generated data across three diverse datasets: Poems, Abstracts, and Essays. Our results indicate that traditional methods demonstrate a high level of accuracy in identifying machine-generated data, reflecting the documented effectiveness of popular pre-trained models like RoBERT. We note that machine-generated texts tend to be shorter and exhibit less word variety compared to human-generated content. While specific domain-related keywords commonly utilized by humans, albeit disregarded by current LLMs (Large Language Models), may contribute to this high detection accuracy, we show that deeper word representations like word2vec can capture subtle semantic variances. Furthermore, readability, bias, moral, and affect comparisons reveal a discernible contrast between machine-generated and human generated content. There are variations in expression styles and potentially underlying biases in the data sources (human and machine-generated). This study provides valuable insights into the advancing capacities and challenges associated with machine-generated content across various domains.
• 摘要：随着深度神经网络 (DNN) 和生成人工智能等先进的现代系统不断增强其生成令人信服且真实的内容的能力，区分用户生成的内容和机器生成的内容的需求变得越来越明显。在这项研究中，我们对八种传统机器学习算法进行了比较评估，以区分三个不同数据集（诗歌、摘要和散文）中机器生成的数据和人类生成的数据。我们的结果表明，传统方法在识别机器生成的数据方面表现出很高的准确性，反映了 RoBERT 等流行的预训练模型的有效性。我们注意到，与人类生成的内容相比，机器生成的文本往往更短，单词多样性也更少。虽然人类常用的特定领域相关关键字（尽管当前的 LLM（大型语言模型）忽略了这些关键字）可能有助于实现如此高的检测精度，但我们表明，像 word2vec 这样的更深层次的单词表示可以捕获微妙的语义差异。此外，可读性、偏见、道德和情感比较揭示了机器生成的内容和人类生成的内容之间明显的对比。数据源（人类和机器生成）的表达风格存在差异，并且存在潜在的潜在偏差。这项研究为各个领域的机器生成内容的进步能力和挑战提供了宝贵的见解。

Title: A Benchmark Evaluation of Clinical Named Entity Recognition in French

• Authors: Nesrine Bannour (STL), Christophe Servan (STL), Aurélie Névéol (STL), Xavier Tannier (LIMICS)
• Subjects: cs.CL, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2403.19726
• Pdf URL: https://arxiv.org/pdf/2403.19726
• Copy Paste: [[2403.19726]] A Benchmark Evaluation of Clinical Named Entity Recognition in French(https://arxiv.org/abs/2403.19726)
• Keywords: language model, llm
• Abstract: Background: Transformer-based language models have shown strong performance on many Natural LanguageProcessing (NLP) tasks. Masked Language Models (MLMs) attract sustained interest because they can be adaptedto different languages and sub-domains through training or fine-tuning on specific corpora while remaining lighterthan modern Large Language Models (LLMs). Recently, several MLMs have been released for the biomedicaldomain in French, and experiments suggest that they outperform standard French counterparts. However, nosystematic evaluation comparing all models on the same corpora is available. Objective: This paper presentsan evaluation of masked language models for biomedical French on the task of clinical named entity recognition.Material and methods: We evaluate biomedical models CamemBERT-bio and DrBERT and compare them tostandard French models CamemBERT, FlauBERT and FrALBERT as well as multilingual mBERT using three publicallyavailable corpora for clinical named entity recognition in French. The evaluation set-up relies on gold-standardcorpora as released by the corpus developers. Results: Results suggest that CamemBERT-bio outperformsDrBERT consistently while FlauBERT offers competitive performance and FrAlBERT achieves the lowest carbonfootprint. Conclusion: This is the first benchmark evaluation of biomedical masked language models for Frenchclinical entity recognition that compares model performance consistently on nested entity recognition using metricscovering performance and environmental impact.
• 摘要：背景：基于 Transformer 的语言模型在许多自然语言处理 (NLP) 任务中表现出了强大的性能。掩码语言模型 (MLM) 吸引了持续的兴趣，因为它们可以通过对特定语料库的训练或微调来适应不同的语言和子领域，同时保持比现代大型语言模型 (LLM) 更轻的性能。最近，针对生物医学领域的多个 MLM 发布了法语版本，实验表明它们的性能优于标准的法语版本。然而，还没有比较同一语料库上所有模型的系统评估。目的：本文提出了对临床命名实体识别任务中的生物医学法语屏蔽语言模型的评估。材料和方法：我们评估生物医学模型 CamemBERT-bio 和 DrBERT，并将它们与标准法语模型 CamemBERT、FlauBERT 和 FrALBERT 以及多语言模型进行比较mBERT 使用三个公开可用的语料库进行法语临床命名实体识别。评估设置依赖于语料库开发人员发布的黄金标准语料库。结果：结果表明 CamemBERT-bio 的性能始终优于 DrBERT，而 FlauBERT 提供有竞争力的性能，FrAlBERT 实现最低的碳足迹。结论：这是对用于法国临床实体识别的生物医学掩码语言模型的第一个基准评估，该评估使用涵盖性能和环境影响的指标来一致地比较嵌套实体识别上的模型性能。

Title: GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

• Authors: Mohsen Gholami, Mohammad Akbari, Cindy Hu, Vaden Masrani, Z. Jane Wang, Yong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19754
• Pdf URL: https://arxiv.org/pdf/2403.19754
• Copy Paste: [[2403.19754]] GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation(https://arxiv.org/abs/2403.19754)
• Keywords: language model, llm
• Abstract: Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks. The code is available.
• 摘要：法学硕士的知识提炼对于语言模型的有效部署至关重要。之前的工作已经提出使用法学硕士来生成数据来准备蒸馏模型。我们认为，使用法学硕士生成数据很容易主要从原始内容分发的中心进行采样。这种限制阻碍了蒸馏模型学习真实的底层数据分布并忘记分布的尾部（概率较低的样本）。为此，我们提出了 GOLD，一种与任务无关的数据生成和知识蒸馏框架，它为法学硕士采用了迭代的非分布引导反馈机制。因此，生成的数据提高了精炼模型的通用性。还引入了基于能量的 OOD 评估方法来处理生成的噪声数据。我们对 NLP 中 10 种不同的分类和序列到序列任务进行的广泛实验表明，GOLD 分别优于现有技术和 LLM，平均提高了 5% 和 14%。我们还将证明所提出的方法适用于较少探索和新颖的任务。代码可用。

Title: Developing Healthcare Language Model Embedding Spaces

• Authors: Niall Taylor, Dan Schofield, Andrey Kormilitzin, Dan W Joyce, Alejo Nevado-Holgado
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19802
• Pdf URL: https://arxiv.org/pdf/2403.19802
• Copy Paste: [[2403.19802]] Developing Healthcare Language Model Embedding Spaces(https://arxiv.org/abs/2403.19802)
• Keywords: language model, llm
• Abstract: Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text. We explore specialized pre-training to adapt smaller LLMs to different healthcare datasets. Three methods are assessed: traditional masked language modeling, Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel pre-training objective utilizing metadata categories from the healthcare settings. These schemes are evaluated on downstream document classification tasks for each dataset, with additional analysis of the resultant embedding spaces. Contrastively trained models outperform other approaches on the classification tasks, delivering strong performance from limited labeled data and with fewer model parameter updates required. While metadata-based pre-training does not further improve classifications across the datasets, it yields interesting embedding cluster separability. All domain adapted LLMs outperform their publicly available general base LLM, validating the importance of domain-specialization. This research illustrates efficient approaches to instill healthcare competency in compact LLMs even under tight computational budgets, an essential capability for responsible and sustainable deployment in local healthcare settings. We provide pre-training guidelines for specialized healthcare LLMs, motivate continued inquiry into contrastive objectives, and demonstrates adaptation techniques to align small LLMs with privacy-sensitive medical tasks.
• 摘要：预训练的大型语言模型 (LLM) 通常难以处理域外数据集，例如以医疗保健为重点的文本。我们探索专门的预培训，以使较小的法学硕士适应不同的医疗数据集。评估了三种方法：传统的屏蔽语言模型、无监督文本表示的深度对比学习 (DeCLUTR) 以及利用医疗保健环境中的元数据类别的新颖预训练目标。这些方案在每个数据集的下游文档分类任务上进行评估，并对所得嵌入空间进行额外分析。对比训练的模型在分类任务上优于其他方法，可以利用有限的标记数据提供强大的性能，并且所需的模型参数更新更少。虽然基于元数据的预训练并不能进一步改善数据集的分类，但它产生了有趣的嵌入集群可分离性。所有适应领域的法学硕士都优于其公开的通用基础法学硕士，验证了领域专业化的重要性。这项研究说明了即使在计算预算紧张的情况下也可以向紧凑型法学硕士灌输医疗保健能力的有效方法，这是在当地医疗保健环境中负责任和可持续部署的基本能力。我们为专业医疗保健法学硕士提供预培训指南，激励对对比目标的持续探究，并演示使小型法学硕士与隐私敏感的医疗任务保持一致的适应技术。

Title: Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs

• Authors: Kanishka Misra, Kyle Mahowald
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19827
• Pdf URL: https://arxiv.org/pdf/2403.19827
• Copy Paste: [[2403.19827]] Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs(https://arxiv.org/abs/2403.19827)
• Keywords: language model
• Abstract: Language models learn rare syntactic phenomena, but it has been argued that they rely on rote memorization, as opposed to grammatical generalization. Training on a corpus of human-scale in size (100M words), we iteratively trained transformer language models on systematically manipulated corpora and then evaluated their learning of a particular rare grammatical phenomenon: the English Article+Adjective+Numeral+Noun (AANN) construction (``a beautiful five days''). We first compared how well this construction was learned on the default corpus relative to a counterfactual corpus in which the AANN sentences were removed. AANNs were still learned better than systematically perturbed variants of the construction. Using additional counterfactual corpora, we suggest that this learning occurs through generalization from related constructions (e.g., ``a few days''). An additional experiment showed that this learning is enhanced when there is more variability in the input. Taken together, our results provide an existence proof that models learn rare grammatical phenomena by generalization from less rare phenomena. Code available at https://github.com/kanishkamisra/aannalysis
• 摘要：语言模型学习罕见的句法现象，但有人认为它们依赖于死记硬背，而不是语法概括。在人类规模的语料库（1 亿字）上进行训练，我们在系统操作的语料库上迭代训练 Transformer 语言模型，然后评估它们对一种特殊罕见语法现象的学习：英语冠词+形容词+数字+名词（AANN）结构（“美好的五天”）。我们首先比较了默认语料库和删除了 AANN 句子的反事实语料库中这种结构的学习效果。 AANN 的学习效果仍然比结构的系统扰动变体更好。使用额外的反事实语料库，我们建议这种学习是通过相关结构（例如“几天”）的概括来进行的。另一项实验表明，当输入存在更多变化时，这种学习能力会得到增强。总而言之，我们的结果提供了一个存在证明，即模型可以通过从不太罕见的现象中进行泛化来学习罕见的语法现象。代码可在 https://github.com/kanishkamisra/aannaanalysis 获取

Title: Target Span Detection for Implicit Harmful Content

• Authors: Nazanin Jafari, James Allan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19836
• Pdf URL: https://arxiv.org/pdf/2403.19836
• Copy Paste: [[2403.19836]] Target Span Detection for Implicit Harmful Content(https://arxiv.org/abs/2403.19836)
• Keywords: language model, llm
• Abstract: Identifying the targets of hate speech is a crucial step in grasping the nature of such speech and, ultimately, in improving the detection of offensive posts on online forums. Much harmful content on online platforms uses implicit language especially when targeting vulnerable and protected groups such as using stereotypical characteristics instead of explicit target names, making it harder to detect and mitigate the language. In this study, we focus on identifying implied targets of hate speech, essential for recognizing subtler hate speech and enhancing the detection of harmful content on digital platforms. We define a new task aimed at identifying the targets even when they are not explicitly stated. To address that task, we collect and annotate target spans in three prominent implicit hate speech datasets: SBIC, DynaHate, and IHC. We call the resulting merged collection Implicit-Target-Span. The collection is achieved using an innovative pooling method with matching scores based on human annotations and Large Language Models (LLMs). Our experiments indicate that Implicit-Target-Span provides a challenging test bed for target span detection methods.
• 摘要：识别仇恨言论的目标是掌握此类言论的性质并最终提高对在线论坛上攻击性帖子的检测的关键一步。在线平台上的许多有害内容都使用隐性语言，尤其是在针对弱势群体和受保护群体时，例如使用刻板特征而不是明确的目标名称，从而使检测和减轻语言变得更加困难。在这项研究中，我们专注于识别仇恨言论的隐含目标，这对于识别更微妙的仇恨言论和加强对数字平台上有害内容的检测至关重要。我们定义了一项新任务，旨在识别目标，即使目标没有明确说明。为了解决该任务，我们在三个著名的隐式仇恨言论数据集中收集并注释目标跨度：SBIC、DynaHate 和 IHC。我们将生成的合并集合称为隐式目标跨度。该集合是使用创新的池方法实现的，该方法具有基于人工注释和大型语言模型 (LLM) 的匹配分数。我们的实验表明，Implicit-Target-Span 为目标跨度检测方法提供了一个具有挑战性的测试平台。

Title: Localizing Paragraph Memorization in Language Models

• Authors: Niklas Stoehr, Mitchell Gordon, Chiyuan Zhang, Owen Lewis
• Subjects: cs.CL, cs.CR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.19851
• Pdf URL: https://arxiv.org/pdf/2403.19851
• Copy Paste: [[2403.19851]] Localizing Paragraph Memorization in Language Models(https://arxiv.org/abs/2403.19851)
• Keywords: language model
• Abstract: Can we localize the weights and mechanisms used by a language model to memorize and recite entire paragraphs of its training data? In this paper, we show that while memorization is spread across multiple layers and model components, gradients of memorized paragraphs have a distinguishable spatial pattern, being larger in lower model layers than gradients of non-memorized examples. Moreover, the memorized examples can be unlearned by fine-tuning only the high-gradient weights. We localize a low-layer attention head that appears to be especially involved in paragraph memorization. This head is predominantly focusing its attention on distinctive, rare tokens that are least frequent in a corpus-level unigram distribution. Next, we study how localized memorization is across the tokens in the prefix by perturbing tokens and measuring the caused change in the decoding. A few distinctive tokens early in a prefix can often corrupt the entire continuation. Overall, memorized continuations are not only harder to unlearn, but also to corrupt than non-memorized ones.
• 摘要：我们能否本地化语言模型用于记忆和背诵其训练数据的整个段落的权重和机制？在本文中，我们表明，虽然记忆分布在多个层和模型组件中，但记忆段落的梯度具有可区分的空间模式，在较低模型层中比未记忆示例的梯度更大。此外，可以通过仅微调高梯度权重来忘记记忆的示例。我们定位了一个低层注意力头，它似乎特别参与段落记忆。该头主要将注意力集中在语料库级一元分布中最不常见的独特、稀有标记上。接下来，我们通过扰动标记并测量解码中引起的变化来研究前缀中标记的本地化记忆如何。前缀中早期的一些独特标记通常会破坏整个延续。总体而言，记忆的延续不仅更难忘记，而且比未记忆的延续更容易损坏。

Title: Jamba: A Hybrid Transformer-Mamba Language Model

• Authors: Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, Yoav Shoham
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19887
• Pdf URL: https://arxiv.org/pdf/2403.19887
• Copy Paste: [[2403.19887]] Jamba: A Hybrid Transformer-Mamba Language Model(https://arxiv.org/abs/2403.19887)
• Keywords: language model
• Abstract: We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. In the particular configuration we have implemented, we end up with a powerful model that fits in a single 80GB GPU. Built at large scale, Jamba provides high throughput and small memory footprint compared to vanilla Transformers, and at the same time state-of-the-art performance on standard language model benchmarks and long-context evaluations. Remarkably, the model presents strong results for up to 256K tokens context length. We study various architectural decisions, such as how to combine Transformer and Mamba layers, and how to mix experts, and show that some of them are crucial in large scale modeling. We also describe several interesting properties of these architectures which the training and evaluation of Jamba have revealed, and plan to release checkpoints from various ablation runs, to encourage further exploration of this novel architecture. We make the weights of our implementation of Jamba publicly available under a permissive license.
• 摘要：我们推出了 Jamba，这是一种基于新型混合 Transformer-Mamba 专家混合 (MoE) 架构的新基础大型语言模型。具体来说，Jamba 将 Transformer 层和 Mamba 层的块交错在一起，享受这两个模型系列的优点。 MoE 添加到其中一些层中，以提高模型容量，同时保持活动参数使用的可控性。这种灵活的架构允许特定于资源和目标的配置。在我们实现的特定配置中，我们最终得到了一个适合单个 80GB GPU 的强大模型。与普通 Transformer 相比，Jamba 是大规模构建的，可提供高吞吐量和较小的内存占用，同时在标准语言模型基准测试和长上下文评估上具有最先进的性能。值得注意的是，该模型在高达 256K 令牌上下文长度的情况下呈现出强劲的结果。我们研究了各种架构决策，例如如何组合 Transformer 和 Mamba 层，以及如何混合专家，并表明其中一些决策对于大规模建模至关重要。我们还描述了 Jamba 的训练和评估所揭示的这些架构的几个有趣的属性，并计划从各种消融运行中发布检查点，以鼓励进一步探索这种新颖的架构。我们在许可下公开 Jamba 实施的权重。

Title: Towards a Robust Retrieval-Based Summarization System

• Authors: Shengjie Liu, Jing Wu, Jingyuan Bao, Wenyi Wang, Naira Hovakimyan, Christopher G Healey
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19889
• Pdf URL: https://arxiv.org/pdf/2403.19889
• Copy Paste: [[2403.19889]] Towards a Robust Retrieval-Based Summarization System(https://arxiv.org/abs/2403.19889)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: This paper describes an investigation of the robustness of large language models (LLMs) for retrieval augmented generation (RAG)-based summarization tasks. While LLMs provide summarization capabilities, their performance in complex, real-world scenarios remains under-explored. Our first contribution is LogicSumm, an innovative evaluation framework incorporating realistic scenarios to assess LLM robustness during RAG-based summarization. Based on limitations identified by LogiSumm, we then developed SummRAG, a comprehensive system to create training dialogues and fine-tune a model to enhance robustness within LogicSumm's scenarios. SummRAG is an example of our goal of defining structured methods to test the capabilities of an LLM, rather than addressing issues in a one-off fashion. Experimental results confirm the power of SummRAG, showcasing improved logical coherence and summarization quality. Data, corresponding model weights, and Python code are available online.
• 摘要：本文描述了对基于检索增强生成 (RAG) 的摘要任务的大型语言模型 (LLM) 鲁棒性的研究。虽然法学硕士提供总结能力，但它们在复杂的现实场景中的表现仍未得到充分探索。我们的第一个贡献是 LogicSumm，这是一个创新的评估框架，结合了现实场景，用于评估基于 RAG 的总结期间 LLM 的稳健性。根据 LogiSumm 发现的局限性，我们随后开发了 SummRAG，这是一个综合系统，用于创建训练对话并微调模型，以增强 LogicSumm 场景中的稳健性。 SummRAG 是我们的目标的一个例子，我们的目标是定义结构化方法来测试法学硕士的能力，而不是一次性解决问题。实验结果证实了 SummRAG 的强大功能，展示了逻辑一致性和摘要质量的改进。数据、相应的模型权重和 Python 代码均可在线获取。

Title: MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models

• Authors: Peng Ding, Jiading Fang, Peng Li, Kangrui Wang, Xiaochen Zhou, Mo Yu, Jing Li, Matthew R. Walter, Hongyuan Mei
• Subjects: cs.CL, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.19913
• Pdf URL: https://arxiv.org/pdf/2403.19913
• Copy Paste: [[2403.19913]] MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models(https://arxiv.org/abs/2403.19913)
• Keywords: language model, gpt, chat
• Abstract: Large language models such as ChatGPT and GPT-4 have recently achieved astonishing performance on a variety of natural language processing tasks. In this paper, we propose MANGO, a benchmark to evaluate their capabilities to perform text-based mapping and navigation. Our benchmark includes 53 mazes taken from a suite of textgames: each maze is paired with a walkthrough that visits every location but does not cover all possible paths. The task is question-answering: for each maze, a large language model reads the walkthrough and answers hundreds of mapping and navigation questions such as "How should you go to Attic from West of House?" and "Where are we if we go north and east from Cellar?". Although these questions are easy to humans, it turns out that even GPT-4, the best-to-date language model, performs poorly at answering them. Further, our experiments suggest that a strong mapping and navigation ability would benefit large language models in performing relevant downstream tasks, such as playing textgames. Our MANGO benchmark will facilitate future research on methods that improve the mapping and navigation capabilities of language models. We host our leaderboard, data, code, and evaluation program at https://mango.ttic.edu and https://github.com/oaklight/mango/.
• 摘要：ChatGPT 和 GPT-4 等大型语言模型最近在各种自然语言处理任务上取得了惊人的性能。在本文中，我们提出了 MANGO，这是一个评估其执行基于文本的地图和导航能力的基准。我们的基准测试包括取自一套文本游戏的 53 个迷宫：每个迷宫都配有一个演练，该演练会访问每个位置，但不会涵盖所有可能的路径。任务是问答：对于每个迷宫，大型语言模型都会读取演练并回答数百个地图和导航问题，例如“您应该如何从房子西边去阁楼？”和“如果我们从地窖向北和向东走，我们会在哪里？”。尽管这些问题对人类来说很容易，但事实证明，即使是迄今为止最好的语言模型 GPT-4 在回答这些问题时也表现不佳。此外，我们的实验表明，强大的映射和导航能力将有利于大型语言模型执行相关的下游任务，例如玩文本游戏。我们的 MANGO 基准将促进未来对提高语言模型的映射和导航能力的方法的研究。我们在 https://mango.ttic.edu 和 https://github.com/oaklight/mango/ 托管排行榜、数据、代码和评估程序。

Title: DiJiang: Efficient Large Language Models through Compact Kernelization

• Authors: Hanting Chen, Zhicheng Liu, Xutao Wang, Yuchuan Tian, Yunhe Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19928
• Pdf URL: https://arxiv.org/pdf/2403.19928
• Copy Paste: [[2403.19928]] DiJiang: Efficient Large Language Models through Compact Kernelization(https://arxiv.org/abs/2403.19928)
• Keywords: language model
• Abstract: In an effort to reduce the computational load of Transformers, research on linear attention has gained significant momentum. However, the improvement strategies for attention mechanisms typically necessitate extensive retraining, which is impractical for large language models with a vast array of parameters. In this paper, we present DiJiang, a novel Frequency Domain Kernelization approach that enables the transformation of a pre-trained vanilla Transformer into a linear complexity model with little training costs. By employing a weighted Quasi-Monte Carlo method for sampling, the proposed approach theoretically offers superior approximation efficiency. To further reduce the training computational complexity, our kernelization is based on Discrete Cosine Transform (DCT) operations. Extensive experiments demonstrate that the proposed method achieves comparable performance to the original Transformer, but with significantly reduced training costs and much faster inference speeds. Our DiJiang-7B achieves comparable performance with LLaMA2-7B on various benchmark while requires only about 1/50 training cost. Code is available at https://github.com/YuchuanTian/DiJiang.
• 摘要：为了减少 Transformer 的计算负载，线性注意力的研究取得了巨大的进展。然而，注意力机制的改进策略通常需要大量的再训练，这对于具有大量参数的大型语言模型来说是不切实际的。在本文中，我们提出了 DiJiang，一种新颖的频域核化方法，可以将预训练的普通 Transformer 转换为线性复杂度模型，且训练成本极低。通过采用加权准蒙特卡罗方法进行采样，该方法理论上提供了优越的逼近效率。为了进一步降低训练计算复杂度，我们的核化基于离散余弦变换（DCT）运算。大量实验表明，所提出的方法实现了与原始 Transformer 相当的性能，但训练成本显着降低，推理速度更快。我们的 DiJiang-7B 在各种基准测试中都达到了与 LLaMA2-7B 相当的性能，而仅需要约 1/50 的训练成本。代码可在 https://github.com/YuchuanTian/DiJiang 获取。

Title: Are LLMs Effective Backbones for Fine-tuning? An Experimental Investigation of Supervised LLMs on Chinese Short Text Matching

• Authors: Shulin Liu, Chengcheng Xu, Hao Liu, Tinghao Yu, Tao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19930
• Pdf URL: https://arxiv.org/pdf/2403.19930
• Copy Paste: [[2403.19930]] Are LLMs Effective Backbones for Fine-tuning? An Experimental Investigation of Supervised LLMs on Chinese Short Text Matching(https://arxiv.org/abs/2403.19930)
• Keywords: language model, llm, prompt
• Abstract: The recent success of Large Language Models (LLMs) has garnered significant attention in both academia and industry. Prior research on LLMs has primarily focused on enhancing or leveraging their generalization capabilities in zero- and few-shot settings. However, there has been limited investigation into effectively fine-tuning LLMs for a specific natural language understanding task in supervised settings. In this study, we conduct an experimental analysis by fine-tuning LLMs for the task of Chinese short text matching. We explore various factors that influence performance when fine-tuning LLMs, including task modeling methods, prompt formats, and output formats.
• 摘要：大型语言模型（LLM）最近的成功引起了学术界和工业界的高度关注。之前对法学硕士的研究主要集中在增强或利用其在零样本和少样本设置中的泛化能力。然而，对于在监督环境下针对特定自然语言理解任务有效微调法学硕士的研究还很有限。在本研究中，我们通过微调法学硕士来进行中文短文本匹配任务的实验分析。我们探讨了微调 LLM 时影响性能的各种因素，包括任务建模方法、提示格式和输出格式。

Title: Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning

• Authors: Qinhao Zhou, Zihan Zhang, Xiang Xiang, Ke Wang, Yuchuan Wu, Yongbin Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19962
• Pdf URL: https://arxiv.org/pdf/2403.19962
• Copy Paste: [[2403.19962]] Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning(https://arxiv.org/abs/2403.19962)
• Keywords: language model, gpt, llm, hallucination, prompt, chat, agent
• Abstract: Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.
• 摘要：开源预训练大型语言模型 (LLM) 表现出强大的语言理解和生成能力，使其在各种任务中取得了巨大成功。然而，当用作处理现实世界中复杂问题的代理时，它们的性能远远不如ChatGPT和GPT-4等大型商业模型。作为智能代理，法学硕士需要具备任务规划、长期记忆的能力，以及利用外部工具实现令人满意的绩效的能力。人们提出了各种方法来增强法学硕士的代理能力。一方面，方法涉及构建特定于代理的数据和微调模型。另一方面，一些方法侧重于设计有效激活法学硕士推理能力的提示。我们在 7B 和 13B 模型上探索这两种策略。我们提出了一种使用 GPT-4 构建特定于代理的数据的综合方法。通过对构造数据进行监督微调，我们发现对于这些参数数量相对较少的模型，监督微调可以显着减少代理任务中的幻觉输出和格式错误。此外，多路径推理和任务分解等技术可以有效降低问题复杂性并提高LLM作为代理的性能。我们在 AgentBench 的五个代理任务上评估我们的方法并取得令人满意的结果。

Title: Large Language Model based Situational Dialogues for Second Language Learning

• Authors: Shuyao Xu, Long Qin, Tianyang Chen, Zhenzhou Zha, Bingxue Qiu, Weizhi Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20005
• Pdf URL: https://arxiv.org/pdf/2403.20005
• Copy Paste: [[2403.20005]] Large Language Model based Situational Dialogues for Second Language Learning(https://arxiv.org/abs/2403.20005)
• Keywords: language model, llm
• Abstract: In second language learning, scenario-based conversation practice is important for language learners to achieve fluency in speaking, but students often lack sufficient opportunities to practice their conversational skills with qualified instructors or native speakers. To bridge this gap, we propose situational dialogue models for students to engage in conversational practice. Our situational dialogue models are fine-tuned on large language models (LLMs), with the aim of combining the engaging nature of an open-ended conversation with the focused practice of scenario-based tasks. Leveraging the generalization capabilities of LLMs, we demonstrate that our situational dialogue models perform effectively not only on training topics but also on topics not encountered during training. This offers a promising solution to support a wide range of conversational topics without extensive manual work. Additionally, research in the field of dialogue systems still lacks reliable automatic evaluation metrics, leading to human evaluation as the gold standard (Smith et al., 2022), which is typically expensive. To address the limitations of existing evaluation methods, we present a novel automatic evaluation method that employs fine-tuned LLMs to efficiently and effectively assess the performance of situational dialogue models.
• 摘要：在第二语言学习中，基于场景的会话练习对于语言学习者实现流利的口语非常重要，但学生往往缺乏足够的机会与合格的教师或母语人士一起练习会话技能。为了弥补这一差距，我们提出了情景对话模型，供学生进行对话练习。我们的情景对话模型在大型语言模型（LLM）上进行了微调，旨在将开放式对话的吸引力与基于场景的任务的集中实践相结合。利用法学硕士的泛化能力，我们证明了我们的情境对话模型不仅在训练主题上有效，而且在训练期间未遇到的主题上也有效。这提供了一个有前景的解决方案，无需大量手动工作即可支持广泛的对话主题。此外，对话系统领域的研究仍然缺乏可靠的自动评估指标，导致人类评估成为黄金标准（Smith et al., 2022），而这通常是昂贵的。为了解决现有评估方法的局限性，我们提出了一种新颖的自动评估方法，该方法采用微调的 LLM 来高效且有效地评估情境对话模型的性能。

Title: On Large Language Models' Hallucination with Regard to Known Facts

• Authors: Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, Jie Zhou
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.20009
• Pdf URL: https://arxiv.org/pdf/2403.20009
• Copy Paste: [[2403.20009]] On Large Language Models' Hallucination with Regard to Known Facts(https://arxiv.org/abs/2403.20009)
• Keywords: language model, llm, hallucination
• Abstract: Large language models are successful in answering factoid questions but are also prone to hallucination.We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics, an area not previously covered in studies on hallucinations.We are able to conduct this analysis via two key ideas.First, we identify the factual questions that query the same triplet knowledge but result in different answers. The difference between the model behaviors on the correct and incorrect outputs hence suggests the patterns when hallucinations happen. Second, to measure the pattern, we utilize mappings from the residual streams to vocabulary space. We reveal the different dynamics of the output token probabilities along the depths of layers between the correct and hallucinated cases. In hallucinated cases, the output token's information rarely demonstrates abrupt increases and consistent superiority in the later stages of the model. Leveraging the dynamic curve as a feature, we build a classifier capable of accurately detecting hallucinatory predictions with an 88\% success rate. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
• 摘要：大型语言模型可以成功地回答事实性问题，但也容易产生幻觉。我们从推理动力学的角度研究了法学硕士拥有正确答案知识但仍然产生幻觉的现象，这是以前幻觉研究中没有涉及的领域。我们能够通过两个关键思想进行此分析。首先，我们确定查询相同三元组知识但产生不同答案的事实问题。因此，正确和错误输出的模型行为之间的差异表明了幻觉发生时的模式。其次，为了测量模式，我们利用从残差流到词汇空间的映射。我们揭示了正确情况和幻觉情况之间沿层深度的输出令牌概率的不同动态。在幻觉情况下，输出令牌的信息很少在模型的后期阶段表现出突然增加和一致的优越性。利用动态曲线作为特征，我们构建了一个能够准确检测幻觉预测的分类器，成功率高达 88%。我们的研究有助于理解法学硕士根据已知事实产生幻觉的原因，更重要的是，有助于准确预测他们何时产生幻觉。

Title: Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

• Authors: Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, Qin Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20041
• Pdf URL: https://arxiv.org/pdf/2403.20041
• Copy Paste: [[2403.20041]] Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs(https://arxiv.org/abs/2403.20041)
• Keywords: language model, llm, chat
• Abstract: The Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite's performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.
• 摘要：大语言模型（LLM）广泛应用于智能助理、文本摘要、翻译和手机上的多模态等任务。然而，当前设备上LLM部署方法的推理速度较慢，导致用户体验较差。为了促进设备 GPU 上高效的 LLM 部署，我们提出了四种优化技术：（a）基于符号表达式的方法来支持动态形状模型推理； (b) 操作员优化和执行优先级设置，以提高推理速度并减少手机延迟； (c) 称为 M0E4 的 FP4 量化方法，用于减少反量化开销； (d) 基于子张量的技术，无需在 LLM 推理后复制 KV 缓存。此外，我们在移动推理引擎 Transformer-Lite 中实现了这些方法，该引擎与 Qualcomm 和 MTK 处理器兼容。我们使用具有从 2B 到 14B 不同架构和参数的 LLM 评估了 Transformer-Lite 的性能。具体来说，我们在 ChatGLM2 6B 中分别实现了 121 个令牌/秒和 14 个令牌/秒的预填充和解码速度，在较小的 Gemma 2B 中分别实现了 330 个令牌/秒和 30 个令牌/秒。与基于CPU的FastLLM和基于GPU的MLC-LLM相比，我们的引擎在预填充速度方面获得了超过10倍的加速，在解码速度方面获得了2~3倍的加速。

Title: Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning

• Authors: Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20046
• Pdf URL: https://arxiv.org/pdf/2403.20046
• Copy Paste: [[2403.20046]] Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning(https://arxiv.org/abs/2403.20046)
• Keywords: llm, prompt, chain-of-thought
• Abstract: Recent works have shown the benefits to LLMs from fine-tuning golden-standard Chain-of-Thought (CoT) rationales or using them as correct examples in few-shot prompting. While humans can indeed imitate correct examples, learning from our mistakes is another vital aspect of human cognition. Hence, a question naturally arises: \textit{can LLMs learn and benefit from their mistakes, especially for their reasoning? } This study investigates this problem from both the prompting and model-tuning perspectives. We begin by introducing \textsc{CoTErrorSet}, a new benchmark with 609,432 questions, each designed with both correct and error references, and demonstrating the types and reasons for making such mistakes. To explore the effectiveness of those mistakes, we design two methods: (1) \textbf{Self-rethinking} prompting guides LLMs to rethink whether they have made similar previous mistakes; and (2) \textbf{Mistake tuning} involves finetuning models in both correct and incorrect reasoning domains, rather than only tuning models to learn ground truth in traditional methodology. We conduct a series of experiments to prove LLMs can obtain benefits from mistakes in both directions. Our two methods offer potentially cost-effective strategies by leveraging errors to enhance reasoning capabilities, which costs significantly less than creating meticulously hand-crafted golden references. We ultimately make a thorough analysis of the reasons behind LLMs' errors, which provides directions that future research needs to overcome. \textsc{CoTErrorSet} will be published soon on \texttt{Anonymity Link}.
• 摘要：最近的研究表明，微调黄金标准思想链（CoT）基本原理或将其用作几次提示中的正确示例对法学硕士有好处。虽然人类确实可以模仿正确的例子，但从错误中学习是人类认知的另一个重要方面。因此，一个问题自然而然地出现了：\textit{法学硕士能否从他们的错误中学习并受益，特别是他们的推理？这项研究从提示和模型调整的角度研究了这个问题。我们首先介绍 \textsc{CoTErrorSet}，这是一个包含 609,432 个问题的新基准，每个问题都设计有正确和错误参考，并演示了犯此类错误的类型和原因。为了探索这些错误的有效性，我们设计了两种方法：（1）\textbf{自我反思}提示引导LLM重新思考他们以前是否犯过类似的错误； (2) \textbf{错误调整}涉及在正确和错误推理领域中对模型进行微调，而不是仅在传统方法中调整模型以学习基本事实。我们进行了一系列实验来证明法学硕士可以从两个方向的错误中获益。我们的两种方法通过利用错误来增强推理能力，提供了潜在的具有成本效益的策略，其成本明显低于创建精心手工制作的黄金参考。我们最终对LLM的错误背后的原因进行了彻底的分析，这为未来的研究需要克服的方向提供了方向。 \textsc{CoTErrorSet} 将很快在 \texttt{匿名链接} 上发布。

Title: Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets

• Authors: Shadi Manafi, Nikhil Krishnaswamy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20056
• Pdf URL: https://arxiv.org/pdf/2403.20056
• Copy Paste: [[2403.20056]] Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets(https://arxiv.org/abs/2403.20056)
• Keywords: language model, llm
• Abstract: Multilingual Language Models (MLLMs) exhibit robust cross-lingual transfer capabilities, or the ability to leverage information acquired in a source language and apply it to a target language. These capabilities find practical applications in well-established Natural Language Processing (NLP) tasks such as Named Entity Recognition (NER). This study aims to investigate the effectiveness of a source language when applied to a target language, particularly in the context of perturbing the input test set. We evaluate on 13 pairs of languages, each including one high-resource language (HRL) and one low-resource language (LRL) with a geographic, genetic, or borrowing relationship. We evaluate two well-known MLLMs--MBERT and XLM-R--on these pairs, in native LRL and cross-lingual transfer settings, in two tasks, under a set of different perturbations. Our findings indicate that NER cross-lingual transfer depends largely on the overlap of entity chunks. If a source and target language have more entities in common, the transfer ability is stronger. Models using cross-lingual transfer also appear to be somewhat more robust to certain perturbations of the input, perhaps indicating an ability to leverage stronger representations derived from the HRL. Our research provides valuable insights into cross-lingual transfer and its implications for NLP applications, and underscores the need to consider linguistic nuances and potential limitations when employing MLLMs across distinct languages.
• 摘要：多语言语言模型 (MLLM) 表现出强大的跨语言迁移能力，或者说能够利用从源语言获取的信息并将其应用到目标语言。这些功能在命名实体识别 (NER) 等成熟的自然语言处理 (NLP) 任务中找到了实际应用。本研究旨在调查源语言应用于目标语言时的有效性，特别是在扰乱输入测试集的情况下。我们评估了 13 对语言，每对包括一种高资源语言 (HRL) 和一种具有地理、遗传或借用关系的低资源语言 (LRL)。我们在本地 LRL 和跨语言传输设置中，在两个任务中，在一组不同的扰动下，在这些对上评估了两个著名的 MLLM——MBERT 和 XLM-R。我们的研究结果表明，NER 跨语言迁移很大程度上取决于实体块的重叠。源语言和目标语言的共同实体越多，迁移能力就越强。使用跨语言迁移的模型似乎对于输入的某些扰动也更加稳健，这可能表明能够利用从 HRL 导出的更强的表示。我们的研究为跨语言迁移及其对 NLP 应用的影响提供了宝贵的见解，并强调在跨不同语言使用 MLLM 时需要考虑语言的细微差别和潜在的限制。

Title: An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models

• Authors: Fahim Faisal, Antonios Anastasopoulos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20088
• Pdf URL: https://arxiv.org/pdf/2403.20088
• Copy Paste: [[2403.20088]] An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models(https://arxiv.org/abs/2403.20088)
• Keywords: language model
• Abstract: The capacity and effectiveness of pre-trained multilingual models (MLMs) for zero-shot cross-lingual transfer is well established. However, phenomena of positive or negative transfer, and the effect of language choice still need to be fully understood, especially in the complex setting of massively multilingual LMs. We propose an \textit{efficient} method to study transfer language influence in zero-shot performance on another target language. Unlike previous work, our approach disentangles downstream tasks from language, using dedicated adapter units. Our findings suggest that some languages do not largely affect others, while some languages, especially ones unseen during pre-training, can be extremely beneficial or detrimental for different target languages. We find that no transfer language is beneficial for all target languages. We do, curiously, observe languages previously unseen by MLMs consistently benefit from transfer from almost any language. We additionally use our modular approach to quantify negative interference efficiently and categorize languages accordingly. Furthermore, we provide a list of promising transfer-target language configurations that consistently lead to target language performance improvements. Code and data are publicly available: https://github.com/ffaisal93/neg_inf
• 摘要：用于零样本跨语言迁移的预训练多语言模型 (MLM) 的能力和有效性已得到充分证实。然而，正迁移或负迁移现象以及语言选择的影响仍然需要充分理解，特别是在大规模多语言 LM 的复杂环境中。我们提出了一种 \textit{efficient} 方法来研究迁移语言对另一种目标语言的零样本性能的影响。与之前的工作不同，我们的方法使用专用的适配器单元将下游任务与语言分开。我们的研究结果表明，某些语言不会对其他语言产生很大影响，而某些语言，尤其是在预训练期间未见过的语言，可能对不同的目标语言极其有益或有害。我们发现没有一种迁移语言对所有目标语言都有利。奇怪的是，我们确实观察到传销以前未见过的语言始终受益于几乎任何语言的迁移。我们还使用模块化方法来有效量化负面干扰并相应地对语言进行分类。此外，我们还提供了一系列有前途的迁移目标语言配置，这些配置始终能够提高目标语言的性能。代码和数据公开：https://github.com/ffaisal93/neg_inf

Title: User Modeling Challenges in Interactive AI Assistant Systems

• Authors: Megan Su, Yuwei Bao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20134
• Pdf URL: https://arxiv.org/pdf/2403.20134
• Copy Paste: [[2403.20134]] User Modeling Challenges in Interactive AI Assistant Systems(https://arxiv.org/abs/2403.20134)
• Keywords: language model
• Abstract: Interactive Artificial Intelligent(AI) assistant systems are designed to offer timely guidance to help human users to complete a variety tasks. One of the remaining challenges is to understand user's mental states during the task for more personalized guidance. In this work, we analyze users' mental states during task executions and investigate the capabilities and challenges for large language models to interpret user profiles for more personalized user guidance.
• 摘要：交互式人工智能（AI）辅助系统旨在提供及时指导，帮助人类用户完成各种任务。剩下的挑战之一是了解用户在任务过程中的心理状态，以获得更个性化的指导。在这项工作中，我们分析了任务执行期间用户的心理状态，并研究了大型语言模型解释用户配置文件以提供更个性化的用户指导的能力和挑战。

Title: Fine-tuning Large Language Models for Automated Diagnostic Screening Summaries

• Authors: Manjeet Yadav, Nilesh Kumar Sahu, Mudita Chaturvedi, Snehil Gupta, Haroon R Lone
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20145
• Pdf URL: https://arxiv.org/pdf/2403.20145
• Copy Paste: [[2403.20145]] Fine-tuning Large Language Models for Automated Diagnostic Screening Summaries(https://arxiv.org/abs/2403.20145)
• Keywords: language model, llm
• Abstract: Improving mental health support in developing countries is a pressing need. One potential solution is the development of scalable, automated systems to conduct diagnostic screenings, which could help alleviate the burden on mental health professionals. In this work, we evaluate several state-of-the-art Large Language Models (LLMs), with and without fine-tuning, on our custom dataset for generating concise summaries from mental state examinations. We rigorously evaluate four different models for summary generation using established ROUGE metrics and input from human evaluators. The results highlight that our top-performing fine-tuned model outperforms existing models, achieving ROUGE-1 and ROUGE-L values of 0.810 and 0.764, respectively. Furthermore, we assessed the fine-tuned model's generalizability on a publicly available D4 dataset, and the outcomes were promising, indicating its potential applicability beyond our custom dataset.
• 摘要：改善发展中国家的心理健康支持是迫切需要的。一种潜在的解决方案是开发可扩展的自动化系统来进行诊断筛查，这可能有助于减轻心理健康专业人员的负担。在这项工作中，我们在自定义数据集上评估了几种最先进的大型语言模型（LLM），无论是否经过微调，以便从心理状态检查中生成简洁的摘要。我们使用已建立的 ROUGE 指标和人类评估者的输入来严格评估四种不同的摘要生成模型。结果表明，我们的最佳微调模型优于现有模型，ROUGE-1 和 ROUGE-L 值分别为 0.810 和 0.764。此外，我们评估了微调模型在公开可用的 D4 数据集上的普遍性，结果令人鼓舞，表明其潜在适用性超出了我们的自定义数据集。

Title: IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context

• Authors: Nihar Ranjan Sahoo, Pranamya Prashant Kulkarni, Narjis Asad, Arif Ahmad, Tanu Goyal, Aparna Garimella, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20147
• Pdf URL: https://arxiv.org/pdf/2403.20147
• Copy Paste: [[2403.20147]] IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context(https://arxiv.org/abs/2403.20147)
• Keywords: language model, gpt, llm, chat
• Abstract: The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India's unique socio-cultural nuances. To bridge this gap, we introduce IndiBias, a comprehensive benchmarking dataset designed specifically for evaluating social biases in the Indian context. We filter and translate the existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian context in Hindi language. Additionally, we leverage LLMs including ChatGPT and InstructGPT to augment our dataset with diverse societal biases and stereotypes prevalent in India. The included bias dimensions encompass gender, religion, caste, age, region, physical appearance, and occupation. We also build a resource to address intersectional biases along three intersectional dimensions. Our dataset contains 800 filtered sentences from the CrowS-Pairs dataset and tuples for bias measurement across different demographics. It is made available in English and Hindi languages, providing a size comparable to existing benchmark datasets. Furthermore, using IndiBias we compare ten different language models on multiple bias measurement metrics. We observed that the language models exhibit more bias across a majority of the intersectional groups.
• 摘要：语言数据中社会偏见的普遍影响引发了对捕捉和评估大型语言模型（LLM）中这些偏见的基准数据集的需求。现有的工作主要集中在英语和西方背景上，为概括印度独特的社会文化细微差别的可靠数据集留下了空白。为了弥补这一差距，我们引入了 IndiBias，这是一个专门为评估印度背景下的社会偏见而设计的综合基准数据集。我们过滤并翻译现有的 CrowS-Pairs 数据集，以创建适合印地语印度语境的基准数据集。此外，我们利用包括 ChatGPT 和 InstructGPT 在内的法学硕士来扩充我们的数据集，以反映印度普遍存在的各种社会偏见和刻板印象。所包含的偏见维度包括性别、宗教、种姓、年龄、地区、外貌和职业。我们还建立了一个资源来解决三个交叉维度上的交叉偏见。我们的数据集包含来自 CrowS-Pairs 数据集的 800 个过滤句子以及用于测量不同人口统计数据的偏差的元组。它提供英语和印地语版本，其大小与现有基准数据集相当。此外，我们使用 IndiBias 比较了十种不同语言模型的多个偏差测量指标。我们观察到，语言模型在大多数交叉群体中表现出更多的偏见。

Title: ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models

• Authors: Zehao Wen, Rabih Younes
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.20158
• Pdf URL: https://arxiv.org/pdf/2403.20158
• Copy Paste: [[2403.20158]] ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models(https://arxiv.org/abs/2403.20158)
• Keywords: language model, gpt, llm, chat
• Abstract: In our rapidly evolving digital sphere, the ability to discern media bias becomes crucial as it can shape public sentiment and influence pivotal decisions. The advent of large language models (LLMs), such as ChatGPT, noted for their broad utility in various natural language processing (NLP) tasks, invites exploration of their efficacy in media bias detection. Can ChatGPT detect media bias? This study seeks to answer this question by leveraging the Media Bias Identification Benchmark (MBIB) to assess ChatGPT's competency in distinguishing six categories of media bias, juxtaposed against fine-tuned models such as BART, ConvBERT, and GPT-2. The findings present a dichotomy: ChatGPT performs at par with fine-tuned models in detecting hate speech and text-level context bias, yet faces difficulties with subtler elements of other bias detections, namely, fake news, racial, gender, and cognitive biases.
• 摘要：在我们快速发展的数字领域，辨别媒体偏见的能力变得至关重要，因为它可以塑造公众情绪并影响关键决策。 ChatGPT 等大型语言模型 (LLM) 的出现，因其在各种自然语言处理 (NLP) 任务中的广泛实用性而闻名，促使人们探索其在媒体偏见检测方面的功效。 ChatGPT 可以检测媒体偏见吗？本研究旨在通过利用媒体偏见识别基准 (MBIB) 来评估 ChatGPT 区分六类媒体偏见的能力，并与 BART、ConvBERT 和 GPT-2 等微调模型并列，从而回答这个问题。研究结果呈现出一种二分法：ChatGPT 在检测仇恨言论和文本级上下文偏见方面与微调模型表现不相上下，但在其他偏见检测的微妙元素（即假新闻、种族、性别和认知偏见）方面面临困难。

Title: Measuring Taiwanese Mandarin Language Understanding

• Authors: Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, Yun-Nung Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20180
• Pdf URL: https://arxiv.org/pdf/2403.20180
• Copy Paste: [[2403.20180]] Measuring Taiwanese Mandarin Language Understanding(https://arxiv.org/abs/2403.20180)
• Keywords: language model, llm, chain-of-thought
• Abstract: The evaluation of large language models (LLMs) has drawn substantial attention in the field recently. This work focuses on evaluating LLMs in a Chinese context, specifically, for Traditional Chinese which has been largely underrepresented in existing benchmarks. We present TMLU, a holistic evaluation suit tailored for assessing the advanced knowledge and reasoning capability in LLMs, under the context of Taiwanese Mandarin. TMLU consists of an array of 37 subjects across social science, STEM, humanities, Taiwan-specific content, and others, ranging from middle school to professional levels. In addition, we curate chain-of-thought-like few-shot explanations for each subject to facilitate the evaluation of complex reasoning skills. To establish a comprehensive baseline, we conduct extensive experiments and analysis on 24 advanced LLMs. The results suggest that Chinese open-weight models demonstrate inferior performance comparing to multilingual proprietary ones, and open-weight models tailored for Taiwanese Mandarin lag behind the Simplified-Chinese counterparts. The findings indicate great headrooms for improvement, and emphasize the goal of TMLU to foster the development of localized Taiwanese-Mandarin LLMs. We release the benchmark and evaluation scripts for the community to promote future research.
• 摘要：大型语言模型（LLM）的评估最近引起了该领域的广泛关注。这项工作的重点是在中文背景下评估法学硕士，特别是繁体中文，而繁体中文在现有基准中的代表性很大程度上不足。我们推出了TMLU，这是一套为评估法学硕士在台湾普通话背景下的高级知识和推理能力而量身定制的整体评估套装。 TMLU 包含 37 个科目，涵盖社会科学、STEM、人文、台湾特定内容等，涵盖从中学到专业水平。此外，我们还为每个主题策划了类似思维链的少量解释，以方便评估复杂的推理技能。为了建立全面的基线，我们对 24 名高级法学硕士进行了广泛的实验和分析。结果表明，与多语言专有模型相比，中文开放权重模型的性能较差，并且为台湾普通话定制的开放权重模型落后于简体中文模型。研究结果显示了巨大的改进空间，并强调了 TMLU 促进本地化台语法学硕士发展的目标。我们为社区发布基准和评估脚本以促进未来的研究。

Title: Using LLMs to Model the Beliefs and Preferences of Targeted Populations

• Authors: Keiichi Namikoshi, Alex Filipowicz, David A. Shamma, Rumen Iliev, Candice L. Hogan, Nikos Arechiga
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.20252
• Pdf URL: https://arxiv.org/pdf/2403.20252
• Copy Paste: [[2403.20252]] Using LLMs to Model the Beliefs and Preferences of Targeted Populations(https://arxiv.org/abs/2403.20252)
• Keywords: language model, llm
• Abstract: We consider the problem of aligning a large language model (LLM) to model the preferences of a human population. Modeling the beliefs, preferences, and behaviors of a specific population can be useful for a variety of different applications, such as conducting simulated focus groups for new products, conducting virtual surveys, and testing behavioral interventions, especially for interventions that are expensive, impractical, or unethical. Existing work has had mixed success using LLMs to accurately model human behavior in different contexts. We benchmark and evaluate two well-known fine-tuning approaches and evaluate the resulting populations on their ability to match the preferences of real human respondents on a survey of preferences for battery electric vehicles (BEVs). We evaluate our models against their ability to match population-wide statistics as well as their ability to match individual responses, and we investigate the role of temperature in controlling the trade-offs between these two. Additionally, we propose and evaluate a novel loss term to improve model performance on responses that require a numeric response.
• 摘要：我们考虑调整大型语言模型（LLM）来模拟人群偏好的问题。对特定人群的信念、偏好和行为进行建模可用于各种不同的应用，例如为新产品进行模拟焦点小组、进行虚拟调查和测试行为干预措施，特别是对于昂贵、不切实际、或不道德。现有的工作利用法学硕士来准确模拟不同环境下的人类行为，取得了不同程度的成功。我们对两种著名的微调方法进行了基准测试和评估，并在电池电动汽车 (BEV) 偏好调查中评估了所得人群与真实人类受访者偏好的匹配能力。我们根据模型匹配人群统计数据的能力以及匹配个体反应的能力来评估我们的模型，并研究温度在控制这两者之间的权衡中的作用。此外，我们提出并评估了一种新的损失项，以提高模型在需要数字响应的响应上的性能。

Title: ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models

• Authors: Thibaut Thonet, Jos Rozen, Laurent Besacier
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.20262
• Pdf URL: https://arxiv.org/pdf/2403.20262
• Copy Paste: [[2403.20262]] ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models(https://arxiv.org/abs/2403.20262)
• Keywords: language model, gpt, llm, long context
• Abstract: Research on Large Language Models (LLMs) has recently witnessed an increasing interest in extending models' context size to better capture dependencies within long documents. While benchmarks have been proposed to assess long-range abilities, existing efforts primarily considered generic tasks that are not necessarily aligned with real-world applications. In contrast, our work proposes a new benchmark for long-context LLMs focused on a practical meeting assistant scenario. In this scenario, the long contexts consist of transcripts obtained by automatic speech recognition, presenting unique challenges for LLMs due to the inherent noisiness and oral nature of such data. Our benchmark, named ELITR-Bench, augments the existing ELITR corpus' transcripts with 271 manually crafted questions and their ground-truth answers. Our experiments with recent long-context LLMs on ELITR-Bench highlight a gap between open-source and proprietary models, especially when questions are asked sequentially within a conversation. We also provide a thorough analysis of our GPT-4-based evaluation method, encompassing insights from a crowdsourcing study. Our findings suggest that while GPT-4's evaluation scores are correlated with human judges', its ability to differentiate among more than three score levels may be limited.
• 摘要：最近，大型语言模型 (LLM) 的研究发现，人们越来越关注扩展模型的上下文大小，以更好地捕获长文档中的依赖关系。虽然已经提出了基准来评估远程能力，但现有的工作主要考虑的是不一定与实际应用程序一致的通用任务。相比之下，我们的工作为长上下文法学硕士提出了一个新的基准，重点关注实用的会议助理场景。在这种情况下，长上下文由自动语音识别获得的转录本组成，由于此类数据固有的噪音和口头性质，给法学硕士带来了独特的挑战。我们的基准测试名为 ELITR-Bench，通过 271 个手动制作的问题及其真实答案来增强现有 ELITR 语料库的成绩单。我们最近在 ELITR-Bench 上对长上下文法学硕士进行的实验凸显了开源模型和专有模型之间的差距，特别是当在对话中按顺序提出问题时。我们还对基于 GPT-4 的评估方法进行了全面分析，其中包含来自众包研究的见解。我们的研究结果表明，虽然 GPT-4 的评估分数与人类评委的分数相关，但其区分三个以上分数水平的能力可能有限。

Title: Latxa: An Open Language Model and Evaluation Suite for Basque

• Authors: Julen Etxaniz, Oscar Sainz, Naiara Perez, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, Aitor Soroa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.20266
• Pdf URL: https://arxiv.org/pdf/2403.20266
• Copy Paste: [[2403.20266]] Latxa: An Open Language Model and Evaluation Suite for Basque(https://arxiv.org/abs/2403.20266)
• Keywords: language model, gpt, llm
• Abstract: We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the scarcity of high-quality benchmarks for Basque, we further introduce 4 multiple choice evaluation datasets: EusProficiency, comprising 5,169 questions from official language proficiency exams; EusReading, comprising 352 reading comprehension questions; EusTrivia, comprising 1,715 trivia questions from 5 knowledge areas; and EusExams, comprising 16,774 questions from public examinations. In our extensive evaluation, Latxa outperforms all previous open models we compare to by a large margin. In addition, it is competitive with GPT-4 Turbo in language proficiency and understanding, despite lagging behind in reading comprehension and knowledge-intensive tasks. Both the Latxa family of models, as well as our new pretraining corpora and evaluation datasets, are publicly available under open licenses at https://github.com/hitz-zentroa/latxa. Our suite enables reproducible research on methods to build LLMs for low-resource languages.
• 摘要：我们引入 Latxa，这是一个巴斯克语大型语言模型家族，参数范围从 7 到 700 亿个参数。 Latxa 基于 Llama 2，我们继续在包含 430 万个文档和 4.2B 个标记的新巴斯克语料库上对其进行预训练。针对巴斯克语高质量基准的缺乏，我们进一步引入了 4 个多项选择评估数据集：EusProficiency，包含来自官方语言能力考试的 5,169 个问题； EusReading，包含 352 道阅读理解问题； EusTrivia，包含来自 5 个知识领域的 1,715 个问答题；和 EusExams，包含来自公开考试的 16,774 个问题。在我们的广泛评估中，Latxa 的性能远远优于我们之前比较的所有开放模型。此外，尽管在阅读理解和知识密集型任务方面落后，但它在语言能力和理解方面与 GPT-4 Turbo 具有竞争力。 Latxa 系列模型以及我们新的预训练语料库和评估数据集均可在开放许可下公开获取：https://github.com/hitz-zentroa/latxa。我们的套件可以对为低资源语言构建法学硕士的方法进行可重复的研究。

Title: LUQ: Long-text Uncertainty Quantification for LLMs

• Authors: Caiqi Zhang, Fangyu Liu, Marco Basaldella, Nigel Collier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.20279
• Pdf URL: https://arxiv.org/pdf/2403.20279
• Copy Paste: [[2403.20279]] LUQ: Long-text Uncertainty Quantification for LLMs(https://arxiv.org/abs/2403.20279)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capability in a variety of NLP tasks. Despite their effectiveness, these models are prone to generate nonfactual content. Uncertainty Quantification (UQ) is pivotal in enhancing our understanding of a model's confidence in its generated content, thereby aiding in the mitigation of nonfactual outputs. Existing research on UQ predominantly targets short text generation, typically yielding brief, word-limited responses. However, real-world applications frequently necessitate much longer responses. Our study first highlights the limitations of current UQ methods in handling long text generation. We then introduce \textsc{Luq}, a novel sampling-based UQ approach specifically designed for long text. Our findings reveal that \textsc{Luq} outperforms existing baseline methods in correlating with the model's factuality scores (negative coefficient of -0.85 observed for Gemini Pro). With \textsc{Luq} as the tool for UQ, we investigate behavior patterns of several popular LLMs' response confidence spectrum and how that interplays with the response' factuality. We identify that LLMs lack confidence in generating long text for rare facts and a factually strong model (i.e. GPT-4) tends to reject questions it is not sure about. To further improve the factual accuracy of LLM responses, we propose a method called \textsc{Luq-Ensemble} that ensembles responses from multiple models and selects the response with the least uncertainty. The ensembling method greatly improves the response factuality upon the best standalone LLM.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务中表现出了卓越的能力。尽管它们很有效，但这些模型很容易生成不真实的内容。不确定性量化 (UQ) 对于增强我们对模型对其生成内容的置信度的理解至关重要，从而有助于减少非事实输出。昆士兰大学现有的研究主要针对短文本生成，通常会产生简短的、字数有限的回复。然而，现实世界的应用程序经常需要更长的响应。我们的研究首先强调了当前昆士兰大学方法在处理长文本生成方面的局限性。然后我们介绍 \textsc{Luq}，这是一种专门为长文本设计的基于采样的新型 UQ 方法。我们的研究结果表明，\textsc{Luq} 在与模型真实性得分（Gemini Pro 观察到的负系数 -0.85）相关方面优于现有基线方法。使用 \textsc{Luq} 作为 UQ 的工具，我们研究了几个流行的 LLM 的响应置信谱的行为模式以及它如何与响应的事实性相互作用。我们发现法学硕士对为罕见事实生成长文本缺乏信心，而事实强大的模型（即 GPT-4）往往会拒绝它不确定的问题。为了进一步提高 LLM 响应的事实准确性，我们提出了一种名为 \textsc{Luq-Ensemble} 的方法，该方法集成来自多个模型的响应并选择不确定性最小的响应。集成方法极大地提高了最佳独立法学硕士的响应真实性。

Title: Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain

• Authors: Burcu Sayin, Pasquale Minervini, Jacopo Staiano, Andrea Passerini
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.20288
• Pdf URL: https://arxiv.org/pdf/2403.20288
• Copy Paste: [[2403.20288]] Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain(https://arxiv.org/abs/2403.20288)
• Keywords: language model, llm, prompt
• Abstract: We explore the potential of Large Language Models (LLMs) to assist and potentially correct physicians in medical decision-making tasks. We evaluate several LLMs, including Meditron, Llama2, and Mistral, to analyze the ability of these models to interact effectively with physicians across different scenarios. We consider questions from PubMedQA and several tasks, ranging from binary (yes/no) responses to long answer generation, where the answer of the model is produced after an interaction with a physician. Our findings suggest that prompt design significantly influences the downstream accuracy of LLMs and that LLMs can provide valuable feedback to physicians, challenging incorrect diagnoses and contributing to more accurate decision-making. For example, when the physician is accurate 38% of the time, Mistral can produce the correct answer, improving accuracy up to 74% depending on the prompt being used, while Llama2 and Meditron models exhibit greater sensitivity to prompt choice. Our analysis also uncovers the challenges of ensuring that LLM-generated suggestions are pertinent and useful, emphasizing the need for further research in this area.
• 摘要：我们探索大型语言模型（LLM）在协助和潜在纠正医生医疗决策任务方面的潜力。我们评估了多个法学硕士，包括 Meditron、Llama2 和 Mistral，以分析这些模型在不同场景下与医生有效交互的能力。我们考虑来自 PubMedQA 的问题和多项任务，从二元（是/否）响应到长答案生成，其中模型的答案是在与医生互动后生成的。我们的研究结果表明，及时设计会显着影响法学硕士的下游准确性，并且法学硕士可以为医生提供有价值的反馈，挑战错误的诊断并有助于更准确的决策。例如，当医生的准确率达到 38% 时，Mistral 可以给出正确答案，根据所使用的提示将准确率提高高达 74%，而 Llama2 和 Meditron 模型对提示选择表现出更高的敏感性。我们的分析还揭示了确保法学硕士生成的建议相关且有用的挑战，强调了在该领域进一步研究的必要性。

Title: Gecko: Versatile Text Embeddings Distilled from Large Language Models

• Authors: Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, Iftekhar Naim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.20327
• Pdf URL: https://arxiv.org/pdf/2403.20327
• Copy Paste: [[2403.20327]] Gecko: Versatile Text Embeddings Distilled from Large Language Models(https://arxiv.org/abs/2403.20327)
• Keywords: language model, llm
• Abstract: We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language models (LLMs) into a retriever. Our two-step distillation process begins with generating diverse, synthetic paired data using an LLM. Next, we further refine the data quality by retrieving a set of candidate passages for each query, and relabeling the positive and hard negative passages using the same LLM. The effectiveness of our approach is demonstrated by the compactness of the Gecko. On the Massive Text Embedding Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing entries with 768 embedding size. Gecko with 768 embedding dimensions achieves an average score of 66.31, competing with 7x larger models and 5x higher dimensional embeddings.
• 摘要：我们推出了 Gecko，一种紧凑且多功能的文本嵌入模型。 Gecko 通过利用一个关键思想实现了强大的检索性能：将大型语言模型 (LLM) 中的知识提取到检索器中。我们的两步蒸馏过程首先使用法学硕士生成多样化的合成配对数据。接下来，我们通过检索每个查询的一组候选段落，并使用相同的 LLM 重新标记正向和硬负向段落，进一步细化数据质量。 Gecko 的紧凑性证明了我们方法的有效性。在大规模文本嵌入基准 (MTEB) 上，嵌入尺寸为 256 的 Gecko 优于嵌入尺寸为 768 的所有现有条目。具有 768 个嵌入维度的 Gecko 平均得分为 66.31，与大 7 倍的模型和高 5 倍维度的嵌入竞争。

Title: ReALM: Reference Resolution As Language Modeling

• Authors: Joel Ruben Antony Moniz, Soundarya Krishnan, Melis Ozyildirim, Prathamesh Saraf, Halim Cagri Ates, Yuan Zhang, Hong Yu, Nidhi Rajshree
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.20329
• Pdf URL: https://arxiv.org/pdf/2403.20329
• Copy Paste: [[2403.20329]] ReALM: Reference Resolution As Language Modeling(https://arxiv.org/abs/2403.20329)
• Keywords: language model, gpt, llm
• Abstract: Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds. This context includes both previous turns and context that pertains to non-conversational entities, such as entities on the user's screen or those running in the background. While LLMs have been shown to be extremely powerful for a variety of tasks, their use in reference resolution, particularly for non-conversational entities, remains underutilized. This paper demonstrates how LLMs can be used to create an extremely effective system to resolve references of various types, by showing how reference resolution can be converted into a language modeling problem, despite involving forms of entities like those on screen that are not traditionally conducive to being reduced to a text-only modality. We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it.
• 摘要：参考解析是一个重要的问题，对于理解和成功处理不同类型的上下文至关重要。此上下文包括先前的轮次和与非会话实体相关的上下文，例如用户屏幕上的实体或在后台运行的实体。虽然法学硕士已被证明对于各种任务都非常强大，但它们在参考解析中的使用，特别是对于非会话实体，仍然没有得到充分利用。本文通过展示如何将引用解析转换为语言建模问题，展示了如何使用法学硕士来创建一个极其有效的系统来解析各种类型的引用，尽管涉及到传统上不利于解决问题的屏幕上的实体形式。被简化为纯文本模式。我们展示了对不同类型参考具有类似功能的现有系统的巨大改进，我们最小的模型在屏幕参考上获得了超过 5% 的绝对增益。我们还针对 GPT-3.5 和 GPT-4 进行了基准测试，我们最小的模型实现了与 GPT-4 相当的性能，而我们较大的模型则远远优于 GPT-4。