2024-03-08

Title: Can Large Language Models do Analytical Reasoning?

• Authors: Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, Fei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04031
• Pdf URL: https://arxiv.org/pdf/2403.04031
• Copy Paste: [[2403.04031]] Can Large Language Models do Analytical Reasoning?(https://arxiv.org/abs/2403.04031)
• Keywords: language model, gpt, prompt
• Abstract: This paper explores the cutting-edge Large Language Model with analytical reasoning on sports. Our analytical reasoning embodies the tasks of letting large language models count how many points each team scores in a quarter in the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find among all the models we employed, GPT-4 stands out in effectiveness, followed by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind. Specifically, we compare three different prompting techniques and a divide-and-conquer approach, we find that the latter was the most effective. Our divide-and-conquer approach breaks down play-by-play data into smaller, more manageable segments, solves each piece individually, and then aggregates them together. Besides the divide-and-conquer approach, we also explore the Chain of Thought (CoT) strategy, which markedly improves outcomes for certain models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing significantly. However, the CoT strategy has negligible or even detrimental effects on the performance of other models like GPT-3.5 and Gemini-Pro. Secondly, to our surprise, we observe that most models, including GPT-4, struggle to accurately count the total scores for NBA quarters despite showing strong performance in counting NFL quarter scores. This leads us to further investigate the factors that impact the complexity of analytical reasoning tasks with extensive experiments, through which we conclude that task complexity depends on the length of context, the information density, and the presence of related information. Our research provides valuable insights into the complexity of analytical reasoning tasks and potential directions for developing future large language models.
• 摘要：本文探讨了具有体育分析推理功能的前沿大语言模型。我们的分析推理体现了让大型语言模型计算 NBA 和 NFL 比赛中每支球队在一个季度内得分的任务。我们的主要发现有两个方面。首先，我们发现在我们使用的所有模型中，GPT-4 的有效性最为突出，其次是 Claude-2.1，GPT-3.5、Gemini-Pro 和 Llama-2-70b 落后。具体来说，我们比较了三种不同的提示技术和分而治之的方法，我们发现后者是最有效的。我们的分而治之的方法将逐个比赛数据分解为更小、更易于管理的部分，单独解决每个部分，然后将它们聚合在一起。除了分而治之的方法之外，我们还探索了思想链（CoT）策略，该策略显着改善了某些模型的结果，特别是 GPT-4 和 Claude-2.1，其准确率显着提高。然而，CoT 策略对 GPT-3.5 和 Gemini-Pro 等其他模型的性能影响可以忽略不计甚至有害。其次，令我们惊讶的是，我们观察到大多数模型（包括 GPT-4）都难以准确计算 NBA 季度的总得分，尽管在计算 NFL 季度得分方面表现出色。这促使我们通过大量实验进一步研究影响分析推理任务复杂性的因素，通过这些实验我们得出结论：任务复杂性取决于上下文的长度、信息密度和相关信息的存在。我们的研究为分析推理任务的复杂性和开发未来大型语言模型的潜在方向提供了宝贵的见解。

Title: Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis

• Authors: Abdelrahman Abdallah, Daniel Eberharter, Zoe Pfister, Adam Jatowt
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04080
• Pdf URL: https://arxiv.org/pdf/2403.04080
• Copy Paste: [[2403.04080]] Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis(https://arxiv.org/abs/2403.04080)
• Keywords: language model
• Abstract: This paper presents a comprehensive survey of research works on the topic of form understanding in the context of scanned documents. We delve into recent advancements and breakthroughs in the field, highlighting the significance of language models and transformers in solving this challenging task. Our research methodology involves an in-depth analysis of popular documents and forms of understanding of trends over the last decade, enabling us to offer valuable insights into the evolution of this domain. Focusing on cutting-edge models, we showcase how transformers have propelled the field forward, revolutionizing form-understanding techniques. Our exploration includes an extensive examination of state-of-the-art language models designed to effectively tackle the complexities of noisy scanned documents. Furthermore, we present an overview of the latest and most relevant datasets, which serve as essential benchmarks for evaluating the performance of selected models. By comparing and contrasting the capabilities of these models, we aim to provide researchers and practitioners with useful guidance in choosing the most suitable solutions for their specific form understanding tasks.
• 摘要：本文对扫描文档背景下的形式理解主题的研究工作进行了全面的调查。我们深入研究了该领域的最新进展和突破，强调了语言模型和转换器在解决这一具有挑战性的任务中的重要性。我们的研究方法涉及对过去十年流行文献和趋势理解形式的深入分析，使我们能够对该领域的演变提供有价值的见解。我们专注于尖端模型，展示 Transformer 如何推动该领域向前发展，彻底改变形式理解技术。我们的探索包括对最先进的语言模型的广泛检查，旨在有效解决噪声扫描文档的复杂性。此外，我们还概述了最新和最相关的数据集，这些数据集可作为评估所选模型性能的基本基准。通过比较和对比这些模型的功能，我们的目标是为研究人员和实践者提供有用的指导，帮助他们为特定的形式理解任务选择最合适的解决方案。

Title: Metric-aware LLM inference

• Authors: Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, Sanjiv Kumar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04182
• Pdf URL: https://arxiv.org/pdf/2403.04182
• Copy Paste: [[2403.04182]] Metric-aware LLM inference(https://arxiv.org/abs/2403.04182)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated strong results on a range of NLP tasks. Typically, outputs are obtained via autoregressive sampling from the LLM's underlying distribution. We show that this inference strategy can be suboptimal for a range of tasks and associated evaluation metrics. As a remedy, we propose metric aware LLM inference: a decision theoretic approach optimizing for custom metrics at inference time. We report improvements over baselines on academic benchmarks and publicly available models.
• 摘要：大型语言模型 (LLM) 在一系列 NLP 任务上表现出了强劲的结果。通常，输出是通过从 LLM 的基础分布中进行自回归采样获得的。我们表明，这种推理策略对于一系列任务和相关的评估指标来说可能不是最优的。作为补救措施，我们提出了度量感知 LLM 推理：一种在推理时优化自定义度量的决策理论方法。我们报告了学术基准和公开模型的基线改进。

Title: Large Language Models are In-Context Molecule Learners

• Authors: Jiatong Li, Wei Liu, Zhihao Ding, Wenqi Fan, Yuqiang Li, Qing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04197
• Pdf URL: https://arxiv.org/pdf/2403.04197
• Copy Paste: [[2403.04197]] Large Language Models are In-Context Molecule Learners(https://arxiv.org/abs/2403.04197)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated exceptional performance in biochemical tasks, especially the molecule caption translation task, which aims to bridge the gap between molecules and natural language texts. However, previous methods in adapting LLMs to the molecule-caption translation task required extra domain-specific pre-training stages, suffered weak alignment between molecular and textual spaces, or imposed stringent demands on the scale of LLMs. To resolve the challenges, we propose In-Context Molecule Adaptation (ICMA), as a new paradigm allowing LLMs to learn the molecule-text alignment from context examples via In-Context Molecule Tuning. Specifically, ICMA incorporates the following three stages: Cross-modal Retrieval, Post-retrieval Re-ranking, and In-context Molecule Tuning. Initially, Cross-modal Retrieval utilizes BM25 Caption Retrieval and Molecule Graph Retrieval to retrieve informative context examples. Additionally, we also propose Post-retrieval Re-ranking with Sequence Reversal and Random Walk to further improve the quality of retrieval results. Finally, In-Context Molecule Tuning unlocks the in-context molecule learning capability of LLMs with retrieved examples and adapts the parameters of LLMs for the molecule-caption translation task. Experimental results demonstrate that ICMT can empower LLMs to achieve state-of-the-art or comparable performance without extra training corpora and intricate structures, showing that LLMs are inherently in-context molecule learners.
• 摘要：大型语言模型（LLM）在生化任务中表现出了卓越的性能，尤其是分子标题翻译任务，该任务旨在弥合分子和自然语言文本之间的差距。然而，以前使法学硕士适应分子标题翻译任务的方法需要额外的特定领域的预训练阶段，分子和文本空间之间的对齐较弱，或者对法学硕士的规模提出了严格的要求。为了解决这些挑战，我们提出了上下文分子适应（ICMA），作为一种新范式，允许法学硕士通过上下文分子调整从上下文示例中学习分子文本对齐。具体来说，ICMA 包含以下三个阶段：跨模态检索、检索后重新排序和上下文分子调谐。最初，跨模式检索利用 BM25 标题检索和分子图检索来检索信息丰富的上下文示例。此外，我们还提出使用序列反转和随机游走进行检索后重新排序，以进一步提高检索结果的质量。最后，上下文分子调整通过检索到的示例解锁了法学硕士的上下文分子学习能力，并针对分子标题翻译任务调整了法学硕士的参数。实验结果表明，ICMT 可以使法学硕士能够实现最先进的或可比的性能，而无需额外的训练语料库和复杂的结构，这表明法学硕士本质上是上下文中的分子学习者。

Title: Self-Evaluation of Large Language Model based on Glass-box Features

• Authors: Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04222
• Pdf URL: https://arxiv.org/pdf/2403.04222
• Copy Paste: [[2403.04222]] Self-Evaluation of Large Language Model based on Glass-box Features(https://arxiv.org/abs/2403.04222)
• Keywords: language model, llm, prompt
• Abstract: The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect - model-aware glass-box features - is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable indicator for quality evaluation. Furthermore, we propose two strategies to enhance the evaluation by incorporating features derived from references. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.
• 摘要：开源大型语言模型 (LLM) 的激增凸显了对评估方法的迫切需求。现有的工作主要依靠外部评估者，侧重于培训和激励策略。然而，一个关键的方面——模型感知玻璃盒功能——却被忽视了。在本研究中，我们探讨了玻璃盒特征在自我评估场景下的效用，即应用法学硕士来评估自己的输出。我们研究了各种玻璃盒特征组，发现 softmax 分布可以作为质量评估的可靠指标。此外，我们提出了两种策略来通过结合参考文献中的特征来增强评估。公共基准的实验结果验证了利用玻璃盒特征进行法学硕士自我评估的可行性。

Title: Aligners: Decoupling LLMs and Alignment

• Authors: Lilian Ngweta, Mayank Agarwal, Subha Maity, Alex Gittens, Yuekai Sun, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04224
• Pdf URL: https://arxiv.org/pdf/2403.04224
• Copy Paste: [[2403.04224]] Aligners: Decoupling LLMs and Alignment(https://arxiv.org/abs/2403.04224)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. Alignment is challenging, costly, and needs to be repeated for every LLM and alignment criterion. We propose to decouple LLMs and alignment by training aligner models that can be used to align any LLM for a given criteria on an as-needed basis, thus also reducing the potential negative impacts of alignment on performance. Our recipe for training the aligner models solely relies on synthetic data generated with a (prompted) LLM and can be easily adjusted for a variety of alignment criteria. We illustrate our method by training an "ethical" aligner and verify its efficacy empirically.
• 摘要：大型语言模型 (LLM) 需要符合人类期望，以确保其在大多数应用中的安全性和实用性。对齐是具有挑战性的、成本高昂的，并且需要针对每个法学硕士和对齐标准重复进行。我们建议通过训练对齐器模型来解耦法学硕士和对齐，该模型可用于根据需要对齐任何给定标准的法学硕士，从而减少对齐对性能的潜在负面影响。我们训练对准器模型的方法仅依赖于（提示的）LLM 生成的合成数据，并且可以轻松调整以适应各种对准标准。我们通过训练“道德”矫正器来说明我们的方法，并根据经验验证其功效。

Title: DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning

• Authors: Xingwei Qu, Yiming Liang, Yucheng Wang, Tianyu Zheng, Tommy Yue, Lei Ma, Stephen W. Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04233
• Pdf URL: https://arxiv.org/pdf/2403.04233
• Copy Paste: [[2403.04233]] DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning(https://arxiv.org/abs/2403.04233)
• Keywords: language model, llm
• Abstract: It has long been assumed that the sheer number of parameters in large language models (LLMs) drives in-context learning (ICL) capabilities, enabling remarkable performance improvements by leveraging task-specific demonstrations. Challenging this hypothesis, we introduce DEEP-ICL, a novel task Definition Enriched ExPert Ensembling methodology for ICL. DEEP-ICL explicitly extracts task definitions from given demonstrations and generates responses through learning task-specific examples. We argue that improvement from ICL does not directly rely on model size, but essentially stems from understanding task definitions and task-guided learning. Inspired by this, DEEP-ICL combines two 3B models with distinct roles (one for concluding task definitions and the other for learning task demonstrations) and achieves comparable performance to LLaMA2-13B. Furthermore, our framework outperforms conventional ICL by overcoming pretraining sequence length limitations, by supporting unlimited demonstrations. We contend that DEEP-ICL presents a novel alternative for achieving efficient few-shot learning, extending beyond the conventional ICL.
• 摘要：长期以来，人们一直认为大型语言模型 (LLM) 中的参数数量巨大，可驱动上下文学习 (ICL) 功能，从而通过利用特定于任务的演示来实现显着的性能改进。为了挑战这一假设，我们引入了 DEEP-ICL，这是一种用于 ICL 的新颖任务定义丰富的专家集成方法。 DEEP-ICL 从给定的演示中显式提取任务定义，并通过学习特定于任务的示例生成响应。我们认为 ICL 的改进并不直接依赖于模型大小，而本质上源于对任务定义和任务引导学习的理解。受此启发，DEEP-ICL 结合了两个具有不同角色的 3B 模型（一个用于总结任务定义，另一个用于学习任务演示），并实现了与 LLaMA2-13B 相当的性能。此外，我们的框架通过克服预训练序列长度限制并支持无限演示，优于传统 ICL。我们认为，DEEP-ICL 为实现高效的小样本学习提供了一种新颖的替代方案，超越了传统的 ICL。

Title: UltraWiki: Ultra-fine-grained Entity Set Expansion with Negative Seed Entities

• Authors: Yangning Li, Qingsong Lv, Tianyu Yu, Yinghui Li, Shulin Huang, Tingwei Lu, Xuming Hu, Wenhao JIang, Hai-Tao Zheng, Hui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04247
• Pdf URL: https://arxiv.org/pdf/2403.04247
• Copy Paste: [[2403.04247]] UltraWiki: Ultra-fine-grained Entity Set Expansion with Negative Seed Entities(https://arxiv.org/abs/2403.04247)
• Keywords: language model, chain-of-thought
• Abstract: Entity Set Expansion (ESE) aims to identify new entities belonging to the same semantic class as a given set of seed entities. Traditional methods primarily relied on positive seed entities to represent a target semantic class, which poses challenge for the representation of ultra-fine-grained semantic classes. Ultra-fine-grained semantic classes are defined based on fine-grained semantic classes with more specific attribute constraints. Describing it with positive seed entities alone cause two issues: (i) Ambiguity among ultra-fine-grained semantic classes. (ii) Inability to define "unwanted" semantic. Due to these inherent shortcomings, previous methods struggle to address the ultra-fine-grained ESE (Ultra-ESE). To solve this issue, we first introduce negative seed entities in the inputs, which belong to the same fine-grained semantic class as the positive seed entities but differ in certain attributes. Negative seed entities eliminate the semantic ambiguity by contrast between positive and negative attributes. Meanwhile, it provide a straightforward way to express "unwanted". To assess model performance in Ultra-ESE, we constructed UltraWiki, the first large-scale dataset tailored for Ultra-ESE. UltraWiki encompasses 236 ultra-fine-grained semantic classes, where each query of them is represented with 3-5 positive and negative seed entities. A retrieval-based framework RetExpan and a generation-based framework GenExpan are proposed to comprehensively assess the efficacy of large language models from two different paradigms in Ultra-ESE. Moreover, we devised three strategies to enhance models' comprehension of ultra-fine-grained entities semantics: contrastive learning, retrieval augmentation, and chain-of-thought reasoning. Extensive experiments confirm the effectiveness of our proposed strategies and also reveal that there remains a large space for improvement in Ultra-ESE.
• 摘要：实体集扩展（ESE）旨在识别与给定种子实体集属于同一语义类的新实体。传统方法主要依靠正种子实体来表示目标语义类，这对超细粒度语义类的表示提出了挑战。超细粒度语义类是在具有更具体属性约束的细粒度语义类的基础上定义的。仅用正面种子实体来描述它会导致两个问题：（i）超细粒度语义类别之间的歧义。 (ii) 无法定义“不需要的”语义。由于这些固有的缺点，以前的方法很难解决超细粒度的 ESE（Ultra-ESE）。为了解决这个问题，我们首先在输入中引入负种子实体，它们与正种子实体属于相同的细粒度语义类，但在某些属性上有所不同。负面种子实体通过正面和负面属性之间的对比来消除语义歧义。同时，它提供了一种直接的方式来表达“不想要的”。为了评估 Ultra-ESE 中的模型性能，我们构建了 UltraWiki，这是第一个为 Ultra-ESE 定制的大型数据集。 UltraWiki 包含 236 个超细粒度语义类，其中每个查询都用 3-5 个正负种子实体表示。提出了基于检索的框架 RetExpan 和基于生成的框架 GenExpan 来综合评估 Ultra-ESE 中两种不同范式的大型语言模型的有效性。此外，我们设计了三种策略来增强模型对超细粒度实体语义的理解：对比学习、检索增强和思想链推理。大量的实验证实了我们提出的策略的有效性，也表明 Ultra-ESE 仍有很大的改进空间。

Title: Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy

• Authors: Yu Zhu, Chuxiong Sun, Wenfei Yang, Wenqiang Wei, Bo Tang, Tianzhu Zhang, Zhiyu Li, Shifeng Zhang, Feiyu Xiong, Jie Hu, Mingchuan yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04283
• Pdf URL: https://arxiv.org/pdf/2403.04283
• Copy Paste: [[2403.04283]] Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy(https://arxiv.org/abs/2403.04283)
• Keywords: language model, llm
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1\% of the training parameters of other methods.
• 摘要：根据人类反馈进行强化学习 (RLHF) 是确保大型语言模型 (LLM) 符合人类价值观的主流方法。然而，现有的 RLHF 方法需要较高的计算成本，一个主要原因是 RLHF 将生成和比对任务同时分配给 LLM。在本文中，我们介绍了 Proxy-RLHF，它将 LLM 的生成和对齐过程解耦，以更低的计算成本实现与人类价值观的对齐。我们从专为对齐过程设计的新型马尔可夫决策过程 (MDP) 开始，并采用强化学习 (RL) 来训练简化的代理模型，该模型可监督 LLM 的代币生成，而无需更改 LLM 本身。实验表明，我们的方法仅用其他方法 1% 的训练参数就达到了相当的对齐水平。

Title: HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild

• Authors: Zhiying Zhu, Zhiqing Sun, Yiming Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04307
• Pdf URL: https://arxiv.org/pdf/2403.04307
• Copy Paste: [[2403.04307]] HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild(https://arxiv.org/abs/2403.04307)
• Keywords: language model, gpt, llm, hallucination, retrieval-augmented generation
• Abstract: Hallucinations pose a significant challenge to the reliability of large language models (LLMs) in critical domains. Recent benchmarks designed to assess LLM hallucinations within conventional NLP tasks, such as knowledge-intensive question answering (QA) and summarization, are insufficient for capturing the complexities of user-LLM interactions in dynamic, real-world settings. To address this gap, we introduce HaluEval-Wild, the first benchmark specifically designed to evaluate LLM hallucinations in the wild. We meticulously collect challenging (adversarially filtered by Alpaca) user queries from existing real-world user-LLM interaction datasets, including ShareGPT, to evaluate the hallucination rates of various LLMs. Upon analyzing the collected queries, we categorize them into five distinct types, which enables a fine-grained analysis of the types of hallucinations LLMs exhibit, and synthesize the reference answers with the powerful GPT-4 model and retrieval-augmented generation (RAG). Our benchmark offers a novel approach towards enhancing our comprehension and improvement of LLM reliability in scenarios reflective of real-world interactions.
• 摘要：幻觉对关键领域的大型语言模型 (LLM) 的可靠性提出了重大挑战。最近设计用于评估传统 NLP 任务（例如知识密集型问答 (QA) 和总结）中的 LLM 幻觉的基准不足以捕捉动态、现实环境中用户与 LLM 交互的复杂性。为了解决这一差距，我们推出了 HaluEval-Wild，这是第一个专门为评估野外 LLM 幻觉而设计的基准。我们从现有的现实世界用户与法学硕士交互数据集（包括 ShareGPT）中精心收集具有挑战性（由 Alpaca 进行对抗性过滤）的用户查询，以评估各种法学硕士的幻觉率。在分析收集到的查询后，我们将它们分为五种不同的类型，这使得能够对法学硕士表现出的幻觉类型进行细粒度分析，并利用强大的 GPT-4 模型和检索增强生成 (RAG) 综合参考答案。我们的基准提供了一种新颖的方法，可以增强我们对反映现实世界交互场景中法学硕士可靠性的理解和改进。

Title: Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders

• Authors: Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hang Su, Hwanjun Song, Saab Mansour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04314
• Pdf URL: https://arxiv.org/pdf/2403.04314
• Copy Paste: [[2403.04314]] Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders(https://arxiv.org/abs/2403.04314)
• Keywords: language model, llm, prompt
• Abstract: Conversational systems often rely on embedding models for intent classification and intent clustering tasks. The advent of Large Language Models (LLMs), which enable instructional embeddings allowing one to adjust semantics over the embedding space using prompts, are being viewed as a panacea for these downstream conversational tasks. However, traditional evaluation benchmarks rely solely on task metrics that don't particularly measure gaps related to semantic understanding. Thus, we propose an intent semantic toolkit that gives a more holistic view of intent embedding models by considering three tasks-- (1) intent classification, (2) intent clustering, and (3) a novel triplet task. The triplet task gauges the model's understanding of two semantic concepts paramount in real-world conversational systems-- negation and implicature. We observe that current embedding models fare poorly in semantic understanding of these concepts. To address this, we propose a pre-training approach to improve the embedding model by leveraging augmentation with data generated by an auto-regressive model and a contrastive loss term. Our approach improves the semantic understanding of the intent embedding model on the aforementioned linguistic dimensions while slightly effecting their performance on downstream task metrics.
• 摘要：会话系统通常依赖嵌入模型来执行意图分类和意图聚类任务。大型语言模型（LLM）的出现被视为这些下游对话任务的灵丹妙药，它支持指令嵌入，允许人们使用提示调整嵌入空间上的语义。然而，传统的评估基准仅依赖于任务指标，而这些指标并不特别衡量与语义理解相关的差距。因此，我们提出了一个意图语义工具包，通过考虑三个任务——（1）意图分类、（2）意图聚类和（3）新颖的三元组任务，可以更全面地了解意图嵌入模型。三元组任务衡量模型对现实世界会话系统中最重要的两个语义概念的理解——否定和暗示。我们观察到当前的嵌入模型在这些概念的语义理解方面表现不佳。为了解决这个问题，我们提出了一种预训练方法，通过利用自回归模型和对比损失项生成的数据进行增强来改进嵌入模型。我们的方法提高了对上述语言维度上的意图嵌入模型的语义理解，同时略微影响了它们在下游任务指标上的性能。

Title: Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models

• Authors: Changjiang Gao, Jixing Li, Jiajun Chen, Shujian Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04325
• Pdf URL: https://arxiv.org/pdf/2403.04325
• Copy Paste: [[2403.04325]] Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models(https://arxiv.org/abs/2403.04325)
• Keywords: language model
• Abstract: The process of meaning composition, wherein smaller units like morphemes or words combine to form the meaning of phrases and sentences, is essential for human sentence comprehension. Despite extensive neurolinguistic research into the brain regions involved in meaning composition, a computational metric to quantify the extent of composition is still lacking. Drawing on the key-value memory interpretation of transformer feed-forward network blocks, we introduce the Composition Score, a novel model-based metric designed to quantify the degree of meaning composition during sentence comprehension. Experimental findings show that this metric correlates with brain clusters associated with word frequency, structural processing, and general sensitivity to words, suggesting the multifaceted nature of meaning composition during human sentence comprehension.
• 摘要：意义构成的过程，即语素或单词等较小的单位组合起来形成短语和句子的含义，对于人类句子理解至关重要。尽管对参与意义构成的大脑区域进行了广泛的神经语言学研究，但仍然缺乏量化构成程度的计算指标。利用变压器前馈网络块的键值记忆解释，我们引入了组合分数，这是一种基于模型的新颖度量，旨在量化句子理解过程中的意义组合程度。实验结果表明，该指标与与词频、结构处理和对单词的一般敏感性相关的大脑簇相关，表明人类句子理解过程中意义构成的多方面性质。

Title: Computational Modelling of Plurality and Definiteness in Chinese Noun Phrases

• Authors: Yuqi Liu, Guanyi Chen, Kees van Deemter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04376
• Pdf URL: https://arxiv.org/pdf/2403.04376
• Copy Paste: [[2403.04376]] Computational Modelling of Plurality and Definiteness in Chinese Noun Phrases(https://arxiv.org/abs/2403.04376)
• Keywords: language model
• Abstract: Theoretical linguists have suggested that some languages (e.g., Chinese and Japanese) are "cooler" than other languages based on the observation that the intended meaning of phrases in these languages depends more on their contexts. As a result, many expressions in these languages are shortened, and their meaning is inferred from the context. In this paper, we focus on the omission of the plurality and definiteness markers in Chinese noun phrases (NPs) to investigate the predictability of their intended meaning given the contexts. To this end, we built a corpus of Chinese NPs, each of which is accompanied by its corresponding context, and by labels indicating its singularity/plurality and definiteness/indefiniteness. We carried out corpus assessments and analyses. The results suggest that Chinese speakers indeed drop plurality and definiteness markers very frequently. Building on the corpus, we train a bank of computational models using both classic machine learning models and state-of-the-art pre-trained language models to predict the plurality and definiteness of each NP. We report on the performance of these models and analyse their behaviours.
• 摘要：理论语言学家认为，某些语言（例如中文和日语）比其他语言“更酷”，因为他们观察到这些语言中短语的预期含义更多地取决于其上下文。因此，这些语言中的许多表达方式都被缩短了，并且它们的含义是从上下文中推断出来的。在本文中，我们重点关注中文名词短语（NP）中复数和确定性标记的省略，以研究在给定上下文的情况下其预期含义的可预测性。为此，我们建立了一个中文 NP 语料库，每个 NP 都附有其相应的上下文，以及指示其单数/复数和确定性/不确定性的标签。我们进行了语料库评估和分析。结果表明，说汉语的人确实经常丢弃复数和确定性标记。在语料库的基础上，我们使用经典的机器学习模型和最先进的预训练语言模型来训练一组计算模型来预测每个 NP 的复数性和确定性。我们报告这些模型的性能并分析它们的行为。

Title: Acceleron: A Tool to Accelerate Research Ideation

• Authors: Harshit Nigam, Manasi Patwardhan, Lovekesh Vig, Gautam Shroff
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04382
• Pdf URL: https://arxiv.org/pdf/2403.04382
• Copy Paste: [[2403.04382]] Acceleron: A Tool to Accelerate Research Ideation(https://arxiv.org/abs/2403.04382)
• Keywords: language model, llm, hallucination, agent
• Abstract: Several tools have recently been proposed for assisting researchers during various stages of the research life-cycle. However, these primarily concentrate on tasks such as retrieving and recommending relevant literature, reviewing and critiquing the draft, and writing of research manuscripts. Our investigation reveals a significant gap in availability of tools specifically designed to assist researchers during the challenging ideation phase of the research life-cycle. To aid with research ideation, we propose `Acceleron', a research accelerator for different phases of the research life cycle, and which is specially designed to aid the ideation process. Acceleron guides researchers through the formulation of a comprehensive research proposal, encompassing a novel research problem. The proposals motivation is validated for novelty by identifying gaps in the existing literature and suggesting a plausible list of techniques to solve the proposed problem. We leverage the reasoning and domain-specific skills of Large Language Models (LLMs) to create an agent-based architecture incorporating colleague and mentor personas for LLMs. The LLM agents emulate the ideation process undertaken by researchers, engaging researchers in an interactive fashion to aid in the development of the research proposal. Notably, our tool addresses challenges inherent in LLMs, such as hallucinations, implements a two-stage aspect-based retrieval to manage precision-recall trade-offs, and tackles issues of unanswerability. As evaluation, we illustrate the execution of our motivation validation and method synthesis workflows on proposals from the ML and NLP domain, given by 3 distinct researchers. Our observations and evaluations provided by the researchers illustrate the efficacy of the tool in terms of assisting researchers with appropriate inputs at distinct stages and thus leading to improved time efficiency.
• 摘要：最近提出了几种工具来在研究生命周期的各个阶段为研究人员提供帮助。但主要集中在相关文献的检索和推荐、草稿的审阅和批评、研究稿件的撰写等任务上。我们的调查显示，专门设计用于在研究生命周期充满挑战的构思阶段为研究人员提供帮助的工具的可用性存在显着差距。为了辅助研究构思，我们提出了“Acceleron”，这是一种针对研究生命周期不同阶段的研究加速器，专门为辅助构思过程而设计。 Acceleron 指导研究人员制定涵盖新研究问题的综合研究计划。通过识别现有文献中的差距并提出解决所提出问题的合理技术列表来验证提案动机的新颖性。我们利用大型语言模型 (LLM) 的推理和特定领域技能来创建一个基于代理的架构，其中包含 LLM 的同事和导师角色。法学硕士代理模仿研究人员进行的构思过程，以互动方式吸引研究人员来帮助研究提案的制定。值得注意的是，我们的工具解决了法学硕士固有的挑战，例如幻觉，实现了两阶段基于方面的检索来管理精确回忆权衡，并解决了无法回答的问题。作为评估，我们展示了对来自 ML 和 NLP 领域提案的动机验证和方法综合工作流程的执行情况，这些提案由 3 位不同的研究人员提供。研究人员提供的观察和评估说明了该工具的有效性，可以帮助研究人员在不同阶段提供适当的输入，从而提高时间效率。

Title: Promising and worth-to-try future directions for advancing state-of-the-art surrogates methods of agent-based models in social and health computational sciences

• Authors: Atiyah Elsheikh
• Subjects: cs.CL, cs.AI, eess.SY, math.DS
• Abstract URL: https://arxiv.org/abs/2403.04417
• Pdf URL: https://arxiv.org/pdf/2403.04417
• Copy Paste: [[2403.04417]] Promising and worth-to-try future directions for advancing state-of-the-art surrogates methods of agent-based models in social and health computational sciences(https://arxiv.org/abs/2403.04417)
• Keywords: agent
• Abstract: The execution and runtime performance of model-based analysis tools for realistic large-scale ABMs (Agent-Based Models) can be excessively long. This due to the computational demand exponentially proportional to the model size (e.g. Population size) and the number of model parameters. Even the runtime of a single simulation of a realistic ABM may demand huge computational resources when attempting to employ realistic population size. The main aim of this ad-hoc brief report is to highlight some of surrogate models that were adequate and computationally less demanding for nonlinear dynamical models in various modeling application areas.To the author knowledge, these methods have been not, at least extensively, employed for ABMs within the field of (SHCS) Social Health Computational Sciences, yet. Thus, they might be, but not necessarily, useful in progressing state of the art for establishing surrogate models for ABMs in the field of SHCS.
• 摘要：用于现实大规模 ABM（基于代理的模型）的基于模型的分析工具的执行和运行时性能可能会过长。这是因为计算需求与模型大小（例如群体大小）和模型参数的数量成指数比例。当尝试采用真实的人口规模时，即使是真实 ABM 的单次模拟的运行时间也可能需要大量的计算资源。这份临时简短报告的主要目的是强调一些替代模型，这些模型对于各种建模应用领域中的非线性动力学模型来说是足够的且计算要求较低。据作者所知，这些方法至少还没有被广泛采用对于（SHCS）社会健康计算科学领域内的 ABM，尚未。因此，它们可能但不一定有助于推动 SHCS 领域建立 ABM 替代模型的最新技术水平。

Title: Classist Tools: Social Class Correlates with Performance in NLP

• Authors: Amanda Cercas Curry, Giuseppe Attanasio, Zeerak Talat, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04445
• Pdf URL: https://arxiv.org/pdf/2403.04445
• Copy Paste: [[2403.04445]] Classist Tools: Social Class Correlates with Performance in NLP(https://arxiv.org/abs/2403.04445)
• Keywords: language model
• Abstract: Since the foundational work of William Labov on the social stratification of language (Labov, 1964), linguistics has made concentrated efforts to explore the links between sociodemographic characteristics and language production and perception. But while there is strong evidence for socio-demographic characteristics in language, they are infrequently used in Natural Language Processing (NLP). Age and gender are somewhat well represented, but Labov's original target, socioeconomic status, is noticeably absent. And yet it matters. We show empirically that NLP disadvantages less-privileged socioeconomic groups. We annotate a corpus of 95K utterances from movies with social class, ethnicity and geographical language variety and measure the performance of NLP systems on three tasks: language modelling, automatic speech recognition, and grammar error correction. We find significant performance disparities that can be attributed to socioeconomic status as well as ethnicity and geographical differences. With NLP technologies becoming ever more ubiquitous and quotidian, they must accommodate all language varieties to avoid disadvantaging already marginalised groups. We argue for the inclusion of socioeconomic class in future language technologies.
• 摘要：自从威廉·拉波夫（William Labov）关于语言的社会分层的基础性工作（Labov，1964）以来，语言学一直致力于探索社会人口特征与语言产生和感知之间的联系。但是，尽管有强有力的证据表明语言具有社会人口特征，但它们很少用于自然语言处理（NLP）。年龄和性别在某种程度上得到了很好的体现，但拉博夫最初的目标——社会经济地位——却明显缺失。但这很重要。我们的经验表明，NLP 对社会经济地位较低的群体不利。我们对包含社会阶层、种族和地理语言多样性的电影中的 95K 话语进行了语料库的注释，并测量了 NLP 系统在三个任务上的性能：语言建模、自动语音识别和语法错误纠正。我们发现显着的绩效差异可归因于社会经济地位以及种族和地域差异。随着 NLP 技术变得越来越普遍和日常化，它们必须适应所有语言品种，以避免使已经被边缘化的群体处于不利地位。我们主张将社会经济阶层纳入未来的语言技术中。

Title: Low-Resource Court Judgment Summarization for Common Law Systems

• Authors: Shuaiqi Liu, Jiannong Cao, Yicong Li, Ruosong Yang, Zhiyuan Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04454
• Pdf URL: https://arxiv.org/pdf/2403.04454
• Copy Paste: [[2403.04454]] Low-Resource Court Judgment Summarization for Common Law Systems(https://arxiv.org/abs/2403.04454)
• Keywords: language model, llm
• Abstract: Common law courts need to refer to similar precedents' judgments to inform their current decisions. Generating high-quality summaries of court judgment documents can facilitate legal practitioners to efficiently review previous cases and assist the general public in accessing how the courts operate and how the law is applied. Previous court judgment summarization research focuses on civil law or a particular jurisdiction's judgments. However, judges can refer to the judgments from all common law jurisdictions. Current summarization datasets are insufficient to satisfy the demands of summarizing precedents across multiple jurisdictions, especially when labeled data are scarce for many jurisdictions. To address the lack of datasets, we present CLSum, the first dataset for summarizing multi-jurisdictional common law court judgment documents. Besides, this is the first court judgment summarization work adopting large language models (LLMs) in data augmentation, summary generation, and evaluation. Specifically, we design an LLM-based data augmentation method incorporating legal knowledge. We also propose a legal knowledge enhanced evaluation metric based on LLM to assess the quality of generated judgment summaries. Our experimental results verify that the LLM-based summarization methods can perform well in the few-shot and zero-shot settings. Our LLM-based data augmentation method can mitigate the impact of low data resources. Furthermore, we carry out comprehensive comparative experiments to find essential model components and settings that are capable of enhancing summarization performance.
• 摘要：普通法法院需要参考类似先例的判决来为当前的判决提供信息。生成高质量的法院判决书摘要可以帮助法律从业者高效地审查过往案件，并帮助公众了解法院如何运作和如何适用法律。以往的法院判决摘要研究主要集中于民法或特定法域的判决。然而，法官可以参考所有普通法司法管辖区的判决。当前的汇总数据集不足以满足汇总多个司法管辖区的先例的需求，特别是当许多司法管辖区缺乏标记数据时。为了解决数据集缺乏的问题，我们提出了 CLSum，这是第一个用于总结多司法管辖区普通法法院判决文件的数据集。此外，这是第一个在数据增强、摘要生成和评估中采用大型语言模型（LLM）的法院判决摘要工作。具体来说，我们设计了一种结合法律知识的基于法学硕士的数据增强方法。我们还提出了一种基于LLM的法律知识增强评估指标，以评估生成的判决摘要的质量。我们的实验结果验证了基于 LLM 的摘要方法可以在少样本和零样本设置中表现良好。我们基于法学硕士的数据增强方法可以减轻数据资源不足的影响。此外，我们进行了全面的比较实验，以找到能够增强摘要性能的基本模型组件和设置。

Title: Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset

• Authors: Minjin Kim, Minju Kim, Hana Kim, Beong-woo Kwak, Soyeon Chun, Hyunseo Kim, SeongKu Kang, Youngjae Yu, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04460
• Pdf URL: https://arxiv.org/pdf/2403.04460
• Copy Paste: [[2403.04460]] Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset(https://arxiv.org/abs/2403.04460)
• Keywords: language model, llm
• Abstract: Conversational recommender system is an emerging area that has garnered an increasing interest in the community, especially with the advancements in large language models (LLMs) that enable diverse reasoning over conversational input. Despite the progress, the field has many aspects left to explore. The currently available public datasets for conversational recommendation lack specific user preferences and explanations for recommendations, hindering high-quality recommendations. To address such challenges, we present a novel conversational recommendation dataset named PEARL, synthesized with persona- and knowledge-augmented LLM simulators. We obtain detailed persona and knowledge from real-world reviews and construct a large-scale dataset with over 57k dialogues. Our experimental results demonstrate that utterances in PEARL include more specific user preferences, show expertise in the target domain, and provide recommendations more relevant to the dialogue context than those in prior datasets.
• 摘要：会话推荐系统是一个新兴领域，它引起了社区越来越多的兴趣，特别是随着大型语言模型（LLM）的进步，它可以对会话输入进行多样化的推理。尽管取得了进展，但该领域还有许多方面有待探索。目前用于会话推荐的公共数据集缺乏特定的用户偏好和推荐解释，阻碍了高质量的推荐。为了应对这些挑战，我们提出了一个名为 PEARL 的新型会话推荐数据集，该数据集由角色和知识增强的 LLM 模拟器综合而成。我们从现实世界的评论中获取详细的角色和知识，并构建一个包含超过 57k 对话的大规模数据集。我们的实验结果表明，PEARL 中的话语包含更具体的用户偏好，显示出目标领域的专业知识，并提供比先前数据集中的对话上下文更相关的建议。

Title: Do Large Language Model Understand Multi-Intent Spoken Language ?

• Authors: Shangjian Yin, Peijie Huang, Yuhong Xu, Haojing Huang, Jiatian Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04481
• Pdf URL: https://arxiv.org/pdf/2403.04481
• Copy Paste: [[2403.04481]] Do Large Language Model Understand Multi-Intent Spoken Language ?(https://arxiv.org/abs/2403.04481)
• Keywords: language model, llm
• Abstract: This study marks a significant advancement by harnessing Large Language Models (LLMs) for multi-intent spoken language understanding (SLU), proposing a unique methodology that capitalizes on the generative power of LLMs within an SLU context. Our innovative technique reconfigures entity slots specifically for LLM application in multi-intent SLU environments and introduces the concept of Sub-Intent Instruction (SII), enhancing the dissection and interpretation of intricate, multi-intent communication within varied domains. The resultant datasets, dubbed LM-MixATIS and LM-MixSNIPS, are crafted from pre-existing benchmarks. Our research illustrates that LLMs can match and potentially excel beyond the capabilities of current state-of-the-art multi-intent SLU models. It further explores LLM efficacy across various intent configurations and dataset proportions. Moreover, we introduce two pioneering metrics, Entity Slot Accuracy (ESA) and Combined Semantic Accuracy (CSA), to provide an in-depth analysis of LLM proficiency in this complex field.
• 摘要：这项研究标志着利用大型语言模型 (LLM) 进行多意图口语理解 (SLU) 的重大进步，提出了一种独特的方法，在 SLU 背景下利用 LLM 的生成能力。我们的创新技术专门针对多意图 SLU 环境中的 LLM 应用重新配置了实体槽，并引入了子意图指令 (SII) 的概念，增强了对不同领域内复杂的多意图通信的剖析和解释。由此产生的数据集被称为 LM-MixATIS 和 LM-MixSNIPS，是根据预先存在的基准精心制作的。我们的研究表明，法学硕士可以与当前最先进的多意图 SLU 模型相媲美，并有可能超越其能力。它进一步探讨了法学硕士在各种意图配置和数据集比例中的功效。此外，我们引入了两个开创性的指标：实体槽准确度（ESA）和组合语义准确度（CSA），以深入分析法学硕士在这个复杂领域的熟练程度。

Title: Where does In-context Translation Happen in Large Language Models

• Authors: Suzanna Sia, David Mueller, Kevin Duh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04510
• Pdf URL: https://arxiv.org/pdf/2403.04510
• Copy Paste: [[2403.04510]] Where does In-context Translation Happen in Large Language Models(https://arxiv.org/abs/2403.04510)
• Keywords: language model, gpt, prompt, chat
• Abstract: Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration examples. In this work, we attempt to characterize the region where large language models transition from in-context learners to translation models. Through a series of layer-wise context-masking experiments on \textsc{GPTNeo2.7B}, \textsc{Bloom3B}, \textsc{Llama7b} and \textsc{Llama7b-chat}, we demonstrate evidence of a "task recognition" point where the translation task is encoded into the input representations and attention to context is no longer necessary. We further observe correspondence between the low performance when masking out entire layers, and the task recognition layers. Taking advantage of this redundancy results in 45\% computational savings when prompting with 5 examples, and task recognition achieved at layer 14 / 32. Our layer-wise fine-tuning experiments indicate that the most effective layers for MT fine-tuning are the layers critical to task recognition.
• 摘要：自监督大型语言模型已经展示了通过上下文学习执行机器翻译 (MT) 的能力，但对于模型在提示指令和演示示例方面执行任务的位置知之甚少。在这项工作中，我们试图描述大型语言模型从上下文学习器过渡到翻译模型的区域。通过对 \textsc{GPTNeo2.7B}、\textsc{Bloom3B}、\textsc{Llama7b} 和 \textsc{Llama7b-chat} 进行一系列分层上下文屏蔽实验，我们证明了“任务识别”点的证据其中翻译任务被编码到输入表示中，并且不再需要注意上下文。我们进一步观察了屏蔽整个层时的低性能与任务识别层之间的对应关系。利用这种冗余，当用 5 个示例进行提示时，可以节省 45% 的计算量，并在第 14 / 32 层实现任务识别。我们的逐层微调实验表明，最有效的 MT 微调层是对任务识别至关重要。

Title: QAQ: Quality Adaptive Quantization for LLM KV Cache

• Authors: Shichen Dong, Wen Cheng, Jiayu Qin, Wei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04643
• Pdf URL: https://arxiv.org/pdf/2403.04643
• Copy Paste: [[2403.04643]] QAQ: Quality Adaptive Quantization for LLM KV Cache(https://arxiv.org/abs/2403.04643)
• Keywords: llm
• Abstract: The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress the KV cache and improve model throughput. However, heuristics used by these strategies may wrongly evict essential KV cache, which can significantly degrade model performance. In this paper, we propose QAQ, a Quality Adaptive Quantization scheme for the KV cache. We theoretically demonstrate that key cache and value cache exhibit distinct sensitivities to quantization, leading to the formulation of separate quantization strategies for their non-uniform quantization. Through the integration of dedicated outlier handling, as well as an improved attention-aware approach, QAQ achieves up to 10x the compression ratio of the KV cache size with a neglectable impact on model performance. QAQ significantly reduces the practical hurdles of deploying LLMs, opening up new possibilities for longer-context applications. The code is available at github.com/ClubieDong/KVCacheQuantization.
• 摘要：法学硕士的出现引发了 NLP 应用领域的新一轮突破，特别是在问答系统和文本生成等领域。随着对更长上下文的需求增长，由于键值（KV）缓存随上下文长度线性扩展，模型部署出现了重大瓶颈。现有的方法主要依赖于各种假设，例如根据替换或驱逐的注意力分数对 KV 缓存进行排序，以压缩 KV 缓存并提高模型吞吐量。然而，这些策略使用的启发式方法可能会错误地驱逐必要的 KV 缓存，从而显着降低模型性能。在本文中，我们提出了 QAQ，一种用于 KV 缓存的质量自适应量化方案。我们从理论上证明，键缓存和值缓存对量化表现出不同的敏感性，从而导致为其非均匀量化制定单独的量化策略。通过集成专用异常值处理以及改进的注意力感知方法，QAQ 实现了 KV 缓存大小高达 10 倍的压缩比，同时对模型性能的影响可以忽略不计。 QAQ 显着减少了部署法学硕士的实际障碍，为更长上下文的应用程序开辟了新的可能性。该代码可在 github.com/ClubieDong/KVCacheQuantization 获取。

Title: Yi: Open Foundation Models by 01.AI

• Authors: 01.AI: Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, Zonghong Dai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04652
• Pdf URL: https://arxiv.org/pdf/2403.04652
• Copy Paste: [[2403.04652]] Yi: Open Foundation Models by 01.AI(https://arxiv.org/abs/2403.04652)
• Keywords: language model, long context, chat
• Abstract: We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.
• 摘要：我们引入了 Yi 模型家族，这是一系列表现出强大的多维能力的语言和多模态模型。 Yi模型系列基于6B和34B预训练语言模型，然后我们将它们扩展到聊天模型、200K长上下文模型、深度升级模型和视觉语言模型。我们的基础模型在 MMLU 等各种基准测试中取得了优异的性能，而我们经过微调的聊天模型在 AlpacaEval 和 Chatbot Arena 等主要评估平台上提供了强大的人类偏好率。基于我们可扩展的超级计算基础设施和经典的 Transformer 架构，我们将 Yi 模型的性能主要归功于我们的数据工程工作所产生的数据质量。对于预训练，我们使用级联重复数据删除和质量过滤管道构建了 3.1 万亿个英语和中文语料库。为了进行微调，我们通过多次迭代完善了小规模（小于 10K）的指令数据集，以便我们的机器学习工程师直接验证每个实例。对于视觉语言，我们将聊天语言模型与视觉转换器编码器结合起来，并训练模型以将视觉表示与语言模型的语义空间对齐。我们通过轻量级持续预训练进一步将上下文长度扩展到 200K，并展示了强大的大海捞针检索性能。我们表明，通过持续预训练扩展预训练检查点的深度可以进一步提高性能。我们相信，鉴于我们目前的结果，继续使用彻底优化的数据扩大模型参数将导致更强大的前沿模型。

Title: Chain of Thought Explanation for Dialogue State Tracking

• Authors: Lin Xu, Ningxin Peng, Daquan Zhou, See-Kiong Ng, Jinlan Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04656
• Pdf URL: https://arxiv.org/pdf/2403.04656
• Copy Paste: [[2403.04656]] Chain of Thought Explanation for Dialogue State Tracking(https://arxiv.org/abs/2403.04656)
• Keywords: chain-of-thought
• Abstract: Dialogue state tracking (DST) aims to record user queries and goals during a conversational interaction achieved by maintaining a prede- fined set of slots and their corresponding values. Current approaches decide slot values opaquely, while humans usually adopt a more deliberate approach by collecting information from relevant dialogue turns and then reasoning the appropriate values. In this work, we focus on the steps needed to figure out slot values by proposing a model named Chain-of-Thought-Explanation (CoTE) for the DST task. CoTE, which is built on the generative DST framework, is designed to create detailed explanations step by step after determining the slot values. This process leads to more accurate and reliable slot values. More-over, to improve the reasoning ability of the CoTE, we further construct more fluent and high-quality explanations with automatic paraphrasing, leading the method CoTE-refined. Experimental results on three widely recognized DST benchmarks-MultiWOZ 2.2, WoZ 2.0, and M2M-demonstrate the remarkable effectiveness of the CoTE. Furthermore, through a meticulous fine-grained analysis, we observe significant benefits of our CoTE on samples characterized by longer dialogue turns, user responses, and reasoning steps.
• 摘要：对话状态跟踪（DST）旨在通过维护一组预定义的槽及其相应的值来记录对话交互期间的用户查询和目标。当前的方法不透明地决定槽值，而人类通常采用更审慎的方法，通过从相关对话轮次收集信息，然后推理出适当的值。在这项工作中，我们通过为 DST 任务提出一个名为 Chain-of-Thought-Explanation (CoTE) 的模型来重点关注计算槽值所需的步骤。 CoTE 建立在生成式 DST 框架之上，旨在在确定槽值后逐步创建详细解释。此过程可产生更准确、更可靠的槽值。此外，为了提高CoTE的推理能力，我们通过自动释义进一步构建更流畅、高质量的解释，引导CoTE方法细化。三个广泛认可的 DST 基准（MultiWOZ 2.2、WoZ 2.0 和 M2M）上的实验结果证明了 CoTE 的显着有效性。此外，通过细致的细粒度分析，我们观察到 CoTE 在具有较长对话轮次、用户响应和推理步骤的样本上具有显着优势。

Title: Telecom Language Models: Must They Be Large?

• Authors: Nicola Piovesan, Antonio De Domenico, Fadhel Ayed
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04666
• Pdf URL: https://arxiv.org/pdf/2403.04666
• Copy Paste: [[2403.04666]] Telecom Language Models: Must They Be Large?(https://arxiv.org/abs/2403.04666)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: The increasing interest in Large Language Models (LLMs) within the telecommunications sector underscores their potential to revolutionize operational efficiency. However, the deployment of these sophisticated models is often hampered by their substantial size and computational demands, raising concerns about their viability in resource-constrained environments. Addressing this challenge, recent advancements have seen the emergence of small language models that surprisingly exhibit performance comparable to their larger counterparts in many tasks, such as coding and common-sense reasoning. Phi-2, a compact yet powerful model, exemplifies this new wave of efficient small language models. This paper conducts a comprehensive evaluation of Phi-2's intrinsic understanding of the telecommunications domain. Recognizing the scale-related limitations, we enhance Phi-2's capabilities through a Retrieval-Augmented Generation approach, meticulously integrating an extensive knowledge base specifically curated with telecom standard specifications. The enhanced Phi-2 model demonstrates a profound improvement in accuracy, answering questions about telecom standards with a precision that closely rivals the more resource-intensive GPT-3.5. The paper further explores the refined capabilities of Phi-2 in addressing problem-solving scenarios within the telecom sector, highlighting its potential and limitations.
• 摘要：电信行业对大型语言模型 (LLM) 的兴趣日益浓厚，突显了它们彻底改变运营效率的潜力。然而，这些复杂模型的部署往往因其庞大的规模和计算需求而受到阻碍，引发了人们对其在资源有限环境中的可行性的担忧。为了应对这一挑战，最近的进展见证了小型语言模型的出现，这些模型在许多任务（例如编码和常识推理）中令人惊讶地表现出与大型语言模型相当的性能。 Phi-2 是一个紧凑而强大的模型，体现了这一新一波高效小语言模型。本文对Phi-2对电信领域的内在理解进行了全面的评估。认识到与规模相关的局限性，我们通过检索增强生成方法增强了 Phi-2 的功能，精心集成了专门根据电信标准规范策划的广泛知识库。增强的 Phi-2 模型在准确性方面有了显着的提高，回答有关电信标准的问题的精度可与资源密集型的 GPT-3.5 相媲美。本文进一步探讨了 Phi-2 在解决电信领域问题解决方案方面的改进能力，强调了其潜力和局限性。

Title: Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification

• Authors: Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04696
• Pdf URL: https://arxiv.org/pdf/2403.04696
• Copy Paste: [[2403.04696]] Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification(https://arxiv.org/abs/2403.04696)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. Such hallucinations can be dangerous, as occasional factual inaccuracies in the generated text might be obscured by the rest of the output being generally factual, making it extremely hard for the users to spot them. Current services that leverage LLMs usually do not provide any means for detecting unreliable generations. Here, we aim to bridge this gap. In particular, we propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification. Uncertainty scores leverage information encapsulated in the output of a neural network or its layers to detect unreliable predictions, and we show that they can be used to fact-check the atomic claims in the LLM output. Moreover, we present a novel token-level uncertainty quantification method that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. Our method Claim Conditioned Probability (CCP) measures only the uncertainty of particular claim value expressed by the model. Experiments on the task of biography generation demonstrate strong improvements for CCP compared to the baselines for six different LLMs and three languages. Human evaluation reveals that the fact-checking pipeline based on uncertainty quantification is competitive with a fact-checking tool that leverages external knowledge.
• 摘要：大型语言模型（LLM）因产生幻觉而臭名昭著，即在其输出中产生错误的声明。这种幻觉可能很危险，因为生成的文本中偶尔出现的事实错误可能会被通常真实的输出的其余部分所掩盖，从而使用户很难发现它们。当前利用法学硕士的服务通常不提供任何检测不可靠代的方法。在这里，我们的目标是弥合这一差距。特别是，我们提出了一种基于令牌级不确定性量化的新型事实检查和幻觉检测管道。不确定性分数利用封装在神经网络或其层的输出中的信息来检测不可靠的预测，并且我们表明它们可用于对法学硕士输出中的原子声明进行事实检查。此外，我们提出了一种新颖的令牌级不确定性量化方法，该方法消除了当前步骤生成什么声明以及使用什么表面形式的不确定性的影响。我们的方法索赔条件概率 (CCP) 仅测量模型所表达的特定索赔值的不确定性。传记生成任务的实验表明，与六种不同的法学硕士和三种语言的基线相比，CCP 有了显着的改进。人工评估表明，基于不确定性量化的事实检查流程与利用外部知识的事实检查工具具有竞争力。

Title: Common 7B Language Models Already Possess Strong Math Capabilities

• Authors: Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, Houwen Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04706
• Pdf URL: https://arxiv.org/pdf/2403.04706
• Copy Paste: [[2403.04706]] Common 7B Language Models Already Possess Strong Math Capabilities(https://arxiv.org/abs/2403.04706)
• Keywords: language model
• Abstract: Mathematical capabilities were previously believed to emerge in common language models only at a very large scale or require extensive math-related pre-training. This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities, as evidenced by its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks, respectively, when selecting the best response from 256 random generations. The primary issue with the current base model is the difficulty in consistently eliciting its inherent mathematical capabilities. Notably, the accuracy for the first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks, respectively. We find that simply scaling up the SFT data can significantly enhance the reliability of generating correct answers. However, the potential for extensive scaling is constrained by the scarcity of publicly available math questions. To overcome this limitation, we employ synthetic data, which proves to be nearly as effective as real data and shows no clear saturation when scaled up to approximately one million samples. This straightforward approach achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B models, surpassing previous models by 14.2% and 20.8%, respectively. We also provide insights into scaling behaviors across different reasoning complexities and error types.
• 摘要：此前人们认为，数学能力只能在非常大规模的通用语言模型中出现，或者需要大量与数学相关的预训练。本文表明，经过普通预训练的 LLaMA-2 7B 模型已经表现出了强大的数学能力，从 256 个随机数据中选择最佳响应时，其在 GSM8K 和 MATH 基准上分别达到 97.7% 和 72.0% 的令人印象深刻的准确率就证明了这一点。几代人。当前基本模型的主要问题是难以一致地得出其固有的数学能力。值得注意的是，第一个答案的准确度在 GSM8K 和 MATH 基准上分别下降至 49.5% 和 7.9%。我们发现，简单地扩展 SFT 数据就可以显着提高生成正确答案的可靠性。然而，广泛扩展的潜力受到公开数学问题稀缺的限制。为了克服这一限制，我们采用了合成数据，事实证明，合成数据几乎与真实数据一样有效，并且当扩展到大约一百万个样本时，没有显示出明显的饱和度。这种简单的方法使用 LLaMA-2 7B 模型在 GSM8K 上实现了 82.6% 的准确率，在 MATH 上实现了 40.6% 的准确率，分别超过了之前的模型 14.2% 和 20.8%。我们还提供了跨不同推理复杂性和错误类型的扩展行为的见解。

Title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

• Authors: Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04746
• Pdf URL: https://arxiv.org/pdf/2403.04746
• Copy Paste: [[2403.04746]] LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error(https://arxiv.org/abs/2403.04746)
• Keywords: language model, gpt, llm
• Abstract: Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.
• 摘要：工具对于大型语言模型 (LLM) 获取最新信息并在外部环境中采取相应行动至关重要。工具增强法学硕士的现有工作主要侧重于工具的广泛覆盖范围和添加新工具的灵活性。然而，令人惊讶的是，一个关键方面却没有得到足够的研究，那就是法学硕士如何准确地使用其接受过培训的工具。我们发现现有的 LLM，包括 GPT-4 和专门针对工具使用进行微调的开源 LLM，正确率仅达到 30% 至 60% 范围内，远未在实践中可靠使用。我们提出了一种受生物学启发的工具增强法学硕士方法，即模拟试错（STE），它协调了生物系统中成功使用工具行为的三个关键机制：试错、想象力和记忆。具体来说，STE 利用法学硕士的“想象力”来模拟使用工具的合理场景，之后法学硕士与该工具进行交互，从其执行反馈中学习。短期记忆和长期记忆分别用于提高探索的深度和广度。 ToolBench 上的综合实验表明，STE 在上下文学习和微调设置下都显着改善了 LLM 的工具学习，为 Mistral-Instruct-7B 带来 46.7% 的提升，使其性能超越 GPT-4。我们还通过简单的体验重放策略展示了对工具的有效持续学习。