2024-03-15

Title: From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation

• Authors: Claire Augusta Bergey, Simon DeDeo
• Subjects: cs.CL, cs.IT, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.08890
• Pdf URL: https://arxiv.org/pdf/2403.08890
• Copy Paste: [[2403.08890]] From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation(https://arxiv.org/abs/2403.08890)
• Keywords: language model
• Abstract: Conversation demands attention. Speakers must call words to mind, listeners must make sense of them, and both together must negotiate this flow of information, all in fractions of a second. We used large language models to study how this works in a large-scale dataset of English-language conversation, the CANDOR corpus. We provide a new estimate of the information density of unstructured conversation, of approximately 13 bits/second, and find significant effects associated with the cognitive load of both retrieving, and presenting, that information. We also reveal a role for backchannels -- the brief yeahs, uh-huhs, and mhmms that listeners provide -- in regulating the production of novelty: the lead-up to a backchannel is associated with declining information rate, while speech downstream rebounds to previous rates. Our results provide new insights into long-standing theories of how we respond to fluctuating demands on cognitive resources, and how we negotiate those demands in partnership with others.
• 摘要：谈话需要注意力。演讲者必须记住单词，听众必须理解它们，双方必须共同协商信息流，所有这些都在不到一秒的时间内完成。我们使用大型语言模型来研究它在大型英语对话数据集 CANDOR 语料库中的工作原理。我们对非结构化对话的信息密度进行了新的估计，约为 13 位/秒，并发现与检索和呈现该信息的认知负荷相关的显着影响。我们还揭示了反向渠道——听众提供的简短的“是”、“嗯嗯”和“嗯嗯”——在调节新奇事物的产生方面的作用：反向渠道的引导与信息率下降有关，而下游语音则反弹至以前的费率。我们的研究结果为我们如何应对认知资源波动的需求以及我们如何与他人合作协商这些需求的长期理论提供了新的见解。

Title: Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topics

• Authors: Tyler A. Chang, Katrin Tomanek, Jessica Hoffmann, Nithum Thain, Erin van Liemt, Kathleen Meier-Hellstern, Lucas Dixon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08904
• Pdf URL: https://arxiv.org/pdf/2403.08904
• Copy Paste: [[2403.08904]] Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topics(https://arxiv.org/abs/2403.08904)
• Keywords: llm, hallucination, chat, retrieval augmented generation
• Abstract: We explore a strategy to handle controversial topics in LLM-based chatbots based on Wikipedia's Neutral Point of View (NPOV) principle: acknowledge the absence of a single true answer and surface multiple perspectives. We frame this as retrieval augmented generation, where perspectives are retrieved from a knowledge base and the LLM is tasked with generating a fluent and faithful response from the given perspectives. As a starting point, we use a deterministic retrieval system and then focus on common LLM failure modes that arise during this approach to text generation, namely hallucination and coverage errors. We propose and evaluate three methods to detect such errors based on (1) word-overlap, (2) salience, and (3) LLM-based classifiers. Our results demonstrate that LLM-based classifiers, even when trained only on synthetic errors, achieve high error detection performance, with ROC AUC scores of 95.3% for hallucination and 90.5% for coverage error detection on unambiguous error cases. We show that when no training data is available, our other methods still yield good results on hallucination (84.0%) and coverage error (85.2%) detection.
• 摘要：我们基于维基百科的中立观点（NPOV）原则探索了一种处理基于 LLM 的聊天机器人中有争议主题的策略：承认不存在单一真实答案并提出多种观点。我们将其描述为检索增强生成，其中从知识库中检索观点，而法学硕士的任务是从给定的观点生成流畅且忠实的响应。作为起点，我们使用确定性检索系统，然后重点关注这种文本生成方法中出现的常见 LLM 失败模式，即幻觉和覆盖错误。我们提出并评估了三种基于（1）单词重叠、（2）显着性和（3）基于 LLM 的分类器来检测此类错误的方法。我们的结果表明，基于 LLM 的分类器，即使仅针对合成错误进行训练，也能实现较高的错误检测性能，在明确错误情况下，幻觉的 ROC AUC 分数为 95.3%，覆盖错误检测的 ROC AUC 分数为 90.5%。我们表明，当没有可用的训练数据时，我们的其他方法在幻觉（84.0％）和覆盖错误（85.2％）检测方面仍然产生良好的结果。

Title: LMStyle Benchmark: Evaluating Text Style Transfer for Chatbots

• Authors: Jianlin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08943
• Pdf URL: https://arxiv.org/pdf/2403.08943
• Copy Paste: [[2403.08943]] LMStyle Benchmark: Evaluating Text Style Transfer for Chatbots(https://arxiv.org/abs/2403.08943)
• Keywords: language model, gpt, llm, chat
• Abstract: Since the breakthrough of ChatGPT, large language models (LLMs) have garnered significant attention in the research community. With the development of LLMs, the question of text style transfer for conversational models has emerged as a natural extension, where chatbots may possess their own styles or even characters. However, standard evaluation metrics have not yet been established for this new settings. This paper aims to address this issue by proposing the LMStyle Benchmark, a novel evaluation framework applicable to chat-style text style transfer (C-TST), that can measure the quality of style transfer for LLMs in an automated and scalable manner. In addition to conventional style strength metrics, LMStyle Benchmark further considers a novel aspect of metrics called appropriateness, a high-level metrics take account of coherence, fluency and other implicit factors without the aid of reference samples. Our experiments demonstrate that the new evaluation methods introduced by LMStyle Benchmark have a higher correlation with human judgments in terms of appropriateness. Based on LMStyle Benchmark, we present a comprehensive list of evaluation results for popular LLMs, including LLaMA, Alpaca, and Vicuna, reflecting their stylistic properties, such as formality and sentiment strength, along with their appropriateness.
• 摘要：自从 ChatGPT 取得突破以来，大语言模型 (LLM) 引起了研究界的广泛关注。随着法学硕士的发展，对话模型的文本风格迁移问题已经成为自然的延伸，聊天机器人可能拥有自己的风格甚至性格。然而，尚未为这一新设置建立标准评估指标。本文旨在通过提出 LMStyle Benchmark 来解决这个问题，这是一种适用于聊天式文本风格迁移（C-TST）的新颖评估框架，可以以自动化和可扩展的方式衡量法学硕士的风格迁移质量。除了传统的风格强度指标之外，LMStyle Benchmark 还考虑了指标的一个新颖方面，即适当性，这是一种高级指标，无需参考样本的帮助即可考虑连贯性、流畅性和其他隐含因素。我们的实验表明，LMStyle Benchmark引入的新评估方法在适当性方面与人类判断具有更高的相关性。基于 LMStyle Benchmark，我们为流行的法学硕士（包括 LLaMA、Alpaca 和 Vicuna）提供了一份全面的评估结果列表，反映了它们的风格特征，例如形式和情感强度以及它们的适当性。

Title: AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents

• Authors: Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, Honglak Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08978
• Pdf URL: https://arxiv.org/pdf/2403.08978
• Copy Paste: [[2403.08978]] AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents(https://arxiv.org/abs/2403.08978)
• Keywords: language model, llm, agent
• Abstract: The primary limitation of large language models (LLMs) is their restricted understanding of the world. This poses significant difficulties for LLM-based agents, particularly in domains where pre-trained LLMs lack sufficient knowledge. In this paper, we introduce a novel framework, called AutoGuide, that bridges the knowledge gap in pre-trained LLMs by leveraging implicit knowledge in offline experiences. Specifically, AutoGuide effectively extracts knowledge embedded in offline data by extracting a set of state-aware guidelines. Importantly, each state-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the state where it is applicable. As such, the resulting guidelines enable a principled way to provide helpful knowledge pertinent to an agent's current decision-making process. We show that our approach outperforms competitive LLM-based baselines by a large margin in sequential decision-making benchmarks.
• 摘要：大型语言模型 (LLM) 的主要限制是它们对世界的理解有限。这给基于法学硕士的代理带来了巨大的困难，特别是在预训练的法学硕士缺乏足够知识的领域。在本文中，我们介绍了一种名为 AutoGuide 的新颖框架，该框架通过利用离线体验中的隐性知识来弥补预训练法学硕士的知识差距。具体来说，AutoGuide 通过提取一组状态感知指南来有效地提取嵌入离线数据中的知识。重要的是，每个状态感知指南都以简洁的自然语言表达，并遵循条件结构，清楚地描述了它适用的状态。因此，由此产生的指南能够以一种有原则的方式来提供与代理当前决策过程相关的有用知识。我们表明，我们的方法在连续决策基准中大幅优于基于 LLM 的竞争基线。

Title: Ethos: Rectifying Language Models in Orthogonal Parameter Space

• Authors: Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, Murali Annavaram
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08994
• Pdf URL: https://arxiv.org/pdf/2403.08994
• Copy Paste: [[2403.08994]] Ethos: Rectifying Language Models in Orthogonal Parameter Space(https://arxiv.org/abs/2403.08994)
• Keywords: language model
• Abstract: Language models (LMs) have greatly propelled the research on natural language processing. However, LMs also raise concerns regarding the generation of biased or toxic content and the potential disclosure of private information from the training dataset. In this work, we present a new efficient approach, Ethos, that rectifies LMs to mitigate toxicity and bias in outputs and avoid privacy leakage. Ethos is built on task arithmetic. However, unlike current task arithmetic algorithms, Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors. Specifically, Ethos first obtains a set of principal components from the pre-trained models using singular value decomposition. Then, by projecting the task vector onto principal components, Ethos identifies the principal components that encode general or undesired knowledge. Ethos performs negating using the task vector with undesired knowledge only, thereby minimizing collateral damage on general model utility. We demonstrate the efficacy of our approach on three different tasks: debiasing, detoxification, and memorization unlearning. Evaluations show Ethos is more effective in removing undesired knowledge and maintaining the overall model performance compared to current task arithmetic methods.
• 摘要：语言模型（LM）极大地推动了自然语言处理的研究。然而，LM 也对产生偏见或有毒内容以及训练数据集中私人信息的潜在泄露表示担忧。在这项工作中，我们提出了一种新的有效方法 Ethos，它可以纠正 LM，以减轻输出中的毒性和偏差，并避免隐私泄露。 Ethos 建立在任务算术的基础上。然而，与当前的任务算术算法不同，Ethos 在重建任务向量时区分一般有益和不需要的知识。具体来说，Ethos 首先使用奇异值分解从预训练模型中获取一组主成分。然后，通过将任务向量投影到主成分上，Ethos 识别出编码一般或不需要的知识的主成分。 Ethos 仅使用具有不需要的知识的任务向量来执行否定，从而最大限度地减少对一般模型效用的附带损害。我们展示了我们的方法在三个不同任务上的有效性：去偏见、解毒和记忆忘却。评估表明，与当前的任务算术方法相比，Ethos 在消除不需要的知识和保持整体模型性能方面更有效。

Title: AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

• Authors: Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, Mohamed Zaytoon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09017
• Pdf URL: https://arxiv.org/pdf/2403.09017
• Copy Paste: [[2403.09017]] AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic(https://arxiv.org/abs/2403.09017)
• Keywords: language model, gpt, llm, prompt
• Abstract: The swift progress and widespread acceptance of artificial intelligence (AI) systems highlight a pressing requirement to comprehend both the capabilities and potential risks associated with AI. Given the linguistic complexity, cultural richness, and underrepresented status of Arabic in AI research, there is a pressing need to focus on Large Language Models (LLMs) performance and safety for Arabic related tasks. Despite some progress in their development, there is a lack of comprehensive trustworthiness evaluation benchmarks which presents a major challenge in accurately assessing and improving the safety of LLMs when prompted in Arabic. In this paper, we introduce AraTrust 1, the first comprehensive trustworthiness benchmark for LLMs in Arabic. AraTrust comprises 516 human-written multiple-choice questions addressing diverse dimensions related to truthfulness, ethics, safety, physical health, mental health, unfairness, illegal activities, privacy, and offensive language. By introducing AraTrust, we aim to promote collaborative efforts to create safer and more trustworthy LLMs for Arabic users. We evaluated a set of LLMs against our benchmark to assess its trustworthiness. GPT-4 showed to be the most trustworthy regarding Arabic language.
• 摘要：人工智能（AI）系统的快速进步和广泛接受凸显了理解与人工智能相关的功能和潜在风险的迫切要求。鉴于阿拉伯语的语言复杂性、文化丰富性以及人工智能研究中代表性不足的地位，迫切需要关注阿拉伯语相关任务的大型语言模型 (LLM) 性能和安全性。尽管其发展取得了一些进展，但缺乏全面的可信度评估基准，这对准确评估和提高阿拉伯语提示的法学硕士的安全性提出了重大挑战。在本文中，我们介绍 AraTrust 1，这是第一个针对阿拉伯语法学硕士的综合可信度基准。 AraTrust 包含 516 个人工撰写的多项选择题，涉及真实性、道德、安全、身体健康、心理健康、不公平、非法活动、隐私和攻击性语言等不同维度。通过引入 AraTrust，我们的目标是促进合作，为阿拉伯语用户创建更安全、更值得信赖的法学硕士。我们根据我们的基准评估了一组法学硕士，以评估其可信度。 GPT-4 在阿拉伯语方面被证明是最值得信赖的。

Title: Semiparametric Token-Sequence Co-Supervision

• Authors: Hyunji Lee, Doyoung Kim, Jihoon Jun, Sejune Joo, Joel Jang, Kyoung-Woon On, Minjoon Seo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09024
• Pdf URL: https://arxiv.org/pdf/2403.09024
• Copy Paste: [[2403.09024]] Semiparametric Token-Sequence Co-Supervision(https://arxiv.org/abs/2403.09024)
• Keywords: language model
• Abstract: In this work, we introduce a semiparametric token-sequence co-supervision training method. It trains a language model by simultaneously leveraging supervision from the traditional next token prediction loss which is calculated over the parametric token embedding space and the next sequence prediction loss which is calculated over the nonparametric sequence embedding space. The nonparametric sequence embedding space is constructed by a separate language model tasked to condense an input text into a single representative embedding. Our experiments demonstrate that a model trained via both supervisions consistently surpasses models trained via each supervision independently. Analysis suggests that this co-supervision encourages a broader generalization capability across the model. Especially, the robustness of parametric token space which is established during the pretraining step tends to effectively enhance the stability of nonparametric sequence embedding space, a new space established by another language model.
• 摘要：在这项工作中，我们引入了一种半参数令牌序列共同监督训练方法。它通过同时利用传统的下一个标记预测损失（在参数标记嵌入空间上计算）和下一个序列预测损失（在非参数序列嵌入空间上计算）的监督来训练语言模型。非参数序列嵌入空间由单独的语言模型构建，该模型的任务是将输入文本压缩为单个代表性嵌入。我们的实验表明，通过两个监督训练的模型始终优于通过每个监督独立训练的模型。分析表明，这种共同监督鼓励整个模型具有更广泛的泛化能力。特别是，在预训练步骤中建立的参数标记空间的鲁棒性往往会有效增强非参数序列嵌入空间（由另一种语言模型建立的新空间）的稳定性。

Title: ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning

• Authors: Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09028
• Pdf URL: https://arxiv.org/pdf/2403.09028
• Copy Paste: [[2403.09028]] ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning(https://arxiv.org/abs/2403.09028)
• Keywords: llm
• Abstract: Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.
• 摘要：图表提供数据的可视化表示，广泛用于分析信息、解决查询以及向他人传达见解。最近出现了各种与图表相关的下游任务，例如问答和总结。解决这些任务的常见策略是微调最初在视觉任务语言上训练的各种模型。然而，此类特定于任务的模型无法解决广泛的与图表相关的任务，从而限制了它们的实际适用性。为了克服这些挑战，我们引入了 ChartInstruct：一种新颖的图表特定视觉语言指令跟踪数据集，包含由 71K 图表生成的 191K 指令。然后，我们提出了两种不同的系统，用于对此类数据集进行指令调整：（1）将用于图表理解的视觉编码器与法学硕士连接起来的端到端模型； (2) 管道模型，采用两步方法提取图表数据表并将其输入到法学硕士中。在四个下游任务的实验中，我们首先展示了我们模型的有效性——实现了一组新的最先进的结果。进一步的评估表明，我们的指令调整方法支持广泛的现实世界图表理解和推理场景，从而将我们的模型的范围和适用性扩展到新类型的任务。

Title: RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems

• Authors: Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09040
• Pdf URL: https://arxiv.org/pdf/2403.09040
• Copy Paste: [[2403.09040]] RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems(https://arxiv.org/abs/2403.09040)
• Keywords: language model, retrieval augmented generation, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) greatly benefits language models (LMs) by providing additional context for tasks such as document-based question answering (DBQA). Despite its potential, the power of RAG is highly dependent on its configuration, raising the question: What is the optimal RAG configuration? To answer this, we introduce the RAGGED framework to analyze and optimize RAG systems. On a set of representative DBQA tasks, we study two classic sparse and dense retrievers, and four top-performing LMs in encoder-decoder and decoder-only architectures. Through RAGGED, we uncover that different models suit substantially varied RAG setups. While encoder-decoder models monotonically improve with more documents, we find decoder-only models can only effectively use < 5 documents, despite often having a longer context window. RAGGED offers further insights into LMs' context utilization habits, where we find that encoder-decoder models rely more on contexts and are thus more sensitive to retrieval quality, while decoder-only models tend to rely on knowledge memorized during training.
• 摘要：检索增强生成 (RAG) 通过为基于文档的问答 (DBQA) 等任务提供额外的上下文，极大地有益于语言模型 (LM)。尽管具有潜力，但 RAG 的功能高度依赖于其配置，这就提出了一个问题：最佳的 RAG 配置是什么？为了回答这个问题，我们引入了 RAGGED 框架来分析和优化 RAG 系统。在一组代表性的 DBQA 任务中，我们研究了两种经典的稀疏和密集检索器，以及编码器-解码器和仅解码器架构中的四种表现最佳的 LM。通过 RAGGED，我们发现不同的模型适合不同的 RAG 设置。虽然编码器-解码器模型随着更多文档的增加而单调改进，但我们发现仅解码器模型只能有效地使用 < 5 个文档，尽管通常具有更长的上下文窗口。 RAGGED 提供了对 LM 上下文使用习惯的进一步见解，我们发现编码器-解码器模型更多地依赖于上下文，因此对检索质量更加敏感，而仅解码器模型往往依赖于训练期间记忆的知识。

Title: A Continued Pretrained LLM Approach for Automatic Medical Note Generation

• Authors: Dong Yuan, Eti Rastogi, Gautam Naik, Jai Chintagunta, Sree Prasanna Rajagopal, Fen Zhao, Sagar Goyal, Jeff Ward
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09057
• Pdf URL: https://arxiv.org/pdf/2403.09057
• Copy Paste: [[2403.09057]] A Continued Pretrained LLM Approach for Automatic Medical Note Generation(https://arxiv.org/abs/2403.09057)
• Keywords: gpt, llm
• Abstract: LLMs are revolutionizing NLP tasks. However, the most powerful LLM, like GPT-4, is too costly for most domain-specific scenarios. We present the first continuously trained 13B Llama2-based LLM that is purpose-built for medical conversations and measured on automated scribing. Our results show that our model outperforms GPT-4 in PubMedQA with 76.6\% accuracy and matches its performance in summarizing medical conversations into SOAP notes. Notably, our model exceeds GPT-4 in capturing a higher number of correct medical concepts and outperforms human scribes with higher correctness and completeness.
• 摘要：法学硕士正在彻底改变 NLP 任务。然而，最强大的 LLM（如 GPT-4）对于大多数特定领域的场景来说成本太高。我们推出第一个持续训练的基于 13B Llama2 的法学硕士，专为医学对话而设计，并通过自动划线进行测量。我们的结果表明，我们的模型在 PubMedQA 中的表现优于 GPT-4，准确率达到 76.6%，并且在将医学对话总结为 SOAP 笔记方面的表现与其相匹配。值得注意的是，我们的模型在捕获更多正确医学概念方面超过了 GPT-4，并在更高的正确性和完整性方面优于人类抄写员。

Title: LAMP: A Language Model on the Map

• Authors: Pasquale Balsebre, Weiming Huang, Gao Cong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09059
• Pdf URL: https://arxiv.org/pdf/2403.09059
• Copy Paste: [[2403.09059]] LAMP: A Language Model on the Map(https://arxiv.org/abs/2403.09059)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Large Language Models (LLMs) are poised to play an increasingly important role in our lives, providing assistance across a wide array of tasks. In the geospatial domain, LLMs have demonstrated the ability to answer generic questions, such as identifying a country's capital; nonetheless, their utility is hindered when it comes to answering fine-grained questions about specific places, such as grocery stores or restaurants, which constitute essential aspects of people's everyday lives. This is mainly because the places in our cities haven't been systematically fed into LLMs, so as to understand and memorize them. This study introduces a novel framework for fine-tuning a pre-trained model on city-specific data, to enable it to provide accurate recommendations, while minimizing hallucinations. We share our model, LAMP, and the data used to train it. We conduct experiments to analyze its ability to correctly retrieving spatial objects, and compare it to well-known open- and closed- source language models, such as GPT-4. Finally, we explore its emerging capabilities through a case study on day planning.
• 摘要：大型语言模型 (LLM) 有望在我们的生活中发挥越来越重要的作用，为各种任务提供帮助。在地理空间领域，法学硕士已经证明了回答一般问题的能力，例如识别一个国家的首都；然而，当涉及到回答有关特定地点（例如杂货店或餐馆）的细粒度问题时，它们的效用受到了阻碍，这些地点构成了人们日常生活的重要方面。这主要是因为我们城市的地方还没有系统地输入到LLM中，以便理解和记忆。这项研究引入了一种新颖的框架，用于根据特定城市的数据微调预训练模型，使其能够提供准确的建议，同时最大限度地减少幻觉。我们共享我们的模型 LAMP 以及用于训练它的数据。我们进行实验来分析其正确检索空间对象的能力，并将其与著名的开源和闭源语言模型（例如 GPT-4）进行比较。最后，我们通过有关日计划的案例研究探索其新兴功能。

Title: Large Language Models are Parallel Multilingual Learners

• Authors: Yongyu Mu, Peinan Feng, Zhiquan Cao, Yuzhang Wu, Bei Li, Chenglong Wang, Tong Xiao, Kai Song, Tongran Liu, Chunliang Zhang, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09073
• Pdf URL: https://arxiv.org/pdf/2403.09073
• Copy Paste: [[2403.09073]] Large Language Models are Parallel Multilingual Learners(https://arxiv.org/abs/2403.09073)
• Keywords: language model, llm
• Abstract: In this study, we reveal an in-context learning (ICL) capability of multilingual large language models (LLMs): by translating the input to several languages, we provide Parallel Input in Multiple Languages (PiM) to LLMs, which significantly enhances their comprehension abilities. To test this capability, we design extensive experiments encompassing 8 typical datasets, 7 languages and 8 state-of-the-art multilingual LLMs. Experimental results show that (1) incorporating more languages help PiM surpass the conventional ICL further; (2) even combining with the translations that are inferior to baseline performance can also help. Moreover, by examining the activated neurons in LLMs, we discover a counterintuitive but interesting phenomenon. Contrary to the common thought that PiM would activate more neurons than monolingual input to leverage knowledge learned from diverse languages, PiM actually inhibits neurons and promotes more precise neuron activation especially when more languages are added. This phenomenon aligns with the neuroscience insight about synaptic pruning, which removes less used neural connections, strengthens remainders, and then enhances brain intelligence.
• 摘要：在这项研究中，我们揭示了多语言大语言模型（LLM）的上下文学习（ICL）能力：通过将输入翻译成多种语言，我们为LLM提供多语言并行输入（PiM），这显着增强了他们的理解力能力。为了测试这种能力，我们设计了广泛的实验，涵盖 8 个典型数据集、7 种语言和 8 个最先进的多语言法学硕士。实验结果表明：（1）融入更多的语言有助于PiM进一步超越传统的ICL； (2) 即使与低于基线性能的翻译相结合也会有所帮助。此外，通过检查法学硕士中激活的神经元，我们发现了一个违反直觉但有趣的现象。人们普遍认为 PiM 会比单语言输入激活更多的神经元来利用从不同语言中学到的知识，但 PiM 实际上会抑制神经元并促进更精确的神经元激活，尤其是在添加更多语言时。这种现象与神经科学关于突触修剪的见解相一致，即删除较少使用的神经连接，加强剩余部分，然后增强大脑智力。

Title: Meaningful Learning: Advancing Abstract Reasoning in Large Language Models via Generic Fact Guidance

• Authors: Kai Xiong, Xiao Ding, Ting Liu, Bing Qin, Dongliang Xu, Qing Yang, Hongtao Liu, Yixin Cao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09085
• Pdf URL: https://arxiv.org/pdf/2403.09085
• Copy Paste: [[2403.09085]] Meaningful Learning: Advancing Abstract Reasoning in Large Language Models via Generic Fact Guidance(https://arxiv.org/abs/2403.09085)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have developed impressive performance and strong explainability across various reasoning scenarios, marking a significant stride towards mimicking human-like intelligence. Despite this, when tasked with simple questions supported by a generic fact, LLMs often fail to provide consistent and precise answers, indicating a deficiency in abstract reasoning abilities. This has sparked a vigorous debate about whether LLMs are genuinely reasoning or merely memorizing. In light of this, we design a preliminary study to quantify and delve into the abstract reasoning abilities of existing LLMs. Our findings reveal a substantial discrepancy between their general reasoning and abstract reasoning performances. To relieve this problem, we tailor an abstract reasoning dataset (AbsR) together with a meaningful learning paradigm to teach LLMs how to leverage generic facts for reasoning purposes. The results show that our approach not only boosts the general reasoning performance of LLMs but also makes considerable strides towards their capacity for abstract reasoning, moving beyond simple memorization or imitation to a more nuanced understanding and application of generic facts.
• 摘要：大型语言模型 (LLM) 在各种推理场景中都表现出了令人印象深刻的性能和强大的可解释性，标志着在模仿类人智能方面迈出了一大步。尽管如此，当处理由一般事实支持的简单问题时，法学硕士通常无法提供一致和精确的答案，这表明抽象推理能力的缺陷。这引发了关于法学硕士到底是真正的推理还是仅仅是记忆的激烈争论。有鉴于此，我们设计了一项初步研究来量化和深入研究现有法学硕士的抽象推理能力。我们的研究结果表明，他们的一般推理和抽象推理表现之间存在巨大差异。为了缓解这个问题，我们定制了一个抽象推理数据集（AbsR）和一个有意义的学习范式，以教导法学硕士如何利用通用事实进行推理。结果表明，我们的方法不仅提高了法学硕士的一般推理能力，而且还使他们的抽象推理能力取得了长足的进步，超越了简单的记忆或模仿，达到了对一般事实的更细致的理解和应用。

Title: AI on AI: Exploring the Utility of GPT as an Expert Annotator of AI Publications

• Authors: Autumn Toney-Wails, Christian Schoeberl, James Dunham
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09097
• Pdf URL: https://arxiv.org/pdf/2403.09097
• Copy Paste: [[2403.09097]] AI on AI: Exploring the Utility of GPT as an Expert Annotator of AI Publications(https://arxiv.org/abs/2403.09097)
• Keywords: language model, gpt, prompt, chat
• Abstract: Identifying scientific publications that are within a dynamic field of research often requires costly annotation by subject-matter experts. Resources like widely-accepted classification criteria or field taxonomies are unavailable for a domain like artificial intelligence (AI), which spans emerging topics and technologies. We address these challenges by inferring a functional definition of AI research from existing expert labels, and then evaluating state-of-the-art chatbot models on the task of expert data annotation. Using the arXiv publication database as ground-truth, we experiment with prompt engineering for GPT chatbot models to identify an alternative, automated expert annotation pipeline that assigns AI labels with 94% accuracy. For comparison, we fine-tune SPECTER, a transformer language model pre-trained on scientific publications, that achieves 96% accuracy (only 2% higher than GPT) on classifying AI publications. Our results indicate that with effective prompt engineering, chatbots can be used as reliable data annotators even where subject-area expertise is required. To evaluate the utility of chatbot-annotated datasets on downstream classification tasks, we train a new classifier on GPT-labeled data and compare its performance to the arXiv-trained model. The classifier trained on GPT-labeled data outperforms the arXiv-trained model by nine percentage points, achieving 82% accuracy.
• 摘要：识别动态研究领域内的科学出版物通常需要主题专家进行昂贵的注释。对于像人工智能 (AI) 这样涵盖新兴主题和技术的领域来说，广泛接受的分类标准或领域分类法等资源是不可用的。我们通过从现有专家标签推断人工智能研究的功能定义，然后评估专家数据注释任务的最先进的聊天机器人模型来应对这些挑战。使用 arXiv 出版物数据库作为基础事实，我们对 GPT 聊天机器人模型进行即时工程实验，以确定替代的自动化专家注释管道，该管道可以以 94% 的准确度分配 AI 标签。为了进行比较，我们对 SPECTRE 进行了微调，这是一种针对科学出版物进行预训练的 Transformer 语言模型，它在对 AI 出版物进行分类时达到了 96% 的准确率（仅比 GPT 高 2%）。我们的结果表明，通过有效的提示工程，即使在需要学科领域专业知识的情况下，聊天机器人也可以用作可靠的数据注释器。为了评估聊天机器人注释的数据集在下游分类任务中的效用，我们在 GPT 标记的数据上训练一个新的分类器，并将其性能与 arXiv 训练的模型进行比较。在 GPT 标记数据上训练的分类器比 arXiv 训练的模型高出 9 个百分点，达到 82% 的准确率。

Title: ProSwitch: Knowledge-Guided Language Model Fine-Tuning to Generate Professional and Non-Professional Styled Text

• Authors: Chang Zong, Yuyan Chen, Weiming Lu, Jian Shao, Yueting Zhuang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09131
• Pdf URL: https://arxiv.org/pdf/2403.09131
• Copy Paste: [[2403.09131]] ProSwitch: Knowledge-Guided Language Model Fine-Tuning to Generate Professional and Non-Professional Styled Text(https://arxiv.org/abs/2403.09131)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated efficacy in various linguistic applications, including text summarization and controlled text generation. However, studies into their capacity of switching between styles via fine-tuning remain underexplored. This study concentrates on textual professionalism and introduces a novel methodology, named ProSwitch, which equips a language model with the ability to produce both professional and non-professional responses through knowledge-guided instruction tuning. ProSwitch unfolds across three phases: data preparation for gathering domain knowledge and training corpus; instruction tuning for optimizing language models with multiple levels of instruction formats; and comprehensive evaluation for assessing the professionalism discrimination and reference-based quality of generated text. Comparative analysis of ProSwitch against both general and specialized language models reveals that our approach outperforms baselines in switching between professional and non-professional text generation.
• 摘要：大型语言模型 (LLM) 已在各种语言应用中展示了功效，包括文本摘要和受控文本生成。然而，对它们通过微调在风格之间切换的能力的研究仍未得到充分探索。这项研究专注于文本专业性，并引入了一种名为 ProSwitch 的新颖方法，该方法使语言模型能够通过知识引导的指令调整来产生专业和非专业响应。 ProSwitch 分为三个阶段：收集领域知识和训练语料库的数据准备；指令调优，用于优化具有多级指令格式的语言模型；综合评估，评估生成文本的专业歧视和基于参考的质量。 ProSwitch 与通用和专业语言模型的比较分析表明，我们的方法在专业和非专业文本生成之间的切换方面优于基线。

Title: Evaluating LLMs for Gender Disparities in Notable Persons

• Authors: Lauren Rhue, Sofie Goethals, Arun Sundararajan
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.09148
• Pdf URL: https://arxiv.org/pdf/2403.09148
• Copy Paste: [[2403.09148]] Evaluating LLMs for Gender Disparities in Notable Persons(https://arxiv.org/abs/2403.09148)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: This study examines the use of Large Language Models (LLMs) for retrieving factual information, addressing concerns over their propensity to produce factually incorrect "hallucinated" responses or to altogether decline to even answer prompt at all. Specifically, it investigates the presence of gender-based biases in LLMs' responses to factual inquiries. This paper takes a multi-pronged approach to evaluating GPT models by evaluating fairness across multiple dimensions of recall, hallucinations and declinations. Our findings reveal discernible gender disparities in the responses generated by GPT-3.5. While advancements in GPT-4 have led to improvements in performance, they have not fully eradicated these gender disparities, notably in instances where responses are declined. The study further explores the origins of these disparities by examining the influence of gender associations in prompts and the homogeneity in the responses.
• 摘要：这项研究考察了大型语言模型（LLM）在检索事实信息方面的用途，解决了人们对其倾向于产生事实上不正确的“幻觉”反应或完全拒绝回答提示的担忧。具体来说，它调查了法学硕士对事实调查的答复中是否存在基于性别的偏见。本文采用多管齐下的方法，通过评估回忆、幻觉和拒绝等多个维度的公平性来评估 GPT 模型。我们的研究结果揭示了 GPT-3.5 生成的响应中存在明显的性别差异。虽然 GPT-4 的进步带来了性能的提高，但它们并没有完全消除这些性别差异，特别是在回复率下降的情况下。该研究通过检查提示中性别关联的影响和反应的同质性，进一步探讨了这些差异的根源。

Title: Unveiling the Generalization Power of Fine-Tuned Large Language Models

• Authors: Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng Ann Heng, Wai Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09162
• Pdf URL: https://arxiv.org/pdf/2403.09162
• Copy Paste: [[2403.09162]] Unveiling the Generalization Power of Fine-Tuned Large Language Models(https://arxiv.org/abs/2403.09162)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, fine-tuning these models on downstream, domain-specific datasets is often necessary to yield superior performance on test sets compared to their counterparts without fine-tuning. However, the comprehensive effects of fine-tuning on the LLMs' generalization ability are not fully understood. This paper delves into the differences between original, unmodified LLMs and their fine-tuned variants. Our primary investigation centers on whether fine-tuning affects the generalization ability intrinsic to LLMs. To elaborate on this, we conduct extensive experiments across five distinct language tasks on various datasets. Our main findings reveal that models fine-tuned on generation and classification tasks exhibit dissimilar behaviors in generalizing to different domains and tasks. Intriguingly, we observe that integrating the in-context learning strategy during fine-tuning on generation tasks can enhance the model's generalization ability. Through this systematic investigation, we aim to contribute valuable insights into the evolving landscape of fine-tuning practices for LLMs.
• 摘要：虽然大型语言模型 (LLM) 已表现出卓越的多任务处理能力，但通常需要在下游、特定领域的数据集上对这些模型进行微调，以便与未经微调的模型相比，在测试集上产生卓越的性能。然而，微调对法学硕士泛化能力的综合影响尚不完全清楚。本文深入探讨了原始的、未经修改的法学硕士及其微调变体之间的差异。我们的主要研究集中在微调是否会影响法学硕士固有的泛化能力。为了详细说明这一点，我们在各种数据集上针对五种不同的语言任务进行了广泛的实验。我们的主要发现表明，在生成和分类任务上进行微调的模型在推广到不同领域和任务时表现出不同的行为。有趣的是，我们观察到，在生成任务的微调过程中整合上下文学习策略可以增强模型的泛化能力。通过这项系统性调查，我们的目标是为法学硕士微调实践的不断发展提供宝贵的见解。

Title: Caveat Lector: Large Language Models in Legal Practice

• Authors: Eliza Mik
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.09163
• Pdf URL: https://arxiv.org/pdf/2403.09163
• Copy Paste: [[2403.09163]] Caveat Lector: Large Language Models in Legal Practice(https://arxiv.org/abs/2403.09163)
• Keywords: language model, llm
• Abstract: The current fascination with large language models, or LLMs, derives from the fact that many users lack the expertise to evaluate the quality of the generated text. LLMs may therefore appear more capable than they actually are. The dangerous combination of fluency and superficial plausibility leads to the temptation to trust the generated text and creates the risk of overreliance. Who would not trust perfect legalese? Relying recent findings in both technical and legal scholarship, this Article counterbalances the overly optimistic predictions as to the role of LLMs in legal practice. Integrating LLMs into legal workstreams without a better comprehension of their limitations, will create inefficiencies if not outright risks. Notwithstanding their unprecedented ability to generate text, LLMs do not understand text. Without the ability to understand meaning, LLMs will remain unable to use language, to acquire knowledge and to perform complex reasoning tasks. Trained to model language on the basis of stochastic word predictions, LLMs cannot distinguish fact from fiction. Their knowledge of the law is limited to word strings memorized in their parameters. It is also incomplete and largely incorrect. LLMs operate at the level of word distributions, not at the level of verified facts. The resulting propensity to hallucinate, to produce statements that are incorrect but appear helpful and relevant, is alarming in high-risk areas like legal services. At present, lawyers should beware of relying on text generated by LLMs.
• 摘要：当前对大型语言模型（LLM）的着迷源于许多用户缺乏评估生成文本质量的专业知识。因此，法学硕士可能看起来比实际上更有能力。流畅性和表面合理性的危险结合会导致人们倾向于相信生成的文本，并产生过度依赖的风险。谁会不相信完美的法律术语？本文根据技术和法律学术领域的最新研究结果，平衡了对法学硕士在法律实践中的作用的过于乐观的预测。在没有更好地理解其局限性的情况下将法学硕士纳入法律工作流程，即使不是彻底的风险，也会造成效率低下。尽管法学硕士具有前所未有的生成文本的能力，但他们并不理解文本。如果没有理解意义的能力，法学硕士将仍然无法使用语言、获取知识和执行复杂的推理任务。法学硕士接受过基于随机单词预测的语言建模训练，无法区分事实与虚构。他们的法律知识仅限于参数中记忆的字符串。它也是不完整的并且很大程度上是不正确的。法学硕士在单词分布的层面上运作，而不是在已验证的事实层面上运作。由此产生的产生幻觉的倾向，即产生不正确但看似有用且相关的陈述的倾向，在法律服务等高风险领域令人震惊。目前，律师应谨防依赖法学硕士生成的文本。

Title: Exploring the Comprehension of ChatGPT in Traditional Chinese Medicine Knowledge

• Authors: Li Yizhen, Huang Shaohan, Qi Jiaxing, Quan Lei, Han Dongran, Luan Zhongzhi
• Subjects: cs.CL, stat.AP
• Abstract URL: https://arxiv.org/abs/2403.09164
• Pdf URL: https://arxiv.org/pdf/2403.09164
• Copy Paste: [[2403.09164]] Exploring the Comprehension of ChatGPT in Traditional Chinese Medicine Knowledge(https://arxiv.org/abs/2403.09164)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: No previous work has studied the performance of Large Language Models (LLMs) in the context of Traditional Chinese Medicine (TCM), an essential and distinct branch of medical knowledge with a rich history. To bridge this gap, we present a TCM question dataset named TCM-QA, which comprises three question types: single choice, multiple choice, and true or false, to examine the LLM's capacity for knowledge recall and comprehensive reasoning within the TCM domain. In our study, we evaluate two settings of the LLM, zero-shot and few-shot settings, while concurrently discussing the differences between English and Chinese prompts. Our results indicate that ChatGPT performs best in true or false questions, achieving the highest precision of 0.688 while scoring the lowest precision is 0.241 in multiple-choice questions. Furthermore, we observed that Chinese prompts outperformed English prompts in our evaluations. Additionally, we assess the quality of explanations generated by ChatGPT and their potential contribution to TCM knowledge comprehension. This paper offers valuable insights into the applicability of LLMs in specialized domains and paves the way for future research in leveraging these powerful models to advance TCM.
• 摘要：此前还没有研究过大型语言模型 (LLM) 在中医 (TCM) 背景下的表现，中医是医学知识的一个重要且独特的分支，具有丰富的历史。为了弥补这一差距，我们提出了一个名为 TCM-QA 的中医问题数据集，其中包括三种问题类型：单选、多选和对错题，以检验法学硕士在中医领域内的知识回忆和综合推理能力。在我们的研究中，我们评估了法学硕士的两种设置，零样本和少样本设置，同时讨论英语和中文提示之间的差异。我们的结果表明，ChatGPT 在对错题中表现最好，达到最高精确度 0.688，而在多项选择题中得分最低，精确度为 0.241。此外，我们观察到，在我们的评估中，中文提示优于英文提示。此外，我们还评估了 ChatGPT 生成的解释的质量及其对中医知识理解的潜在贡献。本文对法学硕士在专业领域的适用性提供了宝贵的见解，并为未来研究利用这些强大的模型推进中医发展铺平了道路。

Title: Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapse

• Authors: Jianwei Sun, Chaoyang Mei, Linlin Wei, Kaiyu Zheng, Na Liu, Ming Cui, Tianyi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09167
• Pdf URL: https://arxiv.org/pdf/2403.09167
• Copy Paste: [[2403.09167]] Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapse(https://arxiv.org/abs/2403.09167)
• Keywords: language model, llm, prompt
• Abstract: The efficacy of large language models (LLMs) is heavily dependent on the quality of the underlying data, particularly within specialized domains. A common challenge when fine-tuning LLMs for domain-specific applications is the potential degradation of the model's generalization capabilities. To address these issues, we propose a two-stage approach for the construction of production prompts designed to yield high-quality data. This method involves the generation of a diverse array of prompts that encompass a broad spectrum of tasks and exhibit a rich variety of expressions. Furthermore, we introduce a cost-effective, multi-dimensional quality assessment framework to ensure the integrity of the generated labeling data. Utilizing a dataset comprised of service provider and customer interactions from the real estate sector, we demonstrate a positive correlation between data quality and model performance. Notably, our findings indicate that the domain-specific proficiency of general LLMs can be enhanced through fine-tuning with data produced via our proposed method, without compromising their overall generalization abilities, even when exclusively domain-specific data is employed for fine-tuning.
• 摘要：大型语言模型 (LLM) 的功效在很大程度上取决于基础数据的质量，尤其是在专业领域内。针对特定领域的应用程序微调法学硕士时，一个常见的挑战是模型泛化能力的潜在下降。为了解决这些问题，我们提出了一种两阶段的方法来构建生产提示，旨在产生高质量的数据。该方法涉及生成各种提示，这些提示涵盖广泛的任务并表现出丰富的表达方式。此外，我们引入了一个具有成本效益的多维质量评估框架，以确保生成的标签数据的完整性。利用由房地产行业的服务提供商和客户交互组成的数据集，我们证明了数据质量和模型性能之间的正相关性。值得注意的是，我们的研究结果表明，即使仅使用特定领域的数据进行微调，也可以通过对我们提出的方法生成的数据进行微调来提高普通法学硕士的特定领域熟练程度，而不会影响其整体泛化能力。

Title: TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Sematic Tasks

• Authors: Viktor Moskvoretskii, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko, Irina Nikishina
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09207
• Pdf URL: https://arxiv.org/pdf/2403.09207
• Copy Paste: [[2403.09207]] TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Sematic Tasks(https://arxiv.org/abs/2403.09207)
• Keywords: llm
• Abstract: In this paper, we explore the capabilities of LLMs in capturing lexical-semantic knowledge from WordNet on the example of the LLaMA-2-7b model and test it on multiple lexical semantic tasks. As the outcome of our experiments, we present TaxoLLaMA, the everything-in-one model, lightweight due to 4-bit quantization and LoRA. It achieves 11 SotA results, 4 top-2 results out of 16 tasks for the Taxonomy Enrichment, Hypernym Discovery, Taxonomy Construction, and Lexical Entailment tasks. Moreover, it demonstrates very strong zero-shot performance on Lexical Entailment and Taxonomy Construction with no fine-tuning. We also explore its hidden multilingual and domain adaptation capabilities with a little tuning or few-shot learning. All datasets, code, and model are available online at https://github.com/VityaVitalich/TaxoLLaMA
• 摘要：在本文中，我们以 LLaMA-2-7b 模型为例，探讨了法学硕士从 WordNet 捕获词汇语义知识的能力，并在多个词汇语义任务上进行测试。作为我们实验的结果，我们推出了 TaxoLLaMA，这是一种全合一模型，由于 4 位量化和 LoRA，它是轻量级的。它在分类丰富、上位词发现、分类构建和词汇蕴涵任务的 16 个任务中获得了 11 个 SotA 结果、4 个前 2 个结果。此外，它在词汇蕴涵和分类构建上表现出了非常强大的零样本性能，无需进行微调。我们还通过一些调整或几次学习来探索其隐藏的多语言和领域适应能力。所有数据集、代码和模型均可在线获取：https://github.com/VityaVitalich/TaxoLLaMA

Title: Retrieval augmented text-to-SQL generation for epidemiological question answering using electronic health records

• Authors: Angelo Ziletti, Leonardo D'Ambrosi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09226
• Pdf URL: https://arxiv.org/pdf/2403.09226
• Copy Paste: [[2403.09226]] Retrieval augmented text-to-SQL generation for epidemiological question answering using electronic health records(https://arxiv.org/abs/2403.09226)
• Keywords: language model, prompt, retrieval augmented generation
• Abstract: Electronic health records (EHR) and claims data are rich sources of real-world data that reflect patient health status and healthcare utilization. Querying these databases to answer epidemiological questions is challenging due to the intricacy of medical terminology and the need for complex SQL queries. Here, we introduce an end-to-end methodology that combines text-to-SQL generation with retrieval augmented generation (RAG) to answer epidemiological questions using EHR and claims data. We show that our approach, which integrates a medical coding step into the text-to-SQL process, significantly improves the performance over simple prompting. Our findings indicate that although current language models are not yet sufficiently accurate for unsupervised use, RAG offers a promising direction for improving their capabilities, as shown in a realistic industry setting.
• 摘要：电子健康记录 (EHR) 和索赔数据是反映患者健康状况和医疗保健利用率的丰富的真实数据来源。由于医学术语的复杂性和复杂 SQL 查询的需要，查询这些数据库来回答流行病学问题具有挑战性。在这里，我们介绍了一种端到端方法，该方法将文本到 SQL 生成与检索增强生成 (RAG) 相结合，以使用 EHR 和索赔数据回答流行病学问题。我们表明，我们的方法将医疗编码步骤集成到文本到 SQL 的过程中，与简单的提示相比，显着提高了性能。我们的研究结果表明，尽管当前的语言模型对于无人监督的使用还不够准确，但 RAG 为提高其能力提供了一个有希望的方向，正如现实行业环境中所显示的那样。

Title: Komodo: A Linguistic Expedition into Indonesia's Regional Languages

• Authors: Louis Owen, Vishesh Tripathi, Abhay Kumar, Biddwan Ahmed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09362
• Pdf URL: https://arxiv.org/pdf/2403.09362
• Copy Paste: [[2403.09362]] Komodo: A Linguistic Expedition into Indonesia's Regional Languages(https://arxiv.org/abs/2403.09362)
• Keywords: language model, gpt, llm, chat
• Abstract: The recent breakthroughs in Large Language Models (LLMs) have mostly focused on languages with easily available and sufficient resources, such as English. However, there remains a significant gap for languages that lack sufficient linguistic resources in the public domain. Our work introduces Komodo-7B, 7-billion-parameter Large Language Models designed to address this gap by seamlessly operating across Indonesian, English, and 11 regional languages in Indonesia. Komodo-7B is a family of LLMs that consist of Komodo-7B-Base and Komodo-7B-Instruct. Komodo-7B-Instruct stands out by achieving state-of-the-art performance in various tasks and languages, outperforming the benchmarks set by OpenAI's GPT-3.5, Cohere's Aya-101, Llama-2-Chat-13B, Mixtral-8x7B-Instruct-v0.1, Gemma-7B-it , and many more. This model not only demonstrates superior performance in both language-specific and overall assessments but also highlights its capability to excel in linguistic diversity. Our commitment to advancing language models extends beyond well-resourced languages, aiming to bridge the gap for those with limited linguistic assets. Additionally, Komodo-7B-Instruct's better cross-language understanding contributes to addressing educational disparities in Indonesia, offering direct translations from English to 11 regional languages, a significant improvement compared to existing language translation services. Komodo-7B represents a crucial step towards inclusivity and effectiveness in language models, providing to the linguistic needs of diverse communities.
• 摘要：大型语言模型（LLM）最近的突破主要集中在资源容易获得且资源充足的语言上，例如英语。然而，公共领域缺乏足够语言资源的语言仍然存在巨大差距。我们的工作引入了 Komodo-7B，这是一个拥有 70 亿参数的大型语言模型，旨在通过跨印度尼西亚语、英语和印度尼西亚 11 种地方语言无缝运行来解决这一差距。 Komodo-7B 是一个法学硕士系列，由 Komodo-7B-Base 和 Komodo-7B-Instruct 组成。 Komodo-7B-Instruct 因在各种任务和语言中实现最先进的性能而脱颖而出，超越了 OpenAI 的 GPT-3.5、Cohere 的 Aya-101、Llama-2-Chat-13B、Mixtral-8x7B 设定的基准- Instruct-v0.1、Gemma-7B-it 等等。该模型不仅在特定语言和整体评估方面表现出卓越的性能，而且还凸显了其在语言多样性方面的卓越能力。我们对推进语言模型的承诺不仅限于资源丰富的语言，旨在弥合语言资源有限的语言之间的差距。此外，Komodo-7B-Instruct 更好的跨语言理解有助于解决印度尼西亚的教育差异，提供从英语到 11 种地区语言的直接翻译，与现有的语言翻译服务相比，这是一个显着的改进。 Komodo-7B 代表了语言模型朝着包容性和有效性迈出的关键一步，满足了不同社区的语言需求。

Title: Rectifying Demonstration Shortcut in In-Context Learning

• Authors: Joonwon Jang, Sanghwan Jang, Wonbin Kweon, Minjin Jeon, Hwanjo Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09488
• Pdf URL: https://arxiv.org/pdf/2403.09488
• Copy Paste: [[2403.09488]] Rectifying Demonstration Shortcut in In-Context Learning(https://arxiv.org/abs/2403.09488)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) are able to solve various tasks with only a few demonstrations utilizing their in-context learning (ICL) abilities. However, LLMs often rely on their pre-trained semantic priors of demonstrations rather than on the input-label relationships to proceed with ICL prediction. In this work, we term this phenomenon as the `Demonstration Shortcut'. While previous works have primarily focused on improving ICL prediction results for predefined tasks, we aim to rectify the Demonstration Shortcut, thereby enabling the LLM to effectively learn new input-label relationships from demonstrations. To achieve this, we introduce In-Context Calibration, a demonstration-aware calibration method. We evaluate the effectiveness of the proposed method in two settings: (1) the Original ICL Task using the standard label space and (2) the Task Learning setting, where the label space is replaced with semantically unrelated tokens. In both settings, In-Context Calibration demonstrates substantial improvements, with results generalized across three LLM families (OPT, GPT, and Llama2) under various configurations.
• 摘要：大型语言模型 (LLM) 只需进行几次演示即可利用其上下文学习 (ICL) 能力来解决各种任务。然而，法学硕士通常依赖于其预先训练的演示语义先验，而不是依赖于输入标签关系来进行 ICL 预测。在这项工作中，我们将这种现象称为“演示捷径”。虽然之前的工作主要集中在改进预定义任务的 ICL 预测结果，但我们的目标是纠正演示快捷方式，从而使法学硕士能够从演示中有效地学习新的输入标签关系。为了实现这一目标，我们引入了上下文校准，这是一种演示感知校准方法。我们在两种设置中评估所提出方法的有效性：（1）使用标准标签空间的原始 ICL 任务和（2）任务学习设置，其中标签空间被语义上不相关的标记替换。在这两种设置中，上下文校准都展示了显着的改进，其结果在不同配置下的三个 LLM 系列（OPT、GPT 和 Llama2）中得到推广。

Title: Leveraging Prototypical Representations for Mitigating Social Bias without Demographic Information

• Authors: Shadi Iskander, Kira Radinsky, Yonatan Belinkov
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09516
• Pdf URL: https://arxiv.org/pdf/2403.09516
• Copy Paste: [[2403.09516]] Leveraging Prototypical Representations for Mitigating Social Bias without Demographic Information(https://arxiv.org/abs/2403.09516)
• Keywords: language model
• Abstract: Mitigating social biases typically requires identifying the social groups associated with each data sample. In this paper, we present DAFair, a novel approach to address social bias in language models. Unlike traditional methods that rely on explicit demographic labels, our approach does not require any such information. Instead, we leverage predefined prototypical demographic texts and incorporate a regularization term during the fine-tuning process to mitigate bias in the model's representations. Our empirical results across two tasks and two models demonstrate the effectiveness of our method compared to previous approaches that do not rely on labeled data. Moreover, with limited demographic-annotated data, our approach outperforms common debiasing approaches.
• 摘要：减轻社会偏见通常需要识别与每个数据样本相关的社会群体。在本文中，我们提出了 DAFair，一种解决语言模型中社会偏见的新方法。与依赖明确的人口统计标签的传统方法不同，我们的方法不需要任何此类信息。相反，我们利用预定义的原型人口统计文本，并在微调过程中纳入正则化项，以减轻模型表示中的偏差。我们在两个任务和两个模型中的实证结果证明了我们的方法与以前不依赖标记数据的方法相比的有效性。此外，由于人口统计注释数据有限，我们的方法优于常见的去偏方法。

Title: MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation

• Authors: Jiahuan Li, Shanbo Cheng, Shujian Huang, Jiajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09522
• Pdf URL: https://arxiv.org/pdf/2403.09522
• Copy Paste: [[2403.09522]] MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation(https://arxiv.org/abs/2403.09522)
• Keywords: language model, llm
• Abstract: Large Language Models (LLM) have demonstrated their strong ability in the field of machine translation (MT), yet they suffer from high computational cost and latency. Therefore, transferring translation knowledge from giant LLMs to medium-sized machine translation models is a promising research direction. However, traditional knowledge distillation methods do not take the capability of student and teacher models into consideration, therefore repeatedly teaching student models on the knowledge they have learned, and failing to extend to novel contexts and knowledge. In this paper, we propose a framework called MT-Patcher, which transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. Considering the current translation ability of student MT models, we only identify and correct their translation errors, instead of distilling the whole translation from the teacher. Leveraging the strong language abilities of LLMs, we instruct LLM teachers to synthesize diverse contexts and anticipate more potential errors for the student. Experiment results on translating both specific language phenomena and general MT benchmarks demonstrate that finetuning the student MT model on about 10% examples can achieve comparable results to the traditional knowledge distillation method, and synthesized potential errors and diverse contexts further improve translation performances on unseen contexts and words.
• 摘要：大型语言模型（LLM）在机器翻译（MT）领域展现了强大的能力，但其计算成本和延迟较高。因此，将巨型法学硕士的翻译知识转移到中型机器翻译模型是一个有前途的研究方向。然而，传统的知识蒸馏方法没有考虑学生模型和教师模型的能力，因此反复教授学生模型所学的知识，而无法扩展到新的情境和知识。在本文中，我们提出了一个名为 MT-Patcher 的框架，该框架以选择性、全面和主动的方式将法学硕士的知识转移到现有的 MT 模型。考虑到学生机器翻译模型目前的翻译能力，我们只是识别并纠正他们的翻译错误，而不是从老师那里提取整个翻译。利用法学硕士强大的语言能力，我们指导法学硕士教师综合不同的背景并预测学生更多的潜在错误。翻译特定语言现象和通用 MT 基准的实验结果表明，在大约 10% 的示例上微调学生 MT 模型可以达到与传统知识蒸馏方法相当的结果，并且综合了潜在错误和不同的上下文，进一步提高了在未见过的上下文和情况下的翻译性能。字。

Title: Logits of API-Protected LLMs Leak Proprietary Information

• Authors: Matthew Finlayson, Swabha Swayamdipta, Xiang Ren
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09539
• Pdf URL: https://arxiv.org/pdf/2403.09539
• Copy Paste: [[2403.09539]] Logits of API-Protected LLMs Leak Proprietary Information(https://arxiv.org/abs/2403.09539)
• Keywords: language model, gpt, llm
• Abstract: The commercialization of large language models (LLMs) has led to the common practice of high-level API-only access to proprietary models. In this work, we show that even with a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under $1,000 for OpenAI's gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We show that this lends itself to a model image or a model signature which unlocks several capabilities with affordable cost: efficiently discovering the LLM's hidden size, obtaining full-vocabulary outputs, detecting and disambiguating different model updates, identifying the source LLM given a single full LLM output, and even estimating the output layer parameters. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI's gpt-3.5-turbo to be about 4,096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.
• 摘要：大型语言模型 (LLM) 的商业化导致了仅通过高级 API 访问专有模型的常见做法。在这项工作中，我们表明，即使对模型架构采取保守假设，也可以从相对少量的 API 查询中了解有关受 API 保护的 LLM 的大量非公开信息（例如，成本计算在OpenAI 的 gpt-3.5-turbo 售价 1,000 美元）。我们的研究结果集中在一个关键的观察上：大多数现代法学硕士都存在 softmax 瓶颈，该瓶颈将模型输出限制在整个输出空间的线性子空间。我们证明，这有助于模型图像或模型签名，以可承受的成本解锁多种功能：有效发现 LLM 的隐藏大小、获取完整词汇输出、检测和消除不同模型更新的歧义、在给定单个完整模型的情况下识别源 LLM LLM输出，甚至估计输出层参数。我们的实证研究表明了我们方法的有效性，这使我们能够估计 OpenAI 的 gpt-3.5-turbo 的嵌入大小约为 4,096。最后，我们讨论了 LLM 提供商防范这些攻击的方法，以及如何通过提高透明度和问责制将这些功能视为一项功能（而不是错误）。

Title: Less is More: Data Value Estimation for Visual Instruction Tuning

• Authors: Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-Rong Wen
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09559
• Pdf URL: https://arxiv.org/pdf/2403.09559
• Copy Paste: [[2403.09559]] Less is More: Data Value Estimation for Visual Instruction Tuning(https://arxiv.org/abs/2403.09559)
• Keywords: language model, llm
• Abstract: Visual instruction tuning is the key to building multimodal large language models (MLLMs), which greatly improves the reasoning capabilities of large language models (LLMs) in vision scenario. However, existing MLLMs mostly rely on a mixture of multiple highly diverse visual instruction datasets for training (even more than a million instructions), which may introduce data redundancy. To investigate this issue, we conduct a series of empirical studies, which reveal a significant redundancy within the visual instruction datasets, and show that greatly reducing the amount of several instruction dataset even do not affect the performance. Based on the findings, we propose a new data selection approach TIVE, to eliminate redundancy within visual instruction data. TIVE first estimates the task-level and instance-level value of the visual instructions based on computed gradients. Then, according to the estimated values, TIVE determines the task proportion within the visual instructions, and selects representative instances to compose a smaller visual instruction subset for training. Experiments on LLaVA-1.5 show that our approach using only about 7.5% data can achieve comparable performance as the full-data fine-tuned model across seven benchmarks, even surpassing it on four of the benchmarks. Our code and data will be publicly released.
• 摘要：视觉指令调优是构建多模态大语言模型（MLLM）的关键，它极大地提高了大语言模型（LLM）在视觉场景下的推理能力。然而，现有的MLLM大多依赖于多个高度多样化的视觉指令数据集的混合来进行训练（甚至超过一百万条指令），这可能会引入数据冗余。为了研究这个问题，我们进行了一系列的实证研究，这些研究揭示了视觉指令数据集中存在显着的冗余，并表明大大减少多个指令数据集的数量甚至不会影响性能。基于这些发现，我们提出了一种新的数据选择方法 TIVE，以消除视觉指令数据中的冗余。 TIVE 首先根据计算的梯度估计视觉指令的任务级和实例级值。然后，TIVE根据估计值确定视觉指令中的任务比例，并选择代表性实例组成更小的视觉指令子集进行训练。 LLaVA-1.5 上的实验表明，我们的方法仅使用约 7.5% 的数据就可以在七个基准上实现与全数据微调模型相当的性能，甚至在四个基准上超过它。我们的代码和数据将公开发布。

Title: Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey

• Authors: Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, Julian McAuley, Wei Ai, Furong Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09606
• Pdf URL: https://arxiv.org/pdf/2403.09606
• Copy Paste: [[2403.09606]] Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey(https://arxiv.org/abs/2403.09606)
• Keywords: language model, llm
• Abstract: Causal inference has shown potential in enhancing the predictive accuracy, fairness, robustness, and explainability of Natural Language Processing (NLP) models by capturing causal relationships among variables. The emergence of generative Large Language Models (LLMs) has significantly impacted various NLP domains, particularly through their advanced reasoning capabilities. This survey focuses on evaluating and improving LLMs from a causal view in the following areas: understanding and improving the LLMs' reasoning capacity, addressing fairness and safety issues in LLMs, complementing LLMs with explanations, and handling multimodality. Meanwhile, LLMs' strong reasoning capacities can in turn contribute to the field of causal inference by aiding causal relationship discovery and causal effect estimations. This review explores the interplay between causal inference frameworks and LLMs from both perspectives, emphasizing their collective potential to further the development of more advanced and equitable artificial intelligence systems.
• 摘要：因果推理通过捕获变量之间的因果关系，显示出提高自然语言处理 (NLP) 模型的预测准确性、公平性、鲁棒性和可解释性的潜力。生成式大型语言模型 (LLM) 的出现对各个 NLP 领域产生了重大影响，特别是通过其先进的推理能力。这项调查的重点是从因果角度评估和改进法学硕士在以下领域：理解和提高法学硕士的推理能力，解决法学硕士的公平和安全问题，用解释补充法学硕士，以及处理多模态。同时，法学硕士强大的推理能力可以通过帮助因果关系发现和因果效应估计来为因果推理领域做出贡献。本综述从两个角度探讨了因果推理框架和法学硕士之间的相互作用，强调它们在进一步发展更先进和更公平的人工智能系统方面的集体潜力。

Title: Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

• Authors: Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, Noah D. Goodman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09629
• Pdf URL: https://arxiv.org/pdf/2403.09629
• Copy Paste: [[2403.09629]] Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking(https://arxiv.org/abs/2403.09629)
• Keywords: language model, agent
• Abstract: When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%$\rightarrow$10.9%) and CommonsenseQA (36.3%$\rightarrow$47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.
• 摘要：在写作和说话时，人们有时会停下来思考。尽管以推理为中心的作品通常将推理视为回答问题或完成代理任务的方法，但推理几乎隐含在所有书面文本中。例如，这适用于证明的字里行间未说明的步骤或对话背后的心理理论。在自学推理机（STaR，Zelikman 等人，2022）中，有用的思维是通过从问答中的少数例子中推断基本原理并从那些导致正确答案的例子中学习来学习的。这是一个高度受限的环境——理想情况下，语言模型可以学习推断任意文本中未阐明的基本原理。我们提出了 Quiet-STaR，这是 STaR 的推广，其中 LM 学习为每个标记生成基本原理来解释未来的文本，从而改进他们的预测。我们解决了关键挑战，包括 1) 生成延续的计算成本，2) LM 最初不知道如何生成或使用内部思想，3) 需要预测超出单个下一个标记的需求。为了解决这些问题，我们提出了一种标记并行采样算法，使用可学习的标记来指示思想的开始和结束，以及扩展的教师强制技术。令人鼓舞的是，生成的理由很大程度上有助于对难以预测的标记进行建模，并提高 LM 直接回答难题的能力。特别是，在使用 Quiet-STaR 对互联网文本语料库继续对 LM 进行预训练后，我们发现 GSM8K (5.9%$\rightarrow$10.9%) 和 CommonsenseQA (36.3%$\rightarrow$47.2%) 上有零样本改进，并观察到自然文本中困难标记的困惑度改进。至关重要的是，这些改进不需要对这些任务进行微调。 Quiet-STARaR 标志着 LM 迈出了一步，可以以更通用和可扩展的方式学习推理。

Title: Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models

• Authors: Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia, Jungho Jung, Harshith Goka, Haejun Lee
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09635
• Pdf URL: https://arxiv.org/pdf/2403.09635
• Copy Paste: [[2403.09635]] Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models(https://arxiv.org/abs/2403.09635)
• Keywords: language model
• Abstract: In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern the moments of the forward and backward signal through the transformer model. Our framework can be used to understand and mitigate vanishing/exploding gradients, rank collapse, and instability associated with high attention scores. We also propose DeepScaleLM, an initialization and scaling scheme that conserves unit output/gradient moments throughout the model, enabling the training of very deep models with 100s of layers. We find that transformer models could be much deeper - our deep models with fewer parameters outperform shallow models in Language Modeling, Speech Translation, and Image Classification, across Encoder-only, Decoder-only and Encoder-Decoder variants, for both Pre-LN and Post-LN transformers, for multiple datasets and model sizes. These improvements also translate into improved performance on downstream Question Answering tasks and improved robustness for image classification.
• 摘要：尽管 Transformer 模型取得了巨大成功，但仍然难以深入扩展。在这项工作中，我们开发了统一的信号传播理论，并提供了通过变压器模型控制前向和后向信号矩的公式。我们的框架可用于理解和减轻梯度消失/爆炸、排名崩溃以及与高注意力分数相关的不稳定性。我们还提出了 DeepScaleLM，这是一种初始化和缩放方案，可以在整个模型中保留单位输出/梯度矩，从而能够训练具有 100 层的非常深的模型。我们发现 Transformer 模型可以更深——我们的参数较少的深层模型在语言建模、语音翻译和图像分类中，在仅编码器、仅解码器和编码器-解码器变体中，对于 Pre-LN 和LN 后变压器，适用于多个数据集和模型大小。这些改进还转化为下游问答任务性能的提高以及图像分类鲁棒性的提高。

Title: Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

• Authors: Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, Edoardo M. Ponti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09636
• Pdf URL: https://arxiv.org/pdf/2403.09636
• Copy Paste: [[2403.09636]] Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference(https://arxiv.org/abs/2403.09636)
• Keywords: language model, llm
• Abstract: Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value representations for past tokens, whose size scales linearly with the input sequence length and batch size. As a solution, we propose Dynamic Memory Compression (DMC), a method for on-line key-value cache compression at inference time. Most importantly, the model learns to apply different compression rates in different heads and layers. We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers, achieving up to ~3.7x throughput increase in auto-regressive inference on a NVIDIA H100 GPU. DMC is applied via continued pre-training on a negligible percentage of the original data without adding any extra parameters. We find that DMC preserves the original downstream performance with up to 4x cache compression, outperforming up-trained grouped-query attention (GQA). GQA and DMC can be even combined to obtain compounded gains. As a result DMC fits longer contexts and larger batches within any given memory budget.
• 摘要：Transformer 已成为大型语言模型 (LLM) 的支柱。然而，由于需要在内存中存储过去标记的键值表示的缓存，生成仍然效率低下，其大小与输入序列长度和批量大小线性缩放。作为解决方案，我们提出了动态内存压缩（DMC），这是一种在推理时进行在线键值缓存压缩的方法。最重要的是，该模型学习在不同的头和层中应用不同的压缩率。我们将 Llama 2（7B、13B 和 70B）等预训练的 LLM 改造为 DMC Transformer，在 NVIDIA H100 GPU 上实现自回归推理吞吐量高达约 3.7 倍的提升。 DMC 通过对原始数据的可忽略百分比进行持续预训练来应用，无需添加任何额外参数。我们发现 DMC 通过高达 4 倍的缓存压缩保留了原始的下游性能，优于经过训练的分组查询注意力 (GQA)。 GQA 和 DMC 甚至可以结合起来以获得复合收益。因此，DMC 在任何给定的内存预算内都适合更长的上下文和更大的批次。