2024-07-31

Title: Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions

• Authors: Jinsung Yoon, Raj Sinha, Sercan O Arik, Tomas Pfister
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.20243
• Pdf URL: https://arxiv.org/pdf/2407.20243
• Copy Paste: [[2407.20243]] Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions(https://arxiv.org/abs/2407.20243)
• Keywords: language model, llm
• Abstract: Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval. While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost. To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings. Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness. Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs. Also, it exhibits efficacy in both unsupervised and supervised learning settings. A rigorous evaluation conducted across a diverse corpus of English, multilingual, and multimodal datasets consistently reveals substantial gains with Matryoshka-Adaptor. Notably, with Google and OpenAI Embedding APIs, Matryoshka-Adaptor achieves a reduction in dimensionality ranging from two- to twelve-fold without compromising performance across multiple BEIR datasets.
• 摘要：大型语言模型 (LLM) 的嵌入已成为各种应用（尤其是信息检索）中的关键组件。虽然高维嵌入通常表现出优异的性能，因为它们包含更显著的信息，但它们的实际应用经常受到计算延迟增加和相关成本增加的阻碍。为了应对这些挑战，我们提出了 Matryoshka-Adaptor，这是一种专为定制 LLM 嵌入而设计的新型调整框架。Matryoshka-Adaptor 有助于大幅降低维数，同时保持可比的性能水平，从而显著提高计算效率和成本效益。我们的框架直接修改了预训练 LLM 的嵌入，旨在与任何 LLM 架构无缝集成，包括那些仅通过黑盒 API 访问的架构。此外，它在无监督和监督学习环境中都表现出有效性。对多种英语、多语言和多模态数据集进行的严格评估始终表明 Matryoshka-Adaptor 取得了显着的进步。值得注意的是，借助 Google 和 OpenAI Embedding API，Matryoshka-Adaptor 实现了维度减少 2 到 12 倍，同时又不影响多个 BEIR 数据集的性能。

Title: Steamroller Problems: An Evaluation of LLM Reasoning Capability with Automated Theorem Prover Strategies

• Authors: Lachlan McGinness, Peter Baumgartner
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20244
• Pdf URL: https://arxiv.org/pdf/2407.20244
• Copy Paste: [[2407.20244]] Steamroller Problems: An Evaluation of LLM Reasoning Capability with Automated Theorem Prover Strategies(https://arxiv.org/abs/2407.20244)
• Keywords: language model, gpt, llm
• Abstract: This study presents the first examination of the ability of Large Language Models (LLMs) to follow reasoning strategies that are used to guide Automated Theorem Provers (ATPs). We evaluate the performance of GPT4, GPT3.5 Turbo and Google's recent Gemini model on problems from a steamroller domain. In addition to determining accuracy we make use of the Natural Language Processing library spaCy to explore new methods of investigating LLM's reasoning capabilities. This led to one alarming result, the low correlation between correct reasoning and correct answers for any of the tested models. We found that the models' performance when using the ATP reasoning strategies was comparable to one-shot chain of thought and observe that attention to uncertainty in the accuracy results is critical when drawing conclusions about model performance. Consistent with previous speculation we confirm that LLMs have a preference for, and are best able to follow, bottom up reasoning processes. However, the reasoning strategies can still be beneficial for deriving small and relevant sets of formulas for external processing by a trusted inference engine.
• 摘要：本研究首次考察了大型语言模型 (LLM) 遵循用于指导自动定理证明器 (ATP) 的推理策略的能力。我们评估了 GPT4、GPT3.5 Turbo 和 Google 最近的 Gemini 模型在压路机领域问题上的表现。除了确定准确性之外，我们还利用自然语言处理库 spaCy 探索研究 LLM 推理能力的新方法。这导致了一个令人震惊的结果，即任何测试模型的正确推理和正确答案之间的相关性都很低。我们发现，使用 ATP 推理策略时，模型的性能与一次性思维链相当，并且观察到在得出有关模型性能的结论时，关注准确性结果中的不确定性至关重要。与之前的推测一致，我们确认 LLM 偏爱自下而上的推理过程，并且最能遵循这种推理过程。然而，推理策略仍然有利于通过可信推理引擎导出小而相关的公式集以供外部处理。

Title: LAPIS: Language Model-Augmented Police Investigation System

• Authors: Heedou Kim, Dain Kim, Jiwoo Lee, Chanwoong Yoon, Donghee Choi, Mogan Gim, Jaewoo Kang
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.20248
• Pdf URL: https://arxiv.org/pdf/2407.20248
• Copy Paste: [[2407.20248]] LAPIS: Language Model-Augmented Police Investigation System(https://arxiv.org/abs/2407.20248)
• Keywords: language model, gpt, prompt
• Abstract: Crime situations are race against time. An AI-assisted criminal investigation system, providing prompt but precise legal counsel is in need for police officers. We introduce LAPIS (Language Model Augmented Police Investigation System), an automated system that assists police officers to perform rational and legal investigative actions. We constructed a finetuning dataset and retrieval knowledgebase specialized in crime investigation legal reasoning task. We extended the dataset's quality by incorporating manual curation efforts done by a group of domain experts. We then finetuned the pretrained weights of a smaller Korean language model to the newly constructed dataset and integrated it with the crime investigation knowledgebase retrieval approach. Experimental results show LAPIS' potential in providing reliable legal guidance for police officers, even better than the proprietary GPT-4 model. Qualitative analysis on the rationales generated by LAPIS demonstrate the model's reasoning ability to leverage the premises and derive legally correct conclusions.
• 摘要：犯罪案件的侦破与时间赛跑。警察需要一套人工智能辅助刑事调查系统，为警察提供及时而精准的法律咨询。我们推出了 LAPIS（语言模型增强型警察调查系统），这是一种自动化系统，可协助警察执行合理合法的调查行动。我们构建了一个专门用于犯罪调查法律推理任务的微调数据集和检索知识库。我们通过结合一组领域专家的手动管理工作来扩展数据集的质量。然后，我们将较小的韩语模型的预训练权重微调到新构建的数据集，并将其与犯罪调查知识库检索方法相结合。实验结果表明，LAPIS 有潜力为警察提供可靠的法律指导，甚至比专有的 GPT-4 模型更好。对 LAPIS 生成的理由进行定性分析，证明了该模型利用前提并得出合法正确结论的推理能力。

Title: An Efficient Inference Framework for Early-exit Large Language Models

• Authors: Ruijie Miao, Yihan Yan, Xinshuo Yao, Tong Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.20272
• Pdf URL: https://arxiv.org/pdf/2407.20272
• Copy Paste: [[2407.20272]] An Efficient Inference Framework for Early-exit Large Language Models(https://arxiv.org/abs/2407.20272)
• Keywords: language model, llm
• Abstract: Building efficient inference framework has gained increasing interests for research community. Early-exit models, a variant of LLMs, improves the inference efficiency of LLMs by skipping rest layers and directly generate output tokens when they are confident enough. However, there is no work of LLM inference framework that takes early-exit models into consideration. This is non-trivial as prior art on LLM inference cannot be directly applied to early-exit models. In this work, we solves two key challenges in building efficient inference framework for early-exit models: (1) batch inference at iteration-level granularity; and (2) KV cache management. For the former, we propose to process the batch until all sequences surpass the early-exit confidence threshold. For the latter, we propose to fill the KV cache of rest layers before the iteration terminates. Our evaluation shows that, compared with the original vLLM operating at full layers, our solution achieves up to 1.25x speed up.
• 摘要：构建高效的推理框架引起了研究界越来越多的兴趣。早期退出模型是 LLM 的一种变体，它通过跳过剩余层并在足够置信时直接生成输出标记来提高 LLM 的推理效率。然而，目前还没有将早期退出模型考虑在内的 LLM 推理框架工作。这并不容易，因为 LLM 推理的现有技术不能直接应用于早期退出模型。在这项工作中，我们解决了构建早期退出模型高效推理框架的两个关键挑战：（1）迭代级粒度的批量推理；（2）KV 缓存管理。对于前者，我们建议处理批次，直到所有序列都超过早期退出置信度阈值。对于后者，我们建议在迭代终止前填充剩余层的 KV 缓存。我们的评估表明，与原始在完整层上运行的 vLLM 相比，我们的解决方案实现了高达 1.25 倍的速度提升。

Title: What if Red Can Talk? Dynamic Dialogue Generation Using Large Language Models

• Authors: Navapat Nananukul, Wichayaporn Wongkamjan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20382
• Pdf URL: https://arxiv.org/pdf/2407.20382
• Copy Paste: [[2407.20382]] What if Red Can Talk? Dynamic Dialogue Generation Using Large Language Models(https://arxiv.org/abs/2407.20382)
• Keywords: language model, gpt, llm
• Abstract: Role-playing games (RPGs) provide players with a rich, interactive world to explore. Dialogue serves as the primary means of communication between developers and players, manifesting in various forms such as guides, NPC interactions, and storytelling. While most games rely on written scripts to define the main story and character personalities, player immersion can be significantly enhanced through casual interactions between characters. With the advent of large language models (LLMs), we introduce a dialogue filler framework that utilizes LLMs enhanced by knowledge graphs to generate dynamic and contextually appropriate character interactions. We test this framework within the environments of Final Fantasy VII Remake and Pokemon, providing qualitative and quantitative evidence that demonstrates GPT-4's capability to act with defined personalities and generate dialogue. However, some flaws remain, such as GPT-4 being overly positive or more subtle personalities, such as maturity, tend to be of lower quality compared to more overt traits like timidity. This study aims to assist developers in crafting more nuanced filler dialogues, thereby enriching player immersion and enhancing the overall RPG experience.
• 摘要：角色扮演游戏 (RPG) 为玩家提供了一个丰富的互动世界供其探索。对话是开发者与玩家之间沟通的主要方式，以各种形式体现，例如指南、NPC 交互和讲故事。虽然大多数游戏都依赖书面脚本来定义主要故事和角色个性，但通过角色之间的随意互动可以显著增强玩家的沉浸感。随着大型语言模型 (LLM) 的出现，我们引入了一个对话填充框架，该框架利用知识图增强的 LLM 来生成动态且适合上下文的角色交互。我们在《最终幻想 VII 重制版》和《口袋妖怪》的环境中测试了这个框架，提供了定性和定量证据，证明了 GPT-4 能够以明确的个性行事并生成对话。然而，一些缺陷仍然存在，例如 GPT-4 过于积极，或者成熟等更微妙的个性往往与胆怯等更明显的特征相比质量较低。这项研究旨在帮助开发人员制作更细致入微的填充对话，从而丰富玩家的沉浸感并增强整体 RPG 体验。

Title: Through the Looking Glass, and what Horn Clause Programs Found There

• Authors: Paul Tarau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20413
• Pdf URL: https://arxiv.org/pdf/2407.20413
• Copy Paste: [[2407.20413]] Through the Looking Glass, and what Horn Clause Programs Found There(https://arxiv.org/abs/2407.20413)
• Keywords: llm
• Abstract: Dual Horn clauses mirror key properties of Horn clauses. This paper explores the ``other side of the looking glass'' to reveal some expected and unexpected symmetries and their practical uses. We revisit Dual Horn clauses as enablers of a form of constructive negation that supports goal-driven forward reasoning and is valid both intuitionistically and classically. In particular, we explore the ability to falsify a counterfactual hypothesis in the context of a background theory expressed as a Dual Horn clause program. With Dual Horn clause programs, by contrast to negation as failure, the variable bindings in their computed answers provide explanations for the reasons why a statement is successfully falsified. Moreover, in the propositional case, by contrast to negation as failure as implemented with stable models semantics in ASP systems, and similarly to Horn clause programs, Dual Horn clause programs have polynomial complexity. After specifying their execution model with a metainterpreter, we devise a compilation scheme from Dual Horn clause programs to Horn clause programs, ensuring their execution with no performance penalty and we design the embedded SymLP language to support combined Horn clause and Dual Horn clause programs. As a (motivating) application, we cast LLM reasoning chains into propositional Horn and Dual Horn clauses that work together to constructively prove and disprove goals and enhance Generative AI with explainability of reasoning chains.
• 摘要：对偶霍恩子句反映了霍恩子句的关键属性。本文探索了“镜子的另一面”，以揭示一些预期和意外的对称性及其实际用途。我们重新审视了对偶霍恩子句，将其作为一种建设性否定形式的推动者，这种否定形式支持目标驱动的前向推理，并且既符合直觉又符合经典。特别是，我们探索了在以对偶霍恩子句程序表示的背景理论背景下证伪反事实假设的能力。对于对偶霍恩子句程序，与否定失败相比，其计算答案中的变量绑定为成功证伪陈述的原因提供了解释。此外，在命题情况下，与在 ASP 系统中使用稳定模型语义实现的否定失败相比，与霍恩子句程序类似，对偶霍恩子句程序具有多项式复杂度。在使用元解释器指定执行模型后，我们设计了从双 Horn 子句程序到 Horn 子句程序的编译方案，确保它们的执行不会影响性能，并且我们设计了嵌入式 SymLP 语言来支持组合 Horn 子句和双 Horn 子句程序。作为一个（激励）应用，我们将 LLM 推理链转换为命题 Horn 和双 Horn 子句，它们共同努力以建设性地证明和反驳目标，并通过推理链的可解释性增强生成式 AI。

Title: A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder

• Authors: Hyun Rae Jo, Dong Kun Shin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.20485
• Pdf URL: https://arxiv.org/pdf/2407.20485
• Copy Paste: [[2407.20485]] A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder(https://arxiv.org/abs/2407.20485)
• Keywords: language model, llm
• Abstract: Recently, large language models (LLM) based on transformers are facing memory bottleneck issues due to KV cache, especially in long sequence handling. Previous researches proposed KV cache compression techniques that identify insignificant tokens based on Accumulative Attention Scores and removes their items from KV cache, noting that only few tokens play an important role in attention operations. However, we have observed that the existing Accumulative Attention Score is not suitable for the transformer decoder structure. In the decoder model, the number of times the Attention Score accumulates varies depending on the order of token appearance due to the effect of masking, causing an uneven comparison between tokens. To solve this, we propose Accumulative Attention Score with Forgetting Factor (A2SF) technique, which introduces a Forgetting Factor in the Attention Score accumulation process. A2SF applies a penalty to the past Attention Score generated from old tokens by repeatedly multiplying the Forgetting Factor to the Attention Score over time. Therefore, older tokens receive a larger penalty, providing fairness among different ages of tokens. Through the fair comparison among tokens, we can more effectively select important tokens. We have verified the accuracy improvement through A2SF in the OPT and LLaMA models and A2SF improves the accuracy of LLaMA 2 by up to 7.8% and 5.1% on 1-shot and 0-shot.
• 摘要：最近，基于 transformer 的大型语言模型 (LLM) 因 KV 缓存而面临内存瓶颈问题，尤其是在处理长序列时。先前的研究提出了基于累积注意力得分的 KV 缓存压缩技术，该技术根据累积注意力得分识别不重要的 token 并将其项从 KV 缓存中删除，并指出只有少数 token 在注意力操作中发挥重要作用。然而，我们观察到现有的累积注意力得分并不适合 transformer 解码器结构。在解码器模型中，由于掩蔽效应，注意力得分累积的次数会根据 token 出现的顺序而变化，从而导致 token 之间的比较不均匀。为了解决这个问题，我们提出了带遗忘因子的累积注意力得分 (A2SF) 技术，该技术在注意力得分累积过程中引入了遗忘因子。A2SF 通过将遗忘因子随时间反复乘以注意力得分来对由旧 token 生成的过去注意力得分施加惩罚。因此，较旧的 token 受到的惩罚更大，从而为不同年龄的 token 提供公平性。通过对 token 进行公平比较，我们可以更有效地选择重要的 token。我们已经在 OPT 和 LLaMA 模型中验证了 A2SF 带来的准确率提升，其中 A2SF 在 1-shot 和 0-shot 上分别将 LLaMA 2 的准确率提升了 7.8% 和 5.1%。

Title: Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language

• Authors: Hossein Rajaby Faghihi, Aliakbar Nafar, Andrzej Uszok, Hamid Karimian, Parisa Kordjamshidi
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.20513
• Pdf URL: https://arxiv.org/pdf/2407.20513
• Copy Paste: [[2407.20513]] Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language(https://arxiv.org/abs/2407.20513)
• Keywords: language model, prompt
• Abstract: This paper presents a conversational pipeline for crafting domain knowledge for complex neuro-symbolic models through natural language prompts. It leverages large language models to generate declarative programs in the DomiKnowS framework. The programs in this framework express concepts and their relationships as a graph in addition to logical constraints between them. The graph, later, can be connected to trainable neural models according to those specifications. Our proposed pipeline utilizes techniques like dynamic in-context demonstration retrieval, model refinement based on feedback from a symbolic parser, visualization, and user interaction to generate the tasks' structure and formal knowledge representation. This approach empowers domain experts, even those not well-versed in ML/AI, to formally declare their knowledge to be incorporated in customized neural models in the DomiKnowS framework.
• 摘要：本文介绍了一种对话式流程，用于通过自然语言提示为复杂的神经符号模型构建领域知识。它利用大型语言模型在 DomiKnowS 框架中生成声明性程序。该框架中的程序将概念及其关系表达为图形，以及它们之间的逻辑约束。随后，可以根据这些规范将图形连接到可训练的神经模型。我们提出的流程利用动态上下文演示检索、基于符号解析器反馈的模型细化、可视化和用户交互等技术来生成任务的结构和正式知识表示。这种方法使领域专家（即使是那些不熟悉 ML/AI 的专家）能够正式宣布他们的知识将被纳入 DomiKnowS 框架中的定制神经模型中。

Title: CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

• Authors: Tianshi Zheng, Jiaxin Bai, Yicheng Wang, Tianqing Fang, Yue Guo, Yauwai Yim, Yangqiu Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20564
• Pdf URL: https://arxiv.org/pdf/2407.20564
• Copy Paste: [[2407.20564]] CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge(https://arxiv.org/abs/2407.20564)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: While large language models (LLMs) have demonstrated impressive capabilities across various natural language processing tasks by acquiring rich factual knowledge from their broad training data, their ability to synthesize and logically reason with this knowledge in complex ways remains underexplored. In this work, we present a systematic evaluation of state-of-the-art LLMs' complex logical reasoning abilities through a novel benchmark of automatically generated complex reasoning questions over general domain and biomedical knowledge graphs. Our extensive experiments, employing diverse in-context learning techniques, reveal that LLMs excel at reasoning over general world knowledge but face significant challenges with specialized domain-specific knowledge. We find that prompting with explicit Chain-of-Thought demonstrations can substantially improve LLM performance on complex logical reasoning tasks with diverse logical operations. Interestingly, our controlled evaluations uncover an asymmetry where LLMs display proficiency at set union operations, but struggle considerably with set intersections - a key building block of logical reasoning. To foster further work, we will publicly release our evaluation benchmark and code.
• 摘要：虽然大型语言模型 (LLM) 通过从广泛的训练数据中获取丰富的事实知识，在各种自然语言处理任务中表现出令人印象深刻的能力，但它们以复杂方式综合和逻辑推理这些知识的能力仍未得到充分探索。在这项工作中，我们通过对一般领域和生物医学知识图谱自动生成的复杂推理问题的新基准，对最先进的 LLM 的复杂逻辑推理能力进行了系统评估。我们进行了广泛的实验，采用了多种上下文学习技术，结果表明 LLM 擅长推理一般世界知识，但在专门的特定领域知识方面面临重大挑战。我们发现，使用明确的思路链演示进行提示可以显著提高 LLM 在具有多种逻辑运算的复杂逻辑推理任务上的表现。有趣的是，我们的受控评估发现了一种不对称现象，即 LLM 在集合并运算方面表现出色，但在集合交集方面却举步维艰——这是逻辑推理的关键组成部分。为了促进进一步的工作，我们将公开发布我们的评估基准和代码。

Title: Comparison of Large Language Models for Generating Contextually Relevant Questions

• Authors: Ivo Lodovico Molina, Valdemar Švábenský, Tsubasa Minematsu, Li Chen, Fumiya Okubo, Atsushi Shimada
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.20578
• Pdf URL: https://arxiv.org/pdf/2407.20578
• Copy Paste: [[2407.20578]] Comparison of Large Language Models for Generating Contextually Relevant Questions(https://arxiv.org/abs/2407.20578)
• Keywords: language model, gpt, llm, chat
• Abstract: This study explores the effectiveness of Large Language Models (LLMs) for Automatic Question Generation in educational settings. Three LLMs are compared in their ability to create questions from university slide text without fine-tuning. Questions were obtained in a two-step pipeline: first, answer phrases were extracted from slides using Llama 2-Chat 13B; then, the three models generated questions for each answer. To analyze whether the questions would be suitable in educational applications for students, a survey was conducted with 46 students who evaluated a total of 246 questions across five metrics: clarity, relevance, difficulty, slide relation, and question-answer alignment. Results indicate that GPT-3.5 and Llama 2-Chat 13B outperform Flan T5 XXL by a small margin, particularly in terms of clarity and question-answer alignment. GPT-3.5 especially excels at tailoring questions to match the input answers. The contribution of this research is the analysis of the capacity of LLMs for Automatic Question Generation in education.
• 摘要：本研究探讨了大型语言模型 (LLM) 在教育环境中自动生成问题的有效性。比较了三款 LLM 在未经微调的情况下从大学幻灯片文本创建问题的能力。问题通过两步流程获得：首先，使用 Llama 2-Chat 13B 从幻灯片中提取答案短语；然后，这三个模型为每个答案生成问题。为了分析这些问题是否适合学生的教育应用，对 46 名学生进行了调查，他们评估了五个指标中的总共 246 个问题：清晰度、相关性、难度、幻灯片关系和问答对齐。结果表明，GPT-3.5 和 Llama 2-Chat 13B 的表现略胜 Flan T5 XXL，尤其是在清晰度和问答对齐方面。GPT-3.5 尤其擅长定制问题以匹配输入答案。本研究的贡献是分析了 LLM 在教育领域自动生成问题的能力。

Title: Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings

• Authors: Gili Goldin (1), Shuly Wintner (1) ((1) Department of Computer Science, University of Haifa, Israel)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20581
• Pdf URL: https://arxiv.org/pdf/2407.20581
• Copy Paste: [[2407.20581]] Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings(https://arxiv.org/abs/2407.20581)
• Keywords: language model
• Abstract: We present Knesset-DictaBERT, a large Hebrew language model fine-tuned on the Knesset Corpus, which comprises Israeli parliamentary proceedings. The model is based on the DictaBERT architecture and demonstrates significant improvements in understanding parliamentary language according to the MLM task. We provide a detailed evaluation of the model's performance, showing improvements in perplexity and accuracy over the baseline DictaBERT model.
• 摘要：我们推出了 Knesset-DictaBERT，这是一个大型希伯来语模型，基于包含以色列议会会议记录的 Knesset Corpus 进行了微调。该模型基于 DictaBERT 架构，在 MLM 任务中表现出对议会语言理解的显著提升。我们对该模型的性能进行了详细评估，结果表明困惑度和准确度均优于基线 DictaBERT 模型。

Title: Pruning Large Language Models with Semi-Structural Adaptive Sparse Training

• Authors: Weiyu Huang, Guohao Jian, Yuezhou Hu, Jun Zhu, Jianfei Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20584
• Pdf URL: https://arxiv.org/pdf/2407.20584
• Copy Paste: [[2407.20584]] Pruning Large Language Models with Semi-Structural Adaptive Sparse Training(https://arxiv.org/abs/2407.20584)
• Keywords: language model, llm
• Abstract: Transformer-based Large Language Models (LLMs) have demonstrated remarkable success across various challenging tasks. However, the deployment of LLMs is hindered by their substantial parameter count and memory consumption. Recently, numerous studies have attempted to compress LLMs by pruning them using training-free methods. However, these pruned models often experience significant performance degradation on complex tasks. To address this issue, we propose a novel training pipeline for semi-structured sparse models, named Adaptive Sparse Trainer (AST). By distilling the knowledge stored in its dense counterpart, we prevent the sparse model from overfitting and ensure a stable training process. Moreover, AST allows the model to adaptively select better lottery tickets (e.g., masks) during training. Additionally, we discovered that adding extra well-initialized parameters can further enhance model performance with only a small increase in memory footprint. Our method significantly narrows the performance gap between dense and sparse models while maintaining limited computational cost. Furthermore, when combined with existing quantization methods, AST can compress language models by up to 16x compared to dense FP32 precision models with minimal performance loss. AST outperforms previous state-of-the-art methods by reducing the zero-shot accuracy gap between dense and semi-structured sparse models to 1.12% across multiple zero-shot tasks on Llama2-7B, using less than 0.4% of the pretraining tokens.
• 摘要：基于 Transformer 的大型语言模型 (LLM) 在各种具有挑战性的任务中都取得了显著的成功。然而，LLM 的部署受到其大量参数数量和内存消耗的阻碍。最近，许多研究尝试通过使用无需训练的方法修剪 LLM 来压缩它们。然而，这些修剪过的模型在复杂任务上通常会经历显著的性能下降。为了解决这个问题，我们提出了一种用于半结构化稀疏模型的新型训练管道，称为自适应稀疏训练器 (AST)。通过提炼存储在其密集对应物中的知识，我们可以防止稀疏模型过度拟合并确保稳定的训练过程。此外，AST 允许模型在训练期间自适应地选择更好的彩票（例如，掩码）。此外，我们发现添加额外的初始化良好的参数可以进一步提高模型性能，而内存占用只会略有增加。我们的方法在保持有限计算成本的同时显著缩小了密集和稀疏模型之间的性能差距。此外，与现有的量化方法相结合，AST 可以将语言模型压缩高达 16 倍（与密集的 FP32 精度模型相比），同时将性能损失降至最低。AST 的表现优于之前最先进的方法，在 Llama2-7B 上的多个零样本任务中，使用不到 0.4% 的预训练标记，将密集和半结构化稀疏模型之间的零样本准确率差距缩小到 1.12%。

Title: Enhancing Agricultural Machinery Management through Advanced LLM Integration

• Authors: Emily Johnson, Noah Wilson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20588
• Pdf URL: https://arxiv.org/pdf/2407.20588
• Copy Paste: [[2407.20588]] Enhancing Agricultural Machinery Management through Advanced LLM Integration(https://arxiv.org/abs/2407.20588)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: The integration of artificial intelligence into agricultural practices, specifically through Consultation on Intelligent Agricultural Machinery Management (CIAMM), has the potential to revolutionize efficiency and sustainability in farming. This paper introduces a novel approach that leverages large language models (LLMs), particularly GPT-4, combined with multi-round prompt engineering to enhance decision-making processes in agricultural machinery management. We systematically developed and refined prompts to guide the LLMs in generating precise and contextually relevant outputs. Our approach was evaluated using a manually curated dataset from various online sources, and performance was assessed with accuracy and GPT-4 Scores. Comparative experiments were conducted using LLama-2-70B, ChatGPT, and GPT-4 models, alongside baseline and state-of-the-art methods such as Chain of Thought (CoT) and Thought of Thought (ThoT). The results demonstrate that our method significantly outperforms these approaches, achieving higher accuracy and relevance in generated responses. This paper highlights the potential of advanced prompt engineering techniques in improving the robustness and applicability of AI in agricultural contexts.
• 摘要：将人工智能融入农业实践，特别是通过智能农业机械管理咨询 (CIAMM)，有可能彻底改变农业的效率和可持续性。本文介绍了一种新方法，该方法利用大型语言模型 (LLM)，特别是 GPT-4，结合多轮提示工程来增强农业机械管理中的决策过程。我们系统地开发和改进了提示，以指导 LLM 生成精确且与上下文相关的输出。我们使用来自各种在线来源的手动整理数据集来评估我们的方法，并使用准确度和 GPT-4 分数来评估性能。使用 LLama-2-70B、ChatGPT 和 GPT-4 模型以及基线和最先进的方法（如思想链 (CoT) 和思想思想 (ThoT)）进行了比较实验。结果表明，我们的方法明显优于这些方法，在生成的响应中实现了更高的准确性和相关性。本文强调了先进的提示工程技术在提高农业环境中人工智能的稳健性和适用性方面的潜力。

Title: Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian

• Authors: Serena Auriemma, Martina Miliani, Mauro Madeddu, Alessandro Bondielli, Lucia Passaro, Alessandro Lenci
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20654
• Pdf URL: https://arxiv.org/pdf/2407.20654
• Copy Paste: [[2407.20654]] Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian(https://arxiv.org/abs/2407.20654)
• Keywords: language model, llm, prompt
• Abstract: Addressing the challenge of limited annotated data in specialized fields and low-resource languages is crucial for the effective use of Language Models (LMs). While most Large Language Models (LLMs) are trained on general-purpose English corpora, there is a notable gap in models specifically tailored for Italian, particularly for technical and bureaucratic jargon. This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in these specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. We evaluated the models on downstream tasks such as document classification and entity typing and conducted intrinsic evaluations using Pseudo-Log-Likelihood. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting. Furthermore, the application of calibration techniques and in-domain verbalizers significantly enhances the efficacy of encoder models. These domain-specialized models prove to be particularly advantageous in scenarios where in-domain resources or expertise are scarce. In conclusion, our findings offer new insights into the use of Italian models in specialized contexts, which may have a significant impact on both research and industrial applications in the digital transformation era.
• 摘要：解决专业领域和资源匮乏语言中注释数据有限的挑战对于有效使用语言模型 (LM) 至关重要。虽然大多数大型语言模型 (LLM) 都是在通用英语语料库上训练的，但专门针对意大利语的模型存在明显差距，尤其是针对技术和官僚术语。本文探讨了在这些专业环境中使用较小的、特定领域的编码器 LM 以及提示技术来提高性能的可行性。我们的研究集中在意大利官僚语言和法律语言上，尝试了通用和进一步预训练的编码器专用模型。我们在文档分类和实体类型等下游任务上评估了模型，并使用伪对数似然法进行了内在评估。结果表明，虽然进一步预训练的模型在一般知识方面可能表现出较差的稳健性，但它们在特定领域任务中表现出卓越的适应性，即使在零样本设置中也是如此。此外，校准技术和域内言语化器的应用显著提高了编码器模型的功效。这些领域专业模型在领域资源或专业知识稀缺的情况下被证明特别有利。总之，我们的研究结果为意大利模型在专业环境中的使用提供了新的见解，这可能会对数字化转型时代的研究和工业应用产生重大影响。

Title: Label-Guided Prompt for Multi-label Few-shot Aspect Category Detection

• Authors: ChaoFeng Guan, YaoHui Zhu, Yu Bai, LingYun Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20673
• Pdf URL: https://arxiv.org/pdf/2407.20673
• Copy Paste: [[2407.20673]] Label-Guided Prompt for Multi-label Few-shot Aspect Category Detection(https://arxiv.org/abs/2407.20673)
• Keywords: language model, prompt
• Abstract: Multi-label few-shot aspect category detection aims at identifying multiple aspect categories from sentences with a limited number of training instances. The representation of sentences and categories is a key issue in this task. Most of current methods extract keywords for the sentence representations and the category representations. Sentences often contain many category-independent words, which leads to suboptimal performance of keyword-based methods. Instead of directly extracting keywords, we propose a label-guided prompt method to represent sentences and categories. To be specific, we design label-specific prompts to represent sentences by combining crucial contextual and semantic information. Further, the label is introduced into a prompt to obtain category descriptions by utilizing a large language model. This kind of category descriptions contain the characteristics of the aspect categories, guiding the construction of discriminative category prototypes. Experimental results on two public datasets show that our method outperforms current state-of-the-art methods with a 3.86% - 4.75% improvement in the Macro-F1 score.
• 摘要：多标签小样本方面类别检测旨在利用有限数量的训练实例从句子中识别出多个方面类别。句子和类别的表示是该任务的关键问题。当前大多数方法都提取句子表示和类别表示的关键词。句子通常包含许多与类别无关的词，这导致基于关键词的方法性能不佳。与直接提取关键词不同，我们提出了一种标签引导的提示方法来表示句子和类别。具体而言，我们结合关键的上下文和语义信息设计特定于标签的提示来表示句子。进一步，利用大型语言模型将标签引入提示中以获得类别描述。这种类别描述包含方面类别的特征，指导构建判别性类别原型。在两个公共数据集上的实验结果表明，我们的方法优于目前最先进的方法，Macro-F1 分数提高了 3.86% - 4.75%。

Title: Adapting Safe-for-Work Classifier for Malaysian Language Text: Enhancing Alignment in LLM-Ops Framework

• Authors: Aisyah Razak, Ariff Nazhan, Kamarul Adha, Wan Adzhar Faiq Adzlan, Mas Aisyah Ahmad, Ammar Azman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20729
• Pdf URL: https://arxiv.org/pdf/2407.20729
• Copy Paste: [[2407.20729]] Adapting Safe-for-Work Classifier for Malaysian Language Text: Enhancing Alignment in LLM-Ops Framework(https://arxiv.org/abs/2407.20729)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) become increasingly integrated into operational workflows (LLM-Ops), there is a pressing need for effective guardrails to ensure safe and aligned interactions, including the ability to detect potentially unsafe or inappropriate content across languages. However, existing safe-for-work classifiers are primarily focused on English text. To address this gap for the Malaysian language, we present a novel safe-for-work text classifier tailored specifically for Malaysian language content. By curating and annotating a first-of-its-kind dataset of Malaysian text spanning multiple content categories, we trained a classification model capable of identifying potentially unsafe material using state-of-the-art natural language processing techniques. This work represents an important step in enabling safer interactions and content filtering to mitigate potential risks and ensure responsible deployment of LLMs. To maximize accessibility and promote further research towards enhancing alignment in LLM-Ops for the Malaysian context, the model is publicly released at this https URL.
• 摘要：随着大型语言模型 (LLM) 越来越多地融入到操作工作流 (LLM-Ops) 中，迫切需要有效的防护措施来确保安全和一致的交互，包括能够检测跨语言的潜在不安全或不适当内容。然而，现有的安全工作分类器主要集中在英语文本上。为了解决马来西亚语的这一空白，我们提出了一种专门针对马来西亚语内容的新型安全工作文本分类器。通过整理和注释涵盖多个内容类别的首创马来西亚文本数据集，我们训练了一个分类模型，该模型能够使用最先进的自然语言处理技术识别潜在的不安全材料。这项工作是实现更安全的交互和内容过滤以减轻潜在风险并确保负责任地部署 LLM 的重要一步。为了最大限度地提高可访问性并促进进一步研究以增强马来西亚语背景下的 LLM-Ops 一致性，该模型在此 https URL 上公开发布。

Title: Meltemi: The first open Large Language Model for Greek

• Authors: Leon Voukoutis, Dimitris Roussis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavasileiou, Athanasios Katsamanis, Stelios Piperidis, Vassilis Katsouros
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20743
• Pdf URL: https://arxiv.org/pdf/2407.20743
• Copy Paste: [[2407.20743]] Meltemi: The first open Large Language Model for Greek(https://arxiv.org/abs/2407.20743)
• Keywords: language model, prompt, chat
• Abstract: We describe the development and capabilities of Meltemi 7B, the first open Large Language Model for the Greek language. Meltemi 7B has 7 billion parameters and is trained on a 40 billion token Greek corpus. For the development of Meltemi 7B, we adapt Mistral, by continuous pretraining on the Greek Corpus. Meltemi 7B contains up-to-date information up to September 2023. Furthermore, we have translated and curated a Greek instruction corpus, which has been used for the instruction-tuning of a chat model, named Meltemi 7B Instruct. Special care has been given to the alignment and the removal of toxic content for the Meltemi 7B Instruct. The developed models are evaluated on a broad set of collected evaluation corpora, and examples of prompts and responses are presented. Both Meltemi 7B and Meltemi 7B Instruct are available at this https URL under the Apache 2.0 license.
• 摘要：我们描述了 Meltemi 7B 的开发和功能，这是第一个开放的希腊语大型语言模型。Meltemi 7B 有 70 亿个参数，并在 400 亿个希腊语标记语料库上进行训练。为了开发 Meltemi 7B，我们通过对希腊语语料库进行持续预训练来调整 Mistral。Meltemi 7B 包含截至 2023 年 9 月的最新信息。此外，我们还翻译并整理了一个希腊语指令语料库，该语料库已用于聊天模型的指令调整，名为 Meltemi 7B Instruct。我们特别注意了 Meltemi 7B Instruct 的对齐和有害内容的删除。在广泛收集的评估语料库上对开发的模型进行了评估，并提供了提示和响应的示例。Meltemi 7B 和 Meltemi 7B Instruct 均可在 Apache 2.0 许可下的此 https URL 上获得。

Title: Effective Black Box Testing of Sentiment Analysis Classification Networks

• Authors: Parsa Karbasizadeh, Fathiyeh Faghih, Pouria Golshanrad
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.20884
• Pdf URL: https://arxiv.org/pdf/2407.20884
• Copy Paste: [[2407.20884]] Effective Black Box Testing of Sentiment Analysis Classification Networks(https://arxiv.org/abs/2407.20884)
• Keywords: language model
• Abstract: Transformer-based neural networks have demonstrated remarkable performance in natural language processing tasks such as sentiment analysis. Nevertheless, the issue of ensuring the dependability of these complicated architectures through comprehensive testing is still open. This paper presents a collection of coverage criteria specifically designed to assess test suites created for transformer-based sentiment analysis networks. Our approach utilizes input space partitioning, a black-box method, by considering emotionally relevant linguistic features such as verbs, adjectives, adverbs, and nouns. In order to effectively produce test cases that encompass a wide range of emotional elements, we utilize the k-projection coverage metric. This metric minimizes the complexity of the problem by examining subsets of k features at the same time, hence reducing dimensionality. Large language models are employed to generate sentences that display specific combinations of emotional features. The findings from experiments obtained from a sentiment analysis dataset illustrate that our criteria and generated tests have led to an average increase of 16\% in test coverage. In addition, there is a corresponding average decrease of 6.5\% in model accuracy, showing the ability to identify vulnerabilities. Our work provides a foundation for improving the dependability of transformer-based sentiment analysis systems through comprehensive test evaluation.
• 摘要：基于 Transformer 的神经网络在情绪分析等自然语言处理任务中表现出色。然而，通过全面测试确保这些复杂架构的可靠性的问题仍然存在。本文介绍了一组专门用于评估为基于 Transformer 的情绪分析网络创建的测试套件的覆盖标准。我们的方法利用输入空间划分（一种黑盒方法），通过考虑与情感相关的语言特征，例如动词、形容词、副词和名词。为了有效地生成涵盖广泛情感元素的测试用例，我们使用了 k 投影覆盖度量。该度量通过同时检查 k 个特征的子集来最大限度地降低问题的复杂性，从而降低维度。大型语言模型用于生成显示特定情感特征组合的句子。从情绪分析数据集获得的实验结果表明，我们的标准和生成的测试已使测试覆盖率平均增加了 16%。此外，模型准确率也相应平均下降了 6.5%，显示出识别漏洞的能力。我们的工作为通过全面的测试评估提高基于 Transformer 的情绪分析系统的可靠性奠定了基础。

Title: Automated Review Generation Method Based on Large Language Models

• Authors: Shican Wu, Xiao Ma, Dehui Luo, Lulu Li, Xiangcheng Shi, Xin Chang, Xiaoyun Lin, Ran Luo, Chunlei Pei, Zhi-Jian Zhao, Jinlong Gong
• Subjects: cs.CL, cs.AI, physics.data-an
• Abstract URL: https://arxiv.org/abs/2407.20906
• Pdf URL: https://arxiv.org/pdf/2407.20906
• Copy Paste: [[2407.20906]] Automated Review Generation Method Based on Large Language Models(https://arxiv.org/abs/2407.20906)
• Keywords: language model, llm, hallucination
• Abstract: Literature research, vital for scientific advancement, is overwhelmed by the vast ocean of available information. Addressing this, we propose an automated review generation method based on Large Language Models (LLMs) to streamline literature processing and reduce cognitive load. In case study on propane dehydrogenation (PDH) catalysts, our method swiftly generated comprehensive reviews from 343 articles, averaging seconds per article per LLM account. Extended analysis of 1041 articles provided deep insights into catalysts' composition, structure, and performance. Recognizing LLMs' hallucinations, we employed a multi-layered quality control strategy, ensuring our method's reliability and effective hallucination mitigation. Expert verification confirms the accuracy and citation integrity of generated reviews, demonstrating LLM hallucination risks reduced to below 0.5% with over 95% confidence. Released Windows application enables one-click review generation, aiding researchers in tracking advancements and recommending literature. This approach showcases LLMs' role in enhancing scientific research productivity and sets the stage for further exploration.
• 摘要：文献研究对科学进步至关重要，但海量的信息让其应接不暇。为了解决这个问题，我们提出了一种基于大型语言模型 (LLM) 的自动化评论生成方法，以简化文献处理并减少认知负荷。在丙烷脱氢 (PDH) 催化剂案例研究中，我们的方法迅速从 343 篇文章中生成了全面的评论，平均每个 LLM 帐户每篇文章只需几秒。对 1041 篇文章的扩展分析提供了对催化剂的成分、结构和性能的深入了解。认识到 LLM 的幻觉，我们采用了多层质量控制策略，确保我们方法的可靠性和有效的幻觉缓解。专家验证证实了生成的评论的准确性和引用完整性，表明 LLM 幻觉风险降低到 0.5% 以下，置信度超过 95%。发布的 Windows 应用程序支持一键生成评论，帮助研究人员跟踪进展和推荐文献。这种方法展示了 LLM 在提高科研生产力方面的作用，并为进一步探索奠定了基础。

Title: Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection

• Authors: Jinfa Huang, Jinsheng Pan, Zhongwei Wan, Hanjia Lyu, Jiebo Luo
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.21004
• Pdf URL: https://arxiv.org/pdf/2407.21004
• Copy Paste: [[2407.21004]] Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection(https://arxiv.org/abs/2407.21004)
• Keywords: prompt
• Abstract: Recent advances show that two-stream approaches have achieved outstanding performance in hateful meme detection. However, hateful memes constantly evolve as new memes emerge by fusing progressive cultural ideas, making existing methods obsolete or ineffective. In this work, we explore the potential of Large Multimodal Models (LMMs) for hateful meme detection. To this end, we propose Evolver, which incorporates LMMs via Chain-of-Evolution (CoE) Prompting, by integrating the evolution attribute and in-context information of memes. Specifically, Evolver simulates the evolving and expressing process of memes and reasons through LMMs in a step-by-step manner. First, an evolutionary pair mining module retrieves the top-k most similar memes in the external curated meme set with the input meme. Second, an evolutionary information extractor is designed to summarize the semantic regularities between the paired memes for prompting. Finally, a contextual relevance amplifier enhances the in-context hatefulness information to boost the search for evolutionary processes. Extensive experiments on public FHM, MAMI, and HarM datasets show that CoE prompting can be incorporated into existing LMMs to improve their performance. More encouragingly, it can serve as an interpretive tool to promote the understanding of the evolution of social memes.
• 摘要：最近的进展表明，双流方法在仇恨模因检测方面取得了出色的表现。然而，随着新模因的出现，仇恨模因通过融合进步的文化理念而不断发展，使现有方法变得过时或无效。在这项工作中，我们探索了大型多模因模型 (LMM) 在仇恨模因检测中的潜力。为此，我们提出了 Evolver，它通过进化链 (CoE) 提示整合 LMM，通过整合模因的进化属性和上下文信息。具体来说，Evolver 通过 LMM 逐步模拟模因和原因的进化和表达过程。首先，进化对挖掘模块从外部策划的模因集中检索与输入模因最相似的前 k 个模因。其次，进化信息提取器旨在总结成对模因之间的语义规律以进行提示。最后，上下文相关性放大器增强了上下文仇恨信息，以促进对进化过程的搜索。在公开的 FHM、MAMI 和 HarM 数据集上进行的大量实验表明，CoE 提示可以融入现有的 LMM 中，以提高其性能。更令人鼓舞的是，它可以作为一种解释工具，促进对社交模因演变的理解。

Title: ThinK: Thinner Key Cache by Query-Driven Pruning

• Authors: Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, Doyen Sahoo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.21018
• Pdf URL: https://arxiv.org/pdf/2407.21018
• Copy Paste: [[2407.21018]] ThinK: Thinner Key Cache by Query-Driven Pruning(https://arxiv.org/abs/2407.21018)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications by leveraging increased model sizes and sequence lengths. However, the associated rise in computational and memory costs poses significant challenges, particularly in managing long sequences due to the quadratic complexity of the transformer attention mechanism. This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference. Unlike existing approaches that optimize the memory based on the sequence lengths, we uncover that the channel dimension of the KV cache exhibits significant redundancy, characterized by unbalanced magnitude distribution and low-rank structure in attention weights. Based on these observations, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in memory costs by over 20% compared with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and Mistral models across various long-sequence datasets confirm the efficacy of ThinK, setting a new precedent for efficient LLM deployment without compromising performance. We also outline the potential of extending our method to value cache pruning, demonstrating ThinK's versatility and broad applicability in reducing both memory and computational overheads.
• 摘要：大型语言模型 (LLM) 彻底改变了自然语言处理领域，通过利用增加的模型大小和序列长度，在各种应用中实现了前所未有的性能。然而，计算和内存成本的增加带来了重大挑战，特别是在管理长序列时，因为 Transformer 注意力机制的二次复杂度。本文重点关注长上下文场景，解决推理过程中 KV 缓存内存消耗的低效问题。与现有的基于序列长度优化内存的方法不同，我们发现 KV 缓存的通道维度表现出明显的冗余，其特点是注意力权重的幅度分布不均衡和低秩结构。基于这些观察，我们提出了 ThinK，这是一种新颖的查询相关 KV 缓存修剪方法，旨在最大限度地减少注意力权重损失，同时有选择地修剪最不重要的通道。与普通的 KV 缓存驱逐方法相比，我们的方法不仅可以保持或提高模型准确性，而且还可以实现内存成本降低 20% 以上。在各种长序列数据集上对 LLaMA3 和 Mistral 模型进行的广泛评估证实了 ThinK 的有效性，为高效部署 LLM 树立了新先例，同时又不影响性能。我们还概述了将我们的方法扩展到价值缓存修剪的潜力，展示了 ThinK 在减少内存和计算开销方面的多功能性和广泛适用性。